How PlanOut works
PlanOut works by hashing input data into numbers, and using these numbers to generate pseudo-random numbers to pick values of parameters. All PlanOut operators include basic unit tests to verify that they generate assignments with the expected distribution.
Good randomization procedures produce assignments that are independent of one another. Below, we show how PlanOut uses experiment-level and parameter-level “salts” (strings that get appended to the data being hashed) to make sure that random assignment among variables within and across experiments remain independent.
Pseudo-random assignment through hashing
Consider the following experiment that manipulates a button label:
class SharingExperiment(SimpleExperiment): def setup(self): self.name = 'sharing_name' self.salt = 'sharing_salt' def assign(self, params, userid): params.button_text = UniformChoice( choices=['OK', 'Share', 'Share with friends'], unit=userid )
All experiments either implicitly or explicitly define an experiment name,
which is used to identify the experiment in log data, and an experiment salt,
which gets used in the hashing procedure. Developers can explicitly define
these variables via the
setup() method. If these values are not defined by
the programmer, the name of the experiment will be set to the class name,
and the default value for the salt will be the name of the experiment.
You can always see both values in the log data.
assign() method, we set a single randomized parameter,
The assignment is generated by first composing a string containing
sharing_salt, the parameter-level salt
and the input unit. By default, PlanOut uses the variable name as the
When we choose the button text for a particular unit, e.g.,
PlanOut would compute the SHA1 checksum for:
and then use the last few digits of this checksum to index into the given list of
choices. Since SHA1 is cryptographically safe, even minor changes to the hashing string (e.g., considering
userid=41 instead of 4) will result in a totally different number.
Multiple units are handled through concatenation. Had the
PlanOut would perform the hashing based on the SHA1 checksum for:
Note that because PlanOut simply concatenates the units, the order in which you specify lists of units matters.
Salts are strings that get appended to input data before they are hashed. There are three ways that salts can enter into the assignment of units to treatments.
Experiment-level salts can be manually in the
(as we have above). If the salt is not specified, then the experiment name is
used as the salt. With SimpleExperiment, if the name is not set in
setup(), then the name of the class is used as the experiment name.
The parameter name is automatically used to salt random assignment operations, but parameter level salts can be specified manually. For example, in the following code
params.x = UniformChoice(choices=['a','b'], unit=userid) params.y = UniformChoice(choices=['a','b'], unit=userid, salt='x')
y will always be assigned to the same exact same value.
This lets you change the name of the variable you are logging without changing the assignment. Use parameter-level salts with caution, since they might lead to failures in randomization.
Salts with namespaces
Namespaces are a way to manage concurrent and iterative experiments. When using
SimpleNamespace, the namespace-level salt is appended to the experiment-level salt. This ensures that random assignment is independent across experiments with the same name running under different namespaces.