How PlanOut works

PlanOut works by hashing input data into numbers, and using these numbers to generate pseudo-random numbers to pick values of parameters. All PlanOut operators include basic unit tests to verify that they generate assignments with the expected distribution.

Good randomization procedures produce assignments that are independent of one another. Below, we show how PlanOut uses experiment-level and parameter-level “salts” (strings that get appended to the data being hashed) to make sure that random assignment among variables within and across experiments remain independent.

Pseudo-random assignment through hashing

Consider the following experiment that manipulates a button label:

class SharingExperiment(SimpleExperiment):
  def setup(self):
    self.name = 'sharing_name'
    self.salt = 'sharing_salt'

  def assign(self, params, userid):
    params.button_text = UniformChoice(
      choices=['OK', 'Share', 'Share with friends'],
      unit=userid
    )

All experiments either implicitly or explicitly define an experiment name, which is used to identify the experiment in log data, and an experiment salt, which gets used in the hashing procedure. Developers can explicitly define these variables via the setup() method. If these values are not defined by the programmer, the name of the experiment will be set to the class name, and the default value for the salt will be the name of the experiment. You can always see both values in the log data.

In the assign() method, we set a single randomized parameter, button_text. The assignment is generated by first composing a string containing experiment-level salt, sharing_salt, the parameter-level salt button_text, and the input unit. By default, PlanOut uses the variable name as the parameter-level salt.

When we choose the button text for a particular unit, e.g.,

SharingExperiment(userid=4).get('button_text')

PlanOut would compute the SHA1 checksum for:

sharing_salt.button_text.4

and then use the last few digits of this checksum to index into the given list of choices. Since SHA1 is cryptographically safe, even minor changes to the hashing string (e.g., considering userid=41 instead of 4) will result in a totally different number.

Multiple units are handled through concatenation. Had the unit parameter been unit=[userid, url],

SharingExperiment(userid=4, url='http://www.facebook.com').get('button_text')

PlanOut would perform the hashing based on the SHA1 checksum for:

sharing_salt.button_text.4.http://www.facebook.com

Note that because PlanOut simply concatenates the units, the order in which you specify lists of units matters.

Salts

Salts are strings that get appended to input data before they are hashed. There are three ways that salts can enter into the assignment of units to treatments.

Experiment-level salts

Experiment-level salts can be manually in the setup() method (as we have above). If the salt is not specified, then the experiment name is used as the salt. With SimpleExperiment, if the name is not set in setup(), then the name of the class is used as the experiment name.

Parameter-level salts

The parameter name is automatically used to salt random assignment operations, but parameter level salts can be specified manually. For example, in the following code

params.x = UniformChoice(choices=['a','b'], unit=userid)
params.y = UniformChoice(choices=['a','b'], unit=userid, salt='x')

both x and y will always be assigned to the same exact same value.

This lets you change the name of the variable you are logging without changing the assignment. Use parameter-level salts with caution, since they might lead to failures in randomization.

Salts with namespaces

Namespaces are a way to manage concurrent and iterative experiments. When using SimpleNamespace, the namespace-level salt is appended to the experiment-level salt. This ensures that random assignment is independent across experiments with the same name running under different namespaces.