Getting started with the interpreter

The PlanOut language has a syntax that is similar to JavaScript and includes a limited number of functions or “operators” for defining experiments, including the random assignment operators introduced in other parts of the guide.

Consider the script, which is saved to mydemo.planout,

button_text = uniformChoice(choices=['Purchase', 'Buy'], unit=userid);
has_discount = bernoulliTrial(p=0.3, unit=userid);

This code can then be compiled either using the PlanOut editor or the node.js-based command-line tool.

Using the command-line tool, one would type something like:

node planout.js mydemo.planout > mydemo.json

The SimpleInterpretedExperiment class

The SimpleInterpretedExperiment object behaves like SimpleExperiment, except rather than defining an execute() method, one defines a loadScript() method that loads a serialized experiment definition:

from planout.experiment import SimpleInterpretedExperiment
import json

class Exp1(SimpleInterpretedExperiment):
    def loadScript(self):
      self.script = json.loads(open("mydemo.json").read())

Let’s try out the experiment:

for i in xrange(5):
    e = Exp1(userid=i)
    print i, e.get_params()

This outputs:

0 {u'button_text': u'Purchase', u'has_discount': 0}
1 {u'button_text': u'Buy', u'has_discount': 0}
2 {u'button_text': u'Purchase', u'has_discount': 0}
3 {u'button_text': u'Buy', u'has_discount': 1}
4 {u'button_text': u'Purchase', u'has_discount': 0}

Centralizing experiment definitions

Suppose we had a centralized store or database of experiments and their associated, JSON-encoded PlanOut scripts. Equipped with a getDefinition(name) function that retrieves these definitions for any particular experiment name, we can write a function to generate full-fledged experiment objects from the data. Here, we use the name and script properties of SimpleInterpretedExperiment to do this:

def genExperiment(name, **inputs):
  e = SimpleInterpretedExperiment(**inputs)
  e.name = name
  e.script = getDefinition(name)
  return e

Then, to retrieve the params associated with a user for an experiment called 'feedback_experiment':

x = genExperiment('feedback_experiment', userid=42).get_params()

Even though we use the same class for multiple experiments, any two experiments will have orthogonal random assignments because PlanOut salts its hashing functions with the experiment name. Furthermore, because logs are associated with experiment names, each experiment will have its own log file. This concept is the basis for namepsaces, which are used to manage multiple experiments.

Algorithmically generating new experiments

Serialized PlanOut code can also be generated via high-level functions or user interfaces. Let’s consider a hypothetical graphical user interface for constructing full factorial experiments. We assume that developers have a web-based frontend and make an AJAX request to the server, including dictionary-like objects whose keys are factors (parameters) and values are lists of possible values.

We would then use a function like this to generate PlanOut code for the given input data:

def gen_full_factorial(config, unit='userid'):
  def gen_factor(var, choices):
    return {
      "op": "set",
      "var": var,
      "value": {
        "op": "uniformChoice",
        "choices": choices,
        "unit": {"op": "get", "var": unit}
      }
    }
    items = [gen_factor(k, v) for k,v in config.iteritems()]
    return {"op":"seq", "seq": items}

So that, e.g.,

my_config = {'button_text':['Post', 'Share', 'Update'], 'button_color':['#00ff00', '#55ee44']}
gen_full_factorial(my_config)

generates

{
  "op": "seq",
  "seq": [
    {
      "op": "set",
      "value": {
        "choices": ["#00ff00", "#55ee44"],
        "op": "uniformChoice",
        "unit": {"op": "get", "var": "userid"}
      },
      "var": "button_color"
    },
    {
      "op": "set",
      "value": {
        "choices": ["Post", "Share", "Update"],
        "op": "uniformChoice",
        "unit": {"op": "get", "var": "userid"}
      },
      "var": "button_text"
    }
  ]
}

Using this function, we can then generate experiments on the fly from any factorial configuration file:

def gen_experiment(experiment_name, config, **inputs):
  e = SimpleInterpretedExperiment(**inputs)
  e.name = experiment_name
  e.script = gen_full_factorial(config)
  return e

Now let’s perform some random assignments. You can also see the log file, custom_experiment.log.

for i in xrange(5):
  print i, gen_experiment('first_algo_experiment', my_config, userid=i).get_params()

outputs:

0 {'button_color': '#55ee44', 'button_text': 'Post'}
1 {'button_color': '#00ff00', 'button_text': 'Share'}
2 {'button_color': '#00ff00', 'button_text': 'Update'}
3 {'button_color': '#00ff00', 'button_text': 'Post'}
4 {'button_color': '#55ee44', 'button_text': 'Share'}

For more on the mechanics of working with the lower-level interpreter class, see the PlanOut tutorial IPython notebook from PyData ‘14.