The PlanOut language

While much of this guide is focused on the nuts-and-bolts of the native Python-based API, PlanOut is first and formost intended to be a parsimonious, high-level language for thinking about and specifying complex experiments in a way that is useful to researchers as well as engineers building production systems.

The PlanOut language as a way of serializing experiments

PlanOut language code gets compiled into a JSON representation which can be stored and retrieved at runtime. This aspect is key to scaling up an organization’s experimentation infrastructure.

For example, a 2x2 factorial design with unequal probability weights can be expressed in a two line script written in the PlanOut domain-specific language:

button_text = uniformChoice(choices=['Purchase', 'Buy'], unit=userid);
has_discount = bernoulliTrial(p=0.3, unit=userid);

The above script is simple enough for non-programmers to understand, and precise enough to be considered a fully executable program. Using the PlanOut compiler, this code can be translated into a serialized, JSON-encoded data structure:

{
  "op": "seq",
  "seq": [
    {
      "op": "set",
      "var": "button_text",
      "value": {
        "choices": {
          "op": "array",
          "values": [
            "Purchase",
            "Buy"
          ]
        },
        "unit": {
          "op": "get",
          "var": "userid"
        },
        "op": "uniformChoice"
      }
    },
    {
      "op": "set",
      "var": "has_discount",
      "value": {
        "p": 0.3,
        "unit": {
          "op": "get",
          "var": "userid"
          },
          "op": "bernoulliTrial"
        }
      }
      ]
    }

This format can then be executed by a PlanOut interpreter which can be written in any language. The result is that no matter what platform you use, the above code will always map the same subjects to the same treatments.

Scaling up experimentation using serialized experiment definitions

The PlanOut serialization layer is particularly well suited for organizations that would like to scale up their experimentation infrastructure. Having a JSON-based representation gives developers the ability to centrally manage and store experiment definitions in a datastore or database. These experimental scripts may be written in the PlanOut language and compiled to JSON, or generated via graphical user interfaces. Once these experiments are defined and named, they can easily be managed using namespaces. This allows Web services to run parallel non-overlapping experiments and follow-on experiments in a statistically sound fashion. All of the functionality provided by the native Python API described in this guide is fully compatible with the interpreter.