PlanOut

A framework for online field experiments

Focus on parameters

PlanOut provides randomized parameter values to Internet services. Instead of using constants for user interface elements or switches controlling the rollout of a new feature or ranking model, just use PlanOut to determine the value of these parameters. Now you have an experiment.

Simple or complex

It is easy to implement simple A/B tests in PlanOut. But it's also not much harder to implement factorial or within-subjects designs, or more complex designs involving multiple types of units (e.g., users and content), and conditional logic. Easily run experiments that help you understand your product and answer important questions.

Extensible

PlanOut was developed at Facebook for running experiments involving hundreds of millions of people. This open-source release mirrors parts of Facebook's infrastructure for implementing, logging and deploying online experiments. PlanOut is written in a way that can be easily integrated with your own stack.

PlanOut is a framework for online field experiments. It was created to make it easy to run and iterate on sophisticated experiments in a statistically sound manner while satisfying the constraints of deployed Internet services.

Developers implement experiments using PlanOut by providing a high-level specification for how units (e.g., users, cookie IDs) should be mapped to parameters that control the user experience. For example, to create a 2x2 factorial experiment that randomizes both the color and the text on a button, using the Python-based framework, one can simply create a class like this:

class MyExperiment(SimpleExperiment):
  def assign(self, params, userid):
    params.button_color = UniformChoice(choices=['#ff0000', '#00ff00'],
      unit=userid)
    params.button_text = UniformChoice(choices=['I voted', 'I am a voter'],
      unit=userid)

Then, in the application code, instead of just using a constant (e.g., '#ff00000'), you query an instance of the class to find out what values should be used for the current user:

my_exp = MyExperiment(userid=42)
color = my_exp.get('button_color')
text = my_exp.get('button_text')

PlanOut takes care of correctly randomizing each userid to parameter values. It does so by determnisitcally hashing the input, so each userid will always map onto the same parameter values for that experiment. As soon as you access any of the parameters, core parts of the data you need to analyze your experiment are automatically logged.

The PlanOut framework includes:

  • Extensible classes for implementing and testing experiments, which automatically log important data.

  • A system for managing and deploying multiple mutually exclusive experiments, called namespaces

  • The PlanOut language, which lets you define, serialize, store, and execute experiment definitions in a platform-independent way.

Who is PlanOut for?

PlanOut is for researchers, businesses, and students wanting to run experiments. It's designed to be easy to get up and running with for first-time experimenters, and extensible for those wanting to use it in a large production environments. This open source implementation shares many of the key design decisions of Facebook's experimentation stack, which is used to conduct experiments with hundreds of millions of people.

Learn more

Continue on to the documentation page or try out the interactive PlanOut editor. For an in-depth look at how we use PlanOut at Facebook, see our paper, Designing and Deploying Online Field Experiments.

If you are publishing research using PlanOut, please consider referencing our work,

@inproceedings{bakshy2014www,
    Author = {Bakshy, E. and Eckles, D. and Bernstein, M. S.},
    Booktitle = {Proceedings of the 23rd ACM conference on the World Wide Web},
    Organization = {ACM},
    Title = {Designing and Deploying Online Field Experiments},
    Year = {2014}
}