Work on storage serving live traffic
Your tasks may consume the same data resources (disk IOPs, CPU, …) as your live traffic. Bistro only starts tasks if the relevant resource slots are available on the data hosts, rack switches, or any other physical bottleneck, which might otherwise impact live traffic. Bistro does not dictate how you store your data. At Facebook, we have used it with:
Divide work in space or time
Map your work over nodes — Bistro’s way of modeling multi-part computations, e.g.
- prioritized workqueue: 1 node per job
- map-only jobs: nodes are map shards
- sharded, replicated database: a DAG of nodes — the DB hosts are sinks (for tracking resources), and DB instances are children of their hosts
- periodic work (like
cron): a data node has a child node for each time period
If needed, MapReduce computations can be modeled as a sequence of 3 dependent jobs: map, shuffle, reduce.
Flexible, extensible, scriptable
- Concise JSON configuration
- Command-line tooling
- Tunable, scalable web UI
- REST-style scheduler API
- Customizable via small plugins
- Powerful job model
- host placement
- node filters
- Many deployment styles
- one scheduler, many workers
- sharded share-nothing workers, with results aggregated by the UI
Why Use Bistro?
You should consider using Bistro instead of rolling your own solution, because in our experience at Facebook, every heavily used task-execution service eventually has to build out a very similar set of layers:
- Execution shell: Start, wait for, kill tasks
- Worker pool: Maintain the distributed state of what tasks are running where
- Data model: Currently available tasks; resource accounting
- Configuration: Add/remove jobs; tune parameters
- Persistence: If the service crashes, don’t forget task completions
- Scheduling logic: Priorities; dependencies; resource constraints; host placement
- Task logs: Make logs accessible, while controlling disk usage
- Command-line tools: Scriptable, yet friendly
- Web UI: Minimal learning curve; great for visualizing data
- RPC API: When command-line scripting is too slow or clunky
Bistro provides all of the above, and improves continuously, so instead of re-inventing, you can focus on innovating.
Although Bistro has been in production at Facebook for over 3 years, the present public release is partial, including just the server components. The CLI tools and web UI will be shipping shortly.
The good news is that the parts that are released are continuously verified by build & unit tests on Travis CI, so they should work out of the box.
There is already a lot of code, with which to play, so you have any thoughts — write us a note! Open-source projects thrive and grow on encouragement and feedback.
As flexible as a library
Although Bistro comes as a ready-to-deploy service, it may be helpful to think of it as a library. In our experience at Facebook, many teams’ distributed computing needs are highly individual, requiring more customization over time.
Luckily, Bistro can adapt to your needs in three different ways.
1. Configuration: most aspects of a Bistro deployment can be tuned:
- Configuration: continuously re-read a file, or a more custom solution
- Persistence: use no database, SQLite, or something fancier
- Code execution: run alongside the scheduler, or on remote workers
- Data shards: manually configured, periodic, or script-generated
2. Plugins: Bistro’s architecture is highly pluggable, so adding your own code is a cinch:
- Can’t find the right scheduling policy? Write your own in about 50 lines of C++.
- Need a different DB? Shard source? Configuration source?
Every plugin you contribute back to the community makes the next person’s customization easier.
Getting help, and helping out
Questions and bug reports
Tips for questions and bug reports:
- Describe what you want to do, and what is going wrong.
- Include your OS, the git hash of your build, the commands you ran, the scheduler & worker logs.
We gladly welcome contributions to both code and documentation — whether you refactor a whole module, or fix one misspelling. Please send pull requests for code against the master branch and for the website or documentation against the gh-pages branch.
Case study: data-parallel job with data and worker resources
While Bistro can be a capable traditional task queue, one of its more unique features is the ability to account not just for the resources of the machines executing the tasks, but also for any bottlenecks imposed by the data being queried.
Here is a simple example:
- Your data resides on 500 database hosts serving live traffic.
- Each database host has 10-20 logical databases.
- The hosts have enough performance headroom to run at most two batch tasks per host, without impacting production traffic.
- You have some batch jobs, which must process all the databases.
- Bistro will use 1 scheduler, and a worker pool of 100 machines.
To configure Bistro in such a setup, first describe the structure of the computation in terms of nodes at several levels:
hostnodes — to limit tasks per DB host.
databasenodes, each connected to its DB host node. A running task locks its logical DB.
- If remote worker hosts are in use, a node per
workerhost is automatically created.
- If global concurrency constraints are desired, a top-level scheduler
instancenode is also available.
In a typical setup, each running task associates with one node on each level:
database: The logical DB is the work shard being processed.
host: What host to query for data?
worker: Where is the process running?
Before running jobs, define resources for the nodes at various levels:
hostnodes should be protected from excessive queries to avoid slowing down live production traffic — allocate 2
host_concurrencyslots to honor the requirement above.
workerresources might be, e.g.
ram_giagabytes, etc. Bistro supports non-uniform resource availability among its worker hosts.
Hosts fail, and therefore our
will change continuously. To deal with this, Bistro continuously polls for
node updates — it can periodically run your “node fetcher” script, or read
a file, or poll a custom in-process plugin. One of the standard plugins
creates and removes nodes corresponding to time intervals, akin to Cron.
When nodes change, tasks are started on the new nodes, and stopped on the
deleted ones. Bistro’s node model also naturally accommodates database
Lastly, specify the jobs to run — a path to a binary will suffice, and the
binary will be run against every
database node found above. Many
other job options and metadata are available. You can even implement your
tasks as custom RPC requests.
Ready to go
The above configuration is continuously polled from a file (or other configuration source) by the Bistro scheduler. Bistro’s worker processes need only minimal configuration — tell them how to connect to the scheduler, and you are in business (they do have plenty of advanced reliability-related settings).
You can then interact with the scheduler (and tasks) via HTTP REST, via a command-line tool, or via a web UI.
Read our USENIX ATC’15 paper for a more principled explanation of this resource model. Note that the model Bistro implements in practice is an extension of the “resource forest model” that is the focus of the paper. Bistro’s data model is being refined and improved, so in the future you can expect it to handle even more applications with ease.