Run Agent Benchmark Tests

Overview

Running Agent benchmark tests consists of building the benchmark binaries and then running them to observe the output.

If you run into issues, try checking our troubleshooting guide.

After each section, check the Overall Outcomes below to ensure everything is completed.

Overall Outcomes

Setup

• an FBOSS forwarding stack Docker container to perform builds in

Build and Test Agent Benchmarks

• benchmark binaries can be built
• benchmark binaries can be run

Setup

Step 1: Clone the FBOSS Repository

git clone https://github.com/facebook/fboss.git
cd fboss

Step 2: Stop Existing Containers and Clean Docker Artifacts

sudo docker container kill -a && sudo docker container prune -f
sudo docker image prune -af

Step 3: Build the FBOSS Docker Image

# Builds a docker container image that is tagged as fboss_image:latest
sudo docker build . -t fboss_image -f fboss/oss/docker/Dockerfile

Step 4: Start a Forwarding Stack Container

# Use stable commits
rm -rf build/deps/github_hashes/
tar xvzf fboss/oss/stable_commits/latest_stable_hashes.tar.gz

# If you know your host has enough space for the build, just make the SDK
# artifacts available by mounting them with the -v flag.
#
# E.g. you have a directory /path_to_sdk/ containing lib/ and include/
# /path_to_sdk/
# ├── include
# │   ├── file1.h
# │   └── file2.h
# └── lib
#     └── libsai_impl.a
#
# this will mount lib/ and include/ to the container like so:
# /path_to_sdk/lib/     -> /opt/sdk/lib/
# /path_to_sdk/include/ -> /opt/sdk/include/
sudo docker run -d \
    -it --name=FBOSS_BUILD_CONTAINER \
    -v $PWD:/var/FBOSS/fboss:z \
    -v /path_to_sdk:/opt/sdk:z \
    fboss_image:latest bash

# A full FBOSS build may take significant space (>50GB of storage). You
# can mount a volume with more storage for building by using the -v flag
sudo docker run -d \
    -it --name=FBOSS_BUILD_CONTAINER \
    -v $PWD:/var/FBOSS/fboss:z \
    -v /path_to_sdk:/opt/sdk:z \
    -v /opt/app/localbuild:/var/FBOSS/tmp_bld_dir:z \
    fboss_image:latest bash

At this point, you should have a Docker container which can be used to build the required artifacts in later steps.

Build and Test Agent Benchmarks

Step 1: Build Agent Benchmarks

Use the forwarding stack Docker container from the Setup step.

Step 1.1: Enter the Container

# Attaches our current terminal to a new bash shell in the docker container so
# that we can perform the build within it
sudo docker exec -it FBOSS_BUILD_CONTAINER bash

Step 1.2: Build Agent Benchmark Binaries

The run-getdeps.py script accepts flags to configure the SAI implementation and SDK version. Run ./fboss/oss/scripts/run-getdeps.py -h to see all options and Meta supported values. Note the --benchmark-install flag which is required to install benchmark binaries.

#!/bin/bash
# Navigate to the right directory
cd /var/FBOSS/fboss || exit

# Start the build with --benchmark-install to include benchmark binaries
# NOTE: Choose the appropriate --npu-sai-impl and --npu-sai-sdk-version values
# for your platform. Run ./fboss/oss/scripts/run-getdeps.py -h to see
# Meta officially supported values.
time ./fboss/oss/scripts/run-getdeps.py \
  --npu-sai-impl SAI_BRCM_IMPL \
  --npu-sai-sdk-version SAI_VERSION_14_0_EA_ODP \
  --npu-sai-version 1.16.1 \
  --npu-libsai-impl-path /opt/sdk \
  --npu-experiments-path /opt/sdk/experimental \
  --benchmark-install \
  build \
  --allow-system-packages \
  --build-type MinSizeRel \
  --extra-cmake-defines='{"CMAKE_CXX_STANDARD": "20", "RANGE_V3_TESTS": "OFF", "RANGE_V3_PERF": "OFF"}' \
  --scratch-path /var/FBOSS/tmp_bld_dir \
  fboss

Step 2: Send Build Artifacts to the Switch

Step 2.1: Package the Benchmark Binaries

# On the container used for building FBOSS:

# Clean any existing packages
rm -rf /var/FBOSS/tmp_bld_dir/fboss_bins*

# Navigate to the FBOSS repository
cd /var/FBOSS/fboss

# Creates a package directory with prefix /var/FBOSS/tmp_bld_dir/fboss_bins
./fboss/oss/scripts/package-fboss.py --copy-root-libs --scratch-path /var/FBOSS/tmp_bld_dir/

# or

# Creates a tarball called "fboss_bins.tar.zst" under /var/FBOSS/tmp_bld_dir/
./fboss/oss/scripts/package-fboss.py --copy-root-libs --scratch-path /var/FBOSS/tmp_bld_dir/ --compress

Step 2.2: Copy and Set Up the Package

# From the host that the container is running on, use the appropriate command to
# copy the FBOSS package directory or tarball to the switch:

# If copying the directory
scp -r /opt/app/FBOSS_DIR/tmp_bld_dir/fboss_bins-$PKG_ID root@$SWITCHNAME:/opt/

# If copying the tarball
scp /opt/app/FBOSS_DIR/tmp_bld_dir/fboss_bins.tar.zst root@$SWITCHNAME:/opt/



# On the switch, use the appropriate command to set up the package:

# If using the directory
cd /opt
ln -s /opt/fboss_bins-$PKG_ID /opt/fboss

# If using the tarball
cd /opt
mkdir fboss && mv fboss_bins.tar.zst fboss/
cd fboss && tar -xvf fboss_bins.tar.zst



# You will have a directory /opt/fboss/ which contains a bin/, lib/, and share/
# directory

Step 3: Run Benchmark Tests

All benchmarks are consolidated into two binaries: sai_all_benchmarks-sai_impl (mono, the default) and sai_multi_switch_all_benchmarks-sai_impl (multi-switch). Individual benchmarks are selected at runtime via --bm_regex, with each test running in its own process for full setup/run/teardown isolation.

Option 1: Using the run_test.py Script (Recommended)

The run_test.py script discovers all benchmarks from the binary via --bm_list, pre-filters known-bad tests, then runs each test individually with full setup/run/teardown isolation. Pass --agent-run-mode multi_switch to use the multi-switch binary; mono is the default.

cd /opt/fboss
source ./bin/setup_fboss_env

# Run all benchmarks (mono mode is the default)
./bin/run_test.py benchmark \
--config /opt/fboss/share/hw_test_configs/montblanc.agent.materialized_JSON \
--skip-known-bad-tests brcm/13.3.0.0_odp/tomahawk5 \
--test-run-timeout 1800

# Run all benchmarks against the multi-switch binary
./bin/run_test.py benchmark --agent-run-mode multi_switch \
--config /opt/fboss/share/hw_test_configs/montblanc.agent.materialized_JSON \
--skip-known-bad-tests brcm/13.3.0.0_odp/tomahawk5 \
--test-run-timeout 1800

# Run only benchmarks listed in a user-provided file (one test name per line)
./bin/run_test.py benchmark \
--config /opt/fboss/share/hw_test_configs/montblanc.agent.materialized_JSON \
--skip-known-bad-tests brcm/13.3.0.0_odp/tomahawk5 \
--filter_file /path/to/my_benchmarks.conf

# Run a single benchmark by name (regex match)
./bin/run_test.py benchmark \
--config /opt/fboss/share/hw_test_configs/montblanc.agent.materialized_JSON \
--skip-known-bad-tests brcm/13.3.0.0_odp/tomahawk5 \
--filter HwEcmpGroupShrink

# Run all route scale benchmarks
./bin/run_test.py benchmark \
--config /opt/fboss/share/hw_test_configs/montblanc.agent.materialized_JSON \
--skip-known-bad-tests brcm/13.3.0.0_odp/tomahawk5 \
--filter ".*Route.*Benchmark"

# List available benchmarks without running
./bin/run_test.py benchmark \
--config /opt/fboss/share/hw_test_configs/montblanc.agent.materialized_JSON \
--list_tests

Benchmark-specific flags:

Flag	Description
--agent-run-mode	mono (default) or multi_switch. Multi-switch passes --multi_switch --hw_agent_for_testing to the binary and manages the hw_agent service lifecycle automatically.
--num-npus	Number of NPUs for multi-switch mode (1 or 2, default 1). When 2, adds --multi_npu_platform_mapping.
--filter	Regex matched against discovered benchmark names to narrow which tests run.
--filter_file	File containing BENCHMARK() registered names to run (one per line).
--skip-known-bad-tests	Platform key (e.g. brcm/13.3.0.0_odp/tomahawk5) used both to pre-filter known-bad tests before running and to look up per-platform thresholds for result validation. Without it, all tests run and threshold validation is skipped (NO_THRESHOLD).
--test-run-timeout	Timeout for each test run in seconds. Default is 1200.

When a benchmark has no threshold configured for the platform, the runner prints a WILL ALWAYS PASS warning so missing entries are visible.

Results are written to a timestamped CSV file (e.g., benchmark_results_20260119_143022.csv) with these columns:

benchmark_binary_name, benchmark_test_name, benchmark_time_ps,
test_status, cpu_time_usec, max_rss, cpu_rx_pps, cpu_tx_pps,
threshold_status, threshold_details

Option 2: Running the Binary Directly

cd /opt/fboss
source ./bin/setup_fboss_env

# List all available benchmarks (no hardware init)
./bin/sai_all_benchmarks-sai_impl --bm_list

# Same for the multi-switch binary
./bin/sai_multi_switch_all_benchmarks-sai_impl --bm_list

# Run a single benchmark (mono)
./bin/sai_all_benchmarks-sai_impl --json \
--bm_regex "^HwEcmpGroupShrink$" \
--config ./share/hw_test_configs/<platform>.agent.materialized_JSON [--logging <level>]

# Run a single benchmark (multi-switch)
./bin/sai_multi_switch_all_benchmarks-sai_impl --json \
--multi_switch --hw_agent_for_testing \
--bm_regex "^HwEcmpGroupShrink$" \
--config ./share/hw_test_configs/<platform>.agent.materialized_JSON [--logging <level>]