Skip to main content

Run Agent Benchmark Tests

Overview​

Running Agent benchmark tests consists of building the benchmark binaries and then running them to observe the output.

If you run into issues, try checking our troubleshooting guide.

After each section, check the Overall Outcomes below to ensure everything is completed.

Overall Outcomes​

Setup​

  • an FBOSS forwarding stack Docker container to perform builds in

Build and Test Agent Benchmarks​

  • benchmark binaries can be built
  • benchmark binaries can be run

Setup​

Step 1: Clone the FBOSS Repository​

git clone https://github.com/facebook/fboss.git
cd fboss

Step 2: Stop Existing Containers and Clean Docker Artifacts​

sudo docker container kill -a && sudo docker container prune -f
sudo docker image prune -af

Step 3: Build the FBOSS Docker Image​

# Builds a docker container image that is tagged as fboss_image:latest
sudo docker build . -t fboss_image -f fboss/oss/docker/Dockerfile

Step 4: Start a Forwarding Stack Container​

# Use stable commits
rm -rf build/deps/github_hashes/
tar xvzf fboss/oss/stable_commits/latest_stable_hashes.tar.gz

# If you know your host has enough space for the build, just make the SDK
# artifacts available by mounting them with the -v flag.
#
# E.g. you have a directory /path_to_sdk/ containing lib/ and include/
# /path_to_sdk/
# ├── include
# │   ├── file1.h
# │   └── file2.h
# └── lib
#     └── libsai_impl.a
#
# this will mount lib/ and include/ to the container like so:
# /path_to_sdk/lib/     -> /opt/sdk/lib/
# /path_to_sdk/include/ -> /opt/sdk/include/
sudo docker run -d \
    -it --name=FBOSS_BUILD_CONTAINER \
    -v $PWD:/var/FBOSS/fboss:z \
    -v /path_to_sdk:/opt/sdk:z \
    fboss_image:latest bash

# A full FBOSS build may take significant space (>50GB of storage). You
# can mount a volume with more storage for building by using the -v flag
sudo docker run -d \
    -it --name=FBOSS_BUILD_CONTAINER \
    -v $PWD:/var/FBOSS/fboss:z \
    -v /path_to_sdk:/opt/sdk:z \
    -v /opt/app/localbuild:/var/FBOSS/tmp_bld_dir:z \
    fboss_image:latest bash

At this point, you should have a Docker container which can be used to build the required artifacts in later steps.

Build and Test Agent Benchmarks​

Step 1: Build Agent Benchmarks​

Use the forwarding stack Docker container from the Setup step.

Step 1.1: Enter the Container​

# Attaches our current terminal to a new bash shell in the docker container so
# that we can perform the build within it
sudo docker exec -it FBOSS_BUILD_CONTAINER bash

Step 1.2: Build Agent Benchmark Binaries​

The run-getdeps.py script accepts flags to configure the SAI implementation and SDK version. Run ./fboss/oss/scripts/run-getdeps.py -h to see all options and Meta supported values. Note the --benchmark-install flag which is required to install benchmark binaries.

#!/bin/bash
# Navigate to the right directory
cd /var/FBOSS/fboss || exit

# Start the build with --benchmark-install to include benchmark binaries
# NOTE: Choose the appropriate --npu-sai-impl and --npu-sai-sdk-version values
# for your platform. Run ./fboss/oss/scripts/run-getdeps.py -h to see
# Meta officially supported values.
time ./fboss/oss/scripts/run-getdeps.py \
  --npu-sai-impl SAI_BRCM_IMPL \
  --npu-sai-sdk-version SAI_VERSION_14_0_EA_ODP \
  --npu-sai-version 1.16.1 \
  --npu-libsai-impl-path /opt/sdk/libsai_impl.a \
  --npu-experiments-path /opt/sdk/experimental \
  --benchmark-install \
  build \
  --allow-system-packages \
  --build-type MinSizeRel \
  --extra-cmake-defines='{"CMAKE_CXX_STANDARD": "20", "RANGE_V3_TESTS": "OFF", "RANGE_V3_PERF": "OFF"}' \
  --scratch-path /var/FBOSS/tmp_bld_dir \
  fboss

Step 2: Send Build Artifacts to the Switch​

Step 2.1: Package the Benchmark Binaries​

# On the container used for building FBOSS:

# Clean any existing packages
rm -rf /var/FBOSS/tmp_bld_dir/fboss_bins*

# Navigate to the FBOSS repository
cd /var/FBOSS/fboss

# Creates a package directory with prefix /var/FBOSS/tmp_bld_dir/fboss_bins
./fboss/oss/scripts/package-fboss.py --copy-root-libs --scratch-path /var/FBOSS/tmp_bld_dir/

# or

# Creates a tarball called "fboss_bins.tar.zst" under /var/FBOSS/tmp_bld_dir/
./fboss/oss/scripts/package-fboss.py --copy-root-libs --scratch-path /var/FBOSS/tmp_bld_dir/ --compress

Step 2.2: Copy and Set Up the Package​

# From the host that the container is running on, use the appropriate command to
# copy the FBOSS package directory or tarball to the switch:

# If copying the directory
scp -r /opt/app/FBOSS_DIR/tmp_bld_dir/fboss_bins-$PKG_ID root@$SWITCHNAME:/opt/

# If copying the tarball
scp /opt/app/FBOSS_DIR/tmp_bld_dir/fboss_bins.tar.zst root@$SWITCHNAME:/opt/



# On the switch, use the appropriate command to set up the package:

# If using the directory
cd /opt
ln -s /opt/fboss_bins-$PKG_ID /opt/fboss

# If using the tarball
cd /opt
mkdir fboss && mv fboss_bins.tar.zst fboss/
cd fboss && tar -xvf fboss_bins.tar.zst



# You will have a directory /opt/fboss/ which contains a bin/, lib/, and share/
# directory

Step 3: Run Benchmark Binaries​

The run_test.py script provides automated execution of benchmark suites with CSV output:

cd /opt/fboss
source ./bin/setup_fboss_env

# Run all benchmarks (T1 + T2 + additional)
./bin/run_test.py benchmark

# Run T1 benchmark suite
./bin/run_test.py benchmark \
--filter_file ./share/hw_benchmark_tests/t1_benchmarks.conf

# Run T2 benchmark suite
./bin/run_test.py benchmark \
--filter_file ./share/hw_benchmark_tests/t2_benchmarks.conf

# Run only additional benchmarks (not in T1 or T2)
./bin/run_test.py benchmark \
--filter_file ./share/hw_benchmark_tests/additional_benchmarks.conf

Results are written to a timestamped CSV file (e.g., benchmark_results_20260119_143022.csv) with detailed metrics.

Option 2: Running Individual Binaries​

Run individual benchmark binaries directly from the bin directory:

cd /opt/fboss
source ./bin/setup_fboss_env

./bin/sai_fsw_scale_route_add_speed-sai_impl
./bin/sai_hgrid_du_scale_route_add_speed-sai_impl
./bin/sai_hgrid_uu_scale_route_del_speed-sai_impl
./bin/sai_th_alpm_scale_route_add_speed-sai_impl
./bin/sai_fsw_scale_route_del_speed-sai_impl
./bin/sai_ecmp_shrink_with_competing_route_updates_speed-sai_impl
./bin/sai_th_alpm_scale_route_del_speed-sai_impl
./bin/sai_ecmp_shrink_speed-sai_impl
./bin/sai_hgrid_uu_scale_route_add_speed-sai_impl
./bin/sai_hgrid_du_scale_route_del_speed-sai_impl
./bin/sai_stats_collection_speed-sai_impl
./bin/sai_tx_slow_path_rate-sai_impl
./bin/sai_rx_slow_path_rate-sai_impl
./bin/sai_init_and_exit_40Gx10G-sai_impl
./bin/sai_init_and_exit_100Gx10G-sai_impl
./bin/sai_init_and_exit_100Gx25G-sai_impl
./bin/sai_init_and_exit_100Gx50G-sai_impl
./bin/sai_init_and_exit_100Gx100G-sai_impl
./bin/sai_rib_resolution_speed-sai_impl
./bin/sai_switch_reachability_change_speed-sai_impl