Skip to main content

Run Agent Benchmark Tests

Overview​

Running Agent benchmark tests consists of building the benchmark binaries and then running them to observe the output.

If you run into issues, try checking our troubleshooting guide.

After each section, check the Overall Outcomes below to ensure everything is completed.

Overall Outcomes​

Setup​

  • an FBOSS forwarding stack Docker container to perform builds in

Build and Test Agent Benchmarks​

  • benchmark binaries can be built
  • benchmark binaries can be run

Setup​

Step 1: Clone the FBOSS Repository​

git clone https://github.com/facebook/fboss.git
cd fboss

Step 2: Stop Existing Containers and Clean Docker Artifacts​

sudo docker container kill -a && sudo docker container prune -f
sudo docker image prune -af

Step 3: Build the FBOSS Docker Image​

# Builds a docker container image that is tagged as fboss_docker:latest
sudo docker build . -t fboss_docker -f fboss/oss/docker/Dockerfile

Step 4: Start a Forwarding Stack Container​

# Use stable commits
rm -rf build/deps/github_hashes/
tar xvzf fboss/oss/stable_commits/latest_stable_hashes.tar.gz

# If you know your host has enough space for the build, just make the SDK
# artifacts available by mounting them with the -v flag.
#
# E.g. you have a directory /path_to_sdk/ containing lib/ and include/
# /path_to_sdk/
# ├── include
# │   ├── file1.h
# │   └── file2.h
# └── lib
#     └── libsai_impl.a
#
# this will mount lib/ and include/ to the container like so:
# /path_to_sdk/lib/     -> /opt/sdk/lib/
# /path_to_sdk/include/ -> /opt/sdk/include/
sudo docker run -d \
    -it --name=FBOSS_DOCKER_CONTAINER \
    -v $PWD:/var/FBOSS/fboss:z \
    -v /path_to_sdk:/opt/sdk:z \
    fboss_docker:latest bash

# A full FBOSS build may take significant space (>50GB of storage). You
# can mount a volume with more storage for building by using the -v flag
sudo docker run -d \
    -it --name=FBOSS_DOCKER_CONTAINER \
    -v $PWD:/var/FBOSS/fboss:z \
    -v /path_to_sdk:/opt/sdk:z \
    -v /opt/app/localbuild:/var/FBOSS/tmp_bld_dir:z \
    fboss_docker:latest bash

At this point, you should have a Docker container which can be used to build the required artifacts in later steps.

Build and Test Agent Benchmarks​

Step 1: Build Agent Benchmarks​

Use the forwarding stack Docker container from the Setup step.

Step 1.1: Enter the Container​

# Attaches our current terminal to a new bash shell in the docker container so
# that we can perform the build within it
sudo docker exec -it FBOSS_DOCKER_CONTAINER bash

Step 1.2: Run the Build Helper​

# By default, `build-helper.py` will use SAI version 1.16.3. If you are planning
# on building against a different version of SAI, you must add another param to
# the build-helper.py command

# Supported values:

# 1.13.2
# 1.14.0
# 1.15.0
# 1.15.3
# 1.16.0
# 1.16.1
# 1.16.3

# Run the build helper to stage the SDK in preparation for the build step
./fboss/oss/scripts/build-helper.py /opt/sdk/lib/libsai_impl.a /opt/sdk/include/ /var/FBOSS/sai_impl_output

# Run the build helper using SAI version 1.15.3 instead of the default 1.16.3
./fboss/oss/scripts/build-helper.py /opt/sdk/lib/libsai_impl.a /opt/sdk/include/ /var/FBOSS/sai_impl_output 1.15.3

Step 1.3: Set Important Environment Variables​

# The following environment variables should be set depending on which platform
# and SDK version you are building

# Set to 1 if building against brcm-sai SDK
export SAI_BRCM_IMPL=1

# Can be omitted if you are using SAI 1.16.3. If using a more recent version of
# SAI from https://github.com/opencomputeproject/SAI, this should be set to the
# semantic version e.g. 1.16.1
export SAI_VERSION="1.16.1"

# Should be set to a string depending on which version of the brcm-sai SDK you
# are building. Default value is "SAI_VERSION_11_0_EA_DNX_ODP".
# Supported values can be found in
# https://github.com/facebook/fboss/blob/main/fboss/agent/hw/sai/api/SaiVersion
# but are listed below for convenience:

# SAI_VERSION_8_2_0_0_ODP
# SAI_VERSION_8_2_0_0_SIM_ODP
# SAI_VERSION_9_2_0_0_ODP
# SAI_VERSION_9_0_EA_SIM_ODP
# SAI_VERSION_10_0_EA_ODP
# SAI_VERSION_10_0_EA_SIM_ODP
# SAI_VERSION_10_2_0_0_ODP
# SAI_VERSION_11_0_EA_ODP
# SAI_VERSION_11_0_EA_SIM_ODP
# SAI_VERSION_11_3_0_0_ODP
# SAI_VERSION_11_7_0_0_ODP
# SAI_VERSION_10_0_EA_DNX_ODP
# SAI_VERSION_10_0_EA_DNX_SIM_ODP
# SAI_VERSION_11_0_EA_DNX_ODP
# SAI_VERSION_11_0_EA_DNX_SIM_ODP
# SAI_VERSION_11_3_0_0_DNX_ODP
# SAI_VERSION_11_7_0_0_DNX_ODP
# SAI_VERSION_12_0_EA_DNX_ODP
# SAI_VERSION_13_0_EA_ODP
# SAI_VERSION_14_0_EA_ODP

# Set the SAI SDK version
export SAI_SDK_VERSION="SAI_VERSION_14_0_EA_ODP"

# Set to 1 to install benchmark binaries
export BENCHMARK_INSTALL=1

# Set to 1 to skip installing FBOSS artifacts
export SKIP_ALL_INSTALL=1

# Set to 1 to enable sanitization checking
export WITH_ASAN=1

For Agent benchmark binaries, you also need to set another env variable:

export BENCHMARK_INSTALL=1

Step 1.4: Build Agent Benchmark Binaries​

Remember to set the BENCHMARK_INSTALL env variable from the previous step.

# Navigate to the right directory
cd /var/FBOSS/fboss

# Set environment variables appropriate for your build
export SAI_BRCM_IMPL=1

# Start the build
time ./build/fbcode_builder/getdeps.py build --allow-system-packages \
--extra-cmake-defines='{"CMAKE_BUILD_TYPE": "MinSizeRel", "CMAKE_CXX_STANDARD": "20"}' \
--scratch-path /var/FBOSS/tmp_bld_dir fboss

Step 2: Send Build Artifacts to the Switch​

Step 2.1: Package the Benchmark Binaries​

# On the container used for building FBOSS:

# Clean any existing packages
rm -rf /var/FBOSS/tmp_bld_dir/fboss_bins*

# Navigate to the FBOSS repository
cd /var/FBOSS/fboss

# Creates a package directory with prefix /var/FBOSS/tmp_bld_dir/fboss_bins
./fboss/oss/scripts/package-fboss.py --copy-root-libs --scratch-path /var/FBOSS/tmp_bld_dir/

# or

# Creates a tarball called "fboss_bins.tar.zst" under /var/FBOSS/tmp_bld_dir/
./fboss/oss/scripts/package-fboss.py --copy-root-libs --scratch-path /var/FBOSS/tmp_bld_dir/ --compress

Step 2.2: Copy and Set Up the Package​

# From the host that the container is running on, use the appropriate command to
# copy the FBOSS package directory or tarball to the switch:

# If copying the directory
scp -r /opt/app/FBOSS_DIR/tmp_bld_dir/fboss_bins-$PKG_ID root@$SWITCHNAME:/opt/

# If copying the tarball
scp /opt/app/FBOSS_DIR/tmp_bld_dir/fboss_bins.tar.zst root@$SWITCHNAME:/opt/



# On the switch, use the appropriate command to set up the package:

# If using the directory
cd /opt
ln -s /opt/fboss_bins-$PKG_ID /opt/fboss

# If using the tarball
cd /opt
mkdir fboss && mv fboss_bins.tar.zst fboss/
cd fboss && tar -xvf fboss_bins.tar.zst



# You will have a directory /opt/fboss/ which contains a bin/, lib/, and share/
# directory

Step 3: Run Benchmark Binaries​

Run the relevant binaries that can be found in the bin directory on the switch:

cd /opt/fboss
source ./bin/setup_fboss_env

./bin/sai_fsw_scale_route_add_speed-sai_impl
./bin/sai_hgrid_du_scale_route_add_speed-sai_impl
./bin/sai_hgrid_uu_scale_route_del_speed-sai_impl
./bin/sai_th_alpm_scale_route_add_speed-sai_impl
./bin/sai_fsw_scale_route_del_speed-sai_impl
./bin/sai_ecmp_shrink_with_competing_route_updates_speed-sai_impl
./bin/sai_th_alpm_scale_route_del_speed-sai_impl
./bin/sai_ecmp_shrink_speed-sai_impl
./bin/sai_hgrid_uu_scale_route_add_speed-sai_impl
./bin/sai_hgrid_du_scale_route_del_speed-sai_impl
./bin/sai_stats_collection_speed-sai_impl
./bin/sai_tx_slow_path_rate-sai_impl
./bin/sai_rx_slow_path_rate-sai_impl
./bin/sai_init_and_exit_40Gx10G-sai_impl
./bin/sai_init_and_exit_100Gx10G-sai_impl
./bin/sai_init_and_exit_100Gx25G-sai_impl
./bin/sai_init_and_exit_100Gx50G-sai_impl
./bin/sai_init_and_exit_100Gx100G-sai_impl
./bin/sai_rib_resolution_speed-sai_impl
./bin/sai_switch_reachability_change_speed-sai_impl