Skip to main content

Building FBOSS on Docker Containers

This document covers how to build FBOSS binaries and all its library dependencies on Docker containers.

Clone the FBOSS Repository​

We will be working out of the root of the repository:

git clone https://github.com/facebook/fboss.git
cd fboss

Set Up the FBOSS Container​

Stop Existing Containers and Clean Docker Artifacts​

This isn't a strictly required step, although in some cases you may want to build a completely fresh container to use. The below commands will stop any existing Docker containers and clean the image cache so that the subsequent steps will build the container from scratch:

sudo docker container kill -a && sudo docker container prune -f
sudo docker image prune -af

Build the FBOSS Docker Image​

The FBOSS GitHub repository contains a Dockerfile that can be used to create the Docker container image for building FBOSS binaries. The Dockerfile is located under fboss/oss/docker/Dockerfile. Use the below command to build the image, which installs required dependencies. Note that the path to the Dockerfile is relative, so the below commands assume you are currently running them from the root of the repository:

# Builds a docker container image that is tagged as fboss_docker:latest
sudo docker build . -t fboss_docker -f fboss/oss/docker/Dockerfile

Start the FBOSS Container​

Start a Container for Platform Stack​

# Use stable commits
rm -rf build/deps/github_hashes/
tar xvzf fboss/oss/stable_commits/latest_stable_hashes.tar.gz

# If you know your host has enough space for the build
sudo docker run -d \
    -it --name=FBOSS_DOCKER_CONTAINER \
    -v $PWD:/var/FBOSS/fboss:z \
    fboss_docker:latest bash

# A full FBOSS build may take significant space (>50GB of storage). You
# can mount a volume with more storage for building by using the -v flag
sudo docker run -d \
    -it --name=FBOSS_DOCKER_CONTAINER \
    -v $PWD:/var/FBOSS/fboss:z \
    -v /opt/app/localbuild:/var/FBOSS/tmp_bld_dir:z \
    fboss_docker:latest bash

Start a Container for Forwarding Stack​

# Use stable commits
rm -rf build/deps/github_hashes/
tar xvzf fboss/oss/stable_commits/latest_stable_hashes.tar.gz

# If you know your host has enough space for the build, just make the SDK
# artifacts available by mounting them with the -v flag.
#
# E.g. you have a directory /path_to_sdk/ containing lib/ and include/
# /path_to_sdk/
# ├── include
# │   ├── file1.h
# │   └── file2.h
# └── lib
#     └── libsai_impl.a
#
# this will mount lib/ and include/ to the container like so:
# /path_to_sdk/lib/     -> /opt/sdk/lib/
# /path_to_sdk/include/ -> /opt/sdk/include/
sudo docker run -d \
    -it --name=FBOSS_DOCKER_CONTAINER \
    -v $PWD:/var/FBOSS/fboss:z \
    -v /path_to_sdk:/opt/sdk:z \
    fboss_docker:latest bash

# A full FBOSS build may take significant space (>50GB of storage). You
# can mount a volume with more storage for building by using the -v flag
sudo docker run -d \
    -it --name=FBOSS_DOCKER_CONTAINER \
    -v $PWD:/var/FBOSS/fboss:z \
    -v /path_to_sdk:/opt/sdk:z \
    -v /opt/app/localbuild:/var/FBOSS/tmp_bld_dir:z \
    fboss_docker:latest bash

Enter the Container​

# Attaches our current terminal to a new bash shell in the docker container so
# that we can perform the build within it
sudo docker exec -it FBOSS_DOCKER_CONTAINER bash

At this point, you should be dropped into a root shell within the Docker container and can proceed with the next steps to start the build.

Build FBOSS Binaries​

Instructions for building FBOSS binaries may have slight differences based on which SDK you are linking against.

Build Platform Stack​

You only need one command to build the Platform Stack, where $TARGET is fboss_platform_services:

# Navigate to the right directory
cd /var/FBOSS/fboss

# Build using a cmake target
time ./build/fbcode_builder/getdeps.py build --allow-system-packages \
--extra-cmake-defines='{"CMAKE_BUILD_TYPE": "MinSizeRel", "CMAKE_CXX_STANDARD": "20"}' \
--scratch-path /var/FBOSS/tmp_bld_dir --cmake-target $TARGET fboss

Build Forwarding Stack​

This section assumes that you have a precompiled ASIC SDK library which you want to link against. More specifically, you'll need the static library libsai_impl.a for the SDK which you are trying to link against, as well as the associated set of SAI headers. In order to run the build:

Run the Build Helper​

# By default, `build-helper.py` will use SAI version 1.16.3. If you are planning
# on building against a different version of SAI, you must add another param to
# the build-helper.py command

# Supported values:

# 1.13.2
# 1.14.0
# 1.15.0
# 1.15.3
# 1.16.0
# 1.16.1
# 1.16.3

# Run the build helper to stage the SDK in preparation for the build step
./fboss/oss/scripts/build-helper.py /opt/sdk/lib/libsai_impl.a /opt/sdk/include/ /var/FBOSS/sai_impl_output

# Run the build helper using SAI version 1.15.3 instead of the default 1.16.3
./fboss/oss/scripts/build-helper.py /opt/sdk/lib/libsai_impl.a /opt/sdk/include/ /var/FBOSS/sai_impl_output 1.15.3

Set Important Environment Variables​

# The following environment variables should be set depending on which platform
# and SDK version you are building

# Set to 1 if building against brcm-sai SDK
export SAI_BRCM_IMPL=1

# Can be omitted if you are using SAI 1.16.3. If using a more recent version of
# SAI from https://github.com/opencomputeproject/SAI, this should be set to the
# semantic version e.g. 1.16.1
export SAI_VERSION="1.16.1"

# Should be set to a string depending on which version of the brcm-sai SDK you
# are building. Default value is "SAI_VERSION_11_0_EA_DNX_ODP".
# Supported values can be found in
# https://github.com/facebook/fboss/blob/main/fboss/agent/hw/sai/api/SaiVersion
# but are listed below for convenience:

# SAI_VERSION_8_2_0_0_ODP
# SAI_VERSION_8_2_0_0_SIM_ODP
# SAI_VERSION_9_2_0_0_ODP
# SAI_VERSION_9_0_EA_SIM_ODP
# SAI_VERSION_10_0_EA_ODP
# SAI_VERSION_10_0_EA_SIM_ODP
# SAI_VERSION_10_2_0_0_ODP
# SAI_VERSION_11_0_EA_ODP
# SAI_VERSION_11_0_EA_SIM_ODP
# SAI_VERSION_11_3_0_0_ODP
# SAI_VERSION_11_7_0_0_ODP
# SAI_VERSION_10_0_EA_DNX_ODP
# SAI_VERSION_10_0_EA_DNX_SIM_ODP
# SAI_VERSION_11_0_EA_DNX_ODP
# SAI_VERSION_11_0_EA_DNX_SIM_ODP
# SAI_VERSION_11_3_0_0_DNX_ODP
# SAI_VERSION_11_7_0_0_DNX_ODP
# SAI_VERSION_12_0_EA_DNX_ODP
# SAI_VERSION_13_0_EA_ODP
# SAI_VERSION_14_0_EA_ODP

# Set the SAI SDK version
export SAI_SDK_VERSION="SAI_VERSION_14_0_EA_ODP"

# Set to 1 to install benchmark binaries
export BENCHMARK_INSTALL=1

# Set to 1 to skip installing FBOSS artifacts
export SKIP_ALL_INSTALL=1

# Set to 1 to enable sanitization checking
export WITH_ASAN=1

Build Against the SDK​

Ensure you are in the right directory, set your relevant environment variables, and start the build:

# Navigate to the right directory
cd /var/FBOSS/fboss

# Set environment variables appropriate for your build
export SAI_BRCM_IMPL=1

# Start the build
time ./build/fbcode_builder/getdeps.py build --allow-system-packages \
--extra-cmake-defines='{"CMAKE_BUILD_TYPE": "MinSizeRel", "CMAKE_CXX_STANDARD": "20"}' \
--scratch-path /var/FBOSS/tmp_bld_dir fboss

Build QSFP Targets Against PHY SAI SDK​

For some project that needs to use XPHY(Retimer) which has its own SAI SDK, we need to build QSFP targets to link with the precompiled PHY SDK library just like we did above for Agent targets linking with ASIC SDK. However, we won't be able to support building both Agent and QSFP targets with the single getdeps.py build fboss command like above, mainly because ASIC SAI SDK and PHY SAI SDK usually don't use the same SAI version and implementation. Therefore, for any platform that needs linking both ASIC SAI SDK and PHY SAI SDK, we highly recommend to do it in the following order:

  • Run the commands from Build Against the SDK to build Agent and other services
  • Run the following command to build QSFP targets
# Navigate to the right directory
cd /var/FBOSS/fboss

# Prepare BRCM_PAI SDK artifacts directory for linking
mkdir -p /var/FBOSS/pai_impl
mkdir -p /var/FBOSS/pai_impl/lib
mkdir -p /var/FBOSS/pai_impl/include

# Copy the three sdk artifacts to `lib` directory
# You should see the following three precompiled artifacts as follow
# ls /var/FBOSS/pai_impl/lib
# libepdm.a  libpai.a  libphymodepil.a

# Copy the header folders from PAI sdk to `include directory`
# NOTE: Adjust the correct directory based on your setup
cp -r /opt/sdk/PAI_3.15/inc/sai /var/FBOSS/pai_impl/include
cp -r /opt/sdk/PAI_3.15/inc/pai_macsec /var/FBOSS/pai_impl/include

# Set environment variables to build the qsfp targets with the BRCM PAI
# Make sure to unset any existing variables to link with ASIC SDK as PHY
# SDK needs different sai
unset SAI_BRCM_IMPL
unset CHENAB_SAI_SDK
unset SAI_TAJO_IMPL
export SAI_BRCM_PAI_IMPL=1

# Start the build with specific cmake-target `qsfp_targets`
time ./build/fbcode_builder/getdeps.py build --allow-system-packages \
--extra-cmake-defines='{"CMAKE_BUILD_TYPE": "MinSizeRel", "CMAKE_CXX_STANDARD": "20"}' \
--scratch-path /var/FBOSS/tmp_bld_dir --cmake-target qsfp_targets fboss

Limit the Build to a Specific Target​

You can limit the build to a specific target by using the --cmake-target flag. Buildable targets can be found by examining the cmake scripts in the repository. Any buildable target will be specified in the cmake scripts either by add_executable or add_library. Example command below:

# Navigate to the right directory
cd /var/FBOSS/fboss

# Build using a cmake target
time ./build/fbcode_builder/getdeps.py build --allow-system-packages \
--extra-cmake-defines='{"CMAKE_BUILD_TYPE": "MinSizeRel", "CMAKE_CXX_STANDARD": "20"}' \
--scratch-path /var/FBOSS/tmp_bld_dir --cmake-target $TARGET fboss