Building FBOSS on Docker Containers

This document covers how to build FBOSS binaries and all its library dependencies on Docker containers.

Clone the FBOSS Repository

We will be working out of the root of the repository:

git clone https://github.com/facebook/fboss.git
cd fboss

Set Up the FBOSS Container

Stop Existing Containers and Clean Docker Artifacts

This isn't a strictly required step, although in some cases you may want to build a completely fresh container to use. The below commands will stop any existing Docker containers and clean the image cache so that the subsequent steps will build the container from scratch:

sudo docker container kill -a && sudo docker container prune -f
sudo docker image prune -af

Build the FBOSS Docker Image

The FBOSS GitHub repository contains a Dockerfile that can be used to create the Docker container image for building FBOSS binaries. The Dockerfile is located under fboss/oss/docker/Dockerfile. Use the below command to build the image, which installs required dependencies. Note that the path to the Dockerfile is relative, so the below commands assume you are currently running them from the root of the repository:

# Builds a docker container image that is tagged as fboss_image:latest
sudo docker build . -t fboss_image -f fboss/oss/docker/Dockerfile

Start the FBOSS Container

Start a Container for Platform Stack

# Use stable commits
rm -rf build/deps/github_hashes/
tar xvzf fboss/oss/stable_commits/latest_stable_hashes.tar.gz

# If you know your host has enough space for the build
sudo docker run -d \
    -it --name=FBOSS_BUILD_CONTAINER \
    -v $PWD:/var/FBOSS/fboss:z \
    fboss_image:latest bash

# A full FBOSS build may take significant space (>50GB of storage). You
# can mount a volume with more storage for building by using the -v flag
sudo docker run -d \
    -it --name=FBOSS_BUILD_CONTAINER \
    -v $PWD:/var/FBOSS/fboss:z \
    -v /opt/app/localbuild:/var/FBOSS/tmp_bld_dir:z \
    fboss_image:latest bash

Start a Container for Forwarding Stack

# Use stable commits
rm -rf build/deps/github_hashes/
tar xvzf fboss/oss/stable_commits/latest_stable_hashes.tar.gz

# If you know your host has enough space for the build, just make the SDK
# artifacts available by mounting them with the -v flag.
#
# E.g. you have a directory /path_to_sdk/ containing lib/ and include/
# /path_to_sdk/
# ├── include
# │   ├── file1.h
# │   └── file2.h
# └── lib
#     └── libsai_impl.a
#
# this will mount lib/ and include/ to the container like so:
# /path_to_sdk/lib/     -> /opt/sdk/lib/
# /path_to_sdk/include/ -> /opt/sdk/include/
sudo docker run -d \
    -it --name=FBOSS_BUILD_CONTAINER \
    -v $PWD:/var/FBOSS/fboss:z \
    -v /path_to_sdk:/opt/sdk:z \
    fboss_image:latest bash

# A full FBOSS build may take significant space (>50GB of storage). You
# can mount a volume with more storage for building by using the -v flag
sudo docker run -d \
    -it --name=FBOSS_BUILD_CONTAINER \
    -v $PWD:/var/FBOSS/fboss:z \
    -v /path_to_sdk:/opt/sdk:z \
    -v /opt/app/localbuild:/var/FBOSS/tmp_bld_dir:z \
    fboss_image:latest bash

Enter the Container

# Attaches our current terminal to a new bash shell in the docker container so
# that we can perform the build within it
sudo docker exec -it FBOSS_BUILD_CONTAINER bash

At this point, you should be dropped into a root shell within the Docker container and can proceed with the next steps to start the build.

Build FBOSS Binaries

Instructions for building FBOSS binaries may have slight differences based on which SDK you are linking against.

Build Platform Stack

You only need one command to build the Platform Stack, where $TARGET is fboss_platform_services:

#!/bin/bash
# Navigate to the right directory
cd /var/FBOSS/fboss || exit

# Build using a cmake target
time ./fboss/oss/scripts/run-getdeps.py \
  build \
  --allow-system-packages \
  --build-type MinSizeRel \
  --extra-cmake-defines='{"CMAKE_CXX_STANDARD": "20", "RANGE_V3_TESTS": "OFF", "RANGE_V3_PERF": "OFF"}' \
  --scratch-path /var/FBOSS/tmp_bld_dir \
  --cmake-target $TARGET \
  fboss

Build Forwarding Stack

This section assumes that you have a precompiled ASIC SDK library which you want to link against. More specifically, you'll need the static library libsai_impl.a for the SDK which you are trying to link against, as well as the associated set of SAI headers. In order to run the build:

Build Against the SDK

Ensure you are in the right directory and start the build. The run-getdeps.py script accepts flags to configure the SAI implementation and SDK version (run ./fboss/oss/scripts/run-getdeps.py -h to see all options and Meta supported values):

#!/bin/bash
# Navigate to the right directory
cd /var/FBOSS/fboss || exit

# Start the build
# NOTE: Choose the appropriate --npu-sai-impl and --npu-sai-sdk-version values
# for your platform. Run ./fboss/oss/scripts/run-getdeps.py -h to see
# Meta officially supported values.
time ./fboss/oss/scripts/run-getdeps.py \
  --npu-sai-impl SAI_BRCM_IMPL \
  --npu-sai-sdk-version SAI_VERSION_14_0_EA_ODP \
  --npu-sai-version 1.16.1 \
  --npu-libsai-impl-path /opt/sdk/libsai_impl.a \
  --npu-experiments-path /opt/sdk/experimental \
  build \
  --allow-system-packages \
  --build-type MinSizeRel \
  --extra-cmake-defines='{"CMAKE_CXX_STANDARD": "20", "RANGE_V3_TESTS": "OFF", "RANGE_V3_PERF": "OFF"}' \
  --scratch-path /var/FBOSS/tmp_bld_dir \
  fboss

Build QSFP Targets Against PHY SAI SDK

For some project that needs to use XPHY(Retimer) which has its own SAI SDK, we need to build QSFP targets to link with the precompiled PHY SDK library just like we did above for Agent targets linking with ASIC SDK. However, we won't be able to support building both Agent and QSFP targets with the single run-getdeps.py build fboss command like above, mainly because ASIC SAI SDK and PHY SAI SDK usually don't use the same SAI version and implementation. Therefore, for any platform that needs linking both ASIC SAI SDK and PHY SAI SDK, we highly recommend to do it in the following order:

Run the commands from Build Against the SDK to build Agent and other services
Run the following command to build QSFP targets

#!/bin/bash
# Navigate to the right directory
cd /var/FBOSS/fboss || exit

# Prepare BRCM_PAI SDK artifacts directory for linking
mkdir -p /var/FBOSS/pai_impl
mkdir -p /var/FBOSS/pai_impl/lib
mkdir -p /var/FBOSS/pai_impl/include
mkdir -p /var/FBOSS/pai_impl/include/epdm

# Copy the three sdk artifacts to `lib` directory
# You should see the following three precompiled artifacts as follow
# ls /var/FBOSS/pai_impl/lib
# libepdm.a  libpai.a  libphymodepil.a

# Copy the header folders from PAI sdk to `include directory`
# NOTE: Adjust the correct directory based on your setup
cp -r /opt/sdk/PAI_4.0/inc/sai /var/FBOSS/pai_impl/include
cp -r /opt/sdk/PAI_4.0/inc/pai_macsec /var/FBOSS/pai_impl/include
# Copy header from EPDM
# NOTE: Adjust the correct directory based on your setup
cp /opt/sdk/epdm_4_5_2/*.h /var/FBOSS/pai_impl/include/epdm/

# Start the build with specific cmake-target `qsfp_targets`
update-alternatives --set gcc /usr/local/llvm/bin/clang
time ./fboss/oss/scripts/run-getdeps.py \
  --phy-sai-impl SAI_BRCM_PAI_IMPL \
  build \
  --allow-system-packages \
  --build-type MinSizeRel \
  --extra-cmake-defines='{"CMAKE_CXX_STANDARD": "20", "RANGE_V3_TESTS": "OFF", "RANGE_V3_PERF": "OFF"}' \
  --scratch-path /var/FBOSS/tmp_bld_dir \
  --cmake-target qsfp_targets \
  fboss

Build Type Options

The getdeps.py script supports different build types via the --build-type flag. This controls optimization levels and debug information:

Build Type	Description	Use Case
Debug	No optimization, full debug symbols	Development and debugging
RelWithDebInfo	Optimized with debug symbols	Default - Balances performance with debuggability
MinSizeRel	Optimized for size, no debug symbols	Production deployments and resource-constrained environments
Release	Full optimization, no debug symbols	Maximum performance (may have compatibility issues)

Note: If --build-type is not specified, the build defaults to RelWithDebInfo, which balances performance with the ability to debug issues.

Limit the Build to a Specific Target

You can limit the build to a specific target by using the --cmake-target flag. Buildable targets can be found by examining the cmake scripts in the repository. Any buildable target will be specified in the cmake scripts either by add_executable or add_library. Example command below:

#!/bin/bash
# Navigate to the right directory
cd /var/FBOSS/fboss || exit

# Build using a cmake target
time ./fboss/oss/scripts/run-getdeps.py \
  build \
  --allow-system-packages \
  --build-type MinSizeRel \
  --extra-cmake-defines='{"CMAKE_CXX_STANDARD": "20", "RANGE_V3_TESTS": "OFF", "RANGE_V3_PERF": "OFF"}' \
  --scratch-path /var/FBOSS/tmp_bld_dir \
  --cmake-target $TARGET \
  fboss

Clone the FBOSS Repository​

Set Up the FBOSS Container​

Stop Existing Containers and Clean Docker Artifacts​

Build the FBOSS Docker Image​

Start the FBOSS Container​

Start a Container for Platform Stack​

Start a Container for Forwarding Stack​

Enter the Container​

Build FBOSS Binaries​

Build Platform Stack​

Build Forwarding Stack​

Build Against the SDK​

Build QSFP Targets Against PHY SAI SDK​

Build Type Options​

Limit the Build to a Specific Target​