rshim and the BFB path

1RShim: four device files and one status file

RShim is the SoC management interface of a BlueField. It “enables management, monitoring, and debugging of the device … firmware updates, system status checks, Arm console access, and network communication through device files (e.g., boot, misc, console) and the RShim network interface (i.e., tmfifo_net0)”.^[2] The host-side daemon (rshim) probes backends in the order IOMMU, UIO, Direct Map, and creates one directory per attached target.^[2] List them with ls -la /dev/ | grep rshim.^[1]

Each /dev/rshim<N>/ holds four files:^[3]

The misc file is the one you read most. At the default DISPLAY_LEVEL 0 the README shows these fields:^[3]

DISPLAY_LEVEL   0 (0:basic, 1:advanced, 2:log)
BOOT_MODE       1 (0:rshim, 1:emmc, 2:emmc-boot-swap)
BOOT_TIMEOUT    100 (seconds)
DROP_MODE       0 (0:normal, 1:drop)
SW_RESET        0 (1: reset)
DEV_NAME        usb-3.3
DEV_INFO        BlueField-3(Rev 1)
OPN_STR         9009D3B400ENEA
UP_TIME         179752(s)
SECURE_NIC_MODE 1 (0:no, 1:yes)

DISPLAY_LEVEL 1 adds PEER_MAC (the Arm-side MAC of the rshim network interface) and PXE_ID; DISPLAY_LEVEL 2 streams the boot log, which is where the install markers appear.^[3][2] On a PCIe backend DEV_NAME reads like pcie-04:00.2: the RShim PF is the .2 function of the BlueField PCIe device, so this field joins an rshim index to an lspci address.^[2] Two writes matter: echo "SW_RESET 1" > /dev/rshim0/misc resets the Arm cores, and echo "DISPLAY_LEVEL 2" > /dev/rshim0/misc turns on the log.^[2] When SECURE_NIC_MODE is 1 “the NIC firmware is in Secure NIC mode and most rshim functionalities are disabled” (PCIe backend only).^[3] OPN_STR is the card’s ordering part number and DEV_INFO its silicon revision; both are worth pasting into a ticket.

One error to recognise on a fresh host: “failed to setup CUSE rshim” means the cuse kernel module is missing; install kernel-modules-extra, modprobe cuse, restart the service.^[2]

sudo apt install -y doca-all        # DEB; RPM: sudo yum install -y doca-all  (or doca-ofed / doca-networking)
sudo mst start
systemctl status rshim

● rshim.service - rshim driver for BlueField SoC
     Active: active (running)

2tmfifo_net0 and the ping that lies

The rshim daemon also creates a network interface, tmfifo_net0, that rides the same channel. The BF-Bundle guide configures the host side with ifconfig tmfifo_net0 192.168.100.1/24 and then connects with ssh ubuntu@192.168.100.2; the Arm side is preconfigured at .2.^[1] The troubleshooting guide shows the Arm end as ifconfig tmfifo_net0 192.168.100.2 netmask 255.255.255.252 up, a /30; either mask works as long as host and Arm agree.^[2] Some DOCA-Host installs bridge tmfifo_net0 into a tm-br bridge, so the address may sit on the bridge instead of the interface.^[9]

The trap: a ping 192.168.100.2 that succeeds can be answered by the host itself when the address landed on the wrong interface. Run ip route get 192.168.100.2 first and accept only dev tmfifo_net0 or dev tm-br in the output.^[9][10] Only then trust SSH failures as the Arm’s fault.

First login on a fresh image is ubuntu/ubuntu, and the system forces a password change unless ubuntu_PASSWORD was set in bf.cfg at install time.^[1] The Arm-side anchors to read once you are in are cat /etc/mlnx-release and sudo bf-info.^[1]

3bf-bundle, bf-fwbundle, and bfb-install

The Linux installation guide lists five ways to put software on a BlueField: a BFB image (“BlueField bootstream (BFB) image for OS deployment on the BlueField DPUs”), standard apt update or yum update on the Arm, an ISO image, bfb-build for people who build their own image, and the BF-FW bundle, “a firmware-only BFB image which includes UEFI, NIC firmware, and BMC firmware”.^[6] The BF-Bundle itself is the DOCA runtime plus the Arm OS plus the platform firmware and UEFI.^[11]

Names tell you which you hold. A full bundle looks like bf-bundle-2.7.0_24.04_ubuntu-22.04_prod.bfb (the guide’s example); the firmware-only image is bf-fwbundle-<version>_prod.^[1] Because 3.5.0’s default BlueField OS is “Ubuntu 24.04 64k”, expect ubuntu-24.04 in a 3.5.0 bundle name, but read the exact file name from the downloads page rather than from this lesson.^[8] The 3.5.0 archive page confirms bf-fwbundle upgrades ATF-UEFI, NIC firmware and BMC firmware only, without OS overwrite, and covers both BlueField-2 and BlueField-3.^[7] The component versions inside a given 3.5.0 image are not tabulated on the pages fetched for this lesson; read them with sudo bf-info on the Arm after the install.^[1]

The push itself:^[1][5]

sudo bfb-install --rshim rshim<N> --bfb <image_path.bfb>
sudo bfb-install --rshim rshim<N> --bfb <image_path.bfb> --config bf.cfg
# short form from the BSP guide
bfb-install -r <rshim device> -c <path/to/bf.cfg> -b <BFB>
# what bfb-install does for you
cat <BFB> <path/to/bf.cfg> > /dev/<rshim device>/boot
# or bake the config into a new image
cat <BFB> <path/to/bf.cfg> > <new BFB>

While it runs, sudo cat /dev/rshim<N>/misc (at DISPLAY_LEVEL 2) shows the boot log. The markers you wait for after the reboot are INFO[MISC]: Linux up and INFO[MISC]: DPU is ready^[1]; the failure string seen during installs is INFO[MISC]: NIC firmware update failed.^[9] The deployment skill’s rule: “Do NOT trust bfb-install exit code 0 alone.” Require the markers, then the tmfifo route, then SSH, then cat /etc/mlnx-release and bfver.^[10] For a card that is not yet healthy it gives a six-state ladder: installer-still-running, uefi-only, linux-up-tmfifo-down, tmfifo-up-ssh-down, arm-ok-host-pfs-unbound, host-bf-version-mismatch.^[10][9] Name the state before you act; each has a different next command.

An Arm-side alternative exists, doca-installer -b <image_path.bfb> -c <path-to-config>, with --show-target-fw, --show-running-fw and --compare; its logs land under /var/log/doca_installer_logs/.^[1]

4bf.cfg: keys and hooks

bf.cfg is a shell-style file appended to the BFB. The BSP guide’s syntax rule is short: “Do not use spaces around the ‘=’.”^[5] The keys fall into four groups (defaults from the BSP 4.15.0 page):^[5]

One default is uncertain across versions: the 4.15.0 page lists WITH_NIC_FW_UPDATE as yes, while older BSP text described it as disabled by default. Read the bf.cfg reference for the BSP inside your bundle before assuming a push will or will not touch NIC firmware.^[5]

Three hook functions let you run shell during the install:^[5]

• bfb_pre_install() runs before storage partitioning.
• bfb_modify_os() is “called after the file system is extracted on the target partitions … target file system mounted under /mnt”.
• bfb_post_install() is the final step before reboot, with partitions unmounted.

The deployment skill uses bfb_modify_os() as the documented place to inject authorized_keys and to apply mode-related mlxconfig at install time, with the rule “decide the DPU mode BEFORE the push”.^[9][10] The guide’s password example is echo ubuntu_PASSWORD='$1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1' > bf.cfg.^[1]

# bf.cfg — minimal lab example (no spaces around '=')
ubuntu_PASSWORD='$1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1'
UPDATE_BMC_FW=no
bfb_modify_os()
{
  mkdir -p /mnt/home/ubuntu/.ssh
  cat >> /mnt/home/ubuntu/.ssh/authorized_keys <<'KEY'
ssh-ed25519 AAAA...your-lab-key... lab@dell
KEY
  chmod 700 /mnt/home/ubuntu/.ssh
  chmod 600 /mnt/home/ubuntu/.ssh/authorized_keys
}

The key material and the chmod lines are ordinary Linux; the hook name, the /mnt prefix and the no-spaces rule are the BlueField-specific facts.^[5]

5A full install transcript

The transcript below is the BF-Bundle guide’s own example (a 2.7.0 image on Ubuntu 22.04; a 3.5.0 push prints the same stages with a different file name).^[1]

host# echo ubuntu_PASSWORD='$1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1' > bf.cfg
host# sudo bfb-install --rshim rshim0 --bfb bf-bundle-2.7.0_24.04_ubuntu-22.04_prod.bfb --config bf.cfg
Pushing bfb 1.41GiB 0:02:02 [11.7MiB/s] [           <=>                     ]
Collecting BlueField booting status. Press Ctrl+C to stop
 INFO[PSC]: PSC BL1 START
 INFO[BL2]: start
 INFO[BL2]: boot mode (rshim)
 INFO[BL2]: VDDQ: 1120 mV
 INFO[BL2]: DDR POST passed
 INFO[BL2]: UEFI loaded
 INFO[BL31]: start
 INFO[BL31]: lifecycle GA Secured
 INFO[BL31]: VDD: 850 mV
 INFO[BL31]: runtime
 INFO[BL31]: MB ping success
 INFO[UEFI]: eMMC init
 INFO[UEFI]: eMMC probed
 INFO[UEFI]: UPVS valid
 INFO[UEFI]: PMI: updates started
 INFO[UEFI]: PMI: total updates: 1
 INFO[UEFI]: PMI: updates completed, status 0
 INFO[UEFI]: PCIe enum start
 INFO[UEFI]: PCIe enum end
 INFO[UEFI]: UEFI Secure Boot
 INFO[UEFI]: PK configured
 INFO[UEFI]: Redfish enabled
 INFO[UEFI]: exit Boot Service
 INFO[MISC]: Found bf.cfg
 INFO[MISC]: Ubuntu installation started
 INFO[MISC]: Installing OS image
 INFO[MISC]: Changing the default password for user ubuntu
 INFO[MISC]: Ubuntu installation completed
 INFO[MISC]: Updating NIC firmware...
 INFO[MISC]: NIC firmware update done
 INFO[MISC]: Installation finished

host# echo ____='$1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1' > bf.cfg
host# sudo bfb-install --rshim ____ --bfb bf-bundle-<ver>_ubuntu-<os>_prod.bfb --config ____
Pushing bfb ...
Collecting BlueField booting status. Press Ctrl+C to stop
 INFO[BL2]: boot mode (____)          <- proves the card booted from the stream
 INFO[BL2]: DDR POST ____
 INFO[UEFI]: PMI: updates completed, status ____
 INFO[MISC]: Found ____               <- your config reached the installer
 INFO[MISC]: Ubuntu installation started
 INFO[MISC]: Installing OS image
 INFO[MISC]: Changing the default password for user ____
 INFO[MISC]: Ubuntu installation completed
 INFO[MISC]: Updating NIC firmware...
 INFO[MISC]: NIC firmware update ____
 INFO[MISC]: Installation ____