CVM Base Image and Binary Building

This document explains how to build the prerequisite base images and binaries required by the CVM builder before running nvflare provision.

The image builder is shipped as part of the NVFlare source tree under nvflare/lighter/cc/image_builder/ and contains the cvm_build.sh script along with Ansible playbooks and helper scripts. It is typically installed at ~/cc/image_builder, which is the path referenced by build_image_cmd in project.yml.

The following artifacts must be built and placed in the image builder directory:

• base_images/ubuntu_base.qcow2 — Ubuntu base disk image

• base_images/OVMF.amdsev.fd — Firmware with kernel-hashes=on support

• binaries/snpguest — Tool for interacting with the TEE

• binaries/kbs-client — Tool for communicating with Trustee KBS

Note

In examples below, <builder_root> refers to the directory containing the cvm_build.sh script.

Build Ubuntu Base Image

The Ubuntu base image must be built on a Ubuntu 25.04 host with an Ubuntu 24.04 guest.

Note

The OS versions differ between host and guest. This is the only tested combination.

The following instructions are adapted from NVIDIA’s Deployment Guide for SecureAI: https://docs.nvidia.com/cc-deployment-guide-snp.pdf

Download GPU Admin Tools

cd /shared/
git clone https://github.com/NVIDIA/gpu-admin-tools

Autoload VFIO

Create /etc/modules-load.d/vfio.conf with the following content:

vfio
vfio_pci

Restart the host to load the VFIO modules:

sudo reboot

Download Ubuntu Installation Image

Download the ISO file for Ubuntu 24.04.2:

cd /shared
wget https://releases.ubuntu.com/24.04.2/ubuntu-24.04.2-live-server-amd64.iso

Create a Drive Image

Create a drive image large enough to hold the OS. A minimum of 30GB is required to install Ubuntu and the GPU drivers. The builder will extend it as needed.

qemu-img create -f qcow2 /shared/ubuntu_base.qcow2 30G

Install Ubuntu Guest

Create the file /shared/launch_vm.sh with the following content:

#!/bin/bash

CORES=16
MEM=32
VDD_IMAGE=/shared/ubuntu_base.qcow2
FWDPORT=9899
CDROM=/shared/ubuntu-24.04.2-live-server-amd64.iso

doecho=false
docc=true
sev=""

while getopts "exp:c:" flag
do
  case ${flag} in
    e) doecho=true;;
    x) docc=false;;
    p) FWDPORT=${OPTARG};;
    c) sev=${OPTARG};;
  esac
done

NVIDIA_GPU=$(lspci -d 10de: | awk '/NVIDIA/{print $1}')
NVIDIA_PASSTHROUGH=$(lspci -n -s $NVIDIA_GPU | awk -F: '{print $4}' | awk '{print $1}')

if [ "$doecho" = true ]; then
  echo 10de $NVIDIA_PASSTHROUGH > /sys/bus/pci/drivers/vfio-pci/new_id
fi

get_cbitpos() {
    modprobe cpuid
    EBX=$(dd if=/dev/cpu/0/cpuid ibs=16 count=32 skip=134217728 | tail -c 16 | od -An -t u4 -j 4 -N 4 | sed -re 's|^ *||')
    CBITPOS=$((EBX & 0x3f))
}

if [ "$docc" = true ]; then
  if [ -n "$sev" ]; then
       case "$sev" in
         sev|sev-es|sev-snp)
           SEV_MODE="$sev"
           USE_CC=true
           get_cbitpos
           ;;
         *)
           echo "Error: unsupported SEV mode '$sev'."
           echo "Use '-c' with valid options: sev, sev-es, sev-snp."
           echo "Or use '-x' to boot without CC modes"
           exit 1
           ;;
       esac
     fi
fi

qemu-system-x86_64 \
  -bios /usr/share/ovmf/OVMF.fd \
  -nographic \
  ${USE_CC:+  -machine confidential-guest-support=sev0,vmport=off} \
  ${USE_CC:+$( [ "$SEV_MODE" = sev ] && \
   echo "  -object sev-guest,id=sev0,cbitpos=${CBITPOS},reduced-phys-bits=1,policy=0x1" )} \
  ${USE_CC:+$( [ "$SEV_MODE" = sev-es ] && \
   echo "  -object sev-guest,id=sev0,cbitpos=${CBITPOS},reduced-phys-bits=1,policy=0x5" )} \
  ${USE_CC:+$( [ "$SEV_MODE" = sev-snp ] && \
   echo "  -object sev-snp-guest,id=sev0,cbitpos=${CBITPOS},reduced-phys-bits=1,policy=0x30000" )} \
  -vga none \
  -enable-kvm -no-reboot \
  -cpu EPYC-v4 \
  -machine q35 -smp $CORES -m ${MEM}G,slots=2,maxmem=512G \
  -drive file=$VDD_IMAGE,if=none,id=disk0,format=qcow2 \
  -device virtio-scsi-pci,id=scsi0,disable-legacy=on,iommu_platform=true,romfile= \
  -device scsi-hd,drive=disk0 \
  -netdev user,id=vmnic,hostfwd=tcp::$FWDPORT-:22 \
  -cdrom $CDROM \
  -device virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=vmnic,romfile= \
  -object iommufd,id=iommufd0 \
  -device pcie-root-port,id=pci.1,bus=pcie.0 \
  -device vfio-pci,host=${NVIDIA_GPU},bus=pci.1,iommufd=iommufd0,romfile=

Launch the VM to start the Ubuntu installation:

chmod +x /shared/launch_vm.sh
sudo /shared/launch_vm.sh -ex

Install a minimal Ubuntu 24.04. All required software will be installed by the builder later. After the guest OS is installed, Ubuntu will prompt you to reboot — the VM will terminate and return you to the host.

Save Base Image

Copy the VM image to the base_images folder:

cp /shared/ubuntu_base.qcow2 <builder_root>/base_images

Fetch Firmware

Check if OVMF.amdsev.fd is already available in /usr/share/ovmf. If not, install it from the Ubuntu proposed repository:

echo 'deb http://archive.ubuntu.com/ubuntu plucky-proposed main restricted universe multiverse' | \
  sudo tee /etc/apt/sources.list.d/plucky-proposed.list

sudo tee /etc/apt/preferences.d/99-plucky-proposed <<'EOF'
Package: *
Pin: release a=plucky-proposed
Pin-Priority: 100
EOF

sudo apt update
sudo apt install -t plucky-proposed ovmf

Copy the firmware to the base_images folder:

cp /usr/share/ovmf/OVMF.amdsev.fd <builder_root>/base_images

Build snpguest

The snpguest tool is needed to interact with the TEE (Trusted Execution Environment).

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"

sudo apt install -y build-essential

# Checkout source code
git clone https://github.com/virtee/snpguest.git
cd snpguest
git checkout v0.9.2
cargo build -r

cp target/release/snpguest <builder_root>/binaries

Build kbs-client

The kbs-client tool is used to interact with Trustee. Its version must match the Trustee server version. The tested commit is a2570329cc33daf9ca16370a1948b5379bb17fbe.

# Install dependencies
sudo apt install -y pkg-config libtss2-dev

# Checkout source code
git clone https://github.com/confidential-containers/trustee.git
cd trustee/tools/kbs-client
git checkout a2570329cc33daf9ca16370a1948b5379bb17fbe

# Build
make -C ../../kbs cli

cp ../../target/release/kbs-client <builder_root>/binaries