Part 2 of the series: software images, categories, node provisioning, vGPU guest driver, Slurm with GPU GRES
This guide continues from “Deploying NVIDIA Base Command Manager 11 with High Availability”. The HA head-node pair is up with shared storage on NFS. Here we prepare separate software images for CPU and GPU nodes, provision three compute nodes over PXE, install the NVIDIA vGPU guest driver into the GPU image, and run CPU and GPU jobs through Slurm.
Part1- https://kdinesh.in/bcm/
Lab note: the compute nodes are VMware VMs (no iDRAC/BMC), so nodes are powered on by hand and BCM cannot power-cycle them. The GPU nodes use an NVIDIA vGPU profile, so they need the vGPU (GRID) guest driver and a license. On bare-metal GPU servers use the NVIDIA datacenter driver instead and no license is needed.
Environment
Item | Value |
|---|---|
Cluster manager | NVIDIA Base Command Manager 11.34.0, HA head pair head1/head2 (from Part 1) |
Node OS | Ubuntu 24.04.4 LTS, kernel 6.8.0-106-generic (from the BCM default-image) |
Network | Type 1; nodes PXE boot on internalnet 172.16.20.0/24 (VLAN20), one NIC each |
Shared storage | /home and /cm/shared on NFS 172.16.20.5 (mounted by the nodes) |
Primary disk | 500 GB on every compute node (single disk, BCM default partition layout) |
GPU | NVIDIA vGPU, profile nvidia_rtxa6000-12q (RTX A6000, 12 GB framebuffer) |
GPU guest driver | NVIDIA vGPU (GRID) guest driver 595.112.07, licensed through NLS |
Workload manager | Slurm (installed with BCM), GPU GRES autodetected with NVML |
Power control | None in the lab (no BMC); nodes powered on manually |
Node | Role | vCPU | RAM | Disk | GPU | Category | Software image | IP |
|---|---|---|---|---|---|---|---|---|
node001 | CPU | 8 | 12 GB | 500 GB | none | compute | compute-image | 172.16.20.21 |
node002 | GPU | 12 | 16 GB | 500 GB | RTXA6000-12Q | gpu | gpu-image | 172.16.20.22 |
node003 | GPU | 12 | 16 GB | 500 GB | RTXA6000-12Q | gpu | gpu-image | 172.16.20.23 |
With vGPU, the GPU VMs must have all guest memory reserved, so node002/node003 run with 16 GB fully reserved.
Part 1: Prepare the software images
Step 1: Update the default software image
Compute nodes boot from a software image stored on the head node under /cm/images. First bring the shipped default-image up to date inside its chroot.
- cm-chroot-sw-img /cm/images/default-image enters the image with /proc, /sys and /dev mounted.
- Run apt update and apt upgrade inside, then exit.

cm-chroot-sw-img /cm/images/default-image
apt update && apt -y upgrade
exit
Always leave the chroot with exit so the helper unmounts /proc, /sys and /dev from the image.
Step 2: Sync the image to the passive head node
In an HA cluster the passive head node also serves images. After any image change, push it to head2 with updateprovisioners.

A passive head shown as “out of date” for an image until this runs is normal.
Step 3: Clone images for CPU and GPU nodes
Keep the GPU driver only on GPU nodes by cloning two images from default-image: compute-image for CPU nodes and gpu-image for GPU nodes.
- softwareimage list shows all three images on kernel 6.8.0-106-generic.
- Wait until each clone finishes before you use it.

cmsh
softwareimage
clone default-image compute-image
commit
clone default-image gpu-image
commit
list
quit
cmsh -c “softwareimage updateprovisioners”
Part 2: Categories and node registration
Step 4: Create the compute and gpu categories
A category groups nodes that share settings: software image, roles, filesystem mounts, disk layout. Clone the default category twice and point each clone at its image.
cmsh
category
clone default compute
set softwareimage compute-image
commit
clone default gpu
set softwareimage gpu-image
commit
list
quit
The cloned categories keep the default disk setup, so each node uses its single 500 GB primary disk.
Step 5: Register each node by MAC and category
The installer pre-created node001 to node003 (IP offset .20 on internalnet). Set each node’s PXE NIC MAC address and category so BCM recognises it when it boots.
- Base View → Devices now shows the three nodes with category, image and IP, still Down.

cmsh
device use node001; set mac <node001-MAC>; set category compute; commit
device use node002; set mac <node002-MAC>; set category gpu; commit
device use node003; set mac <node003-MAC>; set category gpu; commit
quit
Production: on bare metal, identify nodes by switch port (set ethernetswitch on each node) or let the node-installer ask on the console; MAC entry is the simplest option for a small lab.
Part 3: Provision the nodes
Step 6: PXE boot the nodes
Power on the nodes with network boot first. They get an address from the head node and load the BCM PXE menu. The default entry AUTO runs the node-installer.

Step 7: Watch the node-installer progress
Base View → Devices tracks each node as it moves from Booting to Installing (partitioning the 500 GB disk and syncing the image) to Up.


Step 8: Confirm all nodes are Up
All three compute nodes are Up with their image and category.

cmsh -c “device status”
Step 9: Check node health
latesthealthdata shows the health checks for a node. On the GPU nodes the ManagedServicesOk check fails on cuda-dcgm because no NVIDIA driver is loaded yet. This clears once the driver is installed in Part 4.

cmsh -c “device use node002; latesthealthdata”
Part 4: NVIDIA vGPU guest driver in gpu-image
The driver is installed into gpu-image, not onto a running node, so every GPU node gets it on every boot.
Step 10: Get the guest driver
From the NVIDIA vGPU software package, take the Linux guest driver that matches the host vGPU manager: nvidia-linux-grid-595_595.112.07_amd64.deb for Ubuntu.

Step 11: Get the licence client token from NLS
In the NVIDIA Licensing Portal the entitlement shows the vGPU features (RTX Virtual Workstation, Virtual Applications). Create a licence server allocation, deploy a Cloud License Service (CLS) instance and download the client configuration token.

Step 12: Copy the files into the image and remove the shipped driver
Copy the .deb and the token into /cm/images/gpu-image/root/ and enter the chroot. The image ships the open 580 datacenter driver; vGPU needs the GRID guest driver, so purge the 580 packages first (a dry run apt -s purge lists what will go).

cp nvidia-linux-grid-595_595.112.07_amd64.deb /cm/images/gpu-image/root/
cp client_configuration_token_*.tok /cm/images/gpu-image/root/
cm-chroot-sw-img /cm/images/gpu-image
apt purge -y nvidia-driver-580-open nvidia-open-580 nvidia-dkms-580-open \
nvidia-kernel-source-580-open nvidia-kernel-common-580 nvidia-firmware-580 \
‘libnvidia-*-580’ xserver-xorg-video-nvidia-580 nvidia-persistenced \
nvidia-modprobe nvidia-settings
apt purge -y libnvidia-egl-gbm1 libnvidia-egl-wayland1 libnvidia-egl-xcb1 libnvidia-egl-xlib1
apt install -y linux-headers-6.8.0-149-generic dkms
The four libnvidia-egl-* packages conflict with the GRID package (file 15_nvidia_gbm.json), so remove them too. Install headers for every kernel in the image so DKMS can build the module.
Step 13: Install the guest driver and licence token
Still inside the chroot, install the GRID driver from the local .deb, then place the token and enable the licensing and persistence services.

apt install -y /root/nvidia-linux-grid-595_595.112.07_amd64.deb
dkms status
mkdir -p /etc/nvidia/ClientConfigToken
cp /root/client_configuration_token_*.tok /etc/nvidia/ClientConfigToken/
chmod 744 /etc/nvidia/ClientConfigToken/*.tok
cp /etc/nvidia/gridd.conf.template /etc/nvidia/gridd.conf
systemctl enable nvidia-gridd nvidia-persistenced
exit
dkms status must show the nvidia module built for each installed kernel. DCGM needs persistence mode, so keep nvidia-persistenced enabled.
Step 14: Rebuild the ramdisk, sync and reboot the GPU nodes
Regenerate the gpu-image ramdisk, push the image to head2, then reboot the GPU nodes so they pick up the new image.

cmsh -c “softwareimage use gpu-image; createramdisk –wait”
cmsh -c “softwareimage updateprovisioners”
cmsh -c “device reboot -n node002,node003”
Step 15: Verify the GPU on the node
nvidia-smi on node002 shows NVIDIA RTXA6000-12Q, 12288 MiB, driver 595.112.07, CUDA 13.2.

ssh node002 nvidia-smi
ssh node002 “nvidia-smi -q | grep -i -A2 license”
ssh node002 “nvidia-smi –query-gpu=persistence_mode –format=csv”
Step 16: GPU metrics in Base View
With the driver loaded, BCM collects GPU metrics through DCGM. Base View → Monitoring shows GPU utilization for the GPU nodes. Re-run latesthealthdata on node002 to confirm the cuda-dcgm check now passes.

Part 5: Slurm with GPUs
Step 17: Check Slurm: no nodes yet
sinfo shows 0 nodes. After install, the Slurm configuration overlays only include the default category, and our nodes are now in compute and gpu.

module load slurm
sinfo
cmsh -c “configurationoverlay; list”
Step 18: Attach the categories to the Slurm overlays
Add both categories to the submit overlay, the CPU category to slurm-client, and clone a slurm-client-gpu overlay for the GPU category with a higher priority.

cmsh
configurationoverlay
use slurm-submit; append categories compute gpu; commit
use slurm-client; append categories compute; commit
clone slurm-client slurm-client-gpu; set categories gpu; set priority 510; commit
list
Step 19: Add the GPU generic resource (GRES)
On the GPU overlay’s slurmclient role, define one autodetected gpu resource, and let Slurm autodetect GPUs with NVML.
use slurm-client-gpu
roles
use slurmclient
genericresources
add autodetected-gpus
set name gpu
set count 1
set addtogresconfig no
commit
home
wlm use slurm
set gpuautodetect nvml
commit
quit
Step 20: Check the Slurm cluster in Base View
Base View → HPC → Workload Management Clusters shows the Slurm servers (head1, head2) and the clients node001 to node003.

Step 21: Run CPU and GPU test jobs
- sinfo now shows the three nodes idle.
- scontrol show node node002 reports Gres=gpu:1 (CfgTRES cpu=12, mem=14390M).
- srun runs nvidia-smi on one GPU node, then on two nodes at once

sinfo
scontrol show node node002 | grep -i gres
srun -w node002 –gres=gpu:1 nvidia-smi -L
srun -N2 -w node002,node003 –gres=gpu:1 nvidia-smi -L
Step 22: Submit a batch job as a normal user
Run batch jobs as a user whose home is on the shared /home, so the output file is visible from the head node and every compute node.
cmsh -c “user add testuser; set password; commit”
su – testuser
module load slurm
cat > gpujob.sh <<‘EOF’
#!/bin/bash
#SBATCH –job-name=gputest
#SBATCH –gres=gpu:1
#SBATCH –output=%x-%j.out
nvidia-smi
EOF
sbatch gpujob.sh
squeue
cat gputest-*.out
root’s home (/root) is local to each node. A job submitted as root writes its output file on the compute node, not on the head node, so it looks as if the output is missing.
Step 23: Check jobs in Base View
Base View → HPC → Jobs lists the test jobs with their state (completed or failed), nodes and run time.

Step 24: Workload metrics
Base View → Monitoring shows the Slurm queue metrics for the default queue (defq): running, pending and completed jobs over time.

Summary
- Three compute nodes provisioned over PXE from two images: compute-image (CPU) and gpu-image (GPU), each on a single 500 GB disk.
- vGPU guest driver 595.112.07 and NLS licence token baked into gpu-image; nvidia-smi and DCGM metrics work.
- Slurm sees all three nodes; GPU nodes advertise gpu:1 GRES; interactive and batch GPU jobs complete.
Production: on bare-metal nodes use the NVIDIA datacenter driver (or the CUDA packages BCM provides) instead of the vGPU guest driver, configure BMCs so BCM can power-control nodes, and give GPU nodes a GPU partition or QOS limits to suit your users.
Part3 – Deploy Kubernetes with GPU Using NVIDIA BCM 11 – https://kdinesh.in/bcmk8s/
