Part 2 of the series: software images, categories, node provisioning, vGPU guest driver, Slurm with GPU GRES

This guide continues from “Deploying NVIDIA Base Command Manager 11 with High Availability”. The HA head-node pair is up with shared storage on NFS. Here we prepare separate software images for CPU and GPU nodes, provision three compute nodes over PXE, install the NVIDIA vGPU guest driver into the GPU image, and run CPU and GPU jobs through Slurm.

Part1- https://kdinesh.in/bcm/

Lab note: the compute nodes are VMware VMs (no iDRAC/BMC), so nodes are powered on by hand and BCM cannot power-cycle them. The GPU nodes use an NVIDIA vGPU profile, so they need the vGPU (GRID) guest driver and a license. On bare-metal GPU servers use the NVIDIA datacenter driver instead and no license is needed.

Environment

Item

Value

Cluster manager

NVIDIA Base Command Manager 11.34.0, HA head pair head1/head2 (from Part 1)

Node OS

Ubuntu 24.04.4 LTS, kernel 6.8.0-106-generic (from the BCM default-image)

Network

Type 1; nodes PXE boot on internalnet 172.16.20.0/24 (VLAN20), one NIC each

Shared storage

/home and /cm/shared on NFS 172.16.20.5 (mounted by the nodes)

Primary disk

500 GB on every compute node (single disk, BCM default partition layout)

GPU

NVIDIA vGPU, profile nvidia_rtxa6000-12q (RTX A6000, 12 GB framebuffer)

GPU guest driver

NVIDIA vGPU (GRID) guest driver 595.112.07, licensed through NLS

Workload manager

Slurm (installed with BCM), GPU GRES autodetected with NVML

Power control

None in the lab (no BMC); nodes powered on manually

Node

Role

vCPU

RAM

Disk

GPU

Category

Software image

IP

node001

CPU

8

12 GB

500 GB

none

compute

compute-image

172.16.20.21

node002

GPU

12

16 GB

500 GB

RTXA6000-12Q

gpu

gpu-image

172.16.20.22

node003

GPU

12

16 GB

500 GB

RTXA6000-12Q

gpu

gpu-image

172.16.20.23

With vGPU, the GPU VMs must have all guest memory reserved, so node002/node003 run with 16 GB fully reserved.

Part 1: Prepare the software images

Step 1: Update the default software image

Compute nodes boot from a software image stored on the head node under /cm/images. First bring the shipped default-image up to date inside its chroot.

  • cm-chroot-sw-img /cm/images/default-image enters the image with /proc, /sys and /dev mounted.
  • Run apt update and apt upgrade inside, then exit.

Screenshot 211237

cm-chroot-sw-img /cm/images/default-image

apt update && apt -y upgrade

exit

Always leave the chroot with exit so the helper unmounts /proc, /sys and /dev from the image.

Step 2: Sync the image to the passive head node

In an HA cluster the passive head node also serves images. After any image change, push it to head2 with updateprovisioners.

Screenshot 213042

A passive head shown as “out of date” for an image until this runs is normal.

Step 3: Clone images for CPU and GPU nodes

Keep the GPU driver only on GPU nodes by cloning two images from default-image: compute-image for CPU nodes and gpu-image for GPU nodes.

  • softwareimage list shows all three images on kernel 6.8.0-106-generic.
  • Wait until each clone finishes before you use it.


Screenshot 104129

cmsh

softwareimage

clone default-image compute-image

commit

clone default-image gpu-image

commit

list

quit

cmsh -c “softwareimage updateprovisioners”

Part 2: Categories and node registration

Step 4: Create the compute and gpu categories

A category groups nodes that share settings: software image, roles, filesystem mounts, disk layout. Clone the default category twice and point each clone at its image.

cmsh

category

clone default compute

set softwareimage compute-image

commit

clone default gpu

set softwareimage gpu-image

commit

list

quit

The cloned categories keep the default disk setup, so each node uses its single 500 GB primary disk.

Step 5: Register each node by MAC and category

The installer pre-created node001 to node003 (IP offset .20 on internalnet). Set each node’s PXE NIC MAC address and category so BCM recognises it when it boots.

  • Base View → Devices now shows the three nodes with category, image and IP, still Down.

Screenshot 113338

cmsh

device use node001; set mac <node001-MAC>; set category compute; commit

device use node002; set mac <node002-MAC>; set category gpu; commit

device use node003; set mac <node003-MAC>; set category gpu; commit

quit

Production: on bare metal, identify nodes by switch port (set ethernetswitch on each node) or let the node-installer ask on the console; MAC entry is the simplest option for a small lab.

Part 3: Provision the nodes

Step 6: PXE boot the nodes

Power on the nodes with network boot first. They get an address from the head node and load the BCM PXE menu. The default entry AUTO runs the node-installer.

Screenshot 113604

Step 7: Watch the node-installer progress

Base View → Devices tracks each node as it moves from Booting to Installing (partitioning the 500 GB disk and syncing the image) to Up.

Screenshot 113613

Screenshot 114215

Step 8: Confirm all nodes are Up

All three compute nodes are Up with their image and category.

Screenshot 114850

cmsh -c “device status”

Step 9: Check node health

latesthealthdata shows the health checks for a node. On the GPU nodes the ManagedServicesOk check fails on cuda-dcgm because no NVIDIA driver is loaded yet. This clears once the driver is installed in Part 4.

Screenshot 115225

cmsh -c “device use node002; latesthealthdata”

Part 4: NVIDIA vGPU guest driver in gpu-image

The driver is installed into gpu-image, not onto a running node, so every GPU node gets it on every boot.

Step 10: Get the guest driver

From the NVIDIA vGPU software package, take the Linux guest driver that matches the host vGPU manager: nvidia-linux-grid-595_595.112.07_amd64.deb for Ubuntu.

Screenshot 115549

Step 11: Get the licence client token from NLS

In the NVIDIA Licensing Portal the entitlement shows the vGPU features (RTX Virtual Workstation, Virtual Applications). Create a licence server allocation, deploy a Cloud License Service (CLS) instance and download the client configuration token.

Screenshot 120223

Step 12: Copy the files into the image and remove the shipped driver

Copy the .deb and the token into /cm/images/gpu-image/root/ and enter the chroot. The image ships the open 580 datacenter driver; vGPU needs the GRID guest driver, so purge the 580 packages first (a dry run apt -s purge lists what will go).

Screenshot 122948

cp nvidia-linux-grid-595_595.112.07_amd64.deb /cm/images/gpu-image/root/

cp client_configuration_token_*.tok /cm/images/gpu-image/root/

cm-chroot-sw-img /cm/images/gpu-image

apt purge -y nvidia-driver-580-open nvidia-open-580 nvidia-dkms-580-open \

nvidia-kernel-source-580-open nvidia-kernel-common-580 nvidia-firmware-580 \

‘libnvidia-*-580’ xserver-xorg-video-nvidia-580 nvidia-persistenced \

nvidia-modprobe nvidia-settings

apt purge -y libnvidia-egl-gbm1 libnvidia-egl-wayland1 libnvidia-egl-xcb1 libnvidia-egl-xlib1

apt install -y linux-headers-6.8.0-149-generic dkms

The four libnvidia-egl-* packages conflict with the GRID package (file 15_nvidia_gbm.json), so remove them too. Install headers for every kernel in the image so DKMS can build the module.

Step 13: Install the guest driver and licence token

Still inside the chroot, install the GRID driver from the local .deb, then place the token and enable the licensing and persistence services.

Screenshot 123201

apt install -y /root/nvidia-linux-grid-595_595.112.07_amd64.deb

dkms status

mkdir -p /etc/nvidia/ClientConfigToken

cp /root/client_configuration_token_*.tok /etc/nvidia/ClientConfigToken/

chmod 744 /etc/nvidia/ClientConfigToken/*.tok

cp /etc/nvidia/gridd.conf.template /etc/nvidia/gridd.conf

systemctl enable nvidia-gridd nvidia-persistenced

exit

dkms status must show the nvidia module built for each installed kernel. DCGM needs persistence mode, so keep nvidia-persistenced enabled.

Step 14: Rebuild the ramdisk, sync and reboot the GPU nodes

Regenerate the gpu-image ramdisk, push the image to head2, then reboot the GPU nodes so they pick up the new image.

Screenshot 123916

cmsh -c “softwareimage use gpu-image; createramdisk –wait”

cmsh -c “softwareimage updateprovisioners”

cmsh -c “device reboot -n node002,node003”

Step 15: Verify the GPU on the node

nvidia-smi on node002 shows NVIDIA RTXA6000-12Q, 12288 MiB, driver 595.112.07, CUDA 13.2.

Screenshot 125213

ssh node002 nvidia-smi

ssh node002 “nvidia-smi -q | grep -i -A2 license”

ssh node002 “nvidia-smi –query-gpu=persistence_mode –format=csv”

Step 16: GPU metrics in Base View

With the driver loaded, BCM collects GPU metrics through DCGM. Base View → Monitoring shows GPU utilization for the GPU nodes. Re-run latesthealthdata on node002 to confirm the cuda-dcgm check now passes.

Screenshot 125321

Part 5: Slurm with GPUs

Step 17: Check Slurm: no nodes yet

sinfo shows 0 nodes. After install, the Slurm configuration overlays only include the default category, and our nodes are now in compute and gpu.

Screenshot 125607

module load slurm

sinfo

cmsh -c “configurationoverlay; list”

Step 18: Attach the categories to the Slurm overlays

Add both categories to the submit overlay, the CPU category to slurm-client, and clone a slurm-client-gpu overlay for the GPU category with a higher priority.

Screenshot 125756

cmsh

configurationoverlay

use slurm-submit; append categories compute gpu; commit

use slurm-client; append categories compute; commit

clone slurm-client slurm-client-gpu; set categories gpu; set priority 510; commit

list

Step 19: Add the GPU generic resource (GRES)

On the GPU overlay’s slurmclient role, define one autodetected gpu resource, and let Slurm autodetect GPUs with NVML.

use slurm-client-gpu

roles

use slurmclient

genericresources

add autodetected-gpus

set name gpu

set count 1

set addtogresconfig no

commit

home

wlm use slurm

set gpuautodetect nvml

commit

quit

Step 20: Check the Slurm cluster in Base View

Base View → HPC → Workload Management Clusters shows the Slurm servers (head1, head2) and the clients node001 to node003.

Screenshot 125829

Step 21: Run CPU and GPU test jobs

  • sinfo now shows the three nodes idle.
  • scontrol show node node002 reports Gres=gpu:1 (CfgTRES cpu=12, mem=14390M).
  • srun runs nvidia-smi on one GPU node, then on two nodes at once


Screenshot 130216

sinfo

scontrol show node node002 | grep -i gres

srun -w node002 –gres=gpu:1 nvidia-smi -L

srun -N2 -w node002,node003 –gres=gpu:1 nvidia-smi -L

Step 22: Submit a batch job as a normal user

Run batch jobs as a user whose home is on the shared /home, so the output file is visible from the head node and every compute node.

cmsh -c “user add testuser; set password; commit”

su – testuser

module load slurm

cat > gpujob.sh <<‘EOF’

#!/bin/bash

#SBATCH –job-name=gputest

#SBATCH –gres=gpu:1

#SBATCH –output=%x-%j.out

nvidia-smi

EOF

sbatch gpujob.sh

squeue

cat gputest-*.out

root’s home (/root) is local to each node. A job submitted as root writes its output file on the compute node, not on the head node, so it looks as if the output is missing.

Step 23: Check jobs in Base View

Base View → HPC → Jobs lists the test jobs with their state (completed or failed), nodes and run time.

Screenshot 130344

Step 24: Workload metrics

Base View → Monitoring shows the Slurm queue metrics for the default queue (defq): running, pending and completed jobs over time.

Screenshot 130525

Summary

  • Three compute nodes provisioned over PXE from two images: compute-image (CPU) and gpu-image (GPU), each on a single 500 GB disk.
  • vGPU guest driver 595.112.07 and NLS licence token baked into gpu-image; nvidia-smi and DCGM metrics work.
  • Slurm sees all three nodes; GPU nodes advertise gpu:1 GRES; interactive and batch GPU jobs complete.


Production: on bare-metal nodes use the NVIDIA datacenter driver (or the CUDA packages BCM provides) instead of the vGPU guest driver, configure BMCs so BCM can power-control nodes, and give GPU nodes a GPU partition or QOS limits to suit your users.

Part3 – Deploy Kubernetes with GPU Using NVIDIA BCM 11 – https://kdinesh.in/bcmk8s/