Part 3 of the series: cm-kubernetes-setup wizard, HA control plane, GPU Operator, MetalLB, verification

This guide continues from “Adding Compute Nodes and Testing Slurm on NVIDIA BCM 11”. The HA head pair and three provisioned compute nodes (one CPU, two vGPU) are in place. Here we run the BCM cm-kubernetes-setup wizard with a two-head control plane and three-member etcd, install the NVIDIA GPU Operator, and verify that a pod can use a GPU.

Part1 :- https://kdinesh.in/bcm/

Part2:- https://kdinesh.in/hpc/

Lab note: everything runs on VMware VMs (no iDRAC/BMC). The GPU nodes use vGPU, and the vGPU guest driver is already baked into gpu-image (see the previous article), so the GPU Operator is installed with its driver container disabled. On bare-metal GPU servers you can either keep the driver in the BCM image (as here) or let the GPU Operator deploy the datacenter driver.

Environment

Item

Value

Cluster manager

NVIDIA Base Command Manager 11.34.0, HA head pair head1/head2

Kubernetes

v1.36.3 (wizard: v1.36, patch level latest), cluster name k8s, domain cluster.local

Control plane

head1, head2

Workers

categories compute and gpu: node001 (CPU), node002 and node003 (vGPU RTXA6000-12Q)

etcd

head1, head2, node001 (odd number of members), spool /var/lib/etcd

Node disks

every compute node runs on its 500 GB primary disk; no extra disk

Networks

nodes on internalnet 172.16.20.0/24; pod network 172.29.0.0/16; service network 10.150.0.0/16

API server

exposed externally as https://bcmlab.gpu.lab:10443 (API server proxy port 10443)

CNI

Calico (Tigera Operator)

Policy

Kyverno policy engine + Kyverno policies (Pod Security Standards)

Operators

NVIDIA GPU Operator v26.3.0 (driver disabled), Kgateway, Kubernetes Metrics Server, Kube State Metrics, MetalLB

Gateway / LB

MetalLB L2, BCM gateway IP 172.16.20.240, gateway proxy on the heads, port 443

Storage

Local Path StorageClass (default) at /cm/shared/apps/kubernetes/k8s/var/volumes

Note: the lab licence covers 5 nodes, so before this deployment I moved the three compute nodes from Slurm to Kubernetes (removed the compute and gpu categories from the Slurm configuration overlays).

Part 1: Run the cm-kubernetes-setup wizard

Step 1: Start the wizard

Run cm-kubernetes-setup on the active head node and choose Deploy (Kubernetes installation wizard).

Screenshot 182813

Screenshot 182825

Step 2: Choose the Kubernetes version

Select Kubernetes v1.36 (versions marked * are certified for NVIDIA AI Enterprise), then patch level latest.

Screenshot 182846

Screenshot 182904

Step 3: Registry mirror

Optional Docker Hub registry mirror. Left empty in the lab.

Screenshot 182916

Production: set a mirror or pull-through cache to avoid Docker Hub rate limits, or use a local registry in air-gapped sites.

Step 4: Basic cluster values

  • Cluster name k8s, domain cluster.local.
  • External FQDN bcmlab.gpu.lab (this name goes into the API server certificate).
  • Service network 10.150.0.0/16 and pod network 172.29.0.0/16; neither may overlap any BCM network.



Screenshot 183232

Create the DNS record for the external FQDN before clients use the API server.

Step 5: Expose the API server

Choose yes to expose the Kubernetes API server on the external network, so kubectl can be used from outside the cluster.

Screenshot 183238

Step 6: Internal network for Kubernetes

Select internalnet, the network the compute nodes boot and talk on.

Screenshot 183256

Step 7: Control-plane nodes

Select head1 and head2, so the control plane is highly available like the BCM head pair.

Screenshot 183423

Step 8: Worker nodes

Select the compute and gpu categories as workers. On the next screen leave the head nodes unselected as individual workers, so user pods do not run on the heads.

Screenshot 183521

Screenshot 183529

Step 9: etcd nodes

etcd needs an odd number of members for quorum. Select head1, head2 and node001.

Screenshot 183623

BCM treats an etcd member as a data node. Production: run etcd on three dedicated or control-plane nodes with fast local disks.

Step 10: Main component values

Keep the API server proxy port 10443 and the etcd spool directory /var/lib/etcd.

Screenshot 183632

Step 11: Network plugin (CNI)

Select Tigera Operator – Calico (recommended).

Screenshot 183702

Step 12: Kyverno policy engine and policies

Install Kyverno and the Kyverno policies based on the Kubernetes Pod Security Standards. The wizard adds exclusions for the operators it installs.

Screenshot 183708

Screenshot 183717

Step 13: Choose the operators

Select NVIDIA GPU Operator, Kgateway, Kubernetes Metrics Server, Kubernetes State Metrics and MetalLB.

Screenshot 184820

Other operators (Knative, Kubeflow Training, MPI, LeaderWorkerSet, Prometheus) can be added later; Run:ai installs the ones it needs.

Step 14: Kyverno namespace exclusions

Namespaces to exclude from each cluster-wide policy (host volumes, host namespaces, host ports, privileged containers, capabilities). Kept at the defaults.

Screenshot 183923

Step 15: NVIDIA GPU Operator version

Select v26.3.0 (certified for NVIDIA AI Enterprise, marked *). Leave the custom Helm values file empty to get the basic options on the next screen.

Screenshot 184018

Screenshot 184026

Step 16: GPU Operator options

  • toolkit.enabled: yes, deploy the NVIDIA Container Toolkit as a container.
  • driver.enabled: no, the vGPU guest driver is already in gpu-image.
  • nfd.enabled: yes, Node Feature Discovery labels the GPU nodes.


Screenshot 184039

Step 17: MetalLB mode and gateway IP

Select L2 (ARP advertisement), then give the BCM gateway a free IP on internalnet: 172.16.20.240.

Screenshot 184112

Screenshot 184209

Choose an address outside the BCM DHCP range (172.16.20.100-199) and outside the node IPs.

Step 18: MetalLB address pools

No extra address pools are needed for the BCM gateway. Move to Ok with Tab.

Screenshot 184248

Step 19: Gateway proxy

Choose yes to expose Kubernetes gateways on the external network through the gateway proxy on the head nodes, port 443.

Screenshot 184901

Screenshot 184908

Step 20: Permission Manager

Permission Manager is needed only if non-root users will use the Kubernetes cluster.

Screenshot 184920

Step 21: Storage class

Enable Local path as the default StorageClass, storing data under /cm/shared/apps/kubernetes/k8s/var/volumes (on the NFS-backed /cm/shared).

Screenshot 184929

Screenshot 184939

Production: use a real CSI driver (NFS CSI, Ceph, vendor storage) for persistent volumes.

Step 22: Save the configuration and deploy

Choose Save config & deploy and save the answers to /root/cm-kubernetes-setup.conf.

Screenshot 185013

Screenshot 185024

cp /root/cm-kubernetes-setup.conf /cm/shared/

Keep a copy of the saved answers on shared storage so the same settings are available from either head node.

Step 23: Deployment runs

The wizard checks the cluster, then installs packages, etcd, the control plane, workers, Calico, operators, MetalLB and the gateway. In this lab it took about 21 minutes and ended with Kubernetes Setup finished, API server https://bcmlab.gpu.lab:10443.

Screenshot 185058

Screenshot 191245

Part 2: Verify

Step 24: Nodes and GPU Operator

  • All five nodes are Ready on v1.36.3: head1/head2 as control-plane, node001-003 as workers.
  • The GPU Operator validators run on node002 and node003 and the CUDA validator pods complete.


Screenshot 191340

module load kubernetes

kubectl get nodes

kubectl get pods -A | grep -v Running

kubectl get pod -n gpu-operator -l app=nvidia-operator-validator -o wide

Step 25: Kubernetes cluster in Base View

Base View → Containers → Kubernetes Clusters → k8s shows the version, namespaces, nodes and pods.

Screenshot 210520

Step 26: Run a GPU test pod

A pod that requests nvidia.com/gpu: 1 is scheduled on a GPU node (node002) and nvidia-smi prints the RTXA6000-12Q with 12288 MiB, driver 595.112.07, CUDA 13.2.

Screenshot 210750

apiVersion: v1

kind: Pod

metadata:

name: gpu-test

spec:

restartPolicy: Never

containers:

– name: cuda

image: nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04

command: [“nvidia-smi”]

resources:

limits:

nvidia.com/gpu: 1

kubectl apply -f gpu-test.yaml

kubectl get pod gpu-test -o wide

kubectl logs gpu-test

kubectl delete pod gpu-test

Step 27: Check etcd health

The kubernetes module sets the ETCDCTL_* variables, so etcdctl runs without endpoint or certificate flags.

kubectl get –raw=’/readyz?verbose’ | grep etcd

etcdctl member list

etcdctl endpoint health

etcdctl endpoint status

If the BCM Etcd health check on a head reports https://0.0.0.0:2379 unhealthy while etcdctl endpoint health is fine, it is cosmetic: the check uses the listen URL (all interfaces). Do not change the listen URL.

Summary

  • Kubernetes v1.36.3 deployed by cm-kubernetes-setup: HA control plane on head1/head2, three-member etcd, Calico, Kyverno.
  • GPU Operator v26.3.0 uses the vGPU driver already in gpu-image; GPU pods get nvidia.com/gpu.
  • MetalLB L2 and the BCM gateway on 172.16.20.240, gateway proxy on the heads, Local Path default StorageClass.


Production: dedicated etcd/control-plane nodes, a CSI storage backend, a registry mirror, BMC power control, and separate node pools for Slurm and Kubernetes.

Part4:- Deploy and Test NVIDIA Run:ai on BCM 11 Kubernetes – https://kdinesh.in/runai/