Part 3 of the series: cm-kubernetes-setup wizard, HA control plane, GPU Operator, MetalLB, verification
This guide continues from “Adding Compute Nodes and Testing Slurm on NVIDIA BCM 11”. The HA head pair and three provisioned compute nodes (one CPU, two vGPU) are in place. Here we run the BCM cm-kubernetes-setup wizard with a two-head control plane and three-member etcd, install the NVIDIA GPU Operator, and verify that a pod can use a GPU.
Part1 :- https://kdinesh.in/bcm/
Part2:- https://kdinesh.in/hpc/
Lab note: everything runs on VMware VMs (no iDRAC/BMC). The GPU nodes use vGPU, and the vGPU guest driver is already baked into gpu-image (see the previous article), so the GPU Operator is installed with its driver container disabled. On bare-metal GPU servers you can either keep the driver in the BCM image (as here) or let the GPU Operator deploy the datacenter driver.
Environment
Item | Value |
|---|---|
Cluster manager | NVIDIA Base Command Manager 11.34.0, HA head pair head1/head2 |
Kubernetes | v1.36.3 (wizard: v1.36, patch level latest), cluster name k8s, domain cluster.local |
Control plane | head1, head2 |
Workers | categories compute and gpu: node001 (CPU), node002 and node003 (vGPU RTXA6000-12Q) |
etcd | head1, head2, node001 (odd number of members), spool /var/lib/etcd |
Node disks | every compute node runs on its 500 GB primary disk; no extra disk |
Networks | nodes on internalnet 172.16.20.0/24; pod network 172.29.0.0/16; service network 10.150.0.0/16 |
API server | exposed externally as https://bcmlab.gpu.lab:10443 (API server proxy port 10443) |
CNI | Calico (Tigera Operator) |
Policy | Kyverno policy engine + Kyverno policies (Pod Security Standards) |
Operators | NVIDIA GPU Operator v26.3.0 (driver disabled), Kgateway, Kubernetes Metrics Server, Kube State Metrics, MetalLB |
Gateway / LB | MetalLB L2, BCM gateway IP 172.16.20.240, gateway proxy on the heads, port 443 |
Storage | Local Path StorageClass (default) at /cm/shared/apps/kubernetes/k8s/var/volumes |
Note: the lab licence covers 5 nodes, so before this deployment I moved the three compute nodes from Slurm to Kubernetes (removed the compute and gpu categories from the Slurm configuration overlays).
Part 1: Run the cm-kubernetes-setup wizard
Step 1: Start the wizard
Run cm-kubernetes-setup on the active head node and choose Deploy (Kubernetes installation wizard).


Step 2: Choose the Kubernetes version
Select Kubernetes v1.36 (versions marked * are certified for NVIDIA AI Enterprise), then patch level latest.


Step 3: Registry mirror
Optional Docker Hub registry mirror. Left empty in the lab.

Production: set a mirror or pull-through cache to avoid Docker Hub rate limits, or use a local registry in air-gapped sites.
Step 4: Basic cluster values
- Cluster name k8s, domain cluster.local.
- External FQDN bcmlab.gpu.lab (this name goes into the API server certificate).
- Service network 10.150.0.0/16 and pod network 172.29.0.0/16; neither may overlap any BCM network.

Create the DNS record for the external FQDN before clients use the API server.
Step 5: Expose the API server
Choose yes to expose the Kubernetes API server on the external network, so kubectl can be used from outside the cluster.

Step 6: Internal network for Kubernetes
Select internalnet, the network the compute nodes boot and talk on.

Step 7: Control-plane nodes
Select head1 and head2, so the control plane is highly available like the BCM head pair.

Step 8: Worker nodes
Select the compute and gpu categories as workers. On the next screen leave the head nodes unselected as individual workers, so user pods do not run on the heads.


Step 9: etcd nodes
etcd needs an odd number of members for quorum. Select head1, head2 and node001.

BCM treats an etcd member as a data node. Production: run etcd on three dedicated or control-plane nodes with fast local disks.
Step 10: Main component values
Keep the API server proxy port 10443 and the etcd spool directory /var/lib/etcd.

Step 11: Network plugin (CNI)
Select Tigera Operator – Calico (recommended).

Step 12: Kyverno policy engine and policies
Install Kyverno and the Kyverno policies based on the Kubernetes Pod Security Standards. The wizard adds exclusions for the operators it installs.


Step 13: Choose the operators
Select NVIDIA GPU Operator, Kgateway, Kubernetes Metrics Server, Kubernetes State Metrics and MetalLB.

Other operators (Knative, Kubeflow Training, MPI, LeaderWorkerSet, Prometheus) can be added later; Run:ai installs the ones it needs.
Step 14: Kyverno namespace exclusions
Namespaces to exclude from each cluster-wide policy (host volumes, host namespaces, host ports, privileged containers, capabilities). Kept at the defaults.

Step 15: NVIDIA GPU Operator version
Select v26.3.0 (certified for NVIDIA AI Enterprise, marked *). Leave the custom Helm values file empty to get the basic options on the next screen.


Step 16: GPU Operator options
- toolkit.enabled: yes, deploy the NVIDIA Container Toolkit as a container.
- driver.enabled: no, the vGPU guest driver is already in gpu-image.
- nfd.enabled: yes, Node Feature Discovery labels the GPU nodes.

Step 17: MetalLB mode and gateway IP
Select L2 (ARP advertisement), then give the BCM gateway a free IP on internalnet: 172.16.20.240.


Choose an address outside the BCM DHCP range (172.16.20.100-199) and outside the node IPs.
Step 18: MetalLB address pools
No extra address pools are needed for the BCM gateway. Move to Ok with Tab.

Step 19: Gateway proxy
Choose yes to expose Kubernetes gateways on the external network through the gateway proxy on the head nodes, port 443.


Step 20: Permission Manager
Permission Manager is needed only if non-root users will use the Kubernetes cluster.

Step 21: Storage class
Enable Local path as the default StorageClass, storing data under /cm/shared/apps/kubernetes/k8s/var/volumes (on the NFS-backed /cm/shared).


Production: use a real CSI driver (NFS CSI, Ceph, vendor storage) for persistent volumes.
Step 22: Save the configuration and deploy
Choose Save config & deploy and save the answers to /root/cm-kubernetes-setup.conf.


cp /root/cm-kubernetes-setup.conf /cm/shared/
Keep a copy of the saved answers on shared storage so the same settings are available from either head node.
Step 23: Deployment runs
The wizard checks the cluster, then installs packages, etcd, the control plane, workers, Calico, operators, MetalLB and the gateway. In this lab it took about 21 minutes and ended with Kubernetes Setup finished, API server https://bcmlab.gpu.lab:10443.


Part 2: Verify
Step 24: Nodes and GPU Operator
- All five nodes are Ready on v1.36.3: head1/head2 as control-plane, node001-003 as workers.
- The GPU Operator validators run on node002 and node003 and the CUDA validator pods complete.

module load kubernetes
kubectl get nodes
kubectl get pods -A | grep -v Running
kubectl get pod -n gpu-operator -l app=nvidia-operator-validator -o wide
Step 25: Kubernetes cluster in Base View
Base View → Containers → Kubernetes Clusters → k8s shows the version, namespaces, nodes and pods.

Step 26: Run a GPU test pod
A pod that requests nvidia.com/gpu: 1 is scheduled on a GPU node (node002) and nvidia-smi prints the RTXA6000-12Q with 12288 MiB, driver 595.112.07, CUDA 13.2.

apiVersion: v1
kind: Pod
metadata:
name: gpu-test
spec:
restartPolicy: Never
containers:
– name: cuda
image: nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
command: [“nvidia-smi”]
resources:
limits:
nvidia.com/gpu: 1
kubectl apply -f gpu-test.yaml
kubectl get pod gpu-test -o wide
kubectl logs gpu-test
kubectl delete pod gpu-test
Step 27: Check etcd health
The kubernetes module sets the ETCDCTL_* variables, so etcdctl runs without endpoint or certificate flags.
kubectl get –raw=’/readyz?verbose’ | grep etcd
etcdctl member list
etcdctl endpoint health
etcdctl endpoint status
If the BCM Etcd health check on a head reports https://0.0.0.0:2379 unhealthy while etcdctl endpoint health is fine, it is cosmetic: the check uses the listen URL (all interfaces). Do not change the listen URL.
Summary
- Kubernetes v1.36.3 deployed by cm-kubernetes-setup: HA control plane on head1/head2, three-member etcd, Calico, Kyverno.
- GPU Operator v26.3.0 uses the vGPU driver already in gpu-image; GPU pods get nvidia.com/gpu.
- MetalLB L2 and the BCM gateway on 172.16.20.240, gateway proxy on the heads, Local Path default StorageClass.
Production: dedicated etcd/control-plane nodes, a CSI storage backend, a registry mirror, BMC power control, and separate node pools for Slurm and Kubernetes.
Part4:- Deploy and Test NVIDIA Run:ai on BCM 11 Kubernetes – https://kdinesh.in/runai/
