This guide installs the first BCM 11 head node from the ISO, licenses the cluster for high availability, builds a second head node as a failover clone with cmha-setup, and moves /home and /cm/shared onto external NFS storage.
Lab note: the servers are VMware VMs, so there is no iDRAC/BMC. BMC configuration is skipped and failover is manual. On bare-metal servers configure the BMC so BCM can power-cycle (fence) a failed head node and fail over automatically.
Environment
Item | Value |
|---|---|
Cluster manager | NVIDIA Base Command Manager 11.34.0 (installer v11.34.0, UBUNTU2404) |
Operating system | Ubuntu Server 24.04.4 LTS, kernel 6.8.0-106-generic |
Head nodes | head1 and head2: 12 vCPU, 24 GB RAM, ~150-160 GB disk, 3 NICs |
NIC layout (both heads) | ens33 = internalnet (VLAN20), ens34 = externalnet (VLAN30), ens35 = failovernet (VLAN50) |
Network topology | Type 1: compute nodes on a private internal network behind the head nodes |
Shared storage | NFS server 172.16.20.5: /srv/bcm/home and /srv/bcm/cm-shared |
DNS / domain | DNS server 172.16.10.5, search domain gpu.lab |
Time | Asia/Singapore, NTP 0/1/2.pool.ntp.org |
Workload manager | Slurm |
Licence | BCM Advanced, HA licence (two head-node MACs), 5 nodes |
Power control | None in the lab (no BMC); manual failover |
Network | Subnet | Gateway | head1 | head2 | Shared IP (VIP) |
|---|---|---|---|---|---|
internalnet (VLAN20) | 172.16.20.0/24 | active head | 172.16.20.11 | 172.16.20.12 | 172.16.20.10 (ens33:ha) |
externalnet (VLAN30) | 172.16.30.0/24 | 172.16.30.1 | 172.16.30.11 | 172.16.30.12 | 172.16.30.10 (ens34:ha) |
failovernet (VLAN50) | 172.16.50.0/24 | none | 172.16.50.11 | 172.16.50.12 | n/a |
Other addresses | Value |
|---|---|
Compute nodes | node001-node003 = 172.16.20.21-23 (IP offset 0.0.0.20) |
BCM DHCP/PXE dynamic range | 172.16.20.100-172.16.20.199 |
NFS server | 172.16.20.5 |
Part 1: Install the first head node (head1)
Step 1: Boot the BCM ISO
Boot the server from the BCM 11 ISO and choose Start Base Command Manager Graphical Installer.

Step 2: Start the installer
Click Start installation.

Step 3: Accept the licence agreements
Tick I agree for the NVIDIA software licence and then for the Ubuntu Server 24.04 licence. Click Next after each.


Step 4: Kernel modules
Keep the default kernel module list. Click Next.

Step 5: Hardware info
Check that all hardware was detected: three NICs (ens33, ens34, ens35) and the install disk (/dev/sda).

Step 6: Installation source
Select the install media (/dev/sr0) and click Run media integrity check. Wait for “Validation … successful”.

Step 7: General cluster settings
Organization name: GPULAB
Time zone: Asia/Singapore; time servers: 0/1/2.pool.ntp.org
Nameservers: 172.16.10.5, 8.8.8.8; search domain: gpu.lab

Tip: use only the internal DNS server (172.16.10.5) as nameserver. With a public resolver in the list, BCM’s DNS can forward queries for the private gpu.lab domain to it and get “not found” answers.
Step 8: Workload manager
Select Slurm. The Slurm controller runs on the head node.

Step 9: Network topology
Select Type1: nodes on a private internal network, reachable from outside only through the head node.

Step 10: Head node settings
Hostname head1, set the administrator (root) password, hardware manufacturer Other.

Step 11: Compute nodes
Racks 1, number of nodes 3, start number 1, base name node, digits 3 (node001-node003).

Step 12: BMC configuration
Head node and compute nodes: No (the lab has no BMC).

Production: select Yes and enter the IPMI/iDRAC/iLO network so BCM can power-control every server.
Step 13: External network
Tab externalnet: DHCP off, base IP 172.16.30.0, netmask 255.255.255.0 (/24), gateway 172.16.30.1, domain gpu.lab, MTU 1500.

Step 14: Internal network
Tab internalnet: base IP 172.16.20.0 /24, dynamic range 172.16.20.100-172.16.20.199, domain eth.cluster, gateway empty (the head node becomes the nodes’ gateway).

Step 15: Head node network interfaces
ens33 on internalnet = 172.16.20.11; ens34 on externalnet = 172.16.30.11. The third NIC (ens35) is configured later by the HA setup.

Step 16: Compute node network interfaces
BOOTIF on internalnet with IP offset 0.0.0.20, so node001 gets 172.16.20.21.

Step 17: Disk layout
Select the install drive /dev/sda.

Step 18: Disk layout settings
The disk is smaller than the recommended 500 GB, so the installer defaults to One big partition for the head node and compute nodes. Keep it.

Step 19: Additional software
Leave CUDA unticked. GPU drivers are added later, in the GPU nodes’ software image.

Step 20: Summary
Review the summary (IPs, nameservers, time zone, Slurm, no BMC) and click Start.

Step 21: Installation progress
The installer works through 14 stages: partitioning, Ubuntu, BCM packages and the default software image.

Step 22: Reboot
When “Installation completed!” appears, click Yes to reboot into the installed head node.

Part 2: First login and Base View
Step 23: Head node landing page
Browse to the head node (here http://head1.gpu.lab). The landing page links to Base View, the API docs and the User Portal.

Step 24: Log in to Base View
Base View is at https://head1.gpu.lab:8081/base-view. Log in as admin with the administrator password.

Step 25: Cluster overview
The dashboard shows the cluster: one head node up, three compute nodes defined (down until they are provisioned).

Step 26: Health checks
Scroll down to Health Checks: all head-node checks pass (18/18).

Useful first checks from the shell:
ip -br addr
cmsh -c “device status”
apt update && apt -y upgrade
Part 3: Licence the cluster for HA
An HA licence must be installed before HA setup. It is bound to the MAC addresses of both head nodes, so prepare head2 first (same hardware and NIC order) and note the MAC of its first NIC.
Step 27: Run request-license
On head1 run request-license and enter the product key.

Step 28: Licence details
Enter country, state, locality, organisation (GPULAB), unit and cluster name (bcmlab). Then:
- MAC of primary head node (head1, ens33): accept the default
- Use a high-availability setup with 2 head nodes: y
- MAC of secondary head node: head2’s ens33 MAC
Submit the request (y), and when the licence is granted install it (y).

Step 29: Licence installed
BCM backs up the old licence, installs the new one and restarts CMDaemon.

Verify:
verify-license info | grep -i mac
cmsh -c “main licenseinfo”
Head nodes count as licensed nodes. Never change a head node’s MAC address after the licence is issued.
Part 4: Configure failover on head1
Step 30: Start cmha-setup
Run cmha-setup on head1 and choose Setup.

Step 31: Configure
Choose Configure (configure failover setup).

Step 32: Check licence MACs
The two MAC addresses from the licence are shown. Click CONTINUE.

Step 33: Shared internal IP
Shared internal interface ens33:ha, IP 172.16.20.10. Nodes always reach the active head on this IP.

Step 34: Shared external IP
Shared external interface ens34:ha, IP 172.16.30.10. Administrators log in on this IP.

Step 35: Secondary head hostname
Name: head2.

Step 36: Failover network
Name failovernet, base address 172.16.50.0 (the network address, not a host IP), netmask 255.255.255.0, domain failover.cluster.

Step 37: Failover interfaces
Primary ens35 = 172.16.50.11, secondary ens35 = 172.16.50.12.

Step 38: Secondary head IPs
head2: ens33 172.16.20.12, ens34 172.16.30.12.

Step 39: Clone information
Enter the MAC of head2’s interface used for cloning (ens33). CM args: –failover.

Step 40: Summary
Check the shared internal/external interfaces and the failover network, then EXIT the summary.

Step 41: Proceed
Confirm Yes to write the failover configuration.

Step 42: MySQL root password
Enter the MySQL root password (set during install).

Step 43: MySQL replication scope
Choose Yes to replicate all non-system databases (cmdaemon, slurm_acct_db and third-party databases) to head2.

Step 44: Failover initialised
All tasks finish OK: shared interfaces, failover network, head node clone image, secondary configuration, CMDaemon restart.

Step 45: Next instructions
The tool now asks you to boot head2 into the rescue environment and run /cm/cm-clone-install –failover.

Part 5: Clone the second head node (head2)
Step 46: Boot head2 into rescue
Power on head2 with network (PXE) boot from head1 and choose RESCUE in the PXE menu. The rescue environment logs in automatically.

Step 47: Run the clone install
Run /cm/cm-clone-install –failover and answer:
Network interface to use: ens33
Password of the head node (head1 root password)
Disk layout prompt: c (continue)

Step 48: head2 installed
After the clone finishes head2 reboots from its disk. Log in: only ens33 (172.16.20.12) is up until the setup is finalised.

Part 6: Finalize the HA setup
Step 49: Finalize
On head1: cmha-setup → Setup → Finalize.

Step 50: Confirm
Finalize updates head2’s MAC in the CMDaemon database and clones the MySQL databases. Click CONTINUE.

Step 51: MySQL password
Enter the MySQL root password again.

Step 52: Finalize complete
All steps report OK, including MySQL replication and the failover network check.

Step 53: Reboot head2
Click REBOOT to reboot the secondary head node.

Step 54: Check head2 interfaces
After the reboot all three interfaces on head2 are up: 172.16.20.12, 172.16.30.12, 172.16.50.12.

Step 55: Check failover status
Run cmha status on head1. failoverping, mysql, ping and status are OK in both directions; * marks the active head.

Part 7: Move /home and /cm/shared to NFS
Prerequisite: the NFS server exports both paths to the internal network, for example /srv/bcm/home 172.16.20.0/24(rw,sync,no_root_squash,no_subtree_check) and the same for /srv/bcm/cm-shared.
Step 56: Shared Storage menu
cmha-setup → Shared Storage (configure shared storage).

Step 57: Storage type
Select NAS (Network Attached Storage).

Step 58: Resources to share
Select both /home and /cm/shared.

Step 59: NAS parameters
NAS host 172.16.20.5, path to /home /srv/bcm/home, path to /cm/shared /srv/bcm/cm-shared.

Type the paths exactly as exported. A double slash or a missing hyphen makes the wizard report “not mountable on active”.
Step 60: Summary
Check the NAS host and paths, then EXIT to apply.

Step 61: Data copied and mounted
BCM copies the existing data to the NAS, mounts it, and replaces the old fsmounts/fsexports.

Part 8: Verify the HA cluster
Step 62: Mounts and failover status
On head1: both shares are mounted from 172.16.20.5 and cmha status is OK.

df -h | grep -E “/home|/cm/shared”
cmha status
Step 63: Data on the NFS server
On the NFS server the data is now present (here /home 52K, /cm/shared 1.7G).

Set the failover mode
Without BMC power control BCM cannot fence a failed head, so automatic failover is disabled and failover is done by hand:
cmsh
partition failover base
set disableautomaticfailover yes
commit
- Planned switchover: run cmha makeactive on the passive head.
- Head crash: power off the failed head, then run cmha makeactive on the survivor.
- Always log in through the shared IP 172.16.30.10; shared filesystems are mounted only on the active head.
- Updates: update the active head first; on Ubuntu update mysql before cmdaemon.
Production: configure BMC power control for both heads and keep automatic failover enabled.
Part 2 : Adding Compute Nodes and Testing Slurm on NVIDIA BCM 11 – https://kdinesh.in/hpc/
