This guide installs the first BCM 11 head node from the ISO, licenses the cluster for high availability, builds a second head node as a failover clone with cmha-setup, and moves /home and /cm/shared onto external NFS storage.

Lab note: the servers are VMware VMs, so there is no iDRAC/BMC. BMC configuration is skipped and failover is manual. On bare-metal servers configure the BMC so BCM can power-cycle (fence) a failed head node and fail over automatically.

Environment

Item

Value

Cluster manager

NVIDIA Base Command Manager 11.34.0 (installer v11.34.0, UBUNTU2404)

Operating system

Ubuntu Server 24.04.4 LTS, kernel 6.8.0-106-generic

Head nodes

head1 and head2: 12 vCPU, 24 GB RAM, ~150-160 GB disk, 3 NICs

NIC layout (both heads)

ens33 = internalnet (VLAN20), ens34 = externalnet (VLAN30), ens35 = failovernet (VLAN50)

Network topology

Type 1: compute nodes on a private internal network behind the head nodes

Shared storage

NFS server 172.16.20.5: /srv/bcm/home and /srv/bcm/cm-shared

DNS / domain

DNS server 172.16.10.5, search domain gpu.lab

Time

Asia/Singapore, NTP 0/1/2.pool.ntp.org

Workload manager

Slurm

Licence

BCM Advanced, HA licence (two head-node MACs), 5 nodes

Power control

None in the lab (no BMC); manual failover

Network

Subnet

Gateway

head1

head2

Shared IP (VIP)

internalnet (VLAN20)

172.16.20.0/24

active head

172.16.20.11

172.16.20.12

172.16.20.10 (ens33:ha)

externalnet (VLAN30)

172.16.30.0/24

172.16.30.1

172.16.30.11

172.16.30.12

172.16.30.10 (ens34:ha)

failovernet (VLAN50)

172.16.50.0/24

none

172.16.50.11

172.16.50.12

n/a

Other addresses

Value

Compute nodes

node001-node003 = 172.16.20.21-23 (IP offset 0.0.0.20)

BCM DHCP/PXE dynamic range

172.16.20.100-172.16.20.199

NFS server

172.16.20.5

Part 1: Install the first head node (head1)

Step 1: Boot the BCM ISO

Boot the server from the BCM 11 ISO and choose Start Base Command Manager Graphical Installer.

Screenshot 164541

Step 2: Start the installer

Click Start installation.

Screenshot 164628

Step 3: Accept the licence agreements

Tick I agree for the NVIDIA software licence and then for the Ubuntu Server 24.04 licence. Click Next after each.

Screenshot 164640

Screenshot 164656

Step 4: Kernel modules

Keep the default kernel module list. Click Next.

Screenshot 164714

Step 5: Hardware info

Check that all hardware was detected: three NICs (ens33, ens34, ens35) and the install disk (/dev/sda).

Screenshot 164759

Step 6: Installation source

Select the install media (/dev/sr0) and click Run media integrity check. Wait for “Validation … successful”.

Screenshot 164928

Step 7: General cluster settings


  • Organization name: GPULAB

  • Time zone: Asia/Singapore; time servers: 0/1/2.pool.ntp.org

  • Nameservers: 172.16.10.5, 8.8.8.8; search domain: gpu.lab

Screenshot 165248

Tip: use only the internal DNS server (172.16.10.5) as nameserver. With a public resolver in the list, BCM’s DNS can forward queries for the private gpu.lab domain to it and get “not found” answers.

Step 8: Workload manager

Select Slurm. The Slurm controller runs on the head node.

Screenshot 165309

Step 9: Network topology

Select Type1: nodes on a private internal network, reachable from outside only through the head node.

Screenshot 165358

Step 10: Head node settings

Hostname head1, set the administrator (root) password, hardware manufacturer Other.

Screenshot 165651

Step 11: Compute nodes

Racks 1, number of nodes 3, start number 1, base name node, digits 3 (node001-node003).

Screenshot 165824

Step 12: BMC configuration

Head node and compute nodes: No (the lab has no BMC).

Screenshot 165847

Production: select Yes and enter the IPMI/iDRAC/iLO network so BCM can power-control every server.

Step 13: External network

Tab externalnet: DHCP off, base IP 172.16.30.0, netmask 255.255.255.0 (/24), gateway 172.16.30.1, domain gpu.lab, MTU 1500.

Screenshot 170055

Step 14: Internal network

Tab internalnet: base IP 172.16.20.0 /24, dynamic range 172.16.20.100-172.16.20.199, domain eth.cluster, gateway empty (the head node becomes the nodes’ gateway).

Screenshot 170338

Step 15: Head node network interfaces

ens33 on internalnet = 172.16.20.11; ens34 on externalnet = 172.16.30.11. The third NIC (ens35) is configured later by the HA setup.

Screenshot 170444

Step 16: Compute node network interfaces

BOOTIF on internalnet with IP offset 0.0.0.20, so node001 gets 172.16.20.21.

Screenshot 170912

Step 17: Disk layout

Select the install drive /dev/sda.

Screenshot 170920

Step 18: Disk layout settings

The disk is smaller than the recommended 500 GB, so the installer defaults to One big partition for the head node and compute nodes. Keep it.

Screenshot 170930

Step 19: Additional software

Leave CUDA unticked. GPU drivers are added later, in the GPU nodes’ software image.

Screenshot 171039

Step 20: Summary

Review the summary (IPs, nameservers, time zone, Slurm, no BMC) and click Start.

Screenshot 171129

Step 21: Installation progress

The installer works through 14 stages: partitioning, Ubuntu, BCM packages and the default software image.

Screenshot 171226

Step 22: Reboot

When “Installation completed!” appears, click Yes to reboot into the installed head node.

Screenshot 173027

Part 2: First login and Base View

Step 23: Head node landing page

Browse to the head node (here http://head1.gpu.lab). The landing page links to Base View, the API docs and the User Portal.

Screenshot 173422

Step 24: Log in to Base View

Base View is at https://head1.gpu.lab:8081/base-view. Log in as admin with the administrator password.

Screenshot 173456

Step 25: Cluster overview

The dashboard shows the cluster: one head node up, three compute nodes defined (down until they are provisioned).

Screenshot 175142

Step 26: Health checks

Scroll down to Health Checks: all head-node checks pass (18/18).

Screenshot 182216

Useful first checks from the shell:

ip -br addr

cmsh -c “device status”

apt update && apt -y upgrade

Part 3: Licence the cluster for HA

An HA licence must be installed before HA setup. It is bound to the MAC addresses of both head nodes, so prepare head2 first (same hardware and NIC order) and note the MAC of its first NIC.

Step 27: Run request-license

On head1 run request-license and enter the product key.

Screenshot 183825

Step 28: Licence details

Enter country, state, locality, organisation (GPULAB), unit and cluster name (bcmlab). Then:

  • MAC of primary head node (head1, ens33): accept the default
  • Use a high-availability setup with 2 head nodes: y
  • MAC of secondary head node: head2’s ens33 MAC

Submit the request (y), and when the licence is granted install it (y).

Screenshot 184700

Step 29: Licence installed

BCM backs up the old licence, installs the new one and restarts CMDaemon.

Screenshot 184830

Verify:

verify-license info | grep -i mac

cmsh -c “main licenseinfo”

Head nodes count as licensed nodes. Never change a head node’s MAC address after the licence is issued.

Part 4: Configure failover on head1

Step 30: Start cmha-setup

Run cmha-setup on head1 and choose Setup.

Screenshot 185222

Step 31: Configure

Choose Configure (configure failover setup).

Screenshot 185327

Step 32: Check licence MACs

The two MAC addresses from the licence are shown. Click CONTINUE.

Screenshot 185407

Step 33: Shared internal IP

Shared internal interface ens33:ha, IP 172.16.20.10. Nodes always reach the active head on this IP.

Screenshot 185515

Step 34: Shared external IP

Shared external interface ens34:ha, IP 172.16.30.10. Administrators log in on this IP.

Screenshot 185817

Step 35: Secondary head hostname

Name: head2.

Screenshot 185830

Step 36: Failover network

Name failovernet, base address 172.16.50.0 (the network address, not a host IP), netmask 255.255.255.0, domain failover.cluster.

Screenshot 190022

Step 37: Failover interfaces

Primary ens35 = 172.16.50.11, secondary ens35 = 172.16.50.12.

Screenshot 190149

Step 38: Secondary head IPs

head2: ens33 172.16.20.12, ens34 172.16.30.12.

Screenshot 190321

Step 39: Clone information

Enter the MAC of head2’s interface used for cloning (ens33). CM args: –failover.

Screenshot 190350

Step 40: Summary

Check the shared internal/external interfaces and the failover network, then EXIT the summary.

Screenshot 190507

Step 41: Proceed

Confirm Yes to write the failover configuration.

Screenshot 190544

Step 42: MySQL root password

Enter the MySQL root password (set during install).

Screenshot 190611

Step 43: MySQL replication scope

Choose Yes to replicate all non-system databases (cmdaemon, slurm_acct_db and third-party databases) to head2.

Screenshot 190658

Step 44: Failover initialised

All tasks finish OK: shared interfaces, failover network, head node clone image, secondary configuration, CMDaemon restart.

Screenshot 191406

Step 45: Next instructions

The tool now asks you to boot head2 into the rescue environment and run /cm/cm-clone-install –failover.

Screenshot 191436

Part 5: Clone the second head node (head2)

Step 46: Boot head2 into rescue

Power on head2 with network (PXE) boot from head1 and choose RESCUE in the PXE menu. The rescue environment logs in automatically.

Screenshot 191635

Step 47: Run the clone install

Run /cm/cm-clone-install –failover and answer:


  • Network interface to use: ens33

  • Password of the head node (head1 root password)

  • Disk layout prompt: c (continue)

Screenshot 191952

Step 48: head2 installed

After the clone finishes head2 reboots from its disk. Log in: only ens33 (172.16.20.12) is up until the setup is finalised.

Screenshot 193506

Part 6: Finalize the HA setup

Step 49: Finalize

On head1: cmha-setup → Setup → Finalize.

Screenshot 193845

Step 50: Confirm

Finalize updates head2’s MAC in the CMDaemon database and clones the MySQL databases. Click CONTINUE.

Screenshot 193859

Step 51: MySQL password

Enter the MySQL root password again.

Screenshot 193905

Step 52: Finalize complete

All steps report OK, including MySQL replication and the failover network check.

Screenshot 194442

Step 53: Reboot head2

Click REBOOT to reboot the secondary head node.

Screenshot 194449

Step 54: Check head2 interfaces

After the reboot all three interfaces on head2 are up: 172.16.20.12, 172.16.30.12, 172.16.50.12.

Screenshot 194609

Step 55: Check failover status

Run cmha status on head1. failoverping, mysql, ping and status are OK in both directions; * marks the active head.

Screenshot 194726

Part 7: Move /home and /cm/shared to NFS

Prerequisite: the NFS server exports both paths to the internal network, for example /srv/bcm/home 172.16.20.0/24(rw,sync,no_root_squash,no_subtree_check) and the same for /srv/bcm/cm-shared.

Step 56: Shared Storage menu

cmha-setup → Shared Storage (configure shared storage).

Screenshot 195822

Step 57: Storage type

Select NAS (Network Attached Storage).

Screenshot 195827

Step 58: Resources to share

Select both /home and /cm/shared.

Screenshot 195834

Step 59: NAS parameters

NAS host 172.16.20.5, path to /home /srv/bcm/home, path to /cm/shared /srv/bcm/cm-shared.

Screenshot 200348

Type the paths exactly as exported. A double slash or a missing hyphen makes the wizard report “not mountable on active”.

Step 60: Summary

Check the NAS host and paths, then EXIT to apply.

Screenshot 200402

Step 61: Data copied and mounted

BCM copies the existing data to the NAS, mounts it, and replaces the old fsmounts/fsexports.

Screenshot 202252

Part 8: Verify the HA cluster

Step 62: Mounts and failover status

On head1: both shares are mounted from 172.16.20.5 and cmha status is OK.

Screenshot 202420

df -h | grep -E “/home|/cm/shared”

cmha status

Step 63: Data on the NFS server

On the NFS server the data is now present (here /home 52K, /cm/shared 1.7G).

Screenshot 202439

Set the failover mode

Without BMC power control BCM cannot fence a failed head, so automatic failover is disabled and failover is done by hand:

cmsh

partition failover base

set disableautomaticfailover yes

commit

  • Planned switchover: run cmha makeactive on the passive head.
  • Head crash: power off the failed head, then run cmha makeactive on the survivor.
  • Always log in through the shared IP 172.16.30.10; shared filesystems are mounted only on the active head.
  • Updates: update the active head first; on Ubuntu update mysql before cmdaemon.

Production: configure BMC power control for both heads and keep automatic failover enabled.

Part 2 : Adding Compute Nodes and Testing Slurm on NVIDIA BCM 11 – https://kdinesh.in/hpc/