Part 4 of the series: prerequisites, Run:ai 3.0 install, UI access, departments and projects, GPU workload and LLM inference

Part1:- https://kdinesh.in/bcm/

Part2:- https://kdinesh.in/hpc/

Part3:- https://kdinesh.in/bcmk8s/

This guide continues from “Deploy Kubernetes with GPU Using NVIDIA BCM 11”. The BCM Kubernetes cluster k8s is running with the GPU Operator, Kgateway and MetalLB. Here we prepare the prerequisites, install the open-source NVIDIA Run:ai 3.0 control plane and cluster with the guided installer, log in to the UI, create a department and a project, run a GPU workload, and serve an LLM with vLLM.

Lab note: everything runs on VMware VMs (no iDRAC/BMC) with vGPU (RTXA6000-12Q). Run:ai officially supports GPU passthrough/bare-metal GPUs only, so vGPU is used here for learning, and GPU fractions are not available. Production: use bare-metal GPU nodes or passthrough.

Note: before this deployment I increased the CPU and memory of the compute nodes to meet the Run:ai requirements (the Run:ai Nodes page shows 24 cores per compute node). Node disks stay at the 500 GB primary disk; the installer preflight checks ephemeral storage and passed.

Environment

Item

Value

Kubernetes

BCM cluster k8s, v1.36; control plane head1/head2; workers node001 (CPU), node002/node003 (vGPU RTXA6000-12Q)

Run:ai

NVIDIA Run:ai 3.0.20 (open source), guided runai-installer, control plane + cluster on the same Kubernetes

Control plane URL

https://runai.gpu.lab (namespace runai-backend)

Cluster

gpulab (namespace runai), workload routing by host

Inference domain

*.runai-inference.gpu.lab

Ingress

Gateway API: Gateway runai-gateway (kgateway) on MetalLB IP 172.16.20.241

MetalLB pool

runai-pool 172.16.20.241-242 (BCM gateway keeps 172.16.20.240)

TLS

Lab CA + one certificate for runai.gpu.lab, *.runai.gpu.lab, *.runai-inference.gpu.lab

DNS (172.16.10.5, zone gpu.lab)

runai, *.runai, *.runai-inference → 172.16.20.241

Prerequisites

GPU Operator from BCM (skip in installer), Kube-Prometheus Stack via Helm (skip in installer)

Installed by Run:ai

Knative Serving, Kubeflow Training Operator, MPI Operator, LeaderWorkerSet

Organisation

Department gpulab (GPU quota 2), project test-a (namespace runai-test-a, quota 2)

Part 1: Prerequisites

Step 1: Exclude the Run:ai namespaces from the Kyverno policies

BCM installs the Kyverno Pod Security policies in enforce mode. Add monitoring, runai and runai-backend to each policy’s exclusions through the kyverno-policies Helm values (BCM Containerization Manual, policy exclusions).

helm get values -n kyverno kyverno-policies | tee /root/kyverno-policies-values.orig.yaml

# edit a copy: /root/kyverno-policies-values.yaml

# add monitoring, runai, runai-backend under each policyExclude entry

helm upgrade kyverno-policies kyverno/kyverno-policies –version 3.8.2 -n kyverno \

-f /root/kyverno-policies-values.yaml

kubectl get clusterpolicy

After any later “Modify operators” run of cm-kubernetes-setup, check the values and re-apply if needed.

Step 2: Install Kube-Prometheus Stack

Run:ai needs Prometheus but does not install it. The version BCM offers is older than the Run:ai 3.0 minimum, so install it with Helm.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \

–version 93.3.0 -n monitoring –create-namespace –set grafana.enabled=false

kubectl get pods -n monitoring -o wide

Step 3: Add a MetalLB address pool for Run:ai

The BCM gateway pool (172.16.20.240) stays dedicated to BCM. Create a separate pool, not auto-assigned, for the Run:ai gateway.

apiVersion: metallb.io/v1beta1

kind: IPAddressPool

metadata:

name: runai-pool

namespace: metallb-system

spec:

addresses:

– 172.16.20.241-172.16.20.242

autoAssign: false

—

apiVersion: metallb.io/v1beta1

kind: L2Advertisement

metadata:

name: runai-l2

namespace: metallb-system

spec:

ipAddressPools:

– runai-pool

kubectl apply -f /root/runai-metallb.yaml

Use free IPs outside the BCM DHCP range (172.16.20.100-199).

Step 4: Create the DNS records

On the DNS server (zone gpu.lab) point the control plane name and both wildcards to the Run:ai gateway IP, then check resolution from the head node.

runai.gpu.lab A 172.16.20.241

*.runai.gpu.lab A 172.16.20.241

*.runai-inference.gpu.lab A 172.16.20.241

for n in runai.gpu.lab test.runai.gpu.lab test.runai-inference.gpu.lab; do

echo “$n -> $(dig +short $n @172.16.10.5)”; done

getent hosts runai.gpu.lab

If a name was looked up before the record existed, flush the cached negative answer on both heads: rndc flush; resolvectl flush-caches.

Step 5: Create the TLS certificate

Create a lab CA and one server certificate whose SAN covers the control plane name, the workload wildcard and the inference wildcard.

mkdir -p /cm/shared/runai/certs && cd /cm/shared/runai/certs

openssl req -x509 -new -nodes -newkey rsa:4096 -sha256 -days 3650 \

-keyout ca.key -out ca.crt -subj “/O=GPULAB/CN=GPULAB Root CA” \

-addext “basicConstraints=critical,CA:TRUE” -addext “keyUsage=critical,keyCertSign,cRLSign”

openssl req -new -nodes -newkey rsa:2048 -sha256 -keyout runai.key -out runai.csr \

-subj “/O=GPULAB/CN=runai.gpu.lab”

cat > runai.ext <<‘EOF’

basicConstraints=CA:FALSE

keyUsage=critical,digitalSignature,keyEncipherment

extendedKeyUsage=serverAuth

subjectAltName=DNS:runai.gpu.lab,DNS:*.runai.gpu.lab,DNS:*.runai-inference.gpu.lab

EOF

openssl x509 -req -in runai.csr -CA ca.crt -CAkey ca.key -CAcreateserial \

-days 825 -sha256 -extfile runai.ext -out runai.crt

cat runai.crt ca.crt > fullchain.pem

chmod 600 ca.key runai.key

openssl verify -CAfile ca.crt runai.crt

Production: use a certificate from your corporate CA.

Step 6: Create the Run:ai Gateway

With externalConnection.type: gatewayApi the installer needs an existing Gateway. Create the TLS secrets and a kgateway Gateway with three HTTPS listeners, pinned to 172.16.20.241.

kubectl create namespace runai

cd /cm/shared/runai/certs

for s in runai-cluster-domain-tls-secret runai-cluster-domain-star-tls-secret runai-cluster-inference-tls-secret; do

kubectl create secret tls $s -n runai –cert=fullchain.pem –key=runai.key

done

apiVersion: gateway.networking.k8s.io/v1

kind: Gateway

metadata:

name: runai-gateway

namespace: runai

spec:

gatewayClassName: kgateway

infrastructure:

annotations:

metallb.io/loadBalancerIPs: 172.16.20.241

listeners:

– name: https

protocol: HTTPS

port: 443

hostname: runai.gpu.lab

tls:

mode: Terminate

certificateRefs:

– name: runai-cluster-domain-tls-secret

allowedRoutes:

namespaces:

from: All

– name: https-wildcard

protocol: HTTPS

port: 443

hostname: “*.runai.gpu.lab”

tls:

mode: Terminate

certificateRefs:

– name: runai-cluster-domain-star-tls-secret

allowedRoutes:

namespaces:

from: All

– name: https-inference

protocol: HTTPS

port: 443

hostname: “*.runai-inference.gpu.lab”

tls:

mode: Terminate

certificateRefs:

– name: runai-cluster-inference-tls-secret

allowedRoutes:

namespaces:

from: All

kubectl apply -f /root/runai-gateway.yaml

kubectl get gateway,svc,pods -n runai

curl -sv –cacert /cm/shared/runai/certs/ca.crt https://runai.gpu.lab 2>&1 | grep -Ei “verify|HTTP/”

Expected: address 172.16.20.241, PROGRAMMED True, “SSL certificate verify ok” and HTTP 404 (no routes yet). allowedRoutes: from: All is needed because Run:ai and Knative create HTTPRoutes in other namespaces.

Part 2: Install Run:ai

Step 7: Download the installer and write the configuration

Get the open-source Run:ai 3.0.20 installer and describe the install in config.yaml: full installation (control plane + cluster), Gateway API, our certificate, and skip the GPU Operator and Prometheus because they already exist.

mkdir -p /root/runai && cd /root/runai

curl -fLO https://github.com/dsx-ai-factory/run-ai/releases/download/3.0.20/runai-installer-linux-amd64

chmod +x runai-installer-linux-amd64

install:

mode: full_installation

connectivity: connected

externalConnection:

type: gatewayApi

gatewayApi:

name: runai-gateway

namespace: runai

controlPlane:

enabled: true

domain: runai.gpu.lab

cluster:

enabled: true

name: gpulab

workloadRouting: host

inference:

wildcardDomain: “*.runai-inference.gpu.lab”

tls:

certPath: /cm/shared/runai/certs/fullchain.pem

keyPath: /cm/shared/runai/certs/runai.key

caChainPath: /cm/shared/runai/certs/ca.crt

generateSelfSigned: false

registry:

type: ngc

prerequisites:

gpuOperator:

mode: skip

prometheus:

mode: skip

networkOperator:

mode: skip

features:

knative:

mode: install

kubeflow:

mode: install

mpiOperator:

mode: install

lws:

mode: install

Step 8: Preflight and dry run

Run the preflight checks and a dry run before installing. The dry run lists the plan: control plane in runai-backend, cluster gpulab in runai, and the Knative, Kubeflow Training, MPI and LWS features.

./runai-installer-linux-amd64 install –config config.yaml –kubeconfig /root/.kube/config-k8s –preflight-only

./runai-installer-linux-amd64 install –config config.yaml –kubeconfig /root/.kube/config-k8s –dry-run

Step 9: Install Run:ai

Run the install with longer step timeouts. It registers the cluster gpulab, installs the five optional features and prints how to read the admin user.

Screenshot 161953

./runai-installer-linux-amd64 install –config config.yaml –kubeconfig /root/.kube/config-k8s \

–step-timeout 40m –execution-timeout 120m –logs-dir /root/runai/logs

The local-path StorageClass puts PostgreSQL on /cm/shared (NFS) and the first image pulls are large, so the default step timeout can be too short. If a step still times out: abort, kubectl delete pvc -n runai-backend –all, and install again.

Step 10: Verify the installation

All control-plane pods in runai-backend are Running, HTTPRoutes for runai.gpu.lab and *.runai-inference.gpu.lab are attached, and the UI answers HTTP 200.

Screenshot 162053

kubectl get pods -n runai-backend

kubectl get pods -n runai | grep -vE “Running|Completed”

kubectl get httproute -A

curl -s -o /dev/null -w “%{http_code}\n” –cacert /cm/shared/runai/certs/ca.crt https://runai.gpu.lab

kubectl -n runai-backend get configmap runai-backend-tenants-manager -o jsonpath='{.data.ADMIN_USERNAME}’; echo

kubectl -n runai-backend get secret runai-backend-gen-tenants-manager -o jsonpath='{.data.ADMIN_PASSWORD}’ | base64 -d; echo

Part 3: Access the Run:ai UI

Step 11: Log in

Import the lab CA (ca.crt) into the client’s trusted root store, open https://runai.gpu.lab and sign in with the admin user from the previous step.

Screenshot 202934

Change the admin password after the first login.

Step 12: First-login wizard

The welcome wizard offers SSO, an email server and a first research team. For the lab, Skip each page and finish with Start exploring.

Screenshot 203214

Screenshot 203227

Screenshot 203251

Production: configure SSO (SAML 2.0 or OpenID Connect) and an SMTP server for invitations and notifications.

Step 13: Overview dashboard

The dashboard shows 2/2 GPU nodes and 2/2 GPU devices ready.

Screenshot 203326

Step 14: Nodes and cluster status

  • Resources → Nodes lists all five nodes; node002 and node003 each have NVIDIA-RTXA6000-12Q, 1/1 GPU.
  • Resources → Clusters shows gpulab on Run:ai 3.0.20 with status Missing prerequisites: the container toolkit from GPU Operator 26.3.0 is older than Run:ai needs for GPU fractions. Full GPUs work normally.

Screenshot 203334

Screenshot 203401

Part 4: Department and project

Step 15: Create the department

Organization → Departments → New department: cluster gpulab, name gpulab, then Create department & continue.

Screenshot 204005

Step 16: Department GPU quota

After install the built-in default department holds all GPUs, so the quota warning appears. Open Node pool overview, set the default department to 0, then give gpulab a quota of 2 GPUs. Keep over-quota allowed (weight 2, rank Medium Low), then close on the access rules page.

Screenshot 204145

Screenshot 204242

Screenshot 204317

Step 17: Create the project

Organization → Projects → New project: scope department gpulab, name test-a (namespace runai-test-a), GPU quota 2, over-quota allowed. Scheduling and access rules are optional.

Screenshot 204603

Screenshot 204611

Screenshot 204616

A project’s department cannot be changed later, so check the scope before creating it.

Step 18: Project ready

Project test-a is Ready under department gpulab with GPU quota 2.00.

Screenshot 204632

Part 5: Run a GPU workload

Step 19: Create a workload

Workload manager → Workloads → New workload. Choose cluster gpulab, project test-a, Start from scratch, and name the workload.

Screenshot 204908

Screenshot 204916

Workload types: Workspace is interactive and not preemptible by default; Training is a batch job that runs to completion and can use over-quota GPUs. Either works for this test.

Step 20: Image and compute resources

Image kubeflow/pytorch-dist-mnist:latest (a PyTorch MNIST trainer), 1 GPU device, then create the workload.

Screenshot 205143

Screenshot 205148

Step 21: Scheduling

The workload is Pending while the image is pulled. Event history shows the runai-scheduler assigning the pod to node002 in node pool default, then the binder binding it.

Screenshot 205206

Screenshot 205941

The image is about 10.7 GB; the first pull took around 30 minutes. Pre-pull large images on the GPU nodes with crictl pull to save time.

Step 22: Training on the GPU

The pod runs on node002, the logs show the training loss falling epoch by epoch, and nvidia-smi on node002 shows the python process using the RTXA6000-12Q.

Screenshot 212624

kubectl get pods -n runai-test-a -o wide

kubectl logs -n runai-test-a <pod> –tail=10

ssh node002 nvidia-smi

Part 6: LLM inference with vLLM

Step 23: Pre-pull the vLLM image

The vLLM image is about 9.7 GB. Pull it on both GPU nodes first, otherwise Knative’s start deadline can expire during the download.

for n in node002 node003; do

ssh $n “crictl –runtime-endpoint unix:///run/containerd/containerd.sock pull docker.io/vllm/vllm-openai:latest”

done

Step 24: Create the inference workload

Workload manager → Workloads → New workload → Inference → Standard → Start from scratch → inference server vLLM, name chat-qwen.

Screenshot 213817

Step 25: Model and GPU

Model Qwen/Qwen2.5-1.5B-Instruct from Hugging Face, 1 GPU device per replica.

Screenshot 221834

Step 26: Endpoint and runtime settings

Serving endpoint External (public access). Under Runtime settings the model argument and HF_HOME are already set; append the memory arguments so vLLM fits on the 12 GB vGPU, then Create inference.

Screenshot 221946

–model Qwen/Qwen2.5-1.5B-Instruct –gpu-memory-utilization 0.80 –max-model-len 4096

The 12Q vGPU profile reserves part of the framebuffer, so vLLM’s default memory target does not fit. On a full bare-metal GPU the defaults work.

Step 27: Inference running

chat-qwen is Running with 1/1 pods and 1 GPU allocated; the training workload shows Completed.

Screenshot 000707

Step 28: Test the model endpoint

Call the OpenAI-compatible API through the inference URL. The model answers the question.

Screenshot 001214

URL=https://chat-qwenrunai-test-a.runai-inference.gpu.lab

curl -s –cacert /cm/shared/runai/certs/ca.crt $URL/v1/models

curl -s –cacert /cm/shared/runai/certs/ca.crt $URL/v1/chat/completions \

-H “Content-Type: application/json” \

-d ‘{“model”:”Qwen/Qwen2.5-1.5B-Instruct”,”messages”:[{“role”:”user”,”content”:”Explain in 3 sentences what a GPU cluster is used for.”}]}’ \

| python3 -c ‘import sys,json; print(json.load(sys.stdin)[“choices”][0][“message”][“content”])’

Request path: DNS → 172.16.20.241 (MetalLB) → runai-gateway (TLS) → HTTPRoute → Knative → vLLM pod on node002. The URL is an API, not a web page: open /v1/models in a browser; chat needs a POST.

Summary

  • Prerequisites on the BCM Kubernetes cluster: Kyverno exclusions, Kube-Prometheus Stack, MetalLB pool, DNS, TLS and a Gateway API gateway.
  • Run:ai 3.0.20 control plane and cluster gpulab installed with the guided installer; UI at https://runai.gpu.lab.
  • Department gpulab and project test-a with GPU quota; a GPU training job and a vLLM inference endpoint ran on node002.

Production: bare-metal or passthrough GPUs (GPU fractions need a supported toolkit and driver), a CSI storage class for the control plane database, SSO, a corporate certificate, and a registry mirror for large images.