Part 4 of the series: prerequisites, Run:ai 3.0 install, UI access, departments and projects, GPU workload and LLM inference
Part1:- https://kdinesh.in/bcm/
Part2:- https://kdinesh.in/hpc/
Part3:- https://kdinesh.in/bcmk8s/
This guide continues from “Deploy Kubernetes with GPU Using NVIDIA BCM 11”. The BCM Kubernetes cluster k8s is running with the GPU Operator, Kgateway and MetalLB. Here we prepare the prerequisites, install the open-source NVIDIA Run:ai 3.0 control plane and cluster with the guided installer, log in to the UI, create a department and a project, run a GPU workload, and serve an LLM with vLLM.
Lab note: everything runs on VMware VMs (no iDRAC/BMC) with vGPU (RTXA6000-12Q). Run:ai officially supports GPU passthrough/bare-metal GPUs only, so vGPU is used here for learning, and GPU fractions are not available. Production: use bare-metal GPU nodes or passthrough.
Note: before this deployment I increased the CPU and memory of the compute nodes to meet the Run:ai requirements (the Run:ai Nodes page shows 24 cores per compute node). Node disks stay at the 500 GB primary disk; the installer preflight checks ephemeral storage and passed.
Environment
Item | Value |
|---|---|
Kubernetes | BCM cluster k8s, v1.36; control plane head1/head2; workers node001 (CPU), node002/node003 (vGPU RTXA6000-12Q) |
Run:ai | NVIDIA Run:ai 3.0.20 (open source), guided runai-installer, control plane + cluster on the same Kubernetes |
Control plane URL | https://runai.gpu.lab (namespace runai-backend) |
Cluster | gpulab (namespace runai), workload routing by host |
Inference domain | *.runai-inference.gpu.lab |
Ingress | Gateway API: Gateway runai-gateway (kgateway) on MetalLB IP 172.16.20.241 |
MetalLB pool | runai-pool 172.16.20.241-242 (BCM gateway keeps 172.16.20.240) |
TLS | Lab CA + one certificate for runai.gpu.lab, *.runai.gpu.lab, *.runai-inference.gpu.lab |
DNS (172.16.10.5, zone gpu.lab) | runai, *.runai, *.runai-inference → 172.16.20.241 |
Prerequisites | GPU Operator from BCM (skip in installer), Kube-Prometheus Stack via Helm (skip in installer) |
Installed by Run:ai | Knative Serving, Kubeflow Training Operator, MPI Operator, LeaderWorkerSet |
Organisation | Department gpulab (GPU quota 2), project test-a (namespace runai-test-a, quota 2) |
Part 1: Prerequisites
Step 1: Exclude the Run:ai namespaces from the Kyverno policies
BCM installs the Kyverno Pod Security policies in enforce mode. Add monitoring, runai and runai-backend to each policy’s exclusions through the kyverno-policies Helm values (BCM Containerization Manual, policy exclusions).
helm get values -n kyverno kyverno-policies | tee /root/kyverno-policies-values.orig.yaml
# edit a copy: /root/kyverno-policies-values.yaml
# add monitoring, runai, runai-backend under each policyExclude entry
helm upgrade kyverno-policies kyverno/kyverno-policies –version 3.8.2 -n kyverno \
-f /root/kyverno-policies-values.yaml
kubectl get clusterpolicy
After any later “Modify operators” run of cm-kubernetes-setup, check the values and re-apply if needed.
Step 2: Install Kube-Prometheus Stack
Run:ai needs Prometheus but does not install it. The version BCM offers is older than the Run:ai 3.0 minimum, so install it with Helm.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack \
–version 93.3.0 -n monitoring –create-namespace –set grafana.enabled=false
kubectl get pods -n monitoring -o wide
Step 3: Add a MetalLB address pool for Run:ai
The BCM gateway pool (172.16.20.240) stays dedicated to BCM. Create a separate pool, not auto-assigned, for the Run:ai gateway.
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: runai-pool
namespace: metallb-system
spec:
addresses:
– 172.16.20.241-172.16.20.242
autoAssign: false
—
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: runai-l2
namespace: metallb-system
spec:
ipAddressPools:
– runai-pool
kubectl apply -f /root/runai-metallb.yaml
Use free IPs outside the BCM DHCP range (172.16.20.100-199).
Step 4: Create the DNS records
On the DNS server (zone gpu.lab) point the control plane name and both wildcards to the Run:ai gateway IP, then check resolution from the head node.
runai.gpu.lab A 172.16.20.241
*.runai.gpu.lab A 172.16.20.241
*.runai-inference.gpu.lab A 172.16.20.241
for n in runai.gpu.lab test.runai.gpu.lab test.runai-inference.gpu.lab; do
echo “$n -> $(dig +short $n @172.16.10.5)”; done
getent hosts runai.gpu.lab
If a name was looked up before the record existed, flush the cached negative answer on both heads: rndc flush; resolvectl flush-caches.
Step 5: Create the TLS certificate
Create a lab CA and one server certificate whose SAN covers the control plane name, the workload wildcard and the inference wildcard.
mkdir -p /cm/shared/runai/certs && cd /cm/shared/runai/certs
openssl req -x509 -new -nodes -newkey rsa:4096 -sha256 -days 3650 \
-keyout ca.key -out ca.crt -subj “/O=GPULAB/CN=GPULAB Root CA” \
-addext “basicConstraints=critical,CA:TRUE” -addext “keyUsage=critical,keyCertSign,cRLSign”
openssl req -new -nodes -newkey rsa:2048 -sha256 -keyout runai.key -out runai.csr \
-subj “/O=GPULAB/CN=runai.gpu.lab”
cat > runai.ext <<‘EOF’
basicConstraints=CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:runai.gpu.lab,DNS:*.runai.gpu.lab,DNS:*.runai-inference.gpu.lab
EOF
openssl x509 -req -in runai.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-days 825 -sha256 -extfile runai.ext -out runai.crt
cat runai.crt ca.crt > fullchain.pem
chmod 600 ca.key runai.key
openssl verify -CAfile ca.crt runai.crt
Production: use a certificate from your corporate CA.
Step 6: Create the Run:ai Gateway
With externalConnection.type: gatewayApi the installer needs an existing Gateway. Create the TLS secrets and a kgateway Gateway with three HTTPS listeners, pinned to 172.16.20.241.
kubectl create namespace runai
cd /cm/shared/runai/certs
for s in runai-cluster-domain-tls-secret runai-cluster-domain-star-tls-secret runai-cluster-inference-tls-secret; do
kubectl create secret tls $s -n runai –cert=fullchain.pem –key=runai.key
done
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: runai-gateway
namespace: runai
spec:
gatewayClassName: kgateway
infrastructure:
annotations:
metallb.io/loadBalancerIPs: 172.16.20.241
listeners:
– name: https
protocol: HTTPS
port: 443
hostname: runai.gpu.lab
tls:
mode: Terminate
certificateRefs:
– name: runai-cluster-domain-tls-secret
allowedRoutes:
namespaces:
from: All
– name: https-wildcard
protocol: HTTPS
port: 443
hostname: “*.runai.gpu.lab”
tls:
mode: Terminate
certificateRefs:
– name: runai-cluster-domain-star-tls-secret
allowedRoutes:
namespaces:
from: All
– name: https-inference
protocol: HTTPS
port: 443
hostname: “*.runai-inference.gpu.lab”
tls:
mode: Terminate
certificateRefs:
– name: runai-cluster-inference-tls-secret
allowedRoutes:
namespaces:
from: All
kubectl apply -f /root/runai-gateway.yaml
kubectl get gateway,svc,pods -n runai
curl -sv –cacert /cm/shared/runai/certs/ca.crt https://runai.gpu.lab 2>&1 | grep -Ei “verify|HTTP/”
Expected: address 172.16.20.241, PROGRAMMED True, “SSL certificate verify ok” and HTTP 404 (no routes yet). allowedRoutes: from: All is needed because Run:ai and Knative create HTTPRoutes in other namespaces.
Part 2: Install Run:ai
Step 7: Download the installer and write the configuration
Get the open-source Run:ai 3.0.20 installer and describe the install in config.yaml: full installation (control plane + cluster), Gateway API, our certificate, and skip the GPU Operator and Prometheus because they already exist.
mkdir -p /root/runai && cd /root/runai
curl -fLO https://github.com/dsx-ai-factory/run-ai/releases/download/3.0.20/runai-installer-linux-amd64
chmod +x runai-installer-linux-amd64
install:
mode: full_installation
connectivity: connected
externalConnection:
type: gatewayApi
gatewayApi:
name: runai-gateway
namespace: runai
controlPlane:
enabled: true
domain: runai.gpu.lab
cluster:
enabled: true
name: gpulab
workloadRouting: host
inference:
wildcardDomain: “*.runai-inference.gpu.lab”
tls:
certPath: /cm/shared/runai/certs/fullchain.pem
keyPath: /cm/shared/runai/certs/runai.key
caChainPath: /cm/shared/runai/certs/ca.crt
generateSelfSigned: false
registry:
type: ngc
prerequisites:
gpuOperator:
mode: skip
prometheus:
mode: skip
networkOperator:
mode: skip
features:
knative:
mode: install
kubeflow:
mode: install
mpiOperator:
mode: install
lws:
mode: install
Step 8: Preflight and dry run
Run the preflight checks and a dry run before installing. The dry run lists the plan: control plane in runai-backend, cluster gpulab in runai, and the Knative, Kubeflow Training, MPI and LWS features.
./runai-installer-linux-amd64 install –config config.yaml –kubeconfig /root/.kube/config-k8s –preflight-only
./runai-installer-linux-amd64 install –config config.yaml –kubeconfig /root/.kube/config-k8s –dry-run
Step 9: Install Run:ai
Run the install with longer step timeouts. It registers the cluster gpulab, installs the five optional features and prints how to read the admin user.

./runai-installer-linux-amd64 install –config config.yaml –kubeconfig /root/.kube/config-k8s \
–step-timeout 40m –execution-timeout 120m –logs-dir /root/runai/logs
The local-path StorageClass puts PostgreSQL on /cm/shared (NFS) and the first image pulls are large, so the default step timeout can be too short. If a step still times out: abort, kubectl delete pvc -n runai-backend –all, and install again.
Step 10: Verify the installation
All control-plane pods in runai-backend are Running, HTTPRoutes for runai.gpu.lab and *.runai-inference.gpu.lab are attached, and the UI answers HTTP 200.

kubectl get pods -n runai-backend
kubectl get pods -n runai | grep -vE “Running|Completed”
kubectl get httproute -A
curl -s -o /dev/null -w “%{http_code}\n” –cacert /cm/shared/runai/certs/ca.crt https://runai.gpu.lab
kubectl -n runai-backend get configmap runai-backend-tenants-manager -o jsonpath='{.data.ADMIN_USERNAME}’; echo
kubectl -n runai-backend get secret runai-backend-gen-tenants-manager -o jsonpath='{.data.ADMIN_PASSWORD}’ | base64 -d; echo
Part 3: Access the Run:ai UI
Step 11: Log in
Import the lab CA (ca.crt) into the client’s trusted root store, open https://runai.gpu.lab and sign in with the admin user from the previous step.

Change the admin password after the first login.
Step 12: First-login wizard
The welcome wizard offers SSO, an email server and a first research team. For the lab, Skip each page and finish with Start exploring.



Production: configure SSO (SAML 2.0 or OpenID Connect) and an SMTP server for invitations and notifications.
Step 13: Overview dashboard
The dashboard shows 2/2 GPU nodes and 2/2 GPU devices ready.

Step 14: Nodes and cluster status
- Resources → Nodes lists all five nodes; node002 and node003 each have NVIDIA-RTXA6000-12Q, 1/1 GPU.
- Resources → Clusters shows gpulab on Run:ai 3.0.20 with status Missing prerequisites: the container toolkit from GPU Operator 26.3.0 is older than Run:ai needs for GPU fractions. Full GPUs work normally.


Part 4: Department and project
Step 15: Create the department
Organization → Departments → New department: cluster gpulab, name gpulab, then Create department & continue.

Step 16: Department GPU quota
After install the built-in default department holds all GPUs, so the quota warning appears. Open Node pool overview, set the default department to 0, then give gpulab a quota of 2 GPUs. Keep over-quota allowed (weight 2, rank Medium Low), then close on the access rules page.



Step 17: Create the project
Organization → Projects → New project: scope department gpulab, name test-a (namespace runai-test-a), GPU quota 2, over-quota allowed. Scheduling and access rules are optional.



A project’s department cannot be changed later, so check the scope before creating it.
Step 18: Project ready
Project test-a is Ready under department gpulab with GPU quota 2.00.

Part 5: Run a GPU workload
Step 19: Create a workload
Workload manager → Workloads → New workload. Choose cluster gpulab, project test-a, Start from scratch, and name the workload.


Workload types: Workspace is interactive and not preemptible by default; Training is a batch job that runs to completion and can use over-quota GPUs. Either works for this test.
Step 20: Image and compute resources
Image kubeflow/pytorch-dist-mnist:latest (a PyTorch MNIST trainer), 1 GPU device, then create the workload.


Step 21: Scheduling
The workload is Pending while the image is pulled. Event history shows the runai-scheduler assigning the pod to node002 in node pool default, then the binder binding it.


The image is about 10.7 GB; the first pull took around 30 minutes. Pre-pull large images on the GPU nodes with crictl pull to save time.
Step 22: Training on the GPU
The pod runs on node002, the logs show the training loss falling epoch by epoch, and nvidia-smi on node002 shows the python process using the RTXA6000-12Q.

kubectl get pods -n runai-test-a -o wide
kubectl logs -n runai-test-a <pod> –tail=10
ssh node002 nvidia-smi
Part 6: LLM inference with vLLM
Step 23: Pre-pull the vLLM image
The vLLM image is about 9.7 GB. Pull it on both GPU nodes first, otherwise Knative’s start deadline can expire during the download.
for n in node002 node003; do
ssh $n “crictl –runtime-endpoint unix:///run/containerd/containerd.sock pull docker.io/vllm/vllm-openai:latest”
done
Step 24: Create the inference workload
Workload manager → Workloads → New workload → Inference → Standard → Start from scratch → inference server vLLM, name chat-qwen.

Step 25: Model and GPU
Model Qwen/Qwen2.5-1.5B-Instruct from Hugging Face, 1 GPU device per replica.

Step 26: Endpoint and runtime settings
Serving endpoint External (public access). Under Runtime settings the model argument and HF_HOME are already set; append the memory arguments so vLLM fits on the 12 GB vGPU, then Create inference.

–model Qwen/Qwen2.5-1.5B-Instruct –gpu-memory-utilization 0.80 –max-model-len 4096
The 12Q vGPU profile reserves part of the framebuffer, so vLLM’s default memory target does not fit. On a full bare-metal GPU the defaults work.
Step 27: Inference running
chat-qwen is Running with 1/1 pods and 1 GPU allocated; the training workload shows Completed.

Step 28: Test the model endpoint
Call the OpenAI-compatible API through the inference URL. The model answers the question.

URL=https://chat-qwenrunai-test-a.runai-inference.gpu.lab
curl -s –cacert /cm/shared/runai/certs/ca.crt $URL/v1/models
curl -s –cacert /cm/shared/runai/certs/ca.crt $URL/v1/chat/completions \
-H “Content-Type: application/json” \
-d ‘{“model”:”Qwen/Qwen2.5-1.5B-Instruct”,”messages”:[{“role”:”user”,”content”:”Explain in 3 sentences what a GPU cluster is used for.”}]}’ \
| python3 -c ‘import sys,json; print(json.load(sys.stdin)[“choices”][0][“message”][“content”])’
Request path: DNS → 172.16.20.241 (MetalLB) → runai-gateway (TLS) → HTTPRoute → Knative → vLLM pod on node002. The URL is an API, not a web page: open /v1/models in a browser; chat needs a POST.
Summary
- Prerequisites on the BCM Kubernetes cluster: Kyverno exclusions, Kube-Prometheus Stack, MetalLB pool, DNS, TLS and a Gateway API gateway.
- Run:ai 3.0.20 control plane and cluster gpulab installed with the guided installer; UI at https://runai.gpu.lab.
- Department gpulab and project test-a with GPU quota; a GPU training job and a vLLM inference endpoint ran on node002.
Production: bare-metal or passthrough GPUs (GPU fractions need a supported toolkit and driver), a CSI storage class for the control plane database, SSO, a corporate certificate, and a registry mirror for large images.
