Retrieval-Augmented Generation (RAG) is a way of getting an LLM to answer questions using your own documents instead of just what it was trained on. Instead of fine-tuning a model, you store your documents as searchable vectors, pull out the relevant pieces at question time, and hand those to the model along with the question. NVIDIA’s RAG pipeline builds this using its own NIM microservices for the AI models. This guide is a hands-on deployment of that pipeline on a self-managed Kubernetes cluster with two A100 GPUs — the goal is to understand every piece well enough to explain it, not just get it running.

How it works, end to end

There are two separate flows worth understanding before touching any config:

Ingesting a document: the text gets split into chunks, each chunk is turned into a vector by the embedding model, and those vectors are stored in a vector database.
Answering a question: the question itself is turned into a vector the same way, the vector database returns the closest matching chunks, a reranking model re-scores those matches against the actual question, and the LLM writes the final answer using the question plus the best-ranked chunks.

Everything in both flows runs as its own service inside the cluster — the pieces are described below.

A quick picture of it

The chain server is the only piece that talks to the models and the database directly; the browser and the UI never call them on their own.

What’s actually running

Module	What it does
rag-playground	The chat window opened in a browser. Handles document upload and the question/answer UI.
chain-server	The orchestrator behind the UI. Takes a question, calls the embedding model, searches the vector database, calls the reranker, then calls the LLM to produce the answer. This is the only piece that talks to the models and the database directly.
Embedding model (nv-embedqa-e5-v5)	Turns text into vectors so it can be compared and searched. Used for both document chunks and incoming questions.
Reranker (llama-nemotron-rerank-1b-v2)	Re-scores the vector database’s rough matches against the actual question, so the most relevant chunks are what the LLM sees first.
LLM (Llama 3.1 8B, vLLM backend)	Generates the final answer from the question plus the reranked chunks.
Milvus	The vector database. Stores every document chunk as a vector and serves the similarity search.
NIM Operator	A Kubernetes controller that watches model-cache and model-service definitions, downloads the model weights, and runs the model containers — nothing is pulled or started by hand.
GPU Operator	Exposes the physical GPUs to Kubernetes and runs the device plugin that lets more than one model share a GPU (time-slicing).
k3s	The Kubernetes distribution the whole cluster runs on — lightweight enough for a single node.

This setup vs. a normal production setup

This deployment is built to run the full pipeline on a single node for learning purposes. A normal production deployment of the same architecture would differ in a few important ways:

Piece	This setup	Typical production setup
Cluster	k3s, single node	Multi-node Kubernetes cluster (e.g. EKS, GKE, OpenShift) with the control plane separate from worker nodes
GPUs	2× A100 80GB, shared four ways with time-slicing	One or more dedicated GPUs per model, no sharing
Traffic in	NodePort	An ingress controller in front of the UI, with TLS and a real domain
Vector database	Single Milvus pod, no redundancy	Managed or replicated vector database with backups
Storage	Local disk on the node, wiped if the VM is deleted	Persistent, replicated storage
Access to the UI	No authentication	Authentication or SSO in front of the UI
Scaling	One fixed replica per service	Autoscaling based on load

Setting it up

I’m using an Ubuntu 24 server.

The config files referenced below are all in the linked git repo. Each step applies one of those files — the repo has the exact commands for each.

https://github.com/Dineshk1205/nvidiaRAG.git

Run install-nvidia-driver.sh — installs the NVIDIA driver (this reboots the machine at the end). Expect: nvidia-smi lists the GPUs after it comes back up.

Run install-container-toolkit.sh — installs the NVIDIA Container Toolkit, needed before k3s starts. Expect: nvidia-ctk –version prints a version.

Run install-k3s-helm.sh — installs k3s (single-node Kubernetes) and Helm, and points k3s’s data directory at the large disk. Expect: kubectl get nodes shows one node, Ready.

Run install-gpu-operator.sh — adds the NGC Helm repo and installs the GPU Operator, which exposes the GPUs to Kubernetes. Expect: kubectl get clusterpolicy shows ready. Then apply gpu-test-pod.yaml, a one-off CUDA test pod. Expect: the pod’s logs end with “Test PASSED”.

Source setup-ngc-key.sh (an NGC API key is generated once, manually, on NVIDIA’s site — the script only exports it) — needed for every step from here on that pulls a model or image from NGC. Expect: no error after pasting the key.

Run install-nim-operator.sh — installs the NIM Operator (the controller that manages model downloads) and creates the NGC secret it needs to pull NVIDIA’s models. Expect: the operator pod is Running and the secret shows up under kubectl get secret.

Apply nim-caches.yaml — starts downloading the three models’ weights into persistent storage. Expect: kubectl get nimcache shows all three as Ready once the downloads finish (can take a while).

Apply time-slicing-config.yaml — reconfigures the GPU device plugin to split each physical GPU into multiple schedulable slots. Expect: the allocatable GPU count on the node jumps up (for example, from 2 to 4).

Apply nim-services-retrieval.yaml, then nim-service-llm.yaml — starts the embedding, reranker and LLM models as running services. Expect: all three show Ready, and a port-forwarded call to /v1/models returns the model list.

Install Milvus with install mulvus.sh — deploys the vector database. Expect: three pods (etcd, object storage, Milvus itself) Running.

Install the RAG app with install-rag-app.sh — deploys the chain-server and the playground UI, wired to the services above. Expect: two pods Running, and a playground service listening on its port.

Run patch-chain-server.sh — applies a required fix to the chain server, explained below. Expect: the script’s final test command returns a real answer instead of an error.

Expose the UI — publish the playground service (NodePort, or an ingress if one is set up). Expect: the playground loads in a browser, and a question about an uploaded document gets answered using its content.