Nvidia NIMs

NVIDIA Inference Microservices (NIMs) are a set of pre‑packaged, production‑ready containers that expose state‑of‑the‑art AI models (LLMs, vision, speech, embeddings, etc.) as HTTP/REST or gRPC services. The containers bundle the model, all required libraries (TensorRT, cuDNN, CUDA), and a tiny inference server that requests routing, batching, and metrics. You can spin‑up an inference endpoint with a single command, no need to write your own TensorRT‑or‑PyTorch serving code.

NIMs provides many useful features, including:

• GPU‑accelerated inference: The container contains a pre‑compiled TensorRT (or Torch‑script) engine that runs on a GPU at ~10‑30 ms per request, far faster than a CPU‑only model.
• Versioned, language‑agnostic REST API: Each NIM image ships with a tiny HTTP server (nim_server) that exposes a stable, documented JSON API (/v1/embeddings, /v1/text2img, /v1/asr, …). You can call it from any language that can do an HTTP POST.
• Stateless microservice: Each request is independent; the server can be horizontally scaled by launching more containers or by using SLURM’s job array feature. It fits naturally into batch‑or‑online workloads managed by SLURM.
• Automatic request batching: If several clients hit the same endpoint within a few milliseconds, the server groups the requests into a single GPU kernel launch, dramatically increasing throughput.
• Zero‑install for users: All heavy AI libraries (PyTorch, TensorRT, CUDA, cuDNN…) live inside the container. The host system only needs Apptainer and the GPU driver.
• Observability: By setting a couple of environment variables you can get Prometheus metrics (/metrics) and fine‑grained logs without rebuilding the image.
• Model‑cache on shared filesystem: The container can be pointed at a model directory that lives in your home or shared Project2 storage allocation, so the model stays on disk after between jobs.

NIMs offer the same performance as a hand‑written TensorRT application without any code. The service can be kept alive for hours/days and shared by many users. Upgrades are a single apptainer pull—you never have to touch the host system.

This guide will teach you how to use NIMs with CARC OnDemand and on the cluster.

1 Prerequisites

1. GPU: Any GPU, so you must submit your job to gpu partition or your condo node with a GPU
2. Apptainer: Load the module with module load apptainer before running the container
3. NGC / NIM account: Free registration at https://ngc.nvidia.com. You need an API key to pull private NIM images. Once you’ve signed up/in, go to Create, Setup, and copy the 32‑char string API key.
4. Storage space: Models can take a significant amount of space (100GB+). Check your quota with myquota command.

Flowchart of the process:

flowchart TD
    A["Your laptop / Browser"]
    B["Discovery login node"]
    C["CARC OnDemand web portal"]
    D["JupyterLab (no GPU) python kernel"]
    E["SLURM Job with (GPU) - NIM service"]
    A -- "SSH" --> B
    A -- "HTTPS" --> C
    C -- "(OOD JupyterLab App)" --> D
    D -- "REST API" --> E
    B --> E

2 Set up your NGC API key

Run the following command to set up your API key. This only needs to be done once.

# Put this in your ~/.bashrc (or ~/.profile) on the login node
export NGC_API_KEY="YOUR_NGC_PERSONAL_ACCESS_TOKEN"

After you add the line, run source ~/.bashrc (or log out/in again).

3 NIMs with CARC OnDemand

3.0.1 Pick a model and pull the container

Below we use BERT‑base‑uncased (sentence embeddings) as the first example; later we also show Stable Diffusion 2.1 for a more visual demo. The steps are identical for any NIM image.

# Create a single place for all SIF files
mkdir -p /project2/ttrojan_123/apptainer_images
cd /project2/ttrojan_123/apptainer_images

# Pull the BERT NIM container from NGC (the SIF will be called bert.sif)
apptainer pull bert.sif \
    docker://nvcr.io/nim/bert:1.0.0   # version tag may change - check NGC page

apptainer pull contacts NGC, authenticates with $NGC_API_KEY, downloads the OCI image, and converts it into an immutable SIF (bert.sif). The file lives on your home directory (or any shared filesystem)—it is read‑only and can be used by any job.

To inspect the container, run:

apptainer exec bert.sif cat /opt/nim/README.md | less

The README contains the exact REST endpoints, required JSON fields and sample curl commands. Keep it handy, you’ll need the endpoint names when you write the Jupyter notebook.

3.0.2 Create a small shared directory for discovery & logs

mkdir -p /project2/ttrojan_123/nim_shared
# This directory will be visible from every node (login, compute, OnDemand)

Inside this directory we will write a tiny text file (server_info.txt) that contains the host‑name and port of the running NIM service. The JupyterLab notebook will read that file to know where to send its HTTP requests.

3.0.3 Write the SLURM job script that starts the NIM server

File: bert_server.sbatch

#!/bin/bash
#SBATCH --job-name=bert_nim_server
#SBATCH --partition=gpu                     # GPU partition name
#SBATCH --gres=gpu:1                        # one GPU
#SBATCH --cpus-per-task=12                  # TensorRT likes several CPU threads
#SBATCH --mem=64G                           # enough RAM for the model
#SBATCH --account=ttrojan_123               # CARC Project account
#SBATCH --time=12:00:00                     # job time limit
#SBATCH --output=%x_%j.out                  # stdout
#SBATCH --error=%x_%j.err                   # stderr

module purge
module load apptainer

# -------------------------------------------------------------
#  Variables you may want to change
# -------------------------------------------------------------
CONTAINER=/project2/ttrojan_123/apptainer_images/bert.sif
PORT=8500                         # any free port >1024
DISCOVERY_FILE=/project2/ttrojan_123/nim_shared/server_info.txt

# -------------------------------------------------------------
#  Write the node name to a shared location so clients can discover it
# -------------------------------------------------------------
echo "$HOSTNAME $PORT" > $DISCOVERY_FILE

# -------------------------------------------------------------
#  Run the NIM microservice
# -------------------------------------------------------------
#  --nv   = give the container access to the GPU(s) allocated by SLURM
#  The server stays in the foreground; SLURM will consider the job
#  finished only when the process exits.
apptainer exec \
    --nv \
    $CONTAINER \
    python -m nim_server \
        --model_dir /opt/nim/models/bert-base-uncased \
        --port $PORT \
        --log_level INFO

3.0.4 Submit the server job

sbatch bert_server.sbatch

You can see the job with squeue -u $USER and follow its output:

tail -f /project2/ttrojan_123/nim_shared/bert_nim_server_${SERVER_JOB_ID}.out

(If you want the log in a separate file, add --bind /project2/ttrojan_123/nim_logs:/var/log to the apptainer exec line and configure the server to write there.)

The service will now be listening on port 8500 of the compute node that SLURM allocated. It will stay alive for the 12 h wall‑time (or until you cancel it).

JupyterLab is not started in the same job. Open OnDemand already gives you a web‑based JupyterLab that runs on a CPU compute node and is automatically authenticated with your cluster credentials.

By keeping the inference service separate:

• The GPU node is dedicated to the heavy inference work (no extra CPU cycles wasted on the notebook UI).
• Multiple users can share the same service (just point their notebooks at the same URL).
• The OnDemand portal can proxy the notebook’s HTTP traffic to the compute node without you having to manage SSH tunnels.

3.0.5 Start a JupyterLab session via CARC OnDemand

1. Log in at https://ondemand.carc.usc.edu

2. Go to “Interactive Apps and select JupyterLab“ from the dropdown menu

3. In the Launch dialog you can optionally request a node in main partition—no GPU is required because the notebook only talks to the remote NIM service.

4. Click Launch.

   • OnDemand will start a SLURM job for the notebook and then give you a browser‑based JupyterLab UI in the My Interactive Sessions menu at:

   https://ondemand.carc.usc.edu/pun/sys/dashboard/batch_connect/sessions

   The notebook server runs inside an Apptainer container (the same NIM image is used just for a clean Python environment).

When the UI appears, you will see a JupyterLab file browser rooted at a directory you bound‑mounted (e.g. ~/jupyter_notebooks). Any notebook you create there will be stored on the shared filesystem and survive after you log out. Because you are already authenticated by OnDemand, you can safely start JupyterLab without a token or password.

3.0.6 Sample discovery code

The NIM server script wrote a tiny discovery file:

/project2/ttrojan_123/nim_shared/server_info.txt   ->   <node‑hostname> <port>

The Jupyter notebook can read this file, resolve the hostname to an IP address (the compute node’s internal infiniband IP), and build the service URL. CARC OnDemand’s proxy will forward any request that the browser makes to that internal IP/port, so the notebook never needs a manual SSH tunnel.

import pathlib, socket, os

# Path is the same on the login node (where the notebook runs) because
# /project2/ttrojan_123 points to the same shared filesystem.
DISCOVERY_FILE = pathlib.Path.home() / "nim_shared" / "server_info.txt"

if not DISCOVERY_FILE.is_file():
    raise FileNotFoundError(f"Discovery file not found: {DISCOVERY_FILE}")

with open(DISCOVERY_FILE) as f:
    hostname, port_str = f.read().strip().split()
    port = int(port_str)

# Resolve the hostname to the internal IP of the compute node.
# This works because the login node (where Jupyter runs) can resolve
# the compute node’s name via the cluster’s DNS.
host_ip = socket.gethostbyname(hostname)

BASE_URL = f"http://{host_ip}:{port}"
print(f"NIM service URL -> {BASE_URL}")

An example output:

NIM service URL -> http://10.2.5.123:8500

Now you can call the service exactly as you would with any HTTP client.

3.0.7 Optional: Turn on Prometheus metrics for observability

Add the following environment variables to the SLURM server script before the apptainer exec line:

export NIM_METRICS_PORT=9100      # metrics endpoint will be http://host:9100/metrics
export NIM_LOG_LEVEL=DEBUG       # more verbose logging

You can then query the metrics from the login node (or via a Prometheus server) with:

curl http://$(hostname -i):9100/metrics | head

3.0.8 Cancel the server when you’re done

scancel $SERVER_JOB

The discovery file (server_info.txt) will stay on the shared filesystem, but the hostname will refer to a node that no longer exists - your notebook will raise a connection error, which indicates the service is stopped.

3.0.9 Understanding the CARC OnDemand reverse proxy

When you request a resource from a notebook (e.g. http://10.2.5.123:8500/v1/embeddings), the browser sends the request to the gateway (<cluster‑gateway>). The portal’s proxy inspects the destination IP—if it is a private IP belonging to a compute node, it forwards the traffic over the internal network to that node and back to the browser. No firewall changes are required. The proxy is transparent and you just use the real internal IP/port as if you were on the same network.

If your cluster has not enabled the proxy for arbitrary ports, please submit a ticket to add the port range (8500‑8600) to the proxy whitelist. Most installations already allow a wide range for interactive apps.

3.0.10 Fine‑tuning NIMs server behavior with environment variables

You can add these before the python -m nim_server call in the SLURM script:

• NIM_LOG_LEVEL=DEBUG More verbose logging (written to stdout).
• NIM_METRICS_PORT=9100 Exposes Prometheus metrics on the given port (same node).
• NIM_MODEL_CACHE_DIR=/shared/models Directory where the container will store downloaded model files (use a fast shared filesystem).
• NIM_MAX_BATCH_SIZE=64 Upper bound for automatic request batching (default is 32).

Example snippet:

export NIM_LOG_LEVEL=DEBUG
export NIM_METRICS_PORT=9100
export NIM_MAX_BATCH_SIZE=64

3.0.11 Cleaning up

When you are finished with a model:

# Cancel the SLURM job
scancel $SERVER_JOB_ID

# (Optional) Remove the container if you need space
rm /project2/ttrojan_123/apptainer_images/bert.sif

The notebooks you saved in the OnDemand file system remain untouched.

3.0.12 Troubleshooting

3.0.13 Best practices for a multi‑user environment

1. Use a shared discovery directory (/project2/ttrojan_123/nim_shared) so that all users can read the server location (the file is world‑readable by default).
2. Keep the container SIF immutable - never edit it. If you need a different version, just apptainer pull a new file (bert_v2.sif).
3. Write logs to a central location (e.g., /project2/ttrojan_123/nim_logs) and rotate them manually if they grow large.
4. Set resource limits (#SBATCH --time, --mem) to avoid runaway jobs that hog the GPU.
5. Document the service URL in a shared wiki so users know which port to look at (or simply point them to the discovery file).

4 NIMs on the cluster

For a quick start guide, refer to this page.

Start by pulling a NIM image. Here we will pull the BERT‑base‑english inference microservice as an example.

# Create a directory for Apptainer cache, containers and temporary files (highly recommended as models can take a lot of space)

mkdir -p /project2/ttrojan_123/apptainer_images
mkdir -p /project2/ttrojan_123/apptainer_tmp
mkdir -p /project2/ttrojan_123/apptainer_cache

# Pull the container; the file will be named after the last path component (bert_latest.sif)

export APPTAINER_CACHE=/project2/ttrojan_123/apptainer_cache  
export TMPDIR=/project2/ttrojan_123/apptainer_tmp

cd /project2/ttrojan_123/apptainer_images

apptainer pull \
    --disable-cache \
    --env NGC_API_KEY=$NGC_API_KEY \
    bert_latest.sif \
    docker://nvcr.io/nim/bert:latest

The script components are:

• pull: Download a remote image and convert it into a local .sif (Singularity Image File).
• --disable-cache: Forces a fresh download; useful the first time or when you need the newest tag.
• --env NGC_API_KEY=$NGC_API_KEY: Passes the API key to the remote registry for authentication.
• bert_latest.sif: Name of the local SIF file you will run later.
• docker://nvcr.io/nim/bert:latest: The OCI reference (Docker‑style) to the NIM image on NGC.

You now have a single file (bert_latest.sif) that contains the whole container, ready to be executed on any node that has Apptainer and the NVIDIA driver.

Some other useful models include:

Replace the URI in the apptainer pull command accordingly.

4.0.1 Running a NIM container locally

Before we embed the container in a SLURM job, it is helpful to test it interactively.

# Example: start the BERT microservice on port 8000, using 1 GPU
apptainer exec \
    --nv \                     # Enable NVIDIA GPU support
    --bind /etc/hosts:/etc/hosts \   # optional: ensure proper hostname resolution
    bert_latest.sif \
    python -m nim_server \
        --model_dir /opt/nim/models/bert \
        --port 8000 \
        --log_level INFO

Explanation of flags:

• exec: Run a command inside the container.
• --nv: Automatically bind the NVIDIA driver libraries (/usr/lib/x86_64-linux-gnu) and expose GPUs to the container.
• --bind: Map a host directory/file into the container (here we bind /etc/hosts just as an example).
• python -m nim_server …: The entrypoint used by most NIM images; it starts the HTTP inference server.
• --model_dir: Path inside the container where the model weights are stored (usually pre‑bundled).
• --port: Port on the host that will be opened. Apptainer forwards it automatically.
• --log_level: Verbosity of server logs.

You can now test the service from the login node (or a separate client) with curl:

curl -X POST http://$(hostname -I | awk '{print $1}'):8000/v1/embeddings \
     -H "Content-Type: application/json" \
     -d '{"input": "Hello world"}'

If you see a JSON response with an embeddings array, the container works!

4.0.2 Integrating NIMs with SLURM

Create a minimal SLURM batch script (single‑GPU, HTTP server) with a file called run_bert_nim.sbatch:

#!/bin/bash
#
# SBATCH directives
#SBATCH --job-name=bert_nim
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1            # 1 GPU
#SBATCH --cpus-per-task=4
#SBATCH --mem=32G
#SBATCH --time=02:00:00
#SBATCH --account=ttrojan_123
#SBATCH --output=bert_nim_%j.out
#SBATCH --error=bert_nim_%j.err

# ---------------------------------------------------------
# 1) Load modules / set environment (cluster‑specific)
# ---------------------------------------------------------
module purge
module load apptainer

# ---------------------------------------------------------
# 2) Define where the container lives
# ---------------------------------------------------------
CONTAINER=/project2/ttrojan_123/apptainer_images/bert_latest.sif

# ---------------------------------------------------------
# 3) Choose a port that is free on the compute node.
#    SLURM does not manage ports, so pick a high, unlikely-to-conflict value.
# ---------------------------------------------------------
PORT=8500

# ---------------------------------------------------------
# 4) Run the NIM container with Apptainer
# ---------------------------------------------------------
apptainer exec \
    --nv \
    --bind /etc/hosts:/etc/hosts \
    $CONTAINER \
    python -m nim_server \
        --model_dir /opt/nim/models/bert \
        --port $PORT \
        --log_level INFO

# ---------------------------------------------------------
# 5) When the server exits, the job ends.
# ---------------------------------------------------------

Submit the script with:

sbatch run_bert_nim.sbatch

After a few seconds you’ll see the job in the queue:

squeue -u $USER -o "%.8i %.9P %.8j %.8u %.2t %.10M %.6D %R"

When the job starts, the container launches the BERT microservice on the compute node’s IP address. To query it from a login node (or from another node that can reach the compute node), you need the compute node’s hostname:

# Inside a separate terminal (after the job has started)
NODE=$(squeue -j <jobid> -h -o "%R")   # %R prints the allocated node list
IP=$(getent hosts $NODE | awk '{print $1}')
curl -X POST http://${IP}:8500/v1/embeddings \
     -H "Content-Type: application/json" \
     -d '{"input":"CARC at USC is amazing"}'

4.0.3 Batch inference with a job array

Sometimes you want the NIM server to run once and then have many short jobs submit inference requests (e.g., a data‑processing pipeline). The pattern is:

1. Job 0 - starts the NIM server (keeps running).
2. Job 1…N - each task pulls a piece of data, sends an HTTP request, writes the result.

4.0.3.1 Server job (server.sbatch)

#!/bin/bash
#SBATCH --job-name=bert_server
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=4
#SBATCH --mem=32G
#SBATCH --time=04:00:00
#SBATCH --account=ttrojan_123
#SBATCH --output=bert_server_%j.out
#SBATCH --error=bert_server_%j.err

module purge
module load apptainer

CONTAINER=/project2/ttrojan_123/apptainer_images/bert_latest.sif
PORT=8600

# Write the node name to a shared location so workers can read it
echo $HOSTNAME > /project2/ttrojan_123/bert_server_node.txt

apptainer exec \
    --nv \
    $CONTAINER \
    python -m nim_server \
        --model_dir /opt/nim/models/bert \
        --port $PORT \
        --log_level INFO

Submit it once and note the job id ($SERVER_JOB):

SERVER_JOB=$(sbatch server.sbatch | awk '{print $4}')

4.0.3.2 Worker job array (worker.sbatch)

#!/bin/bash
#SBATCH --job-name=bert_worker
#SBATCH --partition=gpu          # no GPUs needed for the client
#SBATCH --cpus-per-task=2
#SBATCH --mem=8G
#SBATCH --time=00:30:00
#SBATCH --array=1-100                 # 100 independent tasks
#SBATCH --account=ttrojan_123
#SBATCH --output=bert_worker_%A_%a.out
#SBATCH --error=bert_worker_%A_%a.err

module purge
module load apptainer

# ------------------------------------------------------------------
# 1) Find the server node & port (written by the server job)
# ------------------------------------------------------------------
SERVER_NODE=$(cat /project2/ttrojan_123/bert_server_node.txt)
SERVER_PORT=8600
SERVER_IP=$(getent hosts $SERVER_NODE | awk '{print $1}')

# ------------------------------------------------------------------
# 2) Example payload - each array task could read a line from a file
# ------------------------------------------------------------------
INPUT_LINE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" /project2/ttrojan_123/input_sentences.txt)

# ------------------------------------------------------------------
# 3) Send request (using curl inside a tiny Apptainer container that
#    has curl and SSL libs; we can reuse the same NIM image for simplicity)
# ------------------------------------------------------------------
apptainer exec \
    /project2/ttrojan_123/apptainer_images/bert_latest.sif \
    curl -s -X POST http://${SERVER_IP}:${SERVER_PORT}/v1/embeddings \
         -H "Content-Type: application/json" \
         -d "{\"input\":\"${INPUT_LINE}\"}" \
    > /project2/ttrojan_123/outputs/emb_${SLURM_ARRAY_TASK_ID}.json

The server runs on a GPU node for the entire 4‑hour window. Each worker reads a line from a shared text file (input_sentences.txt), contacts the server via HTTP, and writes the JSON response to a per‑task output file. Because the workers are lightweight (no GPU), you can schedule many of them on standard CPU nodes, maximizing cluster utilization.

4.0.3.3 Submit the server and worker jobs

# 1) Server (run first)
SERVER_JOB=$(sbatch server.sbatch | awk '{print $4}')
echo "Server job id = $SERVER_JOB"

# 2) Give the server a few seconds to start, then launch workers
sleep 30   # simple wait; in production you might poll for the node file
sbatch worker.sbatch

4.0.4 Customizing NIM containers (advanced)

Most NIM images expose a handful of environment variables you can set at runtime (via --env in Apptainer). Below are the most useful ones:

Pass the environmental variables with:

apptainer exec \
    --nv \
    --env NIM_BATCH_SIZE=16 \
    --env NIM_LOG_LEVEL=DEBUG \
    $CONTAINER \
    python -m nim_server --port 8000

If you need to mount an external model directory (e.g., a custom fine‑tuned checkpoint), add a bind mount:

# Assume your model lives in /project2/ttrojan_123/custom_llama2/
apptainer exec \
    --nv \
    --bind /project2/ttrojan_123/custom_llama2:/opt/custom_model \
    --env NIM_MODEL_DIR=/opt/custom_model \
    $CONTAINER \
    python -m nim_server --port 8000

4.0.5 Monitoring and logging

Here is a quick health‑check script you can run on the login node:

#!/usr/bin/env bash
JOBID=$1
NODE=$(squeue -j $JOBID -h -o "%R")
IP=$(getent hosts $NODE | awk '{print $1}')
PORT=8500   # adjust to the port you used

# simple ping
curl -s -o /dev/null -w "%{http_code}" http://${IP}:${PORT}/health || echo "Server not reachable"

4.0.6 Troubleshooting

5 Additional resources

| Container security | Read Apptainer’s “User namespaces” and “Security options” to run containers in a sandboxed mode. |