Ollama on CARC

This guide walks through the complete workflow of running the Ollama model-serving engine with an Apptainer container, pulling a specific model, and generating a response - all from a JupyterLab terminal.

1 Prerequisites

• A compute node with GPU (the example used below requires at GPU with about 40GB of VRAM - A40, A100, L40s)

• Sufficient disk space: a 27 B-parameter model requires 15–30 GB. A directory in your scratch folder is a good option: /scratch1/ttrojan

2 Login to CARC OnDemand

Navigate to CARC Ondemand to get started. You will need to be connected to a USC VPN to access OnDemand (see more details here).

2.1 Start Jupyterlab in Interactive Apps

From the OnDemand dashboard, select Interactive Apps and then choose the newest version of JupyterLab. You can also select JupyterLab from the pinned apps on the dashboard.

Fill in the fields with the following information:

• Working directory: If you plan on working in your group project directory put /project2/ttrojan_123, replacing the ttrojan_123 with your group directory. Otherwise, use your scratch /scratch1/ttrojan
• Cluster: Discovery
• Project account: ttrojan_123 (choose your group’s account from the dropdown menu)_
• Partition: gpu
• Number of CPUs: 8
• Memory (GB): 120
• GPU model: A40
• Number of GPUs: 1
• Number of hours: 4 (at minimum)

Launch the session. Once the session starts click Connect to Jupyterlab.

3 Set up the Ollama container

3.1 Open a JupyterLab terminal and prepare working space

In JupyterLab, select File -> New -> Terminal (or use an existing terminal tab). This opens a bash shell with your default user environment.

Go to your scratch with the following command (replace ttrojan with your username):

cd /scratch1/ttrojan

Create the ollama directory which will contain models and the Ollama container image:

mkdir ollama
cd ollama

Finish by loading the Apptainer module:

module purge
module load apptainer

3.2 Pull the Ollama container from Docker Hub

The following command prompts Apptainer to contact Docker Hub, download the official ollama/ollama image layers, and assemble them into a single SIF (Singularity Image Format) file named ollama_latest.sif in your current working directory:

apptainer pull --disable-cache docker://ollama/ollama:latest

This process can take a few minutes depending on network speed.

3.3 Create a local model directory

Inside the container, Ollama stores all downloaded model weights under /root/.ollama.

Create a host-side directory and bind-mounting it with:

mkdir -p ollama-models

Running this command makes it so the models persist even if the container instance is stopped or deleted.

3.4 Start the Ollama server as a background instance

Start the Ollama instance:

apptainer instance start \
  --env OLLAMA_HOST=0.0.0.0:11434 \
  --env OLLAMA_MODELS=/ollama-models \
  --nv \
  --bind /scratch1/ttrojan/ollama-models:/ollama-models \
  ollama_latest.sif \
  ollama-server

Start Ollama server:

apptainer exec \
  --env OLLAMA_HOST=0.0.0.0:11434 \
  --env OLLAMA_MODELS=/ollama-models \
  instance://ollama-server \
  ollama serve

Flag breakdown:

The Ollama server (ollama serve) starts automatically inside the container and listens on 127.0.0.1:11434. The command returns immediately because instance start runs the service in the background.

3.5 Prepare your work terminal

Open a new terminal in JupyterLab: In JupyterLab, select File -> New -> Terminal. Navigate to your Ollama directory and load the apptainer module:

cd /scratch1/ttrojan/ollama
module purge
module load apptainer

3.6 Verify the instance is running

To verify, run:

apptainer instance list

Expected output (showing the instance name, PID, and image):

NAME              PID      IMAGE
ollama-server     12345    /scratch1/ttrojan/ollama/ollama_latest.sif

4 Download the qwen3.8:27b model

4.1 Pull the model

apptainer exec \
  --env OLLAMA_HOST=0.0.0.0:11434 \
  --env OLLAMA_MODELS=/ollama-models \
  instance://ollama-server \
  ollama pull qwen3.8:27b

What this command does:

1. apptainer exec instance://ollama-server opens a shell inside the already-running ollama-server instance - sharing the same filesystem, network namespace, and bind mounts.

2. ollama pull qwen3.8:27b tells the Ollama server (reachable at 127.0.0.1:11434 inside the container) to download the model manifest and all associated weight files from Ollama’s model registry.

3. Files are written to /ollama-models inside the container, which maps back to /scratch1/ttrojan/ollama-models on the host.

Disk-space warning: A 27 B-parameter model in a quantized format typically requires 15–30 GB of storage. Ensure /scratch1/ttrojan/ollama-models has sufficient free space (df -sh /scratch1/ttrojan/ollama-models). Download time depends on your internet connection (10–30+ minutes is common).

Model-name note: The name qwen3.8:27b is used exactly as specified. If Ollama reports “model not found,” verify the name at https://ollama.com/library - common Qwen variants include qwen2.5:32b, qwen2.5:72b, etc.

4.2 Confirm the model was downloaded

Confirm with:

apptainer exec \
  --env OLLAMA_HOST=0.0.0.0:11434 \
  --env OLLAMA_MODELS=/ollama-models \
  instance://ollama-server \
  ollama list

Expected output:

NAME           ID              SIZE      MODIFIED      
qwen3.8:27b    22130167c4c2    17 GB     7 minutes ago    

The model is now stored locally in /scratch1/ttrojan/ollama-models and ready for inference.

5 Connect to the model and run your first prompt

5.1 Run the prompt “Hi”

Start with the following:

apptainer exec \
  --env OLLAMA_HOST=0.0.0.0:11434 \
  --env OLLAMA_MODELS=/ollama-models \
  instance://ollama-server \
  ollama run qwen3.8:27b "Hi" \
  --verbose

What this command does:

1. apptainer exec instance://ollama-server enters the running container instance.

2. ollama run qwen3.8:27b "Hi" sends a generate request to the Ollama server at 127.0.0.1:11434.

3. The server loads the model weights into GPU (or CPU) memory - the first run incurs a cold-start delay (30–90 seconds for a 27 B model).

4. The model processes the prompt "Hi" and returns a generated text response, which is printed to the terminal. Consecutive prompts will be faster.

Example output:

Hi there! How can I help you today?

5.2 Interactive mode

To enter an ongoing chat session (type prompts one at a time, press Ctrl+D to exit):

apptainer exec \
  --env OLLAMA_HOST=0.0.0.0:11434 \
  --env OLLAMA_MODELS=/ollama-models \
  instance://ollama-server \
  ollama run qwen3.8:27b \
  --verbose

6 Using the Ollama REST API from a JupyterLab notebook

If you started the instance with --env OLLAMA_HOST=0.0.0.0:11434 and --env OLLAMA_MODELS=/ollama-models, you can call the model from a notebook cell:

import requests, json

response = requests.post(
    "http://127.0.0.1:11434/api/generate",
    json={
        "model": "qwen3.8:27b",
        "prompt": "Hi",
        "stream": False
    }
)

print(response.json()["response"])

7 Stopping the instance

When you are finished, stop the background instance to release GPU/CPU and memory resources:

apptainer instance stop ollama-server

To restart later (models are preserved in /scratch1/ttrojan/ollama-models):

apptainer instance start \
  --env OLLAMA_HOST=0.0.0.0:11434 \
  --env OLLAMA_MODELS=/ollama-models \
  --bind /scratch1/ttrojan/ollama-models:/ollama-models \
  --nv \
  ollama_latest.sif \
  ollama-server

8 Quick reference (all commands in sequence)

# 1. Create and go to your ollama working directory
mkdir /scratch1/ttrojan/ollama
cd /scratch1/ttrojan/ollama

# 1. Pull container
apptainer pull docker://ollama/ollama:latest

# 2. Create persistent model directory
mkdir -p /scratch1/ttrojan/ollama/ollama-models

# 3. Start Ollama server (background)
apptainer instance start \
  --env OLLAMA_HOST=0.0.0.0:11434 \
  --env OLLAMA_MODELS=/ollama-models \
  --bind /scratch1/ttrojan/ollama-models:/ollama-models \
  --nv \
  ollama_latest.sif \
  ollama-server

# 4. Verify instance is up
apptainer instance list

# 5. Pull the model
apptainer exec \
  --env OLLAMA_HOST=0.0.0.0:11434 \
  --env OLLAMA_MODELS=/ollama-models \
  instance://ollama-server ollama pull qwen3.8:27b

# 6. Verify model is downloaded
apptainer exec \
  --env OLLAMA_HOST=0.0.0.0:11434 \
  --env OLLAMA_MODELS=/ollama-models \
  instance://ollama-server ollama list

# 7. Run the first prompt
apptainer exec \
  --env OLLAMA_HOST=0.0.0.0:11434 \
  --env OLLAMA_MODELS=/ollama-models \
  instance://ollama-server ollama run qwen3.8:27b "Hi"

# 8. (Optional) Stop when done
apptainer instance stop ollama-server