MPI and Multi-node

This guide covers building and running MPI programs on Topanga AI Computing, on one node or across several.

1 How multi-node sessions work on Topanga

When you launch a multi-node session, Topanga gives you one container on each node. The container your terminal lands in is called main1; the others are sub1, sub2, and so on. You never need to log in to the sub nodes—you work entirely from main1 and mpirun starts and manages the processes on the other nodes for you. The nodes reach each other by those hostnames, which is why the MPI hostfile below simply lists them.

All nodes in a session see the network storage systems: Topanga’s data folder as well as /home1, /scratch1, and /project2. A program built once in any of these locations is immediately visible on every node, so there is no copying or staging step.

2 Quick start

# 1. Environment: use HPC-X MPI, not the spack OpenMPI
source /apps/lmod/set-lmod.sh
module load gcc/13.3.0 hpcx-mpi/2.20
export LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LIBRARY_PATH   # linker needs this image's crt1.o

# 2. Build
mpif90 -O3 -o myapp myapp.f90        # Fortran
mpicc  -O3 -o myapp myapp.c          # C

# 3a. Run on a single node
mpirun -n 50 --bind-to none ./myapp

# 3b. Run across multiple nodes (full-speed InfiniBand)
mpirun --hostfile hostfile -np 200 --bind-to none -x UCC_TL_SHM_TUNE=0 ./myapp

hostfile lists your session’s nodes and how many ranks each may hold:

main1 slots=62
sub1  slots=62
sub2  slots=62
sub3  slots=62

slots=62 shows the number of CPU cores that each of your nodes have and the number of subs depend on the size of the cluster that you launched.

See below for explanations of the non-obvious flags and how to recognize common errors.

3 Building

3.1 Use HPC-X, not the Spack OpenMPI

The Spack openmpi/5.0.5 module is built against slurm/munge, which Topanga does not have, so its compiler wrappers fail before they can compile anything:

error while loading shared libraries: libmunge.so.2: cannot open shared object file

Use hpcx-mpi/2.20 instead (NVIDIA HPC-X, which is OpenMPI 4.1.7 + UCX). It has no munge dependency and drives the InfiniBand network natively. Load it together with gcc/13.3.0.

3.2 Two environment fixes for this container image

The container image keeps its C runtime and system headers in places the Spack toolchain does not search by default:

export LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LIBRARY_PATH    # crt1.o, needed for any linking
export CPATH=/usr/include/x86_64-linux-gnu:/usr/include:$CPATH # system headers, only if gcc reports them missing

LIBRARY_PATH is required for every build. CPATH is only needed if your C code includes system headers that gcc reports as missing (for example fatal error: sys/ipc.h: No such file or directory).

4 Running

4.1 Single node

mpirun -n 50 --bind-to none ./myapp

--bind-to none is required everywhere on Topanga—the containers do not allow MPI to pin processes to specific CPU cores, so without this flag mpirun fails at startup.

4.2 Multiple nodes

mpirun --hostfile hostfile -np 200 --bind-to none -x UCC_TL_SHM_TUNE=0 ./myapp

This runs at full InfiniBand speed between nodes and over shared memory within each node. No network tuning is needed—the only addition over the single-node command is the hostfile and one flag, explained below.

4.3 One extra multi-node flag

Without -x UCC_TL_SHM_TUNE=0, a multi-node run hangs at the first collective operation (Bcast, Allreduce, Barrier, …) and floods the screen with:

tl_shm_team.c ... Child failed to attach to shmseg, errno: Invalid argument

HPC-X ships with an add-on called UCC that tries to speed up collective operations. One of UCC’s tricks is a special shared-memory channel between ranks on the same node, and that particular trick is broken inside Topanga’s containers when many ranks run per node. The flag tells UCC to skip that one trick; everything else works normally. This is a bug in the UCC library itself, not a system limit, so there is nothing to fix on your side or to request from the admins.

Any one of these three flags fixes it (all three were verified to give identical, correct results; the first and third were also timed back-to-back and show no performance difference):

5 Troubleshooting

6 Complete example script

#!/bin/bash
# Build + run an MPI code across all nodes of a Topanga multi-node session.
set -e
cd /project2/<your_project>/<your_dir>     # any shared location works (/home1, /scratch1, /project2)

NP=${NP:-200}                              # total ranks across all nodes

# Environment: HPC-X MPI + this image's C runtime for the linker
source /apps/lmod/set-lmod.sh
module load gcc/13.3.0 hpcx-mpi/2.20
export LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LIBRARY_PATH

# Build
mpif90 -O3 -o myapp myapp.f90

# Run over full-speed InfiniBand
#   --bind-to none      : containers do not allow CPU core pinning
#   UCC_TL_SHM_TUNE=0   : skip UCC's broken shared-memory collective path
mpirun --hostfile hostfile -np "$NP" --bind-to none \
  -x UCC_TL_SHM_TUNE=0 \
  ./myapp