Your First Days on an HPC Cluster: A Primer for Statisticians

Most statisticians meet a compute cluster the same way: someone sends a two-page guide with lines like “10 CPU nodes, each 2×64-core EPYC, 512 GB RAM, NVMe /tmp” and a handful of commands to copy. The commands work, but nothing explains why. This post fills that gap: what each piece is, why the rules exist, and how to check every claim on your own cluster.

It assumes you are comfortable with R and a terminal, and nothing about hardware or system administration. Cluster names, hostnames and paths below are placeholders (mycluster, netid, cluster.example.edu).

1. The Mental Model

A cluster is a set of separate computers in a server room, linked by a fast network. Each computer is a node. You never sit at any of them.

Four moving parts:

HPC (High-Performance Computing) is the umbrella term for all of this.

Login node versus compute node

On many clusters the hardware is identical. What differs is the role:

2. Inside a Node: Reading the Hardware Line

A typical spec line reads: 10 CPU nodes, 2 GPU nodes; each 2×64-core EPYC 7763, 512 GB RAM, 2 TB NVMe /tmp. “10 CPU” means ten CPU nodes — computers without graphics cards — not ten processors. Names like c[01-10] are Slurm shorthand for c01, …, c10.

Sockets, cores and hardware threads

“CPU” is used at three levels, and you need to know which one a sentence means:

Linux and Slurm usually count threads as “CPUs”. A node with 128 cores therefore shows up as 256 CPUs.

What a hardware thread buys you. A core’s adders and multipliers often sit idle while a program waits for data from RAM — about 100 ns, hundreds of clock cycles. Simultaneous multithreading (SMT) keeps two program states per core and interleaves them to fill those gaps: one cook working two orders while a pot boils. Memory-bound code (sparse matrices, MCMC with many small steps) typically gains 10–30% from the second thread; dense matrix algebra gains close to nothing, because both threads compete for the same multipliers. Rule of thumb: two hardware threads ≈ 1.0–1.3 cores, not 2.

1
2
3

lscpu | grep -E '^(Model name|Socket|Core|Thread|CPU\(s\))'   # on any node
scontrol show node c01 | grep -E 'CPUTot|ThreadsPerCore|RealMemory|Gres'
sinfo                                                          # partitions and nodes

Memory

RAM is working memory where R keeps the objects in your session. A server node commonly has hundreds of GB to 1 TB, shared by all its cores.

Disks: SSD, NVMe and the node-local /tmp

An SSD stores data in flash chips with no moving parts, while the older HDD spins a magnetic platter. NVMe is the fastest common way to attach an SSD: it plugs into the PCIe lanes wired straight to the CPU, whereas SATA goes through a disk controller designed for spinning disks. NVMe also accepts thousands of command queues, so many cores reading at once don’t wait in line.

Compute nodes usually have their own NVMe drive mounted at /tmp. Inside a job, Slurm often points the variable TMPDIR at it. It is fast and usually wiped when the job ends so use it for heavy temporary I/O and copy results out before the job finishes.

The CPU itself

A chip like the AMD EPYC 7763 (3rd-gen “Milan”, Zen 3, 2021) has 64 cores / 128 threads, 2.45 GHz base and 3.5 GHz boost, 256 MB of L3 cache, 8 DDR4 memory channels, and a list price around $8K per chip. EPYC is AMD’s server line, and Intel’s equivalent is Xeon. Each core is slower than one core of a recent laptop, but a two-socket node has 128 of them.

GPUs

A GPU has thousands of small cores that are weak alone, but extremely fast at the same simple operation on huge arrays, mostly matrix multiplication. That is why deep learning lives on GPUs.

Datacenter cards cost more because of strong double precision, error-correcting (ECC) memory, 24/7 server design, and licence terms that keep consumer GeForce cards out of datacenters. Prices move so treat them as orders of magnitude.

3. Speed and Numerical Precision

FLOPS and “peak”

A FLOP is one addition or multiplication of two floating-point numbers; FLOPS counts them per second; TFLOPS is $10^{12}$ FLOPS. Peak is the theoretical maximum — cores × clock × FLOPs per core per cycle. Real code reaches 70–90% of peak on large dense matrix multiplies and often only 1–20% when limited by memory traffic.

Multiplying two $n\times n$ matrices costs about $2n^3$ FLOPs. For $n=10^4$ that is $2\times10^{12}$, about 0.2 s on a 10-TFLOPS card at peak.

A useful comparison for statisticians — one EPYC 7763 in double precision:

$$ 64 \text{ cores} \times 2.45\times10^{9}\ \tfrac{\text{cycles}}{\text{s}} \times 16\ \tfrac{\text{FLOPs}}{\text{cycle}} \approx 2.5 \text{ TFLOPS}, $$

so a two-socket node delivers about 5 TFLOPS of FP64, which is roughly one A30 without its tensor cores. For double-precision statistics, a CPU node is not weak next to a GPU. The GPU advantage is large mainly in single and lower precision.

Floating-point formats

“Lower precision” means fewer bits per number.

Machine ε is the gap between 1 and the next representable number: in FP32, 1 + 1e-8 is exactly 1. Fewer bits are faster because more numbers move per memory transfer and a short multiplier needs less silicon; going from FP64 to FP16 on tensor cores is a 16–30× speedup.

Why deep learning gets away with it and statistics usually can’t

Deep learning tolerates low precision because:

1. SGD is already noisy. In $\theta_{t+1} = \theta_t - \eta,(\nabla L(\theta_t) + \xi_t)$ the minibatch noise $\xi_t$ is a few percent; FP16 rounding is about $10^{-3}$.
2. Nobody needs $\hat\theta$ to many digits — flat basins predict equally well.
3. Normalisation keeps activations of order 1, where floating point is most accurate.
4. Outputs are robust: an argmax or a softmax sample rarely changes under a $10^{-3}$ perturbation.

Where it does break, standard fixes exist — together called mixed-precision training:

For inference, weights are quantised further (INT8, 4-bit): a 70B-parameter model is 140 GB in BF16 and about 35 GB in 4-bit, and since generating text is limited by memory bandwidth, fewer bytes means more tokens per second.

4. Where Your Files Live

Most clusters give you several storage areas with different trade-offs:

“Scratch” is an HPC convention (as in scratch paper), not a Linux reserved word; your admins choose the path and usually set the variable in a startup file under /etc/profile.d/.

Quotas count files, not just bytes

Many clusters print something like this at login (or via a command such as showquota or quota):

1
2
3

Disk                 Disk Usage   Limit   File Usage   Limit
/home/netid              1.2G       50G       31.0K       5M
/scratch/netid           4.8G     500G         112      10M

File Usage counts files. You can hit that limit with plenty of space left — one tiny output file per simulation replicate, or a conda environment with tens of thousands of files. At either limit, every write fails, including inside running jobs.

One copy, seen everywhere

Check where your folders physically live:

1

df -hT $HOME $SCRATCH /tmp

Lines of type nfs/nfs4 (or lustre, gpfs) come from a storage server and are identical on every node: a file saved in your home on the login node is instantly visible to your job on a compute node. A line of type xfs/ext4 on /dev/nvme… is the node’s own local disk.

5. Connecting

An SSH host alias, not a shell alias

A shell alias (alias mycluster='ssh netid@cluster.example.edu' in ~/.zshrc) only works when you type it at the prompt; rsync, scp, git and VS Code never see it. An SSH host alias in ~/.ssh/config is read by every SSH-based program, and it can carry options:

1
2
3
4
5
6
7

Host mycluster                         # the nickname
    HostName cluster.example.edu       # the real address
    User netid                         # your login name
    ControlMaster auto                 # connection sharing:
    ControlPath ~/.ssh/cm-%r@%h:%p     #   log in once, reuse the
    ControlPersist 4h                  #   connection for 4 hours
    ServerAliveInterval 60             # keep idle connections alive

• Lines 1–3 are the alias itself.
• Control… — the first connection becomes a “master” and leaves a socket file at ControlPath (%r, %h, %p = remote user, host, port). Later ssh, scp and rsync calls ride through it with no new password or two-factor prompt.
• ServerAliveInterval 60 sends a small “still here” message every minute so firewalls and VPNs don’t cut idle sessions.
• Recent OpenSSH (10.1 and later) warns when a server lacks post-quantum key exchange; WarnWeakCrypto no-pq-kex silences that one warning. Older OpenSSH rejects the option as a “Bad configuration option”, so add it only if you see the warning and ssh -V reports 10.1+.

Setting it up

1
2
3
4
5
6
7

mkdir -p ~/.ssh && chmod 700 ~/.ssh
nano ~/.ssh/config            # paste the block; Ctrl-O, Enter, Ctrl-X
chmod 600 ~/.ssh/config

ssh -G mycluster | grep -E '^(hostname|user|controlpersist) '   # parse check, no connection
ssh mycluster                 # first time: type "yes" to accept the server's fingerprint
ssh -O check mycluster        # in a second tab: "Master running (pid=…)"

Passwords do not echo while you type — that’s normal.

What chmod 700 and chmod 600 mean

Unix permissions give three classes — owner, group, others — three rights each: read (4), write (2), execute (1). Each digit of the mode is a sum:

On a folder, “execute” means “may enter”. Why lock SSH files down: whoever can edit your config can add a ProxyCommand that runs commands as you; whoever can read a private key can log in as you. SSH refuses a config file that others can write (Bad owner or permissions) and a private key that others can read.

VS Code Remote-SSH

1. Install the Remote - SSH extension by Microsoft (ms-vscode-remote.remote-ssh). It is easy to confuse with “Remote Repositories” or “Dev Containers”, which are different.
2. Command palette → Remote-SSH: Connect to Host… → mycluster. It reuses your SSH config. The first time, VS Code installs a server program in ~/.vscode-server on the cluster — several hundred MB of your home quota.
3. A badge SSH: mycluster in the bottom-left corner confirms the window runs on the cluster.
4. File → Open Folder → your project folder, not your whole home: opening home makes VS Code lint its own server files and fill the Problems panel with noise.
5. Ctrl-` opens a terminal — on the login node.

Gotcha: the bare bash-5.1$ prompt

If the VS Code terminal shows bash-5.1$ while ssh gives a proper prompt, your account probably has no ~/.bashrc. Bash reads different files depending on how it starts:

With no ~/.bashrc, the VS Code terminal loads nothing — no prompt, no aliases, possibly no module command. The standard fix:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

cat > ~/.bashrc <<'EOF'
# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi
EOF

cat > ~/.bash_profile <<'EOF'
if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi
EOF

Open a new terminal and check that the prompt is normal and type module prints “module is a function”.

6. Shell Essentials

Variables, $ and export

A variable is a name holding text; a dollar sign in front means “substitute the value” before the command runs. So ls $SCRATCH/project runs ls /scratch/netid/project. Names are case-sensitive; capitals for system variables are a convention.

1
2
3
4
5

PROJ=~/project                       # set: no spaces around =, no $
echo $PROJ                           # use: with $
bash -c 'echo "child: $PROJ"'        # child:            ← not exported
export PROJ
bash -c 'echo "child: $PROJ"'        # child: /home/netid/project

Every program you start — R, Python, a Slurm job — is a child process and receives only exported variables. export moves a variable into that set; env lists them. That is why R’s Sys.getenv("PROJ") returns "" unless you exported it.

Quoting

1
2
3

echo "$HOME"      # /home/netid     double quotes: substituted
echo '$HOME'      # $HOME           single quotes: literal
echo "\$HOME"     # $HOME           backslash: literal for one $

An unclosed quote leaves a > continuation prompt; Ctrl-C gets you out.

Small commands worth knowing

Unix commands are silent on success. Aliases from your laptop (eza, zoxide’s z, …) don’t exist on the cluster; plain aliases in ~/.bashrc such as alias cdp='cd ~/project' cover the few places you visit.

7. Software: Modules

On a cluster, almost nothing is on your PATH by default — which R typically finds nothing. Many versions are installed side by side, and the module system lets you choose:

“Spider” as in web crawler: it crawls the whole module tree, including modules that only become loadable after a prerequisite such as a compiler is loaded. module load just edits PATH and library paths for the current shell.

Reading a module name. In R/4.4.1-gfbf-2023b, 4.4.1 is the R version and gfbf-2023b is the toolchain it was built with (on EasyBuild-managed clusters, gfbf = GCC + FlexiBLAS + FFTW; foss adds MPI). The “Other possible modules matches” list that spider prints is a loose name match, not a list of R packages — but skim it for useful things like RStudio-Server or JupyterLab.

Check what the module already ships before installing anything:

1
2
3

ip <- rownames(installed.packages())
length(ip); c("data.table", "arrow", "Matrix") %in% ip
.libPaths()

If .libPaths() lists only a system folder, your first install.packages() will offer to create a personal library in your home folder — say yes.

8. Running Work with Slurm

An interactive session, flag by flag

1

srun --partition=short --ntasks=1 --cpus-per-task=4 --mem=8G --time=02:00:00 --pty bash -i

When the prompt changes to a compute node’s name, you’re in; hostname confirms. Afterwards, sacct -j JOBID (or your site’s jobstats/seff) shows what the job actually used, so you can size the next request.

Slurm may round you up to whole cores

Run scontrol show config | grep SelectTypeParameters. If it says CR_Core… and nodes have 2 threads per core, Slurm hands out whole cores: requesting 1 CPU gives you 1 core = 2 CPUs, requesting 3 gives you 2 cores. This isolates jobs — two users never share a physical core, which keeps timings predictable and avoids side-channel leaks between threads. For 16 cores’ worth of dense numerics, request 32 CPUs.

1

srun --cpus-per-task=1 bash -c 'echo task $SLURM_PROCID of $SLURM_NTASKS'

Batch scripts

A batch script is a bash script whose #SBATCH lines are resource requests. Slurm runs the body once, on the first node of the allocation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

#!/bin/bash
#SBATCH --job-name=test
#SBATCH --partition=short
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
#SBATCH --time=00:05:00
#SBATCH --output=logs/%x_%j.out     # %x = job name, %j = job ID
module purge
module load R/4.4.1-gfbf-2023b
hostname                            # should print a compute node, not the login node
Rscript -e 'R.version.string'

Submit with sbatch test.sh, watch with squeue -u $USER, follow output with tail -f logs/test_*.out. Ordinary lines run once; only a line prefixed with srun inside the script launches a job step that may run several copies.

screen: keep interactive sessions alive

The srun client for an interactive session runs on the login node and is the job’s lifeline. If your laptop sleeps or Wi-Fi drops, the SSH connection dies, srun dies, and Slurm ends your job. screen (or tmux) sits upstream of srun on the login node and survives the drop:

Running screen on the compute node wouldn’t help: you only reach the compute node through srun, which lives on the login node. And screen itself is a light text relay, so it’s fine on the login node.

The name comes from GNU Screen, a “screen manager” that keeps several virtual terminal screens inside one window.

Putting it together: installing an R package

1
2
3
4

screen -S work
srun --partition=short --ntasks=1 --cpus-per-task=4 --mem=8G --time=02:00:00 --pty bash -i
module load R/4.4.1-gfbf-2023b
R

1
2

Sys.setenv(MAKEFLAGS = "-j4")     # let the compiler use the CPUs you requested
install.packages("arrow")         # answer "yes" twice to create a personal library

Packages with large C++ dependencies can take many minutes to compile. If the download fails, compute nodes may have no internet access — ask your cluster’s help desk. When done: q(), then exit to free the node, then exit again to close screen.

Which way to run R

RStudio Desktop on your laptop cannot use the cluster’s CPUs. Whatever holds an interactive allocation holds it while you’re idle too — exit when done.

Cheat Sheet

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# ── laptop ──────────────────────────────────────────────
ssh mycluster
rsync -avP ./scripts/ mycluster:~/project/scripts/      # push
rsync -avP mycluster:~/project/results/ ./results/      # pull

# ── login node ──────────────────────────────────────────
sinfo                                  # partitions and node states
module spider R                        # which versions exist
screen -S work   /   screen -r work    # start / reattach (Ctrl-a d detaches)
sbatch job.sh                          # submit
squeue -u $USER                        # my jobs
scancel JOBID                          # cancel one
sacct -j JOBID                         # what a finished job used

# ── compute node ────────────────────────────────────────
srun --partition=short --ntasks=1 --cpus-per-task=4 --mem=8G --time=02:00:00 --pty bash -i
module load R/<version>
R
exit                                   # free the node

Rules of thumb

• Never compute on the login node — VS Code terminals and notebook kernels opened there count.
• Always --ntasks=1 for non-MPI work; Slurm “CPUs” may be half-cores.
• Scripts in home, big regenerable output in scratch, shared data read in place.
• module purge at the top of batch scripts; keep module load lines out of ~/.bashrc.
• mkdir -p logs before your first sbatch.
• Stay in FP64 for likelihoods and Cholesky factors.