Cluster usage
The Data Lab's 16 compute VMs (datalab-i-01 … i-16, 24 cores and 512 GB RAM each) as the VMs themselves see them: memory measured inside every VM every 15 minutes, and every Slurm job since . Most of the memory holds the weights of large language models for the Data Lab's CPU LLM optimization work — fast, exact inference of the largest open models on this cluster, split across nodes — so it stays full even when no process is computing.
Memory of every node, right now
Measured inside each VM (/proc/meminfo, page-cache residency of the model files from fincore). Model weights = LLM weight
files held in RAM, read for every generated token; processes = memory of running programs; other cache = other files and kernel memory.
The right column is how much model-weight data each node stores on its local disk, as a multiple of its RAM: more than fits, so the models compete for memory.
Memory reserved by Slurm jobs
Share of the cluster's 8.2 TB RAM and 384 cores held by running jobs, from Slurm accounting (daily average, and the busiest hour of each day).
Compute per day, by user group
Core-hours of all Slurm jobs per day (a job's cores × its running time, split over the days it ran).
Memory in use over time
Whole cluster, every 15 minutes (the in-VM collector started on ; Slurm history above goes further back).
LLM decoding runs at the memory-bandwidth limit
Generating a token reads every active weight from RAM once, so speed is set by memory bandwidth. Share of each node's measured bandwidth the engine uses while generating (median of recent benchmarks, single-node models).
Job report by user group
All Slurm jobs since . Reserved memory = memory requested × running time. Individual people are grouped for privacy.
Recent jobs
The latest 80 jobs, plus each group's latest 20 (seconds-long node probes left out; they are counted above).