SLURM/ClusterStatus

2026-02-06T17:57:27Z

Ackumar: added slurm-web as a way to view cluster status

=Cluster Status=
SLURM offers a variety of tools to check the general status of nodes/partitions in a cluster.

==sinfo==
The sinfo command will show you the status of partitions in the cluster, in alphabetical order. Passing the -N flag will show each node individually, also in alphabetical order.
<pre>
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
cbcb up infinite 9 mix cbcb[00,03,22,26-28],legacy[00,10,20]
cbcb up infinite 3 alloc cbcb[01-02,29]
cbcb up infinite 49 idle cbcb[04-20,23-25],legacy[01-09,11,13-19,21-28,30-31,34-35]
...
vulcan-scavenger up infinite 14 mix brigid[16-17],vulcan[23-25,27-30,32-33,36-37,45]
vulcan-scavenger up infinite 1 alloc vulcan35
vulcan-scavenger up infinite 34 idle brigid[18-19],vulcan[00-22,26,34,38-44]
</pre>

<pre>
$ sinfo -N
NODELIST NODES PARTITION STATE
brigid16 1 vulcan-cpu mix
brigid16 1 scavenger mix
brigid16 1 vulcan-dpart mix
brigid16 1 vulcan-scavenger mix
...
vulcan45 1 vulcan-ramani mix
vulcan45 1 vulcan-scavenger mix
vulcan45 1 scavenger mix
</pre>

==scontrol==
The scontrol command can be used to view the status/configuration of the nodes in the cluster. If passed specific node name(s) only information about those node(s) will be displayed, otherwise all nodes will be listed. To specify multiple nodes, separate each node name by a comma (no spaces).
<pre>
$ scontrol show nodes tron05,tron13
NodeName=tron05 Arch=x86_64 CoresPerSocket=16
CPUAlloc=28 CPUTot=32 CPULoad=47.32
AvailableFeatures=rhel8,x86_64,Zen,EPYC-7302,Ampere
ActiveFeatures=rhel8,x86_64,Zen,EPYC-7302,Ampere
Gres=gpu:rtxa6000:8
NodeAddr=tron05 NodeHostName=tron05 Version=21.08.5
OS=Linux 4.18.0-348.20.1.el8_5.x86_64 #1 SMP Tue Mar 8 12:56:54 EST 2022
RealMemory=257538 AllocMem=157696 FreeMem=197620 Sockets=2 Boards=1
State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=100 Owner=N/A MCS_label=N/A
Partitions=scavenger,tron
BootTime=2022-04-21T17:40:51 SlurmdStartTime=2022-04-21T18:00:56
LastBusyTime=2022-04-22T11:21:16
CfgTRES=cpu=32,mem=257538M,billing=346,gres/gpu=8,gres/gpu:rtxa6000=8
AllocTRES=cpu=28,mem=154G,gres/gpu=7,gres/gpu:rtxa6000=7
CapWatts=n/a
CurrentWatts=0 AveWatts=0
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

NodeName=tron13 Arch=x86_64 CoresPerSocket=16
CPUAlloc=1 CPUTot=16 CPULoad=8.41
AvailableFeatures=rhel8,x86_64,Zen,EPYC-7302P,Ampere
ActiveFeatures=rhel8,x86_64,Zen,EPYC-7302P,Ampere
Gres=gpu:rtxa4000:4
NodeAddr=tron13 NodeHostName=tron13 Version=21.08.5
OS=Linux 4.18.0-348.20.1.el8_5.x86_64 #1 SMP Tue Mar 8 12:56:54 EST 2022
RealMemory=128525 AllocMem=65536 FreeMem=33463 Sockets=1 Boards=1
State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=10 Owner=N/A MCS_label=N/A
Partitions=scavenger,tron
BootTime=2022-04-21T17:40:46 SlurmdStartTime=2022-04-21T17:54:51
LastBusyTime=2022-04-22T13:04:57
CfgTRES=cpu=16,mem=128525M,billing=173,gres/gpu=4,gres/gpu:rtxa4000=4
AllocTRES=cpu=1,mem=64G,gres/gpu=4,gres/gpu:rtxa4000=4
CapWatts=n/a
CurrentWatts=0 AveWatts=0
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
</pre>

==sacctmgr==
The sacctmgr command shows cluster accounting information. One of the helpful commands is to list the available QoSes.

<pre>
$ sacctmgr list qos format=Name,Priority,MaxWall,MaxJobsPU
Name Priority MaxWall MaxJobsPU
---------- ---------- ----------- ---------
normal 0
dpart 0 2-00:00:00 8
gpu 0 08:00:00 2
</pre>

==slurm-web==
Slurm-web is a web dashboard for our Slurm HPC cluster. It provides cluster statistics and Slurm information for the Nexus cluster.
* URL: https://slurm-web.umiacs.umd.edu

UMIACS - User contributions [en]

SLURM/ClusterStatus