UMIACS - User contributions [en]

MonthlyMaintenanceWindow

2026-05-29T00:16:16Z

Mbaney:

[[HelpDesk | UMIACS staff]] takes a monthly maintenance window to patch and reboot all UMIACS-supported hosts and services. This provides a way for staff to ensure security updates are installed and applied on the numerous different platforms and appliances that UMIACS runs.

The window for each month is calculated by adding 9 days to [https://en.wikipedia.org/wiki/Patch_Tuesday Microsoft's Patch Tuesday] to allow for enough time to marshal patches released that month from Microsoft, Red Hat, Apple, and other OS and application vendors and have enough time to get systems prepared to reboot. This translates to the window being on the '''Thursday that occurs between the 17th and the 23rd (inclusive)''' of each month. The window lasts from '''5pm-8pm'''.

[[Nexus]] will always have a reservation in place from 4:45pm-8pm on the day of the upcoming window to prevent jobs from being scheduled on compute nodes. The 15-minute addition before the start of the window is to allow jobs to fully end. Any job submitted before the reservation begins that has a time limit that would run into the reservation will be held until at least the end of the reservation - 8pm on the day of the window. This is to prevent issues with jobs failing to end properly causing delays in work we have scheduled during the window.

A list of upcoming maintenance windows is as follows, with the next one in bold. Again, the window is on the '''Thursday that occurs between the 17th and the 23rd (inclusive)''' of each month, and lasts from '''5pm-8pm'''.

* '''June 18th 2026'''
* July 23rd 2026
* August 20th 2026
* September 17th 2026
* October 22nd 2026
* November 19th 2026
* December 17th 2026

==Archives==
* January 17th 2013 - BEGIN time of 8pm-12am for this window through February 20th 2020
* February 21st 2013
* March 21st 2013
* April 18th 2013
* May 23rd 2013
* June 20th 2013
* July 18th 2013
* August 22nd 2013
* September 19th 2013
* October 17th 2013
* December 19th 2013
* January 23rd 2014
* February 20th 2014
* March 20th 2014
* April 17th 2014
* May 22nd 2014
* June 19th 2014
* July 17th 2014
* August 21st 2014
* September 18th 2014
* October 23rd 2014
* November 20th 2014
* December 18th 2014
* January 22nd 2015
* February 19th 2015
* March 19th 2015
* May 21st 2015
* June 18th 2015
* July 23rd 2015
* August 20th 2015
* September 17th 2015
* October 22nd 2015
* November 19th 2015
* December 17th 2015
* January 21st 2016
* February 18th 2016
* March 12th 2016 (Adjusted date for AVW power outage)
* April 21st 2016
* May 19th 2016
* June 23rd 2016
* July 21st 2016
* August 18th 2016
* September 22nd 2016
* October 20th 2016
* November 17th 2016
* December 22nd 2016
* January 19th 2017
* February 23rd 2017
* March 23rd 2017
* April 20th 2017
* May 18th 2017
* June 22nd 2017
* July 20th 2017
* August 17th 2017
* September 21st 2017
* October 19th 2017
* December 21st 2017
* January 18th 2018
* February 22nd 2018
* March 22nd 2018
* April 19th 2018
* May 17th 2018
* June 21st 2018
* July 19th 2018
* August 23rd 2018
* September 20th 2018
* October 18th 2018
* December 20th 2018
* January 24th 2019
* February 21st 2019
* April 18th 2019
* May 23rd 2019
* June 20th 2019
* July 18th 2019
* August 22nd 2019
* September 19th 2019
* October 17th 2019
* November 21st 2019
* December 19th 2019
* January 23rd 2020
* February 20th 2020
* April 23rd 2020 - BEGIN time of 5pm-7pm for this window through August 19th 2021
* June 18th 2020
* July 23rd 2020
* August 20th 2020
* September 17th 2020
* October 22nd 2020
* November 19th 2020
* December 17th 2020
* January 21st 2021
* February 18th 2021
* March 25th 2021 (Adjusted date for extended Spring Break)
* April 22nd 2021
* May 20th 2021
* June 17th 2021
* July 22nd 2021
* August 19th 2021
* September 23rd 2021 - BEGIN time of 5pm-8pm for this window and all others below
* October 21st 2021
* November 18th 2021
* January 20th 2022
* February 17th 2022
* March 24th 2022 (Adjusted date for Spring Break)
* April 21st 2022
* May 19th 2022
* June 23rd 2022
* July 21st 2022
* August 18th 2022
* September 22nd 2022
* October 20th 2022
* November 17th 2022
* January 19th 2023
* February 23rd 2023
* April 20th 2023
* May 18th 2023
* June 22nd 2023
* July 20th 2023
* August 17th 2023
* September 21st 2023
* October 19th 2023
* December 20th 2023 (Adjusted date for early Winter Break)
* January 18th 2024
* February 22nd 2024
* March 21st 2024
* April 18th 2024
* May 23th 2024
* June 20th 2024
* July 18th 2024
* August 22nd 2024
* September 19th 2024
* October 17th 2024
* November 21st 2024
* December 19th 2024
* January 23rd 2025
* February 20th 2025
* March 20th 2025
* April 17th 2025
* May 22nd 2025
* June 19th 2025
* July 17th 2025
* August 21st 2025
* September 18th 2025
* October 23rd 2025
* November 20th 2025
* December 18th 2025
* January 22nd 2026
* February 19th 2026
* March 19th 2026
* April 23rd 2026
* May 28th 2026 (Adjusted date for CIO-imposed network change freeze)

Nexus

2026-05-19T15:28:45Z

Mbaney: /* Partition QoS */

{{Note|UMIACS Technical Staff will begin the process of upgrading the operating system version on all Nexus cluster nodes in Summer 2026. Please see [[Nexus/ClusterOSUpgrade]] for more information.}}

The Nexus is the combined scheduler of resources in UMIACS. The resource manager for Nexus is [[SLURM]]. Resources are arranged into partitions where users are able to schedule computational jobs. Users are arranged into a number of SLURM accounts based on faculty, lab, or center investments.

= Getting Started =
All accounts in UMIACS are sponsored. If you don't already have a UMIACS account, please see [[Accounts]] for information on getting one. You need a full UMIACS account - not a [[Accounts/Collaborator | collaborator account]] - in order to access Nexus.

== Access ==
Your access to submission nodes (alternatively called login nodes) for Nexus computational resources is determined by your account sponsor's department, center, or lab affiliation. You can log into the [https://intranet.umiacs.umd.edu/directory/cr/ UMIACS Directory CR application] and select the Computational Resource (CR) in the list that has the prefix <code>nexus</code>. The Hosts section lists your available submission nodes - generally a pair of nodes with hostnames of the format <tt>nexus<department, lab, or center abbreviation>[00,01]</tt>, e.g., <tt>nexusgroup00</tt> and <tt>nexusgroup01</tt>.

Once you have identified your submission nodes, you can [[SSH]] into them [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0016076 after connecting to UMD's GlobalProtect VPN]. From there, you are able to submit to the cluster via our [[SLURM]] workload manager. You need to make sure that your submitted jobs have the correct account, partition, and qos.

Please read our [[Nexus/Submission_Node_Policy|Submission Node Policy]] for guidance on appropriate usage of a submission node. If a submission node becomes unresponsive due to disregarding this policy, we may kill user processes on these nodes to resolve the issue. We reserve the right to take action on users who repeatedly cause issues on submission nodes.

== Jobs ==
[[SLURM]] jobs are [[SLURM/JobSubmission | submitted]] by either <code>srun</code> or <code>sbatch</code> depending if you are doing an interactive job or batch job, respectively. You need to provide the where/how/who to run the job and specify the resources you need to run with.

For the who/where/how, you may be required to specify <code>--account</code>, <code>--partition</code>, and/or <code>--qos</code> (respectively) to be able to adequately submit jobs to the Nexus.

For resources, you may need to specify <code>--time</code> for time, <code>--cpus-per-task</code> for CPUs, <code>--mem</code> for RAM, and <code>--gres=gpu</code> for GPUs in your submission arguments to meet your requirements. There are defaults for all four; if you don't specify something, you will get the default value for that resource, which is minimal (e.g., by default, NO GPUs are included if you do not specify <code>--gres=gpu</code>). For more information about submission flags for GPU resources, see [[SLURM/JobSubmission#Requesting_GPUs | here]]. You may also use <code>--ntasks</code> to specify the number of parallel processes to run, with each task having its own set of the resources specified above. You can run <code>man srun</code> on your submission node for a complete list of available submission arguments.

For a list of available GPU types on Nexus and their specs, please see [[Nexus/GPUs]].

For details on how the network for Nexus is architected, please see [[Nexus/Network]]. This can be important if you wish to optimize performance of your jobs.

=== Interactive ===
Once logged into a submission node, you can run simple interactive jobs. If your session is interrupted from the submission node, the job will be killed. As such, we encourage use of a terminal multiplexer such as [[Tmux]].

<pre>
$ srun --pty --cpus-per-task=4 --mem=2gb --gres=gpu:1 bash
srun: Job account was unset; set to user default of 'nexus'
srun: Job partition was unset; set to cluster default of 'tron'
srun: Job QoS was unset; set to association default of 'default'
srun: Job time limit was unset; set to partition default of 60 minutes
srun: job 1 queued and waiting for resources
srun: job 1 has been allocated resources
$ hostname
tron62.umiacs.umd.edu
$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-daad6a04-a2ce-1183-ce53-b267048f750a)
</pre>

=== Batch ===
Batch jobs are scheduled with a script file with an optional ability to embed job scheduling parameters via variables that are defined by <code>#SBATCH</code> lines at the top of the file. You can find some examples in our [[SLURM/JobSubmission]] documentation.

= Partitions =
The SLURM resource manager uses partitions to act as job queues which can restrict size, time and user limits. The Nexus has a number of different partitions of resources. Different Centers, Labs, and Faculty are able to invest in computational resources that are restricted to approved users through these partitions.

'''Partitions usable by all non-[[ClassAccounts |class account]] users:'''
* [[Nexus/Tron]] - Pool of resources available to all non-class accounts sponsored by either UMIACS or CSD faculty.
* Scavenger - [https://slurm.schedmd.com/preempt.html Preemption] partition that contains [https://en.wikipedia.org/wiki/X86-64 x86_64] architecture nodes from multiple other partitions. More resources are available to schedule simultaneously than in other partitions, however jobs are subject to preemption rules. You are responsible for ensuring your jobs handle this preemption correctly. The SLURM scheduler will simply restart a preempted job with the same submission arguments when it is available to run again. For an overview of things you can check within scripts to determine if your job was preempted/resumed, see [[SLURM/Preemption]].
* Scavenger (aarch64) - Preemption partition identical in design to <tt>scavenger</tt>, but only contains [https://en.wikipedia.org/wiki/AArch64 aarch64] architecture nodes.

'''Partitions usable by [[ClassAccounts]]:'''
* [[ClassAccounts#Cluster_Usage | Class]] - Pool of resources available to class accounts sponsored by either UMIACS or CSD faculty.

'''Partitions usable by specific lab/center users:'''
* [[Nexus/CBCB]] - CBCB lab pool available for CBCB lab members.
* [[Nexus/CLIP]] - CLIP lab pool available for CLIP lab members.
* [[Nexus/CML]] - CML lab pool available for CML lab members.
* [[Nexus/GAMMA]] - GAMMA lab pool available for GAMMA lab members.
* [[Nexus/MBRC]] - MBRC lab pool available for MBRC lab members.
* [[Nexus/MC2]] - MC2 lab pool available for MC2 lab members.
* [[Nexus/QuICS]] - QuICS lab pool available for QuICS lab members.
* [[Nexus/Vulcan]] - Vulcan lab pool available for Vulcan lab members.

You can view the partitions that you have access to by using the <code>show_partitions</code> command. By default, the command will show only the partitions that are available to you.

<pre>
$ show_partitions
Name AllowAccounts AllowQos MaxNodes Nodes
------------------------ ----------------------- ------------------------------ ----------- ----------------------------
scavenger scavenger scavenger UNLIMITED brigid[16-19]
cbcb[00-29]
clip[00-13]
cml[00,02-13,15-28,30-33]
cmlcpu[00-04,06-07]
gammagpu[00-21]
legacy[00-11,13-28,30-36]
legacygpu[00-07]
quics00
tron[00-44,46-69]
vulcan[00-45]
------------------------------------------------------------------------------------------------------------------------
scavenger-aarch64 scavenger scavenger-aarch64 UNLIMITED oasis[00-39]
------------------------------------------------------------------------------------------------------------------------
tron nexus default UNLIMITED tron[00-44,46-69]
high
medium
</pre>

If you want to see information for all of the partitions, including those that you do not have access to, you can use the <code>show_partitions --all</code> command.

<pre>
$ show_partitions --all
Name AllowAccounts AllowQos MaxNodes Nodes
------------------------ ----------------------- ------------------------------ ----------- ----------------------------
cbcb cbcb default UNLIMITED cbcb[00-20,22-29]
medium legacy[00-11,13-28,30-36]
high
huge-long
highmem
------------------------------------------------------------------------------------------------------------------------
cbcb-heng cbcb-heng default UNLIMITED cbcb[26-29]
medium
high
huge-long
highmem
------------------------------------------------------------------------------------------------------------------------
cbcb-interactive cbcb interactive UNLIMITED cbcb21
...
</pre>

= Quality of Service (QoS) =
SLURM uses Quality of Service (QoS) both to provide limits on job sizes (termed by us as "job QoS") as well as to limit resources used by all jobs running in a partition, either per user or per group (termed by us as "partition QoS").

=== Job QoS ===
Job QoS are used to provide limits on the size of job that you can run. You should try to allocate only the resources your job actually needs, as resources that each of your jobs schedules are counted against your [[SLURM/Priority#Fair-share | fair-share priority]] in the future.
* default - Default job QoS. Limited to 4 CPU cores, 1 GPU, and 32GB RAM per job. The maximum wall time per job is 3 days.
* medium - Limited to 8 CPU cores, 2 GPUs, and 64GB RAM per job. The maximum wall time per job is 2 days.
* high - Limited to 16 CPU cores, 4 GPUs, and 128GB RAM per job. The maximum wall time per job is 1 day.
* scavenger - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. 11% of the total resources available for each trackable resource type in the partition (CPUs/GPUs/RAM) is permitted simultaneously across all of your jobs running with this job QoS, enforced via the corresponding partition QoS (below) for the scavenger partition. This job QoS is paired one-to-one with the scavenger partition. To use this job QoS, include <code>--partition=scavenger</code> and <code>--account=scavenger</code> in your submission arguments. Do not include any job QoS argument other than <code>--qos=scavenger</code> (optional) or submission will fail.
* scavenger-aarch64 - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. This job QoS is paired one-to-one with the scavenger-aarch64 partition. To use this job QoS, include <code>--partition=scavenger-aarch64</code>, <code>--account=scavenger</code>, and <code>--qos=scavenger-aarch64</code> in your submission arguments.

You can display these job QoS from the command line using the <code>show_qos</code> command. By default, the command will only show job QoS that you can access. The above five job QoS are the ones that everyone can access.

<pre>
$ show_qos
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
scavenger 3-00:00:00
scavenger-aarch64 3-00:00:00
</pre>

If you want to see all job QoS, including those that you do not have access to, you can use the <code>show_qos --all</code> command.

<pre>
$ show_qos --all
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
cml-cpu 7-00:00:00 8
cml-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
cml-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
cml-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
cml-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
cml-scavenger 3-00:00:00 gres/gpu=24
cml-very_high 1-12:00:00 cpu=32,gres/gpu=8,mem=256G 8 gres/gpu=12
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
gamma-huge-long 10-00:00:00 cpu=32,gres/gpu=16,mem=256G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
highmem 21-00:00:00 cpu=128,mem=2T
huge-long 10-00:00:00 cpu=32,gres/gpu=8,mem=256G
interactive 12:00:00 cpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
oasis-exempt 10-00:00:00 cpu=160,mem=28114M
scavenger 3-00:00:00
scavenger-aarch64 3-00:00:00
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-exempt 7-00:00:00 cpu=32,gres/gpu=8,mem=256G 2
vulcan-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
vulcan-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-sailon 3-00:00:00 cpu=32,gres/gpu=8,mem=256G gres/gpu=48
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
vulcan-scavenger-mu+ 3-00:00:00 cpu=288,gres/gpu=72,mem=1152G
</pre>

You are able to submit to any partition that is listed in the <code>show_partitions</code> command. If you need to use an account other than the default account <tt>nexus</tt>, you will need to specify it via the <code>--account</code> submission argument.

=== Partition QoS ===
Partition QoS are used to limit resources used by all jobs running in a partition, either per user (MaxTRESPU) or per group (GrpTRES).

To view partition QoS, use the <code>show_partition_qos</code> command.

<pre>
$ show_partition_qos
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
scavenger-aarch64_part 500
scavenger_part 500 cpu=11%,gres/gpu=11%,mem=11%
tron 500 cpu=32,gres/gpu=4,mem=262144M
</pre>

The scavenger_part partition QoS has relative TRES limits based on the current hardware in a given partition, represented with percentages. To see the current actual TRES limits of this partition QoS, you can use the <code>-r/--real</code> argument.

<pre>
$ show_partition_qos -r
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
scavenger-aarch64_part 500
scavenger_part 500 cpu=888,gres/gpu=140,mem=12574G
tron 500 cpu=32,gres/gpu=4,mem=262144M
</pre>

If you want to see all partition QoS, including those that you do not have access to, you can use the <code>show_partition_qos --all</code> command.

<pre>
$ show_partition_qos --all
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
cbcb 500 cpu=1406,mem=50359G
cbcb-heng 500
cbcb-interactive 500
class 500 cpu=32,gres/gpu=4,mem=262144M
clip 500 cpu=726,mem=6939G
cml 500 cpu=1226,mem=12116G
cml-cpu 500
cml-director 500
cml-furongh 500
cml-scavenger 500 gres/gpu=24
cml-sfeizi 500
cml-wriva 500
cml-wriva-high 500
csd-h200 500
gamma 500 cpu=906,mem=7675G
mbrc 500 cpu=370,mem=3571G
mc2 500 cpu=330,mem=3201G
oasis 500
quics 500 cpu=458,mem=4710G
scavenger-aarch64_part 500
scavenger_part 500 cpu=11%,gres/gpu=11%,mem=11%
tron 500 cpu=32,gres/gpu=4,mem=262144M
vulcan 500 cpu=1402,mem=12936G
vulcan-ampere 500
vulcan-cpu 500
vulcan-ramani 500
vulcan-scavenger 500
vulcan-scavenger-multi 500
</pre>

'''NOTE''': These QoS cannot be used directly when submitting jobs. Partition QoS limits apply to all jobs running on a given partition, regardless of what job QoS is used.

For example, in the default non-preemption partition (<tt>tron</tt>), you are restricted to 32 total CPU cores, 4 total GPUs, and 256GB total RAM at once across all jobs you have running in the partition.

Lab/group-specific partitions may also have their own user limits, and/or may also have group limits on the total number of resources consumed simultaneously by all users that are using their partition, codified by the line in the output above that matches their lab/group name. Note that the values listed above in the two "TRES" columns are not fixed and may fluctuate per-partition as more resources are added to or removed from each partition.

'''All partitions also only allow a maximum of 500 submitted (running (R) or pending (PD)) jobs per user in the partition simultaneously.''' This is to prevent excess pending jobs causing [https://slurm.schedmd.com/sched_config.html#backfill backfill] issues with the SLURM scheduler.
* If you need to submit more than 500 jobs in batch at once, you can develop and run an "outer submission script" that repeatedly attempts to run an "inner submission script" (your original submission script) to submit jobs in the batch periodically, until all job submissions are successful. The outer submission script should use looping logic to check if you are at the max job limit and should then retry submission after waiting for some time interval.
: An example outer submission script is as follows. In this example, <code>example_inner.sh</code> is your inner submission script and is not an [[SLURM/ArrayJobs | array job]], and you want to run 1000 jobs. If your inner submission script is an array job, adjust the number of jobs accordingly. Array jobs must be of size 500 or less.
<pre>
#!/bin/bash
numjobs=1000
i=0
while [ $i -lt $numjobs ]
do
while [[ "$(sbatch example_inner.sh 2>&1)" =~ "QOSMaxSubmitJobPerUserLimit" ]]
do
echo "Currently at maximum job submissions allowed by the partition's QoS."
echo "Waiting for 5 minutes before trying to submit more jobs."
sleep 300
done
i=$(( $i + 1 ))
echo "Submitted job $i of $numjobs"
done
</pre>

It is suggested that you run the outer submission script in a [[Tmux]] session to keep the terminal window executing it from being interrupted.

= Storage =
All network storage available in Nexus is currently [[NFS]] based, and comes in a few different flavors. Compute nodes also have local scratch storage that can be used.

== Home Directories ==
{{Nfshomes}}

== Scratch Directories ==
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Nexus compute infrastructure:
* Network scratch directories
* Local scratch directories

Please note that [[ClassAccounts | class accounts]] do not have network scratch directories.

=== Network Scratch Directories ===
You are allocated 200GB of scratch space via NFS from <code>/fs/nexus-scratch/<USERNAME></code> where <USERNAME> is your UMIACS username. '''It is not backed up or protected in any way.''' This directory is '''[[Automounter | automounted]]'''; you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access it.

You can view your quota usage by running <code>df -h /fs/nexus-scratch/<USERNAME></code>.

You may request a permanent increase of up to 400GB total space without any faculty approval by [[HelpDesk | contacting staff]]. If you need space beyond 400GB, you will need faculty approval and/or a [[#Project_Allocations | project allocation]] for this. If you choose to increase your scratch space beyond 400GB, the increased space is also subject to the 270 TB days limit mentioned in the project allocation section before we check back in for renewal. For example, if you request 1.4TB total space, you may have this for 270 days (1TB beyond the 400GB permanent increase). The amount increased beyond 400GB will also count against your faculty member's 20TB total storage limit mentioned below.

This file system is available on all submission, data management, and computational nodes within the cluster.

=== Local Scratch Directories ===
Each computational node that you can schedule compute jobs on also has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. and '''are not backed up or protected in any way.''' These directories are almost always more performant than any other storage available to the job as they are mounted from disks directly attached to the compute node. However, you must stage your data within the confines of your job and extract the relevant resultant data elsewhere before the end of your job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month during our [[MonthlyMaintenanceWindow | monthly maintenance windows]]. Please make sure you secure any resultant data you wish to keep from these directories at the end of your job.

== Faculty Allocations ==
Each faculty member can be allocated 1TB of permanent lab space upon request. We can also support grouping these individual allocations together into larger center, lab, or research group allocations if desired by the faculty. Please [[HelpDesk | contact staff]] to inquire.

Lab space storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

== Project Allocations ==
Project allocations are available per user for 270 TB days; you can have a 1TB allocation for up to 270 days, a 3TB allocation for 90 days, etc..

A single faculty member can not have more than 20TB of project allocations across all of their sponsored accounts active simultaneously. Network scratch allocation space increases beyond the 400GB permanent maximum also have the increase count against this limit (i.e., a 1TB network scratch allocation would have 600GB counted towards this limit).

Project storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

The maximum allocation length you can request is 540 days (500GB space) and the maximum storage space you can request is 9TB (30 day length).

To request an allocation, please [[HelpDesk | contact staff]] with the faculty member(s) that the project is under involved in the conversation. Please include the following details:
* Project Name (short)
* Description
* Size (1TB, 2TB, etc.)
* Length in days (270 days, 135 days, etc.)
* Other user(s) that need to access the allocation, if any

These allocations are available via <code>/fs/nexus-projects/<project name></code>. '''Renewal is not guaranteed to be available due to limits on the amount of total storage.''' Near the end of the allocation period, staff will contact you and ask if you are still in need of the storage allocation. If renewal is available, you can renew for up to another 270 TB days with reapproval from the original faculty approver.
* If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation.
* If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible.
** If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible.
** If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

== Datasets ==
We have read-only dataset storage available at <code>/fs/nexus-datasets</code>. If there are datasets that you would like to see curated and made available, please see [[Datasets | this page]].

The list of Nexus datasets we currently host can be viewed [https://info.umiacs.umd.edu/datasets/list/?q=Nexus here].

MonthlyMaintenanceWindow

2026-05-18T14:34:22Z

Mbaney:

[[HelpDesk | UMIACS staff]] takes a monthly maintenance window to patch and reboot all UMIACS-supported hosts and services. This provides a way for staff to ensure security updates are installed and applied on the numerous different platforms and appliances that UMIACS runs.

The window for each month is calculated by adding 9 days to [https://en.wikipedia.org/wiki/Patch_Tuesday Microsoft's Patch Tuesday] to allow for enough time to marshal patches released that month from Microsoft, Red Hat, Apple, and other OS and application vendors and have enough time to get systems prepared to reboot. This translates to the window being on the '''Thursday that occurs between the 17th and the 23rd (inclusive)''' of each month. The window lasts from '''5pm-8pm'''.

[[Nexus]] will always have a reservation in place from 4:45pm-8pm on the day of the upcoming window to prevent jobs from being scheduled on compute nodes. The 15-minute addition before the start of the window is to allow jobs to fully end. Any job submitted before the reservation begins that has a time limit that would run into the reservation will be held until at least the end of the reservation - 8pm on the day of the window. This is to prevent issues with jobs failing to end properly causing delays in work we have scheduled during the window.

A list of upcoming maintenance windows is as follows, with the next one in bold. Again, the window is on the '''Thursday that occurs between the 17th and the 23rd (inclusive)''' of each month, and lasts from '''5pm-8pm'''.

* '''May 28th 2026''' (Adjusted date for CIO-imposed network change freeze)
* June 18th 2026
* July 23rd 2026
* August 20th 2026
* September 17th 2026
* October 22nd 2026
* November 19th 2026
* December 17th 2026

==Archives==
* January 17th 2013 - BEGIN time of 8pm-12am for this window through February 20th 2020
* February 21st 2013
* March 21st 2013
* April 18th 2013
* May 23rd 2013
* June 20th 2013
* July 18th 2013
* August 22nd 2013
* September 19th 2013
* October 17th 2013
* December 19th 2013
* January 23rd 2014
* February 20th 2014
* March 20th 2014
* April 17th 2014
* May 22nd 2014
* June 19th 2014
* July 17th 2014
* August 21st 2014
* September 18th 2014
* October 23rd 2014
* November 20th 2014
* December 18th 2014
* January 22nd 2015
* February 19th 2015
* March 19th 2015
* May 21st 2015
* June 18th 2015
* July 23rd 2015
* August 20th 2015
* September 17th 2015
* October 22nd 2015
* November 19th 2015
* December 17th 2015
* January 21st 2016
* February 18th 2016
* March 12th 2016 (Adjusted date for AVW power outage)
* April 21st 2016
* May 19th 2016
* June 23rd 2016
* July 21st 2016
* August 18th 2016
* September 22nd 2016
* October 20th 2016
* November 17th 2016
* December 22nd 2016
* January 19th 2017
* February 23rd 2017
* March 23rd 2017
* April 20th 2017
* May 18th 2017
* June 22nd 2017
* July 20th 2017
* August 17th 2017
* September 21st 2017
* October 19th 2017
* December 21st 2017
* January 18th 2018
* February 22nd 2018
* March 22nd 2018
* April 19th 2018
* May 17th 2018
* June 21st 2018
* July 19th 2018
* August 23rd 2018
* September 20th 2018
* October 18th 2018
* December 20th 2018
* January 24th 2019
* February 21st 2019
* April 18th 2019
* May 23rd 2019
* June 20th 2019
* July 18th 2019
* August 22nd 2019
* September 19th 2019
* October 17th 2019
* November 21st 2019
* December 19th 2019
* January 23rd 2020
* February 20th 2020
* April 23rd 2020 - BEGIN time of 5pm-7pm for this window through August 19th 2021
* June 18th 2020
* July 23rd 2020
* August 20th 2020
* September 17th 2020
* October 22nd 2020
* November 19th 2020
* December 17th 2020
* January 21st 2021
* February 18th 2021
* March 25th 2021 (Adjusted date for extended Spring Break)
* April 22nd 2021
* May 20th 2021
* June 17th 2021
* July 22nd 2021
* August 19th 2021
* September 23rd 2021 - BEGIN time of 5pm-8pm for this window and all others below
* October 21st 2021
* November 18th 2021
* January 20th 2022
* February 17th 2022
* March 24th 2022 (Adjusted date for Spring Break)
* April 21st 2022
* May 19th 2022
* June 23rd 2022
* July 21st 2022
* August 18th 2022
* September 22nd 2022
* October 20th 2022
* November 17th 2022
* January 19th 2023
* February 23rd 2023
* April 20th 2023
* May 18th 2023
* June 22nd 2023
* July 20th 2023
* August 17th 2023
* September 21st 2023
* October 19th 2023
* December 20th 2023 (Adjusted date for early Winter Break)
* January 18th 2024
* February 22nd 2024
* March 21st 2024
* April 18th 2024
* May 23th 2024
* June 20th 2024
* July 18th 2024
* August 22nd 2024
* September 19th 2024
* October 17th 2024
* November 21st 2024
* December 19th 2024
* January 23rd 2025
* February 20th 2025
* March 20th 2025
* April 17th 2025
* May 22nd 2025
* June 19th 2025
* July 17th 2025
* August 21st 2025
* September 18th 2025
* October 23rd 2025
* November 20th 2025
* December 18th 2025
* January 22nd 2026
* February 19th 2026
* March 19th 2026
* April 23rd 2026

BarracudaSpamFirewall

2026-05-11T19:57:33Z

Mbaney:

===Introduction===
UMIACS has deployed a system with two Barracuda Networks spam firewalls. This allows for enterprise level Virus and Spam scoring and filtering for our email architecture. You can log in to the system either of the two firewalls using your UMIACS email address (username@umiacs.umd.edu) and UMD directory passphrase:
*[https://bubs.umiacs.umd.edu bubs.umiacs.umd.edu]
*[https://pompom.umiacs.umd.edu pompom.umiacs.umd.edu]

===Mail Flow Through Barracudas===
*The first time your mail flows through one of the Barracudas it will send you a mail with a new username and password. Subsequently, you will receive, every day (unless you configure otherwise), a mail at approximately 3:30pm US Eastern from the Barracuda with your quarantine summary. '''Please note that the links provides in the Actions column in the summary do not work anymore due to security hardening on the Barracudas. Please use the links provided above to log in.'''

===Scoring===
The Barracudas will score every message that passes through them and inject varying message headers based on that score. You can then create email filters based on these headers to filter out messages that are tagged as spam by the Barracudas. See [[BarracudaSpamFirewall/Scoring]] for more details.

===Quarantine===
*Mail that has been deemed as spam will be kept on the Barracudas in quarantine. It will not be delivered to your mailbox unless you configure the Barracudas to do so.
*Your quarantine will be preserved for '''21 days'''. Individual mail messages are purged if they are still in the quarantine 22 days after they are first received.
*You can search your spam quarantine by following the steps [[BarracudaSpamFirewall/SearchingQuarantine | here]].

===Quarantine Passthrough===
*If you wish to have the mail that would ordinarily be quarantined by Barracuda delivered to your mailbox instead you can configure this using the Barracuda web configuration.
*You can enable this functionality by following the steps [[BarracudaSpamFirewall/QuarantinePassthrough | here]].

===Allow Lists, Block Lists, and Bayesian Filtering===
*You may also setup allow lists, block lists, and Bayesian filtering options through the Preferences tab at the top of the Barracuda web portal.

===More Information===
*For more information on how to use the Barracuda please download the user's guide:

https://wiki.umiacs.umd.edu/umiacs/images/5/5a/Barracuda_usersguide.pdf

BarracudaSpamFirewall

2026-05-11T19:56:37Z

Mbaney:

===Introduction===
UMIACS has deployed a system with two Barracuda Networks spam firewalls. This allows for enterprise level Virus and Spam scoring and filtering for our email architecture. You can log in to the system either of the two firewalls using your UMIACS email address (username@umiacs.umd.edu) and UMD passphrase:
*[https://bubs.umiacs.umd.edu bubs.umiacs.umd.edu]
*[https://pompom.umiacs.umd.edu pompom.umiacs.umd.edu]

===Mail Flow Through Barracudas===
*The first time your mail flows through one of the Barracudas it will send you a mail with a new username and password. Subsequently, you will receive, every day (unless you configure otherwise), a mail at approximately 3:30pm US Eastern from the Barracuda with your quarantine summary. '''Please note that the links provides in the Actions column in the summary do not work anymore due to security hardening on the Barracudas. Please use the links provided above to log in.'''

===Scoring===
The Barracudas will score every message that passes through them and inject varying message headers based on that score. You can then create email filters based on these headers to filter out messages that are tagged as spam by the Barracudas. See [[BarracudaSpamFirewall/Scoring]] for more details.

===Quarantine===
*Mail that has been deemed as spam will be kept on the Barracudas in quarantine. It will not be delivered to your mailbox unless you configure the Barracudas to do so.
*Your quarantine will be preserved for '''21 days'''. Individual mail messages are purged if they are still in the quarantine 22 days after they are first received.
*You can search your spam quarantine by following the steps [[BarracudaSpamFirewall/SearchingQuarantine | here]].

===Quarantine Passthrough===
*If you wish to have the mail that would ordinarily be quarantined by Barracuda delivered to your mailbox instead you can configure this using the Barracuda web configuration.
*You can enable this functionality by following the steps [[BarracudaSpamFirewall/QuarantinePassthrough | here]].

===Allow Lists, Block Lists, and Bayesian Filtering===
*You may also setup allow lists, block lists, and Bayesian filtering options through the Preferences tab at the top of the Barracuda web portal.

===More Information===
*For more information on how to use the Barracuda please download the user's guide:

https://wiki.umiacs.umd.edu/umiacs/images/5/5a/Barracuda_usersguide.pdf

BarracudaSpamFirewall

2026-05-11T19:51:52Z

Mbaney:

===Introduction===
UMIACS has deployed a system with two Barracuda Networks spam firewalls. This allows for enterprise level Virus and Spam scoring and filtering for our email architecture. You can log in to the system either of the two firewalls using your UMIACS email address (username@umiacs.umd.edu) and UMD passphrase:
*[https://bubs.umiacs.umd.edu bubs.umiacs.umd.edu]
*[https://pompom.umiacs.umd.edu pompom.umiacs.umd.edu]

===Mail Flow Through Barracudas===
*The first time your mail flows through one of the Barracudas it will send you a mail with a new username and password. Subsequently, you will receive, every day (unless you configure otherwise), a mail at approximately 3:30pm US Eastern from the Barracuda with your quarantine summary. '''Please note that auto log-in through the link provided in the summary does not work anymore due to security hardening on the Barracudas. Please use the links provided above to log in.'''

===Scoring===
The Barracudas will score every message that passes through them and inject varying message headers based on that score. You can then create email filters based on these headers to filter out messages that are tagged as spam by the Barracudas. See [[BarracudaSpamFirewall/Scoring]] for more details.

===Quarantine===
*Mail that has been deemed as spam will be kept on the Barracudas in quarantine. It will not be delivered to your mailbox unless you configure the Barracudas to do so.
*Your quarantine will be preserved for '''21 days'''. Individual mail messages are purged if they are still in the quarantine 22 days after they are first received.
*You can search your spam quarantine by following the steps [[BarracudaSpamFirewall/SearchingQuarantine | here]].

===Quarantine Passthrough===
*If you wish to have the mail that would ordinarily be quarantined by Barracuda delivered to your mailbox instead you can configure this using the Barracuda web configuration.
*You can enable this functionality by following the steps [[BarracudaSpamFirewall/QuarantinePassthrough | here]].

===Whitelists, Blacklists & Bayesian Filtering===
*You may also setup whitelists, blacklists, and Bayesian filtering options through the Preferences tab at the top of the Barracuda web portal.

===More Information===
*For more information on how to use the Barracuda please download the user's guide:

https://wiki.umiacs.umd.edu/umiacs/images/5/5a/Barracuda_usersguide.pdf

Iribe/Faxing

2026-05-05T13:53:04Z

Mbaney:

The HP LaserJet Enterprise Flow MFP M633 labelled "ps432-4208" in room 4208 in the Iribe Center, in addition to being a black/white printer and copier, also has faxing capability configured (send/receive). The fax number is 301-314-9115 (x49115). HP's guide for using this model of fax can be found [[Media:M633_manual.pdf | here in Chapter 7 (Fax)]].

Please keep in mind UMD phone syntax still applies when using this machine; you will need to [https://it.umd.edu/phone#place dial 9 and then 1 before faxing to any number external to the university].

File:M633 manual.pdf

2026-05-05T13:50:44Z

Mbaney:

UMIACS:Users

2026-05-01T15:23:07Z

Mbaney: Created page with "https://intranet.umiacs.umd.edu/staff/people"

https://intranet.umiacs.umd.edu/staff/people

GitLab

2026-04-30T18:41:01Z

Mbaney:

GitLab is source code collaboration software based on [https://git-scm.com/ Git] source control management. It allows users to create their own repositories and share them with other users/groups or publicly. It supports built-in project wikis, code review, and issue tracking for each project. A full list of features can be found on the [https://www.gitlab.com/gitlab-ce GitLab] website.

All code is hosted on-site on UMIACS servers and is backed up nightly. We give all non-[[ClassAccounts |class account]] UMIACS users 5 projects by default, with a hard limit of 800MB per pboject. Faculty can request an extension to 25 projects. Projects that you create inside of a group namespace still count towards your project limit. UMIACS Staff can help create Lab groups and delegate authority to manage the group to one or more faculty members. Permissions are assigned within GitLab.

To get started, navigate to the following URL in your browser, which will redirect you to UMD's CAS to log in:

https://gitlab.umiacs.umd.edu

==GitLab Offsite Collaborators==
Any UMIACS user now create an unlimited number of offsite collaborator accounts for our Security Groups, GitLab and Object Store. These accounts can not create repositories or groups, but may be given access to your repositories or groups. You can find this utility in our [https://intranet.umiacs.umd.edu/requests Requests] application under [https://intranet.umiacs.umd.edu/requests/accounts UMIACS Collaborators].

==User Documentation==
GitLab provides user documentation that can be accessed at https://gitlab.umiacs.umd.edu/help

Some of the most useful sections include:
* Permissions and user roles - https://gitlab.umiacs.umd.edu/help/user/permissions.md
* GitLab markdown for wikis and READMEs - https://gitlab.umiacs.umd.edu/help/user/markdown.md

==Repository URLs==
Users can clone git repos over either SSH or HTTPS. '''We recommend using SSH because it is more convenient for most users.''' The blue "Clone" button in the upper-right corner of your GitLab project page will contain the clone URL for whichever authentication method you prefer.

===SSH===
To use SSH, add an SSH key to your GitLab account. Instructions to do so can be found [https://gitlab.umiacs.umd.edu/help/user/ssh.md here]. Once you have the SSH key added, you can authenticate without having to enter any credentials from the machine and account for which you created the SSH key (if your key is not password-protected).

===HTTPS===
The HTTPS URL can be used as well, but you will need to setup Personal Access Tokens in order to connect. Instructions to create one can be found [https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html here]. Once you have a token, you can use that as your password when prompted. You will have to re-enter your credentials for GitLab every single time that you attempt to commit to, push to, pull from, or otherwise access the remote repository.

==Make GitLab projects public==
Making projects public grants access to everyone. Projects can only made public by the owner. To make your projects on GitLab public, follow the steps below:

1. Go to your project page.

2. Click on '''Settings'''.

3. Scroll down to '''Project Visibility'''. Choose '''Public''' from the drop-down menu.

4. Scroll down and click on '''Save Changes'''

5. Scroll down to '''Repository''' under '''Project Visibility''' and select '''Everyone with access''' from the drop-down menu.

6. Click on '''Save Changes'''.

Now your project has become public with its repository accessible by anyone.

Nexus

2026-04-30T18:04:52Z

Mbaney:

{{Note|UMIACS Technical Staff will begin the process of upgrading the operating system version on all Nexus cluster nodes in Summer 2026. Please see [[Nexus/ClusterOSUpgrade]] for more information.}}

The Nexus is the combined scheduler of resources in UMIACS. The resource manager for Nexus is [[SLURM]]. Resources are arranged into partitions where users are able to schedule computational jobs. Users are arranged into a number of SLURM accounts based on faculty, lab, or center investments.

= Getting Started =
All accounts in UMIACS are sponsored. If you don't already have a UMIACS account, please see [[Accounts]] for information on getting one. You need a full UMIACS account - not a [[Accounts/Collaborator | collaborator account]] - in order to access Nexus.

== Access ==
Your access to submission nodes (alternatively called login nodes) for Nexus computational resources is determined by your account sponsor's department, center, or lab affiliation. You can log into the [https://intranet.umiacs.umd.edu/directory/cr/ UMIACS Directory CR application] and select the Computational Resource (CR) in the list that has the prefix <code>nexus</code>. The Hosts section lists your available submission nodes - generally a pair of nodes with hostnames of the format <tt>nexus<department, lab, or center abbreviation>[00,01]</tt>, e.g., <tt>nexusgroup00</tt> and <tt>nexusgroup01</tt>.

Once you have identified your submission nodes, you can [[SSH]] into them [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0016076 after connecting to UMD's GlobalProtect VPN]. From there, you are able to submit to the cluster via our [[SLURM]] workload manager. You need to make sure that your submitted jobs have the correct account, partition, and qos.

Please read our [[Nexus/Submission_Node_Policy|Submission Node Policy]] for guidance on appropriate usage of a submission node. If a submission node becomes unresponsive due to disregarding this policy, we may kill user processes on these nodes to resolve the issue. We reserve the right to take action on users who repeatedly cause issues on submission nodes.

== Jobs ==
[[SLURM]] jobs are [[SLURM/JobSubmission | submitted]] by either <code>srun</code> or <code>sbatch</code> depending if you are doing an interactive job or batch job, respectively. You need to provide the where/how/who to run the job and specify the resources you need to run with.

For the who/where/how, you may be required to specify <code>--account</code>, <code>--partition</code>, and/or <code>--qos</code> (respectively) to be able to adequately submit jobs to the Nexus.

For resources, you may need to specify <code>--time</code> for time, <code>--cpus-per-task</code> for CPUs, <code>--mem</code> for RAM, and <code>--gres=gpu</code> for GPUs in your submission arguments to meet your requirements. There are defaults for all four; if you don't specify something, you will get the default value for that resource, which is minimal (e.g., by default, NO GPUs are included if you do not specify <code>--gres=gpu</code>). For more information about submission flags for GPU resources, see [[SLURM/JobSubmission#Requesting_GPUs | here]]. You may also use <code>--ntasks</code> to specify the number of parallel processes to run, with each task having its own set of the resources specified above. You can run <code>man srun</code> on your submission node for a complete list of available submission arguments.

For a list of available GPU types on Nexus and their specs, please see [[Nexus/GPUs]].

For details on how the network for Nexus is architected, please see [[Nexus/Network]]. This can be important if you wish to optimize performance of your jobs.

=== Interactive ===
Once logged into a submission node, you can run simple interactive jobs. If your session is interrupted from the submission node, the job will be killed. As such, we encourage use of a terminal multiplexer such as [[Tmux]].

<pre>
$ srun --pty --cpus-per-task=4 --mem=2gb --gres=gpu:1 bash
srun: Job account was unset; set to user default of 'nexus'
srun: Job partition was unset; set to cluster default of 'tron'
srun: Job QoS was unset; set to association default of 'default'
srun: Job time limit was unset; set to partition default of 60 minutes
srun: job 1 queued and waiting for resources
srun: job 1 has been allocated resources
$ hostname
tron62.umiacs.umd.edu
$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-daad6a04-a2ce-1183-ce53-b267048f750a)
</pre>

=== Batch ===
Batch jobs are scheduled with a script file with an optional ability to embed job scheduling parameters via variables that are defined by <code>#SBATCH</code> lines at the top of the file. You can find some examples in our [[SLURM/JobSubmission]] documentation.

= Partitions =
The SLURM resource manager uses partitions to act as job queues which can restrict size, time and user limits. The Nexus has a number of different partitions of resources. Different Centers, Labs, and Faculty are able to invest in computational resources that are restricted to approved users through these partitions.

'''Partitions usable by all non-[[ClassAccounts |class account]] users:'''
* [[Nexus/Tron]] - Pool of resources available to all non-class accounts sponsored by either UMIACS or CSD faculty.
* Scavenger - [https://slurm.schedmd.com/preempt.html Preemption] partition that contains [https://en.wikipedia.org/wiki/X86-64 x86_64] architecture nodes from multiple other partitions. More resources are available to schedule simultaneously than in other partitions, however jobs are subject to preemption rules. You are responsible for ensuring your jobs handle this preemption correctly. The SLURM scheduler will simply restart a preempted job with the same submission arguments when it is available to run again. For an overview of things you can check within scripts to determine if your job was preempted/resumed, see [[SLURM/Preemption]].
* Scavenger (aarch64) - Preemption partition identical in design to <tt>scavenger</tt>, but only contains [https://en.wikipedia.org/wiki/AArch64 aarch64] architecture nodes.

'''Partitions usable by [[ClassAccounts]]:'''
* [[ClassAccounts#Cluster_Usage | Class]] - Pool of resources available to class accounts sponsored by either UMIACS or CSD faculty.

'''Partitions usable by specific lab/center users:'''
* [[Nexus/CBCB]] - CBCB lab pool available for CBCB lab members.
* [[Nexus/CLIP]] - CLIP lab pool available for CLIP lab members.
* [[Nexus/CML]] - CML lab pool available for CML lab members.
* [[Nexus/GAMMA]] - GAMMA lab pool available for GAMMA lab members.
* [[Nexus/MBRC]] - MBRC lab pool available for MBRC lab members.
* [[Nexus/MC2]] - MC2 lab pool available for MC2 lab members.
* [[Nexus/QuICS]] - QuICS lab pool available for QuICS lab members.
* [[Nexus/Vulcan]] - Vulcan lab pool available for Vulcan lab members.

You can view the partitions that you have access to by using the <code>show_partitions</code> command. By default, the command will show only the partitions that are available to you.

<pre>
$ show_partitions
Name AllowAccounts AllowQos MaxNodes Nodes
------------------------ ----------------------- ------------------------------ ----------- ----------------------------
scavenger scavenger scavenger UNLIMITED brigid[16-19]
cbcb[00-29]
clip[00-13]
cml[00,02-13,15-28,30-33]
cmlcpu[00-04,06-07]
gammagpu[00-21]
legacy[00-11,13-28,30-36]
legacygpu[00-07]
quics00
tron[00-44,46-69]
vulcan[00-45]
------------------------------------------------------------------------------------------------------------------------
scavenger-aarch64 scavenger scavenger-aarch64 UNLIMITED oasis[00-39]
------------------------------------------------------------------------------------------------------------------------
tron nexus default UNLIMITED tron[00-44,46-69]
high
medium
</pre>

If you want to see information for all of the partitions, including those that you do not have access to, you can use the <code>show_partitions --all</code> command.

<pre>
$ show_partitions --all
Name AllowAccounts AllowQos MaxNodes Nodes
------------------------ ----------------------- ------------------------------ ----------- ----------------------------
cbcb cbcb default UNLIMITED cbcb[00-20,22-29]
medium legacy[00-11,13-28,30-36]
high
huge-long
highmem
------------------------------------------------------------------------------------------------------------------------
cbcb-heng cbcb-heng default UNLIMITED cbcb[26-29]
medium
high
huge-long
highmem
------------------------------------------------------------------------------------------------------------------------
cbcb-interactive cbcb interactive UNLIMITED cbcb21
...
</pre>

= Quality of Service (QoS) =
SLURM uses Quality of Service (QoS) both to provide limits on job sizes (termed by us as "job QoS") as well as to limit resources used by all jobs running in a partition, either per user or per group (termed by us as "partition QoS").

=== Job QoS ===
Job QoS are used to provide limits on the size of job that you can run. You should try to allocate only the resources your job actually needs, as resources that each of your jobs schedules are counted against your [[SLURM/Priority#Fair-share | fair-share priority]] in the future.
* default - Default job QoS. Limited to 4 CPU cores, 1 GPU, and 32GB RAM per job. The maximum wall time per job is 3 days.
* medium - Limited to 8 CPU cores, 2 GPUs, and 64GB RAM per job. The maximum wall time per job is 2 days.
* high - Limited to 16 CPU cores, 4 GPUs, and 128GB RAM per job. The maximum wall time per job is 1 day.
* scavenger - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. 11% of the total resources available for each trackable resource type in the partition (CPUs/GPUs/RAM) is permitted simultaneously across all of your jobs running with this job QoS, enforced via the corresponding partition QoS (below) for the scavenger partition. This job QoS is paired one-to-one with the scavenger partition. To use this job QoS, include <code>--partition=scavenger</code> and <code>--account=scavenger</code> in your submission arguments. Do not include any job QoS argument other than <code>--qos=scavenger</code> (optional) or submission will fail.
* scavenger-aarch64 - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. This job QoS is paired one-to-one with the scavenger-aarch64 partition. To use this job QoS, include <code>--partition=scavenger-aarch64</code>, <code>--account=scavenger</code>, and <code>--qos=scavenger-aarch64</code> in your submission arguments.

You can display these job QoS from the command line using the <code>show_qos</code> command. By default, the command will only show job QoS that you can access. The above five job QoS are the ones that everyone can access.

<pre>
$ show_qos
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
scavenger 3-00:00:00
scavenger-aarch64 3-00:00:00
</pre>

If you want to see all job QoS, including those that you do not have access to, you can use the <code>show_qos --all</code> command.

<pre>
$ show_qos --all
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
cml-cpu 7-00:00:00 8
cml-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
cml-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
cml-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
cml-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
cml-scavenger 3-00:00:00 gres/gpu=24
cml-very_high 1-12:00:00 cpu=32,gres/gpu=8,mem=256G 8 gres/gpu=12
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
gamma-huge-long 10-00:00:00 cpu=32,gres/gpu=16,mem=256G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
highmem 21-00:00:00 cpu=128,mem=2T
huge-long 10-00:00:00 cpu=32,gres/gpu=8,mem=256G
interactive 12:00:00 cpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
oasis-exempt 10-00:00:00 cpu=160,mem=28114M
scavenger 3-00:00:00
scavenger-aarch64 3-00:00:00
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-exempt 7-00:00:00 cpu=32,gres/gpu=8,mem=256G 2
vulcan-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
vulcan-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-sailon 3-00:00:00 cpu=32,gres/gpu=8,mem=256G gres/gpu=48
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
vulcan-scavenger-mu+ 3-00:00:00 cpu=288,gres/gpu=72,mem=1152G
</pre>

You are able to submit to any partition that is listed in the <code>show_partitions</code> command. If you need to use an account other than the default account <tt>nexus</tt>, you will need to specify it via the <code>--account</code> submission argument.

=== Partition QoS ===
Partition QoS are used to limit resources used by all jobs running in a partition, either per user (MaxTRESPU) or per group (GrpTRES).

To view partition QoS, use the <code>show_partition_qos</code> command.

<pre>
$ show_partition_qos
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
scavenger-aarch64_part 500
scavenger_part 500 cpu=11%,gres/gpu=11%,mem=11%
tron 500 cpu=32,gres/gpu=4,mem=262144M
</pre>

The scavenger_part partition QoS has relative TRES limits based on the current hardware in a given partition, represented with percentages. To see the current actual TRES limits of this partition QoS, you can use the <code>-r/--real</code> argument.

<pre>
$ show_partition_qos -r
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
scavenger-aarch64_part 500
scavenger_part 500 cpu=888,gres/gpu=140,mem=12574G
tron 500 cpu=32,gres/gpu=4,mem=262144M
</pre>

If you want to see all partition QoS, including those that you do not have access to, you can use the <code>show_partition_qos --all</code> command.

<pre>
$ show_partition_qos --all
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
cbcb 500 cpu=1406,mem=50359G
cbcb-heng 500
cbcb-interactive 500
class 500 cpu=32,gres/gpu=4,mem=262144M
clip 500 cpu=726,mem=6939G
cml 500 cpu=1226,mem=12116G
cml-cpu 500
cml-director 500
cml-furongh 500
cml-scavenger 500 gres/gpu=24
cml-sfeizi 500
cml-wriva 500
cml-wriva-high 500
csd-h200 500
gamma 500 cpu=906,mem=7675G
mbrc 500 cpu=370,mem=3571G
mc2 500 cpu=330,mem=3201G
oasis 500
quics 500 cpu=458,mem=4710G
scavenger-aarch64_part 500
scavenger_part 500 cpu=11%,gres/gpu=11%,mem=11%
tron 500 cpu=32,gres/gpu=4,mem=262144M
vulcan 500 cpu=1402,mem=12936G
vulcan-ampere 500
vulcan-cpu 500
vulcan-ramani 500
vulcan-scavenger 500
vulcan-scavenger-multi 500
</pre>

'''NOTE''': These QoS cannot be used directly when submitting jobs. Partition QoS limits apply to all jobs running on a given partition, regardless of what job QoS is used.

For example, in the default non-preemption partition (<tt>tron</tt>), you are restricted to 32 total CPU cores, 4 total GPUs, and 256GB total RAM at once across all jobs you have running in the partition.

Lab/group-specific partitions may also have their own user limits, and/or may also have group limits on the total number of resources consumed simultaneously by all users that are using their partition, codified by the line in the output above that matches their lab/group name. Note that the values listed above in the two "TRES" columns are not fixed and may fluctuate per-partition as more resources are added to or removed from each partition.

'''All partitions also only allow a maximum of 500 submitted (running (R) or pending (PD)) jobs per user in the partition simultaneously.''' This is to prevent excess pending jobs causing [https://slurm.schedmd.com/sched_config.html#backfill backfill] issues with the SLURM scheduler.
* If you need to submit more than 500 jobs in batch at once, you can develop and run an "outer submission script" that repeatedly attempts to run an "inner submission script" (your original submission script) to submit jobs in the batch periodically, until all job submissions are successful. The outer submission script should use looping logic to check if you are at the max job limit and should then retry submission after waiting for some time interval.
: An example outer submission script is as follows. In this example, <code>example_inner.sh</code> is your inner submission script and is not an [[SLURM/ArrayJobs | array job]], and you want to run 1000 jobs. If your inner submission script is an array job, adjust the number of jobs accordingly. Array jobs must be of size 500 or less.
<pre>
#!/bin/bash
numjobs=1000
i=0
while [ $i -lt $numjobs ]
do
while [[ "$(sbatch example_inner.sh 2>&1)" =~ "QOSMaxSubmitJobPerUserLimit" ]]
do
echo "Currently at maximum job submissions allowed for this QoS."
echo "Waiting for 5 minutes before trying to submit more jobs."
sleep 300
done
i=$(( $i + 1 ))
echo "Submitted job $i of $numjobs"
done
</pre>

It is suggested that you run the outer submission script in a [[Tmux]] session to keep the terminal window executing it from being interrupted.

= Storage =
All network storage available in Nexus is currently [[NFS]] based, and comes in a few different flavors. Compute nodes also have local scratch storage that can be used.

== Home Directories ==
{{Nfshomes}}

== Scratch Directories ==
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Nexus compute infrastructure:
* Network scratch directories
* Local scratch directories

Please note that [[ClassAccounts | class accounts]] do not have network scratch directories.

=== Network Scratch Directories ===
You are allocated 200GB of scratch space via NFS from <code>/fs/nexus-scratch/<USERNAME></code> where <USERNAME> is your UMIACS username. '''It is not backed up or protected in any way.''' This directory is '''[[Automounter | automounted]]'''; you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access it.

You can view your quota usage by running <code>df -h /fs/nexus-scratch/<USERNAME></code>.

You may request a permanent increase of up to 400GB total space without any faculty approval by [[HelpDesk | contacting staff]]. If you need space beyond 400GB, you will need faculty approval and/or a [[#Project_Allocations | project allocation]] for this. If you choose to increase your scratch space beyond 400GB, the increased space is also subject to the 270 TB days limit mentioned in the project allocation section before we check back in for renewal. For example, if you request 1.4TB total space, you may have this for 270 days (1TB beyond the 400GB permanent increase). The amount increased beyond 400GB will also count against your faculty member's 20TB total storage limit mentioned below.

This file system is available on all submission, data management, and computational nodes within the cluster.

=== Local Scratch Directories ===
Each computational node that you can schedule compute jobs on also has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. and '''are not backed up or protected in any way.''' These directories are almost always more performant than any other storage available to the job as they are mounted from disks directly attached to the compute node. However, you must stage your data within the confines of your job and extract the relevant resultant data elsewhere before the end of your job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month during our [[MonthlyMaintenanceWindow | monthly maintenance windows]]. Please make sure you secure any resultant data you wish to keep from these directories at the end of your job.

== Faculty Allocations ==
Each faculty member can be allocated 1TB of permanent lab space upon request. We can also support grouping these individual allocations together into larger center, lab, or research group allocations if desired by the faculty. Please [[HelpDesk | contact staff]] to inquire.

Lab space storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

== Project Allocations ==
Project allocations are available per user for 270 TB days; you can have a 1TB allocation for up to 270 days, a 3TB allocation for 90 days, etc..

A single faculty member can not have more than 20TB of project allocations across all of their sponsored accounts active simultaneously. Network scratch allocation space increases beyond the 400GB permanent maximum also have the increase count against this limit (i.e., a 1TB network scratch allocation would have 600GB counted towards this limit).

Project storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

The maximum allocation length you can request is 540 days (500GB space) and the maximum storage space you can request is 9TB (30 day length).

To request an allocation, please [[HelpDesk | contact staff]] with the faculty member(s) that the project is under involved in the conversation. Please include the following details:
* Project Name (short)
* Description
* Size (1TB, 2TB, etc.)
* Length in days (270 days, 135 days, etc.)
* Other user(s) that need to access the allocation, if any

These allocations are available via <code>/fs/nexus-projects/<project name></code>. '''Renewal is not guaranteed to be available due to limits on the amount of total storage.''' Near the end of the allocation period, staff will contact you and ask if you are still in need of the storage allocation. If renewal is available, you can renew for up to another 270 TB days with reapproval from the original faculty approver.
* If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation.
* If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible.
** If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible.
** If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

== Datasets ==
We have read-only dataset storage available at <code>/fs/nexus-datasets</code>. If there are datasets that you would like to see curated and made available, please see [[Datasets | this page]].

The list of Nexus datasets we currently host can be viewed [https://info.umiacs.umd.edu/datasets/list/?q=Nexus here].

SLURM/Priority

2026-04-30T16:40:32Z

Mbaney:

[[SLURM]] at UMIACS is configured to prioritize jobs based on a number of factors, termed [https://slurm.schedmd.com/priority_multifactor.html multifactor priority] in SLURM. Each job submitted to the scheduler is assigned a priority value, which can be viewed in the output of <code>scontrol show job <jobid></code>.

Example:
<pre>
$ scontrol show job 1
JobId=1 JobName=bash
UserId=username(13337) GroupId=username(13337) MCS_label=N/A
Priority=2000841 Nice=0 Account=nexus QOS=default
...
</pre>

==Pending Jobs==
If the partition that you submit your job to cannot start your job instantly due to no compute node(s) in the partition having the resources free to run it, your job will remain in the Pending state with the listed reason <tt>(Resources)</tt>. If there is another job already pending with this reason in a partition, you submit a job to the same partition, and your job gets assigned a lower priority value than that pending job, your job will instead remain in the Pending state with reason <tt>(Priority)</tt>. If there are multiple jobs pending and your job is not the highest priority job pending, the scheduler will only start execution of your job if doing so would not push the start times for any higher priority jobs in the same partition further back.

Lowering some combination of the resources you are requesting and/or the time limit may allow submitted jobs to start sooner (or instantly) during times where a partition is under resource pressure. The command <code>squeue -j <jobid> --start</code> can be used to provide a time estimate for when your job will start, where <jobid> is the job ID you receive from either srun or sbatch. This time is subject to change depending on if other users' jobs end sooner or more jobs get submitted.

You can use the command alias <code>[[SLURM/JobSubmission#show_available_nodes | show_available_nodes]]</code> with a variety of different submission arguments to get a better idea of what jobs may be able to start sooner, but the output of this command alias is not definitive, for reasons mentioned in the footnotes on the page linked to.

==Priority Factors==
The priority factors in use at UMIACS are, from most-heavily to least-heavily weighted:
* Partition job was submitted to
* Fair-share of resources within SLURM account
* Age of job, i.e., time spent waiting to run in the queue
* Association/SLURM account being used
* "Nice" value that job was submitted with

===Partition===
The partitions whose names are or are prefixed with <code>scavenger</code> on our clusters are always in a lower priority tier and always have lower priority factors for their jobs than all other partitions on that cluster. As mentioned in other UMIACS cluster-specific documentation, jobs submitted to these partitions are also [https://slurm.schedmd.com/preempt.html preemptable]. These two design choices give the partitions their names; jobs submitted to <code>scavenger</code> named or prefixed partitions "scavenge" for available resources on the cluster rather than consume dedicated resources, and are interrupted by jobs asking to consume dedicated resources.

On [[Nexus]], labs/centers may also have their own scavenger partitions, i.e., <code><labname>-scavenger</code>, if the faculty for the lab/center have decided upon some sort of limit on jobs, such as number of simultaneous jobs, number of actively consumed billing resources, etc., in their non-scavenger partitions. These lab/center scavenger partitions allow for more jobs to be run by members of that lab/center on that lab's/center's nodes only, but jobs on these partitions are preemptable by jobs in that lab's/center's non-scavenger partitions and/or account-specific partitions, if any account-specific partitions containing a given node exist. Jobs submitted to lab/center scavenger partitions will preempt jobs submitted to the institute-wide scavenger partitions (running on nodes that are also in those lab/center scavenger partitions).

In decreasing order of priority (highest first), our priority tiers for partitions are:
# Priority access account-specific partitions
# Account-specific partitions
# Lab/center-specific and institute-wide non-"scavenger" named partitions
# Lab/center-specific "scavenger" named partitions
# Institute-wide "scavenger" named partitions

A job in a specific priority tier will never have a higher priority value than any job in a higher priority tier. Corresponding to the above tiers, the priority values that you will see for jobs in each tier:
# >= 4000000
# 3000000 to 3999999
# 2000000 to 2999999
# 1000000 to 1999999
# < 1000000

As such, '''jobs on specific nodes in some non-"scavenger" named partitions may also be subject to preemption''' based on these priority tiers. Generally speaking, though, most nodes are only in one partition in one of the first three (non-"scavenger") priority tiers, and then in an institute-wide "scavenger" named partition, and a lab/center-specific "scavenger" named partition, if one exists for the lab/center that a given node is a part of.

===Fair-share===
The more resources your jobs have already consumed within an account, the lower priority factor your future jobs will have when compared to other users' jobs in the same account who have used fewer resources (so as to "fair-share" with other users). Additionally, if there are multiple accounts that can submit to a partition, and the sum of resources used by all users' jobs within account A is greater than the sum of resources used by all users' jobs within account B, all future jobs from users in account A will have a lower priority factor when compared to all future jobs from users in account B. (In other words, fair-share is hierarchical.)

You can view the various fair-share statistics with the command <code>sshare -l</code>. It will show your specific FairShare values (always between 0.0 and 1.0) within accounts that you have access to. You can also view other accounts' Level Fairshare (LevelFS).
<pre>
Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS GrpTRESMins TRESRunMins
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------
root 0.000000 68444174744 1.000000 cpu=4797787,mem=70530109515,e+
cbcb 1 0.028571 4454658377 0.065046 0.065046 0.439246 cpu=452139,mem=22276633804,en+
class 1 0.028571 255617290 0.003733 0.003733 7.652841 cpu=7021,mem=74554606,energy=+
clip 1 0.028571 3057933838 0.044674 0.044674 0.639549 cpu=33214,mem=2744443460,ener+
cml 1 0.028571 66866114 0.000975 0.000975 29.299389 cpu=1796,mem=29426756,energy=+
gamma 1 0.028571 2609474948 0.038129 0.038129 0.749334 cpu=34089,mem=360373862,energ+
mbrc 1 0.028571 73411964 0.001073 0.001073 26.635560 cpu=1195,mem=4896358,energy=0+
mc2 1 0.028571 2682557 0.000039 0.000039 728.919551 cpu=0,mem=0,energy=0,node=0,b+
nexus 1 0.028571 5472794067 0.079964 0.079964 0.357302 cpu=278464,mem=3250599000,ene+
nexus username 1 0.000835 69666 0.000001 0.000021 0.457407 37.435501 cpu=0,mem=0,energy=0,node=0,b+
oasis 1 0.028571 330030 0.000005 0.000005 5.9248e+03 cpu=0,mem=0,energy=0,node=0,b+
quics 1 0.028571 4 0.000000 0.000000 4.1683e+08 cpu=0,mem=0,energy=0,node=0,b+
scavenger 1 0.028571 40888195964 0.597419 0.597419 0.047825 cpu=3142204,mem=29902903931,e+
scavenger username 1 0.000835 171 0.000000 0.000000 0.033975 9.8885e+04 cpu=0,mem=0,energy=0,node=0,b+
vulcan 1 0.028571 1247236491 0.018224 0.018224 1.567761 cpu=147273,mem=1161243818,ene+
</pre>

The actual resource billing weights for the three main resources (memory per GB, CPU cores, and number of GPUs if applicable) are per-partition and can be viewed in the <code>TRESBillingWeights</code> line in the output of <code>scontrol show partition</code>. The <code>billing</code> value for a job is the sum of all resource weightings for resources the job has requested. This value is then multiplied by the amount of time a job has run in seconds to get the amount it contributes to the RawUsage for the association within the account it is running under.

====Algorithm====
The algorithm we use for resource weightings differs depending on if there are any GPUs in a partition or not, and is as follows:

=====GPU partitions=====
Each resource (memory/CPU/GPU) is given a weighting value such that their relative billings to each other within the partition are equal (33.33% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU/GPU values are adjusted accordingly.

Different GPU types may also be weighted differently within the GPU relative billing. A baseline GPU type is first chosen. All GPUs of that type and other types that have lower FP32 performance (in [https://en.wikipedia.org/wiki/FLOPS TFLOPS]) are given a weighting factor of 1.0. GPU types with higher FP32 performance than the baseline GPU are given a weighting factor calculated by dividing their FP32 performance by the baseline GPU's FP32 performance. The weighting values for each GPU type are then determined by normalizing the sum of all of GPU cards' billing values multiplied by their weighting factors against the relative billing percentage for GPUs (33.33%).

The current baseline GPU is the [https://www.nvidia.com/en-us/design-visualization/rtx-a4000/ NVIDIA RTX A4000].

=====CPU-only partitions=====
Each resource (memory/CPU) is first given a weighting value such that their relative billings to each other within the partition are equal (50% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU value is adjusted accordingly. The final CPU weight value is then divided by 10, which translates to roughly 90.9% of the billing weight being for memory and 9.1% being for CPU. The division of the CPU value is done so as to not affect accounts' fair-share priority factors as much when running jobs in CPU-only partitions given the popularity of GPGPU computing.

===Age===
The longer a job is eligible to run but cannot due to resources being unavailable or it having a lower priority value than one or more other jobs, the higher the job's priority becomes as it continues to wait in the queue. This is the only priority modifier that can change a job's priority value once it has been submitted, and the priority modifier for this factor reaches its limit after 7 days.

Jobs' age priority factors on our clusters are recalculated every 5 minutes.

===Association===
Some lab/center-specific SLURM accounts have priority values directly attached to them. Jobs run under these accounts gain this many extra points of priority.

===Nice value===
This is a submission argument that you as the user can include when submitting your jobs to deprioritize them. Larger values will deprioritize jobs more, e.g.,
<pre>srun --pty --nice=2 bash</pre>
will have lower priority than
<pre>srun --pty --nice=1 bash</pre>
which will have lower priority than
<pre>srun --pty bash</pre>
assuming all three jobs were submitted at the same time. You cannot use negative values for this argument.

Because this value is absolute, if you want to use it, we would recommend using small numbers - one or two digits - only. Larger numbers may impact your job's ability to run at all as a result of the other factors at play.

SLURM/Priority

2026-04-29T20:53:06Z

Mbaney: /* Pending Jobs */

[[SLURM]] at UMIACS is configured to prioritize jobs based on a number of factors, termed [https://slurm.schedmd.com/priority_multifactor.html multifactor priority] in SLURM. Each job submitted to the scheduler is assigned a priority value, which can be viewed in the output of <code>scontrol show job <jobid></code>.

Example:
<pre>
$ scontrol show job 1
JobId=1 JobName=bash
UserId=username(13337) GroupId=username(13337) MCS_label=N/A
Priority=2000841 Nice=0 Account=nexus QOS=default
...
</pre>

==Pending Jobs==
If the partition that you submit your job to cannot start your job instantly due to no compute node(s) in the partition having the resources free to run it, your job will remain in the Pending state with the listed reason <tt>(Resources)</tt>. If there is another job already pending with this reason in a partition, you submit a job to the same partition, and your job gets assigned a lower priority value than that pending job, your job will instead remain in the Pending state with reason <tt>(Priority)</tt>. If there are multiple jobs pending and your job is not the highest priority job pending, the scheduler will only start execution of your job if starting your job would not push the start times for any higher priority jobs in the same partition further back.

Lowering some combination of the resources you are requesting and/or the time limit may allow submitted jobs to start sooner (or instantly) during times where a partition is under resource pressure. The command <code>squeue -j <jobid> --start</code> can be used to provide a time estimate for when your job will start, where <jobid> is the job ID you receive from either srun or sbatch. This time is subject to change depending on if other users' jobs end sooner or more jobs get submitted.

You can use the command alias <code>[[SLURM/JobSubmission#show_available_nodes | show_available_nodes]]</code> with a variety of different submission arguments to get a better idea of what jobs may be able to start sooner, but the output of this command alias is not definitive, for reasons mentioned in the footnotes on the page linked to.

==Priority Factors==
The priority factors in use at UMIACS are, from most-heavily to least-heavily weighted:
* Partition job was submitted to
* Fair-share of resources within SLURM account
* Age of job, i.e., time spent waiting to run in the queue
* Association/SLURM account being used
* "Nice" value that job was submitted with

===Partition===
The partitions whose names are or are prefixed with <code>scavenger</code> on our clusters are always in a lower priority tier and always have lower priority factors for their jobs than all other partitions on that cluster. As mentioned in other UMIACS cluster-specific documentation, jobs submitted to these partitions are also [https://slurm.schedmd.com/preempt.html preemptable]. These two design choices give the partitions their names; jobs submitted to <code>scavenger</code> named or prefixed partitions "scavenge" for available resources on the cluster rather than consume dedicated resources, and are interrupted by jobs asking to consume dedicated resources.

On [[Nexus]], labs/centers may also have their own scavenger partitions, i.e., <code><labname>-scavenger</code>, if the faculty for the lab/center have decided upon some sort of limit on jobs, such as number of simultaneous jobs, number of actively consumed billing resources, etc., in their non-scavenger partitions. These lab/center scavenger partitions allow for more jobs to be run by members of that lab/center on that lab's/center's nodes only, but jobs on these partitions are preemptable by jobs in that lab's/center's non-scavenger partitions and/or account-specific partitions, if any account-specific partitions containing a given node exist. Jobs submitted to lab/center scavenger partitions will preempt jobs submitted to the institute-wide scavenger partitions (running on nodes that are also in those lab/center scavenger partitions).

In decreasing order of priority (highest first), our priority tiers for partitions are:
# Priority access account-specific partitions
# Account-specific partitions
# Lab/center-specific and institute-wide non-"scavenger" named partitions
# Lab/center-specific "scavenger" named partitions
# Institute-wide "scavenger" named partitions

A job in a specific priority tier will never have a higher priority value than any job in a higher priority tier. Corresponding to the above tiers, the priority values that you will see for jobs in each tier:
# >= 4000000
# 3000000 to 3999999
# 2000000 to 2999999
# 1000000 to 1999999
# < 1000000

As such, '''jobs on specific nodes in some non-"scavenger" named partitions may also be subject to preemption''' based on these priority tiers. Generally speaking, though, most nodes are only in one partition in one of the first three (non-"scavenger") priority tiers, and then in an institute-wide "scavenger" named partition, and a lab/center-specific "scavenger" named partition, if one exists for the lab/center that a given node is a part of.

===Fair-share===
The more resources your jobs have already consumed within an account, the lower priority factor your future jobs will have when compared to other users' jobs in the same account who have used fewer resources (so as to "fair-share" with other users). Additionally, if there are multiple accounts that can submit to a partition, and the sum of resources used by all users' jobs within account A is greater than the sum of resources used by all users' jobs within account B, all future jobs from users in account A will have a lower priority factor when compared to all future jobs from users in account B. (In other words, fair-share is hierarchical.)

You can view the various fair-share statistics with the command <code>sshare -l</code>. It will show your specific FairShare values (always between 0.0 and 1.0) within accounts that you have access to. You can also view other accounts' Level Fairshare (LevelFS).
<pre>
Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS GrpTRESMins TRESRunMins
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------
root 0.000000 68444174744 1.000000 cpu=4797787,mem=70530109515,e+
cbcb 1 0.028571 4454658377 0.065046 0.065046 0.439246 cpu=452139,mem=22276633804,en+
class 1 0.028571 255617290 0.003733 0.003733 7.652841 cpu=7021,mem=74554606,energy=+
clip 1 0.028571 3057933838 0.044674 0.044674 0.639549 cpu=33214,mem=2744443460,ener+
cml 1 0.028571 66866114 0.000975 0.000975 29.299389 cpu=1796,mem=29426756,energy=+
gamma 1 0.028571 2609474948 0.038129 0.038129 0.749334 cpu=34089,mem=360373862,energ+
mbrc 1 0.028571 73411964 0.001073 0.001073 26.635560 cpu=1195,mem=4896358,energy=0+
mc2 1 0.028571 2682557 0.000039 0.000039 728.919551 cpu=0,mem=0,energy=0,node=0,b+
nexus 1 0.028571 5472794067 0.079964 0.079964 0.357302 cpu=278464,mem=3250599000,ene+
nexus username 1 0.000835 69666 0.000001 0.000021 0.457407 37.435501 cpu=0,mem=0,energy=0,node=0,b+
oasis 1 0.028571 330030 0.000005 0.000005 5.9248e+03 cpu=0,mem=0,energy=0,node=0,b+
quics 1 0.028571 4 0.000000 0.000000 4.1683e+08 cpu=0,mem=0,energy=0,node=0,b+
scavenger 1 0.028571 40888195964 0.597419 0.597419 0.047825 cpu=3142204,mem=29902903931,e+
scavenger username 1 0.000835 171 0.000000 0.000000 0.033975 9.8885e+04 cpu=0,mem=0,energy=0,node=0,b+
vulcan 1 0.028571 1247236491 0.018224 0.018224 1.567761 cpu=147273,mem=1161243818,ene+
</pre>

The actual resource billing weights for the three main resources (memory per GB, CPU cores, and number of GPUs if applicable) are per-partition and can be viewed in the <code>TRESBillingWeights</code> line in the output of <code>scontrol show partition</code>. The <code>billing</code> value for a job is the sum of all resource weightings for resources the job has requested. This value is then multiplied by the amount of time a job has run in seconds to get the amount it contributes to the RawUsage for the association within the account it is running under.

====Algorithm====
The algorithm we use for resource weightings differs depending on if there are any GPUs in a partition or not, and is as follows:

=====GPU partitions=====
Each resource (memory/CPU/GPU) is given a weighting value such that their relative billings to each other within the partition are equal (33.33% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU/GPU values are adjusted accordingly.

Different GPU types may also be weighted differently within the GPU relative billing. A baseline GPU type is first chosen. All GPUs of that type and other types that have lower FP32 performance (in [https://en.wikipedia.org/wiki/FLOPS TFLOPS]) are given a weighting factor of 1.0. GPU types with higher FP32 performance than the baseline GPU are given a weighting factor calculated by dividing their FP32 performance by the baseline GPU's FP32 performance. The weighting values for each GPU type are then determined by normalizing the sum of all of GPU cards' billing values multiplied by their weighting factors against the relative billing percentage for GPUs (33.33%).

The current baseline GPU is the [https://www.nvidia.com/en-us/design-visualization/rtx-a4000/ NVIDIA RTX A4000].

=====CPU-only partitions=====
Each resource (memory/CPU) is first given a weighting value such that their relative billings to each other within the partition are equal (50% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU value is adjusted accordingly. The final CPU weight value is then divided by 10, which translates to roughly 90.9% of the billing weight being for memory and 9.1% being for CPU. The division of the CPU value is done so as to not affect accounts' fair-share priority factors as much when running jobs in CPU-only partitions given the popularity of GPGPU computing.

===Age===
The longer a job is eligible to run but cannot due to resources being unavailable or it having a lower priority value than one or more other jobs, the higher the job's priority becomes as it continues to wait in the queue. This is the only priority modifier that can change a job's priority value once it has been submitted, and the priority modifier for this factor reaches its limit after 7 days.

Jobs' age priority factors on our clusters are recalculated every 5 minutes.

===Association===
Some lab/center-specific SLURM accounts have priority values directly attached to them. Jobs run under these accounts gain this many extra points of priority.

===Nice value===
This is a submission argument that you as the user can include when submitting your jobs to deprioritize them. Larger values will deprioritize jobs more, e.g.,
<pre>srun --pty --nice=2 bash</pre>
will have lower priority than
<pre>srun --pty --nice=1 bash</pre>
which will have lower priority than
<pre>srun --pty bash</pre>
assuming all three jobs were submitted at the same time. You cannot use negative values for this argument.

Because this value is absolute, if you want to use it, we would recommend using small numbers - one or two digits - only. Larger numbers may impact your job's ability to run at all as a result of the other factors at play.

SLURM/Priority

2026-04-29T20:50:55Z

Mbaney: /* Age */

[[SLURM]] at UMIACS is configured to prioritize jobs based on a number of factors, termed [https://slurm.schedmd.com/priority_multifactor.html multifactor priority] in SLURM. Each job submitted to the scheduler is assigned a priority value, which can be viewed in the output of <code>scontrol show job <jobid></code>.

Example:
<pre>
$ scontrol show job 1
JobId=1 JobName=bash
UserId=username(13337) GroupId=username(13337) MCS_label=N/A
Priority=2000841 Nice=0 Account=nexus QOS=default
...
</pre>

==Pending Jobs==
If the partition that you submit your job to cannot begin your job instantly due to no compute node(s) in the partition having the resources free to run it, your job will remain in the Pending state with the listed reason <tt>(Resources)</tt>. If there is another job already pending with this reason in a partition, you submit a job to the same partition, and your job gets assigned a lower priority value than that pending job, your job will instead remain in the Pending state with reason <tt>(Priority)</tt>. If there are multiple jobs pending and your job is not the highest priority job pending, the scheduler will only begin execution of your job if starting your job would not push the begin times for any higher priority jobs in the same partition further back.

Lowering some combination of the resources you are requesting and/or the time limit may allow submitted jobs to start sooner (or instantly) during times where a partition is under resource pressure. The command <code>squeue -j <jobid> --start</code> can be used to provide a time estimate for when your job will start, where <jobid> is the job ID you receive from either srun or sbatch. This time is subject to change depending on if other users' jobs end sooner or more jobs get submitted.

You can use the command alias <code>[[SLURM/JobSubmission#show_available_nodes | show_available_nodes]]</code> with a variety of different submission arguments to get a better idea of what jobs may be able to begin sooner, but the output of this command alias is not definitive, for reasons mentioned in the footnotes on the page linked to.

==Priority Factors==
The priority factors in use at UMIACS are, from most-heavily to least-heavily weighted:
* Partition job was submitted to
* Fair-share of resources within SLURM account
* Age of job, i.e., time spent waiting to run in the queue
* Association/SLURM account being used
* "Nice" value that job was submitted with

===Partition===
The partitions whose names are or are prefixed with <code>scavenger</code> on our clusters are always in a lower priority tier and always have lower priority factors for their jobs than all other partitions on that cluster. As mentioned in other UMIACS cluster-specific documentation, jobs submitted to these partitions are also [https://slurm.schedmd.com/preempt.html preemptable]. These two design choices give the partitions their names; jobs submitted to <code>scavenger</code> named or prefixed partitions "scavenge" for available resources on the cluster rather than consume dedicated resources, and are interrupted by jobs asking to consume dedicated resources.

On [[Nexus]], labs/centers may also have their own scavenger partitions, i.e., <code><labname>-scavenger</code>, if the faculty for the lab/center have decided upon some sort of limit on jobs, such as number of simultaneous jobs, number of actively consumed billing resources, etc., in their non-scavenger partitions. These lab/center scavenger partitions allow for more jobs to be run by members of that lab/center on that lab's/center's nodes only, but jobs on these partitions are preemptable by jobs in that lab's/center's non-scavenger partitions and/or account-specific partitions, if any account-specific partitions containing a given node exist. Jobs submitted to lab/center scavenger partitions will preempt jobs submitted to the institute-wide scavenger partitions (running on nodes that are also in those lab/center scavenger partitions).

In decreasing order of priority (highest first), our priority tiers for partitions are:
# Priority access account-specific partitions
# Account-specific partitions
# Lab/center-specific and institute-wide non-"scavenger" named partitions
# Lab/center-specific "scavenger" named partitions
# Institute-wide "scavenger" named partitions

A job in a specific priority tier will never have a higher priority value than any job in a higher priority tier. Corresponding to the above tiers, the priority values that you will see for jobs in each tier:
# >= 4000000
# 3000000 to 3999999
# 2000000 to 2999999
# 1000000 to 1999999
# < 1000000

As such, '''jobs on specific nodes in some non-"scavenger" named partitions may also be subject to preemption''' based on these priority tiers. Generally speaking, though, most nodes are only in one partition in one of the first three (non-"scavenger") priority tiers, and then in an institute-wide "scavenger" named partition, and a lab/center-specific "scavenger" named partition, if one exists for the lab/center that a given node is a part of.

===Fair-share===
The more resources your jobs have already consumed within an account, the lower priority factor your future jobs will have when compared to other users' jobs in the same account who have used fewer resources (so as to "fair-share" with other users). Additionally, if there are multiple accounts that can submit to a partition, and the sum of resources used by all users' jobs within account A is greater than the sum of resources used by all users' jobs within account B, all future jobs from users in account A will have a lower priority factor when compared to all future jobs from users in account B. (In other words, fair-share is hierarchical.)

You can view the various fair-share statistics with the command <code>sshare -l</code>. It will show your specific FairShare values (always between 0.0 and 1.0) within accounts that you have access to. You can also view other accounts' Level Fairshare (LevelFS).
<pre>
Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS GrpTRESMins TRESRunMins
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------
root 0.000000 68444174744 1.000000 cpu=4797787,mem=70530109515,e+
cbcb 1 0.028571 4454658377 0.065046 0.065046 0.439246 cpu=452139,mem=22276633804,en+
class 1 0.028571 255617290 0.003733 0.003733 7.652841 cpu=7021,mem=74554606,energy=+
clip 1 0.028571 3057933838 0.044674 0.044674 0.639549 cpu=33214,mem=2744443460,ener+
cml 1 0.028571 66866114 0.000975 0.000975 29.299389 cpu=1796,mem=29426756,energy=+
gamma 1 0.028571 2609474948 0.038129 0.038129 0.749334 cpu=34089,mem=360373862,energ+
mbrc 1 0.028571 73411964 0.001073 0.001073 26.635560 cpu=1195,mem=4896358,energy=0+
mc2 1 0.028571 2682557 0.000039 0.000039 728.919551 cpu=0,mem=0,energy=0,node=0,b+
nexus 1 0.028571 5472794067 0.079964 0.079964 0.357302 cpu=278464,mem=3250599000,ene+
nexus username 1 0.000835 69666 0.000001 0.000021 0.457407 37.435501 cpu=0,mem=0,energy=0,node=0,b+
oasis 1 0.028571 330030 0.000005 0.000005 5.9248e+03 cpu=0,mem=0,energy=0,node=0,b+
quics 1 0.028571 4 0.000000 0.000000 4.1683e+08 cpu=0,mem=0,energy=0,node=0,b+
scavenger 1 0.028571 40888195964 0.597419 0.597419 0.047825 cpu=3142204,mem=29902903931,e+
scavenger username 1 0.000835 171 0.000000 0.000000 0.033975 9.8885e+04 cpu=0,mem=0,energy=0,node=0,b+
vulcan 1 0.028571 1247236491 0.018224 0.018224 1.567761 cpu=147273,mem=1161243818,ene+
</pre>

The actual resource billing weights for the three main resources (memory per GB, CPU cores, and number of GPUs if applicable) are per-partition and can be viewed in the <code>TRESBillingWeights</code> line in the output of <code>scontrol show partition</code>. The <code>billing</code> value for a job is the sum of all resource weightings for resources the job has requested. This value is then multiplied by the amount of time a job has run in seconds to get the amount it contributes to the RawUsage for the association within the account it is running under.

====Algorithm====
The algorithm we use for resource weightings differs depending on if there are any GPUs in a partition or not, and is as follows:

=====GPU partitions=====
Each resource (memory/CPU/GPU) is given a weighting value such that their relative billings to each other within the partition are equal (33.33% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU/GPU values are adjusted accordingly.

Different GPU types may also be weighted differently within the GPU relative billing. A baseline GPU type is first chosen. All GPUs of that type and other types that have lower FP32 performance (in [https://en.wikipedia.org/wiki/FLOPS TFLOPS]) are given a weighting factor of 1.0. GPU types with higher FP32 performance than the baseline GPU are given a weighting factor calculated by dividing their FP32 performance by the baseline GPU's FP32 performance. The weighting values for each GPU type are then determined by normalizing the sum of all of GPU cards' billing values multiplied by their weighting factors against the relative billing percentage for GPUs (33.33%).

The current baseline GPU is the [https://www.nvidia.com/en-us/design-visualization/rtx-a4000/ NVIDIA RTX A4000].

=====CPU-only partitions=====
Each resource (memory/CPU) is first given a weighting value such that their relative billings to each other within the partition are equal (50% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU value is adjusted accordingly. The final CPU weight value is then divided by 10, which translates to roughly 90.9% of the billing weight being for memory and 9.1% being for CPU. The division of the CPU value is done so as to not affect accounts' fair-share priority factors as much when running jobs in CPU-only partitions given the popularity of GPGPU computing.

===Age===
The longer a job is eligible to run but cannot due to resources being unavailable or it having a lower priority value than one or more other jobs, the higher the job's priority becomes as it continues to wait in the queue. This is the only priority modifier that can change a job's priority value once it has been submitted, and the priority modifier for this factor reaches its limit after 7 days.

Jobs' age priority factors on our clusters are recalculated every 5 minutes.

===Association===
Some lab/center-specific SLURM accounts have priority values directly attached to them. Jobs run under these accounts gain this many extra points of priority.

===Nice value===
This is a submission argument that you as the user can include when submitting your jobs to deprioritize them. Larger values will deprioritize jobs more, e.g.,
<pre>srun --pty --nice=2 bash</pre>
will have lower priority than
<pre>srun --pty --nice=1 bash</pre>
which will have lower priority than
<pre>srun --pty bash</pre>
assuming all three jobs were submitted at the same time. You cannot use negative values for this argument.

Because this value is absolute, if you want to use it, we would recommend using small numbers - one or two digits - only. Larger numbers may impact your job's ability to run at all as a result of the other factors at play.

SLURM/Priority

2026-04-29T20:49:45Z

Mbaney: /* Fair-share */

[[SLURM]] at UMIACS is configured to prioritize jobs based on a number of factors, termed [https://slurm.schedmd.com/priority_multifactor.html multifactor priority] in SLURM. Each job submitted to the scheduler is assigned a priority value, which can be viewed in the output of <code>scontrol show job <jobid></code>.

Example:
<pre>
$ scontrol show job 1
JobId=1 JobName=bash
UserId=username(13337) GroupId=username(13337) MCS_label=N/A
Priority=2000841 Nice=0 Account=nexus QOS=default
...
</pre>

==Pending Jobs==
If the partition that you submit your job to cannot begin your job instantly due to no compute node(s) in the partition having the resources free to run it, your job will remain in the Pending state with the listed reason <tt>(Resources)</tt>. If there is another job already pending with this reason in a partition, you submit a job to the same partition, and your job gets assigned a lower priority value than that pending job, your job will instead remain in the Pending state with reason <tt>(Priority)</tt>. If there are multiple jobs pending and your job is not the highest priority job pending, the scheduler will only begin execution of your job if starting your job would not push the begin times for any higher priority jobs in the same partition further back.

Lowering some combination of the resources you are requesting and/or the time limit may allow submitted jobs to start sooner (or instantly) during times where a partition is under resource pressure. The command <code>squeue -j <jobid> --start</code> can be used to provide a time estimate for when your job will start, where <jobid> is the job ID you receive from either srun or sbatch. This time is subject to change depending on if other users' jobs end sooner or more jobs get submitted.

You can use the command alias <code>[[SLURM/JobSubmission#show_available_nodes | show_available_nodes]]</code> with a variety of different submission arguments to get a better idea of what jobs may be able to begin sooner, but the output of this command alias is not definitive, for reasons mentioned in the footnotes on the page linked to.

==Priority Factors==
The priority factors in use at UMIACS are, from most-heavily to least-heavily weighted:
* Partition job was submitted to
* Fair-share of resources within SLURM account
* Age of job, i.e., time spent waiting to run in the queue
* Association/SLURM account being used
* "Nice" value that job was submitted with

===Partition===
The partitions whose names are or are prefixed with <code>scavenger</code> on our clusters are always in a lower priority tier and always have lower priority factors for their jobs than all other partitions on that cluster. As mentioned in other UMIACS cluster-specific documentation, jobs submitted to these partitions are also [https://slurm.schedmd.com/preempt.html preemptable]. These two design choices give the partitions their names; jobs submitted to <code>scavenger</code> named or prefixed partitions "scavenge" for available resources on the cluster rather than consume dedicated resources, and are interrupted by jobs asking to consume dedicated resources.

On [[Nexus]], labs/centers may also have their own scavenger partitions, i.e., <code><labname>-scavenger</code>, if the faculty for the lab/center have decided upon some sort of limit on jobs, such as number of simultaneous jobs, number of actively consumed billing resources, etc., in their non-scavenger partitions. These lab/center scavenger partitions allow for more jobs to be run by members of that lab/center on that lab's/center's nodes only, but jobs on these partitions are preemptable by jobs in that lab's/center's non-scavenger partitions and/or account-specific partitions, if any account-specific partitions containing a given node exist. Jobs submitted to lab/center scavenger partitions will preempt jobs submitted to the institute-wide scavenger partitions (running on nodes that are also in those lab/center scavenger partitions).

In decreasing order of priority (highest first), our priority tiers for partitions are:
# Priority access account-specific partitions
# Account-specific partitions
# Lab/center-specific and institute-wide non-"scavenger" named partitions
# Lab/center-specific "scavenger" named partitions
# Institute-wide "scavenger" named partitions

A job in a specific priority tier will never have a higher priority value than any job in a higher priority tier. Corresponding to the above tiers, the priority values that you will see for jobs in each tier:
# >= 4000000
# 3000000 to 3999999
# 2000000 to 2999999
# 1000000 to 1999999
# < 1000000

As such, '''jobs on specific nodes in some non-"scavenger" named partitions may also be subject to preemption''' based on these priority tiers. Generally speaking, though, most nodes are only in one partition in one of the first three (non-"scavenger") priority tiers, and then in an institute-wide "scavenger" named partition, and a lab/center-specific "scavenger" named partition, if one exists for the lab/center that a given node is a part of.

===Fair-share===
The more resources your jobs have already consumed within an account, the lower priority factor your future jobs will have when compared to other users' jobs in the same account who have used fewer resources (so as to "fair-share" with other users). Additionally, if there are multiple accounts that can submit to a partition, and the sum of resources used by all users' jobs within account A is greater than the sum of resources used by all users' jobs within account B, all future jobs from users in account A will have a lower priority factor when compared to all future jobs from users in account B. (In other words, fair-share is hierarchical.)

You can view the various fair-share statistics with the command <code>sshare -l</code>. It will show your specific FairShare values (always between 0.0 and 1.0) within accounts that you have access to. You can also view other accounts' Level Fairshare (LevelFS).
<pre>
Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS GrpTRESMins TRESRunMins
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------
root 0.000000 68444174744 1.000000 cpu=4797787,mem=70530109515,e+
cbcb 1 0.028571 4454658377 0.065046 0.065046 0.439246 cpu=452139,mem=22276633804,en+
class 1 0.028571 255617290 0.003733 0.003733 7.652841 cpu=7021,mem=74554606,energy=+
clip 1 0.028571 3057933838 0.044674 0.044674 0.639549 cpu=33214,mem=2744443460,ener+
cml 1 0.028571 66866114 0.000975 0.000975 29.299389 cpu=1796,mem=29426756,energy=+
gamma 1 0.028571 2609474948 0.038129 0.038129 0.749334 cpu=34089,mem=360373862,energ+
mbrc 1 0.028571 73411964 0.001073 0.001073 26.635560 cpu=1195,mem=4896358,energy=0+
mc2 1 0.028571 2682557 0.000039 0.000039 728.919551 cpu=0,mem=0,energy=0,node=0,b+
nexus 1 0.028571 5472794067 0.079964 0.079964 0.357302 cpu=278464,mem=3250599000,ene+
nexus username 1 0.000835 69666 0.000001 0.000021 0.457407 37.435501 cpu=0,mem=0,energy=0,node=0,b+
oasis 1 0.028571 330030 0.000005 0.000005 5.9248e+03 cpu=0,mem=0,energy=0,node=0,b+
quics 1 0.028571 4 0.000000 0.000000 4.1683e+08 cpu=0,mem=0,energy=0,node=0,b+
scavenger 1 0.028571 40888195964 0.597419 0.597419 0.047825 cpu=3142204,mem=29902903931,e+
scavenger username 1 0.000835 171 0.000000 0.000000 0.033975 9.8885e+04 cpu=0,mem=0,energy=0,node=0,b+
vulcan 1 0.028571 1247236491 0.018224 0.018224 1.567761 cpu=147273,mem=1161243818,ene+
</pre>

The actual resource billing weights for the three main resources (memory per GB, CPU cores, and number of GPUs if applicable) are per-partition and can be viewed in the <code>TRESBillingWeights</code> line in the output of <code>scontrol show partition</code>. The <code>billing</code> value for a job is the sum of all resource weightings for resources the job has requested. This value is then multiplied by the amount of time a job has run in seconds to get the amount it contributes to the RawUsage for the association within the account it is running under.

====Algorithm====
The algorithm we use for resource weightings differs depending on if there are any GPUs in a partition or not, and is as follows:

=====GPU partitions=====
Each resource (memory/CPU/GPU) is given a weighting value such that their relative billings to each other within the partition are equal (33.33% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU/GPU values are adjusted accordingly.

Different GPU types may also be weighted differently within the GPU relative billing. A baseline GPU type is first chosen. All GPUs of that type and other types that have lower FP32 performance (in [https://en.wikipedia.org/wiki/FLOPS TFLOPS]) are given a weighting factor of 1.0. GPU types with higher FP32 performance than the baseline GPU are given a weighting factor calculated by dividing their FP32 performance by the baseline GPU's FP32 performance. The weighting values for each GPU type are then determined by normalizing the sum of all of GPU cards' billing values multiplied by their weighting factors against the relative billing percentage for GPUs (33.33%).

The current baseline GPU is the [https://www.nvidia.com/en-us/design-visualization/rtx-a4000/ NVIDIA RTX A4000].

=====CPU-only partitions=====
Each resource (memory/CPU) is first given a weighting value such that their relative billings to each other within the partition are equal (50% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU value is adjusted accordingly. The final CPU weight value is then divided by 10, which translates to roughly 90.9% of the billing weight being for memory and 9.1% being for CPU. The division of the CPU value is done so as to not affect accounts' fair-share priority factors as much when running jobs in CPU-only partitions given the popularity of GPGPU computing.

===Age===
The longer a job is eligible to run but cannot due to resources being unavailable or having a lower priority value than one or more other jobs, the higher the job's priority becomes as it continues to wait in the queue. This is the only priority modifier that can change a job's priority value once it has been submitted, and the priority modifier for this factor reaches its limit after 7 days.

Jobs' age priority factors on our clusters are recalculated every 5 minutes.

===Association===
Some lab/center-specific SLURM accounts have priority values directly attached to them. Jobs run under these accounts gain this many extra points of priority.

===Nice value===
This is a submission argument that you as the user can include when submitting your jobs to deprioritize them. Larger values will deprioritize jobs more, e.g.,
<pre>srun --pty --nice=2 bash</pre>
will have lower priority than
<pre>srun --pty --nice=1 bash</pre>
which will have lower priority than
<pre>srun --pty bash</pre>
assuming all three jobs were submitted at the same time. You cannot use negative values for this argument.

Because this value is absolute, if you want to use it, we would recommend using small numbers - one or two digits - only. Larger numbers may impact your job's ability to run at all as a result of the other factors at play.

SLURM/Priority

2026-04-28T20:51:14Z

Mbaney: /* Age */

[[SLURM]] at UMIACS is configured to prioritize jobs based on a number of factors, termed [https://slurm.schedmd.com/priority_multifactor.html multifactor priority] in SLURM. Each job submitted to the scheduler is assigned a priority value, which can be viewed in the output of <code>scontrol show job <jobid></code>.

Example:
<pre>
$ scontrol show job 1
JobId=1 JobName=bash
UserId=username(13337) GroupId=username(13337) MCS_label=N/A
Priority=2000841 Nice=0 Account=nexus QOS=default
...
</pre>

==Pending Jobs==
If the partition that you submit your job to cannot begin your job instantly due to no compute node(s) in the partition having the resources free to run it, your job will remain in the Pending state with the listed reason <tt>(Resources)</tt>. If there is another job already pending with this reason in a partition, you submit a job to the same partition, and your job gets assigned a lower priority value than that pending job, your job will instead remain in the Pending state with reason <tt>(Priority)</tt>. If there are multiple jobs pending and your job is not the highest priority job pending, the scheduler will only begin execution of your job if starting your job would not push the begin times for any higher priority jobs in the same partition further back.

Lowering some combination of the resources you are requesting and/or the time limit may allow submitted jobs to start sooner (or instantly) during times where a partition is under resource pressure. The command <code>squeue -j <jobid> --start</code> can be used to provide a time estimate for when your job will start, where <jobid> is the job ID you receive from either srun or sbatch. This time is subject to change depending on if other users' jobs end sooner or more jobs get submitted.

You can use the command alias <code>[[SLURM/JobSubmission#show_available_nodes | show_available_nodes]]</code> with a variety of different submission arguments to get a better idea of what jobs may be able to begin sooner, but the output of this command alias is not definitive, for reasons mentioned in the footnotes on the page linked to.

==Priority Factors==
The priority factors in use at UMIACS are, from most-heavily to least-heavily weighted:
* Partition job was submitted to
* Fair-share of resources within SLURM account
* Age of job, i.e., time spent waiting to run in the queue
* Association/SLURM account being used
* "Nice" value that job was submitted with

===Partition===
The partitions whose names are or are prefixed with <code>scavenger</code> on our clusters are always in a lower priority tier and always have lower priority factors for their jobs than all other partitions on that cluster. As mentioned in other UMIACS cluster-specific documentation, jobs submitted to these partitions are also [https://slurm.schedmd.com/preempt.html preemptable]. These two design choices give the partitions their names; jobs submitted to <code>scavenger</code> named or prefixed partitions "scavenge" for available resources on the cluster rather than consume dedicated resources, and are interrupted by jobs asking to consume dedicated resources.

On [[Nexus]], labs/centers may also have their own scavenger partitions, i.e., <code><labname>-scavenger</code>, if the faculty for the lab/center have decided upon some sort of limit on jobs, such as number of simultaneous jobs, number of actively consumed billing resources, etc., in their non-scavenger partitions. These lab/center scavenger partitions allow for more jobs to be run by members of that lab/center on that lab's/center's nodes only, but jobs on these partitions are preemptable by jobs in that lab's/center's non-scavenger partitions and/or account-specific partitions, if any account-specific partitions containing a given node exist. Jobs submitted to lab/center scavenger partitions will preempt jobs submitted to the institute-wide scavenger partitions (running on nodes that are also in those lab/center scavenger partitions).

In decreasing order of priority (highest first), our priority tiers for partitions are:
# Priority access account-specific partitions
# Account-specific partitions
# Lab/center-specific and institute-wide non-"scavenger" named partitions
# Lab/center-specific "scavenger" named partitions
# Institute-wide "scavenger" named partitions

A job in a specific priority tier will never have a higher priority value than any job in a higher priority tier. Corresponding to the above tiers, the priority values that you will see for jobs in each tier:
# >= 4000000
# 3000000 to 3999999
# 2000000 to 2999999
# 1000000 to 1999999
# < 1000000

As such, '''jobs on specific nodes in some non-"scavenger" named partitions may also be subject to preemption''' based on these priority tiers. Generally speaking, though, most nodes are only in one partition in one of the first three (non-"scavenger") priority tiers, and then in an institute-wide "scavenger" named partition, and a lab/center-specific "scavenger" named partition, if one exists for the lab/center that a given node is a part of.

===Fair-share===
The more resources your jobs have already consumed within an account, the lower priority factor your future jobs will have when compared to other users' jobs in the same account who have used fewer resources (so as to "fair-share" with other users). Additionally, if there are multiple accounts that can submit to a partition, and the sum of resources of all users' jobs within account A is greater than the sum of resources of all users' jobs within account B, the lower priority factor all future jobs from users in account A will have when compared to all future jobs from users in account B. (In other words, fair-share is hierarchical.)

You can view the various fair-share statistics with the command <code>sshare -l</code>. It will show your specific FairShare values (always between 0.0 and 1.0) within accounts that you have access to. You can also view other accounts' Level Fairshare (LevelFS).
<pre>
Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS GrpTRESMins TRESRunMins
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------
root 0.000000 68444174744 1.000000 cpu=4797787,mem=70530109515,e+
cbcb 1 0.028571 4454658377 0.065046 0.065046 0.439246 cpu=452139,mem=22276633804,en+
class 1 0.028571 255617290 0.003733 0.003733 7.652841 cpu=7021,mem=74554606,energy=+
clip 1 0.028571 3057933838 0.044674 0.044674 0.639549 cpu=33214,mem=2744443460,ener+
cml 1 0.028571 66866114 0.000975 0.000975 29.299389 cpu=1796,mem=29426756,energy=+
gamma 1 0.028571 2609474948 0.038129 0.038129 0.749334 cpu=34089,mem=360373862,energ+
mbrc 1 0.028571 73411964 0.001073 0.001073 26.635560 cpu=1195,mem=4896358,energy=0+
mc2 1 0.028571 2682557 0.000039 0.000039 728.919551 cpu=0,mem=0,energy=0,node=0,b+
nexus 1 0.028571 5472794067 0.079964 0.079964 0.357302 cpu=278464,mem=3250599000,ene+
nexus username 1 0.000835 69666 0.000001 0.000021 0.457407 37.435501 cpu=0,mem=0,energy=0,node=0,b+
oasis 1 0.028571 330030 0.000005 0.000005 5.9248e+03 cpu=0,mem=0,energy=0,node=0,b+
quics 1 0.028571 4 0.000000 0.000000 4.1683e+08 cpu=0,mem=0,energy=0,node=0,b+
scavenger 1 0.028571 40888195964 0.597419 0.597419 0.047825 cpu=3142204,mem=29902903931,e+
scavenger username 1 0.000835 171 0.000000 0.000000 0.033975 9.8885e+04 cpu=0,mem=0,energy=0,node=0,b+
vulcan 1 0.028571 1247236491 0.018224 0.018224 1.567761 cpu=147273,mem=1161243818,ene+
</pre>

The actual resource billing weights for the three main resources (memory per GB, CPU cores, and number of GPUs if applicable) are per-partition and can be viewed in the <code>TRESBillingWeights</code> line in the output of <code>scontrol show partition</code>. The <code>billing</code> value for a job is the sum of all resource weightings for resources the job has requested. This value is then multiplied by the amount of time a job has run in seconds to get the amount it contributes to the RawUsage for the association within the account it is running under.

====Algorithm====
The algorithm we use for resource weightings differs depending on if there are any GPUs in a partition or not, and is as follows:

=====GPU partitions=====
Each resource (memory/CPU/GPU) is given a weighting value such that their relative billings to each other within the partition are equal (33.33% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU/GPU values are adjusted accordingly.

Different GPU types may also be weighted differently within the GPU relative billing. A baseline GPU type is first chosen. All GPUs of that type and other types that have lower FP32 performance (in [https://en.wikipedia.org/wiki/FLOPS TFLOPS]) are given a weighting factor of 1.0. GPU types with higher FP32 performance than the baseline GPU are given a weighting factor calculated by dividing their FP32 performance by the baseline GPU's FP32 performance. The weighting values for each GPU type are then determined by normalizing the sum of all of GPU cards' billing values multiplied by their weighting factors against the relative billing percentage for GPUs (33.33%).

The current baseline GPU is the [https://www.nvidia.com/en-us/design-visualization/rtx-a4000/ NVIDIA RTX A4000].

=====CPU-only partitions=====
Each resource (memory/CPU) is first given a weighting value such that their relative billings to each other within the partition are equal (50% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU value is adjusted accordingly. The final CPU weight value is then divided by 10, which translates to roughly 90.9% of the billing weight being for memory and 9.1% being for CPU. The division of the CPU value is done so as to not affect accounts' fair-share priority factors as much when running jobs in CPU-only partitions given the popularity of GPGPU computing.

===Age===
The longer a job is eligible to run but cannot due to resources being unavailable or having a lower priority value than one or more other jobs, the higher the job's priority becomes as it continues to wait in the queue. This is the only priority modifier that can change a job's priority value once it has been submitted, and the priority modifier for this factor reaches its limit after 7 days.

Jobs' age priority factors on our clusters are recalculated every 5 minutes.

===Association===
Some lab/center-specific SLURM accounts have priority values directly attached to them. Jobs run under these accounts gain this many extra points of priority.

===Nice value===
This is a submission argument that you as the user can include when submitting your jobs to deprioritize them. Larger values will deprioritize jobs more, e.g.,
<pre>srun --pty --nice=2 bash</pre>
will have lower priority than
<pre>srun --pty --nice=1 bash</pre>
which will have lower priority than
<pre>srun --pty bash</pre>
assuming all three jobs were submitted at the same time. You cannot use negative values for this argument.

Because this value is absolute, if you want to use it, we would recommend using small numbers - one or two digits - only. Larger numbers may impact your job's ability to run at all as a result of the other factors at play.

Accounts/UMD

2026-04-28T14:30:59Z

Mbaney:

Before getting a [[Accounts | UMIACS account]], any prospective UMIACS account holder needs to already hold a UMD account, consisting of a UMD directory ID and associated passphrase. Their UMIACS account will have the same username as the directory ID and also use the same passphrase for authentication, once it is installed.
* [https://terpmail.umd.edu/ TERPmail] accounts are separate from UMD accounts - these accounts are used for student email and exist in perpetuity after a student graduates from UMD, but are '''not''' sufficient for holding UMIACS accounts.

Current UMD students, faculty, and staff should already have UMD accounts. For external collaborators, interns, visitors, etc., at other institutions or otherwise, faculty members will need to request affiliate appointments for them to get them UMD accounts. An overview of the different types of affiliate appointments can be found [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0017232 here], in the attached document. To actually request an appointment, fill out [https://wiki.umiacs.umd.edu/umiacs/images/7/7a/Affiliate_Data_Collection_Form_Blank3.pdf this form] and send it to [mailto:umiacs-payroll@umiacs.umd.edu umiacs-payroll@umiacs.umd.edu].
* Affiliate appointments will also need to be requested for students that have graduated for their UMD (and thus UMIACS) access to remain active.

Once the person gets their appointment and thus their UMD account, determine which type of UMIACS account they need. The two options are [[Accounts#UMIACS_Account | full account]] or [[Accounts/Collaborator | collaborator account]].
* If full: they can now fill out the [https://intranet.umiacs.umd.edu/requests/accounts/new full UMIACS Account Request form] using their directory ID.
* If collaborator: you can now fill out the [https://intranet.umiacs.umd.edu/requests/accounts/collaborators/new Collaborator Account Request form] for them, using their <directoryid>@umd.edu email address. They will receive an email to that address with instructions to set up the account.

Nexus/Vulcan

2026-04-27T19:57:24Z

Mbaney:

The compute nodes from Vulcan's previous standalone cluster have folded into [[Nexus]] as of the scheduled [[MonthlyMaintenanceWindow | maintenance window]] for August 2023 (Thursday 08/17/2023, 5-8pm).

The Nexus cluster already has a large pool of compute resources made possible through college-level funding for UMIACS and CSD faculty. Details on common nodes already in the cluster (Tron partition) can be found [[Nexus/Tron | here]].

Please [[HelpDesk | contact staff]] with any questions or concerns.

==Usage==
You can [[SSH]] to <code>nexusvulcan.umiacs.umd.edu</code> to log in to a submission node.

If you store something in a local directory (/tmp, /scratch0) on one of the two submission nodes, you will need to connect to that same submission node to access it later. The actual submission nodes are:
* <code>nexusvulcan00.umiacs.umd.edu</code>
* <code>nexusvulcan01.umiacs.umd.edu</code>

All partitions, QoSes, and account names from the standalone Vulcan cluster have been moved over to Nexus. However, please note that <code>vulcan-</code> is prepended to all of the values that were present in the standalone Vulcan cluster to distinguish them from existing values in Nexus. The lone exception is the base account that was named <code>vulcan</code> in the standalone cluster (it is also named just <code>vulcan</code> in Nexus).

Here are some before/after examples of job submission with various parameters:

{| class="wikitable"
! Standalone Vulcan cluster submission command
! Nexus cluster submission command
|-
|<code>srun --partition=dpart --qos=medium --account=abhinav --gres=gpu:gtx1080ti:2 --pty bash</code>
|<code>srun --partition=vulcan-dpart --qos=vulcan-medium --account=vulcan-abhinav --gres=gpu:gtx1080ti:2 --pty bash</code>
|-
|<code>srun --partition=cpu --qos=cpu --pty bash</code>
|<code>srun --partition=vulcan-cpu --qos=vulcan-cpu --account=vulcan --pty bash</code>
|-
|<code>srun --partition=scavenger --qos=scavenger --account=vulcan --gres=gpu:4 --pty bash</code>
|<code>srun --partition=vulcan-scavenger --qos=vulcan-scavenger --account=vulcan --gres=gpu:4 --pty bash</code>
|}

Vulcan users (exclusively) can schedule non-interruptible jobs on Vulcan nodes with any non-scavenger job parameters. Please note that the <code>vulcan-dpart</code> partition has a <code>GrpTRES</code> limit of 100% of the available cores/RAM on all vulcan## in aggregate nodes plus 50% of the available cores/RAM on legacy## nodes in aggregate, so your job may need to wait if all available cores/RAM (or GPUs) are in use. It also has a max submission limit of 500 jobs per user simultaneously so as to not overload the cluster. This is codified by the partition QoS named '''vulcan'''.

Please note that the Vulcan compute nodes are also in the institute-wide <code>scavenger</code> partition in Nexus. Vulcan users still have scavenging priority over these nodes via the <code>vulcan-scavenger</code> partition (i.e., all <code>vulcan-</code> partition jobs (other than <code>vulcan-scavenger</code>) can preempt both <code>vulcan-scavenger</code> and <code>scavenger</code> partition jobs, and <code>vulcan-scavenger</code> partition jobs can preempt <code>scavenger</code> partition jobs).

==Compute Nodes==
There are currently 46 [[Nexus/Vulcan/GPUs | GPU nodes]] available, named vulcan[00-45], running a mixture of NVIDIA RTX A6000, NVIDIA RTX A5000, NVIDIA RTX A4000, and a number of different older generation cards. There are also 4 CPU-only nodes available, named brigid[16-19].

All nodes are scheduled with the [[SLURM]] resource manager.

==Network==
The network infrastructure supporting the Vulcan partition consists of:
# One pair of network switches connected to each other via dual 100GbE links for redundancy, serving the following compute nodes:
#* brigid[16-17],vulcan[29-45]: Two 100GbE links per node, one to each switch in the pair (redundancy).
#* vulcan46: Four 100GbE links, two to each switch in the pair (redundancy and increased bandwidth).
# One pair of network switches connected to the above pair of switches through several sets of intermediary switches, and to each other via dual 10GbE links for redundancy. The immediate connection to these sets of intermediary switches is via two 40GbE links to a pair of them, one between the first two switches in each pair and one between the second two switches in each pair for redundancy. This pair serves the following compute nodes:
#* brigid[18-19],vulcan[00-28]: Two 10GbE links per node, one to each switch in the pair (redundancy).

The fileserver hosting all Vulcan [[Nexus/Vulcan#Scratch_Directories | scratch]], [[Nexus/Vulcan#Project_Storage | project]], and [[Nexus/Vulcan#Datasets | dataset]] allocations first connects to a pair of intermediary switches and then the first pair of switches mentioned [[Nexus/Tron#Network | here (Tron page's network section)]]. It then connects to the first pair of switches mentioned on this page through a set of four (different) intermediary switches. The last hop from the four intermediary switches to the first pair of switches mentioned on this page is via 32 100GbE links, four from each switch in the set to each switch in the first pair mentioned on this page for redundancy and increased bandwidth.

For a broader overview of the network infrastructure supporting the Nexus cluster, please see [[Nexus/Network]].

==Partitions==
There are three partitions available to general Vulcan [[SLURM]] users. You must specify a partition when submitting your job.

* '''vulcan-dpart''' - This is the default partition. Job allocations are guaranteed. Only nodes with GPUs from architectures older than NVIDIA's [https://www.nvidia.com/en-us/data-center/ampere-architecture/ Ampere architecture] are included in this partition.
* '''vulcan-scavenger''' - This is the alternate partition that allows jobs longer run times and more resources but is preemptable when jobs in other non-scavenger-named <code>vulcan-</code> partitions are ready to be scheduled.
* '''vulcan-cpu''' - This partition is for CPU focused jobs. Job allocations are guaranteed.

There are a few additional partitions available to subsets of Vulcan users based on specific requirements.

* '''vulcan-ampere''' - This partition contains nodes with GPUs from NVIDIA's [https://www.nvidia.com/en-us/data-center/ampere-architecture/ Ampere architecture] or a newer architecture. Job allocations are guaranteed. Please be aware of the following restrictions on this partition:
*: ''Time limit'': there is a 4 hour time limit on interactive jobs in this partition. If you need to run longer jobs, you will need to modify your workflow into a job that can be submitted as a batch script.
*: ''CPU/memory per GPU limit'': there is a limit of 4 CPUs and 48G memory maximum per non-H200 GPU requested by a job, and 16 CPUs and 256G memory maximum per H200 GPU requested by a job. If you need to run jobs with more CPUs/memory, you will either need to request more GPUs in the job or use a different partition.

: Submission is restricted to the Slurm [[#Accounts | accounts]] of the faculty who invested in these nodes:
:* Abhinav Shrivastava (vulcan-abhinav)
:* Jia-Bin Huang (vulcan-jbhuang)
:* Christopher Metzler (vulcan-metzler)
:* Ruoshi Liu (vulcan-ruoshi)
:* Matthias Zwicker (vulcan-zwicker)

* '''vulcan-scavenger-multi''' - This partition allows multi-node jobs (up to 9 total nodes per job) and allows jobs more resources than the vulcan-scavenger partition, but only contains nodes with GTX 1080 Ti, TITAN Xp, and/or RTX 2080 Ti GPUs in them. As with vulcan-scavenger, it is preemptable when jobs in other non-scavenged-named <code>vulcan-</code> partitions are ready to be scheduled.
*: Access to this partition is on a per-use basis. Please contact Abhinav Shrivastava if you would like to be granted access to this partition.

There is one additional partition available solely to Dr. Ramani Duraiswami's sponsored accounts.

* '''vulcan-ramani''' - This partition is for exclusive priority access to Dr. Duraiswami's purchased GPU nodes. Job allocations are guaranteed.

==Accounts==
Vulcan has a base SLURM account <code>vulcan</code> which has a modest number of guaranteed billing resources available to all cluster users at any given time. Other faculty that have invested in Vulcan compute infrastructure have an additional account provided to their sponsored accounts on the cluster.

If you do not specify an account when submitting your job, you will receive the '''vulcan''' account. If your faculty sponsor has their own account, it is recommended to use that account for job submission.

The current faculty accounts are:
* vulcan-abhinav
* vulcan-djacobs
* vulcan-jbhuang
* vulcan-metzler
* vulcan-rama
* vulcan-ramani
* vulcan-ruoshi
* vulcan-yaser
* vulcan-zwicker

<pre>
$ sacctmgr show account format=account%20,description%30,organization%10
Account Descr Org
-------------------- ------------------------------ ----------
... ... ...
vulcan vulcan vulcan
vulcan-abhinav vulcan - abhinav shrivastava vulcan
vulcan-djacobs vulcan - david jacobs vulcan
vulcan-jbhuang vulcan - jia-bin huang vulcan
vulcan-metzler vulcan - chris metzler vulcan
vulcan-rama vulcan - rama chellappa vulcan
vulcan-ramani vulcan - ramani duraiswami vulcan
vulcan-ruoshi vulcan - ruoshi liu vulcan
vulcan-yaser vulcan - yaser yacoob vulcan
vulcan-zwicker vulcan - matthias zwicker vulcan
... ... ...
</pre>

Faculty can manage this list of users via our [https://intranet.umiacs.umd.edu/directory/secgroup/ Directory application] in the Security Groups section. The security group that controls access has the prefix <code>vulcan_</code> and then the faculty username. It will also list <code>slurm://nexusctl.umiacs.umd.edu</code> as the associated URI.

You can check your account associations by running the '''show_assoc''' command to see the accounts you are associated with. Please [[HelpDesk | contact staff]] and include your faculty member in the conversation if you do not see the appropriate association.

<pre>
$ show_assoc
User Account MaxJobs GrpTRES QOS
---------- ---------------- ------- ------------- --------------------------------------------------------------------------------
... ... ... ...
abhinav vulcan 48 vulcan-cpu,vulcan-default,vulcan-medium,vulcan-scavenger
abhinav vulcan-abhinav 48 vulcan-cpu,vulcan-default,vulcan-high,vulcan-medium,vulcan-scavenger
... ... ... ...
</pre>

You can also see the total number of Track-able Resources (TRES) allowed for each account by running the following command. Please make sure you give the appropriate account that you are looking for. As shown below, there is a concurrent limit of 64 total GPUs for all users not in a contributing faculty group.

<pre>
$ sacctmgr show assoc account=vulcan format=user,account,qos,grptres
User Account QOS GrpTRES
---------- ---------- -------------------- -------------
vulcan gres/gpu=64
... ...
</pre>

==QoS==
Vulcan currently has 3 QoS for the '''vulcan-dpart''' partition, 1 QoS for the '''vulcan-scavenger''' partition, and 1 QoS for the '''vulcan-cpu''' partition. If you do not specify a QoS when submitting your job using the <code>--qos</code> parameter, you will receive the <code>vulcan-default</code> QoS assuming you are using a Vulcan account.

The important part here is that in different QoS you can have a shorter/longer maximum wall time, a different total number of jobs running at once, and a different maximum number of track-able resources (TRES) for the job. In the cml-scavenger QoS, one more constraint that you are restricted by is the total number of TRES per user (over multiple jobs).

<pre>
$ show_qos --all | grep vulcan
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
------------------------- ----------- ------------------------------ --------- ------------------------------
...
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-exempt 7-00:00:00 cpu=32,gres/gpu=8,mem=256G 2
vulcan-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
vulcan-high_long 14-00:00:00 cpu=32,gres/gpu=8 8
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-sailon 3-00:00:00 cpu=32,gres/gpu=8,mem=256G gres/gpu=48
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
vulcan-scavenger-multi 3-00:00:00 cpu=288,gres/gpu=72,mem=1152G
...
</pre>

<pre>
$ show_partition_qos --all | grep vulcan
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- ------------------------------ --------------------
...
vulcan 500 cpu=1760,mem=15824G
vulcan-ampere 500
vulcan-cpu 500
vulcan-ramani 500
vulcan-scavenger 500
vulcan-scavenger-multi 500
...
</pre>

==Storage==
Vulcan has the following storage available. Please also review UMIACS [[FilesystemDataStorage | Filesystem Data Storage]] policies including any volume that is labeled as scratch.

Vulcan users can also request [[Nexus#Project_Allocations | Nexus project allocations]].

===Home Directories===
{{Nfshomes}}

===Scratch Directories===
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Vulcan compute infrastructure:
* Network scratch directory
* Local scratch directories

====Network Scratch Directory====
You have 300GB of scratch storage available at <code>/vulcanscratch/<username></code>. '''It is not backed up or protected in any way.''' This directory is '''automounted''' so you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access this.

You may request a temporary increase of up to 500GB total space for a maximum of 120 days without any faculty approval by [[HelpDesk | contacting staff]]. Once the temporary increase period is over, you will be contacted and given a one-week window of opportunity to clean and secure your data before staff will forcibly remove data to get your space back under 300GB. If you need space beyond 500GB or for longer than 120 days, you will need faculty approval and/or a project directory.

This file system is available on all submission and computational nodes within the cluster.

====Local Scratch Directories====
Each computational node that you can schedule compute jobs on has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. These are almost always more performant than any other storage available to the job. However, you must stage their data within the confine of their job and stage the data out before the end of their job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month at 1am. Different nodes will run the maintenance jobs on different days of the month to ensure the cluster is still highly available at all times. Please make sure you secure any data you write to these directories at the end of your job.

===Datasets===
We have read-only dataset storage available at <code>/fs/vulcan-datasets</code>. If there are datasets that you would like to see curated and made available, please see [[Datasets | this page]].

The list of Vulcan datasets we currently host can be viewed [https://info.umiacs.umd.edu/datasets/list/?q=Vulcan here].

===Project Storage===
Users within the Vulcan compute infrastructure can request project based allocations for up to 10TB for up to 180 days by [[HelpDesk | contacting staff]] with approval from the Vulcan faculty manager (Dr. Shrivastava). These allocations will be available from <code>/fs/vulcan-projects</code> under a name that you provide when you request the allocation. Near the end of the allocation period, staff will contact you and ask if you would like to renew the allocation for up to another 180 days (requires re-approval from Dr. Shrivastava).
* If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation.
* If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible.
** If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible.
** If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

Project storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

===Object Storage===
All Vulcan users can request project allocations in the [https://obj.umiacs.umd.edu/obj/help UMIACS Object Store]. Please [[HelpDesk | contact staff]] with a short project name and the amount of storage you will need to get started.

To access this storage, you'll need to use a [[S3Clients | S3 client]] or our [[UMobj]] command line utilities.

An example on how to use the umobj command line utilities can be found [[UMobj/Example | here]]. A full set of documentation for the utilities can be found on the [https://gitlab.umiacs.umd.edu/staff/umobj/blob/master/README.md#umobj umobj Gitlab page].

Nexus

2026-04-27T17:23:45Z

Mbaney:

{{Note|UMIACS Technical Staff will begin the process of upgrading the operating system version on all Nexus cluster nodes in Summer 2026. Please see [[Nexus/ClusterOSUpgrade]] for more information.}}

The Nexus is the combined scheduler of resources in UMIACS. The resource manager for Nexus is [[SLURM]]. Resources are arranged into partitions where users are able to schedule computational jobs. Users are arranged into a number of SLURM accounts based on faculty, lab, or center investments.

= Getting Started =
All accounts in UMIACS are sponsored. If you don't already have a UMIACS account, please see [[Accounts]] for information on getting one. You need a full UMIACS account (not a [[Accounts/Collaborator | collaborator account]]) in order to access Nexus.

== Access ==
Your access to submission nodes (alternatively called login nodes) for Nexus computational resources is determined by your account sponsor's department, center, or lab affiliation. You can log into the [https://intranet.umiacs.umd.edu/directory/cr/ UMIACS Directory CR application] and select the Computational Resource (CR) in the list that has the prefix <code>nexus</code>. The Hosts section lists your available submission nodes - generally a pair of nodes of the format <tt>nexus<department, lab, or center abbreviation>[00,01]</tt>, e.g., <tt>nexusgroup00</tt> and <tt>nexusgroup01</tt>.

Once you have identified your submission nodes, you can [[SSH]] into them [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0016076 after connecting to UMD's GlobalProtect VPN]. From there, you are able to submit to the cluster via our [[SLURM]] workload manager. You need to make sure that your submitted jobs have the correct account, partition, and qos.

Please read our [[Nexus/Submission_Node_Policy|Submission Node Policy]] for guidance on appropriate usage of a submission node. If a submission node becomes unresponsive due to disregarding this policy, we may kill user processes on these nodes to resolve the issue. We reserve the right to take action on users who repeatedly cause issues on submission nodes.

== Jobs ==
[[SLURM]] jobs are [[SLURM/JobSubmission | submitted]] by either <code>srun</code> or <code>sbatch</code> depending if you are doing an interactive job or batch job, respectively. You need to provide the where/how/who to run the job and specify the resources you need to run with.

For the who/where/how, you may be required to specify <code>--account</code>, <code>--partition</code>, and/or <code>--qos</code> (respectively) to be able to adequately submit jobs to the Nexus.

For resources, you may need to specify <code>--time</code> for time, <code>--cpus-per-task</code> for CPUs, <code>--mem</code> for RAM, and <code>--gres=gpu</code> for GPUs in your submission arguments to meet your requirements. There are defaults for all four; if you don't specify something, you will get the default value for that resource, which is minimal (e.g., by default, NO GPUs are included if you do not specify <code>--gres=gpu</code>). For more information about submission flags for GPU resources, see [[SLURM/JobSubmission#Requesting_GPUs | here]]. You may also use <code>--ntasks</code> to specify the number of parallel processes to run, with each task having its own set of the resources specified above. You can run <code>man srun</code> on your submission node for a complete list of available submission arguments.

For a list of available GPU types on Nexus and their specs, please see [[Nexus/GPUs]].

For details on how the network for Nexus is architected, please see [[Nexus/Network]]. This can be important if you wish to optimize performance of your jobs.

=== Interactive ===
Once logged into a submission node, you can run simple interactive jobs. If your session is interrupted from the submission node, the job will be killed. As such, we encourage use of a terminal multiplexer such as [[Tmux]].

<pre>
$ srun --pty --cpus-per-task=4 --mem=2gb --gres=gpu:1 bash
srun: Job account was unset; set to user default of 'nexus'
srun: Job partition was unset; set to cluster default of 'tron'
srun: Job QoS was unset; set to association default of 'default'
srun: Job time limit was unset; set to partition default of 60 minutes
srun: job 1 queued and waiting for resources
srun: job 1 has been allocated resources
$ hostname
tron62.umiacs.umd.edu
$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-daad6a04-a2ce-1183-ce53-b267048f750a)
</pre>

=== Batch ===
Batch jobs are scheduled with a script file with an optional ability to embed job scheduling parameters via variables that are defined by <code>#SBATCH</code> lines at the top of the file. You can find some examples in our [[SLURM/JobSubmission]] documentation.

= Partitions =
The SLURM resource manager uses partitions to act as job queues which can restrict size, time and user limits. The Nexus has a number of different partitions of resources. Different Centers, Labs, and Faculty are able to invest in computational resources that are restricted to approved users through these partitions.

'''Partitions usable by all non-[[ClassAccounts |class account]] users:'''
* [[Nexus/Tron]] - Pool of resources available to all non-class accounts sponsored by either UMIACS or CSD faculty.
* Scavenger - [https://slurm.schedmd.com/preempt.html Preemption] partition that contains [https://en.wikipedia.org/wiki/X86-64 x86_64] architecture nodes from multiple other partitions. More resources are available to schedule simultaneously than in other partitions, however jobs are subject to preemption rules. You are responsible for ensuring your jobs handle this preemption correctly. The SLURM scheduler will simply restart a preempted job with the same submission arguments when it is available to run again. For an overview of things you can check within scripts to determine if your job was preempted/resumed, see [[SLURM/Preemption]].
* Scavenger (aarch64) - Preemption partition identical in design to <tt>scavenger</tt>, but only contains [https://en.wikipedia.org/wiki/AArch64 aarch64] architecture nodes.

'''Partitions usable by [[ClassAccounts]]:'''
* [[ClassAccounts#Cluster_Usage | Class]] - Pool of resources available to class accounts sponsored by either UMIACS or CSD faculty.

'''Partitions usable by specific lab/center users:'''
* [[Nexus/CBCB]] - CBCB lab pool available for CBCB lab members.
* [[Nexus/CLIP]] - CLIP lab pool available for CLIP lab members.
* [[Nexus/CML]] - CML lab pool available for CML lab members.
* [[Nexus/GAMMA]] - GAMMA lab pool available for GAMMA lab members.
* [[Nexus/MBRC]] - MBRC lab pool available for MBRC lab members.
* [[Nexus/MC2]] - MC2 lab pool available for MC2 lab members.
* [[Nexus/QuICS]] - QuICS lab pool available for QuICS lab members.
* [[Nexus/Vulcan]] - Vulcan lab pool available for Vulcan lab members.

You can view the partitions that you have access to by using the <code>show_partitions</code> command. By default, the command will show only the partitions that are available to you.

<pre>
$ show_partitions
Name AllowAccounts AllowQos MaxNodes Nodes
------------------------ ----------------------- ------------------------------ ----------- ----------------------------
scavenger scavenger scavenger UNLIMITED brigid[16-19]
cbcb[00-29]
clip[00-13]
cml[00,02-13,15-28,30-33]
cmlcpu[00-04,06-07]
gammagpu[00-21]
legacy[00-11,13-28,30-36]
legacygpu[00-07]
quics00
tron[00-44,46-69]
vulcan[00-45]
------------------------------------------------------------------------------------------------------------------------
scavenger-aarch64 scavenger scavenger-aarch64 UNLIMITED oasis[00-39]
------------------------------------------------------------------------------------------------------------------------
tron nexus default UNLIMITED tron[00-44,46-69]
high
medium
</pre>

If you want to see information for all of the partitions, including those that you do not have access to, you can use the <code>show_partitions --all</code> command.

<pre>
$ show_partitions --all
Name AllowAccounts AllowQos MaxNodes Nodes
------------------------ ----------------------- ------------------------------ ----------- ----------------------------
cbcb cbcb default UNLIMITED cbcb[00-20,22-29]
medium legacy[00-11,13-28,30-36]
high
huge-long
highmem
------------------------------------------------------------------------------------------------------------------------
cbcb-heng cbcb-heng default UNLIMITED cbcb[26-29]
medium
high
huge-long
highmem
------------------------------------------------------------------------------------------------------------------------
cbcb-interactive cbcb interactive UNLIMITED cbcb21
...
</pre>

= Quality of Service (QoS) =
SLURM uses Quality of Service (QoS) both to provide limits on job sizes (termed by us as "job QoS") as well as to limit resources used by all jobs running in a partition, either per user or per group (termed by us as "partition QoS").

=== Job QoS ===
Job QoS are used to provide limits on the size of job that you can run. You should try to allocate only the resources your job actually needs, as resources that each of your jobs schedules are counted against your [[SLURM/Priority#Fair-share | fair-share priority]] in the future.
* default - Default job QoS. Limited to 4 CPU cores, 1 GPU, and 32GB RAM per job. The maximum wall time per job is 3 days.
* medium - Limited to 8 CPU cores, 2 GPUs, and 64GB RAM per job. The maximum wall time per job is 2 days.
* high - Limited to 16 CPU cores, 4 GPUs, and 128GB RAM per job. The maximum wall time per job is 1 day.
* scavenger - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. 11% of the total resources available for each trackable resource type in the partition (CPUs/GPUs/RAM) is permitted simultaneously across all of your jobs running with this job QoS, enforced via the corresponding partition QoS (below) for the scavenger partition. This job QoS is paired one-to-one with the scavenger partition. To use this job QoS, include <code>--partition=scavenger</code> and <code>--account=scavenger</code> in your submission arguments. Do not include any job QoS argument other than <code>--qos=scavenger</code> (optional) or submission will fail.
* scavenger-aarch64 - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. This job QoS is paired one-to-one with the scavenger-aarch64 partition. To use this job QoS, include <code>--partition=scavenger-aarch64</code>, <code>--account=scavenger</code>, and <code>--qos=scavenger-aarch64</code> in your submission arguments.

You can display these job QoS from the command line using the <code>show_qos</code> command. By default, the command will only show job QoS that you can access. The above five job QoS are the ones that everyone can access.

<pre>
$ show_qos
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
scavenger 3-00:00:00
scavenger-aarch64 3-00:00:00
</pre>

If you want to see all job QoS, including those that you do not have access to, you can use the <code>show_qos --all</code> command.

<pre>
$ show_qos --all
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
cml-cpu 7-00:00:00 8
cml-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
cml-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
cml-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
cml-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
cml-scavenger 3-00:00:00 gres/gpu=24
cml-very_high 1-12:00:00 cpu=32,gres/gpu=8,mem=256G 8 gres/gpu=12
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
gamma-huge-long 10-00:00:00 cpu=32,gres/gpu=16,mem=256G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
highmem 21-00:00:00 cpu=128,mem=2T
huge-long 10-00:00:00 cpu=32,gres/gpu=8,mem=256G
interactive 12:00:00 cpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
oasis-exempt 10-00:00:00 cpu=160,mem=28114M
scavenger 3-00:00:00
scavenger-aarch64 3-00:00:00
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-exempt 7-00:00:00 cpu=32,gres/gpu=8,mem=256G 2
vulcan-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
vulcan-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-sailon 3-00:00:00 cpu=32,gres/gpu=8,mem=256G gres/gpu=48
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
vulcan-scavenger-mu+ 3-00:00:00 cpu=288,gres/gpu=72,mem=1152G
</pre>

You are able to submit to any partition that is listed in the <code>show_partitions</code> command. If you need to use an account other than the default account <tt>nexus</tt>, you will need to specify it via the <code>--account</code> submission argument.

=== Partition QoS ===
Partition QoS are used to limit resources used by all jobs running in a partition, either per user (MaxTRESPU) or per group (GrpTRES).

To view partition QoS, use the <code>show_partition_qos</code> command.

<pre>
$ show_partition_qos
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
scavenger-aarch64_part 500
scavenger_part 500 cpu=11%,gres/gpu=11%,mem=11%
tron 500 cpu=32,gres/gpu=4,mem=262144M
</pre>

The scavenger_part partition QoS has relative TRES limits based on the current hardware in a given partition, represented with percentages. To see the current actual TRES limits of this partition QoS, you can use the <code>-r/--real</code> argument.

<pre>
$ show_partition_qos -r
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
scavenger-aarch64_part 500
scavenger_part 500 cpu=888,gres/gpu=140,mem=12574G
tron 500 cpu=32,gres/gpu=4,mem=262144M
</pre>

If you want to see all partition QoS, including those that you do not have access to, you can use the <code>show_partition_qos --all</code> command.

<pre>
$ show_partition_qos --all
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
cbcb 500 cpu=1406,mem=50359G
cbcb-heng 500
cbcb-interactive 500
class 500 cpu=32,gres/gpu=4,mem=262144M
clip 500 cpu=726,mem=6939G
cml 500 cpu=1226,mem=12116G
cml-cpu 500
cml-director 500
cml-furongh 500
cml-scavenger 500 gres/gpu=24
cml-sfeizi 500
cml-wriva 500
cml-wriva-high 500
csd-h200 500
gamma 500 cpu=906,mem=7675G
mbrc 500 cpu=370,mem=3571G
mc2 500 cpu=330,mem=3201G
oasis 500
quics 500 cpu=458,mem=4710G
scavenger-aarch64_part 500
scavenger_part 500 cpu=11%,gres/gpu=11%,mem=11%
tron 500 cpu=32,gres/gpu=4,mem=262144M
vulcan 500 cpu=1402,mem=12936G
vulcan-ampere 500
vulcan-cpu 500
vulcan-ramani 500
vulcan-scavenger 500
vulcan-scavenger-multi 500
</pre>

'''NOTE''': These QoS cannot be used directly when submitting jobs. Partition QoS limits apply to all jobs running on a given partition, regardless of what job QoS is used.

For example, in the default non-preemption partition (<tt>tron</tt>), you are restricted to 32 total CPU cores, 4 total GPUs, and 256GB total RAM at once across all jobs you have running in the partition.

Lab/group-specific partitions may also have their own user limits, and/or may also have group limits on the total number of resources consumed simultaneously by all users that are using their partition, codified by the line in the output above that matches their lab/group name. Note that the values listed above in the two "TRES" columns are not fixed and may fluctuate per-partition as more resources are added to or removed from each partition.

'''All partitions also only allow a maximum of 500 submitted (running (R) or pending (PD)) jobs per user in the partition simultaneously.''' This is to prevent excess pending jobs causing [https://slurm.schedmd.com/sched_config.html#backfill backfill] issues with the SLURM scheduler.
* If you need to submit more than 500 jobs in batch at once, you can develop and run an "outer submission script" that repeatedly attempts to run an "inner submission script" (your original submission script) to submit jobs in the batch periodically, until all job submissions are successful. The outer submission script should use looping logic to check if you are at the max job limit and should then retry submission after waiting for some time interval.
: An example outer submission script is as follows. In this example, <code>example_inner.sh</code> is your inner submission script and is not an [[SLURM/ArrayJobs | array job]], and you want to run 1000 jobs. If your inner submission script is an array job, adjust the number of jobs accordingly. Array jobs must be of size 500 or less.
<pre>
#!/bin/bash
numjobs=1000
i=0
while [ $i -lt $numjobs ]
do
while [[ "$(sbatch example_inner.sh 2>&1)" =~ "QOSMaxSubmitJobPerUserLimit" ]]
do
echo "Currently at maximum job submissions allowed for this QoS."
echo "Waiting for 5 minutes before trying to submit more jobs."
sleep 300
done
i=$(( $i + 1 ))
echo "Submitted job $i of $numjobs"
done
</pre>

It is suggested that you run the outer submission script in a [[Tmux]] session to keep the terminal window executing it from being interrupted.

= Storage =
All network storage available in Nexus is currently [[NFS]] based, and comes in a few different flavors. Compute nodes also have local scratch storage that can be used.

== Home Directories ==
{{Nfshomes}}

== Scratch Directories ==
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Nexus compute infrastructure:
* Network scratch directories
* Local scratch directories

Please note that [[ClassAccounts | class accounts]] do not have network scratch directories.

=== Network Scratch Directories ===
You are allocated 200GB of scratch space via NFS from <code>/fs/nexus-scratch/<USERNAME></code> where <USERNAME> is your UMIACS username. '''It is not backed up or protected in any way.''' This directory is '''[[Automounter | automounted]]'''; you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access it.

You can view your quota usage by running <code>df -h /fs/nexus-scratch/<USERNAME></code>.

You may request a permanent increase of up to 400GB total space without any faculty approval by [[HelpDesk | contacting staff]]. If you need space beyond 400GB, you will need faculty approval and/or a [[#Project_Allocations | project allocation]] for this. If you choose to increase your scratch space beyond 400GB, the increased space is also subject to the 270 TB days limit mentioned in the project allocation section before we check back in for renewal. For example, if you request 1.4TB total space, you may have this for 270 days (1TB beyond the 400GB permanent increase). The amount increased beyond 400GB will also count against your faculty member's 20TB total storage limit mentioned below.

This file system is available on all submission, data management, and computational nodes within the cluster.

=== Local Scratch Directories ===
Each computational node that you can schedule compute jobs on also has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. and '''are not backed up or protected in any way.''' These directories are almost always more performant than any other storage available to the job as they are mounted from disks directly attached to the compute node. However, you must stage your data within the confines of your job and extract the relevant resultant data elsewhere before the end of your job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month during our [[MonthlyMaintenanceWindow | monthly maintenance windows]]. Please make sure you secure any resultant data you wish to keep from these directories at the end of your job.

== Faculty Allocations ==
Each faculty member can be allocated 1TB of permanent lab space upon request. We can also support grouping these individual allocations together into larger center, lab, or research group allocations if desired by the faculty. Please [[HelpDesk | contact staff]] to inquire.

Lab space storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

== Project Allocations ==
Project allocations are available per user for 270 TB days; you can have a 1TB allocation for up to 270 days, a 3TB allocation for 90 days, etc..

A single faculty member can not have more than 20TB of project allocations across all of their sponsored accounts active simultaneously. Network scratch allocation space increases beyond the 400GB permanent maximum also have the increase count against this limit (i.e., a 1TB network scratch allocation would have 600GB counted towards this limit).

Project storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

The maximum allocation length you can request is 540 days (500GB space) and the maximum storage space you can request is 9TB (30 day length).

To request an allocation, please [[HelpDesk | contact staff]] with the faculty member(s) that the project is under involved in the conversation. Please include the following details:
* Project Name (short)
* Description
* Size (1TB, 2TB, etc.)
* Length in days (270 days, 135 days, etc.)
* Other user(s) that need to access the allocation, if any

These allocations are available via <code>/fs/nexus-projects/<project name></code>. '''Renewal is not guaranteed to be available due to limits on the amount of total storage.''' Near the end of the allocation period, staff will contact you and ask if you are still in need of the storage allocation. If renewal is available, you can renew for up to another 270 TB days with reapproval from the original faculty approver.
* If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation.
* If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible.
** If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible.
** If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

== Datasets ==
We have read-only dataset storage available at <code>/fs/nexus-datasets</code>. If there are datasets that you would like to see curated and made available, please see [[Datasets | this page]].

The list of Nexus datasets we currently host can be viewed [https://info.umiacs.umd.edu/datasets/list/?q=Nexus here].

Nexus

2026-04-27T17:05:35Z

Mbaney:

{{Note|UMIACS Technical Staff will begin the process of upgrading the operating system version on all Nexus cluster nodes beginning in Summer 2026. Please see [[Nexus/ClusterOSUpgrade]] for more information.}}

The Nexus is the combined scheduler of resources in UMIACS. The resource manager for Nexus is [[SLURM]]. Resources are arranged into partitions where users are able to schedule computational jobs. Users are arranged into a number of SLURM accounts based on faculty, lab, or center investments.

= Getting Started =
All accounts in UMIACS are sponsored. If you don't already have a UMIACS account, please see [[Accounts]] for information on getting one. You need a full UMIACS account (not a [[Accounts/Collaborator | collaborator account]]) in order to access Nexus.

== Access ==
Your access to submission nodes (alternatively called login nodes) for Nexus computational resources is determined by your account sponsor's department, center, or lab affiliation. You can log into the [https://intranet.umiacs.umd.edu/directory/cr/ UMIACS Directory CR application] and select the Computational Resource (CR) in the list that has the prefix <code>nexus</code>. The Hosts section lists your available submission nodes - generally a pair of nodes of the format <tt>nexus<department, lab, or center abbreviation>[00,01]</tt>, e.g., <tt>nexusgroup00</tt> and <tt>nexusgroup01</tt>.

Once you have identified your submission nodes, you can [[SSH]] into them [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0016076 after connecting to UMD's GlobalProtect VPN]. From there, you are able to submit to the cluster via our [[SLURM]] workload manager. You need to make sure that your submitted jobs have the correct account, partition, and qos.

Please read our [[Nexus/Submission_Node_Policy|Submission Node Policy]] for guidance on appropriate usage of a submission node. If a submission node becomes unresponsive due to disregarding this policy, we may kill user processes on these nodes to resolve the issue. We reserve the right to take action on users who repeatedly cause issues on submission nodes.

== Jobs ==
[[SLURM]] jobs are [[SLURM/JobSubmission | submitted]] by either <code>srun</code> or <code>sbatch</code> depending if you are doing an interactive job or batch job, respectively. You need to provide the where/how/who to run the job and specify the resources you need to run with.

For the who/where/how, you may be required to specify <code>--account</code>, <code>--partition</code>, and/or <code>--qos</code> (respectively) to be able to adequately submit jobs to the Nexus.

For resources, you may need to specify <code>--time</code> for time, <code>--cpus-per-task</code> for CPUs, <code>--mem</code> for RAM, and <code>--gres=gpu</code> for GPUs in your submission arguments to meet your requirements. There are defaults for all four; if you don't specify something, you will get the default value for that resource, which is minimal (e.g., by default, NO GPUs are included if you do not specify <code>--gres=gpu</code>). For more information about submission flags for GPU resources, see [[SLURM/JobSubmission#Requesting_GPUs | here]]. You may also use <code>--ntasks</code> to specify the number of parallel processes to run, with each task having its own set of the resources specified above. You can run <code>man srun</code> on your submission node for a complete list of available submission arguments.

For a list of available GPU types on Nexus and their specs, please see [[Nexus/GPUs]].

For details on how the network for Nexus is architected, please see [[Nexus/Network]]. This can be important if you wish to optimize performance of your jobs.

=== Interactive ===
Once logged into a submission node, you can run simple interactive jobs. If your session is interrupted from the submission node, the job will be killed. As such, we encourage use of a terminal multiplexer such as [[Tmux]].

<pre>
$ srun --pty --cpus-per-task=4 --mem=2gb --gres=gpu:1 bash
srun: Job account was unset; set to user default of 'nexus'
srun: Job partition was unset; set to cluster default of 'tron'
srun: Job QoS was unset; set to association default of 'default'
srun: Job time limit was unset; set to partition default of 60 minutes
srun: job 1 queued and waiting for resources
srun: job 1 has been allocated resources
$ hostname
tron62.umiacs.umd.edu
$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-daad6a04-a2ce-1183-ce53-b267048f750a)
</pre>

=== Batch ===
Batch jobs are scheduled with a script file with an optional ability to embed job scheduling parameters via variables that are defined by <code>#SBATCH</code> lines at the top of the file. You can find some examples in our [[SLURM/JobSubmission]] documentation.

= Partitions =
The SLURM resource manager uses partitions to act as job queues which can restrict size, time and user limits. The Nexus has a number of different partitions of resources. Different Centers, Labs, and Faculty are able to invest in computational resources that are restricted to approved users through these partitions.

'''Partitions usable by all non-[[ClassAccounts |class account]] users:'''
* [[Nexus/Tron]] - Pool of resources available to all non-class accounts sponsored by either UMIACS or CSD faculty.
* Scavenger - [https://slurm.schedmd.com/preempt.html Preemption] partition that contains [https://en.wikipedia.org/wiki/X86-64 x86_64] architecture nodes from multiple other partitions. More resources are available to schedule simultaneously than in other partitions, however jobs are subject to preemption rules. You are responsible for ensuring your jobs handle this preemption correctly. The SLURM scheduler will simply restart a preempted job with the same submission arguments when it is available to run again. For an overview of things you can check within scripts to determine if your job was preempted/resumed, see [[SLURM/Preemption]].
* Scavenger (aarch64) - Preemption partition identical in design to <tt>scavenger</tt>, but only contains [https://en.wikipedia.org/wiki/AArch64 aarch64] architecture nodes.

'''Partitions usable by [[ClassAccounts]]:'''
* [[ClassAccounts#Cluster_Usage | Class]] - Pool of resources available to class accounts sponsored by either UMIACS or CSD faculty.

'''Partitions usable by specific lab/center users:'''
* [[Nexus/CBCB]] - CBCB lab pool available for CBCB lab members.
* [[Nexus/CLIP]] - CLIP lab pool available for CLIP lab members.
* [[Nexus/CML]] - CML lab pool available for CML lab members.
* [[Nexus/GAMMA]] - GAMMA lab pool available for GAMMA lab members.
* [[Nexus/MBRC]] - MBRC lab pool available for MBRC lab members.
* [[Nexus/MC2]] - MC2 lab pool available for MC2 lab members.
* [[Nexus/QuICS]] - QuICS lab pool available for QuICS lab members.
* [[Nexus/Vulcan]] - Vulcan lab pool available for Vulcan lab members.

You can view the partitions that you have access to by using the <code>show_partitions</code> command. By default, the command will show only the partitions that are available to you.

<pre>
$ show_partitions
Name AllowAccounts AllowQos MaxNodes Nodes
------------------------ ----------------------- ------------------------------ ----------- ----------------------------
scavenger scavenger scavenger UNLIMITED brigid[16-19]
cbcb[00-29]
clip[00-13]
cml[00,02-13,15-28,30-33]
cmlcpu[00-04,06-07]
gammagpu[00-21]
legacy[00-11,13-28,30-36]
legacygpu[00-07]
quics00
tron[00-44,46-69]
vulcan[00-45]
------------------------------------------------------------------------------------------------------------------------
scavenger-aarch64 scavenger scavenger-aarch64 UNLIMITED oasis[00-39]
------------------------------------------------------------------------------------------------------------------------
tron nexus default UNLIMITED tron[00-44,46-69]
high
medium
</pre>

If you want to see information for all of the partitions, including those that you do not have access to, you can use the <code>show_partitions --all</code> command.

<pre>
$ show_partitions --all
Name AllowAccounts AllowQos MaxNodes Nodes
------------------------ ----------------------- ------------------------------ ----------- ----------------------------
cbcb cbcb default UNLIMITED cbcb[00-20,22-29]
medium legacy[00-11,13-28,30-36]
high
huge-long
highmem
------------------------------------------------------------------------------------------------------------------------
cbcb-heng cbcb-heng default UNLIMITED cbcb[26-29]
medium
high
huge-long
highmem
------------------------------------------------------------------------------------------------------------------------
cbcb-interactive cbcb interactive UNLIMITED cbcb21
...
</pre>

= Quality of Service (QoS) =
SLURM uses Quality of Service (QoS) both to provide limits on job sizes (termed by us as "job QoS") as well as to limit resources used by all jobs running in a partition, either per user or per group (termed by us as "partition QoS").

=== Job QoS ===
Job QoS are used to provide limits on the size of job that you can run. You should try to allocate only the resources your job actually needs, as resources that each of your jobs schedules are counted against your [[SLURM/Priority#Fair-share | fair-share priority]] in the future.
* default - Default job QoS. Limited to 4 CPU cores, 1 GPU, and 32GB RAM per job. The maximum wall time per job is 3 days.
* medium - Limited to 8 CPU cores, 2 GPUs, and 64GB RAM per job. The maximum wall time per job is 2 days.
* high - Limited to 16 CPU cores, 4 GPUs, and 128GB RAM per job. The maximum wall time per job is 1 day.
* scavenger - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. 11% of the total resources available for each trackable resource type in the partition (CPUs/GPUs/RAM) is permitted simultaneously across all of your jobs running with this job QoS, enforced via the corresponding partition QoS (below) for the scavenger partition. This job QoS is paired one-to-one with the scavenger partition. To use this job QoS, include <code>--partition=scavenger</code> and <code>--account=scavenger</code> in your submission arguments. Do not include any job QoS argument other than <code>--qos=scavenger</code> (optional) or submission will fail.
* scavenger-aarch64 - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. This job QoS is paired one-to-one with the scavenger-aarch64 partition. To use this job QoS, include <code>--partition=scavenger-aarch64</code>, <code>--account=scavenger</code>, and <code>--qos=scavenger-aarch64</code> in your submission arguments.

You can display these job QoS from the command line using the <code>show_qos</code> command. By default, the command will only show job QoS that you can access. The above five job QoS are the ones that everyone can access.

<pre>
$ show_qos
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
scavenger 3-00:00:00
scavenger-aarch64 3-00:00:00
</pre>

If you want to see all job QoS, including those that you do not have access to, you can use the <code>show_qos --all</code> command.

<pre>
$ show_qos --all
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
cml-cpu 7-00:00:00 8
cml-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
cml-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
cml-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
cml-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
cml-scavenger 3-00:00:00 gres/gpu=24
cml-very_high 1-12:00:00 cpu=32,gres/gpu=8,mem=256G 8 gres/gpu=12
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
gamma-huge-long 10-00:00:00 cpu=32,gres/gpu=16,mem=256G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
highmem 21-00:00:00 cpu=128,mem=2T
huge-long 10-00:00:00 cpu=32,gres/gpu=8,mem=256G
interactive 12:00:00 cpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
oasis-exempt 10-00:00:00 cpu=160,mem=28114M
scavenger 3-00:00:00
scavenger-aarch64 3-00:00:00
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-exempt 7-00:00:00 cpu=32,gres/gpu=8,mem=256G 2
vulcan-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
vulcan-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-sailon 3-00:00:00 cpu=32,gres/gpu=8,mem=256G gres/gpu=48
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
vulcan-scavenger-mu+ 3-00:00:00 cpu=288,gres/gpu=72,mem=1152G
</pre>

You are able to submit to any partition that is listed in the <code>show_partitions</code> command. If you need to use an account other than the default account <tt>nexus</tt>, you will need to specify it via the <code>--account</code> submission argument.

=== Partition QoS ===
Partition QoS are used to limit resources used by all jobs running in a partition, either per user (MaxTRESPU) or per group (GrpTRES).

To view partition QoS, use the <code>show_partition_qos</code> command.

<pre>
$ show_partition_qos
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
scavenger-aarch64_part 500
scavenger_part 500 cpu=11%,gres/gpu=11%,mem=11%
tron 500 cpu=32,gres/gpu=4,mem=262144M
</pre>

The scavenger_part partition QoS has relative TRES limits based on the current hardware in a given partition, represented with percentages. To see the current actual TRES limits of this partition QoS, you can use the <code>-r/--real</code> argument.

<pre>
$ show_partition_qos -r
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
scavenger-aarch64_part 500
scavenger_part 500 cpu=888,gres/gpu=140,mem=12574G
tron 500 cpu=32,gres/gpu=4,mem=262144M
</pre>

If you want to see all partition QoS, including those that you do not have access to, you can use the <code>show_partition_qos --all</code> command.

<pre>
$ show_partition_qos --all
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
cbcb 500 cpu=1406,mem=50359G
cbcb-heng 500
cbcb-interactive 500
class 500 cpu=32,gres/gpu=4,mem=262144M
clip 500 cpu=726,mem=6939G
cml 500 cpu=1226,mem=12116G
cml-cpu 500
cml-director 500
cml-furongh 500
cml-scavenger 500 gres/gpu=24
cml-sfeizi 500
cml-wriva 500
cml-wriva-high 500
csd-h200 500
gamma 500 cpu=906,mem=7675G
mbrc 500 cpu=370,mem=3571G
mc2 500 cpu=330,mem=3201G
oasis 500
quics 500 cpu=458,mem=4710G
scavenger-aarch64_part 500
scavenger_part 500 cpu=11%,gres/gpu=11%,mem=11%
tron 500 cpu=32,gres/gpu=4,mem=262144M
vulcan 500 cpu=1402,mem=12936G
vulcan-ampere 500
vulcan-cpu 500
vulcan-ramani 500
vulcan-scavenger 500
vulcan-scavenger-multi 500
</pre>

'''NOTE''': These QoS cannot be used directly when submitting jobs. Partition QoS limits apply to all jobs running on a given partition, regardless of what job QoS is used.

For example, in the default non-preemption partition (<tt>tron</tt>), you are restricted to 32 total CPU cores, 4 total GPUs, and 256GB total RAM at once across all jobs you have running in the partition.

Lab/group-specific partitions may also have their own user limits, and/or may also have group limits on the total number of resources consumed simultaneously by all users that are using their partition, codified by the line in the output above that matches their lab/group name. Note that the values listed above in the two "TRES" columns are not fixed and may fluctuate per-partition as more resources are added to or removed from each partition.

'''All partitions also only allow a maximum of 500 submitted (running (R) or pending (PD)) jobs per user in the partition simultaneously.''' This is to prevent excess pending jobs causing [https://slurm.schedmd.com/sched_config.html#backfill backfill] issues with the SLURM scheduler.
* If you need to submit more than 500 jobs in batch at once, you can develop and run an "outer submission script" that repeatedly attempts to run an "inner submission script" (your original submission script) to submit jobs in the batch periodically, until all job submissions are successful. The outer submission script should use looping logic to check if you are at the max job limit and should then retry submission after waiting for some time interval.
: An example outer submission script is as follows. In this example, <code>example_inner.sh</code> is your inner submission script and is not an [[SLURM/ArrayJobs | array job]], and you want to run 1000 jobs. If your inner submission script is an array job, adjust the number of jobs accordingly. Array jobs must be of size 500 or less.
<pre>
#!/bin/bash
numjobs=1000
i=0
while [ $i -lt $numjobs ]
do
while [[ "$(sbatch example_inner.sh 2>&1)" =~ "QOSMaxSubmitJobPerUserLimit" ]]
do
echo "Currently at maximum job submissions allowed for this QoS."
echo "Waiting for 5 minutes before trying to submit more jobs."
sleep 300
done
i=$(( $i + 1 ))
echo "Submitted job $i of $numjobs"
done
</pre>

It is suggested that you run the outer submission script in a [[Tmux]] session to keep the terminal window executing it from being interrupted.

= Storage =
All network storage available in Nexus is currently [[NFS]] based, and comes in a few different flavors. Compute nodes also have local scratch storage that can be used.

== Home Directories ==
{{Nfshomes}}

== Scratch Directories ==
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Nexus compute infrastructure:
* Network scratch directories
* Local scratch directories

Please note that [[ClassAccounts | class accounts]] do not have network scratch directories.

=== Network Scratch Directories ===
You are allocated 200GB of scratch space via NFS from <code>/fs/nexus-scratch/<USERNAME></code> where <USERNAME> is your UMIACS username. '''It is not backed up or protected in any way.''' This directory is '''[[Automounter | automounted]]'''; you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access it.

You can view your quota usage by running <code>df -h /fs/nexus-scratch/<USERNAME></code>.

You may request a permanent increase of up to 400GB total space without any faculty approval by [[HelpDesk | contacting staff]]. If you need space beyond 400GB, you will need faculty approval and/or a [[#Project_Allocations | project allocation]] for this. If you choose to increase your scratch space beyond 400GB, the increased space is also subject to the 270 TB days limit mentioned in the project allocation section before we check back in for renewal. For example, if you request 1.4TB total space, you may have this for 270 days (1TB beyond the 400GB permanent increase). The amount increased beyond 400GB will also count against your faculty member's 20TB total storage limit mentioned below.

This file system is available on all submission, data management, and computational nodes within the cluster.

=== Local Scratch Directories ===
Each computational node that you can schedule compute jobs on also has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. and '''are not backed up or protected in any way.''' These directories are almost always more performant than any other storage available to the job as they are mounted from disks directly attached to the compute node. However, you must stage your data within the confines of your job and extract the relevant resultant data elsewhere before the end of your job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month during our [[MonthlyMaintenanceWindow | monthly maintenance windows]]. Please make sure you secure any resultant data you wish to keep from these directories at the end of your job.

== Faculty Allocations ==
Each faculty member can be allocated 1TB of permanent lab space upon request. We can also support grouping these individual allocations together into larger center, lab, or research group allocations if desired by the faculty. Please [[HelpDesk | contact staff]] to inquire.

Lab space storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

== Project Allocations ==
Project allocations are available per user for 270 TB days; you can have a 1TB allocation for up to 270 days, a 3TB allocation for 90 days, etc..

A single faculty member can not have more than 20TB of project allocations across all of their sponsored accounts active simultaneously. Network scratch allocation space increases beyond the 400GB permanent maximum also have the increase count against this limit (i.e., a 1TB network scratch allocation would have 600GB counted towards this limit).

Project storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

The maximum allocation length you can request is 540 days (500GB space) and the maximum storage space you can request is 9TB (30 day length).

To request an allocation, please [[HelpDesk | contact staff]] with the faculty member(s) that the project is under involved in the conversation. Please include the following details:
* Project Name (short)
* Description
* Size (1TB, 2TB, etc.)
* Length in days (270 days, 135 days, etc.)
* Other user(s) that need to access the allocation, if any

These allocations are available via <code>/fs/nexus-projects/<project name></code>. '''Renewal is not guaranteed to be available due to limits on the amount of total storage.''' Near the end of the allocation period, staff will contact you and ask if you are still in need of the storage allocation. If renewal is available, you can renew for up to another 270 TB days with reapproval from the original faculty approver.
* If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation.
* If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible.
** If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible.
** If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

== Datasets ==
We have read-only dataset storage available at <code>/fs/nexus-datasets</code>. If there are datasets that you would like to see curated and made available, please see [[Datasets | this page]].

The list of Nexus datasets we currently host can be viewed [https://info.umiacs.umd.edu/datasets/list/?q=Nexus here].

Nexus/ClusterOSUpgrade

2026-04-27T17:05:27Z

Mbaney: Created page with "==Overview== UMIACS Technical Staff will begin the process of upgrading the operating system version on all Nexus cluster nodes from Red Hat Enterprise Linux (RHEL) 8 to 9 in Summer 2026. RHEL8 is in the Maintenance Support phase of its life cycle and is transitioning to the Extended Life phase in 2029. More information on Red Hat's lifecycle policy for its operating systems can be found [https://access.redhat.com/support/policy/updates/errata here]. We..."

==Overview==
UMIACS Technical Staff will begin the process of upgrading the operating system version on all [[Nexus]] cluster nodes from [[RHEL | Red Hat Enterprise Linux (RHEL)]] 8 to 9 in Summer 2026.

RHEL8 is in the Maintenance Support phase of its life cycle and is transitioning to the Extended Life phase in 2029. More information on Red Hat's lifecycle policy for its operating systems can be found [https://access.redhat.com/support/policy/updates/errata here]. We are staying well ahead of the Extended Life phase date for our cluster nodes by performing these upgrades now.

RHEL9 is still in the Full Support phase of its life cycle and introduces a newer major Linux kernel version and newer [https://www.gnu.org/software/libc glibc] version, improving compatibility with many newer software applications.

==Scheduling==
'''Upgrades for all cluster nodes will begin as early as Monday 06/01/2026 at 9am.''' We expect to be finished with all cluster node upgrades no later than Friday 08/21/2026 at 5pm.

===[[SLURM/JobSubmission | Submission Nodes]]===
'''Submission nodes with the number '01' in their hostnames will be taken offline for upgrade on 06/01/2026 at 9am.''' We expect that they will be back online no later than 5pm on that same day.

Submission nodes with the number '00' in their hostnames will be scheduled for upgrade individually, when all of the compute nodes associated with the same lab/center have been upgraded. Staff will send a notification to individual lab's/center's cluster users to schedule the relevant '00' node's upgrade when applicable. The actual date of each upgrade will be no less than one week after the corresponding notification has been sent.

Data in [[FilesystemDataStorage#UNIX_Filesystem_Storage | UNIX filesystem storage]] spaces on each submission node, i.e., /tmp and /scratch0, will not be preserved during upgrade. If you have any data in any such space on either submission node in a pairing that you want to keep, please ensure you copy it to the other submission node or a [[FilesystemDataStorage#Network-Attached_Filesystem_Storage | network-attached filesystem storage]] space prior to each node's upgrade date. Data in network-attached filesystem storage spaces, such as /nfshomes or /fs/nexus-scratch, will not be affected.

===Compute Nodes===
Due to the large number of compute nodes and the desire to not interrupt running jobs, we are not generally able to schedule each specific compute node upgrade on a specific date. If you find that a specific node is unavailable to schedule jobs on, you can run the command <code>sinfo --list-reasons --long</code> on a submission node and look to see if the node is in the list with the text "RHEL9 upgrade" - if this is present, the upgrade for that node is underway.

We will generally be prioritizing upgrades for nodes based on how available they are across various partitions; nodes that are only available in partitions that contain large numbers of users for a lab/center, e.g., cbcb, clip, cml-dpart, gamma, vulcan-ampere, vulcan-dpart, etc., and corresponding "scavenger" named partitions, will be prioritized over nodes that are only or are also available in faculty-specific / limited-node partitions. All nodes in the tron partition will also generally be prioritized.

If you are a faculty member authoritative for your own partition or a small group's limited-node partition and have scheduling concerns for the nodes in these partitions, please [[HelpDesk | contact staff]] ASAP to let us know about these concerns and we will make our best effort to accommodate them.

==Interoperability==
===Software and Modules===
Please begin transitioning your [[PythonVirtualEnv | virtual environments]], workflows, etc. to work with RHEL9 as soon as possible. You can use the '01' submission node that you have access to after it is back online sometime after 9am on 06/01/2026 for transitioning and light testing - as always, [[Nexus/Submission_Node_Policy | please do not run any computationally intensive processes on this node]]. It is intended to be a host for configuring environments/workflows and submitting jobs only.

The [[Modules | module tree]] for RHEL9 has already been populated with a large number of the same modules that are available in the RHEL8 module tree, although specific modules may have different versions available in the RHEL9 tree as compared to the RHEL8 tree. If you have a dependency on a specific version of a module that is not available in the RHEL9 tree, please [[HelpDesk | contact staff]] and we can get one created.
* If you want to check to see if a specific version is available now but do not have access to any RHEL9 node yet, the current module tree for RHEL9 is located at /fs/UMos/RedHat-9/x86_64/local/stow/.modulefiles and can be viewed from any UMIACS-supported host.

===SLURM Scheduling===
If you want or need to schedule a job on only nodes running RHEL8 (or RHEL9, once you have validated whatever is relevant), you can use the submission arguments <code>--prefer=rhel#</code> or <code>--constraint=rhel#</code> in your job arguments to specify this, where # is replaced by the OS version number. The --prefer argument is a soft limitation on which nodes the job can be scheduled on and the --constraint argument is a hard limitation, i.e., if you use the argument <code>--prefer=rhel8</code> but there are no RHEL8 nodes available at present (with your other submission arguments also satisfied) in the partition you are submitting to, the job will be scheduled on an appropriate RHEL9 node if that would result in an earlier (or instantaneous) start time.

Nexus/Vulcan

2026-04-27T13:42:27Z

Mbaney: /* Accounts */

The compute nodes from Vulcan's previous standalone cluster have folded into [[Nexus]] as of the scheduled [[MonthlyMaintenanceWindow | maintenance window]] for August 2023 (Thursday 08/17/2023, 5-8pm).

The Nexus cluster already has a large pool of compute resources made possible through college-level funding for UMIACS and CSD faculty. Details on common nodes already in the cluster (Tron partition) can be found [[Nexus/Tron | here]].

Please [[HelpDesk | contact staff]] with any questions or concerns.

==Usage==
You can [[SSH]] to <code>nexusvulcan.umiacs.umd.edu</code> to log in to a submission node.

If you store something in a local directory (/tmp, /scratch0) on one of the two submission nodes, you will need to connect to that same submission node to access it later. The actual submission nodes are:
* <code>nexusvulcan00.umiacs.umd.edu</code>
* <code>nexusvulcan01.umiacs.umd.edu</code>

All partitions, QoSes, and account names from the standalone Vulcan cluster have been moved over to Nexus. However, please note that <code>vulcan-</code> is prepended to all of the values that were present in the standalone Vulcan cluster to distinguish them from existing values in Nexus. The lone exception is the base account that was named <code>vulcan</code> in the standalone cluster (it is also named just <code>vulcan</code> in Nexus).

Here are some before/after examples of job submission with various parameters:

{| class="wikitable"
! Standalone Vulcan cluster submission command
! Nexus cluster submission command
|-
|<code>srun --partition=dpart --qos=medium --account=abhinav --gres=gpu:gtx1080ti:2 --pty bash</code>
|<code>srun --partition=vulcan-dpart --qos=vulcan-medium --account=vulcan-abhinav --gres=gpu:gtx1080ti:2 --pty bash</code>
|-
|<code>srun --partition=cpu --qos=cpu --pty bash</code>
|<code>srun --partition=vulcan-cpu --qos=vulcan-cpu --account=vulcan --pty bash</code>
|-
|<code>srun --partition=scavenger --qos=scavenger --account=vulcan --gres=gpu:4 --pty bash</code>
|<code>srun --partition=vulcan-scavenger --qos=vulcan-scavenger --account=vulcan --gres=gpu:4 --pty bash</code>
|}

Vulcan users (exclusively) can schedule non-interruptible jobs on Vulcan nodes with any non-scavenger job parameters. Please note that the <code>vulcan-dpart</code> partition has a <code>GrpTRES</code> limit of 100% of the available cores/RAM on all vulcan## in aggregate nodes plus 50% of the available cores/RAM on legacy## nodes in aggregate, so your job may need to wait if all available cores/RAM (or GPUs) are in use. It also has a max submission limit of 500 jobs per user simultaneously so as to not overload the cluster. This is codified by the partition QoS named '''vulcan'''.

Please note that the Vulcan compute nodes are also in the institute-wide <code>scavenger</code> partition in Nexus. Vulcan users still have scavenging priority over these nodes via the <code>vulcan-scavenger</code> partition (i.e., all <code>vulcan-</code> partition jobs (other than <code>vulcan-scavenger</code>) can preempt both <code>vulcan-scavenger</code> and <code>scavenger</code> partition jobs, and <code>vulcan-scavenger</code> partition jobs can preempt <code>scavenger</code> partition jobs).

==Compute Nodes==
There are currently 46 [[Nexus/Vulcan/GPUs | GPU nodes]] available, named vulcan[00-45], running a mixture of NVIDIA RTX A6000, NVIDIA RTX A5000, NVIDIA RTX A4000, and a number of different older generation cards. There are also 4 CPU-only nodes available, named brigid[16-19].

All nodes are scheduled with the [[SLURM]] resource manager.

==Network==
The network infrastructure supporting the Vulcan partition consists of:
# One pair of network switches connected to each other via dual 100GbE links for redundancy, serving the following compute nodes:
#* brigid[16-17],vulcan[29-45]: Two 100GbE links per node, one to each switch in the pair (redundancy).
#* vulcan46: Four 100GbE links, two to each switch in the pair (redundancy and increased bandwidth).
# One pair of network switches connected to the above pair of switches through several sets of intermediary switches, and to each other via dual 10GbE links for redundancy. The immediate connection to these sets of intermediary switches is via two 40GbE links to a pair of them, one between the first two switches in each pair and one between the second two switches in each pair for redundancy. This pair serves the following compute nodes:
#* brigid[18-19],vulcan[00-28]: Two 10GbE links per node, one to each switch in the pair (redundancy).

The fileserver hosting all Vulcan [[Nexus/Vulcan#Scratch_Directories | scratch]], [[Nexus/Vulcan#Project_Storage | project]], and [[Nexus/Vulcan#Datasets | dataset]] allocations first connects to a pair of intermediary switches and then the first pair of switches mentioned [[Nexus/Tron#Network | here (Tron page's network section)]]. It then connects to the first pair of switches mentioned on this page through a set of four (different) intermediary switches. The last hop from the four intermediary switches to the first pair of switches mentioned on this page is via 32 100GbE links, four from each switch in the set to each switch in the first pair mentioned on this page for redundancy and increased bandwidth.

For a broader overview of the network infrastructure supporting the Nexus cluster, please see [[Nexus/Network]].

==Partitions==
There are three partitions available to general Vulcan [[SLURM]] users. You must specify a partition when submitting your job.

* '''vulcan-dpart''' - This is the default partition. Job allocations are guaranteed. Only nodes with GPUs from architectures older than NVIDIA's [https://www.nvidia.com/en-us/data-center/ampere-architecture/ Ampere architecture] are included in this partition.
* '''vulcan-scavenger''' - This is the alternate partition that allows jobs longer run times and more resources but is preemptable when jobs in other non-scavenger-named <code>vulcan-</code> partitions are ready to be scheduled.
* '''vulcan-cpu''' - This partition is for CPU focused jobs. Job allocations are guaranteed.

There are a few additional partitions available to subsets of Vulcan users based on specific requirements.

* '''vulcan-ampere''' - This partition contains nodes with GPUs from NVIDIA's [https://www.nvidia.com/en-us/data-center/ampere-architecture/ Ampere architecture] or a newer architecture. Job allocations are guaranteed. Please be aware of the following restrictions on this partition:
*: ''Time limit'': there is a 4 hour time limit on interactive jobs in this partition. If you need to run longer jobs, you will need to modify your workflow into a job that can be submitted as a batch script.
*: ''CPU/memory per GPU limit'': there is a limit of 4 CPUs and 48G memory maximum per GPU requested by a job. If you need to run jobs with more CPUs/memory, you will either need to request more GPUs in the job or use a different partition.

: Submission is restricted to the Slurm [[#Accounts | accounts]] of the faculty who invested in these nodes:
:* Abhinav Shrivastava (vulcan-abhinav)
:* Jia-Bin Huang (vulcan-jbhuang)
:* Christopher Metzler (vulcan-metzler)
:* Ruoshi Liu (vulcan-ruoshi)
:* Matthias Zwicker (vulcan-zwicker)

* '''vulcan-scavenger-multi''' - This partition allows multi-node jobs (up to 9 total nodes per job) and allows jobs more resources than the vulcan-scavenger partition, but only contains nodes with GTX 1080 Ti, TITAN Xp, and/or RTX 2080 Ti GPUs in them. As with vulcan-scavenger, it is preemptable when jobs in other non-scavenged-named <code>vulcan-</code> partitions are ready to be scheduled.
*: Access to this partition is on a per-use basis. Please contact Abhinav Shrivastava if you would like to be granted access to this partition.

There is one additional partition available solely to Dr. Ramani Duraiswami's sponsored accounts.

* '''vulcan-ramani''' - This partition is for exclusive priority access to Dr. Duraiswami's purchased GPU nodes. Job allocations are guaranteed.

==Accounts==
Vulcan has a base SLURM account <code>vulcan</code> which has a modest number of guaranteed billing resources available to all cluster users at any given time. Other faculty that have invested in Vulcan compute infrastructure have an additional account provided to their sponsored accounts on the cluster.

If you do not specify an account when submitting your job, you will receive the '''vulcan''' account. If your faculty sponsor has their own account, it is recommended to use that account for job submission.

The current faculty accounts are:
* vulcan-abhinav
* vulcan-djacobs
* vulcan-jbhuang
* vulcan-metzler
* vulcan-rama
* vulcan-ramani
* vulcan-ruoshi
* vulcan-yaser
* vulcan-zwicker

<pre>
$ sacctmgr show account format=account%20,description%30,organization%10
Account Descr Org
-------------------- ------------------------------ ----------
... ... ...
vulcan vulcan vulcan
vulcan-abhinav vulcan - abhinav shrivastava vulcan
vulcan-djacobs vulcan - david jacobs vulcan
vulcan-jbhuang vulcan - jia-bin huang vulcan
vulcan-metzler vulcan - chris metzler vulcan
vulcan-rama vulcan - rama chellappa vulcan
vulcan-ramani vulcan - ramani duraiswami vulcan
vulcan-ruoshi vulcan - ruoshi liu vulcan
vulcan-yaser vulcan - yaser yacoob vulcan
vulcan-zwicker vulcan - matthias zwicker vulcan
... ... ...
</pre>

Faculty can manage this list of users via our [https://intranet.umiacs.umd.edu/directory/secgroup/ Directory application] in the Security Groups section. The security group that controls access has the prefix <code>vulcan_</code> and then the faculty username. It will also list <code>slurm://nexusctl.umiacs.umd.edu</code> as the associated URI.

You can check your account associations by running the '''show_assoc''' command to see the accounts you are associated with. Please [[HelpDesk | contact staff]] and include your faculty member in the conversation if you do not see the appropriate association.

<pre>
$ show_assoc
User Account MaxJobs GrpTRES QOS
---------- ---------------- ------- ------------- --------------------------------------------------------------------------------
... ... ... ...
abhinav vulcan 48 vulcan-cpu,vulcan-default,vulcan-medium,vulcan-scavenger
abhinav vulcan-abhinav 48 vulcan-cpu,vulcan-default,vulcan-high,vulcan-medium,vulcan-scavenger
... ... ... ...
</pre>

You can also see the total number of Track-able Resources (TRES) allowed for each account by running the following command. Please make sure you give the appropriate account that you are looking for. As shown below, there is a concurrent limit of 64 total GPUs for all users not in a contributing faculty group.

<pre>
$ sacctmgr show assoc account=vulcan format=user,account,qos,grptres
User Account QOS GrpTRES
---------- ---------- -------------------- -------------
vulcan gres/gpu=64
... ...
</pre>

==QoS==
Vulcan currently has 3 QoS for the '''vulcan-dpart''' partition, 1 QoS for the '''vulcan-scavenger''' partition, and 1 QoS for the '''vulcan-cpu''' partition. If you do not specify a QoS when submitting your job using the <code>--qos</code> parameter, you will receive the <code>vulcan-default</code> QoS assuming you are using a Vulcan account.

The important part here is that in different QoS you can have a shorter/longer maximum wall time, a different total number of jobs running at once, and a different maximum number of track-able resources (TRES) for the job. In the cml-scavenger QoS, one more constraint that you are restricted by is the total number of TRES per user (over multiple jobs).

<pre>
$ show_qos --all | grep vulcan
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
------------------------- ----------- ------------------------------ --------- ------------------------------
...
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-exempt 7-00:00:00 cpu=32,gres/gpu=8,mem=256G 2
vulcan-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
vulcan-high_long 14-00:00:00 cpu=32,gres/gpu=8 8
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-sailon 3-00:00:00 cpu=32,gres/gpu=8,mem=256G gres/gpu=48
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
vulcan-scavenger-multi 3-00:00:00 cpu=288,gres/gpu=72,mem=1152G
...
</pre>

<pre>
$ show_partition_qos --all | grep vulcan
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- ------------------------------ --------------------
...
vulcan 500 cpu=1760,mem=15824G
vulcan-ampere 500
vulcan-cpu 500
vulcan-ramani 500
vulcan-scavenger 500
vulcan-scavenger-multi 500
...
</pre>

==Storage==
Vulcan has the following storage available. Please also review UMIACS [[FilesystemDataStorage | Filesystem Data Storage]] policies including any volume that is labeled as scratch.

Vulcan users can also request [[Nexus#Project_Allocations | Nexus project allocations]].

===Home Directories===
{{Nfshomes}}

===Scratch Directories===
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Vulcan compute infrastructure:
* Network scratch directory
* Local scratch directories

====Network Scratch Directory====
You have 300GB of scratch storage available at <code>/vulcanscratch/<username></code>. '''It is not backed up or protected in any way.''' This directory is '''automounted''' so you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access this.

You may request a temporary increase of up to 500GB total space for a maximum of 120 days without any faculty approval by [[HelpDesk | contacting staff]]. Once the temporary increase period is over, you will be contacted and given a one-week window of opportunity to clean and secure your data before staff will forcibly remove data to get your space back under 300GB. If you need space beyond 500GB or for longer than 120 days, you will need faculty approval and/or a project directory.

This file system is available on all submission and computational nodes within the cluster.

====Local Scratch Directories====
Each computational node that you can schedule compute jobs on has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. These are almost always more performant than any other storage available to the job. However, you must stage their data within the confine of their job and stage the data out before the end of their job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month at 1am. Different nodes will run the maintenance jobs on different days of the month to ensure the cluster is still highly available at all times. Please make sure you secure any data you write to these directories at the end of your job.

===Datasets===
We have read-only dataset storage available at <code>/fs/vulcan-datasets</code>. If there are datasets that you would like to see curated and made available, please see [[Datasets | this page]].

The list of Vulcan datasets we currently host can be viewed [https://info.umiacs.umd.edu/datasets/list/?q=Vulcan here].

===Project Storage===
Users within the Vulcan compute infrastructure can request project based allocations for up to 10TB for up to 180 days by [[HelpDesk | contacting staff]] with approval from the Vulcan faculty manager (Dr. Shrivastava). These allocations will be available from <code>/fs/vulcan-projects</code> under a name that you provide when you request the allocation. Near the end of the allocation period, staff will contact you and ask if you would like to renew the allocation for up to another 180 days (requires re-approval from Dr. Shrivastava).
* If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation.
* If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible.
** If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible.
** If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

Project storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

===Object Storage===
All Vulcan users can request project allocations in the [https://obj.umiacs.umd.edu/obj/help UMIACS Object Store]. Please [[HelpDesk | contact staff]] with a short project name and the amount of storage you will need to get started.

To access this storage, you'll need to use a [[S3Clients | S3 client]] or our [[UMobj]] command line utilities.

An example on how to use the umobj command line utilities can be found [[UMobj/Example | here]]. A full set of documentation for the utilities can be found on the [https://gitlab.umiacs.umd.edu/staff/umobj/blob/master/README.md#umobj umobj Gitlab page].

Nexus/Vulcan

2026-04-27T13:09:24Z

Mbaney: /* Partitions */

The compute nodes from Vulcan's previous standalone cluster have folded into [[Nexus]] as of the scheduled [[MonthlyMaintenanceWindow | maintenance window]] for August 2023 (Thursday 08/17/2023, 5-8pm).

The Nexus cluster already has a large pool of compute resources made possible through college-level funding for UMIACS and CSD faculty. Details on common nodes already in the cluster (Tron partition) can be found [[Nexus/Tron | here]].

Please [[HelpDesk | contact staff]] with any questions or concerns.

==Usage==
You can [[SSH]] to <code>nexusvulcan.umiacs.umd.edu</code> to log in to a submission node.

If you store something in a local directory (/tmp, /scratch0) on one of the two submission nodes, you will need to connect to that same submission node to access it later. The actual submission nodes are:
* <code>nexusvulcan00.umiacs.umd.edu</code>
* <code>nexusvulcan01.umiacs.umd.edu</code>

All partitions, QoSes, and account names from the standalone Vulcan cluster have been moved over to Nexus. However, please note that <code>vulcan-</code> is prepended to all of the values that were present in the standalone Vulcan cluster to distinguish them from existing values in Nexus. The lone exception is the base account that was named <code>vulcan</code> in the standalone cluster (it is also named just <code>vulcan</code> in Nexus).

Here are some before/after examples of job submission with various parameters:

{| class="wikitable"
! Standalone Vulcan cluster submission command
! Nexus cluster submission command
|-
|<code>srun --partition=dpart --qos=medium --account=abhinav --gres=gpu:gtx1080ti:2 --pty bash</code>
|<code>srun --partition=vulcan-dpart --qos=vulcan-medium --account=vulcan-abhinav --gres=gpu:gtx1080ti:2 --pty bash</code>
|-
|<code>srun --partition=cpu --qos=cpu --pty bash</code>
|<code>srun --partition=vulcan-cpu --qos=vulcan-cpu --account=vulcan --pty bash</code>
|-
|<code>srun --partition=scavenger --qos=scavenger --account=vulcan --gres=gpu:4 --pty bash</code>
|<code>srun --partition=vulcan-scavenger --qos=vulcan-scavenger --account=vulcan --gres=gpu:4 --pty bash</code>
|}

Vulcan users (exclusively) can schedule non-interruptible jobs on Vulcan nodes with any non-scavenger job parameters. Please note that the <code>vulcan-dpart</code> partition has a <code>GrpTRES</code> limit of 100% of the available cores/RAM on all vulcan## in aggregate nodes plus 50% of the available cores/RAM on legacy## nodes in aggregate, so your job may need to wait if all available cores/RAM (or GPUs) are in use. It also has a max submission limit of 500 jobs per user simultaneously so as to not overload the cluster. This is codified by the partition QoS named '''vulcan'''.

Please note that the Vulcan compute nodes are also in the institute-wide <code>scavenger</code> partition in Nexus. Vulcan users still have scavenging priority over these nodes via the <code>vulcan-scavenger</code> partition (i.e., all <code>vulcan-</code> partition jobs (other than <code>vulcan-scavenger</code>) can preempt both <code>vulcan-scavenger</code> and <code>scavenger</code> partition jobs, and <code>vulcan-scavenger</code> partition jobs can preempt <code>scavenger</code> partition jobs).

==Compute Nodes==
There are currently 46 [[Nexus/Vulcan/GPUs | GPU nodes]] available, named vulcan[00-45], running a mixture of NVIDIA RTX A6000, NVIDIA RTX A5000, NVIDIA RTX A4000, and a number of different older generation cards. There are also 4 CPU-only nodes available, named brigid[16-19].

All nodes are scheduled with the [[SLURM]] resource manager.

==Network==
The network infrastructure supporting the Vulcan partition consists of:
# One pair of network switches connected to each other via dual 100GbE links for redundancy, serving the following compute nodes:
#* brigid[16-17],vulcan[29-45]: Two 100GbE links per node, one to each switch in the pair (redundancy).
#* vulcan46: Four 100GbE links, two to each switch in the pair (redundancy and increased bandwidth).
# One pair of network switches connected to the above pair of switches through several sets of intermediary switches, and to each other via dual 10GbE links for redundancy. The immediate connection to these sets of intermediary switches is via two 40GbE links to a pair of them, one between the first two switches in each pair and one between the second two switches in each pair for redundancy. This pair serves the following compute nodes:
#* brigid[18-19],vulcan[00-28]: Two 10GbE links per node, one to each switch in the pair (redundancy).

The fileserver hosting all Vulcan [[Nexus/Vulcan#Scratch_Directories | scratch]], [[Nexus/Vulcan#Project_Storage | project]], and [[Nexus/Vulcan#Datasets | dataset]] allocations first connects to a pair of intermediary switches and then the first pair of switches mentioned [[Nexus/Tron#Network | here (Tron page's network section)]]. It then connects to the first pair of switches mentioned on this page through a set of four (different) intermediary switches. The last hop from the four intermediary switches to the first pair of switches mentioned on this page is via 32 100GbE links, four from each switch in the set to each switch in the first pair mentioned on this page for redundancy and increased bandwidth.

For a broader overview of the network infrastructure supporting the Nexus cluster, please see [[Nexus/Network]].

==Partitions==
There are three partitions available to general Vulcan [[SLURM]] users. You must specify a partition when submitting your job.

* '''vulcan-dpart''' - This is the default partition. Job allocations are guaranteed. Only nodes with GPUs from architectures older than NVIDIA's [https://www.nvidia.com/en-us/data-center/ampere-architecture/ Ampere architecture] are included in this partition.
* '''vulcan-scavenger''' - This is the alternate partition that allows jobs longer run times and more resources but is preemptable when jobs in other non-scavenger-named <code>vulcan-</code> partitions are ready to be scheduled.
* '''vulcan-cpu''' - This partition is for CPU focused jobs. Job allocations are guaranteed.

There are a few additional partitions available to subsets of Vulcan users based on specific requirements.

* '''vulcan-ampere''' - This partition contains nodes with GPUs from NVIDIA's [https://www.nvidia.com/en-us/data-center/ampere-architecture/ Ampere architecture] or a newer architecture. Job allocations are guaranteed. Please be aware of the following restrictions on this partition:
*: ''Time limit'': there is a 4 hour time limit on interactive jobs in this partition. If you need to run longer jobs, you will need to modify your workflow into a job that can be submitted as a batch script.
*: ''CPU/memory per GPU limit'': there is a limit of 4 CPUs and 48G memory maximum per GPU requested by a job. If you need to run jobs with more CPUs/memory, you will either need to request more GPUs in the job or use a different partition.

: Submission is restricted to the Slurm [[#Accounts | accounts]] of the faculty who invested in these nodes:
:* Abhinav Shrivastava (vulcan-abhinav)
:* Jia-Bin Huang (vulcan-jbhuang)
:* Christopher Metzler (vulcan-metzler)
:* Ruoshi Liu (vulcan-ruoshi)
:* Matthias Zwicker (vulcan-zwicker)

* '''vulcan-scavenger-multi''' - This partition allows multi-node jobs (up to 9 total nodes per job) and allows jobs more resources than the vulcan-scavenger partition, but only contains nodes with GTX 1080 Ti, TITAN Xp, and/or RTX 2080 Ti GPUs in them. As with vulcan-scavenger, it is preemptable when jobs in other non-scavenged-named <code>vulcan-</code> partitions are ready to be scheduled.
*: Access to this partition is on a per-use basis. Please contact Abhinav Shrivastava if you would like to be granted access to this partition.

There is one additional partition available solely to Dr. Ramani Duraiswami's sponsored accounts.

* '''vulcan-ramani''' - This partition is for exclusive priority access to Dr. Duraiswami's purchased GPU nodes. Job allocations are guaranteed.

==Accounts==
Vulcan has a base SLURM account <code>vulcan</code> which has a modest number of guaranteed billing resources available to all cluster users at any given time. Other faculty that have invested in Vulcan compute infrastructure have an additional account provided to their sponsored accounts on the cluster.

If you do not specify an account when submitting your job, you will receive the '''vulcan''' account. If your faculty sponsor has their own account, it is recommended to use that account for job submission.

The current faculty accounts are:
* vulcan-abhinav
* vulcan-djacobs
* vulcan-jbhuang
* vulcan-metzler
* vulcan-rama
* vulcan-ramani
* vulcan-yaser
* vulcan-zwicker

<pre>
$ sacctmgr show account format=account%20,description%30,organization%10
Account Descr Org
-------------------- ------------------------------ ----------
... ... ...
vulcan vulcan vulcan
vulcan-abhinav vulcan - abhinav shrivastava vulcan
vulcan-djacobs vulcan - david jacobs vulcan
vulcan-jbhuang vulcan - jia-bin huang vulcan
vulcan-metzler vulcan - chris metzler vulcan
vulcan-rama vulcan - rama chellappa vulcan
vulcan-ramani vulcan - ramani duraiswami vulcan
vulcan-yaser vulcan - yaser yacoob vulcan
vulcan-zwicker vulcan - matthias zwicker vulcan
... ... ...
</pre>

Faculty can manage this list of users via our [https://intranet.umiacs.umd.edu/directory/secgroup/ Directory application] in the Security Groups section. The security group that controls access has the prefix <code>vulcan_</code> and then the faculty username. It will also list <code>slurm://nexusctl.umiacs.umd.edu</code> as the associated URI.

You can check your account associations by running the '''show_assoc''' command to see the accounts you are associated with. Please [[HelpDesk | contact staff]] and include your faculty member in the conversation if you do not see the appropriate association.

<pre>
$ show_assoc
User Account MaxJobs GrpTRES QOS
---------- ---------------- ------- ------------- --------------------------------------------------------------------------------
... ... ... ...
abhinav vulcan 48 vulcan-cpu,vulcan-default,vulcan-medium,vulcan-scavenger
abhinav vulcan-abhinav 48 vulcan-cpu,vulcan-default,vulcan-high,vulcan-medium,vulcan-scavenger
... ... ... ...
</pre>

You can also see the total number of Track-able Resources (TRES) allowed for each account by running the following command. Please make sure you give the appropriate account that you are looking for. As shown below, there is a concurrent limit of 64 total GPUs for all users not in a contributing faculty group.

<pre>
$ sacctmgr show assoc account=vulcan format=user,account,qos,grptres
User Account QOS GrpTRES
---------- ---------- -------------------- -------------
vulcan gres/gpu=64
... ...
</pre>

==QoS==
Vulcan currently has 3 QoS for the '''vulcan-dpart''' partition, 1 QoS for the '''vulcan-scavenger''' partition, and 1 QoS for the '''vulcan-cpu''' partition. If you do not specify a QoS when submitting your job using the <code>--qos</code> parameter, you will receive the <code>vulcan-default</code> QoS assuming you are using a Vulcan account.

The important part here is that in different QoS you can have a shorter/longer maximum wall time, a different total number of jobs running at once, and a different maximum number of track-able resources (TRES) for the job. In the cml-scavenger QoS, one more constraint that you are restricted by is the total number of TRES per user (over multiple jobs).

<pre>
$ show_qos --all | grep vulcan
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
------------------------- ----------- ------------------------------ --------- ------------------------------
...
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-exempt 7-00:00:00 cpu=32,gres/gpu=8,mem=256G 2
vulcan-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
vulcan-high_long 14-00:00:00 cpu=32,gres/gpu=8 8
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-sailon 3-00:00:00 cpu=32,gres/gpu=8,mem=256G gres/gpu=48
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
vulcan-scavenger-multi 3-00:00:00 cpu=288,gres/gpu=72,mem=1152G
...
</pre>

<pre>
$ show_partition_qos --all | grep vulcan
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- ------------------------------ --------------------
...
vulcan 500 cpu=1760,mem=15824G
vulcan-ampere 500
vulcan-cpu 500
vulcan-ramani 500
vulcan-scavenger 500
vulcan-scavenger-multi 500
...
</pre>

==Storage==
Vulcan has the following storage available. Please also review UMIACS [[FilesystemDataStorage | Filesystem Data Storage]] policies including any volume that is labeled as scratch.

Vulcan users can also request [[Nexus#Project_Allocations | Nexus project allocations]].

===Home Directories===
{{Nfshomes}}

===Scratch Directories===
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Vulcan compute infrastructure:
* Network scratch directory
* Local scratch directories

====Network Scratch Directory====
You have 300GB of scratch storage available at <code>/vulcanscratch/<username></code>. '''It is not backed up or protected in any way.''' This directory is '''automounted''' so you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access this.

You may request a temporary increase of up to 500GB total space for a maximum of 120 days without any faculty approval by [[HelpDesk | contacting staff]]. Once the temporary increase period is over, you will be contacted and given a one-week window of opportunity to clean and secure your data before staff will forcibly remove data to get your space back under 300GB. If you need space beyond 500GB or for longer than 120 days, you will need faculty approval and/or a project directory.

This file system is available on all submission and computational nodes within the cluster.

====Local Scratch Directories====
Each computational node that you can schedule compute jobs on has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. These are almost always more performant than any other storage available to the job. However, you must stage their data within the confine of their job and stage the data out before the end of their job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month at 1am. Different nodes will run the maintenance jobs on different days of the month to ensure the cluster is still highly available at all times. Please make sure you secure any data you write to these directories at the end of your job.

===Datasets===
We have read-only dataset storage available at <code>/fs/vulcan-datasets</code>. If there are datasets that you would like to see curated and made available, please see [[Datasets | this page]].

The list of Vulcan datasets we currently host can be viewed [https://info.umiacs.umd.edu/datasets/list/?q=Vulcan here].

===Project Storage===
Users within the Vulcan compute infrastructure can request project based allocations for up to 10TB for up to 180 days by [[HelpDesk | contacting staff]] with approval from the Vulcan faculty manager (Dr. Shrivastava). These allocations will be available from <code>/fs/vulcan-projects</code> under a name that you provide when you request the allocation. Near the end of the allocation period, staff will contact you and ask if you would like to renew the allocation for up to another 180 days (requires re-approval from Dr. Shrivastava).
* If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation.
* If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible.
** If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible.
** If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

Project storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

===Object Storage===
All Vulcan users can request project allocations in the [https://obj.umiacs.umd.edu/obj/help UMIACS Object Store]. Please [[HelpDesk | contact staff]] with a short project name and the amount of storage you will need to get started.

To access this storage, you'll need to use a [[S3Clients | S3 client]] or our [[UMobj]] command line utilities.

An example on how to use the umobj command line utilities can be found [[UMobj/Example | here]]. A full set of documentation for the utilities can be found on the [https://gitlab.umiacs.umd.edu/staff/umobj/blob/master/README.md#umobj umobj Gitlab page].

AddingUMIACSCertificateAuthority

2026-04-24T19:52:50Z

Mbaney:

===Introduction===

When you visit webpages encrypted with SSL, you may be presented with a warning about the site's security certificate not being trusted. These are normally important screens to pay attention to on the internet as a whole and UMIACS goes to great lengths to maintain a secure environment for our users. There are rare times where we may use our own internal certificate authority (CA) instead of a third-party one with wider trust to issue a site's certificate, to keep costs down. If you want to add this CA's root certificate ahead of time, follow these steps to import it based on your preferred web browser.

===Windows===

For most Windows browsers (Chrome, Firefox, Edge and Internet Explorer), certificate authorities are handled by Windows itself. These are the steps required to accept the certificate:

* Click [https://wiki.umiacs.umd.edu/umiacs/images/7/7d/CA-cert.crt UMIACS Certificate Authority] to download the file.
* Open the file and click "Install Certificate".
*; [[File:UMIACSCA1.png|alt=Screenshot of Windows menu highlighting the "Install Certificate" button]]]
* In the dialog box opened, click "Next".
* Choose "Place all certificates in the following store".
* Choose "Browse", in the dialog box opened, Choose "Trusted Root Certification Authorities" and click "Ok".
*; [[File:UMIACSCA2.png|alt=Screenshot of Windows highlighting indicated fields in instructions]]]
* Click "Next" and then "Finish".
* If you get a Security Warning asking if you want to install this certificate, click "Yes".
*; [[File:UMIACSCA3.png|alt=Screenshot of Windows highlighting indicated fields in instructions]]]
* You should receive a success message similar to the following:
*; [[File:UMIACSCA4.png|alt=Screenshot of expected success message in Windows]]
* You may need to restart your browser for the change to take effect.

===Safari and Google Chrome (macOS)===

For most macOS browsers (excluding Firefox), certificate authorities are handled by macOS itself. This process requires administrator access. If you do not have administrator access and you are using a UMIACS-supported Mac, please contact [[HelpDesk|Staff]]. Otherwise, here are the steps required to accept the certificate:

* Click [https://wiki.umiacs.umd.edu/umiacs/images/7/7d/CA-cert.crt UMIACS Certificate Authority] to download the file.
* Open Keychain Access (Located in the Others group in Launchpad)
* Go to the Systems > Certificates
* Open the UMIACS Certificate Authority file by double-clicking it (should be located in your downloads folder).
* Enter your administrator password or use your fingerprint on the dialog box that appears.
[[Image:KeychainAcces1.png|500px|alt=Screenshot of MacOS menu displaying the dialog box requiring authentication]]
* Right-click the certificate that was just added and select the "Get Info" section.
[[Image:KeychainAccess2.png|500px|alt=Screenshot of MacOS showing the results of right clicking the certificate]]
* Select "Always Trust" option in the "When using this certificate" dropdown ("Trust" > "When using this certificate" > "Always Trust")
* Close the certificate and keychain access window.
* Enter the administrator credentials to add this certificate for all users of the system
[[Image:KeychainAccess3.png|500px|alt=Screenshot of MacOS menu to enter administrator credentials]]
* You may need to restart your browser for the change to take effect.

===Other Browsers (Unix)===

If you are using a browser other than Firefox in Unix, the process is more complicated than the above methods and may depend on your particular Unix distribution. If you need assistance with this please contact [[HelpDesk|UMIACS Staff]].

MonthlyMaintenanceWindow

2026-04-24T19:49:08Z

Mbaney:

[[HelpDesk | UMIACS staff]] takes a monthly maintenance window to patch and reboot all UMIACS-supported hosts and services. This provides a way for staff to ensure security updates are installed and applied on the numerous different platforms and appliances that UMIACS runs.

The window for each month is calculated by adding 9 days to [https://en.wikipedia.org/wiki/Patch_Tuesday Microsoft's Patch Tuesday] to allow for enough time to marshal patches released that month from Microsoft, Red Hat, Apple, and other OS and application vendors and have enough time to get systems prepared to reboot. This translates to the window being on the '''Thursday that occurs between the 17th and the 23rd (inclusive)''' of each month. The window lasts from '''5pm-8pm'''.

[[Nexus]] will always have a reservation in place from 4:45pm-8pm on the day of the upcoming window to prevent jobs from being scheduled on compute nodes. The 15-minute addition before the start of the window is to allow jobs to fully end. Any job submitted before the reservation begins that has a time limit that would run into the reservation will be held until at least the end of the reservation - 8pm on the day of the window. This is to prevent issues with jobs failing to end properly causing delays in work we have scheduled during the window.

A list of upcoming maintenance windows is as follows, with the next one in bold. Again, the window is on the '''Thursday that occurs between the 17th and the 23rd (inclusive)''' of each month, and lasts from '''5pm-8pm'''.

* '''May 21st 2026'''
* June 18th 2026
* July 23rd 2026
* August 20th 2026
* September 17th 2026
* October 22nd 2026
* November 19th 2026
* December 17th 2026

==Archives==
* January 17th 2013 - BEGIN time of 8pm-12am for this window through February 20th 2020
* February 21st 2013
* March 21st 2013
* April 18th 2013
* May 23rd 2013
* June 20th 2013
* July 18th 2013
* August 22nd 2013
* September 19th 2013
* October 17th 2013
* December 19th 2013
* January 23rd 2014
* February 20th 2014
* March 20th 2014
* April 17th 2014
* May 22nd 2014
* June 19th 2014
* July 17th 2014
* August 21st 2014
* September 18th 2014
* October 23rd 2014
* November 20th 2014
* December 18th 2014
* January 22nd 2015
* February 19th 2015
* March 19th 2015
* May 21st 2015
* June 18th 2015
* July 23rd 2015
* August 20th 2015
* September 17th 2015
* October 22nd 2015
* November 19th 2015
* December 17th 2015
* January 21st 2016
* February 18th 2016
* March 12th 2016 (Adjusted date for AVW power outage)
* April 21st 2016
* May 19th 2016
* June 23rd 2016
* July 21st 2016
* August 18th 2016
* September 22nd 2016
* October 20th 2016
* November 17th 2016
* December 22nd 2016
* January 19th 2017
* February 23rd 2017
* March 23rd 2017
* April 20th 2017
* May 18th 2017
* June 22nd 2017
* July 20th 2017
* August 17th 2017
* September 21st 2017
* October 19th 2017
* December 21st 2017
* January 18th 2018
* February 22nd 2018
* March 22nd 2018
* April 19th 2018
* May 17th 2018
* June 21st 2018
* July 19th 2018
* August 23rd 2018
* September 20th 2018
* October 18th 2018
* December 20th 2018
* January 24th 2019
* February 21st 2019
* April 18th 2019
* May 23rd 2019
* June 20th 2019
* July 18th 2019
* August 22nd 2019
* September 19th 2019
* October 17th 2019
* November 21st 2019
* December 19th 2019
* January 23rd 2020
* February 20th 2020
* April 23rd 2020 - BEGIN time of 5pm-7pm for this window through August 19th 2021
* June 18th 2020
* July 23rd 2020
* August 20th 2020
* September 17th 2020
* October 22nd 2020
* November 19th 2020
* December 17th 2020
* January 21st 2021
* February 18th 2021
* March 25th 2021 (Adjusted date for extended Spring Break)
* April 22nd 2021
* May 20th 2021
* June 17th 2021
* July 22nd 2021
* August 19th 2021
* September 23rd 2021 - BEGIN time of 5pm-8pm for this window and all others below
* October 21st 2021
* November 18th 2021
* January 20th 2022
* February 17th 2022
* March 24th 2022 (Adjusted date for Spring Break)
* April 21st 2022
* May 19th 2022
* June 23rd 2022
* July 21st 2022
* August 18th 2022
* September 22nd 2022
* October 20th 2022
* November 17th 2022
* January 19th 2023
* February 23rd 2023
* April 20th 2023
* May 18th 2023
* June 22nd 2023
* July 20th 2023
* August 17th 2023
* September 21st 2023
* October 19th 2023
* December 20th 2023 (Adjusted date for early Winter Break)
* January 18th 2024
* February 22nd 2024
* March 21st 2024
* April 18th 2024
* May 23th 2024
* June 20th 2024
* July 18th 2024
* August 22nd 2024
* September 19th 2024
* October 17th 2024
* November 21st 2024
* December 19th 2024
* January 23rd 2025
* February 20th 2025
* March 20th 2025
* April 17th 2025
* May 22nd 2025
* June 19th 2025
* July 17th 2025
* August 21st 2025
* September 18th 2025
* October 23rd 2025
* November 20th 2025
* December 18th 2025
* January 22nd 2026
* February 19th 2026
* March 19th 2026
* April 23rd 2026

BarracudaSpamFirewall

2026-04-23T18:08:45Z

Mbaney:

===Introduction===
UMIACS has deployed a system with two Barracuda Networks spam firewalls. This allows for enterprise level Virus and Spam scoring and filtering for our email architecture. You can log in to the system either of the two firewalls using your UMIACS email address (username@umiacs.umd.edu) and UMD passphrase:
*[https://bubs.umiacs.umd.edu bubs.umiacs.umd.edu]
*[https://pompom.umiacs.umd.edu pompom.umiacs.umd.edu]

===Mail Flow Through Barracudas===
*The first time your mail flows through one of the Barracudas it will send you a mail with a new username and password. Subsequently you will receive every day (unless you configure otherwise) a mail at approx. 3:30pm EST from the Barracuda with your quarantine summary. '''Please note that auto log-in through the link provided in the summary does not work anymore due to security hardening on the Barracudas. Please use the links provided above to log in.'''

===Scoring===
The Barracudas will score every message that passes through them and inject varying message headers based on that score. You can then create email filters based on these headers to filter out messages that are tagged as spam by the Barracudas. See [[BarracudaSpamFirewall/Scoring]] for more details.

===Quarantine===
*Mail that has been deemed as spam will be kept on the Barracudas in quarantine. It will not be delivered to your mailbox unless you configure the Barracudas to do so.
*Your quarantine will be preserved for '''21 days'''. Individual mail messages are purged if they are still in the quarantine 22 days after they are first received.
*You can search your spam quarantine by following the steps [[BarracudaSpamFirewall/SearchingQuarantine | here]].

===Quarantine Passthrough===
*If you wish to have the mail that would ordinarily be quarantined by Barracuda delivered to your mailbox instead you can configure this using the Barracuda web configuration.
*You can enable this functionality by following the steps [[BarracudaSpamFirewall/QuarantinePassthrough | here]].

===Whitelists, Blacklists & Bayesian Filtering===
*You may also setup whitelists, blacklists, and Bayesian filtering options through the Preferences tab at the top of the Barracuda web portal.

===More Information===
*For more information on how to use the Barracuda please download the user's guide:

https://wiki.umiacs.umd.edu/umiacs/images/5/5a/Barracuda_usersguide.pdf

Nexus/Submission Node Policy

2026-04-22T16:30:36Z

Mbaney:

Submission nodes are intended for users to submit jobs to the Nexus cluster. User interactivity is critical here, and while some of these submission nodes have a decent amount of cores and memory, highly-intensive computational jobs should not be run on these nodes. We have minimal resource restrictions in place at the moment, but we may explore further CPU/memory restrictions in the future. For the time being, be a good neighbor!

In general, we expect that users on these nodes do not use more than 1-2GB of memory at a time and do not sustain >=100% CPU usage (at least 1 core fully utilized) for extended periods of time. Utilizing more than 1 core is fine for bursty workloads, but sustained high CPU utilization can lead to issues with interactivity, and these issues can be drastically amplified if there is also memory contention on the node, as swapping memory requires CPU time.

=Examples=

==Appropriate submission node uses==

* Job submission
* Using as a SSH jump host for other hosts
* Basic code editing in editors with minimal extensions (e.g. emacs, nano, vim, neovim without an LSP).

==Appropriate submission node uses, with caveats==

These kinds of workloads can be run on submission nodes, but if they are incorrectly configured, they can affect interactivity of the nodes. If we notice repeated behavior like this, we may reach out to you to ask about your environment and suggest ways to lower resource utilization.

* IDEs can be fine, but keep in mind which extensions you install. Some extensions/behaviors may use more memory/CPU than you realize.
** We have some [[VS_Code|guidance for VS Code and its forks]].
* For smaller projects, code compilation can be fine. If a project takes longer than 30 seconds to compile, please ensure compilation is limited to a single thread, or consider compiling the code in a Slurm job on one of the dedicated CPU nodes (we suggest using the scavenger partition).
** Depending on a project's file structure, code compilation can be highly I/O dependent, which can also impact user interactivity.
* Setting up environments with user package managers (e.g. pip, npm, conda/mamba)
** The resource usage of package managers mostly depends on the specific packages being installed. Complex environments can take a while with package managers like conda/mamba due to dependency resolution. Other packages aren't actually binary releases but will instead compile projects transparently to the user. For these reasons, we suggest that non-trivial environments should be set up in a Slurm job (we suggest using the scavenger partition).
* Running lightweight programs to interact with Slurm jobs currently running on the cluster.

==Inappropriate submission node uses==

If tech staff notices processes with the following behavior and node interactivity is affected, we may kill these processes. If a user repeatedly causes issues, further action may be taken on their account.

* Compiling large projects using all threads on a submission node.
* Running nontrivial computation requiring significant CPU and/or memory resources.

=Advice for monitoring/reducing CPU/memory utilization=

Here is some general guidance for tracking your resource utilization on a node, as well as some practices you can follow to slightly reduce CPU usage of certain tasks.

* `top`
** By default, `top` sorts by %CPU, but you can sort by %MEM by pressing "M" (Shift + "m").
** You can limit it to show only processes from your user by starting it with `-u $USER`
** VIRT memory usage doesn't mean too much, as oftentimes programs will request way more memory than they actually need, and the kernel may not actually give them this memory until it is actually required. RSS memory is the actual amount of memory used by a process at a given point. If there is no suffix, the reported value is in KB.
* /sys/fs/cgroup/memory/user.slice/user-$UID_NUMBER.slice/memory.usage_in_bytes
** This is the current combined memory usage of all processes running under your user.

Nexus/GPUs

2026-04-17T19:21:30Z

Mbaney:

There are several different types of [https://www.nvidia.com/en-us/ NVIDIA] GPUs in the [[Nexus]] cluster that are available to be scheduled. They are listed below in order of newest to oldest architecture, and then alphanumerically by name.

The exact quantities of GPUs per type are not listed here since these numbers may change frequently due to additions to or removals from the cluster or during compute node troubleshooting. To see which compute nodes have which GPUs and in what quantities, use the <code>show_nodes</code> command on a submission or compute node. The quantities are listed under the <tt>GRES</tt> column.

{| class="wikitable sortable"
! Name
! GRES string ([[SLURM]])
! [https://www.nvidia.com/en-us/technologies Architecture]
! [https://developer.nvidia.com/cuda-toolkit CUDA] Cores
! Memory Amount and Type
! Memory Bandwidth
! FP32 Performance (TFLOPS)
! [https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/ TF32] Performance ([https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt Dense / Sparse TOPS])
|-
| L40S
| <code>l40s</code>
| Ada Lovelace
| 18176
| 48GB GDDR6
| 864GB/s
| 91.6
| 183/366
|-
| RTX 6000 Ada Generation
| <code>rtx6000ada</code>
| Ada Lovelace
| 18176
| 48GB GDDR6
| 960GB/s
| 91.1
| 182.1/364.2
|-
| H100 NVLink [0]
| <code>h100-nvl</code>
| Hopper
| 33792
| 188GB HBM3
| 7.87TB/s
| 133.8
| not officially published/1671
|-
| H100 SXM
| <code>h100-sxm</code>
| Hopper
| 16896
| 80GB HBM3
| 3.35TB/s
| 66.9
| not officially published/989
|-
| H200 SXM
| <code>h200-sxm</code>
| Hopper
| 16896
| 141GB HBM3e
| 4.89TB/s
| 66.9
| not officially published/989
|-
| A100 PCIe 80GB
| <code>a100</code>
| Ampere
| 6912
| 80GB HBM2e
| 1.94TB/s
| 19.5
| 156/312
|-
| A100 SXM 80GB
| <code>a100</code>
| Ampere
| 6912
| 80GB HBM2e
| 2.04TB/s
| 19.5
| 156/312
|-
| GeForce RTX 3070
| <code>rtx3070</code>
| Ampere
| 5888
| 8GB GDDR6
| 448GB/s
| 20.3
| 20.3/40.6
|-
| GeForce RTX 3090
| <code>rtx3090</code>
| Ampere
| 10496
| 24GB GDDR6X
| 936GB/s
| 35.6
| 35.6/71
|-
| RTX A4000
| <code>rtxa4000</code>
| Ampere
| 6144
| 16GB GDDR6
| 448GB/s
| 19.2
| not officially published
|-
| RTX A5000
| <code>rtxa5000</code>
| Ampere
| 8192
| 24GB GDDR6
| 768GB/s
| 27.8
| not officially published
|-
| RTX A6000
| <code>rtxa6000</code>
| Ampere
| 10752
| 48GB GDDR6
| 768GB/s
| 38.7
| 77.4/154.8
|-
| GeForce RTX 2080 Ti
| <code>rtx2080ti</code>
| Turing
| 4352
| 11GB GDDR5X
| 616GB/s
| 13.4
| n/a
|-
| GeForce GTX 1080 Ti
| <code>gtx1080ti</code>
| Pascal
| 3584
| 11GB GDDR5X
| 484GB/s
| 11.3
| n/a
|-
| Quadro P6000
| <code>p6000</code>
| Pascal
| 3840
| 24GB GDDR5X
| 432GB/s
| 12.6
| n/a
|-
| Tesla P100
| <code>p100</code>
| Pascal
| 3584
| 16GB CoWoS HBM2
| 732GB/s
| 9.3
| n/a
|-
| TITAN X (Pascal)
| <code>titanxpascal</code>
| Pascal
| 3584
| 12GB GDDR5X
| 480GB/s
| 11.0
| n/a
|-
| TITAN Xp
| <code>titanxp</code>
| Pascal
| 3840
| 12GB GDDR5X
| 548GB/s
| 12.1
| n/a
|-
| GeForce GTX TITAN X
| <code>gtxtitanx</code>
| Maxwell
| 3072
| 12GB GDDR5
| 336GB/s
| 6.7
| n/a
|-
|}

[0] - This GPU type is actually a pair of two physical cards connected over [https://www.nvidia.com/en-us/data-center/nvlink NVLink] bridges. NVIDIA's provided specifications for this GPU type are for one physical card; to get these specs, we have hence doubled NVIDIA's provided values.

Windows Patch Management

2026-04-16T13:45:53Z

Mbaney:

UMIACS uses Windows' built-in Windows Update mechanism to patch the Windows operating system, Windows drivers, and other Microsoft products that Microsoft uses Windows Update to push updates for, in tandem with a software distribution tool called [https://umd-dit.atlassian.net/wiki/spaces/DMS/pages/45285463/Patch+My+PC Patch My PC] that operates through the Division of IT's managed [https://umd-dit.atlassian.net/wiki/spaces/DMS/pages/45285467/Intune Intune] service to patch third party applications.

Our previous patch management solution, Ivanti Endpoint Manager, may still have agent software installed on UMIACS-supported [[Windows]] desktops, however it operates only in "read-only" mode and does not perform actual patching anymore. It will be removed in the future.

==Windows Update==
* '''Desktops''' will run updates available through Windows Update daily between 3am and 4am US Eastern.
* '''Laptops''' will run updates available through Windows Update at any time they are on and connected to the internet.

The only updates available through Windows Update that should require computer restarts are Windows operating system monthly rollups, released on [https://en.wikipedia.org/wiki/Patch_Tuesday Microsoft's Patch Tuesday], and new [[WindowsServicing | feature updates]], when they are pushed by us annually.

After a month's monthly rollup is installed on your computer, you will receive a notification stating that your machine needs to be restarted in the next 7 days. You can choose either to restart immediately or to schedule the restart. If you do not restart by the deadline, your computer will automatically restart no more than a day after the deadline is exceeded. During the month that a feature update is pushed, your computer may need to reboot twice.

==Patch My PC==
Patches are deployed as they come out for the Patch My PC catalog. They should install automatically and require little to no input.

Windows Patch Management

2026-04-16T13:45:08Z

Mbaney:

As of Fall 2025, UMIACS uses Windows' built-in Windows Update mechanism to patch the Windows operating system, Windows drivers, and other Microsoft products that Microsoft uses Windows Update to push updates for, in tandem with a software distribution tool called [https://umd-dit.atlassian.net/wiki/spaces/DMS/pages/45285463/Patch+My+PC Patch My PC] that operates through the Division of IT's managed [https://umd-dit.atlassian.net/wiki/spaces/DMS/pages/45285467/Intune Intune] service to patch third party applications.

Our previous patch management solution, Ivanti Endpoint Manager, may still have agent software installed on UMIACS-supported [[Windows]] desktops, however it operates only in "read-only" mode and does not perform actual patching anymore. It will be removed in the future.

==Windows Update==
* '''Desktops''' will run updates available through Windows Update daily between 3am and 4am US Eastern.
* '''Laptops''' will run updates available through Windows Update at any time they are on and connected to the internet.

The only updates available through Windows Update that should require computer restarts are Windows operating system monthly rollups, released on [https://en.wikipedia.org/wiki/Patch_Tuesday Microsoft's Patch Tuesday], and new [[WindowsServicing | feature updates]], when they are pushed by us annually. After a month's monthly rollup is installed on your computer, you will receive a notification stating that your machine needs to be restarted in the next 7 days. You can choose either to restart immediately or to schedule the restart. If you do not restart by the deadline, your computer will automatically restart no more than a day after the deadline is exceeded. During the month that a feature update is released, your computer may need to reboot twice.

==Patch My PC==
Patches are deployed as they come out for the Patch My PC catalog. They should install automatically and require little to no input.

WindowsServicing

2026-04-16T13:43:26Z

Mbaney:

[[Windows]] periodically releases "feature updates" that are designed to bring a large set of new features to all machines that can run Windows. They are typically released annually by Microsoft.

At UMIACS, we release these feature updates to be installed on all UMIACS-supported Windows desktops as well as Enterprise supported laptops and home machines via our [[Windows Patch Management]]. We typically release feature updates to be installed several months after Microsoft releases them to the general public to ensure that the targeted release is compatible with all software that we run.

==Current Feature Updates at UMIACS==
The current feature updates we deploy at UMIACS are:

{| class="wikitable"
!OS Version
!Feature Update
|-
|Windows 11
|25H2
|}

We deployed the current Windows 11 feature update on April 15th, 2026. For a comprehensive list of what is new in this feature update, please see Microsoft's documentation:
* 11: https://learn.microsoft.com/en-us/windows/whats-new/whats-new-windows-11-version-25h2

'''Windows 10 reached end of support on October 14th, 2025.''' If you still have a university-owned computer running Windows 10, please upgrade it to Windows 11 ASAP. Please [[HelpDesk | contact staff]] if help is needed.

==Updating==
Our [[Windows Patch Management]] automatically installs feature updates when we release them to UMIACS-managed devices. You will receive a notification to restart to install the feature update in the same way that other patches are installed. Please see that page for more details.

WindowsServicing

2026-04-16T13:19:33Z

Mbaney:

[[Windows]] periodically releases "feature updates" that are designed to bring a large set of new features to all machines that can run Windows. They are typically released annually by Microsoft.

At UMIACS, we release these feature updates to be installed on all UMIACS-supported Windows desktops as well as Enterprise supported laptops and home machines via our [[Windows Patch Management]]. We typically release feature updates to be installed several months after Microsoft releases them to the general public to ensure that the targeted release is compatible with all software that we run.

==Current Feature Updates at UMIACS==
The current feature updates we deploy at UMIACS are:

{| class="wikitable"
!OS Version
!Feature Update
|-
|Windows 11
|25H2
|}

We deployed the current Windows 11 feature update on April 15th, 2026.

'''Windows 10 reached end of support on October 14th, 2025.''' If you still have a university-owned computer running Windows 10, please upgrade it to Windows 11 ASAP. Please [[HelpDesk | contact staff]] if help is needed.

For a comprehensive list of what is new in this feature update, please see Microsoft's documentation:
* 11: https://learn.microsoft.com/en-us/windows/whats-new/whats-new-windows-11-version-25h2

==Updating==
Our [[Windows Patch Management]] automatically installs feature updates when we release them to UMIACS-managed devices. You will receive a notification to restart to install the feature update in the same way that other patches are installed. Please see that page for more details.

SLURM/Priority

2026-04-15T14:28:59Z

Mbaney: /* Fair-share */

[[SLURM]] at UMIACS is configured to prioritize jobs based on a number of factors, termed [https://slurm.schedmd.com/priority_multifactor.html multifactor priority] in SLURM. Each job submitted to the scheduler is assigned a priority value, which can be viewed in the output of <code>scontrol show job <jobid></code>.

Example:
<pre>
$ scontrol show job 1
JobId=1 JobName=bash
UserId=username(13337) GroupId=username(13337) MCS_label=N/A
Priority=2000841 Nice=0 Account=nexus QOS=default
...
</pre>

==Pending Jobs==
If the partition that you submit your job to cannot begin your job instantly due to no compute node(s) in the partition having the resources free to run it, your job will remain in the Pending state with the listed reason <tt>(Resources)</tt>. If there is another job already pending with this reason in a partition, you submit a job to the same partition, and your job gets assigned a lower priority value than that pending job, your job will instead remain in the Pending state with reason <tt>(Priority)</tt>. If there are multiple jobs pending and your job is not the highest priority job pending, the scheduler will only begin execution of your job if starting your job would not push the begin times for any higher priority jobs in the same partition further back.

Lowering some combination of the resources you are requesting and/or the time limit may allow submitted jobs to start sooner (or instantly) during times where a partition is under resource pressure. The command <code>squeue -j <jobid> --start</code> can be used to provide a time estimate for when your job will start, where <jobid> is the job ID you receive from either srun or sbatch. This time is subject to change depending on if other users' jobs end sooner or more jobs get submitted.

You can use the command alias <code>[[SLURM/JobSubmission#show_available_nodes | show_available_nodes]]</code> with a variety of different submission arguments to get a better idea of what jobs may be able to begin sooner, but the output of this command alias is not definitive, for reasons mentioned in the footnotes on the page linked to.

==Priority Factors==
The priority factors in use at UMIACS are, from most-heavily to least-heavily weighted:
* Partition job was submitted to
* Fair-share of resources within SLURM account
* Age of job, i.e., time spent waiting to run in the queue
* Association/SLURM account being used
* "Nice" value that job was submitted with

===Partition===
The partitions whose names are or are prefixed with <code>scavenger</code> on our clusters are always in a lower priority tier and always have lower priority factors for their jobs than all other partitions on that cluster. As mentioned in other UMIACS cluster-specific documentation, jobs submitted to these partitions are also [https://slurm.schedmd.com/preempt.html preemptable]. These two design choices give the partitions their names; jobs submitted to <code>scavenger</code> named or prefixed partitions "scavenge" for available resources on the cluster rather than consume dedicated resources, and are interrupted by jobs asking to consume dedicated resources.

On [[Nexus]], labs/centers may also have their own scavenger partitions, i.e., <code><labname>-scavenger</code>, if the faculty for the lab/center have decided upon some sort of limit on jobs, such as number of simultaneous jobs, number of actively consumed billing resources, etc., in their non-scavenger partitions. These lab/center scavenger partitions allow for more jobs to be run by members of that lab/center on that lab's/center's nodes only, but jobs on these partitions are preemptable by jobs in that lab's/center's non-scavenger partitions and/or account-specific partitions, if any account-specific partitions containing a given node exist. Jobs submitted to lab/center scavenger partitions will preempt jobs submitted to the institute-wide scavenger partitions (running on nodes that are also in those lab/center scavenger partitions).

In decreasing order of priority (highest first), our priority tiers for partitions are:
# Priority access account-specific partitions
# Account-specific partitions
# Lab/center-specific and institute-wide non-"scavenger" named partitions
# Lab/center-specific "scavenger" named partitions
# Institute-wide "scavenger" named partitions

A job in a specific priority tier will never have a higher priority value than any job in a higher priority tier. Corresponding to the above tiers, the priority values that you will see for jobs in each tier:
# >= 4000000
# 3000000 to 3999999
# 2000000 to 2999999
# 1000000 to 1999999
# < 1000000

As such, '''jobs on specific nodes in some non-"scavenger" named partitions may also be subject to preemption''' based on these priority tiers. Generally speaking, though, most nodes are only in one partition in one of the first three (non-"scavenger") priority tiers, and then in an institute-wide "scavenger" named partition, and a lab/center-specific "scavenger" named partition, if one exists for the lab/center that a given node is a part of.

===Fair-share===
The more resources your jobs have already consumed within an account, the lower priority factor your future jobs will have when compared to other users' jobs in the same account who have used fewer resources (so as to "fair-share" with other users). Additionally, if there are multiple accounts that can submit to a partition, and the sum of resources of all users' jobs within account A is greater than the sum of resources of all users' jobs within account B, the lower priority factor all future jobs from users in account A will have when compared to all future jobs from users in account B. (In other words, fair-share is hierarchical.)

You can view the various fair-share statistics with the command <code>sshare -l</code>. It will show your specific FairShare values (always between 0.0 and 1.0) within accounts that you have access to. You can also view other accounts' Level Fairshare (LevelFS).
<pre>
Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS GrpTRESMins TRESRunMins
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------
root 0.000000 68444174744 1.000000 cpu=4797787,mem=70530109515,e+
cbcb 1 0.028571 4454658377 0.065046 0.065046 0.439246 cpu=452139,mem=22276633804,en+
class 1 0.028571 255617290 0.003733 0.003733 7.652841 cpu=7021,mem=74554606,energy=+
clip 1 0.028571 3057933838 0.044674 0.044674 0.639549 cpu=33214,mem=2744443460,ener+
cml 1 0.028571 66866114 0.000975 0.000975 29.299389 cpu=1796,mem=29426756,energy=+
gamma 1 0.028571 2609474948 0.038129 0.038129 0.749334 cpu=34089,mem=360373862,energ+
mbrc 1 0.028571 73411964 0.001073 0.001073 26.635560 cpu=1195,mem=4896358,energy=0+
mc2 1 0.028571 2682557 0.000039 0.000039 728.919551 cpu=0,mem=0,energy=0,node=0,b+
nexus 1 0.028571 5472794067 0.079964 0.079964 0.357302 cpu=278464,mem=3250599000,ene+
nexus username 1 0.000835 69666 0.000001 0.000021 0.457407 37.435501 cpu=0,mem=0,energy=0,node=0,b+
oasis 1 0.028571 330030 0.000005 0.000005 5.9248e+03 cpu=0,mem=0,energy=0,node=0,b+
quics 1 0.028571 4 0.000000 0.000000 4.1683e+08 cpu=0,mem=0,energy=0,node=0,b+
scavenger 1 0.028571 40888195964 0.597419 0.597419 0.047825 cpu=3142204,mem=29902903931,e+
scavenger username 1 0.000835 171 0.000000 0.000000 0.033975 9.8885e+04 cpu=0,mem=0,energy=0,node=0,b+
vulcan 1 0.028571 1247236491 0.018224 0.018224 1.567761 cpu=147273,mem=1161243818,ene+
</pre>

The actual resource billing weights for the three main resources (memory per GB, CPU cores, and number of GPUs if applicable) are per-partition and can be viewed in the <code>TRESBillingWeights</code> line in the output of <code>scontrol show partition</code>. The <code>billing</code> value for a job is the sum of all resource weightings for resources the job has requested. This value is then multiplied by the amount of time a job has run in seconds to get the amount it contributes to the RawUsage for the association within the account it is running under.

====Algorithm====
The algorithm we use for resource weightings differs depending on if there are any GPUs in a partition or not, and is as follows:

=====GPU partitions=====
Each resource (memory/CPU/GPU) is given a weighting value such that their relative billings to each other within the partition are equal (33.33% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU/GPU values are adjusted accordingly.

Different GPU types may also be weighted differently within the GPU relative billing. A baseline GPU type is first chosen. All GPUs of that type and other types that have lower FP32 performance (in [https://en.wikipedia.org/wiki/FLOPS TFLOPS]) are given a weighting factor of 1.0. GPU types with higher FP32 performance than the baseline GPU are given a weighting factor calculated by dividing their FP32 performance by the baseline GPU's FP32 performance. The weighting values for each GPU type are then determined by normalizing the sum of all of GPU cards' billing values multiplied by their weighting factors against the relative billing percentage for GPUs (33.33%).

The current baseline GPU is the [https://www.nvidia.com/en-us/design-visualization/rtx-a4000/ NVIDIA RTX A4000].

=====CPU-only partitions=====
Each resource (memory/CPU) is first given a weighting value such that their relative billings to each other within the partition are equal (50% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU value is adjusted accordingly. The final CPU weight value is then divided by 10, which translates to roughly 90.9% of the billing weight being for memory and 9.1% being for CPU. The division of the CPU value is done so as to not affect accounts' fair-share priority factors as much when running jobs in CPU-only partitions given the popularity of GPGPU computing.

===Age===
The longer a job is eligible to run but cannot due to resources being unavailable or having a lower priority value than one or more other jobs, the higher the job's priority becomes as it continue to wait in the queue. This is the only priority modifier that can change a job's priority value once it has been submitted, and the priority modifier for this factor reaches its limit after 7 days.

Jobs' age priority factors on our clusters are recalculated every 5 minutes.

===Association===
Some lab/center-specific SLURM accounts have priority values directly attached to them. Jobs run under these accounts gain this many extra points of priority.

===Nice value===
This is a submission argument that you as the user can include when submitting your jobs to deprioritize them. Larger values will deprioritize jobs more, e.g.,
<pre>srun --pty --nice=2 bash</pre>
will have lower priority than
<pre>srun --pty --nice=1 bash</pre>
which will have lower priority than
<pre>srun --pty bash</pre>
assuming all three jobs were submitted at the same time. You cannot use negative values for this argument.

Because this value is absolute, if you want to use it, we would recommend using small numbers - one or two digits - only. Larger numbers may impact your job's ability to run at all as a result of the other factors at play.

WindowsServicing

2026-04-15T14:10:28Z

Mbaney:

[[Windows]] periodically releases "feature updates" that are designed to bring a large set of new features to all machines that can run Windows. They are typically released annually by Microsoft.

At UMIACS, we release these feature updates to be installed on all UMIACS-supported Windows desktops as well as Enterprise supported laptops and home machines via our [[Windows Patch Management]]. We typically release feature updates to be installed several months after Microsoft releases them to the general public to ensure that the targeted release is compatible with all software that we run.

==Current Feature Release at UMIACS==
The current feature releases we deploy at UMIACS are:

{| class="wikitable"
!OS Version
!Feature Release
|-
|Windows 11
|25H2
|}

We deployed the current Windows 11 feature release on April 15th, 2026.

'''Windows 10 reached end of support on October 14th, 2025.''' If you still have a university-owned computer running Windows 10, please upgrade it to Windows 11 ASAP. Please [[HelpDesk | contact staff]] if help is needed.

For a comprehensive list of what is new in this feature update, please see Microsoft's documentation:
* 11: https://learn.microsoft.com/en-us/windows/whats-new/whats-new-windows-11-version-25h2

==Updating==
Our [[Windows Patch Management]] automatically installs feature updates when we release them to UMIACS-managed devices. You will receive a notification to restart to install the feature update in the same way that other patches are installed. Please see that page for more details.

SLURM/Priority

2026-04-15T13:10:43Z

Mbaney:

[[SLURM]] at UMIACS is configured to prioritize jobs based on a number of factors, termed [https://slurm.schedmd.com/priority_multifactor.html multifactor priority] in SLURM. Each job submitted to the scheduler is assigned a priority value, which can be viewed in the output of <code>scontrol show job <jobid></code>.

Example:
<pre>
$ scontrol show job 1
JobId=1 JobName=bash
UserId=username(13337) GroupId=username(13337) MCS_label=N/A
Priority=2000841 Nice=0 Account=nexus QOS=default
...
</pre>

==Pending Jobs==
If the partition that you submit your job to cannot begin your job instantly due to no compute node(s) in the partition having the resources free to run it, your job will remain in the Pending state with the listed reason <tt>(Resources)</tt>. If there is another job already pending with this reason in a partition, you submit a job to the same partition, and your job gets assigned a lower priority value than that pending job, your job will instead remain in the Pending state with reason <tt>(Priority)</tt>. If there are multiple jobs pending and your job is not the highest priority job pending, the scheduler will only begin execution of your job if starting your job would not push the begin times for any higher priority jobs in the same partition further back.

Lowering some combination of the resources you are requesting and/or the time limit may allow submitted jobs to start sooner (or instantly) during times where a partition is under resource pressure. The command <code>squeue -j <jobid> --start</code> can be used to provide a time estimate for when your job will start, where <jobid> is the job ID you receive from either srun or sbatch. This time is subject to change depending on if other users' jobs end sooner or more jobs get submitted.

You can use the command alias <code>[[SLURM/JobSubmission#show_available_nodes | show_available_nodes]]</code> with a variety of different submission arguments to get a better idea of what jobs may be able to begin sooner, but the output of this command alias is not definitive, for reasons mentioned in the footnotes on the page linked to.

==Priority Factors==
The priority factors in use at UMIACS are, from most-heavily to least-heavily weighted:
* Partition job was submitted to
* Fair-share of resources within SLURM account
* Age of job, i.e., time spent waiting to run in the queue
* Association/SLURM account being used
* "Nice" value that job was submitted with

===Partition===
The partitions whose names are or are prefixed with <code>scavenger</code> on our clusters are always in a lower priority tier and always have lower priority factors for their jobs than all other partitions on that cluster. As mentioned in other UMIACS cluster-specific documentation, jobs submitted to these partitions are also [https://slurm.schedmd.com/preempt.html preemptable]. These two design choices give the partitions their names; jobs submitted to <code>scavenger</code> named or prefixed partitions "scavenge" for available resources on the cluster rather than consume dedicated resources, and are interrupted by jobs asking to consume dedicated resources.

On [[Nexus]], labs/centers may also have their own scavenger partitions, i.e., <code><labname>-scavenger</code>, if the faculty for the lab/center have decided upon some sort of limit on jobs, such as number of simultaneous jobs, number of actively consumed billing resources, etc., in their non-scavenger partitions. These lab/center scavenger partitions allow for more jobs to be run by members of that lab/center on that lab's/center's nodes only, but jobs on these partitions are preemptable by jobs in that lab's/center's non-scavenger partitions and/or account-specific partitions, if any account-specific partitions containing a given node exist. Jobs submitted to lab/center scavenger partitions will preempt jobs submitted to the institute-wide scavenger partitions (running on nodes that are also in those lab/center scavenger partitions).

In decreasing order of priority (highest first), our priority tiers for partitions are:
# Priority access account-specific partitions
# Account-specific partitions
# Lab/center-specific and institute-wide non-"scavenger" named partitions
# Lab/center-specific "scavenger" named partitions
# Institute-wide "scavenger" named partitions

A job in a specific priority tier will never have a higher priority value than any job in a higher priority tier. Corresponding to the above tiers, the priority values that you will see for jobs in each tier:
# >= 4000000
# 3000000 to 3999999
# 2000000 to 2999999
# 1000000 to 1999999
# < 1000000

As such, '''jobs on specific nodes in some non-"scavenger" named partitions may also be subject to preemption''' based on these priority tiers. Generally speaking, though, most nodes are only in one partition in one of the first three (non-"scavenger") priority tiers, and then in an institute-wide "scavenger" named partition, and a lab/center-specific "scavenger" named partition, if one exists for the lab/center that a given node is a part of.

===Fair-share===
The more resources your jobs have already consumed within an account, the lower priority factor your future jobs will have when compared to other users' jobs in the same account who have used fewer resources (so as to "fair-share" with other users). Additionally, if there are multiple accounts that can submit to a partition, and the sum of resources of all users' jobs within account A is greater than the sum of resources of all users' jobs within account B, the lower priority factor all future jobs from users in account A will have when compared to all future jobs from users in account B. (In other words, fair-share is hierarchical.)

You can view the various fair-share statistics with the command <code>sshare -l</code>. It will show your specific FairShare values (always between 0.0 and 1.0) within accounts that you have access to. You can also view other accounts' Level Fairshare (LevelFS).
<pre>
Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS GrpTRESMins TRESRunMins
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------
root 0.000000 68444174744 1.000000 cpu=4797787,mem=70530109515,e+
cbcb-heng 1 0.028571 2041564795 0.029831 0.029831 0.957779 cpu=107606,mem=2890646050,ene+
cbcb 1 0.028571 4454658377 0.065046 0.065046 0.439246 cpu=452139,mem=22276633804,en+
class 1 0.028571 255617290 0.003733 0.003733 7.652841 cpu=7021,mem=74554606,energy=+
clip 1 0.028571 3057933838 0.044674 0.044674 0.639549 cpu=33214,mem=2744443460,ener+
cml-abhinav 1 0.028571 39726437 0.000580 0.000580 49.220844 cpu=0,mem=0,energy=0,node=0,b+
cml-director 1 0.028571 672692955 0.009829 0.009829 2.906778 cpu=65977,mem=1080976998,ener+
cml-furongh 1 0.028571 765366431 0.011183 0.011183 2.554814 cpu=78802,mem=1513011200,ener+
cml-hajiagha 1 0.028571 12397205 0.000181 0.000181 157.726572 cpu=0,mem=0,energy=0,node=0,b+
cml-ramani 1 0.028571 0 0.000000 0.000000 1.3575e+11 cpu=0,mem=0,energy=0,node=0,b+
cml-scavenger 1 0.028571 2564733604 0.037475 0.037475 0.762406 cpu=370867,mem=3568986794,ene+
cml-sfeizi 1 0.028571 40715178 0.000595 0.000595 48.025548 cpu=0,mem=0,energy=0,node=0,b+
cml-tokekar 1 0.028571 26458249 0.000387 0.000387 73.903937 cpu=23240,mem=178488320,energ+
cml-tomg 1 0.028571 82800 0.000001 0.000001 2.3615e+04 cpu=0,mem=0,energy=0,node=0,b+
cml-wriva-high 1 0.028571 0 0.000000 0.000000 inf cpu=0,mem=0,energy=0,node=0,b+
cml-wriva 1 0.028571 1047428702 0.015305 0.015305 1.866828 cpu=5366,mem=137386666,energy+
cml 1 0.028571 66866114 0.000975 0.000975 29.299389 cpu=1796,mem=29426756,energy=+
csd-sarahwie 1 0.028571 710055035 0.007034 0.007034 4.061956 cpu=0,mem=0,energy=0,node=0,b+
gamma 1 0.028571 2609474948 0.038129 0.038129 0.749334 cpu=34089,mem=360373862,energ+
mbrc 1 0.028571 73411964 0.001073 0.001073 26.635560 cpu=1195,mem=4896358,energy=0+
mc2 1 0.028571 2682557 0.000039 0.000039 728.919551 cpu=0,mem=0,energy=0,node=0,b+
nexus 1 0.028571 5472794067 0.079964 0.079964 0.357302 cpu=278464,mem=3250599000,ene+
nexus username 1 0.000835 69666 0.000001 0.000021 0.457407 37.435501 cpu=0,mem=0,energy=0,node=0,b+
oasis 1 0.028571 330030 0.000005 0.000005 5.9248e+03 cpu=0,mem=0,energy=0,node=0,b+
quics 1 0.028571 4 0.000000 0.000000 4.1683e+08 cpu=0,mem=0,energy=0,node=0,b+
scavenger 1 0.028571 40888195964 0.597419 0.597419 0.047825 cpu=3142204,mem=29902903931,e+
scavenger username 1 0.000835 171 0.000000 0.000000 0.033975 9.8885e+04 cpu=0,mem=0,energy=0,node=0,b+
vulcan-abhinav 1 0.028571 1742123078 0.025452 0.025452 1.122544 cpu=26166,mem=781842841,energ+
vulcan-djacobs 1 0.028571 912637 0.000013 0.000013 2.1425e+03 cpu=0,mem=0,energy=0,node=0,b+
vulcan-jbhuang 1 0.028571 283430750 0.004141 0.004141 6.898930 cpu=172,mem=5661764,energy=0,+
vulcan-metzler 1 0.028571 262070179 0.003829 0.003829 7.461241 cpu=0,mem=0,energy=0,node=0,b+
vulcan-rama 1 0.028571 0 0.000000 0.000000 1.6942e+14 cpu=0,mem=0,energy=0,node=0,b+
vulcan-ramani 1 0.028571 681333557 0.009955 0.009955 2.869914 cpu=22188,mem=568033280,energ+
vulcan-yaser 1 0.028571 397309 0.000006 0.000006 4.9215e+03 cpu=0,mem=0,energy=0,node=0,b+
vulcan-zwicker 1 0.028571 133204932 0.001946 0.001946 14.679402 cpu=0,mem=0,energy=0,node=0,b+
vulcan 1 0.028571 1247236491 0.018224 0.018224 1.567761 cpu=147273,mem=1161243818,ene+
</pre>

The actual resource billing weights for the three main resources (memory per GB, CPU cores, and number of GPUs if applicable) are per-partition and can be viewed in the <code>TRESBillingWeights</code> line in the output of <code>scontrol show partition</code>. The <code>billing</code> value for a job is the sum of all resource weightings for resources the job has requested. This value is then multiplied by the amount of time a job has run in seconds to get the amount it contributes to the RawUsage for the association within the account it is running under.

====Algorithm====
The algorithm we use for resource weightings differs depending on if there are any GPUs in a partition or not, and is as follows:

=====GPU partitions=====
Each resource (memory/CPU/GPU) is given a weighting value such that their relative billings to each other within the partition are equal (33.33% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU/GPU values are adjusted accordingly.

Different GPU types may also be weighted differently within the GPU relative billing. A baseline GPU type is first chosen. All GPUs of that type and other types that have lower FP32 performance (in [https://en.wikipedia.org/wiki/FLOPS TFLOPS]) are given a weighting factor of 1.0. GPU types with higher FP32 performance than the baseline GPU are given a weighting factor calculated by dividing their FP32 performance by the baseline GPU's FP32 performance. The weighting values for each GPU type are then determined by normalizing the sum of all of GPU cards' billing values multiplied by their weighting factors against the relative billing percentage for GPUs (33.33%).

The current baseline GPU is the [https://www.nvidia.com/en-us/design-visualization/rtx-a4000/ NVIDIA RTX A4000].

=====CPU-only partitions=====
Each resource (memory/CPU) is first given a weighting value such that their relative billings to each other within the partition are equal (50% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU value is adjusted accordingly. The final CPU weight value is then divided by 10, which translates to roughly 90.9% of the billing weight being for memory and 9.1% being for CPU. The division of the CPU value is done so as to not affect accounts' fair-share priority factors as much when running jobs in CPU-only partitions given the popularity of GPGPU computing.

===Age===
The longer a job is eligible to run but cannot due to resources being unavailable or having a lower priority value than one or more other jobs, the higher the job's priority becomes as it continue to wait in the queue. This is the only priority modifier that can change a job's priority value once it has been submitted, and the priority modifier for this factor reaches its limit after 7 days.

Jobs' age priority factors on our clusters are recalculated every 5 minutes.

===Association===
Some lab/center-specific SLURM accounts have priority values directly attached to them. Jobs run under these accounts gain this many extra points of priority.

===Nice value===
This is a submission argument that you as the user can include when submitting your jobs to deprioritize them. Larger values will deprioritize jobs more, e.g.,
<pre>srun --pty --nice=2 bash</pre>
will have lower priority than
<pre>srun --pty --nice=1 bash</pre>
which will have lower priority than
<pre>srun --pty bash</pre>
assuming all three jobs were submitted at the same time. You cannot use negative values for this argument.

Because this value is absolute, if you want to use it, we would recommend using small numbers - one or two digits - only. Larger numbers may impact your job's ability to run at all as a result of the other factors at play.

SLURM/Priority

2026-04-14T20:57:33Z

Mbaney:

[[SLURM]] at UMIACS is configured to prioritize jobs based on a number of factors, termed [https://slurm.schedmd.com/priority_multifactor.html multifactor priority] in SLURM. Each job submitted to the scheduler is assigned a priority value, which can be viewed in the output of <code>scontrol show job <jobid></code>.

Example:
<pre>
$ scontrol show job 1
JobId=1 JobName=bash
UserId=username(13337) GroupId=username(13337) MCS_label=N/A
Priority=2000841 Nice=0 Account=nexus QOS=default
...
</pre>

==Pending Jobs==
If the partition that you submit your job to cannot begin your job instantly due to no compute node(s) in the partition having the resources free to run it, your job will remain in the Pending state with the listed reason <tt>(Resources)</tt>. If there is another job already pending with this reason in a partition, you submit a job to the same partition, and your job gets assigned a lower priority value than that pending job, your job will instead remain in the Pending state with reason <tt>(Priority)</tt>. If there are multiple jobs pending and your job is not the highest priority job pending, the scheduler will only begin execution of your job if starting your job would not push the begin times for any higher priority jobs in the same partition further back.

Lowering some combination of the resources you are requesting and/or the time limit may allow submitted jobs to run more quickly or instantly during times where a partition is under resource pressure. The command <code>squeue -j <jobid> --start</code> can be used to provide a time estimate for when your job will start, where <jobid> is the job ID you receive from either srun or sbatch. This time is subject to change depending on if other users' jobs end sooner or more jobs get submitted.

You can use the command alias <code>[[SLURM/JobSubmission#show_available_nodes | show_available_nodes]]</code> with a variety of different submission arguments to get a better idea of what jobs may be able to begin sooner, but the output of this command alias is not definitive, for reasons mentioned in the footnotes on the page linked to.

==Priority Factors==
The priority factors in use at UMIACS are, from most-heavily to least-heavily weighted:
* Partition job was submitted to
* Fair-share of resources within SLURM account
* Age of job, i.e., time spent waiting to run in the queue
* Association/SLURM account being used
* "Nice" value that job was submitted with

===Partition===
The partitions whose names are or are prefixed with <code>scavenger</code> on our clusters are always in a lower priority tier and always have lower priority factors for their jobs than all other partitions on that cluster. As mentioned in other UMIACS cluster-specific documentation, jobs submitted to these partitions are also [https://slurm.schedmd.com/preempt.html preemptable]. These two design choices give the partitions their names; jobs submitted to <code>scavenger</code> named or prefixed partitions "scavenge" for available resources on the cluster rather than consume dedicated resources, and are interrupted by jobs asking to consume dedicated resources.

On [[Nexus]], labs/centers may also have their own scavenger partitions, i.e., <code><labname>-scavenger</code>, if the faculty for the lab/center have decided upon some sort of limit on jobs, such as number of simultaneous jobs, number of actively consumed billing resources, etc., in their non-scavenger partitions. These lab/center scavenger partitions allow for more jobs to be run by members of that lab/center on that lab's/center's nodes only, but jobs on these partitions are preemptable by jobs in that lab's/center's non-scavenger partitions and/or account-specific partitions, if any account-specific partitions containing a given node exist. Jobs submitted to lab/center scavenger partitions will preempt jobs submitted to the institute-wide scavenger partitions (running on nodes that are also in those lab/center scavenger partitions).

In decreasing order of priority (highest first), our priority tiers for partitions are:
# Priority access account-specific partitions
# Account-specific partitions
# Lab/center-specific and institute-wide non-"scavenger" named partitions
# Lab/center-specific "scavenger" named partitions
# Institute-wide "scavenger" named partitions

A job in a specific priority tier will never have a higher priority value than any job in a higher priority tier. Corresponding to the above tiers, the priority values that you will see for jobs in each tier:
# >= 4000000
# 3000000 to 3999999
# 2000000 to 2999999
# 1000000 to 1999999
# < 1000000

As such, '''jobs on specific nodes in some non-"scavenger" named partitions may also be subject to preemption''' based on these priority tiers. Generally speaking, though, most nodes are only in one partition in one of the first three (non-"scavenger") priority tiers, and then in an institute-wide "scavenger" named partition, and a lab/center-specific "scavenger" named partition, if one exists for the lab/center that a given node is a part of.

===Fair-share===
The more resources your jobs have already consumed within an account, the lower priority factor your future jobs will have when compared to other users' jobs in the same account who have used fewer resources (so as to "fair-share" with other users). Additionally, if there are multiple accounts that can submit to a partition, and the sum of resources of all users' jobs within account A is greater than the sum of resources of all users' jobs within account B, the lower priority factor all future jobs from users in account A will have when compared to all future jobs from users in account B. (In other words, fair-share is hierarchical.)

You can view the various fair-share statistics with the command <code>sshare -l</code>. It will show your specific FairShare values (always between 0.0 and 1.0) within accounts that you have access to. You can also view other accounts' Level Fairshare (LevelFS).
<pre>
Account User RawShares NormShares RawUsage NormUsage EffectvUsage FairShare LevelFS GrpTRESMins TRESRunMins
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------
root 0.000000 68444174744 1.000000 cpu=4797787,mem=70530109515,e+
cbcb-heng 1 0.028571 2041564795 0.029831 0.029831 0.957779 cpu=107606,mem=2890646050,ene+
cbcb 1 0.028571 4454658377 0.065046 0.065046 0.439246 cpu=452139,mem=22276633804,en+
class 1 0.028571 255617290 0.003733 0.003733 7.652841 cpu=7021,mem=74554606,energy=+
clip 1 0.028571 3057933838 0.044674 0.044674 0.639549 cpu=33214,mem=2744443460,ener+
cml-abhinav 1 0.028571 39726437 0.000580 0.000580 49.220844 cpu=0,mem=0,energy=0,node=0,b+
cml-director 1 0.028571 672692955 0.009829 0.009829 2.906778 cpu=65977,mem=1080976998,ener+
cml-furongh 1 0.028571 765366431 0.011183 0.011183 2.554814 cpu=78802,mem=1513011200,ener+
cml-hajiagha 1 0.028571 12397205 0.000181 0.000181 157.726572 cpu=0,mem=0,energy=0,node=0,b+
cml-ramani 1 0.028571 0 0.000000 0.000000 1.3575e+11 cpu=0,mem=0,energy=0,node=0,b+
cml-scavenger 1 0.028571 2564733604 0.037475 0.037475 0.762406 cpu=370867,mem=3568986794,ene+
cml-sfeizi 1 0.028571 40715178 0.000595 0.000595 48.025548 cpu=0,mem=0,energy=0,node=0,b+
cml-tokekar 1 0.028571 26458249 0.000387 0.000387 73.903937 cpu=23240,mem=178488320,energ+
cml-tomg 1 0.028571 82800 0.000001 0.000001 2.3615e+04 cpu=0,mem=0,energy=0,node=0,b+
cml-wriva-high 1 0.028571 0 0.000000 0.000000 inf cpu=0,mem=0,energy=0,node=0,b+
cml-wriva 1 0.028571 1047428702 0.015305 0.015305 1.866828 cpu=5366,mem=137386666,energy+
cml 1 0.028571 66866114 0.000975 0.000975 29.299389 cpu=1796,mem=29426756,energy=+
csd-sarahwie 1 0.028571 710055035 0.007034 0.007034 4.061956 cpu=0,mem=0,energy=0,node=0,b+
gamma 1 0.028571 2609474948 0.038129 0.038129 0.749334 cpu=34089,mem=360373862,energ+
mbrc 1 0.028571 73411964 0.001073 0.001073 26.635560 cpu=1195,mem=4896358,energy=0+
mc2 1 0.028571 2682557 0.000039 0.000039 728.919551 cpu=0,mem=0,energy=0,node=0,b+
nexus 1 0.028571 5472794067 0.079964 0.079964 0.357302 cpu=278464,mem=3250599000,ene+
nexus username 1 0.000835 69666 0.000001 0.000021 0.457407 37.435501 cpu=0,mem=0,energy=0,node=0,b+
oasis 1 0.028571 330030 0.000005 0.000005 5.9248e+03 cpu=0,mem=0,energy=0,node=0,b+
quics 1 0.028571 4 0.000000 0.000000 4.1683e+08 cpu=0,mem=0,energy=0,node=0,b+
scavenger 1 0.028571 40888195964 0.597419 0.597419 0.047825 cpu=3142204,mem=29902903931,e+
scavenger username 1 0.000835 171 0.000000 0.000000 0.033975 9.8885e+04 cpu=0,mem=0,energy=0,node=0,b+
vulcan-abhinav 1 0.028571 1742123078 0.025452 0.025452 1.122544 cpu=26166,mem=781842841,energ+
vulcan-djacobs 1 0.028571 912637 0.000013 0.000013 2.1425e+03 cpu=0,mem=0,energy=0,node=0,b+
vulcan-jbhuang 1 0.028571 283430750 0.004141 0.004141 6.898930 cpu=172,mem=5661764,energy=0,+
vulcan-metzler 1 0.028571 262070179 0.003829 0.003829 7.461241 cpu=0,mem=0,energy=0,node=0,b+
vulcan-rama 1 0.028571 0 0.000000 0.000000 1.6942e+14 cpu=0,mem=0,energy=0,node=0,b+
vulcan-ramani 1 0.028571 681333557 0.009955 0.009955 2.869914 cpu=22188,mem=568033280,energ+
vulcan-yaser 1 0.028571 397309 0.000006 0.000006 4.9215e+03 cpu=0,mem=0,energy=0,node=0,b+
vulcan-zwicker 1 0.028571 133204932 0.001946 0.001946 14.679402 cpu=0,mem=0,energy=0,node=0,b+
vulcan 1 0.028571 1247236491 0.018224 0.018224 1.567761 cpu=147273,mem=1161243818,ene+
</pre>

The actual resource billing weights for the three main resources (memory per GB, CPU cores, and number of GPUs if applicable) are per-partition and can be viewed in the <code>TRESBillingWeights</code> line in the output of <code>scontrol show partition</code>. The <code>billing</code> value for a job is the sum of all resource weightings for resources the job has requested. This value is then multiplied by the amount of time a job has run in seconds to get the amount it contributes to the RawUsage for the association within the account it is running under.

====Algorithm====
The algorithm we use for resource weightings differs depending on if there are any GPUs in a partition or not, and is as follows:

=====GPU partitions=====
Each resource (memory/CPU/GPU) is given a weighting value such that their relative billings to each other within the partition are equal (33.33% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU/GPU values are adjusted accordingly.

Different GPU types may also be weighted differently within the GPU relative billing. A baseline GPU type is first chosen. All GPUs of that type and other types that have lower FP32 performance (in [https://en.wikipedia.org/wiki/FLOPS TFLOPS]) are given a weighting factor of 1.0. GPU types with higher FP32 performance than the baseline GPU are given a weighting factor calculated by dividing their FP32 performance by the baseline GPU's FP32 performance. The weighting values for each GPU type are then determined by normalizing the sum of all of GPU cards' billing values multiplied by their weighting factors against the relative billing percentage for GPUs (33.33%).

The current baseline GPU is the [https://www.nvidia.com/en-us/design-visualization/rtx-a4000/ NVIDIA RTX A4000].

=====CPU-only partitions=====
Each resource (memory/CPU) is first given a weighting value such that their relative billings to each other within the partition are equal (50% each). Memory is typically always the most abundant resource by unit (weighting value of 1.0 per GB) and the CPU value is adjusted accordingly. The final CPU weight value is then divided by 10, which translates to roughly 90.9% of the billing weight being for memory and 9.1% being for CPU. The division of the CPU value is done so as to not affect accounts' fair-share priority factors as much when running jobs in CPU-only partitions given the popularity of GPGPU computing.

===Age===
The longer a job is eligible to run but cannot due to resources being unavailable or having a lower priority value than one or more other jobs, the higher the job's priority becomes as it continue to wait in the queue. This is the only priority modifier that can change a job's priority value once it has been submitted, and the priority modifier for this factor reaches its limit after 7 days.

Jobs' age priority factors on our clusters are recalculated every 5 minutes.

===Association===
Some lab/center-specific SLURM accounts have priority values directly attached to them. Jobs run under these accounts gain this many extra points of priority.

===Nice value===
This is a submission argument that you as the user can include when submitting your jobs to deprioritize them. Larger values will deprioritize jobs more, e.g.,
<pre>srun --pty --nice=2 bash</pre>
will have lower priority than
<pre>srun --pty --nice=1 bash</pre>
which will have lower priority than
<pre>srun --pty bash</pre>
assuming all three jobs were submitted at the same time. You cannot use negative values for this argument.

Because this value is absolute, if you want to use it, we would recommend using small numbers - one or two digits - only. Larger numbers may impact your job's ability to run at all as a result of the other factors at play.

Security

2026-04-10T15:16:21Z

Mbaney:

==Security Info and Practices==
* [[CrowdStrike]]
* [[Phishing]]

Malware/virus removal

2026-04-10T15:16:06Z

Mbaney: Mbaney moved page Malware/virus removal to CrowdStrike

#REDIRECT [[CrowdStrike]]

CrowdStrike

2026-04-10T15:16:06Z

Mbaney: Mbaney moved page Malware/virus removal to CrowdStrike

All UMIACS-supported machines have [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0015939 CrowdStrike] installed on them as mandated by the Division of IT.

CrowdStrike

2026-04-10T15:15:10Z

Mbaney: Replaced content with "All UMIACS-supported machines have [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0015939 CrowdStrike] installed on them as mandated by the Division of IT."

All UMIACS-supported machines have [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0015939 CrowdStrike] installed on them as mandated by the Division of IT.

Security

2026-04-10T15:14:50Z

Mbaney:

==Security Info and Practices==
* [[Phishing]]
* [[Malware/virus removal]]

Services/Support

2026-04-10T15:12:14Z

Mbaney:

[[HelpDesk | UMIACS Technical Staff]] provides hardware and operating system (OS) support services for University-owned systems across its various labs and centers. No support is provided for non-University-owned systems.

==Hardware Services==
* In-warranty hardware diagnostic and replacement
* Hardware integration, i.e., physical installation of additional storage devices, RAM, GPUs, etc. for existing pre-assembled systems. If you are assembling a system entirely from parts initially, it is your responsibility to assemble it.

==Operating System Services==
* OS deployment and integration with other UMIACS [[Services]]
* Security updates
* [[Backups]]
* University-licensed software installation

Services/Support

2026-04-10T15:08:46Z

Mbaney:

Services/Support

2026-04-10T15:08:16Z

Mbaney:

[[HelpDesk | UMIACS Technical Staff]] provides hardware and operating system (OS) support services for university-owned systems across its various labs and centers. No support is provided for non-university-owned systems.

==Hardware Services==
* In-warranty hardware diagnostic and replacement
* Hardware integration, i.e., physical installation of additional storage devices, RAM, GPUs, etc. for existing pre-assembled systems. If you are assembling a system entirely from parts initially, it is your responsibility to assemble it.

==Operating System Services==
* OS deployment and integration with other UMIACS [[Services]]
* Security updates
* [[Backups]]
* University-licensed software installation

===Hardware Base Requirements===
In order for a system to qualify for these operating system services, it must meet these hardware base requirements:
* Business or professional model line from a tier one manufacturer, e.g., Dell, Apple, Lenovo, HP, Microsoft
* 3+ years OEM warranty/support, i.e., no refurbished equipment or third-party vendor warranties
* Readily available manufacturer diagnostic tools
* Readily available hardware drivers

Other hardware may be considered on a case-by-case basis, subject to staff evaluation. Systems assembled entirely from individual parts, i.e., the motherboard / CPU / RAM / GPU / PSU / storage / etc. were purchased separately, will never qualify, however.

Matlab

2026-04-10T15:07:44Z

Mbaney:

UMIACS has institutional Matlab licenses through UMD that cover Linux, Windows, and macOS. To access them, you have to be using a University-owned computer. Please note that only Matlab versions R2020a and newer are supported by the University's license server.

If you would like to install the institutional version of Matlab on a University-owned computer yourself, please [[HelpDesk | contact staff]] and we can provide you install media.

==Linux==
[[Modules]] are the preferred way to interact with Matlab on UMIACS-supported Linux machines. The Modules page will provide information on adding Matlab into your environment and using it.

==Windows==
Matlab is available to install through the Company Portal app on UMIACS-supported Windows machines. More details can be found [[Windows#Software | here]].

==macOS==
Please [[HelpDesk | contact staff]] if you would like Matlab installed on a UMIACS-supported macOS machine.

==License Information==
Campus has a limited number of seats for Matlab itself, as well as the various toolboxes, managed by a cluster of [https://www.mathworks.com/help/install/administer-network-licenses.html network license servers]. As such, you must have network connectivity to be able to use the University's licenses. When calling functions in a toolbox, Matlab will automatically check out a license for that toolbox, making it temporarily unavailable to other users. If, while attempting to use Matlab, you run into a license manager error stating that the maximum number of users for Matlab has been reached, this likely indicates that all of campus' available licenses are in use. To check the number of licenses available for a specific toolbox, please see the below sections:

===Toolbox Shortnames===
The toolbox names themselves don't always work with licensing commands; rather, Matlab has a set of shortnames which can be passed to these commands. The following are the valid shortnames:

Aerospace_Blockset
Aerospace_Toolbox
Antenna_Toolbox
Bioinformatics_Toolbox
Communication_Blocks
Communication_Toolbox
Compiler
Control_Toolbox
Curve_Fitting_Toolbox
Data_Acq_Toolbox
Database_Toolbox
Datafeed_Toolbox
Distrib_Computing_Toolbox
Econometrics_Toolbox
Excel_Link
Fin_Derivatives_Toolbox
Fin_Instruments_Toolbox
Financial_Toolbox
Fixed_Income_Toolbox
Fixed_Point_Toolbox
Fuzzy_Toolbox
GADS_Toolbox
Identification_Toolbox
Image_Acquisition_Toolbox
Image_Toolbox
Instr_Control_Toolbox
MAP_Toolbox
MATLAB
MATLAB_Builder_for_Java
MATLAB_Coder
MATLAB_Excel_Builder
MATLAB_Report_Gen
MPC_Toolbox
Neural_Network_Toolbox
Optimization_Toolbox
OPC_Toolbox
PDE_Toolbox
Phased_Array_System_Toolbox
Real-Time_Workshop
Robotics_System_Toolbox
Robust_Toolbox
RTW_Embedded_Coder
Signal_Blocks
Signal_Toolbox
SimEvents
SimMechanics
Simscape
SIMULINK
Simulink_Control_Design
Simulink_Design_Optim
Stateflow
Stateflow_Coder
Statistics_Toolbox
Symbolic_Toolbox
Video_and_Image_Blockset
Virtual_Reality_Toolbox
Wavelet_Toolbox

If you are having problems, feel free to contact [[HelpDesk | UMIACS Staff]], however please note that the Matlab license servers are hosted by campus' Division of Information Technology.

==Matlab Customizations==
You can tweak your Matlab experience using a file "startup.m". If this file is present in the current working directory where you launch Matlab, it will be executed. You can perform a variety of tasks using this file. These include (but are not limited to) customizing the window environment, setting variables for later use, and enabling use of 3rd party toolboxes. Alternatively, you can use the userpath env variable to set a path other than the current working directory for startup and 3rd party toolbox locations. Please see the following links for more information:

* [http://www.mathworks.com/help/matlab/ref/matlabrc.html Mathworks: Startup.m]
* [https://www.mathworks.com/help/matlab/ref/userpath.html Mathworks: Userpath function]

Matlab

2026-04-10T15:07:14Z

Mbaney:

UMIACS has institutional Matlab licenses through UMD that cover Linux, Windows, and macOS. To access them, you have to be using a University owned computer. Please note that only Matlab versions R2020a and newer are supported by the University's license server.

If you would like to install the institutional version of Matlab on a self-supported machine, please [[HelpDesk | contact staff]] and we can provide you install media.

==Linux==
[[Modules]] are the preferred way to interact with Matlab on UMIACS-supported Linux machines. The Modules page will provide information on adding Matlab into your environment and using it.

==Windows==
Matlab is available to install through the Company Portal app on UMIACS-supported Windows machines. More details can be found [[Windows#Software | here]].

==macOS==
Please [[HelpDesk | contact staff]] if you would like Matlab installed on a UMIACS-supported macOS machine.

==License Information==
Campus has a limited number of seats for Matlab itself, as well as the various toolboxes, managed by a cluster of [https://www.mathworks.com/help/install/administer-network-licenses.html network license servers]. As such, you must have network connectivity to be able to use the University's licenses. When calling functions in a toolbox, Matlab will automatically check out a license for that toolbox, making it temporarily unavailable to other users. If, while attempting to use Matlab, you run into a license manager error stating that the maximum number of users for Matlab has been reached, this likely indicates that all of campus' available licenses are in use. To check the number of licenses available for a specific toolbox, please see the below sections:

===Toolbox Shortnames===
The toolbox names themselves don't always work with licensing commands; rather, Matlab has a set of shortnames which can be passed to these commands. The following are the valid shortnames:

Aerospace_Blockset
Aerospace_Toolbox
Antenna_Toolbox
Bioinformatics_Toolbox
Communication_Blocks
Communication_Toolbox
Compiler
Control_Toolbox
Curve_Fitting_Toolbox
Data_Acq_Toolbox
Database_Toolbox
Datafeed_Toolbox
Distrib_Computing_Toolbox
Econometrics_Toolbox
Excel_Link
Fin_Derivatives_Toolbox
Fin_Instruments_Toolbox
Financial_Toolbox
Fixed_Income_Toolbox
Fixed_Point_Toolbox
Fuzzy_Toolbox
GADS_Toolbox
Identification_Toolbox
Image_Acquisition_Toolbox
Image_Toolbox
Instr_Control_Toolbox
MAP_Toolbox
MATLAB
MATLAB_Builder_for_Java
MATLAB_Coder
MATLAB_Excel_Builder
MATLAB_Report_Gen
MPC_Toolbox
Neural_Network_Toolbox
Optimization_Toolbox
OPC_Toolbox
PDE_Toolbox
Phased_Array_System_Toolbox
Real-Time_Workshop
Robotics_System_Toolbox
Robust_Toolbox
RTW_Embedded_Coder
Signal_Blocks
Signal_Toolbox
SimEvents
SimMechanics
Simscape
SIMULINK
Simulink_Control_Design
Simulink_Design_Optim
Stateflow
Stateflow_Coder
Statistics_Toolbox
Symbolic_Toolbox
Video_and_Image_Blockset
Virtual_Reality_Toolbox
Wavelet_Toolbox

If you are having problems, feel free to contact [[HelpDesk | UMIACS Staff]], however please note that the Matlab license servers are hosted by campus' Division of Information Technology.

==Matlab Customizations==
You can tweak your Matlab experience using a file "startup.m". If this file is present in the current working directory where you launch Matlab, it will be executed. You can perform a variety of tasks using this file. These include (but are not limited to) customizing the window environment, setting variables for later use, and enabling use of 3rd party toolboxes. Alternatively, you can use the userpath env variable to set a path other than the current working directory for startup and 3rd party toolbox locations. Please see the following links for more information:

* [http://www.mathworks.com/help/matlab/ref/matlabrc.html Mathworks: Startup.m]
* [https://www.mathworks.com/help/matlab/ref/userpath.html Mathworks: Userpath function]

Mathematica

2026-04-10T15:05:42Z

Mbaney:

Mathematica is freely available to all University-owned machines.

On our UMIACS-supported Linux hosts, Mathematica can be accessed through our [[Modules]].
*The command <code>module add mathematica</code> will add the default version of Mathematica to your Environment.
*To see the versions of Mathematica that are available use the command <code>module avail mathematica</code>.
*To add a specific version of Mathematica to your Environment (i.e. Mathematica 12.0) use the command <code>module add mathematica/12.0</code>
*Further information can be found on our [[Modules | Modules page]].

For UMIACS-supported Windows or macOS machines, please [[HelpDesk | contact staff]].

==Activation==
There is no automated way to activate Mathematica across our domain. As a result, each computer will have to be registered once against our hosted license server. Any user can go through this process, and it should persist indefinitely so long as the computer is not reinstalled.

* Upon being prompted, click "Other ways to activate" in the bottom row: [[Image:math1.jpg| 500px| Mathematica 10 Activation Screen 1]] 
* Click "Connect to a network license server": [[Image:math2.jpg| 500px| Mathematica 10 Activation Screen 2]] 
* Enter "licserv.umiacs.umd.edu" as the license server. Click "Activate": [[Image:math3.jpg| 500px| Mathematica 10 Activation Screen 3]] 
* Accept the terms, and click ok: [[Image:math4.jpg| 500px| Mathematica 10 Activation Screen 4]]
* Mathematica should now be activated on the computer.

UbuntuPrinting

2026-04-10T15:01:34Z

Mbaney:

==Overview==
Printing from your personally-owned Ubuntu laptop or desktop is supported. If you are sitting at a UMIACS-supported Ubuntu desktop, please see our common UNIX/Linux printing documentation for [[CUPS]] instead.

This documentation may not work for all versions of Ubuntu. If you run into any issues, please [[HelpDesk | contact staff]].

===Printer Accessibility===
In order to print you '''need to be on the UMIACS internal network''', either via a wired proxy drop or an [[SecureShellTunneling | SSH tunnel]].

==Connecting to a printer==
In order for printing to function properly, you must complete '''ALL''' of the following steps:
# Open a web browser and go to <code>localhost:631</code>. This will bring up the CUPS home page.
#: [[File:Cups_home_page_12-2024.png|700px|alt="Screenshot of CUPS home page"]]
# At the top of the screen, select '''Administration'''. If you're given a login prompt, enter your username and password for the computer. Select the 'Add Printer' button.
#: [[File:Cups_admin_page_12-2024.png|700px|alt="Screenshot of CUPS Administration Page"]]
# Open a new tab in your web browser and navigate to https://print.umiacs.umd.edu/printers.
# Find the '''name''' of the printer that you wish to connect to. If you are unsure of which one to pick, see the [[PrinterQueueNaming | selecting a print queue]] page.
# Navigate back to the 'Add Printer' page. Select the 'Internet Printing Protocol (ipp)' radio button, then hit the 'Continue' button.
#: [[File:Cups_ipp_radio_button_12-2024.png|450px|alt="Screenshot of CUPS administration page with indicated selection"]]
# Next to the 'Connection:' field, enter <code>ipp://print.umiacs.umd.edu:631/printers/</code> with the name of the printer added at the end. For instance, to add cps432-3208 (color printer in IRB 3208), you would enter <code>ipp://print.umiacs.umd.edu:631/printers/cps432-3208</code> in the field. Hit 'Continue'.
#: [[File:Cups_connection_entry_12-2024.png|500px|alt="Screenshot of CUPS administration page with indicated fields"]]
# Enter the printer '''name''' in the 'Name:' field. It's recommended to enter useful information in the 'Description:' and 'Location:' fields, but it's not required. Keep 'Share This Printer' unchecked. Hit 'Continue'.
#: [[File:Cups_name_entry_12-2024.png|500px|alt="Screenshot of CUPS administration page with indicated fields"]]
# In the 'Make:' list, select the brand of the printer as noted in step 4, and click 'Continue'.
#: [[File:Cups_make_selection_12-2024.png|400px|alt="Screenshot of make options"]]
# In the 'Model:' list, select the specific model of the printer as noted in step 4. Then click 'Add Printer'.
#: [[File:Cups_model_selection_12-2024.png|500px|alt="Screenshot of model options"]]
# The printer likely requires additional configuration. When adding the printer through the CUPS interface, it is added system-wide. It can be viewed and modified in the 'Printers' menu of the Ubuntu settings, where in Ubuntu 24.04, there is a kebab menu for each printer added. In the kebab menu, select 'Printing Options'. We recommend changing the printer options here instead of in the CUPS browser interface, though you can make changes there if you prefer.
#: [[File:Ubuntu_printer_options-12-2024.png|600px|alt="Screenshot of Ubuntu printer options"]]
# In 'Page Setup', at minimum, change the media size to 'US Letter', unless you're specifically printing from a tray which doesn't use letter paper. Change the 'Pages per side', 'Two-sided', and 'Orientation' settings according to your preferences.
#: [[File:Ubuntu_page_setup_12-2024.png|600px|alt="Screenshot of printer settings"]]
# The available settings depend on the printer you added. For cps432-3208, there is an 'Installable Options' menu. Use this to configure the trays according to the printer's tray configuration.
#: [[File:Ubuntu_tray_setup_12-2024.png|600px|alt="Screenshot of installable options menu"]]
# Verify this process by printing a test page, which is the top left button in the 'Printing Options' submenu in Ubuntu 24.04. You should get the default Ubuntu test page. If you just get a page with a string of text then the printer is not properly configured, and you should verify the settings.

Printing

2026-04-10T15:01:20Z

Mbaney:

==Printing==
===OS-specific printing guides===
* [[WindowsPrinting | Windows Printing Guide]]
* [[OSXPrinting | macOS Printing Guide]]
* [[CUPS | Linux Printing Guide (RHEL and Ubuntu)]]
* [[UbuntuPrinting | Ubuntu Printing Guide for personally-owned devices]]

To print to UMIACS-supported printers, you will need to be either within the UMIACS network border or connected to the [[VPN]]. Afterwards, follow the above instructions for your operating system.

===Other Printing Pages===
* [[PrinterQueueNaming | Selecting A Print Queue (Color/Driver)]]
* [[UNIXPrinting | Legacy UNIX Printing Guide (Linux and Solaris)]]
* [[UMIACS Public Printers]]
* [[PrinterTroubleshooting | Printer troubleshooting]]