UMIACS - User contributions [en]

VS Code

2026-04-20T16:30:33Z

Seide: /* Clean up when done */ Add information about "Remote.SSH: Reconnection Grace Time"

[https://code.visualstudio.com/ Visual Studio (VS) Code] is a multi-platform source-code editor developed by Microsoft. It can be used with a variety of programming languages and can have its base functionality extended through extensions available via its [https://marketplace.visualstudio.com/vscode marketplace]. The base editor itself is free, but some extensions may require paid subscriptions.

There are also forks/open source alternatives of VS Code, such as [https://www.cursor.com Cursor] or [https://vscodium.com/ VSCodium]. The same general principals below apply to these editors as well.

==Cluster Usage==
It can be convenient when using our [[SLURM]] computing clusters to open a connection to a [https://code.visualstudio.com/docs/remote/vscode-server VS Code Server] session on the submission node(s) that you have access to via VS Code's [https://code.visualstudio.com/docs/remote/ssh Remote - SSH extension]. Steps to set this up can be found [https://code.visualstudio.com/docs/remote/ssh#_installation here]. Use your UMIACS username and the [https://en.wikipedia.org/wiki/Fully_qualified_domain_name fully qualified domain name (FQDN)] of the submission node you want to connect to. Password authentication does not provide an option to save your password, so for convenience, you may want to set up an [[SSH/Keys | SSH key]].

For a list of active issues with the Remote - SSH extension, please see Microsoft's GitHub tracker [https://github.com/Microsoft/vscode-remote-release/labels/ssh here].

===Best Practices===
Because of the multi-tenant nature of our submission nodes, using the Remote - SSH extension to connect to a VS Code Server running on a submission node can have adverse affects on other users simultaneously using the submission nodes if not properly managed. The following is a list of "best practices" that we ask you please follow to minimize the chance of impacting others on submission nodes. We have compiled the list through observation as well as through discussion with users.

====Use only as a code editor====
VS Code can perform many common file management operations, such as moving, copying, or deleting files or directories. However, the overhead of providing progress updates / status messages through VS Code's UI can cause load spikes that result in other system-level and user processes on the submission node slowing down. This is especially the case if operating on thousands to millions of small image or video files. In extreme cases, it can seem as though the submission node is wholly unresponsive.

The best way to combat this is to use VS Code as just a code editor as strictly as you can (meaning creating new and/or editing existing source code files only). Use standard command-line programs such as <code>mv</code>/<code>cp</code>/<code>rm</code> in VS Code's terminal window or a separate application's terminal window for local file management instead, or use the techniques advertised [[LocalDataTransfer | here]] for larger data transfers between nodes.

====Do not open large files====
Files that you open in VS Code's text editor on a remote machine through the Remote - SSH extension are loaded entirely into RAM on the remote machine. Because of the persistence of VS Code Server processes, opening several unique files may result in them all remaining in RAM for extended periods of time even if you no longer have tabs open for them in the Tab Bar.

As such, please refrain from opening large files (more than a few MB) in VS Code itself on submission nodes. Use a lighter-weight command-line based text editor such as [https://www.vim.org vim] or otherwise in VS Code's terminal window or a separate application's terminal window for local file editing instead.

If you absolutely need to open one or more large files for a very brief period of time on a submission node, please clean up your server session on that node (see below section) as soon as possible when done.

====Ripgrep====
VS Code comes bundled with [https://github.com/BurntSushi/ripgrep Ripgrep] as the default search program. Even if you aren't explicitly using the search functionality, VS Code will periodically run "rg --files", which will enumerate the list of files that Ripgrep would search. In workspaces with very large numbers of small files, this can result in degraded performance due to NFS overhead amplified by small file I/O. The following tips can ensure that Ripgrep does not significantly impact performance.

=====Keep data separate from code where possible=====
Make sure that datasets are kept outside of your VS Code workspace so that searches do not look through these directories. Alternatively, by default VS Code will not search files listed in a .ignore, .gitignore, or .rgignore file, so adding directories with datasets to one of these files is another option. Any directory with only binary data is a good candidate to add to this file. If you have changed your defaults or want to tweak these exclude settings further, these settings are found under "Settings->Features->Search".

=====Limit maximum number of threads for searches=====
By default, VS Code will automatically determine the number of threads to use when running Ripgrep. You can force a maximum number of threads under "Settings->Features->Search->Ripgrep: Max Threads" in order to limit the performance impact.

====Clean up when done====
By default, VS Code Server will keep the processes used by your session active for a few hours after you disconnect. This takes away CPU time and memory from other users who are actively trying to use the nodes. There are two ways that you can ensure that your session gets closed in a reasonable amount of time.

=====Shortening the Remote-SSH Reconnection Grace Time=====
You can configure the Remote-SSH extension to reduce this time to something reasonable (e.g. 5 minutes).

This can be done by opening the [https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette Command Palette] (Ctrl+Shift+P) and searching "Remote-SSH: Settings".

: [[File:VSCode-remotesshsettings.jpg|alt=Under the Command Palette menu, "Remote-SSH: Settings" is searched.]]

This will open a dialog box with settings for the Remote-SSH extension. From here, you can scroll down or search for "Remote.SSH: Reconnection Grace Time" and configure this to something reasonable.

: [[File:VSCode-reconnectiongracetime.jpg|alt=In the "Settings" dialog box, "Remote.SSH: Reconnection Grace Time" is overridden to 300 seconds.]]

=====Manually killing a remote VS Code server=====
You can also manually kill a remote VS Code server by opening the [https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette Command Palette] (Ctrl+Shift+P) and searching "Kill VS Code Server on Host...".

: [[File:VSCode-killremote1.png|alt=Under the Command Palette menu, "Kill VS Code Server on Host..." is searched.]]

Select the submission node you were using and click on it.

: [[File:VSCode-killremote2.png|alt='nexus' is searched and a 2 submission nodes pop up as matching searches.]]

Wait a few seconds and you should get a message confirming the command went through.

: [[File:VSCode-killremote3.png|alt=Pop-up confirming the kill command went through.]]

The next time you connect to the same submission node, you will have a fresh VS Code session on it.

==Common Issues==
If you are having trouble [[SSH]]'ing to a submission node with VS Code, check if your home directory is full.

# Connect to one of your Nexus submission nodes via [[SSH]] '''not''' using VS Code.
# Navigate to your home directory. <pre>cd ~</pre>
# Check if it is full. <pre>df -h .</pre>
# If it is full (Avail 0), delete some files to free up at least ~100MB of space. <pre>Filesystem Size Used Avail Use% Mounted on
data.isilon.umiacs.umd.edu:/ifs/umiacs/homes 30G 30G 0 100% /fs/nfshomes</pre>
# Try connecting with VS Code again.

File:VSCode-reconnectiongracetime.jpg

2026-04-20T16:27:50Z

Seide: Seide uploaded a new version of File:VSCode-reconnectiongracetime.jpg

== Summary ==
VS Code Remote-SSH: Settings page showing "Reconnection Grace Time" overridden to 300 seconds.

File:VSCode-remotesshsettings.jpg

2026-04-20T16:26:27Z

Seide: Seide uploaded a new version of File:VSCode-remotesshsettings.jpg

== Summary ==
VS Code Command Pallete with "Remote-SSH: Settings" selected.

File:VSCode-reconnectiongracetime.jpg

2026-04-20T16:03:59Z

Seide: VS Code Remote-SSH: Settings page showing "Reconnection Grace Time" overridden to 300 seconds.

== Summary ==
VS Code Remote-SSH: Settings page showing "Reconnection Grace Time" overridden to 300 seconds.

File:VSCode-remotesshsettings.jpg

2026-04-20T16:03:04Z

Seide: VS Code Command Pallete with "Remote-SSH: Settings" selected.

== Summary ==
VS Code Command Pallete with "Remote-SSH: Settings" selected.

Template:Note

2026-03-23T18:34:42Z

Seide: Add alt text to the exclamation point image.

<pre<noinclude></noinclude> style="width: 65%; white-space:pre-wrap; margin-left: 5%;">
{|
|[[File:Exclamation-point.png | 25px | alt=Note]]
|{{{1}}}
|}</pre<noinclude></noinclude>>

VS Code

2026-03-05T21:29:40Z

Seide: /* Search tuning */

[https://code.visualstudio.com/ Visual Studio (VS) Code] is a multi-platform source-code editor developed by Microsoft. It can be used with a variety of programming languages and can have its base functionality extended through extensions available via its [https://marketplace.visualstudio.com/vscode marketplace]. The base editor itself is free, but some extensions may require paid subscriptions.

There are also forks/open source alternatives of VS Code, such as [https://www.cursor.com Cursor] or [https://vscodium.com/ VSCodium]. The same general principals below apply to these editors as well.

==Cluster Usage==
It can be convenient when using our [[SLURM]] computing clusters to open a connection to a [https://code.visualstudio.com/docs/remote/vscode-server VS Code Server] session on the submission node(s) that you have access to via VS Code's [https://code.visualstudio.com/docs/remote/ssh Remote - SSH extension]. Steps to set this up can be found [https://code.visualstudio.com/docs/remote/ssh#_installation here]. Use your UMIACS username and the [https://en.wikipedia.org/wiki/Fully_qualified_domain_name fully qualified domain name (FQDN)] of the submission node you want to connect to. Password authentication does not provide an option to save your password, so for convenience, you may want to set up an [[SSH/Keys | SSH key]].

For a list of active issues with the Remote - SSH extension, please see Microsoft's GitHub tracker [https://github.com/Microsoft/vscode-remote-release/labels/ssh here].

===Best Practices===
Because of the multi-tenant nature of our submission nodes, using the Remote - SSH extension to connect to a VS Code Server running on a submission node can have adverse affects on other users simultaneously using the submission nodes if not properly managed. The following is a list of "best practices" that we ask you please follow to minimize the chance of impacting others on submission nodes. We have compiled the list through observation as well as through discussion with users.

====Use only as a code editor====
VS Code can perform many common file management operations, such as moving, copying, or deleting files or directories. However, the overhead of providing progress updates / status messages through VS Code's UI can cause load spikes that result in other system-level and user processes on the submission node slowing down. This is especially the case if operating on thousands to millions of small image or video files. In extreme cases, it can seem as though the submission node is wholly unresponsive.

The best way to combat this is to use VS Code as just a code editor as strictly as you can (meaning creating new and/or editing existing source code files only). Use standard command-line programs such as <code>mv</code>/<code>cp</code>/<code>rm</code> in VS Code's terminal window or a separate application's terminal window for local file management instead, or use the techniques advertised [[LocalDataTransfer | here]] for larger data transfers between nodes.

====Do not open large files====
Files that you open in VS Code's text editor on a remote machine through the Remote - SSH extension are loaded entirely into RAM on the remote machine. Because of the persistence of VS Code Server processes, opening several unique files may result in them all remaining in RAM for extended periods of time even if you no longer have tabs open for them in the Tab Bar.

As such, please refrain from opening large files (more than a few MB) in VS Code itself on submission nodes. Use a lighter-weight command-line based text editor such as [https://www.vim.org vim] or otherwise in VS Code's terminal window or a separate application's terminal window for local file editing instead.

If you absolutely need to open one or more large files for a very brief period of time on a submission node, please clean up your server session on that node (see below section) as soon as possible when done.

====Ripgrep====
VS Code comes bundled with Ripgrep as the default search program. Even if you aren't explicitly using the search functionality, VS Code will periodically run "rg --files", which will enumerate the list of files that Ripgrep would search. In workspaces with very large numbers of small files, this can result in degraded performance due to NFS overhead amplified by small file I/O. The following tips can ensure that Ripgrep does not significantly impact performance.

=====Keep data separate from code where possible=====
Make sure that datasets are kept outside of your VS Code workspace so that searches do not look through these directories. Alternatively, by default VS Code will not search files listed in a .ignore, .gitignore, or .rgignore file, so adding directories with datasets to one of these files is another option. Any directory with only binary data is a good candidate to add to this file. If you have changed your defaults or want to tweak these exclude settings further, these settings are found under "Settings->Features->Search".

=====Limit maximum number of threads for searches=====
By default, VS Code will automatically determine the number of threads to use when running Ripgrep. You can force a maximum number of threads under "Settings->Features->Search->Ripgrep: Max Threads" in order to limit the performance impact.

====Clean up when done====
If you are done using VS Code for an extended period of time (several hours / overnight), consider manually killing your server session on the submission node when exiting. By default, VS Code Server will keep the processes used by your session active for a few hours after you disconnect, keeping the VS Code Server processes on the submission node you were using alive, despite no longer actively being used.

This can be done by opening the [https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette Command Palette] (Ctrl+Shift+P) and searching "Kill VS Code Server on Host...".

: [[File:VSCode-killremote1.png]]

Select the submission node you were using and click on it.

: [[File:VSCode-killremote2.png]]

Wait a few seconds and you should get a message confirming the command went through.

: [[File:VSCode-killremote3.png]]

The next time you connect to the same submission node, you will have a fresh VS Code session on it.

==Common Issues==
If you are having trouble [[SSH]]'ing to a submission node with VS Code, check if your home directory is full.

# Connect to one of your Nexus submission nodes via [[SSH]] '''not''' using VS Code.
# Navigate to your home directory. <pre>cd ~</pre>
# Check if it is full. <pre>df -h .</pre>
# If it is full (Avail 0), delete some files to free up at least ~100MB of space. <pre>Filesystem Size Used Avail Use% Mounted on
data.isilon.umiacs.umd.edu:/ifs/umiacs/homes 30G 30G 0 100% /fs/nfshomes</pre>
# Try connecting with VS Code again.

Nexus

2026-03-05T21:24:29Z

Seide: /* Access */ Link to "Nexus/Submission Node Policy"

The Nexus is the combined scheduler of resources in UMIACS. The resource manager for Nexus is [[SLURM]]. Resources are arranged into partitions where users are able to schedule computational jobs. Users are arranged into a number of SLURM accounts based on faculty, lab, or center investments.

= Getting Started =
All accounts in UMIACS are sponsored. If you don't already have a UMIACS account, please see [[Accounts]] for information on getting one. You need a full UMIACS account (not a [[Accounts/Collaborator | collaborator account]]) in order to access Nexus.

== Access ==
Your access to submission nodes (alternatively called login nodes) for Nexus computational resources is determined by your account sponsor's department, center, or lab affiliation. You can log into the [https://intranet.umiacs.umd.edu/directory/cr/ UMIACS Directory CR application] and select the Computational Resource (CR) in the list that has the prefix <code>nexus</code>. The Hosts section lists your available submission nodes - generally a pair of nodes of the format <tt>nexus<department, lab, or center abbreviation>[00,01]</tt>, e.g., <tt>nexusgroup00</tt> and <tt>nexusgroup01</tt>.

Once you have identified your submission nodes, you can [[SSH]] into them [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0016076 after connecting to UMD's GlobalProtect VPN]. From there, you are able to submit to the cluster via our [[SLURM]] workload manager. You need to make sure that your submitted jobs have the correct account, partition, and qos.

Please read our [[Nexus/Submission_Node_Policy|Submission Node Policy]] for guidance on appropriate usage of a submission node. If a submission node becomes unresponsive due to disregarding this policy, <b>we may kill user processes on these nodes to resolve the issue</b>. We reserve the right to take action on users who repeatedly cause issues on submission nodes.

== Jobs ==
[[SLURM]] jobs are [[SLURM/JobSubmission | submitted]] by either <code>srun</code> or <code>sbatch</code> depending if you are doing an interactive job or batch job, respectively. You need to provide the where/how/who to run the job and specify the resources you need to run with.

For the who/where/how, you may be required to specify <code>--account</code>, <code>--partition</code>, and/or <code>--qos</code> (respectively) to be able to adequately submit jobs to the Nexus.

For resources, you may need to specify <code>--time</code> for time, <code>--cpus-per-task</code> for CPUs, <code>--mem</code> for RAM, and <code>--gres=gpu</code> for GPUs in your submission arguments to meet your requirements. There are defaults for all four; if you don't specify something, you will get the default value for that resource, which is minimal (e.g., by default, NO GPUs are included if you do not specify <code>--gres=gpu</code>). For more information about submission flags for GPU resources, see [[SLURM/JobSubmission#Requesting_GPUs | here]]. You may also use <code>--ntasks</code> to specify the number of parallel processes to run, with each task having its own set of the resources specified above. You can run <code>man srun</code> on your submission node for a complete list of available submission arguments.

For a list of available GPU types on Nexus and their specs, please see [[Nexus/GPUs]].

For details on how the network for Nexus is architected, please see [[Nexus/Network]]. This can be important if you wish to optimize performance of your jobs.

=== Interactive ===
Once logged into a submission node, you can run simple interactive jobs. If your session is interrupted from the submission node, the job will be killed. As such, we encourage use of a terminal multiplexer such as [[Tmux]].

<pre>
$ srun --pty --cpus-per-task=4 --mem=2gb --gres=gpu:1 bash
srun: Job account was unset; set to user default of 'nexus'
srun: Job partition was unset; set to cluster default of 'tron'
srun: Job QoS was unset; set to association default of 'default'
srun: Job time limit was unset; set to partition default of 60 minutes
srun: job 1 queued and waiting for resources
srun: job 1 has been allocated resources
$ hostname
tron62.umiacs.umd.edu
$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-daad6a04-a2ce-1183-ce53-b267048f750a)
</pre>

=== Batch ===
Batch jobs are scheduled with a script file with an optional ability to embed job scheduling parameters via variables that are defined by <code>#SBATCH</code> lines at the top of the file. You can find some examples in our [[SLURM/JobSubmission]] documentation.

= Partitions =
The SLURM resource manager uses partitions to act as job queues which can restrict size, time and user limits. The Nexus has a number of different partitions of resources. Different Centers, Labs, and Faculty are able to invest in computational resources that are restricted to approved users through these partitions.

'''Partitions usable by all non-[[ClassAccounts |class account]] users:'''
* [[Nexus/Tron]] - Pool of resources available to all non-class accounts sponsored by either UMIACS or CSD faculty.
* Scavenger - [https://slurm.schedmd.com/preempt.html Preemption] partition that contains [https://en.wikipedia.org/wiki/X86-64 x86_64] architecture nodes from multiple other partitions. More resources are available to schedule simultaneously than in other partitions, however jobs are subject to preemption rules. You are responsible for ensuring your jobs handle this preemption correctly. The SLURM scheduler will simply restart a preempted job with the same submission arguments when it is available to run again. For an overview of things you can check within scripts to determine if your job was preempted/resumed, see [[SLURM/Preemption]].
* Scavenger (aarch64) - Preemption partition identical in design to <tt>scavenger</tt>, but only contains [https://en.wikipedia.org/wiki/AArch64 aarch64] architecture nodes.

'''Partitions usable by [[ClassAccounts]]:'''
* [[ClassAccounts#Cluster_Usage | Class]] - Pool of resources available to class accounts sponsored by either UMIACS or CSD faculty.

'''Partitions usable by specific lab/center users:'''
* [[Nexus/CBCB]] - CBCB lab pool available for CBCB lab members.
* [[Nexus/CLIP]] - CLIP lab pool available for CLIP lab members.
* [[Nexus/CML]] - CML lab pool available for CML lab members.
* [[Nexus/GAMMA]] - GAMMA lab pool available for GAMMA lab members.
* [[Nexus/MBRC]] - MBRC lab pool available for MBRC lab members.
* [[Nexus/MC2]] - MC2 lab pool available for MC2 lab members.
* [[Nexus/QuICS]] - QuICS lab pool available for QuICS lab members.
* [[Nexus/Vulcan]] - Vulcan lab pool available for Vulcan lab members.

You can view the partitions that you have access to by using the <code>show_partitions</code> command. By default, the command will show only the partitions that are available to you.

<pre>
$ show_partitions
Name AllowAccounts AllowQos MaxNodes Nodes
------------------------ ----------------------- ------------------------------ ----------- ----------------------------
scavenger scavenger scavenger UNLIMITED brigid[16-19]
cbcb[00-29]
clip[00-13]
cml[00,02-13,15-28,30-33]
cmlcpu[00-04,06-07]
gammagpu[00-21]
legacy[00-11,13-28,30-36]
legacygpu[00-07]
quics00
tron[00-44,46-69]
vulcan[00-45]
------------------------------------------------------------------------------------------------------------------------
scavenger-aarch64 scavenger scavenger-aarch64 UNLIMITED oasis[00-39]
------------------------------------------------------------------------------------------------------------------------
tron nexus default UNLIMITED tron[00-44,46-69]
high
medium
</pre>

If you want to see information for all of the partitions, including those that you do not have access to, you can use the <code>show_partitions --all</code> command.

<pre>
$ show_partitions --all
Name AllowAccounts AllowQos MaxNodes Nodes
------------------------ ----------------------- ------------------------------ ----------- ----------------------------
cbcb cbcb default UNLIMITED cbcb[00-20,22-29]
medium legacy[00-11,13-28,30-36]
high
huge-long
highmem
------------------------------------------------------------------------------------------------------------------------
cbcb-heng cbcb-heng default UNLIMITED cbcb[26-29]
medium
high
huge-long
highmem
------------------------------------------------------------------------------------------------------------------------
cbcb-interactive cbcb interactive UNLIMITED cbcb21
...
</pre>

= Quality of Service (QoS) =
SLURM uses Quality of Service (QoS) both to provide limits on job sizes (termed by us as "job QoS") as well as to limit resources used by all jobs running in a partition, either per user or per group (termed by us as "partition QoS").

=== Job QoS ===
Job QoS are used to provide limits on the size of job that you can run. You should try to allocate only the resources your job actually needs, as resources that each of your jobs schedules are counted against your [[SLURM/Priority#Fair-share | fair-share priority]] in the future.
* default - Default job QoS. Limited to 4 CPU cores, 1 GPU, and 32GB RAM per job. The maximum wall time per job is 3 days.
* medium - Limited to 8 CPU cores, 2 GPUs, and 64GB RAM per job. The maximum wall time per job is 2 days.
* high - Limited to 16 CPU cores, 4 GPUs, and 128GB RAM per job. The maximum wall time per job is 1 day.
* scavenger - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. 11% of the total resources available for each trackable resource type in the partition (CPUs/GPUs/RAM) is permitted simultaneously across all of your jobs running with this job QoS, enforced via the corresponding partition QoS (below) for the scavenger partition. This job QoS is paired one-to-one with the scavenger partition. To use this job QoS, include <code>--partition=scavenger</code> and <code>--account=scavenger</code> in your submission arguments. Do not include any job QoS argument other than <code>--qos=scavenger</code> (optional) or submission will fail.
* scavenger-aarch64 - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. This job QoS is paired one-to-one with the scavenger-aarch64 partition. To use this job QoS, include <code>--partition=scavenger-aarch64</code>, <code>--account=scavenger</code>, and <code>--qos=scavenger-aarch64</code> in your submission arguments.

You can display these job QoS from the command line using the <code>show_qos</code> command. By default, the command will only show job QoS that you can access. The above five job QoS are the ones that everyone can access.

<pre>
$ show_qos
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
scavenger 3-00:00:00
scavenger-aarch64 3-00:00:00
</pre>

If you want to see all job QoS, including those that you do not have access to, you can use the <code>show_qos --all</code> command.

<pre>
$ show_qos --all
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
cml-cpu 7-00:00:00 8
cml-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
cml-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
cml-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
cml-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
cml-scavenger 3-00:00:00 gres/gpu=24
cml-very_high 1-12:00:00 cpu=32,gres/gpu=8,mem=256G 8 gres/gpu=12
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
gamma-huge-long 10-00:00:00 cpu=32,gres/gpu=16,mem=256G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
highmem 21-00:00:00 cpu=128,mem=2T
huge-long 10-00:00:00 cpu=32,gres/gpu=8,mem=256G
interactive 12:00:00 cpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
oasis-exempt 10-00:00:00 cpu=160,mem=28114M
scavenger 3-00:00:00
scavenger-aarch64 3-00:00:00
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-exempt 7-00:00:00 cpu=32,gres/gpu=8,mem=256G 2
vulcan-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
vulcan-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-sailon 3-00:00:00 cpu=32,gres/gpu=8,mem=256G gres/gpu=48
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
vulcan-scavenger-mu+ 3-00:00:00 cpu=288,gres/gpu=72,mem=1152G
</pre>

You are able to submit to any partition that is listed in the <code>show_partitions</code> command. If you need to use an account other than the default account <tt>nexus</tt>, you will need to specify it via the <code>--account</code> submission argument.

=== Partition QoS ===
Partition QoS are used to limit resources used by all jobs running in a partition, either per user (MaxTRESPU) or per group (GrpTRES).

To view partition QoS, use the <code>show_partition_qos</code> command.

<pre>
$ show_partition_qos
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
scavenger-aarch64_part 500
scavenger_part 500 cpu=11%,gres/gpu=11%,mem=11%
tron 500 cpu=32,gres/gpu=4,mem=262144M
</pre>

The scavenger_part partition QoS has relative TRES limits based on the current hardware in a given partition, represented with percentages. To see the current actual TRES limits of this partition QoS, you can use the <code>-r/--real</code> argument.

<pre>
$ show_partition_qos -r
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
scavenger-aarch64_part 500
scavenger_part 500 cpu=888,gres/gpu=140,mem=12574G
tron 500 cpu=32,gres/gpu=4,mem=262144M
</pre>

If you want to see all partition QoS, including those that you do not have access to, you can use the <code>show_partition_qos --all</code> command.

<pre>
$ show_partition_qos --all
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
cbcb 500 cpu=1406,mem=50359G
cbcb-heng 500
cbcb-interactive 500
class 500 cpu=32,gres/gpu=4,mem=262144M
clip 500 cpu=726,mem=6939G
cml 500 cpu=1226,mem=12116G
cml-cpu 500
cml-director 500
cml-furongh 500
cml-scavenger 500 gres/gpu=24
cml-sfeizi 500
cml-wriva 500
cml-wriva-high 500
csd-h200 500
gamma 500 cpu=906,mem=7675G
mbrc 500 cpu=370,mem=3571G
mc2 500 cpu=330,mem=3201G
oasis 500
quics 500 cpu=458,mem=4710G
scavenger-aarch64_part 500
scavenger_part 500 cpu=11%,gres/gpu=11%,mem=11%
tron 500 cpu=32,gres/gpu=4,mem=262144M
vulcan 500 cpu=1402,mem=12936G
vulcan-ampere 500
vulcan-cpu 500
vulcan-ramani 500
vulcan-scavenger 500
vulcan-scavenger-multi 500
</pre>

'''NOTE''': These QoS cannot be used directly when submitting jobs. Partition QoS limits apply to all jobs running on a given partition, regardless of what job QoS is used.

For example, in the default non-preemption partition (<tt>tron</tt>), you are restricted to 32 total CPU cores, 4 total GPUs, and 256GB total RAM at once across all jobs you have running in the partition.

Lab/group-specific partitions may also have their own user limits, and/or may also have group limits on the total number of resources consumed simultaneously by all users that are using their partition, codified by the line in the output above that matches their lab/group name. Note that the values listed above in the two "TRES" columns are not fixed and may fluctuate per-partition as more resources are added to or removed from each partition.

'''All partitions also only allow a maximum of 500 submitted (running (R) or pending (PD)) jobs per user in the partition simultaneously.''' This is to prevent excess pending jobs causing [https://slurm.schedmd.com/sched_config.html#backfill backfill] issues with the SLURM scheduler.
* If you need to submit more than 500 jobs in batch at once, you can develop and run an "outer submission script" that repeatedly attempts to run an "inner submission script" (your original submission script) to submit jobs in the batch periodically, until all job submissions are successful. The outer submission script should use looping logic to check if you are at the max job limit and should then retry submission after waiting for some time interval.
: An example outer submission script is as follows. In this example, <code>example_inner.sh</code> is your inner submission script and is not an [[SLURM/ArrayJobs | array job]], and you want to run 1000 jobs. If your inner submission script is an array job, adjust the number of jobs accordingly. Array jobs must be of size 500 or less.
<pre>
#!/bin/bash
numjobs=1000
i=0
while [ $i -lt $numjobs ]
do
while [[ "$(sbatch example_inner.sh 2>&1)" =~ "QOSMaxSubmitJobPerUserLimit" ]]
do
echo "Currently at maximum job submissions allowed for this QoS."
echo "Waiting for 5 minutes before trying to submit more jobs."
sleep 300
done
i=$(( $i + 1 ))
echo "Submitted job $i of $numjobs"
done
</pre>

It is suggested that you run the outer submission script in a [[Tmux]] session to keep the terminal window executing it from being interrupted.

= Storage =
All network storage available in Nexus is currently [[NFS]] based, and comes in a few different flavors. Compute nodes also have local scratch storage that can be used.

== Home Directories ==
{{Nfshomes}}

== Scratch Directories ==
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Nexus compute infrastructure:
* Network scratch directories
* Local scratch directories

Please note that [[ClassAccounts | class accounts]] do not have network scratch directories.

=== Network Scratch Directories ===
You are allocated 200GB of scratch space via NFS from <code>/fs/nexus-scratch/<USERNAME></code> where <USERNAME> is your UMIACS username. '''It is not backed up or protected in any way.''' This directory is '''[[Automounter | automounted]]'''; you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access it.

You can view your quota usage by running <code>df -h /fs/nexus-scratch/<USERNAME></code>.

You may request a permanent increase of up to 400GB total space without any faculty approval by [[HelpDesk | contacting staff]]. If you need space beyond 400GB, you will need faculty approval and/or a [[#Project_Allocations | project allocation]] for this. If you choose to increase your scratch space beyond 400GB, the increased space is also subject to the 270 TB days limit mentioned in the project allocation section before we check back in for renewal. For example, if you request 1.4TB total space, you may have this for 270 days (1TB beyond the 400GB permanent increase). The amount increased beyond 400GB will also count against your faculty member's 20TB total storage limit mentioned below.

This file system is available on all submission, data management, and computational nodes within the cluster.

=== Local Scratch Directories ===
Each computational node that you can schedule compute jobs on also has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. and '''are not backed up or protected in any way.''' These directories are almost always more performant than any other storage available to the job as they are mounted from disks directly attached to the compute node. However, you must stage your data within the confines of your job and extract the relevant resultant data elsewhere before the end of your job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month during our [[MonthlyMaintenanceWindow | monthly maintenance windows]]. Please make sure you secure any resultant data you wish to keep from these directories at the end of your job.

== Faculty Allocations ==
Each faculty member can be allocated 1TB of permanent lab space upon request. We can also support grouping these individual allocations together into larger center, lab, or research group allocations if desired by the faculty. Please [[HelpDesk | contact staff]] to inquire.

Lab space storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

== Project Allocations ==
Project allocations are available per user for 270 TB days; you can have a 1TB allocation for up to 270 days, a 3TB allocation for 90 days, etc..

A single faculty member can not have more than 20TB of project allocations across all of their sponsored accounts active simultaneously. Network scratch allocation space increases beyond the 400GB permanent maximum also have the increase count against this limit (i.e., a 1TB network scratch allocation would have 600GB counted towards this limit).

Project storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

The maximum allocation length you can request is 540 days (500GB space) and the maximum storage space you can request is 9TB (30 day length).

To request an allocation, please [[HelpDesk | contact staff]] with the faculty member(s) that the project is under involved in the conversation. Please include the following details:
* Project Name (short)
* Description
* Size (1TB, 2TB, etc.)
* Length in days (270 days, 135 days, etc.)
* Other user(s) that need to access the allocation, if any

These allocations are available via <code>/fs/nexus-projects/<project name></code>. '''Renewal is not guaranteed to be available due to limits on the amount of total storage.''' Near the end of the allocation period, staff will contact you and ask if you are still in need of the storage allocation. If renewal is available, you can renew for up to another 270 TB days with reapproval from the original faculty approver.
* If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation.
* If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible.
** If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible.
** If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

== Datasets ==
We have read-only dataset storage available at <code>/fs/nexus-datasets</code>. If there are datasets that you would like to see curated and made available, please see [[Datasets | this page]].

The list of Nexus datasets we currently host can be viewed [https://info.umiacs.umd.edu/datasets/list/?q=Nexus here].

Nexus/Submission Node Policy

2026-03-05T21:19:04Z

Seide: Created page with "Submission nodes are intended for users to submit jobs to the Nexus cluster. User interactivity is critical here, and while some of these submission nodes have a decent amount of cores and memory, highly-intensive computational jobs should not be run on these nodes. We have minimal resource restrictions in place at the moment, but we may explore further CPU/memory restrictions in the future. For the time being, be a good neighbor! In general, we expect that users on the..."

Submission nodes are intended for users to submit jobs to the Nexus cluster. User interactivity is critical here, and while some of these submission nodes have a decent amount of cores and memory, highly-intensive computational jobs should not be run on these nodes. We have minimal resource restrictions in place at the moment, but we may explore further CPU/memory restrictions in the future. For the time being, be a good neighbor!

In general, we expect that users on these nodes do not use more than 1-2GB of memory at a time and do not sustain >=100% CPU usage (at least 1 core fully utilized) for extended periods of time. Utilizing more than 1 core is fine for bursty workloads, but sustained high CPU utilization can lead to issues with interactivity, and these issues can be drastically amplified if there is also memory contention on the node (swapping memory requires CPU time).

=Examples=

==Appropriate submission node uses==

* Job submission
* SSH jump host for user workstations
* Basic code editing in editors with minimal extensions (e.g. emacs, nano, vim, neovim without an LSP).

==Appropriate submission node uses, with caveats==

These kinds of workloads can be run on submission nodes, but if they are incorrectly configured, they can affect interactivity of the nodes. If we notice repeated behavior like this, we may reach out to you to ask about your environment and suggest ways to lower resource utilization.

* IDEs can be fine, but keep in mind which extensions you install. Some extensions/behaviors may use more memory/CPU than you realize.
** We have some [[VS_Code|guidance for VS Code and its forks]].
* For smaller projects, code compilation can be fine. If a project takes longer than 30 seconds to compile, please ensure compilation is limited to a single thread, or consider compiling the code in a Slurm job on one of the dedicated CPU nodes (we suggest using the scavenger partition).
** Depending on a project's file structure, code compilation can be highly I/O dependent, which can also impact user interactivity.
* Setting up environments with user package managers (e.g. pip, npm, conda/mamba)
** The resource usage of package managers mostly depends on the specific packages being installed. Complex environments can take a while with package managers like conda/mamba due to dependency resolution. Other packages aren't actually binary releases but will instead compile projects transparently to the user. For these reasons, we suggest that non-trivial environments should be set up in a Slurm job (we suggest using the scavenger partition).
* Running lightweight programs to interact with Slurm jobs currently running on the cluster.

==Inappropriate submission node uses==

If tech staff notices processes with the following behavior and node interactivity is affected, <b>we may kill these processes</b>. If a user repeatedly causes issues, further action may be taken on their account.

* Compiling large projects using all threads on a submission node.
* Running nontrivial computation requiring significant CPU and/or memory resources.

=Advice for monitoring/reducing CPU/memory utilization=

Here is some general guidance for tracking your resource utilization on a node, as well as some practices you can follow to slightly reduce CPU usage of certain tasks.

* `top`
** By default, `top` sorts by %CPU, but you can sort by %MEM by pressing "M" (Shift + "m").
** You can limit it to show only processes from your user by starting it with `-t $USER`
** VIRT memory usage doesn't mean too much, as oftentimes programs will request way more memory than they actually need, and the kernel may not actually give them this memory until it is actually required. RSS memory is the actual amount of memory used by a process at a given point. If there is no suffix, the reported value is in KB.
* /sys/fs/cgroup/memory/user.slice/user-$UID_NUMBER.slice/memory.usage_in_bytes
** This is the current combined memory usage of all processes running under your user.

VS Code

2026-03-05T20:56:09Z

Seide: /* Keep data separate from code where possible */

[https://code.visualstudio.com/ Visual Studio (VS) Code] is a multi-platform source-code editor developed by Microsoft. It can be used with a variety of programming languages and can have its base functionality extended through extensions available via its [https://marketplace.visualstudio.com/vscode marketplace]. The base editor itself is free, but some extensions may require paid subscriptions.

There are also forks/open source alternatives of VS Code, such as [https://www.cursor.com Cursor] or [https://vscodium.com/ VSCodium]. The same general principals below apply to these editors as well.

==Cluster Usage==
It can be convenient when using our [[SLURM]] computing clusters to open a connection to a [https://code.visualstudio.com/docs/remote/vscode-server VS Code Server] session on the submission node(s) that you have access to via VS Code's [https://code.visualstudio.com/docs/remote/ssh Remote - SSH extension]. Steps to set this up can be found [https://code.visualstudio.com/docs/remote/ssh#_installation here]. Use your UMIACS username and the [https://en.wikipedia.org/wiki/Fully_qualified_domain_name fully qualified domain name (FQDN)] of the submission node you want to connect to. Password authentication does not provide an option to save your password, so for convenience, you may want to set up an [[SSH/Keys | SSH key]].

For a list of active issues with the Remote - SSH extension, please see Microsoft's GitHub tracker [https://github.com/Microsoft/vscode-remote-release/labels/ssh here].

===Best Practices===
Because of the multi-tenant nature of our submission nodes, using the Remote - SSH extension to connect to a VS Code Server running on a submission node can have adverse affects on other users simultaneously using the submission nodes if not properly managed. The following is a list of "best practices" that we ask you please follow to minimize the chance of impacting others on submission nodes. We have compiled the list through observation as well as through discussion with users.

====Use only as a code editor====
VS Code can perform many common file management operations, such as moving, copying, or deleting files or directories. However, the overhead of providing progress updates / status messages through VS Code's UI can cause load spikes that result in other system-level and user processes on the submission node slowing down. This is especially the case if operating on thousands to millions of small image or video files. In extreme cases, it can seem as though the submission node is wholly unresponsive.

The best way to combat this is to use VS Code as just a code editor as strictly as you can (meaning creating new and/or editing existing source code files only). Use standard command-line programs such as <code>mv</code>/<code>cp</code>/<code>rm</code> in VS Code's terminal window or a separate application's terminal window for local file management instead, or use the techniques advertised [[LocalDataTransfer | here]] for larger data transfers between nodes.

====Do not open large files====
Files that you open in VS Code's text editor on a remote machine through the Remote - SSH extension are loaded entirely into RAM on the remote machine. Because of the persistence of VS Code Server processes, opening several unique files may result in them all remaining in RAM for extended periods of time even if you no longer have tabs open for them in the Tab Bar.

As such, please refrain from opening large files (more than a few MB) in VS Code itself on submission nodes. Use a lighter-weight command-line based text editor such as [https://www.vim.org vim] or otherwise in VS Code's terminal window or a separate application's terminal window for local file editing instead.

If you absolutely need to open one or more large files for a very brief period of time on a submission node, please clean up your server session on that node (see below section) as soon as possible when done.

====Search tuning====
VS Code comes bundled with Ripgrep as the default search program. Searching in workspaces with very large numbers of small files can result in degraded performance due to NFS overhead amplified by small file I/O. The following tips can ensure that searches do not significantly impact performance.

=====Keep data separate from code where possible=====
Make sure that datasets are kept outside of your VS Code workspace so that searches do not look through these directories. Alternatively, by default VS Code will not search files listed in a .ignore, .gitignore, or .rgignore file, so adding directories with datasets to one of these files is another option. Any directory with only binary data is a good candidate to add to this file.

=====Limit maximum number of threads for searches=====
By default, VS Code will automatically determine the number of threads to use when searching files. You can force a maximum number of threads under "Settings->Features->Search->Ripgrep: Max Threads".

====Clean up when done====
If you are done using VS Code for an extended period of time (several hours / overnight), consider manually killing your server session on the submission node when exiting. By default, VS Code Server will keep the processes used by your session active for a few hours after you disconnect, keeping the VS Code Server processes on the submission node you were using alive, despite no longer actively being used.

This can be done by opening the [https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette Command Palette] (Ctrl+Shift+P) and searching "Kill VS Code Server on Host...".

: [[File:VSCode-killremote1.png]]

Select the submission node you were using and click on it.

: [[File:VSCode-killremote2.png]]

Wait a few seconds and you should get a message confirming the command went through.

: [[File:VSCode-killremote3.png]]

The next time you connect to the same submission node, you will have a fresh VS Code session on it.

==Common Issues==
If you are having trouble [[SSH]]'ing to a submission node with VS Code, check if your home directory is full.

# Connect to one of your Nexus submission nodes via [[SSH]] '''not''' using VS Code.
# Navigate to your home directory. <pre>cd ~</pre>
# Check if it is full. <pre>df -h .</pre>
# If it is full (Avail 0), delete some files to free up at least ~100MB of space. <pre>Filesystem Size Used Avail Use% Mounted on
data.isilon.umiacs.umd.edu:/ifs/umiacs/homes 30G 30G 0 100% /fs/nfshomes</pre>
# Try connecting with VS Code again.

VS Code

2026-03-05T19:50:13Z

Seide: /* Best Practices */ Add information about searching

[https://code.visualstudio.com/ Visual Studio (VS) Code] is a multi-platform source-code editor developed by Microsoft. It can be used with a variety of programming languages and can have its base functionality extended through extensions available via its [https://marketplace.visualstudio.com/vscode marketplace]. The base editor itself is free, but some extensions may require paid subscriptions.

There are also forks/open source alternatives of VS Code, such as [https://www.cursor.com Cursor] or [https://vscodium.com/ VSCodium]. The same general principals below apply to these editors as well.

==Cluster Usage==
It can be convenient when using our [[SLURM]] computing clusters to open a connection to a [https://code.visualstudio.com/docs/remote/vscode-server VS Code Server] session on the submission node(s) that you have access to via VS Code's [https://code.visualstudio.com/docs/remote/ssh Remote - SSH extension]. Steps to set this up can be found [https://code.visualstudio.com/docs/remote/ssh#_installation here]. Use your UMIACS username and the [https://en.wikipedia.org/wiki/Fully_qualified_domain_name fully qualified domain name (FQDN)] of the submission node you want to connect to. Password authentication does not provide an option to save your password, so for convenience, you may want to set up an [[SSH/Keys | SSH key]].

For a list of active issues with the Remote - SSH extension, please see Microsoft's GitHub tracker [https://github.com/Microsoft/vscode-remote-release/labels/ssh here].

===Best Practices===
Because of the multi-tenant nature of our submission nodes, using the Remote - SSH extension to connect to a VS Code Server running on a submission node can have adverse affects on other users simultaneously using the submission nodes if not properly managed. The following is a list of "best practices" that we ask you please follow to minimize the chance of impacting others on submission nodes. We have compiled the list through observation as well as through discussion with users.

====Use only as a code editor====
VS Code can perform many common file management operations, such as moving, copying, or deleting files or directories. However, the overhead of providing progress updates / status messages through VS Code's UI can cause load spikes that result in other system-level and user processes on the submission node slowing down. This is especially the case if operating on thousands to millions of small image or video files. In extreme cases, it can seem as though the submission node is wholly unresponsive.

The best way to combat this is to use VS Code as just a code editor as strictly as you can (meaning creating new and/or editing existing source code files only). Use standard command-line programs such as <code>mv</code>/<code>cp</code>/<code>rm</code> in VS Code's terminal window or a separate application's terminal window for local file management instead, or use the techniques advertised [[LocalDataTransfer | here]] for larger data transfers between nodes.

====Do not open large files====
Files that you open in VS Code's text editor on a remote machine through the Remote - SSH extension are loaded entirely into RAM on the remote machine. Because of the persistence of VS Code Server processes, opening several unique files may result in them all remaining in RAM for extended periods of time even if you no longer have tabs open for them in the Tab Bar.

As such, please refrain from opening large files (more than a few MB) in VS Code itself on submission nodes. Use a lighter-weight command-line based text editor such as [https://www.vim.org vim] or otherwise in VS Code's terminal window or a separate application's terminal window for local file editing instead.

If you absolutely need to open one or more large files for a very brief period of time on a submission node, please clean up your server session on that node (see below section) as soon as possible when done.

====Search tuning====
VS Code comes bundled with Ripgrep as the default search program. Searching in workspaces with very large numbers of small files can result in degraded performance due to NFS overhead amplified by small file I/O. The following tips can ensure that searches do not significantly impact performance.

=====Keep data separate from code where possible=====
Make sure that datasets are kept outside of your VS Code workspace so that searches do not look through these directories. Alternatively, by default VS Code will not search files listed in a .ignore or .gitignore file, so adding directories with datasets to one of these files is another option. Any directory with only binary data is a good candidate to add to this file.

=====Limit maximum number of threads for searches=====
By default, VS Code will automatically determine the number of threads to use when searching files. You can force a maximum number of threads under "Settings->Features->Search->Ripgrep: Max Threads".

====Clean up when done====
If you are done using VS Code for an extended period of time (several hours / overnight), consider manually killing your server session on the submission node when exiting. By default, VS Code Server will keep the processes used by your session active for a few hours after you disconnect, keeping the VS Code Server processes on the submission node you were using alive, despite no longer actively being used.

This can be done by opening the [https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette Command Palette] (Ctrl+Shift+P) and searching "Kill VS Code Server on Host...".

: [[File:VSCode-killremote1.png]]

Select the submission node you were using and click on it.

: [[File:VSCode-killremote2.png]]

Wait a few seconds and you should get a message confirming the command went through.

: [[File:VSCode-killremote3.png]]

The next time you connect to the same submission node, you will have a fresh VS Code session on it.

==Common Issues==
If you are having trouble [[SSH]]'ing to a submission node with VS Code, check if your home directory is full.

# Connect to one of your Nexus submission nodes via [[SSH]] '''not''' using VS Code.
# Navigate to your home directory. <pre>cd ~</pre>
# Check if it is full. <pre>df -h .</pre>
# If it is full (Avail 0), delete some files to free up at least ~100MB of space. <pre>Filesystem Size Used Avail Use% Mounted on
data.isilon.umiacs.umd.edu:/ifs/umiacs/homes 30G 30G 0 100% /fs/nfshomes</pre>
# Try connecting with VS Code again.

Nexus

2025-11-21T16:14:52Z

Seide: /* Partition QoS */ Update for scavenger/scavenger-aarch64 relative QoS configuration

The Nexus is the combined scheduler of resources in UMIACS. The resource manager for Nexus is [[SLURM]]. Resources are arranged into partitions where users are able to schedule computational jobs. Users are arranged into a number of SLURM accounts based on faculty, lab, or center investments.

= Getting Started =
All accounts in UMIACS are sponsored. If you don't already have a UMIACS account, please see [[Accounts]] for information on getting one. You need a full UMIACS account (not a [[Accounts/Collaborator | collaborator account]]) in order to access Nexus.

== Access ==
Your access to submission nodes (alternatively called login nodes) for Nexus computational resources is determined by your account sponsor's department, center, or lab affiliation. You can log into the [https://intranet.umiacs.umd.edu/directory/cr/ UMIACS Directory CR application] and select the Computational Resource (CR) in the list that has the prefix <code>nexus</code>. The Hosts section lists your available submission nodes - generally a pair of nodes of the format <tt>nexus<department, lab, or center abbreviation>[00,01]</tt>, e.g., <tt>nexusgroup00</tt> and <tt>nexusgroup01</tt>.

Once you have identified your submission nodes, you can [[SSH]] into them [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0016076 after connecting to UMD's GlobalProtect VPN]. From there, you are able to submit to the cluster via our [[SLURM]] workload manager. You need to make sure that your submitted jobs have the correct account, partition, and qos.

== Jobs ==
[[SLURM]] jobs are [[SLURM/JobSubmission | submitted]] by either <code>srun</code> or <code>sbatch</code> depending if you are doing an interactive job or batch job, respectively. You need to provide the where/how/who to run the job and specify the resources you need to run with.

For the who/where/how, you may be required to specify <code>--account</code>, <code>--partition</code>, and/or <code>--qos</code> (respectively) to be able to adequately submit jobs to the Nexus.

For resources, you may need to specify <code>--time</code> for time, <code>--cpus-per-task</code> for CPUs, <code>--mem</code> for RAM, and <code>--gres=gpu</code> for GPUs in your submission arguments to meet your requirements. There are defaults for all four; if you don't specify something, you will get the default value for that resource, which is minimal (e.g., by default, NO GPUs are included if you do not specify <code>--gres=gpu</code>). For more information about submission flags for GPU resources, see [[SLURM/JobSubmission#Requesting_GPUs | here]]. You may also use <code>--ntasks</code> to specify the number of parallel processes to run, with each task having its own set of the resources specified above. You can run <code>man srun</code> on your submission node for a complete list of available submission arguments.

For a list of available GPU types on Nexus and their specs, please see [[Nexus/GPUs]].

For details on how the network for Nexus is architected, please see [[Nexus/Network]]. This can be important if you wish to optimize performance of your jobs.

=== Interactive ===
Once logged into a submission node, you can run simple interactive jobs. If your session is interrupted from the submission node, the job will be killed. As such, we encourage use of a terminal multiplexer such as [[Tmux]].

<pre>
$ srun --pty --cpus-per-task=4 --mem=2gb --gres=gpu:1 bash
srun: Job account was unset; set to user default of 'nexus'
srun: Job partition was unset; set to cluster default of 'tron'
srun: Job QoS was unset; set to association default of 'default'
srun: Job time limit was unset; set to partition default of 60 minutes
srun: job 1 queued and waiting for resources
srun: job 1 has been allocated resources
$ hostname
tron62.umiacs.umd.edu
$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-daad6a04-a2ce-1183-ce53-b267048f750a)
</pre>

=== Batch ===
Batch jobs are scheduled with a script file with an optional ability to embed job scheduling parameters via variables that are defined by <code>#SBATCH</code> lines at the top of the file. You can find some examples in our [[SLURM/JobSubmission]] documentation.

= Partitions =
The SLURM resource manager uses partitions to act as job queues which can restrict size, time and user limits. The Nexus has a number of different partitions of resources. Different Centers, Labs, and Faculty are able to invest in computational resources that are restricted to approved users through these partitions.

'''Partitions usable by all non-[[ClassAccounts |class account]] users:'''
* [[Nexus/Tron]] - Pool of resources available to all non-class accounts sponsored by either UMIACS or CSD faculty.
* Scavenger - [https://slurm.schedmd.com/preempt.html Preemption] partition that contains [https://en.wikipedia.org/wiki/X86-64 x86_64] architecture nodes from multiple other partitions. More resources are available to schedule simultaneously than in other partitions, however jobs are subject to preemption rules. You are responsible for ensuring your jobs handle this preemption correctly. The SLURM scheduler will simply restart a preempted job with the same submission arguments when it is available to run again. For an overview of things you can check within scripts to determine if your job was preempted/resumed, see [[SLURM/Preemption]].
* Scavenger (aarch64) - Preemption partition identical in design to <tt>scavenger</tt>, but only contains [https://en.wikipedia.org/wiki/AArch64 aarch64] architecture nodes.

'''Partitions usable by [[ClassAccounts]]:'''
* [[ClassAccounts | Class]] - Pool of resources available to class accounts sponsored by either UMIACS or CSD faculty.

'''Partitions usable by specific lab/center users:'''
* [[Nexus/CBCB]] - CBCB lab pool available for CBCB lab members.
* [[Nexus/CLIP]] - CLIP lab pool available for CLIP lab members.
* [[Nexus/CML]] - CML lab pool available for CML lab members.
* [[Nexus/GAMMA]] - GAMMA lab pool available for GAMMA lab members.
* [[Nexus/MBRC]] - MBRC lab pool available for MBRC lab members.
* [[Nexus/MC2]] - MC2 lab pool available for MC2 lab members.
* [[Nexus/QuICS]] - QuICS lab pool available for QuICS lab members.
* [[Nexus/Vulcan]] - Vulcan lab pool available for Vulcan lab members.

You can view the partitions that you have access to by using the <code>show_partitions</code> command. By default, the command will show only the partitions that are available to you.

<pre>
$ show_partitions
Name AllowAccounts AllowQos MaxNodes Nodes
------------------------ ----------------------- ------------------------------ ----------- ----------------------------
scavenger scavenger scavenger UNLIMITED brigid[16-19]
cbcb[00-29]
clip[00-13]
cml[00,02-13,15-28,30-33]
cmlcpu[00-04,06-07]
gammagpu[00-21]
legacy[00-11,13-28,30-36]
legacygpu[00-07]
quics00
tron[00-44,46-69]
vulcan[00-45]
------------------------------------------------------------------------------------------------------------------------
scavenger-aarch64 scavenger scavenger-aarch64 UNLIMITED oasis[00-39]
------------------------------------------------------------------------------------------------------------------------
tron nexus default UNLIMITED tron[00-44,46-69]
high
medium
</pre>

If you want to see information for all of the partitions, you can use the <code>show_partitions --all</code> command.

<pre>
$ show_partitions --all
Name AllowAccounts AllowQos MaxNodes Nodes
------------------------ ----------------------- ------------------------------ ----------- ----------------------------
cbcb cbcb default UNLIMITED cbcb[00-20,22-29]
medium legacy[00-11,13-28,30-36]
high
huge-long
highmem
------------------------------------------------------------------------------------------------------------------------
cbcb-heng cbcb-heng default UNLIMITED cbcb[26-29]
medium
high
huge-long
highmem
------------------------------------------------------------------------------------------------------------------------
cbcb-interactive cbcb interactive UNLIMITED cbcb21
...
</pre>

= Quality of Service (QoS) =
SLURM uses Quality of Service (QoS) both to provide limits on job sizes (termed by us as "job QoS") as well as to limit resources used by all jobs running in a partition, either per user or per group (termed by us as "partition QoS").

=== Job QoS ===
Job QoS are used to provide limits on the size of job that you can run. You should try to allocate only the resources your job actually needs, as resources that each of your jobs schedules are counted against your [[SLURM/Priority#Fair-share | fair-share priority]] in the future.
* default - Default job QoS. Limited to 4 CPU cores, 1 GPU, and 32GB RAM per job. The maximum wall time per job is 3 days.
* medium - Limited to 8 CPU cores, 2 GPUs, and 64GB RAM per job. The maximum wall time per job is 2 days.
* high - Limited to 16 CPU cores, 4 GPUs, and 128GB RAM per job. The maximum wall time per job is 1 day.
* scavenger - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. 576 total CPU cores, 72 total GPUs, and 2304GB total RAM are permitted simultaneously across all of your jobs running with this job QoS. This job QoS is paired 1-1 with the scavenger partition. To use this job QoS, include <code>--partition=scavenger</code> and <code>--account=scavenger</code> in your submission arguments. Do not include any job QoS argument other than <code>--qos=scavenger</code> (optional) or submission will fail.
* scavenger-aarch64 - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. 1600 total CPU cores and 281140MB total RAM are permitted simultaneously across all of your jobs running with this job QoS. This job QoS is paired 1-1 with the scavenger-aarch64 partition. To use this job QoS, include <code>--partition=scavenger-aarch64</code>, <code>--account=scavenger</code>, and <code>--qos=scavenger-aarch64</code> in your submission arguments.

You can display these job QoS from the command line using the <code>show_qos</code> command. By default, the command will only show job QoS that you can access. The above five job QoS are the ones that everyone can access.

<pre>
$ show_qos
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
scavenger 3-00:00:00 cpu=576,gres/gpu=72,mem=2304G
scavenger-aarch64 3-00:00:00 cpu=1600,mem=281140M
</pre>

If you want to see all job QoS, including those that you do not have access to, you can use the <code>show_qos --all</code> command.

<pre>
$ show_qos --all
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
cml-cpu 7-00:00:00 8
cml-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
cml-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
cml-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
cml-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
cml-scavenger 3-00:00:00 gres/gpu=24
cml-very_high 1-12:00:00 cpu=32,gres/gpu=8,mem=256G 8 gres/gpu=12
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
gamma-huge-long 10-00:00:00 cpu=32,gres/gpu=16,mem=256G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
highmem 21-00:00:00 cpu=128,mem=2T
huge-long 10-00:00:00 cpu=32,gres/gpu=8,mem=256G
interactive 12:00:00 cpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
oasis-exempt 10-00:00:00 cpu=160,mem=28114M
scavenger 3-00:00:00 cpu=576,gres/gpu=72,mem=2304G
scavenger-aarch64 3-00:00:00 cpu=1600,mem=281140M
tron-exempt cpu=32,gres/gpu=8,mem=383030M 1
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-exempt 7-00:00:00 cpu=32,gres/gpu=8,mem=256G 2
vulcan-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
vulcan-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-sailon 3-00:00:00 cpu=32,gres/gpu=8,mem=256G gres/gpu=48
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
vulcan-scavenger-mu+ 3-00:00:00 cpu=288,gres/gpu=72,mem=1152G
</pre>

You are able to submit to any partition that is listed in the <code>show_partitions</code> command. If you need to use an account other than the default account <tt>nexus</tt>, you will need to specify it via the <code>--account</code> submission argument.

=== Partition QoS ===
Partition QoS are used to limit resources used by all jobs running in a partition, either per user (MaxTRESPU) or per group (GrpTRES).

To view partition QoS, use the <code>show_partition_qos</code> command.

<pre>
$ show_partition_qos
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
scavenger-aarch64_part 500 cpu=11%,mem=11%
scavenger_part 500 cpu=11%,gres/gpu=11%,mem=11%
tron 500 cpu=32,gres/gpu=4,mem=262144M
</pre>

Some partitions QoS have relative TRES limits based on the current hardware in a given partition, represented with percentages. Of the broadly-available partitions, this applies to scavenger and scavenger-aarch64. To see the current actual TRES limits of these QoS, you can use the <code>-r/--real</code> argument.

<pre>
[seide@nexusstaff00 ~]$ show_partition_qos -r
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
scavenger-aarch64_part 500 cpu=651,mem=114423M
scavenger_part 500 cpu=888,gres/gpu=140,mem=12574G
tron 500 cpu=32,gres/gpu=4,mem=262144M
</pre>

If you want to see all partition QoS, including those that you do not have access to, you can use the <code>-a/--all</code> argument.

<pre>
$ show_partition_qos -a
Name MaxSubmitPU MaxTRESPU GrpTRES
------------------------- ----------- -------------------------------- --------------------
cbcb 500 cpu=1406,mem=50359G
cbcb-heng 500
cbcb-interactive 500
class 500 cpu=32,gres/gpu=4,mem=262144M
clip 500 cpu=726,mem=6939G
cml 500 cpu=1226,mem=12116G
cml-cpu 500
cml-director 500
cml-furongh 500
cml-scavenger 500 gres/gpu=24
cml-sfeizi 500
cml-wriva 500
cml-wriva-high 500
csd-h200 500
gamma 500 cpu=906,mem=7675G
mbrc 500 cpu=370,mem=3571G
mc2 500 cpu=330,mem=3201G
oasis 500
quics 500 cpu=458,mem=4710G
scavenger-aarch64_part 500 cpu=11%,mem=11%
scavenger_part 500 cpu=11%,gres/gpu=11%,mem=11%
tron 500 cpu=32,gres/gpu=4,mem=262144M
vulcan 500 cpu=1402,mem=12936G
vulcan-ampere 500
vulcan-cpu 500
vulcan-ramani 500
vulcan-scavenger 500
vulcan-scavenger-multi 500
</pre>

'''NOTE''': These QoS cannot be used directly when submitting jobs. Partition QoS limits apply to all jobs running on a given partition, regardless of what job QoS is used.

For example, in the default non-preemption partition (<tt>tron</tt>), you are restricted to 32 total CPU cores, 4 total GPUs, and 256GB total RAM at once across all jobs you have running in the partition.

Lab/group-specific partitions may also have their own user limits, and/or may also have group limits on the total number of resources consumed simultaneously by all users that are using their partition, codified by the line in the output above that matches their lab/group name. Note that the values listed above in the two "TRES" columns are not fixed and may fluctuate per-partition as more resources are added to or removed from each partition.

'''All partitions also only allow a maximum of 500 submitted (running (R) or pending (PD)) jobs per user in the partition simultaneously.''' This is to prevent excess pending jobs causing [https://slurm.schedmd.com/sched_config.html#backfill backfill] issues with the SLURM scheduler.
* If you need to submit more than 500 jobs in batch at once, you can develop and run an "outer submission script" that repeatedly attempts to run an "inner submission script" (your original submission script) to submit jobs in the batch periodically, until all job submissions are successful. The outer submission script should use looping logic to check if you are at the max job limit and should then retry submission after waiting for some time interval.
: An example outer submission script is as follows. In this example, <code>example_inner.sh</code> is your inner submission script and is not an [[SLURM/ArrayJobs | array job]], and you want to run 1000 jobs. If your inner submission script is an array job, adjust the number of jobs accordingly. Array jobs must be of size 500 or less.
<pre>
#!/bin/bash
numjobs=1000
i=0
while [ $i -lt $numjobs ]
do
while [[ "$(sbatch example_inner.sh 2>&1)" =~ "QOSMaxSubmitJobPerUserLimit" ]]
do
echo "Currently at maximum job submissions allowed."
echo "Waiting for 5 minutes before trying to submit more jobs."
sleep 300
done
i=$(( $i + 1 ))
echo "Submitted job $i of $numjobs"
done
</pre>

It is suggested that you run the outer submission script in a [[Tmux]] session to keep the terminal window executing it from being interrupted.

= Storage =
All network storage available in Nexus is currently [[NFS]] based, and comes in a few different flavors. Compute nodes also have local scratch storage that can be used.

== Home Directories ==
{{Nfshomes}}

== Scratch Directories ==
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Nexus compute infrastructure:
* Network scratch directories
* Local scratch directories

Please note that [[ClassAccounts | class accounts]] do not have network scratch directories.

=== Network Scratch Directories ===
You are allocated 200GB of scratch space via NFS from <code>/fs/nexus-scratch/<USERNAME></code> where <USERNAME> is your UMIACS username. '''It is not backed up or protected in any way.''' This directory is '''[[Automounter | automounted]]'''; you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access it.

You can view your quota usage by running <code>df -h /fs/nexus-scratch/<USERNAME></code>.

You may request a permanent increase of up to 400GB total space without any faculty approval by [[HelpDesk | contacting staff]]. If you need space beyond 400GB, you will need faculty approval and/or a [[#Project_Allocations | project allocation]] for this. If you choose to increase your scratch space beyond 400GB, the increased space is also subject to the 270 TB days limit mentioned in the project allocation section before we check back in for renewal. For example, if you request 1.4TB total space, you may have this for 270 days (1TB beyond the 400GB permanent increase). The amount increased beyond 400GB will also count against your faculty member's 20TB total storage limit mentioned below.

This file system is available on all submission, data management, and computational nodes within the cluster.

=== Local Scratch Directories ===
Each computational node that you can schedule compute jobs on also has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. and '''are not backed up or protected in any way.''' These directories are almost always more performant than any other storage available to the job as they are mounted from disks directly attached to the compute node. However, you must stage your data within the confines of your job and extract the relevant resultant data elsewhere before the end of your job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month during our [[MonthlyMaintenanceWindow | monthly maintenance windows]]. Please make sure you secure any resultant data you wish to keep from these directories at the end of your job.

== Faculty Allocations ==
Each faculty member can be allocated 1TB of permanent lab space upon request. We can also support grouping these individual allocations together into larger center, lab, or research group allocations if desired by the faculty. Please [[HelpDesk | contact staff]] to inquire.

Lab space storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

== Project Allocations ==
Project allocations are available per user for 270 TB days; you can have a 1TB allocation for up to 270 days, a 3TB allocation for 90 days, etc..

A single faculty member can not have more than 20TB of project allocations across all of their sponsored accounts active simultaneously. Network scratch allocation space increases beyond the 400GB permanent maximum also have the increase count against this limit (i.e., a 1TB network scratch allocation would have 600GB counted towards this limit).

Project storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

The maximum allocation length you can request is 540 days (500GB space) and the maximum storage space you can request is 9TB (30 day length).

To request an allocation, please [[HelpDesk | contact staff]] with the faculty member(s) that the project is under involved in the conversation. Please include the following details:
* Project Name (short)
* Description
* Size (1TB, 2TB, etc.)
* Length in days (270 days, 135 days, etc.)
* Other user(s) that need to access the allocation, if any

These allocations are available via <code>/fs/nexus-projects/<project name></code>. '''Renewal is not guaranteed to be available due to limits on the amount of total storage.''' Near the end of the allocation period, staff will contact you and ask if you are still in need of the storage allocation. If renewal is available, you can renew for up to another 270 TB days with reapproval from the original faculty approver.
* If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation.
* If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible.
** If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible.
** If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

== Datasets ==
We have read-only dataset storage available at <code>/fs/nexus-datasets</code>. If there are datasets that you would like to see curated and made available, please see [[Datasets | this page]].

The list of Nexus datasets we currently host can be viewed [https://info.umiacs.umd.edu/datasets/list/?q=Nexus here].

Modules

2024-09-12T17:40:30Z

Seide: RHEL9 desktop node

=GNU Modules=
Many large institutions use the concept of Modules to load software into user environments. They provide a way to add and remove environmental variables that provide access to UMIACS' large set of software that we offer on our Red Hat Enterprise Linux ([[RedHat]]) and [[Ubuntu]] platforms. They work by customizing your shell environment which is done automatically for the two major shell families (bash/sh (default shell) and tcsh/csh). If you use an alternate shell, please look to source the appropriate script for your shell in <tt>/usr/share/Modules/init</tt>.

Initially your module environment is empty, though included in your module path are local operating system specific modules, locally built software modules, and binary software modules (Matlab, Intel Compiler, etc.).

==Available Software==
To see if a piece of software is available, use the <tt>module avail</tt> command. This can be given a trailing prefix on the command line to search a subset of the available software.

<pre>
[username@nexusstaff00 ~]$ module avail cuda
--------------------------- /opt/common/.modulefiles ---------------------------
cuda/3.2.16 cuda/6.5.14 cuda/9.0.176 cuda/11.0.3 cuda/11.8.0
cuda/4.2.9 cuda/7.0.28 cuda/9.1.85 cuda/11.1.1 cuda/12.0.1
cuda/5.0.35 cuda/7.5.18 cuda/9.2.148 cuda/11.2.2 cuda/12.1.1
cuda/5.5.11 cuda/8.0.27rc2 cuda/10.0.130 cuda/11.3.1
cuda/5.5.22 cuda/8.0.44 cuda/10.1.243 cuda/11.4.4
cuda/6.0.37 cuda/8.0.61 cuda/10.2.89 cuda/11.7.0
</pre>

<pre>
[username@nexusstaff00 ~]$ module avail Python3
------------------------- /opt/local/stow/.modulefiles -------------------------
Python3/3.5.2 Python3/3.8.2 Python3/3.8.15 Python3/3.9.16
Python3/3.6.15 Python3/3.8.10 Python3/3.9.5 Python3/3.10.4
Python3/3.7.13 Python3/3.8.12 Python3/3.9.6 Python3/3.10.10
Python3/3.7.16 Python3/3.8.13 Python3/3.9.12 Python3/3.11.2
</pre>

Some pieces of software may have default versions that are loaded if no version is explicitly specified, indicated by <code>(default)</code> coming after their name/version in the output of <tt>module avail</tt>. If a piece of software does not have any default version and you load it without specifying a version, you will load the most recent version of it.

==Adding Modules into your Environment==
You can simply add a module into your environment by using the <tt>module add <module></tt> command.

<pre>
[username@nexusstaff00 ~]$ module add cuda
</pre>

You can also specify a specific version of the software when we have multiple ones available.

<pre>
[username@nexusstaff00 ~]$ module add Python3/3.10.10
</pre>

==Listing Modules==
You can list the currently loaded modules in your environment by using the '''list''' command.

<pre>
[username@nexusstaff00 ~]$ module list
Currently Loaded Modulefiles:
1) Python3/3.10.10 2) cuda/12.1.1
</pre>

==Showing a Module==
You can show what the module is going to add to your environment (and the dependencies that will be added) with the '''show''' command.

<pre>
[username@nexusstaff00 ~]$ module show cuda
-------------------------------------------------------------------
/opt/common/.modulefiles/cuda/12.1.1:

conflict cuda/12.0.1 cuda/11.8.0 cuda/11.7.0 cuda/11.4.4 cuda/11.2.2 cuda/11.1.1 cuda/11.0.3 cuda/10.2.89 cuda/10.1.243 cuda/10.0.130 cuda/9.2.148 cuda/9.1.85 cuda/9.0.176 cuda/8.0.61 cuda/8.0.44 cuda/8.0.27rc2 cuda/7.5.18 cuda/7.0.28 cuda/6.5.14 cuda/6.0.37 cuda/5.5.22 cuda/5.5.11 cuda/5.0.35 cuda/4.2.9 cuda/3.2.16
prepend-path PATH /opt/common/cuda/cuda-12.1.1/bin
prepend-path LD_LIBRARY_PATH /opt/common/cuda/cuda-12.1.1/lib64
prepend-path LD_RUN_PATH /usr/lib64/nvidia:/usr/lib/nvidia:/opt/common/cuda/cuda-12.1.1/lib64:/opt/common/cuda/cuda-12.1.1/lib
prepend-path LIBRARY_PATH /usr/lib64/nvidia:/usr/lib/nvidia:/opt/common/cuda/cuda-12.1.1/lib64:/opt/common/cuda/cuda-12.1.1/lib
prepend-path CPATH /opt/common/cuda/cuda-12.1.1/include
prepend-path PKG_CONFIG_PATH /opt/common/cuda/cuda-12.1.1/pkgconfig
setenv CUDA_HOME /opt/common/cuda/cuda-12.1.1
-------------------------------------------------------------------
</pre>

==Removing Modules in your Environment==
If you want to remove a module because it conflicts or you want to clean up your environment you can by using the <tt>module rm <module></tt> command.

==Using Modules in Scripts==
To use modules within a shell script or interpreted language you will need to load a file from <tt>/usr/share/Modules/init</tt> into your program.

===Bash===
<pre>
. /usr/share/Modules/init/bash
module add gcc
</pre>

===Tcsh===
<pre>
source /usr/share/Modules/init/tcsh
module add gcc
</pre>

==Modules in Non-Interactive Shell Sessions==
In non-interactive shell sessions (non-login shells), the Modules configuration environment will not automatically load. This will also occur if the OS version of the compute node you are scheduled on is different from the OS version of the submission node you are submitting the job from.

If you will need the use of Modules in non-interactive [[SLURM]] jobs, cross-OS jobs, or other similar sessions, you will need to include the following in your shell init scripts:

===Bash===
<pre>
. /usr/share/Modules/init/bash
. /etc/profile.d/ummodules.sh
</pre>

===Tcsh===
<pre>
source /usr/share/Modules/init/tcsh
source /etc/profile.d/ummodules.csh
</pre>

==Modules on RHEL9 Desktop Sessions==
Because of changes made in GNOME between RHEL8 and RHEL9, the <tt>module</tt> command will not work on desktop sessions out of the box. In order to use modules in a shell opened from a desktop session, you must source the modules init script. See [[#Modules in Non-Interactive Shell Sessions]] for the required commands. For convenience, we recommend adding this to the top of your shell init file (e.g ~/.bashrc, ~/.tcshrc), this way it gets sourced automatically with each new shell.

Note that this issue does not affect RHEL9 SSH sessions, only sessions using the desktop GUI.

==Additional Help==
You can type <tt>module</tt> with no arguments for a full list of commands or <tt>man module</tt> for further information.

===Online Resources===
*[http://modules.sourceforge.net/ Project Page (SourceForge)]
*[http://modules.sourceforge.net/docs/Modules-Paper.pdf Introduction to Modules]
*[http://sourceforge.net/p/modules/wiki/FAQ/ Modules FAQ]
*[http://modules.sourceforge.net/docs/user-setup.pdf user-setup]

Nexus

2024-05-10T15:59:41Z

Seide: /* Partitions */

The Nexus is the combined scheduler of resources in UMIACS. The resource manager for Nexus is [[SLURM]]. Resources are arranged into partitions where users are able to schedule computational jobs. Users are arranged into a number of SLURM accounts based on faculty, lab, or center investments.

= Getting Started =
All accounts in UMIACS are sponsored. If you don't already have a UMIACS account, please see [[Accounts]] for information on getting one. You need a full UMIACS account (not a [[Accounts/Collaborator | collaborator account]]) in order to access Nexus.

== Access ==
Your access to submission nodes (alternatively called login nodes) for Nexus computational resources is determined by your account sponsor's department, center, or lab affiliation. You can log into the [https://intranet.umiacs.umd.edu/directory/cr/ UMIACS Directory CR application] and select the Computational Resource (CR) in the list that has the prefix <code>nexus</code>. The Hosts section lists your available submission nodes - generally a pair of nodes of the format <tt>nexus<department, lab, or center abbreviation>[00,01]</tt>, e.g., <tt>nexusgroup00</tt> and <tt>nexusgroup01</tt>.

'''Note''' - UMIACS requires multi-factor authentication through our [[Duo]] instance. This is completely discrete from both UMD's and CSD's Duo instances. You will need to enroll one or more devices to access resources in UMIACS, and will be prompted to enroll when you log into the Directory application for the first time.

Once you have identified your submission nodes, you can [[SSH]] directly into them. From there, you are able to submit to the cluster via our [[SLURM]] workload manager. You need to make sure that your submitted jobs have the correct account, partition, and qos.

== Jobs ==
[[SLURM]] jobs are [[SLURM/JobSubmission | submitted]] by either <code>srun</code> or <code>sbatch</code> depending if you are doing an interactive job or batch job, respectively. You need to provide the where/how/who to run the job and specify the resources you need to run with.

For the who/where/how, you may be required to specify <code>--account</code>, <code>--partition</code>, and/or <code>--qos</code> (respectively) to be able to adequately submit jobs to the Nexus.

For resources, you may need to specify <code>--time</code> for time, <code>--ntasks</code> for CPUs, <code>--mem</code> for RAM, and <code>--gres=gpu</code> for GPUs in your submission arguments to meet your requirements. There are defaults for all four, so if you don't specify something, you may be scheduled with a very minimal set of time and resources (e.g., by default, NO GPUs are included if you do not specify <code>--gres=gpu</code>). For more information about submission flags for GPU resources, see [[SLURM/JobSubmission#Requesting_GPUs | here]]. You can also can run <code>man srun</code> on your submission node for a complete list of available submission arguments.

For a list of available GPU types on Nexus and their specs, please see [[Nexus/GPUs]].

=== Interactive ===
Once logged into a submission node, you can run simple interactive jobs. If your session is interrupted from the submission node, the job will be killed. As such, we encourage use of a terminal multiplexer such as [[Tmux]].

<pre>
$ srun --pty --ntasks=4 --mem=2gb --gres=gpu:1 nvidia-smi -L
GPU 0: NVIDIA RTX A4000 (UUID: GPU-ae5dc1f5-c266-5b9f-58d5-7976e62b3ca1)
</pre>

=== Batch ===
Batch jobs are scheduled with a script file with an optional ability to embed job scheduling parameters via variables that are defined by <code>#SBATCH</code> lines at the top of the file. You can find some examples in our [[SLURM/JobSubmission]] documentation.

= Partitions =
The SLURM resource manager uses partitions to act as job queues which can restrict size, time and user limits. The Nexus has a number of different partitions of resources. Different Centers, Labs, and Faculty are able to invest in computational resources that are restricted to approved users through these partitions.

'''Partitions usable by all non-[[ClassAccounts |class account]] users:'''
* [[Nexus/Tron]] - Pool of resources available to all UMIACS and CSD faculty and graduate students.
* Scavenger - [https://slurm.schedmd.com/preempt.html Preemption] partition that supports nodes from multiple other partitions. More resources are available to schedule simultaneously than in other partitions, however jobs are subject to preemption rules. You are responsible for ensuring your jobs handle this preemption correctly. The SLURM scheduler will simply restart a preempted job with the same submission arguments when it is available to run again. For an overview of things you can check within scripts to determine if your job was preempted/resumed, see [[SLURM/Preemption]].

'''Partitions usable by [[ClassAccounts]]:'''
* [[ClassAccounts | Class]] - Pool available for UMIACS class accounts sponsored by either UMIACS or CSD faculty.

'''Partitions usable by specific lab/center users:'''
* [[Nexus/CBCB]] - CBCB lab pool available for CBCB lab members.
* [[Nexus/CLIP]] - CLIP lab pool available for CLIP lab members.
* [[Nexus/CML]] - CML lab pool available for CML lab members.
* [[Nexus/GAMMA]] - GAMMA lab pool available for GAMMA lab members.
* [[Nexus/MBRC]] - MBRC lab pool available for MBRC lab members.
* [[Nexus/MC2]] - MC2 lab pool available for MC2 lab members.
* [[Nexus/Vulcan]] - Vulcan lab pool available for Vulcan lab members.

= Quality of Service (QoS) =
SLURM uses Quality of Service (QoS) both to provide limits on job sizes (termed by us as "job QoS") as well as to limit resources used by all jobs running in a partition, either per user or per group (termed by us as "partition QoS").

=== Job QoS ===
Job QoS are used to provide limits on the size of job that you can run. You should try to allocate only the resources your job actually needs, as resources that each of your jobs schedules are counted against your [[SLURM/Priority#Fair-share | fair-share priority]] in the future.
* default - Default job QoS. Limited to 4 CPU cores, 1 GPU, and 32GB RAM per job. The maximum wall time per job is 3 days.
* medium - Limited to 8 CPU cores, 2 GPUs, and 64GB RAM per job. The maximum wall time per job is 2 days.
* high - Limited to 16 CPU cores, 4 GPUs, and 128GB RAM per job. The maximum wall time per job is 1 day.
* scavenger - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. 576 total CPU cores, 72 total GPUs, and 2304GB total RAM are permitted simultaneously across all of your jobs running with this job QoS. This job QoS is both only available in the scavenger partition and the only job QoS available in the scavenger partition. To use this job QoS, include <code>--partition=scavenger</code> and <code>--account=scavenger</code> in your submission arguments. Do not include any job QoS argument other than <code>--qos=scavenger</code> (optional) or submission will fail.

You can display these job QoS from the command line using the <code>show_qos</code> command. By default, the command will only show job QoS that you can access. The above four job QoS are the ones that everyone can access.

<pre>
$ show_qos
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
scavenger 3-00:00:00 cpu=576,gres/gpu=72,mem=2304G
</pre>

If you want to see all job QoS, including those that you do not have access to, you can use the <code>show_qos --all</code> command.

<pre>
$ show_qos --all
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
cml-cpu 7-00:00:00 8
cml-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
cml-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
cml-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
cml-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
cml-scavenger 3-00:00:00 gres/gpu=24
cml-very_high 1-12:00:00 cpu=32,gres/gpu=8,mem=256G 8 gres/gpu=12
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
highmem 21-00:00:00 cpu=32,mem=2T
huge-long 10-00:00:00 cpu=32,gres/gpu=8,mem=256G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
scavenger 3-00:00:00 cpu=576,gres/gpu=72,mem=2304G
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-exempt 7-00:00:00 cpu=32,gres/gpu=8,mem=256G 2
vulcan-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
vulcan-janus 3-00:00:00 cpu=32,gres/gpu=10,mem=256G
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-sailon 3-00:00:00 cpu=32,gres/gpu=8,mem=256G gres/gpu=48
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
</pre>

To find out what accounts and partitions you have access to, first use the <code>show_assoc</code> command to show your account/job QoS combinations. Then, use the <code>scontrol show partition</code> command and note the <tt>AllowAccounts</tt> entry for each listed partition. You are able to submit to any partition that allows an account that you have. If you need to use an account other than the default account <tt>nexus</tt>, you will need to specify it via the <code>--account</code> submission argument.

=== Partition QoS ===
Partition QoS are used to limit resources used by all jobs running in a partition, either per user (MaxTRESPU) or per group (GrpTRES).

To view partition QoS, use the <code>show_partition_qos</code> command.

<pre>
$ show_partition_qos
Name MaxSubmitPU MaxTRESPU GrpTRES
-------------------- ----------- ------------------------------ --------------------
scavenger 500 cpu=576,gres/gpu=72,mem=2304G
tron 500 cpu=32,gres/gpu=4,mem=256G
</pre>

If you want to see all partition QoS, including those that you do not have access to, you can use the <code>show_partition_qos --all</code> command.

<pre>
$ show_partition_qos --all
Name MaxSubmitPU MaxTRESPU GrpTRES
-------------------- ----------- ------------------------------ --------------------
cbcb 500 cpu=1004,mem=47840G
class 500 cpu=32,gres/gpu=4,mem=256G
clip 500 cpu=564,mem=5647G
cml 500 cpu=1128,mem=11381G
cml-scavenger 500 gres/gpu=24
gamma 500 cpu=520,mem=4517G
mbrc 500 cpu=240,mem=2378G
mc2 500 cpu=312,mem=3133G
scavenger 500 cpu=576,gres/gpu=72,mem=2304G
tron 500 cpu=32,gres/gpu=4,mem=256G
vulcan 500 cpu=1760,mem=15824G
vulcan-scavenger 500
</pre>

'''NOTE''': These QoS cannot be used directly when submitting jobs, with the exception of scavenger QoS (i.e. they are not in the AllowQos field for their respective partition). Partition QoS limits apply to all jobs running on a given partition, even when using multiple job QoS.

For example, in the default non-preemption partition (<tt>tron</tt>), you are restricted to 32 total CPU cores, 4 total GPUs, and 256GB total RAM at once across all jobs you have running in the partition. You also can only have a maximum of 500 submitted (running (R) or pending (PD)) jobs in the partition simultaneously. The latter restriction is to prevent excess pending jobs causing [https://slurm.schedmd.com/sched_config.html#backfill backfill] issues with the SLURM scheduler.
* If you need to submit more than 500 jobs in batch at once, you can develop and run an "outer submission script" that repeatedly attempts to run the "inner submission script" (your original submission script) to submit jobs in the batch periodically, until all job submissions are successful. The outer submission script should use looping logic to check if you are at the max job limit and should then retry submission after waiting for some time interval.
: An example outer submission script is as follows. In this example, <code>example_inner.sh</code> is your inner submission script and you want to run 1000 jobs. In this example, the inner submission script is not an [[SLURM/ArrayJobs | array job]]. If your inner submission script is an array job, adjust the number of jobs accordingly. Keep in mind that array jobs must be of size 500 or fewer.
<pre>
#!/bin/bash
numjobs=1000
i=0
while [ $i -lt $numjobs ]
do
while [[ "$(sbatch example_inner.sh 2>&1)" =~ "QOSMaxSubmitJobPerUserLimit" ]]
do
echo "Currently at maximum job submissions allowed."
echo "Waiting for 5 minutes before trying to submit more jobs."
sleep 300
done
i=$(( $i + 1 ))
echo "Submitted job $i of $numjobs"
done
</pre>

It is suggested that you run the outer submission script in a [[Tmux]] session to keep the terminal window executing it from being interrupted.

Lab/group-specific partitions may also have partition QoS intended to limit the total number of resources consumed by all users in that lab/group that are using the partition (codified by <tt>GrpTRES</tt> in the output above for the partition QoS name that matches the lab/group partition name). They also have the 500 submitted job maximum. Note that the values listed above in various "TRES" columns are not fixed and may fluctuate as more resources are added to or removed from various partitions.

= Storage =
All network storage available in Nexus is currently [[NFS]] based, and comes in a few different flavors. Compute nodes also have local storage that can be used.

== Home Directories ==
{{Nfshomes}}

== Scratch Directories ==
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Nexus compute infrastructure:
* Network scratch directories
* Local scratch directories

Please note that [[ClassAccounts | class accounts]] do not have network scratch directories.

=== Network Scratch Directories ===
You are allocated 200GB of scratch space via NFS from <code>/fs/nexus-scratch/<USERNAME></code> where <USERNAME> is your UMIACS username. '''It is not backed up or protected in any way.''' This directory is '''automounted''' so you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access this.

You can view your quota usage by running <code>df -h /fs/nexus-scratch/<USERNAME></code>.

You may request a permanent increase of up to 400GB total space without any faculty approval by [[HelpDesk | contacting staff]]. If you need space beyond 400GB, you will need faculty approval and/or a [[#Project_Allocations | project allocation]] for this. If you choose to increase your scratch space beyond 400GB, the increased space is also subject to the 270 TB days limit mentioned in the project allocation section before we check back in for renewal. For example, if you request 1.4TB total space, you may have this for 270 days (1TB beyond the 400GB permanent increase).

This file system is available on all submission, data management, and computational nodes within the cluster.

=== Local Scratch Directories ===
Each computational node that you can schedule compute jobs on also has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. and '''are not backed up or protected in any way.''' These directories are almost always more performant than any other storage available to the job as they are mounted from disks directly attached to the compute node. However, you must stage your data within the confines of your job and extract the relevant resultant data elsewhere before the end of your job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month during our [[MonthlyMaintenanceWindow | monthly maintenance windows]]. Please make sure you secure any resultant data you wish to keep from these directories at the end of your job.

== Faculty Allocations ==
Each faculty member can be allocated 1TB of permanent lab space upon request. We can also support grouping these individual allocations together into larger center, lab, or research group allocations if desired by the faculty. Please [[HelpDesk | contact staff]] to inquire.

Lab space storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

== Project Allocations ==
Project allocations are available per user for 270 TB days; you can have a 1TB allocation for up to 270 days, a 3TB allocation for 90 days, etc.. A single faculty member can not have more than 20TB of sponsored account project allocations active at any point.

Project storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

The maximum allocation length you can request is 540 days (500GB space) and the maximum storage space you can request is 9TB (30 day length).

To request an allocation, please [[HelpDesk | contact staff]] with the faculty member(s) that the project is under involved in the conversation. Please include the following details:
* Project Name (short)
* Description
* Size (1TB, 2TB, etc.)
* Length in days (270 days, 135 days, etc.)
* Other user(s) that need to access the allocation, if any

These allocations are available via <code>/fs/nexus-projects/<project name></code>. '''Renewal is not guaranteed to be available due to limits on the amount of total storage.''' Near the end of the allocation period, staff will contact you and ask if you are still in need of the storage allocation. If renewal is available, you can renew for up to another 270 TB days with reapproval from the original faculty approver.
* If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation.
* If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible.
** If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible.
** If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

== Datasets ==
We have read-only dataset storage available at <code>/fs/nexus-datasets</code>. If there are datasets that you would like to see curated and available, please see [[Datasets | this page]].

The list of Nexus datasets we currently host can be viewed [https://info.umiacs.umd.edu/datasets/list/?q=Nexus here].

SLURM

2024-05-10T15:56:56Z

Seide: /* Documentation */

=Simple Linux Utility for Resource Management (SLURM)=
SLURM is an open-source workload manager designed for Linux clusters of all sizes. It provides three key functions. First, it allocates exclusive or non-exclusive access to resources (computer nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (typically a parallel job) on a set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work.

==Documentation==
:[[SLURM/JobSubmission | Submitting Jobs]]
:[[SLURM/JobStatus | Checking Job Status]]
:[[SLURM/ClusterStatus | Checking Cluster Status]]
:[[SLURM/Priority | Understanding Job Priority]]
:[[SLURM/Preemption | Job Preemption Overview]]
:[http://slurm.schedmd.com/documentation.html Official Documentation]
:[http://slurm.schedmd.com/faq.html FAQ]

==Commands==
Below are some of the common commands used in SLURM. Further information on how to use these commands is found in the documentation linked above. To see all flags available for a command, please check the command's manual by using <code>man <COMMAND></code> on the command line.

====[https://slurm.schedmd.com/srun.html srun]====
srun runs a parallel job on a cluster managed by SLURM. If necessary, it will first create a resource allocation in which to run the parallel job.

====[https://slurm.schedmd.com/salloc.html salloc]====
salloc allocates a SLURM job allocation, which is a set of resources (nodes), possibly with some set of constraints (e.g. number of processors per node). When salloc successfully obtains the requested allocation, it then runs the command specified by the user. Finally, when the user specified command is complete, salloc relinquishes the job allocation. If no command is specified, salloc runs the user's default shell.

====[https://slurm.schedmd.com/sbatch.html sbatch]====
sbatch submits a batch script to SLURM. The batch script may be given to sbatch through a file name on the command line, or if no file name is specified, sbatch will read in a script from standard input. The batch script may contain options preceded with <code>#SBATCH</code> before any executable commands in the script.

====[https://slurm.schedmd.com/squeue.html squeue]====
squeue views job and job step information for jobs managed by SLURM.

====[https://slurm.schedmd.com/scancel.html scancel]====
scancel signals or cancels jobs, job arrays, or job steps. An arbitrary number of jobs or job steps may be signaled using job specification filters or a space separated list of specific job and/or job step IDs.

====[https://slurm.schedmd.com/sacct.html sacct]====
sacct displays job accounting data stored in the job accounting log file or SLURM database in a variety of forms for your analysis. The sacct command displays information on jobs, job steps, status, and exitcodes by default. You can tailor the output with the use of the <code>--format=</code> option to specify the fields to be shown.

====[https://slurm.schedmd.com/sstat.html sstat]====
sstat displays job status information for your analysis. The sstat command displays information pertaining to CPU, Task, Node, Resident Set Size (RSS) and Virtual Memory (VM). You can tailor the output with the use of the <code>--fields=</code> option to specify the fields to be shown.

==Modules==
If you are trying to use [[Modules | GNU Modules]] in a Slurm job, please read the section of our [[Modules]] documentation on [[Modules#Modules_in_Non-Interactive_Shell_Sessions | non-interactive shell sessions]]. This also needs to be done if the OS version of the compute node you are scheduled on is different from the OS version of the submission node you are submitting the job from.

==Running Jupyter Notebook on a Compute Node==
The steps to run a Jupyter Notebook from a compute node are listed below.

<h6>Setting up your Python Virtual Environment</h6>
[[PythonVirtualEnv | Create a Python virtual environment]] on the compute node you are assigned and [[PythonVirtualEnv#Activating_the_VirtualEnv | activate it]]. Next, install Jupyter using pip by following the steps [https://jupyter.readthedocs.io/en/latest/install/notebook-classic.html#alternative-for-experienced-python-users-installing-jupyter-with-pip here]. You may also use other environment management systems such as [https://docs.conda.io/en/latest/ Conda] if desired.

<h6>Running Jupyter Notebook</h6>
After you've set up the Python virtual environment, submit a job, activate the environment within the job, and run the following command on the compute node you are assigned:
<pre>
jupyter notebook --no-browser --port=8889 --ip=0.0.0.0
</pre>

This will start running the notebook on port 8889. <b>Note:</b> You must keep this shell window open to be able to connect. If the submission node for the cluster you are using is not accessible via the public internet, you must also be on a machine connected to the UMIACS network or connected to our [[Network/VPN | VPN]] in order to access the Jupyter notebook once you start the SSH tunnel, so ensure this is the case before starting the tunnel. Then, on your local machine, run
<pre>
ssh -N -f -L localhost:8888:<NODENAME>:8889 <USERNAME>@<SUBMISSIONNODE>.umiacs.umd.edu
</pre>

This will tunnel port 8889 from the compute node to port 8888 on your local machine, using <SUBMISSIONNODE> as an intermediate node. Make sure to replace <USERNAME> with your username, <SUBMISSIONNODE> with the name of the submission node you want to use, and <NODENAME> with the name of the compute node you are assigned. Note that this command will not display any output if the connection is successful due to the included ssh flags. You must also keep this shell window open to be able to connect.

For example, assuming your username is <code>username</code> and that you are using the [[Nexus]] cluster, have been [[Nexus#Access | assigned]] the nexusgroup submission nodes, and are assigned compute node tron00.umiacs.umd.edu:
<pre>
ssh -N -f -L localhost:8888:tron00.umiacs.umd.edu:8889 username@nexusgroup.umiacs.umd.edu
</pre>

You can then open a web browser and type in <code>localhost:8888</code> to access the notebook.

Notes:
* Later versions of Jupyter have token authentication enabled by default - you will need to prepend the <code>/?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</code> part of the URL provided by the terminal output after starting the notebook in order to connect if this is the case. e.g. <code>localhost:8888/?token=fcc6bd0f996e7aa89376c33cb34f7b80890502aacc97d98e</code>
* If the port on the compute node mentioned in the example above (8889) is not working, it may be that someone else has already started a process (Jupyter notebook or otherwise) using that specific port number on that specific compute node. The port number can be replaced with any other [https://en.wikipedia.org/wiki/Ephemeral_port ephemeral port] number you'd like, just make sure to change it in both the command you run on the compute node and the ssh command from your local machine.

=Quick Guide to translate PBS/Torque to SLURM=

{| class="wikitable"
|+User commands
|-
!
!PBS/Torque
!SLURM
|-
!Job submission
|qsub [filename]
|sbatch [filename]
|-
!Job deletion
|qdel [job_id]
|scancel [job_id]
|-
!Job status (by job)
|qstat [job_id]
|squeue --job [job_id]
|-
!Full job status (by job)
|qstat -f [job_id]
|scontrol show job [job_id]
|-
!Job status (by user)
|qstat -u [username]
|squeue --user=[username]
|}

{| class="wikitable"
|+Environment variables
|-
!
!PBS/Torque
!SLURM
|-
!Job ID
|$PBS_JOBID
|$SLURM_JOBID
|-
!Submit Directory
|$PBS_O_WORKDIR
|$SLURM_SUBMIT_DIR
|-
!Node List
|$PBS_NODEFILE
|$SLURM_JOB_NODELIST
|}

{| class="wikitable"
|+Job specification
|-
!
!PBS/Torque
!SLURM
|-
!Script directive
|#PBS
|#SBATCH
|-
!Job Name
| -N [name]
| --job-name=[name] OR -J [name]
|-
!Node Count
| -l nodes=[count]
| --nodes=[min[-max]] OR -N [min[-max]]
|-
!CPU Count
| -l ppn=[count]
| --ntasks-per-node=[count]
|-
!CPUs Per Task
|
| --cpus-per-task=[count]
|-
!Memory Size
| -l mem=[MB]
| --mem=[MB] OR --mem-per-cpu=[MB]
|-
!Wall Clock Limit
| -l walltime=[hh:mm:ss]
| --time=[min] OR --time=[days-hh:mm:ss]
|-
!Node Properties
| -l nodes=4:ppn=8:[property]
| --constraint=[list]
|-
!Standard Output File
| -o [file_name]
| --output=[file_name] OR -o [file_name]
|-
!Standard Error File
| -e [file_name]
| --error=[file_name] OR -e [file_name]
|-
!Combine stdout/stderr
| -j oe (both to stdout)
|(Default if you don't specify --error)
|-
!Job Arrays
| -t [array_spec]
| --array=[array_spec] OR -a [array_spec]
|-
!Delay Job Start
| -a [time]
| --begin=[time]
|}

SLURM/Preemption

2024-05-10T15:54:15Z

Seide: Created page with "If you are submitting to a partition which is eligible for preemption (e.g. scavenger), you are responsible for making sure that your job can be interrupted/restarted gracefully. Here we are documenting some Slurm behaviors that you can use to determine if your job has been cancelled or preempted. You should be able to take different code paths during startup/shutdown of your jobs based on this information. =Flowchart for job cancellation/preemption= ==Cancellation==..."

If you are submitting to a partition which is eligible for preemption (e.g. scavenger), you are responsible for making sure that your job can be interrupted/restarted gracefully. Here we are documenting some Slurm behaviors that you can use to determine if your job has been cancelled or preempted. You should be able to take different code paths during startup/shutdown of your jobs based on this information.

=Flowchart for job cancellation/preemption=

==Cancellation==

# Slurm controller sends an internal cancel signal to Slurm node(s) where the job is currently assigned.
# Slurm node(s) send SIGCONT and SIGTERM around the same time, and the following fields of note are set in the output of <tt>scontrol --json show job $SLURM_JOBID</tt>:
#* <tt>job_state = ['CANCELLED', 'COMPLETING']</tt>
# If the processes don't stop within a certain amount of time, eventually SIGKILL will be sent.

==Preemption/Requeue==

# Slurm controller sends an internal preemption/requeue signal to Slurm node(s) where the job is currently assigned.
# Slurm node(s) send SIGCONT and SIGTERM around the same time, and the following fields of note are set in the output of <tt>scontrol --json show job $SLURM_JOBID</tt>:
#* <tt>job_state = ['PENDING', 'COMPLETING']</tt>
#* <tt>restart_cnt += 1</tt>
# If the processes don't stop within a certain amount of time, eventually SIGKILL will be sent.
# Once the job is stopped, it enters the PENDING state for two minutes, and then is eligible to be run again.
# When the job runs again, an additional environment variable will be defined, <tt>SLURM_RESTART_COUNT</tt>, which reports the number of times the job has been preempted/requeued.

=Key takeaways=

==On job cancellation/preemption==

You can handle the SIGTERM signal and run <tt>scontrol --json show job $SLURM_JOBID</tt> within your script. Based on the value of <tt>job_state</tt>, your script can take a different codepath depending on if it was cancelled or was preempted.

The output of <tt>scontrol --json show job $SLURM_JOBID</tt> is equivalent to the Slurm REST API's endpoint "GET /slurm/v0.0.40/job/{job_id}". For more information on how to parse the output of this scontrol command, please refer to Slurm's REST API documentation. [https://slurm.schedmd.com/rest_api.html#slurmV0040GetJob]

==On resuming==

If your job needs to behave differently based on whether or not it was previously preempted, you can check if the environment variable <tt>SLURM_RESTART_COUNT</tt> is defined.

=Testing=

You can use the following commands to manually cancel and requeue your jobs (requeueing and preemption are handled similarly). With these tools, you can test your scripts to ensure that they gracefully handle these scenarios.

Cancellation: <tt>scancel <job id></tt>

Preemption: <tt>scontrol requeue <job id>

Podman

2024-05-03T19:40:13Z

Seide: /* Storage */

[https://podman.io/ Podman] is a daemonless container engine alternative to [https://www.docker.com/ Docker]. We don't support Docker in many of our environments as it grants trivial administrative control over the host the Docker daemon runs on. Podman on the other hand has the ability to run containers in user namespaces. This means that for every user name space in the kernel you create the processes within it will map to a new uid/gid range. For example, if you are root in your container, you will not be uid 0 outside the container, but instead you will be uid 4294000000.

We still believe that [[Apptainer]] is the best option for running containerized workloads on our clustered based resources. Podman is a good option for developing the containers to be run via Apptainer or building a deliverable for a funding agency. Please [[HelpDesk | contact staff]] if you would like Podman installed on a workstation or standalone server. More information on Podman running [https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md rootless].

== Getting Started ==
To get started there are a few things that you need to configure.

First, run the <code>podman</code> command. If it says command not found or you get an ERRO like the one below about no subuid ranges, and you are on a workstation or standalone (non-cluster) server, please [[HelpDesk | contact staff]] with the error and the host that you are using. We will need to do some steps to setup the host you want ready.

<pre>
$ podman
ERRO[0000] cannot find mappings for user username: No subuid ranges found for user "username" in /etc/subuid
Error: missing command 'podman COMMAND'
Try 'podman --help' for more information.
</pre>

=== Storage ===
Containers are made up of layers for the image and these are stored in the graphroot setting of <code>~/.config/containers/storage.conf</code> which by default will be in your home directory. With our home directories being available over NFS there is an [https://www.redhat.com/sysadmin/rootless-podman-nfs issue] that due to the user name space mapping described above you will not be able to access your home directory when you are building the layers.

You need to update the <code>graphroot</code> setting to a local directory on the host. The file <code>~/.config/containers/storage.conf</code> may not exist until you run <code>podman</code> the first time, however you can manually create it.

<pre>
[storage]
driver = "vfs"
runroot = "/tmp/run-2174"
graphroot = "/scratch0/username/.local/share/containers/storage"
...
</pre>

When building larger images, it may fill up the default directory for imageCopyTmpDir (/var/tmp). If this happens, you will need to specify a different directory using the environment variable TMPDIR. For example:

<pre>export TMPDIR="/scratch0/example_tmp_directory"</pre>

== GPUs ==
Running Podman with the local Nvidia GPUs requires some additional configuration steps that staff has to add to any individual workstation or standalone (non-cluster) server that runs Podman. This includes ensuring the <tt>nvidia-container-runtime</tt> package is installed.

For example you can run <code>nvidia-smi</code> from within the official Nvidia CUDA containers with a command like this, optionally replacing the tag for different CUDA versions/OS images:

<pre>
$ podman run --rm --hooks-dir=/usr/share/containers/oci/hooks.d docker.io/nvidia/cuda:12.2.2-base-ubi8 nvidia-smi
Thu Apr 16 18:47:04 2020
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2 |
+---------------------------------------------------------------------------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA RTX A6000 Off | 00000000:01:00.0 Off | Off |
| 30% 28C P8 6W / 300W | 2MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
</pre>

The full list of tags can be found at https://hub.docker.com/r/nvidia/cuda/tags.

== Example ==
To build your own image you can start from an example we have https://gitlab.umiacs.umd.edu/derek/gpudocker.

First clone the repository, change directory and build the image with podman.
<pre>
git clone https://gitlab.umiacs.umd.edu/derek/gpudocker.git
cd gpudocker
podman build -t gpudocker .
</pre>

Then you can run the test script to verify. Notice that we pass the local directory <code>test</code> as a path into the image so we can run a script. This can also be useful for your data output data as well as if you write anywhere else in the container it will not be available outside the container.
<pre>
$ podman run --volume `pwd`/test:/mnt --hooks-dir=/usr/share/containers/oci/hooks.d gpudocker python3 /mnt/test_torch.py
GPU found 0: GeForce GTX 1080 Ti
tensor([[0.3479, 0.6594, 0.5791],
[0.6065, 0.3415, 0.9328],
[0.9117, 0.3541, 0.9050],
[0.6611, 0.5361, 0.3212],
[0.8574, 0.5116, 0.7021]])
</pre>

If you instead want to push modifications to this example to your own container registry such that you can pull the container image down later, please see the README.md located in the example repository itself.

Tensorflow

2024-04-29T17:17:59Z

Seide:

[https://www.tensorflow.org/ Tensorflow] is a [[Python]] deep learning package from Google. The easiest way to use install it is to build a [[PythonVirtualEnv | Python virtualenv]] with it in it.

First load GPU modules to allow access to accelerated GPGPU training. You can find the list of Tensorflow versions and their corresponding Python/CUDA/cuDNN requirements here. [https://www.tensorflow.org/install/source#gpu] These modules are appropriate for tensorflow-2.16.1, which is the latest tensorflow release as of April 29, 2024.

<pre>module add Python3/3.9.16 cuda/12.4.1 cudnn/v8.9.7</pre>

Next you will want to create a virtualenv and source into it. Note that depending on the version of Tensorflow you need, you may also need to load a module for a more recent version of Python3.

<pre>
$ python3 -m venv env
$ source env/bin/activate
(env) $
</pre>

Then ensure you have a recent copy of pip in your virtualenv.

<pre>
(env) $ pip install --upgrade pip
</pre>

Then install the Tensorflow wheel through pip.

<pre>
(env) $ pip install --upgrade tensorflow
</pre>

Finally, start up a python shell (or install ipython through pip) and import Tensorflow.

<pre>
(env)[username@hostname:/scratch0/username ] $ python
Python 3.9.16 (main, Feb 28 2023, 09:58:09)
[GCC 8.5.0 20210514 (Red Hat 8.5.0-16)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2024-04-29 13:14:09.685077: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-29 13:14:09.737280: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-29 13:14:11.316744: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
>>> tf.__version__
'2.16.1'
</pre>

You can then try a more rigorous test by running the following example. Note that you may need to export XLA_FLAGS in your shell: <code>export XLA_FLAGS=--xla_gpu_cuda_data_dir=/opt/common/cuda/cuda-x.x.x</code>
<pre>
import tensorflow as tf
mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(512, activation=tf.nn.relu),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)
</pre>

<b>To use this install after you close the shell you did this install in, you will need to both add the correct [[CUDA]]/cuDNN modules, export the XLA_FLAGS variable (if needed), and activate the virtualenv by the source command. This includes any time you are submitting to [[SLURM]].</b>

Nexus/GPUs

2024-04-23T16:36:47Z

Seide: h100

There are several different types of [https://www.nvidia.com/en-us/ NVIDIA] GPUs in the [[Nexus]] cluster that are available to be scheduled. They are listed below in order of newest to oldest architecture, and then alphabetically.

We do not list the exact quantities of GPU here since they change frequently due to additions to or removals from the cluster or during compute node troubleshooting. To see which compute nodes have which GPUs and in what quantities, use the <code>show_nodes</code> command on a submission or compute node. The quantities are listed under the <tt>GRES</tt> column.

{| class="wikitable sortable"
! Name
! GRES string (Slurm)
! [https://www.nvidia.com/en-us/technologies Architecture]
! [https://developer.nvidia.com/cuda-toolkit CUDA] Cores
! GPU Memory
! Memory Bandwidth
! FP32 Performance (TFLOPS)
! [https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/ TF32] Performance ([https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt Dense / Sparse TOPS])
|-
| RTX 6000 Ada Generation
| <code>rtx6000ada</code>
| Ada Lovelace
| 18176
| 48GB GDDR6
| 960GB/s
| 91.1
| 182.1/364.2
|-
| H100 SXM5 80GB
| <code>h100</code>
| Hopper
| 16896
| 80GB HBM3
| 3.36TB/s
| 66.91
| not officially published/989
|-
| A100 (80GB)
| <code>a100</code>
| Ampere
| 6912
| 80GB HBM2e
| 1935GB/s
| 19.5
| 156/312
|-
| GeForce RTX 3070
| <code>rtx3070</code>
| Ampere
| 5888
| 8GB GDDR6
| 448GB/s
| 20.3
| 20.3/40.6
|-
| GeForce RTX 3090
| <code>rtx3090</code>
| Ampere
| 10496
| 24GB GDDR6X
| 936GB/s
| 35.6
| 35.6/71
|-
| RTX A4000
| <code>rtxa4000</code>
| Ampere
| 6144
| 16GB GDDR6
| 448GB/s
| 19.2
| not officially published
|-
| RTX A5000
| <code>rtxa5000</code>
| Ampere
| 8192
| 24GB GDDR6
| 768GB/s
| 27.8
| not officially published
|-
| RTX A6000
| <code>rtxa6000</code>
| Ampere
| 10752
| 48GB GDDR6
| 768GB/s
| 38.7
| 77.4/154.8
|-
| GeForce RTX 2080 Ti
| <code>rtx2080ti</code>
| Turing
| 4352
| 11GB GDDR5X
| 616GB/s
| 13.4
| n/a
|-
| GeForce GTX 1080 Ti
| <code>gtx1080ti</code>
| Pascal
| 3584
| 11GB GDDR5X
| 484GB/s
| 11.3
| n/a
|-
| Quadro P6000
| <code>p6000</code>
| Pascal
| 3840
| 24GB GDDR5X
| 432GB/s
| 12.6
| n/a
|-
| Tesla P100
| <code>p100</code>
| Pascal
| 3584
| 16GB CoWoS HBM2
| 732GB/s
| 9.3
| n/a
|-
| TITAN X (Pascal)
| <code>titanxpascal</code>
| Pascal
| 3584
| 12GB GDDR5X
| 480GB/s
| 11.0
| n/a
|-
| TITAN Xp
| <code>titanxp</code>
| Pascal
| 3840
| 12GB GDDR5X
| 548GB/s
| 12.1
| n/a
|-
| GeForce GTX TITAN X
| <code>gtxtitanx</code>
| Maxwell
| 3072
| 12GB GDDR5
| 336GB/s
| 6.7
| n/a
|-
|}

SLURM/JobSubmission

2024-03-06T23:04:58Z

Seide: /* show_available_nodes */

=Job Submission=
SLURM offers a variety of ways to run jobs. It is important to understand the different options available and how to request the resources required for a job in order for it to run successfully. All job submission should be done from submit nodes; any computational code should be run in a job allocation on compute nodes. The following commands outline how to allocate resources on the compute nodes and submit processes to be run on the allocated nodes.

The cluster that everyone with a [[Accounts#UMIACS_Account | UMIACS account]] has access to is [[Nexus]]. Please visit the Nexus page for instructions on how to connect to your assigned submit nodes.

Please note that the hard maximum number of jobs that the SLURM scheduler can handle at once (on Nexus) is 50000. It is best to limit your number of submitted jobs at any given time to significantly less than this amount in the case that another user also wants to submit a large number of jobs.

'''Computational jobs run on submission nodes will be terminated. Please use compute nodes for running computational jobs.'''

For details on how SLURM decides how to schedule jobs when multiple jobs are waiting in a scheduler's queue, please see [[SLURM/Priority]].

==srun==
The <code>srun</code> command is used to run a process on the compute nodes in the cluster. If you pass it a normal shell command (or command that executes a script), it will submit a job to run that shell command/script on a compute node and then return. <code>srun</code> accepts many command line options to specify the resources required by the command passed to it. Some common command line arguments are listed below and full documentation of all available options is available in the man page for <code>srun</code>, which can be accessed by running <code>man srun</code>.

<pre>
$ srun --qos=default --mem=100mb --time=1:00:00 bash -c 'echo "Hello World from" `hostname`'
Hello World from tron33.umiacs.umd.edu
</pre>

It is important to understand that <code>srun</code> is an interactive command. By default input to <code>srun</code> is broadcast to all compute nodes running your process and output from the compute nodes is redirected to <code>srun</code>. This behavior can be changed; however, '''srun will always wait for the command passed to finish before exiting, so if you start a long running process and end your terminal session, your process will stop running on the compute nodes and your job will end'''. To run a non-interactive submission that will remain running after you logout, you will need to wrap your <code>srun</code> commands in a batch script and submit it with [[#sbatch | sbatch]].

===Common srun arguments===
* <code>--job-name=helloWorld</code> ''name of your job''
* <code>--mem=1gb</code> ''request 1GB of memory, if no unit is given MB is assumed''
* <code>--ntasks=2</code> ''request 2 "tasks" which map to cores on a CPU, if passed to srun the given command will be run concurrently on each core''
* <code>--nodes=2</code> ''if passed to srun, the given command will be run concurrently on each node''
* <code>--nodelist=$NODENAME</code> ''request to run your job on the $NODENAME node''
* <code>--time=hh:mm:ss</code> ''time needed to run your job''
* <code>--error=filename</code> ''file to redirect stderr''
* <code>--partition=$PNAME</code> ''request job run in the $PNAME partition''
* <code>--qos=default</code> ''to see the available QOS options on a cluster, run'' <code>show_qos</code>
* <code>--account=accountname</code> ''use qos specific to an account''
* <code>--output=filename</code> ''file to redirect stdout to''
* <code>--requeue</code> ''automatically requeue your job if it is preempted''

===Interactive Shell Sessions===
An interactive shell session on a compute node can be useful for debugging or developing code that isn't ready to be run as a batch job. To get an interactive shell on a node, use <code>srun</code> to invoke a shell:
<pre>
$ srun --pty --qos=default --mem 1gb --time=01:00:00 bash
$ hostname
tron33.umiacs.umd.edu
</pre>
'''Please do not leave interactive shells running for long periods of time when you are not working. This blocks resources from being used by everyone else.'''

==salloc==
The <code>salloc</code> command can also be used to request resources be allocated without needing a batch script. Running salloc with a list of resources will allocate the resources you requested, create a job, and drop you into a subshell with the environment variables necessary to run commands in the newly created job allocation. When your time is up or you exit the subshell, your job allocation will be relinquished.

<pre>
$ salloc --qos=default -N 1 --mem=2gb --time=01:00:00
salloc: Granted job allocation 159
$ srun /usr/bin/hostname
tron33.umiacs.umd.edu
$ exit
exit
salloc: Relinquishing job allocation 159
</pre>

'''Please note that any commands not invoked with srun will be run locally on the submit node. Please be careful when using salloc.'''

==sbatch==
The <code>sbatch</code> command allows you to write a batch script to be submitted and run non-interactively on the compute nodes. To run a simple Hello World command on the compute nodes you could write a file, helloWorld.sh with the following contents:

<pre>
#!/bin/bash

srun bash -c 'echo Hello World from `hostname`'
</pre>

Then you need to submit the script with sbatch and request resources:

<pre>
$ sbatch --qos=default --mem=1gb --time=1:00:00 helloWorld.sh
Submitted batch job 121
</pre>

SLURM will return a job number that you can use to check the status of your job with squeue:

<pre>
$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
121 tron helloWor username R 0:01 1 tron32
</pre>

====Advanced Batch Scripts====
You can also write a batch script with all of your resources/options defined in the script itself. This is useful for jobs that need to be run tens/hundreds/thousands of times. You can then handle any necessary environment setup and run commands on the resources you requested by invoking commands with srun. The srun commands can also be more complex and be told to only use portions of your entire job allocation, each of these distinct srun commands makes up one "job step". The batch script will be run on the first node allocated as part of your job allocation and each job step will be run on whatever resources you tell them to. In the following example, we have a batch job that will request 2 nodes in the cluster. We then load a specific version of [[Python]] into our environment and submit two job steps, each one using one node. Since srun is blocks until the command finishes, we use the '&' operator to background the process so that both job steps can run at once; however, this means that we then need to use the wait command to block processing until all background processes have finished.

<pre>
#!/bin/bash

# Lines that begin with #SBATCH specify commands to be used by SLURM for scheduling

#SBATCH --job-name=helloWorld # sets the job name
#SBATCH --output=helloWorld.out.%j # indicates a file to redirect STDOUT to; %j is the jobid. If set, must be set to a file instead of a directory or else submission will fail.
#SBATCH --error=helloWorld.out.%j # indicates a file to redirect STDERR to; %j is the jobid. If set, must be set to a file instead of a directory or else submission will fail.
#SBATCH --time=00:05:00 # how long you would like your job to run; format=hh:mm:ss
#SBATCH --qos=default # set QOS, this will determine what resources can be requested
#SBATCH --nodes=2 # number of nodes to allocate for your job
#SBATCH --ntasks=4 # request 4 cpu cores be reserved for your node total
#SBATCH --ntasks-per-node=2 # request 2 cpu cores be reserved per node
#SBATCH --mem=1gb # memory required by job; if unit is not specified MB will be assumed

srun -N 1 --mem=512mb bash -c "hostname; python3 --version" & # use srun to invoke commands within your job; using an '&'
srun -N 1 --mem=512mb bash -c "hostname; python3 --version" & # will background the process allowing them to run concurrently
wait # wait for any background processes to complete

# once the end of the batch script is reached your job allocation will be revoked
</pre>

Another useful thing to know is that you can pass additional arguments into your sbatch scripts on the command line and reference them as <code>${1}</code> for the first argument and so on.

====More Examples====
* [[SLURM/ArrayJobs]]

===scancel===
The <code>scancel</code> command can be used to cancel job allocations or job steps that are no longer needed. It can be passed individual job IDs or an option to delete all of your jobs or jobs that meet certain criteria.
*<code>scancel 255</code> ''cancel job 255''
*<code>scancel 255.3</code> ''cancel job step 3 of job 255''
*<code>scancel --user username --partition=tron</code> ''cancel all jobs for username in the tron partition''

=Identifying Resources and Features=
The <code>sinfo</code> command can show you additional features of nodes in the cluster but you need to ask it to show some non-default options using a command like <code>sinfo -o "%40N %8c %8m %35f %35G"</code>.

<pre>
$ sinfo -o "%40N %8c %8m %35f %35G"
NODELIST CPUS MEMORY AVAIL_FEATURES GRES
legacy00 48 125940 rhel8,Zen,EPYC-7402 (null)
legacy[01-11,13-19,22-28,30] 12+ 61804+ rhel8,Xeon,E5-2620 (null)
cbcb[23-24],twist[02-05] 24 255150 rhel8,Xeon,E5-2650 (null)
cbcb26 128 513243 rhel8,Zen,EPYC-7763,Ampere gpu:rtxa5000:8
cbcb27 64 255167 rhel8,Zen,EPYC-7513,Ampere gpu:rtxa6000:8
cbcb[00-21] 32 2061175 rhel8,Zen,EPYC-7313 (null)
cbcb22,cmlcpu[00,06-07],legacy20 24+ 384270+ rhel8,Xeon,E5-2680 (null)
cbcb25 24 255278 rhel8,Xeon,E5-2650,Pascal,Turing gpu:rtx2080ti:1,gpu:gtx1080ti:1
legacy21 8 61746 rhel8,Xeon,E5-2623 (null)
tron[06-09,12-15,21] 16 126214+ rhel8,Zen,EPYC-7302P,Ampere gpu:rtxa4000:4
tron[10-11,16-20,34] 16 126217 rhel8,Zen,EPYC-7313P,Ampere gpu:rtxa4000:4
tron[22-33,35-45] 16 126214+ rhel8,Zen,EPYC-7302,Ampere gpu:rtxa4000:4
clip11 16 126217 rhel8,Zen,EPYC-7313,Ampere gpu:rtxa4000:4
clip00 32 255276 rhel8,Xeon,E5-2683,Pascal gpu:titanxpascal:3
clip02 20 126255 rhel8,Xeon,E5-2630,Pascal gpu:gtx1080ti:3
clip03 20 126243 rhel8,Xeon,E5-2630,Pascal,Turing gpu:rtx2080ti:1,gpu:gtx1080ti:2
clip04 32 255233 rhel8,Zen,EPYC-7302,Ampere gpu:rtx3090:4
clip[05-06] 24 126216 rhel8,Zen,EPYC-7352,Ampere gpu:rtxa6000:2
clip07 8 255263 rhel8,Xeon,E5-2623,Pascal gpu:gtx1080ti:3
clip09 32 383043 rhel8,Xeon,6130,Pascal,Turing gpu:rtx2080ti:5,gpu:gtx1080ti:3
clip13,cml30,vulcan[29-32] 32 255218+ rhel8,Zen,EPYC-7313,Ampere gpu:rtxa6000:8
clip08,vulcan[08-22,25] 32 255258+ rhel8,Xeon,E5-2683,Pascal gpu:gtx1080ti:8
clip12,gammagpu[10-17] 16 126203+ rhel8,Zen,EPYC-7313,Ampere gpu:rtxa6000:4
clip01 32 255276 rhel8,Xeon,E5-2683,Pascal gpu:titanxpascal:1,gpu:titanxp:2
clip10 44 1029404 rhel8,Xeon,E5-2699 (null)
cml[00,02-11,13-14],tron[62-63,65-66,68- 32 351530+ rhel8,Xeon,4216,Turing gpu:rtx2080ti:8
cml01 32 383030 rhel8,Xeon,4216,Turing gpu:rtx2080ti:6
cml12 32 383038 rhel8,Xeon,4216,Turing,Ampere gpu:rtx2080ti:7,gpu:rtxa4000:1
cml[15-16] 32 383038 rhel8,Xeon,4216,Turing gpu:rtx2080ti:7
cml[17-28],gammagpu05 32 255225+ rhel8,Zen,EPYC-7282,Ampere gpu:rtxa4000:8
cml31 32 384094 rhel8,Zen,EPYC-9124,Ampere gpu:a100:1
cml32 64 512999 rhel8,Zen,EPYC-7543,Ampere gpu:a100:4
cmlcpu[01-04] 20 384271 rhel8,Xeon,E5-2660 (null)
gammagpu00 32 255233 rhel8,Zen,EPYC-7302,Ampere gpu:rtxa5000:8
mbrc[00-01] 20 189498 rhel8,Xeon,4114,Turing gpu:rtx2080ti:8
twist[00-01] 8 61727 rhel8,Xeon,E5-1660 (null)
legacygpu08 20 513327 rhel8,Xeon,E5-2640,Maxwell gpu:m40:2
brigid[16-17] 48 512897 rhel8,Zen,EPYC-7443 (null)
brigid[18-19] 20 61739 rhel8,Xeon,E5-2640 (null)
legacygpu06 20 255249 rhel8,Xeon,E5-2699,Maxwell gpu:gtxtitanx:4
tron[00-05] 32 255233 rhel8,Zen,EPYC-7302,Ampere gpu:rtxa6000:8
tron[46-61] 48 255232 rhel8,Zen,EPYC-7352,Ampere gpu:rtxa5000:8
tron[64,67] 32 383028+ rhel8,Xeon,4216,Turing,Ampere gpu:rtx2080ti:7,gpu:rtx3070:1
vulcan00 32 255259 rhel8,Xeon,E5-2683,Pascal gpu:p6000:7,gpu:p100:1
vulcan[01-04,06-07] 32 255259 rhel8,Xeon,E5-2683,Pascal gpu:p6000:8
vulcan05 32 255259 rhel8,Xeon,E5-2683,Pascal gpu:p6000:7
janus[02-04] 40 383025 rhel8,Xeon,6248,Turing gpu:rtx2080ti:10
legacygpu00 20 255249 rhel8,Xeon,E5-2650,Pascal gpu:titanxp:4
legacygpu[01-02,07] 20 255249+ rhel8,Xeon,E5-2650,Maxwell gpu:gtxtitanx:4
legacygpu[03-04] 16 255268 rhel8,Xeon,E5-2630,Maxwell gpu:gtxtitanx:2
legacygpu05 44 513193 rhel8,Xeon,E5-2699,Pascal gpu:gtx1080ti:4
vulcan23 32 383030 rhel8,Xeon,4612,Turing gpu:rtx2080ti:8
vulcan26 24 770126 rhel8,Xeon,6146,Pascal gpu:titanxp:10
vulcan[27-28] 56 770093 rhel8,Xeon,8280,Turing gpu:rtx2080ti:10
vulcan24 16 126216 rhel8,Zen,7282,Ampere gpu:rtxa6000:4
gammagpu[01-04,06-09],vulcan[33-37] 32 255215+ rhel8,Zen,EPYC-7313,Ampere gpu:rtxa5000:8
vulcan[38-44] 32 255215 rhel8,Zen,EPYC-7313,Ampere gpu:rtxa4000:8
</pre>

Note that all of the nodes shown by this may not necessarily be in a partition you are able to submit to.

You can identify further specific information about a node using [[SLURM/ClusterStatus#scontrol | scontrol]] with various flags.

There are also two command aliases developed by UMIACS staff to show various node information in aggregate. They are <code>show_nodes</code> and <code>show_available_nodes</code>.

==show_nodes==
The <code>show_nodes</code> command alias shows each node's name, number of CPUs, memory, {OS, CPU architecture, CPU type, GPU architecture (if the node has GPUs)} (as AVAIL_FEATURES), GRES (GPUs), and State. It essentially wraps the <tt>sinfo</tt> command with some pre-determined output format options and shows each node on its own line, in alphabetical order.

To only view nodes in a specific partition, append <code>-p <partition name></code> to the command alias.

===Examples===
<pre>
$ show_nodes
NODELIST CPUS MEMORY AVAIL_FEATURES GRES STATE
brigid16 48 512897 rhel8,Zen,EPYC-7443 (null) idle
brigid17 48 512897 rhel8,Zen,EPYC-7443 (null) idle
... ... ... ... ... ...
vulcan44 32 255215 rhel8,Zen,EPYC-7313,Ampere gpu:rtxa4000:8 idle
</pre>

(specific partition)
<pre>
$ show_nodes -p tron
NODELIST CPUS MEMORY AVAIL_FEATURES GRES STATE
tron00 32 255233 rhel8,Zen,EPYC-7302,Ampere gpu:rtxa6000:8 idle
tron01 32 255233 rhel8,Zen,EPYC-7302,Ampere gpu:rtxa6000:8 idle
... ... ... ... ... ...
tron69 32 383030 rhel8,Xeon,4216,Turing gpu:rtx2080ti:8 idle
</pre>

==show_available_nodes==
The <code>show_available_nodes</code> command alias takes zero or more arguments that correspond to resources or features that you are looking to request a job for and tells you what nodes could '''theoretically'''[0,1] run a job with these arguments immediately. It assumes your job is a single-node job.

These arguments are:
* <code>--partition</code>: Only include nodes in the specified partition(s).
* <code>--account</code>: Only include nodes from partitions that can use the specified account(s).
* <code>--qos</code>: Only include nodes from partitions that can use the specified QoS(es).
* <code>--cpus</code>: Only include nodes with at least this many CPUs free.
* <code>--mem</code>: Only include nodes with at least this much memory free. The default unit is MB if unspecified, but any of {K,M,G,T} can be suffixed to the number provided (will then be interpreted as KB, MB, GB, or TB, respectively).
* GRES-related arguments:
** <code>--or-gres</code>: Only include nodes whose list of GRES contains any of the specified GRES type/quantity pairings.
** <code>--and-gres</code>: Only include nodes whose list of GRES contains all of the specified GRES type/quantity pairings. Functionally identical to <tt>--or-gres</tt> if only one GRES type/quantity pairing is specified.
* GPU-related arguments:
** <code>--or-gpus</code>: Only include nodes whose list of GPUs (a subset of GRES) contains any of the specified GPU type/quantity pairings.
** <code>--and-gpus</code>: Only include nodes whose list of GPUs (a subset of GRES) contains all of the specified GPU type/quantity pairings. Functionally identical to <tt>--or-gpus</tt> if only one GPU type/quantity pairing is specified.
* Feature-related arguments:
** <code>--or-feature</code>: Only include nodes whose list of features contains any of the specified feature(s).
** <code>--and-feature</code>: Only include nodes whose list of features contains all of the specified feature(s). Functionally identical to <tt>--or-feature</tt> if only one feature is specified.

These arguments are also viewable by running <code>show_available_nodes -h</code>.

===Examples===
TODO

===Footnotes===
[0] - As of now, this command alias does not factor in resources occupied by jobs that could be preempted (based on the partition(s) passed to it, if present). This is soon to come.

[1] - This command alias also does not factor in potentially higher priority jobs in the same partition(s) blocking execution of a job submitted with the arguments checked by the command alias. This is due to the complexity of calculating a job's priority before it is actually submitted.

=Requesting GPUs=
If you need to do processing on a GPU, you will need to request that your job have access to GPUs just as you need to request processors or CPU cores. In SLURM, GPUs are considered "generic resources" also known as GRES. To request some number of GPUs be reserved/available for your job, you can use the flag <code>--gres=gpu:#</code> (with the actual number of GPUs you want). If there are multiple types of GPUs available in the cluster and you need a specific type, you can provide the type option to the gres flag e.g. <code>--gres=gpu:rtxa5000:#</code>. If you do not request a specific type of GPU, you are likely to be scheduled on an older, lower spec'd GPU.

Note that some QoSes may have limits on the number of GPUs you can request per job, so you may need to specify a different QoS to request more GPUs.

<pre>
$ srun --pty --qos=medium --gres=gpu:2 nvidia-smi
...
Wed Mar 6 16:59:39 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 2080 Ti Off | 00000000:3D:00.0 Off | N/A |
| 32% 23C P8 1W / 250W | 0MiB / 11264MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce RTX 2080 Ti Off | 00000000:40:00.0 Off | N/A |
| 32% 25C P8 1W / 250W | 0MiB / 11264MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
</pre>

Please note that your job will only be able to see/access the GPUs you requested. If you only need 1 GPU, please only request 1 GPU. The others on the node (if any) will be left available for other users.

<pre>
$ srun --pty --gres=gpu:rtxa5000:1 nvidia-smi
Thu Aug 25 15:22:15 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.129.06 Driver Version: 470.129.06 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA RTX A5000 Off | 00000000:01:00.0 Off | Off |
| 30% 23C P8 20W / 230W | 0MiB / 24256MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
</pre>

As with all other flags, the <code>--gres</code> flag may also be passed to [[#sbatch | sbatch]] and [[#salloc | salloc]] rather than directly to [[#srun | srun]].

=MPI example=
To run [https://en.wikipedia.org/wiki/Message_Passing_Interface MPI] jobs, you will need to include the <code>--mpi=pmix</code> flag in your submission arguments.

<pre>
#!/usr/bin/bash
#SBATCH --job-name=mpi_test # Job name
#SBATCH --nodes=4 # Number of nodes
#SBATCH --ntasks=8 # Number of MPI ranks
#SBATCH --ntasks-per-node=2 # Number of MPI ranks per node
#SBATCH --ntasks-per-socket=1 # Number of tasks per processor socket on the node
#SBATCH --time=00:30:00 # Time limit hrs:min:sec

srun --mpi=pmix /nfshomes/username/testing/mpi/a.out
</pre>

Nexus/GPUs

2023-12-06T21:41:17Z

Seide:

There are several different types of [https://www.nvidia.com/en-us/ NVIDIA] GPUs in the [[Nexus]] cluster that are available to be scheduled. They are listed below in order of newest to oldest architecture, and then alphabetically.

We do not list the exact quantities of GPU here since they change frequently due to additions to or removals from the cluster or during compute node troubleshooting. To see which compute nodes have which GPUs and in what quantities, use the <code>show_nodes</code> command on a submission or compute node. The quantities are listed under the <tt>GRES</tt> column.

{| class="wikitable sortable"
! Name
! [https://www.nvidia.com/en-us/technologies Architecture]
! [https://developer.nvidia.com/cuda-toolkit CUDA] Cores
! GPU Memory
! Memory Bandwidth
! FP32 Performance (TFLOPS)
! [https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/ TF32] Performance ([https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt Dense / Sparse TOPS])
|-
| GeForce RTX 3070
| Ampere
| 5888
| 8GB GDDR6
| 448GB/s
| 20.3
| 20.3/40.6
|-
| GeForce RTX 3090
| Ampere
| 10496
| 24GB GDDR6X
| 936GB/s
| 35.6
| 35.6/71
|-
| RTX A4000
| Ampere
| 6144
| 16GB GDDR6
| 448GB/s
| 19.2
| not officially published
|-
| RTX A5000
| Ampere
| 8192
| 24GB GDDR6
| 768GB/s
| 27.8
| not officially published
|-
| RTX A6000
| Ampere
| 10752
| 48GB GDDR6
| 768GB/s
| 38.7
| 77.4/154.8
|-
| A100
| Ampere
| 6912
| 80GB HBM2e
| 1935 GB/s
| 19.5
| 156/312
|-
| GeForce RTX 2080 Ti
| Turing
| 4352
| 11GB GDDR5X
| 616GB/s
| 13.4
| n/a
|-
| GeForce GTX 1080 Ti
| Pascal
| 3584
| 11GB GDDR5X
| 484GB/s
| 11.3
| n/a
|-
| GeForce GTX Titan Xp
| Pascal
| 3840
| 12GB GDDR5X
| 548GB/s
| 12.1
| n/a
|-
| Quadro P6000
| Pascal
| 3840
| 24GB GDDR5X
| 432GB/s
| 12.6
| n/a
|-
| Tesla P100
| Pascal
| 3584
| 16GB CoWoS HBM2
| 732GB/s
| 9.3
| n/a
|-
| GeForce GTX Titan X
| Maxwell
| 3072
| 12GB GDDR5
| 336GB/s
| 6.7
| n/a
|-
| Tesla M40
| Maxwell
| 3072
| 12GB GDDR5
| 288GB/s
| 6.8
| n/a
|-
| Tesla K80
| Kepler
| 4992
| 24GB GDDR5
| 480GB/s
| 8.7
| n/a
|-
|}

Nexus/CML

2023-09-22T15:50:08Z

Seide: /* QoS */

The [[CML]] standalone cluster's compute nodes have folded into [[Nexus]] as of the scheduled [[MonthlyMaintenanceWindow | maintenance window]] for August 2023 (Thursday 08/17/2023, 5-8pm).

The Nexus cluster already has a large pool of compute resources made possible through leftover funding for the [[Iribe | Brendan Iribe Center]]. Details on common nodes already in the cluster (Tron partition) can be found [[Nexus/Tron | here]].

The CML cluster's standalone submission node <code>cmlsub00.umiacs.umd.edu</code> will be retired on Thursday, September 21st, 2023 during that month's maintenance window (5-8pm), as it is no longer able to submit jobs to CML compute nodes. Please use <code>nexuscml00.umiacs.umd.edu</code> and <code>nexuscml01.umiacs.umd.edu</code> for any general purpose CML compute needs after this time.

Please see the [[#Timeline | Timeline]] section below for concrete dates in chronological order.

Please [[HelpDesk | contact staff]] with any questions or concerns.

==Usage==
The Nexus cluster submission nodes that are allocated to CML are <code>nexuscml00.umiacs.umd.edu</code> and <code>nexuscml01.umiacs.umd.edu</code>. '''You must use these nodes to submit jobs to CML compute nodes.''' Submission from <code>cmlsub00.umiacs.umd.edu</code> is no longer available.

All partitions, QoSes, and account names from the standalone CML cluster have been moved over to Nexus. However, please note that <code>cml-</code> is prepended to all of the values that were present in the standalone CML cluster to distinguish them from existing values in Nexus. The lone exception is the base account that was named <code>cml</code> in the standalone cluster (it is also named just <code>cml</code> in Nexus).

Here are some before/after examples of job submission with various parameters:

{| class="wikitable"
! Standalone CML cluster submission command
! Nexus cluster submission command
|-
|<code>srun --partition=dpart --qos=medium --account=tomg --gres=gpu:rtx2080ti:2 --pty bash</code>
|<code>srun --partition=cml-dpart --qos=cml-medium --account=cml-tomg --gres=gpu:rtx2080ti:2 --pty bash</code>
|-
|<code>srun --partition=cpu --qos=cpu --pty bash</code>
|<code>srun --partition=cml-cpu --qos=cml-cpu --account=cml --pty bash</code>
|-
|<code>srun --partition=scavenger --qos=scavenger --account=scavenger --gres=gpu:4 --pty bash</code>
|<code>srun --partition=cml-scavenger --qos=cml-scavenger --account=cml-scavenger --gres=gpu:4 --pty bash</code>
|}

CML users (exclusively) can schedule non-interruptible jobs on CML nodes with any non-scavenger job parameters. Please note that the <code>cml-dpart</code> partition has a <code>GrpTRES</code> limit of 100% of the available cores/RAM on all cml## nodes in aggregate plus 50% of the available cores/RAM on legacy## nodes in aggregate, so your job may need to wait if all available cores/RAM (or GPUs) are in use. It also has a max submission limit of 500 jobs per user simultaneously so as to not overload the cluster. This is codified by the partition QoS named '''cml'''.

Please note that the CML compute nodes are also in the institute-wide <code>scavenger</code> partition in Nexus. CML users still have scavenging priority over these nodes via the <code>cml-scavenger</code> partition (i.e., all <code>cml-</code> partition jobs (other than <code>cml-scavenger</code>) can preempt both <code>cml-scavenger</code> and <code>scavenger</code> partition jobs, and <code>cml-scavenger</code> partition jobs can preempt <code>scavenger</code> partition jobs).

==Partitions==
There are three partitions available to general CML [[SLURM]] users. You must specify a partition when submitting your job.

* '''cml-dpart''' - This is the default partition. Job allocations are guaranteed.
* '''cml-scavenger''' - This is the alternate partition that allows jobs longer run times and more resources but is preemptable when jobs in other <code>cml-</code> partitions are ready to be scheduled.
* '''cml-cpu''' - This partition is for CPU focused jobs. Job allocations are guaranteed.

There is one additional partition available solely to Dr. Furong Huang's sponsored accounts.

* '''cml-furongh''' - This partition is for exclusive priority access to Dr. Huang's purchased A6000 node.

==Accounts==
The Center has a base SLURM account <code>cml</code> which has a modest number of guaranteed billing resources available to all cluster users at any given time. Other faculty that have invested in the cluster have an additional account provided to their sponsored accounts on the cluster, which provides a number of guaranteed billing resources corresponding to the amount that they invested.

If you do not specify an account when submitting your job, you will receive the <code>cml</code> account. If your faculty sponsor has their own account, it is recommended to use that account for job submission.

The current faculty accounts are:
* cml-abhinav
* cml-cameron
* cml-furongh
* cml-hajiagha
* cml-john
* cml-ramani
* cml-sfeizi
* cml-tokekar
* cml-tomg
* cml-zhou

<pre>
$ sacctmgr show account format=account%20,description%30,organization%10
Account Descr Org
-------------------- ------------------------------ ----------
... ... ...
cml cml cml
cml-abhinav cml - abhinav shrivastava cml
cml-cameron cml - maria cameron cml
cml-furongh cml - furong huang cml
cml-hajiagha cml - mohammad hajiaghayi cml
cml-john cml - john dickerson cml
cml-ramani cml - ramani duraiswami cml
cml-scavenger cml - scavenger cml
cml-sfeizi cml - soheil feizi cml
cml-tokekar cml - pratap tokekar cml
cml-tomg cml - tom goldstein cml
cml-zhou cml - tianyi zhou cml
... ... ...
</pre>

Faculty can manage this list of users via our [https://intranet.umiacs.umd.edu/directory/secgroup/ Directory application] in the Security Groups section. The security group that controls access has the prefix <code>cml_</code> and then the faculty username. It will also list <code>slurm://nexusctl.umiacs.umd.edu</code> as the associated URI.

You can check your account associations by running the '''show_assoc''' command to see the accounts you are associated with. Please [[HelpDesk | contact staff]] and include your faculty member in the conversation if you do not see the appropriate association.

<pre>
$ show_assoc
User Account MaxJobs GrpTRES QOS
---------- ---------------- ------- ------------- --------------------------------------------------
... ... ...
tomg cml cml-cpu,cml-default,cml-medium
tomg cml-scavenger cml-scavenger
tomg cml-tomg cml-default,cml-high,cml-medium
... ... ...
</pre>

You can also see the total number of Track-able Resources (TRES) allowed for each account by running the following command. Please make sure you give the appropriate account that you are looking for. The billing number displayed here is the sum of [[SLURM/Priority#Modern | resource weightings]] for all nodes appropriated to that account.

<pre>
$ sacctmgr show assoc account=cml format=user,account,qos,grptres
User Account QOS GrpTRES
---------- ---------- -------------------- -------------
cml billing=7732
... ...
</pre>

==QoS==
CML currently has 5 QoS for the '''cml-dpart''' partition (though <code>high_long</code> and <code>very_high</code> may not be available to all faculty accounts), 1 QoS for the '''cml-scavenger''' partition, and 1 QoS for the '''cml-cpu''' partition. If you do not specify a QoS when submitting your job using the <code>--qos</code> parameter, you will receive the '''cml-default''' QoS assuming you are using a CML account.

The important part here is that in different QoS you can have a shorter/longer maximum wall time, a different total number of jobs running at once, and a different maximum number of track-able resources (TRES) for the job. In the cml-scavenger QoS, one more constraint that you are restricted by is the total number of TRES per user (over multiple jobs).

<pre>
$ show_qos
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
...
cml-cpu 7-00:00:00 8
cml-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
cml-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
cml-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
cml-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
cml-scavenger 3-00:00:00 gres/gpu=24
cml-very_high 1-12:00:00 cpu=32,gres/gpu=8,mem=256G 8 gres/gpu=12
...

$ show_partition_qos
Name MaxSubmitPU MaxTRESPU GrpTRES
-------------------- ----------- ------------------------------ --------------------
...
cml 500 cpu=1128,mem=11381G
cml-scavenger 500 gres/gpu=24
...
</pre>

==Storage==
There are 3 types of user storage available to users in the CML:
* Home directories
* Project directories
* Scratch directories

There are also 2 types of read-only storage available for common use among users in the CML:
* Dataset directories
* Model directories

CML users can also request [[Nexus#Project_Allocations | Nexus project allocations]].

===Home Directories===
Home directories in the CML computational infrastructure are available from the Institute's [[NFShomes]] as <code>/nfshomes/USERNAME</code> where USERNAME is your username. These home directories have very limited storage (30GB, cannot be increased) and are intended for your personal files, configuration and source code. Your home directory is '''not''' intended for data sets or other large scale data holdings. Users are encouraged to utilize our [[GitLab]] infrastructure to host your code repositories.

'''NOTE''': To check your quota on this directory you will need to use the <code>quota -s</code> command.

Your home directory data is fully protected and has both [[Snapshots | snapshots]] and is [[NightlyBackups | backed up nightly]].

===Project Directories===
You can request project based allocations for up to 6TB for up to 120 days with approval from a CML faculty member and the director of CML.

To request an allocation, please [[HelpDesk | contact staff]] with the faculty member(s) that the project is under involved in the conversation. Please include the following details:
* Project Name (short)
* Description
* Size (1TB, 2TB, etc.)
* Length in days (30 days, 90 days, etc.)
* Other user(s) that need to access the allocation, if any

These allocations will be available from '''/fs/cml-projects''' under a name that you provide when you request the allocation. Near the end of the allocation period, staff will contact you and ask if you would like to renew the allocation for up to another 120 days (requires re-approval from a CML faculty member and the director of CML). If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation. If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible. If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible. If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

This data is backed up nightly.

===Scratch Directories===
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the CML compute infrastructure:
* Network scratch directory
* Local scratch directories

====Network Scratch Directory====
You are allocated 400GB of scratch space via NFS from <code>/cmlscratch/$username</code>. '''It is not backed up or protected in any way.''' This directory is '''automounted''' so you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access this.

You may request a permanent increase of up to 800GB total space without any faculty approval by [[HelpDesk | contacting staff]]. If you need space beyond 800GB, you will need faculty approval and/or a project directory. Space increases beyond 800GB also have a maximum request period of 120 days (as with project directories), after which they will need to be renewed with re-approval from a CML faculty member and the director of CML.

This file system is available on all submission, data management, and computational nodes within the cluster.

====Local Scratch Directories====
Each computational node that you can schedule compute jobs on has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. These are almost always more performant than any other storage available to the job. However, you must stage data to these directories within the confines of your jobs and stage the data out before the end of your jobs.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month during our monthly maintenance windows. Again, please make sure you secure any data you write to these directories at the end of your job.

===Datasets===
We have read-only dataset storage available at <code>/fs/cml-datasets</code>. If there are datasets that you would like to see curated and available, please see [[Datasets | this page]].

The following is the list of datasets available:
{| class="wikitable"
! Dataset
! Path
|-
| CelebA
| /fs/cml-datasets/CelebA
|-
| CelebA-HQ
| /fs/cml-datasets/CelebA-HQ
|-
| CelebAMask-HQ
| /fs/cml-datasets/CelebAMask-HQ
|-
| Charades
| /fs/cml-datasets/Charades
|-
| Cityscapes
| /fs/cml-datasets/cityscapes
|-
| COCO
| /fs/cml-datasets/coco
|-
| Diversity in Faces [1]
| /fs/cml-datasets/diversity_in_faces
|-
| FFHQ
| /fs/cml-datasets/FFHQ
|-
| ImageNet ILSVRC2012
| /fs/cml-datasets/ImageNet/ILSVRC2012
|-
| LFW
| /fs/cml-datasets/facial_test_data
|-
| LibriSpeech
| /fs/cml-datasets/LibriSpeech
|-
| LSUN
| /fs/cml-datasets/LSUN
|-
| MAG240M
| /fs/cml-datasets/OGB/MAG240M
|-
| MegaFace
| /fs/cml-datasets/megaface
|-
| MS-Celeb-1M
| /fs/cml-datasets/MS_Celeb_aligned_112
|-
| OC20
| /fs/cml-datasets/OC20
|-
| ogbn-papers100M
| /fs/cml-datasets/OGB/ogbn-papers100M
|-
| roberta
| /fs/cml-datasets/roberta
|-
| Salient ImageNet
| /fs/cml-datasets/Salient-ImageNet
|-
| ShapeNetCore.v2
| /fs/cml-datasets/ShapeNetCore.v2
|-
| Tiny ImageNet
| /fs/cml-datasets/tiny_imagenet
|-
| WikiKG90M
| /fs/cml-datasets/OGB/WikiKG90M
|}

[1] - This dataset has restricted access. Please [[HelpDesk | contact staff]] if you are looking to use this dataset.

===Models===
We have read-only model storage available at <code>/fs/cml-models</code>. If there are models that you would like to see downloaded and made available, please see [[Datasets | this page]].

==Timeline==
Each event will be completed within the timeframe specified.

{| class="wikitable"
! Date
! Event
|-
| July 14th 2023
| A single compute node <code>cml03</code> is moved into Nexus so submission can be tested
|-
| [[MonthlyMaintenanceWindow | August 17th 2023, 5-8pm]]
| All other standalone CML cluster compute nodes are moved into Nexus in corresponding <code>cml-</code> named partitions
|-
| [[MonthlyMaintenanceWindow | September 21st 2023, 5-8pm]]
| <code>cmlsub00.umiacs.umd.edu</code> is taken offline
|-
|}

Nexus/Vulcan

2023-09-22T15:46:20Z

Seide: /* QoS */

The [https://wiki.umiacs.umd.edu/cfar/index.php/Vulcan Vulcan] standalone cluster's compute nodes have folded into [[Nexus]] as of the scheduled [[MonthlyMaintenanceWindow | maintenance window]] for August 2023 (Thursday 08/17/2023, 5-8pm).

The Nexus cluster already has a large pool of compute resources made possible through leftover funding for the [[Iribe | Brendan Iribe Center]]. Details on common nodes already in the cluster (Tron partition) can be found [[Nexus/Tron | here]].

The Vulcan cluster's standalone submission nodes <code>vulcansub00.umiacs.umd.edu</code> and <code>vulcansub01.umiacs.umd.edu</code> will be retired on Thursday, September 21st, 2023 during that month's maintenance window (5-8pm), as they are no longer able to submit jobs to Vulcan compute nodes. Please use <code>nexusvulcan00.umiacs.umd.edu</code> and <code>nexusvulcan01.umiacs.umd.edu</code> for any general purpose Vulcan compute needs after this time.

Please see the [[#Timeline | Timeline]] section below for concrete dates in chronological order.

Please [[HelpDesk | contact staff]] with any questions or concerns.

==Usage==
The Nexus cluster submission nodes that are allocated to Vulcan are <code>nexusvulcan00.umiacs.umd.edu</code> and <code>nexusvulcan01.umiacs.umd.edu</code>. '''You must use these nodes to submit jobs to Vulcan compute nodes.''' Submission from <code>vulcansub00.umiacs.umd.edu</code> or <code>vulcansub01.umiacs.umd.edu</code> is no longer available.

All partitions, QoSes, and account names from the standalone Vulcan cluster have been moved over to Nexus. However, please note that <code>vulcan-</code> is prepended to all of the values that were present in the standalone Vulcan cluster to distinguish them from existing values in Nexus. The lone exception is the base account that was named <code>vulcan</code> in the standalone cluster (it is also named just <code>vulcan</code> in Nexus).

Here are some before/after examples of job submission with various parameters:

{| class="wikitable"
! Standalone Vulcan cluster submission command
! Nexus cluster submission command
|-
|<code>srun --partition=dpart --qos=medium --account=abhinav --gres=gpu:rtxa4000:2 --pty bash</code>
|<code>srun --partition=vulcan-dpart --qos=vulcan-medium --account=vulcan-abhinav --gres=gpu:rtxa4000:2 --pty bash</code>
|-
|<code>srun --partition=cpu --qos=cpu --pty bash</code>
|<code>srun --partition=vulcan-cpu --qos=vulcan-cpu --account=vulcan --pty bash</code>
|-
|<code>srun --partition=scavenger --qos=scavenger --account=vulcan --gres=gpu:4 --pty bash</code>
|<code>srun --partition=vulcan-scavenger --qos=vulcan-scavenger --account=vulcan --gres=gpu:4 --pty bash</code>
|}

Vulcan users (exclusively) can schedule non-interruptible jobs on Vulcan nodes with any non-scavenger job parameters. Please note that the <code>vulcan-dpart</code> partition has a <code>GrpTRES</code> limit of 100% of the available cores/RAM on all vulcan## in aggregate nodes plus 50% of the available cores/RAM on legacy## nodes in aggregate, so your job may need to wait if all available cores/RAM (or GPUs) are in use. It also has a max submission limit of 500 jobs per user simultaneously so as to not overload the cluster. This is codified by the partition QoS named '''vulcan'''.

Please note that the Vulcan compute nodes are also in the institute-wide <code>scavenger</code> partition in Nexus. Vulcan users still have scavenging priority over these nodes via the <code>vulcan-scavenger</code> partition (i.e., all <code>vulcan-</code> partition jobs (other than <code>vulcan-scavenger</code>) can preempt both <code>vulcan-scavenger</code> and <code>scavenger</code> partition jobs, and <code>vulcan-scavenger</code> partition jobs can preempt <code>scavenger</code> partition jobs).

==Nodes==
There are currently 45 [[Nexus/Vulcan/GPUs | GPU nodes]] available running a mixture of NVIDIA RTX A6000, NVIDIA RTX A5000, NVIDIA RTX A4000, NVIDIA Quadro P6000, NVIDIA GeForce GTX 1080 Ti, NVIDIA GeForce RTX 2080 Ti, and NVIDIA Tesla P100 cards. There are also 2 CPU-only nodes available.

All nodes are scheduled with the [[SLURM]] resource manager.

==Partitions==
There are three partitions available to general Vulcan [[SLURM]] users. You must specify a partition when submitting your job.

* '''vulcan-dpart''' - This is the default partition. Job allocations are guaranteed.
* '''vulcan-scavenger''' - This is the alternate partition that allows jobs longer run times and more resources but is preemptable when jobs in other <code>vulcan-</code> partitions are ready to be scheduled.
* '''vulcan-cpu''' - This partition is for CPU focused jobs. Job allocations are guaranteed.

There are a few additional partitions available to subsets of Vulcan users based on specific requirements.

==Accounts==
Vulcan has a base SLURM account <code>vulcan</code> which has a modest number of guaranteed billing resources available to all cluster users at any given time. Other faculty that have invested in Vulcan compute infrastructure have an additional account provided to their sponsored accounts on the cluster, which provides a number of guaranteed billing resources corresponding to the amount that they invested.

If you do not specify an account when submitting your job, you will receive the <code>vulcan</code> account. If your faculty sponsor has their own account, it is recommended to use that account for job submission.

The current faculty accounts are:
* vulcan-abhinav
* vulcan-djacobs
* vulcan-jbhuang
* vulcan-lsd
* vulcan-metzler
* vulcan-rama
* vulcan-ramani
* vulcan-yaser
* vulcan-zwicker

<pre>
$ sacctmgr show account format=account%20,description%30,organization%10
Account Descr Org
-------------------- ------------------------------ ----------
... ... ...
vulcan vulcan vulcan
vulcan-abhinav vulcan - abhinav shrivastava vulcan
vulcan-djacobs vulcan - david jacobs vulcan
vulcan-jbhuang vulcan - jia-bin huang vulcan
vulcan-lsd vulcan - larry davis vulcan
vulcan-metzler vulcan - chris metzler vulcan
vulcan-rama vulcan - rama chellappa vulcan
vulcan-ramani vulcan - ramani duraiswami vulcan
vulcan-yaser vulcan - yaser yacoob vulcan
vulcan-zwicker vulcan - matthias zwicker vulcan
... ... ...
</pre>

Faculty can manage this list of users via our [https://intranet.umiacs.umd.edu/directory/secgroup/ Directory application] in the Security Groups section. The security group that controls access has the prefix <code>vulcan_</code> and then the faculty username. It will also list <code>slurm://nexusctl.umiacs.umd.edu</code> as the associated URI.

You can check your account associations by running the '''show_assoc''' command to see the accounts you are associated with. Please [[HelpDesk | contact staff]] and include your faculty member in the conversation if you do not see the appropriate association.

<pre>
$ show_assoc
User Account MaxJobs GrpTRES QOS
---------- ---------------- ------- ------------- --------------------------------------------------------------------------------
... ... ... ...
abhinav abhinav 48 vulcan-cpu,vulcan-default,vulcan-high,vulcan-medium,vulcan-scavenger
abhinav vulcan 48 vulcan-cpu,vulcan-default,vulcan-medium,vulcan-scavenger
... ... ... ...
</pre>

You can also see the total number of Track-able Resources (TRES) allowed for each account by running the following command. Please make sure you give the appropriate account that you are looking for. As shown below, there is a concurrent limit of 64 total GPUs for all users not in a contributing faculty group.

<pre>
$ sacctmgr show assoc account=vulcan format=user,account,qos,grptres
User Account QOS GrpTRES
---------- ---------- -------------------- -------------
vulcan gres/gpu=64
... ...
</pre>

==QoS==
You need to decide the QOS to submit with which will set a certain number of restrictions to your job. If you do not specify a QoS when submitting your job using the <code>--qos</code> parameter, you will receive the '''vulcan-default''' QoS assuming you are using a Vulcan account.

The following <code>sacctmgr</code> command will list the current QOS. Either the <code>vulcan-default</code>, <code>vulcan-medium</code>, or <code>vulcan-high</code> QOS is required for the vulcan-dpart partition. Please note that only faculty accounts (see above) have access to the <code>vulcan-high</code> QoS.

The following example will show you the current limits that the QOS have. The output is truncated to show only relevant Vulcan QoS.
<pre>
$ show_qos
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
...
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-exempt 7-00:00:00 cpu=32,gres/gpu=8,mem=256G 2
vulcan-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
vulcan-janus 3-00:00:00 cpu=32,gres/gpu=10,mem=256G
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-sailon 3-00:00:00 cpu=32,gres/gpu=8,mem=256G gres/gpu=48
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
...

$ show_partition_qos
Name MaxSubmitPU MaxTRESPU GrpTRES
-------------------- ----------- ------------------------------ --------------------
...
vulcan 500 cpu=1760,mem=15824G
vulcan-scavenger 500
...
</pre>

==Storage==
Vulcan has the following storage available. Please also review UMIACS [[LocalDataStorage | Local Data Storage]] policies including any volume that is labeled as scratch.

Vulcan users can also request [[Nexus#Project_Allocations | Nexus project allocations]].

===Home Directory===
You have 30GB of storage available at <code>/nfshomes/<username></code>. It has both [[Snapshots]] and [[TSM | Backups]] available if need be.

Home directories are intended to store personal or configuration files only. We encourage users to not share any data in their home directory.

===Scratch Directories===
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Vulcan compute infrastructure:
* Network scratch directory
* Local scratch directories

====Network Scratch Directory====
You are allocated 300GB of scratch space via NFS from <code>/vulcanscratch/$username</code>. '''It is not backed up or protected in any way.''' This directory is '''automounted''' so you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access this.

You may request a temporary increase of up to 500GB total space for a maximum of 120 days without any faculty approval by contacting [mailto:staff@umiacs.umd.edu staff@umiacs.umd.edu]. Once the temporary increase period is over, you will be contacted and given a one-week window of opportunity to clean and secure your data before staff will forcibly remove data to get your space back under 300GB. If you need space beyond 500GB or for longer than 120 days, you will need faculty approval and/or a project directory.

This file system is available on all submission, data management, and computational nodes within the cluster.

====Local Scratch Directories====
Each computational node that you can schedule compute jobs on has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. These are almost always more performant than any other storage available to the job. However, you must stage their data within the confine of their job and stage the data out before the end of their job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month at 1am. Different nodes will run the maintenance jobs on different days of the month to ensure the cluster is still highly available at all times. Please make sure you secure any data you write to these directories at the end of your job.

===Datasets===
We have read-only dataset storage available at <code>/fs/vulcan-datasets</code>. If there are datasets that you would like to see curated and available, please see [[Datasets | this page]].

The following is the list of datasets available:
{| class="wikitable"
! Dataset
! Path
|-
| 3D-FRONT
| /fs/vulcan-datasets/3d-front
|-
| 3D-FUTURE
| /fs/vulcan-datasets/3d-future
|-
| Action Genome
| /fs/vulcan-datasets/AG
|-
| ActivityNet
| /fs/vulcan-datasets/ActivityNet
|-
| CATER
| /fs/vulcan-datasets/CATER
|-
| COVID-DA
| /fs/vulcan-datasets/COVID-DA
|-
| CelebA
| /fs/vulcan-datasets/CelebA
|-
| CelebA-HQ
| /fs/vulcan-datasets/CelebA-HQ
|-
| CelebAMask-HQ
| /fs/vulcan-datasets/CelebAMask-HQ
|-
| Charades
| /fs/vulcan-datasets/Charades
|-
| CharadesEgo
| /fs/vulcan-datasets/CharadesEgo
|-
| CIFAR10
| /fs/vulcan-datasets/cifar-10-python
|-
| CIFAR100
| /fs/vulcan-datasets/cifar-100-python
|-
| CityScapes
| /fs/vulcan-datasets/cityscapes
|-
| COCO
| /fs/vulcan-datasets/coco
|-
| Conceptual Captions
| /fs/vulcan-datasets/conceptual_captions
|-
| CUB
| /fs/vulcan-datasets/CUB
|-
| DeepFashion
| /fs/vulcan-datasets/DeepFashion
|-
| Digits
| /fs/vulcan-datasets/digits_full
|-
| Edges2handbags
| /fs/vulcan/datasets/edges2handbags
|-
| Edges2shoes
| /fs/vulcan/datasets/edges2shoes
|-
| EGTEA
| /fs/vulcan/datasets/EGTEA
|-
| emnist
| /fs/vulcan-datasets/emnist
|-
| EPIC Kitchens 2018
| /fs/vulcan-datasets/Epics-kitchen-2018
|-
| EPIC Kitchens 2020
| /fs/vulcan-datasets/EPIC-Kitchens-2020
|-
| Facades
| /fs/vulcan/datasets/facades
|-
| from_games (GTA5)
| /fs/vulcan-datasets/from_games
|-
| FFHQ
| /fs/vulcan-datasets/ffhq-dataset
|-
| FineGym
| /fs/vulcan-datasets/FineGym
|-
| Google Landmarks Dataset v2
| /fs/vulcan-datasets/google-landmark-v2
|-
| HAA500
| /fs/vulcan-datasets/haa500
|-
| HICO
| /fs/vulcan-datasets/HICO
|-
| HMDB51
| /fs/vulcan-datasets/HMDB51
|-
| Honda_100h
| /fs/vulcan-datasets/honda_100h
|-
| HPatches
| /fs/vulcan-datasets/HPatches
|-
| Human3.6M
| /fs/vulcan-datasets/human3.6
|-
| IM2GPS (test only)
| /fs/vulcan-datasets/im2gps
|-
| ImageNet
| /fs/vulcan-datasets/imagenet
|-
| iNaturalist Dataset 2021
| /fs/vulcan-datasets/inat_comp_2021
|-
| InteriorNet
| /fs/vulcan-datasets/InteriorNet
|-
| Kinetics-400
| /fs/vulcan-datasets/Kinetics-400
|-
| Labelled Faces in the Wild
| /fs/vulcan-datasets/lfw
|-
| LibriSpeech
| /fs/vulcan-datasets/LibriSpeech
|-
| LSUN
| /fs/vulcan-datasets/LSUN
|-
| LVIS
| /fs/vulcan-datasets/LVIS
|-
| Maps
| /fs/vulcan-datasets/maps
|-
| Matterport3D
| /fs/vulcan-datasets/Matterport3D
|-
| MegaDepth
| /fs/vulcan-datasets/MegaDepth
|-
| MineRL
| /fs/vulcan-datasets/MineRL
|-
| Mini-ImageNet
| /fs/vulcan-datasets/miniImagenet
|-
| MIT Indoor
| /fs/vulcan-datasets/mit_indoor
|-
| MIT Places
| /fs/vulcan-datasets/mit_places
|-
| Multi-PIE Face
| /fs/vulcan-datasets/multipie
|-
| Night2day
| /fs/vulcan-datasets/night2day
|-
| ObjectNet3D
| /fs/vulcan-datasets/ObjectNet3D
|-
| Occluded Video Instance Segmentation
| /fs/vulcan-datasets/ovis-2021
|-
| Office
| /fs/vulcan-datasets/office
|-
| Office-Home
| /fs/vulcan-datasets/office_home
|-
| omniglot
| /fs/vulcan-datasets/omniglot
|-
| OOPS
| /fs/vulcan-datasets/OOPS
|-
| OpenImagesv4
| /fs/vulcan-datasets/OpenImagesv4
|-
| PartNet
| /fs/vulcan-datasets/PartNet
|-
| Pascal VOC
| /fs/vulcan-datasets/pascal_voc
|-
| PIC (HOI-A)
| /fs/vulcan-datasets/PIC
|-
| PubLayNet
| /fs/vulcan-datasets/PubLayNet
|-
| Replica
| /fs/vulcan-datasets/Replica
|-
| ScanNet
| /fs/vulcan-datasets/ScanNet
|-
| ShapeNetCore.v2
| /fs/vulcan-datasets/ShapeNetCore.v2
|-
| Something-Something-V1
| /fs/vulcan-datasets/SomethingV1
|-
| Something-Something-V2
| /fs/vulcan-datasets/SomethingV2
|-
| SYNTHIA-RAND-CITYSCAPES
| /fs/vulcan-datasets/SYNTHIA-RAND-CITYSCAPES
|-
| TAPOS
| /fs/vulcan-datasets/TAPOS
|-
| Tiny ImageNet
| /fs/vulcan-datasets/tiny_imagenet
|-
| Tumblr GIF Description
| /fs/vulcan-datasets/TGIF
|-
| Thingi10K
| /fs/vulcan-datasets/Thingi10K
|-
| UCF101
| /fs/vulcan-datasets/UCF101
|-
| VirtualHomes
| /fs/vulcan-datasets/VirtualHomes
|-
| visda17
| /fs/vulcan-datasets/visda17
|-
| visda17_openset
| /fs/vulcan-datasets/VISDA
|-
| visda19
| /fs/vulcan-datasets/visda
|-
| Visual Genome
| /fs/vulcan-datasets/VG
|-
| Visual Relationship Detection
| /fs/vulcan-datasets/VRD
|-
| VOCdevkit
| /fs/vulcan-datasets/VOCdevkit
|-
| VoxCeleb2
| /fs/vulcan-datasets/VoxCeleb2
|-
| WILDS
| /fs/vulcan-datasets/WILDS
|-
| xView2
| /fs/vulcan-datasets/xView2
|-
| YCB Object Models
| /fs/vulcan-datasets/YCB
|-
| YouTube8M
| /fs/vulcan-datasets/YouTube8M
|-
| YouTubeVIS-2019
| /fs/vulcan-datasets/YouTubeVIS-2019
|-
| YouTubeVIS-2021
| /fs/vulcan-datasets/YouTubeVIS-2021
|}

===Project Storage===
Users within the Vulcan compute infrastructure can request project based allocations for up to 10TB for up to 180 days by [https://wiki.umiacs.umd.edu/umiacs/index.php/HelpDesk contacting staff] with approval from the Vulcan faculty manager (Dr. Shrivastava). These allocations will be available from <code>/fs/vulcan-projects</code> under a name that you provide when you request the allocation. Near the end of the allocation period, staff will contact you and ask if you would like to renew the allocation for up to another 180 days (requires re-approval from Dr. Shrivastava). If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation. If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible. If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible. If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

This data, by default, will be backed up nightly and have a limited snapshot schedule (1 daily snapshot). Upon request, staff can both exclude the data from backups and/or disable snapshots on the project storage volume. We currently have 100TB total to support these projects which includes the snapshot data for this volume.

===Object Storage===
All Vulcan users can request project allocations in the [https://obj.umiacs.umd.edu/obj/help UMIACS Object Store]. Please email staff@umiacs.umd.edu with a short project name and the amount of storage you will need to get started.

An example on how to use the umobj command line utilities can be found [https://wiki.umiacs.umd.edu/umiacs/index.php/UMobj/Example here]. A full set of documentation for the utilities can be found on the [https://gitlab.umiacs.umd.edu/staff/umobj/blob/master/README.md#umobj umobj Gitlab page].

==Timeline==
Each event will be completed within the timeframe specified.

{| class="wikitable"
! Date
! Event
|-
| July 21st 2023
| Two compute nodes <code>vulcan11</code> and <code>vulcan12</code> are moved into Nexus so submission can be tested
|-
| [[MonthlyMaintenanceWindow | August 17th 2023, 5-8pm]]
| All other standalone Vulcan cluster compute nodes are moved into Nexus in corresponding <code>vulcan-</code> named partitions
|-
| [[MonthlyMaintenanceWindow | September 21st 2023, 5-8pm]]
| <code>vulcansub00.umiacs.umd.edu</code> and <code>vulcansub01.umiacs.umd.edu</code> are taken offline
|-
|}

==Migration==
===Home Directories===
The [[Nexus]] uses [[NFShomes]] home directories - if your UMIACS account was created before February 22nd, 2023, you were using <code>/cfarhomes/<username></code> as your home directory on the standalone Vulcan cluster. While <code>/cfarhomes</code> is available on Nexus, your shell initialization scripts from it will not automatically load. Please copy over anything you need to your <code>/nfshomes/<username></code> directory at your earliest convenience, as <code>/cfarhomes</code> will be retired in a two phase process:
* Fri 11/17/2023, 5pm: cfarhomes directories are made read-only
* Thu 12/21/2023, 5pm ([[MonthlyMaintenanceWindow |monthly maintenance window]]): cfarhomes directories are taken offline

Nexus/Vulcan

2023-09-22T15:45:45Z

Seide: /* QoS */

The [https://wiki.umiacs.umd.edu/cfar/index.php/Vulcan Vulcan] standalone cluster's compute nodes have folded into [[Nexus]] as of the scheduled [[MonthlyMaintenanceWindow | maintenance window]] for August 2023 (Thursday 08/17/2023, 5-8pm).

The Nexus cluster already has a large pool of compute resources made possible through leftover funding for the [[Iribe | Brendan Iribe Center]]. Details on common nodes already in the cluster (Tron partition) can be found [[Nexus/Tron | here]].

The Vulcan cluster's standalone submission nodes <code>vulcansub00.umiacs.umd.edu</code> and <code>vulcansub01.umiacs.umd.edu</code> will be retired on Thursday, September 21st, 2023 during that month's maintenance window (5-8pm), as they are no longer able to submit jobs to Vulcan compute nodes. Please use <code>nexusvulcan00.umiacs.umd.edu</code> and <code>nexusvulcan01.umiacs.umd.edu</code> for any general purpose Vulcan compute needs after this time.

Please see the [[#Timeline | Timeline]] section below for concrete dates in chronological order.

Please [[HelpDesk | contact staff]] with any questions or concerns.

==Usage==
The Nexus cluster submission nodes that are allocated to Vulcan are <code>nexusvulcan00.umiacs.umd.edu</code> and <code>nexusvulcan01.umiacs.umd.edu</code>. '''You must use these nodes to submit jobs to Vulcan compute nodes.''' Submission from <code>vulcansub00.umiacs.umd.edu</code> or <code>vulcansub01.umiacs.umd.edu</code> is no longer available.

All partitions, QoSes, and account names from the standalone Vulcan cluster have been moved over to Nexus. However, please note that <code>vulcan-</code> is prepended to all of the values that were present in the standalone Vulcan cluster to distinguish them from existing values in Nexus. The lone exception is the base account that was named <code>vulcan</code> in the standalone cluster (it is also named just <code>vulcan</code> in Nexus).

Here are some before/after examples of job submission with various parameters:

{| class="wikitable"
! Standalone Vulcan cluster submission command
! Nexus cluster submission command
|-
|<code>srun --partition=dpart --qos=medium --account=abhinav --gres=gpu:rtxa4000:2 --pty bash</code>
|<code>srun --partition=vulcan-dpart --qos=vulcan-medium --account=vulcan-abhinav --gres=gpu:rtxa4000:2 --pty bash</code>
|-
|<code>srun --partition=cpu --qos=cpu --pty bash</code>
|<code>srun --partition=vulcan-cpu --qos=vulcan-cpu --account=vulcan --pty bash</code>
|-
|<code>srun --partition=scavenger --qos=scavenger --account=vulcan --gres=gpu:4 --pty bash</code>
|<code>srun --partition=vulcan-scavenger --qos=vulcan-scavenger --account=vulcan --gres=gpu:4 --pty bash</code>
|}

Vulcan users (exclusively) can schedule non-interruptible jobs on Vulcan nodes with any non-scavenger job parameters. Please note that the <code>vulcan-dpart</code> partition has a <code>GrpTRES</code> limit of 100% of the available cores/RAM on all vulcan## in aggregate nodes plus 50% of the available cores/RAM on legacy## nodes in aggregate, so your job may need to wait if all available cores/RAM (or GPUs) are in use. It also has a max submission limit of 500 jobs per user simultaneously so as to not overload the cluster. This is codified by the partition QoS named '''vulcan'''.

Please note that the Vulcan compute nodes are also in the institute-wide <code>scavenger</code> partition in Nexus. Vulcan users still have scavenging priority over these nodes via the <code>vulcan-scavenger</code> partition (i.e., all <code>vulcan-</code> partition jobs (other than <code>vulcan-scavenger</code>) can preempt both <code>vulcan-scavenger</code> and <code>scavenger</code> partition jobs, and <code>vulcan-scavenger</code> partition jobs can preempt <code>scavenger</code> partition jobs).

==Nodes==
There are currently 45 [[Nexus/Vulcan/GPUs | GPU nodes]] available running a mixture of NVIDIA RTX A6000, NVIDIA RTX A5000, NVIDIA RTX A4000, NVIDIA Quadro P6000, NVIDIA GeForce GTX 1080 Ti, NVIDIA GeForce RTX 2080 Ti, and NVIDIA Tesla P100 cards. There are also 2 CPU-only nodes available.

All nodes are scheduled with the [[SLURM]] resource manager.

==Partitions==
There are three partitions available to general Vulcan [[SLURM]] users. You must specify a partition when submitting your job.

* '''vulcan-dpart''' - This is the default partition. Job allocations are guaranteed.
* '''vulcan-scavenger''' - This is the alternate partition that allows jobs longer run times and more resources but is preemptable when jobs in other <code>vulcan-</code> partitions are ready to be scheduled.
* '''vulcan-cpu''' - This partition is for CPU focused jobs. Job allocations are guaranteed.

There are a few additional partitions available to subsets of Vulcan users based on specific requirements.

==Accounts==
Vulcan has a base SLURM account <code>vulcan</code> which has a modest number of guaranteed billing resources available to all cluster users at any given time. Other faculty that have invested in Vulcan compute infrastructure have an additional account provided to their sponsored accounts on the cluster, which provides a number of guaranteed billing resources corresponding to the amount that they invested.

If you do not specify an account when submitting your job, you will receive the <code>vulcan</code> account. If your faculty sponsor has their own account, it is recommended to use that account for job submission.

The current faculty accounts are:
* vulcan-abhinav
* vulcan-djacobs
* vulcan-jbhuang
* vulcan-lsd
* vulcan-metzler
* vulcan-rama
* vulcan-ramani
* vulcan-yaser
* vulcan-zwicker

<pre>
$ sacctmgr show account format=account%20,description%30,organization%10
Account Descr Org
-------------------- ------------------------------ ----------
... ... ...
vulcan vulcan vulcan
vulcan-abhinav vulcan - abhinav shrivastava vulcan
vulcan-djacobs vulcan - david jacobs vulcan
vulcan-jbhuang vulcan - jia-bin huang vulcan
vulcan-lsd vulcan - larry davis vulcan
vulcan-metzler vulcan - chris metzler vulcan
vulcan-rama vulcan - rama chellappa vulcan
vulcan-ramani vulcan - ramani duraiswami vulcan
vulcan-yaser vulcan - yaser yacoob vulcan
vulcan-zwicker vulcan - matthias zwicker vulcan
... ... ...
</pre>

Faculty can manage this list of users via our [https://intranet.umiacs.umd.edu/directory/secgroup/ Directory application] in the Security Groups section. The security group that controls access has the prefix <code>vulcan_</code> and then the faculty username. It will also list <code>slurm://nexusctl.umiacs.umd.edu</code> as the associated URI.

You can check your account associations by running the '''show_assoc''' command to see the accounts you are associated with. Please [[HelpDesk | contact staff]] and include your faculty member in the conversation if you do not see the appropriate association.

<pre>
$ show_assoc
User Account MaxJobs GrpTRES QOS
---------- ---------------- ------- ------------- --------------------------------------------------------------------------------
... ... ... ...
abhinav abhinav 48 vulcan-cpu,vulcan-default,vulcan-high,vulcan-medium,vulcan-scavenger
abhinav vulcan 48 vulcan-cpu,vulcan-default,vulcan-medium,vulcan-scavenger
... ... ... ...
</pre>

You can also see the total number of Track-able Resources (TRES) allowed for each account by running the following command. Please make sure you give the appropriate account that you are looking for. As shown below, there is a concurrent limit of 64 total GPUs for all users not in a contributing faculty group.

<pre>
$ sacctmgr show assoc account=vulcan format=user,account,qos,grptres
User Account QOS GrpTRES
---------- ---------- -------------------- -------------
vulcan gres/gpu=64
... ...
</pre>

==QoS==
You need to decide the QOS to submit with which will set a certain number of restrictions to your job. If you do not specify a QoS when submitting your job using the <code>--qos</code> parameter, you will receive the '''vulcan-default''' QoS assuming you are using a Vulcan account.

The following <code>sacctmgr</code> command will list the current QOS. Either the <code>vulcan-default</code>, <code>vulcan-medium</code>, or <code>vulcan-high</code> QOS is required for the vulcan-dpart partition. Please note that only faculty accounts (see above) have access to the <code>vulcan-high</code> QoS.

The following example will show you the current limits that the QOS have. The output is truncated to show only relevant Vulcan QoS.
<pre>
$ show_qos --all
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
...
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-exempt 7-00:00:00 cpu=32,gres/gpu=8,mem=256G 2
vulcan-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
vulcan-janus 3-00:00:00 cpu=32,gres/gpu=10,mem=256G
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-sailon 3-00:00:00 cpu=32,gres/gpu=8,mem=256G gres/gpu=48
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
...

$ show_partition_qos --all
Name MaxSubmitPU MaxTRESPU GrpTRES
-------------------- ----------- ------------------------------ --------------------
...
vulcan 500 cpu=1760,mem=15824G
vulcan-scavenger 500
...
</pre>

==Storage==
Vulcan has the following storage available. Please also review UMIACS [[LocalDataStorage | Local Data Storage]] policies including any volume that is labeled as scratch.

Vulcan users can also request [[Nexus#Project_Allocations | Nexus project allocations]].

===Home Directory===
You have 30GB of storage available at <code>/nfshomes/<username></code>. It has both [[Snapshots]] and [[TSM | Backups]] available if need be.

Home directories are intended to store personal or configuration files only. We encourage users to not share any data in their home directory.

===Scratch Directories===
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Vulcan compute infrastructure:
* Network scratch directory
* Local scratch directories

====Network Scratch Directory====
You are allocated 300GB of scratch space via NFS from <code>/vulcanscratch/$username</code>. '''It is not backed up or protected in any way.''' This directory is '''automounted''' so you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access this.

You may request a temporary increase of up to 500GB total space for a maximum of 120 days without any faculty approval by contacting [mailto:staff@umiacs.umd.edu staff@umiacs.umd.edu]. Once the temporary increase period is over, you will be contacted and given a one-week window of opportunity to clean and secure your data before staff will forcibly remove data to get your space back under 300GB. If you need space beyond 500GB or for longer than 120 days, you will need faculty approval and/or a project directory.

This file system is available on all submission, data management, and computational nodes within the cluster.

====Local Scratch Directories====
Each computational node that you can schedule compute jobs on has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. These are almost always more performant than any other storage available to the job. However, you must stage their data within the confine of their job and stage the data out before the end of their job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month at 1am. Different nodes will run the maintenance jobs on different days of the month to ensure the cluster is still highly available at all times. Please make sure you secure any data you write to these directories at the end of your job.

===Datasets===
We have read-only dataset storage available at <code>/fs/vulcan-datasets</code>. If there are datasets that you would like to see curated and available, please see [[Datasets | this page]].

The following is the list of datasets available:
{| class="wikitable"
! Dataset
! Path
|-
| 3D-FRONT
| /fs/vulcan-datasets/3d-front
|-
| 3D-FUTURE
| /fs/vulcan-datasets/3d-future
|-
| Action Genome
| /fs/vulcan-datasets/AG
|-
| ActivityNet
| /fs/vulcan-datasets/ActivityNet
|-
| CATER
| /fs/vulcan-datasets/CATER
|-
| COVID-DA
| /fs/vulcan-datasets/COVID-DA
|-
| CelebA
| /fs/vulcan-datasets/CelebA
|-
| CelebA-HQ
| /fs/vulcan-datasets/CelebA-HQ
|-
| CelebAMask-HQ
| /fs/vulcan-datasets/CelebAMask-HQ
|-
| Charades
| /fs/vulcan-datasets/Charades
|-
| CharadesEgo
| /fs/vulcan-datasets/CharadesEgo
|-
| CIFAR10
| /fs/vulcan-datasets/cifar-10-python
|-
| CIFAR100
| /fs/vulcan-datasets/cifar-100-python
|-
| CityScapes
| /fs/vulcan-datasets/cityscapes
|-
| COCO
| /fs/vulcan-datasets/coco
|-
| Conceptual Captions
| /fs/vulcan-datasets/conceptual_captions
|-
| CUB
| /fs/vulcan-datasets/CUB
|-
| DeepFashion
| /fs/vulcan-datasets/DeepFashion
|-
| Digits
| /fs/vulcan-datasets/digits_full
|-
| Edges2handbags
| /fs/vulcan/datasets/edges2handbags
|-
| Edges2shoes
| /fs/vulcan/datasets/edges2shoes
|-
| EGTEA
| /fs/vulcan/datasets/EGTEA
|-
| emnist
| /fs/vulcan-datasets/emnist
|-
| EPIC Kitchens 2018
| /fs/vulcan-datasets/Epics-kitchen-2018
|-
| EPIC Kitchens 2020
| /fs/vulcan-datasets/EPIC-Kitchens-2020
|-
| Facades
| /fs/vulcan/datasets/facades
|-
| from_games (GTA5)
| /fs/vulcan-datasets/from_games
|-
| FFHQ
| /fs/vulcan-datasets/ffhq-dataset
|-
| FineGym
| /fs/vulcan-datasets/FineGym
|-
| Google Landmarks Dataset v2
| /fs/vulcan-datasets/google-landmark-v2
|-
| HAA500
| /fs/vulcan-datasets/haa500
|-
| HICO
| /fs/vulcan-datasets/HICO
|-
| HMDB51
| /fs/vulcan-datasets/HMDB51
|-
| Honda_100h
| /fs/vulcan-datasets/honda_100h
|-
| HPatches
| /fs/vulcan-datasets/HPatches
|-
| Human3.6M
| /fs/vulcan-datasets/human3.6
|-
| IM2GPS (test only)
| /fs/vulcan-datasets/im2gps
|-
| ImageNet
| /fs/vulcan-datasets/imagenet
|-
| iNaturalist Dataset 2021
| /fs/vulcan-datasets/inat_comp_2021
|-
| InteriorNet
| /fs/vulcan-datasets/InteriorNet
|-
| Kinetics-400
| /fs/vulcan-datasets/Kinetics-400
|-
| Labelled Faces in the Wild
| /fs/vulcan-datasets/lfw
|-
| LibriSpeech
| /fs/vulcan-datasets/LibriSpeech
|-
| LSUN
| /fs/vulcan-datasets/LSUN
|-
| LVIS
| /fs/vulcan-datasets/LVIS
|-
| Maps
| /fs/vulcan-datasets/maps
|-
| Matterport3D
| /fs/vulcan-datasets/Matterport3D
|-
| MegaDepth
| /fs/vulcan-datasets/MegaDepth
|-
| MineRL
| /fs/vulcan-datasets/MineRL
|-
| Mini-ImageNet
| /fs/vulcan-datasets/miniImagenet
|-
| MIT Indoor
| /fs/vulcan-datasets/mit_indoor
|-
| MIT Places
| /fs/vulcan-datasets/mit_places
|-
| Multi-PIE Face
| /fs/vulcan-datasets/multipie
|-
| Night2day
| /fs/vulcan-datasets/night2day
|-
| ObjectNet3D
| /fs/vulcan-datasets/ObjectNet3D
|-
| Occluded Video Instance Segmentation
| /fs/vulcan-datasets/ovis-2021
|-
| Office
| /fs/vulcan-datasets/office
|-
| Office-Home
| /fs/vulcan-datasets/office_home
|-
| omniglot
| /fs/vulcan-datasets/omniglot
|-
| OOPS
| /fs/vulcan-datasets/OOPS
|-
| OpenImagesv4
| /fs/vulcan-datasets/OpenImagesv4
|-
| PartNet
| /fs/vulcan-datasets/PartNet
|-
| Pascal VOC
| /fs/vulcan-datasets/pascal_voc
|-
| PIC (HOI-A)
| /fs/vulcan-datasets/PIC
|-
| PubLayNet
| /fs/vulcan-datasets/PubLayNet
|-
| Replica
| /fs/vulcan-datasets/Replica
|-
| ScanNet
| /fs/vulcan-datasets/ScanNet
|-
| ShapeNetCore.v2
| /fs/vulcan-datasets/ShapeNetCore.v2
|-
| Something-Something-V1
| /fs/vulcan-datasets/SomethingV1
|-
| Something-Something-V2
| /fs/vulcan-datasets/SomethingV2
|-
| SYNTHIA-RAND-CITYSCAPES
| /fs/vulcan-datasets/SYNTHIA-RAND-CITYSCAPES
|-
| TAPOS
| /fs/vulcan-datasets/TAPOS
|-
| Tiny ImageNet
| /fs/vulcan-datasets/tiny_imagenet
|-
| Tumblr GIF Description
| /fs/vulcan-datasets/TGIF
|-
| Thingi10K
| /fs/vulcan-datasets/Thingi10K
|-
| UCF101
| /fs/vulcan-datasets/UCF101
|-
| VirtualHomes
| /fs/vulcan-datasets/VirtualHomes
|-
| visda17
| /fs/vulcan-datasets/visda17
|-
| visda17_openset
| /fs/vulcan-datasets/VISDA
|-
| visda19
| /fs/vulcan-datasets/visda
|-
| Visual Genome
| /fs/vulcan-datasets/VG
|-
| Visual Relationship Detection
| /fs/vulcan-datasets/VRD
|-
| VOCdevkit
| /fs/vulcan-datasets/VOCdevkit
|-
| VoxCeleb2
| /fs/vulcan-datasets/VoxCeleb2
|-
| WILDS
| /fs/vulcan-datasets/WILDS
|-
| xView2
| /fs/vulcan-datasets/xView2
|-
| YCB Object Models
| /fs/vulcan-datasets/YCB
|-
| YouTube8M
| /fs/vulcan-datasets/YouTube8M
|-
| YouTubeVIS-2019
| /fs/vulcan-datasets/YouTubeVIS-2019
|-
| YouTubeVIS-2021
| /fs/vulcan-datasets/YouTubeVIS-2021
|}

===Project Storage===
Users within the Vulcan compute infrastructure can request project based allocations for up to 10TB for up to 180 days by [https://wiki.umiacs.umd.edu/umiacs/index.php/HelpDesk contacting staff] with approval from the Vulcan faculty manager (Dr. Shrivastava). These allocations will be available from <code>/fs/vulcan-projects</code> under a name that you provide when you request the allocation. Near the end of the allocation period, staff will contact you and ask if you would like to renew the allocation for up to another 180 days (requires re-approval from Dr. Shrivastava). If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation. If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible. If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible. If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

This data, by default, will be backed up nightly and have a limited snapshot schedule (1 daily snapshot). Upon request, staff can both exclude the data from backups and/or disable snapshots on the project storage volume. We currently have 100TB total to support these projects which includes the snapshot data for this volume.

===Object Storage===
All Vulcan users can request project allocations in the [https://obj.umiacs.umd.edu/obj/help UMIACS Object Store]. Please email staff@umiacs.umd.edu with a short project name and the amount of storage you will need to get started.

An example on how to use the umobj command line utilities can be found [https://wiki.umiacs.umd.edu/umiacs/index.php/UMobj/Example here]. A full set of documentation for the utilities can be found on the [https://gitlab.umiacs.umd.edu/staff/umobj/blob/master/README.md#umobj umobj Gitlab page].

==Timeline==
Each event will be completed within the timeframe specified.

{| class="wikitable"
! Date
! Event
|-
| July 21st 2023
| Two compute nodes <code>vulcan11</code> and <code>vulcan12</code> are moved into Nexus so submission can be tested
|-
| [[MonthlyMaintenanceWindow | August 17th 2023, 5-8pm]]
| All other standalone Vulcan cluster compute nodes are moved into Nexus in corresponding <code>vulcan-</code> named partitions
|-
| [[MonthlyMaintenanceWindow | September 21st 2023, 5-8pm]]
| <code>vulcansub00.umiacs.umd.edu</code> and <code>vulcansub01.umiacs.umd.edu</code> are taken offline
|-
|}

==Migration==
===Home Directories===
The [[Nexus]] uses [[NFShomes]] home directories - if your UMIACS account was created before February 22nd, 2023, you were using <code>/cfarhomes/<username></code> as your home directory on the standalone Vulcan cluster. While <code>/cfarhomes</code> is available on Nexus, your shell initialization scripts from it will not automatically load. Please copy over anything you need to your <code>/nfshomes/<username></code> directory at your earliest convenience, as <code>/cfarhomes</code> will be retired in a two phase process:
* Fri 11/17/2023, 5pm: cfarhomes directories are made read-only
* Thu 12/21/2023, 5pm ([[MonthlyMaintenanceWindow |monthly maintenance window]]): cfarhomes directories are taken offline

SLURM/JobStatus

2023-09-22T15:39:20Z

Seide: /* Job Codes */ show_partition_qos

=Job Status=
SLURM offers a variety of tools to check the status of your jobs before, during, and after execution. When you first submit your job, SLURM should give you a job ID which represents the resources allocated to your job. Individual calls to srun will spawn job steps which can also be queried individually.

==squeue==
The squeue command shows job status in the queue. Helpful flags:
* <code>-u username</code> to show only your jobs (replace username with your UMIACS username)
* <code>--start</code> to estimate start time for a job that has not yet started and the reason why it is waiting
* <code>-s</code> to show the status of individual job steps for a job (e.g. batch jobs)

Examples:
<pre>
username@nexusclip00:squeue -u username
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
162 test2 helloWor username R 0:03 2 tron[00-01]
</pre>

<pre>
username@nexusclip00:squeue --start -u username
JOBID PARTITION NAME USER ST START_TIME NODES SCHEDNODES NODELIST(REASON)
163 test2 helloWo2 username PD 2020-05-11T18:36:49 1 tron02 (Priority)
</pre>

<pre>
username@nexusclip00:squeue -s -u username
STEPID NAME PARTITION USER TIME NODELIST
162.0 sleep test2 username 0:05 tron00
162.1 sleep test2 username 0:05 tron01
</pre>

==sstat==
The sstat command shows metrics from currently running job steps. If you don't specify a job step, the lowest job step is displayed.
<pre>
sstat --format JobID,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize <$JOBID>.<$JOBSTEP>
</pre>
<pre>
username@nexusclip00: sstat --format JobID,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize 171
JobID NTasks Nodelist MaxRSS MaxVMSize AveRSS AveVMSize
------------ -------- -------------------- ---------- ---------- ---------- ----------
171.0 1 tron00 0 186060K 0 107900K
username@nexusclip00: sstat --format JobID,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize 171.1
JobID NTasks Nodelist MaxRSS MaxVMSize AveRSS AveVMSize
------------ -------- -------------------- ---------- ---------- ---------- ----------
171.1 1 tron01 0 186060K 0 107900K
</pre>
Note that if you do not have any jobsteps, sstat will return an error.
<pre>
username@nexusclip00: sstat --format JobID,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize 172
JobID NTasks Nodelist MaxRSS MaxVMSize AveRSS AveVMSize
------------ -------- -------------------- ---------- ---------- ---------- ----------
sstat: error: no steps running for job 237
</pre>
If you do not run any srun commands, you will not create any job steps and metrics will not be available for your job. Your batch scripts should follow this format:
<pre>
#!/bin/bash
#SBATCH ...
#SBATCH ...
# set environment up
module load ...

# launch job steps
srun <command to run> # that would be step 1
srun <command to run> # that would be step 2
</pre>

==sacct==
The sacct command shows metrics from past jobs.
<pre>
username@nexusclip00:sacct
JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
162 helloWorld test2 staff 2 COMPLETED 0:0
162.batch batch staff 1 COMPLETED 0:0
162.0 sleep staff 1 COMPLETED 0:0
162.1 sleep staff 1 COMPLETED 0:0
163 helloWorld test2 staff 2 COMPLETED 0:0
163.batch batch staff 1 COMPLETED 0:0
163.0 sleep staff 1 COMPLETED 0:0
</pre>
To check one specific job, you can run something like the following (if you omit .<$JOBSTEP>, all jobsteps will be shown):
<pre>sacct --format JobID,jobname,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize,Elapsed -j <$JOBID>.<$JOBSTEP></pre>
<pre>
username@nexusclip00:sacct --format JobID,jobname,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize,Elapsed -j 171
JobID JobName NTasks NodeList MaxRSS MaxVMSize AveRSS AveVMSize Elapsed
------------ ---------- -------- --------------- ---------- ---------- ---------- ---------- ----------
171 helloWorld tron[00-01] 00:00:30
171.batch batch 1 tron00 0 119784K 0 113120K 00:00:30
171.0 sleep 1 tron00 0 186060K 0 107900K 00:00:30
171.1 sleep 1 tron01 0 186060K 0 107900K 00:00:30
</pre>

=Job Codes=
If you list the current running jobs and your job is in <code>PD</code> (Pending), SLURM will provide you some information on what the reason for this in the NODELIST parameter. You can use <code>scontrol show job <jobid></code> to get all the parameters for your job to help identify why your job is not running.

<pre>
# squeue -u testuser
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
581530 dpart bash testuser PD 0:00 1 (AssocGrpGRES)
581533 dpart bash testuser PD 0:00 1 (Resources)
581534 dpart bash testuser PD 0:00 1 (QOSMaxGRESPerUser)
581535 dpart bash testuser PD 0:00 1 (ReqNodeNotAvail, Reserved for maintenance)
</pre>

Some common ones are as follows:
* <code>Resources</code> - The cluster does not currently have the resources to fit your job in your selected partition.
* <code>QOSMaxGRESPerUser</code> - The quality of service (QoS) your job is running in has a limit of resources per user. Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the limits and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS.
* <code>AssocGrpGRES</code> or <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on either a specific GRES or the overall billing amount (respectively) available in total for the account. Use <code>sacctmgr show assoc account=<account_name></code> to identify the limit. You can see all jobs running under the account by running <code>squeue -A account_name</code> and then find out more information on each job by <code>scontrol show job <jobid></code>.
* <code>ReqNodeNotAvail</code> - You have requested a specific node for your job and it does not currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow | maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.

SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].

Nexus

2023-09-22T15:36:04Z

Seide: show_qos and show_partition_qos updates

The Nexus is the combined scheduler of resources in UMIACS. Many of our existing computational clusters that have discrete schedulers will be folding into this scheduler in the future (see [[#Migrations | below]]). The resource manager for Nexus is [[SLURM]]. Resources are arranged into partitions where users are able to schedule computational jobs. Users are arranged into a number of SLURM accounts based on faculty, lab, or center investments.

= Getting Started =
All accounts in UMIACS are sponsored. If you don't already have a UMIACS account, please see [[Accounts]] for information on getting one. You need a full UMIACS account (not a [[Accounts/Collaborator | collaborator account]]) in order to access Nexus.

== Access ==
Your access to submission nodes for Nexus computational resources are determined by your account sponsor's department, center, or lab affiliation. You can log into the [https://intranet.umiacs.umd.edu/directory/cr/ UMIACS Directory CR application] and select the Computational Resource (CR) in the list that has the prefix <code>nexus</code>. The Hosts section lists your available submission nodes, generally a pair of nodes of the format <tt>nexus<department, lab, or center abbreviation>[00,01]</tt>, e.g., <tt>nexuscfar00</tt> and <tt>nexuscfar01</tt>.

'''Note''' - UMIACS requires multi-factor authentication through our [[Duo]] instance. This is completely discrete from both UMD's and CSD's Duo instances. You will need to enroll one or more devices to access resources in UMIACS, and will be prompted to enroll when you log into the Directory application for the first time.

Once you have identified your submission nodes, you can [[SSH]] directly into them. From there, you are able to submit to the cluster via our [[SLURM]] workload manager. You need to make sure that your submitted jobs have the correct account, partition, and qos.

== Jobs ==
[[SLURM]] jobs are submitted by either <code>srun</code> or <code>sbatch</code> depending if you are doing an interactive job or batch job, respectively. You need to provide the where/how/who to run the job and specify the resources you need to run with.

For the who/where/how, you may be required to specify <code>--account</code>, <code>--partition</code>, and/or <code>--qos</code> (respectively) to be able to adequately submit jobs to the Nexus.

For resources, you may need to specify <code>--time</code> for time, <code>--tasks</code> for CPUs, <code>--mem</code> for RAM, and <code>--gres=gpu</code> for GPUs in your submission arguments to meet your requirements. There are defaults for all four, so if you don't specify something, you may be scheduled with a very minimal set of time and resources (e.g., by default, NO GPUs are included if you do not specify <code>--gres=gpu</code>). For more information about submission flags for GPU resources, see [[SLURM/JobSubmission#Requesting_GPUs]]. You can also can run <code>man srun</code> on your submission node for a complete list of available submission arguments.

=== Interactive ===
Once logged into a submission node, you can run simple interactive jobs. If your session is interrupted from the submission node, the job will be killed. As such, we encourage use of a terminal multiplexer such as [[Tmux]].

<pre>
$ srun --pty --ntasks 4 --mem=2gb --gres=gpu:1 nvidia-smi -L
GPU 0: NVIDIA RTX A4000 (UUID: GPU-ae5dc1f5-c266-5b9f-58d5-7976e62b3ca1)
</pre>

=== Batch ===
Batch jobs are scheduled with a script file with an optional ability to embed job scheduling parameters via variables that are defined by <code>#SBATCH</code> lines at the top of the file. You can find some examples in our [[SLURM/JobSubmission]] documentation.

= Partitions =
The SLURM resource manager uses partitions to act as job queues which can restrict size, time and user limits. The Nexus has a number of different partitions of resources. Different Centers, Labs, and Faculty are able to invest in computational resources that are restricted to approved users through these partitions.

'''Partitions usable by all non-[[ClassAccounts |class account]] users:'''
* [[Nexus/Tron]] - Pool of resources available to all UMIACS and CSD faculty and graduate students.
* Scavenger - [https://slurm.schedmd.com/preempt.html Preemption] partition that supports nodes from multiple other partitions. More resources are available to schedule simultaneously than in other partitions, however jobs are subject to preemption rules. You are responsible for ensuring your jobs handle this preemption correctly. The SLURM scheduler will simply restart a preempted job with the same submission arguments when it is available to run again.

'''Partitions usable by [[ClassAccounts]]:'''
* [[ClassAccounts | Class]] - Pool available for UMIACS class accounts sponsored by either UMIACS or CSD faculty.

'''Partitions usable by specific lab/center users:'''
* [[Nexus/CBCB]] - CBCB lab pool available for CBCB lab members.
* [[Nexus/CLIP]] - CLIP lab pool available for CLIP lab members.
* [[Nexus/CML]] - CML lab pool available for CML lab members. (all compute nodes from standalone cluster folded in as of 08/17/2023)
* [[Nexus/Gamma]] - GAMMA lab pool available for GAMMA lab members.
* [[Nexus/MBRC]] - MBRC lab pool available for MBRC lab members.
* [[Nexus/MC2]] - MC2 lab pool available for MC2 lab members.
* [[Nexus/Vulcan]] - Vulcan lab pool available for Vulcan lab members. (all compute nodes from standalone cluster folded in as of 08/17/2023)

= Quality of Service (QoS) =
SLURM uses Quality of Service (QoS) to provide limits on job sizes to users. Note that you should still try to only allocate the minimum resources for your jobs, as resources that each of your jobs schedules are counted against your [https://slurm.schedmd.com/fair_tree.html FairShare priority] in the future.
* default - Default job QoS. Limited to 4 cores, 32GB RAM, and 1 GPU per job. The maximum wall time per job is 3 days.
* medium - Limited to 8 cores, 64GB RAM, and 2 GPUs per job. The maximum wall time per job is 2 days.
* high - Limited to 16 cores, 128GB RAM, and 4 GPUs per job. The maximum wall time per job is 1 day.
* scavenger - Limited to 64 cores, 256GB RAM, and 8 GPUs per job. The maximum wall time per job is 2 days. Only 576 total cores, 2304GB total RAM, and 72 total GPUs are permitted simultaneously across all of your jobs running with this job QoS. This job QoS is both only available in the scavenger partition and the only job QoS available in the scavenger partition. To use this job QoS, include <code>--partition=scavenger</code> and <code>--account=scavenger</code> in your submission arguments. Do not include any job QoS argument other than <code>--qos=scavenger</code> (optional) or submission will fail.

You can display these job QoS from the command line using the <code>show_qos</code> command. By default, the command will only show QoS that your user can access. The above four job QoS are the ones that everyone can submit using.

<pre>
[seide@nexusstaff00 ~]$ show_qos
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
huge-long 10-00:00:00 cpu=32,gres/gpu=8,mem=256G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
scavenger 3-00:00:00 cpu=64,gres/gpu=8,mem=256G cpu=576,gres/gpu=72,mem=2304G
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
</pre>

If you want to see all QoS, including those that your user cannot access, you can use the <code>show_qos --all</code> command.

<pre>
[seide@nexusstaff00 ~]$ show_qos --all
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
cml-cpu 7-00:00:00 8
cml-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
cml-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
cml-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
cml-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
cml-scavenger 3-00:00:00 gres/gpu=24
cml-very_high 1-12:00:00 cpu=32,gres/gpu=8,mem=256G 8 gres/gpu=12
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
highmem 21-00:00:00 cpu=32,mem=2000G
huge-long 10-00:00:00 cpu=32,gres/gpu=8,mem=256G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
scavenger 3-00:00:00 cpu=64,gres/gpu=8,mem=256G cpu=576,gres/gpu=72,mem=2304G
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-exempt 7-00:00:00 cpu=32,gres/gpu=8,mem=256G 2
vulcan-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
vulcan-janus 3-00:00:00 cpu=32,gres/gpu=10,mem=256G
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-sailon 3-00:00:00 cpu=32,gres/gpu=8,mem=256G gres/gpu=48
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
</pre>

To find out what accounts and partitions you have access to, first use the <code>show_assoc</code> command to show your account/job QoS combinations. Then, use the <code>scontrol show partition</code> command and note the <tt>AllowAccounts</tt> entry for each listed partition. You are able to submit to any partition that allows an account that you have. If you need to use an account other than the default account <tt>nexus</tt>, you will need to specify an account via the <code>--account</code> submission argument.

===Partition QoS===
In addition to using QoS to provide limits on job parameters, termed by us as "job QoS", SLURM can also have QoS assigned to partitions themselves, termed by us as "partition QoS".

To view partition QoS, use the <code>show_partition_qos</code> command.

<pre>
[seide@nexusstaff00 ~]$ show_partition_qos
Name MaxSubmitPU MaxTRESPU GrpTRES
-------------------- ----------- ------------------------------ --------------------
clip 500 cpu=564,mem=5647G
mbrc 500 cpu=240,mem=2378G
scavenger 500 cpu=576,gres/gpu=72,mem=2304G
tron 500 cpu=32,gres/gpu=4,mem=256G
vulcan 500 cpu=1760,mem=15824G
vulcan-scavenger 500
</pre>

If you want to see all partition QoS, including those that your user cannot access, you can use the <code>show_partition_qos --all</code> command.

<pre>
[seide@nexusstaff00 ~]$ show_partition_qos --all
Name MaxSubmitPU MaxTRESPU GrpTRES
-------------------- ----------- ------------------------------ --------------------
cbcb 500 cpu=1004,mem=47840G
class 500 cpu=32,gres/gpu=4,mem=256G
clip 500 cpu=564,mem=5647G
cml 500 cpu=1128,mem=11381G
cml-scavenger 500 gres/gpu=24
gamma 500 cpu=520,mem=4517G
mbrc 500 cpu=240,mem=2378G
mc2 500 cpu=312,mem=3133G
scavenger 500 cpu=576,gres/gpu=72,mem=2304G
tron 500 cpu=32,gres/gpu=4,mem=256G
vulcan 500 cpu=1760,mem=15824G
vulcan-scavenger 500
</pre>

'''NOTE''': These QoS cannot be used directly when submitting jobs, with the exception of scavenger QoS (i.e. they are not in the AllowQos field for their respective partition). Partition QoS limits apply to all jobs running on a given partition, even when using multiple job QoS.

For example, in the default non-preemption partition (<tt>tron</tt>), you are restricted to 32 total cores, 4 total GPUs, and 256GB total RAM at once across all jobs you have running in the job QoS allowed by the partition. You also can only have a maximum of 500 jobs in the partition in the running (R) or pending (PD) states simultaneously. The latter is to prevent excess pending jobs causing [https://slurm.schedmd.com/sched_config.html#backfill backfill] issues.
* If you need to submit more than 500 jobs in batch at once, you can develop and run an "outer submission script" that repeatedly attempts to run the "inner submission script" (your original submission script) to submit jobs in the batch periodically, until all job submissions are successful. The outer submission script should use looping logic to check if you are at the max job limit and should then retry submission after waiting for some time interval. An example outer submission script is as follows. In this example, <code>example_inner.sh</code> is your inner submission script which is not an array job and you want to run 1000 jobs. If your inner submission script is an array job, adjust the number of jobs accordingly. Keep in mind that array jobs must be of size 500 or fewer.
<pre>
#!/bin/bash
numjobs=1000
i=0
while [ $i -lt $numjobs ]
do
while [[ "$(sbatch example_inner.sh 2>&1)" =~ "QOSMaxSubmitJobPerUserLimit" ]]
do
echo "Currently at maximum job submissions allowed."
echo "Waiting for 5 minutes before trying to submit more jobs."
sleep 300
done
i=$(( $i + 1 ))
echo "Submitted job $i of $numjobs"
done
</pre>

It is suggested that you run the outer submission script in a [[Tmux]] session to keep the terminal window executing it from being interrupted.

Lab/group-specific partitions may also have partition QoS intended to limit the total number of resources consumed by all users in that lab/group that are using the partition (codified by <tt>GrpTRES</tt> in the output above for the partition QoS name that matches the lab/group partition name). They also have the 500 running/pending job maximum. Note that the exact values above for TRES are not fixed and may fluctuate as more resources are added to various partitions.

= Storage =
All storage available in Nexus is currently [[NFS]] based. We will be introducing some changes for Phase 2 to support high performance GPUDirect Storage (GDS). These storage allocation procedures will be revised and approved by the launch of Phase 2 by a joint UMIACS and CSD faculty committee.

== Home Directories ==
Home directories in the Nexus computational infrastructure are available from the Institute's [[NFShomes]] as <code>/nfshomes/USERNAME</code> where USERNAME is your username. These home directories have very limited storage (30GB, cannot be increased) and are intended for your personal files, configuration and source code. Your home directory is '''not''' intended for data sets or other large scale data holdings. Users are encouraged to utilize our [[GitLab]] infrastructure to host your code repositories.

'''NOTE''': To check your quota on this directory you will need to use the <code>quota -s</code> command.

Your home directory data is fully protected and has both [[Snapshots | snapshots]] and is [[NightlyBackups | backed up nightly]].

Other standalone compute clusters have begun to fold into partitions in Nexus. The corresponding home directories used by these clusters (if not <code>/nfshomes</code>) will be gradually phased out in favor of the <code>/nfshomes</code> home directories.

== Scratch Directories ==
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Nexus compute infrastructure:
* Network scratch directories
* Local scratch directories

Please note that [[ClassAccounts | class accounts]] do not have network scratch directories.

=== Network Scratch Directories ===
You are allocated 200GB of scratch space via NFS from <code>/fs/nexus-scratch/$username</code>. '''It is not backed up or protected in any way.''' This directory is '''automounted''' so you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access this.

You can view your quota usage by running <code>df -h /fs/nexus-scratch/$username</code>.

You may request a permanent increase of up to 400GB total space without any faculty approval by [[HelpDesk | contacting staff]]. If you need space beyond 400GB, you will need faculty approval and/or a [[#Project_Allocations | project allocation]] for this. If you choose to increase your scratch space beyond 400GB, the increased space is also subject to the 270 TB days limit mentioned in the project allocation section before we check back in for renewal. For example, if you request 1.4TB total space, you may have this for 270 days (1TB beyond the 400GB permanent increase).

This file system is available on all submission, data management, and computational nodes within the cluster.

=== Local Scratch Directories ===
Each computational node that you can schedule compute jobs on also has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. These are almost always more performant than any other storage available to the job. However, you must stage their data within the confines of your job and stage the data out before the end of your job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month during our monthly maintenance windows. Please make sure you secure any data you write to these directories at the end of your job.

== Faculty Allocations ==
Each faculty member can be allocated 1TB of lab space upon request. We can also support grouping these individual allocations together into larger center, lab, or research group allocations if desired by the faculty. Please [[HelpDesk | contact staff]] to inquire.

This lab space does not have [[Snapshots | snapshots]] by default (but are available if requested), but is [[NightlyBackups | backed up]].

== Project Allocations ==
Project allocations are available per user for 270 TB days; you can have a 1TB allocation for up to 270 days, a 3TB allocation for 90 days, etc.. A single faculty member can not have more than 20TB of sponsored account project allocations active at any point.

The maximum allocation length you can request is 540 days (500GB space) and the maximum storage space you can request is 9TB (30 day length).

To request an allocation, please [[HelpDesk | contact staff]] with the faculty member(s) that the project is under involved in the conversation. Please include the following details:
* Project Name (short)
* Description
* Size (1TB, 2TB, etc.)
* Length in days (270 days, 135 days, etc.)
* Other user(s) that need to access the allocation, if any

These allocations are available via <code>/fs/nexus-projects/$project_name</code>. '''Renewal is not guaranteed to be available due to limits on the amount of total storage.''' Near the end of the allocation period, staff will contact you and ask if you are still in need of the storage allocation. If renewal is available, you can renew for up to another 270 TB days with reapproval from the original faculty approver. If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation. If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible. If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible. If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

== Datasets ==
We have read-only dataset storage available at <code>/fs/nexus-datasets</code>. If there are datasets that you would like to see curated and available, please see [[Datasets | this page]].

We will have a more formal process to approve datasets by Phase 2 of Nexus.

= Migrations =
If you are a user of an existing cluster that is the process of being folded into Nexus now or in the near future, your cluster-specific migration information will be listed here.
* [[Nexus/CML]] - all compute nodes folded in as of 08/17/2023
* [[Nexus/Vulcan]] - all compute nodes folded in as of 08/17/2023

CML

2022-04-18T20:38:42Z

Seide: /* Datasets */ Add Salient ImageNet

The Center for Machine Learning ([https://ml.umd.edu CML]) at the University of Maryland is located within the Institute for Advanced Computer Studies. The CML has a cluster of computational (CPU/GPU) resources that are available to be scheduled.

=Compute Infrastructure=
Each of UMIACS' cluster computational infrastructures is accessed through the submission node. Users will need to submit jobs through the [[SLURM]] resource manager once they have logged into the submission node. Each cluster in UMIACS has different quality of service (QoS) that are '''required''' to be selected upon submission of a job. Many clusters, including this one, also have other resources such as GPUs that need to be requested for a job.

The current submission node(s) for '''CML''' are:
* <code>cmlsub00.umiacs.umd.edu</code>

The Center for Machine Learning GPU resources are a small investment from the base Center funds and a number of investments by individual faculty members. The scheduler's resources are modeled around this concept. This means there are additional Slurm accounts that users will need to be aware of if they are submitting in the non-scavenger partition.

==Partitions==
There are three partitions to the CML [[SLURM]] computational infrastructure. If you do not specify a partition when submitting your job, you will receive the '''dpart''' partition.

* '''dpart''' - This is the default partition. Job allocations are guaranteed.
* '''scavenger''' - This is the alternate partition that allows jobs longer run times and more resources but is preemptable when jobs in other partitions are ready to be scheduled.
* '''cpu''' - This partition is for CPU focused jobs. Job allocations are guaranteed.

==Accounts==
The Center has a base account <code>cml</code> which has a modest number of nodes (currently 16 GPUs) total available in it. Other faculty that have invested in the cluster have an additional account provided to their sponsored accounts on the cluster, which provides a number of guaranteed GPU resources corresponding to the amount that they invested. If you do not specify a account when submitting your job, you will receive the <code>cml</code> account.

<pre>
# sacctmgr show accounts
Account Descr Org
---------- -------------------- --------------------
abhinav abhinav shrivastava cml
cml cml cml
furongh furong huang cml
john john dickerson cml
root default root account root
scavenger scavenger scavenger
sfeizi soheil feizi cml
tomg tom goldstein cml
</pre>

You can check your account associations by running the '''show_assoc''' to see the accounts you are associated with. Please [[HelpDesk | contact staff]] and include your faculty member in the conversation if you do not see the appropriate association.

<pre>
$ show_assoc
User Account Def Acct Def QOS QOS
---------- ---------- ---------- --------- ------------------------------------
tomg tomg default,high,medium
tomg cml cpu,default,medium
tomg scavenger scavenger
</pre>

You can also see the total number of Track-able Resources (TRES) allowed for each account by running the following command. Please make sure you give the appropriate account that you are looking for.

<pre>
$ sacctmgr show assoc account=tomg format=user,account,qos,grptres
User Account QOS GrpTRES
---------- ---------- -------------------- -------------
tomg gres/gpu=48

</pre>

==QoS==
CML currently has 4 QoS for the '''dpart''' partition (though <code>very_high</code> is only available on a single faculty member's account), 1 QoS for the '''scavenger''' partition, and 1 QoS for the '''cpu''' partition. You are '''required''' to specify a QoS when submitting your job. The important part here is that in different QoS you can have a shorter/longer maximum wall time, a different total number of jobs running at once, and a different maximum number of track-able resources (TRES) for the job. In the scavenger QoS, one more constraint that you are restricted by is the total number of TRES per user (over multiple jobs).

<pre>
# show_qos
Name MaxWall MaxJobs MaxTRES MaxTRESPU Priority
---------- ----------- ------- ------------------------------ ------------- ----------
medium 3-00:00:00 1 cpu=8,gres/gpu=2,mem=64G 0
default 7-00:00:00 2 cpu=4,gres/gpu=1,mem=32G 0
high 1-12:00:00 2 cpu=16,gres/gpu=4,mem=128G 0
scavenger 3-00:00:00 gres/gpu=24 0
normal 0
cpu 1-00:00:00 1 0
very_high 1-12:00:00 8 cpu=32,gres/gpu=8,mem=256G gres/gpu=12 0
</pre>

==GPUs==
Jobs that require GPU resources need to explicitly request the resources within their job submission. This is done through Generic Resource Scheduling (GRES). Users may use the most generic identifier (in this case '''gpu'''), a colon, and a number to select without explicitly naming the type of GPU (ie. <code>--gres=gpu:4</code> for 4 GPUs).

<pre>
$ sinfo -o "%20N %10c %10m %25f %40G"
NODELIST CPUS MEMORY AVAIL_FEATURES GRES
cmlgrad05 32 385421 Xeon,4216 gpu:rtx3070:1,gpu:rtx2080ti:6
cml[00-11,13-16 32 353924+ Xeon,4216 gpu:rtx2080ti:8
cmlgrad02 32 385421 Xeon,4216 gpu:rtx2080ti:7,gpu:rtx3070:1
cmlcpu[00,05-07 24 386675+ Xeon,E5-2680 (null)
cmlcpu[01-04] 20 386675 Xeon,E5-2660 (null)
cml12 32 385429 Xeon,4216 gpu:rtx2080ti:7,gpu:rtxa4000:1
</pre>

==Job Submission and Management==
Users should review our [[SLURM]] [[SLURM/JobSubmission | job submission]] and [[SLURM/JobStatus | job management]] documentation.

A very quick start to get an interactive shell is as follows when run on the submission node. This will allocate 1 GPU with 16GB of memory (system RAM) in the QoS default for 4 hours maximum time. If the job goes beyond these limits (either the memory allocation or the maximum time) it will be terminated immediately.

<pre>
srun --pty --gres=gpu:1 --mem=16G --qos=default --time=04:00:00 bash
</pre>

<pre>
[username@cmlsub00:~ ] $ srun --pty --gres=gpu:1 --mem=16G --qos=default --time=04:00:00 bash
[username@cml00:~ ] $ nvidia-smi -L
GPU 0: GeForce RTX 2080 Ti (UUID: GPU-20846848-e66d-866c-ecbe-89f2623f3b9a)
</pre>

If you are going to run in a faculty account instead of the default <code>cml</code> account you will need to specify the <code>--account=</code> flag.

A quick example to run an interactive job using the cpu partition. The cpu partition uses the default account <code>cml</code>.
<pre>
-bash-4.2$ srun --partition=cpu --qos=cpu bash -c 'echo "Hello World from" `hostname`'
</pre>

=Data Storage=
Until the final storage investment arrives we have made available a temporary allocation of storage. This section is subject to change. There are 3 types of storage available to users in the CML:
* Home directories
* Project directories
* Scratch directories

==Home Directories==
Home directories in the CML computational infrastructure are available from the Institute's [[NFShomes]] as <code>/nfshomes/USERNAME</code> where USERNAME is your username. These home directories have very limited storage and are intended for your personal files, configuration and source code. Your home directory is '''not''' intended for data sets or other large scale data holdings. Users are encouraged to utilize our [[GitLab]] infrastructure to host your code repositories.

'''NOTE''': To check your quota on this directory you will need to use the <code>quota -s</code> command.

Your home directory data is fully protected and has both snapshots and is backed up nightly.

==Project Directories==
You can request project based allocations for up to 2TB for up to 120 days by [[HelpDesk | contacting staff]] with approval from a CML faculty member and the director of CML. These allocations will be available from '''/fs/cml-projects''' under a name that you provide when you request the allocation. Once the allocation period is over, you will be contacted and given a 14-day window of opportunity to clean and secure your data before staff will remove the allocation.

This data is backed up nightly.

==Scratch Directories==
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the CML compute infrastructure:
* Network scratch directory
* Local scratch directories

===Network Scratch Directory===
You are allocated 400GB of scratch space via NFS from <code>/cmlscratch/$username</code>. '''It is not backed up or protected in any way.''' This directory is '''automounted''' so you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access this.

You may request a permanent increase of up to 800GB total space without any faculty approval by [[HelpDesk | contacting staff]]. If you need space beyond 800GB, you will need faculty approval and/or a project directory.

This file system is available on all submission, data management, and computational nodes within the cluster.

===Local Scratch Directories===
Each computational node that you can schedule compute jobs on has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. These are almost always more performant than any other storage available to the job. However, you must stage their data within the confine of their job and stage the data out before the end of their job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month at 1am. Please make sure you secure any data you write to these directories at the end of your job.

==Datasets==
We have read-only dataset storage available at <code>/fs/cml-datasets</code>. If there are datasets that you would like to see curated and available, please see [[Datasets | this page]].

The following is the list of datasets available:
{| class="wikitable"
! Dataset
! Path
|-
| CelebA
| /fs/cml-datasets/CelebA
|-
| CelebA-HQ
| /fs/cml-datasets/CelebA-HQ
|-
| CelebAMask-HQ
| /fs/cml-datasets/CelebAMask-HQ
|-
| Charades
| /fs/cml-datasets/Charades
|-
| Cityscapes
| /fs/cml-datasets/cityscapes
|-
| COCO
| /fs/cml-datasets/coco
|-
| Diversity in Faces [1]
| /fs/cml-datasets/diversity_in_faces
|-
| FFHQ
| /fs/cml-datasets/FFHQ
|-
| ImageNet ILSVRC2012
| /fs/cml-datasets/ImageNet/ILSVRC2012
|-
| LFW
| /fs/cml-datasets/facial_test_data
|-
| LibriSpeech
| /fs/cml-datasets/LibriSpeech
|-
| LSUN
| /fs/cml-datasets/LSUN
|-
| MAG240M
| /fs/cml-datasets/OGB/MAG240M
|-
| MegaFace
| /fs/cml-datasets/megaface
|-
| MS-Celeb-1M
| /fs/cml-datasets/MS_Celeb_aligned_112
|-
| OC20
| /fs/cml-datasets/OC20
|-
| ogbn-papers100M
| /fs/cml-datasets/OGB/ogbn-papers100M
|-
| roberta
| /fs/cml-datasets/roberta
|-
|-
| Salient ImageNet
| /fs/cml-datasets/Salient-ImageNet
|-
| ShapeNetCore.v2
| /fs/cml-datasets/ShapeNetCore.v2
|-
| Tiny ImageNet
| /fs/cml-datasets/tiny_imagenet
|-
| WikiKG90M
| /fs/cml-datasets/OGB/WikiKG90M
|}

[1] - This dataset has restricted access. Please [[HelpDesk | contact staff]] if you are looking to use this dataset.

LaTeX

2019-06-24T20:16:27Z

Seide: /* LaTeX on Linux/UNIX */ Revise section to include GNU module information. Trim down the example.

==Background==

From the main [http://www.latex-project.org Project Page]:

LaTeX is a high-quality typesetting system; it includes features designed for the production of technical and scientific documentation. LaTeX is the <i>de facto</i> standard for the communication and publication of scientific documents.

==LaTeX on Windows==

In Windows, it is highly recommended to use a full-featured suite such as [https://www.tug.org/texlive/ TeX Live] or [http://miktex.org/ MiKTeX]. Both of these suites include everything required for end-to-end LaTeX compilation and filetype conversation. Please contact the [[HelpDesk]] for assistance with installing or using these on a UMIACS-supported Windows machine.

==LaTeX on Linux/UNIX==

[https://www.tug.org/mactex/ MacTeX] is available for macOS.

[https://www.tug.org/texlive/ TeX Live] is available for Linux. For editing LaTeX files on RHEL7, [https://www.xm1math.net/texmaker/ Texmaker] is available as a GNU module.

Most of our supported Linux systems come with the LaTeX command-line utilities, as well as TeX Live, installed. Supported RHEL7 workstations come with Tex Live 2013, but you can load a newer version of TeX Live as a GNU module.

===Checking installed packages===

TeX Live comes with tons of packages preinstalled. To get a list of installed packages, you must first load the TeX Live GNU module. You can then use the following command:

<pre>
$ tlmgr list --only-installed</pre>

If you know which package you are looking for, you can pipe the output into <tt>grep</tt> to search for specific packages.

===Compiling to PDF===

The following is an example of compiling <tt>example.tex</tt> to a PDF.

<ol>
<li>
Compile the file into a DVI file by using the <tt>latex</tt> command.

<pre>
$ latex example.tex
This is pdfTeX, Version 3.1415926-2.3-1.40.13 (TeX Live 2013)
[...]
[1] (./example.aux) )
Output written on example.dvi (1 page, 1692 bytes).
Transcript written on example.log.</pre>
</li>
<li>
Use <tt>dvipdfmx</tt> to convert <tt>example.dvi</tt> into a PDF (<tt>dvipdf</tt> or <tt>dvipdfm</tt> usually would work as well).

<pre>
$ dvipdfmx example.dvi
example.dvi -> example.pdf
[1]
12211 bytes written</pre>
</li>
</ol>
If your output PDF does not look quite right, you may need to use different conversion tools. Your workflow may necessarily vary depending on the contents of your document and how they are formatted and rendered.

==Further Reading==

* [http://latex-project.org/guides LaTeX project documentation page]
* [http://mintaka.sdsu.edu/GF/bibliog/latex/LaTeXtoPDF.html More LaTex to PDF options]