UMIACS - User contributions [en]

Nexus

2025-09-05T15:29:53Z

Jolmsted:

The Nexus is the combined scheduler of resources in UMIACS. The resource manager for Nexus is [[SLURM]]. Resources are arranged into partitions where users are able to schedule computational jobs. Users are arranged into a number of SLURM accounts based on faculty, lab, or center investments.

= Getting Started =
All accounts in UMIACS are sponsored. If you don't already have a UMIACS account, please see [[Accounts]] for information on getting one. You need a full UMIACS account (not a [[Accounts/Collaborator | collaborator account]]) in order to access Nexus.

== Access ==
Your access to submission nodes (alternatively called login nodes) for Nexus computational resources is determined by your account sponsor's department, center, or lab affiliation. You can log into the [https://intranet.umiacs.umd.edu/directory/cr/ UMIACS Directory CR application] and select the Computational Resource (CR) in the list that has the prefix <code>nexus</code>. The Hosts section lists your available submission nodes - generally a pair of nodes of the format <tt>nexus<department, lab, or center abbreviation>[00,01]</tt>, e.g., <tt>nexusgroup00</tt> and <tt>nexusgroup01</tt>.

Once you have identified your submission nodes, you can [[SSH]] into them [https://itsupport.umd.edu/itsupport?id=kb_article_view&sysparm_article=KB0016076 after connecting to UMD's GlobalProtect VPN]. From there, you are able to submit to the cluster via our [[SLURM]] workload manager. You need to make sure that your submitted jobs have the correct account, partition, and qos.

== Jobs ==
[[SLURM]] jobs are [[SLURM/JobSubmission | submitted]] by either <code>srun</code> or <code>sbatch</code> depending if you are doing an interactive job or batch job, respectively. You need to provide the where/how/who to run the job and specify the resources you need to run with.

For the who/where/how, you may be required to specify <code>--account</code>, <code>--partition</code>, and/or <code>--qos</code> (respectively) to be able to adequately submit jobs to the Nexus.

For resources, you may need to specify <code>--time</code> for time, <code>--cpus-per-task</code> for CPUs, <code>--mem</code> for RAM, and <code>--gres=gpu</code> for GPUs in your submission arguments to meet your requirements. There are defaults for all four; if you don't specify something, you will get the default value for that resource, which is minimal (e.g., by default, NO GPUs are included if you do not specify <code>--gres=gpu</code>). For more information about submission flags for GPU resources, see [[SLURM/JobSubmission#Requesting_GPUs | here]]. You may also use <code>--ntasks</code> to specify the number of parallel processes to run, with each task having its own set of the resources specified above. You can run <code>man srun</code> on your submission node for a complete list of available submission arguments.

For a list of available GPU types on Nexus and their specs, please see [[Nexus/GPUs]].

For details on how the network for Nexus is architected, please see [[Nexus/Network]]. This can be important if you wish to optimize performance of your jobs.

=== Interactive ===
Once logged into a submission node, you can run simple interactive jobs. If your session is interrupted from the submission node, the job will be killed. As such, we encourage use of a terminal multiplexer such as [[Tmux]].

<pre>
$ srun --pty --cpus-per-task=4 --mem=2gb --gres=gpu:1 bash
srun: Job account was unset; set to user default of 'nexus'
srun: Job partition was unset; set to cluster default of 'tron'
srun: Job QoS was unset; set to association default of 'default'
srun: Job time limit was unset; set to partition default of 60 minutes
srun: job 1 queued and waiting for resources
srun: job 1 has been allocated resources
$ hostname
tron62.umiacs.umd.edu
$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-daad6a04-a2ce-1183-ce53-b267048f750a)
</pre>

=== Batch ===
Batch jobs are scheduled with a script file with an optional ability to embed job scheduling parameters via variables that are defined by <code>#SBATCH</code> lines at the top of the file. You can find some examples in our [[SLURM/JobSubmission]] documentation.

= Partitions =
The SLURM resource manager uses partitions to act as job queues which can restrict size, time and user limits. The Nexus has a number of different partitions of resources. Different Centers, Labs, and Faculty are able to invest in computational resources that are restricted to approved users through these partitions.

'''Partitions usable by all non-[[ClassAccounts |class account]] users:'''
* [[Nexus/Tron]] - Pool of resources available to all non-class accounts sponsored by either UMIACS or CSD faculty.
* Scavenger - [https://slurm.schedmd.com/preempt.html Preemption] partition that contains [https://en.wikipedia.org/wiki/X86-64 x86_64] architecture nodes from multiple other partitions. More resources are available to schedule simultaneously than in other partitions, however jobs are subject to preemption rules. You are responsible for ensuring your jobs handle this preemption correctly. The SLURM scheduler will simply restart a preempted job with the same submission arguments when it is available to run again. For an overview of things you can check within scripts to determine if your job was preempted/resumed, see [[SLURM/Preemption]].
* Scavenger (aarch64) - Preemption partition identical in design to <tt>scavenger</tt>, but only contains [https://en.wikipedia.org/wiki/AArch64 aarch64] architecture nodes.

'''Partitions usable by [[ClassAccounts]]:'''
* [[ClassAccounts | Class]] - Pool of resources available to class accounts sponsored by either UMIACS or CSD faculty.

'''Partitions usable by specific lab/center users:'''
* [[Nexus/CBCB]] - CBCB lab pool available for CBCB lab members.
* [[Nexus/CLIP]] - CLIP lab pool available for CLIP lab members.
* [[Nexus/CML]] - CML lab pool available for CML lab members.
* [[Nexus/GAMMA]] - GAMMA lab pool available for GAMMA lab members.
* [[Nexus/MBRC]] - MBRC lab pool available for MBRC lab members.
* [[Nexus/MC2]] - MC2 lab pool available for MC2 lab members.
* [[Nexus/QuICS]] - QuICS lab pool available for QuICS lab members.
* [[Nexus/Vulcan]] - Vulcan lab pool available for Vulcan lab members.

You can view the partitions that you have access to by using the <code>show_partitions</code> command. By default, the command will show only the partitions that are available to you.

<pre>
$ show_partitions
Name AllowAccounts AllowQos MaxNodes Nodes
------------------------ ----------------------- ------------------------------ ----------- ----------------------------
scavenger scavenger scavenger UNLIMITED brigid[16-19]
cbcb[00-29]
clip[00-13]
cml[00,02-13,15-28,30-33]
cmlcpu[00-04,06-07]
gammagpu[00-21]
legacy[00-11,13-28,30-36]
legacygpu[00-07]
quics00
tron[00-44,46-69]
vulcan[00-45]
------------------------------------------------------------------------------------------------------------------------
scavenger-aarch64 scavenger scavenger-aarch64 UNLIMITED oasis[00-39]
------------------------------------------------------------------------------------------------------------------------
tron nexus default UNLIMITED tron[00-44,46-69]
high
medium
</pre>

If you want to see information for all of the partitions, you can use the <code>show_partitions --all</code> command.

<pre>
$ show_partitions --all
Name AllowAccounts AllowQos MaxNodes Nodes
------------------------ ----------------------- ------------------------------ ----------- ----------------------------
cbcb cbcb default UNLIMITED cbcb[00-20,22-29]
medium legacy[00-11,13-28,30-36]
high
huge-long
highmem
------------------------------------------------------------------------------------------------------------------------
cbcb-heng cbcb-heng default UNLIMITED cbcb[26-29]
medium
high
huge-long
highmem
------------------------------------------------------------------------------------------------------------------------
cbcb-interactive cbcb interactive UNLIMITED cbcb21
...
</pre>

= Quality of Service (QoS) =
SLURM uses Quality of Service (QoS) both to provide limits on job sizes (termed by us as "job QoS") as well as to limit resources used by all jobs running in a partition, either per user or per group (termed by us as "partition QoS").

=== Job QoS ===
Job QoS are used to provide limits on the size of job that you can run. You should try to allocate only the resources your job actually needs, as resources that each of your jobs schedules are counted against your [[SLURM/Priority#Fair-share | fair-share priority]] in the future.
* default - Default job QoS. Limited to 4 CPU cores, 1 GPU, and 32GB RAM per job. The maximum wall time per job is 3 days.
* medium - Limited to 8 CPU cores, 2 GPUs, and 64GB RAM per job. The maximum wall time per job is 2 days.
* high - Limited to 16 CPU cores, 4 GPUs, and 128GB RAM per job. The maximum wall time per job is 1 day.
* scavenger - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. 576 total CPU cores, 72 total GPUs, and 2304GB total RAM are permitted simultaneously across all of your jobs running with this job QoS. This job QoS is paired 1-1 with the scavenger partition. To use this job QoS, include <code>--partition=scavenger</code> and <code>--account=scavenger</code> in your submission arguments. Do not include any job QoS argument other than <code>--qos=scavenger</code> (optional) or submission will fail.
* scavenger-aarch64 - No resource limits per job, only a maximum wall time per job of 3 days. You are responsible for ensuring your job requests multiple nodes if it requests resources beyond what any one node is capable of. 1600 total CPU cores and 281140MB total RAM are permitted simultaneously across all of your jobs running with this job QoS. This job QoS is paired 1-1 with the scavenger-aarch64 partition. To use this job QoS, include <code>--partition=scavenger-aarch64</code>, <code>--account=scavenger</code>, and <code>--qos=scavenger-aarch64</code> in your submission arguments.

You can display these job QoS from the command line using the <code>show_qos</code> command. By default, the command will only show job QoS that you can access. The above five job QoS are the ones that everyone can access.

<pre>
$ show_qos
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
scavenger 3-00:00:00 cpu=576,gres/gpu=72,mem=2304G
scavenger-aarch64 3-00:00:00 cpu=1600,mem=281140M
</pre>

If you want to see all job QoS, including those that you do not have access to, you can use the <code>show_qos --all</code> command.

<pre>
$ show_qos --all
Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
-------------------- ----------- ------------------------------ --------- ------------------------------
cml-cpu 7-00:00:00 8
cml-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
cml-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
cml-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
cml-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
cml-scavenger 3-00:00:00 gres/gpu=24
cml-very_high 1-12:00:00 cpu=32,gres/gpu=8,mem=256G 8 gres/gpu=12
default 3-00:00:00 cpu=4,gres/gpu=1,mem=32G
gamma-huge-long 10-00:00:00 cpu=32,gres/gpu=16,mem=256G
high 1-00:00:00 cpu=16,gres/gpu=4,mem=128G
highmem 21-00:00:00 cpu=128,mem=2T
huge-long 10-00:00:00 cpu=32,gres/gpu=8,mem=256G
interactive 12:00:00 cpu=4,mem=128G
medium 2-00:00:00 cpu=8,gres/gpu=2,mem=64G
oasis-exempt 10-00:00:00 cpu=160,mem=28114M
scavenger 3-00:00:00 cpu=576,gres/gpu=72,mem=2304G
scavenger-aarch64 3-00:00:00 cpu=1600,mem=281140M
tron-exempt cpu=32,gres/gpu=8,mem=383030M 1
vulcan-cpu 2-00:00:00 cpu=1024,mem=4T 4
vulcan-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
vulcan-exempt 7-00:00:00 cpu=32,gres/gpu=8,mem=256G 2
vulcan-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
vulcan-high_long 14-00:00:00 cpu=32,gres/gpu=8 8 gres/gpu=8
vulcan-medium 3-00:00:00 cpu=8,gres/gpu=2,mem=64G 2
vulcan-sailon 3-00:00:00 cpu=32,gres/gpu=8,mem=256G gres/gpu=48
vulcan-scavenger 3-00:00:00 cpu=32,gres/gpu=8,mem=256G
vulcan-scavenger-mu+ 3-00:00:00 cpu=288,gres/gpu=72,mem=1152G
</pre>

You are able to submit to any partition that is listed in the <code>show_partitions</code> command. If you need to use an account other than the default account <tt>nexus</tt>, you will need to specify it via the <code>--account</code> submission argument.

=== Partition QoS ===
Partition QoS are used to limit resources used by all jobs running in a partition, either per user (MaxTRESPU) or per group (GrpTRES).

To view partition QoS, use the <code>show_partition_qos</code> command.

<pre>
$ show_partition_qos
Name MaxSubmitPU MaxTRESPU GrpTRES
-------------------- ----------- ------------------------------ --------------------
scavenger 500 cpu=576,gres/gpu=72,mem=2304G
scavenger-aarch64 500 cpu=1600,mem=281140M
tron 500 cpu=32,gres/gpu=4,mem=256G
</pre>

If you want to see all partition QoS, including those that you do not have access to, you can use the <code>show_partition_qos --all</code> command.

<pre>
$ show_partition_qos --all
Name MaxSubmitPU MaxTRESPU GrpTRES
-------------------- ----------- ------------------------------ --------------------
cbcb 500 cpu=1228,mem=48003G
cbcb-heng 500
cbcb-interactive 500
class 500 cpu=32,gres/gpu=4,mem=256G
clip 500 cpu=564,mem=5590G
cml 500 cpu=1096,mem=10890G
cml-cpu 500
cml-furongh 500
cml-scavenger 500 gres/gpu=24
cml-wriva 500
cml-zhou 500
gamma 500 cpu=616,mem=5204G
mbrc 500 cpu=240,mem=2345G
mc2 500 cpu=312,mem=3092G
oasis 500
quics 500 cpu=328,mem=3484G
scavenger 500 cpu=576,gres/gpu=72,mem=2304G
scavenger-aarch64 500 cpu=1600,mem=281140M
tron 500 cpu=32,gres/gpu=4,mem=256G
vulcan 500 cpu=1272,mem=11710G
vulcan-ampere 500
vulcan-cpu 500
vulcan-ramani 500
vulcan-scavenger 500
</pre>

'''NOTE''': These QoS cannot be used directly when submitting jobs, with the exception of the scavenger QoS (i.e., they are not in the AllowQos field for their respective partition). Partition QoS limits apply to all jobs running on a given partition, regardless of what job QoS is used.

For example, in the default non-preemption partition (<tt>tron</tt>), you are restricted to 32 total CPU cores, 4 total GPUs, and 256GB total RAM at once across all jobs you have running in the partition.

Lab/group-specific partitions may also have their own user limits, and/or may also have group limits on the total number of resources consumed simultaneously by all users that are using their partition, codified by the line in the output above that matches their lab/group name. Note that the values listed above in the two "TRES" columns are not fixed and may fluctuate per-partition as more resources are added to or removed from each partition.

'''All partitions also only allow a maximum of 500 submitted (running (R) or pending (PD)) jobs per user in the partition simultaneously.''' This is to prevent excess pending jobs causing [https://slurm.schedmd.com/sched_config.html#backfill backfill] issues with the SLURM scheduler.
* If you need to submit more than 500 jobs in batch at once, you can develop and run an "outer submission script" that repeatedly attempts to run an "inner submission script" (your original submission script) to submit jobs in the batch periodically, until all job submissions are successful. The outer submission script should use looping logic to check if you are at the max job limit and should then retry submission after waiting for some time interval.
: An example outer submission script is as follows. In this example, <code>example_inner.sh</code> is your inner submission script and is not an [[SLURM/ArrayJobs | array job]], and you want to run 1000 jobs. If your inner submission script is an array job, adjust the number of jobs accordingly. Array jobs must be of size 500 or less.
<pre>
#!/bin/bash
numjobs=1000
i=0
while [ $i -lt $numjobs ]
do
while [[ "$(sbatch example_inner.sh 2>&1)" =~ "QOSMaxSubmitJobPerUserLimit" ]]
do
echo "Currently at maximum job submissions allowed."
echo "Waiting for 5 minutes before trying to submit more jobs."
sleep 300
done
i=$(( $i + 1 ))
echo "Submitted job $i of $numjobs"
done
</pre>

It is suggested that you run the outer submission script in a [[Tmux]] session to keep the terminal window executing it from being interrupted.

= Storage =
All network storage available in Nexus is currently [[NFS]] based, and comes in a few different flavors. Compute nodes also have local scratch storage that can be used.

== Home Directories ==
{{Nfshomes}}

== Scratch Directories ==
Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Nexus compute infrastructure:
* Network scratch directories
* Local scratch directories

Please note that [[ClassAccounts | class accounts]] do not have network scratch directories.

=== Network Scratch Directories ===
You are allocated 200GB of scratch space via NFS from <code>/fs/nexus-scratch/<USERNAME></code> where <USERNAME> is your UMIACS username. '''It is not backed up or protected in any way.''' This directory is '''[[Automounter | automounted]]'''; you will need to <code>cd</code> into the directory or request/specify a fully qualified file path to access it.

You can view your quota usage by running <code>df -h /fs/nexus-scratch/<USERNAME></code>.

You may request a permanent increase of up to 400GB total space without any faculty approval by [[HelpDesk | contacting staff]]. If you need space beyond 400GB, you will need faculty approval and/or a [[#Project_Allocations | project allocation]] for this. If you choose to increase your scratch space beyond 400GB, the increased space is also subject to the 270 TB days limit mentioned in the project allocation section before we check back in for renewal. For example, if you request 1.4TB total space, you may have this for 270 days (1TB beyond the 400GB permanent increase). The amount increased beyond 400GB will also count against your faculty member's 20TB total storage limit mentioned below.

This file system is available on all submission, data management, and computational nodes within the cluster.

=== Local Scratch Directories ===
Each computational node that you can schedule compute jobs on also has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. and '''are not backed up or protected in any way.''' These directories are almost always more performant than any other storage available to the job as they are mounted from disks directly attached to the compute node. However, you must stage your data within the confines of your job and extract the relevant resultant data elsewhere before the end of your job.

These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month during our [[MonthlyMaintenanceWindow | monthly maintenance windows]]. Please make sure you secure any resultant data you wish to keep from these directories at the end of your job.

== Faculty Allocations ==
Each faculty member can be allocated 1TB of permanent lab space upon request. We can also support grouping these individual allocations together into larger center, lab, or research group allocations if desired by the faculty. Please [[HelpDesk | contact staff]] to inquire.

Lab space storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

== Project Allocations ==
Project allocations are available per user for 270 TB days; you can have a 1TB allocation for up to 270 days, a 3TB allocation for 90 days, etc..

A single faculty member can not have more than 20TB of project allocations across all of their sponsored accounts active simultaneously. Network scratch allocation space increases beyond the 400GB permanent maximum also have the increase count against this limit (i.e., a 1TB network scratch allocation would have 600GB counted towards this limit).

Project storage is fully protected. It has [[Snapshots | snapshots]] enabled and is [[NightlyBackups | backed up nightly]].

The maximum allocation length you can request is 540 days (500GB space) and the maximum storage space you can request is 9TB (30 day length).

To request an allocation, please [[HelpDesk | contact staff]] with the faculty member(s) that the project is under involved in the conversation. Please include the following details:
* Project Name (short)
* Description
* Size (1TB, 2TB, etc.)
* Length in days (270 days, 135 days, etc.)
* Other user(s) that need to access the allocation, if any

These allocations are available via <code>/fs/nexus-projects/<project name></code>. '''Renewal is not guaranteed to be available due to limits on the amount of total storage.''' Near the end of the allocation period, staff will contact you and ask if you are still in need of the storage allocation. If renewal is available, you can renew for up to another 270 TB days with reapproval from the original faculty approver.
* If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation.
* If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible.
** If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible.
** If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

== Datasets ==
We have read-only dataset storage available at <code>/fs/nexus-datasets</code>. If there are datasets that you would like to see curated and made available, please see [[Datasets | this page]].

The list of Nexus datasets we currently host can be viewed [https://info.umiacs.umd.edu/datasets/list/?q=Nexus here].

Nexus/GAMMA

2025-08-28T17:25:29Z

Jolmsted:

The [https://gamma.umd.edu/ GAMMA] lab has a partition of GPU nodes available in the [[Nexus]]. Only GAMMA lab members are able to run non-interruptible jobs on these nodes.

=Access=
You can always find out what hosts you have access to submit via the [[Nexus#Access]] page. The GAMMA lab in particular has a special submission node that has additional local storage available.
* <code>nexusgamma00.umiacs.umd.edu</code>

Please do not run anything on the submission node. Always allocate yourself machines on the compute nodes (see instructions below) to run any job.

=Quality of Service=
GAMMA users have access to all of the [[Nexus#Quality_of_Service_.28QoS.29 | standard job QoSes]] in the <code>gamma</code> partition using the <code>gamma</code> account.

The additional job QoSes for the GAMMA partition specifically are:
* <code>huge-long</code>: Allows for longer jobs using higher overall resources.
* <code>gamma-huge-long</code>: Allows for longer jobs using higher overall resources, with even more available GPUs per job.

Please note that the partition has a <code>GrpTRES</code> limit of 100% of the available cores/RAM on the partition-specific nodes in aggregate plus 50% of the available cores/RAM on legacy## nodes in aggregate, so your job may need to wait if all available cores/RAM (or GPUs) are in use.

==Time Limit==
There is a 12 hour time limit on interactive jobs in the gamma partition. If you need to run longer jobs, you will need to modify your workflow into a job that can be submitted as a batch script.

=Compute Nodes=
{| class="wikitable sortable"
! Nodenames
! Type
! Quantity
! CPU cores per node
! Memory per node
! GPUs per node
|-
|gammagpu[00-04,06-09]
|A5000 GPU Node
|9
|32
|256GB
|8
|-
|gammagpu05
|A4000 GPU Node
|1
|32
|256GB
|8
|-
|gammagpu[10-17]
|A6000 GPU Node
|8
|16
|128GB
|4
|-
|gammagpu[18-21]
|L40S GPU Node
|4
|32
|256GB
|4
|- class="sortbottom"
|
!Total
|22
|576
|4608GB
|128
|}

=Network=
The network infrastructure supporting the GAMMA partition consists of:
# One pair of network switches connected to each other via dual 100GbE links for redundancy, serving the following compute nodes:
#* gammagpu[00-19]: Two 100GbE links per node, one to each switch in the pair (redundancy).
#* gammagpu[20-21]: One 100GbE link per node. One of the two nodes links to the first switch in the pair, and the other links to the second switch in the pair. These nodes do not have redundant links because the switches are currently at port capacity.

The fileserver hosting all GAMMA [[Nexus/GAMMA#Project_Directories | project]], [[Nexus/GAMMA#Scratch_Directories | scratch]], and [[Nexus/GAMMA#Datasets | dataset]] allocations first connects to the first pair of switches mentioned [[Nexus/CML#Network | here (CML's page's network section)]] and then to a pair of intermediary switches before reaching the compute nodes. The last hop from the pair of intermediary switches to the first pair of switches mentioned on this page is via eight 100GbE links, two to each switch in a set of four intermediary switches for redundancy and increased bandwidth.

For a broader overview of the network infrastructure supporting the Nexus cluster, please see [[Nexus/Network]].

=Example=
From <code>nexusgamma00.umiacs.umd.edu</code> you can run the following example to submit an interactive job. Please note that you need to specify the <code>--partition</code> and <code>--account</code>. Please refer to our [[SLURM]] documentation about about how to further customize your submissions including making a batch submission. The following command will allocate 8 GPUs for 2 days in an interactive session. Change parameters accordingly to your needs. We discourage use of srun and promote use of sbatch for fair use of GPUs.

<pre>
$ srun --pty --gres=gpu:8 --account=gamma --partition=gamma --qos=huge-long bash
$ hostname
gammagpu05.umiacs.umd.edu
$ nvidia-smi -L
GPU 0: NVIDIA RTX A4000 (UUID: GPU-07ef47fe-072c-5e08-ba19-b87442dd330f)
GPU 1: NVIDIA RTX A4000 (UUID: GPU-7e30eac5-d1e2-ab92-686c-2aee8efbcb79)
GPU 2: NVIDIA RTX A4000 (UUID: GPU-9a31a590-7f83-23a3-6499-05b67357dbf1)
GPU 3: NVIDIA RTX A4000 (UUID: GPU-5c581a49-e0e5-8ec2-124d-b37e043a9086)
GPU 4: NVIDIA RTX A4000 (UUID: GPU-a7061b64-b90e-8fec-ff31-8cc997c88880)
GPU 5: NVIDIA RTX A4000 (UUID: GPU-2e590cba-70fd-4261-9f9f-c5ee813fd305)
GPU 6: NVIDIA RTX A4000 (UUID: GPU-439de936-4b11-5ea6-184c-0a709d35d679)
GPU 7: NVIDIA RTX A4000 (UUID: GPU-de7ef2e7-b6f8-fe81-dfab-a86629553bde)
</pre>

You can also use SBATCH to submit your job. Here are two examples on how to do that.

<pre>
$ sbatch --pty --gres=gpu:8 --account=gamma --partition=gamma --qos=huge-long --time=1-23:00:00 script.sh
</pre>

OR

<pre>
$ sbatch script.sh

// script.sh //

#!/bin/bash
#SBATCH --gres=gpu:8
#SBATCH --account=gamma
#SBATCH --partition=gamma
#SBATCH --qos=huge-long
#SBATCH --time=1-23:00:00

python your_file.py
</pre>

=Storage=
There are 3 types of user storage available to users in GAMMA:
* Home directories
* Project directories
* Scratch directories

There is also read-only storage available for Dataset directories.

GAMMA users can also request [[Nexus#Project_Allocations | Nexus project allocations]].

===Home Directories===
{{Nfshomes}}

===Project Directories===
You can request project based allocations for up to 8TB and up to 180 days with approval from a GAMMA faculty member.

To request an allocation, please [[HelpDesk | contact staff]] with the faculty member(s) that approved the project in the conversation. Please include the following details:
* Project Name (short)
* Description
* Size (1TB, 2TB, etc.)
* Length in days (30 days, 90 days, etc.)
* Other user(s) that need to access the allocation, if any

These allocations will be available from '''/fs/gamma-projects''' under a name that you provide when you request the allocation. Near the end of the allocation period, staff will contact you and ask if you would like to renew the allocation (requires re-approval from a GAMMA faculty member).
* If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation.
* If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible.
** If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible.
** If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

This data is backed up nightly.

===Scratch Directories===
Scratch data has no data protection, there are no snapshots and the data is not backed up.
There are two types of scratch directories:
* Network scratch directory
* Local scratch directories

====Network Scratch Directory====
You are allocated 100GB of scratch space via NFS from <code>/gammascratch/$username</code>. '''It is not backed up or protected in any way.'''

This directory is '''automounted''' so you may not see your directory if you run <code>ls /gammascratch</code> but it will be mounted when you <code>cd</code> into your /gammascratch directory.

You may request a permanent increase of up to 200GB total space without any faculty approval by [[HelpDesk | contacting staff]]. If you need space beyond 200GB, you will need faculty approval.

This file system is available on all submission and computational nodes within the cluster.

====Local Scratch Directories====
These file systems are not available over [[NFS]] and '''there are no backups or snapshots available''' for these file systems.

* Each computational node that you can schedule compute jobs on has one or more local scratch directories. These are always named <code>/scratch0</code>, <code>/scratch1</code>, etc. These directories are local to each node, ie. the <code>/scratch0</code> on two different nodes are completely separate.
** These directories are almost always more performant than any other storage available to the job. However, you must stage data to these directories within the confines of your jobs and stage the data out before the end of your jobs.
** These local scratch directories have a tmpwatch job which will '''delete unaccessed data after 90 days''', scheduled via maintenance jobs to run once a month during our monthly maintenance windows. Again, please make sure you secure any data you write to these directories at the end of your job.
* Gamma has invested in a 20TB NVMe scratch file system on <code>nexusgamma00.umiacs.umd.edu</code> that is available as <code>/scratch1</code>. To utilize this space, you will need to copy data from/to this over SSH from a compute node. To make this easier, you may want to setup [[SSH]] keys that will allow you to copy data without prompting for passwords.
** The <code>/scratch1</code> directory on <code>nexusgamma00.umiacs.umd.edu</code> doesn't have a tmpwatch. The files in this directory need to be manually removed once they are no longer needed.

===Datasets===
We have read-only dataset storage available at <code>/fs/gamma-datasets</code>. If there are datasets that you would like to see curated and made available, please see [[Datasets | this page]].

The list of GAMMA datasets we currently host can be viewed [https://info.umiacs.umd.edu/datasets/list/?q=GAMMA here].

WindowsPrinting

2025-04-24T18:36:25Z

Jolmsted:

==Introduction==
This will guide you through the process of installing and using the printers available within UMIACS.

==Pre-Installation Knowledge on Printing at UMIACS==
[[PrinterQueueNaming| Printer Queue Naming]]

==Printing at UMIACS==
<b>If the computer you are on is not directly connected to UMD's/UMIACS' network, you must first connect to the [[VPN]], and must be on the VPN whenever you print thereafter.</b>
* As alternatives, you can either:
** Hard-wire into a UMD/UMIACS network before continuing.
** Copy your files onto a UMIACS-supported system, such as your available [[Nexus]] submission nodes, and print from there.
** Copy your files onto a USB drive and print via USB at a printer directly.

===Installation of printer queue===
After you are connected, follow these steps for adding a printer queue:
# Go to Start and search for "add a printer" in the search box. Click on the "Add a printer or scanner" option.
#: [[File:Windows_AddPrinter.png]]
# In the top right of the Settings menu that opens, click on "Add device".
#: [[File:Windows_AddPrinterDevice.png|800px]]
# Wait a few seconds for the "Add manually" option to come up, and click it.
#: [[File:Windows_AddPrinterDeviceWait.png|800px]] -> [[File:Windows_AddPrinterDeviceManually.png|800px]]
# In the Add Printer menu that pops up, choose "Select a shared printer by name", and type <code><nowiki>http://print.umiacs.umd.edu:631/printers/<printername></nowiki></code> into the dialog box, where <printername> is replaced with the name of the printer that you want to print to. Click Next.
#: [[File:Windows_TypePrinterName.png]]
#: ''In this example, we are using cps432-3208 as the printer name.''
# Select the manufacturer and model of your printer. Almost all UMIACS-managed printers are HP-brand, so choose HP unless you know for a fact that your printer is not HP-brand. You can either select a model-specific driver if you have one installed and you know the printer's exact model, or a HP Universal Printing driver. Click OK.
#* If you do not have HP as an available manufacturer option at all, [https://support.hp.com/us-en/drivers/hp-universal-print-driver-series-for-windows/4157320 download and install the latest HP Universal Print Driver for Windows]. The PostScript (PS) driver is usually fine unless you want to print with an option only available in PCL6, in which case choose the PCL6 driver.
#** To install the driver, unzip the file you download, then click the "Have Disk..." button. '''Enter the full folder path''' of the extracted zip file and click OK. Do NOT choose the "Browse..." option as this will only let you select individual files, but typing the full file path will let it select the entire folder and determine the right files automatically. Choose either of the two drivers that appear.
#*** For example, if you download the file to C:\Users\<username>\Downloads\upd-ps-x64-7.4.0.25959.zip, and then extract the file also to C:\Users\<username>\Downloads, enter <code>C:\Users\<username>\Downloads\upd-ps-x64-7.4.0.25959</code> in the "Copy manufacturer's files from:" box, and click OK. Make sure to replace <username> with the username you are logged into Windows with.
#: [[File:Windows_SelectPrinterDriver.png]]
#: or
#: [[File:Windows_PrinterInstallFromDisk.png]] -> [[File:Windows_SelectPrinterDriverFromDisk.png]]
# Wait a few seconds for your computer to connect to the printer with the driver you picked. Eventually, a window will pop up saying that the printer was successfully installed with the driver you chose. Click Next.
#: [[File:Windows_WaitForPrinterDriver.png]] -> [[File:Windows_AddPrinterSuccess.png]]
# Choose to set the printer as your default printer or print a test page if you want to, and click Finish. You can now start using the printer.
#: [[File:Windows_DefaultAndTestPagePrinter.png]]

==Troubleshooting==
If you receive the following error when attempting to add a printer queue, the issue may be that your computer is configured for a higher level of printer security than is default at UMIACS as a result of the [https://en.wikipedia.org/wiki/PrintNightmare PrintNightmare vulnerability].
: [[File:win_print_printnightmare.png]]

To resolve this error:
# Click on the Start menu and type "cmd" in the run bar.
#: [[Image:windows_bar.png]]
# ''Right-Click'' on the cmd button, and select "Run as Administrator" - '''THIS IS VERY IMPORTANT!'''
#: [[Image:Example.jpg]]
# Select "Yes" when it asks if you want to allow the following program to make changes.
#: [[Image:Uac.jpg]]
# From the Command Prompt, run these commands in order:
#:<pre>reg add "HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows NT\Printers\PointAndPrint" /v RestrictDriverInstallationToAdministrators /t REG_DWORD /d 0 /f
reg add "HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Print" /v RpcAuthnLevelPrivacyEnabled /t REG_DWORD /d 0 /f
net stop spooler /y && net start spooler</pre>
#: [[File:win_print_printnightmare_cmds.png]]
# Attempt to add the printer queue again.

==Removing a printer queue==
To remove a printer queue, open the Start menu and search for and go to Control Panel. Next, click on "View devices and printers." To remove the queue, right click on the printer and select "Remove device". If you wish to add the printer back, follow the instructions above.
: [[Image:win_print_remove.png]]

==Duplex Printing==
Many printers in UMIACS support duplex (2 sided) printing. To make the duplex printing default, first make sure the appropriate printer queue is installed via the directions above. Afterwards, go to start, then “Devices and Printers.”
Once the printer has been installed proceed by right clicking on the queue then selecting “Printing preferences.”
: [[Image:win_print_preferences.png]]

After a few seconds the following menu will come up. Make sure you are on the “Printing Shortcuts” tab and select the “EcoSMART Settings” option then click on okay. This modification will allow you to print 2-sided whenever you choose to use this printer.
: [[Image:win_print_duplex.png]]

==Store and Print PIN-Protected Jobs==
PIN-protected jobs provide a way to ensure jobs are not printed until you are ready to pick them up. This can be useful if you have a sensitive job that you want to print on a public printer at UMIACS. Follow the below steps to store a job and then print it at the printer.

'''Create a stored job'''
# From the software program, select the '''Print''' option.
# Select the printer from the list of printers, and then select '''Properties''' or '''Preferences''' (name varies by software program)
# Click the '''Job Storage''' tab.
# Under '''Job Storage Mode''', select '''Personal Job'''.
#: [[Image:Pinprint_1_mode.jpg|600px]]
# In the drop-down menu under '''Make Job Private/Secure''', select '''PIN to print''', and type in a 4-digit number.
#: [[Image:Pinprint_2_pin.jpg|600px]]
#: Note: You will need to provide the entered PIN (4-digit number) at the printer's control panel.
# Click the '''OK''' button to close the Document Properties/Preferences dialog box. In the Print dialog box, click '''Print'''.

'''Print a stored job'''

After you have created a stored job using the instructions above, follow these steps to print from the printer's control panel.

# From the Home screen on the printer control panel, select '''Print'''.
# Select '''Print from Job Storage'''.
#: [[Image:Pinprint_3_printfrom.jpg|600px]]
# Select '''Choose''', and then select the name of the folder where the job is stored.
#: [[Image:Pinprint_4_folder.jpg|600px]]
# Select the name of the job, enter your chosen PIN (4-digit number), and click '''Ok'''.
#: [[Image:Pinprint_5_pin.jpg|600px]]
# Select '''Print''' to print the job.
#: [[Image:Pinprint_6_print.jpg|600px]]
#: Note: The print job is deleted from memory after it prints.