UMIACS - Recent changes [en]

Nexus/ClusterOSUpgrade

Mbaney — Fri, 24 Jul 2026 16:51:46 GMT

← Older revision		Revision as of 16:51, 24 July 2026
(One intermediate revision by the same user not shown)
Line 21:		Line 21:
	==Interoperability==		==Interoperability==
	===Software and Modules===		===Software and Modules===
	Please ~~begin transitioning~~ your [[PythonVirtualEnv \| virtual environments]], workflows, etc. to work with RHEL9 as soon as possible. You can use the '01' submission node that you have access to for transitioning and light testing - as always, [[Nexus/Submission_Node_Policy \| please do not run any computationally intensive processes on this node]]. It is intended to be a host for configuring environments/workflows and submitting jobs only.		Please transition your [[PythonVirtualEnv \| virtual environments]], workflows, etc. to work with RHEL9 as soon as possible. You can use the '01' submission node that you have access to for transitioning and light testing - as always, [[Nexus/Submission_Node_Policy \| please do not run any computationally intensive processes on this node]]. It is intended to be a host for configuring environments/workflows and submitting jobs only. If you need to do more involved testing, please schedule a job for this.

	The [[Modules \| module tree]] for RHEL9 ~~has already been~~ populated with a large number of the same modules that are available in the RHEL8 module tree, although specific modules may have different versions available in the RHEL9 tree as compared to the RHEL8 tree. If you have a dependency on a specific version of a module that is not available in the RHEL9 tree, please [[HelpDesk \| contact staff]] and we can get one created.		The [[Modules \| module tree]] for RHEL9 is populated with a large number of the same modules that are available in the RHEL8 module tree, although specific modules may have different versions available in the RHEL9 tree as compared to the RHEL8 tree. If you have a dependency on a specific version of a module that is not available in the RHEL9 tree, please [[HelpDesk \| contact staff]] and we can get one created.

MonthlyMaintenanceWindow

Mbaney — Fri, 24 Jul 2026 16:25:40 GMT

← Older revision		Revision as of 16:25, 24 July 2026
Line 8:		Line 8:


	* '''~~July 23rd~~ 2026'''		* '''August 20th 2026'''
	* August 20th 2026
	* September 17th 2026		* September 17th 2026
	* October 22nd 2026		* October 22nd 2026
Line 167:		Line 166:
	* May 28th 2026 (Adjusted date for CIO-imposed network change freeze)		* May 28th 2026 (Adjusted date for CIO-imposed network change freeze)
	* June 18th 2026		* June 18th 2026
			* July 23rd 2026

Nexus/ClusterOSUpgrade

Mbaney — Fri, 24 Jul 2026 15:41:19 GMT

← Older revision		Revision as of 15:41, 24 July 2026
(3 intermediate revisions by the same user not shown)
Line 1:		Line 1:
	==Overview==		==Overview==
	UMIACS Technical Staff has begun the process of upgrading the operating system version on all [[Nexus]] cluster nodes from [[RHEL \| Red Hat Enterprise Linux (RHEL)]] 8 to 9 as of 9am on 06/01/2026.		UMIACS Technical Staff has begun the process of upgrading the operating system version on all [[Nexus]] cluster nodes from [[RHEL \| Red Hat Enterprise Linux (RHEL)]] 8 to 9 as of 9am on Monday 06/01/2026.

	RHEL8 is in the Maintenance Support phase of its life cycle and is transitioning to the Extended Life phase in 2029. More information on Red Hat's lifecycle policy for its operating systems can be found [https://access.redhat.com/support/policy/updates/errata here]. We are staying well ahead of the Extended Life phase date for our cluster nodes by performing these upgrades now.		RHEL8 is in the Maintenance Support phase of its life cycle and is transitioning to the Extended Life phase in 2029. More information on Red Hat's lifecycle policy for its operating systems can be found [https://access.redhat.com/support/policy/updates/errata here]. We are staying well ahead of the Extended Life phase date for our cluster nodes by performing these upgrades now.
Line 7:		Line 7:

	==Scheduling==		==Scheduling==
	'''Upgrades for all cluster nodes have begun.''' We expect to be finished with all cluster node upgrades no later than Friday 08/21/2026 ~~at 5pm~~.		'''Upgrades for all cluster nodes have begun.''' We expect to be finished with all cluster node upgrades no later than 5pm on Friday 08/21/2026.

	===[[SLURM/JobSubmission \| Submission Nodes]]===		===[[SLURM/JobSubmission \| Submission Nodes]]===
	'''Submission nodes with the number '01' in their hostnames have been upgraded as of 06/01/2026.'''		'''Submission nodes with the number '01' in their hostnames have been upgraded as of 06/01/2026.'''

	Submission nodes with the number '00' in their hostnames will be scheduled for upgrade individually~~, when all of the compute nodes associated with~~ the ~~same lab/center have been upgraded~~. Staff will send a notification to individual lab's/center's cluster users to schedule the relevant '00' node's upgrade when ~~applicable~~. The actual date of each upgrade will be no less than one week after the corresponding notification has been sent.		Submission nodes with the number '00' in their hostnames, if not already upgraded, will be scheduled for upgrade individually in the near future. Staff will send a notification to individual lab's/center's cluster users to schedule the relevant '00' node's upgrade when ready. The actual date of each upgrade will be no less than one week after the corresponding notification has been sent.

	Data in [[FilesystemDataStorage#UNIX_Filesystem_Storage \| UNIX filesystem storage]] spaces on each submission node, i.e., /tmp and /scratch0, will not be preserved during upgrade. If you have any data in any such space on the '00' submission node in a pairing that you want to keep, please ensure you copy it to the '01' submission node or a [[FilesystemDataStorage#Network-Attached_Filesystem_Storage \| network-attached filesystem storage]] space prior to the '00' node's upgrade date. Data in network-attached filesystem storage spaces, such as /nfshomes or /fs/nexus-scratch, will not be affected.		Data in [[FilesystemDataStorage#UNIX_Filesystem_Storage \| UNIX filesystem storage]] spaces on each submission node, i.e., /tmp and /scratch0, will not be preserved during upgrade. If you have any data in any such space on the '00' submission node in a pairing that you want to keep, please ensure you copy it to the '01' submission node or a [[FilesystemDataStorage#Network-Attached_Filesystem_Storage \| network-attached filesystem storage]] space prior to the '00' node's upgrade date. Data in network-attached filesystem storage spaces, such as /nfshomes or /fs/nexus-scratch, will not be affected.

	===Compute Nodes===		===Compute Nodes===
	~~Due to the large number of~~ compute nodes and the desire to not interrupt running jobs, we are not generally able to schedule each specific compute node upgrade on a specific date. If you find that a specific node is unavailable to schedule jobs on, you can run the command <code>sinfo --list-reasons --long</code> on a submission node and look to see if the node is in the list with the text "RHEL9 upgrade" - if this is present, the upgrade for that node is underway.		'''All compute nodes have been upgraded as of 07/24/2026.'''

	~~We will generally be prioritizing upgrades for nodes based on how available they are across various partitions; nodes that are only available in partitions that contain large numbers~~ of ~~users for a lab~~/center, e.g., cbcb, clip, cml-dpart, gamma, vulcan-ampere, vulcan-dpart, etc., and corresponding "scavenger" named partitions, will be prioritized over nodes that are only or are also available in faculty-specific / ~~limited-node partitions~~. ~~All nodes in the tron partition will also generally be prioritized.~~

	~~If you are a faculty member authoritative for your own partition or a small group~~'s limited-node partition and have scheduling concerns for the nodes in these partitions, please [[HelpDesk \| contact staff]] ASAP to let us know about these concerns and we will make our best effort to accommodate them.

	==Interoperability==		==Interoperability==
Line 28:		Line 24:

	The [[Modules \| module tree]] for RHEL9 has already been populated with a large number of the same modules that are available in the RHEL8 module tree, although specific modules may have different versions available in the RHEL9 tree as compared to the RHEL8 tree. If you have a dependency on a specific version of a module that is not available in the RHEL9 tree, please [[HelpDesk \| contact staff]] and we can get one created.		The [[Modules \| module tree]] for RHEL9 has already been populated with a large number of the same modules that are available in the RHEL8 module tree, although specific modules may have different versions available in the RHEL9 tree as compared to the RHEL8 tree. If you have a dependency on a specific version of a module that is not available in the RHEL9 tree, please [[HelpDesk \| contact staff]] and we can get one created.

	~~===SLURM Scheduling===~~
	If you want or need to schedule a job on only nodes running RHEL8 (or RHEL9, once you have validated whatever is relevant), you can use the submission arguments <code>--prefer=rhel#</code> or <code>--constraint=rhel#</code> in your job arguments to specify this, where # is replaced by the OS version number. The --prefer argument is a soft limitation on which nodes the job can be scheduled on and the --constraint argument is a hard limitation, i.e., if you use the argument <code>--prefer=rhel8</code> but there are no RHEL8 nodes available at present (with your other submission arguments also satisfied) in the partition you are submitting to, the job will be scheduled on an appropriate RHEL9 node if that would result in an earlier (or instantaneous) start time.

Nexus/GPUs

Seide — Wed, 22 Jul 2026 15:33:07 GMT

Add B300 SXM6 AC

← Older revision		Revision as of 15:33, 22 July 2026
Line 12:		Line 12:
	! FP32 Performance (TFLOPS)		! FP32 Performance (TFLOPS)
	! [https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/ TF32] Performance ([https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt Dense / Sparse TOPS])		! [https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/ TF32] Performance ([https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt Dense / Sparse TOPS])
			\|-
			\| NVIDIA B300 SXM6 AC
			\| <code>b300-sxm</code>
			\| Blackwell
			\| 37888
			\| 288GB HBM3e
			\| 8.20 TB/s
			\| 153.98
			\| not officially published/not officially published
	\|-		\|-
	\| RTX PRO 6000 Blackwell Max-Q Workstation Edition		\| RTX PRO 6000 Blackwell Max-Q Workstation Edition

Nexus/GPUs

Mbaney — Fri, 17 Jul 2026 18:01:26 GMT

← Older revision		Revision as of 18:01, 17 July 2026
(3 intermediate revisions by the same user not shown)
Line 12:		Line 12:
	! FP32 Performance (TFLOPS)		! FP32 Performance (TFLOPS)
	! [https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/ TF32] Performance ([https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt Dense / Sparse TOPS])		! [https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/ TF32] Performance ([https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt Dense / Sparse TOPS])
			\|-
			\| RTX PRO 6000 Blackwell Max-Q Workstation Edition
			\| <code>rtx6000bw-mq</code>
			\| Blackwell
			\| 24064
			\| 96GB GDDR7
			\| 1.79 TB/s
			\| 109.7
			\| 219.5/438.9
	\|-		\|-
	\| L40S		\| L40S
Line 101:		Line 110:
	\| 448GB/s		\| 448GB/s
	\| 19.2		\| 19.2
	\| not officially published		\| not officially published/not officially published
	\|-		\|-
	\| RTX A5000		\| RTX A5000
Line 110:		Line 119:
	\| 768GB/s		\| 768GB/s
	\| 27.8		\| 27.8
	\| not officially published		\| not officially published/not officially published
	\|-		\|-
	\| RTX A6000		\| RTX A6000
Line 184:		Line 193:
	\| n/a		\| n/a
	\|-		\|-
	~~\| RTX PRO 6000 Blackwell Max-Q Workstation~~
	~~\| <code>rtx6000bw-mq</code>~~
	~~\| Blackwell~~
	~~\| 24064~~
	~~\| 96 GB GDDR7~~
	~~\| 1.79 TB/s~~
	~~\| 126.0~~
	~~\| n/a~~
	~~\| -~~
	\|}		\|}

	[0] - This GPU type is actually a pair of two physical cards connected over [https://www.nvidia.com/en-us/data-center/nvlink NVLink] bridges. NVIDIA's provided specifications for this GPU type are for one physical card; to get these specs, we have hence doubled NVIDIA's provided values.		[0] - This GPU type is actually a pair of two physical cards connected over [https://www.nvidia.com/en-us/data-center/nvlink NVLink] bridges. NVIDIA's provided specifications for this GPU type are for one physical card; to get these specs, we have hence doubled NVIDIA's provided values.

CML

Mbaney — Fri, 17 Jul 2026 17:55:06 GMT

← Older revision		Revision as of 17:55, 17 July 2026
Line 1:		Line 1:
	The Center for Machine Learning ([https://ml.umd.edu CML]) at the University of Maryland is located within the Institute for Advanced Computer Studies. The CML has a cluster of computational (CPU/GPU) resources that are available to be scheduled.		The Center for Machine Learning ([https://ml.umd.edu CML]) at the University of Maryland is located within the Institute for Advanced Computer Studies. The CML has a cluster of computational (CPU/GPU) resources that are available to be scheduled.

	<span style="font-size:150%">'''As of the [[MonthlyMaintenanceWindow \| August 2023 maintenance window]], all compute nodes have moved into the [[Nexus]] cluster.''' Please see [[Nexus/CML]] for more details.</span>

	=Getting Started=		=Getting Started=

Nexus/Vulcan

Mbaney — Fri, 17 Jul 2026 17:52:38 GMT

Partitions

← Older revision		Revision as of 17:52, 17 July 2026
(One intermediate revision by the same user not shown)
Line 11:		Line 11:
	* <code>nexusvulcan00.umiacs.umd.edu</code>		* <code>nexusvulcan00.umiacs.umd.edu</code>
	* <code>nexusvulcan01.umiacs.umd.edu</code>		* <code>nexusvulcan01.umiacs.umd.edu</code>

	All partitions, QoSes, and account names from the standalone Vulcan cluster have been moved over to Nexus. However, please note that <code>vulcan-</code> is prepended to all of the values that were present in the standalone Vulcan cluster to distinguish them from existing values in Nexus. The lone exception is the base account that was named <code>vulcan</code> in the standalone cluster (it is also named just <code>vulcan</code> in Nexus).

	~~Here are some before/after examples of job submission with various parameters:~~

	~~{\| class="wikitable"~~
	~~! Standalone Vulcan cluster submission command~~
	~~! Nexus cluster submission command~~
	\|-
	~~\|<code>srun --partition=dpart --qos=medium --account=abhinav --gres=gpu:gtx1080ti:2 --pty bash</code>~~
	~~\|<code>srun --partition=vulcan-dpart --qos=vulcan-medium --account=vulcan-abhinav --gres=gpu:gtx1080ti:2 --pty bash</code>~~
	\|-
	~~\|<code>srun --partition=cpu --qos=cpu --pty bash</code>~~
	~~\|<code>srun --partition=vulcan-cpu --qos=vulcan-cpu --account=vulcan --pty bash</code>~~
	\|-
	~~\|<code>srun --partition=scavenger --qos=scavenger --account=vulcan --gres=gpu:4 --pty bash</code>~~
	~~\|<code>srun --partition=vulcan-scavenger --qos=vulcan-scavenger --account=vulcan --gres=gpu:4 --pty bash</code>~~
	\|}

	Vulcan users (exclusively) can schedule non-interruptible jobs on Vulcan nodes with any non-scavenger job parameters. Please note that the <code>vulcan-dpart</code> partition has a <code>GrpTRES</code> limit of 100% of the available cores/RAM on all vulcan## in aggregate nodes plus 50% of the available cores/RAM on legacy## nodes in aggregate, so your job may need to wait if all available cores/RAM (or GPUs) are in use. It also has a max submission limit of 500 jobs per user simultaneously so as to not overload the cluster. This is codified by the partition QoS named '''vulcan'''.		Vulcan users (exclusively) can schedule non-interruptible jobs on Vulcan nodes with any non-scavenger job parameters. Please note that the <code>vulcan-dpart</code> partition has a <code>GrpTRES</code> limit of 100% of the available cores/RAM on all vulcan## in aggregate nodes plus 50% of the available cores/RAM on legacy## nodes in aggregate, so your job may need to wait if all available cores/RAM (or GPUs) are in use. It also has a max submission limit of 500 jobs per user simultaneously so as to not overload the cluster. This is codified by the partition QoS named '''vulcan'''.
Line 35:		Line 17:

	==Compute Nodes==		==Compute Nodes==
	There are currently 46 [[Nexus/Vulcan/GPUs \| GPU nodes]] available, named vulcan[00-45], running a mixture of NVIDIA RTX A6000, NVIDIA RTX A5000, NVIDIA RTX A4000, and ~~a number of different older generation~~ cards. There are also 4 CPU-only nodes available, named brigid[16-19].		There are currently 22 [[Nexus/Vulcan/GPUs \| GPU nodes]] available, named vulcan[23-24,27-46], running a mixture of NVIDIA H200, NVIDIA RTX A6000, NVIDIA RTX A5000, NVIDIA RTX A4000, and NVIDIA GeForce RTX 2080 Ti cards. There are also 2 CPU-only nodes available, named brigid[16-17].

	All nodes are scheduled with the [[SLURM]] resource manager.		All nodes are scheduled with the [[SLURM]] resource manager.
Line 71:		Line 53:
	:* Matthias Zwicker (vulcan-zwicker)		:* Matthias Zwicker (vulcan-zwicker)

	* '''vulcan-scavenger-multi''' - This partition allows multi-node jobs (up to 9 total nodes per job) and allows jobs more resources than the vulcan-scavenger partition, but only contains nodes with ~~GTX 1080 Ti, TITAN Xp, and/or~~ RTX 2080 Ti GPUs in them. As with vulcan-scavenger, it is preemptable when jobs in other non-scavenged-named <code>vulcan-</code> partitions are ready to be scheduled.		* '''vulcan-scavenger-multi''' - This partition allows multi-node jobs (up to 9 total nodes per job) and allows jobs more resources than the vulcan-scavenger partition, but only contains nodes with RTX 2080 Ti GPUs in them. As with vulcan-scavenger, it is preemptable when jobs in other non-scavenged-named <code>vulcan-</code> partitions are ready to be scheduled.
	*: Access to this partition is on a per-use basis. Please contact Abhinav Shrivastava if you would like to be granted access to this partition.		*: Access to this partition is on a per-use basis. Please contact Abhinav Shrivastava if you would like to be granted access to this partition.

Nexus/CLIP

Mbaney — Fri, 17 Jul 2026 17:48:57 GMT

Nexus/CBCB

Mbaney — Fri, 17 Jul 2026 17:46:02 GMT

Compute Nodes

← Older revision		Revision as of 17:46, 17 July 2026
(2 intermediate revisions by the same user not shown)
Line 14:		Line 14:
	= Compute Nodes =		= Compute Nodes =
	All compute nodes in CBCB-owned partitions (see below section) owned by CBCB faculty are named in the format <code>cbcb##</code>. The sets of nodes are:		All compute nodes in CBCB-owned partitions (see below section) owned by CBCB faculty are named in the format <code>cbcb##</code>. The sets of nodes are:
	* 22 nodes that were purchased in October 2022 with center-wide funding~~. They are~~ cbcb[00-21].		* 22 nodes that were purchased in October 2022 with center-wide funding: cbcb[00-21]
	* ~~4 nodes~~ from the previous standalone CBCB cluster that moved in as of Summer 2023~~. They are cbcb[22-25].~~		* 1 node from the previous standalone CBCB cluster that moved in as of Summer 2023: cbcb25
	* 4 additional nodes purchased by Dr. Heng Huang~~. They are~~ cbcb[26-29].		* 4 additional nodes purchased by Dr. Heng Huang: cbcb[26-29]
	* 1 additional node purchased by Dr. Mihai Pop~~. It is~~ cbcb30.		* 1 additional node purchased by Dr. Mihai Pop: cbcb30

	{\| class="wikitable sortable"		{\| class="wikitable sortable"
Line 33:		Line 33:
	\|~350GB (SATA SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch0]]), ~2TB (NVMe SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch1]])		\|~350GB (SATA SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch0]]), ~2TB (NVMe SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch1]])
	\|0		\|0
	\|-		\|-
	~~\|cbcb22~~
	\|1
	~~\|28 (Dual [https://ark.intel.com/content/www/us/en/ark/products/91754/intel-xeon-processor-e5-2680-v4-35m-cache-2-40-ghz.html Intel Xeon E5-2680 v4])~~
	~~\|~768GB (DDR4 2400MHz)~~
	~~\|~650GB (SATA SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch0]])~~
	\|0
	\|-
	~~\|cbcb[23-24]~~
	\|2
	~~\|24 (Dual [https://www.intel.com/content/www/us/en/products/sku/91767/intel-xeon-processor-e52650-v4-30m-cache-2-20-ghz/specifications.html Intel Xeon E5-2650 v4])~~
	~~\|~256GB (DDR4 2400MHz)~~
	~~\|~800GB (SATA SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch0]])~~
	\|0
	\|-
	\|cbcb25		\|cbcb25
	\|1		\|1
Line 54:		Line 40:
	\|~1.4TB (SATA SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch0]])		\|~1.4TB (SATA SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch0]])
	\|2 (1x [https://www.nvidia.com/en-gb/geforce/graphics-cards/geforce-gtx-1080-ti/specifications/ NVIDIA GeForce GTX 1080 Ti], 1x [https://www.nvidia.com/en-us/geforce/graphics-cards/compare/?section=compare-20 NVIDIA GeForce RTX 2080 Ti])		\|2 (1x [https://www.nvidia.com/en-gb/geforce/graphics-cards/geforce-gtx-1080-ti/specifications/ NVIDIA GeForce GTX 1080 Ti], 1x [https://www.nvidia.com/en-us/geforce/graphics-cards/compare/?section=compare-20 NVIDIA GeForce RTX 2080 Ti])
	\|-		\|-
	\|cbcb26		\|cbcb26
	\|1		\|1
Line 61:		Line 47:
	\|~3.4TB (NVMe SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch0]]), ~14TB (NVMe SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch1]])		\|~3.4TB (NVMe SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch0]]), ~14TB (NVMe SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch1]])
	\|7 ([https://www.nvidia.com/en-us/design-visualization/rtx-a5000 NVIDIA RTX A5000])		\|7 ([https://www.nvidia.com/en-us/design-visualization/rtx-a5000 NVIDIA RTX A5000])
	\|-		\|-
	\|cbcb27		\|cbcb27
	\|1		\|1
Line 68:		Line 54:
	\|~3.4TB (SATA SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch0]]), ~3.5TB (NVMe SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch1]])		\|~3.4TB (SATA SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch0]]), ~3.5TB (NVMe SSD [[FilesystemDataStorage#UNIX_Filesystem_Storage \| /scratch1]])
	\|8 ([https://www.nvidia.com/en-us/design-visualization/rtx-a6000 NVIDIA RTX A6000])		\|8 ([https://www.nvidia.com/en-us/design-visualization/rtx-a6000 NVIDIA RTX A6000])
	\|-		\|-
	\|cbcb[28-29]		\|cbcb[28-29]
	\|2		\|2
Line 84:		Line 70:
	\|- class="sortbottom"		\|- class="sortbottom"
	!Total		!Total
	\|31		\|28
	\|~~1108~~ (various)		\|1032 (various)
	\|~~~50TB~~ (various)		\|~49TB (various)
	\|~~~105TB~~ (various)		\|~103TB (various)
	\|33 (various)		\|33 (various)
	\|}		\|}
Line 117:		Line 103:
	cbcb20 32 2061175 rhel8,x86_64,Zen,EPYC-7313 (null) idle		cbcb20 32 2061175 rhel8,x86_64,Zen,EPYC-7313 (null) idle
	cbcb21 32 2061175 rhel8,x86_64,Zen,EPYC-7313 (null) idle		cbcb21 32 2061175 rhel8,x86_64,Zen,EPYC-7313 (null) idle
	~~cbcb22 28 771245 rhel8,x86_64,Xeon,E5-2680 (null) idle~~
	~~cbcb23 24 255150 rhel8,x86_64,Xeon,E5-2650 (null) idle~~
	~~cbcb24 24 255150 rhel8,x86_64,Xeon,E5-2650 (null) idle~~
	cbcb25 24 255278 rhel8,x86_64,Xeon,E5-2650,Pascal,Turing gpu:rtx2080ti:1,gpu:gtx1080ti:1 idle		cbcb25 24 255278 rhel8,x86_64,Xeon,E5-2650,Pascal,Turing gpu:rtx2080ti:1,gpu:gtx1080ti:1 idle
	cbcb26 128 513243 rhel8,x86_64,Zen,EPYC-7763,Ampere gpu:rtxa5000:7 idle		cbcb26 128 513243 rhel8,x86_64,Zen,EPYC-7763,Ampere gpu:rtxa5000:7 idle
Line 133:		Line 116:
	#* cbcb[00-21,26-30]: Two 100GbE links per node, one to each switch in the pair (redundancy).		#* cbcb[00-21,26-30]: Two 100GbE links per node, one to each switch in the pair (redundancy).
	# One pair of network switches connected to the above pair of network switches via four 40GbE links, one between every combination of switches across the two pairings for redundancy, and to each other via dual 10GbE links for redundancy.		# One pair of network switches connected to the above pair of network switches via four 40GbE links, one between every combination of switches across the two pairings for redundancy, and to each other via dual 10GbE links for redundancy.
	#* ~~cbcb[22-25]~~: Two 10GbE links ~~per node~~, one to each switch in the pair (redundancy).		#* cbcb25: Two 10GbE links, one to each switch in the pair (redundancy).

	For a broader overview of the network infrastructure supporting the Nexus cluster, please see [[Nexus/Network]].		For a broader overview of the network infrastructure supporting the Nexus cluster, please see [[Nexus/Network]].

Nexus/Vulcan

Mbaney — Fri, 17 Jul 2026 16:11:05 GMT

Network

← Older revision		Revision as of 16:11, 17 July 2026
(One intermediate revision by the same user not shown)
Line 45:		Line 45:
	#* vulcan46: Four 100GbE links, two to each switch in the pair (redundancy and increased bandwidth).		#* vulcan46: Four 100GbE links, two to each switch in the pair (redundancy and increased bandwidth).
	# One pair of network switches connected to the above pair of switches through several sets of intermediary switches, and to each other via dual 10GbE links for redundancy. The immediate connection to these sets of intermediary switches is via two 40GbE links to a pair of them, one between the first two switches in each pair and one between the second two switches in each pair for redundancy. This pair serves the following compute nodes:		# One pair of network switches connected to the above pair of switches through several sets of intermediary switches, and to each other via dual 10GbE links for redundancy. The immediate connection to these sets of intermediary switches is via two 40GbE links to a pair of them, one between the first two switches in each pair and one between the second two switches in each pair for redundancy. This pair serves the following compute nodes:
	#* ~~brigid~~[18-~~19]~~,~~vulcan[00~~-28]: Two 10GbE links per node, one to each switch in the pair (redundancy).		#* vulcan[23-24,27-28]: Two 10GbE links per node, one to each switch in the pair (redundancy).

	The fileserver hosting all Vulcan [[Nexus/Vulcan#Scratch_Directories \| scratch]], [[Nexus/Vulcan#Project_Storage \| project]], and [[Nexus/Vulcan#Datasets \| dataset]] allocations first connects to a pair of intermediary switches and then the first pair of switches mentioned [[Nexus/Tron#Network \| here (Tron page's network section)]]. It then connects to the first pair of switches mentioned on this page through a set of four (different) intermediary switches. The last hop from the four intermediary switches to the first pair of switches mentioned on this page is via 32 100GbE links, four from each switch in the set to each switch in the first pair mentioned on this page for redundancy and increased bandwidth.		The fileserver hosting all Vulcan [[Nexus/Vulcan#Scratch_Directories \| scratch]], [[Nexus/Vulcan#Project_Storage \| project]], and [[Nexus/Vulcan#Datasets \| dataset]] allocations first connects to a pair of intermediary switches and then the first pair of switches mentioned [[Nexus/Tron#Network \| here (Tron page's network section)]]. It then connects to the first pair of switches mentioned on this page through a set of four (different) intermediary switches. The last hop from the four intermediary switches to the first pair of switches mentioned on this page is via 32 100GbE links, four from each switch in the set to each switch in the first pair mentioned on this page for redundancy and increased bandwidth.

Nexus/CML

Mbaney — Fri, 17 Jul 2026 16:10:08 GMT

Network

← Older revision		Revision as of 16:10, 17 July 2026
(One intermediate revision by the same user not shown)
Line 20:		Line 20:
	# One pair of network switches connected to each other via dual 100GbE links for redundancy, serving the following compute nodes:		# One pair of network switches connected to each other via dual 100GbE links for redundancy, serving the following compute nodes:
	#* cml[17-28,30-32,34,37]: Two 100GbE links per node, one to each switch in the pair (redundancy).		#* cml[17-28,30-32,34,37]: Two 100GbE links per node, one to each switch in the pair (redundancy).
	#* cml[35-36]: Four 100GbE links, two to each switch in the pair (redundancy and increased bandwidth).		#* cml[35-36,38]: Four 100GbE links per node, two to each switch in the pair (redundancy and increased bandwidth).
	# One pair of network switches connected to the above pair of network switches via two 100GbE links, one between the first two switches in each pair and one between the second two switches in each pair for redundancy, and to each other via dual 25GbE links for redundancy.		# One pair of network switches connected to the above pair of network switches via two 100GbE links, one between the first two switches in each pair and one between the second two switches in each pair for redundancy, and to each other via dual 25GbE links for redundancy.
	#* cml[00,02-09]: Two 25GbE links per node, one to each switch in the pair (redundancy).		#* cml[00,02-09]: Two 25GbE links per node, one to each switch in the pair (redundancy).

Nexus

Mbaney — Fri, 17 Jul 2026 16:09:16 GMT

← Older revision		Revision as of 16:09, 17 July 2026
Line 75:		Line 75:
	clip[00-13]		clip[00-13]
	cml[00,02-13,15-28,30-33]		cml[00,02-13,15-28,30-33]
	~~cmlcpu[00-04,06-07]~~
	gammagpu[00-21]		gammagpu[00-21]
	legacy[00-11,13-28,30-36]		legacy[00-11,13-28,30-36]
	legacygpu[00-07]		legacygpu[00-07]
	quics00		quics00
	tron[00~~-44,46~~-69]		tron[00-69]
	vulcan[00-45]		vulcan[00-45]
	------------------------------------------------------------------------------------------------------------------------		------------------------------------------------------------------------------------------------------------------------
	scavenger-aarch64 scavenger scavenger-aarch64 UNLIMITED oasis[00-39]		scavenger-aarch64 scavenger scavenger-aarch64 UNLIMITED oasis[00-39]
	------------------------------------------------------------------------------------------------------------------------		------------------------------------------------------------------------------------------------------------------------
	tron nexus default UNLIMITED tron[00~~-44,46~~-69]		tron nexus default UNLIMITED tron[00-69]
	high		high
	medium		medium

SLURM/JobSubmission

Mbaney — Fri, 17 Jul 2026 16:07:43 GMT

← Older revision		Revision as of 16:07, 17 July 2026
Line 129:		Line 129:
	NODELIST CPUS MEMORY AVAIL_FEATURES GRES		NODELIST CPUS MEMORY AVAIL_FEATURES GRES
	cbcb[00-21] 32 2061175 rhel8,x86_64,Zen,EPYC-7313 (null)		cbcb[00-21] 32 2061175 rhel8,x86_64,Zen,EPYC-7313 (null)
	cbcb22~~,cmlcpu[00,06-07]~~,legacy20 24+ 384270+ rhel8,x86_64,Xeon,E5-2680 (null)		cbcb22,legacy20 24+ 384270+ rhel8,x86_64,Xeon,E5-2680 (null)
	cbcb26 128 513243 rhel8,x86_64,Zen,EPYC-7763,Ampere gpu:rtxa5000:7		cbcb26 128 513243 rhel8,x86_64,Zen,EPYC-7763,Ampere gpu:rtxa5000:7
	cbcb27 64 255167 rhel8,x86_64,Zen,EPYC-7513,Ampere gpu:rtxa6000:8		cbcb27 64 255167 rhel8,x86_64,Zen,EPYC-7513,Ampere gpu:rtxa6000:8
Line 149:		Line 149:
	cml32 64 512999 rhel8,x86_64,Zen,EPYC-7543,Ampere gpu:a100:4		cml32 64 512999 rhel8,x86_64,Zen,EPYC-7543,Ampere gpu:a100:4
	cml33 64 1029019 rhel8,x86_64,Xeon,6448Y,Hopper gpu:h100-sxm:4		cml33 64 1029019 rhel8,x86_64,Xeon,6448Y,Hopper gpu:h100-sxm:4
	~~cmlcpu[01-04] 20 384271 rhel8,x86_64,Xeon,E5-2660 (null)~~
	cml[00,15-16] 32 351530+ rhel8,x86_64,Xeon,4216,Turing gpu:rtx2080ti:7		cml[00,15-16] 32 351530+ rhel8,x86_64,Xeon,4216,Turing gpu:rtx2080ti:7
	cml01 32 383030 rhel8,x86_64,Xeon,4216,Turing gpu:rtx2080ti:6		cml01 32 383030 rhel8,x86_64,Xeon,4216,Turing gpu:rtx2080ti:6

Nexus/CML

Mbaney — Thu, 16 Jul 2026 13:06:27 GMT

← Older revision		Revision as of 13:06, 16 July 2026
Line 23:		Line 23:
	# One pair of network switches connected to the above pair of network switches via two 100GbE links, one between the first two switches in each pair and one between the second two switches in each pair for redundancy, and to each other via dual 25GbE links for redundancy.		# One pair of network switches connected to the above pair of network switches via two 100GbE links, one between the first two switches in each pair and one between the second two switches in each pair for redundancy, and to each other via dual 25GbE links for redundancy.
	#* cml[00,02-09]: Two 25GbE links per node, one to each switch in the pair (redundancy).		#* cml[00,02-09]: Two 25GbE links per node, one to each switch in the pair (redundancy).
	#* cml[10-16~~],cmlcpu[01-04,06-07~~]: Two 10GbE links per node, one to each switch in the pair (redundancy).		#* cml[10-16]: Two 10GbE links per node, one to each switch in the pair (redundancy).

	The fileserver hosting all CML [[Nexus/CML#Project_Directories \| project]], [[Nexus/CML#Scratch_Directories \| scratch]], [[Nexus/CML#Datasets \| dataset]], and [[Nexus/CML#Models \| model]] allocations also connects to the same pair of switches supporting cml[17-28,30-32] via fourteen 25GbE links, seven to each switch in the pair for redundancy and increased bandwidth.		The fileserver hosting all CML [[Nexus/CML#Project_Directories \| project]], [[Nexus/CML#Scratch_Directories \| scratch]], [[Nexus/CML#Datasets \| dataset]], and [[Nexus/CML#Models \| model]] allocations also connects to the same pair of switches supporting cml[17-28,30-32] via fourteen 25GbE links, seven to each switch in the pair for redundancy and increased bandwidth.
Line 30:		Line 30:

	==Partitions==		==Partitions==
	There are ~~three~~ partitions available to general CML [[SLURM]] users. You must specify a partition when submitting your job.		There are two partitions available to general CML [[SLURM]] users. You must specify a partition when submitting your job.

	* '''cml-dpart''' - This is the default partition. Job allocations are guaranteed.		* '''cml-dpart''' - This is the default partition. Job allocations are guaranteed.
	* '''cml-scavenger''' - This is the alternate partition that allows jobs longer run times and more resources but is preemptable when jobs in other <code>cml-</code> partitions are ready to be scheduled.		* '''cml-scavenger''' - This is the alternate partition that allows jobs longer run times and more resources but is preemptable when jobs in other <code>cml-</code> partitions are ready to be scheduled.
	* '''cml-cpu''' - This partition is for CPU focused jobs. Job allocations are guaranteed. '''Please note that this partition is being permanently removed on 07/16/2026 at 9am.'''

	There are a few additional partitions available solely to specific faculty members and their sponsored user accounts.		There are a few additional partitions available solely to specific faculty members and their sponsored user accounts.
Line 49:		Line 48:
	The Center has a base SLURM account <code>cml</code> which has a modest number of guaranteed billing resources available to all cluster users at any given time. Other faculty that have invested in the cluster have an additional account provided to their sponsored accounts on the cluster, which provides a number of guaranteed billing resources corresponding to the amount that they invested.		The Center has a base SLURM account <code>cml</code> which has a modest number of guaranteed billing resources available to all cluster users at any given time. Other faculty that have invested in the cluster have an additional account provided to their sponsored accounts on the cluster, which provides a number of guaranteed billing resources corresponding to the amount that they invested.

	If you do not specify an account when submitting your job, you will receive the '''cml''' account, which only has access to the ~~'''cml-cpu''',~~ '''cml-default''', and '''cml-medium''' QoSes (see below section).		If you do not specify an account when submitting your job, you will receive the '''cml''' account, which only has access to the '''cml-default''' and '''cml-medium''' QoSes (see below section).

	If you need access to a different QoS, or if the '''cml''' account is at its billing limit (see below in this section), please use your faculty sponsor's account if they have one available. However, keep in mind that if you use your faculty sponsor has their own named partition (see previous section), using the faculty-specific account in the '''cml-dpart''' partition may block access to resources in the faculty-specific partition, since the billing limit for the account is charged regardless of what partition is being used.		If you need access to a different QoS, or if the '''cml''' account is at its billing limit (see below in this section), please use your faculty sponsor's account if they have one available. However, keep in mind that if you use your faculty sponsor has their own named partition (see previous section), using the faculty-specific account in the '''cml-dpart''' partition may block access to resources in the faculty-specific partition, since the billing limit for the account is charged regardless of what partition is being used.
Line 88:		Line 87:
	---------- ---------------- ------- ------------- --------------------------------------------------		---------- ---------------- ------- ------------- --------------------------------------------------
	... ... ...		... ... ...
	tomg cml ~~cml-cpu,~~cml-default,cml-medium		tomg cml cml-default,cml-medium
	tomg cml-scavenger cml-scavenger		tomg cml-scavenger cml-scavenger
	tomg cml-tomg cml-default,cml-high,cml-medium		tomg cml-tomg cml-default,cml-high,cml-medium
Line 105:		Line 104:

	==QoS==		==QoS==
	CML currently has 5 QoS for the '''cml-dpart''' partition (though <code>cml-high_long</code> and <code>cml-very_high</code> may not be available to all faculty accounts), 1 QoS for the '''cml-scavenger~~''' partition, and 1 QoS for the '''cml-cpu~~''' partition. If you do not specify a QoS when submitting your job using the <code>--qos</code> parameter, you will receive the <code>cml-default</code> QoS assuming you are using a CML account.		CML currently has 5 QoS for the '''cml-dpart''' partition (though <code>cml-high_long</code> and <code>cml-very_high</code> may not be available to all faculty accounts) and 1 QoS for the '''cml-scavenger''' partition. If you do not specify a QoS when submitting your job using the <code>--qos</code> parameter, you will receive the <code>cml-default</code> QoS assuming you are using a CML account.

	If your faculty member's Slurm account does not have one or both of the <code>cml-high_long</code> or <code>cml-very_high</code> QoS available to it, we can add it to their account provided they approve. Please [[HelpDesk \| contact staff]] if this is desired.		If your faculty member's Slurm account does not have one or both of the <code>cml-high_long</code> or <code>cml-very_high</code> QoS available to it, we can add it to their account provided they approve. Please [[HelpDesk \| contact staff]] if this is desired.
Line 115:		Line 114:
	Name MaxWall MaxTRES MaxJobsPU MaxTRESPU		Name MaxWall MaxTRES MaxJobsPU MaxTRESPU
	-------------------- ----------- ------------------------------ --------- ------------------------------		-------------------- ----------- ------------------------------ --------- ------------------------------
	...		...
	~~cml-cpu 7-00:00:00 8~~
	cml-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2		cml-default 7-00:00:00 cpu=4,gres/gpu=1,mem=32G 2
	cml-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2		cml-high 1-12:00:00 cpu=16,gres/gpu=4,mem=128G 2
Line 132:		Line 130:
	...		...
	cml 500 cpu=1128,mem=11T		cml 500 cpu=1128,mem=11T
	~~cml-cpu 500~~
	cml-director 500		cml-director 500
	cml-furongh 500		cml-furongh 500

SLURM/Priority

Mbaney — Fri, 10 Jul 2026 12:57:39 GMT

Age

← Older revision		Revision as of 12:57, 10 July 2026
Line 86:		Line 86:

	===Age===		===Age===
	The longer a job is eligible to run but cannot due to resources being unavailable or it having a lower priority value than one or more other jobs, the higher the job's priority becomes as it continues to wait in the queue. This is the only priority ~~modifier~~ that can change a job's priority value ~~once it has been submitted~~, and the priority modifier for this factor reaches its limit after 7 days.		The longer a job is eligible to run but cannot due to resources being unavailable or it having a lower priority value than one or more other jobs, the higher the job's priority becomes as it continues to wait in the queue. This is the only priority factor that can change a job's priority value after submission, and the priority modifier for this factor reaches its limit after 7 days.

	Jobs' age priority factors on our clusters are recalculated every 5 minutes.		Jobs' age priority factors on our clusters are recalculated every 5 minutes.

SecureShell

Seide — Thu, 09 Jul 2026 22:05:41 GMT

Verify remote host SSH fingerprint: Update documentation for verifying SSH host keys. Add info about "REMOTE HOST IDENTIFICATION HAS CHANGED!" warning.

← Older revision		Revision as of 22:05, 9 July 2026
Line 42:		Line 42:

	==Verify remote host SSH fingerprint==		==Verify remote host SSH fingerprint==
	~~The SSH protocol relies on host keys to verify the identify of a given host. Each host as a unique key for the various different protocols supported.~~

	When connecting to a ~~remove~~ host for the first time~~, or when the remote host's local host key configuration has changed~~, you may see the following prompt:		The SSH protocol relies on host keys to verify the identity of a given host. Each host has a unique key for the various different protocols supported.

			When connecting to a remote host for the first time, you may see the following prompt:

	<pre>		<pre>
	$ ssh username@~~nexusgroup~~.umiacs.umd.edu		$ ssh username@nexusstaff.umiacs.umd.edu
	The authenticity of host '~~nexusgroup~~.umiacs.umd.edu (128.8.~~121~~.73)' can't be established.		The authenticity of host 'nexusstaff.umiacs.umd.edu (128.8.132.230)' can't be established.
	~~RSA~~ key fingerprint is 25:83:~~aa:df:f5:ad:5f:08:c9:8a:a3:5d:97:8b:48:1f~~.		ED25519 key fingerprint is: SHA256:N8+6sOAdq1NwvrpdHehe5MjQdnscNTBxvncqlHpt94o
	Are you sure you want to continue connecting (yes/no)?		This key is not known by any other names.
			Are you sure you want to continue connecting (yes/no/[fingerprint])?
	</pre>		</pre>
	~~It is considered best practice to verify the key fingerprint with the actual key of the host. UMIACS maintains a reference of SSH key fingerprints available at the following link:~~
	~~https://intranet.umiacs.umd.edu/hostkeys~~

	It is important to note that each key type has a different fingerprint. Depending on your local configuration, your client may prefer a specific type of key. The following commands ~~can be used to determine~~ the ~~fingerprint~~ of ~~a given~~ key ~~type on a remote host~~:		It is considered best practice to verify the key fingerprint with the actual key of the host. It is important to note that each key type has a different fingerprint. Depending on your local configuration, your client may prefer a specific type of key. The following commands retrieves all SSH public host keys from the server and prints out the fingerprints of each key in the MD5 format:

	<pre>		<pre>
	$ ssh-keyscan ~~-t rsa nexusgroup~~.umiacs.umd.edu > key		$ ssh-keyscan nexusstaff.umiacs.umd.edu > key
	~~# nexusgroup.umiacs.umd.edu:22 SSH-2.0-OpenSSH_8.0~~		$ ssh-keygen -l -E md5 -f key
	$ ssh-keygen -l -E md5 -f key		3072 MD5:aa:8f:e7:95:3f:ee:7c:23:0f:6f:9a:64:78:49:25:37 nexusstaff.umiacs.umd.edu (RSA)
	~~2048~~ MD5:25:83:aa:df:f5:ad:5f:08:c9:8a:a3:5d:97:8b:48:~~1f nexusgroup~~.umiacs.umd.edu (~~RSA~~)		256 MD5:34:29:ea:73:98:98:e0:7b:d4:c3:83:50:bc:cb:b3:61 nexusstaff.umiacs.umd.edu (ECDSA)
			256 MD5:02:50:b3:3e:d7:65:fb:08:54:30:f3:8d:7d:48:02:fc nexusstaff.umiacs.umd.edu (ED25519)
	</pre>		</pre>

			UMIACS maintains a reference of SSH key fingerprints available at the following link:
			https://intranet.umiacs.umd.edu/hostkeys

	If you have any questions, or notice a discrepancy, please [[HelpDesk \| contact staff]].		If you have any questions, or notice a discrepancy, please [[HelpDesk \| contact staff]].
Line 75:		Line 80:
	* [https://winscp.net/eng/docs/faq_hostkey WinSCP]		* [https://winscp.net/eng/docs/faq_hostkey WinSCP]
	* [https://mobaxterm.mobatek.net/ MobaXterm]		* [https://mobaxterm.mobatek.net/ MobaXterm]

			==Remote Host Identification Has Changed==

			If a host key has changed since you last connected, you may see the following warning:

			<pre>
			$ ssh username@nexusstaff.umiacs.umd.edu
			@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
			@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
			@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
			IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
			Someone could be eavesdropping on you right now (man-in-the-middle attack)!
			It is also possible that a host key has just been changed.
			The fingerprint for the ED25519 key sent by the remote host is
			SHA256:N8+6sOAdq1NwvrpdHehe5MjQdnscNTBxvncqlHpt94o.
			Please contact your system administrator.
			Add correct host key in /Users/username/.ssh/known_hosts to get rid of this message.
			Offending ED25519 key in /Users/username/.ssh/known_hosts:86
			Host key for nexusstaff.umiacs.umd.edu has changed and you have requested strict checking.
			Host key verification failed.
			</pre>

			By default, SSH clients should refuse to connect when a remote host key has changed. If you see this warning, follow the instructions below to resolve.

			# [[#Verify remote host SSH fingerprint \| Verify]] that the new fingerprint matches the fingerprint we publish at https://intranet.umiacs.umd.edu/hostkeys.
			# Once verified, open the known_hosts file and go to the line referenced in the warning (in this example, /Users/username/.ssh/known_hosts at line 86) and remove the entry. Next time you connect, it will prompt you as if it were your first time connecting to the host.
			<strong>NOTE:</strong> This may need to be repeated if your known_hosts file has multiple types of keys for this host.

	==Long Running Processes==		==Long Running Processes==

Nexus/GPUs

Vsammeta — Thu, 09 Jul 2026 17:04:08 GMT

added rtx pro 6000 blackwell entries

← Older revision		Revision as of 17:04, 9 July 2026
Line 184:		Line 184:
	\| n/a		\| n/a
	\|-		\|-
			\| RTX PRO 6000 Blackwell Max-Q Workstation
			\| <code>rtx6000bw-mq</code>
			\| Blackwell
			\| 24064
			\| 96 GB GDDR7
			\| 1.79 TB/s
			\| 126.0
			\| n/a
			\| -
	\|}		\|}

	[0] - This GPU type is actually a pair of two physical cards connected over [https://www.nvidia.com/en-us/data-center/nvlink NVLink] bridges. NVIDIA's provided specifications for this GPU type are for one physical card; to get these specs, we have hence doubled NVIDIA's provided values.		[0] - This GPU type is actually a pair of two physical cards connected over [https://www.nvidia.com/en-us/data-center/nvlink NVLink] bridges. NVIDIA's provided specifications for this GPU type are for one physical card; to get these specs, we have hence doubled NVIDIA's provided values.

Nexus/Tron

Mbaney — Wed, 08 Jul 2026 17:22:08 GMT

← Older revision		Revision as of 17:22, 8 July 2026
Line 57:		Line 57:
	#* tron[62-69]: Two 10GbE links to the switch per node (increased bandwidth).		#* tron[62-69]: Two 10GbE links to the switch per node (increased bandwidth).

	The fileserver hosting all Nexus [[Nexus#Scratch_Directories \| scratch]], [[Nexus#Faculty_Allocations \| faculty]], [[Nexus#Project_Allocations \| project]], and [[Nexus#Datasets \| dataset]] allocations first connects to a pair of intermediary switches before reaching the compute nodes. The last hop from the pair of intermediary switches to the first pair of switches mentioned on this page (same that nodes tron[00~~-44,46~~-61] are on) is via four 100GbE links, one for each combination of switches across each pairing, for redundancy and increased bandwidth.		The fileserver hosting all Nexus [[Nexus#Scratch_Directories \| scratch]], [[Nexus#Faculty_Allocations \| faculty]], [[Nexus#Project_Allocations \| project]], and [[Nexus#Datasets \| dataset]] allocations first connects to a pair of intermediary switches before reaching the compute nodes. The last hop from the pair of intermediary switches to the first pair of switches mentioned on this page (same that nodes tron[00-61] are on) is via four 100GbE links, one for each combination of switches across each pairing, for redundancy and increased bandwidth.

	For a broader overview of the network infrastructure supporting the Nexus cluster, please see [[Nexus/Network]].		For a broader overview of the network infrastructure supporting the Nexus cluster, please see [[Nexus/Network]].

Nexus/CML

Mbaney — Wed, 08 Jul 2026 17:16:55 GMT

Partitions

← Older revision		Revision as of 17:16, 8 July 2026
(3 intermediate revisions by the same user not shown)
Line 11:		Line 11:
	* <code>nexuscml00.umiacs.umd.edu</code>		* <code>nexuscml00.umiacs.umd.edu</code>
	* <code>nexuscml01.umiacs.umd.edu</code>		* <code>nexuscml01.umiacs.umd.edu</code>

	All partitions, QoSes, and account names from the standalone CML cluster have been moved over to Nexus. However, please note that <code>cml-</code> is prepended to all of the values that were present in the standalone CML cluster to distinguish them from existing values in Nexus. The lone exception is the base account that was named <code>cml</code> in the standalone cluster (it is also named just <code>cml</code> in Nexus).

	~~Here are some before/after examples of job submission with various parameters:~~

	~~{\| class="wikitable"~~
	~~! Standalone CML cluster submission command~~
	~~! Nexus cluster submission command~~
	\|-
	~~\|<code>srun --partition=dpart --qos=medium --account=tomg --gres=gpu:rtx2080ti:2 --pty bash</code>~~
	~~\|<code>srun --partition=cml-dpart --qos=cml-medium --account=cml-tomg --gres=gpu:rtx2080ti:2 --pty bash</code>~~
	\|-
	~~\|<code>srun --partition=cpu --qos=cpu --pty bash</code>~~
	~~\|<code>srun --partition=cml-cpu --qos=cml-cpu --account=cml --pty bash</code>~~
	\|-
	~~\|<code>srun --partition=scavenger --qos=scavenger --account=scavenger --gres=gpu:4 --pty bash</code>~~
	~~\|<code>srun --partition=cml-scavenger --qos=cml-scavenger --account=cml-scavenger --gres=gpu:4 --pty bash</code>~~
	\|}

	CML users (exclusively) can schedule non-interruptible jobs on CML nodes with any non-scavenger job parameters. Please note that the <code>cml-dpart</code> partition has a <code>GrpTRES</code> limit of 100% of the available cores/RAM on all cml## nodes in aggregate plus 50% of the available cores/RAM on legacy## nodes in aggregate, so your job may need to wait if all available cores/RAM (or GPUs) are in use. It also has a max submission limit of 500 jobs per user simultaneously so as to not overload the cluster. This is codified by the partition QoS named '''cml'''.		CML users (exclusively) can schedule non-interruptible jobs on CML nodes with any non-scavenger job parameters. Please note that the <code>cml-dpart</code> partition has a <code>GrpTRES</code> limit of 100% of the available cores/RAM on all cml## nodes in aggregate plus 50% of the available cores/RAM on legacy## nodes in aggregate, so your job may need to wait if all available cores/RAM (or GPUs) are in use. It also has a max submission limit of 500 jobs per user simultaneously so as to not overload the cluster. This is codified by the partition QoS named '''cml'''.

	Please note that the CML compute nodes are also in the institute-wide <code>scavenger</code> partition in Nexus. CML users still have scavenging priority over these nodes via the <code>cml-scavenger</code> partition (i.e., all <code>cml-</code> partition jobs (other than <code>cml-scavenger</code>) can preempt both <code>cml-scavenger</code> and <code>scavenger</code> partition jobs, and <code>cml-scavenger</code> partition jobs can preempt <code>scavenger</code> partition jobs).		Please note that the CML compute nodes are also in the institute-wide <code>scavenger</code> partition in Nexus. CML users still have scavenging priority over these nodes via the <code>cml-scavenger</code> partition, i.e., all <code>cml-*</code> partition jobs (other than <code>cml-scavenger</code>) can preempt both <code>cml-scavenger</code> and <code>scavenger</code> partition jobs, and <code>cml-scavenger</code> partition jobs can preempt <code>scavenger</code> partition jobs.

	==Network==		==Network==
	The network infrastructure supporting the CML partition consists of:		The network infrastructure supporting the CML partition consists of:
	# One pair of network switches connected to each other via dual 100GbE links for redundancy, serving the following compute nodes:		# One pair of network switches connected to each other via dual 100GbE links for redundancy, serving the following compute nodes:
	#* cml[17-28,30-32,34]: Two 100GbE links per node, one to each switch in the pair (redundancy).		#* cml[17-28,30-32,34,37]: Two 100GbE links per node, one to each switch in the pair (redundancy).
	#* ~~cml35~~: Four 100GbE links, two to each switch in the pair (redundancy and increased bandwidth).		#* cml[35-36]: Four 100GbE links, two to each switch in the pair (redundancy and increased bandwidth).
	# One pair of network switches connected to the above pair of network switches via two 100GbE links, one between the first two switches in each pair and one between the second two switches in each pair for redundancy, and to each other via dual 25GbE links for redundancy.		# One pair of network switches connected to the above pair of network switches via two 100GbE links, one between the first two switches in each pair and one between the second two switches in each pair for redundancy, and to each other via dual 25GbE links for redundancy.
	#* cml[00,02-09]: Two 25GbE links per node, one to each switch in the pair (redundancy).		#* cml[00,02-09]: Two 25GbE links per node, one to each switch in the pair (redundancy).
	#* cml[10-16],cmlcpu[00-04,06-07]: Two 10GbE links per node, one to each switch in the pair (redundancy).		#* cml[10-16],cmlcpu[01-04,06-07]: Two 10GbE links per node, one to each switch in the pair (redundancy).

	The fileserver hosting all CML [[Nexus/CML#Project_Directories \| project]], [[Nexus/CML#Scratch_Directories \| scratch]], [[Nexus/CML#Datasets \| dataset]], and [[Nexus/CML#Models \| model]] allocations also connects to the same pair of switches supporting cml[17-28,30-32] via fourteen 25GbE links, seven to each switch in the pair for redundancy and increased bandwidth.		The fileserver hosting all CML [[Nexus/CML#Project_Directories \| project]], [[Nexus/CML#Scratch_Directories \| scratch]], [[Nexus/CML#Datasets \| dataset]], and [[Nexus/CML#Models \| model]] allocations also connects to the same pair of switches supporting cml[17-28,30-32] via fourteen 25GbE links, seven to each switch in the pair for redundancy and increased bandwidth.
Line 52:		Line 34:
	* '''cml-dpart''' - This is the default partition. Job allocations are guaranteed.		* '''cml-dpart''' - This is the default partition. Job allocations are guaranteed.
	* '''cml-scavenger''' - This is the alternate partition that allows jobs longer run times and more resources but is preemptable when jobs in other <code>cml-</code> partitions are ready to be scheduled.		* '''cml-scavenger''' - This is the alternate partition that allows jobs longer run times and more resources but is preemptable when jobs in other <code>cml-</code> partitions are ready to be scheduled.
	* '''cml-cpu''' - This partition is for CPU focused jobs. Job allocations are guaranteed.		* '''cml-cpu''' - This partition is for CPU focused jobs. Job allocations are guaranteed. '''Please note that this partition is being permanently removed on 07/16/2026 at 9am.'''

	There are a few additional partitions available solely to specific faculty members and their sponsored user accounts.		There are a few additional partitions available solely to specific faculty members and their sponsored user accounts.

	* '''cml-furongh''' - This partition is for exclusive priority access to Dr. Furong Huang's purchased nodes. Job allocations are guaranteed.		* '''cml-furongh''' - This partition is for exclusive priority access to Dr. Furong Huang's purchased nodes. Job allocations are guaranteed.
			* '''cml-ramani''' - This partition is for exclusive priority access to Dr. Ramani Duraiswami's purchased nodes. Job allocations are guaranteed.
	* '''cml-sfeizi''' - This partition is for exclusive priority access to Dr. Soheil Feizi's purchased nodes. Job allocations are guaranteed.		* '''cml-sfeizi''' - This partition is for exclusive priority access to Dr. Soheil Feizi's purchased nodes. Job allocations are guaranteed.

← Older revision		Revision as of 17:48, 17 July 2026
Line 18:		Line 18:
	The network infrastructure supporting the CLIP partition consists of:		The network infrastructure supporting the CLIP partition consists of:
	# One pair of network switches connected to each other via dual 25GbE links for redundancy, serving the following compute nodes:		# One pair of network switches connected to each other via dual 25GbE links for redundancy, serving the following compute nodes:
	#* ~~clip[00-03,05,07-08,10]~~: Two ~~10GbE~~ links ~~per node~~, one to each switch in the pair (redundancy).		#* clip04: Two 40GbE links, one to each switch in the pair (redundancy).
	#* ~~clip04~~: Two ~~40GbE~~ links per node, one to each switch in the pair (redundancy).		#* clip[05,14]: Two 10GbE links per node, one to each switch in the pair (redundancy).
	#* clip06: Two 25GbE links ~~per node, one to each switch in the pair (redundancy).~~		#* clip06: Two 25GbE links, one to each switch in the pair (redundancy).
	~~#* clip09: Two 1GbE links per node~~, one to each switch in the pair (redundancy).
	#* clip[11-13]: Two 100GbE links per node, one to each switch in the pair (redundancy).		#* clip[11-13]: Two 100GbE links per node, one to each switch in the pair (redundancy).

Line 55:		Line 54:
	Partition=clip AllocNode:Sid=nexusclip00:25443		Partition=clip AllocNode:Sid=nexusclip00:25443
	ReqNodeList=(null) ExcNodeList=(null)		ReqNodeList=(null) ExcNodeList=(null)
	NodeList=~~clip00~~		NodeList=clip04
	BatchHost=~~clip00~~		BatchHost=clip04
	NumNodes=1 NumCPUs=4 NumTasks=4 CPUs/Task=1 ReqB:S:C:T=0:0::		NumNodes=1 NumCPUs=4 NumTasks=4 CPUs/Task=1 ReqB:S:C:T=0:0::
	TRES=cpu=4,mem=8G,node=1,billing=2266		TRES=cpu=4,mem=8G,node=1,billing=2266