Nexus/CML

The compute nodes from CML's previous standalone cluster have folded into Nexus as of the scheduled maintenance window for August 2023 (Thursday 08/17/2023, 5-8pm).

The Nexus cluster already has a large pool of compute resources made possible through college-level funding for UMIACS and CSD faculty. Details on common nodes already in the cluster (Tron partition) can be found here.

Please contact staff with any questions or concerns.

Usage

You can SSH to nexuscml.umiacs.umd.edu to log in to a submission node.

If you store something in a local directory (/tmp, /scratch0) on one of the two submission nodes, you will need to connect to that same submission node to access it later. The actual submission nodes are:

• nexuscml00.umiacs.umd.edu
• nexuscml01.umiacs.umd.edu

All partitions, QoSes, and account names from the standalone CML cluster have been moved over to Nexus. However, please note that cml- is prepended to all of the values that were present in the standalone CML cluster to distinguish them from existing values in Nexus. The lone exception is the base account that was named cml in the standalone cluster (it is also named just cml in Nexus).

Here are some before/after examples of job submission with various parameters:

Standalone CML cluster submission command	Nexus cluster submission command
srun --partition=dpart --qos=medium --account=tomg --gres=gpu:rtx2080ti:2 --pty bash	srun --partition=cml-dpart --qos=cml-medium --account=cml-tomg --gres=gpu:rtx2080ti:2 --pty bash
srun --partition=cpu --qos=cpu --pty bash	srun --partition=cml-cpu --qos=cml-cpu --account=cml --pty bash
srun --partition=scavenger --qos=scavenger --account=scavenger --gres=gpu:4 --pty bash	srun --partition=cml-scavenger --qos=cml-scavenger --account=cml-scavenger --gres=gpu:4 --pty bash

CML users (exclusively) can schedule non-interruptible jobs on CML nodes with any non-scavenger job parameters. Please note that the cml-dpart partition has a GrpTRES limit of 100% of the available cores/RAM on all cml## nodes in aggregate plus 50% of the available cores/RAM on legacy## nodes in aggregate, so your job may need to wait if all available cores/RAM (or GPUs) are in use. It also has a max submission limit of 500 jobs per user simultaneously so as to not overload the cluster. This is codified by the partition QoS named cml.

Please note that the CML compute nodes are also in the institute-wide scavenger partition in Nexus. CML users still have scavenging priority over these nodes via the cml-scavenger partition (i.e., all cml- partition jobs (other than cml-scavenger) can preempt both cml-scavenger and scavenger partition jobs, and cml-scavenger partition jobs can preempt scavenger partition jobs).

Network

The network infrastructure supporting the CML partition consists of:

1. One pair of network switches connected to each other via dual 100GbE links for redundancy, serving the following compute nodes:
   • cml[17-28,30-32]: Two 100GbE links per node, one to each switch in the pair (redundancy).
2. One pair of network switches connected to the above pair of network switches via four 100GbE links, one between every combination of switches across the two pairings for redundancy, and to each other via dual 25GbE links for redundancy.
   • cml[00-09]: (subject to change) Two 25GbE links per node, one to each switch in the pair (redundancy).
   • cml[10-16],cmlcpu[00-04,06-07]: (subject to change) Two 10GbE links per node, one to each switch in the pair (redundancy).

The fileserver hosting all CML project, scratch, dataset, and model allocations also connects to the same pair of switches supporting cml[17-28,30-32] via fourteen 25GbE links, seven to each switch in the pair for redundancy and increased bandwidth.

For a broader overview of the network infrastructure supporting the Nexus cluster, please see Nexus/Network.

Partitions

There are three partitions available to general CML SLURM users. You must specify a partition when submitting your job.

• cml-dpart - This is the default partition. Job allocations are guaranteed.
• cml-scavenger - This is the alternate partition that allows jobs longer run times and more resources but is preemptable when jobs in other cml- partitions are ready to be scheduled.
• cml-cpu - This partition is for CPU focused jobs. Job allocations are guaranteed.

There are two additional partitions available solely to specific faculty members and their sponsored accounts.

• cml-furongh - This partition is for exclusive priority access to Dr. Furong Huang's purchased A6000 node. Job allocations are guaranteed.
• cml-zhou - This partition is for exclusive priority access to Dr. Tianyi Zhou's purchased nodes. Job allocations are guaranteed.

Accounts

The Center has a base SLURM account cml which has a modest number of guaranteed billing resources available to all cluster users at any given time. Other faculty that have invested in the cluster have an additional account provided to their sponsored accounts on the cluster, which provides a number of guaranteed billing resources corresponding to the amount that they invested.

If you do not specify an account when submitting your job, you will receive the cml account, which only has access to the cml-cpu, cml-default, and cml-medium QoSes (see below section).

If you need access to a different QoS, or if the cml account is at its billing limit (see below in this section), please use your faculty sponsor's account if they have one available. However, keep in mind that if you use your faculty sponsor has their own named partition (see previous section), using the faculty-specific account in the cml-dpart partition may block access to resources in the faculty-specific partition, since the billing limit for the account is charged regardless of what partition is being used.

The current faculty accounts are:

• cml-abhinav
• cml-cameron
• cml-furongh
• cml-hajiagha
• cml-john
• cml-ramani
• cml-sfeizi
• cml-tokekar
• cml-tomg
• cml-zhou

$ sacctmgr show account format=account%20,description%30,organization%10
             Account                          Descr        Org
-------------------- ------------------------------ ----------
                 ...                            ...        ...
                 cml                            cml        cml
         cml-abhinav      cml - abhinav shrivastava        cml
         cml-cameron            cml - maria cameron        cml
         cml-furongh             cml - furong huang        cml
        cml-hajiagha      cml - mohammad hajiaghayi        cml
            cml-john           cml - john dickerson        cml
          cml-ramani        cml - ramani duraiswami        cml
       cml-scavenger                cml - scavenger        cml
          cml-sfeizi             cml - soheil feizi        cml
         cml-tokekar           cml - pratap tokekar        cml
            cml-tomg            cml - tom goldstein        cml
            cml-zhou              cml - tianyi zhou        cml
                 ...                            ...        ...

Faculty can manage this list of users via our Directory application in the Security Groups section. The security group that controls access has the prefix cml_ and then the faculty username. It will also list slurm://nexusctl.umiacs.umd.edu as the associated URI.

You can check your account associations by running the show_assoc command to see the accounts you are associated with. Please contact staff and include your faculty member in the conversation if you do not see the appropriate association.

$ show_assoc
      User          Account MaxJobs       GrpTRES                                                QOS
---------- ---------------- ------- ------------- --------------------------------------------------
       ...              ...                                                                      ...
      tomg              cml                                           cml-cpu,cml-default,cml-medium
      tomg    cml-scavenger                                                            cml-scavenger
      tomg         cml-tomg                                          cml-default,cml-high,cml-medium
       ...              ...                                                                      ...

You can also see the total number of Track-able Resources (TRES) allowed for each account by running the following command. Please make sure you give the appropriate account that you are looking for. The billing number displayed here is the sum of resource weightings for all nodes appropriated to that account.

$ sacctmgr show assoc account=cml format=user,account,qos,grptres
      User    Account                  QOS       GrpTRES
---------- ---------- -------------------- -------------
                  cml                       billing=6481
                  ...                                ...

QoS

CML currently has 5 QoS for the cml-dpart partition (though cml-high_long and cml-very_high may not be available to all faculty accounts), 1 QoS for the cml-scavenger partition, and 1 QoS for the cml-cpu partition. If you do not specify a QoS when submitting your job using the --qos parameter, you will receive the cml-default QoS assuming you are using a CML account.

If your faculty member's Slurm account does not have one or both of the cml-high_long or cml-very_high QoS available to it, we can add it to their account provided they approve. Please contact staff if this is desired.

The important part here is that in different QoS you can have a shorter/longer maximum wall time, a different total number of jobs running at once, and a different maximum number of track-able resources (TRES) for the job. In the cml-scavenger QoS, one more constraint that you are restricted by is the total number of TRES per user (over multiple jobs).

$ show_qos --all | grep cml
                Name     MaxWall                        MaxTRES MaxJobsPU                      MaxTRESPU                                                                             
-------------------- ----------- ------------------------------ --------- ------------------------------      
...                                                                       
             cml-cpu  7-00:00:00                                        8
         cml-default  7-00:00:00       cpu=4,gres/gpu=1,mem=32G         2
            cml-high  1-12:00:00     cpu=16,gres/gpu=4,mem=128G         2
       cml-high_long 14-00:00:00              cpu=32,gres/gpu=8         8                     gres/gpu=8
          cml-medium  3-00:00:00       cpu=8,gres/gpu=2,mem=64G         2
       cml-scavenger  3-00:00:00                                                             gres/gpu=24
       cml-very_high  1-12:00:00     cpu=32,gres/gpu=8,mem=256G         8                    gres/gpu=12
...

$ show_partition_qos --all | grep cml
                Name MaxSubmitPU                      MaxTRESPU              GrpTRES 
-------------------- ----------- ------------------------------ -------------------- 
...
                 cml         500                                    cpu=1128,mem=11T
             cml-cpu         500
         cml-furongh         500
       cml-scavenger         500                    gres/gpu=24
           cml-wriva         500
            cml-zhou         500
...

Storage

There are 3 types of user storage available to users in the CML:

• Home directories
• Project directories
• Scratch directories

There are also 2 types of read-only storage available for common use among users in the CML:

• Dataset directories
• Model directories

CML users can also request Nexus project allocations.

Home Directories

You have 30GB of home directory storage available at /nfshomes/<username>. It has both Snapshots and Backups enabled.

Home directories are intended to store personal or configuration files only. We encourage you to not share any data in your home directory. You are encouraged to utilize our GitLab infrastructure to host your code repositories.

NOTE: To check your quota on this directory, use the command df -h ~.

Project Directories

You can request project based allocations for up to 6TB for up to 120 days with one or more approvals:

• Allocations up to and including 3TB require approval from a CML faculty member
• Allocations above 3TB (up to 6TB) require approval from both a CML faculty member and the director of CML

To request an allocation, please contact staff with the faculty member(s) that the project is under involved in the conversation. Please include the following details:

• Project Name (short)
• Description
• Size (1TB, 2TB, etc.)
• Length in days (30 days, 90 days, etc.)
• Other user(s) that need to access the allocation, if any

These allocations will be available from /fs/cml-projects under a name that you provide when you request the allocation. Near the end of the allocation period, staff will contact you and ask if you would like to renew the allocation for up to another 120 days (requires re-approval from a CML faculty member and the director of CML).

• If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation.
• If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible.
  • If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible.
  • If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

This data is backed up nightly.

Scratch Directories

Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the CML compute infrastructure:

• Network scratch directory
• Local scratch directories

Network Scratch Directory

You have 200GB of scratch storage available at /cmlscratch/<username>. It is not backed up or protected in any way. This directory is automounted so you will need to cd into the directory or request/specify a fully qualified file path to access this.

You may request a permanent increase of up to 800GB total space without any faculty approval by contacting staff. If you need space beyond 800GB, you will need faculty approval and/or a project directory. Space increases beyond 800GB also have a maximum request period of 120 days (as with project directories), after which they will need to be renewed with re-approval from a CML faculty member and the director of CML.

• As with project directories, allocations over 3TB total space require approval from the director of CML in addition to your faculty member.

This file system is available on all submission and computational nodes within the cluster.

Local Scratch Directories

Each computational node that you can schedule compute jobs on has one or more local scratch directories. These are always named /scratch0, /scratch1, etc. These are almost always more performant than any other storage available to the job. However, you must stage data to these directories within the confines of your jobs and stage the data out before the end of your jobs.

These local scratch directories have a tmpwatch job which will delete unaccessed data after 90 days, scheduled via maintenance jobs to run once a month during our monthly maintenance windows. Again, please make sure you secure any data you write to these directories at the end of your job.

Datasets

We have read-only dataset storage available at /fs/cml-datasets. If there are datasets that you would like to see curated and made available, please see this page.

The list of CML datasets we currently host can be viewed here.

Models

We have read-only model storage available at /fs/cml-models. If there are models that you would like to see downloaded and made available, please see this page.