Nexus/CML

From UMIACS
Revision as of 22:37, 17 August 2023 by Mbaney (talk | contribs)
Jump to navigation Jump to search

The CML standalone cluster's compute nodes have folded into Nexus as of the scheduled maintenance window for August (Thursday 8/17, 5-8pm).

The Nexus cluster already has a large pool of compute resources made possible through leftover funding for the Brendan Iribe Center. Details on common nodes already in the cluster (Tron partition) can be found here.

The CML cluster's standalone submission node cmlsub00.umiacs.umd.edu will be retired on Thursday, September 21st, 2023 during that month's maintenance window (5-8pm), as it is no longer be able to submit jobs to CML compute nodes. Please use nexuscml00.umiacs.umd.edu and nexuscml01.umiacs.umd.edu for any general purpose CML compute needs after this time.

Please see the Timeline section below for concrete dates in chronological order.

Please contact staff with any questions or concerns.

Usage

The Nexus cluster submission nodes that are allocated to CML are nexuscml00.umiacs.umd.edu and nexuscml01.umiacs.umd.edu. You must use these nodes to submit jobs to CML compute nodes. Submission from cmlsub00.umiacs.umd.edu is no longer available.

All partitions, QoSes, and account names from the standalone CML cluster have been moved over to Nexus. However, please note that cml- is prepended to all of the values that were present in the standalone CML cluster to distinguish them from existing values in Nexus. The lone exception is the base account that was named cml in the standalone cluster (it is also named just cml in Nexus).

Here are some before/after examples of job submission with various parameters:

Standalone CML cluster submission command Nexus cluster submission command
srun --partition=dpart --qos=medium --account=tomg --gres=gpu:rtx2080ti:2 --pty bash srun --partition=cml-dpart --qos=cml-medium --account=cml-tomg --gres=gpu:rtx2080ti:2 --pty bash
srun --partition=cpu --qos=cpu --pty bash srun --partition=cml-cpu --qos=cml-cpu --pty bash
srun --partition=scavenger --qos=scavenger --account=scavenger --gres=gpu:4 --pty bash srun --partition=cml-scavenger --qos=cml-scavenger --account=cml-scavenger --gres=gpu:4 --pty bash

CML users (exclusively) can schedule non-interruptible jobs on the moved nodes with any non-scavenger job parameters. Please note that the cml-dpart partition has a GrpTRES limit of 100% of the available cores/RAM on each set of cml## nodes plus 50% of the available cores/RAM on legacy## nodes, so your job may need to wait if all available cores/RAM (or GPUs) are in use. It also has a max submission limit of 500 jobs per user simultaneously so as to not overload the cluster. This is codified by the partition QoS named cml.

Please note that the CML compute nodes are also in the institute-wide scavenger partition in Nexus. CML users still have scavenging priority over these nodes via the cml-scavenger partition (i.e., all cml- partition jobs (other than cml-scavenger) can preempt both cml-scavenger and scavenger partition jobs, and cml-scavenger partition jobs can preempt scavenger partition jobs).

Partitions

There are three partitions available to general CML SLURM users. You must specify a partition when submitting your job.

  • cml-dpart - This is the default partition. Job allocations are guaranteed.
  • cml-scavenger - This is the alternate partition that allows jobs longer run times and more resources but is preemptable when jobs in other cml- partitions are ready to be scheduled.
  • cml-cpu - This partition is for CPU focused jobs. Job allocations are guaranteed.

There is one additional partition available solely to Furong's sponsored accounts.

  • cml-furongh - This partition is for exclusive priority access to Furong's purchased A6000 node.

Accounts

The Center has a base SLURM account cml which has a modest number of guaranteed billing resources available to all cluster users at any given time. Other faculty that have invested in the cluster have an additional account provided to their sponsored accounts on the cluster, which provides a number of guaranteed billing resources corresponding to the amount that they invested.

If you do not specify an account when submitting your job, you will receive the cml account. If your faculty sponsor has their own account, it is recommended to use that account for job submission.

The current faculty accounts are:

  • cml-abhinav
  • cml-cameron
  • cml-furongh
  • cml-hajiagha
  • cml-john
  • cml-ramani
  • cml-sfeizi
  • cml-tokekar
  • cml-tomg
$ sacctmgr show account format=account%20,description%30,organization%10
             Account                          Descr        Org
-------------------- ------------------------------ ----------
                 ...                            ...        ...
                 cml                            cml        cml
         cml-abhinav      cml - abhinav shrivastava        cml
         cml-cameron            cml - maria cameron        cml
         cml-furongh             cml - furong huang        cml
        cml-hajiagha      cml - mohammad hajiaghayi        cml
            cml-john           cml - john dickerson        cml
          cml-ramani        cml - ramani duraiswami        cml
       cml-scavenger                cml - scavenger        cml
          cml-sfeizi             cml - soheil feizi        cml
         cml-tokekar           cml - pratap tokekar        cml
            cml-tomg            cml - tom goldstein        cml
                 ...                            ...        ...

Faculty can manage this list of users via our Directory application in the Security Groups section. The security group that controls access has the prefix cml_ and then the faculty username. It will also list slurm://nexusctl.umiacs.umd.edu as the associated URI.

You can check your account associations by running the show_assoc command to see the accounts you are associated with. Please contact staff and include your faculty member in the conversation if you do not see the appropriate association. 

$ show_assoc
      User          Account MaxJobs       GrpTRES                                                QOS
---------- ---------------- ------- ------------- --------------------------------------------------
       ...              ...                                                                      ...
      tomg              cml                                           cml-cpu,cml-default,cml-medium
      tomg    cml-scavenger                                                            cml-scavenger
      tomg         cml-tomg                                          cml-default,cml-high,cml-medium
       ...              ...                                                                      ...

You can also see the total number of Track-able Resources (TRES) allowed for each account by running the following command. Please make sure you give the appropriate account that you are looking for. The billing number displayed here is the sum of resource weightings for all nodes appropriated to that account. 

$ sacctmgr show assoc account=cml format=user,account,qos,grptres
      User    Account                  QOS       GrpTRES
---------- ---------- -------------------- -------------
                  cml                       billing=7732
                  ...                                ...

QoS

CML currently has 5 QoS for the cml-dpart partition (though high_long and very_high may not be available to all faculty accounts), 1 QoS for the cml-scavenger partition, and 1 QoS for the cml-cpu partition. If you do not specify a QoS when submitting your job using the --qos parameter, you will receive the cml-default QoS assuming you are using a CML account.

The important part here is that in different QoS you can have a shorter/longer maximum wall time, a different total number of jobs running at once, and a different maximum number of track-able resources (TRES) for the job. In the cml-scavenger QoS, one more constraint that you are restricted by is the total number of TRES per user (over multiple jobs). 

$ show_qos
                Name     MaxWall                        MaxTRES MaxJobsPU MaxSubmitPU                      MaxTRESPU              GrpTRES
-------------------- ----------- ------------------------------ --------- ----------- ------------------------------ --------------------
                 ...         ...                            ...       ...         ...                            ...                  ...
         cml-default  7-00:00:00       cpu=4,gres/gpu=1,mem=32G         2
          cml-medium  3-00:00:00       cpu=8,gres/gpu=2,mem=64G         2
            cml-high  1-12:00:00     cpu=16,gres/gpu=4,mem=128G         2
       cml-scavenger  3-00:00:00                                                                         gres/gpu=24
             cml-cpu  7-00:00:00                                        8
       cml-very_high  1-12:00:00     cpu=32,gres/gpu=8,mem=256G         8                                gres/gpu=12
       cml-high_long 14-00:00:00              cpu=32,gres/gpu=8         8                                 gres/gpu=8
                 cml                                                              500                                   cpu=252,mem=2884G
                 ...         ...                            ...       ...         ...                            ...                  ...

Storage

There are 3 types of user storage available to users in the CML:

  • Home directories
  • Project directories
  • Scratch directories

There are also 2 types of read-only storage available for common use among users in the CML:

  • Dataset directories
  • Model directories

CML users can also request Nexus project allocations.

Home Directories

Home directories in the CML computational infrastructure are available from the Institute's NFShomes as /nfshomes/USERNAME where USERNAME is your username. These home directories have very limited storage (20GB, cannot be increased) and are intended for your personal files, configuration and source code. Your home directory is not intended for data sets or other large scale data holdings. Users are encouraged to utilize our GitLab infrastructure to host your code repositories.

NOTE: To check your quota on this directory you will need to use the quota -s command.

Your home directory data is fully protected and has both snapshots and is backed up nightly.

Project Directories

You can request project based allocations for up to 6TB for up to 120 days with approval from a CML faculty member and the director of CML.

To request an allocation, please contact staff with the faculty member(s) that the project is under involved in the conversation. Please include the following details:

  • Project Name (short)
  • Description
  • Size (1TB, 2TB, etc.)
  • Length in days (30 days, 90 days, etc.)
  • Other user(s) that need to access the allocation, if any

These allocations will be available from /fs/cml-projects under a name that you provide when you request the allocation. Near the end of the allocation period, staff will contact you and ask if you would like to renew the allocation for up to another 120 days (requires re-approval from a CML faculty member and the director of CML). If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation. If you or the original faculty approver do not respond to staff's request or within one month of the end of the allocation period, staff will remove the allocation.

This data is backed up nightly.

Scratch Directories

Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the CML compute infrastructure:

  • Network scratch directory
  • Local scratch directories

Network Scratch Directory

You are allocated 400GB of scratch space via NFS from /cmlscratch/$username. It is not backed up or protected in any way. This directory is automounted so you will need to cd into the directory or request/specify a fully qualified file path to access this.

You may request a permanent increase of up to 800GB total space without any faculty approval by contacting staff. If you need space beyond 800GB, you will need faculty approval and/or a project directory. Space increases beyond 800GB also have a maximum request period of 120 days (as with project directories), after which they will need to be renewed with re-approval from a CML faculty member and the director of CML.

This file system is available on all submission, data management, and computational nodes within the cluster.

Local Scratch Directories

Each computational node that you can schedule compute jobs on has one or more local scratch directories. These are always named /scratch0, /scratch1, etc. These are almost always more performant than any other storage available to the job. However, you must stage data to these directories within the confines of your jobs and stage the data out before the end of your jobs.

These local scratch directories have a tmpwatch job which will delete unaccessed data after 90 days, scheduled via maintenance jobs to run once a month during our monthly maintenance windows. Again, please make sure you secure any data you write to these directories at the end of your job.

Datasets

We have read-only dataset storage available at /fs/cml-datasets. If there are datasets that you would like to see curated and available, please see this page.

The following is the list of datasets available:

Dataset Path
CelebA /fs/cml-datasets/CelebA
CelebA-HQ /fs/cml-datasets/CelebA-HQ
CelebAMask-HQ /fs/cml-datasets/CelebAMask-HQ
Charades /fs/cml-datasets/Charades
Cityscapes /fs/cml-datasets/cityscapes
COCO /fs/cml-datasets/coco
Diversity in Faces [1] /fs/cml-datasets/diversity_in_faces
FFHQ /fs/cml-datasets/FFHQ
ImageNet ILSVRC2012 /fs/cml-datasets/ImageNet/ILSVRC2012
LFW /fs/cml-datasets/facial_test_data
LibriSpeech /fs/cml-datasets/LibriSpeech
LSUN /fs/cml-datasets/LSUN
MAG240M /fs/cml-datasets/OGB/MAG240M
MegaFace /fs/cml-datasets/megaface
MS-Celeb-1M /fs/cml-datasets/MS_Celeb_aligned_112
OC20 /fs/cml-datasets/OC20
ogbn-papers100M /fs/cml-datasets/OGB/ogbn-papers100M
roberta /fs/cml-datasets/roberta
Salient ImageNet /fs/cml-datasets/Salient-ImageNet
ShapeNetCore.v2 /fs/cml-datasets/ShapeNetCore.v2
Tiny ImageNet /fs/cml-datasets/tiny_imagenet
WikiKG90M /fs/cml-datasets/OGB/WikiKG90M

[1] - This dataset has restricted access. Please contact staff if you are looking to use this dataset.

Models

We have read-only model storage available at /fs/cml-models. If there are models that you would like to see downloaded and made available, please see this page.

Timeline

Each event will be completed within the timeframe specified.

Date Event
July 14th 2023 A single compute node cml03 is moved into Nexus so submission can be tested
August 17th 2023, 5-8pm All other standalone CML cluster compute nodes are moved into Nexus in corresponding cml- named partitions
September 21st 2023, 5-8pm cmlsub00.umiacs.umd.edu is taken offline
Retrieved from "https://wiki.umiacs.umd.edu/umiacs/index.php?title=Nexus/CML&oldid=11229"