Nexus/Vulcan

From UMIACS
Revision as of 14:50, 2 December 2024 by Mbaney (talk | contribs) (→‎Datasets)
Jump to navigation Jump to search

The compute nodes from Vulcan's previous standalone cluster have folded into Nexus as of the scheduled maintenance window for August 2023 (Thursday 08/17/2023, 5-8pm).

The Nexus cluster already has a large pool of compute resources made possible through college-level funding for UMIACS and CSD faculty. Details on common nodes already in the cluster (Tron partition) can be found here.

Please contact staff with any questions or concerns.

Usage

You can SSH to nexusvulcan.umiacs.umd.edu to log in to a submission node.

If you store something in a local directory (/tmp, /scratch0) on one of the two submission nodes, you will need to connect to that same submission node to access it later. The actual submission nodes are:

  • nexusvulcan00.umiacs.umd.edu
  • nexusvulcan01.umiacs.umd.edu

All partitions, QoSes, and account names from the standalone Vulcan cluster have been moved over to Nexus. However, please note that vulcan- is prepended to all of the values that were present in the standalone Vulcan cluster to distinguish them from existing values in Nexus. The lone exception is the base account that was named vulcan in the standalone cluster (it is also named just vulcan in Nexus).

Here are some before/after examples of job submission with various parameters:

Standalone Vulcan cluster submission command Nexus cluster submission command
srun --partition=dpart --qos=medium --account=abhinav --gres=gpu:gtx1080ti:2 --pty bash srun --partition=vulcan-dpart --qos=vulcan-medium --account=vulcan-abhinav --gres=gpu:gtx1080ti:2 --pty bash
srun --partition=cpu --qos=cpu --pty bash srun --partition=vulcan-cpu --qos=vulcan-cpu --account=vulcan --pty bash
srun --partition=scavenger --qos=scavenger --account=vulcan --gres=gpu:4 --pty bash srun --partition=vulcan-scavenger --qos=vulcan-scavenger --account=vulcan --gres=gpu:4 --pty bash

Vulcan users (exclusively) can schedule non-interruptible jobs on Vulcan nodes with any non-scavenger job parameters. Please note that the vulcan-dpart partition has a GrpTRES limit of 100% of the available cores/RAM on all vulcan## in aggregate nodes plus 50% of the available cores/RAM on legacy## nodes in aggregate, so your job may need to wait if all available cores/RAM (or GPUs) are in use. It also has a max submission limit of 500 jobs per user simultaneously so as to not overload the cluster. This is codified by the partition QoS named vulcan.

Please note that the Vulcan compute nodes are also in the institute-wide scavenger partition in Nexus. Vulcan users still have scavenging priority over these nodes via the vulcan-scavenger partition (i.e., all vulcan- partition jobs (other than vulcan-scavenger) can preempt both vulcan-scavenger and scavenger partition jobs, and vulcan-scavenger partition jobs can preempt scavenger partition jobs).

Nodes

There are currently 46 GPU nodes available, named vulcan[00-45], running a mixture of NVIDIA RTX A6000, NVIDIA RTX A5000, NVIDIA RTX A4000, NVIDIA Quadro P6000, NVIDIA GeForce GTX 1080 Ti, NVIDIA GeForce RTX 2080 Ti, and NVIDIA Tesla P100 cards. There are also 4 CPU-only nodes available, named brigid[16-19].

All nodes are scheduled with the SLURM resource manager.

Partitions

There are three partitions available to general Vulcan SLURM users. You must specify a partition when submitting your job.

  • vulcan-dpart - This is the default partition. Job allocations are guaranteed. Only nodes with GPUs from architectures before NVIDIA's Ampere architecture are included in this partition.
  • vulcan-scavenger - This is the alternate partition that allows jobs longer run times and more resources but is preemptable when jobs in other vulcan- partitions are ready to be scheduled.
  • vulcan-cpu - This partition is for CPU focused jobs. Job allocations are guaranteed.

There are a few additional partitions available to subsets of Vulcan users based on specific requirements.

  • vulcan-ampere - This partition contains nodes with GPUs from NVIDIA's Ampere architecture. Job allocations are guaranteed.
    As of Thursday 02/29/2024 at 12pm, there is a 4 hour time limit on interactive jobs in this partition. If you need to run longer jobs, you will need to modify your workflow into a job that can be submitted as a batch script.
    As of Thursday 03/21/2024 at 5pm, there is a limit of 4 CPUs and 48G memory maximum per GPU requested by a job. If you need to run jobs with more CPUs/memory, you will either need to request more GPUs in the job or use a different partition.
Submission is restricted to the Slurm accounts of the faculty who invested in these nodes:
  • Abhinav Shrivastava (vulcan-abhinav)
  • Jia-Bin Huang (vulcan-jbhuang)
  • Christopher Metzler (vulcan-metzler)
  • Matthias Zwicker (vulcan-zwicker)

Accounts

Vulcan has a base SLURM account vulcan which has a modest number of guaranteed billing resources available to all cluster users at any given time. Other faculty that have invested in Vulcan compute infrastructure have an additional account provided to their sponsored accounts on the cluster.

If you do not specify an account when submitting your job, you will receive the vulcan account. If your faculty sponsor has their own account, it is recommended to use that account for job submission.

The current faculty accounts are:

  • vulcan-abhinav
  • vulcan-djacobs
  • vulcan-jbhuang
  • vulcan-lsd
  • vulcan-metzler
  • vulcan-rama
  • vulcan-ramani
  • vulcan-yaser
  • vulcan-zwicker
$ sacctmgr show account format=account%20,description%30,organization%10
             Account                          Descr        Org
-------------------- ------------------------------ ----------
                 ...                            ...        ...
              vulcan                         vulcan     vulcan
      vulcan-abhinav   vulcan - abhinav shrivastava     vulcan
      vulcan-djacobs          vulcan - david jacobs     vulcan
      vulcan-jbhuang         vulcan - jia-bin huang     vulcan
          vulcan-lsd           vulcan - larry davis     vulcan
      vulcan-metzler         vulcan - chris metzler     vulcan
         vulcan-rama        vulcan - rama chellappa     vulcan
       vulcan-ramani     vulcan - ramani duraiswami     vulcan
        vulcan-yaser          vulcan - yaser yacoob     vulcan
      vulcan-zwicker      vulcan - matthias zwicker     vulcan
                 ...                            ...        ...

Faculty can manage this list of users via our Directory application in the Security Groups section. The security group that controls access has the prefix vulcan_ and then the faculty username. It will also list slurm://nexusctl.umiacs.umd.edu as the associated URI.

You can check your account associations by running the show_assoc command to see the accounts you are associated with. Please contact staff and include your faculty member in the conversation if you do not see the appropriate association. 

$ show_assoc
      User          Account MaxJobs       GrpTRES                                                                              QOS
---------- ---------------- ------- ------------- --------------------------------------------------------------------------------
       ...              ...     ...                                                                                            ...
   abhinav           vulcan      48                                       vulcan-cpu,vulcan-default,vulcan-medium,vulcan-scavenger
   abhinav   vulcan-abhinav      48                           vulcan-cpu,vulcan-default,vulcan-high,vulcan-medium,vulcan-scavenger
       ...              ...     ...                                                                                            ...

You can also see the total number of Track-able Resources (TRES) allowed for each account by running the following command. Please make sure you give the appropriate account that you are looking for. As shown below, there is a concurrent limit of 64 total GPUs for all users not in a contributing faculty group. 

$ sacctmgr show assoc account=vulcan format=user,account,qos,grptres
      User    Account                  QOS       GrpTRES
---------- ---------- -------------------- -------------
               vulcan                        gres/gpu=64
                  ...                                ...

QoS

You need to decide the QOS to submit with which will set a certain number of restrictions to your job. If you do not specify a QoS when submitting your job using the --qos parameter, you will receive the vulcan-default QoS assuming you are using a Vulcan account.

The following sacctmgr command will list the current QOS. Either the vulcan-default, vulcan-medium, or vulcan-high QOS is required for the vulcan-dpart partition. Please note that only faculty accounts (see above) have access to the vulcan-high QoS.

The following example will show you the current limits that the QOS have. The output is truncated to show only relevant Vulcan QoS. 

$ show_qos
                Name     MaxWall                        MaxTRES MaxJobsPU                      MaxTRESPU 
-------------------- ----------- ------------------------------ --------- ------------------------------ 
...
          vulcan-cpu  2-00:00:00                cpu=1024,mem=4T         4                                
      vulcan-default  7-00:00:00       cpu=4,gres/gpu=1,mem=32G         2                                
       vulcan-exempt  7-00:00:00     cpu=32,gres/gpu=8,mem=256G         2                                
         vulcan-high  1-12:00:00     cpu=16,gres/gpu=4,mem=128G         2                                
        vulcan-janus  3-00:00:00    cpu=32,gres/gpu=10,mem=256G                                          
       vulcan-medium  3-00:00:00       cpu=8,gres/gpu=2,mem=64G         2                                
       vulcan-sailon  3-00:00:00     cpu=32,gres/gpu=8,mem=256G                              gres/gpu=48 
    vulcan-scavenger  3-00:00:00     cpu=32,gres/gpu=8,mem=256G                                          
...

$ show_partition_qos
                Name MaxSubmitPU                      MaxTRESPU              GrpTRES 
-------------------- ----------- ------------------------------ -------------------- 
...
              vulcan         500                                 cpu=1760,mem=15824G 
    vulcan-scavenger         500                                                     
...

Storage

Vulcan has the following storage available. Please also review UMIACS Filesystem Data Storage policies including any volume that is labeled as scratch.

Vulcan users can also request Nexus project allocations.

Home Directories

You have 30GB of home directory storage available at /nfshomes/<username>. It has both Snapshots and Backups enabled.

Home directories are intended to store personal or configuration files only. We encourage you to not share any data in your home directory. You are encouraged to utilize our GitLab infrastructure to host your code repositories.

NOTE: To check your quota on this directory, use the command df -h ~.

Scratch Directories

Scratch data has no data protection including no snapshots and the data is not backed up. There are two types of scratch directories in the Vulcan compute infrastructure:

  • Network scratch directory
  • Local scratch directories

Network Scratch Directory

You have 300GB of scratch storage available at /vulcanscratch/<username>. It is not backed up or protected in any way. This directory is automounted so you will need to cd into the directory or request/specify a fully qualified file path to access this.

You may request a temporary increase of up to 500GB total space for a maximum of 120 days without any faculty approval by contacting staff. Once the temporary increase period is over, you will be contacted and given a one-week window of opportunity to clean and secure your data before staff will forcibly remove data to get your space back under 300GB. If you need space beyond 500GB or for longer than 120 days, you will need faculty approval and/or a project directory.

This file system is available on all submission and computational nodes within the cluster.

Local Scratch Directories

Each computational node that you can schedule compute jobs on has one or more local scratch directories. These are always named /scratch0, /scratch1, etc. These are almost always more performant than any other storage available to the job. However, you must stage their data within the confine of their job and stage the data out before the end of their job.

These local scratch directories have a tmpwatch job which will delete unaccessed data after 90 days, scheduled via maintenance jobs to run once a month at 1am. Different nodes will run the maintenance jobs on different days of the month to ensure the cluster is still highly available at all times. Please make sure you secure any data you write to these directories at the end of your job.

Datasets

We have read-only dataset storage available at /fs/vulcan-datasets. If there are datasets that you would like to see curated and made available, please see this page.

The list of Vulcan datasets we currently host can be viewed here.

Project Storage

Users within the Vulcan compute infrastructure can request project based allocations for up to 10TB for up to 180 days by contacting staff with approval from the Vulcan faculty manager (Dr. Shrivastava). These allocations will be available from /fs/vulcan-projects and /fs/cfar-projects under a name that you provide when you request the allocation. Near the end of the allocation period, staff will contact you and ask if you would like to renew the allocation for up to another 180 days (requires re-approval from Dr. Shrivastava).

  • If you are no longer in need of the storage allocation, you will need to relocate all desired data within two weeks of the end of the allocation period. Staff will then remove the allocation.
  • If you do not respond to staff's request by the end of the allocation period, staff will make the allocation temporarily inaccessible.
    • If you do respond asking for renewal but the original faculty approver does not respond within two weeks of the end of the allocation period, staff will also make the allocation temporarily inaccessible.
    • If one month from the end of the allocation period is reached without both you and the faculty approver responding, staff will remove the allocation.

Project storage is fully protected. It has snapshots enabled and is backed up nightly.

Object Storage

All Vulcan users can request project allocations in the UMIACS Object Store. Please contact staff with a short project name and the amount of storage you will need to get started.

To access this storage, you'll need to use a S3 client or our UMobj command line utilities.

An example on how to use the umobj command line utilities can be found here. A full set of documentation for the utilities can be found on the umobj Gitlab page.

Migration

Home Directories

The Nexus uses NFShomes home directories - if your UMIACS account was created before February 22nd, 2023, you were using /cfarhomes/<username> as your home directory on the standalone Vulcan cluster. While /cfarhomes is available on Nexus, your shell initialization scripts from it will not automatically load. Please copy over anything you need to your /nfshomes/<username> directory at your earliest convenience, as /cfarhomes will be retired in a two phase process:

  • Fri 11/17/2023, 5pm: cfarhomes directories are made read-only
  • Thu 12/21/2023, 5-8pm (monthly maintenance window): cfarhomes directories are taken offline
Retrieved from "https://wiki.umiacs.umd.edu/umiacs/index.php?title=Nexus/Vulcan&oldid=12277"