UMIACS - User contributions [en]

SLURM/JobSubmission

2016-07-14T14:32:48Z

Tgray26:

=Job Submission=

SLURM offers a variety of ways to run jobs. It is important to understand the different options available and how to request the resources required for a job in order for it to run successfully. All job submission should be done from submit nodes; any computational code should be run in a job allocation on compute nodes. The following commands outline how to allocate resources on the compute nodes and submit processes to be run on the allocated nodes.

==srun==
srun is the command used to run a process on the compute nodes in the cluster. It works by passing it a command (this could be a script) which will be run on a compute node and then srun will return. srun accepts many command line options to specify the resources required by the command passed to it, some common command line arguments are listed below and full documentation of all available options is available in the man page for srun which can be accessed by running <code>man srun</code>.
<pre>
tgray26@opensub01:srun --mem=100mb --time=1:00:00 bash -c 'echo "Hello World from" `hostname`'
Hello World from openlab06.umiacs.umd.edu
</pre>
It is important to understand that srun is an interactive command. By default input to srun is broadcast to all compute nodes running your process and output from the compute nodes is redirected to srun, this behavior can be changed; however, '''srun will always wait for the command passed to finish before exiting, so if you start a long running process and end your terminal session, your process will stop running on the compute nodes and your job will end'''. To run a non-interactive session that you can submit to the cluster and will remain running after you logout, you will need to wrap your srun commands in a batch script and submit it with [[#sbatch | sbatch]]
===Common srun arguments===
* <code>--mem=1gb</code> ''if no unit is given MB is assumed''
* <code>--nodes=2</code> ''if passed to srun, the given command will be run concurrently on each node''
* <code>--qos=dpart</code>
* <code>--time=hh:mm:ss</code> ''time needed to run your job''
* <code>--job-name=helloWorld</code>
* <code>--output filename</code> ''file to redirect stdout to''
* <code>--error filename</code> ''file to redirect stderr''
* <code>--partition $PNAME</code> ''request job run in the $PNAME partition''
* <code>--ntasks 2</code> ''request 2 "tasks" which map to cores on a CPU, if passed to srun the given command will be run concurrently on each core''

===Interactive Shell Sessions===
An interactive shell session on a compute node can be useful for debugging or developing code that isn't ready to be run as a batch job. To get an interactive shell on a node, use srun to invoke a shell:
<pre>
tgray26@opensub01:srun --pty --mem 1gb --time=01:00:00 bash
tgray26@openlab06:
</pre>
'''Please do not leave interactive shells running for long periods of time when you are not working. This blocks resources from being used by everyone else.'''

==salloc==
The salloc command can also be used to request resources be allocated without needing a batch script. Running salloc with a list of resources will allocate the resources you requested, create a job, and drop you into a subshell with the environment variables necessary to run commands in the newly created job allocation. When your time is up or you exit the subshell, your job allocation will be relinquished.
<pre>
tgray26@opensub00:salloc -N 1 --mem=2gb --time=01:00:00
salloc: Granted job allocation 159
tgray26@opensub00:srun /usr/bin/hostname
openlab00.umiacs.umd.edu
tgray26@opensub00:exit
exit
salloc: Relinquishing job allocation 159
</pre>
'''Please note that any commands not invoked with srun will be run locally on the submit node. Please be careful when using salloc.'''

==sbatch==
The sbatch command allows you to write a batch script to be submitted and run non-interactively on the compute nodes. To run a simple Hello World command on the compute nodes you could write a file, helloWorld.sh with the following contents:
<pre>
#!/bin/bash

srun bash -c 'echo Hello World from `hostname`'
</pre>
Then you need to submit the script with sbatch and request resources:
<pre>tgray26@opensub00:sbatch --mem=1gb --time=1:00:00 helloWorld.sh
Submitted batch job 121
</pre>
SLURM will return a job number that you can use to check the status of your job with squeue:
<pre>
tgray26@opensub00:squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
121 dpart helloWor tgray26 R 0:01 2 openlab[00-01]
</pre>
====Advanced Batch Scripts====
You can also write a batch script with all of your resources/options defined in the script itself. This is useful for jobs that need to be run 10s/100s/1000s of times. You can then handle any necessary environment setup and run commands on the resources you requested by invoking commands with srun. The srun commands can also be more complex and be told to only use portions of your entire job allocation, each of these distinct srun commands makes up one "job step". The batch script will be run on the first node allocated as part of your job allocation and each job step will be run on whatever resources you tell them to. In the following example I have a batch job that will request 2 nodes in the cluster, then I load a specific version of Python into my environment and submit two job steps, each one using one node. Since srun is blocks until the command finishes, I use the '&' operator to background the process so that both job steps can run at once; however, this means that I then need to use the wait command to block processing until all background processes have finished.
<pre>
#!/bin/bash

# Lines that begin with #SBATCH specify commands to be used by SLURM for scheduling

#SBATCH --job-name=helloWorld # sets the job name
#SBATCH --output helloWorld.out.%j # indicates a file to redirect STDOUT to; %j is the jobid
#SBATCH --error helloWorld.out.%j # indicates a file to redirect STDERR to; %j is the jobid
#SBATCH --time=00:05:00 # how long you think your job will take to complete; format=hh:mm:ss
#SBATCH --qos=dpart # set QOS, this will determine what resources can be requested
#SBATCH --nodes=2 # number of nodes to allocate for your job
#SBATCH --ntasks=4 # request 4 cpu cores be reserved for your node total
#SBATCH --ntasks-per-node=2 # request 2 cpu cores be reserved per node
#SBATCH --mem 1gb # memory required by job; if unit is not specified MB will be assumed

module load Python/2.7.9 # run any commands necessary to setup your environment

srun -N 1 --mem=512mb bash -c "hostname; python --version" & # use srun to invoke commands within your job; using an '&'
srun -N 1 --mem=512mb bash -c "hostname; python --version" & # will background the process allowing them to run concurrently
wait # wait for any background processes to complete

# once the end of the batch script is reached your job allocation will be revoked
</pre>
====More Examples====
More examples of how to use batch scripts to setup your environment for processing will be coming soon
===scancel===
The scancel command can be used to cancel job allocations or job steps that are no longer needed. It can be passed individual job IDs or an option to delete all of your jobs or jobs that meet certain criteria.
*<code>scancel 255</code> ''cancel job 255''
*<code>scancel 255.3</code> ''cancel job step 3 of job 255''
*<code>scancel --user tgray26 --partition dpart</code> ''cancel all jobs for tgray26 in the dpart partition''

=Requesting GPUs=
If you need to do processing on a GPU, you will need to request that your job have access to GPUs just as you need to request processors or cpu cores. You will also need to make sure that you submit your job to the correct partition since nodes with GPUs are often put into their own partition to prevent the nodes from being tied up by jobs that don't utilize GPUs. In SLURM, GPUs are considered "generic resources" also known as GRES. To request some number of GPUs be reserved/available for your job you can use the flag <code>--gres:gpu:2</code> or if there are multiple types of GPUs available in the cluster and you need a specific type, you can provide the type option to the gres flag <code>--gres:k20:1</code>
<pre>
tgray26@opensub01:srun --pty --partition gpu --gres=gpu:2 nvidia-smi
Wed Jul 13 15:33:18 2016
+------------------------------------------------------+
| NVIDIA-SMI 361.28 Driver Version: 361.28 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20c Off | 0000:03:00.0 Off | 0 |
| 30% 24C P0 48W / 225W | 11MiB / 4799MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K20c Off | 0000:84:00.0 Off | 0 |
| 30% 23C P0 52W / 225W | 11MiB / 4799MiB | 93% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
</pre>
Please note that your job will only be able to see/access the GPUs you requested. If you only need 1 GPU, please request only 1 GPU and the other one will be left available for other users:
<pre>
tgray26@opensub01:srun --pty --partition gpu --gres=gpu:k20:1 nvidia-smi
Wed Jul 13 15:31:29 2016
+------------------------------------------------------+
| NVIDIA-SMI 361.28 Driver Version: 361.28 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K20c Off | 0000:03:00.0 Off | 0 |
| 30% 24C P0 50W / 225W | 11MiB / 4799MiB | 92% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
</pre>
The <code>--gres</code> flag may also be passed to [[#sbatch | sbatch]] and [[#salloc | salloc]] rather than directly to [[#srun | srun]]

SLURM/JobStatus

2016-07-14T01:15:05Z

Tgray26: /* sstat */

=Job Status=
SLURM offers a variety of tools to check the status of your jobs before, during, and after execution. When you first submit your job, SLURM should give you a job ID which represents the resources allocated to your job. Individual calls to srun will spawn job steps which can also be queried individually.

==squeue==
The squeue command shows job status in the queue. If your job has not yet started, you can ask for an estimated start time with <code>squeue --start</code>
<pre>
tgray26@opensub00:squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
162 test2 helloWor tgray26 R 0:03 2 openlab[00-01]
</pre>
If you want to see the status of individual job steps you can use <code>squeue -s</code>
<pre>
tgray26@opensub00:squeue -s
STEPID NAME PARTITION USER TIME NODELIST
162.0 sleep test2 tgray26 0:05 openlab00
162.1 sleep test2 tgray26 0:05 openlab01
</pre>

==sstat==
The sstat command shows metrics from currently running job steps. If you don't specify a job step, the lowest job step is displayed.
<pre>
sstat --format JobID,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize <$JOBID>.<$JOBSTEP>
</pre>
<pre>
tgray26@opensub00: sstat --format JobID,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize 171
JobID NTasks Nodelist MaxRSS MaxVMSize AveRSS AveVMSize
------------ -------- -------------------- ---------- ---------- ---------- ----------
171.0 1 openlab00 0 186060K 0 107900K
tgray26@opensub00: sstat --format JobID,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize 171.1
JobID NTasks Nodelist MaxRSS MaxVMSize AveRSS AveVMSize
------------ -------- -------------------- ---------- ---------- ---------- ----------
171.1 1 openlab01 0 186060K 0 107900K
</pre>
Note that if you do not have any jobsteps, sstat will return an error.
<pre>
tgray26@opensub00: sstat --format JobID,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize 172
JobID NTasks Nodelist MaxRSS MaxVMSize AveRSS AveVMSize
------------ -------- -------------------- ---------- ---------- ---------- ----------
sstat: error: no steps running for job 237
</pre>
If you do not run any srun commands, you will not create any job steps and metrics will not be available for your job. Your batch scripts should follow this format:
<pre>
#!/bin/bash
#SBATCH ...
#SBATCH ...
# set environment up
module load ...

# launch job steps
srun <command to run> # that would be step 1
srun <command to run> # that would be step 2
</pre>

==sacct==
The sacct command shows metrics from past jobs.
<pre>
tgray26@opensub00:sacct
JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
162 helloWorld test2 staff 2 COMPLETED 0:0
162.batch batch staff 1 COMPLETED 0:0
162.0 sleep staff 1 COMPLETED 0:0
162.1 sleep staff 1 COMPLETED 0:0
163 helloWorld test2 staff 2 COMPLETED 0:0
163.batch batch staff 1 COMPLETED 0:0
163.0 sleep staff 1 COMPLETED 0:0
</pre>
To check one specific job, you can run something like the following (if you omit .<$JOBSTEP>, all jobsteps will be shown):
<pre>sacct --format JobID,jobname,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize,Elapsed -j <$JOBID>.<$JOBSTEP></pre>
<pre>
tgray26@opensub00:sacct --format JobID,jobname,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize,Elapsed -j 171
JobID JobName NTasks NodeList MaxRSS MaxVMSize AveRSS AveVMSize Elapsed
------------ ---------- -------- --------------- ---------- ---------- ---------- ---------- ----------
171 helloWorld openlab[00-01] 00:00:30
171.batch batch 1 openlab00 0 119784K 0 113120K 00:00:30
171.0 sleep 1 openlab00 0 186060K 0 107900K 00:00:30
171.1 sleep 1 openlab01 0 186060K 0 107900K 00:00:30
</pre>

SLURM/JobSubmission

2016-07-13T20:11:31Z

2015-07-30T20:35:36Z

Tgray26:

InputMethodEditors

2015-05-14T21:49:52Z

2012-02-15T18:56:51Z

Tgray26:

__NOTOC__
We support printing from user managed Mac OS X 10.5/10.6 machines. Please note that you have to be on a UMIACS network directly or connected to the [[VPN]].

[[Image:osx-print00.png|right|100px]]
=System Preferences=
To start you need to open your System Preferences from your Dock or the Applications folder. Once you have opened it you will need to click '''Print & Fax''' from the Hardware panel ( Called '''Print & Scan''' in 10.7 (Lion) ).

[[Image:osx-print01.png|right|100px]]

=Print & Fax=
When the '''Print & Fax''' (or '''Print & Scan''') window appears you will need to create a new local printer by clicking the '''+''' icon in the lower left corner of the first pane in the window.

=Add Printer=
This will bring up a Add Printer dialog. Please ensure that the that you have selected IP.

* Set Protocol to '''Internet Printing Protocol - IPP'''
* Set Address to '''print.umiacs.umd.edu'''
* Set the queue to =printers/queue= in this example for cps3142 it would be '''printers/cps3142'''. You have to make sure the queue is prefixed by '''printers/'''
* Set Name to the queue you are trying to use
* It will always select '''Generic Postscript Printer'''. If you need to access the more advanced features of a queue/printer you will need take extra steps, please see the Advanced section at the bottom of this page.
* Select Add

[[Image:osx-print02.png]]

You should now be able to print to this printer/queue from any Mac OS X print menu.

=Exceptional Printers=
For the following printers you will need to add a different driver because they don't work well with the generic one causing any print job to get stuck in the "Printing" status on on your Mac and with a state of "Stopped" in the CUPS web interface here: http://print.umiacs.umd.edu/jobs. In order to fix this you do not need to download an additional driver, it is already present in Mac OS X.

Printers: '''ps4430b''', '''ps3126'''

* In the previous step, while adding the printer, instead of leaving the selected '''Generic Postscript Printer''' choose '''Select Printer Software...'''

[[Image:MacPrinterDriverPic3.png]]

* Look for '''HP LaserJet p4010''' and select the series.

[[Image:MacPrinterDriverPic4.png]]

* After selecting and clicking OK you will receive the following window, where you will be able to choose options for the printer. Enable duplex, and change Collation in Printer to more than 288 MB

[[Image:MacPrinterDriverPic5.png]]

* After you click on Continue the dialogue box should close and you should see the printer added to your list of installed printers. The printer is now ready for use.

=Advanced=

You will need to first download a copy of the Postcript Printer Definition file from the print server. This can be fetched with any normal browser by going to

http://print.umiacs.umd.edu/printers/queue.ppd

Where '''queue''' is the queue/printer you want to get extra features from. For example here is a few links to our major public printers.

* http://print.umiacs.umd.edu/printers/cps3142.ppd - AV Williams Building room 3142
* http://print.umiacs.umd.edu/printers/ps3142.ppd - AV Williams Building room 3142
* http://print.umiacs.umd.edu/printers/cps4430a.ppd - AV Williams Building room 4430
* http://print.umiacs.umd.edu/printers/cps296-3120Hb.ppd - Biomolecular Sciences Building room 3120H

Please save this file to your hard disk as you will need use the file chooser to select it in the next step. If you don't see the queue you want please just reconstruct the URL above and substitute in your queue name.

Next go back to the '''Print & Fax''' (or '''Print & Scan''') dialog and click the '''Options & Supplies...''' button for the queue you want to modify.

[[Image:osx-print03.png]]

You will need to navigate to the Driver tab and select the option '''Other...''' from the Print Using option. This will bring up a file chooser that you will need to select the file you downloaded from.

[[Image:osx-printingadv00.png]]

This will fill in all the default options and you can then click Okay. If you have questions or need help please contact [mailto:staff@umiacs.umd.edu staff@umiacs.umd.edu].

MacOSPrinting

2012-02-15T18:32:33Z

Tgray26:

__NOTOC__
We support printing from user managed Mac OS X 10.5/10.6 machines. Please note that you have to be on a UMIACS network directly or connected to the [[VPN]].

[[Image:osx-print00.png|right|100px]]
=System Preferences=
To start you need to open your System Preferences from your Dock or the Applications folder. Once you have opened it you will need to click '''Print & Fax''' from the Hardware panel ( Called '''Print & Scan''' in 10.7 (Lion) ).

[[Image:osx-print01.png|right|100px]]

=Print & Fax=
When the '''Print & Fax''' (or '''Print & Scan''') window appears you will need to create a new local printer by clicking the '''+''' icon in the lower left corner of the first pane in the window.

=Add Printer=
This will bring up a Add Printer dialog. Please ensure that the that you have selected IP.

* Set Protocol to '''Internet Printing Protocol - IPP'''
* Set Address to '''print.umiacs.umd.edu'''
* Set the queue to =printers/queue= in this example for cps3142 it would be '''printers/cps3142'''. You have to make sure the queue is prefixed by '''printers/'''
* Set Name to the queue you are trying to use
* It will always select '''Generic Postscript Printer'''. If you need to access the more advanced features of a queue/printer you will need take extra steps, please see the Advanced section at the bottom of this page.
* Select Add

[[Image:osx-print02.png]]

You should now be able to print to this printer/queue from any Mac OS X print menu.

=Exceptional Printers=
For the following printers you will need to add a different driver because they don't work well with the generic one and will throw you an error. You don't have to download the driver, it is already present in Mac OS.

Printers: '''ps4430b''', '''ps3126'''

* In the previous step, while adding the printer, instead of leaving the selected '''Generic Postscript Printer''' choose '''Select Printer Software...'''

[[Image:MacPrinterDriverPic3.png]]

* Look for '''HP LaserJet p4010''' and select the series.

[[Image:MacPrinterDriverPic4.png]]

* After selecting and clicking OK you will receive the following window, where you will be able to choose options for the printer. Enable duplex, and change Collation in Printer to more than 288 MB

[[Image:MacPrinterDriverPic5.png]]

* After you click on Continue it will look like the following and once you add the printer you will be able to use it:

[[Image:MacPrinterDriverPic5.png]]

=Advanced=

You will need to first download a copy of the Postcript Printer Definition file from the print server. This can be fetched with any normal browser by going to

http://print.umiacs.umd.edu/printers/queue.ppd

Where '''queue''' is the queue/printer you want to get extra features from. For example here is a few links to our major public printers.

* http://print.umiacs.umd.edu/printers/cps3142.ppd - AV Williams Building room 3142
* http://print.umiacs.umd.edu/printers/ps3142.ppd - AV Williams Building room 3142
* http://print.umiacs.umd.edu/printers/cps4430a.ppd - AV Williams Building room 4430
* http://print.umiacs.umd.edu/printers/cps296-3120Hb.ppd - Biomolecular Sciences Building room 3120H

Please save this file to your hard disk as you will need use the file chooser to select it in the next step. If you don't see the queue you want please just reconstruct the URL above and substitute in your queue name.

Next go back to the '''Print & Fax''' (or '''Print & Scan''') dialog and click the '''Options & Supplies...''' button for the queue you want to modify.

[[Image:osx-print03.png]]

You will need to navigate to the Driver tab and select the option '''Other...''' from the Print Using option. This will bring up a file chooser that you will need to select the file you downloaded from.

[[Image:osx-printingadv00.png]]

This will fill in all the default options and you can then click Okay. If you have questions or need help please contact [mailto:staff@umiacs.umd.edu staff@umiacs.umd.edu].