SLURM - Revision history

Mbaney at 20:49, 18 February 2025

2025-02-18T20:49:24Z

← Older revision		Revision as of 20:49, 18 February 2025
Line 54:		Line 54:
	</pre>		</pre>

	This will start running the notebook on port 8889. <b>Note:</b> You must keep this shell window open to be able to connect. If the submission node for the cluster you are using is not accessible via the public internet, you must also be on a machine connected to the UMIACS network ~~or connected to our [[Network/VPN \| VPN]]~~ in order to access the Jupyter notebook once you start the SSH tunnel, so ensure this is the case before starting the tunnel. Then, on your local machine, run		This will start running the notebook on port 8889. <b>Note:</b> You must keep this shell window open to be able to connect. If the submission node for the cluster you are using is not accessible via the public internet, you must also be on a machine connected to the UMIACS network in order to access the Jupyter notebook once you start the SSH tunnel, so ensure this is the case before starting the tunnel. Then, on your local machine, run
	<pre>		<pre>
	ssh -N -f -L localhost:8888:<NODENAME>:8889 <USERNAME>@<SUBMISSIONNODE>.umiacs.umd.edu		ssh -N -f -L localhost:8888:<NODENAME>:8889 <USERNAME>@<SUBMISSIONNODE>.umiacs.umd.edu

Wgeckle at 15:37, 14 January 2025

2025-01-14T15:37:20Z

← Older revision		Revision as of 15:37, 14 January 2025
Line 63:		Line 63:
	For example, assuming your username is <code>username</code> and that you are using the [[Nexus]] cluster, have been [[Nexus#Access \| assigned]] the nexusgroup submission nodes, and are assigned compute node tron00.umiacs.umd.edu:		For example, assuming your username is <code>username</code> and that you are using the [[Nexus]] cluster, have been [[Nexus#Access \| assigned]] the nexusgroup submission nodes, and are assigned compute node tron00.umiacs.umd.edu:
	<pre>		<pre>
	ssh -N -f -L localhost:8888:tron00~~.umiacs.umd.edu~~:8889 username@nexusgroup.umiacs.umd.edu		ssh -N -f -L localhost:8888:tron00:8889 username@nexusgroup.umiacs.umd.edu
	</pre>		</pre>

	You can then open a web browser and type in <code>localhost:8888</code> to access the notebook.		You can then open a web browser and type in <code>localhost:8888</code> to access the notebook. To ensure that the notebook is running on the desired compute node, open a Python notebook and write
			<pre>
			import socket
			socket.gethostname()
			</pre>

			into one of the cells. After hitting Shift+Enter, the hostname of the compute node should be displayed.

	Notes:		Notes:
	* Later versions of Jupyter have token authentication enabled by default - you will need to prepend the <code>/?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</code> part of the URL provided by the terminal output after starting the notebook in order to connect if this is the case. e.g. <code>localhost:8888/?token=fcc6bd0f996e7aa89376c33cb34f7b80890502aacc97d98e</code>		* Later versions of Jupyter have token authentication enabled by default - you will need to prepend the <code>/?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</code> part of the URL provided by the terminal output after starting the notebook in order to connect if this is the case. e.g. <code>localhost:8888/?token=fcc6bd0f996e7aa89376c33cb34f7b80890502aacc97d98e</code>
	* If the port on the compute node mentioned in the example above (8889) is not working, it may be that someone else has already started a process (Jupyter notebook or otherwise) using that specific port number on that specific compute node. The port number can be replaced with any other [https://en.wikipedia.org/wiki/Ephemeral_port ephemeral port] number you'd like, just make sure to change it in both the command you run on the compute node and the ssh command from your local machine.		* If the port on the compute node mentioned in the example above (8889) is not working, it may be that someone else has already started a process (Jupyter notebook or otherwise) using that specific port number on that specific compute node. The port number can be replaced with any other [https://en.wikipedia.org/wiki/Ephemeral_port ephemeral port] number you'd like, just make sure to change it in both the command you run on the compute node and the ssh command from your local machine.
			* The SSH tunnel process is not interruptible with Ctrl+C in the terminal, so you cannot exit out of it without rendering the host port unusable. For example, if you use port 8888 on your local machine for the SSH tunnel command, then hit Ctrl+Z in the terminal, port 8888 on your local machine is no longer usable until restart. If you need to get the SSH tunnel back up before restarting your computer, the simplest workaround is to use another ephemeral port on your local machine.

	=Quick Guide to translate PBS/TORQUE to Slurm=		=Quick Guide to translate PBS/TORQUE to Slurm=

Mbaney at 21:25, 23 October 2024

2024-10-23T21:25:38Z

← Older revision		Revision as of 21:25, 23 October 2024
Line 1:		Line 1:
	=~~Simple Linux Utility for Resource Management (SLURM)~~=		=Slurm Workload Manager=
	~~SLURM~~ is an open-source workload manager designed for Linux clusters of all sizes. It provides three key functions. First, it allocates exclusive or non-exclusive access to resources (computer nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (typically a parallel job) on a set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work.		Slurm is an open-source workload manager designed for Linux clusters of all sizes. It provides three key functions. First, it allocates exclusive or non-exclusive access to resources (computer nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (typically a parallel job) on a set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work.

	==Documentation==		==Documentation==
Line 16:		Line 16:

	==Commands==		==Commands==
	Below are some of the common commands used in ~~SLURM~~. Further information on how to use these commands is found in the documentation linked above. To see all flags available for a command, please check the command's manual by using <code>man <COMMAND></code> on the command line.		Below are some of the common commands used in Slurm. Further information on how to use these commands is found in the documentation linked above. To see all flags available for a command, please check the command's manual by using <code>man <COMMAND></code> on the command line.

	====[https://slurm.schedmd.com/srun.html srun]====		====[https://slurm.schedmd.com/srun.html srun]====
	srun runs a parallel job on a cluster managed by ~~SLURM~~. If necessary, it will first create a resource allocation in which to run the parallel job.		srun runs a parallel job on a cluster managed by Slurm. If necessary, it will first create a resource allocation in which to run the parallel job.

	====[https://slurm.schedmd.com/salloc.html salloc]====		====[https://slurm.schedmd.com/salloc.html salloc]====
	salloc allocates a ~~SLURM~~ job allocation, which is a set of resources (nodes), possibly with some set of constraints (e.g. number of processors per node). When salloc successfully obtains the requested allocation, it then runs the command specified by the user. Finally, when the user specified command is complete, salloc relinquishes the job allocation. If no command is specified, salloc runs the user's default shell.		salloc allocates a Slurm job allocation, which is a set of resources (nodes), possibly with some set of constraints (e.g. number of processors per node). When salloc successfully obtains the requested allocation, it then runs the command specified by the user. Finally, when the user specified command is complete, salloc relinquishes the job allocation. If no command is specified, salloc runs the user's default shell.

	====[https://slurm.schedmd.com/sbatch.html sbatch]====		====[https://slurm.schedmd.com/sbatch.html sbatch]====
	sbatch submits a batch script to ~~SLURM~~. The batch script may be given to sbatch through a file name on the command line, or if no file name is specified, sbatch will read in a script from standard input. The batch script may contain options preceded with <code>#SBATCH</code> before any executable commands in the script.		sbatch submits a batch script to Slurm. The batch script may be given to sbatch through a file name on the command line, or if no file name is specified, sbatch will read in a script from standard input. The batch script may contain options preceded with <code>#SBATCH</code> before any executable commands in the script.

	====[https://slurm.schedmd.com/squeue.html squeue]====		====[https://slurm.schedmd.com/squeue.html squeue]====
	squeue views job and job step information for jobs managed by ~~SLURM~~.		squeue views job and job step information for jobs managed by Slurm.

	====[https://slurm.schedmd.com/scancel.html scancel]====		====[https://slurm.schedmd.com/scancel.html scancel]====
Line 34:		Line 34:

	====[https://slurm.schedmd.com/sacct.html sacct]====		====[https://slurm.schedmd.com/sacct.html sacct]====
	sacct displays job accounting data stored in the job accounting log file or ~~SLURM~~ database in a variety of forms for your analysis. The sacct command displays information on jobs, job steps, status, and exitcodes by default. You can tailor the output with the use of the <code>--format=</code> option to specify the fields to be shown.		sacct displays job accounting data stored in the job accounting log file or Slurm database in a variety of forms for your analysis. The sacct command displays information on jobs, job steps, status, and exitcodes by default. You can tailor the output with the use of the <code>--format=</code> option to specify the fields to be shown.

	====[https://slurm.schedmd.com/sstat.html sstat]====		====[https://slurm.schedmd.com/sstat.html sstat]====
Line 72:		Line 72:
	* If the port on the compute node mentioned in the example above (8889) is not working, it may be that someone else has already started a process (Jupyter notebook or otherwise) using that specific port number on that specific compute node. The port number can be replaced with any other [https://en.wikipedia.org/wiki/Ephemeral_port ephemeral port] number you'd like, just make sure to change it in both the command you run on the compute node and the ssh command from your local machine.		* If the port on the compute node mentioned in the example above (8889) is not working, it may be that someone else has already started a process (Jupyter notebook or otherwise) using that specific port number on that specific compute node. The port number can be replaced with any other [https://en.wikipedia.org/wiki/Ephemeral_port ephemeral port] number you'd like, just make sure to change it in both the command you run on the compute node and the ssh command from your local machine.

	=Quick Guide to translate PBS/TORQUE to ~~SLURM~~=		=Quick Guide to translate PBS/TORQUE to Slurm=
	[https://en.wikipedia.org/wiki/TORQUE PBS/TORQUE] was the previous workload manager and job submission framework used at UMIACS prior to ~~SLURM~~'s adoption. Below is a quick guide of how to translate some common PBS/TORQUE commands to ~~SLURM~~ ones.		[https://en.wikipedia.org/wiki/TORQUE PBS/TORQUE] was the previous workload manager and job submission framework used at UMIACS prior to Slurm's adoption. Below is a quick guide of how to translate some common PBS/TORQUE commands to Slurm ones.

	{\| class="wikitable"		{\| class="wikitable"
Line 80:		Line 80:
	!		!
	!PBS/TORQUE		!PBS/TORQUE
	!~~SLURM~~		!Slurm
	\|-		\|-
	!Job submission		!Job submission
Line 108:		Line 108:
	!		!
	!PBS/TORQUE		!PBS/TORQUE
	!~~SLURM~~		!Slurm
	\|-		\|-
	!Job ID		!Job ID
Line 128:		Line 128:
	!		!
	!PBS/TORQUE		!PBS/TORQUE
	!~~SLURM~~		!Slurm
	\|-		\|-
	!Script directive		!Script directive

Mbaney: /* Documentation */

2024-10-15T19:36:46Z

Documentation

← Older revision		Revision as of 19:36, 15 October 2024
Line 12:		Line 12:

	Related documentation:		Related documentation:
			:[[Compute/DataLocality \| Optimizing Storage Performance]]
	:[[VS Code#Cluster_Usage \| Using VS Code]]		:[[VS Code#Cluster_Usage \| Using VS Code]]

Mbaney at 14:22, 25 July 2024

2024-07-25T14:22:02Z

← Older revision		Revision as of 14:22, 25 July 2024
Line 71:		Line 71:
	* If the port on the compute node mentioned in the example above (8889) is not working, it may be that someone else has already started a process (Jupyter notebook or otherwise) using that specific port number on that specific compute node. The port number can be replaced with any other [https://en.wikipedia.org/wiki/Ephemeral_port ephemeral port] number you'd like, just make sure to change it in both the command you run on the compute node and the ssh command from your local machine.		* If the port on the compute node mentioned in the example above (8889) is not working, it may be that someone else has already started a process (Jupyter notebook or otherwise) using that specific port number on that specific compute node. The port number can be replaced with any other [https://en.wikipedia.org/wiki/Ephemeral_port ephemeral port] number you'd like, just make sure to change it in both the command you run on the compute node and the ssh command from your local machine.

	=Quick Guide to translate PBS/~~Torque~~ to SLURM=		=Quick Guide to translate PBS/TORQUE to SLURM=
			[https://en.wikipedia.org/wiki/TORQUE PBS/TORQUE] was the previous workload manager and job submission framework used at UMIACS prior to SLURM's adoption. Below is a quick guide of how to translate some common PBS/TORQUE commands to SLURM ones.

	{\| class="wikitable"		{\| class="wikitable"
Line 77:		Line 78:
	\|-		\|-
	!		!
	!PBS/~~Torque~~		!PBS/TORQUE
	!SLURM		!SLURM
	\|-		\|-
Line 89:		Line 90:
	\|-		\|-
	!Job status (by job)		!Job status (by job)
	\|qstat [job_id]		\|qstat [job_id]
	\|squeue --job [job_id]		\|squeue --job [job_id]
	\|-		\|-
Line 105:		Line 106:
	\|-		\|-
	!		!
	!PBS/~~Torque~~		!PBS/TORQUE
	!SLURM		!SLURM
	\|-		\|-
Line 116:		Line 117:
	\|$SLURM_SUBMIT_DIR		\|$SLURM_SUBMIT_DIR
	\|-		\|-
	!Node List		!Node List
	\|$PBS_NODEFILE		\|$PBS_NODEFILE
	\|$SLURM_JOB_NODELIST		\|$SLURM_JOB_NODELIST
Line 125:		Line 126:
	\|-		\|-
	!		!
	!PBS/~~Torque~~		!PBS/TORQUE
	!SLURM		!SLURM
	\|-		\|-
Line 132:		Line 133:
	\|#SBATCH		\|#SBATCH
	\|-		\|-
	!Job Name		!Job Name
	\| -N [name]		\| -N [name]
	\| --job-name=[name] OR -J [name]		\| --job-name=[name] OR -J [name]
Line 149:		Line 150:
	\|-		\|-
	!Memory Size		!Memory Size
	\| -l mem=[MB]		\| -l mem=[MB]
	\| --mem=[MB] OR --mem-per-cpu=[MB]		\| --mem=[MB] OR --mem-per-cpu=[MB]
	\|-		\|-

Mbaney at 17:18, 15 May 2024

2024-05-15T17:18:09Z

← Older revision		Revision as of 17:18, 15 May 2024
Line 10:		Line 10:
	:[http://slurm.schedmd.com/documentation.html Official Documentation]		:[http://slurm.schedmd.com/documentation.html Official Documentation]
	:[http://slurm.schedmd.com/faq.html FAQ]		:[http://slurm.schedmd.com/faq.html FAQ]

			Related documentation:
			:[[VS Code#Cluster_Usage \| Using VS Code]]

	==Commands==		==Commands==

Seide: /* Documentation */

2024-05-10T15:56:56Z

Documentation

← Older revision		Revision as of 15:56, 10 May 2024
Line 7:		Line 7:
	:[[SLURM/ClusterStatus \| Checking Cluster Status]]		:[[SLURM/ClusterStatus \| Checking Cluster Status]]
	:[[SLURM/Priority \| Understanding Job Priority]]		:[[SLURM/Priority \| Understanding Job Priority]]
			:[[SLURM/Preemption \| Job Preemption Overview]]
	:[http://slurm.schedmd.com/documentation.html Official Documentation]		:[http://slurm.schedmd.com/documentation.html Official Documentation]
	:[http://slurm.schedmd.com/faq.html FAQ]		:[http://slurm.schedmd.com/faq.html FAQ]

Mbaney: /* Running Jupyter Notebook on a Compute Node */

2024-03-15T15:35:25Z

Running Jupyter Notebook on a Compute Node

← Older revision		Revision as of 15:35, 15 March 2024
Line 62:		Line 62:

	You can then open a web browser and type in <code>localhost:8888</code> to access the notebook.		You can then open a web browser and type in <code>localhost:8888</code> to access the notebook.

			Notes:
	* Later versions of Jupyter have token authentication enabled by default - you will need to prepend the <code>/?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</code> part of the URL provided by the terminal output after starting the notebook in order to connect if this is the case. e.g. <code>localhost:8888/?token=fcc6bd0f996e7aa89376c33cb34f7b80890502aacc97d98e</code>		* Later versions of Jupyter have token authentication enabled by default - you will need to prepend the <code>/?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</code> part of the URL provided by the terminal output after starting the notebook in order to connect if this is the case. e.g. <code>localhost:8888/?token=fcc6bd0f996e7aa89376c33cb34f7b80890502aacc97d98e</code>
	* If the port on the compute node mentioned in the example above (8889) is not working, it may be that someone else has already started a process (Jupyter notebook or otherwise) using that specific port number on that specific compute node. The port number can be replaced with any other [https://en.wikipedia.org/wiki/Ephemeral_port ephemeral port] number you'd like, just make sure to change it in both the command you run on the compute node and the ssh command from your local machine.		* If the port on the compute node mentioned in the example above (8889) is not working, it may be that someone else has already started a process (Jupyter notebook or otherwise) using that specific port number on that specific compute node. The port number can be replaced with any other [https://en.wikipedia.org/wiki/Ephemeral_port ephemeral port] number you'd like, just make sure to change it in both the command you run on the compute node and the ssh command from your local machine.

Mbaney at 13:46, 12 March 2024

2024-03-12T13:46:07Z

← Older revision		Revision as of 13:46, 12 March 2024
Line 58:		Line 58:
	For example, assuming your username is <code>username</code> and that you are using the [[Nexus]] cluster, have been [[Nexus#Access \| assigned]] the nexusgroup submission nodes, and are assigned compute node tron00.umiacs.umd.edu:		For example, assuming your username is <code>username</code> and that you are using the [[Nexus]] cluster, have been [[Nexus#Access \| assigned]] the nexusgroup submission nodes, and are assigned compute node tron00.umiacs.umd.edu:
	<pre>		<pre>
	ssh -N -f -L localhost:8888:tron00.umiacs.umd.edu:8889 username@~~nexusgroup00~~.umiacs.umd.edu		ssh -N -f -L localhost:8888:tron00.umiacs.umd.edu:8889 username@nexusgroup.umiacs.umd.edu
	</pre>		</pre>

Mbaney: /* Running Jupyter Notebook on a Compute Node */

2023-11-06T19:59:35Z

Running Jupyter Notebook on a Compute Node

← Older revision		Revision as of 19:59, 6 November 2023
Line 63:		Line 63:
	You can then open a web browser and type in <code>localhost:8888</code> to access the notebook.		You can then open a web browser and type in <code>localhost:8888</code> to access the notebook.
	* Later versions of Jupyter have token authentication enabled by default - you will need to prepend the <code>/?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</code> part of the URL provided by the terminal output after starting the notebook in order to connect if this is the case. e.g. <code>localhost:8888/?token=fcc6bd0f996e7aa89376c33cb34f7b80890502aacc97d98e</code>		* Later versions of Jupyter have token authentication enabled by default - you will need to prepend the <code>/?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</code> part of the URL provided by the terminal output after starting the notebook in order to connect if this is the case. e.g. <code>localhost:8888/?token=fcc6bd0f996e7aa89376c33cb34f7b80890502aacc97d98e</code>
	* If the port on the compute node mentioned in the example above (8889) is not working, it may be that someone else has already started a notebook using that specific port number on that specific compute node. The port number can be replaced with any other [https://en.wikipedia.org/wiki/Ephemeral_port ephemeral port] number you'd like, just make sure to change it in both the command you run on the compute node and the ssh command from your local machine.		* If the port on the compute node mentioned in the example above (8889) is not working, it may be that someone else has already started a process (Jupyter notebook or otherwise) using that specific port number on that specific compute node. The port number can be replaced with any other [https://en.wikipedia.org/wiki/Ephemeral_port ephemeral port] number you'd like, just make sure to change it in both the command you run on the compute node and the ssh command from your local machine.

	=Quick Guide to translate PBS/Torque to SLURM=		=Quick Guide to translate PBS/Torque to SLURM=