SLURM/Preemption - Revision history

Mbaney: /* On job cancellation/preemption */

2026-02-20T19:20:59Z

On job cancellation/preemption

← Older revision		Revision as of 19:20, 20 February 2026
Line 23:		Line 23:
	You can handle the SIGTERM signal and run <code>scontrol --json show job $SLURM_JOBID</code> within your script. Based on the value of <tt>job_state</tt>, your script can take a different codepath depending on if it was cancelled or was preempted.		You can handle the SIGTERM signal and run <code>scontrol --json show job $SLURM_JOBID</code> within your script. Based on the value of <tt>job_state</tt>, your script can take a different codepath depending on if it was cancelled or was preempted.

	The output of <code>scontrol --json show job $SLURM_JOBID</code> is equivalent to the Slurm REST API's endpoint "GET /slurm/v0.0.44/job/{job_id}". For more information on how to parse the output of this scontrol command, please refer to [~~Slurm's REST API documentation~~ https://slurm.schedmd.com/rest_api.html#slurmV0044GetJob].		The output of <code>scontrol --json show job $SLURM_JOBID</code> is equivalent to the Slurm REST API's endpoint "GET /slurm/v0.0.44/job/{job_id}". For more information on how to parse the output of this scontrol command, please refer to [https://slurm.schedmd.com/rest_api.html#slurmV0044GetJob Slurm's REST API documentation].

	==On resuming==		==On resuming==

Mbaney at 19:20, 20 February 2026

2026-02-20T19:20:46Z

← Older revision		Revision as of 19:20, 20 February 2026
Line 23:		Line 23:
	You can handle the SIGTERM signal and run <code>scontrol --json show job $SLURM_JOBID</code> within your script. Based on the value of <tt>job_state</tt>, your script can take a different codepath depending on if it was cancelled or was preempted.		You can handle the SIGTERM signal and run <code>scontrol --json show job $SLURM_JOBID</code> within your script. Based on the value of <tt>job_state</tt>, your script can take a different codepath depending on if it was cancelled or was preempted.

	The output of <code>scontrol --json show job $SLURM_JOBID</code> is equivalent to the Slurm REST API's endpoint "GET /slurm/v0.0.40/job/{job_id}". For more information on how to parse the output of this scontrol command, please refer to Slurm's REST API documentation~~. [~~https://slurm.schedmd.com/rest_api.html#~~slurmV0040GetJob~~]		The output of <code>scontrol --json show job $SLURM_JOBID</code> is equivalent to the Slurm REST API's endpoint "GET /slurm/v0.0.44/job/{job_id}". For more information on how to parse the output of this scontrol command, please refer to [Slurm's REST API documentation https://slurm.schedmd.com/rest_api.html#slurmV0044GetJob].

	==On resuming==		==On resuming==

Mbaney at 18:58, 25 November 2025

2025-11-25T18:58:16Z

← Older revision		Revision as of 18:58, 25 November 2025
Line 1:		Line 1:
	If you are submitting to a partition ~~which~~ is eligible for preemption (e.g., scavenger), you are responsible for making sure that your job can be interrupted/restarted gracefully. Below is some information that you can use to determine if your job has been cancelled or preempted. You should be able to take different code paths during startup/shutdown of your jobs based on this information.		If you are submitting a job in [[SLURM]] to a partition that is eligible for preemption (e.g., scavenger), you are responsible for making sure that your job can be interrupted/restarted gracefully. Below is some information that you can use to determine if your job has been cancelled or preempted. You should be able to take different code paths during startup/shutdown of your jobs based on this information.

	=Flowchart for job cancellation/preemption=		=Flowchart for job cancellation/preemption=

Mbaney at 17:11, 20 October 2025

2025-10-20T17:11:55Z

← Older revision		Revision as of 17:11, 20 October 2025
Line 7:		Line 7:
	# Slurm node(s) send SIGCONT and SIGTERM around the same time, and the following fields of note are set in the output of <code>scontrol --json show job $SLURM_JOBID</code>:		# Slurm node(s) send SIGCONT and SIGTERM around the same time, and the following fields of note are set in the output of <code>scontrol --json show job $SLURM_JOBID</code>:
	#* <tt>job_state = ['CANCELLED', 'COMPLETING']</tt>		#* <tt>job_state = ['CANCELLED', 'COMPLETING']</tt>
	# If the processes don't stop within a certain amount of time, eventually SIGKILL is sent. On the [[Nexus]] cluster, this ~~time~~ is ~~currently~~ 5 minutes.		# If the processes don't stop within a certain amount of time, eventually SIGKILL is sent. On the [[Nexus]] cluster, this is 5 minutes.

	==Preemption/Requeue==		==Preemption/Requeue==
Line 14:		Line 14:
	#* <tt>job_state = ['PENDING', 'COMPLETING']</tt>		#* <tt>job_state = ['PENDING', 'COMPLETING']</tt>
	#* <tt>restart_cnt += 1</tt>		#* <tt>restart_cnt += 1</tt>
	# If the processes don't stop within a certain amount of time, eventually SIGKILL is sent. On the [[Nexus]] cluster, this ~~time~~ is ~~currently~~ 5 minutes.		# If the processes don't stop within a certain amount of time, eventually SIGKILL is sent. On the [[Nexus]] cluster, this is 5 minutes.
	# Once the job is stopped, it enters the PENDING state for ~~two~~ minutes, and then is eligible to be run again.		# Once the job is stopped, it enters the PENDING state for 2 minutes, and then is eligible to be run again.
	# When the job runs again, an additional environment variable <tt>SLURM_RESTART_COUNT</tt> is defined, which reports the number of times the job has been preempted/requeued.		# When the job runs again, an additional environment variable <tt>SLURM_RESTART_COUNT</tt> is defined, which reports the number of times the job has been preempted/requeued.

Mbaney at 15:16, 15 August 2025

2025-08-15T15:16:18Z

← Older revision		Revision as of 15:16, 15 August 2025
Line 5:		Line 5:
	==Cancellation==		==Cancellation==
	# Slurm controller sends an internal cancel signal to Slurm node(s) where the job is currently assigned.		# Slurm controller sends an internal cancel signal to Slurm node(s) where the job is currently assigned.
	# Slurm node(s) send SIGCONT and SIGTERM around the same time, and the following fields of note are set in the output of <tt>scontrol --json show job $SLURM_JOBID</tt>:		# Slurm node(s) send SIGCONT and SIGTERM around the same time, and the following fields of note are set in the output of <code>scontrol --json show job $SLURM_JOBID</code>:
	#* <tt>job_state = ['CANCELLED', 'COMPLETING']</tt>		#* <tt>job_state = ['CANCELLED', 'COMPLETING']</tt>
	# If the processes don't stop within a certain amount of time, eventually SIGKILL is sent. On the [[Nexus]] cluster, this time is currently 5 minutes.		# If the processes don't stop within a certain amount of time, eventually SIGKILL is sent. On the [[Nexus]] cluster, this time is currently 5 minutes.
Line 11:		Line 11:
	==Preemption/Requeue==		==Preemption/Requeue==
	# Slurm controller sends an internal preemption/requeue signal to Slurm node(s) where the job is currently assigned.		# Slurm controller sends an internal preemption/requeue signal to Slurm node(s) where the job is currently assigned.
	# Slurm node(s) send SIGCONT and SIGTERM around the same time, and the following fields of note are set in the output of <tt>scontrol --json show job $SLURM_JOBID</tt>:		# Slurm node(s) send SIGCONT and SIGTERM around the same time, and the following fields of note are set in the output of <code>scontrol --json show job $SLURM_JOBID</code>:
	#* <tt>job_state = ['PENDING', 'COMPLETING']</tt>		#* <tt>job_state = ['PENDING', 'COMPLETING']</tt>
	#* <tt>restart_cnt += 1</tt>		#* <tt>restart_cnt += 1</tt>
Line 21:		Line 21:

	==On job cancellation/preemption==		==On job cancellation/preemption==
	You can handle the SIGTERM signal and run <tt>scontrol --json show job $SLURM_JOBID</tt> within your script. Based on the value of <tt>job_state</tt>, your script can take a different codepath depending on if it was cancelled or was preempted.		You can handle the SIGTERM signal and run <code>scontrol --json show job $SLURM_JOBID</code> within your script. Based on the value of <tt>job_state</tt>, your script can take a different codepath depending on if it was cancelled or was preempted.

	The output of <tt>scontrol --json show job $SLURM_JOBID</tt> is equivalent to the Slurm REST API's endpoint "GET /slurm/v0.0.40/job/{job_id}". For more information on how to parse the output of this scontrol command, please refer to Slurm's REST API documentation. [https://slurm.schedmd.com/rest_api.html#slurmV0040GetJob]		The output of <code>scontrol --json show job $SLURM_JOBID</code> is equivalent to the Slurm REST API's endpoint "GET /slurm/v0.0.40/job/{job_id}". For more information on how to parse the output of this scontrol command, please refer to Slurm's REST API documentation. [https://slurm.schedmd.com/rest_api.html#slurmV0040GetJob]

	==On resuming==		==On resuming==
Line 29:		Line 29:

	=Testing=		=Testing=
	You can use the following commands to manually cancel and requeue your jobs (requeueing and preemption are handled similarly). With these tools, you can test your scripts to ensure that they gracefully handle these scenarios.		You can use the following commands to manually cancel and requeue your jobs (requeueing and preemption are handled similarly). With these tools, you can test your scripts to ensure that they gracefully handle these scenarios. Replace <job_id> with the actual job ID of the job you are testing.

	Cancellation: <tt>scancel <~~job id~~></tt>		Cancellation: <code>scancel <job_id></code>

	Preemption: <tt>scontrol requeue <~~job id~~>		Preemption: <code>scontrol requeue <job_id></code>

Mbaney at 17:16, 26 November 2024

2024-11-26T17:16:26Z

← Older revision		Revision as of 17:16, 26 November 2024
Line 16:		Line 16:
	# If the processes don't stop within a certain amount of time, eventually SIGKILL is sent. On the [[Nexus]] cluster, this time is currently 5 minutes.		# If the processes don't stop within a certain amount of time, eventually SIGKILL is sent. On the [[Nexus]] cluster, this time is currently 5 minutes.
	# Once the job is stopped, it enters the PENDING state for two minutes, and then is eligible to be run again.		# Once the job is stopped, it enters the PENDING state for two minutes, and then is eligible to be run again.
	# When the job runs again, an additional environment variable <tt>SLURM_RESTART_COUNT</tt> ~~will be~~ defined, which reports the number of times the job has been preempted/requeued.		# When the job runs again, an additional environment variable <tt>SLURM_RESTART_COUNT</tt> is defined, which reports the number of times the job has been preempted/requeued.

	=Key takeaways=		=Key takeaways=

Mbaney at 17:13, 26 November 2024

2024-11-26T17:13:01Z

Seide: Created page with "If you are submitting to a partition which is eligible for preemption (e.g. scavenger), you are responsible for making sure that your job can be interrupted/restarted gracefully. Here we are documenting some Slurm behaviors that you can use to determine if your job has been cancelled or preempted. You should be able to take different code paths during startup/shutdown of your jobs based on this information. =Flowchart for job cancellation/preemption= ==Cancellation==..."

2024-05-10T15:54:15Z

Created page with "If you are submitting to a partition which is eligible for preemption (e.g. scavenger), you are responsible for making sure that your job can be interrupted/restarted gracefully. Here we are documenting some Slurm behaviors that you can use to determine if your job has been cancelled or preempted. You should be able to take different code paths during startup/shutdown of your jobs based on this information. =Flowchart for job cancellation/preemption= ==Cancellation==..."

New page

If you are submitting to a partition which is eligible for preemption (e.g. scavenger), you are responsible for making sure that your job can be interrupted/restarted gracefully. Here we are documenting some Slurm behaviors that you can use to determine if your job has been cancelled or preempted. You should be able to take different code paths during startup/shutdown of your jobs based on this information.

=Flowchart for job cancellation/preemption=

==Cancellation==

# Slurm controller sends an internal cancel signal to Slurm node(s) where the job is currently assigned.
# Slurm node(s) send SIGCONT and SIGTERM around the same time, and the following fields of note are set in the output of <tt>scontrol --json show job $SLURM_JOBID</tt>:
#* <tt>job_state = ['CANCELLED', 'COMPLETING']</tt>
# If the processes don't stop within a certain amount of time, eventually SIGKILL will be sent.

==Preemption/Requeue==

# Slurm controller sends an internal preemption/requeue signal to Slurm node(s) where the job is currently assigned.
# Slurm node(s) send SIGCONT and SIGTERM around the same time, and the following fields of note are set in the output of <tt>scontrol --json show job $SLURM_JOBID</tt>:
#* <tt>job_state = ['PENDING', 'COMPLETING']</tt>
#* <tt>restart_cnt += 1</tt>
# If the processes don't stop within a certain amount of time, eventually SIGKILL will be sent.
# Once the job is stopped, it enters the PENDING state for two minutes, and then is eligible to be run again.
# When the job runs again, an additional environment variable will be defined, <tt>SLURM_RESTART_COUNT</tt>, which reports the number of times the job has been preempted/requeued.

=Key takeaways=

==On job cancellation/preemption==

You can handle the SIGTERM signal and run <tt>scontrol --json show job $SLURM_JOBID</tt> within your script. Based on the value of <tt>job_state</tt>, your script can take a different codepath depending on if it was cancelled or was preempted.

The output of <tt>scontrol --json show job $SLURM_JOBID</tt> is equivalent to the Slurm REST API's endpoint "GET /slurm/v0.0.40/job/{job_id}". For more information on how to parse the output of this scontrol command, please refer to Slurm's REST API documentation. [https://slurm.schedmd.com/rest_api.html#slurmV0040GetJob]

==On resuming==

If your job needs to behave differently based on whether or not it was previously preempted, you can check if the environment variable <tt>SLURM_RESTART_COUNT</tt> is defined.

=Testing=

You can use the following commands to manually cancel and requeue your jobs (requeueing and preemption are handled similarly). With these tools, you can test your scripts to ensure that they gracefully handle these scenarios.

Cancellation: <tt>scancel <job id></tt>

Preemption: <tt>scontrol requeue <job id>

← Older revision		Revision as of 17:13, 26 November 2024
Line 1:		Line 1:
	If you are submitting to a partition which is eligible for preemption (e.g. scavenger), you are responsible for making sure that your job can be interrupted/restarted gracefully. ~~Here we are documenting~~ some ~~Slurm behaviors~~ that you can use to determine if your job has been cancelled or preempted. You should be able to take different code paths during startup/shutdown of your jobs based on this information.		If you are submitting to a partition which is eligible for preemption (e.g., scavenger), you are responsible for making sure that your job can be interrupted/restarted gracefully. Below is some information that you can use to determine if your job has been cancelled or preempted. You should be able to take different code paths during startup/shutdown of your jobs based on this information.

	=Flowchart for job cancellation/preemption=		=Flowchart for job cancellation/preemption=

	==Cancellation==		==Cancellation==

	# Slurm controller sends an internal cancel signal to Slurm node(s) where the job is currently assigned.		# Slurm controller sends an internal cancel signal to Slurm node(s) where the job is currently assigned.
	# Slurm node(s) send SIGCONT and SIGTERM around the same time, and the following fields of note are set in the output of <tt>scontrol --json show job $SLURM_JOBID</tt>:		# Slurm node(s) send SIGCONT and SIGTERM around the same time, and the following fields of note are set in the output of <tt>scontrol --json show job $SLURM_JOBID</tt>:
	#* <tt>job_state = ['CANCELLED', 'COMPLETING']</tt>		#* <tt>job_state = ['CANCELLED', 'COMPLETING']</tt>
	# If the processes don't stop within a certain amount of time, eventually SIGKILL ~~will be~~ sent.		# If the processes don't stop within a certain amount of time, eventually SIGKILL is sent. On the [[Nexus]] cluster, this time is currently 5 minutes.

	==Preemption/Requeue==		==Preemption/Requeue==

	# Slurm controller sends an internal preemption/requeue signal to Slurm node(s) where the job is currently assigned.		# Slurm controller sends an internal preemption/requeue signal to Slurm node(s) where the job is currently assigned.
	# Slurm node(s) send SIGCONT and SIGTERM around the same time, and the following fields of note are set in the output of <tt>scontrol --json show job $SLURM_JOBID</tt>:		# Slurm node(s) send SIGCONT and SIGTERM around the same time, and the following fields of note are set in the output of <tt>scontrol --json show job $SLURM_JOBID</tt>:
	#* <tt>job_state = ['PENDING', 'COMPLETING']</tt>		#* <tt>job_state = ['PENDING', 'COMPLETING']</tt>
	#* <tt>restart_cnt += 1</tt>		#* <tt>restart_cnt += 1</tt>
	# If the processes don't stop within a certain amount of time, eventually SIGKILL ~~will be~~ sent.		# If the processes don't stop within a certain amount of time, eventually SIGKILL is sent. On the [[Nexus]] cluster, this time is currently 5 minutes.
	# Once the job is stopped, it enters the PENDING state for two minutes, and then is eligible to be run again.		# Once the job is stopped, it enters the PENDING state for two minutes, and then is eligible to be run again.
	# When the job runs again, an additional environment variable ~~will be defined,~~ <tt>SLURM_RESTART_COUNT</tt>, which reports the number of times the job has been preempted/requeued.		# When the job runs again, an additional environment variable <tt>SLURM_RESTART_COUNT</tt> will be defined, which reports the number of times the job has been preempted/requeued.

	=Key takeaways=		=Key takeaways=

	==On job cancellation/preemption==		==On job cancellation/preemption==

	You can handle the SIGTERM signal and run <tt>scontrol --json show job $SLURM_JOBID</tt> within your script. Based on the value of <tt>job_state</tt>, your script can take a different codepath depending on if it was cancelled or was preempted.		You can handle the SIGTERM signal and run <tt>scontrol --json show job $SLURM_JOBID</tt> within your script. Based on the value of <tt>job_state</tt>, your script can take a different codepath depending on if it was cancelled or was preempted.

Line 29:		Line 26:

	==On resuming==		==On resuming==

	If your job needs to behave differently based on whether or not it was previously preempted, you can check if the environment variable <tt>SLURM_RESTART_COUNT</tt> is defined.		If your job needs to behave differently based on whether or not it was previously preempted, you can check if the environment variable <tt>SLURM_RESTART_COUNT</tt> is defined.

	=Testing=		=Testing=

	You can use the following commands to manually cancel and requeue your jobs (requeueing and preemption are handled similarly). With these tools, you can test your scripts to ensure that they gracefully handle these scenarios.		You can use the following commands to manually cancel and requeue your jobs (requeueing and preemption are handled similarly). With these tools, you can test your scripts to ensure that they gracefully handle these scenarios.