SLURM/JobStatus - Revision history

Mbaney: /* Job Codes */

2025-10-20T17:03:55Z

Job Codes

← Older revision		Revision as of 17:03, 20 October 2025
Line 95:		Line 95:
	[username@nexusclip00 ~]$ squeue -u username		[username@nexusclip00 ~]$ squeue -u username
	JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)		JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
	1 tron bash username PD 0:00 1 (~~AssocGrpGRES~~)		1 tron bash username PD 0:00 1 (AssocGrpBilling)
	2 tron bash username PD 0:00 1 (Resources)		2 tron bash username PD 0:00 1 (Resources)
	3 tron bash username PD 0:00 1 (Priority)		3 tron bash username PD 0:00 1 (Priority)

Mbaney: /* Job Codes */

2024-08-22T19:12:56Z

Job Codes

← Older revision		Revision as of 19:12, 22 August 2024
Line 109:		Line 109:
	* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.		* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.

	SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/~~resource_limits~~.html~~#reasons~~ here].		SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/job_reason_codes.html here].

Mbaney: /* Job Codes */

2024-08-22T19:12:21Z

Job Codes

← Older revision		Revision as of 19:12, 22 August 2024
Line 105:		Line 105:
	* <code>Resources</code> - The cluster does not currently have the resources to fit your job in your selected partition.		* <code>Resources</code> - The cluster does not currently have the resources to fit your job in your selected partition.
	* <code>Priority</code> - The cluster has reserved resources for higher [[SLURM/Priority \| priority]] jobs in your selected partition.		* <code>Priority</code> - The cluster has reserved resources for higher [[SLURM/Priority \| priority]] jobs in your selected partition.
	* <code>QOSMaxPerUser</code> or <code>QOSMaxPerUserLimit</code> - The quality of service (QoS) your job is ~~running in~~ has some limit per user (CPU, mem, GRES, etc.). Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the limit(s) and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS to see the resources they are currently consuming.		* <code>QOSMaxPerUser</code> or <code>QOSMaxPerUserLimit</code> - The quality of service (QoS) your job is requesting to use has some limit per user (CPU, mem, GRES, etc.). Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the limit(s) and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS to see the resources they are currently consuming.
	* <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on the overall billing amount available in total for the account. Use <code>sacctmgr show assoc account=<accountname> where user=</code> to identify the limit, replacing <tt><accountname></tt> with the account you are submitting your job with. You can see all jobs running under the account and their billing values by running <code>squeue -A <accountname> -O "JobId:.18 ,Partition:.9 ,Name:.8 ,UserName:.8 ,StateCompact:.2 ,TimeUsed:.10 ,NumNodes:.6 ,ReasonList:45 ,tres-alloc:80"</code>. The billing value will be part of the <tt>tres-alloc</tt> string for each job.		* <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on the overall billing amount available in total for the account. Use <code>sacctmgr show assoc account=<accountname> where user=</code> to identify the limit, replacing <tt><accountname></tt> with the account you are submitting your job with. You can see all jobs running under the account and their billing values by running <code>squeue -A <accountname> -O "JobId:.18 ,Partition:.9 ,Name:.8 ,UserName:.8 ,StateCompact:.2 ,TimeUsed:.10 ,NumNodes:.6 ,ReasonList:45 ,tres-alloc:80"</code>. The billing value will be part of the <tt>tres-alloc</tt> string for each job.
	* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.		* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.

	SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].		SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].

Mbaney: /* Job Codes */

2024-08-22T19:09:45Z

Job Codes

← Older revision		Revision as of 19:09, 22 August 2024
Line 106:		Line 106:
	* <code>Priority</code> - The cluster has reserved resources for higher [[SLURM/Priority \| priority]] jobs in your selected partition.		* <code>Priority</code> - The cluster has reserved resources for higher [[SLURM/Priority \| priority]] jobs in your selected partition.
	* <code>QOSMaxPerUser</code> or <code>QOSMaxPerUserLimit</code> - The quality of service (QoS) your job is running in has some limit per user (CPU, mem, GRES, etc.). Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the limit(s) and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS to see the resources they are currently consuming.		* <code>QOSMaxPerUser</code> or <code>QOSMaxPerUserLimit</code> - The quality of service (QoS) your job is running in has some limit per user (CPU, mem, GRES, etc.). Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the limit(s) and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS to see the resources they are currently consuming.
	* <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on the overall billing amount available in total for the account. Use <code>sacctmgr show assoc account=<accountname> where user=</code> to identify the limit, replacing <tt><accountname></tt> with the account you are submitting your job with. You can see all jobs running under the account and their billing values by running <code>squeue -A <accountname> -O "JobId:.18 ,Partition:.9 ,Name:.8 ,UserName:.8 ,StateCompact:.2 ,TimeUsed:.10 ,NumNodes:.6 ,ReasonList:45 ,tres-alloc:80"</code>.		* <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on the overall billing amount available in total for the account. Use <code>sacctmgr show assoc account=<accountname> where user=</code> to identify the limit, replacing <tt><accountname></tt> with the account you are submitting your job with. You can see all jobs running under the account and their billing values by running <code>squeue -A <accountname> -O "JobId:.18 ,Partition:.9 ,Name:.8 ,UserName:.8 ,StateCompact:.2 ,TimeUsed:.10 ,NumNodes:.6 ,ReasonList:45 ,tres-alloc:80"</code>. The billing value will be part of the <tt>tres-alloc</tt> string for each job.
	* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.		* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.

	SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].		SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].

Mbaney: /* Job Codes */

2024-08-22T19:08:35Z

Job Codes

← Older revision		Revision as of 19:08, 22 August 2024
Line 105:		Line 105:
	* <code>Resources</code> - The cluster does not currently have the resources to fit your job in your selected partition.		* <code>Resources</code> - The cluster does not currently have the resources to fit your job in your selected partition.
	* <code>Priority</code> - The cluster has reserved resources for higher [[SLURM/Priority \| priority]] jobs in your selected partition.		* <code>Priority</code> - The cluster has reserved resources for higher [[SLURM/Priority \| priority]] jobs in your selected partition.
	* <code>~~QOSMaxGRESPerUser~~</code> - The quality of service (QoS) your job is running in has a limit ~~of resources~~ per user. Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the ~~limits~~ and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS.		* <code>QOSMaxPerUser</code> or <code>QOSMaxPerUserLimit</code> - The quality of service (QoS) your job is running in has some limit per user (CPU, mem, GRES, etc.). Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the limit(s) and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS to see the resources they are currently consuming.
	* <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on the overall billing amount available in total for the account. Use <code>sacctmgr show assoc account=<accountname> where user=</code> to identify the limit, replacing <tt><accountname></tt> with the account you are submitting your job with. You can see all jobs running under the account and their billing values by running <code>squeue -A <accountname> -O "JobId:.18 ,Partition:.9 ,Name:.8 ,UserName:.8 ,StateCompact:.2 ,TimeUsed:.10 ,NumNodes:.6 ,ReasonList:45 ,tres-alloc:80"</code>.		* <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on the overall billing amount available in total for the account. Use <code>sacctmgr show assoc account=<accountname> where user=</code> to identify the limit, replacing <tt><accountname></tt> with the account you are submitting your job with. You can see all jobs running under the account and their billing values by running <code>squeue -A <accountname> -O "JobId:.18 ,Partition:.9 ,Name:.8 ,UserName:.8 ,StateCompact:.2 ,TimeUsed:.10 ,NumNodes:.6 ,ReasonList:45 ,tres-alloc:80"</code>.
	* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.		* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.

	SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].		SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].

Mbaney: /* Job Codes */

2024-08-22T18:59:01Z

Job Codes

← Older revision		Revision as of 18:59, 22 August 2024
Line 106:		Line 106:
	* <code>Priority</code> - The cluster has reserved resources for higher [[SLURM/Priority \| priority]] jobs in your selected partition.		* <code>Priority</code> - The cluster has reserved resources for higher [[SLURM/Priority \| priority]] jobs in your selected partition.
	* <code>QOSMaxGRESPerUser</code> - The quality of service (QoS) your job is running in has a limit of resources per user. Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the limits and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS.		* <code>QOSMaxGRESPerUser</code> - The quality of service (QoS) your job is running in has a limit of resources per user. Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the limits and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS.
	* <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on the overall billing amount available in total for the account. Use <code>sacctmgr show assoc account=<accountname></code> to identify the limit, replacing <tt><accountname></tt> with the account you are submitting your job with. You can see all jobs running under the account and their billing values by running <code>squeue -A <accountname> -O "JobId:.18 ,Partition:.9 ,Name:.8 ,UserName:.8 ,StateCompact:.2 ,TimeUsed:.10 ,NumNodes:.6 ,ReasonList:45 ,tres-alloc:80"</code>.		* <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on the overall billing amount available in total for the account. Use <code>sacctmgr show assoc account=<accountname> where user=</code> to identify the limit, replacing <tt><accountname></tt> with the account you are submitting your job with. You can see all jobs running under the account and their billing values by running <code>squeue -A <accountname> -O "JobId:.18 ,Partition:.9 ,Name:.8 ,UserName:.8 ,StateCompact:.2 ,TimeUsed:.10 ,NumNodes:.6 ,ReasonList:45 ,tres-alloc:80"</code>.
	* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.		* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.

	SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].		SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].

Mbaney: /* Job Codes */

2024-08-22T18:58:08Z

Job Codes

← Older revision		Revision as of 18:58, 22 August 2024
Line 106:		Line 106:
	* <code>Priority</code> - The cluster has reserved resources for higher [[SLURM/Priority \| priority]] jobs in your selected partition.		* <code>Priority</code> - The cluster has reserved resources for higher [[SLURM/Priority \| priority]] jobs in your selected partition.
	* <code>QOSMaxGRESPerUser</code> - The quality of service (QoS) your job is running in has a limit of resources per user. Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the limits and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS.		* <code>QOSMaxGRESPerUser</code> - The quality of service (QoS) your job is running in has a limit of resources per user. Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the limits and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS.
	* <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on the overall billing amount available in total for the account. Use <code>sacctmgr show assoc account=<accountname></code> to identify the limit, replacing <tt><accountname></tt> with the account you are submitting your job with. You can see all jobs running under the account by running <code>squeue -A <accountname>~~</code> and then find out more information on each job by <code>scontrol show job <jobid>~~</code>.		* <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on the overall billing amount available in total for the account. Use <code>sacctmgr show assoc account=<accountname></code> to identify the limit, replacing <tt><accountname></tt> with the account you are submitting your job with. You can see all jobs running under the account and their billing values by running <code>squeue -A <accountname> -O "JobId:.18 ,Partition:.9 ,Name:.8 ,UserName:.8 ,StateCompact:.2 ,TimeUsed:.10 ,NumNodes:.6 ,ReasonList:45 ,tres-alloc:80"</code>.
	* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.		* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.

	SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].		SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].

Mbaney at 18:39, 22 August 2024

2024-08-22T18:39:55Z

← Older revision		Revision as of 18:39, 22 August 2024
Line 106:		Line 106:
	* <code>Priority</code> - The cluster has reserved resources for higher [[SLURM/Priority \| priority]] jobs in your selected partition.		* <code>Priority</code> - The cluster has reserved resources for higher [[SLURM/Priority \| priority]] jobs in your selected partition.
	* <code>QOSMaxGRESPerUser</code> - The quality of service (QoS) your job is running in has a limit of resources per user. Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the limits and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS.		* <code>QOSMaxGRESPerUser</code> - The quality of service (QoS) your job is running in has a limit of resources per user. Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the limits and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS.
	* ~~<code>AssocGrpGRES</code> or~~ <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on ~~either a specific GRES or~~ the overall billing amount ~~(respectively)~~ available in total for the account. Use <code>sacctmgr show assoc account=<accountname></code> to identify the limit, replacing <tt><accountname></tt> with the account you are submitting your job with. You can see all jobs running under the account by running <code>squeue -A <accountname></code> and then find out more information on each job by <code>scontrol show job <jobid></code>.		* <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on the overall billing amount available in total for the account. Use <code>sacctmgr show assoc account=<accountname></code> to identify the limit, replacing <tt><accountname></tt> with the account you are submitting your job with. You can see all jobs running under the account by running <code>squeue -A <accountname></code> and then find out more information on each job by <code>scontrol show job <jobid></code>.
	* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.		* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.

	SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].		SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].

Mbaney: /* Job Codes */

2024-04-02T20:44:32Z

Job Codes

← Older revision		Revision as of 20:44, 2 April 2024
Line 106:		Line 106:
	* <code>Priority</code> - The cluster has reserved resources for higher [[SLURM/Priority \| priority]] jobs in your selected partition.		* <code>Priority</code> - The cluster has reserved resources for higher [[SLURM/Priority \| priority]] jobs in your selected partition.
	* <code>QOSMaxGRESPerUser</code> - The quality of service (QoS) your job is running in has a limit of resources per user. Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the limits and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS.		* <code>QOSMaxGRESPerUser</code> - The quality of service (QoS) your job is running in has a limit of resources per user. Use <code>show_qos</code> and <code>show_partition_qos</code> to identify the limits and then use <code>scontrol show job <jobid></code> for each of your jobs running in that QoS.
	* <code>AssocGrpGRES</code> or <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on either a specific GRES or the overall billing amount (respectively) available in total for the account. Use <code>sacctmgr show assoc account=<~~account_name~~></code> to identify the limit. You can see all jobs running under the account by running <code>squeue -A ~~account_name~~</code> and then find out more information on each job by <code>scontrol show job <jobid></code>.		* <code>AssocGrpGRES</code> or <code>AssocGrpBilling</code> - The SLURM account you are using has a limit on either a specific GRES or the overall billing amount (respectively) available in total for the account. Use <code>sacctmgr show assoc account=<accountname></code> to identify the limit, replacing <tt><accountname></tt> with the account you are submitting your job with. You can see all jobs running under the account by running <code>squeue -A <accountname></code> and then find out more information on each job by <code>scontrol show job <jobid></code>.
	* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.		* <code>ReqNodeNotAvail</code> - None of the nodes that could run your job (based on requested partition/resources) currently have the resources to fit your job. Alternatively, if you also see <code>Reserved for maintenance</code>, there is a reservation in place (often for a [[MonthlyMaintenanceWindow \| maintenance window]]). You can see the current reservations by running <code>scontrol show reservation</code>. Often the culprit is that you have requested a TimeLimit that will conflict with the reservation. You can either lower your TimeLimit such that the job will complete before the reservation begins, or leave your job to wait until the reservation completes.

	SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].		SLURM's full list of reasons/explanations can be found [https://slurm.schedmd.com/resource_limits.html#reasons here].

Mbaney at 20:43, 2 April 2024

2024-04-02T20:43:04Z

← Older revision		Revision as of 20:43, 2 April 2024
Line 10:		Line 10:
	Examples:		Examples:
	<pre>		<pre>
	username@nexusclip00:squeue -u username		[username@nexusclip00 ~]$ squeue -u username
	JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)		JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
	162 tron helloWor username R 0:03 2 tron[00-01]		162 tron helloWor username R 0:03 2 tron[00-01]
Line 16:		Line 16:

	<pre>		<pre>
	username@nexusclip00:squeue --start -u username		[username@nexusclip00 ~]$ squeue --start -u username
	JOBID PARTITION NAME USER ST START_TIME NODES SCHEDNODES NODELIST(REASON)		JOBID PARTITION NAME USER ST START_TIME NODES SCHEDNODES NODELIST(REASON)
	163 tron helloWo2 username PD 2020-05-11T18:36:49 1 tron02 (Priority)		163 tron helloWo2 username PD 2020-05-11T18:36:49 1 tron02 (Priority)
Line 22:		Line 22:

	<pre>		<pre>
	username@nexusclip00:squeue -s -u username		[username@nexusclip00 ~]$ squeue -s -u username
	STEPID NAME PARTITION USER TIME NODELIST		STEPID NAME PARTITION USER TIME NODELIST
	162.0 sleep tron username 0:05 tron00		162.0 sleep tron username 0:05 tron00
Line 34:		Line 34:
	</pre>		</pre>
	<pre>		<pre>
	username@nexusclip00: sstat --format JobID,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize 171		[username@nexusclip00 ~]$ sstat --format JobID,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize 171
	JobID NTasks Nodelist MaxRSS MaxVMSize AveRSS AveVMSize		JobID NTasks Nodelist MaxRSS MaxVMSize AveRSS AveVMSize
	------------ -------- -------------------- ---------- ---------- ---------- ----------		------------ -------- -------------------- ---------- ---------- ---------- ----------
	171.0 1 tron00 0 186060K 0 107900K		171.0 1 tron00 0 186060K 0 107900K
	username@nexusclip00: sstat --format JobID,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize 171.1		[username@nexusclip00 ~]$ sstat --format JobID,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize 171.1
	JobID NTasks Nodelist MaxRSS MaxVMSize AveRSS AveVMSize		JobID NTasks Nodelist MaxRSS MaxVMSize AveRSS AveVMSize
	------------ -------- -------------------- ---------- ---------- ---------- ----------		------------ -------- -------------------- ---------- ---------- ---------- ----------
Line 45:		Line 45:
	Note that if you do not have any jobsteps, sstat will return an error.		Note that if you do not have any jobsteps, sstat will return an error.
	<pre>		<pre>
	username@nexusclip00: sstat --format JobID,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize 172		[username@nexusclip00 ~]$ sstat --format JobID,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize 172
	JobID NTasks Nodelist MaxRSS MaxVMSize AveRSS AveVMSize		JobID NTasks Nodelist MaxRSS MaxVMSize AveRSS AveVMSize
	------------ -------- -------------------- ---------- ---------- ---------- ----------		------------ -------- -------------------- ---------- ---------- ---------- ----------
Line 66:		Line 66:
	The sacct command shows metrics from past jobs.		The sacct command shows metrics from past jobs.
	<pre>		<pre>
	username@nexusclip00:sacct		[username@nexusclip00 ~]$ sacct
	JobID JobName Partition Account AllocCPUS State ExitCode		JobID JobName Partition Account AllocCPUS State ExitCode
	------------ ---------- ---------- ---------- ---------- ---------- --------		------------ ---------- ---------- ---------- ---------- ---------- --------
Line 80:		Line 80:
	<pre>sacct --format JobID,jobname,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize,Elapsed -j <$JOBID>.<$JOBSTEP></pre>		<pre>sacct --format JobID,jobname,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize,Elapsed -j <$JOBID>.<$JOBSTEP></pre>
	<pre>		<pre>
	username@nexusclip00:sacct --format JobID,jobname,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize,Elapsed -j 171		[username@nexusclip00 ~]$ sacct --format JobID,jobname,NTasks,nodelist,MaxRSS,MaxVMSize,AveRSS,AveVMSize,Elapsed -j 171
	JobID JobName NTasks NodeList MaxRSS MaxVMSize AveRSS AveVMSize Elapsed		JobID JobName NTasks NodeList MaxRSS MaxVMSize AveRSS AveVMSize Elapsed
	------------ ---------- -------- --------------- ---------- ---------- ---------- ---------- ----------		------------ ---------- -------- --------------- ---------- ---------- ---------- ---------- ----------
Line 93:		Line 93:

	<pre>		<pre>
	# squeue -u ~~testuser~~		[username@nexusclip00 ~]$ squeue -u username
	JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)		JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
	1 tron bash username PD 0:00 1 (AssocGrpGRES)		1 tron bash username PD 0:00 1 (AssocGrpGRES)