SLURM

JOB SCHEDULER
WHAT IS SLURM?
Slurm is an open source cluster management and job scheduling system for Linux clusters.

1 Keeps track of available resources on the cluster

2 Collects users resources requests for jobs

A B C D

3 Assign priorities to jobs

4 Run jobs on assigned compute nodes

A D
B C

www.slurm.schedmd.com
PARTITIONS

Compute nodes are grouped into logical sets called partitions

depending on their hardware characteristics or function:

production
Standard CPU nodes
(default)
Standard CPU nodes for debug
debug
(fast allocation times)
maxwell Nodes with Nvidia Maxwell GPUs
pascal Nodes with Nvidia Pascal GPUs
mic Nodes with Intel Xeon Phi cards

Ask ACCRE if you would like to get

access to specific partitions.
JOB EXECUTION WORKFLOW

1. DETERMINE THE RESOURCES NECESSARY FOR THE SPECIFIC JOB

2. CREATE A BATCH JOB SCRIPT

3. SUBMIT THE JOB TO THE SCHEDULER

4. CHECK JOB STATUS

5. RETRIEVE JOB INFORMATION

i
DETERMINE RESOURCES FOR JOB

NUMBER OF CPU CORES

• From 1 to the maximum allowed for your group’s account.
• Default is one CPU core. Slurm will immediately kill your
job if your process exceeds the
requested amount of
AMOUNT OF MEMORY GB per node # nodes
resources.
• Up to 246 GB per node. 20 90
• Default is 1 GB per core. 44 45
58 55
120 344
246 44

TIME Slightly overestimate the

requested job resources, but
• Job duration on production can be set up to 14 days.
do not greatly overestimate to
• Default is 15 minutes.
i • DEBUG QUEUE: max 30 minutes avoid unnecessary long wait
times.
DETERMINE RESOURCES FOR JOB - BACKFILL
Backfill scheduling will start lower priority jobs if doing so does not delay the expected start time of
any higher priority job.

1
2 John
3 12 CPU cores
4 1 week
5
6 Mark
7 2 CPU cores
5 hours
8
9
10 Lucy
11 1 CPU core
7 hours
12

i 1 2 3 4 5 6 7 8 hours
DETERMINE RESOURCES FOR JOB - OPTIMIZATION

How to define the right amount of resources for my job?

Select a representative job

Optimized
Overestimate resources requested
resources

Run test job

Check actual resources utilization

Lower
Optimize resources request More
queue
research
wait time

i Run production jobs

CREATE A BATCH JOB SCRIPT
A batch job consists of a sequence of commands listed in a file with the purpose of being executed by
the OS as a single instruction.

SHEBANG
myjob.slurm
• Specify the script interpreter (Bash)
• Must be the first line! #!/bin/bash

#SBATCH --nodes=1 # Nodes

SLURM DIRECTIVES #SBATCH --ntasks=1
• Start with “#SBATCH”: #SBATCH --mem=1G
Parsed by Slurm but ignored by Bash.
• Can be separated by spaces. # Max job duration
• Comments between and after #SBATCH --time=1-06:30:00
directives are allowed. #SBATCH --job-name=myjob
#SBATCH --output=myjob.out
• Must be before actual commands!
# Just a comment
SCRIPT COMMANDS setpkgs -a python

i • Commands you want to execute on the

compute nodes.
./myprogram
CREATE A BATCH JOB SCRIPT - THE ESSENTIALS

--nodes=N
• Request N nodes to be allocated. (Default: N=1)

--ntasks=N
• Request N tasks to be allocated. (Default: N=1)
• Unless otherwise specified, one task maps to one CPU core.

--mem=NG
• Request N gigabytes of memory per node. (Default: N=1)

--time=d-hh:mm:ss
• Request d days, hh hours, mm minutes and ss seconds. (Default: 00:15:00)

--job-name=<string>
• Specify a name for the job allocation. (Default: batch file name)

--output=<file_name>

i •
•
Write the batch script’s standard output in the specified file.
If not specified the output will be saved in the file: slurm-<jobid>.out
CREATE A BATCH JOB SCRIPT - EMAIL NOTIFICATION

--mail-user=<address>
• Send email to address.
• It accepts multiple comma separated addresses.

--mail-type=<event>
• Define the events for which you want to be notified:

BEGIN Job begins

END Job ends
FAIL Job fails
ALL BEGIN+END+FAIL
TIME_LIMIT_50 Elapsed time reaches 50% of allocated time
TIME_LIMIT_80 Elapsed time reaches 80% of allocated time
TIME_LIMIT_90 Elapsed time reaches 90% of allocated time

i
SUBMIT JOB TO THE SCHEDULER

sbatch batch_file
• Submit batch_file to Slurm.
• If successful, it returns the job ID of the submitted job.

SUBMISSION PRIORITY WAIT ALLOCATION

AND
Job is added to A priority value Job waits in queue until: EXECUTION
the queue is assigned to the
1. Resources are
job.
available
2. There are no jobs
with higher priority
in queue

scancel jobid
• Cancel the job corresponding to the
i How do I remove a job from the queue?
given jobid from the queue.
SUBMIT JOB TO THE SCHEDULER

How is my job’s priority calculated?

FAIRSHARE AGE

Prioritizes jobs belonging to under-

FAIRSHARE AGE The longer the job waits in
queue, the larger its age
serviced accounts.
factor becomes.
It reflects:

1. The share of resources

contributed by your research
group.
JOB SIZE
JOB SIZE
2. The historical amount of
computing resources consumed Jobs requesting more
by your account. CPUs are favored.

i PRIORITY
CHECK JOB STATUS

squeue -u vunetid
• Show the queued jobs for user vunetid.

NODELIST (REASON)
• For running jobs shows the allocated nodes.
• For pending jobs shows the wait reason:
STATUS
Priority Other jobs in queue have higher priority.
R = Running
Resources Insufficient resources available on the cluster.
PD = Pending
Reached maximum number of allocated CPUs by
CA = Cancelled AssocGrpCpuLimit
all jobs belonging to the user’s account.
Reached maximum amount of allocated memory
AssocGrpMemLimit
by all jobs belonging to the user’s account.

i AssocGrpTimeLimit
Reached maximum amount of allocated time by
all jobs belonging to the user’s account.
RETRIEVE JOB INFORMATION

rtracejob jobid
• Print requested and utilized resources (and more) for the given jobid.

The used memory may not be

an exact value.
Take it with reservations.

i
JOB ARRAYS

Submit multiple similar jobs with a single job batch script.

To each job within the array is assigned a unique task ID. All jobs in a job array must
have the same resource
requirements.
--array=start-end[:step][%limit]

• Define task ID interval from start to end as unsigned integer values.

• The step between successive values can be set after colon sign.
• Set the limit to the number of simultaneously running jobs with “%”. The maximum array size is
• Individual task IDs can be specified as a comma separated values list. 30,000 jobs.

--array=0-7 0, 1, 2, 3, 4, 5, 6, 7

--array=1-13:3 1, 4, 7, 10, 13 Significantly shorter

submission times than
--array=2,3,6,15 2, 3, 6, 15 submitting jobs
individually.
JOB ARRAYS

How to select different input/output for each job in the array?

Use Slurm environment variable: SLURM_ARRAY_TASK_ID

The task ID for the specific job in the array.

EXECUTED COMMAND OUTPUT FILE

my_program file_1 job_1234567_task_1.out

#!/bin/bash
#SBATCH --ntasks=1
#SBATCH --time=00:05:00 my_program file_2 job_1234567_task_2.out
#SBATCH --job-name=job_array
#SBATCH --array=1-4
#SBATCH --output=job_%A_task_%a.out my_program file_3 job_1234567_task_3.out

my_program file_${SLURM_ARRAY_TASK_ID}
my_program file_4 job_1234567_task_4.out
JOB ARRAYS

What if my input files do not have a numerical index?

#!/bin/bash
#SBATCH
…
1 2 3 4
myfile=$( ls DataDir | awk -v line=${SLURM_ARRAY_TASK_ID} ‘{if (NR==line) print $0}’ )

5 my_program ${myfile}

1 Get the list of files names in the data directory in alphabetical order
2 Send the list to awk
3 Pass the value of the bash variable SLURM_ARRAY_TASK_ID to the awk variable “line”
4 Print only the NRth line in the list of files names for which NR corresponds to the job task ID
5 Pass the file name in the myfile variable to the main program
MULTITHREADED JOBS

POSIX THREADS

• Single task with multiple concurrent execution threads.

• Each thread uses a single CPU core.
• All threads share the same allocated memory.

Single node only!

1 node 1 node
1 task 2 tasks
8 CPUs per task 4 CPUs per task
MULTITHREADED JOBS

--cpus-per-task=N
• Request N CPU cores to be allocated for each task.

With OpenMP in your batch script don’t forget to set:

export OMP_NUM_THREADS = $SLURM_CPUS_PER_TASK
DISTRIBUTED MEMORY JOBS

MESSAGE PASSING INTERFACE (MPI)

• Multiple tasks with private memory allocations.

• Tasks exchange data through communications.
• Tasks can reside on the same node or on multiple nodes.

Single or multiple nodes

2 nodes
8 tasks per node
1 CPU per task
DISTRIBUTED MEMORY JOBS

--nodes=N
• Request N nodes to be allocated.

--tasks-per-node=N
• Request N tasks per node.
• Unless otherwise specified, one task maps to one CPU core.

In the batch script, run the MPI program with:

srun ./program_name

• Run MPI program called program_name.

Do not use mpirun or mpiexec! For OpenMPI only, add the

srun will use the correct following flag to srun:
launcher for the MPI library you
selected via setpkgs. srun --mpi=pmi2 ./program_name
INTERACTIVE SHELL JOB

salloc options
• Obtain job allocation with shell access.
• Accepts all the same options previously seen for sbatch.

Gateway

Compute node

Recommended for debugging and

benchmarking sessions.
TROUBLESHOOTING

Why is my job still pending? SlurmActive -m mem

• Show the overall cluster utilization.
• Count as available cores only the ones with at least mem
amount of memory (in GB). Default: 1GB

Check overall cluster utilization

Check you account’s resources use

Check your account limits

TROUBLESHOOTING

Why is my job still pending?

qSummary -g group
• Show the total number of jobs and CPU cores allocated
or waiting for allocation for the selected group.
Check overall cluster utilization

Check you account’s resources use

Check your account limits

TROUBLESHOOTING

Why is my job still pending?

showLimits -g group

• Show the cluster resources limits for a specific group.

Check overall cluster utilization

Check you account’s resources use

Check your account limits

Users in the same group share

the same amount of resources.
TROUBLESHOOTING

Why did my job fail?

Check with rtracejob:

1 A non-zero exit code means your

Check the job’s output
file for error messages. 2 application failed.

Check your Slurm batch job script for syntax

3 or logic errors.
www.accre.vanderbilt.edu/slurm
NEED MORE HELP?

Check our Frequently Asked Question webpage:

www.accre.vanderbilt.edu/faq

Submit a ticket from the helpdesk:

www.accre.vanderbilt.edu/help

Open a ticket to request an appointment

with an ACCRE specialist.

DO NOT submit tickets in “Rush cluster”!

Rush tickets are for cluster-wide issues only.