SLURM Job Arrays

Introduction and Submitting Arrays

To submit a number of identical jobs without having drive the submission with an external script use the SLURM's feature of array jobs. You can learn how to submit them at Submitting Array Jobs.

Note: There is a maximum limit of 3000 jobs per user on HiPerGator.

Using the array ID Index

SLURM will provide a $SLURM_ARRAY_TASK_ID variable to each task. It can be used inside the job script to handle input and output files for that task.

For instance, for a 100-task job array the input files can be named seq_1.fa, seq_2.fa and so on through seq_100.fa. In a job script for a blastn job they can be referenced as blastn -query seq_${SLURM_ARRAY_TASK_ID}.fa. The output files can be handled in the same way.

One common application of array jobs is to run many input files. While it is easy if the files are numbered as in the example above, this is not needed. If for example you have a folder of 100 files that end in .txt, you can use the following approach to get the name of the file for each task automatically:

file=$(ls *.txt | sed -n ${SLURM_ARRAY_TASK_ID}p)
myscript -in $file

If, alternatively, you use an input file (e.g. 'input.list') with a list of samples/datasets (one per line) to process you can pick an item from the list as follows:

SAMPLE_LIST=($(<input.list))
SAMPLE=${SAMPLE_LIST[${SLURM_ARRAY_TASK_ID}]}

Array ID use in Scripts

When running custom code written in Python or R use the respective module that allows you to read environment variables to read the SLURM array task id of the current job and use it to perform analysis on the correct input file or data column/row. For example:

Python

import sys
jobid = sys.getenv('SLURM_ARRAY_TASK_ID')

R

task_id <- Sys.getenv("SLURM_ARRAY_TASK_ID")

Extended Example

This shell portion of a SLURM job script sets input and output directories as variables. Then, it sets a RUN variable to the SLURM task id in a job or to a value of '1' if you are running a quick test before submitting a job. The RUN value is used as an index value to pick a full path to a dataset from the input directory, determine the file name, and remove the specified extension to create a value that may represent a sample name to be used in forming an output path. Finally, the automated command is printed to stdout to be recorded in the job log and executed as a command.

Expand to see an explicit example of using SLURM Arrays and automating handling of input and output datasets.

INPUT_DIR=/blue/group/user/project/run/input
OUTPUT_DIR=/blue/group/user/project/run/output

RUN=${SLURM_ARRAY_TASK_ID:-1}

echo "Run: ${RUN}"

module load plink/1.90b3.39

INPUT_PATH=$(ls ${INPUT_DIR}/*.vcf.gz | sed -n ${RUN}p)
INPUT_FILE=$(basename ${INPUT_PATH})
SAMPLE=$(basename ${INPUT_FILE} .vcf.gz)

read -d '' cmd << EOF
plink \
--vcf ${INPUT_PATH} \
--out ${OUTPUT_DIR}/${SAMPLE} \
--allow-no-sex \
--keep-allele-order
EOF
echo ${cmd}

eval ${cmd}

Running many short tasks

While SLURM array jobs make it easy to run many similar tasks, if each task is short (seconds or even a few minutes), array jobs quickly bog down the scheduler and more time is spent managing jobs than actually doing any work for you. This also negatively impacts other users.

If you have hundreds or thousands of tasks, it is unlikely that a simple array job is the best solution. That does not mean that array jobs are not helpful in these cases, but that a little more thought needs to go into them for efficient use of the resources.

[10 min, 16sec] Watch the video discussing some of the issues and walking through the details of the example script below.

As an example let's imagine I have 5,000 runs of a program to do, with each run taking about 30 seconds to complete. Rather than running an array job with 5,000 tasks, it would be much more efficient to run 5 tasks where each completes 1,000 runs.

Expand to view a sample script to accomplish this by combining array jobs with bash loops.

#!/bin/sh
#SBATCH --job-name=mega_array   # Job name
#SBATCH --mail-type=ALL         # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=gatorlink@ufl.edu # Where to send mail	
#SBATCH --nodes=1                   # Use one node
#SBATCH --ntasks=1                  # Run a single task
#SBATCH --mem-per-cpu=1gb           # Memory per processor
#SBATCH --time=00:10:00             # Time limit hrs:min:sec
#SBATCH --output=array_%A-%a.out    # Standard output and error log
#SBATCH --array=1-5                 # Array range
# This is an example script that combines array tasks with
# bash loops to process many short runs. Array jobs are convenient
# for running lots of tasks, but if each task is short, they
# quickly become inefficient, taking more time to schedule than
# they spend doing any work and bogging down the scheduler for
# all users. 
pwd; hostname; date

#Set the number of runs that each SLURM task should do
PER_TASK=1000

# Calculate the starting and ending values for this task based
# on the SLURM task and the number of runs per task.
START_NUM=$(( ($SLURM_ARRAY_TASK_ID - 1) * $PER_TASK + 1 ))
END_NUM=$(( $SLURM_ARRAY_TASK_ID * $PER_TASK ))

# Print the task and run range
echo This is task $SLURM_ARRAY_TASK_ID, which will do runs $START_NUM to $END_NUM

# Run the loop of runs for this task.
for (( run=$START_NUM; run<=END_NUM; run++ )); do
  echo This is SLURM task $SLURM_ARRAY_TASK_ID, run number $run
  #Do your stuff here
done

date

Deleting job arrays and tasks

To delete all of the tasks of an array job, use scancel with the job ID:

scancel 292441

To delete a single task, add the task ID:

scancel 292441_5

Controlling Job emails

By default in SLURM, the emails for events BEGIN, END and FAIL apply to the job array as a whole rather than individual tasks. So:

#SBATCH --mail-type=BEGIN,END,FAIL

would result in one email per job, not per task. If you want per task emails, specify:

 #SBATCH --mail-type=BEGIN,END,FAIL,ARRAY_TASKS

which will send emails for each task in the array.