Sample SLURM Scripts

HiPerGator 2.0 documentation

Sample SLURM Scripts

Below are a number of sample scripts that can be used as a template for building your own SLURM submission scripts for use on HiPerGator 2.0. These scripts are also located at: /ufrc/data/training/SLURM/, and can be copied from there.

Basic, single-processor job

This script can serve as the template for many single-processor applications. The mem-per-cpu flag can be used to request the appropriate amount of memory for your job. Please make sure to test you application and set this value to a reasonable number based on actual memory use. The %j in the -o (can also use --output) line tells SLURM to substitute the job ID in the name of the output file. You can also add a -e or --error with an error file name to separate output and error logs.

Download the [{{#fileLink: single_job}} single_processor_job.sh] script {{#fileAnchor: single_job}}

#!/bin/sh
#SBATCH --job-name=serial_job_test # Job name
#SBATCH --mail-type=ALL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=ENTER_YOUR_EMAIL_HERE # Where to send mail	
#SBATCH --nodes=1 # Use one node
#SBATCH --ntasks=1 # Run a single task
#SBATCH --mem=1gb # Memory limit
#SBATCH --time=00:05:00 # Time limit hrs:min:sec
#SBATCH --output=serial_test_%j.out # Standard output and error log
pwd; hostname; date

module load gcc/5.2.0 python/2.7.10

echo "Running plot program on a single CPU core"

python /ufrc/data/training/SLURM/plot_template.py

date

Threaded or multi-processor job

This script can serve as a template for applications that are capable of using multiple processors on a single server or physical computer. These applications are commonly referred to as threaded, OpenMP, PTHREADS, or shared memory applications. While they can use multiple processors, they cannot make use of multiple servers and all the processors must be on the same node.

Download the [{{#fileLink: parallel_job}} multi_processor_job.sh] script {{#fileAnchor: parallel_job}}

#!/bin/sh
#SBATCH --job-name=parallel_job_test # Job name
#SBATCH --mail-type=ALL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=ENTER_YOUR_EMAIL_HERE # Where to send mail	
#SBATCH --nodes=1 # Use one node
#SBATCH --ntasks=1 # Run a single task	
#SBATCH --cpus-per-task=4 # Number of CPU cores per task
#SBATCH --mem=4gb # Total memory limit
#SBATCH --time=00:05:00 # Time limit hrs:min:sec
#SBATCH --output=parallel_test_%j.out # Standard output and error log
pwd; hostname; date

echo "Running prime number generator program on $SLURM_CPUS_ON_NODE CPU cores"

module load gcc/5.2.0 

/ufrc/data/training/SLURM/prime/prime

date

MPI job

This script can serve as a template for MPI, or message passing interface, applications. These are applications that can use multiple processors that may, or may not, be on multiple servers. In this example, each MPI rank has a single processor, and the number of nodes is not specified. This gives SLURM the most flexibility in finding resources for the job, but may not result in the best performance. Users can add the --nodes flag to specify the number of nodes they wish the job to use.

Download the [{{#fileLink: mpi_job}} mpi_job.sh] script {{#fileAnchor: mpi_job}}

#!/bin/sh
#SBATCH --job-name=mpi_job_test # Job name
#SBATCH --mail-type=ALL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=ENTER_YOUR_EMAIL_HERE # Where to send mail	
#SBATCH --ntasks=4 # Number of MPI ranks
#SBATCH --cpus-per-task=1 # Number of cores per MPI rank 
#SBATCH --mem-per-cpu=1gb # Memory per processor
#SBATCH --time=00:05:00 # Time limit hrs:min:sec
#SBATCH --output=mpi_test_%j.out # Standard output and error log
pwd; hostname; date

echo "Running prime number generator program on $SLURM_JOB_NUM_NODES nodes with $SLURM_NTASKS tasks, each with $SLURM_CPUS_PER_TASK cores."

module load intel/2016.0.109 openmpi/1.10.2

mpiexec /ufrc/data/training/SLURM/prime/prime_mpi

date

Hybrid MPI/Threaded job

This script can serve as a template for hybrid MPI/Threaded applications. These are MPI applications where each MPI rank is threaded and can use multiple processors. There are many options for specifying core layout and users are urged to consult the SLURM sbatch documentation as well as test different options and their effect on performance.

Download the [{{#fileLink: hybrid_job}} hybrid_job.sh] script {{#fileAnchor: hybrid_job}}

#!/bin/sh
#SBATCH --job-name=hybrid_job_test # Job name
#SBATCH --mail-type=ALL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=ENTER_YOUR_EMAIL_HERE # Where to send mail	
#SBATCH --ntasks=5 # Number of MPI ranks
#SBATCH --cpus-per-task=5 # Number of cores per MPI rank 
#SBATCH --mem-per-cpu=100mb # Memory per core
#SBATCH --time=00:05:00 # Time limit hrs:min:sec
#SBATCH --output=hybrid_test_%j.out # Standard output and error log
pwd; hostname; date

module load intel/2016.0.109 openmpi/1.10.2 raxml/8.2.8

mpiexec raxmlHPC-HYBRID-SSE3 -T $SLURM_CPUS_PER_TASK \
      -f a -m GTRGAMMA -s /ufrc/data/training/SLURM/dna.phy -p $RANDOM \
      -x $RANDOM -N 500 -n dna

date

• Note that MPI gets -np from SLURM automatically.
• Note there are many directives available to control processor layout.
  • Some to pay particular attention to are:
    • --nodes if you care exactly how many nodes are used
    • --ntasks-per-node to limit number of tasks on a node
    • --distribution one of several directives (see also --contiguous, --cores-per-socket, --mem_bind, --ntasks-per-socket, --sockets-per-node) to control how tasks, cores and memory are distributed among nodes, sockets and cores. While SLURM will generally make appropriate decisions for setting up jobs, careful use of these directives can significantly enhance job performance and users are encouraged to profile application performance under different conditions.

Array Jobs

Please see the SLURM_Job_Arrays page for information on job arrays. Note that we use the simplest 'single-threaded' process example from above and extending it to an array of jobs. Modify the following script using the parallel, mpi, or hybrid job layout as needed.

Download the [{{#fileLink: array_job}} array_job.sh] script {{#fileAnchor: array_job}}

#!/bin/sh
#SBATCH --job-name=array_job_test # Job name
#SBATCH --mail-type=ALL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=ENTER_YOUR_EMAIL_HERE # Where to send mail	
#SBATCH --nodes=1 # Use one node
#SBATCH --ntasks=1 # Run a single task
#SBATCH --mem-per-cpu=1gb # Memory per processor
#SBATCH --time=00:05:00 # Time limit hrs:min:sec
#SBATCH --output=array_test_%A-$a.out # Standard output and error log
#SBATCH --array=1-5 # Array range
pwd; hostname; date

echo This is task $SLURM_ARRAY_TASK_ID

date

Note the use of %A for the master job ID of the array, and the %a for the task ID in the output filename.