Sample SLURM Scripts

HiPerGator 2.0 documentation

Sample SLURM Scripts

Below are a number of sample scripts that can be used as a template for building your own SLURM submission scripts for use on HiPerGator 2.0. These scripts are also located at: /ufrc/data/training/SLURM/, and can be copied from there.

Basic, single-processor job

This script can serve as the template for many single-processor applications. The mem-per-cpu flag can be used to request the appropriate amount of memory for your job. Please make sure to test you application and set this value to a reasonable number based on actual memory use. The %j in the -o (can also use --output) line tells SLURM to substitute the job ID in the name of the output file. You can also add a -e or --error with an error file name to separate output and error logs.

#!/bin/sh
#SBATCH --job-name=find_coins    #Job name
#SBATCH --mail-type=ALL   # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=ENTER_YOUR_EMAIL_HERE   # Where to send mail	
#SBATCH --mem-per-cpu=1gb   # Per processor memory
#SBATCH -t 00:05:00    # Walltime hrs:min:sec
#SBATCH -o plot_example_%j.out	   # Name output file 
#
pwd; hostname; date

module load gcc/5.2.0 python/2.7.10

python /ufrc/data/training/SLURM/plot_template.py

date

Threaded or multi-processor job

This script can serve as a template for applications that are capable of using multiple processors on a single server or physical computer. These applications are commonly referred to as threaded, OpenMP, PTHREADS, or shared memory applications. While they can use multiple processors, they cannot make use of multiple servers and all the processors must be on the same node.

#!/bin/sh
#SBATCH --job-name=prime_threaded   #Job name	
#SBATCH --mail-type=ALL   # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=ENTER_YOUR_EMAIL_HERE   # Where to send mail	
#SBATCH --cpus-per-task=4   # Number of cores: Can also use -c=4 
#SBATCH --mem-per-cpu=1gb   # Per processor memory
#SBATCH -t 00:05:00   # Walltime
#SBATCH -o prime_threaded.%j.out   # Name output file 
#
pwd; hostname; date

echo Running prime number generator program

echo There are $SLURM_CPUS_ON_NODE cores available.

module load gcc/5.2.0 

/ufrc/data/training/SLURM/prime/prime

date

MPI job

This script can serve as a template for MPI, or message passing interface, applications. These are applications that can use multiple processors that may, or may not, be on multiple servers. In this example, each MPI rank has a single processor, and the number of nodes is not specified. This gives SLURM the most flexibility in finding resources for the job, but may not result in the best performance. Users can add the --nodes flag to specify the number of nodes they wish the job to use.

#!/bin/sh
#SBATCH --job-name=MPI_job    #Job name	
#SBATCH --mail-type=ALL   # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=ENTER_YOUR_EMAIL_HERE   # Where to send mail	
#SBATCH --ntasks=4   # Number of MPI ranks
#SBATCH --cpus-per-task=1   # Number of cores per MPI rank 
#SBATCH --mem-per-cpu=1gb   # Per processor memory
#SBATCH -t 00:05:00     # Walltime
#SBATCH -o MPI_ex_%j.out   # Name output file 
#
pwd; hostname; date

echo Running prime number generator program

echo Running on $SLURM_JOB_NUM_NODES nodes with $SLURM_NTASKS tasks, each with $SLURM_CPUS_PER_TASK cores.

module load intel/2016.0.109 openmpi/1.10.2

mpiexec /ufrc/data/training/SLURM/prime/prime_mpi

date

Hybrid MPI/Threaded job

This script can serve as a template for hybrid MPI/Threaded applications. These are MPI applications where each MPI rank is threaded and can use multiple processors. There are many options for specifying core layout and users are urged to consult the SLURM sbatch documentation as well as test different options and their effect on performance.

#!/bin/sh
#SBATCH --job-name=RAxML
#SBATCH --mail-type=ALL
#SBATCH --mail-user=YOUR_EMAIL_HERE	
#SBATCH -o raxml.%j.log
#SBATCH --ntasks=5
#SBATCH --cpus-per-task=5
#SBATCH --mem-per-cpu=100mb

module load intel/2016.0.109 openmpi/1.10.2 raxml/8.2.8

mpiexec raxmlHPC-HYBRID-SSE3 -T $SLURM_CPUS_PER_TASK \
      -f a -m GTRGAMMA -s /ufrc/data/training/SLURM/dna.phy -p $RANDOM \
      -x $RANDOM -N 500 -n dna

• Note that MPI gets -np from SLURM.
• Note there are many directives available to control processor layout.
  • Some to pay particular attention to are:
    • --nodes if you care exactly how many nodes are used
    • --ntasks-per-node to limit number of tasks on a node
    • --distribution one of several directives (see also --contiguous, --cores-per-socket, --mem_bind, --ntasks-per-socket, --sockets-per-node) to control how tasks, cores and memory are distributed among nodes, sockets and cores. While SLURM will generally make appropriate decisions for setting up jobs, careful use of these directives can significantly enhance job performance and users are encouraged to profile application performance under different conditions.