Difference between revisions of "Sample SLURM Scripts"

From UFRC
Jump to navigation Jump to search
m (Fix $a in array output name)
Line 9: Line 9:
 
This script can serve as the template for many single-processor applications. The mem-per-cpu flag can be used to request the appropriate amount of memory for your job. Please make sure to test your application and set this value to a reasonable number based on actual memory use. The %j in the -o (can also use --output) line tells SLURM to substitute the job ID in the name of the output file. You can also add a -e or --error with an error file name to separate output and error logs.
 
This script can serve as the template for many single-processor applications. The mem-per-cpu flag can be used to request the appropriate amount of memory for your job. Please make sure to test your application and set this value to a reasonable number based on actual memory use. The %j in the -o (can also use --output) line tells SLURM to substitute the job ID in the name of the output file. You can also add a -e or --error with an error file name to separate output and error logs.
  
Download the [{{#fileLink: single_job.sh}} single_processor_job.sh] script
+
Download the [{{#fileLink: single_core_job_annotated.sh}} single_core_job_annotated.sh] script
{{#fileAnchor: single_job.sh}}
+
{{#fileAnchor: single_core_job_annotated.sh}}
 
<source lang=bash>
 
<source lang=bash>
 
#!/bin/bash
 
#!/bin/bash
 
#SBATCH --job-name=serial_job_test    # Job name
 
#SBATCH --job-name=serial_job_test    # Job name
#SBATCH --mail-type=ALL              # Mail events (NONE, BEGIN, END, FAIL, ALL)
+
#SBATCH --mail-type=END,FAIL          # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=<email_address>  # Where to send mail
+
#SBATCH --mail-user=email@ufl.edu    # Where to send mail
 
#SBATCH --ntasks=1                    # Run on a single CPU
 
#SBATCH --ntasks=1                    # Run on a single CPU
#SBATCH --mem=600mb                  # Memory limit
+
#SBATCH --mem=1gb                    # Job memory request
 
#SBATCH --time=00:05:00              # Time limit hrs:min:sec
 
#SBATCH --time=00:05:00              # Time limit hrs:min:sec
#SBATCH --output=serial_test_%j.out   # Standard output and error log
+
#SBATCH --output=serial_test_%j.log   # Standard output and error log
 
 
 
pwd; hostname; date
 
pwd; hostname; date
  
Line 31: Line 30:
 
date
 
date
 
</source>
 
</source>
 +
  
 
==Threaded or multi-processor job==
 
==Threaded or multi-processor job==
Line 44: Line 44:
  
  
Download the [{{#fileLink: parallel_job.sh}} multi_processor_job.sh] script
+
Download the [{{#fileLink: multicore_job.sh}} multicore_job.sh] script
{{#fileAnchor: parallel_job.sh}}
+
{{#fileAnchor: multicore_job.sh}}
 
<source lang=bash>
 
<source lang=bash>
 
#!/bin/bash
 
#!/bin/bash
#SBATCH --job-name=parallel_job_test # Job name
+
#SBATCH --job-name=parallel_job      # Job name
#SBATCH --mail-type=ALL              # Mail events (NONE, BEGIN, END, FAIL, ALL)
+
#SBATCH --mail-type=END,FAIL        # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=<email_address>  # Where to send mail
+
#SBATCH --mail-user=email@ufl.edu    # Where to send mail
#SBATCH --nodes=1                    # Use one node
 
 
#SBATCH --ntasks=1                  # Run a single task
 
#SBATCH --ntasks=1                  # Run a single task
 
#SBATCH --cpus-per-task=4            # Number of CPU cores per task
 
#SBATCH --cpus-per-task=4            # Number of CPU cores per task
#SBATCH --mem=600mb                  # Total memory limit
+
#SBATCH --mem=1gb                    # Job memory request
 
#SBATCH --time=00:05:00              # Time limit hrs:min:sec
 
#SBATCH --time=00:05:00              # Time limit hrs:min:sec
#SBATCH --output=parallel_%j.out     # Standard output and error log
+
#SBATCH --output=parallel_%j.log     # Standard output and error log
 
 
 
pwd; hostname; date
 
pwd; hostname; date
  
Line 77: Line 75:
 
#!/bin/bash
 
#!/bin/bash
 
#SBATCH --job-name=parallel_job_test # Job name
 
#SBATCH --job-name=parallel_job_test # Job name
#SBATCH --mail-type=ALL              # Mail events (NONE, BEGIN, END, FAIL, ALL)
+
#SBATCH --mail-type=END,FAIL        # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=<email_address>  # Where to send mail
+
#SBATCH --mail-user=email@ufl.edu    # Where to send mail
#SBATCH --nodes=1                    # Use one node
 
 
#SBATCH --ntasks=1                  # Run a single task
 
#SBATCH --ntasks=1                  # Run a single task
 
#SBATCH --cpus-per-task=4            # Number of CPU cores per task
 
#SBATCH --cpus-per-task=4            # Number of CPU cores per task
 
#SBATCH --mem=600mb                  # Total memory limit
 
#SBATCH --mem=600mb                  # Total memory limit
 
#SBATCH --time=00:05:00              # Time limit hrs:min:sec
 
#SBATCH --time=00:05:00              # Time limit hrs:min:sec
#SBATCH --output=parallel_%j.out     # Standard output and error log
+
#SBATCH --output=parallel_%j.log     # Standard output and error log
 +
date;hostname;pwd
  
 
export OMP_NUM_THREADS=4
 
export OMP_NUM_THREADS=4
  
# Load required modules; for example, if your program was
 
# compiled with Intel compiler, use the following
 
 
module load intel
 
module load intel
  
# Run your program with correct path and command line options
 
 
./YOURPROGRAM INPUT
 
./YOURPROGRAM INPUT
 +
 +
date
 
</source>
 
</source>
  
Line 127: Line 124:
 
#!/bin/bash
 
#!/bin/bash
 
#SBATCH --job-name=mpi_job_test      # Job name
 
#SBATCH --job-name=mpi_job_test      # Job name
#SBATCH --mail-type=ALL              # Mail events (NONE, BEGIN, END, FAIL, ALL)
+
#SBATCH --mail-type=END,FAIL        # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=<email_address>  # Where to send mail
+
#SBATCH --mail-user=email@ufl.edu    # Where to send mail
 
#SBATCH --ntasks=24                  # Number of MPI ranks
 
#SBATCH --ntasks=24                  # Number of MPI ranks
 
#SBATCH --cpus-per-task=1            # Number of cores per MPI rank  
 
#SBATCH --cpus-per-task=1            # Number of cores per MPI rank  
Line 137: Line 134:
 
#SBATCH --mem-per-cpu=600mb          # Memory per processor
 
#SBATCH --mem-per-cpu=600mb          # Memory per processor
 
#SBATCH --time=00:05:00              # Time limit hrs:min:sec
 
#SBATCH --time=00:05:00              # Time limit hrs:min:sec
#SBATCH --output=mpi_test_%j.out     # Standard output and error log
+
#SBATCH --output=mpi_test_%j.log     # Standard output and error log
 
pwd; hostname; date
 
pwd; hostname; date
  
Line 161: Line 158:
 
#!/bin/bash
 
#!/bin/bash
 
#SBATCH --job-name=hybrid_job_test      # Job name
 
#SBATCH --job-name=hybrid_job_test      # Job name
#SBATCH --mail-type=ALL                # Mail events (NONE, BEGIN, END, FAIL, ALL)
+
#SBATCH --mail-type=END,FAIL            # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=<email_address>    # Where to send mail
+
#SBATCH --mail-user=email@ufl.edu      # Where to send mail
 
#SBATCH --ntasks=8                      # Number of MPI ranks
 
#SBATCH --ntasks=8                      # Number of MPI ranks
 
#SBATCH --cpus-per-task=4              # Number of cores per MPI rank  
 
#SBATCH --cpus-per-task=4              # Number of cores per MPI rank  
Line 170: Line 167:
 
#SBATCH --mem-per-cpu=100mb            # Memory per core
 
#SBATCH --mem-per-cpu=100mb            # Memory per core
 
#SBATCH --time=00:05:00                # Time limit hrs:min:sec
 
#SBATCH --time=00:05:00                # Time limit hrs:min:sec
#SBATCH --output=hybrid_test_%j.out     # Standard output and error log
+
#SBATCH --output=hybrid_test_%j.log     # Standard output and error log
 
 
 
pwd; hostname; date
 
pwd; hostname; date
 
   
 
   
Line 181: Line 177:
 
   
 
   
 
date
 
date
 
 
</source>
 
</source>
  
Line 193: Line 188:
 
#SBATCH --job-name=LAMMPS
 
#SBATCH --job-name=LAMMPS
 
#SBATCH --output=LAMMPS_%j.out
 
#SBATCH --output=LAMMPS_%j.out
#SBATCH --mail-type=ALL
+
#SBATCH --mail-type=END,FAIL
 
#SBATCH --mail-user=<email_address>
 
#SBATCH --mail-user=<email_address>
 
#SBATCH --nodes=4              # Number of nodes
 
#SBATCH --nodes=4              # Number of nodes
Line 202: Line 197:
 
#SBATCH --mem-per-cpu=2000mb  # Per processor memory request
 
#SBATCH --mem-per-cpu=2000mb  # Per processor memory request
 
#SBATCH --time=4-00:00:00      # Walltime in hh:mm:ss or d-hh:mm:ss
 
#SBATCH --time=4-00:00:00      # Walltime in hh:mm:ss or d-hh:mm:ss
 
+
date;hostname;pwd
date
 
hostname
 
  
 
module load intel/2016.0.109 openmpi/1.10.2
 
module load intel/2016.0.109 openmpi/1.10.2
Line 211: Line 204:
  
 
srun --mpi=pmi2 /path/to/app/lmp_gator2 < in.Cu.v.24nm.eq_xrd
 
srun --mpi=pmi2 /path/to/app/lmp_gator2 < in.Cu.v.24nm.eq_xrd
 +
 +
date
 
</source>
 
</source>
  
Line 228: Line 223:
 
#!/bin/bash
 
#!/bin/bash
 
#SBATCH --job-name=array_job_test  # Job name
 
#SBATCH --job-name=array_job_test  # Job name
#SBATCH --mail-type=ALL            # Mail events (NONE, BEGIN, END, FAIL, ALL)
+
#SBATCH --mail-type=FAIL            # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=<email_address> # Where to send mail
+
#SBATCH --mail-user=email@ufl.edu  # Where to send mail
#SBATCH --nodes=1                  # Use one node
 
 
#SBATCH --ntasks=1                  # Run a single task
 
#SBATCH --ntasks=1                  # Run a single task
#SBATCH --mem-per-cpu=1gb           # Memory per processor
+
#SBATCH --mem=1gb                   # Job Memory
 
#SBATCH --time=00:05:00            # Time limit hrs:min:sec
 
#SBATCH --time=00:05:00            # Time limit hrs:min:sec
#SBATCH --output=array_%A-%a.out   # Standard output and error log
+
#SBATCH --output=array_%A-%a.log   # Standard output and error log
 
#SBATCH --array=1-5                # Array range
 
#SBATCH --array=1-5                # Array range
 
pwd; hostname; date
 
pwd; hostname; date
Line 260: Line 254:
 
#SBATCH --time=12:00:00
 
#SBATCH --time=12:00:00
 
#SBATCH --mem-per-cpu=2000
 
#SBATCH --mem-per-cpu=2000
#SBATCH --mail-type=ALL
+
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=taylor@rc.ufl.edu
+
#SBATCH --mail-user=email@ufl.edu
 
#SBATCH --account=ufhpc
 
#SBATCH --account=ufhpc
 
#SBATCH --qos=ufhpc-b
 
#SBATCH --qos=ufhpc-b
 
#SBATCH --partition=hpg1-gpu
 
#SBATCH --partition=hpg1-gpu
 
#SBATCH --gres=gpu:tesla:2
 
#SBATCH --gres=gpu:tesla:2
 +
date;hostname;pwd
  
 
module load cuda/8.0
 
module load cuda/8.0
Line 281: Line 276:
 
wait
 
wait
  
 +
date
 
</source>
 
</source>

Revision as of 18:26, 5 December 2017

Hpg2 wiki logo.png

HiPerGator 2.0 documentation

Sample SLURM Scripts

Below are a number of sample scripts that can be used as a template for building your own SLURM submission scripts for use on HiPerGator 2.0. These scripts are also located at: /ufrc/data/training/SLURM/, and can be copied from there. If you choose to copy one of these sample scripts, please make sure you understand what each line of the sbatch directives before using it to submit your jobs. Otherwise, you may not get the result you want and may waste valuable computing resources.

Basic, single-processor job

This script can serve as the template for many single-processor applications. The mem-per-cpu flag can be used to request the appropriate amount of memory for your job. Please make sure to test your application and set this value to a reasonable number based on actual memory use. The %j in the -o (can also use --output) line tells SLURM to substitute the job ID in the name of the output file. You can also add a -e or --error with an error file name to separate output and error logs.

Download the [{{#fileLink: single_core_job_annotated.sh}} single_core_job_annotated.sh] script {{#fileAnchor: single_core_job_annotated.sh}}

#!/bin/bash
#SBATCH --job-name=serial_job_test    # Job name
#SBATCH --mail-type=END,FAIL          # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=email@ufl.edu     # Where to send mail	
#SBATCH --ntasks=1                    # Run on a single CPU
#SBATCH --mem=1gb                     # Job memory request
#SBATCH --time=00:05:00               # Time limit hrs:min:sec
#SBATCH --output=serial_test_%j.log   # Standard output and error log
pwd; hostname; date

module load python

echo "Running plot script on a single CPU core"

python /ufrc/data/training/SLURM/plot_template.py

date


Threaded or multi-processor job

This script can serve as a template for applications that are capable of using multiple processors on a single server or physical computer. These applications are commonly referred to as threaded, OpenMP, PTHREADS, or shared memory applications. While they can use multiple processors, they cannot make use of multiple servers and all the processors must be on the same node.

These applications required shared memory and can only run on one node; as such it is important to remember the following:

  • You must set --nodes=1, and then set --cpus-per-task to the number of OpenMP threads you wish to use.
  • You must make the application aware of how many processors to use. How that is done depends on the application:
    • For some applications, set OMP_NUM_THREADS to a value less than or equal to the number of cpus-per-task you set.
    • For some applications, use a command line option when calling that application.


Download the [{{#fileLink: multicore_job.sh}} multicore_job.sh] script {{#fileAnchor: multicore_job.sh}}

#!/bin/bash
#SBATCH --job-name=parallel_job      # Job name
#SBATCH --mail-type=END,FAIL         # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=email@ufl.edu    # Where to send mail	
#SBATCH --ntasks=1                   # Run a single task	
#SBATCH --cpus-per-task=4            # Number of CPU cores per task
#SBATCH --mem=1gb                    # Job memory request
#SBATCH --time=00:05:00              # Time limit hrs:min:sec
#SBATCH --output=parallel_%j.log     # Standard output and error log
pwd; hostname; date

echo "Running prime number generator program on $SLURM_CPUS_ON_NODE CPU cores"

module load gcc/5.2.0 

/ufrc/data/training/SLURM/prime/prime

date


Another example, setting OMP_NUM_THREADS:

Download the [{{#fileLink: parallel_job2.sh}} multi_processor_job2.sh] script {{#fileAnchor: parallel_job2.sh}}

#!/bin/bash
#SBATCH --job-name=parallel_job_test # Job name
#SBATCH --mail-type=END,FAIL         # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=email@ufl.edu    # Where to send mail	
#SBATCH --ntasks=1                   # Run a single task	
#SBATCH --cpus-per-task=4            # Number of CPU cores per task
#SBATCH --mem=600mb                  # Total memory limit
#SBATCH --time=00:05:00              # Time limit hrs:min:sec
#SBATCH --output=parallel_%j.log     # Standard output and error log
date;hostname;pwd

export OMP_NUM_THREADS=4

module load intel

./YOURPROGRAM INPUT

date

MPI job

This script can serve as a template for MPI, or message passing interface, applications. These are applications that can use multiple processors that may, or may not, be on multiple servers.

Our testing has found that it is best to be very specific about how you want your MPI ranks laid out across nodes and even sockets (multi-core CPUs). SLURM and OpenMPI have some conflicting behavior if you leave too much to chance. Please refer to the full SLURM sbatch documentation, but the following directives are the main directives to pay attention to:

  • -c, --cpus-per-task=<ncpus>
    • Advise the Slurm controller that ensuing job steps will require ncpus number of processors per task.
  • -m, --distribution=arbitrary|<block|cyclic|plane=<options>[:block|cyclic|fcyclic]>
    • Specify alternate distribution methods for remote processes.
    • We recommend -m cyclic:cyclic, which tells SLURM to distribute tasks cyclically over nodes and sockets.
  • -N, --nodes=<minnodes[-maxnodes]>
    • Request that a minimum of minnodes nodes be allocated to this job.
  • -n, --ntasks=<number>
    • Number of tasks (MPI ranks)
  • --ntasks-per-node=<ntasks>
    • Request that ntasks be invoked on each node
  • --ntasks-per-socket=<ntasks>
    • Request the maximum ntasks be invoked on each socket
    • Notes on socket layout:
      • hpg2-compute nodes have 2 sockets, each with 16 cores.
      • hpg1-compute nodes have 4 sockets, each with 16 cores.


The following example requests 24 tasks, each with one core. It further specifies that these should be split evenly on 2 nodes, and within the nodes, the 12 tasks should be evenly split on the two sockets. So each CPU on the two nodes will have 6 tasks, each with its own dedicated core. The distribution option will ensure that MPI ranks are distributed cyclically on nodes and sockets.

SLURM is very flexible and allows users to be very specific about their resource requests. Thinking about your application and doing some testing will be important to determine the best request for your specific use.

Download the [{{#fileLink: mpi_job.sh}} mpi_job.sh] script {{#fileAnchor: mpi_job.sh}}

#!/bin/bash
#SBATCH --job-name=mpi_job_test      # Job name
#SBATCH --mail-type=END,FAIL         # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=email@ufl.edu    # Where to send mail	
#SBATCH --ntasks=24                  # Number of MPI ranks
#SBATCH --cpus-per-task=1            # Number of cores per MPI rank 
#SBATCH --nodes=2                    # Number of nodes
#SBATCH --ntasks-per-node=12         # How many tasks on each node
#SBATCH --ntasks-per-socket=6        # How many tasks on each CPU or socket
#SBATCH --distribution=cyclic:cyclic # Distribute tasks cyclically on nodes and sockets
#SBATCH --mem-per-cpu=600mb          # Memory per processor
#SBATCH --time=00:05:00              # Time limit hrs:min:sec
#SBATCH --output=mpi_test_%j.log     # Standard output and error log
pwd; hostname; date

echo "Running prime number generator program on $SLURM_JOB_NUM_NODES nodes with $SLURM_NTASKS tasks, each with $SLURM_CPUS_PER_TASK cores."

module load intel/2016.0.109 openmpi/1.10.2

srun --mpi=pmi2 /ufrc/data/training/SLURM/prime/prime_mpi

date

Hybrid MPI/Threaded job

This script can serve as a template for hybrid MPI/Threaded applications. These are MPI applications where each MPI rank is threaded and can use multiple processors.

Our testing has found that it is best to be very specific about how you want your MPI ranks laid out across nodes and even sockets (multi-core CPUs). SLURM and OpenMPI have some conflicting behavior if you leave too much to chance. Please refer to the full SLURM sbatch documentation, as well as the information in the MPI example above.

The following example requests 8 tasks, each with 4 cores. It further specifies that these should be split evenly on 2 nodes, and within the nodes, the 4 tasks should be evenly split on the two sockets. So each CPU on the two nodes will have 2 tasks, each with 4 cores. The distribution option will ensure that MPI ranks are distributed cyclically on nodes and sockets.

Download the [{{#fileLink: hybrid_pthreads_job.sh}} hybrid_pthreads_job.sh] script {{#fileAnchor: hybrid_pthreads_job.sh}}

#!/bin/bash
#SBATCH --job-name=hybrid_job_test      # Job name
#SBATCH --mail-type=END,FAIL            # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=email@ufl.edu       # Where to send mail	
#SBATCH --ntasks=8                      # Number of MPI ranks
#SBATCH --cpus-per-task=4               # Number of cores per MPI rank 
#SBATCH --nodes=2                       # Number of nodes
#SBATCH --ntasks-per-node=4             # How many tasks on each node
#SBATCH --ntasks-per-socket=2           # How many tasks on each CPU or socket
#SBATCH --mem-per-cpu=100mb             # Memory per core
#SBATCH --time=00:05:00                 # Time limit hrs:min:sec
#SBATCH --output=hybrid_test_%j.log     # Standard output and error log
pwd; hostname; date
 
module load intel/2016.0.109 openmpi/1.10.2 raxml/8.2.8
 
srun --mpi=pmi2 raxmlHPC-HYBRID-SSE3 -T $SLURM_CPUS_PER_TASK \
      -f a -m GTRGAMMA -s /ufrc/data/training/SLURM/dna.phy -p $RANDOM \
      -x $RANDOM -N 500 -n dna
 
date

The following example requests 8 tasks, each with 8 cores. It further specifies that these should be split evenly on 4 nodes, and within the nodes, the 2 tasks should be split, one on each of the two sockets. So each CPU on the two nodes will have 1 task, each with 8 cores. The distribution option will ensure that MPI ranks are distributed cyclically on nodes and sockets.

Also note setting OMP_NUM_THREADS so that OpenMP knows how many threads to use per task.

Download the [{{#fileLink: hybrid_OpenMP_job.sh}} hybrid_OpenMP_job.sh] script

#!/bin/bash
#SBATCH --job-name=LAMMPS
#SBATCH --output=LAMMPS_%j.out
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=<email_address>
#SBATCH --nodes=4              # Number of nodes
#SBATCH --ntasks=8             # Number of MPI ranks
#SBATCH --ntasks-per-node=2    # Number of MPI ranks per node
#SBATCH --ntasks-per-socket=1  # Number of tasks per processor socket on the node
#SBATCH --cpus-per-task=8      # Number of OpenMP threads for each MPI process/rank
#SBATCH --mem-per-cpu=2000mb   # Per processor memory request
#SBATCH --time=4-00:00:00      # Walltime in hh:mm:ss or d-hh:mm:ss
date;hostname;pwd

module load intel/2016.0.109 openmpi/1.10.2

export OMP_NUM_THREADS=8

srun --mpi=pmi2 /path/to/app/lmp_gator2 < in.Cu.v.24nm.eq_xrd

date
  • Note that MPI gets -np from SLURM automatically.
  • Note there are many directives available to control processor layout.
    • Some to pay particular attention to are:
      • --nodes if you care exactly how many nodes are used
      • --ntasks-per-node to limit number of tasks on a node
      • --distribution one of several directives (see also --contiguous, --cores-per-socket, --mem_bind, --ntasks-per-socket, --sockets-per-node) to control how tasks, cores and memory are distributed among nodes, sockets and cores. While SLURM will generally make appropriate decisions for setting up jobs, careful use of these directives can significantly enhance job performance and users are encouraged to profile application performance under different conditions.

Array job

Please see the SLURM_Job_Arrays page for information on job arrays. Note that we use the simplest 'single-threaded' process example from above and extending it to an array of jobs. Modify the following script using the parallel, mpi, or hybrid job layout as needed.

Download the [{{#fileLink: array_job.sh}} array_job.sh] script {{#fileAnchor: array_job.sh}}

#!/bin/bash
#SBATCH --job-name=array_job_test   # Job name
#SBATCH --mail-type=FAIL            # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=email@ufl.edu   # Where to send mail	
#SBATCH --ntasks=1                  # Run a single task
#SBATCH --mem=1gb                   # Job Memory
#SBATCH --time=00:05:00             # Time limit hrs:min:sec
#SBATCH --output=array_%A-%a.log    # Standard output and error log
#SBATCH --array=1-5                 # Array range
pwd; hostname; date

echo This is task $SLURM_ARRAY_TASK_ID

date

Note the use of %A for the master job ID of the array, and the %a for the task ID in the output filename.

GPU job

Please see GPU_Access for more information regarding the use of HiPerGator GPUs.

Download the [{{#fileLink: Gpu_job.zip}} gpu_job.sh] script

#!/bin/bash
#SBATCH --job-name=gpuMemTest
#SBATCH --output=gpuMemTest.out
#SBATCH --error=gpuMemTest.err
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=1
#SBATCH --distribution=cyclic:cyclic
#SBATCH --time=12:00:00
#SBATCH --mem-per-cpu=2000
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=email@ufl.edu
#SBATCH --account=ufhpc
#SBATCH --qos=ufhpc-b
#SBATCH --partition=hpg1-gpu
#SBATCH --gres=gpu:tesla:2
date;hostname;pwd

module load cuda/8.0

cudaMemTest=/ufrc/ufhpc/chasman/Cuda/cudaMemTest/cuda_memtest

cudaDevs=$(echo $CUDA_VISIBLE_DEVICES | sed -e 's/,/ /g')

for cudaDev in $cudaDevs
do
  echo cudaDev = $cudaDev
  #srun --gres=gpu:tesla:1 -n 1 --exclusive ./gpuMemTest.sh > gpuMemTest.out.$cudaDev 2>&1 &
  $cudaMemTest --num_passes 1 --device $cudaDev > gpuMemTest.out.$cudaDev 2>&1 &
done
wait

date