Temporary Directories
When a SLURM job starts, the scheduler creates a temporary directory for the job on the compute node's local hard drive. This directory accessible via the $SLURM_TMPDIR or $TMPDIR variable as a filesystem path is very useful for jobs that need to use or generate a large number of small files, as the main RC parallel filesystem is optimized for large file streaming and is less suitable for small files.
The directory is owned by the user running the job. The path to the temporary directory is made available as the $SLURM_TMPDIR variable. At the end of the job, the temporary directory is automatically removed unless the job fails at which time it's temporarily retained for troubleshooting.
You can use the ${SLURM_TMPDIR} variable in job scripts to copy temporary data to the temporary job directory. If necessary, it can also be used as argument for applications that accept a temporary directory argument.
Note - Default Paths
Many applications and programming languages use the $TMPDIR environment variable, if available, as the default temporary directory path. If this variable is not set, the applications will default to using the /tmp directory, which is not desirable. SLURM will set $TMPDIR to the same value as $SLURM_TMPDIR unless $TMPDIR has already been set, in which case it will be ignored. Check your job script(s) and shell initialization files like .bashrc and .bash_profile to make sure you do not have $TMPDIR set.
If a personal Singularity container is used, make sure that the $SINGULARITYENV_TMPDIR variable is set within the job to export the local scratch location into the Singularity container.