Difference between revisions of "Nvidia CUDA Toolkit"

From UFRC
Jump to navigation Jump to search
Line 22: Line 22:
  
  
* For single task, non MPI programs
+
=== For single task, non MPI programs===
  
 
  module load cuda/9.1.85 intel/2017
 
  module load cuda/9.1.85 intel/2017
Line 30: Line 30:
 
  module load cuda/9.1.85 gcc/7.3.0
 
  module load cuda/9.1.85 gcc/7.3.0
  
* For MPI programs
+
===For MPI programs ===
  
 
  module load cuda/9.1.85 intel/2017 openmpi/3.0.0
 
  module load cuda/9.1.85 intel/2017 openmpi/3.0.0

Revision as of 16:31, 27 July 2018

Description

cuda website  
CUDA™ is a parallel computing platform and programming model invented by NVIDIA. It enables dramatic increases in computing performance by harnessing the power of the graphics processing unit (GPU). With millions of CUDA-enabled GPUs sold to date, software developers, scientists and researchers are finding broad-ranging uses for GPU computing with CUDA.

Required Modules

For single task, non MPI programs

module load cuda/9.1.85 intel/2017

or

module load cuda/9.1.85 gcc/7.3.0

For MPI programs

module load cuda/9.1.85 intel/2017 openmpi/3.0.0

System Variables

  • HPC_{{#uppercase:cuda}}_DIR
  • HPC_{{#uppercase:cuda}}_BIN
  • HPC_{{#uppercase:cuda}}_INC
  • HPC_{{#uppercase:cuda}}_LIB

Available GPUs

Research Computing has a significant investment in GPU-enabled servers. Each supports from two to eight Nvidia GPUs (see table below).

Available GPUs under Torque/Moab on HPG1

GPU Quantity Host Quantity Host Architecture Host Memory Host Interconnect Host Attributes Notes
M2070 8 4 Intel E5675 24 GB QDR IB fermi,m2070 cobra[1-4]
M2070 8 1 Intel E5620 24 GB GigE fermi,m2070 vette
M2090 4 1 Intel E5-2643 64 GB FDR IB fermi,m2090
M2090 14 7 AMD Opteron 6220 32 GB QDR IB fermi,m2090

Available GPUs under SLURM on HPG1

GPU Quantity Host Quantity Host Architecture Host Memory Host Interconnect Host Attributes Notes
M2090 26 13 AMD Opteron 6220 32 GB QDR IB fermi,m2090

Available GPUs under SLURM on HPG2

GPU Quantity Host Quantity Host Architecture Host Memory Host Interconnect Host Attributes Notes
Tesla K80 32 8 INTEL E5-2683 132 GB QDR IB fermi,m2090

Usage Policy

Interactive Use

If you need interactive access to a gpu for development and testing you may do so by requesting an interactive session through the batch system.

In order to gain interactive access to a GPU server you should run similar to the one that follows.

Under SLURM

To access 1 GPU for a default 10-minute session:

srun -p hpg1-gpu --pty -u bash -i

OR

srun -p hpg2-gpu --pty -u bash -i

To access 2 Tesla GPUs on one node for a 1-hour session:

srun -p hpg1-gpu --gres=gpu:tesla:2 --time=01:00:00  --pty -u bash -i  

OR

srun -p hpg2-gpu --gres=gpu:tesla:2 --time=01:00:00  --pty -u bash -i  

If a gpu is available, you will get a prompt on a gpu-enabled host within a minute or two. Otherwise, you will have to wait or try another time. If you choose to wait, you will be connected when a gpu is available. The default walltime limit for the gpu queue is 10 minutes. You should request the amount of time you need but be sure to log out and end your session when you are finished so that the GPU will be available to others.

Batch Jobs

For batch jobs, to access a host with Tesla GPUs, you can add the following to your submission script.

Under SLURM:

#SBATCH --partition=hpg1-gpu
#SBATCH --gres=gpu:tesla:2

OR

#SBATCH --partition=hpg2-gpu
#SBATCH --gres=gpu:tesla:2

Exclusive Mode

The GPUs are configured to run in exclusive mode. This means that the gpu driver will only allow one process at a time to access the GPU. If GPU 0 is in use and your application tries to use it, it will simply block. If your application does not call cudaSetDevice(), the CUDA runtime should assign it to a free GPU. Since everyone will be accessing the GPUs through the batch system, there should be no over-subscription of the GPUs.

Environment

For CUDA development please load the "cuda" module. Doing so will ensure that your environment is set up correctly for the use of the CUDA compiler, header files, and libraries. Currently cuda/8.0 and cuda/8.061 are the only versions supported on hipergator.

$ module spider cuda
--------------------------------------------------------
  cuda:
--------------------------------------------------------
    Description:
      NVIDIA CUDA Toolkit

     Versions:
        cuda/8.0
        cuda/8.0.61
----------------------------------------------------------
  
For detailed information about a specific cuda module (including how to load the modules) use the module full name.
  For example:

     $ module spider cuda/8.0
----------------------------------------------------------
$ module load cuda/8.0

$ which nvcc
/apps/cuda/8.0/bin/nvcc

$ printenv | grep CUDA
HPC_CUDA_LIB=/apps/cuda/8.0/lib64
HPC_CUDA_DIR=/apps/cuda/8.0
HPC_CUDA_BIN=/apps/cuda/8.0/bin
HPC_CUDA_INC=/apps/cuda/8.0/include

Sample GPU Batch Job Scripts

SLURM Job Scripts

See the Example_SLURM-GPU-Job-Scripts page for an example.

CUDA Examples

See the Nvidia CUDA Toolkit_Examples page for CUDA development examples.