Difference between revisions of "GPU Access"

From UFRC
Jump to navigation Jump to search
 
(86 intermediate revisions by 10 users not shown)
Line 1: Line 1:
[[Category:SLURM]]
+
[[Category:Scheduler]][[Category:GPU]]
Researchers may use GPUs in the form of Normalized Graphics Processor Units (NGUs), which include all of the infrastructure (memory, network, rack space, cooling), necessary for GPU-accelerated computation.
 
  
Groups that do not have GPU allocations can invest into GPUs by filling out the purchase form at: https://www.rc.ufl.edu/services/purchase-request/.
+
{{Note|Interactive Jobs in the GPU partition are limited to 12 hrs|warn}}
  
=GPU-enabled Servers=
+
Normalized Graphics Processor Units (NGUs) include all of the infrastructure (memory, network, rack space, cooling) necessary for GPU-accelerated computation. Each NGU is equivalent to 1 GPU presently, however newer GPUs such as the A100s may require more than 1 NGU to access in the future.
  
We have two types of GPU services for two different kinds of applications.  
+
Researchers can add NGUs to their allocations by filling out the [https://www.rc.ufl.edu/get-started/purchase-allocation/ Purchase Form] or requesting a [https://www.rc.ufl.edu/services/request-trial-allocation/ Trial Allocation].
 +
 
 +
=GPU-enabled Services=
 +
 
 +
Types of GPUs are listed below. Two partitions contain GPUs - the hwgui partition for visualization and the gpu partition for general computation.  
  
 
== Hardware Accelerated GUI ==
 
== Hardware Accelerated GUI ==
  
GPUs are used for hardware accelerated graphical applications. To run this type of applications on HiPerGator, please use SLURM partition "'''hwgui'''" and refer to '''[[Hardware Accelerated GUI Sessions]]''' for more information on the usage.
+
GPUs in these servers are used to accelerate rendering for graphical applications. These servers are in the SLURM "'''hwgui'''" partition. Refer to the '''[[Hardware Accelerated GUI Sessions]]''' page for more information on available resources and usage.
  
 
== GPU Assisted Computation ==
 
== GPU Assisted Computation ==
  
A number of high performance applications installed on HiPerGator implement GPU-accelerated computing functions via CUDA to achieve significant speed-up over CPU implementations. Please use SLURM partition '''"gpu"''' to run GPU enabled computational applications.  
+
A number of high performance applications installed on HiPerGator implement GPU-accelerated computing functions via CUDA to achieve significant speed-up over CPU calculations. These servers are in the SLURM '''"gpu"''' partition (<code>--partition=gpu</code>).
  
=== GPU Specification for GPU Partition===
+
=== Hardware Specifications for the GPU Partition===
 +
We have the following types of NVIDIA GPU nodes available in the "gpu" partition:
  
We have two types of NVIDIA GPU nodes currently available in "gpu" partition:
+
* NVIDIA GeForce GTX 1080Ti, with 2 GPUs per node. See [https://www.geforce.com/hardware/desktop-gpus/geforce-gtx-1080-ti/specifications technical specifications] for reference.
 
+
* NVIDIA GeForce RTX 2080Ti, with 8 GPUs per node. See [https://www.nvidia.com/en-us/geforce/graphics-cards/rtx-2080-ti technical specifications] for reference.
* Nvidia K80s, with 2 GPUs per K80 card and 2 K80 cards in one host. Please refer to [https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/tesla-product-literature/Tesla-K80-BoardSpec-07317-001-v05.pdf K80 technical specs]
+
* NVIDIA Quadro RTX 6000, with 8 GPUs per node. These GPUs have SLI bridging See [https://www.nvidia.com/en-us/design-visualization/quadro/rtx-6000/ technical specifications] for reference.
* Nvidia GeForce GTX 1080 Ti, with 1 GPU per 1080Ti card and 2 1080Ti cards in one host. Please refer to [https://www.geforce.com/hardware/desktop-gpus/geforce-gtx-1080-ti/specifications 1080Ti technical specs]
+
* AI NVIDIA DGX A100 SuperPod, with 8 GPUs per node. These GPUs have [https://www.nvidia.com/en-us/data-center/nvlink/ NVSWITCH] interconnects See [https://www.nvidia.com/en-us/data-center/a100/ technical specifications] for reference.
* Nvidia GeForce RTX 2080Ti, with 1 GPU per 2080Ti card and 8 2080Ti cards in one host. Please refer to [https://www.nvidia.com/en-us/geforce/graphics-cards/rtx-2080-ti 280Ti technical specs]
 
  
 
{| style="margin-left: 5px; width:80%"
 
{| style="margin-left: 5px; width:80%"
 
|
 
|
 
{| class="wikitable" style="text-align: center"
 
{| class="wikitable" style="text-align: center"
!GPU!!Quantity!!Host Quantity!!Host Architecture!!Host Memory!!Host Interconnect!!CPUs per Host!!GPUs per Host!!Memory per GPU
+
!GPU!!Host Quantity!!Host Architecture!!Host Memory!!Host Interconnect!!CPUs per Host!!CPUS per Socket!!GPUs per Host!!CPUs per GPU!!Memory per GPU!!SLURM Feature!!GRES GPU type
 +
|-
 +
| style="width: 14%;"|GeForce 1080Ti||1||Intel Haswell||128 GB||FDR IB||28||14||2||14||11GB||n/a||geforce
 +
|-
 +
| style="width: 14%;"|GeForce 2080Ti||32||Intel Skylake||187 GB||EDR IB||32||16||8||4||11GB||2080ti||geforce
 
|-
 
|-
| style="width: 12%;"|Tesla K80||80||20||INTEL E5-2683||128 GB||FDR IB||28||4||12GB
+
| style="width: 14%;"|GeForce 2080Ti||38||Intel Cascade Lake||187 GB||EDR IB||32||16||8||4||11GB||2080ti||geforce
 
|-
 
|-
| style="width: 12%;"|GeForce 1080Ti||2||1||INTEL E5-2683||128 GB||FDR IB||28||2||11GB
+
| style="width: 14%;"|Quadro RTX 6000||6||Intel Cascade Lake||187 GB||EDR IB||32||16||8||4||23GB||rtx6000||quadro
 
|-
 
|-
| style="width: 12%;"|GeForce 2080Ti||104||13||INTEL Gold 6142||192 GB||EDR IB||32||8||11GB
+
| style="width: 14%;"|NVIDIA A100 ||140||AMD EPYC ROME||2 TB||HDR IB||128||16||8||32||80GB||a100||a100 - changed to 'a100' at 8:00am on 8/30/21
 
|}
 
|}
 
|}
 
|}
  
== Compile CUDA Enabled Programs ==
+
For a list of additional node features, see the [[Available Node Features]] page.
 +
 
 +
To select a specific type of GPU within a partition please use either a SLURM constraint (e.g. --constraint=rtx6000) or a GRES with the needed GPU type (--gres or --gpu=a100:1). See more examples below.
  
To compile CUDA programs, please refer to [[Nvidia CUDA Toolkit]]
+
= Compiling CUDA Enabled Programs =
== GPU Use Policy ==
 
  
'''Warning''':
+
The most direct way to develop a custom GPU accelerated algorithm is with the CUDA programming, please refer to the [[Nvidia CUDA Toolkit]] page. The current CUDA environment is cuda/11. However, C++ or Python packages numba and PyCuda are other ways to program GPU algorithms.
* GPUs are allocated only via the investment QOS. There is no burst QOS in the gpu partition. There are few GPUs on HiPerGator because of the high cost of GPU cards, so there is no spare capacity. Purchased GPUs need to be available for users who invested into GPU resources.
 
* Time Limit for the gpu partition is 7 days (at most <code>#SBATCH --time=7-00:00:00</code>) to increase the availability of GPU resources.
 
  
===Interactive Access (SLURM)===
+
= Slurm and GPU Use =
In order to request interactive access to a GPU under SLURM, use commands similar to those that follow.
 
  
:'''•''' To request access to one GPU (of any type) for a default 10-minute session:
+
== Policy ==
::<source lang=bash>srun -p gpu --gres=gpu:1 --pty -u bash -i</source>
+
* After purchase, NGU allocations are included in your groups resources (quality of service).
:'''•''' To request access to two Tesla GPUs on a single node for a 1-hour session:
+
* To increase the availability of GPU resources, the time limit for the gpu partition is 7-days (at most <code>#SBATCH --time=7-00:00:00</code>). If you have a workload requiring more time, please create a [https://support.rc.ufl.edu/enter_bug.cgi help request].
::<source lang=bash>srun -p gpu --gres=gpu:tesla:2 --time=01:00:00 --pty -u bash -i</source>
 
  
If no units are accessible, your request will be queued and your connection established once the next GPU becomes available. Otherwise, you may choose to try connecting again at a later time. If you have requested for a longer time than is needed, please be sure to end your session so that the GPU will be available for other users.
+
==Interactive Access ==
 +
In order to request interactive command line access to a GPU under SLURM, use commands similar to these:
  
=== Batch Jobs (SLURM) ===
+
*To request access to one GPU (of any type) for a default 1 hour session:
For batch jobs, to request GPU resources, use lines similar to the following in your submission script.
+
srun -p gpu --gpus=1 --pty -u bash -i
 +
*To request access to two A100 GPUs on a single node for a 3-hour session with 300gb RAM:
 +
srun -p gpu --nodes=1 --gpus=a100:2 --time=03:00:00 --mem=300gb  --pty -u bash -i
 +
*To request access to two GeForce GPUs with multiple CPUs:
 +
srun -p gpu --nodes=1 --gpus=geforce:2 --time=01:00:00 --ntasks=1 --cpus-per-task=8 --mem 300gb  --pty -u bash -i
  
:'''•''' In this example, two Tesla GPUs on a single server (--nodes defaults to "1") will be allocated to the job:
+
Interactive sessions are limited to 12 hours.
<source lang=bash>
+
 
 +
==Open On Demand Access ==
 +
To access GPUs using Open-On-Demand, please check the form for your application.  If your application supports multiple GPU types, choose the GPU partition and specify number of GPUs and type:
 +
 
 +
*To request access to one GPU (of any type, use this gres string):
 +
gpu:1
 +
 
 +
*To request multiple GPUs (of any type, use this gres string were n is the number of GPUs you need):
 +
gpu:n
 +
 
 +
*To request a specific type of GPU, use this gres string (requesting geforce GPUs in this example):
 +
gpu:geforce:1
 +
 
 +
*To request a A100 GPU, use this gres string:
 +
gpu:a100:1
 +
 
 +
== Batch Jobs ==
 +
For batch jobs, to request GPU resources, use lines similar to the following:  
 +
 
 +
*In this example, two A100 GPUs on a single server (--nodes defaults to "1") will be allocated to the job:
 +
<pre>
 
#SBATCH --partition=gpu
 
#SBATCH --partition=gpu
#SBATCH --gres=gpu:tesla:2
+
#SBATCH --gpus=a100:2
</source>
+
</pre>
  
===Exclusive Mode===
+
*In this example, two GeForce GPUs on a single server (--nodes defaults to "1") will be allocated to the job:
The GPUs are configured to run in '''exclusive''' mode.  This means that the gpu driver will only allow one process at a time to access the GPU. If GPU 0 is in use and your application tries to use it, it will simply block. If your application does not call cudaSetDevice(), the CUDA runtime should assign it to a free GPU.  Since everyone will be accessing the GPUs through the batch system, there should be no over-subscription of the GPUs.
+
<pre>
 +
#SBATCH --partition=gpu
 +
#SBATCH --gpus=geforce:2
 +
</pre>
 +
 
 +
 
 +
Alternatively, use '<code>--gres=gpu:1</code>' or '<code>--gres=gpu:geforce:1</code>' format. Note, if '--gpus=' format is used SLURM will not provide the data on GPU usage to slurmInfo and those GPUs will not be shown in slurmInfo output.
 +
 
 +
If no GPUs are available, your request will be queued and your connection established once the next GPU becomes available. Otherwise, you may cancel your job and try lowering requested resources. If you have requested a longer time than is needed, please be sure to end your session so that the GPU will be available for other users.
 +
 
 +
== SLURM Options for A100 GPUs ==
 +
To use A100 GPUs for interactive sessions or batch jobs, please use one of the following SLURM parameters:
 +
<pre>
 +
--partition=gpu
 +
--gpus=a100:2
 +
</pre>
  
 
== Job Script Examples ==
 
== Job Script Examples ==
===Hybrid MPI/Threaded===
+
===MPI Parallel===
  
This is a sample script for a hybrid MPI/threaded Gromacs job requesting and using GPUs under SLURM:
+
This is a sample script for MPI parallel VASP job requesting and using GPUs under SLURM:
  
<source lang=bash>
+
<pre>
 
#!/bin/bash
 
#!/bin/bash
#SBATCH --job-name=gromacs_gpu
+
#SBATCH --job-name=vasptest
#SBATCH --output=gromacs_%j.out
+
#SBATCH --output=vasp.out
#SBATCH --error=gromacs_%j.err
+
#SBATCH --error=vasp.err
 +
#SBATCH --mail-type=ALL
 +
#SBATCH --mail-user=email@ufl.edu
 +
#SBATCH --nodes=1
 +
#SBATCH --ntasks=8
 +
#SBATCH --cpus-per-task=1
 +
#SBATCH --ntasks-per-node=8
 +
#SBATCH --ntasks-per-socket=4
 +
#SBATCH --mem-per-cpu=7000mb
 +
#SBATCH --distribution=cyclic:cyclic
 
#SBATCH --partition=gpu
 
#SBATCH --partition=gpu
#SBATCH --mail-type=END,FAIL
+
#SBATCH --gres=gpu:geforce:4
#SBATCH --mail-user=user@some.domain.com
+
#SBATCH --time=00:30:00
#SBATCN --nodes=1
 
#SBATCH --ntasks=2
 
#SBATCH --cpus-per-task=7
 
#SBATCH --ntasks-per-socket=1
 
#SBATCH --mem-per-cpu=2600mb
 
#SBATCH --distribution=cyclic:block
 
#SBATCH --gres=gpu:tesla:2
 
#SBATCH --time=6:00:00
 
  
 
echo "Date      = $(date)"
 
echo "Date      = $(date)"
Line 94: Line 139:
 
echo "Directory = $(pwd)"
 
echo "Directory = $(pwd)"
  
module load intel/2017 openmpi/3.0.0 cuda/9.1.85 gromacs/2018
+
module purge
 
+
module load cuda/10.0.130  intel/2018  openmpi/4.0.0 vasp/5.4.4
GROMACS=gmx
 
export OMP_NUM_THREADS=7
 
  
 
T1=$(date +%s)
 
T1=$(date +%s)
srun --mpi=pmix $GROMACS mdrun -v -deffnm topol
+
srun --mpi=pmix_v3 vasp_gpu
 
T2=$(date +%s)
 
T2=$(date +%s)
  
 
ELAPSED=$((T2 - T1))
 
ELAPSED=$((T2 - T1))
 
echo "Elapsed Time = $ELAPSED"
 
echo "Elapsed Time = $ELAPSED"
</source>
+
</pre>

Latest revision as of 21:35, 23 September 2022


Interactive Jobs in the GPU partition are limited to 12 hrs

Normalized Graphics Processor Units (NGUs) include all of the infrastructure (memory, network, rack space, cooling) necessary for GPU-accelerated computation. Each NGU is equivalent to 1 GPU presently, however newer GPUs such as the A100s may require more than 1 NGU to access in the future.

Researchers can add NGUs to their allocations by filling out the Purchase Form or requesting a Trial Allocation.

GPU-enabled Services

Types of GPUs are listed below. Two partitions contain GPUs - the hwgui partition for visualization and the gpu partition for general computation.

Hardware Accelerated GUI

GPUs in these servers are used to accelerate rendering for graphical applications. These servers are in the SLURM "hwgui" partition. Refer to the Hardware Accelerated GUI Sessions page for more information on available resources and usage.

GPU Assisted Computation

A number of high performance applications installed on HiPerGator implement GPU-accelerated computing functions via CUDA to achieve significant speed-up over CPU calculations. These servers are in the SLURM "gpu" partition (--partition=gpu).

Hardware Specifications for the GPU Partition

We have the following types of NVIDIA GPU nodes available in the "gpu" partition:

GPU Host Quantity Host Architecture Host Memory Host Interconnect CPUs per Host CPUS per Socket GPUs per Host CPUs per GPU Memory per GPU SLURM Feature GRES GPU type
GeForce 1080Ti 1 Intel Haswell 128 GB FDR IB 28 14 2 14 11GB n/a geforce
GeForce 2080Ti 32 Intel Skylake 187 GB EDR IB 32 16 8 4 11GB 2080ti geforce
GeForce 2080Ti 38 Intel Cascade Lake 187 GB EDR IB 32 16 8 4 11GB 2080ti geforce
Quadro RTX 6000 6 Intel Cascade Lake 187 GB EDR IB 32 16 8 4 23GB rtx6000 quadro
NVIDIA A100 140 AMD EPYC ROME 2 TB HDR IB 128 16 8 32 80GB a100 a100 - changed to 'a100' at 8:00am on 8/30/21

For a list of additional node features, see the Available Node Features page.

To select a specific type of GPU within a partition please use either a SLURM constraint (e.g. --constraint=rtx6000) or a GRES with the needed GPU type (--gres or --gpu=a100:1). See more examples below.

Compiling CUDA Enabled Programs

The most direct way to develop a custom GPU accelerated algorithm is with the CUDA programming, please refer to the Nvidia CUDA Toolkit page. The current CUDA environment is cuda/11. However, C++ or Python packages numba and PyCuda are other ways to program GPU algorithms.

Slurm and GPU Use

Policy

  • After purchase, NGU allocations are included in your groups resources (quality of service).
  • To increase the availability of GPU resources, the time limit for the gpu partition is 7-days (at most #SBATCH --time=7-00:00:00). If you have a workload requiring more time, please create a help request.

Interactive Access

In order to request interactive command line access to a GPU under SLURM, use commands similar to these:

  • To request access to one GPU (of any type) for a default 1 hour session:
srun -p gpu --gpus=1 --pty -u bash -i
  • To request access to two A100 GPUs on a single node for a 3-hour session with 300gb RAM:
srun -p gpu --nodes=1 --gpus=a100:2 --time=03:00:00 --mem=300gb  --pty -u bash -i
  • To request access to two GeForce GPUs with multiple CPUs:
srun -p gpu --nodes=1 --gpus=geforce:2 --time=01:00:00 --ntasks=1 --cpus-per-task=8 --mem 300gb  --pty -u bash -i

Interactive sessions are limited to 12 hours.

Open On Demand Access

To access GPUs using Open-On-Demand, please check the form for your application. If your application supports multiple GPU types, choose the GPU partition and specify number of GPUs and type:

  • To request access to one GPU (of any type, use this gres string):
gpu:1
  • To request multiple GPUs (of any type, use this gres string were n is the number of GPUs you need):
gpu:n
  • To request a specific type of GPU, use this gres string (requesting geforce GPUs in this example):
gpu:geforce:1
  • To request a A100 GPU, use this gres string:
gpu:a100:1

Batch Jobs

For batch jobs, to request GPU resources, use lines similar to the following:

  • In this example, two A100 GPUs on a single server (--nodes defaults to "1") will be allocated to the job:
#SBATCH --partition=gpu
#SBATCH --gpus=a100:2
  • In this example, two GeForce GPUs on a single server (--nodes defaults to "1") will be allocated to the job:
#SBATCH --partition=gpu
#SBATCH --gpus=geforce:2


Alternatively, use '--gres=gpu:1' or '--gres=gpu:geforce:1' format. Note, if '--gpus=' format is used SLURM will not provide the data on GPU usage to slurmInfo and those GPUs will not be shown in slurmInfo output.

If no GPUs are available, your request will be queued and your connection established once the next GPU becomes available. Otherwise, you may cancel your job and try lowering requested resources. If you have requested a longer time than is needed, please be sure to end your session so that the GPU will be available for other users.

SLURM Options for A100 GPUs

To use A100 GPUs for interactive sessions or batch jobs, please use one of the following SLURM parameters:

--partition=gpu
--gpus=a100:2

Job Script Examples

MPI Parallel

This is a sample script for MPI parallel VASP job requesting and using GPUs under SLURM:

#!/bin/bash
#SBATCH --job-name=vasptest
#SBATCH --output=vasp.out
#SBATCH --error=vasp.err
#SBATCH --mail-type=ALL
#SBATCH --mail-user=email@ufl.edu
#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --cpus-per-task=1
#SBATCH --ntasks-per-node=8
#SBATCH --ntasks-per-socket=4
#SBATCH --mem-per-cpu=7000mb
#SBATCH --distribution=cyclic:cyclic
#SBATCH --partition=gpu
#SBATCH --gres=gpu:geforce:4
#SBATCH --time=00:30:00

echo "Date      = $(date)"
echo "host      = $(hostname -s)"
echo "Directory = $(pwd)"

module purge
module load cuda/10.0.130  intel/2018  openmpi/4.0.0 vasp/5.4.4

T1=$(date +%s)
srun --mpi=pmix_v3 vasp_gpu
T2=$(date +%s)

ELAPSED=$((T2 - T1))
echo "Elapsed Time = $ELAPSED"