Difference between revisions of "Choosing QOS for a Job"
Jump to navigation
Jump to search
(Created page with "==Choosing QOS for a Job== When choosing between the high-priority investment QOS and the 9x larger low-priority burst QOS, you should start by considering the overall resourc...") |
|||
| (3 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| − | + | Back to [[Account and QOS limits under SLURM]]<br> | |
When choosing between the high-priority investment QOS and the 9x larger low-priority burst QOS, you should start by considering the overall resource requirements for the job. For smaller allocations the investment QOS may not be large enough for some jobs, whereas for other smaller jobs the wait time in the burst QOS could be too long. In addition, consider the current state of the account you are planning to use for your job. | When choosing between the high-priority investment QOS and the 9x larger low-priority burst QOS, you should start by considering the overall resource requirements for the job. For smaller allocations the investment QOS may not be large enough for some jobs, whereas for other smaller jobs the wait time in the burst QOS could be too long. In addition, consider the current state of the account you are planning to use for your job. | ||
| Line 13: | Line 13: | ||
$ module load ufrc | $ module load ufrc | ||
$ slurmInfo | $ slurmInfo | ||
| − | for the primary account or | + | for the primary account (group name) or |
$ slurmInfo <account> | $ slurmInfo <account> | ||
| − | for another account | + | for another account (group name) |
'''Example:''' <code>$ slurmInfo ufgi</code>: | '''Example:''' <code>$ slurmInfo ufgi</code>: | ||
| Line 31: | Line 31: | ||
---------------------------------------------------------------------- | ---------------------------------------------------------------------- | ||
HiPerGator Utilization | HiPerGator Utilization | ||
| − | + | CPUs: Used/Total MEM(GB): Used/Total GPUs: Used/Total | |
| − | ---------------------------------------------------------------------- | + | -------------------------------------------------------------------------- |
| − | + | Total : 47150/79620 59% 250529/600009 41% 512/1234 41% | |
| − | ---------------------------------------------------------------------- | + | -------------------------------------------------------------------------- |
| + | HiPerGator GPU Utilization by type | ||
| + | Partition A100 geforce quadro | ||
| + | -------------------------------------------------------------------------- | ||
| + | gpu : 467/664 70% 43/528 8% 0/0 0% | ||
| + | hwgui : 0/0 0% 0/0 0% 2/42 4% | ||
| + | -------------------------------------------------------------------------- | ||
* Burst QOS uses idle cores at low priority with a 4-day time limit | * Burst QOS uses idle cores at low priority with a 4-day time limit | ||
| + | * Duplicate partition(s): hpg-ai / gpu | ||
| + | * Reserved nodes excluded from HPG utilization metrics | ||
Run 'slurmInfo -h' to see all available options | Run 'slurmInfo -h' to see all available options | ||
| + | |||
</pre> | </pre> | ||
| − | The output shows that the investment QOS for the <code style="color:black; background:WhiteSmoke; border:1px solid gray;">ufgi</code> account is actively used. Since 100 CPU cores out of 150 available are used only 50 cores are available. In the same vein since 280GB out of 527GB in the investment QOS are used 247GB are still available. The <code>ufgi-b</code> burst QOS is unused. | + | The output shows that the investment QOS for the <code style="color:black; background:WhiteSmoke; border:1px solid gray;">ufgi</code> account is actively used. Since 100 CPU cores out of 150 available are used only 50 cores are available. In the same vein since 280GB out of 527GB in the investment QOS are used 247GB are still available. The <code>ufgi-b</code> burst QOS is unused. The total HiPerGator use is 59% of all CPU cores and 41% of all memory on compute nodes, which means that there is some available capacity from which burst resources can be drawn. In this case a job submitted to the <code style="color:black; background:WhiteSmoke; border:1px solid gray;">ufgi-b</code> QOS would likely take a long time to start. If the overall utilization is below 80% it would be easier to start a burst job within a reasonable amount of time. When the HiPerGator load is high, or if the burst QOS is actively used, the investment QOS is more appropriate for a smaller job. The output also includes GPU utilization per GPU type cluster-wide. |
Latest revision as of 14:45, 1 July 2024
Back to Account and QOS limits under SLURM
When choosing between the high-priority investment QOS and the 9x larger low-priority burst QOS, you should start by considering the overall resource requirements for the job. For smaller allocations the investment QOS may not be large enough for some jobs, whereas for other smaller jobs the wait time in the burst QOS could be too long. In addition, consider the current state of the account you are planning to use for your job.
For any individual jobs submitted to the Burst QOS we do not guarantee that they will ever start, although historical data shows that burst jobs do start and provide significant additional throughput to groups that use them correctly as 'long queues' i.e.
- Submit only non-time-critical jobs to the Burst QOS.
- Parallelize analyses to make sure they can run within the 4-day window.
- Let the scheduler take its time to find unused resources to run burst jobs.
To show the status of any SLURM account as well as the overall usage of HiPerGator resources, use the following command from the UFRC module:
$ module load ufrc $ slurmInfo
for the primary account (group name) or
$ slurmInfo <account>
for another account (group name)
Example: $ slurmInfo ufgi:
----------------------------------------------------------------------
Allocation summary: Time Limit Hardware Resources
Investment QOS Hours CPU MEM(GB) GPU
----------------------------------------------------------------------
ufgi 744 150 527 0
----------------------------------------------------------------------
CPU/MEM Usage: Running Pending Total
CPU MEM(GB) CPU MEM(GB) CPU MEM(GB)
----------------------------------------------------------------------
Investment (ufgi): 100 280 0 0 100 280
----------------------------------------------------------------------
HiPerGator Utilization
CPUs: Used/Total MEM(GB): Used/Total GPUs: Used/Total
--------------------------------------------------------------------------
Total : 47150/79620 59% 250529/600009 41% 512/1234 41%
--------------------------------------------------------------------------
HiPerGator GPU Utilization by type
Partition A100 geforce quadro
--------------------------------------------------------------------------
gpu : 467/664 70% 43/528 8% 0/0 0%
hwgui : 0/0 0% 0/0 0% 2/42 4%
--------------------------------------------------------------------------
* Burst QOS uses idle cores at low priority with a 4-day time limit
* Duplicate partition(s): hpg-ai / gpu
* Reserved nodes excluded from HPG utilization metrics
Run 'slurmInfo -h' to see all available options
The output shows that the investment QOS for the ufgi account is actively used. Since 100 CPU cores out of 150 available are used only 50 cores are available. In the same vein since 280GB out of 527GB in the investment QOS are used 247GB are still available. The ufgi-b burst QOS is unused. The total HiPerGator use is 59% of all CPU cores and 41% of all memory on compute nodes, which means that there is some available capacity from which burst resources can be drawn. In this case a job submitted to the ufgi-b QOS would likely take a long time to start. If the overall utilization is below 80% it would be easier to start a burst job within a reasonable amount of time. When the HiPerGator load is high, or if the burst QOS is actively used, the investment QOS is more appropriate for a smaller job. The output also includes GPU utilization per GPU type cluster-wide.