View source for NCU and QOS limits under SLURM

[[Category:SLURM]]
{{HPG2}}
=Account and QOS Use=
In the HiperGator 2.0 SLURM implementation a job's priority and resource limits are determined by its QOS (Quality of Service).  Every group with an HPG2 investment will have a corresponding SLURM '''account'''. Each SLURM account has two associated QOSs - the normal or high-priority QOS and a low-priority '''burst''' QOS. In turn, each user who is a member of '''vested''' group has a SLURM association.  It is the association which determines which QOSs are a available to a user.  In addition to the primary group association, users who have secondary group membership (in the Unix/Linux sense) will also have secondary group associations which will afford them access to the normal and burst QOSs of that group's account.

To see your SLURM associations, you can run the ''showAssoc'' command.  For example,

<source lang=bash>
[root@slurm1 ~]# showAssoc magitz
              User    Account   Def Acct   Def QOS                                      QOS 
------------------ ---------- ---------- --------- ---------------------------------------- 
magitz                zoo6927      ufhpc     ufhpc zoo6927,zoo6927-b                        
magitz                  ufhpc      ufhpc     ufhpc ufhpc,ufhpc-b                            
magitz                 soltis      ufhpc    soltis soltis,soltis-b                          
magitz                  borum      ufhpc     borum borum,borum-b   
</source>                

The output above shows that the user ''magitz'' has four SLURM associations and thus, access to 8 different QOSs.  By convention, a user's default account is always the account of their primary group.  Additionally, their default QOS is the normal (high-priority) QOS of their default QOS.  If a user does not explicitly request a specific account and QOS, the user's default account and QOS will be assigned to the job.

However, if the user ''magitz'' wanted to use the borum group's burst QOS (to which he has access by virture of the borum account association), he would specify the account as QOS in his batch script as follows.

<source lang=bash>
    #SBATCH  --account=borum
    #SBATCH  --qos=borum-b
</source>

Note that both must be specified.  Otherwise, SLURM will assume the default '''ufhpc''' account is intended and the '''borum-b''' QOS will not be available to the job. Consequently, SLURM would deny the submission.

These sbatch directives can also be given as command line arguments to "srun" as in,

<source lang=bash>
    srun --account=borum --qos=borum-b some_command
</source>

==QOS Limits==

SLURM refers to the resources (NCUs (cores), Memory (RAM), accelerators, software licenses, etc.) as '''Trackable Resources''' (TRES).  The TRES available to a given group are determined by the group's investments and are limited by parameters assigned to the QOS.  

Continuing with the example above, we can see the trackable resource limits placed on the borum group's primary QOS by running,

<source lang=bash>
# showQos borum
                Name                          Descr                                       GrpTRES  GrpCPUs 
-------------------- ------------------------------ --------------------------------------------- -------- 
borum                borum qos                      cpu=41,mem=125952,gres/gpu=0,gres/mic=0             41 
</source>

from which we can see that when submitting jobs under the borum group's primary QOS, users have access to a total of 41 cores, 125 GB of RAM with no access to accelerators (GPUs, MICs). These resource will be shared among all members of the borum group running jobs under the primary QOS.  Similarly, we can see the trackable resources available under the burst QOS with,

<source lang=bash>
[root@slurm1 bin]# showQos borum-b
                Name                          Descr                                       GrpTRES  GrpCPUs 
-------------------- ------------------------------ --------------------------------------------- -------- 
borum-b              borum burst qos                cpu=369,mem=1133568,gres/gpu=0,gres/mic=0          369 
</source>

There are additional limits and parameters associated with QOSs in addition to the TRES limits.  Among them are the maximum wall time available under the QOS and the base priority assigned to the job.  We can see this with,

<source lang=bash>
# sacctmgr show qos format="name%-20,Description%-30,priority,maxwall" borum borum-b
                Name                          Descr   Priority     MaxWall 
-------------------- ------------------------------ ---------- ----------- 
borum                borum qos                           36000 31-00:00:00 
borum-b              borum burst qos                       900  4-00:00:00 
</source>

From which we see that normal and burst QOS jobs are limited to a maximum duration of 31 and 4 days, respectively.  Additionally, the base priority of a burst QOS job is 1/40th that of a normal QOS job.  It is important to remember that the base priority is only one component of the jobs overall priority and that the priority will change over time as the job waits in the queue.

By policy, the burst QOS cpu and memory limits are always nine times (9x) those of the normal QOS and are intended to allow groups to take advantage of unused resources beyond those that they have purchased for short periods of time.
 
== Cores (CPUs) and Memory (RAM)==
The resources of any computing device are limited.  This is to say that the number of cores, the amount of memory, the memory bandwidth, the I/O bandwidth, etc. - all are limited.  Once you have used up all of the available cores on a machine, it is fully consumed and useless to other users.  This is true whether you use 1 byte of RAM or all of the RAM on the machine.  Likewise, if your application uses all of the memory available to the machine, whether it uses 1 core or all the cores, the machine is consumed and unavailable to others.  Because of this, we place limits on both the number of cores available to a group (based on the NCU investments) '''and''' the amount of memory available to a group (NCUs x 3GB).  

Why do we limit the amount of memory you can use?  Because it is necessary to be fair to all investors.  Consider a system where a PI has invested in 10 NCUs.  His TRES cpu limit will be "10".  With no memory limits he could submit ten jobs requesting 1 cpu and 120GB each.  Because each machine only has about 120GB available for applications, each job would be started on a separate machine leaving no memory available for any other jobs to run on the machine.  The result.  The user has invested in 10 NCUS but is consuming 320.  Such a scenario is not tenable and would quickly result in a grossly unfair allocation of resources.  Thus, we place limits on both CPUs and memory.

The total memory limit is calculated as 'QOS NCU * 3 GB'. For example, a primary QOS of 30 NCUs will have a group memory limit of 90gb while the burst qos for that group will be equal to '30 * 3gb * 9 = 810gb'. If the group memory limit is reached you will see a '(QOSGrpMemLimit)' status in the 'NODELIST(REASON)' column of the squeue output. For example,

<source lang=bash>
 squeue | grep MemLimit | head -n 1
            123456    bigmem test_job   jdoe PD       0:00      1 (QOSGrpMemLimit)
</source>

The above message can only be seen in the output of the 'squeue' command and does not interfere with job submission, but the job will remained queued until the group falls below its memory limit. 

If the submitted job is so large that its resource request falls outside the total resource limit of the requested QOS, SLURM will refuse the job submission altogether and produce the following error

<source lang=bash>
;sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)
</source>

=Examples=

A hypothetical group ($GROUP in the examples below) has an investment of 42 NCUs. That's the group's so-called ''soft limit'' for  HiPerGator jobs in the main qos for up to 744 hours at high priority. The hard limit accessible through the so-called ''burst qos'' is +9 times that giving a group potentially a total of 10x the invested resources i.e. 420 NCUs with burst qos providing 378 NCUs of that capacity for up to 96 hours at low priority.

Let's test:

[marvin@gator ~]$ srun --mem=126gb --pty bash -i

srun: job 123456 queued and waiting for resources

<Looks good, let's terminate the request with Ctrl+C>

^C

srun: Job allocation 123456 has been revoked

srun: Force Terminated job 123456


On the other hand, going even 1gb over that limit results in the already encountered job limit error

[marvin@gator ~]$ srun --mem=127gb --pty bash -i

srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits



At this point the group can try using the 'burst' QOS with

#SBATCH --qos=$GROUP-b


Let's test:

[marvin@gator3 ~]$ srun -p bigmem --mem=400gb --time=96:00:00 --qos=$GROUP-b --pty bash -i

srun: job  123457 queued and waiting for resources

<Looks good, let's terminate with Ctrl+C>

^C

srun: Job allocation 123457 has been revoked

srun: Force Terminated job 123457


However, now there's the burst qos time limit to consider:

[marvin@gator ~]$ srun --mem=400gb --time=300:00:00 --pty bash -i

srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits


Let's reduce the time limit to what burst qos supports and try again:



[marvin@gator ~]$ srun --mem=400gb --time=96:00:00 --pty bash -i

srun: job  123458 queued and waiting for resources

<Looks good, let's terminate with Ctrl+C>

^C

srun: Job allocation 123458 has been revoked

srun: Force Terminated job

=Pending Job Reasons=

The following ''Reasons'' can be seen in the '<code>NODELIST(REASON)</code>' column of the job queue listing for a group when the group reaches the resource limit for the respective account/qos combination:

;QOSGrpCpuLimit
:means that all CPU cores available for the listed account within the respective QOS are in use.

;QOSGrpMemLimit
:means that all memory available for the listed account within the respective QOS as described in the previous section is in use.