Checking and Using Secondary Resources

Back to Account and QOS limits under SLURM

Using the resources from a secondary group

By default, when you submit a job on HiPerGator, it will use the resources from your primary group. You can easily see your primary and secondary groups with the id command:

[agator@login4 ~]$ id
uid=12345(agator) gid=12345(gator-group) groups=12345(gator-group),12346(second-group),12347(third-group)

As shown above, our fictional user agator's primary group is gator-group and they also have secondary groups of second-group and third-group.

To use the resources of one of their secondary groups rather than their primary group, agator can use the --account and --qos flags, in the submit script, in the sbatch command or in the boxes in the Open on Demand interface. For example, to use the orange-group they could:

How to set account and qos options to use resources from a secondary group in Open on Demand

1. In a submit script, add these lines:
#SBATCH --account=second-group
#SBATCH --qos=second-group
2. In the sbatch command: sbatch --account=second-group --qos=second-group my_script.sh
3. Using Open on Demand:
4. Note: Jupyterhub can only use your primary group's resources and cannot be used for accessing secondary group resources.
5. Note: To use a secondary group's Burst QOS the --account= parameter is still 'second-group', while the --qos= parameter is 'second-group-b'. The QOS is different, but the account remains the same. This may make more sense by viewing the output of the showAssoc command in the See your associations section immediately below.

See your associations

On the command line, you can view your SLURM associations with the showAssoc command:

$ showAssoc <username>

Example: $ showAssoc magitz output:

              User    Account   Def Acct   Def QOS                                      QOS 
------------------ ---------- ---------- --------- ---------------------------------------- 
magitz                zoo6927      ufhpc     ufhpc zoo6927,zoo6927-b                        
magitz                  ufhpc      ufhpc     ufhpc ufhpc,ufhpc-b                            
magitz                 soltis      ufhpc    soltis soltis,soltis-b                          
magitz                  borum      ufhpc     borum borum,borum-b   

The output shows that the user magitz has four account associations and 8 different QOSes. By convention, a user's default account is always the account of their primary group. Additionally, their default QOS is the investment (high-priority) QOS.

If a user does not explicitly request a specific account and QOS, the user's default account and QOS will be assigned to the job. If the user magitz wanted to use the borum group's account - which he has access by virtue of the borum account association - he would specify the account and the chosen QOS in his batch script as follows:

 #SBATCH  --account=borum
 #SBATCH  --qos=borum

Or, for the burst QOS:

 #SBATCH  --account=borum
 #SBATCH  --qos=borum-b

Note that both --account and --qos must be specified. Otherwise scheduler will assume the default ufhpc account is intended, and neither the borum nor borum-b QOSes will be available to the job. Consequently, scheduler would deny the submission. In addition, you cannot mix and match resources from different allocations.

These sbatch directives can also be given as command line arguments to srun. For example:

$ srun --account=borum --qos=borum-b <example_command>