HiPerGator Metrics

Accessing the HiPerGator Status Dashboard

1. You must have a valid HiPerGator account. If you need to request an account, see the Account Request page.
2. Use your browser to access https://metrics.rc.ufl.edu
3. You will be directed to the UF GatorLink login page (it's possible this step will be skipped if you have already authenticated to other UF resources)
4. Once authenticated, you will be shown a Grafana login page

5. Enter your GatorLink credentials and click Log In
6. You will be directed to the HiPerGator Status dashboard which should look like this:

If you do not land on this page, please contact Support or file a Bugzilla ticket.

Dashboard Panels Explained

5 Minute Load Average

This panel shows the 5 minute load average of each login node.
A red threshold line has been added to indicate a higher than expected load average.
Being over the threshold does not necessarily imply a problem with that node.
As with all panels, you can hover your mouse over a point on the line and it will display the value at that point.
If a value is not displayed, move your cursor slightly left or right.

Slurm Jobs Started per Minute

This panel shows the number of jobs started by the Slurm scheduler every minute.
To view the value of a specific point, you can hover your mouse over a bar.
Each bar should display a value.

GPU Availability

This panel shows how many GPUs of each product family are available for scheduling.
Please note this does not factor in GPUs that have been reserved, so the actual number available could be less.

1 Minute Load Average

This panel shows the 1 minute load average of the login nodes.
It is similar to the 5 minute average but this one can spike higher at times.
If the 1 minute load average is high, but the 5 minute average is not, then it's an indication of a short spike of activity.
If both the 1 and 5 minute averages are high, then it's possible someone has started a CPU intensive process.
There is also a threshold value set to indicate a higher than expected load.

CPU Status

This panel shows the number of CPUs that are allocated, idle and reserved.
It also displays both the current value and average value. Average is calculated based on chosen time range.
Data is collected every 15 minutes and are a snapshot of that point in time.
You can hover over the points in the line to get the value for that time.
Allocated CPUs are actively performing job functions.
Idle CPUs are available to be scheduled.
Reserved CPUs are generally unavailable for scheduling and most often are held for system maintenance.

GPU Allocation

This panel shows the allocated percentage of GPUs per product family and overall.
It also displays both the current and average allocation, calculated over the chosen time range.
The overall allocation includes all GPU families.

Number of Users

This panel shows the number of users per login node.
There is a visual threshold set at 75 users. Values over this threshold do not necessarily indicate a problem.
This panel can be used to identify an unbalanced load across the login nodes.
Often, if a node has zero or very few users, it's likely the node is being drained for maintenance purposes and/or has been removed from the pool.

General Dashboard Usage

The dashboard has several restrictions, but there are some areas that can be changed.

Changing the Time Range

You are able to change the time range of the dashboard by clicking on the box in the top right corner with the clock icon.
It is set to a default of the "Last 3 hours". When you click this, you will be presented with several preset options.
It's best to use one of the presets. Be advised, when using a longer range, the dashboard panels may be more difficult to view.

Changing the Refresh Frequency

You are able to adjust how often the dashboard panels refresh.
Click the icon in the top right with the refresh icon. It is set to a default of 5 minutes.
Be advised, making the refresh interval less than 5 minutes will increase the load on backend servers and may result in poor performance of the dashboard.
Also, some of the panels are only designed to collect data every 15 minutes, or longer, so shortening the refresh may have little to no effect.

Changing the Partition

You can select partitions of interest from the drop down menu at the top left of the dashboard.
Currently, this will only have an effect on the CPU Status panel.
You may select multiple entries from the selector, or choose All (the default).

Maximize a Panel

If you want to view only a single panel, simply click on the panel title then click View.
This will present only that panel and fill the entire dashboard making it easier to read.