CARC Grafana Dashboards

Dashboard-as-code for CARC (Center for Advanced Research Computing, UNM) using Grafonnet.

Environment

These dashboards target the PLG stack running as rootful podman quadlets on vortex.alliance.unm.edu:

• Grafana 13.0.1 — OIDC login via Keycloak (carc realm), reverse-proxied by nginx at https://vortex.alliance.unm.edu/.
• Prometheus v3 — http://prometheus:9090 inside the monitoring podman network, 90d / 800GB retention, remote-write enabled.
• Loki 3 — http://loki:3100, currently ingesting vortex's own systemd journal via Alloy (cluster/node log shipping from compute nodes is planned but not yet wired up — see loki-ingest.conf / mTLS TODO in nginx config).
• snmp_exporter — UPS monitoring (Mitsubishi + Liebert units) via job="snmp_ups" in Prometheus.

Config lives under /etc/monitoring/, data under /var/lib/monitoring/ on that host; dashboards from this repo are provisioned into Grafana from two folders, each its own file provider in default.yml:

• /var/lib/monitoring/grafana/dashboards/carc → CARC folder (provider carc) — ups-power, researcher-overview, public-status. Left on default/inherited folder permissions, so anyone with the Viewer org role can see them with no per-dashboard permission step.
• /var/lib/monitoring/grafana/dashboards/carc-admin → CARC Admin folder (provider carc-admin) — admin-overview. This one has the Viewer row deliberately removed from the folder's permissions, restricting it to Editor+.

Both are separate from the slurm-*.json dashboards, which live directly under /var/lib/monitoring/grafana/dashboards (the default provider, root folder) and aren't authored here.

make install (see Usage below) pushes compiled dashboard JSON onto the two provisioned directories above. Everything else on that host (the folders/permissions themselves, provisioning config, the box in general) is still Ansible-managed — ask before assuming this repo should take over more than dashboard content.

Known Prometheus scrape jobs today (/etc/monitoring/prometheus/prometheus.yml):

job	targets	notes
snmp_ups	2 UPS units via snmp_exporter	role=ups, vendor=Mitsubishi|Liebert
slurm_exporter	easley-sn, hopper-sn (:9341)	cluster=easley|hopper
node	easley login + controller nodes only so far	more clusters/nodes pending
prometheus	self

The /var/lib/monitoring/grafana/dashboards/slurm-*.json dashboards already on the box are the upstream slurm-exporter defaults, not authored here.

Layout

dashboards/    one .jsonnet file per dashboard
lib/           shared helpers: datasource variable (datasource.libsonnet),
               common panel builders (panels.libsonnet), slurm_exporter
               cluster/partition template variables (slurm.libsonnet)
vendor/        jsonnet-bundler dependencies (gitignored, run `make vendor`)
build/         compiled dashboard JSON (gitignored, run `make`)

Usage

make vendor   # jb install — fetch grafonnet into vendor/
make          # compile dashboards/*.jsonnet -> build/*.json
make fmt      # jsonnetfmt -i
make install  # build + copy build/*.json onto vortex at the provisioned paths (needs root)

make install (scripts/install-dashboards.sh) refreshes dashboard JSON in the live carc / carc-admin provisioning directories on vortex — it's the repeatable counterpart to the one-time runbook-carc-folder-split.sh migration, which has to be run first to create those folders and their permissions. Grafana's file provider polls every 30s, so no restart is needed for a content-only update.

Every dashboard exposes a datasource template variable (type datasource, query prometheus) and references it as ${datasource} on every panel target, matching the pattern used by the existing slurm dashboards — this keeps dashboards portable instead of hardcoding a datasource uid.

Dashboards

• public-status — unauthenticated status board for the website/lobby. Aggregate-only cluster stats (nodes, CPU/GPU %, jobs running/pending) for Easley and Hopper. No usernames, accounts, or per-job detail. Hidden timepicker/datasource variable, 30s refresh — meant to be dropped straight onto a screen with no interaction.
• researcher-overview — for any logged-in user (default Viewer role). Partition load, running/pending queue by user (the same thing squeue already shows any cluster user, not a new disclosure), node status, and login-node storage capacity, filterable by cluster/partition.
• admin-overview — for CARC ops staff (Keycloak systems / monitor-admins groups, mapped to Grafana Editor/Admin below). Scheduler internals, exporter self-health, per-account fairshare/usage, and full login/controller node + filesystem health. Links to ups-power.
• ups-power — UPS battery/power status for the datacenter units, metrics confirmed live against the snmp_exporter endpoint on vortex.

Access tiers

Grafana OIDC (grafana.ini) already maps Keycloak groups to roles: monitor-admins → Admin, systems → Editor, everyone else → Viewer. That covers the researcher/admin split for free — admin-overview just needs to land in a folder whose permissions require Editor+, and researcher-overview in a folder any Viewer can read. Resolved as the CARC / CARC Admin folder split described above.

public-status needs actual unauthenticated access, which this repo doesn't and can't provision — it's an instance-level Grafana setting ([auth.anonymous] in grafana.ini, currently enabled = false) or a per-dashboard share via Grafana's Public Dashboards feature. Either is a deploy/infra change on vortex (Ansible-managed, per above), not something the compiled dashboard JSON controls — flag it to whoever owns that host rather than assuming this repo should flip it.