Understanding the GPU monitoring dashboards

The GPU monitoring dashboards are populated by the NVIDIA DCGM Exporter, which runs on every GPU node in your cluster. This page explains what each dashboard is for and how to read its metrics. For the exact definition of any metric, see the DCGM Field Identifiers reference.

What DCGM exporter exposes

The DCGM exporter reads metrics straight from the GPU and sends them to Prometheus. It sees the GPU at the hardware level: how much of the GPU is being used, how much VRAM is occupied and health signals from the device itself. Kubernetes only knows that a pod asked for a GPU (nvidia.com/gpu); it can’t see what happens inside it. DCGM fills that gap.

Two dashboards, two perspectives

You get two dashboards, both built on the same DCGM data.

Kubernetes / GPU Resources / Nodes answers “how is each physical GPU doing?”. Metrics are grouped per node and per GPU. Use it for hardware health and per-device usage trends.
Kubernetes / GPU Resources / Pods answers “which workload is using the GPU?”. DCGM works with the kubelet to link metrics to the pod holding the GPU and groups them by namespace and pod.

To right-size a workload or find a heavy user, start with Pods. To investigate the hardware or driver, start with Nodes.

How to read utilisation

Three panels show how busy the GPU is, each as a percentage from 0 to 100:

GPU Utilization (DCGM_FI_DEV_GPU_UTIL). The share of time the GPU was doing at least some work. This is the headline “how hard is the GPU working” number, but it doesn’t tell you how much of the GPU was in use, only that it was busy. Treat it as a busy/idle signal, not a measure of intensity.
Memory Bandwidth Utilization (DCGM_FI_DEV_MEM_COPY_UTIL). The share of time GPU memory was being read or written. Sustained high values mean the workload is memory-bound: it spends more time moving data than computing. This is common with large-batch inference and some data-preprocessing.
Encoder / Decoder Utilization (DCGM_FI_DEV_ENC_UTIL, DCGM_FI_DEV_DEC_UTIL). Dedicated engines for video encode/decode. Only relevant for video pipelines; ignore them for pure ML workloads.

Rule of thumb: sustained high GPU Utilization with low Memory Bandwidth Utilization usually means the workload is compute-bound and the GPU is a good fit. The reverse suggests the GPU is waiting on memory more than it is computing.

How to read memory

Two panels track VRAM (NVIDIA calls it “framebuffer”):

GPU Memory Used (DCGM_FI_DEV_FB_USED), in MB.
GPU Memory Used % (from DCGM_FI_DEV_FB_USED and DCGM_FI_DEV_FB_FREE), as a percentage of the total.

Running out of VRAM is the most common reason a GPU workload fails. If a workload asks for more VRAM than the GPU has, it fails immediately with an out-of-memory error, there’s no swap or graceful fallback. Watching “Memory Used %” over a full workload cycle is the best way to right-size it.

Clocks

SM Clock (DCGM_FI_DEV_SM_CLOCK). The GPU’s compute clock speed.
Memory Clock (DCGM_FI_DEV_MEM_CLOCK). The GPU memory clock speed.

Normally both rise and fall with demand on their own. Clocks that stay low under load can point to thermal throttling or a power cap, worth checking alongside the health metrics below. For steady workloads, clocks are usually a secondary signal.

Hardware health

These metrics tell you the GPU hardware or driver has hit a problem. Because AWS manages the physical hardware, the fix for a persistent fault is to replace the node rather than repair anything in software.

XID Errors (DCGM_FI_DEV_XID_ERRORS). The code of the last GPU fault the driver recorded (XID is NVIDIA’s fault-coding scheme). A non-zero value means something went wrong; the code tells you whether it was minor (e.g. a workload running out of memory) or serious (e.g. a hardware fault). NVIDIA’s XID error reference explains each code.
ECC Errors (DCGM_FI_DEV_ECC_SBE_*, DCGM_FI_DEV_ECC_DBE_*). ECC (Error-Correcting Code) lets GPU memory detect and fix bit errors. Single-Bit Errors (SBE) are fixed automatically; a few over time is normal. Double-Bit Errors (DBE) can’t be fixed and usually crash the workload; even one is worth investigating. The “volatile” counter resets on power cycles; the “aggregate” counter covers the GPU’s whole lifetime.
Retired Pages (DCGM_FI_DEV_RETIRED_*). When one spot in memory hits too many ECC errors, the GPU permanently stops using it. A rising count means memory is wearing out.
Remapped Rows (DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS, DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS, DCGM_FI_DEV_ROW_REMAP_FAILURE). Newer GPUs (Ampere and later) keep spare memory rows to swap in when one wears out. Read these by severity: correctable remaps are routine wear-and-tear and need no action; a rising count of uncorrectable remaps means the memory is degrading and the node should be scheduled for replacement; a remap failure means the GPU has exhausted its spare rows (or the remap itself failed) and can no longer self-heal, so replace the node promptly.

Known issues

Per-pod attribution is limited when time-slicing is enabled

If a cluster uses NVIDIA GPU time-slicing to share one physical GPU across several pods, DCGM can only attribute metrics to one of them. This is a known exporter limitation, documented by NVIDIA in dcgm-exporter#307. The Nodes dashboard is unaffected and still shows accurate per-GPU usage; the Pods dashboard shows only one of the sharing workloads.

On clusters with time-slicing, treat the Pods dashboard as best-effort and rely on the Nodes dashboard for saturation questions.

Dashboards only show data while GPU nodes exist

DCGM runs only on GPU nodes. If your GPU nodepool scales to zero when idle (a common Karpenter setup), there’s nothing to scrape during those periods, so Prometheus has no data. Past data is kept (subject to your Prometheus retention) and gaps in the graphs simply mean no GPU nodes existed at that time.

Utilisation is busy-or-idle, not intensity

DCGM_FI_DEV_GPU_UTIL measures how much of the time the GPU was doing work, not how much of its capacity was in use. A workload using 10% of the GPU non-stop shows 100% utilisation, the same as one using all of it. For finer detail (e.g. per-SM occupancy) you’d need DCGM’s profiling metrics (DCGM_FI_PROF_*), which we don’t collect today because they conflict with EKS’s own GPU health checks.