Modernizing a Grafana and Prometheus observability stack
The stack had drifted behind: Grafana past end-of-life, Prometheus eight minor versions old, and the heavy dashboards taking tens of seconds to render. Here is how I brought both to LTS and made those dashboards about 30 times faster, with no data loss and no alerts touched.
The work combined version upgrades, recording rules and operational validation.
Contribution at a glance
- My responsibility
- Staged monitoring upgrades, dashboard cleanup and recording-rule work.
- Scope
- Grafana and Prometheus modernization with backups and operational validation.
- Collaboration & boundaries
- I maintained the monitoring stack and checked query and dashboard behavior through version changes.
- Outcome
- Updated the monitoring stack and persisted migrated dashboards; this case does not establish a fleet-wide latency improvement.
Observability stacks rot quietly. The upgrades feel risky, so they get deferred, and one day you are on an end-of-life Grafana with dashboards nobody trusts and queries that take half a minute. This is a write-up of pulling one back to current: two major Grafana versions, a Prometheus LTS bump, and the recording rules that were the real win.
Grafana: end-of-life to 11.6 LTS
Grafana was on an end-of-life release, one major version short of the Angular plugin runtime being removed entirely. I took it to 11.6 LTS in a deliberate, reversible way:
- Two-step image bump (through an intermediate minor) rather than one big jump, so each step was small and easy to roll back.
- Angular runtime removal. Grafana 11 disables the Angular plugin runtime, so I uninstalled the legacy Angular panel plugins first, then walked the old-schema dashboards through the auto-migration so new React panel types persisted.
- Dashboard cleanup. Exported every dashboard from the live instance into git (it had drifted from the database), then deleted abandoned ones.
- Defensive backups. Disk snapshots before every change. Both went unused, which is exactly how backups should end.
- Zero disruption. All alert rules and datasources came through the upgrade unchanged. A later pass took it to the next major with a full dashboard audit.
The real win: Prometheus recording rules
The heavy ingress dashboards were painful: 8 to 33 seconds per render. The cause was histogram_quantile() being recomputed from tens of thousands of bucket series (about 57k) on every single refresh. The fix is a classic one that is easy to forget to reach for: recording rules.
I added three recording rules per environment that pre-aggregate the P50, P90 and P99 of the request-duration histogram, and pointed the dashboards at the pre-computed series instead of the raw buckets.
| Environment | Before | After | Speedup |
|---|---|---|---|
| Production (higher cardinality) | 8 to 33 s | ~0.66 s | ~30× |
| Sandbox (lower cardinality) | 3.5 to 9 s | ~0.66 s | ~5 to 13× |
Prometheus: 2.45 to 2.53 LTS
I patched Prometheus up eight minor versions to 2.53 LTS, staged safely: the sandbox cluster first, then production after a 24-hour soak. One real-world constraint worth naming: the stack runs the managed (GMP) fork of Prometheus because the environment's Cloud Monitoring alerts and dashboards depend on its built-in exporter. Upstream Prometheus is already on 3.x, but the managed fork tops out at 2.53 LTS, so that was the ceiling. Knowing which fork you are on, and why, is half the upgrade.
A config refactor worth doing
The per-environment Prometheus configs had drifted into different section orders, which made diffing one environment against another a chore. I reordered them to a single canonical structure (persistent volumes, config, RBAC, StatefulSet, services, ingress). No functional change, but now a side-by-side diff shows only the differences that matter.
A latent bug, found in passing
While validating the recording rules I found a panel rendering "No data" over a query that clearly had data. The cause was a Grafana thresholds config (thresholdsStyle: line+area) with a trailing threshold step missing its value. The kind of bug that hides for months until someone looks closely at the right panel.
What I would tell someone starting this
- Reach for recording rules before you blame the dashboard. If a panel recomputes a quantile over tens of thousands of series on every render, pre-aggregate it.
- Upgrade in small, reversible steps, and snapshot before each one. The jump from end-of-life to LTS is less scary as two small bumps than one big one.
- Know your fork. The managed Prometheus fork and upstream are not the same release train; pick the highest LTS your fork actually ships.
- Keep config in a canonical order across environments. Drift in structure hides drift in meaning.
Decisions and next steps
A retrospective view of the approach and its limits. The next improvement is a proposal, not a delivered result.
Alternative and rationale
A single large upgrade reduces steps but combines failure modes. Staged changes and backups make incompatibilities easier to isolate.
Trade-off
Recording rules reduce repeated query work but introduce additional definitions whose meaning and freshness must be maintained.
What I would improve next
Next improvement: keep representative dashboard queries and rule tests in a repeatable upgrade acceptance suite.