← Back to changelog

2026-09-23

Cluster add-on upgrades, with Tempo 3.0 and a monitoring stack refresh

#add-on  #kubernetes  #update  #upgrade  #component  #eks  #observability 

We are rolling out another round of cluster add-on upgrades. It ships to non-production clusters first, with production to follow. Most of it is routine patch and minor releases, and none of them need any action on your side. One change is worth a closer look: Tempo 3.0.

What changed

  • Tempo 3.0 (tracing). Tempo’s new architecture replaces its ingester and compactor with a live store and a backend scheduler. In the single-binary mode we run, more than one Tempo instance is no longer supported, so Tempo now runs as a single replica. Traces flow as before. What is new: when the Tempo pod restarts, for example during a rollout or a node replacement, new traces wait in Alloy’s queue, which now retries for 15 minutes, and stored traces stay queryable throughout. For the upgrade itself we made the old Tempo write its in-flight traces to storage before handing over, so nothing is lost at the switch. The observability.tracing.tempo.replicas setting no longer exists in the cluster definition; no cluster set it. If you need highly available tracing, talk to us: that is a different deployment mode with its own message queue.
  • Monitoring stack refresh. kube-prometheus-stack moves to chart v91 (Prometheus Operator v0.94.0) and takes Grafana from 13.1 to 13.2 on a slimmer, hardened image. The CRD updates are applied automatically by our GitOps setup; dashboards, data sources and plugins keep working unchanged.
  • Vertical Pod Autoscaler chart 0.12. The chart now manages its own CRDs; our GitOps setup adopts them automatically. Nothing changes in how the VPA behaves.
  • GPU nodes. The new EKS node image reworks how the NVIDIA driver is installed (R580 and R595 available, chosen per instance type), alongside NVIDIA device plugin 0.20 and DCGM exporter 4.8.4. GPU workloads need no change.
  • external-dns stays on 0.21 for now. Version 0.22 changes how it reads hostname annotations. We are holding that upgrade back and will ship it separately, with its own announcement, once it is prepared properly.

Add-on upgrades

We also picked up the latest EKS build revisions for CoreDNS, kube-proxy, metrics-server and the snapshot-controller (packaging updates only, no version change).