← Back to changelog

2026-11-12

Platform components no longer read your Secrets

#security  #kubernetes  #component  #observability  #aikido 

A follow-up to last week’s hardening. Grafana, Loki and Alloy could read every Secret in your cluster, your application’s credentials included, while they only need ConfigMaps or nothing at all. They now get only what they use, so a compromised monitoring component can’t reach your credentials. The changes roll out with our next platform release: non-production clusters first, production after a soak period.

What changed

  • Grafana reads ConfigMaps only. Grafana loads dashboards, datasources and alerts from ConfigMaps labelled grafana_dashboard, grafana_datasource or grafana_alert. It could also read Secrets for this, and no longer can. Grafana restarts once during the rollout; where it runs a single replica, that’s a short interruption of the Grafana UI.
  • Loki no longer talks to the Kubernetes API. Loki had a sidecar that watched for alerting rules in ConfigMaps and Secrets, but Loki keeps its rules in S3 and never read that sidecar’s output. We removed it, along with Loki’s API access.
  • Alloy gets only what it uses. The Alloy instances that receive your metrics and traces now have no Kubernetes API access at all. The one that collects Kubernetes events can read events, and nothing else.
  • Security fixes for the Secrets Store CSI driver. The driver that syncs secrets from AWS Secrets Manager now runs v1.5.7, with updated images that fix known vulnerabilities. We stay on the 1.5 line until upstream ships a fix for a bug in 1.6 that deleted synced Secrets.
  • Tests keep it this way. Our nightly tests now check that none of these components can read Secrets, so a future chart upgrade that widens their access again gets caught before it reaches your cluster.

What you need to do

Nothing, unless you ship Grafana dashboards, datasources or alerts as Secrets instead of ConfigMaps. We checked every cluster and found none, but if you do, move them to ConfigMaps as described in the Grafana guide. For datasource credentials, add the datasource through the Grafana UI instead.

What you’ll see in Aikido

If you’ve connected your cluster to Aikido, the “Workload with cluster takeover roles” findings on Grafana, Loki and Alloy should go away after Aikido’s next scan of your cluster.