Changelog
This changelog lists all updates, improvements and new features our Engineering team develops for our Skyscrapers Reference Developer Platform. These are rolled out automatically to all DevOps-as-a-Service customers.
2023 Q1
- 2023-03-13
Maintenance
Vault upgraded to 1.13.0
All Vault setups have been updated to the latest version 1.13.0. Please refer to the upstream changelogs to see what’s changed: Vault 1.13.0 Release notes
- 2023-03-01
Maintenance
INFO - Empty response for external.metrics.k8s.io/v1beta1 errors with kubectl and helm
We noticed that some of our customers are having questions about the error messages they’re getting when deploying their workload using helm and/or using kubectl: E0228 08:25:03.223480 19443 memcache.go:255] couldn't get resource list for …
- 2023-02-28
Maintenance
Post Mortem - Loki log loss
After deploying our Grafana Loki refactor, several issues started popping up, cascading to a loss of logs for a maximum of 12 hours on 22/02 or 23/02. All environments using Loki as main logging provider were affected. Environments logging to other systems …
- 2023-02-24
Maintenance
Upgraded Teleport to version 12.0.2
We’ve upgraded all Teleport clusters from version 11.1.1 to 12.0.2. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded …
- 2023-02-20
Maintenance
Upgraded cluster add-ons
Update 2023-02-28: These updates have been rolled out to all environments. As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated and our gradually rolling out to all our managed clusters. As usual, focus of …
- 2023-02-13
Maintenance
Improving Loki performance & scalability
Update 2023-02-15: These changes have now been rolled out everywhere. In the coming days we are rolling out a significant change to the Loki setup, migrating to the “simple scalable deployment mode”. With this model, the Loki services will be …
- 2023-02-03
Maintenance
Node Termination Handler Slack notifications disabled by default (AWS EKS)
Based on customer feedback, we’ve now disabled posting AWS Node Termination Handler (NTH) notifications to Slack by default. The NTH is responsible for reacting to node state changes, by properly draining a node for example when a Spot Instance …
- 2023-01-27
Maintenance
Upgraded K8s clusters to 1.24
Update 2023-02-10: All clusters have been upgraded to v1.24. We have started rolling out AKS and EKS 1.24. This brings our supported AKS platforms to v1.24.6 and EKS to v1.24.8. Upon writing we have already upgraded all non-production clusters. Production …
2022 Q4
- 2022-12-26
Maintenance
Upgraded Teleport to version 11.1.4 for security fixes
We’ve upgraded all Teleport clusters from version 11.1.2 to 11.1.4. This upgrade was done on all Teleport servers to fix potential vulnerabilties: [Critical] RBAC bypass in SSH TCP tunneling When establishing a direct-tcpip channel, Teleport did not …
- 2022-12-21
Maintenance
Upgraded cluster add-ons
As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated. These changes will be rolling out to all clusters soon. Update 2023-01-06: Changes have been rolled out to all clusters. EKS specific aws-efs-csi-driver …
- 2022-12-13
Maintenance
Vault upgraded to 1.12.2
All Vault setups have been updated from 1.12.0 to the latest version 1.12.2. This release brings small improvements and bug fixes. Please refer to the upstream changelogs to see what’s changed: Vault 1.12.1 Release notes Vault 1.12.2 Release notes
- 2022-12-13
Maintenance
New feature: Use External-DNS for managing custom DNS records
Whenever you deploy an Ingress resource, external-dns is responsible for creating the matching DNS record. We have now enabled the “CRD” feature of this component, which allows you to manage any DNS records of your choice through external-dns. …
- 2022-12-13
Maintenance
Istio upgraded to version 1.16.1
We have upgraded Istio on all clusters that use it. The version was upgraded from 1.15.2 to 1.16.1. What’s new: External Authorization Promoted to Beta Kubernetes Gateway API Implementation Promoted to Beta JWT Claim Based Routing Promoted to Alpha …
- 2022-12-02
Maintenance
Upgraded Teleport to version 11.1.1 for security fix
We’ve upgraded all Teleport clusters from version 11.0.3 to 11.1.1. This upgrade was done on all Teleport servers to fix a potential vulnerabilty: Fixed issue where an attacker with physical access to user’s computer and raw access to the …
- 2022-11-28
Maintenance
Upgraded Teleport to version 11.0.3
We’ve upgraded all Teleport clusters from version 10.1.4 to 11.0.3. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded …
- 2022-11-21
Maintenance
New feature: Kubernetes descheduler
Today we’re adding a new fearure in our Kubernetes reference solution. It is now possible to deploy the Kubernetes descheduler on your cluster(s). For now while we are testing this add-on this is an optional component. If all goes well we’ll …
- 2022-11-18
Maintenance
Upgraded cluster add-ons
As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated. We’ve already rolled these out to all non-production clusters. Production upgrades are scheduled to happen next week during business hours. As usual, …
- 2022-11-17
Maintenance
Upgraded K8s clusters to 1.23
We have started rolling out AKS and EKS 1.23. This brings our supported AKS platforms to v1.22.12 and EKS to v1.22.13. Upon writing we have already upgraded all non-production clusters. Production clusters will follow next week after some extra validation. …
- 2022-11-10
Maintenance
Upgraded ingress-nginx, fixing CVE-2022-32149, CVE-2022-27664 and CVE-2022-1996
In response to several CVEs, the following Kubernetes cluster components have been updated. These changes have already been rolled out to all clusters. ingress-nginx v1.5.1 CVE fixes CVE-2022-32149, CVE-2022-27664, CVE-2022-1996 v1.4.0 updates …
- 2022-11-09
Maintenance
Upgraded monitoring add-ons, fixing Grafana CVE-2022-32149
In response to CVE-2022-32149, the following Kubernetes cluster components have been updated. These changes have already been rolled out to all clusters. kube-prometheus-stack chart 41.7.3 This brings updates to alertmanager, prometheus-operator, …
- 2022-11-04
Maintenance
AWS EKS AMI recalled
We use the AWS-published EKS AMI (Amazon Machine Image) as a base to build our custom image for our managed Kubernetes clusters, which in turn is based on Amazon linux 2. Our CI system monitors the published AWS AMIs and automatically builds our custom …
- 2022-10-28
Maintenance
Improved monitoring for the RDS snapshot cross-account replicator module
We’ve implemented several improvements on the monitoring of our RDS snapshot cross-account replicator module, which have been rolled out to all customers that are currently using it. That module is used to take regular snapshots of RDS instances or …
- 2022-10-25
Maintenance
Vault upgraded to 1.12.0
All Vault setups have been updated to the latest version 1.12.0. Please refer to the upstream changelogs to see what’s changed: Vault 1.10.0 Release notes Ability to view client counts per auth and changes to clients over months, therefore, providing …
- 2022-10-18
Maintenance
CVE-2022-27665 patches
A security issue was discovered in Golang where a user can cause a denial of service because an HTTP/2 connection can hang during closing if shutdown were preempted by a fatal error. In order to mitigate this problem we updated the following components: …
- 2022-10-18
Maintenance
Concourse upgraded to 7.8.3
We have upgraded our Concourse setups to the latest version 7.8.3. This patch release comes with some security fixes. You can check the full changelog in the Concourse releases page. Fix team name overwritten bug All Concourse versions prior to v7.8.3 is …