Changelog
This changelog lists all updates, improvements and new features our Engineering team develops for our Skyscrapers Reference Developer Platform. These are rolled out automatically to all DevOps-as-a-Service customers.
2023 Q3
- 2023-09-29
Maintenance
Removed Calico as NetworkPolicies engine - this functionality has merged with the AWS VPC CNI
Historically we’ve been using Calico as controller to provide NetworkPolicies support. This was offered as an optional feature only, considering the resource (and thus possible cost) impact of running this component. As announced in our K8s 1.27 …
- 2023-09-20
Maintenance
Upgrading K8s clusters to v1.27
Update 2023-09-28: All clusters have been upgraded to v1.27. We are rolling out AKS and EKS v1.27. This brings our supported AKS platforms to v1.27.3 and EKS to v1.27(.4). Please make sure to update to our recommended client versions matching this upgrade …
- 2023-09-11
Maintenance
New experimental component Karpenter for cost-effective node scaling
Karpenter is a new big feature that we offer in our AWS reference solution Today we manage Kubernetes nodes in a classic way through AWS autoscaling groups. However doing it this way brings a couple restrictions, most importantly that nodes are not …
- 2023-09-07
Maintenance
Upgraded Teleport to version 13.3.8
We’ve upgraded all Teleport clusters from version 13.3.0 to 13.3.8. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded …
- 2023-08-04
Maintenance
Upgraded Teleport to version 13.3.0
We’ve upgraded all Teleport clusters from version 13.0.3 to13.3.0. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded …
- 2023-07-19
Maintenance
Disable kubernetes-dashboard by default
In an effort to further reduce the footprint of the reference solution we are no longer going to deploy the kubernetes-dashboard* by default. The Skyscrapers team is using k9s as a tool to manage our clusters and we think its a worhty replacement. This …
- 2023-07-14
Maintenance
Upgraded cluster add-ons
Update 2023-07-18: These updates have been rolled out to all environments. As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated and our gradually rolling out to all our managed clusters. Click each link to go …
- 2023-07-07
Maintenance
Upgraded cluster add-ons
Update 2023-07-18: These updates have been rolled out to all environments. As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated and our gradually rolling out to all our managed clusters. Focus of most changes …
- 2023-07-05
Maintenance
Dedicated system node pool + reduced system component footprint
In order to improve our services we changed the way the Kubernetes nodepools are structured. Previously there was a default nodepool that had a mix of both Kubernetes add-ons and application deployments. This made things more complex than it needed to be. …
2023 Q2
- 2023-06-22
Maintenance
Upgrading K8s clusters to v1.26
We are rolling out AKS and EKS v1.26. This brings our supported AKS platforms to v1.26.3 and EKS to v1.26(.5). Please make sure to update to our recommended client versions matching this upgrade Timeline: Testing & rollout to non-production clusters …
- 2023-06-09
Maintenance
Vault upgraded to 1.13.3
All Vault setups have been updated to the latest version 1.13.3. Please refer to the upstream changelogs to see what’s changed: Vault 1.13.3 Release notes
- 2023-06-09
Maintenance
Upgrading OpenSearch clusters to 2.5
We’re starting to gradually upgrade all our managed OpenSearch clusters to the latest supported version in Amazon OpenSearch Service (V2.5). Depending on which version your OpenSearch domain comes from, this upgrade brings up several new features and …
- 2023-05-31
Maintenance
Upgraded Teleport to version 13.0.3
We’ve upgraded all Teleport clusters from version 12.2.1 to 13.0.3. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded …
- 2023-05-30
Maintenance
Post-mortem - A word on Pods stuck with ContainerCreating status problems
During the past days we’ve had multiple customer reports that launching new Pods got stuck with the ContainerCreating status, mostly affecting CronJobs. Even if you were not directly affected, you might have noticed the messages in your channels that …
- 2023-05-25
Maintenance
Upgraded fluent-bit and moving to built-in loki output
We have upgraded Fluent Bit form v2.0.1 to v2.1.3 and this is gradually rolling out to all our managed clusters. Fluent Bit is the component responsible for collecting all logs, filtering, and shippiing this to several outputs with Grafana Loki as our …
- 2023-05-11
Maintenance
Upgraded cluster add-ons for monitoring & logging
As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated and our gradually rolling out to all our managed clusters. Click each link to go to the full upstream changelog. cloudwatch-exporter 0.15.0 -> 0.15.3 …
- 2023-05-10
Maintenance
[Action required] Update external-dns behavior for private Route53 zones
We’re updating the behavior of external-dns when managing records in private Route53 zones. Note that this new behavior is still in the works and will be rolled out after 2023-06-09. Since it brings breaking changes for domains in private zones, …
- 2023-05-04
Maintenance
Upgraded cluster add-ons
As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated and our gradually rolling out to all our managed clusters. Focus of most changes resolve several CVEs, bug fixes and minor improvements. AKS specific Click …
- 2023-04-17
Maintenance
Istio upgraded to version 1.16.4
We have upgraded Istio on all clusters that use it. The version was upgraded from 1.16.3 to 1.16.4. These release mostly focuses on many security fixes, ranging from moderate to high. You can check the full release notes here. We’ve also upgraded …
- 2023-04-11
Maintenance
Upgraded Teleport to version 12.2.1
We’ve upgraded all Teleport clusters from version to 12.2.1. Teleport is a tool we mostly use internally to provide secure and auditted access to (EC2) instances, Kubernetes clusters and several dashboards. The nodes will gradually be upgraded to the …
2023 Q1
- 2023-03-31
Maintenance
Defaulting to capacity-optimized for Spot nodepools
Most of our EKS clusters leverage Spot instances as a cost-efficient way to provide compute nodes. Historically we’ve been defaulting to the “lowest price” allocation strategy to maximize possible cost savings. However this can lead to …
- 2023-03-30
Maintenance
[Action required] Upgrades starting for K8s clusters to 1.25
Update 2023-05-04: All clusters have been upgraded to v1.25. We are starting to roll out AKS and EKS 1.25. This brings our supported AKS platforms to v1.25.5 and EKS to v1.25.6. This release includes the removal of several deprecated API Version, so please …
- 2023-03-27
Maintenance
Upgraded EKS cluster add-ons
As part of our regular upgrade cycle, the following EKS cluster components have been updated. We’ve already rolled these out to all non-production clusters. Production upgrades are scheduled to happen in the next few days during business hours. As …
- 2023-03-20
Maintenance
Deletion prevention on OpenSearch S3 backups
We have made an update to the S3 buckets that are used to take external backups from our managed OpenSearch clusters. Previously everyone who had access to these buckets were able to modifications to the objects in that bucket. In order to prevent …
- 2023-03-16
Maintenance
Istio upgraded to version 1.16.3
We have upgraded Istio on all clusters that use it. The version was upgraded from 1.16.1 to 1.16.3. These releases contain bug fixes to improve robustness and security fixes in the underlying Go packages. You can check the full release notes here. …