Changelog
This changelog lists all updates, improvements and new features our Engineering team develops for our Skyscrapers Reference Developer Platform. These are rolled out automatically to all DevOps-as-a-Service customers.
2021 Q4
- 2021-12-06
Maintenance
Upgraded cluster add-ons
As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated. We’ve already rolled these out to all clusters. Most of the updates are patches with minor changes. The most relevant of them are: Loki with several …
- 2021-11-26
Maintenance
Upgraded Teleport to version 8.0.0
We’ve upgraded all Teleport clusters to version 8.0.0. This is a major release, coming with many new features: Windows Desktop Access Preview TLS Routing AWS CLI support Application and Database Dynamic Registration RDS Automatic Discovery WebAuthn …
- 2021-11-25
Maintenance
Istio upgraded to version 1.12.0
We have upgraded Istio on all clusters that use it. The version was upgraded from 1.11.2 to 1.12.0. You can check the full release notes here. We’ve taken the chance to also upgrade Kiali to the latest version, from 1.40.0 to 1.43.0. This only …
- 2021-11-15
Maintenance
Introducing alerts for Fluent Bit errors
Considering we’re moving more and more log processing to Fluent Bit, it’s important to get notified when logs are not making it to the storage solutions (“outputs”) like Elasticsearch, Logz.io and S3. We’ve added 2 new alerts, …
- 2021-10-18
Maintenance
Concourse upgraded to v7.5.0
We have upgraded our Concourse setups to the latest version 7.5.0. Changes There is now the clear-resource-cache command so you can clear the cache of a resource The build page shows the name of who triggered the build in the header of the build page You …
- 2021-10-11
Maintenance
A note on Let's Encrypt chain issues due to DST Root CA X3 expiry
Let’s Encrypt certificates are (usually) cross-signed with the DST Root CA X3 root certificate, however this root certificate expired on September 30th 2021. From the upstream Let’s Encrypt documentation on the DST Root CA X3 Expiration: Let’s …
- 2021-10-07
Maintenance
Making our Terraform helper script public
Every piece of infrastructure we create is managed via Terraform. This is to ensure that everything we deploy is repeatable, follows best practices and is fully tracked. Over the years, to make our work a bit more convenient, we’ve developed some …
- 2021-10-07
Maintenance
Grafana security patch following High Severity CVE-2021-39226
On the 5th of October a notice for CVE-2021-39226 with a severity of high went out, impacting the Grafana deployments. Unauthenticated and authenticated users are able to view the snapshot with the lowest database key by accessing the literal paths: …
- 2021-10-06
Maintenance
RDS snapshots cross-account replication available
In some cases, a disaster recovery plan might require RDS snapshots to be replicated / copied over to a different AWS account and region. We can now set up this replication process for the managed RDS instances of our customers. Note that this will work in …
2021 Q3
- 2021-09-27
Maintenance
Upgraded cluster add-ons
As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated. We’ve already rolled these out to all clusters. Some highlights: Support for m6i AWS EC2 instances Many bug fixes with eg. aad-pod-identity, …
- 2021-09-27
Maintenance
Guaranteed QoS for all critical system and infrastructure Pods
We’ve seen in multiple occasions that, due to resource starvation in a cluster, the kubelet starts evicting critical infrastructure Pods. This can lead to important downtimes and disruptions in multiple occasions. We have added a custom PriorityClass …
- 2021-09-20
Maintenance
Improved EC2 instance interruption notifications
We’ve improved the looks and the content of the EC2 instance interruption notifications that we receive in Slack. For those clusters that run on spot instances, AWS can reclaim them at any moment, and when that happens the Pods running on those …
- 2021-09-16
Maintenance
Upgrade AKS and EKS clusters to 1.21. Actions to take!
We have started rolling out AKS and EKS 1.21. This brings both our supported AKS and EKS platforms on Kubernetes v1.21.2. Upon writing we have already upgraded all non-production clusters. Production clusters will follow in the next days after some extra …
- 2021-09-13
Maintenance
Istio upgraded to version 1.11.2
We have upgraded Istio on all clusters that use it. The version was upgraded from 1.10.0 to 1.11.2. The new version comes with some features meant for operators and no breaking changes that you should be concerned of. You can check the full release notes …
- 2021-09-10
Maintenance
Mute critical KubeAPIErrorBudgetBurn alerts
We have muted the critical KubeAPIErrorBudgetBurn alerts. The KubeAPIErrorBudgetBurn alerts are used to measure the SLO and is meant to point to issues or bad performance of the APIServer. While we will still receive the warning alerts we won’t route …
- 2021-09-09
Maintenance
VPA enabled by default
During the last year we have tested out the Vertical Pod Autoscaler on several of our workloads and customers. These results were positive and therefore we decided to roll out the VPA on all our clusters. By default we deploy autoscaling rules for …
- 2021-09-07
Maintenance
Downgraded Grafana to v7.5
Last month we rolled out a major Grafana update, going from 7.5 to 8.1. While initially everything looked in order, some customers experienced issues with their custom dashboards which were working perfectly in the previous release. Mainly data coming from …
- 2021-09-06
Maintenance
Cert-manager upgraded to 1.4.4
We’ve upgraded Cert-manager to the version 1.4.4 on all our Kubernetes clusters. This patch upgrade contains a bug-fix for a renewal time issue that affected some of our clusters. The upgrade is already released on all clusters and environments.
- 2021-08-31
Maintenance
Vault upgraded to 1.8.2
We are in the process of upgrading our Kubernetes based Vault setups to the latest version 1.8.2. This release contains features and bug fixes. For full details and other changes, please refer to the upstream Vault release notes: …
- 2021-08-26
Maintenance
Vault upgraded to 1.8.1
We are in the process of upgrading our Kubernetes based Vault setups to the latest version 1.8.1. This release contains features and bug fixes. For full details and other changes, please refer to the upstream Vault release notes: …
- 2021-08-10
Maintenance
Upgraded cluster components & increased Pod density. Actions to take!
As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated. We’ve already rolled these out to all clusters. Some highlights: Max pod density per node increased to 110! Major releases for Grafana IngressClasses …
- 2021-08-04
Maintenance
Reduced memory usage of Cluster Autoscaler
In an effort to optmize as much as possible the resources being used by the infrastrucutre components running on our Reference Solution Kubernetes platforms, we’ve considerably reduced the memory used by the Cluster Autoscaler by optimizing its …
- 2021-08-02
Maintenance
Upgraded Teleport to version 6.2.8
We’ve upgraded all Teleport clusters to version 6.2.8. Coming from version 4.x, this is a (double) major release, coming with many new features: Application access: users can use Teleport to securely proxy to (web-)applications with SSO, as an …
- 2021-07-22
Maintenance
Concourse upgraded to v7.3.2
We have upgraded our Concourse setups to the latest version 7.3.2. This upgrade only contains features and bugfixes, no breaking changes. You can check the full changelog here.
- 2021-07-19
Maintenance
Using encryption at rest for Prometheus and Alertmanager
We’re switching to encrypted volumes for all our Prometheus and Alertmanager set ups. These were the last of our managed infrastructure components to receive encryption at rest. You don’t need to take any further actions regarding this matter, …