Changelog
This changelog lists all updates, improvements and new features our Engineering team develops for our Skyscrapers Reference Developer Platform. These are rolled out automatically to all DevOps-as-a-Service customers.
2022 Q1
- 2022-03-01
Maintenance
AKS component upgrades
As part of our regular upgrade cycle, the following Azure specific Kubernetes cluster components have been updated. We’ve already rolled these out to all non-production clusters. Production clusters will follow once we validated everything is stable. …
- 2022-02-24
Maintenance
Upgraded Grafana and Prometheus
As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated. We’ve already rolled these out to all non-production clusters. Production clusters will follow once we validated everything is stable. There are no …
- 2022-02-14
Maintenance
Upgraded Teleport to version 8.2.0
We’ve upgraded all Teleport clusters from version 8.0.7 to 8.2.0. This is a minor release, coming with mostly bug and performance fixes. You can find more information on this release in the Teleport changelog.
- 2022-02-07
Maintenance
Github Actions Runner Controller
We’re adding support for the Github actions-runner-controller as a managed add-on for our Kubernetes platforms. With this controller, the customers using Github Actions will be able to easily deploy self-hosted runners on their clusters. This is …
- 2022-02-07
Maintenance
AKS rollouts are now automated
We manage multiple Kubernetes clusters and regularly set up new ones from scratch. There are also a bunch of extra components deployed on each cluster, that we also need to maintain and keep up to date. Rolling out these changes on AWS EKS have been …
- 2022-02-02
Maintenance
Calico NetworkPolicy controller upgraded on EKS
On AWS EKS clusters we use Calico for providing NetworkPolicy functionality. With these NetworkPolicies you can control the traffic flow within a Kubernetes cluster between Pods, Services and external resources. In earlier versions we used the AWS provided …
- 2022-01-28
Maintenance
VPA enabled by for metrics-server
We have already configured the VPA for many of our workloads (ExternalDNS, cert-manager, Prometheus and more). Today we also configured this for the metrics-server workload. This means that for those workloads we need less manual configuration changes when …
- 2022-01-27
Maintenance
Let's Encrypt revocations affecting TLS-ALPN-01 certificates
On 26 January 2022, Let’s Encrypt notified subscribers that most certificates issued in the last 90 days and validated with the TLS-ALPN-01 challenge will be revoked on 28 January 2022 and should be immediatelly renewed. This revocation only affects …
- 2022-01-27
Maintenance
Adding support for the AWS Load Balancer controller
The AWS Load Balancer Controller is the successor of the ALB Ingress Controller, with many new features. This controller allows creating both ALBs and NLBs dynamically. It satisfies Kubernetes Ingress resources by provisioning Application Load Balancers. …
- 2022-01-21
Maintenance
Standardizing on Fluent Bit and Loki updates
For a while we’ve offered Grafana Loki as default logging solution. For shipping logs to Loki we were using the included Promtail. However, more recently, we’ve also supported other logging solutions, like Elasticsearch and Logz.io for …
- 2022-01-21
Announcements
CVE-2021-25742 in ingress-nginx
A security issue was discovered in ingress-nginx where a user that can create or update ingress objects can use the custom snippets feature to obtain all secrets in the cluster. This vulnerability has a high impact on clusters where there are strict RBAC …
- 2022-01-07
Maintenance
VPA enabled for Vault
We have added Vault to the list of autoscaling rules we deploy by default. By doing this we can allow the VPA to set the optimal resource requests and limits within the boundaries that we provide. You can also make use of the features that the VPA …
- 2022-01-07
Maintenance
Upgraded Teleport to version 8.0.7
We’ve upgraded all Teleport clusters from version 8.0.0 to 8.0.7. This is a minor release, coming with mostly bug and security fixes. You can find more information on this release in the Teleport changelog.
- 2022-01-07
Maintenance
Module updated for AWS OpenSearch and started upgrades
AWS ElasticSearch Service has been rebranded to AWS OpenSearch for some time now, and thus we’ve decided to rename our Terraform module for managing this service accordingly. This open source module will setup an OpenSearch 1.1 domain by default, …
2021 Q4
- 2021-12-16
Maintenance
Monitoring for Grafana Loki in case of discarded logs
During a routine monitoring review, we’ve noticed some Promtail pods were using significantly more CPU than the generic request. This pointed us to two issues: Although using the Vertical Pod Autoscaler, CPU requests for Promtail pods was not being …
- 2021-12-16
Maintenance
Add support for mixed node pools in EKS
We have added support for mixed node pools on AWS. Previously our EKS solution supported only a single type of spot instances in a pool. This caused problems when a certain type is out of capacity or when there’s lower prices for another instance …
- 2021-12-14
Maintenance
Critical CVE-2021-44228 in Log4j - Check your application workloads!
Update 2021-12-16: The patched Log4j 2.15.0 was found to still have a possible vulnerability. We’ve updated the action below to update to (at least) version 2.16.0. On Thursday the 9th of December a 0-day exploit in the popular Java logging library …
- 2021-12-10
Maintenance
Add support for AWS Secrets Manager in EKS
We’ve added support for using secrets from AWS Secrets Manager in EKS clusters. This support is optional and disabled by default. When enabled, two extra components will be deployed on the cluster: the Secrets Store CSI driver and the AWS Secrets …
- 2021-12-06
Maintenance
Upgraded cluster add-ons
As part of our regular upgrade cycle, the following Kubernetes cluster components have been updated. We’ve already rolled these out to all clusters. Most of the updates are patches with minor changes. The most relevant of them are: Loki with several …
- 2021-11-26
Maintenance
Upgraded Teleport to version 8.0.0
We’ve upgraded all Teleport clusters to version 8.0.0. This is a major release, coming with many new features: Windows Desktop Access Preview TLS Routing AWS CLI support Application and Database Dynamic Registration RDS Automatic Discovery WebAuthn …
- 2021-11-25
Maintenance
Istio upgraded to version 1.12.0
We have upgraded Istio on all clusters that use it. The version was upgraded from 1.11.2 to 1.12.0. You can check the full release notes here. We’ve taken the chance to also upgrade Kiali to the latest version, from 1.40.0 to 1.43.0. This only …
- 2021-11-15
Maintenance
Introducing alerts for Fluent Bit errors
Considering we’re moving more and more log processing to Fluent Bit, it’s important to get notified when logs are not making it to the storage solutions (“outputs”) like Elasticsearch, Logz.io and S3. We’ve added 2 new alerts, …
- 2021-10-18
Maintenance
Concourse upgraded to v7.5.0
We have upgraded our Concourse setups to the latest version 7.5.0. Changes There is now the clear-resource-cache command so you can clear the cache of a resource The build page shows the name of who triggered the build in the header of the build page You …
- 2021-10-11
Maintenance
A note on Let's Encrypt chain issues due to DST Root CA X3 expiry
Let’s Encrypt certificates are (usually) cross-signed with the DST Root CA X3 root certificate, however this root certificate expired on September 30th 2021. From the upstream Let’s Encrypt documentation on the DST Root CA X3 Expiration: Let’s …
- 2021-10-07
Maintenance
Making our Terraform helper script public
Every piece of infrastructure we create is managed via Terraform. This is to ensure that everything we deploy is repeatable, follows best practices and is fully tracked. Over the years, to make our work a bit more convenient, we’ve developed some …