Changelog
This changelog lists all updates, improvements and new features our Engineering team develops for our Skyscrapers Reference Developer Platform. These are rolled out automatically to all DevOps-as-a-Service customers.
2020 Q1
- 2020-01-14
Maintenance
Teleport setups and updates are now fully automated
In our quest to automate most of the components of our infrastructure, we’ve set up CI/CD pipelines to automate the rollout of Teleport servers and their nodes. As you might know, we use Teleport to access the instances of our (and your) …
2019 Q4
- 2019-12-12
Maintenance
Upgrade Concourse to version 5.7.2
During the coming days, we’ll roll out Concourse version 5.7.2 to all our setups. This is a minor version upgrade, comming from version 5.5.3, and it includes the following: a highlight of features you might be interested in: A new experimental …
- 2019-12-12
Maintenance
Spot termination alerts on Slack
We have updated the alert routing from k8s-spot-termination-handler to notify in our shared Slack channel to increase visibility. We’ve rolled this change out to all our clusters during the last couple of days. The k8s-spot-termination-handler is …
- 2019-12-12
Maintenance
Improved Prometheus-based ElasticSearch monitoring
It has come to our attention that in certain cases our Prometheus-based ElasticSearch monitoring wasn’t correctly detecting issues and sending alerts. This problem would arise when deployed through Helm with a long release name. Kubernetes object …
- 2019-12-12
Maintenance
Fixed Kubernetes cluster-autoscaler ASG auto-detectionn
Some earlier changes in how we label our AWS AutoScaling Groups (ASGs) and which labels the Kubernetes cluster-autoscaler uses for automatically detecting these ASGs caused the scaler to not work properly. This could result in clusters not automatically …
- 2019-12-05
Maintenance
Several other K8s Reference Solution improvemens
To complement today’s barrage of changelog updates, here’s some miscellaneous additions that didn’t make it in onther post 😁: Usage of auth-proxy for accessing the Kubernetes Dashboard on EKS clusters, removing the need to re-generate a …
- 2019-12-05
Maintenance
Kubernetes monitoring upgrades
In the past week we’ve rolled out a bunch of updates to our Kubernetes cluster-monitoring stack. Except for the updated compoments (see below) we have also added a couple of new features: Allow enabling of custom Grafana plugins through the cluster …
- 2019-12-05
Maintenance
Heavily reduced resource reservations for our default cluster Add-ons
In the past weeks we’ve revisited the resource reservations (requests and limits) we made for running all the cluster Add-ons. These Add-ons provide you with features like Ingress, DNS, certificates, monitoring and logging, backups and so forth. …
- 2019-12-05
Maintenance
Better support for different AWS regions and AZs for our Kubernetes reference solution
We’ve made several improvements to our Kubernetes stacks, allowing us to deploy in different AWS Regions (eg. us-east-1) and allowing more dynamic usage of the Availability Zones in those regions. Before we were moslty focussed on the eu-west-1 …
- 2019-10-22
Maintenance
Moving from Calico to the AWS VPC CNI for EKS clusters
Previously we used Project Calico as networking plugin (CNI) on all our Kubernetes clusters (KOPS & EKS). However with our move to EKS as base for our reference solution, we will be defaulting to AWS’ own VPC CNI. Using a custom CNI with EKS, …
- 2019-10-17
Maintenance
Upgrade Vault to 1.2.3
We’ve recently upgraded our Vault setups to version 1.2.3, which is the latest Vault version available at the moment. Compared to version 1.0.1, there are a bunch of bug fixes and multiple improvements under the hood. You can check the full changelog …
- 2019-10-17
Maintenance
Introducing Grafana Loki to the k8s reference solution
Previously we shipped your logs with Fluentd to CloudWatch Logs and optionally send them to an ElasticSearch/Kibana cluster (“EFK” stack) for analytics. This setup however was expensive, had quite some problems scaling and was overkill for most …
- 2019-10-17
Maintenance
CVE-2019-14287
On tuesday a notice for CVE-2019-14287 affecting Sudo versions prior to 1.8.28. CVE-2019-11253 is a vulnerability that when sudo is configured to allow a user to run commands as an arbitrary user via the ALL keyword in a Runas specification, it is possible …
- 2019-10-17
Maintenance
CVE-2019-11253
Yesterday a notice for CVE-2019-11253 with a severity of High went out, impacting all versions of Kubernetes. CVE-2019-11253 is a denial of service vulnerability in the kube-apiserver, allowing authorized users sending malicious YAML or JSON payloads to …
- 2019-10-04
Maintenance
Upgrade to Terraform 0.12
Terraform is an automation tool that allows you to define infrastructure as code, and we use it to manage most of our customer’s infrastructure. In order to get to that point, we’ve developed a lot of Terraform code during the last few years, …
- 2019-10-01
Maintenance
CVE-2019-16276 Upgrade Concourse to 5.5.3
During today, we’ll roll out Concourse version 5.5.3 to all our setups. This is a Security patch using GoLang v1.13.1 that address a recently reported issue with Go net/http (CVE-2019-16276). This upgrade will not require a rotation of the workers so …
2019 Q3
- 2019-09-30
Maintenance
Improved Kubernetes clusters automation
We manage multiple Kubernetes clusters and regularly set up new ones from scratch. There are also a bunch of extra components deployed on each cluster, that we also need to maintain and keep up to date. In order to become more scalable, we’ve been …
- 2019-09-25
Maintenance
Upgrade EKS to 1.14
We have updated our EKS control planes and nodes to the latest version: 1.14. In the process of upgrading EKS we updated: CoreDNS from 1.2.6 to 1.3.1 KubeProxy from 1.13.10 to 1.14.6 Actions to take No specific actions are rerquired. Upstream resources EKS …
- 2019-09-25
Maintenance
Upgrade Concourse to version 5.5.1
During the coming days, we’ll roll out Concourse version 5.5.1 to all our setups. This is a minor version upgrade, comming from version 5.4.1, and it includes the following: a bunch of features a new container placement strategy more autocomplete …
- 2019-09-24
Maintenance
Upgrade Calico to 3.8.2
We have updated Calico on our AWS EKS-based reference solution to the latest version: 3.8.2. Calico is the CNI (Container Network Interface) plugin we use in our Kubernetes clusters. It’s responsible for setting up all the cluster networking and it uses …
- 2019-09-19
Maintenance
Concourse docker-image deprecation and how to migrate to the new registry-image
In the new Concourse 5.0.0 version, a new resource was released to track and upload Docker images to a registry, the registry-image-resource. This new resource is intended to replace the current docker-image-resource, as it’s more lightweight and …
- 2019-09-17
Maintenance
Switch Terraform Dynamodb tables and Vault Dynamodb backend to pay per request
Previously we were still on the default provisioned capacity for our tables. This however led to over provisioned tables and/or autoscaling to be in place. As of now we defaulted to the PPR cost type for internal Dynamodb tables and for the Dynamodb …
- 2019-09-05
Maintenance
Upgrade kops-based clusters to Kubernetes 1.11.10
We are in the process of upgrading our kops-managed Kubernetes clusters from v1.11.9 to v1.11.10. This is a bug fix release. For the complete Kubernetes 1.11.10 changelog, see https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v11110. …
- 2019-09-02
Maintenance
Upgrade to EKS 1.13
We have updated our AWS EKS-based reference solution to be compatible with Kubernetes 1.13. More specifically, EKS uses K8s v1.13.10. Actions to take No specific actions are rerquired. Notable features The following features are now supported in Kubernetes …
- 2019-09-02
Maintenance
Upgrade Concourse to version 5.4.1
During the comming days, we’ll roll out Concourse version 5.4.1 to all our setups. This is a minor version upgrade, comming from version 5.0.1, and it includes the following: a lot of internal fixes performance optimisations UI fixes. This upgrade …