Changelog
This changelog lists all updates, improvements and new features our Engineering team develops for our Skyscrapers Reference Developer Platform. These are rolled out automatically to all DevOps-as-a-Service customers.
2019 Q4
- 2019-10-17
Maintenance
Upgrade Vault to 1.2.3
We’ve recently upgraded our Vault setups to version 1.2.3, which is the latest Vault version available at the moment. Compared to version 1.0.1, there are a bunch of bug fixes and multiple improvements under the hood. You can check the full changelog …
- 2019-10-17
Maintenance
Introducing Grafana Loki to the k8s reference solution
Previously we shipped your logs with Fluentd to CloudWatch Logs and optionally send them to an ElasticSearch/Kibana cluster (“EFK” stack) for analytics. This setup however was expensive, had quite some problems scaling and was overkill for most …
- 2019-10-17
Maintenance
CVE-2019-14287
On tuesday a notice for CVE-2019-14287 affecting Sudo versions prior to 1.8.28. CVE-2019-11253 is a vulnerability that when sudo is configured to allow a user to run commands as an arbitrary user via the ALL keyword in a Runas specification, it is possible …
- 2019-10-17
Maintenance
CVE-2019-11253
Yesterday a notice for CVE-2019-11253 with a severity of High went out, impacting all versions of Kubernetes. CVE-2019-11253 is a denial of service vulnerability in the kube-apiserver, allowing authorized users sending malicious YAML or JSON payloads to …
- 2019-10-04
Maintenance
Upgrade to Terraform 0.12
Terraform is an automation tool that allows you to define infrastructure as code, and we use it to manage most of our customer’s infrastructure. In order to get to that point, we’ve developed a lot of Terraform code during the last few years, …
- 2019-10-01
Maintenance
CVE-2019-16276 Upgrade Concourse to 5.5.3
During today, we’ll roll out Concourse version 5.5.3 to all our setups. This is a Security patch using GoLang v1.13.1 that address a recently reported issue with Go net/http (CVE-2019-16276). This upgrade will not require a rotation of the workers so …
2019 Q3
- 2019-09-30
Maintenance
Improved Kubernetes clusters automation
We manage multiple Kubernetes clusters and regularly set up new ones from scratch. There are also a bunch of extra components deployed on each cluster, that we also need to maintain and keep up to date. In order to become more scalable, we’ve been …
- 2019-09-25
Maintenance
Upgrade EKS to 1.14
We have updated our EKS control planes and nodes to the latest version: 1.14. In the process of upgrading EKS we updated: CoreDNS from 1.2.6 to 1.3.1 KubeProxy from 1.13.10 to 1.14.6 Actions to take No specific actions are rerquired. Upstream resources EKS …
- 2019-09-25
Maintenance
Upgrade Concourse to version 5.5.1
During the coming days, we’ll roll out Concourse version 5.5.1 to all our setups. This is a minor version upgrade, comming from version 5.4.1, and it includes the following: a bunch of features a new container placement strategy more autocomplete …
- 2019-09-24
Maintenance
Upgrade Calico to 3.8.2
We have updated Calico on our AWS EKS-based reference solution to the latest version: 3.8.2. Calico is the CNI (Container Network Interface) plugin we use in our Kubernetes clusters. It’s responsible for setting up all the cluster networking and it uses …
- 2019-09-19
Maintenance
Concourse docker-image deprecation and how to migrate to the new registry-image
In the new Concourse 5.0.0 version, a new resource was released to track and upload Docker images to a registry, the registry-image-resource. This new resource is intended to replace the current docker-image-resource, as it’s more lightweight and …
- 2019-09-17
Maintenance
Switch Terraform Dynamodb tables and Vault Dynamodb backend to pay per request
Previously we were still on the default provisioned capacity for our tables. This however led to over provisioned tables and/or autoscaling to be in place. As of now we defaulted to the PPR cost type for internal Dynamodb tables and for the Dynamodb …
- 2019-09-05
Maintenance
Upgrade kops-based clusters to Kubernetes 1.11.10
We are in the process of upgrading our kops-managed Kubernetes clusters from v1.11.9 to v1.11.10. This is a bug fix release. For the complete Kubernetes 1.11.10 changelog, see https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v11110. …
- 2019-09-02
Maintenance
Upgrade to EKS 1.13
We have updated our AWS EKS-based reference solution to be compatible with Kubernetes 1.13. More specifically, EKS uses K8s v1.13.10. Actions to take No specific actions are rerquired. Notable features The following features are now supported in Kubernetes …
- 2019-09-02
Maintenance
Upgrade Concourse to version 5.4.1
During the comming days, we’ll roll out Concourse version 5.4.1 to all our setups. This is a minor version upgrade, comming from version 5.0.1, and it includes the following: a lot of internal fixes performance optimisations UI fixes. This upgrade …
- 2019-08-29
Maintenance
Prometheus-blackbox-exporter available as optional cluster addon
We’ve added the prometheus-blackbox-exporter as a K8s cluster addon which can be enabled upon request. The blackbox exporter can be used for probing HTTP(S), DNS, TCP and ICMP endpoints, for example to check whether an external resource is up/down. …
- 2019-08-29
Maintenance
Concourse task that checks the status of the service after the deployment
We extended the functionallity for the ECS deployments with concourse. After the service gets deployed Concourse would just exit because Terraform doesn’t take the deployment itself into account. This resulted in having false deploys sometimes …
- 2019-08-28
Maintenance
Redshift monitoring via Prometheus
We have updated our stacks to support Redshift monitoring via the Prometheus Operator running on our K8s clusters. If you have Redshift running, you will now be able to see alerts in Alertmanager and on slack when there is something wrong with the cluster. …
- 2019-08-26
Maintenance
Neo4j monitoring via Prometheus
We have updated our stacks to support Neo4j monitoring (Neo4j >= 3.4) via the Prometheus Operator running on our K8s clusters. If you have Neo4j running, you will see metrics appearing in the new Neo4j Grafana dashboard. Previously we still monitored …
- 2019-08-20
Maintenance
Fix for dashboards HTTP 500 error when refreshing token
Since our SSO overhaul you might’ve been noticing sudden HTTP 500 errors while using the Alertmanager, Kubernetes of Prometheus dashboards when your token’s TTL expires. Normally when your OIDC token expires, your session should automatically …
- 2019-08-19
Maintenance
A note on CVE-2019-11247
Two weeks ago a patch for Kubernetes vulnerability CVE-2019-11247 was released for K8s 1.13, 1.14 and 1.15. Unfortunately as of writing clusters using older K8s versions (like our kops-based 1.11 clusters) are still vulnerable. In short this vulnerability …
- 2019-08-12
Maintenance
Kubernetes dashboards ERR_TOO_MANY_REDIRECTS bug
During the past days you might’ve been getting ERR_TOO_MANY_REDIRECTS and or Bad Request - Login session expired errors. This bug was introduced during last week’s cluster add-ons upgrade. We have reverted the change that’s causing these …
- 2019-08-09
Maintenance
Add Bitbucket, GitLab and Google authentication to Concourse
By default we only allowed authenticating to Concourse through GitHub and local users. It’s now possible to plug into other systems like Bitbucket, GitLab or Google. Let us know if you’d like to change to any of these authentication systems.
- 2019-08-06
Maintenance
Kubernetes add-on upgrades
In the following days we’ll be rolling-out a bunch of upgrades to the deployed add-ons on your clusters. You don’t have to do anything to apply these upgrades, we’ll do that for you. And it won’t cause any downtime to the cluster or …
- 2019-08-01
Maintenance
We're moving to EKS
The past months we’ve beeen heavily re-evaluting and testing AWS EKS as base for our reference solution. Today we can consider our platform GA and moving forward all new clusters will be setup using EKS. Naturally we’ll keep on supporting and …