Your cluster upgrades now roll out through Skyflow

Every change to your EKS clusters (a new kubernetes-stack release, an add-on change, an edit to a cluster definition) now rolls out through Skyflow, our new rollout system. It replaces the Concourse pipelines we ran for you. All clusters, staging and production, are switched as of today.

What you gain:

• Safer: no more long-lived credentials, every customer in its own isolated environment, and every risky change checked by an engineer before it is applied.
• Faster: a release reaches all clusters of a stage in one coordinated rollout, and a change you make to a cluster definition is picked up within a minute.
• More predictable: every release proves itself on staging before it can reach production, and every rollout is recorded: which version, which stage, who promoted it.

The outcome stays the same: OpenTofu provisions the resources and the rendered Flux manifests land in your repository, which Flux then deploys. For staging clusters that now happens without a pull request to merge; for production clusters we still get a pull request to review, as before. What changes is how a release gets there:

    flowchart LR
    rel["kubernetes-stack<br>release"] --> staging["Stage: staging<br>all staging clusters<br>automatic"]
    staging -- "soak, then promoted<br>by our team" --> prod["Stage: production<br>all production clusters"]
    staging --> run["One run per cluster<br>plan, gate, apply, render"]
    prod --> run
    change["Change in your<br>cluster definition"] --> run
    run --> pr["Pull request<br>in your repository"] --> flux["Flux deploys<br>to the cluster"]
  

What changed

Where we previously used Concourse to help us with our rollouts, our new Skyflow system is built on Kargo and Argo Workflows. Kargo decides which release goes to which stage and keeps the record of who promoted what, and when. Argo Workflows does the work: one run per cluster that plans and applies the OpenTofu stacks, renders the Flux manifests and commits them to your repository.

Releases now move through stages. A new release first rolls out to every staging cluster under our management. It has to succeed there and sit for a few hours before our team can promote it to the production stage, and only then does it reach production clusters. Every cluster of a stage gets the same version at the same time, so a release lands on all our platforms in one coordinated rollout instead of pipeline by pipeline. A change to a cluster definition in your repository follows the same path: it is picked up and rolled out at the version your cluster already runs.

Because the new system is more automated we had to build in more safeguards. Every rollout runs behind a gate. Before anything is applied, the run plans the change and inspects the plan. A plan that would destroy or replace resources, or change the Kubernetes version, stops and waits for one of our engineers to look at it and approve before it continues. Nothing risky happens without a person having seen it.

Fewer static credentials. Concourse worked with long-lived AWS access keys and a shared GitHub token. Skyflow holds none of those: every run gets short-lived AWS credentials through a role chain that can only reach your own accounts, and GitHub access comes from a per-customer GitHub App, installed on your repository alone, whose tokens are created per step and expire within the hour. On top of that, every customer runs in its own isolated Kubernetes namespace with its own identities and secrets: a run for one customer cannot see or reach another customer’s runs, credentials or repository. Nothing to leak, nothing to rotate, nothing shared.

What you will see

The pull requests in your repository look familiar. They are still titled auto: Update system Flux manifests (<version>) and still carry the rendered manifests under flux/system/<cluster>/, but they now come from the sks-skycontrol-<yourname> app instead of our Concourse bot. For staging clusters that app also merges the pull request itself and the run waits for Flux to report the cluster healthy at the new commit. For production clusters the pull request stays open for our human review and merge, as before.

The first Skyflow run on each cluster opened a one-off pull request that adds a STACK_VERSION marker file and re-encrypts the existing encrypted files under flux/system/<cluster>/ for the new identity. It changes nothing on the cluster itself.

If your cluster definition sets a Slack channel for pipeline results, each rollout of that cluster is posted there as before, with improved details on what was changed where.