Bitbucket Pipelines runners

You can run self-hosted Bitbucket Pipelines runners on your Kubernetes cluster, scaled on demand by Atlassian’s runners-autoscaler. The autoscaler registers runners in your Bitbucket workspace or repository, starts a runner Job on the cluster for each one, and removes them again once they’ve been idle for a while.

The runners are disabled by default but we can quickly enable them upon request.

How it works

The autoscaler runs in the bitbucket-runners namespace and polls the Bitbucket API on a fixed interval. Runners are organised in groups: each group targets a workspace (or a single repository), carries a set of labels, and scales between a minimum and maximum number of runners. When most of a group’s runners are busy, the autoscaler registers extra runners and starts a Job for each; when most are idle, it removes them.

Skyscrapers manages the configuration with OpenTofu. Our CI renders the autoscaler manifests into your repository, under flux/system/<cluster>/addons/bitbucket-runners/, and Flux applies them to the cluster. Those files carry a DO NOT EDIT header: ask us for changes instead of editing them, as the next rollout overwrites them.

The Bitbucket credential never passes through Skyscrapers or your repository: you create it as a Kubernetes Secret directly on the cluster, and the manifests only reference it by name.

Bitbucket authentication

The autoscaler authenticates to Bitbucket with an Atlassian API token. The token lets its holder register runners, and registered runners receive your pipeline steps together with their variables and secrets, so treat it as a high-value credential:

• Create the token under a dedicated service account, not a personal account, so it survives people leaving and its permissions stay limited to what the runners need.
• The account needs permission to manage runners where they are registered: workspace admin for workspace runners, or admin on the repository for repository runners. Prefer repository runners when they cover your needs, so the account doesn’t need workspace-wide rights.
• API tokens expire (at most one year after creation). Set a reminder to rotate it before it does, or the runners stop scaling.

To create the token and hand it to the autoscaler:

1. Log in to Atlassian as the service account and go to API tokens in the account security settings.

2. Choose Create API token with scopes, give it a recognisable name (e.g. bitbucket-runners-<cluster>) and an expiry date, and select Bitbucket as the app.

3. Select the following scopes, which the autoscaler requires:

   • read:repository:bitbucket
   • read:workspace:bitbucket
   • read:runner:bitbucket
   • write:runner:bitbucket

4. Create the token and copy it. Atlassian only shows it once.

5. Share the email address of the service account with Skyscrapers. It isn’t secret: it goes into the autoscaler configuration next to the token.

6. Once Skyscrapers has rolled out the runners (the rollout creates the bitbucket-runners namespace), create the Secret holding the token in that namespace (remember to set the correct kubectl context before running this command):

   kubectl create secret generic bitbucket-runner-credentials \
     -n bitbucket-runners \
     --from-literal=atlassianApiToken=<api-token>

   Until the Secret exists, the autoscaler pods wait in CreateContainerConfigError; they start on their own once it’s there.

Runner configuration

Skyscrapers sets up the runners with Terragrunt/OpenTofu, using our bitbucket-runners stack. To configure it, we need from you:

• The Bitbucket workspace, and the repository if you want repository runners
• The labels for each group of runners (see Known behaviours before you pick them)
• The minimum and maximum number of runners per group
• The resources your pipeline steps need, if the defaults (1 CPU and 2 GiB for the runner, 1 CPU and 2 GiB for the Docker daemon) aren’t enough

A typical configuration looks like this:

inputs = {
  atlassian_account_email = "<service-account@example.com>"

  # Run the runner Jobs on the CI node pool
  tolerations = [
    { key = "role", operator = "Equal", value = "cicd", effect = "NoSchedule" },
  ]
  node_selector = {
    role = "cicd"
  }

  groups = [
    {
      name      = "<workspace> runners"
      workspace = "<workspace>"
      # repository = "<repository>"  # set for repository runners, omit for workspace runners
      labels = ["self.hosted", "linux", "k8s", "autoscaled"]
      min    = 1
      max    = 5

      requests_cpu    = "2000m"
      requests_memory = "4Gi"
      limits_cpu      = "2000m"
      limits_memory   = "4Gi"
    },
  ]
}

Each runner Job runs a Docker daemon (docker-in-docker) next to the runner, so steps that build images or use Docker services work out of the box.

Use the runners in your pipelines

Once the runners are online, you can see them under your workspace or repository settings in Bitbucket (Runners). To run a step on them, set runs-on to the group’s labels in your bitbucket-pipelines.yml:

pipelines:
  default:
    - step:
        name: Build
        runs-on:
          - self.hosted
          - linux
          - k8s
          - autoscaled
        script:
          - make build

Bitbucket sends the step to any online runner that has at least the labels listed in runs-on, so a step listing only self.hosted and linux can also land on these runners.

Known behaviours

Labels must differ from your existing runners

The autoscaler claims every runner in the workspace whose labels exactly match one of its groups, including runners it didn’t create. If you already have static self-hosted runners with the same labels, the autoscaler adopts them and disables them at its next scale-down.

When you move from static runners to the autoscaler, start with a distinct label set (e.g. add autoscaled), move a low-risk repository onto it, and only reuse the old labels once the static runners and their Bitbucket registrations are gone.

For the same reason, changing a group’s labels later makes the autoscaler lose track of the runners it already registered. Plan a label change with Skyscrapers so the old runners get cleaned up.

At least one runner per group stays online

The autoscaler can’t scale a group to zero: it only scales up from runners that are online, so each group’s min is at least 1. That baseline runner keeps a small amount of capacity reserved on the cluster at all times, and also means a pipeline step never waits for a cold start.

Scaling reacts on the polling interval

The autoscaler checks Bitbucket every 10 minutes by default. During a burst, steps queue on the existing runners until the next poll adds capacity, and idle runners are removed after a 5-minute cool-down at the earliest. Both intervals can be tuned; tell us if your pipelines need faster scale-up.

Node pool and resources

Runner Jobs are scheduled with the tolerations and node selector from the configuration, typically your CI node pool, so builds don’t compete with your applications. Each Job requests the resources configured for its group plus those of the Docker daemon, which Karpenter uses to size the nodes. If your steps are heavier than the defaults, ask us to raise the requests; a step that exceeds its memory limit gets killed.

Rotate the API token

Create a new token as described in Bitbucket authentication, then replace the Secret and restart the autoscaler so it picks up the new value:

kubectl create secret generic bitbucket-runner-credentials \
  -n bitbucket-runners \
  --from-literal=atlassianApiToken=<new-api-token> \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n bitbucket-runners rollout restart deployment/runner-controller deployment/runner-controller-cleaner

Revoke the old token in Atlassian once the autoscaler is running with the new one.