Concourse usage

Login with fly

fly can be used to access multiple Concourse servers, to accomplish that it has the concept of targets. Each target represents a different Concourse setup, and you need to provide the alias of such target to all the fly commands with the -t flag, so it knows which Concourse server to reach. You can choose the name of your targets, it can be an arbitrary string, and it’ll only apply to your computer.

fly -t youralias login --team-name yourcompanyname --concourse-url https://ci.yourcompanyname.com

Note: from here on you’ll need to specify the youralias target in each fly command with -t youralias.

Useful commands and procedures

Set pipelines

fly -t youralias set-pipeline -p <pipeline-name> -c <path-to-the-ci-dir>/pipelines/<pipeline-name>.yaml

For example:

fly -t youralias set-pipeline -p k8s-staging -c ci/pipelines/k8s-staging.yaml

Get the newest fly

Either download the newest binary from your Concourse web UI or execute fly -t youralias sync

Removing a stalled worker

Show Concourse workers: fly -t youralias workers
Terminate the stalled worker instance (via EC2 dashboard, cli, whatever)
Prune the worker from Concourse: fly -t youralias prune-worker -w <worker name>

Secrets

Sometimes you’ll need to set secrets or other sensitive data in your pipelines, such as AWS or DockerHub credentials. You can handle this in a few ways: by integrating with Kubernetes or AWS Secrets Manager, or by using pipeline parameters to keep sensitive data out of the pipeline definitions.

Kubernetes integration

Concourse can read secrets from the Kubernetes cluster. This is one of the most straight forward ways to provide secrets to your pipelines. Get in touch with us through Slack or GH issue to get you started to setup the needed namespace(s) and/or RBAC.

Auto-update pipelines

Concourse pipelines can also be automated through Concourse itself, so you don’t need to create and update pipelines manually via fly anymore. This is done using the Concourse pipeline resource. We highly encourage you to go over the resource documentation to know how to set it up, although we’ll provide you with a simple example below.

mother-of-pipelines.yaml:

---
resource_types:
  - name: concourse-pipeline
    type: docker-image
    source:
      repository: concourse/concourse-pipeline-resource
      tag: latest

resources:
  - name: git-ci
    type: git
    source:
      uri: git@github.com:example/ci.git
      branch: master

  - name: concourse
    type: concourse-pipeline
    source:
      target: https://ci.example.com
      teams:
      - name: yourteamname
        username: concourse
        password: ((CONCOURSE_PASSWORD))

jobs:
  - name: deploy-pipelines
    plan:
      - get: git-ci
        trigger: true
      - put: concourse
        params:
          pipelines_file: git-ci/pipelines-file.yaml

pipelines-file.yaml being:

---
pipelines:
  - name: puppet
    team: yourteamname
    config_file: git-ci/puppet/pipeline.yaml
    unpaused: true
  - name: charts
    team: yourteamname
    config_file: git-ci/charts/pipeline.yaml
    unpaused: true
  - name: docker-images
    team: yourteamname
    config_file: git-ci/docker-images/pipeline.yaml
    unpaused: true

With the previous example, Concourse will automatically update all pipelines specified in the pipelines-file.yaml file whenever their config files get updated. Also, if you need to add a new pipeline, you just specify it in the pipelines-file.yaml file and Concourse will create it for you. Of course, you still need to run fly set-pipeline to create the main mother-of-pipelines pipeline and keep it up-to-date.

There’s one important thing to consider here though: whether you’re using a secrets manager for providing the pipeline ((params)) or not. If you’re using a secrets manager, then you don’t need to make further changes in your current pipelines, as they’ll keep picking up their ((params)) from the secret store. But if you’re using an external params.yaml or secrets.yaml file you might have some extra work ahead. If this is your case, keep reading.

docker-image deprecation

In the new Concourse 5.0.0 version, a new resource was released to track and upload Docker images to a registry, the registry-image-resource. This new resource is intended to replace the current docker-image-resource, as it’s more lightweight and simpler. Concourse announced that they intend to deprecate the current docker-image-resource in the future.

The only caveat is that the new registry-image-resource doesn’t build images, so you need to do that a task before the put step.

Example:

---
resources:
# Git repos
 - name: docker-mongodb-exporter-git
  type: git
  icon: github-circle
  source:
    uri: https://github.com/dcu/mongodb_exporter.git
    branch: master

 - name: mongodb-exporter-image
-  type: docker-image
+  type: registry-image
  icon: docker
  check_every: 24h
  source:
    repository: skyscrapers/mongodb-exporter
    username: ((DOCKER_HUB_SKYSCRAPERS_CREDS.username))
    password: ((DOCKER_HUB_SKYSCRAPERS_CREDS.password))

jobs:
  plan:
    - get: docker-mongodb-exporter-git
      trigger: true
+    - task: docker-build
+      privileged: true
+      config:
+        platform: linux
+        image_resource:
+          type: registry-image
+          source:
+            repository: vito/oci-build-task
+        inputs:
+          - name: docker-mongodb-exporter-git
+            path: .
+        outputs:
+          - name: image
+        caches:
+          - path: cache
+        run:
+          path: build
    - put: mongodb-exporter-image
      params:
-        build: docker-mongodb-exporter-git
-        tag_as_latest: true #(tag_as_latest is default in the new registry-image resource)
+        image: image/image.tar

Adding extra auth credentials for registry-image build

DockerHub

Example to add DockerHub credentals, eg. when being rate-limited for pulls:

jobs:
  plan:
    - get: docker-mongodb-exporter-git
      trigger: true
    - task: create_creds
      config:
        platform: linux
        image_resource:
          type: registry-image
          source:
            repository: alpine
            tag: latest
            username: ((DOCKERHUB_USERNAME))
            password: ((DOCKERHUB_PASSWORD))
        outputs:
          - name: docker_creds
        run:
          path: sh
          args:
            - -exc
            - |
              AUTH="$(echo -n '((DOCKERHUB_USERNAME)):((DOCKERHUB_PASSWORD))' | base64 -w 0)"
              mkdir -p docker_creds
              cat > docker_creds/config.json <<EOF
              { "auths": { "https://index.docker.io/v1/": { "auth": "$AUTH" }}}
              EOF
    - task: docker-build
      privileged: true
      config:
        platform: linux
        image_resource:
          type: registry-image
          source:
            repository: vito/oci-build-task
        params:
          DOCKERFILE: docker/Dockerfile
          DOCKER_CONFIG: docker_creds
        inputs:
          - name: docker-mongodb-exporter-git
            path: .
          - name: docker_creds
        outputs:
          - name: image
        caches:
          - path: cache
        run:
          path: build

ECR

Example to add login with aws to authenticate to ECR:

jobs:
  plan:
    - get: docker-mongodb-exporter-git
      trigger: true
    - task: create_creds
      config:
        platform: linux
        image_resource:
          type: registry-image
          source:
            repository: skyscrapers/concourse-ecr-login
            tag: latest
        outputs:
          - name: docker_creds
        run:
          path: sh
          args:
            - -exc
            - |
              export AWS_ACCESS_KEY_ID=((OPS_AWS_ACCESS_KEY_ID))
              export AWS_SECRET_ACCESS_KEY=((OPS_AWS_SECRET_ACCESS_KEY))
              aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin ((OPS_AWS_ACCOUNT_ID)).dkr.ecr.((OPS_AWS_REGION)).amazonaws.com
              cp /root/.docker/config.json docker_creds/config.json
    - task: docker-build
      privileged: true
      config:
        platform: linux
        image_resource:
          type: registry-image
          source:
            repository: vito/oci-build-task
        params:
          DOCKERFILE: docker/Dockerfile
          DOCKER_CONFIG: docker_creds
        inputs:
          - name: docker-mongodb-exporter-git
            path: .
          - name: docker_creds
        outputs:
          - name: image
        caches:
          - path: cache
        run:
          path: build

Last updated on August 25, 2026