Argo Workflows

Argo Workflows runs multi-step jobs on your cluster: data pipelines, batch processing, scheduled jobs, anything you can describe as a series of containers. We run Argo for you, one installation per cluster, and set up a namespace per team with access and artifact storage. You write and own the workflows.

The component is off by default. This page explains what you get, how to turn it on, and the rules your workflows run under.

What you get

• The workflow controller, which runs your Workflows, CronWorkflows and WorkflowTemplates. We install, configure and upgrade it, like every other component of your platform.
• The Argo UI and API on https://argo.<clusterFQDN>. It is only reachable on the internal ingress, so over your VPN, never from the internet.
• Single sign-on through your cluster’s Dex, so people log in with your own identity provider. What they see after logging in depends on their groups.
• Workflow namespaces, one per team or environment, each with three roles, its own share of an S3 bucket for artifacts, and its own IAM role.
• Metrics and alerts. Prometheus scrapes the controller’s metrics, so you can graph and alert on your workflows in Grafana. The control plane itself is covered by our platform alerts and our on-call, like any other platform component. We don’t alert on your workflows failing: that is your logic, and you know best which failures matter.

Turn it on

Ask us to enable it, or add an argoWorkflows block to your cluster definition (k8s-clusters/<cluster>.yaml). Your cluster needs Kubernetes stack release 15.12.0 or later, and the internal Traefik, which most clusters already have. A typical setup:

spec:
  argoWorkflows:
    enabled: true
    artifactRepository:
      expirationDays: 30
    workflowDefaults:
      tolerations:
        - key: role
          operator: Equal
          value: workflows
          effect: NoSchedule
    namespaces:
      - name: data-pipelines
        labels:
          cost-center: data
        serviceAccounts:
          - data-exporter
        admins:
          groups:
            - data_infrastructure
          ssoGroups:
            - Data Infrastructure
        developers:
          groups:
            - data_developers
          ssoGroups:
            - Data
        viewers:
          groups:
            - platform_team

What each field does:

• namespaces[]: the namespaces your workflows run in. We create each one, or take over one that already exists. name is also part of the namespace’s IAM role name (<cluster>-argo-<name>), which has to stay within 64 characters, so keep it short.
• admins, developers, viewers: who holds which role in that namespace. Each takes two lists, and you’ll usually fill in both for the same team:
  • groups are Kubernetes groups, the ones your EKS access entries map your IAM roles to. They apply to kubectl, the argo CLI and CI jobs that talk to the cluster with their own credentials.
  • ssoGroups are group names as your identity provider sends them through Dex, for example the name of an Okta or Google group. They apply when someone logs in to the UI.
• labels: extra labels on the namespace, for example for cost allocation. You can’t set prometheus; we use that one.
• serviceAccounts: other ServiceAccounts that pods in the namespace may run as. By default, workflows run as argo-workflow. See Guardrails before you add one.
• workflowDefaults: fields every Workflow in the cluster gets unless it sets them itself, such as tolerations or a nodeSelector to run workflows on a dedicated node pool. A key you set here replaces our default for that key. Pod labels and annotations are merged instead. You can’t set serviceAccountName, automountServiceAccountToken or executor; we set those so workflow pods don’t carry a token they don’t need.
• controller.resources, server.resources, executor.resources: CPU and memory for the controller, the UI/API server, and the small init and wait containers Argo adds to every workflow pod. The defaults suit most clusters, and the controller and server scale their memory up automatically when they need more. Raise the executor’s only if your workflows move large artifacts.
• artifactRepository.expirationDays: how long artifacts stay in the bucket. Default: 30 days.

To run workflows on a dedicated, tainted node pool, a toleration alone isn’t enough: it lets workflow pods onto the pool but doesn’t send them there. Add a nodeSelector as well, in your Workflows or WorkflowTemplates:

spec:
  nodeSelector:
    karpenter.sh/nodepool: workflows

You can also put the nodeSelector in workflowDefaults, next to the toleration, to send every Workflow in the cluster to that pool.

Out of the box, every Workflow also gets these defaults: pods of a successful Workflow are cleaned up right away, a finished Workflow is deleted after 24 hours, and the controller keeps at most 100 completed, 100 failed and 100 errored Workflows. Set your own podGC or ttlStrategy on a Workflow, or in workflowDefaults, to change that.

Who can do what

Each workflow namespace has three roles. They only apply to that namespace.

When you log in to the UI, you get the role your ssoGroups give you in each namespace. If you’re in groups for two roles, you get the wider one.

Open the UI straight on your namespace: https://argo.<clusterFQDN>/workflows/<namespace>. The start page lists the workflows of all namespaces at once, so it shows a “Permission denied … list workflows in namespace argo” error. That’s expected: access is per namespace, and nobody has access to every namespace.

The argo CLI and CI jobs use their own Kubernetes credentials, so they get what your groups give them.

Skyscrapers engineers are viewers in every workflow namespace. We can look at your workflows and logs to help you troubleshoot, but we can’t change or run them. Admin rights stay with your own teams.

Artifacts

Each cluster gets one S3 bucket, <cluster>-argo-artifacts, encrypted, private and reachable over TLS only. Every workflow namespace is the default artifact repository for its own prefix in that bucket, so artifacts work without any configuration in your Workflows. They’re stored under <namespace>/<workflow>/<pod>.

Per namespace, we create an IAM role and policy named <cluster>-argo-<namespace>. The role is used by the argo-workflow ServiceAccount, and the policy only grants access to that namespace’s prefix. A workflow in one namespace can’t read another namespace’s artifacts.

If a workflow runs under one of your own ServiceAccounts (listed in serviceAccounts), attach the <cluster>-argo-<namespace> policy to that ServiceAccount’s IAM role so it can use the artifact bucket too.

Artifacts are deleted after expirationDays (30 by default). Keep anything you need for longer somewhere else.

Deliver your workflows

We set up Argo and the namespaces; the workflow definitions are yours. Keep them in a Git repository and deploy them in whichever of these ways suits you.

With Flux

Add a Flux Kustomization per workflow namespace to your infrastructure repository, in flux/clusters/<cluster>/, next to the other cluster manifests. Flux then applies everything under the path you choose:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: workflows-data-pipelines
  namespace: flux-system
spec:
  dependsOn:
    - name: system-addons
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./flux/workflows/data-pipelines
  targetNamespace: data-pipelines
  prune: true

• dependsOn: system-addons waits until the platform, including Argo and the namespace, is in place, so a fresh cluster doesn’t fail on missing Argo CRDs.
• targetNamespace puts everything in the workflow namespace, so the manifests don’t need a namespace: of their own.
• Put your WorkflowTemplates, CronWorkflows and WorkflowEventBindings under path. Flux keeps the namespace in line with Git, and removes what you delete there.

If your workflows live in a separate repository, point sourceRef at a GitRepository for it instead. The Flux page explains how.

From CI

A CI job can apply manifests with kubectl apply or submit runs with argo submit. It needs an IAM role that your EKS access entries map to a Kubernetes group listed in the namespace’s admins or developers. A developer can submit Workflows; changing templates or CronWorkflows needs admin.

From the UI or CLI

People in the right groups can submit a Workflow from an existing template in the UI, or with the argo CLI. That’s handy for a one-off run or a test. For anything you want to keep, put it in Git.

Guardrails

Which ServiceAccounts pods can use

In a workflow namespace, pods may only run as argo-workflow or as an account in that namespace’s serviceAccounts list. A policy on the cluster refuses any other pod. This applies to every pod in the namespace, workflow or not, so don’t run your applications in a workflow namespace.

The same policy refuses pods that run as, or mount the tokens of, the accounts Argo uses internally for logins and for its executor. Without it, a developer could write a Workflow that runs with admin rights.

Every developer of the namespace can run a Workflow under any account on the serviceAccounts list. Only list accounts whose permissions every developer may have.

Long jobs aren’t interrupted by node consolidation

Karpenter regularly moves pods to make better use of your nodes. Workflow pods carry the karpenter.sh/do-not-disrupt annotation, so Karpenter leaves a node alone while a workflow step runs on it. A long job finishes instead of being restarted halfway. The flip side: a node running a long step stays up until the step ends, which can delay consolidation and node upgrades.

No default deadline

We don’t put a time limit on Workflows. How long a run may take depends on the job, so set activeDeadlineSeconds on your Workflows or templates yourself. Without one, a stuck workflow keeps running, and keeps its node, until you stop it.

Turn it off or remove a namespace

Remove the namespace from namespaces[], or set argoWorkflows.enabled to false. We remove the roles, ServiceAccounts and IAM role, but we never delete a workflow namespace: it holds your workflows, templates and running pods. Delete the namespace yourself when you no longer need it.

Turning Argo off also removes the artifact bucket. Copy out any artifacts you want to keep first.