Use AWS Secrets Manager secrets in Kubernetes

It is possible to automatically mount and use secrets from AWS Secrets Manager in EKS workloads. This documentation page will show you how. Note that this feature is optional for our EKS platform, and it’s disabled by default. If you’d like to enable it for your cluster(s) please reach out to us.

When enabled, two extra components will be deployed on the EKS cluster: the Secrets Store CSI driver and the AWS Secrets Manager provider. The CSI driver provides the in-cluster functionality to be able to mount secrets from external providers into Pods via a Container Storage Interface (CSI) volume, and the AWS Secrets Manager provider integrates it with the AWS service.

Note that besides mounting secrets into the container’s file system, it’s also possible to sync them to actual Kubernetes Secret objects and use them as environment variables in Pods.

Usage

To be able to mount a secret from AWS Secrets Manager, you first need to create a SecretProviderClass in the namespace you’re going to be using it:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "arn:aws:secretsmanager:eu-west-1:[111122223333]:secret:MySecret-00AACC"
        jmesPath:
          - path: user.username
            objectAlias: dbusername
          - path: user.password
            objectAlias: dbpassword
      - objectName: "arn:aws:secretsmanager:eu-west-1:[111122223333]:secret:MySecret2-00AABB"
        objectAlias: foobar

You can specify multiple secrets in the same SecretProviderClass, and all of them will be mounted in the same directory when referenced in the Pod. For secrets in JSON format, you can use the jmesPath attribute to filter which attributes from the JSON object to map. Provided the following JSON object, the example SecretProviderClass above would result in two files, dbusername and dbpassword with the respective contents john.doe and 123456, plus the file foobar for the secret MySecret2-00AABB, which will contain the whole secret value, regardless of its format.

{
  "user": {
    "username": "john.doe",
    "password": "123456"
  }
}

Then the SecretProviderClass needs to be referenced from the Pod as a volume, like:

kind: Pod
apiVersion: v1
metadata:
  name: test-secrets
spec:
  containers:
    - name: busybox
      image: registry.k8s.io/e2e-test-images/busybox:1.29
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
        - name: secrets
          mountPath: "/mnt/secrets-store"
          readOnly: true
  volumes:
    - name: secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: aws-secrets

The secretProviderClass in volumeAttributes needs to match the name of the SecretsProviderClass defined above.

To be able to use a secret value as an environment variable, you need to specify the secretObjects attribute in the SecretProviderClass object, like:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  secretObjects:
    - secretName: foosecret
      type: Opaque
      data:
        - objectName: foobar # name of the mounted content to sync. this could be the object name or object alias 
          key: username
  parameters:
    objects: |
      - objectName: "arn:aws:secretsmanager:eu-west-1:[111122223333]:secret:MySecret-00AACC"
        jmesPath:
          - path: user.username
            objectAlias: dbusername
          - path: user.password
            objectAlias: dbpassword
      - objectName: "arn:aws:secretsmanager:eu-west-1:[111122223333]:secret:MySecret2-00AABB"
        objectAlias: foobar

This SecretProviderClass will instruct the CSI driver to create and sync the specified Secret objects with the contents of the referenced mounted secrets. To be able to use the Secret you still need to define the volumes and volumeMounts in the Pod specification. It is important that the volume is mounted in at least a container, even if it’s not going to be used, otherwise the controller won’t sync the Secret object.

kind: Pod
apiVersion: v1
metadata:
  name: test-secrets
spec:
  containers:
    - name: busybox
      image: registry.k8s.io/e2e-test-images/busybox:1.29
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
        - name: secrets
          mountPath: "/mnt/secrets-store"
          readOnly: true
      env:
        - name: SECRET_USERNAME
          valueFrom:
            secretKeyRef:
              name: foosecret
              key: username
  volumes:
    - name: secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: aws-secrets

Note that the synced Secret object will only exist as long as there’s a Pod mounting the SecretProviderClass. If no Pods are using the SecretProviderClass, the CSI driver will stop syncing the Secret object and delete it.

More information about this limitation and others can be found in the CSI driver documentation.

Automatic secret rotation

Instead of writing the secrets once, the CSI driver can poll Secrets Manager and update the mounted files and the synced Secret objects in place, without restarting your Pods. This is optional and disabled by default. To enable it, add the following to your cluster definition:

spec:
  secretsManagerCSIDriver:
    enabled: true
    secretRotation:
      enabled: true
      pollInterval: 2m

• pollInterval: how often the driver checks Secrets Manager for new values, as a Go duration string (e.g. 2m, 15m, 1h). Defaults to 2m.

The driver polls Secrets Manager; it doesn’t get notified of changes. A new secret value takes up to one pollInterval to reach your Pods.

With rotation enabled:

• The mounted files are rewritten with the new value. Your application has to re-read the file; if it only reads the secret at startup, it still needs a restart.
• The synced Secret objects are updated, including when you add or remove objects or change keys in secretObjects.
• Environment variables are not updated. A container’s environment is set when it starts, so a Pod that consumes the synced Secret through secretKeyRef keeps the old value until it restarts. Mount the secret as a file instead, or restart your Deployment after a rotation.

The driver records each rotation cycle as an event on the Pod, with reason MountRotationComplete or SecretRotationComplete on success and MountRotationFailed or SecretRotationFailed on failure:

kubectl describe pod <pod>

When a rotation fails, the Pod keeps running with the last value that was fetched successfully. To see which secret versions a Pod currently has:

kubectl get secretproviderclasspodstatus <pod>-<namespace>-<secretproviderclass> -o yaml

Secret rotation is an alpha feature of the Secrets Store CSI driver. If you’d like to enable it for your cluster(s), or need help picking a pollInterval, please reach out to us.

Documentation references

You can read more on how the Secrets Store CSI driver works in the official documentation page. Similar with the AWS Secrets Manager provider, there’s some more information in the official AWS documentation.