Skip to content

Vault Agent Sidecar for Applications

How CWIQ application containers receive secrets at runtime using a Vault Agent sidecar, without storing credentials on disk or in environment variables baked into the image.

For long-running application containers, secrets cannot be fetched once at job start the way CI/CD jobs do. Application containers need their secrets renewed automatically and they must never write credentials to disk. The Vault Agent sidecar pattern solves both requirements.

Architecture

flowchart TB
    subgraph COMPOSE["Docker Compose Stack"]
        AGENT["Vault Agent\n(sidecar container)"]
        APP["Application Container\n(e.g. orchestrator-server)"]
        VOL[("tmpfs volume\n/vault/secrets/")]

        AGENT -->|"AppRole auth\n→ renewable token"| VAULT["Vault\nvault.shared.cwiq.io"]
        VAULT -->|"secret values"| AGENT
        AGENT -->|"render .env template\nto tmpfs"| VOL
        APP -->|"source .env\nat container startup"| VOL
    end

    AGENT -->|"token renewal\nevery 45 min"| VAULT
    AGENT -->|"secret refresh\nevery 5 min"| VAULT

The tmpfs volume is a RAM-backed filesystem. Its contents are never written to the host's disk and disappear when the container stops. This satisfies the requirement that secrets are never persisted to storage.

How It Works

  1. Vault Agent authenticates to Vault using AppRole credentials (Role ID + Secret ID) provided as environment variables at container start.
  2. Vault Agent reads a template file (/vault/templates/.env.tpl) that describes which secrets to fetch and how to format them.
  3. Vault Agent renders the template with the secret values and writes the output to /vault/secrets/.env on the shared tmpfs volume.
  4. Vault Agent monitors the token and renews it before expiry (default: every 45 minutes).
  5. Vault Agent re-renders the template whenever a secret changes (default: every 5 minutes).
  6. The application container starts with a depends_on: condition: service_healthy on the Vault Agent, ensuring secrets are available before the app process launches.
  7. The application entrypoint sources the .env file: set -a && . /vault/secrets/.env && set +a && exec <app-command>.

Template Syntax

Templates use Go's text/template syntax with Vault Agent's built-in secret function. The with secret block returns an error if the secret path does not exist, which causes Vault Agent to exit and prevents the application from starting with missing credentials.

{{- with secret "secret/data/cwiq/dev/orchestrator/database" }}
DATABASE_URL=postgresql+asyncpg://{{ .Data.data.pg_user }}:{{ .Data.data.pg_password }}@{{ .Data.data.pg_host }}:{{ .Data.data.pg_port }}/{{ .Data.data.pg_name }}
{{- end }}

{{- with secret "secret/data/cwiq/dev/orchestrator/config" }}
SECRET_KEY={{ .Data.data.secret_key }}
{{- end }}

The rendered output is a standard shell .env file:

DATABASE_URL=postgresql+asyncpg://orchestrator:s3cret@10.1.35.46:5432/orchestrator
SECRET_KEY=abc123...

Note the data.data double key.

KV v2 wraps the actual secret data inside a data envelope. When reading secret/data/cwiq/..., the secret values are at .Data.data.field, not .Data.field. This is a common source of confusion when first writing templates.

Docker Compose Pattern

# docker-compose.yml (excerpt)
services:
  vault-agent:
    image: hashicorp/vault:1.17.0
    command: ["vault", "agent", "-config=/vault/config/config.hcl"]
    restart: on-failure
    environment:
      VAULT_ADDR: https://vault.shared.cwiq.io
      VAULT_ROLE_ID: ${VAULT_ROLE_ID}
      VAULT_SECRET_ID: ${VAULT_SECRET_ID}
    volumes:
      - ./vault/config.hcl:/vault/config/config.hcl:ro
      - ./vault/templates:/vault/templates:ro
    tmpfs:
      - /vault/secrets:mode=0700,uid=1000,gid=1000
    healthcheck:
      test: ["CMD", "test", "-f", "/vault/secrets/.env"]
      interval: 5s
      timeout: 3s
      retries: 10

  server:
    image: nexus.shared.cwiq.io:8444/orchestrator-server:latest
    depends_on:
      vault-agent:
        condition: service_healthy
    volumes:
      - /vault/secrets:/vault/secrets:ro
    entrypoint:
      - /bin/sh
      - -c
      - "set -a && . /vault/secrets/.env && set +a && exec python -m orchestrator"

volumes:
  vault-secrets: {}

Secrets live on tmpfs only — never on a named volume.

The tmpfs: key on the vault-agent service mounts /vault/secrets as a RAM filesystem. If you use a named Docker volume instead of tmpfs:, secrets will be written to the host filesystem, violating the security requirement. Always use tmpfs: for the secrets mount on the sidecar.

depends_on: condition: service_healthy prevents race conditions.

Without this condition, the application container may start before Vault Agent has rendered the .env file, causing the entrypoint to fail when sourcing a non-existent file. The healthcheck on vault-agent uses test -f /vault/secrets/.env to signal readiness.

Vault Agent Configuration File

# vault/config.hcl
vault {
  address = "https://vault.shared.cwiq.io"
}

auto_auth {
  method "approle" {
    config = {
      role_id_env_var   = "VAULT_ROLE_ID"
      secret_id_env_var = "VAULT_SECRET_ID"
    }
  }

  sink "file" {
    config = {
      path = "/vault/secrets/.vault-token"
    }
  }
}

template {
  source      = "/vault/templates/.env.tpl"
  destination = "/vault/secrets/.env"
  perms       = "0640"
}

Token Lifecycle

Vault Agent manages the entire token lifecycle automatically. No application code needs to handle token renewal.

Event When What Happens
Initial auth Container start AppRole auth, token issued with configured TTL
Token renewal Every 45 min (configurable) Vault Agent renews the token before expiry
Secret refresh Every 5 min (configurable) Vault Agent re-renders templates with latest values
Auth failure If Role ID / Secret ID invalid Vault Agent exits with non-zero code; container restarts
Secret rotation When a secret value changes in Vault Template re-rendered on next refresh cycle

Provisioning Role ID and Secret ID

The Role ID and Secret ID for each application are provisioned by Ansible during deployment. The Ansible role writes them as environment variables to the host's docker-compose.env file, which Docker Compose reads at startup via env_file:.

Never hardcode Role ID or Secret ID values in Docker Compose files or application configuration. They are infrastructure-level credentials managed exclusively by Ansible.