Skip to content

Vault Agent Sidecar

The Vault Agent sidecar pattern injects secrets into application containers as environment variables from a tmpfs volume. No secrets are stored on disk or visible via docker inspect.

How It Works

Vault Server
    | AppRole auth (role-id + secret-id)
    v
vault-agent container (sidecar)
    | reads template /vault/templates/.env.tpl
    | fetches secrets from Vault
    | renders to /vault/secrets/.env (tmpfs, RAM only)
    v
application containers
    | mount /vault/secrets as read-only volume
    | source .env at startup via shell command

The application container reads secrets from the shared tmpfs volume. The sidecar auto-refreshes every 5 minutes and renews the Vault token every 45 minutes.

Security Benefits

Before (plain env vars) After (Vault Agent)
Secrets in group_vars/all.yml Only Vault address + AppRole config stored
.env files on disk Secrets only in tmpfs (RAM, never on disk)
Visible via docker inspect Not visible via docker inspect
Manual rotation Automatic refresh with TTL
No audit trail Full audit trail of every access

Directory Structure on the Host

The vault_agent Ansible role deploys this structure:

/data/{app-name}/vault/
├── config/
│   ├── config.hcl       # Vault Agent configuration
│   ├── role-id          # AppRole Role ID (stable)
│   └── secret-id        # AppRole Secret ID (one-time use)
└── templates/
    └── .env.tpl         # Jinja2-style template with Vault secret references

The secrets/ directory is a Docker tmpfs volume — it exists only in RAM and is never written to disk.

Docker Compose Integration

volumes:
  vault-secrets:
    driver: local
    driver_opts:
      type: tmpfs
      device: tmpfs

services:
  vault-agent:
    image: hashicorp/vault:latest
    container_name: ${APP_NAME}-vault-agent
    restart: unless-stopped
    command: ["agent", "-config=/vault/config/config.hcl"]
    volumes:
      - /data/${APP_NAME}/vault/config:/vault/config:ro
      - /data/${APP_NAME}/vault/templates:/vault/templates:ro
      - vault-secrets:/vault/secrets
    healthcheck:
      test: ["CMD", "test", "-f", "/vault/secrets/.env"]
      interval: 5s
      timeout: 3s
      retries: 12

  app:
    image: ${APP_IMAGE}
    container_name: ${APP_NAME}
    restart: unless-stopped
    # Source the .env file at startup — do NOT use env_file directive
    command: ["sh", "-c", "set -a && . /vault/secrets/.env && set +a && exec app-binary"]
    volumes:
      - vault-secrets:/vault/secrets:ro
    depends_on:
      vault-agent:
        condition: service_healthy

Do not use Docker env_file with Vault Agent

Docker's env_file is evaluated at container creation time, before Vault Agent has rendered the secrets file. The file will not exist yet. Always source the .env file via a shell command at runtime.

Template Syntax

Templates use Go template syntax with Vault functions. The key pattern is:

# /vault/templates/.env.tpl

{{ with secret "secret/data/cwiq/shared/<app>/database" }}
DB_HOST={{ .Data.data.pg_host }}
DB_PASSWORD={{ .Data.data.pg_password }}
DB_PORT={{ or .Data.data.pg_port "5432" }}
{{ end }}

{{ with secret "secret/data/cwiq/shared/<app>/config" }}
SECRET_KEY={{ .Data.data.secret_key }}
{{ end }}

# Static values (not from Vault)
LOG_LEVEL=info

The .Data.data.<key> path reflects the KV v2 API structure. The outer .Data is the Vault response wrapper; the inner .data is the actual secret payload.

Ansible Role: vault_agent

Use the shared vault_agent role to deploy the sidecar configuration:

# In your application's setup.yml
- name: Deploy Vault Agent sidecar
  include_role:
    name: vault_agent
  vars:
    vault_agent_app_name: "<app-name>"
    vault_role_id: "{{ vault_role_id }}"         # From group_vars/all.yml
    vault_secret_id: "{{ vault_secret_id }}"     # From group_vars/all.yml
    vault_agent_template_src: "{{ playbook_dir }}/templates/<app>-env.tpl.j2"
    vault_agent_template_dest: ".env"
    vault_agent_render_interval: "5m"
  when: vault_enabled | default(false)

Role Variables Reference

Variable Required Default Description
vault_addr No https://vault.shared.cwiq.io Vault server URL
vault_agent_app_name Yes — Application name (used for paths)
vault_role_id Yes — AppRole Role ID
vault_secret_id Yes — AppRole Secret ID
vault_agent_template_src Yes — Path to template file
vault_agent_template_dest No .env Rendered output filename
vault_agent_render_interval No 5m Secret refresh interval

Verifying the Sidecar is Working

# Check vault-agent container is healthy
ssh ec2-user@<hostname>-cwiq-io \
  "docker ps --filter 'name=<app>-vault-agent' --format '{{.Status}}'"
# Expected: "healthy"

# Check secrets file exists and is non-empty (tmpfs)
ssh ec2-user@<hostname>-cwiq-io \
  "docker exec <app>-vault-agent ls -la /vault/secrets/"
# Expected: .env file present

# Check vault-agent logs for errors
ssh ec2-user@<hostname>-cwiq-io \
  "docker logs <app>-vault-agent --tail 20"
# Look for: "template rendered", no "error" lines