Vault Integration¶

Authentik HA uses a Vault Agent sidecar to inject secrets at runtime. Secrets are rendered to a tmpfs volume — they never touch disk on the EC2 instances.

Overview¶

The Vault Agent sidecar pattern provides: - No raw secrets on disk — all secrets rendered to RAM (tmpfs) - Automatic secret refresh every 5 minutes - Centralized secret management in HashiCorp Vault - AppRole machine-to-machine authentication with short-lived tokens (1h TTL)

Secret Paths¶

All Authentik secrets live under secret/cwiq/shared/authentik/:

Secret	Vault Path	Injected As
RDS host	`secret/data/cwiq/shared/authentik/database` → `pg_host`	`AUTHENTIK_POSTGRESQL__HOST`
RDS user	`secret/data/cwiq/shared/authentik/database` → `pg_user`	`AUTHENTIK_POSTGRESQL__USER`
RDS password	`secret/data/cwiq/shared/authentik/database` → `pg_password`	`AUTHENTIK_POSTGRESQL__PASSWORD`
RDS database	`secret/data/cwiq/shared/authentik/database` → `pg_name`	`AUTHENTIK_POSTGRESQL__NAME`
Secret key	`secret/data/cwiq/shared/authentik/config` → `secret_key`	`AUTHENTIK_SECRET_KEY`

Secrets Injection Flow¶

Ansible deploys: role-id, secret-id, config.hcl to /data/authentik/vault/config/
        ↓
Container startup:
  vault-agent reads role-id + secret-id
  vault-agent POSTs to Vault: /v1/auth/approle/login
  Vault returns token (1h TTL)
  vault-agent GETs secrets: /v1/secret/data/cwiq/shared/authentik/database
  vault-agent renders .env.tpl → /vault/secrets/.env (tmpfs)
  vault-agent health check passes (.env exists)
        ↓
authentik-server and authentik-worker start
(both depend_on: vault-agent healthy)
  Each sources /vault/secrets/.env on startup
  Connects to RDS using credentials from .env
        ↓
Runtime (every 5 minutes):
  vault-agent re-fetches secrets from Vault
  vault-agent re-renders .env if changed
  Token renewed at 45 min (before 1h expiry)

Initial Setup¶

Step 1: Create Vault Policy¶

vault policy write cwiq-shared-authentik-read - <<'EOF'
path "secret/data/cwiq/shared/authentik/*" {
  capabilities = ["read", "list"]
}
path "secret/metadata/cwiq/shared/authentik/*" {
  capabilities = ["read", "list"]
}
EOF

Step 2: Create AppRole¶

# Enable AppRole auth if not already enabled
vault auth enable approle 2>/dev/null || echo "Already enabled"

# Create Authentik AppRole
vault write auth/approle/role/authentik \
  token_policies="cwiq-shared-authentik-read" \
  token_ttl=1h \
  token_max_ttl=4h \
  secret_id_ttl=720h \
  secret_id_num_uses=0

# Get Role ID (stable — safe to store in config)
vault read auth/approle/role/authentik/role-id

# Generate Secret ID (sensitive — treat as a credential)
vault write -f auth/approle/role/authentik/secret-id

Step 3: Store Secrets in Vault¶

# Generate RDS password
RDS_PASSWORD=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | head -c 32)

# Store database credentials
vault kv put secret/cwiq/shared/authentik/database \
  pg_host="cwiq-shared-authentik-postgres.xxxxx.us-west-2.rds.amazonaws.com" \
  pg_user="authentik" \
  pg_password="${RDS_PASSWORD}" \
  pg_name="authentik" \
  pg_port="5432"

# Generate and store secret key (must be 50+ chars)
AUTHENTIK_SECRET_KEY=$(openssl rand -base64 64 | tr -dc 'a-zA-Z0-9' | head -c 64)
vault kv put secret/cwiq/shared/authentik/config \
  secret_key="${AUTHENTIK_SECRET_KEY}"

Step 4: Update RDS Endpoint After Terraform Apply¶

# Get actual RDS endpoint from Terraform
cd terraform-plan/organization/environments/shared-services/rds/authentik
RDS_ENDPOINT=$(terraform output -raw db_address)

# Update Vault with the actual endpoint
vault kv patch secret/cwiq/shared/authentik/database \
  pg_host="${RDS_ENDPOINT}"

Step 5: Deploy Authentik with Vault Agent¶

ssh ansible@ansible-shared-cwiq-io
ansible-helper
cd authentik

ROLE_ID=$(vault read -field=role_id auth/approle/role/authentik/role-id)
SECRET_ID=$(vault write -f -field=secret_id auth/approle/role/authentik/secret-id)

ansible-playbook -i inventory.ini setup.yml \
  -e "vault_enabled=true" \
  -e "vault_role_id=${ROLE_ID}" \
  -e "vault_secret_id=${SECRET_ID}"

Files Involved¶

File	Location on EC2	Purpose
`config.hcl`	`/data/authentik/vault/config/`	Vault Agent config (address, auth, template)
`role-id`	`/data/authentik/vault/config/`	AppRole identifier (stable, deployed by Ansible)
`secret-id`	`/data/authentik/vault/config/`	AppRole credential (one-time, deployed at startup)
`.env.tpl`	`/data/authentik/vault/templates/`	Template with Vault secret references
`.env`	`/vault/secrets/` (tmpfs)	Rendered secrets — RAM only, never persisted

Rotating Secrets¶

Rotate RDS Password¶

NEW_PASSWORD=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | head -c 32)

# 1. Update Vault
vault kv patch secret/cwiq/shared/authentik/database pg_password="${NEW_PASSWORD}"

# 2. Update RDS
aws rds modify-db-instance \
  --db-instance-identifier cwiq-shared-authentik-postgres \
  --master-user-password "${NEW_PASSWORD}" \
  --apply-immediately \
  --profile shared-services

# 3. Wait up to 5 minutes for Vault Agent to pick up and re-render
# Or restart Authentik containers to force immediate pickup:
ansible-playbook -i inventory.ini restart.yml

Rotate AUTHENTIK_SECRET_KEY¶

Rotating the secret key invalidates all user sessions

All users will be logged out. Schedule a maintenance window.

NEW_KEY=$(openssl rand -base64 64 | tr -dc 'a-zA-Z0-9' | head -c 64)
vault kv patch secret/cwiq/shared/authentik/config secret_key="${NEW_KEY}"

# Restart all Authentik instances to pick up new key
ansible-playbook -i inventory.ini restart.yml

Regenerate Secret ID¶

If the Vault AppRole Secret ID is compromised or lost:

# Generate new Secret ID
vault write -f auth/approle/role/authentik/secret-id

# Redeploy with new Secret ID
ansible-playbook -i inventory.ini setup.yml \
  -e "vault_enabled=true" \
  -e "vault_role_id=<existing_role_id>" \
  -e "vault_secret_id=<new_secret_id>"

Ansible Policy Requirement¶

The ansible user's Vault policy must include AppRole management capabilities. Use + (single path segment wildcard) — not *:

# Correct — + matches single segment
path "auth/approle/role/+/role-id" {
  capabilities = ["read"]
}
path "auth/approle/role/+/secret-id" {
  capabilities = ["create", "update"]
}

After updating the policy, clear the cached token on the ansible server:

rm -f ~/.vault-token
vault-login
vault token capabilities auth/approle/role/authentik/role-id
# Expected: read

Troubleshooting¶

Vault Agent Cannot Authenticate¶

# Check vault-agent logs
docker compose logs vault-agent --tail 50

# Test AppRole auth manually
ROLE_ID=$(cat /data/authentik/vault/config/role-id)
SECRET_ID=$(cat /data/authentik/vault/config/secret-id)
vault write auth/approle/login role_id="${ROLE_ID}" secret_id="${SECRET_ID}"

Secrets Not Rendered¶

# Check if .env file exists in tmpfs
docker compose exec server cat /vault/secrets/.env

# Manually trigger re-render by restarting vault-agent
docker compose restart vault-agent
# Wait ~10 seconds for vault-agent to re-authenticate and render
docker compose restart server worker

Verify Current Secret Values¶

vault kv get secret/cwiq/shared/authentik/database
vault kv get secret/cwiq/shared/authentik/config

Security Properties¶

Property	Detail
Role ID	Stable — safe to store in config files on disk
Secret ID	Sensitive — passed at deployment time, not stored long-term
Token TTL	1 hour — Vault Agent auto-renews at 45 minutes
Secret refresh	Every 5 minutes — live rotation without restart
tmpfs volume	Rendered secrets never touch EC2 disk
Encryption	All Vault secrets encrypted at rest (AES-256-GCM) and in transit (TLS)

Architecture — HA infrastructure, Vault Agent sidecar diagram
SSL: Architecture — Cert-server manages SSL certificates (separate from Vault)