Skip to content

Vault Architecture

HashiCorp Vault is the secrets management system for the CWIQ infrastructure. It uses AWS KMS for auto-unseal, RDS PostgreSQL as the storage backend, and Tailscale for network isolation — no secrets are accessible from the public internet.

Overview

Property Value
Version HashiCorp Vault (community edition)
Host vault-shared-cwiq-io
URL https://vault.shared.cwiq.io
Storage RDS PostgreSQL (vault-shared-storage)
Auto-unseal AWS KMS (us-west-2)
Network Tailscale only — no public exposure
Playbook vault-server/setup.yml

Architecture

Internet (BLOCKED — Tailscale only)
        |
vault-shared-cwiq-io (Tailscale: vault-shared-cwiq-io)
  Vault container
  TLS: Let's Encrypt (vault.shared.cwiq.io)
    |  storage
    v
  RDS PostgreSQL (vault-shared-storage)
  Private subnet — accessible only from Vault EC2
    |
    |  auto-unseal
    v
  AWS KMS key (us-west-2)

Security Controls

Control Status Description
Network isolation Active Tailscale-only access (100.64.0.0/10), no public internet
TLS 1.2+ only Active HTTPS enforced, no HTTP
Memory locking Active Prevents secrets from swapping to disk
Audit logging Active All operations logged to /vault/logs/vault-audit.log
OIDC via Authentik Active SSO with MFA for interactive access
Root token Active Revoked after initial setup; regenerate only via recovery keys
Rate limiting Active 100 req/s with 200 burst
KV v2 versioning Active 10 versions retained per secret
Least privilege Active Per-application AppRole policies with minimal access

Deployment Sequence

The Vault infrastructure was deployed in five steps:

  1. Terraform: KMS key — AWS KMS key for auto-unseal
  2. Terraform: RDS — PostgreSQL instance in private subnet
  3. Terraform: EC2 + DNS — vault-shared-cwiq-io instance and Route53 record
  4. Ansible: setup.yml — Docker Compose, TLS certificates, Vault config
  5. Vault initialization — One-time init, recovery keys distributed, KMS auto-unseal configured, AppRole and OIDC auth methods enabled, root token revoked

Terraform modules are at terraform-plan/organization/environments/shared-services/ec2-instances/vault/.

How Auto-Unseal Works

Vault stores its encrypted encryption keys in RDS PostgreSQL. On startup, it calls the AWS KMS API to decrypt the root key, enabling access to secrets — no manual unseal keyholders needed.

If the KMS key is revoked or the EC2 IAM role loses kms:Decrypt permission, Vault will remain sealed after restart.

# Check seal status
curl -s https://vault.shared.cwiq.io/v1/sys/health | jq .sealed

# Response codes:
# 200 — Initialized, unsealed, active
# 429 — Unsealed, standby
# 503 — Sealed

Auth Methods

Method Used By Path
OIDC (Authentik) Interactive human login auth/oidc/
AppRole Ansible, applications, Vault Agent sidecars auth/approle/
JWT GitLab CI/CD pipelines auth/jwt/

See Auth Methods for configuration details and CI/CD Integration for the JWT workflow.

Secret Structure

All application secrets are stored under the KV v2 engine mounted at secret/:

secret/
├── cwiq/
│   ├── shared/
│   │   ├── authentik/database     — Authentik PostgreSQL credentials
│   │   ├── authentik/config       — Authentik secret key
│   │   ├── gitlab/oidc            — GitLab OIDC client credentials
│   │   ├── taiga/database         — Taiga DB credentials
│   │   └── icinga/database        — Icinga DB credentials
│   └── dev/
│       └── ...
├── sonarqube/
│   ├── admin                      — SonarQube admin credentials
│   └── svc-orchestrator           — SonarQube CI token
├── defectdojo/
│   ├── admin
│   └── svc-orchestrator
├── nexus/
│   ├── admin
│   └── svc-orchestrator
├── identity-db/
│   ├── admin
│   └── dev/roles
├── slack/webhooks                 — Slack incoming webhook URLs
└── orchestrator/e2e-test-user     — E2E test user credentials

See Secret Paths Reference for the full table with all fields.

Accessing Vault

# Interactive (opens browser for Authentik SSO)
export VAULT_ADDR="https://vault.shared.cwiq.io"
vault login -method=oidc

# From the Vault server (VAULT_ADDR pre-configured)
ssh vault-shared-cwiq-io
vault login -method=oidc

# From the Ansible server (vault-login helper)
ssh ansible-shared-cwiq-io
sudo su - ansible
vault-login