Skip to content

SSL: Renewal and Deployment

Certificates are renewed automatically twice daily by a systemd timer on the cert-server. The ssl-renew-deploy.yml playbook handles the full pipeline: certbot renew, deploy to all EC2 hosts, and import to ACM for ALB-backed services.

Automated Renewal Pipeline

The ssl-renew-deploy.yml playbook executes three stages in sequence:

1. certbot renew --quiet
   └── Checks all 21 certificates, renews those expiring within 30 days
       (Let's Encrypt issues 90-day certs; renewal begins at 60 days)

2. ssl-deploy-all.yml
   └── Deploys renewed certificates to all EC2 hosts (23 deployment targets)
       Each host runs its reload_command after cert files are updated

3. acm-import.yml (sso.shared.cwiq.io)
   └── Re-imports Authentik cert to AWS ACM for ALB SSL termination

4. acm-import.yml (gitlab.shared.cwiq.io)
   └── Re-imports GitLab cert to AWS ACM for ALB SSL termination

If a certificate was not renewed (still more than 30 days until expiry), certbot skips it and the deploy/import steps are no-ops for that domain.

Systemd Timer

The timer is configured by running ansible-playbook setup-auto-renewal.yml once on the cert-server. It creates two systemd unit files:

Setting Value
Timer unit ssl-renew-deploy.timer
Service unit ssl-renew-deploy.service
Schedule *-*-* 00,12:00:00 (midnight and noon UTC)
Random delay Up to 3600 seconds (1 hour) — staggers Let's Encrypt requests
Runs as user ansible
Working directory /data/ansible/cwiq-ansible-playbooks/cert-server
Log file /var/log/ssl-renewal.log
Log rotation Weekly, 4 rotations kept, compressed

The service runs as the ansible user because that user owns the ~/.ssh/cwiq-ansible SSH key and has the known_hosts entries for all target servers.

Managing the Timer

# Set up timer for the first time (one-time)
ansible-playbook setup-auto-renewal.yml

# Check if timer is active
systemctl status ssl-renew-deploy.timer
systemctl list-timers ssl-renew-deploy.timer

# View renewal logs
tail -f /var/log/ssl-renewal.log

# Manually trigger a renewal run (for testing)
systemctl start ssl-renew-deploy.service

# Check service status after manual trigger
systemctl status ssl-renew-deploy.service
journalctl -u ssl-renew-deploy.service --since "5 minutes ago"

Issue Certificates (Initial Setup)

Before the timer can renew certificates, they must be issued once. The ssl-issue-all.yml playbook uses DNS-01 validation through Route53:

ssh ansible@ansible-shared-cwiq-io
cd /data/ansible/cwiq-ansible-playbooks/cert-server

# Issue certificates for all domains in cert_domains (group_vars/all.yml)
ansible-playbook ssl-issue-all.yml

# Issue for a single domain (dry-run test first)
certbot certonly --dns-route53 --dry-run -d newservice.dev.cwiq.io
certbot certonly --dns-route53 -d newservice.dev.cwiq.io

Deploy to Servers

Deploy All (23 targets)

ansible-playbook -i inventory.yml ssl-deploy-all.yml

This deploys to every host in inventory.yml. Use --limit to target a subset.

Deploy to a Single Host

ansible-playbook -i inventory.yml ssl-deploy-all.yml --limit gitlab-dev-cwiq-io

Service-Specific Playbooks

Each service has a dedicated playbook for targeted deployments:

Playbook Hosts Targeted
ssl-deploy-gitlab.yml gitlab-dev-cwiq-io
ssl-deploy-gitlab-shared.yml gitlab-shared-cwiq-io
ssl-deploy-taiga.yml taiga-dev-cwiq-io
ssl-deploy-icinga.yml icinga-dev-cwiq-io
ssl-deploy-zammad.yml support-dev-cwiq-io
ssl-deploy-vault.yml vault-shared-cwiq-io
ssl-deploy-nexus.yml Both Nexus instances (use --limit for one)
ssl-deploy-semaphore.yml semaphore-shared-cwiq-io
ssl-deploy-grafana.yml grafana-shared-cwiq-io
ssl-deploy-prometheus.yml prometheus-shared-cwiq-io
ssl-deploy-sonarqube.yml sonarqube-shared-cwiq-io
ssl-deploy-defectdojo.yml defectdojo-shared-cwiq-io
ssl-deploy-reportportal.yml reportportal-shared-cwiq-io
ssl-deploy-orchestrator.yml Both Orchestrator environments (use --limit for one)

Nexus (two instances)

# Both Nexus instances
ansible-playbook -i inventory.yml ssl-deploy-nexus.yml

# Dev only
ansible-playbook -i inventory.yml ssl-deploy-nexus.yml -l nexus-dev-cwiq-io

# Shared only
ansible-playbook -i inventory.yml ssl-deploy-nexus.yml -l nexus-shared-cwiq-io

Orchestrator (two environments)

# Both environments
ansible-playbook -i inventory.yml ssl-deploy-orchestrator.yml

# Dev only
ansible-playbook -i inventory.yml ssl-deploy-orchestrator.yml --limit orchestrator-dev-cwiq-io

# Demo only
ansible-playbook -i inventory.yml ssl-deploy-orchestrator.yml --limit orchestrator-demo-cwiq-io

Automatic Service Reload

Each host in inventory.yml has a reload_command that runs automatically after certificate files are updated. This ensures the service immediately picks up the new certificate without a full restart.

How Reload Commands Work

The deploy playbooks: 1. Copy fullchain.pem and privkey.pem to /data/ssl/<domain>/ on the target host 2. Set file ownership to ssl_owner:ssl_group (per-host configuration) 3. Run the reload_command for that host

Vault Special Case

Vault: privkey.pem ownership

Vault runs as UID 100 (the vault container user), which does not correspond to the vault OS user (UID 1001). The ssl-deploy-vault.yml playbook explicitly sets privkey.pem ownership to UID 100:100 after deployment to allow the container to read the private key. This is handled automatically — no manual action needed.

Full Renewal Run (Manual)

To manually trigger a full renewal cycle (same as what the timer does):

ssh ansible@ansible-shared-cwiq-io
cd /data/ansible/cwiq-ansible-playbooks/cert-server

ansible-playbook -i inventory.yml ssl-renew-deploy.yml

Watch the log during renewal:

# In another terminal
tail -f /var/log/ssl-renewal.log

Verify Certificates

After initial setup or a manual renewal run:

# Dev services
curl -I https://gitlab.dev.cwiq.io
curl -I https://taiga.dev.cwiq.io
curl -I https://nexus.dev.cwiq.io
curl -I https://orchestrator.dev.cwiq.io

# Shared services
curl -I https://sso.shared.cwiq.io
curl -I https://vault.shared.cwiq.io
curl -I https://gitlab.shared.cwiq.io
curl -I https://nexus.shared.cwiq.io
curl -I https://grafana.shared.cwiq.io

# Demo
curl -I https://orchestrator.demo.cwiq.io

# Check timer is running
systemctl list-timers ssl-renew-deploy.timer

Check certificate expiry on the cert-server:

certbot certificates

Check a specific certificate from a client:

openssl s_client -connect vault.shared.cwiq.io:443 \
  -servername vault.shared.cwiq.io < /dev/null 2>/dev/null | \
  openssl x509 -noout -subject -dates