Skip to content

Icinga Health Checks

Icinga2 master-satellite architecture provides infrastructure health checks across all 21 monitored hosts, routing alerts to Slack based on environment.

Architecture

icinga-shared-cwiq-io (MASTER)
  Icinga2 2.15.2 + IcingaWeb2 + IcingaDB + MariaDB + Redis
  https://icinga.shared.cwiq.io
  Port 5665 (cluster API)
        |
        | PKI-authenticated cluster protocol
        | Master pushes zone configs to satellite
        | Satellite submits check results
        v
icinga-dev-cwiq-io (SATELLITE)
  Icinga2 daemon only (no web UI, no database)
  Zone: dev — checks DEV/Demo hosts locally
Component Version
Icinga2 2.15.2
IcingaWeb2 2.12.6
IcingaDB 1.5.1
MariaDB 10.11
Redis 7-alpine

Zone Layout

Zone Managed By Checks
global-templates Master Templates, common services, notifications (synced to all zones)
master Master 14 Shared Services host configs
dev Master (pushed to satellite) 7 DEV/Demo host configs

Monitored Hosts

Shared Zone (14 hosts — checked by master)

Host SSH HTTPS HTTP Containers
ansible-shared-cwiq-io Yes — — —
gitlab-shared-cwiq-io Yes Yes — Yes (gitlab)
grafana-shared-cwiq-io Yes Yes — Yes (grafana, nginx)
prometheus-shared-cwiq-io Yes Yes — Yes (prometheus, alertmanager)
loki-shared-cwiq-io Yes — Yes (:3100/ready) Yes (loki)
nexus-shared-cwiq-io Yes Yes — Yes (nexus, nginx)
vault-shared-cwiq-io Yes Yes — Yes (vault)
icinga-shared-cwiq-io Yes Yes — Yes (icinga2, etc.)
authentik-shared-cwiq-io-1 Yes Yes — Yes
authentik-shared-cwiq-io-2 Yes Yes — Yes
semaphore-shared-cwiq-io Yes Yes — Yes
defectdojo-shared-cwiq-io Yes Yes — Yes
sonarqube-shared-cwiq-io Yes Yes — Yes
reportportal-shared-cwiq-io Yes Yes — Yes

Dev Zone (7 hosts — checked by satellite)

Host SSH HTTPS Docker Containers
orchestrator-dev-cwiq-io Yes Yes Yes
orchestrator-demo-cwiq-io Yes Yes Yes
icinga-dev-cwiq-io Yes — Yes
langfuse-dev-cwiq-io Yes Yes Yes
datastorea-dev-cwiq-io Yes — Yes
datastoreb-dev-cwiq-io Yes — Yes
identity-db-dev-cwiq-io Yes — Yes

Docker Container Checks

Container health is verified using check_by_ssh, which runs docker inspect on the target host.

SSH key mount path

The Icinga2 container mounts its SSH key at /var/lib/icinga2/.ssh/ (NOT /var/lib/nagios/.ssh/). The container user is UID 5665. Tailscale SSH ACLs must permit tag:icinga → target hosts.

Container checks are defined per-host as vars.docker_containers["<name>"] in the host config file:

vars.docker_containers["orchestrator-server"] = {
  display_name = "Server container"
}

Notifications

Icinga routes alerts to Slack based on the host's environment variable:

Environment Slack Channel
shared #cwiq-shared-infra-alerts
dev #cwiq-dev-infra-alerts

Notification config is in icinga/conf.d/notifications.conf.j2, deployed to the global-templates zone.

Adding a New Host

1. Create the host config file

For DEV hosts, create icinga/conf.d/hosts/dev/<hostname>-dev.conf:

object Host "<hostname>-dev-cwiq-io" {
  import "cwiq-dev-host"

  address = "<hostname>-dev-cwiq-io"
  display_name = "<Service Name> DEV"

  vars.environment = "dev"
  vars.os = "AlmaLinux"

  vars.http_vhosts["HTTPS"] = {
    http_address = "<service>.dev.cwiq.io"
    http_ssl = true
    http_vhost = "<service>.dev.cwiq.io"
    http_uri = "/health"
    http_port = 443
  }
}

For Shared hosts, use conf.d/hosts/shared/<hostname>-shared.conf and import cwiq-shared-host.

2. Update icinga/README.md

Add a row to the Monitored Hosts table.

3. Deploy the configuration

ssh ansible@ansible-shared-cwiq-io
ansible-helper
cd icinga

# DEV zone configs
ansible-playbook -i inventory/shared.yml deploy-config.yml --tags dev

# Shared zone configs
ansible-playbook -i inventory/shared.yml deploy-config.yml --tags shared

Playbook Reference

Playbook Purpose
setup.yml Full initial deployment (master or satellite)
deploy-config.yml Deploy zone configs (host objects, services)
deploy-config.yml --tags dev Update DEV zone only
deploy-config.yml --tags shared Update Shared zone only

Web UI

Access the Icinga2 dashboard at https://icinga.shared.cwiq.io. Login with admin credentials from Vault at secret/icinga/admin.