Skip to content

Pipeline Stages

A detailed breakdown of every stage in the CWIQ CI/CD pipeline: which jobs run, what tools they use, what artifacts they produce, and whether they block the pipeline on failure.

Stage Reference

Stage Jobs Tools Artifacts Runner Tag Blocking
validate lint, typecheck, semgrep, trivy-fs-scan ruff / eslint, mypy / tsc, Semgrep, Trivy semgrep SARIF + JSON, trivy SARIF + JSON small Yes — Trivy secrets findings block the pipeline
test test, integration-test pytest / vitest coverage.xml or coverage/ small / large Yes
build build-push Kaniko build.env containing IMAGE_TAG medium Yes
push push-latest Docker latest image tag in Nexus small No (main only)
scan sonarqube-scan, trivy-image-scan, defectdojo-import sonar-scanner-cli, Trivy, curl scan reports small No (allow_failure: true)
release release release-cli GitLab release entry small No (version tags only)
deploy-dev deploy-dev, manual-dev-deploy SSH, docker compose — small Yes
migrate migrate-dev Alembic — small Yes
verify verify-dev curl health status small Yes

Stage Details

validate

The first stage runs on every push to every branch. Its purpose is to catch problems before a Docker image is built.

lint — Runs ruff check (Python) or eslint (Node.js) against the source directory. Fails the pipeline on any lint error.

typecheck — Runs mypy (Python) or tsc --noEmit (TypeScript). Fails on type errors.

semgrep — Runs Semgrep SAST against the full repository. Produces semgrep-results.sarif and semgrep-results.json, which are later consumed by sonarqube-scan.

trivy-fs-scan — Scans the repository filesystem for secrets and misconfigurations. Produces trivy-fs-results.sarif. The job is configured to exit non-zero if secrets are detected (--exit-code 1 --severity CRITICAL), which blocks the pipeline.

Trivy secrets findings are a hard block

If trivy-fs-scan finds a secret committed to the repository (API key, private key, password), the validate stage fails immediately and the pipeline does not proceed to test or build. Fix the finding and force-push to continue.

test

test — Runs the full unit test suite. For Python services, this is pytest with --cov and --cov-report=xml, producing coverage.xml. For the UI, this is vitest --coverage. Runs on the small runner.

integration-test — Runs integration tests that require a live database and Redis (provided as GitLab services). This job only runs on main and develop branches. It uses the large runner for projects that require more memory (executor).

The sonarqube-scan job in the scan stage uses needs: to depend on the test job's coverage.xml artifact. Make sure your project's coverage report path matches what the template expects.

build

build-push — Builds the Docker image using Kaniko and pushes it to Nexus. At the end of the job, it writes IMAGE_TAG=main-{short-sha} (or the appropriate branch tag) to build.env. All downstream jobs that reference the image use this variable rather than constructing the tag themselves, ensuring consistency.

See Kaniko Docker Builds for the full Kaniko configuration and tag strategy.

push

push-latest — Pulls the image built in the build stage, adds the latest tag, and pushes it back to Nexus. This job only runs on the main branch and is not blocking (allow_failure: true).

scan

The scan stage runs only on the main branch. All three jobs are non-blocking (allow_failure: true) so that a scan service outage never prevents a deployment.

sonarqube-scan — Uses sonar-scanner-cli to analyse the codebase and upload results to SonarQube. This job uses needs: to pull artifacts from three upstream jobs:

needs:
  - job: test
    artifacts: true       # coverage.xml
  - job: semgrep
    artifacts: true       # semgrep-results.sarif
  - job: trivy-fs-scan
    artifacts: true       # trivy-fs-results.sarif

Vault JWT authentication is used to fetch the SonarQube token at runtime. See SonarQube Setup for configuration details.

trivy-image-scan — Scans the built Docker image for CVEs. Uses build.env to determine which image tag to pull. Produces trivy-image-results.json.

defectdojo-import — Uploads trivy-image-results.json to DefectDojo via its REST API. Uses Vault JWT to fetch the DefectDojo API token.

release

release — Runs only when a version tag matching v* is pushed (e.g., v1.2.0). Uses release-cli to create a GitLab release entry with auto-generated release notes from merge request titles since the previous tag.

deploy-dev

deploy-dev — The main deployment job. SSHs to the DEV server using $SSH_PRIVATE_KEY and $SSH_USER, pulls the new Docker image from Nexus, and runs docker compose up -d. Runs automatically on every push to main.

manual-dev-deploy — Identical to deploy-dev but requires a manual click in the GitLab UI. Triggered when MANUAL_DEPLOY=true is passed as a pipeline variable.

Only the deploy stage connects to remote servers

All validation, build, and scan jobs run entirely within the EKS pod and never reach outside the cluster. Only deploy-dev, migrate-dev, and verify-dev SSH to the DEV server.

migrate

migrate-dev — Runs database migrations via Alembic (alembic upgrade head) over an SSH connection to the DEV server. Runs after deploy-dev completes. Only applies to services with a database (server, iam-api, audit-api, monitoring-api, notification-api, ai-catalogue-api).

Services without a database skip this stage by not defining a migrate-dev job.

verify

verify-dev — Performs a final health check by curling https://orchestrator.dev.cwiq.io/api/health (or the service-specific health endpoint) and asserting a 200 response. Uses $DEV_SERVER_URL from the group-level CI/CD variables.

If this job fails, it signals that the deployment did not produce a healthy service — investigate the service logs on the DEV server.

Cross-Stage Dependencies (needs:)

GitLab's needs: keyword allows a job to start as soon as its listed dependencies complete, bypassing the normal stage ordering. CWIQ uses this in the scan stage to avoid waiting for the entire build stage when only test artifacts are needed.

Key needs: relationships:

flowchart LR
    TEST[test] -- coverage.xml --> SQ[sonarqube-scan]
    SEMGREP[semgrep] -- semgrep-results.sarif --> SQ
    TRIVY_FS[trivy-fs-scan] -- trivy-fs-results.sarif --> SQ
    BUILD[build-push] -- build.env IMAGE_TAG --> TRIVY_IMG[trivy-image-scan]
    TRIVY_IMG -- trivy-image-results.json --> DD[defectdojo-import]
    BUILD -- build.env IMAGE_TAG --> DEPLOY[deploy-dev]