Skip to content

CI/CD Pipeline Overview

All CWIQ platform services share the same pipeline structure, defined by shared CI templates from orchestrator/ci-templates. This page explains how the pipeline is assembled and how data flows between stages.

Shared Template Architecture

Rather than duplicating CI/CD logic in every repository, CWIQ uses a single source of truth: the orchestrator/ci-templates project. Every platform service includes these templates in its .gitlab-ci.yml. When a template is updated, all projects pick up the change automatically on their next pipeline run.

flowchart TD
    TMPL[orchestrator/ci-templates]
    TMPL --> C1[.gitlab-ci-common.yml]
    TMPL --> C2[.gitlab-ci-python.yml]
    TMPL --> C3[.gitlab-ci-node.yml]
    TMPL --> C4[.gitlab-ci-trivy.yml]
    TMPL --> C5[.gitlab-ci-scan.yml]

    C1 & C2 & C4 & C5 --> SVC1[server pipeline]
    C1 & C2 & C4 & C5 --> SVC2[iam-api pipeline]
    C1 & C3 & C4 & C5 --> SVC3[ui pipeline]
    C1 & C2 & C4 & C5 --> SVC4[executor pipeline]

Template Files

Template File What It Provides
.gitlab-ci-common.yml Global variables, workflow rules, pipeline stages, default runner tag (medium), Docker build/push templates, Ansible deploy template, health-check template
.gitlab-ci-python.yml Python lint (ruff), typecheck (mypy), test (pytest), Semgrep SAST, wheel build, and PyPI publish templates
.gitlab-ci-node.yml Node.js lint (eslint), typecheck (tsc), test (vitest), Semgrep SAST, E2E Playwright, npm build, and npm publish templates
.gitlab-ci-trivy.yml Trivy filesystem scan, Trivy Docker image scan, and DefectDojo import templates
.gitlab-ci-scan.yml SonarQube analysis template with Vault JWT authentication

Including Templates

# Backend service (FastAPI, worker, CLI)
include:
  - project: 'orchestrator/ci-templates'
    ref: main
    file:
      - '/.gitlab-ci-common.yml'
      - '/.gitlab-ci-python.yml'
      - '/.gitlab-ci-trivy.yml'
      - '/.gitlab-ci-scan.yml'

# Frontend service (React / TypeScript)
include:
  - project: 'orchestrator/ci-templates'
    ref: main
    file:
      - '/.gitlab-ci-common.yml'
      - '/.gitlab-ci-node.yml'
      - '/.gitlab-ci-trivy.yml'
      - '/.gitlab-ci-scan.yml'

Pipeline Stage Flow

Every service pipeline runs through the same ordered stages. Jobs within a stage run in parallel; the next stage starts only when all jobs in the current stage succeed (unless allow_failure: true is set).

flowchart LR
    V[validate] --> T[test] --> B[build] --> P[push] --> S[scan] --> R[release] --> D[deploy-dev] --> M[migrate] --> VR[verify]
Stage Key Jobs Runs On
validate lint, typecheck, semgrep, trivy-fs-scan All branches
test test, integration-test All branches
build build-push (Kaniko) All branches
push push-latest main only
scan sonarqube-scan, trivy-image-scan, defectdojo-import main only
release release (release-cli) Version tags (v*) only
deploy-dev deploy-dev main auto / feature manual
migrate migrate-dev (Alembic) main only
verify verify-dev main only

Artifact Flow Between Jobs

Several jobs produce artifacts that downstream jobs consume. The diagram below shows the key data dependencies across stages.

flowchart TD
    TEST[test job<br/>coverage.xml] --> SQ[sonarqube-scan]
    SEM[semgrep job<br/>semgrep-results.sarif] --> SQ
    TRIVY_FS[trivy-fs-scan<br/>trivy-fs-results.sarif] --> SQ
    BUILD[build job<br/>build.env with IMAGE_TAG] --> PUSH[push-latest<br/>re-tags as latest]
    BUILD --> TRIVY_IMG[trivy-image-scan<br/>trivy-image-results.json]
    TRIVY_IMG --> DD[defectdojo-import<br/>uploads to DefectDojo]
    SQ --> SONAR_DASH[SonarQube Dashboard<br/>sonarqube.shared.cwiq.io]
    DD --> DD_DASH[DefectDojo Dashboard<br/>defectdojo.shared.cwiq.io]

Key artifact handoffs:

  • build.env — Written by the build job, contains IMAGE_TAG=main-{sha}. Consumed by push, trivy-image-scan, and deploy-dev to ensure all jobs use the exact same image tag.
  • coverage.xml — Generated by pytest with --cov. Consumed by sonarqube-scan for code coverage overlay in the SonarQube UI.
  • semgrep-results.sarif — SAST findings from Semgrep. Consumed by sonarqube-scan as an external issue source.
  • trivy-fs-results.sarif — Filesystem scan findings. Consumed by sonarqube-scan.
  • trivy-image-results.json — Docker image CVE scan results. Consumed by defectdojo-import.