computer-playbook/docs/contributing/artefact/git/pipeline.md

Continuous Integration 🔄

This document describes the CI pipeline structure, entry points, and gates used in Infinito.Nexus. For the catalog of every workflow file (description, triggers, inputs) see workflows.md.

Overview 🗺️

CI is triggered automatically on pull-request events and on pushes to the branches listed in the entry-workflow row in workflows.md. The pipeline is composed of reusable workflow files under .github/workflows/. The central coordinator is call-orchestrator.yml, which is called by the entry workflows.

Entry Points 🚪

Every external CI trigger MUST route through one of the entry workflows. Each entry translates its event into a call to the orchestrator; the catalog of triggers and inputs lives in workflows.md.

• entry-pr-change-orchestrate.yml: detects PR scope and conditionally calls the orchestrator.
• entry-push-latest.yml: calls the orchestrator for pushes on the supported branch prefixes and additionally invokes the release workflow on version tags. For the contributor-facing release procedure see release.md.
• entry-manual-steer.yml: dispatches the orchestrator manually for a chosen distro set and whitelist.

The push entry synchronizes the current repository's main branch from the configured source repository before CI policy and deploy discovery run. The default source is infinito-nexus/core, and CI_SYNC_MAIN_SOURCE_REPOSITORY MAY override or disable the sync as documented in configuration.md. This sync force-overwrites the local main, so on a synced repository you MUST NOT work directly in main; use a feature/**, fix/**, or hotfix/** branch and treat main as a read-only mirror.

PR Scope Detection 🔎

Before the full pipeline runs, CI detects the scope of the PR based on the changed files and the branch prefix (see branch.md).

• Documentation-only and agent-only PRs MUST pass a lightweight gate without running the full orchestrator.
• All other scopes MUST run the full call-orchestrator.yml.
• The branch prefix MUST match the detected scope, otherwise the validate-pr-branch-prefix job fails.

Pipeline Stages 🏗️

The orchestrator runs the following stages in order:

1. Fork Prerequisites ⏳

For fork PRs, CI waits for privileged images built by the pull_request_target event before proceeding. For an untrusted fork that build runs from the safe base merge ref without organization secrets (only the per-job GITHUB_TOKEN); a maintainer-applied 🛡️ Trusted label switches the build source to the PR head, unlocks the mirror credentials, and skips the wait — see Trusted fork PRs.

2. Security 🛡️

The security-codeql job runs the CodeQL scan (catalogued in the Security and linting table of workflows.md). A failing scan MUST fail code-quality-gate and thereby block all downstream deploy, install, and environment tests.

3. Linting 🧹

The lint job runs call-lint.yml: make lint fans every lint check out in parallel, plus a hadolint SARIF job. It MUST pass before the pipeline continues.

4. CI Image Build 🐳

The build-ci-images stage uses call-images-build-ci.yml to build Docker images for the target distributions (arch, debian, ubuntu, fedora, centos). These images are consumed by all subsequent test jobs.

5. Code Tests 🧪

The test job runs call-test.yml: make test fans the unit, integration, lint and external suites out in parallel on the runner host. It MUST pass code-quality-gate.

6. Code Quality Gate 🚦

The code-quality-gate requires linting, code tests, and security checks to all pass before deploy tests are triggered.

7. DNS Tests 🌐

call-test-dns.yml validates DNS configuration across all target distributions.

8. Image Mirroring 🪞

call-images-mirror-missing.yml mirrors any missing upstream images in parallel with the DNS tests so later deploy jobs pull from GHCR instead of Docker Hub, MCR, or other external registries. This shields CI from upstream rate limits, geo-blocking, and transient registry outages.

Fork PRs cannot publish mirror packages directly, so their runs wait for the pull_request_target mirror producer before continuing. The producer mirrors with the Docker Hub credentials only for trusted PRs (🛡️ Trusted); untrusted forks mirror anonymously (no organization secrets). See mirror.md for the mirroring architecture and naming convention.

GHCR publication uses the workflow GITHUB_TOKEN; optional Docker Hub secrets are used only to reduce source-side Docker Hub rate limits while mirroring. See authentication.md.

9. Deploy Tests 🚀

The two deploy-test workflows listed in the Infrastructure tests table of workflows.md (compose and swarm modes) run in parallel once DNS and mirroring have completed. Both cover all deployment types (INFINITO_DISCOVER_ALL, lifecycle-filtered to the INFINITO_LIFECYCLES envelope), not just the web-* server scope.

Diff-driven app selection 🎯

call-test-deploy.yml narrows its app matrix to the set of roles actually impacted by the branch's diff against origin/main. The discover job resolves an effective whitelist before output_apps.sh runs, using the following precedence:

1. Sentinel __ALL__ in the whitelist input (case-insensitive). The diff logic MUST be skipped and an empty whitelist MUST be emitted, which deploys everything in the workflow's scope. This is the explicit opt-out from diff narrowing for manual dispatch.
2. Any other non-empty whitelist (forwarded from entry-manual-steer.yml and similar). The explicit value MUST win over the diff and is passed through verbatim.
3. Diff vs origin/main, applied only when the whitelist input is empty:
   • Skipped paths: .md and .rst files and everything under tests/ outside roles/<role>/... neither seed a role nor widen the deploy.
   • No diff at all, OR any other changed path outside roles/<role>/..., OR only skipped paths: no whitelist is set, so the full deploy across the workflow's scope runs.
   • All remaining changed paths under roles/<role>/...: the changed roles become the seed set, and the whitelist is the transitive closure of those seeds expanded upwards over run_after, dependencies, and shared services. In other words, every role whose prerequisite set as defined by combined resolver contains one of the seeds is included, together with the seeds themselves.

Each role in that closure is then narrowed to the variants that actually reach a seed: variant i of a role is kept only when round i's closure — walked on the same variant-merged services maps the matrix planner builds its rounds from, and therefore variant-aware at every generation of the chain — contains one of the seeds. A role whose variants do not all qualify enters the whitelist as a role#0,2 selection token, so an image bump in a provider no longer redeploys the variants that switch that provider off. This applies to every auto-derived whitelist — the weekly update PRs, Dependabot, and any other diff-driven run — because they all resolve through this one path.

The reverse closure is fail-safe: any seed that is not modellable in the resolver (no application_id and not referenced by any role's run_after) MUST trigger a fall-back to a full deploy, as MUST any resolver runtime error. The closure MUST NOT silently shrink the deploy matrix on partial information. The variant narrowing follows the same rule: a role reached only through a run_after edge (which the round closure does not follow) keeps every variant, and so does a seed role itself, whose own files changed.

The PR-scope short-circuits in scope.sh (documentation-only, agent-only) still apply at the entry layer. They skip the orchestrator entirely and are independent of the diff-driven whitelist resolution above.

The reverse closure is implemented in affected resolver and invoked from affected_roles.sh. The workflow glue lives in effective_whitelist.sh.

Subset label 🧩

Add a ## Roles section with a fenced roles: YAML list to the PR body and apply the 🧩 Subset label:

## Roles

```yaml
roles:
  - web-app-nextcloud
  - web-app-matomo
  - sys-version
```

With the label set, the detect-affected-roles job in entry-pr-change-orchestrate.yml calls pr_affected_roles.sh, which runs the cli.meta.ci.subset_roles module. The module reads the first fenced block whose YAML carries a roles: key (the heading text is not matched) and emits that list as the whitelist output, which flows into the deploy tests as precedence rung 2 above. Without the label, the diff-driven precedence applies.

The subset path fails the run when:

• the YAML is invalid,
• no roles: block is found,
• the roles: list is empty, or
• any listed id is not an existing roles/<id> directory.

The 🧩 Subset label must exist in the repository's label set and is applied through the PR labels UI. Both the pull_request and pull_request_target events carry a labeled trigger, so adding it starts a fresh run.

10. Installation Tests 📦

The install-test workflows listed in the Infrastructure tests table of workflows.md run in parallel once code-quality-gate is green. All of them MUST pass test-install-gate.

11. Development Environment 🛠️

The test-workspace stage runs the workspace workflow listed in the Infrastructure tests table of workflows.md.

12. Done 🏁

The final done job aggregates all deploy, install, and development gates. CI is considered green only when this job succeeds.

Concurrency 🔀

• PR pipelines use cancel-in-progress: true so only the newest run per PR and event type is active. The group does not always reap a long orchestrator sweep, which then leaves the newer run parked on pending; entry-cancel-superseded.yml cancels those leftovers through the API, see cancel/README.md.
• The orchestrator and the push entry workflow (entry-push-latest.yml) cancel in-progress runs on every ref except main, which no automatic trigger cancels. A pull_request_target run inherits refs/heads/main as its ref, so the orchestrator does not cancel there either; per-PR supersession is handled by the entry workflow's own pr-ci-* group. See configuration.md.
• The manual entry workflow (entry-manual-steer.yml) sets cancel-in-progress: true on its own group, and the orchestrator's group cancels for workflow_dispatch regardless of ref, so a manual sweep supersedes on both levels, main included.

See configuration.md for all repository variables that control CI behaviour.

Fork PRs 🍴

Fork PRs run under the pull_request event without write permissions. Privileged steps (image builds, package writes) run via pull_request_target. The fork CI waits for those privileged images before proceeding. See pull-request.md for the contributor-facing fork workflow.

By default — for an untrusted fork PR — the privileged pull_request_target build checks out the base merge ref (refs/pull/<N>/merge) from the base repository: the PR's changes merged onto base, with all workflow and build-orchestration scripts coming from base, never from the fork. Because actions/checkout otherwise refuses to fetch a PR ref under pull_request_target, the source checkout sets allow-unsafe-pr-checkout: true. The untrusted build runs without organization secrets — secrets: inherit is not passed and it authenticates to GHCR with the per-job GITHUB_TOKEN only, so a malicious fork cannot read repository/organization secrets. (The pull_request_target token still carries packages: write, so the 🛡️ Trusted label remains the real trust boundary for anything beyond image publication.)

Trusted fork PRs 🔓

When a maintainer applies the 🛡️ Trusted label, the fork PR is treated as deliberately released code:

• The orchestrator's wait-fork-prereq-run job skips the wait for the privileged fork images.
• The privileged pull_request_target build switches its build source from the base merge ref to the raw PR head (head.repo.full_name @ head.sha).
• The trusted build additionally receives the Docker Hub mirror credentials (passed only when the label is present); untrusted builds mirror anonymously.
• The built image keeps the merge-SHA tag (ci-<merge_sha>) so the pull_request orchestrator finds it unchanged — no consumer workflow has to know about the label.

Security gates:

• The 🛡️ Trusted label flips the build source to the PR head and unlocks the Docker Hub credentials; without it the safe base merge ref is built with no organization secrets. Restrict who may set the label via repository label permissions / branch protection so only maintainers can grant it.
• Build-orchestration scripts (scripts/image/push.sh and the helper checkout) always come from the base repo at github.sha; only the Docker build context (_src) is the fork head. persist-credentials: false keeps the token out of the source checkout, and USE_NIX_TOKEN stays disabled for fork-sourced builds.
• Fork pull_request runs always receive a read-only token (GitHub-enforced); the label never grants write tokens or secrets to the untrusted event.

Both events carry a labeled trigger, so applying the 🛡️ Trusted label starts fresh runs that already observe it: the pull_request_target build switches to the PR head and the pull_request orchestrator skips the wait — no extra push is needed. (Re-running an existing run does not pick up a new label, because GitHub replays the original event payload.) The trade-off is that any label change starts a fresh PR CI run.