kevinveenbirkenbachandClaude Opus 5.5
b055eece05
The workflows pin `runs-on: ubuntu-NN.NN`, so no job moves to a new Ubuntu release behind our back, but nothing raised the pin either. A sixth job of `🔄 Update versions` closes that: `utils/update/runner.py` reads the image table of actions/runner-images, takes the newest x64 Ubuntu label whose image carries no preview or beta badge, and raises every older bare `runs-on: ubuntu-NN.NN` to it; arch-suffixed labels stay untouched. The job opens the change as one PR over `.github/workflows`, so its CI run tests the new image before it reaches main. This commit carries the updater's first result: every job now runs on ubuntu-26.04 (49 `runs-on`). actionlint 1.7.12, the newest release, does not know `ubuntu-26.04` yet (rhysd/actionlint#682, fix proposed in rhysd/actionlint#743), so `.github/actionlint.yaml` declares the label until a release knows it. act maps each `ubuntu-NN.NN` label the workflows use to the local platform image, derived from `.github/workflows`, instead of a fixed list that did not know ubuntu-26.04. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
5.3 KiB
CI Failures and Debugging 🐛
If CI fails, follow a clean debugging workflow:
- Download every failed job's logs and artefacts with the tooling below.
- Inspect the raw logs to find each root failure, then cluster the failures that share a cause.
- Decide whether each failure belongs to your branch or to something unrelated.
- Fix related failures in the same branch.
- Open an issue for unrelated failures instead of mixing them into your branch.
Fetching and Clustering 📥
The CLI under cli/meta/ci/logs pulls a run's job logs and artefacts in one shot and clusters them by root failure.
# All failed jobs' logs + artefacts into /tmp/logs/<run-id>/ (logs/, artifacts/, jobs.json, summary.tsv)
python -m cli.meta.ci.logs.download <run-url> -f
# Group the downloaded logs by their shared failing task, largest cluster first
python -m cli.meta.ci.logs.analyze /tmp/logs/<run-id>
download accepts conclusion filters (-s success, -f failed, -c cancelled, -k skipped; none fetches all), -d for the destination, and -j for the worker count.
The per-job logs carry the real failing task and its error, and the artefacts carry the Playwright error-context.md, screenshots, and playwright-junit.xml.
Important 🚨
- You MUST NOT debug from screenshots alone. Use raw logs.
- You MUST NOT commit log files to the repository.
- If a Playwright job fails, you MUST also download the Playwright assets, store them in
/tmp/, and analyze them together with the logs.
Manual CI Jobs 🎯
You SHOULD use targeted manual CI jobs instead of rerunning the full pipeline if you only need one focused check.
Prefer the manual workflow in entry-manual-steer.yml:
- Select your branch.
- Use
debianunless you have a clear reason to use a different distro. - Limit the run to the affected app when possible.
This gives faster feedback and protects shared CI runners.
More information here.
Reproducing Runner Resource Limits Locally 💾
Some CI failures (most notably OOM-kills such as #162, PeerTube plugin install with rc=137) only surface under the memory and CPU ceiling of the GitHub-hosted runner and cannot be reproduced on a workstation with plenty of RAM. To reproduce these failures locally, you MUST cap the resources of the top-level infinito service in compose.yml to values that match (or undercut) the runner specs below.
GitHub-Hosted Runner Specs
All workflows target runs-on: ubuntu-26.04, the GitHub-hosted standard Linux runner pinned to one Ubuntu release. GitHub still refreshes that image weekly. The 🏃 Update GitHub-hosted runner labels job of cron-update.yml opens a pull request when a newer GA x64 Ubuntu image is listed in actions/runner-images, and tests/lint/repository/test_workflow_runner_pinned.py rejects *-latest labels. The effective ceiling depends on the repository's visibility and plan:
| Repository / plan | vCPU | RAM | SSD |
|---|---|---|---|
Public repo (this repo, ubuntu-26.04) |
4 | 16 GB | 14 GB |
| Private repo on Free / Pro / Team | 2 | 7 GB | 14 GB |
| Private repo on Enterprise | 4 | 16 GB | 14 GB |
The effective budget for the actual workload is lower than the table suggests because the runner itself plus the nested infinito container (systemd, nested Docker, Ansible) reserve a non-trivial baseline. Plan for roughly 1–2 GB of overhead before any role deploys a service.
For the authoritative specs, see the upstream docs for standard hosted runners and larger runners.
Env-Var Knobs
compose.yml exposes three env vars on the infinito service that cap its memory, swap, and CPU ceiling. For the authoritative variable reference (names, defaults, purpose) see the SPOT in compose.yml.md. This page SHOULD NOT duplicate that table and MUST link back to it when a contributor needs the precise definitions.
Canonical Reproduction Profiles
Pick the profile that matches the runner you want to mimic:
# Public repo runner (ubuntu-26.04, 4 vCPU / 16 GB): the default CI target
INFINITO_MEM_LIMIT=16g INFINITO_MEMSWAP_LIMIT=16g INFINITO_CPUS=4 make <target>
# Private repo runner on Free/Pro/Team (2 vCPU / 7 GB)
INFINITO_MEM_LIMIT=7g INFINITO_MEMSWAP_LIMIT=7g INFINITO_CPUS=2 make <target>
# Aggressive OOM reproducer: forces the memory spike that surfaces #162
INFINITO_MEM_LIMIT=4g INFINITO_MEMSWAP_LIMIT=4g INFINITO_CPUS=2 make <target>
Verify the caps applied with docker compose -f compose.yml config | grep -E 'mem_limit|memswap_limit|cpus' before starting the deploy.