CentOS jobs also die on "Failed to download metadata for repo 'epel':
Cannot download repomd.xml ... All mirrors were tried" (feature/store-i18n
run 39658756830, Baserow#0 and Nextcloud#8 aarch64): the failing host's
librepo log shows mirrors that answered and then failed the checksum the
metalink expects, the stale-metalink signature already pinned away for
the CentOS Stream repos, plus TLS errors from two mirrors.
centos_mirror.sh now also rewrites the EPEL repo files: each section's
own commented `#baseurl=https://download.example/pub/epel/...` template
becomes the baseurl on dl.fedoraproject.org and the Fedora metalink line
goes. Using the file's template keeps the path variables of whichever
epel-release is installed (`$releasever${stream:+s}` on CentOS Stream,
`releasever_minor` in newer releases). A file whose metalinks lack that
template stops the script with an error instead of losing its source.
The Dockerfile runs the script again right after
scripts/install/filesystem.sh. Its first run precedes the EPEL install
(filesystem.sh adds epel-release for btrfs-progs), so the image shipped an
unpinned epel.repo, and the failing call, sys-svc-ssh's OpenSSH install,
runs before roles/update/tasks/centos.yml pins anything.
On CentOS Stream the template resolves to epel/10s/, which on
dl.fedoraproject.org serves the same aarch64 repomd.xml as epel/10.4/, the
only path the epel-10s metalink lists.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Under the dnf4 backend (CentOS) ansible.builtin.dnf reports
`Failed to download packages: <reason>` (module_utils/_embed/dnf.py),
under dnf5 (Fedora) the bare `Failed to download packages`. The exact
comparison re-raised the CentOS form instead of retrying. The rescue now
re-raises everything that does not start with that message.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Since 2026-10-21 18:45Z part of the metalinks mirrors.centos.org hands
to CI runners still expects the previous BaseOS repomd.xml (sha512
bed4f4…) while every mirror and the master already serve the current one
(2f4550…). dnf then rejects every mirror ("checksum doesn't match",
"All mirrors were tried", Nextcloud#2, Nextcloud#10, Mattermost#1 in run
39684179395) or resolves BaseOS from a mirror still on the previous push
while AppStream is current ("nothing provides rpm = 4.19.1.1-27.el10
needed by rpm-build-4.19.1.1-27.el10", Baserow#0, Akaunting#0).
roles/update/files/centos_mirror.sh replaces every metalink of the
CentOS Stream repo files with the matching baseurl on
mirror.stream.centos.org and prints what it pinned. The Dockerfile runs
it before its first dnf call, so the dev and swarm-node images never
consult a metalink; roles/update reaches fedora.yml through a new
centos.yml that runs it first, so production hosts resolve against the
same source. The script is idempotent and prints nothing once every
section is pinned.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Fedora deploys failed in `update (dnf)` with "Failed to download
packages": the node's metadata still listed builds the master had
already replaced, and every mirror answered 404 (#1138: less-704-4.fc44
four hours after the push; run 39684179395: OpenBao#0 swarm-mgr-01 and
Nextcloud#9, less-704-4 and ca-certificates-2026.2.90).
Two mechanisms lead there:
- Under dnf5, ansible.builtin.dnf's `update_cache: true` expires no
repository: it queries the repos before loading them (0 available
repos before create_repos_from_system_configuration, 16 after), so
metadata already cached on the node stays in use until
metadata_expire. An explicit `dnf makecache --refresh` now runs
before the upgrade.
- A metalink can still hand out repodata from a mirror on the previous
push. The upgrade now runs in a block whose rescue re-raises every
other failure unchanged and, on exactly "Failed to download
packages", runs `dnf clean metadata` and upgrades once more.
Fixes#1138
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six of thirteen failures in run 30937913632 landed on the one package
acquisition path that carries no retry: package_call() built every ordinary
distro-repo install with retry=None, and the enable_existing branch did the
same. EXTERNAL_FETCH was attached to AUR, COPR, PPA and source builds only,
and its docstring said so outright.
The dichotomy that justified it - external endpoints are flaky, distro repos
are not - is one the repo already rejects elsewhere: the image build wraps
every dnf call in RETRIES=3, and sys-svc-ssh hand-rolls retries: 5 for apt
with the comment 'nested CI egress is flaky'. Line 14 of that same file then
hands the RedHat branch to the unarmoured path, which is where lam 2 and
prometheus 0 died.
The measurement that settles it, same task across the run:
2.98s / 5.34s / 5.63s / 6.78s / 18.09s / 20.46s / 68.41s / 367.71s (FAIL) /
769.29s (FAIL) / 815.18s (SUCCESS). The slowest run is a success - failure
and success sit on one continuous tail, so the split is not mirror-up versus
mirror-down. xwiki 2 hit the identical epel metadata error, same checksum,
404 from ten mirrors including dl.fedoraproject.org, and is green only
because the Docker build retried it five seconds later.
The three upstreams that failed - EPEL, updates-testing and www.centos.org -
are exactly the three the package cache does not proxy, so the cache cannot
substitute for this.
test_a_plain_repository_install_is_not_retried was the defect's own tripwire
and inverts accordingly.
Also here because it is the same class: roles/update pinned dnf to retries=5,
below dnf's own default of 10 - a hardening task that halved what it meant to
raise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the raw package modules at 76 call sites with package_install, moves
each role's distribution mapping into its own meta/packages.yml, and turns the
inventory PACKAGES variable from source-keyed literal names into a list of
declared package ids. The lint that forbids the raw modules is enabled here,
because the migration burns it down from 76 offenders to zero.
Removes sys-aur, sys-aur-install and sys-package. The AUR build user, the
passwordless pacman rule and the inventory-driven install are part of the plan
the action plugin executes, so the roles no longer carry them.
Declaring the mappings surfaced breakage the raw modules had hidden: on Debian
the `which` binary comes from debianutils, jrnl is in no Debian or Ubuntu
suite, chromium-browser is absent from Debian stable, JDK 11 is unobtainable on
RedHat, wireguard had no RedHat branch at all, qbittorrent was built from the
AUR although Arch ships it in extra, and desk-docker installed a Docker variant
that conflicts with the one sys-svc-container installs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pacman's XferCommand curl gains --retry-all-errors --retry-delay 2 so
connection resets during package downloads retry instead of failing the
update; plain --retry does not cover curl exit 35. No --max-time here:
package downloads are size-unbounded and a ceiling would truncate large
transfers. dnf gets retries=5 via its native config, matching the apt
Acquire::Retries hardening already in place for Debian.
Settle the egress lint follow-ups: collapse the nested retry checks per
ruff SIM102 and reformat the curl filter tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Transient CDN/mirror TLS resets are a recurring hard-fail class (docker.com, PyPI). Harden all three egress paths at their SPOT: the curl filter now emits --retry-all-errors (retries=0 opts a call out), so every ansible curl retries; a new lint requires --retry-all-errors on file-downloading curl and --tries on wget in verbatim files/*.sh (which cannot use the filter), fixing the moodle/alias downloads it flagged; and the update role configures native retries for every package manager - apt Acquire::Retries, dnf retries=, pacman XferCommand over retrying curl - covering all five CI distros (ubuntu shares apt, centos shares dnf).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh: rename AUTOFORMAT_PARALLEL to PARALLEL,
declare it at file start with default `true`; still env-overridable.
Reword the worker comment so the literal `# shellcheck` token no
longer appears at start-of-line and trips SC1072/SC1073.
* scripts/lint/galaxy.sh: drop the GALAXY_FATAL / GALAXY_NAMESPACE /
GALAXY_PARALLEL env vars (over-engineered for a single repo). Failure
is always fatal. NAMESPACE is an internal constant. Add the same
PARALLEL env var (default `true`, env-overridable); falsy demotes the
xargs fan-out to one worker.
* docs/contributing/environment/variables.md: collapse the four
AUTOFORMAT_*/GALAXY_* rows into one `PARALLEL` row covering both
consumers.
* docs/contributing/tools/make.md: refer to `PARALLEL=0` everywhere;
add the lint-galaxy parallelism note.
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Makefile lint integration:
* scripts/install/lint.sh: replace the prior checkmake binary install with
ensure_mbake (mbake is pip-installable, parses our Makefile correctly
whereas checkmake bailed on .ONESHELL: / .SHELLFLAGS: / ifneq).
* scripts/lint/makefile.sh: new runner — `mbake format --check Makefile`
reports formatting drift and missing .PHONY declarations.
* scripts/lint/autoformat.sh: add a two-pass `mbake format Makefile`
(mbake's formatter is not strictly idempotent on the first pass).
* Makefile: split the multi-line `.PHONY:` block into 18 single-line
`.PHONY:` declarations so mbake's parser can validate it; add the
previously-missing `install-python-dev` and `build-dependency` PHONY
targets; add `lint-makefile` aggregate-and-direct target.
* .pre-commit-config.yaml: new make-lint-makefile hook scoped to
`^Makefile$`.
* .github/workflows/lint-makefile.yml: new CI workflow (pip-install mbake
+ make lint-makefile).
* .github/workflows/ci-orchestrator.yml: wire lint-makefile into
lint-gate.needs.
Misc:
* .gitignore: add roles/importer_result.json (tooling artefact written by
galaxy-importer when it runs in roles/).
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cover all five default CI distributions (Archlinux, Debian, Ubuntu,
Fedora, CentOS) from a single role. tasks/main.yml dispatches via
ansible_facts['distribution'] to one of three task files grouped by
package-manager family — Debian and Ubuntu both use apt, Fedora and
CentOS both use dnf, so each pair shares a single file:
| distribution | task file | module |
|---------------------|------------------|----------------------------|
| Archlinux | arch.yml | community.general.pacman |
| Debian / Ubuntu | debian.yml | ansible.builtin.apt |
| Fedora / CentOS | fedora.yml | ansible.builtin.dnf |
| other | unsupported.yml | ansible.builtin.fail w/msg |
The dispatcher is wrapped in `when: run_once_update is not defined`
and finishes by calling tasks/utils/once/flag.yml so subsequent
invocations within one play are no-ops.
Migration impact: only one external caller exists
(tasks/stages/01_constructor.yml line 143, "update device") and is
migrated to the new name. The auto-generated tasks/groups/update-roles.yml
(prefix-matched on `update-*`) becomes empty; it had no consumers.
Closes the GHA universal-deploy `update-apt on arch` failure path,
since update-apt no longer exists as a standalone role and the new
dispatcher routes Arch to pacman.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
