apt installs recommends by default. wireguard-tools recommends the virtual
wireguard-modules, whose only providers are kernel images, so a plain install
pulls a realtime kernel into a container, fails there, and leaves the metapackage
broken -- after which every later apt call on that host exits 100, including the
retries of the task that caused it.
Every other apt call in the repository already passes --no-install-recommends;
the registry was the one path that did not.
Two arch swarm jobs in fork run 32667447816 died in the ganesha AUR
install at 07:41Z, 42s apart, on two different network errors. All three
EXTERNAL_FETCH attempts ran and lost - the ~40s window is sized for
mirror pools where a bad member rotates out in seconds, but the AUR is
one origin host: when it flaps, it flaps for minutes. A sibling job
fetched the same package successfully 2.5 minutes earlier, and the
module (kewlfft.aur 0.13.0) hardcodes the AUR URLs with no timeout or
mirror knob, so a wider client-side window is the only available layer.
6x60s rides out a multi-minute flap and costs its 5 minutes only when
the AUR is actually down; every mirror-pool source keeps the tighter
EXTERNAL_FETCH.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six of thirteen failures in run 30937913632 landed on the one package
acquisition path that carries no retry: package_call() built every ordinary
distro-repo install with retry=None, and the enable_existing branch did the
same. EXTERNAL_FETCH was attached to AUR, COPR, PPA and source builds only,
and its docstring said so outright.
The dichotomy that justified it - external endpoints are flaky, distro repos
are not - is one the repo already rejects elsewhere: the image build wraps
every dnf call in RETRIES=3, and sys-svc-ssh hand-rolls retries: 5 for apt
with the comment 'nested CI egress is flaky'. Line 14 of that same file then
hands the RedHat branch to the unarmoured path, which is where lam 2 and
prometheus 0 died.
The measurement that settles it, same task across the run:
2.98s / 5.34s / 5.63s / 6.78s / 18.09s / 20.46s / 68.41s / 367.71s (FAIL) /
769.29s (FAIL) / 815.18s (SUCCESS). The slowest run is a success - failure
and success sit on one continuous tail, so the split is not mirror-up versus
mirror-down. xwiki 2 hit the identical epel metadata error, same checksum,
404 from ten mirrors including dl.fedoraproject.org, and is green only
because the Docker build retried it five seconds later.
The three upstreams that failed - EPEL, updates-testing and www.centos.org -
are exactly the three the package cache does not proxy, so the cache cannot
substitute for this.
test_a_plain_repository_install_is_not_retried was the defect's own tripwire
and inverts accordingly.
Also here because it is the same class: roles/update pinned dnf to retries=5,
below dnf's own default of 10 - a hardening task that halved what it meant to
raise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The inline repo definition pinned gpgkey to www.centos.org, a single
non-CDN origin. In run 30937913632 it was unreachable for at least
17m25s and killed 2 of 23 centos ganesha installs; dnf's own retries=5
was already active on the host and did not help.
centos-release-nfs-ganesha11 ships the same repo with metalink failover
and gpgkey=file://, planting the key through Requires:
centos-release-storage-common. Verified end to end against
quay.io/centos/centos:latest and ghcr.io/kevinveenbirkenbach/pkgmgr-centos:stable.
The BOOTSTRAP_BASEURL entry keeps the availability probe verifying both
names instead of degrading them to an inconclusive warning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A module that dies on an unhandled exception carries no msg: the AnsiballZ wrapper returns only failed and exception, whose event holds the sole account of what went wrong. The aggregation defaulted to the literal string package install failed and discarded it, so an AUR fetch that threw in run 30713666732 reported nothing at all and took a full investigation to place. Walk msg, exception.event.msg, exception, module_stderr, stderr instead, and tag every result with the module that produced it. This applies to apt, dnf, pacman and every other module the plugin drives, not only to the AUR.
The aur module fetches the RPC record and the snapshot tarball through bare open_url calls with no error handling, so one blip kills a deploy; distribution repositories need none of this because apt, dnf and pacman retry inside their own transport. Give the calls that reach an endpoint of their own a single shared policy. Only the AUR path is declared today; copr, ppa and build carry it against the day they are.
Split the result along its responsibilities so no file passes 250 lines: calls.py holds the vocabulary, sources.py one builder per acquisition path, plan.py only the dispatcher, outcome.py the reporting and facts.py the fact shapes. The action plugin is left with executing modules. Tests follow the code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
categories.yml moves from roles/ to the repository-root meta/, joining packages.yml. utils/roles/categories.py owns its location plus the tree-walking helpers that were stranded in entity/name.py, so no caller spells the path or duplicates the lookup.
meta/distros.yml becomes the SPOT for every distribution: os_family, Galaxy platform name, dev-environment base image, and the pkgmgr/pkgmgr-virgin/environment reference templates. It replaces seven scattered copies (DISTRO_FAMILY, VALID_DISTROS, _DISTROS, _CANONICAL_PLATFORMS, two shell arrays, a Makefile literal). INFINITO_DISTROS is no longer a static default but generated by utils/env/handlers/infinito/distros.py; the shell consumers read it through the env layer.
Drops dead CLI surface found along the way: --categories-file and --roles-dir on the provision CLI plus three unused parameters of generate_dynamic_inventory, and the positional roles_file of the invokable categories command.
tests/lint/repository/test_distros_spot.py pins the copies that cannot call the SPOT (workflow literals, docs table) and forbids composing a pkgmgr reference outside it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ansible.builtin.package` has no module implementation. It is a documentation-only stub whose logic lives in Ansible's own action plugin, so `_execute_module` shipped that stub to the target and every deploy died at the first package install with "module (ansible.builtin.package) is missing interpreter line".
Resolve the generic marker to the host's `ansible_facts.pkg_mgr` before execution.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A role states which package it needs; the registry owns every distribution
mapping and every acquisition path. Declarations live in the repository-root
meta/packages.yml when more than one consumer installs them, and in a role's
own meta/packages.yml otherwise.
schema.py owns the declaration shape, registry.py finds and indexes the
declarations, plan.py turns a resolved spec into ordered module calls, and the
package_install action plugin only executes them. Keeping the plan pure makes
every acquisition path unit-testable without an Ansible connection:
distribution repositories, the AUR with its unprivileged build user, COPR, PPA,
an EPEL bootstrap package, the pacman multilib section and source builds.
The data is guarded by lints: a package id and a concrete distribution package
name each belong to exactly one declaration, a role installs only its own or
shared ids, every id resolves for all five default distributions with an
empty mapping justified by a nocheck, declarations read in ascending order,
every plan builds and names only whitelisted modules, and an external test
queries the real package indexes of all five distributions.
This commit adds the mechanism and the shared collection; no consumer uses it
yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
evangelostsak
kevinveenbirkenbachandClaude Opus 5