Two arch swarm jobs in fork run 32667447816 died in the ganesha AUR
install at 07:41Z, 42s apart, on two different network errors. All three
EXTERNAL_FETCH attempts ran and lost - the ~40s window is sized for
mirror pools where a bad member rotates out in seconds, but the AUR is
one origin host: when it flaps, it flaps for minutes. A sibling job
fetched the same package successfully 2.5 minutes earlier, and the
module (kewlfft.aur 0.13.0) hardcodes the AUR URLs with no timeout or
mirror knob, so a wider client-side window is the only available layer.
6x60s rides out a multi-minute flap and costs its 5 minutes only when
the AUR is actually down; every mirror-pool source keeps the tighter
EXTERNAL_FETCH.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six of thirteen failures in run 30937913632 landed on the one package
acquisition path that carries no retry: package_call() built every ordinary
distro-repo install with retry=None, and the enable_existing branch did the
same. EXTERNAL_FETCH was attached to AUR, COPR, PPA and source builds only,
and its docstring said so outright.
The dichotomy that justified it - external endpoints are flaky, distro repos
are not - is one the repo already rejects elsewhere: the image build wraps
every dnf call in RETRIES=3, and sys-svc-ssh hand-rolls retries: 5 for apt
with the comment 'nested CI egress is flaky'. Line 14 of that same file then
hands the RedHat branch to the unarmoured path, which is where lam 2 and
prometheus 0 died.
The measurement that settles it, same task across the run:
2.98s / 5.34s / 5.63s / 6.78s / 18.09s / 20.46s / 68.41s / 367.71s (FAIL) /
769.29s (FAIL) / 815.18s (SUCCESS). The slowest run is a success - failure
and success sit on one continuous tail, so the split is not mirror-up versus
mirror-down. xwiki 2 hit the identical epel metadata error, same checksum,
404 from ten mirrors including dl.fedoraproject.org, and is green only
because the Docker build retried it five seconds later.
The three upstreams that failed - EPEL, updates-testing and www.centos.org -
are exactly the three the package cache does not proxy, so the cache cannot
substitute for this.
test_a_plain_repository_install_is_not_retried was the defect's own tripwire
and inverts accordingly.
Also here because it is the same class: roles/update pinned dnf to retries=5,
below dnf's own default of 10 - a hardening task that halved what it meant to
raise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A module that dies on an unhandled exception carries no msg: the AnsiballZ wrapper returns only failed and exception, whose event holds the sole account of what went wrong. The aggregation defaulted to the literal string package install failed and discarded it, so an AUR fetch that threw in run 30713666732 reported nothing at all and took a full investigation to place. Walk msg, exception.event.msg, exception, module_stderr, stderr instead, and tag every result with the module that produced it. This applies to apt, dnf, pacman and every other module the plugin drives, not only to the AUR.
The aur module fetches the RPC record and the snapshot tarball through bare open_url calls with no error handling, so one blip kills a deploy; distribution repositories need none of this because apt, dnf and pacman retry inside their own transport. Give the calls that reach an endpoint of their own a single shared policy. Only the AUR path is declared today; copr, ppa and build carry it against the day they are.
Split the result along its responsibilities so no file passes 250 lines: calls.py holds the vocabulary, sources.py one builder per acquisition path, plan.py only the dispatcher, outcome.py the reporting and facts.py the fact shapes. The action plugin is left with executing modules. Tests follow the code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5