The probe scraped a directory listing under dl.fedoraproject.org, which
redirects to whichever mirror is near the caller. A mirror that does not
carry the directory answers 404, and 404 is not retried, so every EPEL
package came back uncheckable and the suite failed on runners whose nearest
mirror is incomplete -- twice in a row, while the same URLs answer 200 from
elsewhere.
mdapi already backs the Fedora probe and serves the epel10 branch from a
single host. Routing EPEL through it removes the mirror roulette and keeps
the verdict sharp: a name it does not know is still reported absent rather
than unknown.
Checked against the live branch: ShellCheck and keepassxc resolve, an
invented name reports absent, and the suite passes.
bd84664e8 moved debian and arch to a cached Packages index but left ubuntu
on packages.ubuntu.com, one HTTP request per package. That endpoint is the
same anti-bot territory: 96 requests behind a 2-per-host semaphore, and a
single timeout fails the target. Three workspace jobs of one run died that
way on different packages (tor/traceroute, msmtp), which is a lottery, not
a defect.
The debian loader is now generic over mirror, path, pockets and components,
so ubuntu answers from the same 12-request bulk fetch through the existing
_get retry layer. It also probes the pockets a default install resolves
against: ubuntu ships 0ad only in noble-updates, and the release-pocket
index alone reports it absent. The old HTML probe called it present because
the "not available" page does not match the "no such package" regex, so the
per-package route was reporting a false positive as well.
Verified: 96/96 identical verdicts old vs new across every ubuntu probe,
454 probes and 23 inconclusive unchanged, and test-external green inside
the container through the package-cache frontend (8 passed in 108s, against
1025s and red in CI).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-package endpoints are anti-bot territory: packages.debian.org shed
request-level 503s for >13h (2 CI runs and 2 local quality gates red, 7
resp. 91 probes unverified), and archlinux.org connection resets are the
chronic class 00784105f already fought. One Packages.xz/.db fetch per
distro answers every probe from a CDN meant for machines, through the
existing _get retry/semaphore layer; the index is cached per run behind a
lock, and an unreadable index stays a loud unknown.
Debian goes over http because the dev container hijacks deb.debian.org
onto the package-cache frontend, which serves the name on port 80 only -
the same transport the stack's own apt list uses. The stricter index also
exposed three registry declarations that never existed in Debian stable;
they are fixed in the follow-up commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A name that fails to resolve was re-raised on the first attempt, so a
resolver hiccup - the local one drops its :53 sockets whenever docker
restarts a container - failed the whole external suite.
The carve-out protected nothing: the exception is re-raised after the
last attempt either way, so a genuinely unresolvable host still fails.
It only bought a faster failure. Dropping it also matches how the rest
of the repository already groups gaierror with OSError.
A connection reset while reading the response body surfaces as
http.client.RemoteDisconnected, which is an OSError but not a URLError,
so it escaped the retry entirely and failed the whole external suite on
a single flaky AUR request.
archlinux.org answers 429 and resets connections when the suite fires its
16 workers at it: 22 arch probes in parallel, 12 answered and 4 refused in
a reproduction inside the test container, while each name resolves fine on
its own. Those 22 self-inflicted failures then took the warning path,
because an unreachable index is supposed to say nothing about the
declaration — so the suite reported green on 22 packages it never checked.
_get now holds at most two requests per host and retries 429 and 5xx with
backoff; a DNS failure is not retried, it is not transient. That alone
removed all 22 warnings, 45 down to 23, with the runtime unchanged.
The remaining 23 are declared: a third-party repository the role
configures itself, or a name the index legitimately does not list because
the package is virtual. Outcome carries that as a flag now, so those stay
warnings while anything else that could not be checked fails the suite and
names each case. Verified by pointing the arch probe at an unreachable
host: 94 packages reported as unchecked, and the run went red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The inline repo definition pinned gpgkey to www.centos.org, a single
non-CDN origin. In run 30937913632 it was unreachable for at least
17m25s and killed 2 of 23 centos ganesha installs; dnf's own retries=5
was already active on the host and did not help.
centos-release-nfs-ganesha11 ships the same repo with metalink failover
and gpgkey=file://, planting the key through Requires:
centos-release-storage-common. Verified end to end against
quay.io/centos/centos:latest and ghcr.io/kevinveenbirkenbach/pkgmgr-centos:stable.
The BOOTSTRAP_BASEURL entry keeps the availability probe verifying both
names instead of degrading them to an inconclusive warning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A role states which package it needs; the registry owns every distribution
mapping and every acquisition path. Declarations live in the repository-root
meta/packages.yml when more than one consumer installs them, and in a role's
own meta/packages.yml otherwise.
schema.py owns the declaration shape, registry.py finds and indexes the
declarations, plan.py turns a resolved spec into ordered module calls, and the
package_install action plugin only executes them. Keeping the plan pure makes
every acquisition path unit-testable without an Ansible connection:
distribution repositories, the AUR with its unprivileged build user, COPR, PPA,
an EPEL bootstrap package, the pacman multilib section and source builds.
The data is guarded by lints: a package id and a concrete distribution package
name each belong to exactly one declaration, a role installs only its own or
shared ids, every id resolves for all five default distributions with an
empty mapping justified by a nocheck, declarations read in ascending order,
every plan builds and names only whitelisted modules, and an external test
queries the real package indexes of all five distributions.
This commit adds the mechanism and the shared collection; no consumer uses it
yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
evangelostsak
kevinveenbirkenbachandClaude Opus 5