- Declare `__all__` in every package `__init__.py` whose `PROJECT_ROOT`
is imported by its modules (163 packages), so the constant counts as
exported; list every public name where the package defines more.
- Empty the 15 package `__init__.py` files whose `PROJECT_ROOT` no module
imports.
- Declare the public API of `utils.github.variant.axes` in `__all__`,
re-exports included, instead of silencing them with `noqa: F401`; the
list matches the one `feature/store-i18n` already carries.
- Use `import unittest.mock as mock` in `test_applications.py`, as the
rest of the suite does.
- Spell `_NON_PRINTABLE` in `utils/yaml_bootstrap.py` as the equivalent
BMP-only positive class; it matches exactly the same 2111 code points
as the old negated class across U+0000..U+10FFFF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The probe scraped a directory listing under dl.fedoraproject.org, which
redirects to whichever mirror is near the caller. A mirror that does not
carry the directory answers 404, and 404 is not retried, so every EPEL
package came back uncheckable and the suite failed on runners whose nearest
mirror is incomplete -- twice in a row, while the same URLs answer 200 from
elsewhere.
mdapi already backs the Fedora probe and serves the epel10 branch from a
single host. Routing EPEL through it removes the mirror roulette and keeps
the verdict sharp: a name it does not know is still reported absent rather
than unknown.
Checked against the live branch: ShellCheck and keepassxc resolve, an
invented name reports absent, and the suite passes.
The Docker updater watches image plus version, the repository updater
repository plus ref. A dozen pins fell outside both - an app_version, a
release, a tag in a registry neither speaks - so nothing ever told anyone
they had gone stale.
Such a pin now names its upstream next to itself in an `update:` block:
git_tags, registry_tags, npm, http_regex or a role-local script, with
`match` selecting a tag flavour and `strip` removing a prefix the pin does
not carry. utils/update/source.py resolves them beside the two existing
updaters, `python -m cli.contributing.update.source` bumps them, the
update-version-sources job opens one PR per role, and
tests/external/update/source warns while a bump is pending.
Two lints hold the rule: every semver pin is watched by one of the three
updaters or opted out with a reason, and a semver only ever sits behind
`version`, `<what>_version` or `ref` - which renames moodle's release to
app_version and simpleicons' local_version to local_tag, the latter being
the tag of a locally built image rather than an upstream version. A
declaration naming a key the entity does not carry, or a type the resolver
does not know, fails the lint instead of resolving nothing in silence.
Picked from feature/store-i18n (94d607fd0a) onto current main. Two things
differ from the original: svc-ai-lmstudio no longer carries an app_version
pin, so its declaration is dropped, and web-app-stalwart's webui_version,
added since, declares the git tags of the webui repository.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
c6e8990e25 kept the daemon's auth key off the shared volume and made the red
job green, but it left multi-replica operation as broken as it found it: three
daemons still shared one /root/.lmstudio, where LM Studio rewrites
extensions/plugins with rm -rf at every start, writes settings.json in place and
resets it to defaults on a torn read, and re-copies a 102 MB bin/lms while peers
execute it.
Splitting the tree by access pattern fixes that. /root/.lmstudio is node-local,
because it is per-daemon state the daemon rewrites in place. models/ stays
shared, because a blob is written once and read forever.
That leaves one question, and it is temporal rather than mechanical: the daemon
discovers models by an unconditional scan at startup and by an inotify watcher,
and inotify never sees writes made by another NFS client. So a model seeded
after a daemon started is invisible to it, whichever tool wrote the file. The
previous design answered that with `container service update --force`, which
restarts the whole inference backend svc-ai-litellm points at, on every deploy.
The seed now runs before the stack does. 00_core.yml places the blob on the NFS
export ahead of the compose include that deploys the stack, so every daemon
finds a complete tree at its own startup scan and nothing has to be restarted.
Measured on a three-node swarm with NFS:
Fetch the models into the shared store 136.14s one 531 MB download
Pre-pull on mgr-01 0.69s
Pre-pull on wrk-01 0.50s
Pre-pull on wrk-02 0.49s
Half a second is not a download. All three daemons recognised the pre-placed
model, so the cluster now stores and transfers it once instead of three times.
The nested mount across two devices raises no EXDEV, and the per-node disk guard
correctly skipped itself four times because the pre-seed covers that check.
Vendor coupling is gone from the role. `repo` and `url` are declared fields
rather than pieces cut out of a HuggingFace URL with a regex; a model on any
host needs only those two. `lms get` still knows only HuggingFace, so `source`
stays for it, and a non-HF model simply leaves it unset: the blob is seeded and
the startup scan picks it up. The external check follows, comparing against the
HuggingFace API only for entries whose url actually points there.
The guard arithmetic was wrong and is now derived rather than guessed. Peak
usage is the missing bytes plus the largest single model, not twice the declared
total, so the read-back moved ahead of the check and subtracts what the store
already holds. Before, the requirement grew after every successful seed. Staging
moved beside the tree instead of inside it, so an aborted transfer cannot leave
orphans where the indexer scans.
A deployment test ships with it and states its findings rather than its exit
code: it fails on `authPacket` by naming the shared key, on a missing or
mismatching blob by naming path and digest, and on a model that is on disk but
absent from /v1/models by saying the startup scan did not happen.
Gone with the restart: import.sh and its container-cp suppression, seed.yml,
rescan.yml, and the disk-copy factor of three that only existed because the
import staged through the container.
Residual: compose is untested here. There the pre-seed is off, the node-local
guard is the one that runs, and `lms get` is the only fetch, so a non-HF model
is not loadable in that mode. Seeding node-local volumes is the follow-up that
closes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A wedged LM Studio daemon cost CI job 105558224377 three hours: three kills at
the global 60-minute `CONTAINER_EXEC_TIMEOUT`, for a 531 MB model that a healthy
node pulls in 27 seconds. Nothing in the path was sized against the thing being
fetched, and nothing checked whether it could land at all.
Size. Each model now declares `bytes`, taken from the value the deploy already
printed as "OK (531068224 bytes)", and the pre-pull derives its own timeout per
model: max(preload_floor_seconds, bytes / preload_min_bytes_per_second). The
floor of 600s protects a small model from a brief stall; the rate of 1 MB/s sits
twenty times below the 21.98 MB/s a healthy pull reached and seven thousand
times above the 142 B/s the wedged one managed, so it separates the two without
guessing. Evaluated against ansible-core's filters: 531 MB and 10 MB both land
on the floor, 4 GB gives 4000s.
Disk. The seed measures free space with `df -P -B1` before fetching and refuses
with a message naming both figures in GiB, the factor and the file to edit. The
factor is three because the node cache, the staging copy inside the container
and the models volume all exist at once during the import; that is measured, not
padding.
Fallback. A model may list `mirrors`, extra bases tried after the source. The
mirror task is a no-op once the source delivered, because get_url skips a dest
whose checksum already matches: measured at 145.24s for the first pass against
0.64s for the second. The list ships empty, because the obvious candidate is
not one: hf-mirror.com answers 206 but resolves to us.aws.cdn.hf.co, so it is a
front end for HuggingFace and adds no redundancy against a HuggingFace outage.
The mechanism is here; an independent copy is the operator's to name.
The external check grew with the declaration: `bytes` is compared against the
size the API publishes, and every declared mirror must still serve the pinned
file, or it is redundancy on paper only.
01_manager_ops.yml's rescue rationale claimed that restarting the container
clears a wedged transfer. Job 105558224377 shows the restart returning rc 0
without clearing it, so the claim now says what was measured instead.
Residual: the mirror path has never run in a deploy, since no mirror is
declared. Its loop expression is verified against ansible-core's subelements
over four shapes, including the zero-iteration case when nothing is mirrored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An addon may pin `config.archive` so the deploy fetches that release instead of
letting the app store choose. The pin is worth exactly as long as the artifact
outlives it: when upstream retires the release, every deploy of the role fails
at install time with a 404 and nothing in the failure names the declaration as
the cause.
That is not hypothetical. ONLYOFFICE listed onlyoffice 10.2.0 in the Nextcloud
appstore and never published the artifact, which made Nextcloud#4 red in both
runs 35329488812 and 35330645276 on the same commit. This check answers the
mirror image of that question: does the release we pinned still exist.
The split follows the sibling model-pin check. The declaration's shape is a
repository fact and is asserted; whether the archive is still served is a
network fact, so a timeout or a refused connection reports unverified rather
than failing, because a firewalled runner must not turn a good pin red. A 4xx
is an answer that contradicts the pin and does fail.
Negative control: pointing the pin at the retired v10.2.0 produces
web-app-nextcloud.onlyoffice: .../v10.2.0/onlyoffice.tar.gz answered HTTP 404
so the check would have caught today's failure before it reached CI. The
unverified path is exercised too: with the sandbox blocking the host, the run
warns instead of failing.
Residual: this proves reachability, not compatibility. A pin whose max-version
sits below the deployed server major passes here and fails loudly later at
`app:enable`. Today the single pin is v10.1.2, which the store declares for
>=33.0.0 <35.0.0 against a role that deploys 34.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The preload is the only artifact this repository pulls raw from the public
internet. Everything else, including the LM Studio image itself, arrives
through the mirror. `lms get` gets exactly one attempt under async, cannot
resume, and wedges its own daemon when a transfer dies, so one dropped socket
costs a whole job. XWiki#2 of run 35152776835 died 28 seconds into the
transfer at 3 MB/s and rising, with 3600 seconds of budget left.
The model is now fetched on the hosting node with get_url_retry, which gives
13 attempts and verifies a pinned sha256, and handed to the container with
`lms import`. Both steps are best effort: when the warm-up fails, `lms get`
runs exactly as before, so the change adds a faster path without removing the
old one. The pattern is web-app-mediawiki's extension install, down to the
`container cp` in the script.
Verified against the binary the role actually deploys, extracted from
lmstudio/llmster-preview:cpu and built 2025-08-28, not against upstream HEAD:
`lms import` takes `<file-path>`, `-y, --yes` and `--user-repo <user/repo>`;
the destination is join(modelsFolderPath, user, repo, basename(path)) under
join(lmstudioHome, "models"); the image ships no settings.json, so
downloadsFolder cannot move that target. The already-present branch of
`lms get` carries no process.exit, so it returns 0 and the task's existing
`until: rc == 0` accepts it.
The pin carries file and sha256 next to the source URL, and the digest is
quoted because an all-digit hash would otherwise parse as an integer and fail
the shape check for the wrong reason. tests/external guards the pin against
what HuggingFace publishes, comparing digests through the API without
transferring the file, and reports a network error as unverified rather than
failing a good pin on a firewalled runner.
Two limits worth stating. `lms get` still queries HuggingFace before it
reaches the download step, so this removes 531 MB of exposure rather than the
network dependency. And the tolerated substring 'already downloaded' in the
task's until, failed_when and reap list matches no output this version emits,
which says "You already have this model."; that branch is dead today and is
left untouched here because the return code carries the contract.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing here was typed by hand. The docs generator rebuilt the MCP audit
table against pins that had already landed, so fider reads v0.36.0 instead
of stable, jenkins 2.568.3-lts instead of lts, pretix 2026.7.0 instead of
stable, and flowise 24-slim instead of 24-bookworm-slim. That is the table
catching up with da665ab69 and 10c5b35e8, reading the same declaration the
updater reads.
Autoformat wrapped three test files from the same series: one condition
pulled onto a single line, three assertEqual calls broken across lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ranking the last commits built toward is now enforced: a semver tag,
else a digest, and nothing else without a written reason. Two states fail.
A pin that is neither names something the registry can move underneath us,
and nothing here can tell whether it still points where it did when it was
written. A digest on an upstream that does publish semver takes the
fallback for no gain and hides the entry from the update job.
Thirteen entries stood in the way when the check was first run. Four were
resolved by the channel grammar, which made the Jitsi components orderable
without touching their pins. Three needed nothing once a moving tag could
declare itself with 'docker-version'. Two are fixed here and four are
declared.
LiteLLM leaves 'main-v1.77.3.dynamic_rates'. That tag exists exactly once
among 2609, so it is a dead end no updater can follow, and it misreports
itself: the image inside answers 1.77.2. The pin moves to
'main-v1.77.7-stable', five patches along the same minor and onto a channel
with 134 builds. Started against this repository's config shape, 1.77.7
registers both model entry forms and accepts drop_params and the
os.environ master key.
Pixelfed leaves 'latest' for '20260917'. The upstream stamps its dates as
one eight-digit number, which is depth 1 and orderable with no tooling
change at all, and the two tags resolve to the same digest, so nothing
about the deployed image moves.
The four declarations carry the count that justifies them rather than a
claim: LM Studio publishes two tags and neither is a version, the MIG
registry carries one, and the two Nextcloud Talk images publish 174 and 104
AIO build stamps between them.
Both branches of the gate were negative-controlled. The digest branch
skips in this repository because no digest pin exists at all, so it was
exercised by introducing one and watching it fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A digest pin was emitted as `name:sha256:...`, which no registry accepts:
image_source('ghcr.io/.../csp-checker', 'sha256:d559...') returned
'ghcr.io/.../csp-checker:3.0.0'. The separator depends on the pin class, so
the class has to be decided before the reference is built.
utils/docker/image/pin.py names the three classes in preference order
(semver, then digest, then a general ref that no registry can order) and
maps each to the nocheck marker that declares it. Both discovery entry
points now end in pull_reference(), so '@' versus ':' is decided in one
place instead of at every call site.
The semver pin lint points a digest at 'docker-digest' instead of
'docker-version', so the two fallbacks stay distinguishable in the
suppression catalog. It stays warn-only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ubuntu CI image build (job 103277644636) hung on Canonical's
archive: apt fetched 26 MB at 417 kB/s, then 61.6 kB in 29 s, then sat
on the package download until the job was cancelled after 23 minutes,
while download.docker.com delivered 42.8 MB in under a second on the
same runner. Locally archive.ubuntu.com and security.ubuntu.com took
20-26 s to the first byte against 2 s for a university mirror.
INFINITO_APT_UBUNTU_MIRRORS in default.env is now the one list of
Ubuntu mirrors, in failover order: de.archive.ubuntu.com, ftp.fau.de,
ftp.halifax.rwth-aachen.de, archive.ubuntu.com.
- Image builds: the Dockerfile sources the list alongside the install
scripts and runs scripts/install/apt-mirrors.sh first. In an image
with ubuntu.sources it points both stanzas at apt's mirror+file
method, priorities in list order, and sets Acquire::http::Timeout to
10 s, so apt moves on from a mirror that fails or goes silent; other
distros are untouched. In ubuntu:latest apt-get update took 8 s
against 110 s on the stock sources, a dead first mirror was left
behind within 25 s against 62 s on apt's defaults, and the swarm node
packages (80 debs) came in 17 s. The Dockerfile's own RUN lines ran
clean on ubuntu, debian, fedora and arch.
- Package cache: package.sh proxies the first entry as the primary
Ubuntu repos and the second as their fallback. Nexus revalidates
metadata on every request here (metadataMaxAge 0), which costs a
full download from a mirror without an ETag; both de.archive and
ftp.fau.de send one, and the 15 MB universe index now comes back
from the cache in 0.02 s.
- tests/external/repository/test_apt_ubuntu_mirrors.py keeps the list
honest: the first mirror must answer InRelease within 5 s (median of
three) or the test names a faster later one, no mirror's -updates may
lag the freshest by more than 24 h, and every entry must be the
primary archive, a Canonical country alias or an enabled official
mirror in Launchpad's registry. It caught the first draft of the
list: mirrors.edge.kernel.org is not an official Ubuntu mirror.
After pulling, an existing checkout needs `make dotenv-force`: plain
`make dotenv` re-reads the old .env through BASH_ENV and keeps its
values.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The info-media marker moved to meta/info.yml one commit ago, and this rule is
where the reason it could not move earlier was written down: a comment-only
info.yml parses to an empty document, which the schema lint used to reject. That
obstacle is gone, so the docstring now states a justification that no longer
holds, and the two video-related exemptions sit in different files for no reason
a reader can reconstruct.
So the marker is read from info.yml, the failure message names info.yml, and the
docstring says why the marker lives there rather than why it cannot. No role
opts out of this rule today, so nothing under roles/ changes and the move cannot
break in either direction.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The root-README overview renders a Video link per role and a reader following it expects a demo, but 26 of the 62 entries addressed a channel, a playlist or a product page: youtube.com/@zammadhq, /channel/UC..., framatube.org/c/mobilizon/videos, pgadmin.org/videos. None of those can be embedded and none shows the product without a second click.
Each replacement was resolved through YouTube's oEmbed endpoint, which returns the real title and channel, so a candidate could be checked rather than assumed. That caught a supposed LibreOffice intro published by an unrelated channel, a Discourse overview belonging to a hosting vendor, and a Zoom video answering 403 with embedding disabled. Fourteen of the new entries come from the vendor itself and two are PeerTube.
The reachability half needed the same endpoint for a different reason: YouTube answers a removed video with HTTP 200 and a Video-unavailable page, so a plain fetch proves only that YouTube is up, while oEmbed answers 400. Measured both ways before relying on it.
The check is external because reachability needs live HTTP, and like its siblings it reports an unanswered probe as unverified rather than failing, so a firewalled runner cannot redden a good URL. Only the shape is asserted, because that is a repository fact.
README.md follows rather than leads: overview.py renders the Video column from this field.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The version cron updates docker image tags and repository refs, and tests/external/update covers docker, addons and repository. Neither sees a version inside an Ansible task variable, and dependabot reads pyproject.toml rather than roles/. A pin set once therefore froze and nothing thawed it: backup-docker-to-local has not moved since it was written, and the automtu pin one commit back would have gone the same way.
utils/update/pip mirrors the docker module: collect the pins, ask PyPI, rewrite in place. Only exact == pins are collected, because a range or a VCS reference states an intent an updater must not overrule. The external check warns rather than fails, like its docker sibling, so a slow PyPI cannot redden a run and a release the repository has deliberately not taken is not a regression.
The cron job is the docker job with one script swapped, so a changed pin arrives as its own PR under update/pip-pins.
Executed: the collector finds both pins in the repository; a synthetic file rewrites from 1.0.0 to 1.1.1 with the indent, quoting and neighbouring line untouched; automtu==1.0.0 would be reported while a bare name and a git+ reference are passed over.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The connectivity canary fetched https://pypi.org/simple/ with the suite's
ten-second read timeout. That path is the complete PyPI index: measured at
45 MB here, it times out long before it finishes, and the suite then aborts
with "This host cannot reach the infrastructure the repository depends on"
even though pypi.org answers 200. _probe_url uses requests.get, so the whole
body is downloaded to decide a yes/no question.
https://pypi.org/simple/ansible/ answers the same question against the same
host and service in 199 KB and 0.41s, and ansible is a declared dependency in
pyproject.toml, so the canary still sits on a path this repository actually
depends on.
The guard keeps its teeth. 7b5c462af2 added it because a networkless run had
reported green over 244 unanswered probes, 110 of them real 404s; an
unreachable pypi.org still fails this canary and still aborts the suite.
Only the volume needed to establish reachability changes.
The other two canaries were measured and left alone: registry.npmjs.org
returns 2 bytes in 0.25s and github.com is likewise small.
make test-external went from FAILED TARGETS with exit 2 to 11 passed.
Two gate runs and a CI workspace job each reddened on a single opencollective funding link from package-lock.json, and all three URLs answered 200 on every later probe. A one-off 4xx from a third-party host is not a broken reference, so the suite now repeats such a probe once after a pause, the way _probe_canary already treats an unreachable canary. A status that repeats still fails, and a transport error is not retried because it is already only a warning.
The retry runs inside the worker that owns the URL, so the pool keeps its concurrency: the added wall-clock is one pause per wave of failures, and the runs measured 84s and 88s against 100s without it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each of these is invisible because every individual setting is valid on its own. A sandboxed workload and the host container socket are both legitimate, only their combination is not: the socket hands over the host daemon, which can start a privileged container on the host kernel, and the isolating runtime becomes decoration while the deploy stays green. An embodiable agent that forgets the robot admission include deploys privileged onto a shared node and nothing says a word, and the include cannot live in svc-ai-robot because the service loader preloads it a stage earlier.
A provider bearer handed to a browser through playwright.env.j2 lands in the retained trace, its headers, its environment dump and its network log, and outlives the deploy that minted it inside a downloadable CI artifact. A served surface without its own mcp RBAC role leaves a client nothing to scope a grant to, and the fallback is not nobody: an empty grant reads as every administrator. An image bump to an amd64-only tag turns an arm64 promise into a scheduling failure on the Pi rather than a red test here, because platform_arch exists to pin a role to one architecture and a role that must run everywhere declares nothing.
Four of the five derive their scan from declarations rather than a list, and each carries a vacuum guard that fails when its scan comes back empty, since a rule whose derivation broke is otherwise a rule that passes forever.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bd84664e8 moved debian and arch to a cached Packages index but left ubuntu
on packages.ubuntu.com, one HTTP request per package. That endpoint is the
same anti-bot territory: 96 requests behind a 2-per-host semaphore, and a
single timeout fails the target. Three workspace jobs of one run died that
way on different packages (tor/traceroute, msmtp), which is a lottery, not
a defect.
The debian loader is now generic over mirror, path, pockets and components,
so ubuntu answers from the same 12-request bulk fetch through the existing
_get retry layer. It also probes the pockets a default install resolves
against: ubuntu ships 0ad only in noble-updates, and the release-pocket
index alone reports it absent. The old HTML probe called it present because
the "not available" page does not match the "no such package" regex, so the
per-package route was reporting a false positive as well.
Verified: 96/96 identical verdicts old vs new across every ubuntu probe,
454 probes and 23 inconclusive unchanged, and test-external green inside
the container through the package-cache frontend (8 passed in 108s, against
1025s and red in CI).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-package endpoints are anti-bot territory: packages.debian.org shed
request-level 503s for >13h (2 CI runs and 2 local quality gates red, 7
resp. 91 probes unverified), and archlinux.org connection resets are the
chronic class 00784105f already fought. One Packages.xz/.db fetch per
distro answers every probe from a CDN meant for machines, through the
existing _get retry/semaphore layer; the index is cached per run behind a
lock, and an unreadable index stays a loud unknown.
Debian goes over http because the dev container hijacks deb.debian.org
onto the package-cache frontend, which serves the name on port 80 only -
the same transport the stack's own apt list uses. The stricter index also
exposed three registry declarations that never existed in Debian stable;
they are fixed in the follow-up commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A name that fails to resolve was re-raised on the first attempt, so a
resolver hiccup - the local one drops its :53 sockets whenever docker
restarts a container - failed the whole external suite.
The carve-out protected nothing: the exception is re-raised after the
last attempt either way, so a genuinely unresolvable host still fails.
It only bought a faster failure. Dropping it also matches how the rest
of the repository already groups gaierror with OSError.
A connection reset while reading the response body surfaces as
http.client.RemoteDisconnected, which is an OSError but not a URLError,
so it escaped the retry entirely and failed the whole external suite on
a single flaky AUR request.
archlinux.org answers 429 and resets connections when the suite fires its
16 workers at it: 22 arch probes in parallel, 12 answered and 4 refused in
a reproduction inside the test container, while each name resolves fine on
its own. Those 22 self-inflicted failures then took the warning path,
because an unreachable index is supposed to say nothing about the
declaration — so the suite reported green on 22 packages it never checked.
_get now holds at most two requests per host and retries 429 and 5xx with
backoff; a DNS failure is not retried, it is not transient. That alone
removed all 22 warnings, 45 down to 23, with the runtime unchanged.
The remaining 23 are declared: a third-party repository the role
configures itself, or a name the index legitimately does not list because
the package is virtual. Outcome carries that as a flag now, so those stay
warnings while anything else that could not be checked fails the suite and
names each case. Verified by pointing the arch probe at an unreachable
host: 94 packages reported as unchecked, and the run went red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A networkless run reported green over 244 unanswered probes, among them 110 lockfile URLs that were in fact HTTP 404. Probe github.com, pypi.org and registry.npmjs.org before anything else and abort the suite when one of them stays unreachable, so a run that cannot verify anything can no longer pass.
Per-URL transport failures stay warnings: blocked, geo-fenced and rendered placeholder hosts would otherwise redden CI. Onion addresses are skipped, since no clearnet probe reaches them without a Tor SOCKS proxy this suite does not use. Each canary retries once so a single blip cannot fail the run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workflow directory mixed entry points, reusable stages and scheduled jobs into one flat list sorted by topic, so the first question about any file - what starts it - was the one the name did not answer. Every workflow now carries one of three prefixes, derived from its on: block: cron- for a schedule trigger, call- for one only reachable through workflow_call, entry- for everything a repository event or a human starts. Scheduled workflows additionally carry an alarm clock as the first character of their name.
Two lints keep it that way. One derives the expected prefix from the triggers and compares it against the file name; it flagged all 17 offenders before the rename and none after. The other resolves every .github/workflows/<file> string in the repository against the directory, which is what a rename silently breaks in Markdown links, Python constants and shell API URLs alike.
Renaming pulled 69 pre-existing comments into the comment lint's changed-file scope. The vendored CodeQL template got the nocheck marker its scorecard sibling already carried, two workflow headers moved above name: where they count as file headers, and the remaining banners were deleted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The inline repo definition pinned gpgkey to www.centos.org, a single
non-CDN origin. In run 30937913632 it was unreachable for at least
17m25s and killed 2 of 23 centos ganesha installs; dnf's own retries=5
was already active on the host and did not help.
centos-release-nfs-ganesha11 ships the same repo with metalink failover
and gpgkey=file://, planting the key through Requires:
centos-release-storage-common. Verified end to end against
quay.io/centos/centos:latest and ghcr.io/kevinveenbirkenbach/pkgmgr-centos:stable.
The BOOTSTRAP_BASEURL entry keeps the availability probe verifying both
names instead of degrading them to an inconclusive warning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An over-declared min_storage makes a host look too small: it feeds the storage guard, the runner packer and the sufficient-storage filter, so a deploy that would have fit prunes a cache it did not need to lose. Nothing checked the declared number against reality.
The new external test measures what a pull of the service's own image:version actually transfers - config blob plus every layer blob, resolved for linux/amd64 - and fails when min_storage exceeds it by more than 1GB. A service whose data volume genuinely dominates keeps its value by carrying '# nocheck: min-storage-headroom' together with a 'Reason:', the same two-marker shape meta/volumes.yml already uses for nfs: false.
Sizes come from the repo's own registry client rather than 'docker buildx imagetools': make test-external runs inside the compose container, which has no docker socket. manifest_transfer_size reuses the existing bearer-challenge flow, keeps the tri-state contract of manifest_exists (an indeterminate registry warns instead of failing), and resolves a multi-platform index to its amd64 manifest - the case a raw jq over '.layers' would miss entirely. Distinct image refs are measured concurrently through resolve_max_fetch_workers().
A min_storage that parses out of the YAML but cannot be located in the file text fails the test, so a declaration can never silently lose its marker resolution.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A role states which package it needs; the registry owns every distribution
mapping and every acquisition path. Declarations live in the repository-root
meta/packages.yml when more than one consumer installs them, and in a role's
own meta/packages.yml otherwise.
schema.py owns the declaration shape, registry.py finds and indexes the
declarations, plan.py turns a resolved spec into ordered module calls, and the
package_install action plugin only executes them. Keeping the plan pure makes
every acquisition path unit-testable without an Ansible connection:
distribution repositories, the AUR with its unprivileged build user, COPR, PPA,
an EPEL bootstrap package, the pacman multilib section and source builds.
The data is guarded by lints: a package id and a concrete distribution package
name each belong to exactly one declaration, a role installs only its own or
shared ids, every id resolves for all five default distributions with an
empty mapping justified by a nocheck, declarations read in ascending order,
every plan builds and names only whitelisted modules, and an external test
queries the real package indexes of all five distributions.
This commit adds the mechanism and the shared collection; no consumer uses it
yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ruff toolchain was unpinned, so host venvs, the lint container and CI
installed different releases over time. That drift is why lint-python only
failed on CI: locally ruff 0.15.x preview-gated ISC004/FURB192 and (falsely)
fired S310 on urllib.request.Request, while CI's 0.16.0 has ISC004/FURB192
stable and drops the S310-on-Request false positive.
Pin ruff==0.16.0 in pyproject.toml's dev extras (Dependabot keeps it current)
and have utils.install.lint.ruff read that pin and reinstall whenever the
found version drifts, so every environment resolves the same ruff. Then fix
what 0.16.0 flags: parenthesise the implicit string concatenations (ISC004),
use min() over sorted()[0] (FURB192), drop the now-unused S310 noqa on the
Request() builders (RUF100), and delete the narration comments that the
comment lint surfaces in the touched files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the operator cheatsheet's copy-paste prompts into per-workflow i8- skills
that each route to their authoritative procedure doc, and reduce the cheatsheet
to a table index plus a usage paragraph, enforced by a new lint. Add skills for
develop, test, quality, refactor, the debug and requirement flows, and
commit / pull-request / push-trigger-pull; rename the iteration skills to
i8-iterate-* and rename the index to i8-help. Wire the active-listening,
dialectic, and robot skills into the flows for clarification, uncertain root
causes, and autonomous execution. Drop the Infinito-specific agent aliases
(alias.md, INFINITO_ALIAS_MD, the make alias agent section, and the two alias
tests); terminal aliases stay.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Atlassian's CDN served the CI runner a 404 for a live page because the probe announced itself as a bot; the reachability question is whether a link works in a browser, so the probe now sends a browser user agent like standard link checkers do. Touching the file pulled it into the changed-scope comment lint, so its narration blocks are gone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
elastic.co answers the probe's Accept/UA with 406 while the page is
alive; classify it with 401/403/405/415 as server-alive instead of a
dead link.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add the i8* terminal aliases (make install-alias) and ai8* agent
conversation shortcuts (docs/.../agents/alias.md), move skill
distribution to the external INFINITO_SKILLS_REPOSITORY (make
install-skills), and add make cheat, make alias and make onboard. Introduce
custom.env operator overrides (win over defaults and handlers) and pin
INFINITO_ALIAS_MD / INFINITO_ALIAS_REPOSITORY / INFINITO_SKILLS_REPOSITORY
in default.env. Drop the in-repo skills-lock.json and its update-skills CI
job. AGENTS.md instructs agents to suggest the matching cli/prompt alias
when an operator writes something an alias covers. Lints enforce alias
sorting, i8/ai8 prefixes, no consonant clusters, full make-target
coverage, and terminal/agent shortcut non-overlap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a generic Docker Registry v2 client (WWW-Authenticate bearer-token
challenge) so tag listing and reachability work for any registry (Docker
Hub, GHCR, MCR, Quay, GitLab/opencode), and route the version updater
through it so quay.io and opencode.de images auto-update like the rest.
Add an external test that hard-fails when a pinned image:version tag is
unreachable (404) and warns on incomplete image/version pairs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
keepass.info responds with 415 Unsupported Media Type to the probe's
GET; the previous 25a77bc7e widening landed 418/429/451 but missed 415.
Treat 415 like 401/403/405 (server alive, request format rejected).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes the gap left by test_image_versions, which silently skips any
non-semver pin (latest, stable, branch tags). The new test_image_reachable
walks every entry from iter_role_images and probes the Docker Registry v2
manifest endpoint via HTTPS with anonymous bearer-token negotiation
(docker.io, ghcr.io, quay.io, mcr.microsoft.com). HTTP 404 hard-fails as
a broken pin; auth/rate-limit/5xx/network errors warn. Bounded by per-
request, per-probe, and global deadlines and parallelized via
INFINITO_WORKER_FETCH.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new lint tests close gaps in the INFINITO_* SPOT contract:
* tests/lint/filesystem/env/test_default_env_keys_handled.py
Every default.env key must be wired through a handler.
* tests/lint/filesystem/env/test_used_keys_declared.py
Every INFINITO_* in .sh/.py must live in env/{default,ci}.env or
a handler; per-line `# nocheck:` suppresses runtime-only refs.
* tests/lint/filesystem/python/test_no_infinito_get_defaults.py
Forbid ANY default in `.get("INFINITO_*", ...)` -- including the
empty-string fallback (which silently masks an unset key). The
nocheck check walks every physical line the call spans so an
autoformat-split call still honours the marker.
Cache namespace flattens to a single prefix:
INFINITO_PACKAGE_CACHE_* -> INFINITO_CACHE_PACKAGE_*
INFINITO_REGISTRY_CACHE_* -> INFINITO_CACHE_REGISTRY_*
The five previously driver-injected paths (APT_LIST, NPMRC,
PIP_CONF, REGISTRY_PROXY_CONF, FRONTEND_CA_FILE) move into
env/default.env and passthrough.STATIC_KEYS; the
`cache_env_overrides()` helper and its two call sites are gone
along with the test-fixture override for FRONTEND_CA_DIR.
SPOT-strict rewrites:
- 19 sites: `something.get("INFINITO_X", "")` -> subscript
`something["INFINITO_X"]` (loud KeyError when env loader did
not run; all keys live in env/default.env).
- 2 handler-internal bootstrap reads suppressed with
`# nocheck: handler-bootstrap-read`.
- 30 runtime-only / driver-injected / test-fixture references
annotated with `# nocheck: <reason>` per-line.
- Unit tests that previously removed SPOT-guaranteed keys via
`{k: v ... if k \!= KEY}` or `clear=True` switch to setting
the key to "" explicitly, matching the new contract.
Side fixes:
- INFINITO_DNS_DOMAIN + INFINITO_DNS_HOSTS_FILE move from
shell-side defaults in common.sh to env/default.env and
passthrough.STATIC_KEYS.
- dns_systemd_is_operational() drops the dead
INFINITO_DNS_FORCE_HOSTS_FALLBACK override branch (the
variable was never set anywhere).
- INFINITO_WORKER_FETCH switches from `os.environ.get(KEY, "1")`
to `os.environ[KEY]`.
- scripts/tests/code/wrapper.sh exec args move into a bash
array so `# nocheck:` on INFINITO_TEST_TYPE can live on its
own element line (SC2215 fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The URL-reachability probe now bounds its wall-clock end-to-end:
allow_redirects=False (a redirect chain previously stacked
timeouts), a 5h30m global deadline that marks in-flight probes
as warnings on expiry, and live `[done/total]` progress every 50
probes. Submission order is shuffled so same-host URLs no longer
batch at the front and self-DoS one origin.
The Docker image-version test drops its bespoke sequential
fetch loop and calls find_outdated_updates() from utils.update.docker,
so it now shares the INFINITO_WORKER_FETCH-sized worker pool with
the repository-refs check.
`scripts/tests/code/wrapper.sh` propagates BASH_ENV into the
container exec — without it INFINITO_WORKER_FETCH was unset
inside the container and the URL probe collapsed to 1 worker.
`make test` gains test-external; `make dotenv-force` regenerates
.env with a stripped environment so stale BASH_ENV-preloaded
INFINITO_* values cannot pin old numbers via setdefault.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename INFINITO_PARALLEL -> INFINITO_WORKER_ENABLED and add two derived
keys so consumers no longer multiply by their own factor: the env
generator resolves the cascade once at `make dotenv` time and bakes
the integer counts into `.env`.
Chain:
- INFINITO_WORKER_ENABLED (operator toggle, default true)
- INFINITO_WORKER_CPU = cpu_count() when ENABLED truthy, else 1
- INFINITO_WORKER_FETCH = CPU * 20 for I/O-bound bulk fetches,
collapsing to 1 when CPU == 1
Handlers:
- utils/env/handlers/infinito_worker.py (the prior single key)
removed
- utils/env/handlers/infinito_worker_cpu.py + infinito_worker_fetch.py
added; registry orders them after `passthrough` so the cascade
reads its dependency from the env-builder state.
Consumers (now 1-liners):
- tests/external/repository/test_urls_reachable.py reads
INFINITO_WORKER_FETCH directly.
- utils/update/base.py::resolve_max_fetch_workers() reads it directly;
utils/update/{docker,repository}.py keep using that helper plus a
ThreadPoolExecutor for the actual bulk fetch (with `dict(pool.map(...))`
satisfying PERF403).
- scripts/lint/autoformat.sh + docs/contributing/tools/make.md
rename INFINITO_PARALLEL -> INFINITO_WORKER_ENABLED.
Tests:
- EXPECTED_BASELINE_KEYS in test_dotenv_generator updated to track the
three new keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
env/static.env:
Drop the stale "and 'make lint-galaxy'" reference from
INFINITO_PARALLEL's comment (lint-galaxy was removed earlier).
tests/external/repository/test_urls_reachable.py:
Make _MAX_WORKERS network-aware. URL reachability is I/O-bound, so
cpu_count() is the wrong proxy; scale by 20x to saturate the wait
budget. Respects INFINITO_PARALLEL=false for sequential fallback.
tests/unit/utils/install/test_github_release.py:
Replace the github.com/x/y/ dummy URLs with example.com/x/y/ so the
URL-reachability test (which scans source for HTTP literals) skips
them via its example.* allowlist. The github.com URLs were probed
for real and returned 404.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add MCR_REGISTRY constant, is_mcr / mcr_repo / fetch_mcr_tags helpers
mirroring the Docker Hub and GHCR pattern. Tags are fetched via the
Docker Registry V2 API with RFC 5988 Link-header pagination (anonymous
access; MCR is a public read mirror). Wire MCR into both the external
test_image_versions warning surface and the find_outdated_updates
orchestrator in utils.update.docker.
- Bump mcr.microsoft.com/playwright in test-e2e-playwright from
v1.59.1-noble to v1.60.0-noble — the update was surfaced by the new
MCR detection path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Folds the parallel Docker-image and git-ref version-update plumbing
into a single subsystem so the auto-update workflow, the external
freshness tests, and the unit tests all share one module layout.
utils/
update/
base.py # semver primitives shared by both backends
docker.py # OCI registry lookups + version: line rewriter
repository.py # git ls-remote walker + ref: line rewriter
cli/contributing/update/
docker/ # python -m cli.contributing.update.docker
repository/ # python -m cli.contributing.update.repository
scripts/github/update/
image_versions.sh
repository_refs.sh
Tests mirror the same shape under tests/{unit,external}/update/{docker,repository}/.
Also widens the Docker bump scope to all roles (previously gated on
`web-*`) so non-web roles like sys-svc-container get bumped too;
pins Bookwyrm's python image at 3.11-bookworm via
`# nocheck: docker-version` with a TODO until upstream supports
Python >= 3.12.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `tests/external/repository/test_repository_versions.py` — the
git-source counterpart to `tests/external/docker/test_image_versions.py`.
Walks every role's `meta/services.yml` recursively, collects every
`(repository, ref)` pair (top-level entity, sub-entity, plugin map),
filters to semver-compatible refs, queries upstream via `git ls-remote
--tags`, and emits GitHub Actions `::warning::` annotations when a
newer semver tag exists. Always passes — outdated refs are warnings,
not hard failures, so the suite stays green when remotes are slow or
unreachable. Suppress per-entry with `# nocheck: repository-version`
on the line directly above the `ref:` key.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Widens the cache-read lint guard from ``tests/lint/`` to every ``.py``
under ``tests/``. Integration tests show the same compounding cost
pattern (one role's compose template / nginx conf / scrape fragment
read across many shape tests) that motivated the original lint for
lint/.
Converts 91 violations across 28 test files:
- **Cache (utils.cache.files.read_text)** for project-tracked reads
and single-read tempdir fixtures, e.g. test_prometheus.py (32x),
test_keycloak_import_flow.py (11x), test_dashboard_oidc.py (3x),
test_changelog_md.py (8x), test_package_mirror.py (8x), plus
test_logging.py / test_summarize.py / test_version_updater.py /
test_get_url_retry.py / test_pull_specific_host.py.
- **Cache + UnicodeDecodeError handler** for project-walk scanners
that previously used ``errors="replace"`` / ``errors="ignore"`` to
defend against non-UTF-8 files (compose volumes, password quoting,
template image, mailu dependency, URL reachability). Wraps each
read in ``try/except (OSError, UnicodeDecodeError): continue``,
which has the same observable effect (skip undecodable files).
- **Refactor away the binary heuristic** in test_no_raw_docker.py:
``read_bytes() + b"\x00" not in raw`` becomes
``read_text() / except UnicodeDecodeError``. Same files skipped,
with cache benefit.
- **Variable extraction** in test_changelog_md.py / test_package_mirror.py
for cases where ``self.assertEqual(path.read_text(...), expected)``
exceeded ruff's 88-char limit. Ruff was wrapping the marker onto a
line outside the AST span of the read_text call.
- **Compute-from-setup** for two test methods that previously read
``self.changelog`` twice (before/after) to verify a no-op:
``before`` is now derived from the known input, dropping the first
read.
Five ``# nocheck: cache-read`` markers remain, each annotated inline
with the reason: idempotency tests where the second read must hit
disk to catch non-idempotent regressions, and tests with intermediate
``write_text`` whose subsequent read would return stale cache data.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidates image declarations under a single schema: every role's
`<svc>.{image,version}` lives in `meta/services.yml` alongside compose
service entries. The `image` lookup now reads from disk per role,
`iter_role_images` collapses to a single source, and the mirror chain
(resolver + apply_mirror_overrides) drops the dead `images:` branch.
`test-e2e-playwright` resolves `TEST_E2E_PLAYWRIGHT_IMAGE` directly via
`lookup('image', 'playwright', 'ref')` in vars/main.yml; the runtime
set_fact + override-empty-string dance is gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_role_file_spot.py: AST-walks every .py
file under the production trees and tests/, flags any reference to a
role file (standalone literal, f-string segment, pathlib `/` chain,
`+` concat chain, os.path.join / Path.joinpath / posixpath.join /
ntpath.join call) whose value or joined trailing operands match a
ROLE_FILE_* constant in utils.roles.mapping. The flag set is
auto-discovered from the module so adding a new constant extends
coverage without further edits. ROLE_FILE_README is exempt as too
generic. Per-line `# nocheck: role-file-spot` (same-or-above) is the
documented opt-out, catalog row added in
docs/contributing/actions/testing/suppression.md.
Migrate 139 .py files (377 occurrences) to use the constants:
- 135 files via direct constant import (`from utils.roles.mapping
import ROLE_FILE_*`), with the symbol used in pathlib chains, in
os.path.join / Path.joinpath calls, in f-string interpolation, and
in glob patterns.
- 4 files via `# nocheck: role-file-spot` where the literal lives in
argparse help text or in an allowlist tuple meant to declare the
literal path.
Adjust utils/roles/dependency_resolver.py to use lazy-formatting
positional args in the logger.exception call (G003-clean) instead of
string concatenation against the constant.
Drop a redundant `if cond: return True; return False` guard in the
new test for `return cond` (SIM103-clean).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
