`roles/` set as inline code reads as a literal path, yet the directory
holds an index page the build resolves. Writing it as a link costs nothing
and gives the reader the page; leaving it as code hides it, which is how
this branch degraded six working references in one pass.
The lint strips markdown links from a line before scanning it, or it would
report the code span inside a finished link, and it skips fenced blocks. It
reads the index filenames from the same place the link lint does.
Two false-positive classes came out of the first run and shaped the rule. A
span of nothing but dots and slashes is a path word, not a path: '/' names a
filesystem in a table of mount points and '../' names the traversal an
argument must not perform. An absolute span is skipped because this
repository writes Keycloak group paths that way, and /roles/web-app-bluesky
is a group, not a folder.
35 spans across 12 files become links, and the documentation build resolves
every one of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three shapes of the same failure, taken from the build's own message list
rather than from a reading pass.
Eight unescaped inline characters in prose: a '*', '**', backtick or '|'
opens markup that never closes. 'roles/*/vars/main.yml', the regex
'^(web-|svc-).*', 'scripts/system/tls/trust/*.sh'. In ansible_log an inline
literal ended on a space, which reStructuredText forbids; the space carried
meaning, so it moved into the prose.
Thirteen example blocks after a single ':', which makes the indented block
a block quote instead of a literal one.
Three bullet lists with no blank line to the paragraph above, which
docutils reads as a continuation of that paragraph.
Two markdown files carried a '---' where reStructuredText allows no
transition: directly after a heading, and at the end of a document.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sys-svc-compose already emits a transient build.extra_hosts override for every
service that has a build, driven by INFINITO_CACHE_PACKAGE_FRONTEND_IP and
gated on the frontend CA being present. Its host list held the three apt hosts
and not argos-net.com, so 3af579449b rebuilt the same thing inside the
libretranslate role.
test_no_ansible_env_lookups found it: a role may not pull a default.env key
with lookup('env', ...), because default.env is the dev and CI contract rather
than something deployment-agnostic Ansible should reach into. Adding the host
to the shared list removes the rule violation and the duplicate in one move,
and with it LIBRETRANSLATE_CACHE_FRONTEND_IP, LIBRETRANSLATE_BUILD_HOSTS and
the container_extra_hosts call.
The URL lint probes every URL the repository mentions and a bare GET on
argos-net.com answers 404, measured, while the model paths Nexus prefixes
underneath /v1/ answer 206. The base URL is what Nexus needs, so the line is
marked the way the deb.debian.org redirector two lines above it already is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
argos-net.com hosts the .argosmodel files the LibreTranslate image pulls
at build time, 70 to 150 MB each and around 100 of them, so every rebuild
refetched several gigabytes from upstream. Route the host through the
existing Nexus raw proxy and the nginx MITM frontend that already covers
pypi, npm, alpine and the GitHub raw hosts.
The argos server block overrides the global proxy_read_timeout. Nexus
emits no byte of an object until it has fetched and stored the whole
thing, so the wait for the response header is the wait for the entire
cold fetch, which outran the global 300s once eight download lanes
competed for the upstream.
INFINITO_CACHE_PACKAGE_FRONTEND_CONF replaces the hardcoded relative bind
for the frontend's upstream map. One cache stack serves every checkout on
the host, so the map follows the primary checkout by default;
INFINITO_CACHE_CONF_SOURCE=worktree points it at the current one to test a
change before it merges. dotenv-force threads that variable through its
env -i allowlist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
server() now resolves a translation endpoint in three steps: use the running
service, rebuild when the host has a GPU but no accelerated deploy left its
stamp, otherwise deploy. A plain run therefore costs nothing, and the rebuild
fires once, on the move from a CPU-built container to an accelerated one.
deploy() brings the runner up, waits for State.Running and deploys the role
inside it. The exec steps address `docker compose exec i18n` directly:
cli.administration.deploy.development exec hardcodes the `infinito` service
(compose.py:261), so every deploy through it silently ran in the main
container, where mailu already holds port 25 and postfix cannot start.
check=True stops a failing step instead of carrying on against the wrong
target.
The deploy router writes its pid to build/deploy.pid, which survives the exec
into the subscripts, and deploying() refuses a translation while that process
lives. The running deploy is left untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The router now prints one line before it routes:
=== compose-deploy: mode=reinstall apps=svc-ai-litellm bundles= variant= \
full_cycle=false purge=false disable= playwright_keep=
Rewriting a deploy command is where its other arguments get lost. Switching
apps= from svc-ai-agent-broker to svc-ai-litellm dropped
disable="matomo,dashboard,prometheus,email,css" and
INFINITO_PLAYWRIGHT_KEEP=true in the same edit, after three deploys that had
carried both, and the run went on to deploy matomo and dashboard. That matters
beyond the wasted minutes: test-e2e-cli runs in the destructor stage, after
every role, so the providers are paid for before the CLI test the deploy exists
for even starts.
main.sh is the single entry every compose-deploy passes through, so the line
cannot be skipped and does not depend on remembering to look. It reads the
knobs and exports nothing; 8e293bd921 is on record for what happens when these
same variables reach places that persist them.
compose.md loses the checklist this first carried. The rules were already
stated twice in that file and a third statement is not what was missing; the
misses were not caused by not knowing them. It keeps the two facts that carry
information: why dropping disable= delays the evidence rather than merely
costing resources, and that full_cycle=true is the expensive form rather than
the complete one, since it adds pass 2, credential rotation and the recovery
drill while the CLI tests gate on RUNTIME instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
load.sh treats every already-set INFINITO_* as caller intent and restores it
after sourcing .env. In a worktree those values come from a different checkout,
so make requested the primary instance's subnet and docker refused the network
with "invalid pool request: Pool overlaps with other one on this address
space". Five retries, then the deploy died.
INFINITO_INSTANCE names the slot a value belongs to. When the environment
carries a different one than this checkout's .env, the whole set is foreign and
is dropped before .env loads. A caller in the same slot still wins, which eight
scripts under scripts/ rely on, and an unreadable .env yields no slot and
leaves the previous behaviour untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A pull request against main extracts the core and docs domains, fans out
over the languages LibreTranslate supports, and opens one translation PR
per language against the branch that introduced the strings. Only the
entries a catalog leaves empty or fuzzy reach the translator.
The job matrix and the translator read the same list through
translatable(), so a code cannot enter the matrix and then be refused by
the run it starts.
The pipeline stays off until CI_ENABLE_AUTO_TRANSLATIONS is set, and
pull requests from a fork are skipped because the app token cannot push
to their head branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
git diff reports no change for a path whose only change is an untracked
file, while open/pr.sh stages that path with `git add` and commits it.
Every gated job therefore skipped its PR step whenever an updater added
a file instead of editing one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Docker updater watches image plus version, the repository updater
repository plus ref. Twelve pins fell outside both - an app_version, a
release, a tag in a registry neither speaks - so nothing ever told anyone
they had gone stale.
Such a pin now names its upstream next to itself in an `update:` block:
git_tags, registry_tags, npm, http_regex or a role-local script, with
`match` selecting a tag flavour and `strip` removing a prefix the pin does
not carry. utils/update/source.py resolves them beside the two existing
updaters, `python -m cli.contributing.update.source` bumps them, the
update-version-sources job opens one PR per role, and
tests/external/update/source warns while a bump is pending.
Two lints hold the rule: every semver pin is watched by one of the three
updaters or opted out with a reason, and a semver only ever sits behind
`version`, `<what>_version` or `ref` - which renames moodle's release to
app_version and simpleicons' local_version to local_tag, the latter being
the tag of a locally built image rather than an upstream version. A
declaration naming a key the entity does not carry, or a type the resolver
does not know, fails the lint instead of resolving nothing in silence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
769163ca7 computed DASHBOARD_URL and MATOMO_URL from INFINITO_DOMAIN
right after load_repo_env in common.sh. The orchestrator sources
common.sh inside the bare runtime image before Python is installed,
where load_repo_env returns without loading the env, so the guard
aborted the debian, ubuntu and fedora workspace jobs before their first
step.
common.sh now offers stack_url, which reads INFINITO_DOMAIN when it is
called. The compose steps resolve their URLs in plain assignments after
their own source of common.sh, so a missing domain still aborts the step
instead of handing an empty URL to an assertion that expects 000.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The recovery added in 2f5f52bd1 ran 'npx puppeteer browsers clear',
which asks yes/No on stdin and deletes nothing without a terminal. A
cut-off download therefore still left chrome-headless-shell without its
executable, and puppeteer kept refusing every later install of that
build, so lint-mermaid failed on every run.
A failed provisioning now removes the chrome-headless-shell cache folder
itself, the step puppeteer's own error message asks for, and retries.
Against a stack whose cache still held the broken folder, all 287
Markdown files rendered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The chrome-headless-shell download had no time limit, so a stalled
network kept lint-mermaid, and with it the whole quality gate, waiting
for 148 minutes. The install now runs under a 900 second timeout.
An interrupted download leaves the browser folder without its
executable, and puppeteer then refuses every later install of that
build. A failed provisioning therefore clears the puppeteer browser
cache once and tries again before falling back to the existing warning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Building every architecture in one job through QEMU took the image job
from under ten minutes to over two hours: each arm64 layer after
'COPY .' reran emulated on every commit, on a four-vCPU runner.
Each architecture builds on its own runner again, but push.sh now
pushes by digest only and records the digest per distro and
architecture, so no per-architecture tag exists. A merge job collects
every architecture's digests and manifest_all.sh publishes the plain
tag as one manifest list, refusing a distro that lacks a digest for any
architecture in the pool. The build cache is per architecture again so
the two jobs do not overwrite each other's cache. The binfmt mirror
entry goes with the emulation, and ADR 0002 records the measured
reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An image bump in a provider redeployed every variant of every consumer,
including the variants that switch that provider off and therefore never
deploy the bumped image at all. The diff-derived whitelist now narrows each
affected role to the variants whose own round closure still reaches a seed,
emitted as `role#0,2` selection tokens. That closure is walked on the same
variant-merged services maps the matrix planner builds its rounds from, so the
narrowing follows a parent's variant choice across every generation of the
chain. It covers every auto-derived whitelist: the weekly update PRs,
Dependabot, and any other diff-driven run.
The narrowing never shrinks the matrix on partial information. A seed role
keeps every variant, because its own files changed. A role reached only
through a run_after edge keeps every variant too, because the round closure
does not follow that edge and a role whose hits are empty is therefore
unnarrowed rather than dropped.
A pinned token naming a role the discovery query does not return now warns
instead of aborting the run. The diff-derived whitelist legitimately names
roles the lifecycle envelope filters out, and once it pins variants such a
token would abort every chunk; role names are still checked against roles/
before the run starts, so a typo is caught one layer earlier.
The round's variant merge moves out of the deploy inventory package into
utils/roles/applications/variants.py. The CI selection has to build the same
maps, and importing the deploy package for them pulls in a module that reads
INFINITO_SRC_DIR at import time, which the runner host does not set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
default.env now sets INFINITO_DOMAIN to main.infinito.test, and CI
deploys under it, but the workspace suite still asserted
https://dashboard.infinito.test. curl could not resolve that host, so
the debian and fedora compose tracks failed after every deploy had
passed. Both URLs now follow INFINITO_DOMAIN once load_repo_env has
loaded it, and an unset domain aborts instead of probing a guess.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CI images were built by one job per architecture, each pushing a
<tag>-<arch> tag, and a third job merged them into the plain tag. One
job on an amd64 runner now registers QEMU through a pinned
tonistiigi/binfmt and hands the whole architecture pool to a single
buildx --platform build, so each distro gets exactly one tag, a
manifest list from the start, and one build cache.
binfmt installs with 'all' because that list leaves out the native
architecture; naming amd64 explicitly on an amd64 host would route the
host's own binaries through QEMU. The registry assertion moves into
the build job, manifest_all.sh goes, and the cleanup stops protecting
per-architecture children that no longer exist. ADR 0002 records the
changed decision.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The filesystem install ran in the RUN after 'COPY .', so any commit
rebuilt the zfs userland: the AUR build on Arch and, on arm64, the
source RPM rebuild on Fedora and CentOS, all of it under emulation now
that one job builds every architecture. It now runs in its own layer
right after default.env and scripts/install are copied, which only
those two paths invalidate. apt-mirrors.sh moves with it and still runs
first.
Running before the Python step, filesystem.sh no longer finds a system
that step already upgraded, so its pacman call does the full -Syu
instead of a partial -Sy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zfsonlinux serves no aarch64 metadata for either Fedora or EL. dnf5
skipped the missing repository and fell through to the source rebuild,
but dnf4 on CentOS Stream aborted every transaction that loaded it, so
the arm64 centos image died at the repoquery.
The repoquery now skips an unavailable zfs repository, and the source
path disables it on every call and removes zfs-release right after the
source RPM is downloaded. The pieces that differ between dnf
generations follow the installed one: dnf-plugins-core or dnf5-plugins,
'download --source' (dnf5 keeps it as an alias of --srpm), and 'mark
remove' or 'mark dependency'. EL enables CRB for builddep, where the
devel dependencies live, and the package snapshot moves ahead of the
repoquery so the plugins count as build tooling too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The arm64 Fedora image failed because zfsonlinux publishes Fedora RPMs
for x86_64 only, so repoquery found no zfs to pin the dummy kmod to.
An empty repoquery now downloads the source RPM of the same release,
takes the version from it, rebuilds it and installs the runtime
packages only.
Everything installed from rpm-build onwards is marked as a dependency
and autoremoved afterwards. A plain dnf remove refuses because builddep
pulls the protected systemd, and taking the snapshot after rpm-build
left gcc held by redhat-rpm-config's annobin requirement.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every stack answered to *.infinito.test, and the host resolver pointed that
zone at 127.0.0.1, so a worktree's stack on its own bind IP could not be opened
in a browser while the primary stack existed.
INFINITO_DOMAIN is now derived per checkout: main.infinito.test for the
primary one, and for a linked worktree its branch, lowercased with everything
outside a-z0-9_- turned into "-", below INFINITO_DNS_DOMAIN
(feature/store-i18n -> feature-store-i18n.infinito.test). The branch is read
from the worktree's gitdir HEAD instead of running git, because git refuses the
mounted checkout inside the stack container. GitHub and act runs keep the
static default, custom.env still wins.
The handler overrides the caller's value on purpose: inside the stack
container that value is the previous .env, which make setup rewrote on every
deploy, so the old domain kept feeding itself back into the next compose up.
make network-dns-setup routes each stack's domain to the INFINITO_BIND_IP in
that checkout's .env, with a local= line so AAAA answers NODATA instead of the
primary stack's ::1. It skips domains outside the base zone, which a stale
.env still carries, and fails when two checkouts claim one domain.
worktree-up prints the domain and the follow-up command.
Existing checkouts switch after make dotenv-force and a redeploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every distribution built FROM `base-debian`, so the arch, ubuntu, fedora and
centos images were all asking the registry for the debian parent.
push.sh exports INFINITO_DISTRO from MATRIX_DISTRO and then sources the env
layer, which is correct as far as it goes: load.sh snapshots caller-set values
and restores them, so the distro survives. INFINITO_PARENT_IMAGE is not set by
the caller, though, and arrives already materialised in .env, rendered for
whichever distro generated that file. Setting the distro in a loop therefore
never reaches the reference derived from it.
The new resolver renders it through utils.distros.base_image() rather than
composing the string, because test_distros_spot rejects an image reference
built anywhere but the SPOT. It sits beside local.sh, ci.sh and service_ref.sh,
which resolve the other image names the same way.
Verified: arch, fedora and ubuntu now resolve to three different parents where
they previously resolved to one. shellcheck is clean and test_distros_spot
passes, which is what proves the resolver stays inside the SPOT.
Not verified: the build step that calls it. push.sh runs only in CI, and the
base images are not published yet, so a real run still stops at the pull before
it reaches this line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two install workflows differed in a name, one env key and the script they
ran, and the orchestrator wired each separately. They become one file.
The pkgmgr half goes with them. Installing through the package manager is no
longer the plan, and the images have already moved: since the switch to
base-images every container is built FROM base-<distro>, so that test was the
last thing still pulling a pkgmgr image. The pkgmgr CLI itself is untouched;
roles keep calling it.
The mermaid graph in .github/workflows/README.md named both files as nodes and
as edges, which test_workflow_references caught after the rename. The workflow
table in the docs follows.
Verified: actionlint accepts every workflow, and 263 tests pass across
tests/lint/repository, tests/lint/filesystem and tests/integration/meta/workflows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The axis was complete in CI and unusable from a workstation. Four separate
gaps, each invisible on a real runner, and each found by running the swarm loop
against it:
`INFINITO_ARCHITECTURES` as an exported variable did nothing, because the
workflow reads its pool from `inputs.architectures`. Passing the input alone did
not help either: ACT_ENV sets INFINITO_APP_DISCOVERY_RUNNER=host, so the
discovery that builds the matrix runs outside the act container, where a
workflow-level `env:` does not reach. This is why the distro axis is handed over
twice, as `distros=` and as `INFINITO_DISTROS`, and SWARM_ARCH now does the same
for the architecture.
The third gap is the sharpest: act does not resolve `${{ matrix.* }}` inside a
job-level `env:` block, so `INFINITO_ARCHITECTURE` reached the assertion empty
every time. Earlier runs only appeared to work because a stale local `.env`
supplied a value, and the assertion was therefore checking a leftover rather
than the row. It surfaced the moment that file was regenerated.
Fourth, act had no mapping for `ubuntu-24.04-arm`, the label RUNNERS
introduced, and skipped the job with a hint instead of failing. It maps to the
same runner image: locally the architecture is delivered by emulation, and the
runner's own hardware is not the promise being made.
`make binfmt` registers that emulation. The handler already existed for
compose-up; it gains a subcommand so the registration is reachable without a
privileged docker invocation, which is what an agent would otherwise have to
ask for every time. The emulation does not survive a swarm run, because act
starts further containers whose systemd clears the shared binfmt registry, so
this is needed often.
Without `arch=`, SWARM_ARCH resolves to the host's own architecture rather than
staying empty, which would now abort every unpinned local swarm run at the
assertion.
Verified: `make binfmt arch=s390x` takes a platform from `exec format error` to
`s390x`; `make -n swarm-zombie arch=arm64` carries arm64 through all three
levers and defaults to amd64 without it; shellcheck passes; a live run reached
`"architecture":"arm64"` with the arm64 glyph in its job title.
Not verified: that the runner-label mapping carries a swarm deploy to the end.
That run was stopped so a parallel agent could use the shared stack.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
INFINITO_ARCHITECTURE means two different things. In CI it is the runner's
hardware, delivered by `runs-on`. Locally it is the platform app containers are
created for, delivered by qemu. The check asserted the first everywhere, so a
local run that honoured the assignment by emulation was reported as a mismatch,
with an error blaming RUNNERS where nothing was wrong.
Each context is now checked against its own promise. On a real runner the
uname must match, unchanged. Anywhere else the host must be able to execute the
assigned platform, which is verified by running a container of it rather than
by reading /proc/sys/fs/binfmt_misc, since that directory reads as empty from a
sandboxed shell whether or not a handler is registered.
Nothing is skipped and nothing passes quietly. A host that cannot deliver the
assigned architecture fails here, in the first second of the job, and names the
two commands that fix it. Five arm64 deploys died instead at
`exec /bin/sh: exec format error` deep inside an Ansible role, each after
fifteen to twenty minutes of setup.
This replaces the `exit 0` under act that an earlier version of this change
introduced. That version swallowed exactly what this check exists to catch: a
row assigned arm64 ran entirely on amd64 and would have reported green.
Verified, all four branches: amd64 native passes; arm64 with emulation present
passes as emulated; arm64 without it exits 1 with the remedy; a real-CI
mismatch exits 1 with the RUNNERS hint. shellcheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
run.sh drives every suite with -n auto and no scope, so xdist hands out single
test METHODS. Each worker is its own process, so setUpClass runs once per worker
the class happens to be scattered over, and an expensive fixture is paid as many
times as the scheduler split its methods.
The integration report shows the shape without interpretation:
TestMcpEnvRender's nine methods came back as six at 277-287s and three at
0.003s. The fast three landed on workers that had already paid. That is 1776 of
the suite's 2282 CPU-seconds spent computing one identical result six times,
twenty Ansible templar renders of the MCP providers each time.
--dist loadscope groups by class, so a class occupies one worker and its
fixture runs once.
1a908a6e5b introduced -n auto and hoisted fixtures in the same commit, so the
hoisting was undercut from the start: work moved from per-method to per-class,
and the scheduler it arrived with promptly charged it per worker instead. This
finishes that commit rather than correcting it.
Measured against the same tree, four suites, before and after:
integration 296.83s -> 219.32s
lint 105.64s -> 84.27s
unit 65.74s -> 53.02s
external 75.69s -> 71.99s
Phase 1's critical path drops from 4m57s to 3m40s. That unit and lint move too
says the pattern was not unique to the MCP class.
Coverage is unchanged, which is the point of the change rather than a side note.
All four suites report identical counts: unit 5857 passed / 9 skipped / 2301
subtests, external 19 / 2, lint 521 / 1 / 374, integration 328 / 2 / 1562.
A second effect is worth naming because it cuts the right way. Several sweeps
keep their non-empty guard in a separate method, e.g. test_fragments.py:139
proving the corpus is not empty before the sweep asserts over it. Under
--dist load those two ran in different processes with separate caches, so the
guard proved nothing about the process the sweep ran in. loadscope puts them
back together.
Residual: loadscope trades balance for locality, and here one class sets the
floor. TestMcpEnvRender holds a worker for ~250s on its own, so the suite cannot
go below that until the ~12s per render is addressed. That cost is the render
guard in utils/cache/applications.py:415 returning the merged map without
caching it, which is production config resolution and deliberately left alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lint fan-out listed lint-php twice, once before and once after
lint-playwright. Two commits added it independently: 82218087c1 for the PHP and
Ruby syntax checks, 6fee7ca136 for the per-language unit suites, and the second
did not see the first because lint-playwright had since been inserted between
them.
The wasted slot was the harmless half. parallel.sh keys target_start,
target_dur, target_rc and target_log by target NAME, so a repeat overwrites the
first run's log path and exit code, and the final pass reads each name once.
Two consequences: drain_one prints the second run's log twice and the first
run's output never appears, and a failing first invocation is reported green as
soon as the second one passes. A duplicate therefore turns a target into a blind
spot rather than into a slow one.
The gate showed the symptom plainly, two lint-php rows with an identical
0m 03s, which reads as a rendering quirk rather than as one target run twice.
Both halves are fixed: the list loses the repeat, and parallel.sh refuses a
duplicate before it loads the environment or starts a single make, naming the
offender and why a repeat cannot be tolerated. The two callers are Makefile:440
and Makefile:822; the test list has no duplicates, so the guard is inert there.
Negative control: `parallel.sh lint-php lint-python lint-php` exits 2 with
"duplicate target(s): lint-php" and launches nothing. The full quality-high run
afterwards dispatches 15 lint targets with lint-php once and stays green.
Residual: the guard covers scripts/make/parallel.sh only. scripts/tests/dns/
parallel.sh is a separate script with its own argument handling and was not
examined.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both architectures were claimed to work and only one was ever exercised. The
deploy matrix now carries an architecture the way it carries mode, tor, distro
and filesystem, so every run proves both.
A job cannot ask for an architecture the way it asks for a distro, because the
architecture is the machine rather than something the job installs on it. The
matrix therefore carries a runner label and `runs-on` reads it, and an
architecture without a label in RUNNERS is a build error rather than a row that
lands on the wrong hardware and still reports green. assert_architecture.sh
checks the runner it actually got, because a label that stops resolving to
arm64 would otherwise go unnoticed forever.
Assignment is rotation, not chance: two architectures against five distros and
fifteen distro/filesystem positions are coprime enough that every combination
comes round. A role narrows the pool through `architectures` in its
meta/services.yml; without one it takes whatever the rotation hands it. An
empty intersection aborts instead of silently removing the role from the run.
Images are built per architecture and pushed under an architecture-suffixed
tag, then merged into the plain tag by manifest_all.sh; pushing the plain tag
from both builds would let the last one win and send half the rows to the wrong
platform. cleanup.sh now protects the `ci-<sha>-<architecture>` children of a
protected tag, because the plain tag is an index over them and deleting a child
leaves it unpullable on that platform.
label.py is split out of axes.py, which the 500-line limit no longer fits.
Writing a job title and reading one back belong together: a retrigger that
recovers an axis wrongly replays a different deploy than the one that failed,
and a drifting artifact name makes the failure reporter download nothing. The
leading `.*` in LABEL_RE is greedy on purpose so it anchors on the last mode
glyph; a reusable-workflow caller path carries a glyph of its own, and
test_selection.py guards the case.
Verified: 313 unit tests and 48 subtests across utils/github/variant,
utils/roles and cli/meta/ci. Locally, a deploy pinned to arm64 reaches ok=1444
and its image builds log aarch64-linux-gnu paths.
Not verified here: actionlint over the seven changed workflows, and a full
`make test`. The matrix doubles the row count, so runtime and runner
availability change for every CI run from here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The container this project builds installs its own python, ansible and docker
CLI, so the package manager riding along in the image it built FROM was never
used. That layer now lives in its own repository, which publishes one
multi-platform image per distribution, and meta/distros.yml points at it:
`base` replaces `pkgmgr_virgin`, and utils.distros grows base_image() where
pkgmgr_virgin_image() stood. The pkgmgr image itself stays, because the
`pkgmgr install` test still boots it.
Three references carried the pkgmgr-era tag `stable`, which base-images does
not publish: it pushes `latest` and the major release, and for a rolling
distribution only `latest`, so `latest` is the one value that exists for all
five. INFINITO_PARENT_IMAGE_TAG and the two install tests move to it.
The guard that keeps image references inside the SPOT only matched
`pkgmgr-[{$]`. Renaming the family made it blind to exactly half its surface,
which is how scripts/tests/install/make.sh kept `base-${d}:stable` while
pkgmgr.sh was caught. It now matches `base-` too.
INFINITO_ARCHITECTURE and INFINITO_ARCHITECTURES appear here because
default.env is shared with the architecture axis that follows. They are empty
and inert until it lands; the alternative was a commit whose build fails by
construction, with `base-` declared and `stable` still requested.
From here nothing builds until base-images has published: `docker pull` of the
parent answers `manifest unknown`, and every make target that depends on
build-dependency fails there.
Verified: tests/lint/repository/test_distros_spot.py passes, and the widened
pattern is shown to catch the line the old one missed rather than merely not
breaking.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
71888d86af moved twelve workflow files behind trigger prefixes and updated the
references GitHub validates. Three mentions in prose kept the old names: an echo
line, a file header and a lint's failure message. The same commit already fixed
the REST dispatch URL two lines below the echo it missed.
roles/svc-runner/files/test/external.sh:36 test-runner-smoke.yml
scripts/github/runner/cleanup_docker.sh:4 images-build-ci.yml
tests/lint/repository/test_distros_spot.py:116 test-workspace.yml
None of them executes; the workflow-reference lint deliberately does not reach
them, because outside .github/workflows a bare *.yml collides with services.yml,
compose.yml and variants.yml. A repo-wide grep over all eleven legacy names now
returns nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image runs this module from a layer that holds three files of utils:
__init__.py, install/__init__.py and install/collections.py, the slice
ef45d69dc3 cut so the source tree cannot invalidate the collection install.
67f5398985 established that the module therefore stays on the standard
library, and d4ae5cf9f0 broke it 15 minutes later by routing the reads
through utils.cache.files to satisfy the cache-read lint. The import raises
ModuleNotFoundError: No module named 'utils.cache' in that layer, which no
import inside a full checkout can show, because this repository installs
utils into site-packages.
Nothing turned red. The caller reads only the exit code, and the module's
contract is exit 1 with the missing collections on stdout, so a crash - exit
1, nothing on stdout - is indistinguishable from work to do. Every image
build since has reinstalled every collection with --force-with-deps, which
is the cost the check exists to avoid. The deploy job the traceback was
found in finished green.
The reads go back to Path.read_text with a cache-read marker each. The cache
was never earning its place here: read_text is an lru_cache keyed on the
path, and one run reads the requirements file once and one MANIFEST.json per
collection, eight distinct paths, none of them twice. Copying utils/cache
into the slice instead would work - the closure is three stdlib-only files -
but it binds a package with 51 commits in the last twelve months to the
layer that installs ansible, in front of a cache with no possible hit.
The caller now exits 3 when a non-zero status arrives with an empty stdout,
a state the contract cannot produce, so the next crash stops the install
instead of quietly reinstalling everything. all.sh runs under set -e and
touches the stamp only after ansible.sh returns, so the failure also leaves
the stamp stale rather than recording an install that did not happen.
The probe that would have caught this builds the slice in a tempdir from the
Dockerfile's own COPY lines and runs the module inside it. It has to pass
python -S: without it the interpreter resolves every submodule from the
installed checkout however the slice is built, and the test passes on a
module that cannot import in the image. Verified by injecting the import
back into the tempdir copy, which reproduces the traceback verbatim.
Delta measured by loading the committed module and this one side by side:
identical on both requirements files, against the real collections directory
and an empty one, on a missing file, on non-UTF-8 input and on a corrupt
MANIFEST.json. The one behavioural difference is that a file changed within
a process is now read fresh rather than from the memo.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A codename carries no ordering, so 'bookworm' and 'trixie' are two opaque
strings and no updater can tell that one supersedes the other. A pin that
names its Debian release that way is frozen on that release for as long as
it stands, and the freeze is invisible because the tag keeps resolving.
Numeric tags are comparable and stay inside the freshness pipeline, so the
rule is worth holding for the whole repository rather than fixing case by
case.
The check reads two shapes, because a codename reaches the tree in two
spellings. A 'name:tag' reference is recognised by the absence of a space
after the colon, which is what separates 'debian:bookworm' from the YAML
'image: python' and covers Dockerfile FROM lines and heredocs without a
second parser. A 'version:' value is read as a key, because '3.11-bookworm'
carries no colon at all.
False positives are kept out by two independent gates: a word-boundary
prefilter, so 'residual' and 'inside' are not read as 'sid', and a split of
the tag on '-', '.' and '_' that requires a whole component to be a
codename. That is what lets 'cache buster', '.stretched-link',
'fa-bullseye' and a tags list containing 'stretch' through. Every codename
occurrence in the repository was checked against the matcher by hand, and
the check was negative-controlled in both shapes, including an untracked
file.
Four references had to move for the rule to hold. meta/distros.yml takes
'debian:13' rather than '13.6': at depth 1 the updater can raise it across
a major, while depth 2 pins it inside the 13.x line forever. The whole
'make install' path was run in a debian:13 container to back that: all
twenty-four Depends and Recommends resolve on trixie, the docker repository
bootstrap derives the right suite and installs docker-ce-cli, and the
package builds and installs.
The workspace DiD probe moves to 'debian:13-slim'. An earlier version of
this change carried a suppression claiming the probe had to follow the
suite the apt proxy mirrors. That was wrong twice over: it cited a function
that has nothing to do with the probe, and the coupling does not exist,
because the cache frontend rewrites by path prefix and the suite never
enters a routing decision. Replaying the probe verbatim against
debian:13-slim fetches all three trixie suites through the
bookworm-configured proxies, so the suppression is gone rather than
reworded.
The rule ships with an escape hatch and no uses of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Penpot job 105068912535 and MediaWiki job 105068912565 both died on one lab node whose dnsmasq kept its :53 socket bound and rejected every query with 'ignoring query from non-local network', once for a container address and once for 127.0.0.1. Rejecting loopback as non-local only makes sense if --local-service is active and dnsmasq's own interface list went stale, but the bundle records neither the flag nor where it would come from.
Capture the process command line, /etc/default/dnsmasq, the unit including its drop-ins, and the config that actually landed on disk. Addresses and listening sockets were already collected and are not duplicated.
Read-only. Every call falls back to a text line when the file or the unit is absent, so a node without dnsmasq reports that rather than failing the collector.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
For a fork PR the build step is skipped -- the images come from the
privileged run -- so the job went green in 29 seconds while that run was
still pushing. Everything gated on it inherited the lie: the DNS test
pulled arch 3m30s too early on one run and 53s too early on the next,
both times 'manifest unknown', and the four deploy chunks were spared
only by starting later.
The job now polls until every distro image for its tag is in the
registry. It runs unconditionally: where the job built the images itself
the first poll passes, and a push that silently failed is caught too.
The poll drives scripts/meta/resolve/missing/ci_images.sh, which already
owned that check for the release workflow, so the ref shape is not
written a fourth time. A resolver that cannot run aborts with 2 rather
than reading as 'not yet pushed' and hiding behind the full budget.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The job could sit silent for its full 30-minute budget and only then
print five groups, so a hanging distro was indistinguishable from a slow
one. Each container's output is stamped with [distro][HH:MM:SS] as it
arrives, and the per-distro verdict is printed when that distro finishes
rather than at the end.
A pipeline hands $! the pid of tee rather than of the test, which would
have turned every DNS failure into a green job, so each run goes through
a subshell under pipefail. The ::error:: annotation, the per-distro logs
and the container cleanup are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Computing the distro list no longer justifies a runner of its own: it is
two steps, and the job that consumes them already checks the repository
out. The workflow now occupies one runner instead of seven.
The step also went silent for minutes at a time, printing five pids and
then nothing until every build had finished. Each build's output is piped
through a [distro][HH:MM:SS] stamp as it arrives, so the log shows what is
happening while it happens; tee still files the per-distro logs for the
artifact, and the replay block at the end is gone as duplication.
A pipeline hands $! the pid of tee rather than of the build, which would
have swallowed every failure, so each build runs in a subshell under
pipefail and wait reads that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gate flagged six of my own changes. INFINITO_ANSIBLE_COLLECTIONS_SOURCE
reached default.env without a handler, so the generator dropped it
silently; ansible.sh then carried a shell-side default for the same key,
a second source that drifts. Both are now one declaration, read as
required, so a missing key fails loudly instead of falling back.
File reads route through the session cache except where the same run
rewrites the file, which is the documented opt-out and now says so per
call. The updater reads yaml through utils.cache.yaml rather than calling
safe_load itself.
restore_collection_history.sh was committed without its executable bit,
and the new workflow's name was 31 characters where the dispatch menu
truncates at 25; the explanatory wording lives in run-name instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The parent directory was created and chowned on every run, before the
script looked at whether the virtualenv exists. Where the parent is not
writable that reached for sudo, which a sandboxed or no-new-privileges
container refuses -- so an install that had nothing to do failed with
'The "no new privileges" flag is set'.
Nothing needs preparing once the interpreter is there, so the check moves
ahead of the preparation and returns.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The matrix spent five runners on work that is almost entirely network
wait: the collection install measured 0% CPU for its whole duration, and
the parent-image pull, the package managers and the push are no
different. One runner carries all five concurrently and frees the other
four.
Losing the matrix loses fail-fast: false with it, so push_all.sh awaits
every build and reports each one by name before exiting non-zero. A
broken distro no longer hides the other four, which is what made today's
triage find three causes instead of one.
Starts are spread by fibonacci seconds because each build opens with a
parent-image pull of several hundred MB. Per-distro logs are grouped in
the job output and uploaded as an artifact, and the pressure report now
carries free -h: memory is the one resource this consolidation contends
for, and an OOM kill would otherwise read as a build error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ignore-error traded a loud fault for a silent one. The cache writes to
the same GHCR repository as the image and under the same packages:write
token, so every shared failure mode -- permissions, outage, rate limit --
already fails --push. What remains is buildkit's own cache manifest type,
which a registry can reject while accepting an ordinary image push.
That case is exactly the one worth hearing about. Swallowed, it leaves CI
green while the cache never fills and every build silently pays the full
855-899s again; raised, the first run answers it once and for good.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Which of the two sources is faster is not a constant. galaxy.ansible.com
degraded from 207s to 687s over one day and then served 502s outright,
while the pinned git fallback carried the same install in 31s; a week
later the order may be the other way round. The daily job times both and
swaps the default only when the preferred one's median reaches three
times the other's over twelve samples.
A failed install is recorded as None and ranks worse than any duration,
so an outage counts instead of dropping out of the statistic. Below a
full window nothing changes, which keeps a single bad day from flipping
the default.
History lives in an artifact rather than the Actions cache: a schedule
run resolves to the default branch and has read-only cache access there,
so it could not carry the window forward.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pinning the collections froze them, and nothing thaws a pin: dependabot
reads pyproject.toml, the docker updater reads image tags, and neither
sees a version inside requirements/. The new job resolves upstream
releases with git ls-remote --tags and opens a PR the same way the pip,
image and repository-ref updaters do.
Both requirement files are bumped together, because the git fallback has
to install what the Galaxy file declares; moving only one would break
that. A collection whose two files disagree, or whose version is not a
semver, is left alone -- that covers the branch ref and the commit pin,
which state an intent a bumper must not guess.
The job calls pr.sh rather than role_prs.sh: the latter splits per
sub-directory and would open no PR at all for flat paths like
requirements/*.yml.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
COPY . preceded every expensive RUN, so each commit rebuilt the whole
chain and ansible-galaxy ran on every build of every distro. The
collection install now sits in its own layer fed only by default.env,
scripts/install, the dev-python shell scripts, requirements/ and the
three files the version check imports. Everything that needs the full
tree, the project's own distro package included, stays behind COPY ..
Measured on a role-only change: the collection layer stays CACHED and
the build drops to 3m10s against a CI baseline of 855-899s. An
unchanged rebuild is 15 cached layers in 1.5s.
The gha cache imported and exported nothing in a plain run step, so the
export moves to the registry the job already authenticates against.
ignore-error keeps a cache write failure from taking the build with it,
which it otherwise does: proven against an unreachable registry, where
the build fails hard without it and exits 0 with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The requirements named seven collections without versions, so every
install re-resolved them against galaxy.ansible.com and --force-with-deps
redownloaded all 37 MB even when nothing had changed. Exact pins make the
installed set checkable, and the check short-circuits the whole roundtrip:
make install-ansible drops from 8m24s to 0.077s, and a Galaxy outage stops
mattering once the pinned set is present.
The git fallback is pinned to the same versions so it installs what the
Galaxy file declares rather than whatever the branches currently hold.
kewlfft.aur pins a commit because its tags carry no galaxy.yml and master
has moved past the released version.
The lint installer kept a second, weaker truth about the same directory,
guessing from directory presence for three hardcoded collections; it now
shares the version-exact check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-net-tor aborts at roles/svc-net-tor/tasks/router.yml:2 when networks.internet.dns is empty, and rightly so: dnsmasq runs with no-resolv, so an empty upstream leaves the node answering .onion and nothing else. That value is set in exactly two places, both under inventories/development/, and the documented production path reaches neither - the provisioner writes no networks key at all. A reader following the role README verbatim therefore cannot deploy a Tor node.
CI has been proving this since 2026-08-31 without anyone seeing it: the README replay rides on the role's smallest variant and is skipped when the deploy is red, so the only row that ever reached it is one where svc-net-tor itself went green. That happened first on Ubuntu, where the local-service defect abb6f37297 fixes never fired.
The template now emits the DNS assignment, the -e passthrough and the networks entry in --vars for a role whose tasks name networks.internet.dns, which today is svc-net-tor alone; the generator derives that rather than taking a declaration, so a second role asking for the value documents it by doing so. scripts/github/guide/compose_deploy.sh substitutes the placeholder from INFINITO_DNS_IP, the same SPOT inventories/development/default.yml reads, next to the existing domain substitution.
This follows 972596a758, which added the administrator key to the same block for the same reason. No fallback inside the role: defaulting to NETWORK_PUBLIC_DNS_RESOLVERS would route a Tor node's clearnet queries to Cloudflare without the operator choosing it, which is what the assert exists to prevent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34893363518 lost the fedora workspace job at certs.sh with exit 125. docker run resolved alpine:latest itself, the token request to auth.docker.io came back as 'read: connection reset by peer', and that single transport fault ended the job. Five of the six workspace jobs pulled the same image successfully in the same run, so the cause is transport, not a Docker Hub policy response, whose signature would be toomanyrequests after a successful token exchange.
Every other network operation on this path already retries: scripts/github/runner/pull_with_retry.sh serves two workflows, scripts/image/build.sh has its own ladder for the build frontend, and cli/administration/deploy/development/compose.py retries compose up six times. This call site had no attempts at all, so pre-pull through the existing helper rather than a new loop.
The attempt budget is 3x10s instead of the helper's 7x20s default because this runs as a prerequisite on the interactive local path, where two minutes of sleeping without a network would be felt. The call stays non-fatal so a locally cached image still carries an unreachable registry, the same semantics 0df75d83a6 established for the buildx bootstrap; set -euo pipefail is why it needs the if-guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
