compose names the swarm-test volume after its project, so every cluster
left a <cluster>_nfs-export behind that no teardown path touched. At
roughly 5GB each they filled the host disk after ~30 runs, which showed
up as unexplained playwright timeouts once writes started stalling.
Remove it per cluster in teardown, per worktree subnet in lab-reclaim,
and host-wide in swarm-clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make quality rejected the new reclaim script for missing the executable
bit, and the S310 suppression in the Home Assistant token probe for
carrying no reason. A security suppression must state why it is safe, so
the loopback justification goes back on the urlopen call; ruff dropped
the second marker on Request(), where S310 never fires.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lab subnet is pinned per worktree slot while lab network names carry
the cluster id, so a cluster kept from one app blocked the next run of
another with 'invalid pool request: Pool overlaps'. The pre-run teardown
only cleared the same cluster id, which never matched. Matching on the
IPAM subnet instead clears exactly the blocker and leaves a sibling
worktree's cluster alone; swarm-clean stays the host-wide tool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
git worktree remove deletes the checkout first and the metadata dir
second. Under the agent sandbox, which holds config.worktree and
commondir as read-only bind mounts, it therefore dies with EBUSY after
the checkout is already gone, aborting make worktree-down with rc 255
even though the branch is in fact free. down.sh now falls back to the
unregistration prune.sh already used (drop gitdir + HEAD, best-effort
rm -rf) and reports any dir the sandbox still pins. That routine moves
into lib.sh so both callers cannot drift apart, and gains a third
outcome for the case where gitdir/HEAD themselves survive: only then is
the branch still claimed, and only then must the teardown fail.
The discriminator for the fallback is whether the checkout survived,
not a match on "Device or resource busy" — the identical half-removed
state arises from EACCES, EPERM and EROFS.
A failing git status probe no longer reads as clean: on a broken .git
link it exits 128 with empty output, which used to satisfy the guard,
tear down the compose stack, and only then die at git worktree remove
with an unexplained 128.
Measured against the unfixed scripts with real fixtures: pinned
metadata dir 255 -> 0, broken .git link 128 -> 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CI deploy bootstrap hand-listed two packages and let the ubuntu-latest runner image supply the rest, so a new host-side import surfaced as a red deploy job instead of a failed install. Size [project.dependencies] to what runs outside the project image and move the deployment stack into a deploy extra, then have every install site read that instead of naming packages itself: the image build and make install-python take the deploy extra, the CI host and the mirror and instructions workflows take the base.
Declare cryptography, which utils/manager/value_generator.py imports directly and which only reached the environment transitively through ansible. Drop numpy, which nothing in the repository imports. Guard the boundary with a lint test that blocks every deploy package and asserts the runner-host CLI still builds its parser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The storage-constrained work put utils.storage.constrained into the host-side import chain of cli.administration.deploy.development, which pulls humanfriendly. The CI dependency line installed only ruamel.yaml and PyYAML, so every compose and host deploy job aborted at parser build time with ModuleNotFoundError.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It shipped with the persistent-base change; the constraint it described - git forbids a branch feature beside feature/x, so the nested directories cannot collide - belongs in that commit's message, where it already is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The topology SPOT read default.env only, so every lab took the same subnet no matter what the generated .env said. Two labs on one host therefore collided with "invalid pool request: Pool overlaps", which is what stopped a worktree lab from running beside the primary checkout's.
Collect the generated .env first, then default.env, and emit each assignment as := so precedence becomes exported environment, then .env, then default.env. default.env stays the fallback because the workflow sources this file before make dotenv has run, which was the original reason for reading it directly.
A line that does not parse as a balanced assignment now aborts naming the line. Sourced as-is it would have become an unconditional assignment and silently defeated the precedence this change exists to establish.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An aborted Playwright run - watchdog SIGTERM, task timeout - returns a result dict with neither stdout nor rc. The reporting task dereferenced both, so it became the fatal error itself: the log ended on "object of type 'dict' has no attribute 'stdout'" while the real cause, a disk-exhaustion abort with rc 75, sat eight thousand lines earlier. Guard all three reads, fall back to msg for stderr because that is what Ansible fills on an abort, and treat an absent rc as a failure so the run is recorded and named by the aggregate check rather than vanishing behind a template error.
The distro sweep's summary also mislabelled the distros it never started: they kept the initial "not reached within the budget" note even when the sweep aborted on a failure with 12214 seconds of budget left, pointing at a budget problem that did not exist. Say what happened instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
act merges its --env-file, which defaults to .env, after the workflow-level env block (pkg/runner/run_context.go:84 at v0.2.89) and interpolates only afterwards, so INFINITO_DISTROS: ${{ inputs.distros }} was replaced by the repository's own five-distro list before the expression was ever evaluated. Every workspace leg therefore swept all five distros instead of its own: 25 distro deploys where 5 were intended, which is what walked the five legs into their 345-minute cap. The drill step's name kept rendering the single distro because it is interpolated from the workflow model rather than from the env map, which is why the logs read as a contradiction.
Pass INFINITO_DISTROS as an --env pair instead, the layer act itself wins with, where --env beats the env-file because the file only fills keys that are absent (cmd/root.go:364-366). The expression was duplicated across ACT_ENV and ACT_INPUTS, so name it once.
Wrapping the assignment for readability required the splitter to tolerate whitespace: inside double quotes the shell removes the backslash-newline but keeps the continuation line's indentation, so every pair after the first arrived with a leading tab.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pipeline gate only listed twelve of seventeen jobs in its needs, leaving out wait-fork-prereq-run, fork-prereqs-ready, build-ci-images, images-mirror-missing and test-dns. A failure in any of them turns its dependents into skipped rather than failure, and the gate's check matches only failure and cancelled, so the run would have concluded green without images ever being built.
A caller-supplied whitelist was also passed through verbatim. Run 30492550883 was dispatched with web-app-prometheu; discovery dropped the unknown id without a warning, the empty resolution became [], the deploy workflows skipped their whole matrix on that condition and the parent job still concluded success. Validate every id against roles/<id> and exit naming the offenders, so a typo cannot quietly remove an application from a run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default base was /tmp, which is tmpfs on this host: a reboot took a whole worktree and its uncommitted work with it. Derive the base from the origin remote instead, as ~/.local/share/worktrees/<domain>/<account>/<repo>, and let the branch keep its slashes so it nests, which also makes the infinito-wt- prefix redundant. Without an origin remote it now fails loudly rather than inventing a location.
up.sh bumped only INFINITO_SWARM_LAB_SUBNET per slot while the five swarm lab node IPs stayed at default.env's 192.168.244.x, so every worktree from slot 1 on got a lab network in a different /24 than the nodes it places in it, and collided with the primary checkout's lab. Pin all six values from one octet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge of the swarm-nfs work moved utils/unmount_nfs_mounts.sh to utils/unmount/nfs_mounts.sh and updated teardown.sh and the bootstrap, but clean/all.sh kept the old path. make swarm-clean died at the unmount step with "No such file or directory" before it reached container removal, which is exactly the state that needs cleaning up. The host-state release is not added here because all.sh already unmounts DIR_VAR_LIB itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f660dd6bd wrote the placeholder into a parameter expansion, `${INFINITO_IMAGE:-<none, bootstrap will build it>}`, which tests/lint/filesystem/shell/test_no_infinito_defaults.py rejects: an INFINITO_* default in a shell file is a second source next to default.env that drifts silently, and only the empty form is allowed. It was the sole failure of test-lint.
Split the message into the resolved and the unresolved case instead of suppressing the lint; the output is also no longer self-contradictory, since 'published <none>' could not both be true.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
act fills the GITHUB_REPOSITORY* vars with its own placeholders, so the
existing owner guard passes and the resolver hands back ghcr.io/nektos/act/
<distro>. That path does not exist and the swarm node build died on a 403
three minutes into every local run. Returning nothing is what the callers
already treat as 'build it locally'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f660dd6bd introduced `${INFINITO_IMAGE_TAG:?}` in the image resolution, a hard requirement the replaced call deliberately did not have -- ci.sh exits 0 before its own assert when no registry owner is in scope. The check sat after the EXIT trap was armed and before the env layer is sourced, so a missing tag ran collect_and_teardown against a cluster that was never created, and the variable appeared in neither the # Param header nor the strict guard block that carries the other four required inputs.
Move it into that guard block, which runs before mktemp and before the trap, and drop the now-redundant inline assert.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
841a5e0e1 claimed to keep entries whose checkout cannot be inspected, but the check resolved dirname(dirname(gitdir)) -- the checkout's PARENT, e.g. /tmp -- which always exists. The branch was dead code and every entry fell through to `rm -f gitdir HEAD`, including a live worktree whose metadata git can no longer read: that path drops it from the liveness list and it was then unregistered, which is the reported symptom.
Make the checkout directory itself the criterion, and refuse to conclude anything when its parent is not traversable -- `test -d` succeeds on a directory without +x, so a first attempt using -d alone reproduced the same blind spot. Reproduced in throwaway repos: a live worktree with a 0-byte commondir keeps its registration, a deleted checkout is pruned, and a metadata corpse without gitdir is cleaned.
Remaining limit: `git worktree prune`, which the script runs first, shares the blind spot and wins where it can unlink; this guard therefore only decides in the sandbox-pinned case that produced the bug.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every workspace leg pinned INFINITO_DISTRO=debian and drilled all five distros sequentially, pulling its swarm nodes from ghcr <distro>:latest -- an image from 2026-03-21 whose nfs-ganesha 7.2 segfaults on any accepted TCP connection on 2049. The controller mount then failed with ECONNREFUSED for 26 minutes and took run 30446813110 down after 5h13m, with 3 of 5 distros never reached.
Derive the distro from the leg's own matrix entry, build that image, and drill only it: swarm-zombie takes an optional distros= that defaults to INFINITO_DISTRO, and 00_one.sh prefers the locally built image over the published one. 12_swarm.sh refuses to start when that image is missing, so a broken build fails loudly instead of being masked by a stale registry pull.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Staleness was decided by reading gitdir inside the metadata dir. Under the agent sandbox that dir is pinned as read-only bind mounts, so a live worktree can read as having no gitdir at all -- prune then removed gitdir and HEAD and left the checkout as 'fatal: not a git repository' with no way back. Decide liveness from git's own worktree list plus each checkout's .git pointer instead, keep entries whose checkout base is not visible rather than guessing, drop the expected EBUSY noise, and report the pinned leftovers once with the command that clears them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test-environment.yml and test-guide.yml become test-workspace.yml and
test-instructions.yml. Orchestrator job ids, the distros-SPOT matrix
lint, the act example, the diagnostics slug and the rescue artefact
names follow.
Replace the roles_only input with per-job workspace/instructions
tri-state inputs (auto|true|false). 'auto' derives the former
roles_only skip straight from whitelist: both jobs run on a global
scope (whitelist empty or __ALL__) and are skipped on a role-scoped
run, which keeps the PR behaviour identical because the diff resolver
never emits an empty whitelist. entry-manual exposes both as choice
dropdowns defaulting to false, so a manual run opts in explicitly.
The now redundant roles_only output is dropped from
pr_affected_roles.sh and subset_roles.py.
Branch-protection required-status-check names change along with the
job renames.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The artifact of the wedged run carried journal, unit status and mounts but no process data, so the hang could be localized to the export config commit and not pinned to a syscall. wchan and syscall per thread close that gap.
MainPID 0 is its own likely state given Restart=on-failure with the start-rate limit disabled, so that branch reports the unit counters instead. Every read keeps 2>&1 because /proc stack and syscall are EPERM on hardened kernels.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
git worktree prune removes the metadata directory as a whole, but the agent
sandbox pins config.worktree and commondir as read-only bind mounts, so the
unlink fails with EBUSY and the stale entry keeps claiming its branch.
Dropping gitdir and HEAD unregisters it without needing the pinned files gone.
The LUKS step of the DR drill died on `The file /dev/mapper/usbdrill does
not exist and no size was specified.` -- from mkfs, not from cryptsetup,
because luksOpen had exited 0. The node did have a udev, and it had
created the mapper entry: as a symlink to ../dm-1. Modern udev no longer
mknods, it only manages symlinks, and the /dev/dm-N target it points at is
a devtmpfs node that a container's tmpfs /dev never receives. Without a
udev libdevmapper creates the node itself, which is why the debian nodes
passed -- their image ships no udevd at all, while arch cannot install
systemd without one.
Masking udevd in the node image removes the whole class rather than one
call site. DM_DISABLE_UDEV rides on the node containers instead of the
script because the drill opens the volume a second time through the
recover CLI in a separate docker exec, which would not inherit an export.
The block-device check then fails loudly with both prerequisites named,
and closes the mapping first so an abort cannot leak it onto the runner --
device-mapper is not namespaced and every distro round shares the VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ganesha unregisters its RPC programs from inside nfs_Init_svc(), before
the sockets are ever put into listen(). That unregister reaches
libntirpc's rpcb_unset -> CLNT_DESTROY -> svc_vc_destroy_it, which spins
on nanosleep({0,0}) with no timeout -- and it only takes that path when
the AF_LOCAL connect to /var/run/rpcbind.sock succeeds. So a reachable
rpcbind is what wedges the server, not a missing one.
The wedged instance burned 5m30s of CPU while serving nothing, never
logged NFS SERVER INITIALIZED, and left the controller mount retrying
into ECONNREFUSED for 1706s before the deploy failed 20 minutes further
downstream. Masking and stopping both rpcbind units before ganesha starts
makes the spin unreachable: without a local socket rpcb_unset returns
false immediately. Wants=rpcbind.service is not Requires=, so a masked
unit is logged and ignored, and v4 registration is explicitly optional in
ganesha -- which is why this holds only while the export set stays
NFSv4-only.
The unit list is probed first so distros without rpcbind are skipped
rather than failing the play, and the probe distinguishes "no such unit"
from an unreachable manager so the mask can never be skipped silently.
The Type=forking unit reports started while the daemon is still spinning,
so a wait_for on 2049 turns a 28-minute mystery into a named failure, and
the diagnostics now carry rpcbind's journal plus what actually listens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guide job builds its boot image from a pinned pkgmgr-arch parent that
can be weeks old, and `pacman -Sy systemd` against it is the partial
upgrade arch upstream documents as unsupported. It pulled systemd 261.2-1
next to the image's systemd-sysvcompat, which pins systemd exactly, so the
transaction could not be satisfied and every arch guide run died before
the README sequence started.
Refreshing archlinux-keyring first and then running a full -Syu --needed
fixes it; the cache wipe matches what the apt and dnf branches already do.
The upgrade lands in the PREP container, which runs sleep infinity, so
systemd is only ever replaced before it becomes PID 1 in the committed
image.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The controller mount of DIR_VAR_LIB survived the teardown, so the next
distro cycle inherited an NFS session whose server container was already
gone: the kernel logged 'server not responding, timed out' and every
access under /var/lib/infinito wedged, taking the deploy down with it.
All five swarm jobs of the last manual run died this way, each on a
different distro but always on the third lab cycle.
Both halves of the old teardown were unable to clear it. The guard
`mountpoint -q` blocks on exactly the dead mount it is asked about, so
the timeout dropped the unmount that was needed; and `umount -lf` ran
without sudo on a non-root runner, so the EPERM went into `|| true`.
Detection now reads /proc/self/mountinfo, which touches no filesystem,
the unmount elevates when the caller is not root, and a mount that
survives all of it fails loudly instead of silently poisoning the next
cycle. The bootstrap runs the same release before bringing a lab up, so
a cycle whose teardown never ran cannot hand the mount on either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The loop may drop a distro when the budget runs out, and both skip paths fell through to an implicit exit 0, so a run that never touched two distros still reported success. That hid the coverage gap: arch, fedora and centos get no package cache (cache.override.yml is apt-only), are therefore the slowest, and are the first the remaining < max_seen heuristic drops.
Skipping stays non-fatal and the order stays randomised, but every distro now shows up in a GITHUB_STEP_SUMMARY table in execution order, as passed, failed with its rc, or skipped with the reason. The table is also rendered before an aborting exit, since that is the run whose table matters. Two contract tests cover it; the order assertion checks the logged shuf order rather than the input list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The update-ca-trust branch hardcoded /etc/ca-certificates/trust-source/anchors, which only arch ships. Fedora's ca-certificates rpm ships only /etc/pki/ca-trust/source/anchors and CentOS Stream is identical, so on those nodes install -d created a directory nothing reads, update-ca-trust extract rebuilt a bundle without the CA, and the script reported success. The playbook then routes the node dockerd through the MITM proxy unconditionally, so every image pull failed with x509 while bootstrap looked green.
The dir is now probed, an unknown layout is a hard error, and the rebuild no longer swallows stderr.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d9a4498b3 collapsed eleven CI steps into one, which broke the assumption every handover in the drill rested on: a $GITHUB_ENV write only comes back in the NEXT step. Inside one step nobody sees it. matrix.py reads os.environ["MGR_IP"], so every swarm job died with KeyError before the first deploy round; 05 -> 06 -> 07 passed INITIAL_NODE/NEW_NODE the same way and would have died next.
The node IPs now come from the topology SPOT: compose.yml already pins every node to those INFINITO_SWARM_*_IP keys, so the docker-inspect capture only read back what compose had set. It goes, together with the $GITHUB_ENV publish and lab_net. lab_domain_target reads the env instead, guarded by an assert task because lookup('env') yields '' for a missing key and an empty dnsmasq `address=/domain/` is valid syntax meaning NODATA -- the failure would have surfaced 30 retries later as a DNS problem.
INITIAL_NODE/NEW_NODE move to a SWARM_DRILL_ENV file, created fresh per distro so the previous distro's node cannot leak into the next drain. The out-of-drill callers of _context.sh keep working through a visible guard; both writers stay strict via :?.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
guide_select picked the role, the mode and the distro in one draw, so a run proved the README instructions worked on exactly one randomly chosen distribution. The distro list was also a literal in the workflow, which meant the manual-CI distros input never reached the guide at all.
The distro draw is removed. guide_select still picks the role and derives the mode, while the shared loop from 1bbe7a0eb drives the distro dimension, and ci-orchestrator forwards its distros input.
The four mode-specific steps collapse into guide/one.sh, which the loop runs once per distro: it renders the pkgmgr base image for the current distro, then either execs host_deploy.sh or brings the compose server up and replays the README block against it.
one.sh releases the compose server in an EXIT trap. Without it the second distro would start against the stack the first one left behind. Rescue diagnostics gain a distro segment for the same reason the swarm collectors did.
The job timeout goes from 90 to 345 minutes because up to five rounds now have to fit; the loop's own budget still decides how many actually run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The discover job was named after the full distro list and the plan summary printed all five, but test-deploy-swarm.yml never set INFINITO_DISTROS, so every run fell back to the default.env distro and only debian was ever exercised. The job name has been advertising coverage that did not exist.
The workflow now passes the distros input into the env and hands the drill to the shared loop from 1bbe7a0eb. Bootstrap, setup, matrix deploy, seed, drain and assert move out of eleven separate steps into routine/00_one.sh, which is what the loop runs once per distro.
Collection and teardown move with them, into an EXIT trap: it replaces the removed 'if: always()' and 'if: failure()' step conditions and returns the original exit code so cleanup cannot mask the deploy failure. The always() teardown step stays as a second net for a step that times out as a whole.
Playwright reports and rescue diagnostics gain a distro segment, matching the layout the compose path already writes, because a second distro would otherwise overwrite the first one's evidence. The structure summary carries the distro in its heading for the same reason.
The README described the old shape and is corrected: the loop entries are listed, topology_summary.sh was missing, and 01_bootstrap.sh is no longer 'one CI step'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The serial distro loop (shuffle, time budget, fast-fail skip, exit on the first failing distro) lived inline in scripts/tests/deploy/ci/all.sh and was therefore reachable only from the compose and host deploys. Swarm and guide have to iterate distros with the same semantics, so the loop moves into scripts/tests/deploy/distros.sh and takes the per-distro command as "$@".
ci/all.sh keeps the app-level preamble and execs the loop with the new ci/one.sh, which resolves the CI image and dispatches to the per-app override or dedicated.sh.
Image resolution leaves the loop as well: scripts/meta/resolve/image/ci.sh renders the reference through cli.meta.ci.image_ref instead of respelling ghcr.io/... locally, which meta/distros.yml declares as the single source.
The two CI time knobs are declared in default.env instead of being repeated as literals, so no caller has to invent a budget.
Three paths are dropped as dead: the empty budget that disabled the limit, the failed counter that could only ever print 0, and the branch for an unset deadline.
Both new scripts are pinned by contract tests. Those tests clear BASH_ENV because the test container exports it as the env loader, which would otherwise re-inject every stripped key and make the assertions pass for the wrong reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
git writes MERGE_HEAD only after the pre-merge-commit hook returns, so the
gate never found it and aborted every non-fast-forward merge with "No merge
in progress"; git then wrote the merge state via abort_commit(), leaving a
half-finished merge that git status reported as "still merging". The gate
had therefore never verified a single signature.
git also skips pre-merge-commit entirely once a merge conflicts, so the
conflict-resolution commit was unguarded as well. prepare-commit-msg fires
with MERGE_HEAD present on both paths and no-ops on ordinary commits.
Verified in a scratch repo with a real pre-commit install: ordinary commit
passes silently, clean automerge and conflict-resolution commit both block
an unsigned incoming commit, and the extra stash/restore cycle round-trips
unstaged changes byte-identically.
pre_commit install never removes hook types dropped from the config, so the
install script now also uninstalls the stale pre-merge-commit hook.
Parallel checkouts collided on the compose subnet, host ports and container
names, so a quality run in a worktree died at compose-up. Slot each checkout
via custom.env and layer two overrides on top: the shared cache network in
place of a second cache stack, and the shared .git directory without which
git aborts inside the container.
Both overlays gate on the resource they need, not on the slot number, so a
stray INFINITO_INSTANCE cannot make compose demand a value nobody set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
categories.yml moves from roles/ to the repository-root meta/, joining packages.yml. utils/roles/categories.py owns its location plus the tree-walking helpers that were stranded in entity/name.py, so no caller spells the path or duplicates the lookup.
meta/distros.yml becomes the SPOT for every distribution: os_family, Galaxy platform name, dev-environment base image, and the pkgmgr/pkgmgr-virgin/environment reference templates. It replaces seven scattered copies (DISTRO_FAMILY, VALID_DISTROS, _DISTROS, _CANONICAL_PLATFORMS, two shell arrays, a Makefile literal). INFINITO_DISTROS is no longer a static default but generated by utils/env/handlers/infinito/distros.py; the shell consumers read it through the env layer.
Drops dead CLI surface found along the way: --categories-file and --roles-dir on the provision CLI plus three unused parameters of generate_dynamic_inventory, and the positional roles_file of the invokable categories command.
tests/lint/repository/test_distros_spot.py pins the copies that cannot call the SPOT (workflow literals, docs table) and forbids composing a pkgmgr reference outside it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
moby/buildkit was hardcoded in four places across two workflows and catthehacker/ubuntu in two act scripts, none of them reachable by the image mirror: utils/docker/image/discovery.py only scans roles/**/meta/services.yml. So the one image the CI build cannot proceed without was pulled straight from Docker Hub, unmirrored, which is what broke the fedora build in run 30160077828.
Declare both as mirror-only services on svc-runner, following the test-e2e-playwright precedent, so the existing mirror pipeline picks them up unchanged. New scripts/meta/resolve/image/service_ref.sh reads image+version back out of the declaration and is the single source for consumers outside Ansible; its mode argument is mandatory (upstream never touches the network, mirror prefers the GHCR tag and falls back to upstream). The fallback is why the mirror job can keep running in parallel with the build instead of gating it.
ACT_PLATFORM_IMAGE moves to an env handler so act consumers read the declared value via BASH_ENV, and workflow.sh now demands it instead of defaulting to a literal. Operator overrides still win through eb.setdefault.
Verified: service_ref.sh in all four paths (upstream, mirror hit, missing mode, invalid mode); the mirror path matches GHCRProvider.image_base() character for character and resolves against the live registry anonymously; ACT_PLATFORM_IMAGE resolves through the real BASH_ENV mechanism; make quality green on all four targets.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The container image ships with /var/lib/apt/lists emptied, so puppeteer's
--install-deps resolved no package name and chrome-headless-shell then died on
a missing libnspr4. All 262 markdown files with a mermaid block failed to
render, and the wrapper discarded the reason into /dev/null.
Refresh the lists when apt-get is present, and surface the provisioning output
instead of swallowing it, so a failure on a distribution puppeteer cannot serve
names its own cause.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill configured svc-bkp-remote-2-local with a fixed manager plus NFS provider list for every app, while the manager only runs a backup repository when the app's include closure pulls svc-bkp-volume-2-local or svc-bkp-secrets-2-local in. For an app whose container_backup consumer is statically off, such as svc-dns-unbound, user-backup never ran on the manager, so the pull was rejected with 'Permission denied (publickey,password)' and the fail-if-any wrapper took the whole unit down at drill step 4/9. Four roles reach that step today: svc-dns-unbound, svc-db-redis, svc-registry-cache and svc-registry-docker.
backup_repos.py expresses the placement once and matrix derives the provider list from it per round. derive_includes accepts the round's variant map because INFINITO_APP_VARIANTS only reaches the subprocess callers. The extras file keeps the device paths the drill reads back and loses the provider list nothing read from it, and the drill banners stop claiming backup legs that were never installed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill configured svc-bkp-remote-2-local with a fixed manager plus NFS provider list for every app, while the manager only runs a backup repository when the app's include closure pulls svc-bkp-volume-2-local or svc-bkp-secrets-2-local in. For an app whose container_backup consumer is statically off, such as svc-dns-unbound, user-backup never ran on the manager, so the pull was rejected with 'Permission denied (publickey,password)' and the fail-if-any wrapper took the whole unit down at drill step 4/9. Four roles reach that step today: svc-dns-unbound, svc-db-redis, svc-registry-cache and svc-registry-docker.
backup_repos.py expresses the placement once and matrix derives the provider list from it per round. derive_includes accepts the round's variant map because INFINITO_APP_VARIANTS only reaches the subprocess callers. The extras file keeps the device paths the drill reads back and loses the provider list nothing read from it, and the drill banners stop claiming backup legs that were never installed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Restarting docker does not clear a container whose shim never sent an exit
event: that shim belongs to containerd, a separate unit that survives the
restart. Measured on a host carrying two such remnants - docker.service had
been restarted hours earlier while containerd.service had been up for three
days, and `rm -f`, plain `rm` and `stop -t 1` all still failed afterwards.
The hint also has to say what the restart costs. It wipes every exec instance,
so following it during a deploy kills the run: an act deploy died after 1h48m
with `failed to inspect exec: No such exec instance`, and the log carried no
ansible error at all, only post-steps failing in milliseconds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
