Parallel checkouts collided on the compose subnet, host ports and container
names, so a quality run in a worktree died at compose-up. Slot each checkout
via custom.env and layer two overrides on top: the shared cache network in
place of a second cache stack, and the shared .git directory without which
git aborts inside the container.
Both overlays gate on the resource they need, not on the slot number, so a
stray INFINITO_INSTANCE cannot make compose demand a value nobody set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The template carried a three-way conditional across three {% set %} lines,
which was already at the edge of readability. Move the decision into
lookup('smtp_host') as plain control flow and pin all six cases in tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A workstation has no GHCR mirror and no node registry cache, so the same
deploy that CI finishes in ~70min median needs far longer locally and
died at the 345min step limit. The CI value and its 15min headroom to
the platform cap stay untouched; swarm-zombie raises it locally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Host-only backup role: the missing compose/swarm mode declarations
defaulted to enabled, so the roundtrip ran both — swarm produced an
empty deploy matrix and failed. Mirror the svc-bkp-secrets sibling.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nodes outside the mesh (nfs, backup) can neither resolve the onion
mail host (no tor resolver) nor reach a locally published SMTP port;
hlth-msmtp died with status=68/NOHOST. The manager's ingress publishes
the mail ports on every node IP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ansible resolves task includes relative to the including playbook, so the
playbook_dir prefix was never needed — and it hid the file from the
include-existence check, which the nocheck directive then papered over.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DinD nodes fetched every image straight from upstream while the warm
host cache sat unused beside them; only the DiD container was wired to
it. CA and proxy drop-in are now placed during bootstrap, before the
deploy installs docker, so the daemon picks them up on its first start.
NO_PROXY carries the node names because the in-cluster registry is
addressed by hostname and the proxy intercepts pulls only. Without a
local cache stack, as in CI, the CA is never mounted and the block is
skipped entirely.
Cut the prometheus swarm step from a 5h45 timeout to 4h03 green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
categories.yml moves from roles/ to the repository-root meta/, joining packages.yml. utils/roles/categories.py owns its location plus the tree-walking helpers that were stranded in entity/name.py, so no caller spells the path or duplicates the lookup.
meta/distros.yml becomes the SPOT for every distribution: os_family, Galaxy platform name, dev-environment base image, and the pkgmgr/pkgmgr-virgin/environment reference templates. It replaces seven scattered copies (DISTRO_FAMILY, VALID_DISTROS, _DISTROS, _CANONICAL_PLATFORMS, two shell arrays, a Makefile literal). INFINITO_DISTROS is no longer a static default but generated by utils/env/handlers/infinito/distros.py; the shell consumers read it through the env layer.
Drops dead CLI surface found along the way: --categories-file and --roles-dir on the provision CLI plus three unused parameters of generate_dynamic_inventory, and the positional roles_file of the invokable categories command.
tests/lint/repository/test_distros_spot.py pins the copies that cannot call the SPOT (workflow literals, docs table) and forbids composing a pkgmgr reference outside it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sandbox tier runs on gVisor, so its upstream documentation is a routine
lookup while working on svc-virt-kata.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The daemon configuration named fuse-overlayfs as the storage driver for every
host it was rendered on, while the package that provides it is installed under
the opposite condition: only inside a nested container. A host that is not
docker-in-docker therefore received a daemon pointed at a graphdriver nothing
had installed, and dockerd does not start on one.
That path is reached by a bare host in the svc-virt-kata group, which is
exactly the shape the agent tier is meant to run on. It stayed invisible
because every deploy in this repository nests through the stack container, so
the condition that installs the package and the condition that renders the file
were never false at the same time.
fuse-overlayfs exists for the nested case, where overlay2 cannot stack on
overlay2. On a bare host the native default is both correct and faster, so the
key is now emitted only where it is needed and the JSON comma structure moves
with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sandbox tier carried no check of its own. Three agent replicas once ran on
the shared kernel while every test stayed green, because nothing between the
runtime selection and the running container ever asked what the container
actually got. Each guard below sits where the promise first breaks.
A node is declared a sandbox target by a label that routes every sandboxed
service to it. It now has to prove it can carry them first: the node reads its
own default runtime and refuses the label while that is still runc. Doing it
before the label rather than after is the point, since the label is what makes
the promise.
Compose has no node-wide default to inspect, so the agent roles read the
runtime the container actually received once the stack is up. The check is
skipped in embodied mode, where the runtime is dropped on purpose because a
microVM between the agent and the hardware it drives defeats the point, and it
keys on the same services.kata switch that already drives the placement
constraint and the compose pin.
A runtime is only as good as the template around it. The new lint refuses a
privileged container, a host network, pid, ipc or user namespace, and a host
docker socket bind in any role that declares a kata service. It fails when it
finds no sandboxed role at all, so renaming the service cannot silently retire
the guard.
It caught its own subject on the first run: both agent templates emitted
privileged: true next to the device allowlist, which grants every device and
made the allowlist decorative. The requirement is explicit that no device
outside the allowlist reaches the agent, so the flag is gone and the role
README no longer claims a limit that was not enforced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The onion note listed manjaro, which is the base image rather than a
declared distro id; the SPOT lint flags any list that diverges from it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The file moved from roles/ to meta/ in the swarm branch; the onion
requirement still linked the old path and failed the markdown link check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge dropped the import while six os.environ uses remained, so the
module died with NameError before collecting any test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
categories.yml moves from roles/ to the repository-root meta/, joining packages.yml. utils/roles/categories.py owns its location plus the tree-walking helpers that were stranded in entity/name.py, so no caller spells the path or duplicates the lookup.
meta/distros.yml becomes the SPOT for every distribution: os_family, Galaxy platform name, dev-environment base image, and the pkgmgr/pkgmgr-virgin/environment reference templates. It replaces seven scattered copies (DISTRO_FAMILY, VALID_DISTROS, _DISTROS, _CANONICAL_PLATFORMS, two shell arrays, a Makefile literal). INFINITO_DISTROS is no longer a static default but generated by utils/env/handlers/infinito/distros.py; the shell consumers read it through the env layer.
Drops dead CLI surface found along the way: --categories-file and --roles-dir on the provision CLI plus three unused parameters of generate_dynamic_inventory, and the positional roles_file of the invokable categories command.
tests/lint/repository/test_distros_spot.py pins the copies that cannot call the SPOT (workflow literals, docs table) and forbids composing a pkgmgr reference outside it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
package_install resolves ids against the including role, not the role
owning the task file. web-app-mailu and sys-svc-mail-msmtp both include
sys-svc-mail-smtp's disable/enable tasks, so postfix has to live in the
shared root registry; only the owning role had declared it, which aborted
the mailu deploy.
The new lint walks cross-role task includes and reports every id the
including role cannot reach, so this fails in seconds instead of hours
into a deploy. It flags both call sites when the fix is reverted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drop a return-only temporary and state autoescape explicitly — the test
renders a dnsmasq config, where html escaping would corrupt the output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible resolves task includes relative to the including playbook, so the
playbook_dir prefix was never needed — and it hid the file from the
include-existence check, which the nocheck directive then papered over.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three waits in the login path were raw literals while their neighbours
already went through resolveTimeout, so they kept a flat 30s where a tor
circuit needs the onion multiplier.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The egress router reached for ansible.builtin.package with a literal
name, which bypasses the per-distro mapping the registry exists for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects, the second only visible once the first was gone.
The hub answered every proxied request with HTTP 400. In swarm it joins the
proxy's own overlay, so the forwarded request arrives from that network and not
from the hub's, while `trusted_proxies` listed only the hub's own subnet. The
container log named the address it rejected, and the two networks resolve to
192.168.107.0/24 for the proxy against 192.168.79.0/24 for the hub. Compose has
no such split, which is why this never surfaced there. Use the project-wide
trusted-proxy CIDRs instead, the same list three other roles already consume.
With the proxy trusted the failure moved to HTTP 502: the backend was
crash-looping. Home Assistant guards its config directory with a PID lock, so
the second and third replica on the shared volume abort with "Another Home
Assistant instance is already running" and never come up. The service sat at
2/3 with `task: non-zero exit (1)` cycling over all three nodes. Upstream
supports no multi-instance mode, so the hub is pinned to a single replica like
xwiki and jellyfin before it.
Verified by a full swarm run: deploy, e2e and the drain-and-reschedule step all
green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
package_install resolves ids against the including role, not the role
owning the task file. web-app-mailu and sys-svc-mail-msmtp both include
sys-svc-mail-smtp's disable/enable tasks, so postfix has to live in the
shared root registry; only the owning role had declared it, which aborted
the mailu deploy.
The new lint walks cross-role task includes and reports every id the
including role cannot reach, so this fails in seconds instead of hours
into a deploy. It flags both call sites when the fix is reverted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reaching an onion address required each role to point its own http client
at the tor socks proxy, which then carried clearnet traffic too — fatal
in the test network, which has no exit, and the reason nextcloud needed a
per-host exclusion list.
The egress router already redirects tor's virtual range to the TransPort
and dnsmasq already forwards .onion to tor's DNSPort; the piece missing
was reachability. dnsmasq listened on loopback only, which docker
discards, so containers never saw it. It now also answers on the docker
bridge, the daemon points there with the clearnet resolver kept as
fallback, and the role's own onion resolves to a container-reachable
backend instead of loopback.
Nextcloud drops its proxy block and the exclusion list; every other role
gets the same behaviour without configuring anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ansible.builtin.package` has no module implementation. It is a documentation-only stub whose logic lives in Ansible's own action plugin, so `_execute_module` shipped that stub to the target and every deploy died at the first package install with "module (ansible.builtin.package) is missing interpreter line".
Resolve the generic marker to the host's `ansible_facts.pkg_mgr` before execution.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ansible.builtin.package` has no module implementation. It is a documentation-only stub whose logic lives in Ansible's own action plugin, so `_execute_module` shipped that stub to the target and every deploy died at the first package install with "module (ansible.builtin.package) is missing interpreter line".
Resolve the generic marker to the host's `ansible_facts.pkg_mgr` before execution.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
moby/buildkit was hardcoded in four places across two workflows and catthehacker/ubuntu in two act scripts, none of them reachable by the image mirror: utils/docker/image/discovery.py only scans roles/**/meta/services.yml. So the one image the CI build cannot proceed without was pulled straight from Docker Hub, unmirrored, which is what broke the fedora build in run 30160077828.
Declare both as mirror-only services on svc-runner, following the test-e2e-playwright precedent, so the existing mirror pipeline picks them up unchanged. New scripts/meta/resolve/image/service_ref.sh reads image+version back out of the declaration and is the single source for consumers outside Ansible; its mode argument is mandatory (upstream never touches the network, mirror prefers the GHCR tag and falls back to upstream). The fallback is why the mirror job can keep running in parallel with the build instead of gating it.
ACT_PLATFORM_IMAGE moves to an env handler so act consumers read the declared value via BASH_ENV, and workflow.sh now demands it instead of defaulting to a literal. Operator overrides still win through eb.setdefault.
Verified: service_ref.sh in all four paths (upstream, mirror hit, missing mode, invalid mode); the mirror path matches GHCRProvider.image_base() character for character and resolves against the live registry anonymously; ACT_PLATFORM_IMAGE resolves through the real BASH_ENV mechanism; make quality green on all four targets.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
download.docker.com now ships a repo for resolute with docker-ce
packages, which is what the removed todo was waiting for. 26.10
(stonking) still has none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BigBlueButton is never served over Tor (WebRTC needs UDP/ICE), but in an
onion deployment its issuer only exists as an onion address. The test
browser reaches clearnet or the onion net, never both, since the private
tor network has no exit — so the login round-trip timed out. A real Tor
Browser handles this fine, which is why the gate belongs in the spec and
not in the role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role routes all outgoing traffic through tor so it can reach
keycloak on its onion address, but the test tor network has no exit, so
every app download ran into a two-minute timeout and the deploy aborted
on the mandatory oidc_login plugin. All 20 required apps resolve to
githubusercontent hosts after redirects, so excluding those plus the
appstore API is enough; onion traffic still goes through the proxy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A workstation has no GHCR mirror and no node registry cache, so the same
deploy that CI finishes in ~70min median needs far longer locally and
died at the 345min step limit. The CI value and its 15min headroom to
the platform cap stay untouched; swarm-zombie raises it locally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
seaweedfs disables its frontend on tor nodes by contract, which drops
the filer/master vhosts and leaves nginx.conf unrendered. The readiness
probe still hit the canonical filer domain and died on
TLSV1_UNRECOGNIZED_NAME, and the config mount kept nginx.conf in the
stack file, which docker stack deploy validates and rejects. Both now
carry the same frontend condition the role already renders into its
test env.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Chutney pins controlport_base at 8000 and the tor container runs in the
host network namespace, so its nodes squatted 8000-8006 — right inside
the app port block that starts at 8001 and runs near-contiguously to
8088. nextcloud, wordpress, mediawiki and confluence could not bind
their proxy port. Shifting the base to 18000 frees the range; the grep
fails the build if upstream ever rewords the line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DinD nodes fetched every image straight from upstream while the warm
host cache sat unused beside them; only the DiD container was wired to
it. CA and proxy drop-in are now placed during bootstrap, before the
deploy installs docker, so the daemon picks them up on its first start.
NO_PROXY carries the node names because the in-cluster registry is
addressed by hostname and the proxy intercepts pulls only. Without a
local cache stack, as in CI, the CA is never mounted and the block is
skipped entirely.
Cut the prometheus swarm step from a 5h45 timeout to 4h03 green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drop the `# syntax=docker/dockerfile:1` line from the root Dockerfile and
add a lint test that keeps it out of every Dockerfile in the repo.
The directive makes BuildKit pull an external frontend image from Docker
Hub at build time. That tag floats to the newest 1.x release, Podman and
Buildah have no frontend dispatch and ignore it entirely, and the pull
turns every build into a network dependency that fails whenever Hub is
unreachable or rate-limits the runner.
The root Dockerfile used no frontend-only feature, so the build behaviour
is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Environment matrix was the only CI path that produced no artifact at
all: no Playwright reports, no container state, no container logs. When
all five jobs of run 30254851245 failed on a dead prometheus backend, the
job log carried the 502 and nothing else, so the reason the container was
not serving could not be determined from CI at all.
Reuse the existing rescue-diagnostics SPOT instead of adding a second
capture path. Three things make it work here:
- the job mounts the runner docker socket, so `make compose-up` creates
infinito_nexus_debian on the runner daemon and it outlives the `--rm` of
the orchestrator container;
- the app stack sits one level deeper inside that container's own dockerd,
and container.py recurses into any running container carrying python3
and a container runtime;
- nothing tears the stack down on failure: 00_orchestrator.sh runs under
`set -euo pipefail` with no trap, and 10_teardown.sh is the only
`make compose-down` in the suite and never reached.
The image ref needs a slug because artifact names cannot contain `/` or
`:`, and the matrix values are refs like quay.io/centos/centos:latest.
`|| true` mirrors dedicated.sh: container.py always returns 1 by design so
a failing pipeline stays failing.
Probed locally against a running infinito_nexus_debian: recursion reaches
containers/infinito_nexus_debian/nested/, per-container inspect and logs
are captured with State.ExitCode, State.OOMKilled and RestartCount intact,
24 containers in roughly 20 seconds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/tmp/infinito is created by Docker as the parent of the CA bind mounts and
is owned by root with mode 0755, so an image running as a non-root user
cannot write there. The combined-bundle redirect was the only unguarded
write in the wrapper, and under `set -eu` its EACCES killed the entrypoint
before it could exec the wrapped command.
Every image carrying a USER directive plus /bin/sh therefore crash-looped
from the second deploy of its stack onwards, which is when compose_ca.py
starts injecting the entrypoint wrapper: the CA override is generated
before `compose pull`, so on a first-ever deploy the image is absent,
has_sh is false and no wrapper is applied.
In run 30254851245 this took down prometheus, alertmanager and
node-exporter while blackbox, cadvisor, redis and the sso-proxy stayed up.
The dead published port surfaced as a 502 on the prometheus vhost and
failed the Playwright suite in all five Environment jobs.
Write the bundle to /tmp and guard the redirect the way every other write
in the script is already guarded, falling back to the single-CA env with a
warning.
Verified inside prom/prometheus:v3.13.1 with the real bind-mount shape:
the previous version aborts with "can't create
/tmp/infinito/ca-trust-combined.crt: Permission denied" and never execs
the wrapped command; the new one builds the bundle as nobody and execs
normally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
packages.ubuntu.com serves the Ubuntu leg of the external package availability
test, next to the packages.debian.org entry that is already allowed.
pkgs.alpinelinux.org and pkgs.org are general package searches with no caller
in the repository.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the raw package modules at 76 call sites with package_install, moves
each role's distribution mapping into its own meta/packages.yml, and turns the
inventory PACKAGES variable from source-keyed literal names into a list of
declared package ids. The lint that forbids the raw modules is enabled here,
because the migration burns it down from 76 offenders to zero.
Removes sys-aur, sys-aur-install and sys-package. The AUR build user, the
passwordless pacman rule and the inventory-driven install are part of the plan
the action plugin executes, so the roles no longer carry them.
Declaring the mappings surfaced breakage the raw modules had hidden: on Debian
the `which` binary comes from debianutils, jrnl is in no Debian or Ubuntu
suite, chromium-browser is absent from Debian stable, JDK 11 is unobtainable on
RedHat, wireguard had no RedHat branch at all, qbittorrent was built from the
AUR although Arch ships it in extra, and desk-docker installed a Docker variant
that conflicts with the one sys-svc-container installs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A role states which package it needs; the registry owns every distribution
mapping and every acquisition path. Declarations live in the repository-root
meta/packages.yml when more than one consumer installs them, and in a role's
own meta/packages.yml otherwise.
schema.py owns the declaration shape, registry.py finds and indexes the
declarations, plan.py turns a resolved spec into ordered module calls, and the
package_install action plugin only executes them. Keeping the plan pure makes
every acquisition path unit-testable without an Ansible connection:
distribution repositories, the AUR with its unprivileged build user, COPR, PPA,
an EPEL bootstrap package, the pacman multilib section and source builds.
The data is guarded by lints: a package id and a concrete distribution package
name each belong to exactly one declaration, a role installs only its own or
shared ids, every id resolves for all five default distributions with an
empty mapping justified by a nocheck, declarations read in ascending order,
every plan builds and names only whitelisted modules, and an external test
queries the real package indexes of all five distributions.
This commit adds the mechanism and the shared collection; no consumer uses it
yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The container image ships with /var/lib/apt/lists emptied, so puppeteer's
--install-deps resolved no package name and chrome-headless-shell then died on
a missing libnspr4. All 262 markdown files with a mermaid block failed to
render, and the wrapper discarded the reason into /dev/null.
Refresh the lists when apt-get is present, and surface the provisioning output
instead of swallowing it, so a failure on a distribution puppeteer cannot serve
names its own cause.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reading the packaged nfs-ganesha spec is how the Fedora build of the
package was checked against its libntirpc dependency. Add the domain to
the WebFetch allowlist so that lookup does not need an approval prompt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both iteration guides and the nfs_flavor lookup docstring still described
the flavor as RUNTIME conditional, selecting ganesha only for dev and act
and kernel everywhere else. Since the per-variant switching was dropped,
meta/services.yml pins ganesha for every runtime and kernel is a
per-inventory override.
State that, name meta/services.yml as the single point of truth, and
record which os families ship ganesha.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sys-aur-install gains an extra_args passthrough so callers can hand
flags to the AUR helper. Nested include_role inherits every variable the
inner call does not shadow, so a caller flag would also reach the
system-wide yay upgrade sys-aur runs at the end of its bootstrap, where
makepkg-only flags are not valid.
Shadow the passthrough in that nested call so it applies to the caller's
package only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
