The 16.2.0 release commit (e2501cb1de) shipped a locally generated
`compose/package-cache-frontend/upstreams.conf`, the output of the cache
generator that only exists on the feature branches (230d9a1e5e). It adds
TLS server blocks for argos-net.com, huggingface.co, index.crates.io,
registry.ollama.ai and static.crates.io, but main's `cert-gen.sh` issues
no certificates for them. nginx then dies on start with
cannot load certificate "/etc/nginx/certs/argos-net.com.crt"
the frontend stays unhealthy, pypi.org on the runner refuses connections,
and `make install-lint` fails in every workspace job (run 39482585010).
Restore the file byte for byte to its state before the release (blob
b5c5ef08e7), the version every workspace job passed with in run
39443109355.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
5aa7506e36 traced the enumeration bug: a veth that disappears between the RTM_GETADDR snapshot and the SIOCGIFFLAGS ioctl makes the AF_INET6 pass return 0, and network.c:836 then skips the AF_INET pass entirely. That commit removed bind-dynamic, which took the listener collector out of the blast radius. It left the other consumer of daemon->interface_addrs in place.
forward.c:1669 opens a rejection gate under OPT_LOCAL_SERVICE and walks the same list. With the IPv4 half missing, every IPv4 source is rejected, loopback included, and the once-only warning at forward.c:1695 makes a permanent outage look like a single incident. Penpot job 105068912535 and MediaWiki job 105068912565 each lost one lab node this way: PID 332 ran with --local-service and is the PID that logged the rejection; the sampler shows 179 consecutive failures over 47 minutes with the socket still bound.
Debian's init-system-common appends --local-service unconditionally, so no packaging knob removes it. option.c:6240 resets it when any of if_names, if_except, if_addrs or authserver is set, no matter which config file carried the directive. interface=* fills if_names and, through wildcard_match, still admits every interface including a docker_gwbridge that does not exist at render time. bind-interfaces and bind-dynamic are absent from that predicate and would not have helped.
Measured against the rendered template with --local-service on argv as Debian appends it: the DNS service limited to local subnets line is gone and dig still answers from the lab zone. The nodes run Debian and Ubuntu, so the task name's previous claim about fedora and centos, and about local-service implying bind-interfaces, was wrong twice over and is replaced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CI zone served A and AAAA for INFINITO_DOMAIN and let every other type fall through to the public resolvers, which return NXDOMAIN for the reserved .test TLD. Per RFC 2308 that denial covers the whole name, not the queried type, so any caching resolver in front of CoreDNS applies it to the A record the deployment depends on.
Without such a cache nobody noticed: Mastodon's EmailMxValidator asks MX, takes the NXDOMAIN, then asks A anyway and gets the address. On a Tor node svc-net-tor puts dnsmasq in front, the denial is cached, and tootctl accounts create ends on 'Failure/Error: email unreachable'. It cost Fediwall#0 and Fediwall#2 in run 34653542922 and Fediwall#2 in run 34653754175, all three compose plus tor; the same two jobs passed in run 34614940508 as compose without tor, same command, same administrator@infinito.test, rc 0.
A template with no answer returns NOERROR with an empty answer section and the SOA in authority, so the zone denies a type without denying the name. It sits after the A and AAAA templates, which would otherwise have their names answered NODATA, and before forward, so no type escapes.
Measured against 172.30.0.53 in the running stack: infinito.test MX went from NXDOMAIN with an authority record to NOERROR with zero answers and the aa flag, infinito.test A and auth.infinito.test A still return 172.30.0.10, and deb.debian.org still resolves through the forward.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
INFINITO_DOMAIN_RE carried an already escaped regex through the env
chain: EnvBuilder.setdefault prefers a value already in the process env,
quote_for_dotenv escapes every backslash, and load.sh unescapes it again.
One inherited value with a doubled backslash was enough to write
"infinito\\\\.test" into .env, the rendered Corefile then matched a
literal backslash, and getent hosts test.infinito.test failed with rc 2.
The renderer now substitutes DOMAIN_RE = re.escape(INFINITO_DOMAIN) at
render time, so no backslash travels through the env or .env at all. The
template placeholder is renamed because it is a derived render value, not
an env key, and INFINITO_DOMAIN_RE is removed from default.env and the
passthrough keys, leaving INFINITO_DOMAIN as the single source.
A renderer test renders the real template and requires a single-escaped
match with no doubled backslash and no leftover placeholder. Verified
live: the Corefile renders ^infinito\.test\.$, test.infinito.test
resolves to 172.30.0.10 and test.infinitoXtest does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ubuntu CI image build (job 103277644636) hung on Canonical's
archive: apt fetched 26 MB at 417 kB/s, then 61.6 kB in 29 s, then sat
on the package download until the job was cancelled after 23 minutes,
while download.docker.com delivered 42.8 MB in under a second on the
same runner. Locally archive.ubuntu.com and security.ubuntu.com took
20-26 s to the first byte against 2 s for a university mirror.
INFINITO_APT_UBUNTU_MIRRORS in default.env is now the one list of
Ubuntu mirrors, in failover order: de.archive.ubuntu.com, ftp.fau.de,
ftp.halifax.rwth-aachen.de, archive.ubuntu.com.
- Image builds: the Dockerfile sources the list alongside the install
scripts and runs scripts/install/apt-mirrors.sh first. In an image
with ubuntu.sources it points both stanzas at apt's mirror+file
method, priorities in list order, and sets Acquire::http::Timeout to
10 s, so apt moves on from a mirror that fails or goes silent; other
distros are untouched. In ubuntu:latest apt-get update took 8 s
against 110 s on the stock sources, a dead first mirror was left
behind within 25 s against 62 s on apt's defaults, and the swarm node
packages (80 debs) came in 17 s. The Dockerfile's own RUN lines ran
clean on ubuntu, debian, fedora and arch.
- Package cache: package.sh proxies the first entry as the primary
Ubuntu repos and the second as their fallback. Nexus revalidates
metadata on every request here (metadataMaxAge 0), which costs a
full download from a mirror without an ETag; both de.archive and
ftp.fau.de send one, and the 15 MB universe index now comes back
from the cache in 0.02 s.
- tests/external/repository/test_apt_ubuntu_mirrors.py keeps the list
honest: the first mirror must answer InRelease within 5 s (median of
three) or the test names a faster later one, no mirror's -updates may
lag the freshest by more than 24 h, and every entry must be the
primary archive, a Canonical country alias or an enabled official
mirror in Launchpad's registry. It caught the first draft of the
list: mirrors.edge.kernel.org is not an official Ubuntu mirror.
After pulling, an existing checkout needs `make dotenv-force`: plain
`make dotenv` re-reads the old .env through BASH_ENV and keeps its
values.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_packages_available failed locally in every quality pass with
"91 package(s) could not be checked at all ... archive.ubuntu.com:
timed out". The dev stack routes archive.ubuntu.com to the package-cache
frontend, which re-runs a request against the -mirror repo only once
the primary Nexus repo answers 502/503/504. From this host's docker
network Canonical's IPv4 addresses take seconds to connect and then
send nothing, and Nexus tries all nine addresses of archive.ubuntu.com
in turn before it gives up: apt-ubuntu answered 502 after 21-25 s, so
the frontend's first byte arrived after 27-46 s, past the probe's 20 s
timeout. apt-ubuntu-mirror (de.archive.ubuntu.com) served the same
files in 0.08 s.
The four primary locations now carry proxy_read_timeout 8s: nginx
gives up on Nexus itself, its own 504 hits the existing error_page, and
the mirror answers. A shorter Nexus upstream timeout was tried first and
dropped, because Nexus spends it once per address and still took
9-21 s. The mirror locations keep the global 300 s, and a healthy
primary answers well inside the bound, as before.
Measured after a compose restart: the ubuntu index arrives after
8-12.6 s with Origin: Ubuntu, debian still in 0.03-0.16 s, npm and PyPI
in 0.13 s; make test-external passed 13 of 13 and the full
make quality-high went green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
00478b8ead moved the Corefile substitution in-process, to string.Template,
and left this file describing envsubst. That is the page a reader consults
before wiring gettext back into CI, which is what b3686a977 just removed.
grep for envsubst across the tree now returns one hit, the docstring saying
why it is not used.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32118850138 lost the :53 listener, so the sampler counts listeners.
Run 32538854067 lost DNS with the listener still bound: sockets-udp.txt
showed dnsmasq holding 0.0.0.0:53 while every forwarded query timed out,
and the sampler reported that hour as healthy.
Each tick now also records what a glibc caller gets back — for the zone
this resolver owns, derived from its own address= line, and for one name
it must forward — with the latency that caller paid. The rescue collector
probes the in-zone name too: probing external names alone cannot separate
"forwarding is gone" from "dnsmasq is gone".
The sampler now also runs on the swarm lab DNS nodes; it only ever ran in
the compose path, which is why the Magento artifact carried no sampler log
at all. Its location comes from group_vars FILE_DNS53_SAMPLER_LOG and is
required at every call site, so a run cannot quietly collect nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
address=/<zone>/<ipv4> covers only A records, so every AAAA lookup for
the lab zone was forwarded to the public resolvers. When the manager
node lost its upstream mid-run, glibc callers (Playwright request
contexts) paid the full resolver timeout per lookup: 20s each, which
blew the 30s request budget and the 60s persona budget.
local=/<zone>/ makes dnsmasq authoritative for the zone, so AAAA is
answered as NODATA from config instead of leaving the lab. Same idiom
the tor onion template already uses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The frontend proxies to the package-cache by name, and nginx resolves a
literal proxy_pass hostname once while parsing the config and keeps it for the
worker's lifetime; the resolver only re-resolves targets that contain a
variable. The package-cache was the one cache service without a pinned
address, so recreating it left the frontend dialling a dead one, answering 502
to everything and reporting unhealthy.
That surfaces far from its cause: pip.conf deliberately keeps the real index
URL and extra_hosts points pypi.org at the frontend, so a pip install reports
no versions at all and looks like a network fault.
Pins it the way the registry-cache and the frontend are already pinned, and
carries the address into the runner slots.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without the bindings the ansible apt module installs them itself mid-play, and the dist-upgrade that follows deadlocks: the module spins in user code while apt-get blocks writing to a full stderr pipe. It ran that way on all five nodes for three hours.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The five DinD lab nodes run privileged with cgroupns=host, a writable host cgroupfs and seccomp/apparmor off, and carried no resource limit of any kind. A runaway node therefore competes with the Actions runner agent itself, and the platform reports that only as 'lost communication', never as an OOM.
Only pids_limit for now. A memory cap needs a measured per-node peak, and every measurement available is a failure-time snapshot, i.e. a lower bound: 4.4 GiB on the busiest node of a variant-0 drill. Sizing the cap off that risks OOM-killing a legitimate container instead of the runaway node. The pid ceiling has no such problem: 8192 against a measured 981 is 8.3x.
The value is a static default.env key on purpose. A handler-derived key is absent from .env on a fresh runner, and the topology export step runs before the install step that generates it, so compose interpolation would have failed every swarm job before anything deployed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mirror failover turned the four apt locations into regex locations.
nginx only synthesises the `/debian` -> `/debian/` redirect for prefix
locations, so the slash-less form started answering 404. Inside a runner
`deb.debian.org` is DNS-hijacked to this frontend and the URL
reachability test probes exactly that form, which took the debian,
fedora and centos workspace jobs down.
The locations are prefix locations again, byte for byte as before. The
only thing that stays is the error path: the named location picks up the
already-rewritten `$uri` and rewrites it once more onto the mirror repo,
so the failover survives without a capture variable.
Measured against a simulated hijack: both bare paths 200 again, all four
suite indexes 200, and with `apt-debian` pointed at a dead remote
`dists/bookworm/InRelease` still arrives with 151 kB through
apt-debian-mirror.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A debian:bookworm workspace job died on `apt-get update`: the mounted
sources list carried the Ubuntu entry too, Nexus had auto-blocked
archive.ubuntu.com, and apt fails the whole update on one bad source.
Three holes, three plugs:
- One sources list per INFINITO_DISTRO under compose/package-cache/apt/,
selected by the compose overrides. A Debian runner no longer fetches
an Ubuntu suite it has no use for. The non-apt distros get an empty
file so the bind mount has a source to point at.
- autoBlock off. While a remote is auto-blocked Nexus answers `404
Remote Auto Blocked` to every client instead of falling back to the
copy it already has cached.
- Mirror failover in the frontend. Every apt suite gets a second proxy
repo on an independent upstream; a 5xx from the primary re-runs the
request against it. Nexus has no group repository type for apt, so
the retry lives in nginx.
Measured on a live Nexus 3.91.1: dead remote without a cached copy
answers 502, with a cached copy 200 regardless of metadataMaxAge. The
frontend serves 149 kB of real bookworm `Release` from the mirror repo
when the primary is dead, and the new regex locations produce upstream
request lines byte-identical to the `rewrite … break` form they replace,
including `%3a` epochs in pool paths.
INFINITO_CACHE_PACKAGE_APT_LIST is gone; the mount interpolates
INFINITO_DISTRO directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In job 91983553540 all five lab nodes derived the same pool name,
infinito_ci_unknown, and the run died on `cannot destroy
'infinito_ci_unknown': pool is busy`. The name came from `hostname`, which
returns non-zero inside the node containers, so the `|| echo unknown` fallback
answered for every one of them. One node won the race and created the pool; the
others found it, took it for a leftover, and tried to tear down a pool a
sibling was actively using.
234c1158c already fixed this collision class three days ago by deriving the
name from `hostname` instead of a constant, and its message named the zfs
exclusion in resolve.sh as the only thing keeping the collision out of sight.
That fix was never exercised, because the exclusion stayed up until ed1121053
removed it - and it does not work, because the call it rests on fails in the
containers it was written for. Guessing the identity is the defect; the
playbook has known it all along as inventory_hostname, and ci/one.sh knows its
own deploy mode. Both now state it, as a required fourth argument with no
default, the way SIZE is already stated. verdict() stops guessing too: every
row it wrote into the job summary said `unknown`, which is why the summary was
never worth reading.
The second half is independent of the naming. reclaim_stale_pool ran
`zpool destroy -f` as a bare command under `set -euo pipefail`, the only
failure path in this script that bypassed decline(). A pool that refuses to be
destroyed is by definition not a leftover - something holds it - so that case
now declines through the usual contract: a drawn pick falls back and reports,
a stated one fails loudly as required-but-unavailable. A reclaim can no longer
be more destructive than the state it exists to clean up.
Verified over the real invocation path rather than a model of it: an
ansible-playbook run using the same `script:` task and docker connection as
compose/swarm/playbook.yml, against three containers in one play, has each node
address its own pool - infinito_ci_race-mgr-01, -wrk-01, -nfs-01 - with
failed=0 throughout. A stubbed-pool harness covers the paths that need a kernel
module this machine does not have: a sibling holding the pool declines with rc
0 instead of ending the run, the same case with a stated pick exits 1 through
the contract, a genuine leftover is still destroyed, and a missing identity is
a hard error. The earlier replacement harness (ext4 recognised, btrfs
replacing it, no loop leaked) still passes against the stricter signature.
Pool names now carry dashes, which zfs_namecheck.c accepts; at 42 characters
they sit far inside the 240 limit and match no reserved prefix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dockerd's zfs graphdriver resolves `zfs` on PATH and opens /dev/zfs before it
accepts a data root (moby daemon/graphdriver/zfs/zfs.go:54,59), so the userland
has to sit in the image, not on the host. Only the apt images carried it, which
is why resolve.sh's userland_for() kept zfs out of every entry containing arch,
centos or fedora - that is every automatic swarm sweep, so the zfs data root and
the zfs branch of the backup launcher were never once exercised. Run
30781723102 shows it: "zfs left out: not every distro of this entry carries the
userland".
scripts/install/zfs.sh joins the three existing install scripts in the image
build and takes a different route per distro, each one run in a real container
of the matching base image before landing here:
debian trixie 2.3.2 zfsutils-linux, contrib enabled first
ubuntu 26.04 2.4.1 zfsutils-linux
arch 2.4.3 the AUR PKGBUILD, which configures --with-config=user
fedora 44 2.4.3 zfsonlinux repo plus a dummy zfs-kmod provider
centos 10 2.2.10 the same, from the epel path
The dummy is what keeps the RPM route usable: the `zfs` package carries an
ungated `Requires: %{name}-kmod`, so a plain `dnf install zfs` resolves through
zfs-dkms and installs 389 extra packages and 1.44 GB of kernel and toolchain to
build a module against a kernel the container never boots. Providing zfs-kmod
from an empty package keeps the same install at 60 packages with no kernel
package at all, and the transaction is checked afterwards so a regression fails
the build instead of bloating the image. The repository is removed again once
the userland is in, because the dummy pins one version and an upgrade would
otherwise re-resolve the kmod at deploy time.
Version drift against the host module is not a concern: zpool create enables a
feature only when the loaded module reports it through /sys/module/zfs/features
.pool (cmd/zpool/zpool_main.c, module/zcommon/zfeature_common.c), and libzfs
compares no versions at all (lib/libzfs/os/linux/libzfs_util_os.c:123-134), so a
newer userland degrades rather than breaking.
With every image carrying it, userland_for() and every_distro_carries() have
nothing left to decide and go. Two defects that only became reachable once zfs
could be drawn are fixed in the same move:
* /dev/zfs is a misc device, so the old lookup in /proc/devices could never
find its major and the node was never created. A node container's /dev is
populated at creation and udevd is masked, so nothing else fills the gap
either: the first bring-up of every job would have declined, blaming a
module that modprobe had just loaded. The minor now comes from /proc/misc.
* Pools live in the kernel the lab nodes share with the runner, while the
teardown only removes containers, and the node hostnames repeat across the
distro sweep. From the second distro on, zpool create hit the previous
iteration's pool. It is destroyed and its loop detached first.
The node image drops its own apt-only zfs branch and instead masks the six boot
units the userland enables by preset, which on a node can only fail: the pool is
created after boot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every lab node got an identical 30 GiB loop device because docker_dataroot.sh
defaulted SIZE to 30G and no caller ever stated it. The swarm manager hosts the
in-cluster registry, ~60 containers, and - by deliberate placement, since /data
is NFS-unsafe - the SeaweedFS object store, whose sizing filter claims 14 GiB
of slots plus a 2 GiB minFreeSpace floor. The idle nfs-server node got the same
30 GiB and used 1% of it.
In run 30859490335 the manager's device stood at 30G/30G (98%), SeaweedFS
crossed its floor at 02:04:45 and marked all 14 volumes unwritable, permanently.
Nextcloud in variant 0 stores objects in that SeaweedFS via OBJECTSTORE_S3_*,
so every authenticated page render performed an object write that failed as a
10-attempt assign ladder - 15 requests, 15 bursts of exactly 10, burst end
matching the nginx 500 to the second. That is the "🐝 web-app-nextcloud 0"
failure: 5 addon specs, 3 attempts each, all dead on the post-login page.
The default is gone: SIZE is now a required argument and a missing value dies
naming the call site. The swarm playbook states it per node - 60G for the
manager, 30G for the rest. 60G covers the observed demand (>=31 GiB: a full
device plus the denied writes) with headroom; the image is sparse, so the
raised ceiling costs only what is actually written, and the runner-side
watchdog at 6 GiB free remains the backstop. The compose and host workflows
state their unchanged 30G explicitly.
Not addressed here: what fills the first 28 GiB (registry + image layers +
volumes composition is unmeasured - the rescue artifact holds exactly one df
snapshot), and the drill-side shadow restore that supplied the last ~1 GiB;
both are separate changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docker_dataroot.sh installs zfsutils-linux at runtime, so the zfs pick depends
on the package cache being up. In run 30781723102 that decided the two swarm
drills differently: the cache stack was removed at 06:39:49 and rebuilt at
07:18:25, so the mariadb drill had no zfsutils candidate at all and declined,
while the postgres drill got one and reached zpool create. Installing it at
build time removes that dependency.
Debian keeps zfsutils-linux in contrib, which the base image does not enable, so
the branch adds a contrib source first - only for ID=debian, since Ubuntu ships
it in its default components. A failed update after adding the source removes
the source again rather than leaving a broken repo line behind.
The install is deliberately separate from the mandatory packages and
non-fatal. Node images for centos, fedora and arch have no zfs userland either
and resolve.sh's userland_for() already accounts for that, so a missing zfs
userland is a supported state - whereas folding it into the mandatory install
would let a packaging error break the node image for all five distros. A failure
now prints to stderr and the run declines the pick visibly.
Verified against the real bookworm contrib index (deb.debian.org, HTTP 200):
zfsutils-linux 2.1.11-1+deb12u1 exists there, all four libzfs libraries are
Depends rather than Recommends so --no-install-recommends keeps them, and the
only Recommends are zfs-modules|zfs-dkms and zfs-zed, which we do not want - the
module comes from the host via /lib/modules. Control flow exercised with a
stubbed apt-get across six combinations of distro, package availability and
update failure; every one leaves the build at rc 0.
Not covered here: whether a 2.1 userland can drive the runner's newer host
module. zfs-version.txt and modules.txt from 264556a31 answer that next run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
18abce480 added kmod so the data-root setup can run its one explicit `modprobe
zfs`. It also handed a working loader to the four modprobe@ instances systemd
pulls at every node boot - configfs, drm, efi_pstore and fuse - which until now
failed on the missing binary, as every node's journal in
rescue-diagnostics-workspace-debian-bookworm shows. They would have started
loading host modules from a privileged container, on every node of every swarm
job in all five distros.
Masking the template keeps them exactly as dead as they are today and leaves the
loader available to the caller that needs it, so the behaviour change of
18abce480 is confined to the one site it was meant for. It sits with the
existing masks for systemd-resolved and systemd-udevd.
The image was not built; that the mask covers every instance of the template is
read from systemd's behaviour, not measured. The next run shows it: the units
report masked instead of failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node image ships no module loader, so `modprobe zfs` in
scripts/tests/deploy/utils/filesystem/docker_dataroot.sh was a no-op whose error
was additionally discarded by `2>/dev/null || true`. Every node's own systemd
says so at boot, in rescue-diagnostics-workspace-debian-bookworm of run
30781723102:
(modprobe)[27]: modprobe@configfs.service: Unable to locate executable
'/sbin/modprobe': No such file or directory
The module itself is reachable: the nodes run privileged with /lib/modules
bind-mounted read-only, and their df confirms it. What was missing is the
loader, not the module. zfs-load-module.service is installed and enabled in the
image and sits inactive for the same reason.
This does not by itself make the debian lane apply zfs. `The ZFS modules are not
loaded.` in job 91588086713 came from libzfs at `zpool create`, not from the
suppressed modprobe - the kernel ring buffer shows five 30 GiB loop attachments
at 07:37:32, i.e. the script was past the /dev/zfs gate. And a Debian bookworm
node only has a zfsutils-linux candidate while the package cache stack is up:
the cache was removed at 06:39:49 and rebuilt at 07:18:25, which is why the
mariadb drill declined at the userland check while the postgres drill reached
zpool create. That cache dependency is untouched here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dnsmasq 2.93 with bind-dynamic enumerates IPv6 before IPv4 (network.c:836-875). A veth that disappears between the RTM_GETADDR snapshot and the SIOCGIFFLAGS ioctl makes that pass return 0, which skips the IPv4 pass entirely while the interface-record collector still runs and releases every listener not marked found. The v6 addresses already dumped survive; the whole IPv4 set does not. Recovery would come from the next netlink address event, so the loss becomes permanent when the churn stops on a release.
That is what killed web-app-matrix 0: the manager kept only [::1]:53 and eight fe80 link-locals while ip -4 addr still showed all four addresses, so ghcr.io stopped resolving and the image pull failed ten times. Both workers, same image and same config, held the complete set on the same fds. daemon.json points container DNS at 172.17.0.1, so every container on that node lost resolution at the same instant.
Reproduced on the same dnsmasq version: 889 full IPv4 release/recreate cycles under a veth storm, with ::1 never released once; the same storm without bind-dynamic produced no listener events at all. Without the directive dnsmasq binds one wildcard pair and reads the arrival interface from IP_PKTINFO, and wildcard listeners carry iface == NULL, which the collector skips outright.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role runs ganesha at -N NIV_EVENT, and roughly eighteen init steps between the CLIENT commits and the listener log below that level, so the silence in run 30699990912 marked the edge of visibility rather than the fault. Raise INIT and DISPATCH under MODE_DEBUG: nfs_Init_svc(), which binds 2049, logs under COMPONENT_DISPATCH, so INIT alone would have missed exactly the step in question.
Add a userspace backtrace beside the kernel stack dump, which only ever says futex_do_wait. eu-stack names the function holding the lock; elfutils goes into the lab image rather than the role, since a production NFS server should not carry a debugger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The summary named the filesystem that was picked, never the one the run got.
A host that declines - no root, missing tooling, no /dev/zfs - still deploys,
and nothing recorded on what, so a red run could not be reproduced.
docker_dataroot.sh now ends every path with a verdict that measures the data
root with `stat -f -c %T` and reports applied, declined, already-prepared,
unchanged or required-but-unavailable. Compose and host write it straight into
the step summary; swarm runs the script inside each node, where that variable
does not exist, so the playbook lifts the verdict line into the deploy log and
report.sh aggregates it afterwards.
`hostname` is resolved with a fallback: it is not installed by default on the
RedHat family, and under `set -euo pipefail` a bare command substitution would
have turned the decline path - which exits 0 - into a hard failure.
test-deploy-host.yml gains the `filesystem` input. single-node has been passing
it since the per-matrix selection landed, and a reusable workflow that receives
an undeclared input fails the whole run with a startup_failure whose only trace
is a banner on the run page. The single-node line was skipped in the run that
would have shown it. A lint now pairs every local `uses:` with the callee's
declared inputs, and also demands that required inputs without a default are
supplied.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backup snapshot mode only engages on btrfs or zfs, so on CI - where the
data root is whatever the runner or the node image happens to provide - it
was never exercised at all. Nothing proved the snapshot path beyond unit
tests and a synthetic probe.
`entry-manual` gains a `filesystem` choice that reaches both deploy modes.
Empty means each matrix entry draws its own, so coverage spreads over runs
instead of pinning one path forever; the pick and its origin go to the step
summary, because reproducing a red run means re-stating it verbatim. The
draw is restricted to what every distribution of that entry can deliver -
centos stream has no btrfs at all, and zfs is a third-party repository
outside debian and ubuntu - so a run never dies on a packaging gap. A
stated filesystem skips that restriction: it is an instruction, and a host
that cannot deliver it fails loudly rather than deploying on something
else.
docker_dataroot.sh formats a sparse loop image, mounts it at /mnt/docker-fs
and carves the data root inside it, then points the daemon there through
daemon.json - merged, not overwritten, so the registry-cache entry
survives. It runs on the runner for compose and inside each node for swarm,
before anything is pulled, and installs the filesystem tooling it needs.
Two container facts it has to work around: a privileged container's /dev is
a fresh tmpfs, so neither /dev/loopN nor /dev/zfs exists there and both are
mknod'ed from /proc/devices before use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The LUKS step of the DR drill died on `The file /dev/mapper/usbdrill does
not exist and no size was specified.` -- from mkfs, not from cryptsetup,
because luksOpen had exited 0. The node did have a udev, and it had
created the mapper entry: as a symlink to ../dm-1. Modern udev no longer
mknods, it only manages symlinks, and the /dev/dm-N target it points at is
a devtmpfs node that a container's tmpfs /dev never receives. Without a
udev libdevmapper creates the node itself, which is why the debian nodes
passed -- their image ships no udevd at all, while arch cannot install
systemd without one.
Masking udevd in the node image removes the whole class rather than one
call site. DM_DISABLE_UDEV rides on the node containers instead of the
script because the drill opens the volume a second time through the
recover CLI in a separate docker exec, which would not inherit an export.
The block-device check then fails loudly with both prerequisites named,
and closes the mapping first so an abort cannot leak it onto the runner --
device-mapper is not namespaced and every distro round shares the VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backup DR drill could not pull from the manager on centos and fedora:
the backup user's ssh died with "Connection closed", exit 255. Pubkey auth
had actually succeeded -- sshd only calls the PAM account phase once
authenticated -- and that phase failed because the RHEL parents ship
/etc/shadow mode 0000. Their setuid-root unix_chkpwd then needs
CAP_DAC_READ_SEARCH, which the host's unix-chkpwd AppArmor profile denies
right through the container: profiles attach by executable path, so the
lab nodes hit it despite already running privileged with
apparmor=unconfined.
chmod u+rw adds only the owner bit, which is what arch ships and what a
setuid helper needs. A hard 0600 would strip the group bit debian and
ubuntu rely on for their setgid-shadow helper and break the distros that
work today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node image layers on a pinned pkgmgr-arch parent that can be weeks old, and `pacman -Sy <pkg>` against it is the partial upgrade arch upstream documents as unsupported. Two failure modes: the parent's archlinux-keyring predates any packager key added since, so a package signed by one aborts the RUN with unknown trust; and -Sy without u installs fresh binaries next to the image's older libraries, which pacman cannot catch because it matches SONAMEs, not symbol versions (cryptsetup declares libcrypto.so=3-64 and installs happily onto an older openssl).
Refreshing archlinux-keyring first and then running a full -Syu --needed fixes both. The cache wipe matches what the apt and dnf branches already do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d9a4498b3 collapsed eleven CI steps into one, which broke the assumption every handover in the drill rested on: a $GITHUB_ENV write only comes back in the NEXT step. Inside one step nobody sees it. matrix.py reads os.environ["MGR_IP"], so every swarm job died with KeyError before the first deploy round; 05 -> 06 -> 07 passed INITIAL_NODE/NEW_NODE the same way and would have died next.
The node IPs now come from the topology SPOT: compose.yml already pins every node to those INFINITO_SWARM_*_IP keys, so the docker-inspect capture only read back what compose had set. It goes, together with the $GITHUB_ENV publish and lab_net. lab_domain_target reads the env instead, guarded by an assert task because lookup('env') yields '' for a missing key and an empty dnsmasq `address=/domain/` is valid syntax meaning NODATA -- the failure would have surfaced 30 retries later as a DNS problem.
INITIAL_NODE/NEW_NODE move to a SWARM_DRILL_ENV file, created fresh per distro so the previous distro's node cannot leak into the next drain. The out-of-drill callers of _context.sh keep working through a visible guard; both writers stay strict via :?.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fedora and CentOS Stream ship `local-service` in the stock /etc/dnsmasq.conf. It implies bind-interfaces, which dnsmasq refuses next to the lab config's bind-dynamic, so every DNS node died at 'cannot set --bind-interfaces and --bind-dynamic'. A drop-in cannot fix it: conf-dir files are read after the main config and a boolean already set there cannot be taken back. Writing the main config instead gives all five distros the same option set. On arch it also starts working at all -- the package ships no /etc/dnsmasq.d and its unit passes no --conf-dir, so the drop-in was a silent no-op there.
The restart timeout goes to 120s because arch's unit is the only Type=dbus one: systemd waits its full 90s DefaultTimeoutStartSec for the bus name, so the old 60s ansible timeout fired first and turned every arch-side DNS fault into a bare timeout with no journal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The simulated cluster's DNS config lived as an inline `copy: content:` block in the lab playbook. It moves to compose/swarm/templates/dnsmasq.conf.j2 so the template module can render it; the playbook switch follows separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Parallel checkouts collided on the compose subnet, host ports and container
names, so a quality run in a worktree died at compose-up. Slot each checkout
via custom.env and layer two overrides on top: the shared cache network in
place of a second cache stack, and the shared .git directory without which
git aborts inside the container.
Both overlays gate on the resource they need, not on the slot number, so a
stray INFINITO_INSTANCE cannot make compose demand a value nobody set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible resolves task includes relative to the including playbook, so the
playbook_dir prefix was never needed — and it hid the file from the
include-existence check, which the nocheck directive then papered over.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DinD nodes fetched every image straight from upstream while the warm
host cache sat unused beside them; only the DiD container was wired to
it. CA and proxy drop-in are now placed during bootstrap, before the
deploy installs docker, so the daemon picks them up on its first start.
NO_PROXY carries the node names because the in-cluster registry is
addressed by hostname and the proxy intercepts pulls only. Without a
local cache stack, as in CI, the CA is never mounted and the block is
skipped entirely.
Cut the prometheus swarm step from a 5h45 timeout to 4h03 green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-node dnsmasq answered the app-domain wildcard with 127.0.0.1 on the manager; node dockerd hands that resolver to every task container, where loopback is the task's own netns, so any healthcheck reaching for a published service port (friendica's msmtp send to mail.<domain>:465 with email enabled) got connection-refused although the port was ingress-published and mailu healthy - manager-placed replicas could never become healthy and the stack sat at 0/3 until the converge wait expired. Answer with the manager's lab IP for every consumer; host-side processes reach published ports through it identically.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every DinD node is attached to a second network by the bootstrap cache connect, so the embedded DNS returns two A records per node name in undefined order; workers that resolved the manager onto the cache network could not reach the registry bound to the lab IP and rejected locally built images with 'No such image', pushing all replicas onto the manager. Pin a fixed lab ipv4_address per node and every node name via extra_hosts (files-first resolution wins over DNS); the subnet and IPs live in default.env as the INFINITO_SWARM_* SPOT consumed by compose interpolation and the topology exports. Proven by A/B: the identical friendica variant-2 swarm job went from worker rejects to fully green including the drain test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The DR drill ran in every variant job because its gate checked the
job-local plan index, which is 0 in each of the single-variant CI jobs;
gate it on the absolute round index so the destructive drill and the
backup host deploy run once per app, in the variant-0 job. Guard the
matrix steps with a disk watchdog that aborts visibly below 6G free
instead of letting the Actions Worker die silently on ENOSPC with no
artifacts. Cap node journald (256M) and DiD container logs (10m x 2),
which were unbounded across five nodes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Image builds on the swarm test nodes inherit the node /etc/hosts, and
build containers carry neither the package-cache frontend CA nor any
skip-cert config, so every HTTPS-defaulting hijack broke them with
untrusted-cert errors (openresty apk via dl-cdn.alpinelinux.org,
simpleicons npm install via registry.npmjs.org). Align the node wiring
with the compose-mode build design: hijack only the HTTP apt mirrors
and keep apt.list, dropping the pip/npm hijacks and their configs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Node image now builds FROM the distro infinito image (python, ansible, CLI baked in) instead of installing a per-node .deb, so there is no per-node bootstrap or timeout; the swarm CI job depends on the distribution build. Move compose/swarm.yml to compose/swarm/compose.yml and relocate the swarm matrix orchestrator from cli.administration.deploy.swarm to utils.tests.swarm (test code out of the shipped CLI).
Back the NFS export root with a disk-backed named volume instead of a RUNNER_TEMP bind: local-act RUNNER_TEMP resolves onto tmpfs and ganesha FSAL_VFS cannot export a tmpfs, so it loaded zero exports and the controller mount got ENOENT. Add optional compose/swarm/cache.override.yml wiring swarm nodes to the package-cache frontend when present, leaving CI without a cache unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The node image baked one machine-id into every container, so the pull
target directory backup/<machine-id>/ was shared by all providers and
the per-provider rsyncs raced each other with --delete (DR drill step
4/9, run 29186471592 job 86634020663). Emptying /etc/machine-id makes
systemd generate a distinct id on first boot; real hosts always had
their own.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The DiD nodes boot degraded, so systemd never reaches the point where
systemd-user-sessions removes /run/nologin and pam blocked every
non-root ssh login with 'System is booting up'. The bootstrap play now
starts the unit on every node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The provisioner copied the manager host_vars only to the workers, so the
service-loader preload on the nfs and backup nodes merged a defaults-only
applications dict (credentials lookups failed), and the bootstrap play
seeded /etc/hosts only on manager and nfs, so the backup node's msmtp
healthcheck died with EX_NOHOST. Both nodes now get the host_vars copy
and the app-domain hosts entries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
