5aa7506e36 traced the enumeration bug: a veth that disappears between the RTM_GETADDR snapshot and the SIOCGIFFLAGS ioctl makes the AF_INET6 pass return 0, and network.c:836 then skips the AF_INET pass entirely. That commit removed bind-dynamic, which took the listener collector out of the blast radius. It left the other consumer of daemon->interface_addrs in place.
forward.c:1669 opens a rejection gate under OPT_LOCAL_SERVICE and walks the same list. With the IPv4 half missing, every IPv4 source is rejected, loopback included, and the once-only warning at forward.c:1695 makes a permanent outage look like a single incident. Penpot job 105068912535 and MediaWiki job 105068912565 each lost one lab node this way: PID 332 ran with --local-service and is the PID that logged the rejection; the sampler shows 179 consecutive failures over 47 minutes with the socket still bound.
Debian's init-system-common appends --local-service unconditionally, so no packaging knob removes it. option.c:6240 resets it when any of if_names, if_except, if_addrs or authserver is set, no matter which config file carried the directive. interface=* fills if_names and, through wildcard_match, still admits every interface including a docker_gwbridge that does not exist at render time. bind-interfaces and bind-dynamic are absent from that predicate and would not have helped.
Measured against the rendered template with --local-service on argv as Debian appends it: the DNS service limited to local subnets line is gone and dig still answers from the lab zone. The nodes run Debian and Ubuntu, so the task name's previous claim about fedora and centos, and about local-service implying bind-interfaces, was wrong twice over and is replaced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32118850138 lost the :53 listener, so the sampler counts listeners.
Run 32538854067 lost DNS with the listener still bound: sockets-udp.txt
showed dnsmasq holding 0.0.0.0:53 while every forwarded query timed out,
and the sampler reported that hour as healthy.
Each tick now also records what a glibc caller gets back — for the zone
this resolver owns, derived from its own address= line, and for one name
it must forward — with the latency that caller paid. The rescue collector
probes the in-zone name too: probing external names alone cannot separate
"forwarding is gone" from "dnsmasq is gone".
The sampler now also runs on the swarm lab DNS nodes; it only ever ran in
the compose path, which is why the Magento artifact carried no sampler log
at all. Its location comes from group_vars FILE_DNS53_SAMPLER_LOG and is
required at every call site, so a run cannot quietly collect nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In job 91983553540 all five lab nodes derived the same pool name,
infinito_ci_unknown, and the run died on `cannot destroy
'infinito_ci_unknown': pool is busy`. The name came from `hostname`, which
returns non-zero inside the node containers, so the `|| echo unknown` fallback
answered for every one of them. One node won the race and created the pool; the
others found it, took it for a leftover, and tried to tear down a pool a
sibling was actively using.
234c1158c already fixed this collision class three days ago by deriving the
name from `hostname` instead of a constant, and its message named the zfs
exclusion in resolve.sh as the only thing keeping the collision out of sight.
That fix was never exercised, because the exclusion stayed up until ed1121053
removed it - and it does not work, because the call it rests on fails in the
containers it was written for. Guessing the identity is the defect; the
playbook has known it all along as inventory_hostname, and ci/one.sh knows its
own deploy mode. Both now state it, as a required fourth argument with no
default, the way SIZE is already stated. verdict() stops guessing too: every
row it wrote into the job summary said `unknown`, which is why the summary was
never worth reading.
The second half is independent of the naming. reclaim_stale_pool ran
`zpool destroy -f` as a bare command under `set -euo pipefail`, the only
failure path in this script that bypassed decline(). A pool that refuses to be
destroyed is by definition not a leftover - something holds it - so that case
now declines through the usual contract: a drawn pick falls back and reports,
a stated one fails loudly as required-but-unavailable. A reclaim can no longer
be more destructive than the state it exists to clean up.
Verified over the real invocation path rather than a model of it: an
ansible-playbook run using the same `script:` task and docker connection as
compose/swarm/playbook.yml, against three containers in one play, has each node
address its own pool - infinito_ci_race-mgr-01, -wrk-01, -nfs-01 - with
failed=0 throughout. A stubbed-pool harness covers the paths that need a kernel
module this machine does not have: a sibling holding the pool declines with rc
0 instead of ending the run, the same case with a stated pick exits 1 through
the contract, a genuine leftover is still destroyed, and a missing identity is
a hard error. The earlier replacement harness (ext4 recognised, btrfs
replacing it, no loop leaked) still passes against the stricter signature.
Pool names now carry dashes, which zfs_namecheck.c accepts; at 42 characters
they sit far inside the 240 limit and match no reserved prefix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every lab node got an identical 30 GiB loop device because docker_dataroot.sh
defaulted SIZE to 30G and no caller ever stated it. The swarm manager hosts the
in-cluster registry, ~60 containers, and - by deliberate placement, since /data
is NFS-unsafe - the SeaweedFS object store, whose sizing filter claims 14 GiB
of slots plus a 2 GiB minFreeSpace floor. The idle nfs-server node got the same
30 GiB and used 1% of it.
In run 30859490335 the manager's device stood at 30G/30G (98%), SeaweedFS
crossed its floor at 02:04:45 and marked all 14 volumes unwritable, permanently.
Nextcloud in variant 0 stores objects in that SeaweedFS via OBJECTSTORE_S3_*,
so every authenticated page render performed an object write that failed as a
10-attempt assign ladder - 15 requests, 15 bursts of exactly 10, burst end
matching the nginx 500 to the second. That is the "🐝 web-app-nextcloud 0"
failure: 5 addon specs, 3 attempts each, all dead on the post-login page.
The default is gone: SIZE is now a required argument and a missing value dies
naming the call site. The swarm playbook states it per node - 60G for the
manager, 30G for the rest. 60G covers the observed demand (>=31 GiB: a full
device plus the denied writes) with headroom; the image is sparse, so the
raised ceiling costs only what is actually written, and the runner-side
watchdog at 6 GiB free remains the backstop. The compose and host workflows
state their unchanged 30G explicitly.
Not addressed here: what fills the first 28 GiB (registry + image layers +
volumes composition is unmeasured - the rescue artifact holds exactly one df
snapshot), and the drill-side shadow restore that supplied the last ~1 GiB;
both are separate changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The summary named the filesystem that was picked, never the one the run got.
A host that declines - no root, missing tooling, no /dev/zfs - still deploys,
and nothing recorded on what, so a red run could not be reproduced.
docker_dataroot.sh now ends every path with a verdict that measures the data
root with `stat -f -c %T` and reports applied, declined, already-prepared,
unchanged or required-but-unavailable. Compose and host write it straight into
the step summary; swarm runs the script inside each node, where that variable
does not exist, so the playbook lifts the verdict line into the deploy log and
report.sh aggregates it afterwards.
`hostname` is resolved with a fallback: it is not installed by default on the
RedHat family, and under `set -euo pipefail` a bare command substitution would
have turned the decline path - which exits 0 - into a hard failure.
test-deploy-host.yml gains the `filesystem` input. single-node has been passing
it since the per-matrix selection landed, and a reusable workflow that receives
an undeclared input fails the whole run with a startup_failure whose only trace
is a banner on the run page. The single-node line was skipped in the run that
would have shown it. A lint now pairs every local `uses:` with the callee's
declared inputs, and also demands that required inputs without a default are
supplied.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backup snapshot mode only engages on btrfs or zfs, so on CI - where the
data root is whatever the runner or the node image happens to provide - it
was never exercised at all. Nothing proved the snapshot path beyond unit
tests and a synthetic probe.
`entry-manual` gains a `filesystem` choice that reaches both deploy modes.
Empty means each matrix entry draws its own, so coverage spreads over runs
instead of pinning one path forever; the pick and its origin go to the step
summary, because reproducing a red run means re-stating it verbatim. The
draw is restricted to what every distribution of that entry can deliver -
centos stream has no btrfs at all, and zfs is a third-party repository
outside debian and ubuntu - so a run never dies on a packaging gap. A
stated filesystem skips that restriction: it is an instruction, and a host
that cannot deliver it fails loudly rather than deploying on something
else.
docker_dataroot.sh formats a sparse loop image, mounts it at /mnt/docker-fs
and carves the data root inside it, then points the daemon there through
daemon.json - merged, not overwritten, so the registry-cache entry
survives. It runs on the runner for compose and inside each node for swarm,
before anything is pulled, and installs the filesystem tooling it needs.
Two container facts it has to work around: a privileged container's /dev is
a fresh tmpfs, so neither /dev/loopN nor /dev/zfs exists there and both are
mknod'ed from /proc/devices before use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d9a4498b3 collapsed eleven CI steps into one, which broke the assumption every handover in the drill rested on: a $GITHUB_ENV write only comes back in the NEXT step. Inside one step nobody sees it. matrix.py reads os.environ["MGR_IP"], so every swarm job died with KeyError before the first deploy round; 05 -> 06 -> 07 passed INITIAL_NODE/NEW_NODE the same way and would have died next.
The node IPs now come from the topology SPOT: compose.yml already pins every node to those INFINITO_SWARM_*_IP keys, so the docker-inspect capture only read back what compose had set. It goes, together with the $GITHUB_ENV publish and lab_net. lab_domain_target reads the env instead, guarded by an assert task because lookup('env') yields '' for a missing key and an empty dnsmasq `address=/domain/` is valid syntax meaning NODATA -- the failure would have surfaced 30 retries later as a DNS problem.
INITIAL_NODE/NEW_NODE move to a SWARM_DRILL_ENV file, created fresh per distro so the previous distro's node cannot leak into the next drain. The out-of-drill callers of _context.sh keep working through a visible guard; both writers stay strict via :?.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fedora and CentOS Stream ship `local-service` in the stock /etc/dnsmasq.conf. It implies bind-interfaces, which dnsmasq refuses next to the lab config's bind-dynamic, so every DNS node died at 'cannot set --bind-interfaces and --bind-dynamic'. A drop-in cannot fix it: conf-dir files are read after the main config and a boolean already set there cannot be taken back. Writing the main config instead gives all five distros the same option set. On arch it also starts working at all -- the package ships no /etc/dnsmasq.d and its unit passes no --conf-dir, so the drop-in was a silent no-op there.
The restart timeout goes to 120s because arch's unit is the only Type=dbus one: systemd waits its full 90s DefaultTimeoutStartSec for the bus name, so the old 60s ansible timeout fired first and turned every arch-side DNS fault into a bare timeout with no journal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible resolves task includes relative to the including playbook, so the
playbook_dir prefix was never needed — and it hid the file from the
include-existence check, which the nocheck directive then papered over.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DinD nodes fetched every image straight from upstream while the warm
host cache sat unused beside them; only the DiD container was wired to
it. CA and proxy drop-in are now placed during bootstrap, before the
deploy installs docker, so the daemon picks them up on its first start.
NO_PROXY carries the node names because the in-cluster registry is
addressed by hostname and the proxy intercepts pulls only. Without a
local cache stack, as in CI, the CA is never mounted and the block is
skipped entirely.
Cut the prometheus swarm step from a 5h45 timeout to 4h03 green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-node dnsmasq answered the app-domain wildcard with 127.0.0.1 on the manager; node dockerd hands that resolver to every task container, where loopback is the task's own netns, so any healthcheck reaching for a published service port (friendica's msmtp send to mail.<domain>:465 with email enabled) got connection-refused although the port was ingress-published and mailu healthy - manager-placed replicas could never become healthy and the stack sat at 0/3 until the converge wait expired. Answer with the manager's lab IP for every consumer; host-side processes reach published ports through it identically.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Node image now builds FROM the distro infinito image (python, ansible, CLI baked in) instead of installing a per-node .deb, so there is no per-node bootstrap or timeout; the swarm CI job depends on the distribution build. Move compose/swarm.yml to compose/swarm/compose.yml and relocate the swarm matrix orchestrator from cli.administration.deploy.swarm to utils.tests.swarm (test code out of the shipped CLI).
Back the NFS export root with a disk-backed named volume instead of a RUNNER_TEMP bind: local-act RUNNER_TEMP resolves onto tmpfs and ganesha FSAL_VFS cannot export a tmpfs, so it loaded zero exports and the controller mount got ENOENT. Add optional compose/swarm/cache.override.yml wiring swarm nodes to the package-cache frontend when present, leaving CI without a cache unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The DiD nodes boot degraded, so systemd never reaches the point where
systemd-user-sessions removes /run/nologin and pam blocked every
non-root ssh login with 'System is booting up'. The bootstrap play now
starts the unit on every node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The provisioner copied the manager host_vars only to the workers, so the
service-loader preload on the nfs and backup nodes merged a defaults-only
applications dict (credentials lookups failed), and the bootstrap play
seeded /etc/hosts only on manager and nfs, so the backup node's msmtp
healthcheck died with EX_NOHOST. Both nodes now get the host_vars copy
and the app-domain hosts entries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each task carries a timeout sized to its I/O class (probes 10-30s,
service restart 60s, repo tar/unarchive 300s, .deb install 600s), so a
wedged node fails the play loudly instead of hanging the bootstrap step;
the systemd wait keeps its rescue diagnostics for exactly that case.
Bare integers instead of the timeout lookup because the standalone
-c docker run does not load repo plugins.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both files are bind mounts inside the node containers, so ansible's
atomic tempfile+rename fails with EBUSY and every node dropped out of
the bootstrap play. unsafe_writes lets the two tasks write in place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the per-node docker-run bring-up with a single node image
(compose/swarm/Dockerfile bakes python3/dnsmasq/openssh/rsync/cryptsetup)
declared in compose/swarm.yml, one compose up, and one ansible play
(compose/swarm/playbook.yml over ansible_connection: docker) that waits for
systemd, captures IPs, wires lab DNS and installs the .deb per node. Reshape
the harness into topology/ (naming SPOT), routine/ (01_bootstrap + the
sequenced 02-08 steps) and utils/ (clean/, collect/, shared helpers); the
workflow, matrix.py and Makefile follow the new paths. swarm init/join was
already ansible-owned (svc-swarm-node), corrected in the README diagram.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
