5aa7506e36 traced the enumeration bug: a veth that disappears between the RTM_GETADDR snapshot and the SIOCGIFFLAGS ioctl makes the AF_INET6 pass return 0, and network.c:836 then skips the AF_INET pass entirely. That commit removed bind-dynamic, which took the listener collector out of the blast radius. It left the other consumer of daemon->interface_addrs in place.
forward.c:1669 opens a rejection gate under OPT_LOCAL_SERVICE and walks the same list. With the IPv4 half missing, every IPv4 source is rejected, loopback included, and the once-only warning at forward.c:1695 makes a permanent outage look like a single incident. Penpot job 105068912535 and MediaWiki job 105068912565 each lost one lab node this way: PID 332 ran with --local-service and is the PID that logged the rejection; the sampler shows 179 consecutive failures over 47 minutes with the socket still bound.
Debian's init-system-common appends --local-service unconditionally, so no packaging knob removes it. option.c:6240 resets it when any of if_names, if_except, if_addrs or authserver is set, no matter which config file carried the directive. interface=* fills if_names and, through wildcard_match, still admits every interface including a docker_gwbridge that does not exist at render time. bind-interfaces and bind-dynamic are absent from that predicate and would not have helped.
Measured against the rendered template with --local-service on argv as Debian appends it: the DNS service limited to local subnets line is gone and dig still answers from the lab zone. The nodes run Debian and Ubuntu, so the task name's previous claim about fedora and centos, and about local-service implying bind-interfaces, was wrong twice over and is replaced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32118850138 lost the :53 listener, so the sampler counts listeners.
Run 32538854067 lost DNS with the listener still bound: sockets-udp.txt
showed dnsmasq holding 0.0.0.0:53 while every forwarded query timed out,
and the sampler reported that hour as healthy.
Each tick now also records what a glibc caller gets back — for the zone
this resolver owns, derived from its own address= line, and for one name
it must forward — with the latency that caller paid. The rescue collector
probes the in-zone name too: probing external names alone cannot separate
"forwarding is gone" from "dnsmasq is gone".
The sampler now also runs on the swarm lab DNS nodes; it only ever ran in
the compose path, which is why the Magento artifact carried no sampler log
at all. Its location comes from group_vars FILE_DNS53_SAMPLER_LOG and is
required at every call site, so a run cannot quietly collect nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
address=/<zone>/<ipv4> covers only A records, so every AAAA lookup for
the lab zone was forwarded to the public resolvers. When the manager
node lost its upstream mid-run, glibc callers (Playwright request
contexts) paid the full resolver timeout per lookup: 20s each, which
blew the 30s request budget and the 60s persona budget.
local=/<zone>/ makes dnsmasq authoritative for the zone, so AAAA is
answered as NODATA from config instead of leaving the lab. Same idiom
the tor onion template already uses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without the bindings the ansible apt module installs them itself mid-play, and the dist-upgrade that follows deadlocks: the module spins in user code while apt-get blocks writing to a full stderr pipe. It ran that way on all five nodes for three hours.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The five DinD lab nodes run privileged with cgroupns=host, a writable host cgroupfs and seccomp/apparmor off, and carried no resource limit of any kind. A runaway node therefore competes with the Actions runner agent itself, and the platform reports that only as 'lost communication', never as an OOM.
Only pids_limit for now. A memory cap needs a measured per-node peak, and every measurement available is a failure-time snapshot, i.e. a lower bound: 4.4 GiB on the busiest node of a variant-0 drill. Sizing the cap off that risks OOM-killing a legitimate container instead of the runaway node. The pid ceiling has no such problem: 8192 against a measured 981 is 8.3x.
The value is a static default.env key on purpose. A handler-derived key is absent from .env on a fresh runner, and the topology export step runs before the install step that generates it, so compose interpolation would have failed every swarm job before anything deployed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A debian:bookworm workspace job died on `apt-get update`: the mounted
sources list carried the Ubuntu entry too, Nexus had auto-blocked
archive.ubuntu.com, and apt fails the whole update on one bad source.
Three holes, three plugs:
- One sources list per INFINITO_DISTRO under compose/package-cache/apt/,
selected by the compose overrides. A Debian runner no longer fetches
an Ubuntu suite it has no use for. The non-apt distros get an empty
file so the bind mount has a source to point at.
- autoBlock off. While a remote is auto-blocked Nexus answers `404
Remote Auto Blocked` to every client instead of falling back to the
copy it already has cached.
- Mirror failover in the frontend. Every apt suite gets a second proxy
repo on an independent upstream; a 5xx from the primary re-runs the
request against it. Nexus has no group repository type for apt, so
the retry lives in nginx.
Measured on a live Nexus 3.91.1: dead remote without a cached copy
answers 502, with a cached copy 200 regardless of metadataMaxAge. The
frontend serves 149 kB of real bookworm `Release` from the mirror repo
when the primary is dead, and the new regex locations produce upstream
request lines byte-identical to the `rewrite … break` form they replace,
including `%3a` epochs in pool paths.
INFINITO_CACHE_PACKAGE_APT_LIST is gone; the mount interpolates
INFINITO_DISTRO directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In job 91983553540 all five lab nodes derived the same pool name,
infinito_ci_unknown, and the run died on `cannot destroy
'infinito_ci_unknown': pool is busy`. The name came from `hostname`, which
returns non-zero inside the node containers, so the `|| echo unknown` fallback
answered for every one of them. One node won the race and created the pool; the
others found it, took it for a leftover, and tried to tear down a pool a
sibling was actively using.
234c1158c already fixed this collision class three days ago by deriving the
name from `hostname` instead of a constant, and its message named the zfs
exclusion in resolve.sh as the only thing keeping the collision out of sight.
That fix was never exercised, because the exclusion stayed up until ed1121053
removed it - and it does not work, because the call it rests on fails in the
containers it was written for. Guessing the identity is the defect; the
playbook has known it all along as inventory_hostname, and ci/one.sh knows its
own deploy mode. Both now state it, as a required fourth argument with no
default, the way SIZE is already stated. verdict() stops guessing too: every
row it wrote into the job summary said `unknown`, which is why the summary was
never worth reading.
The second half is independent of the naming. reclaim_stale_pool ran
`zpool destroy -f` as a bare command under `set -euo pipefail`, the only
failure path in this script that bypassed decline(). A pool that refuses to be
destroyed is by definition not a leftover - something holds it - so that case
now declines through the usual contract: a drawn pick falls back and reports,
a stated one fails loudly as required-but-unavailable. A reclaim can no longer
be more destructive than the state it exists to clean up.
Verified over the real invocation path rather than a model of it: an
ansible-playbook run using the same `script:` task and docker connection as
compose/swarm/playbook.yml, against three containers in one play, has each node
address its own pool - infinito_ci_race-mgr-01, -wrk-01, -nfs-01 - with
failed=0 throughout. A stubbed-pool harness covers the paths that need a kernel
module this machine does not have: a sibling holding the pool declines with rc
0 instead of ending the run, the same case with a stated pick exits 1 through
the contract, a genuine leftover is still destroyed, and a missing identity is
a hard error. The earlier replacement harness (ext4 recognised, btrfs
replacing it, no loop leaked) still passes against the stricter signature.
Pool names now carry dashes, which zfs_namecheck.c accepts; at 42 characters
they sit far inside the 240 limit and match no reserved prefix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dockerd's zfs graphdriver resolves `zfs` on PATH and opens /dev/zfs before it
accepts a data root (moby daemon/graphdriver/zfs/zfs.go:54,59), so the userland
has to sit in the image, not on the host. Only the apt images carried it, which
is why resolve.sh's userland_for() kept zfs out of every entry containing arch,
centos or fedora - that is every automatic swarm sweep, so the zfs data root and
the zfs branch of the backup launcher were never once exercised. Run
30781723102 shows it: "zfs left out: not every distro of this entry carries the
userland".
scripts/install/zfs.sh joins the three existing install scripts in the image
build and takes a different route per distro, each one run in a real container
of the matching base image before landing here:
debian trixie 2.3.2 zfsutils-linux, contrib enabled first
ubuntu 26.04 2.4.1 zfsutils-linux
arch 2.4.3 the AUR PKGBUILD, which configures --with-config=user
fedora 44 2.4.3 zfsonlinux repo plus a dummy zfs-kmod provider
centos 10 2.2.10 the same, from the epel path
The dummy is what keeps the RPM route usable: the `zfs` package carries an
ungated `Requires: %{name}-kmod`, so a plain `dnf install zfs` resolves through
zfs-dkms and installs 389 extra packages and 1.44 GB of kernel and toolchain to
build a module against a kernel the container never boots. Providing zfs-kmod
from an empty package keeps the same install at 60 packages with no kernel
package at all, and the transaction is checked afterwards so a regression fails
the build instead of bloating the image. The repository is removed again once
the userland is in, because the dummy pins one version and an upgrade would
otherwise re-resolve the kmod at deploy time.
Version drift against the host module is not a concern: zpool create enables a
feature only when the loaded module reports it through /sys/module/zfs/features
.pool (cmd/zpool/zpool_main.c, module/zcommon/zfeature_common.c), and libzfs
compares no versions at all (lib/libzfs/os/linux/libzfs_util_os.c:123-134), so a
newer userland degrades rather than breaking.
With every image carrying it, userland_for() and every_distro_carries() have
nothing left to decide and go. Two defects that only became reachable once zfs
could be drawn are fixed in the same move:
* /dev/zfs is a misc device, so the old lookup in /proc/devices could never
find its major and the node was never created. A node container's /dev is
populated at creation and udevd is masked, so nothing else fills the gap
either: the first bring-up of every job would have declined, blaming a
module that modprobe had just loaded. The minor now comes from /proc/misc.
* Pools live in the kernel the lab nodes share with the runner, while the
teardown only removes containers, and the node hostnames repeat across the
distro sweep. From the second distro on, zpool create hit the previous
iteration's pool. It is destroyed and its loop detached first.
The node image drops its own apt-only zfs branch and instead masks the six boot
units the userland enables by preset, which on a node can only fail: the pool is
created after boot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every lab node got an identical 30 GiB loop device because docker_dataroot.sh
defaulted SIZE to 30G and no caller ever stated it. The swarm manager hosts the
in-cluster registry, ~60 containers, and - by deliberate placement, since /data
is NFS-unsafe - the SeaweedFS object store, whose sizing filter claims 14 GiB
of slots plus a 2 GiB minFreeSpace floor. The idle nfs-server node got the same
30 GiB and used 1% of it.
In run 30859490335 the manager's device stood at 30G/30G (98%), SeaweedFS
crossed its floor at 02:04:45 and marked all 14 volumes unwritable, permanently.
Nextcloud in variant 0 stores objects in that SeaweedFS via OBJECTSTORE_S3_*,
so every authenticated page render performed an object write that failed as a
10-attempt assign ladder - 15 requests, 15 bursts of exactly 10, burst end
matching the nginx 500 to the second. That is the "🐝 web-app-nextcloud 0"
failure: 5 addon specs, 3 attempts each, all dead on the post-login page.
The default is gone: SIZE is now a required argument and a missing value dies
naming the call site. The swarm playbook states it per node - 60G for the
manager, 30G for the rest. 60G covers the observed demand (>=31 GiB: a full
device plus the denied writes) with headroom; the image is sparse, so the
raised ceiling costs only what is actually written, and the runner-side
watchdog at 6 GiB free remains the backstop. The compose and host workflows
state their unchanged 30G explicitly.
Not addressed here: what fills the first 28 GiB (registry + image layers +
volumes composition is unmeasured - the rescue artifact holds exactly one df
snapshot), and the drill-side shadow restore that supplied the last ~1 GiB;
both are separate changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docker_dataroot.sh installs zfsutils-linux at runtime, so the zfs pick depends
on the package cache being up. In run 30781723102 that decided the two swarm
drills differently: the cache stack was removed at 06:39:49 and rebuilt at
07:18:25, so the mariadb drill had no zfsutils candidate at all and declined,
while the postgres drill got one and reached zpool create. Installing it at
build time removes that dependency.
Debian keeps zfsutils-linux in contrib, which the base image does not enable, so
the branch adds a contrib source first - only for ID=debian, since Ubuntu ships
it in its default components. A failed update after adding the source removes
the source again rather than leaving a broken repo line behind.
The install is deliberately separate from the mandatory packages and
non-fatal. Node images for centos, fedora and arch have no zfs userland either
and resolve.sh's userland_for() already accounts for that, so a missing zfs
userland is a supported state - whereas folding it into the mandatory install
would let a packaging error break the node image for all five distros. A failure
now prints to stderr and the run declines the pick visibly.
Verified against the real bookworm contrib index (deb.debian.org, HTTP 200):
zfsutils-linux 2.1.11-1+deb12u1 exists there, all four libzfs libraries are
Depends rather than Recommends so --no-install-recommends keeps them, and the
only Recommends are zfs-modules|zfs-dkms and zfs-zed, which we do not want - the
module comes from the host via /lib/modules. Control flow exercised with a
stubbed apt-get across six combinations of distro, package availability and
update failure; every one leaves the build at rc 0.
Not covered here: whether a 2.1 userland can drive the runner's newer host
module. zfs-version.txt and modules.txt from 264556a31 answer that next run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
18abce480 added kmod so the data-root setup can run its one explicit `modprobe
zfs`. It also handed a working loader to the four modprobe@ instances systemd
pulls at every node boot - configfs, drm, efi_pstore and fuse - which until now
failed on the missing binary, as every node's journal in
rescue-diagnostics-workspace-debian-bookworm shows. They would have started
loading host modules from a privileged container, on every node of every swarm
job in all five distros.
Masking the template keeps them exactly as dead as they are today and leaves the
loader available to the caller that needs it, so the behaviour change of
18abce480 is confined to the one site it was meant for. It sits with the
existing masks for systemd-resolved and systemd-udevd.
The image was not built; that the mask covers every instance of the template is
read from systemd's behaviour, not measured. The next run shows it: the units
report masked instead of failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node image ships no module loader, so `modprobe zfs` in
scripts/tests/deploy/utils/filesystem/docker_dataroot.sh was a no-op whose error
was additionally discarded by `2>/dev/null || true`. Every node's own systemd
says so at boot, in rescue-diagnostics-workspace-debian-bookworm of run
30781723102:
(modprobe)[27]: modprobe@configfs.service: Unable to locate executable
'/sbin/modprobe': No such file or directory
The module itself is reachable: the nodes run privileged with /lib/modules
bind-mounted read-only, and their df confirms it. What was missing is the
loader, not the module. zfs-load-module.service is installed and enabled in the
image and sits inactive for the same reason.
This does not by itself make the debian lane apply zfs. `The ZFS modules are not
loaded.` in job 91588086713 came from libzfs at `zpool create`, not from the
suppressed modprobe - the kernel ring buffer shows five 30 GiB loop attachments
at 07:37:32, i.e. the script was past the /dev/zfs gate. And a Debian bookworm
node only has a zfsutils-linux candidate while the package cache stack is up:
the cache was removed at 06:39:49 and rebuilt at 07:18:25, which is why the
mariadb drill declined at the userland check while the postgres drill reached
zpool create. That cache dependency is untouched here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dnsmasq 2.93 with bind-dynamic enumerates IPv6 before IPv4 (network.c:836-875). A veth that disappears between the RTM_GETADDR snapshot and the SIOCGIFFLAGS ioctl makes that pass return 0, which skips the IPv4 pass entirely while the interface-record collector still runs and releases every listener not marked found. The v6 addresses already dumped survive; the whole IPv4 set does not. Recovery would come from the next netlink address event, so the loss becomes permanent when the churn stops on a release.
That is what killed web-app-matrix 0: the manager kept only [::1]:53 and eight fe80 link-locals while ip -4 addr still showed all four addresses, so ghcr.io stopped resolving and the image pull failed ten times. Both workers, same image and same config, held the complete set on the same fds. daemon.json points container DNS at 172.17.0.1, so every container on that node lost resolution at the same instant.
Reproduced on the same dnsmasq version: 889 full IPv4 release/recreate cycles under a veth storm, with ::1 never released once; the same storm without bind-dynamic produced no listener events at all. Without the directive dnsmasq binds one wildcard pair and reads the arrival interface from IP_PKTINFO, and wildcard listeners carry iface == NULL, which the collector skips outright.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role runs ganesha at -N NIV_EVENT, and roughly eighteen init steps between the CLIENT commits and the listener log below that level, so the silence in run 30699990912 marked the edge of visibility rather than the fault. Raise INIT and DISPATCH under MODE_DEBUG: nfs_Init_svc(), which binds 2049, logs under COMPONENT_DISPATCH, so INIT alone would have missed exactly the step in question.
Add a userspace backtrace beside the kernel stack dump, which only ever says futex_do_wait. eu-stack names the function holding the lock; elfutils goes into the lab image rather than the role, since a production NFS server should not carry a debugger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The summary named the filesystem that was picked, never the one the run got.
A host that declines - no root, missing tooling, no /dev/zfs - still deploys,
and nothing recorded on what, so a red run could not be reproduced.
docker_dataroot.sh now ends every path with a verdict that measures the data
root with `stat -f -c %T` and reports applied, declined, already-prepared,
unchanged or required-but-unavailable. Compose and host write it straight into
the step summary; swarm runs the script inside each node, where that variable
does not exist, so the playbook lifts the verdict line into the deploy log and
report.sh aggregates it afterwards.
`hostname` is resolved with a fallback: it is not installed by default on the
RedHat family, and under `set -euo pipefail` a bare command substitution would
have turned the decline path - which exits 0 - into a hard failure.
test-deploy-host.yml gains the `filesystem` input. single-node has been passing
it since the per-matrix selection landed, and a reusable workflow that receives
an undeclared input fails the whole run with a startup_failure whose only trace
is a banner on the run page. The single-node line was skipped in the run that
would have shown it. A lint now pairs every local `uses:` with the callee's
declared inputs, and also demands that required inputs without a default are
supplied.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backup snapshot mode only engages on btrfs or zfs, so on CI - where the
data root is whatever the runner or the node image happens to provide - it
was never exercised at all. Nothing proved the snapshot path beyond unit
tests and a synthetic probe.
`entry-manual` gains a `filesystem` choice that reaches both deploy modes.
Empty means each matrix entry draws its own, so coverage spreads over runs
instead of pinning one path forever; the pick and its origin go to the step
summary, because reproducing a red run means re-stating it verbatim. The
draw is restricted to what every distribution of that entry can deliver -
centos stream has no btrfs at all, and zfs is a third-party repository
outside debian and ubuntu - so a run never dies on a packaging gap. A
stated filesystem skips that restriction: it is an instruction, and a host
that cannot deliver it fails loudly rather than deploying on something
else.
docker_dataroot.sh formats a sparse loop image, mounts it at /mnt/docker-fs
and carves the data root inside it, then points the daemon there through
daemon.json - merged, not overwritten, so the registry-cache entry
survives. It runs on the runner for compose and inside each node for swarm,
before anything is pulled, and installs the filesystem tooling it needs.
Two container facts it has to work around: a privileged container's /dev is
a fresh tmpfs, so neither /dev/loopN nor /dev/zfs exists there and both are
mknod'ed from /proc/devices before use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The LUKS step of the DR drill died on `The file /dev/mapper/usbdrill does
not exist and no size was specified.` -- from mkfs, not from cryptsetup,
because luksOpen had exited 0. The node did have a udev, and it had
created the mapper entry: as a symlink to ../dm-1. Modern udev no longer
mknods, it only manages symlinks, and the /dev/dm-N target it points at is
a devtmpfs node that a container's tmpfs /dev never receives. Without a
udev libdevmapper creates the node itself, which is why the debian nodes
passed -- their image ships no udevd at all, while arch cannot install
systemd without one.
Masking udevd in the node image removes the whole class rather than one
call site. DM_DISABLE_UDEV rides on the node containers instead of the
script because the drill opens the volume a second time through the
recover CLI in a separate docker exec, which would not inherit an export.
The block-device check then fails loudly with both prerequisites named,
and closes the mapping first so an abort cannot leak it onto the runner --
device-mapper is not namespaced and every distro round shares the VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backup DR drill could not pull from the manager on centos and fedora:
the backup user's ssh died with "Connection closed", exit 255. Pubkey auth
had actually succeeded -- sshd only calls the PAM account phase once
authenticated -- and that phase failed because the RHEL parents ship
/etc/shadow mode 0000. Their setuid-root unix_chkpwd then needs
CAP_DAC_READ_SEARCH, which the host's unix-chkpwd AppArmor profile denies
right through the container: profiles attach by executable path, so the
lab nodes hit it despite already running privileged with
apparmor=unconfined.
chmod u+rw adds only the owner bit, which is what arch ships and what a
setuid helper needs. A hard 0600 would strip the group bit debian and
ubuntu rely on for their setgid-shadow helper and break the distros that
work today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node image layers on a pinned pkgmgr-arch parent that can be weeks old, and `pacman -Sy <pkg>` against it is the partial upgrade arch upstream documents as unsupported. Two failure modes: the parent's archlinux-keyring predates any packager key added since, so a package signed by one aborts the RUN with unknown trust; and -Sy without u installs fresh binaries next to the image's older libraries, which pacman cannot catch because it matches SONAMEs, not symbol versions (cryptsetup declares libcrypto.so=3-64 and installs happily onto an older openssl).
Refreshing archlinux-keyring first and then running a full -Syu --needed fixes both. The cache wipe matches what the apt and dnf branches already do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d9a4498b3 collapsed eleven CI steps into one, which broke the assumption every handover in the drill rested on: a $GITHUB_ENV write only comes back in the NEXT step. Inside one step nobody sees it. matrix.py reads os.environ["MGR_IP"], so every swarm job died with KeyError before the first deploy round; 05 -> 06 -> 07 passed INITIAL_NODE/NEW_NODE the same way and would have died next.
The node IPs now come from the topology SPOT: compose.yml already pins every node to those INFINITO_SWARM_*_IP keys, so the docker-inspect capture only read back what compose had set. It goes, together with the $GITHUB_ENV publish and lab_net. lab_domain_target reads the env instead, guarded by an assert task because lookup('env') yields '' for a missing key and an empty dnsmasq `address=/domain/` is valid syntax meaning NODATA -- the failure would have surfaced 30 retries later as a DNS problem.
INITIAL_NODE/NEW_NODE move to a SWARM_DRILL_ENV file, created fresh per distro so the previous distro's node cannot leak into the next drain. The out-of-drill callers of _context.sh keep working through a visible guard; both writers stay strict via :?.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fedora and CentOS Stream ship `local-service` in the stock /etc/dnsmasq.conf. It implies bind-interfaces, which dnsmasq refuses next to the lab config's bind-dynamic, so every DNS node died at 'cannot set --bind-interfaces and --bind-dynamic'. A drop-in cannot fix it: conf-dir files are read after the main config and a boolean already set there cannot be taken back. Writing the main config instead gives all five distros the same option set. On arch it also starts working at all -- the package ships no /etc/dnsmasq.d and its unit passes no --conf-dir, so the drop-in was a silent no-op there.
The restart timeout goes to 120s because arch's unit is the only Type=dbus one: systemd waits its full 90s DefaultTimeoutStartSec for the bus name, so the old 60s ansible timeout fired first and turned every arch-side DNS fault into a bare timeout with no journal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The simulated cluster's DNS config lived as an inline `copy: content:` block in the lab playbook. It moves to compose/swarm/templates/dnsmasq.conf.j2 so the template module can render it; the playbook switch follows separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible resolves task includes relative to the including playbook, so the
playbook_dir prefix was never needed — and it hid the file from the
include-existence check, which the nocheck directive then papered over.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DinD nodes fetched every image straight from upstream while the warm
host cache sat unused beside them; only the DiD container was wired to
it. CA and proxy drop-in are now placed during bootstrap, before the
deploy installs docker, so the daemon picks them up on its first start.
NO_PROXY carries the node names because the in-cluster registry is
addressed by hostname and the proxy intercepts pulls only. Without a
local cache stack, as in CI, the CA is never mounted and the block is
skipped entirely.
Cut the prometheus swarm step from a 5h45 timeout to 4h03 green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-node dnsmasq answered the app-domain wildcard with 127.0.0.1 on the manager; node dockerd hands that resolver to every task container, where loopback is the task's own netns, so any healthcheck reaching for a published service port (friendica's msmtp send to mail.<domain>:465 with email enabled) got connection-refused although the port was ingress-published and mailu healthy - manager-placed replicas could never become healthy and the stack sat at 0/3 until the converge wait expired. Answer with the manager's lab IP for every consumer; host-side processes reach published ports through it identically.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every DinD node is attached to a second network by the bootstrap cache connect, so the embedded DNS returns two A records per node name in undefined order; workers that resolved the manager onto the cache network could not reach the registry bound to the lab IP and rejected locally built images with 'No such image', pushing all replicas onto the manager. Pin a fixed lab ipv4_address per node and every node name via extra_hosts (files-first resolution wins over DNS); the subnet and IPs live in default.env as the INFINITO_SWARM_* SPOT consumed by compose interpolation and the topology exports. Proven by A/B: the identical friendica variant-2 swarm job went from worker rejects to fully green including the drain test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The DR drill ran in every variant job because its gate checked the
job-local plan index, which is 0 in each of the single-variant CI jobs;
gate it on the absolute round index so the destructive drill and the
backup host deploy run once per app, in the variant-0 job. Guard the
matrix steps with a disk watchdog that aborts visibly below 6G free
instead of letting the Actions Worker die silently on ENOSPC with no
artifacts. Cap node journald (256M) and DiD container logs (10m x 2),
which were unbounded across five nodes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Image builds on the swarm test nodes inherit the node /etc/hosts, and
build containers carry neither the package-cache frontend CA nor any
skip-cert config, so every HTTPS-defaulting hijack broke them with
untrusted-cert errors (openresty apk via dl-cdn.alpinelinux.org,
simpleicons npm install via registry.npmjs.org). Align the node wiring
with the compose-mode build design: hijack only the HTTP apt mirrors
and keep apt.list, dropping the pip/npm hijacks and their configs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Node image now builds FROM the distro infinito image (python, ansible, CLI baked in) instead of installing a per-node .deb, so there is no per-node bootstrap or timeout; the swarm CI job depends on the distribution build. Move compose/swarm.yml to compose/swarm/compose.yml and relocate the swarm matrix orchestrator from cli.administration.deploy.swarm to utils.tests.swarm (test code out of the shipped CLI).
Back the NFS export root with a disk-backed named volume instead of a RUNNER_TEMP bind: local-act RUNNER_TEMP resolves onto tmpfs and ganesha FSAL_VFS cannot export a tmpfs, so it loaded zero exports and the controller mount got ENOENT. Add optional compose/swarm/cache.override.yml wiring swarm nodes to the package-cache frontend when present, leaving CI without a cache unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The node image baked one machine-id into every container, so the pull
target directory backup/<machine-id>/ was shared by all providers and
the per-provider rsyncs raced each other with --delete (DR drill step
4/9, run 29186471592 job 86634020663). Emptying /etc/machine-id makes
systemd generate a distinct id on first boot; real hosts always had
their own.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The DiD nodes boot degraded, so systemd never reaches the point where
systemd-user-sessions removes /run/nologin and pam blocked every
non-root ssh login with 'System is booting up'. The bootstrap play now
starts the unit on every node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The provisioner copied the manager host_vars only to the workers, so the
service-loader preload on the nfs and backup nodes merged a defaults-only
applications dict (credentials lookups failed), and the bootstrap play
seeded /etc/hosts only on manager and nfs, so the backup node's msmtp
healthcheck died with EX_NOHOST. Both nodes now get the host_vars copy
and the app-domain hosts entries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each task carries a timeout sized to its I/O class (probes 10-30s,
service restart 60s, repo tar/unarchive 300s, .deb install 600s), so a
wedged node fails the play loudly instead of hanging the bootstrap step;
the systemd wait keeps its rescue diagnostics for exactly that case.
Bare integers instead of the timeout lookup because the standalone
-c docker run does not load repo plugins.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both files are bind mounts inside the node containers, so ansible's
atomic tempfile+rename fails with EBUSY and every node dropped out of
the bootstrap play. unsafe_writes lets the two tasks write in place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the per-node docker-run bring-up with a single node image
(compose/swarm/Dockerfile bakes python3/dnsmasq/openssh/rsync/cryptsetup)
declared in compose/swarm.yml, one compose up, and one ansible play
(compose/swarm/playbook.yml over ansible_connection: docker) that waits for
systemd, captures IPs, wires lab DNS and installs the .deb per node. Reshape
the harness into topology/ (naming SPOT), routine/ (01_bootstrap + the
sequenced 02-08 steps) and utils/ (clean/, collect/, shared helpers); the
workflow, matrix.py and Makefile follow the new paths. swarm init/join was
already ansible-owned (svc-swarm-node), corrected in the README diagram.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
