01_bootstrap.sh pins the five swarm nodes into the shared cache network
at 200 + INFINITO_INSTANCE * 5. That band fits a /24 only for instances
0 to 10; from instance 11 on the script aborted with "pushes the node
band past ...254", so no worktree past slot 10 could start a swarm
cluster while the package cache runs. Five nodes per slot cannot fit
255 slots in one /24 with any fixed formula.
Instances whose band fits keep their pinned addresses, so the primary
checkout and CI behave as before. Above that, the nodes join the cache
network without --ip and Docker assigns them from its pool (the upper
half of the subnet), the same way every worktree stack container joins
it. Nothing in the repository reads the nodes' cache-network addresses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`shfmt.ensure()` returned as soon as any `shfmt` was on PATH and tried
the distribution package before the release pinned in
roles/sys-lint/meta/services.yml. The Debian dev container therefore
formats with its packaged 3.8.0 (/usr/bin/shfmt), Fedora with its packaged
3.7.0, the CentOS one with the pinned v3.14.1 (/usr/local/bin/shfmt), and
they disagree on
scripts/tests/deploy/swarm/utils/unmount/nfs_mounts.sh: every gate in a
CentOS stack moved the `then` of the heredoc `if` below the terminator.
ensure() now follows ruff's installer: it reads `shfmt --version`, keeps
the binary only when it reports the pinned release, and otherwise installs
the release binary into /usr/local/bin. It fails when PATH still resolves
another version afterwards instead of formatting with it. The system
package path is gone.
nfs_mounts.sh is committed in the pinned release's format (byte-identical
to the v3.14.1 output from the CentOS container), so the first gate on any
distro leaves the tree clean.
The install-lint stamp now also depends on roles/sys-lint/meta/services.yml,
the pin file of actionlint, hadolint and shfmt, so a bumped pin reaches
environments whose stamp predates it; until now only pyproject.toml and
scripts/install/lint.sh invalidated it.
Fixes#1155
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
diagnostics.sh runs under a 900-second deadline that the sweep deadline
can shorten further. The package manager section came last, after the NFS,
ganesha backtrace and controller probes, so a cut-off run would have lost
exactly the log this collection exists for. It now runs first.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Six Fedora jobs of main runs 39494396920 and 39590304501 (Postmarks#0,
OpenBao#1, Jenkins#1, Magento#0, Matrix#2, OpenResty#0) died in dnf with
`Failed to download packages`. Ansible's dnf5 module reports nothing
more: `failures: []`. Its file logger writes the download errors, with
mirror, URL and status, at debug level to /var/log/dnf5.log, but no
rescue bundle collects that file, so the cause of five of them stays
unknown.
The swarm stack diagnostics now append the last 3000 lines of
/var/log/dnf5.log and, for dnf4 hosts, /var/log/dnf.librepo.log of every
node. A node without them records the `tail` error and the capture goes
on.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On ubuntu-26.04 runners the swarm deploy fails at svc-storage-nfs-client's
probe-write, right after a successful probe-mount, on every client node:
`Destination directory /run/svc-storage-nfs-client/probe does not exist`.
The rescue bundles cannot say why: ganesha writes to
/var/log/ganesha/ganesha.log, which the swarm diagnostics never collected,
and its NFS_V4 component logged nothing below EVENT.
Debug runs now log NFS_V4 at DEBUG, and the diagnostics append the tail
of ganesha.log to the nfs-boundary section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On swarm nodes /var/lib/infinito is the ganesha NFSv4.2 state share, so
every backup repository landed on it. ganesha 6.5 refuses user.* extended
attributes, and baudolo 7.1.x copies volumes with `rsync -aHAX`, so the
volume backup of OpenCloud (user.oc.* metadata) failed with
`fsetxattr(... "user.oc.id") failed: Operation not supported (95)`.
After the state mount the client role now binds the node's own
/var/lib/infinito-backup over /var/lib/infinito/backup. Every consumer
keeps its path; the data stays on local disk with its attributes, and the
backup host pulls each provider over ssh as before. The bind carries
_netdev: without it, systemd-fstab-generator puts the bind into
local-fs.target below its NFS parent in remote-fs.target, and the boot
transaction cycles. The DR drill remounts the bind after its lazy remount
of the state share, because `umount -l` on the parent detaches it.
Closes#1012
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ubuntu-latest is rolling over to ubuntu-26.04, whose sudo is sudo-rs. It
ignores `-E` ("preserving the entire environment is not supported") and
hands the child only the explicit `env` assignments, so the swarm matrix
died at import with `KeyError: 'INFINITO_SRC_DIR'` before any inventory
existed (PR #1009, Nextcloud#0). Jobs that landed on ubuntu-24.04 kept
passing.
Every `sudo -E` now passes the caller's exported names through
`--preserve-env="$(compgen -e | paste -sd, -)"`, which both classic sudo
and sudo-rs honour. A lint rejects `sudo -E` and bare `--preserve-env`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.
The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.
call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The three-arm comparison on web-app-gitea#0 put the meshed arm at 414ms
per task over the direct arm, and the tunnel at 4% of that. What it could not
split is the rest: the arms differ both in connection plugin (docker against
ssh) and in escalation (root with no prompt against administrator with a
password sudo on every task).
A fixed set of module calls, timed either side of the transport switch with
become off and on, gives four numbers that separate the two. It costs about a
minute where a fourth deploy arm would have cost three hours, and it needs no
root login and no sudoers change.
INFINITO_SWARM_TRANSPORT_BENCH guards it, so a normal round times nothing.
The arm existed to answer one question: how much of the meshed arm's
cost is the connection plugin and how much is the tunnel. Runs 37592061960
(docker), 37753506661 (SSH on the underlay) and 37621556682 (SSH over
WireGuard) answered it on web-app-gitea#0, debian/ext4:
pass 1 5964s / 8511s / 8832s
pass 2 5319s / 7472s / 7705s
Leaving ansible_connection: docker costs +4700s, the tunnel +554s - 89.5%
against 10.5%. The answer is recorded; the scaffolding does not need to ship.
Reverts d792519725, 1b28f87b37, 89b8a8c8d1, 0470491e1a, 529c5ace43,
dd3d10a9a2, c20dd0e5f5, 458d8956e1, 63ade1e03b, d281555945 and 4adfb6e3d2.
95cfc8b7b7 is already superseded by bbfd82f89, which sets the profiling limit
upstream.
Narrowing the meshed drill to its backup half bought budget by giving
up the recovery legs over the mesh. The budget overrun is worth measuring
before it is designed around, so the pre-drill pass and the full teardown and
recover come back. Reverts a27b093c4.
DRILL_STAGE=backup returns once the marker has travelled from the live
export through the deployed units to the LUKS device, before step 6 tears the
stack down. The loop device is closed on the way out, which the full path does
inline at step 7. Default stays full, so the direct arm is unchanged.
A round that drills has to deploy twice, so the caller needs the answer
before it deploys anything. Answering here keeps the rule deciding which
applications have something to restore in the script that already owns it,
rather than restating it in the caller.
The verdict is printed rather than signalled by exit status: a probe that
dies on the way to the decision must not read as 'no drill' and leave the
caller skipping the deploy the drill depends on.
SWARM_REQUIRED_SERVICES no longer force-keeps wireguard: with the axis off the
role is absent from the inventory, so there is no disabled-service conflict to
work around.
The drill strips the services it cannot run without from the incoming disable
list, and wireguard was not among them. For an app whose own graph does not
reach it, the provision step disabled the service while extend_inventory had
already placed svc-net-wireguard on every swarm host, and the services
disabler rejected the pair:
service 'wireguard' is disabled, but provider role 'svc-net-wireguard' is
still active in .../devices.yml
svc-swarm-node declares the service required for exactly this reason, so the
drill has no more business disabling it than it has disabling node.
Penpot job 105068912535 and MediaWiki job 105068912565 both died on one lab node whose dnsmasq kept its :53 socket bound and rejected every query with 'ignoring query from non-local network', once for a container address and once for 127.0.0.1. Rejecting loopback as non-local only makes sense if --local-service is active and dnsmasq's own interface list went stale, but the bundle records neither the flag nor where it would come from.
Capture the process command line, /etc/default/dnsmasq, the unit including its drop-ins, and the config that actually landed on disk. Addresses and listening sockets were already collected and are not duplicated.
Read-only. Every call falls back to a text line when the file or the unit is absent, so a node without dnsmasq reports that rather than failing the collector.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two places in the DR drill ruled on a systemd unit without looking at what the
unit was doing.
[8b/9] took the exit code of `systemctl try-restart nfs-ganesha` as the verdict.
The unit carries Restart=on-failure from the role's own drop-in, and that
drop-in also hangs a 20 second readiness probe on port 2049 as ExecStartPost.
Right after the backing FS is wiped and restored, that probe runs out while the
container still waits on its own stale NFS mount, so systemd kills the daemon it
just started and reports a failed start - then restarts it three seconds later
and succeeds. Measured in run 34699757369, jobs 103572236910 (Matrix#0) and
103638497583 (Gitea#0): 'Control process exited, code=exited, status=124',
'Failed to start', then 'Scheduled restart job, restart counter is at 1'. The
restart attempt may now fail; the assertion is a bounded wait on is-active, and
it aborts loudly with systemctl status when the unit never arrives. The
coherence probe that follows is unchanged and still has to pass.
[0/9] waited for in-flight backups with `systemctl is-active --quiet
'svc-bkp-*.service'` and did not wait. In job 103588089327 (Moodle#0) the volume
unit ran from 01:00:56 to 01:01:12 while that loop returned after 0.36s for four
nodes. The drill then attached to the running job instead of starting its own,
and [3b/9] refused the generation it found because it predated the drill by ten
seconds.
That second change is a hardening, not a proven fix: the reason the loop
returned early is still unknown. The obvious explanation does not hold - the
systemd manual says is-active "Returns an exit code 0 if at least one is
active", so a pattern matching one running unit among four idle ones should have
blocked. list-units --state=active expresses the same intent without relying on
that semantics. Closing the gap needs the call run on a node with exactly that
mix, which no local container offers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
From triage of run 33923190220. Each fix is independently complete.
penpot: ansible templates delegate_to before the task's merged when, and AnsibleUndefinedVariable escapes get_delegated_vars_and_hostname. PENPOT_BACKEND_HOST_NODE is set only by the resolve include, so the bootstrap task hard-failed on every host that include skipped instead of skipping with it. A block-level when cannot prevent this; move the tasks into a dynamic include.
matrix: the MDAD reconfigure looked for volume matrix_matrix_mdad_state (real name matrix_mdad_state) and container matrix-mdad (real name matrix). Both if-bodies were skipped while the task still reported changed, so a rotated S3 secret never reached synapse and every media write got SignatureDoesNotMatch. Move the shell into its own script and pass the real names through the environment. Also add no_log to the vars.yml render, whose diff printed 16 secrets in cleartext.
n8n: when the settings file and N8N_ENCRYPTION_KEY disagree, oclif's findCommand throws for every command, so 'n8n export:credentials' is unreachable in exactly the state that would need it. The re-key is one-way; pin it rotatable: false.
swarm-drill: distros.sh sees only rc=124 and cannot tell a wedged deploy from one the sweep deadline cut short. Say so where the governor caps it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0b1565c493 went in with --no-verify and without the autoformat pass, so the
trailing nocheck comment in run_within_deadline() sat sixteen columns right
of the comment column the next line uses. The quality gate's autoformat step
realigns it on every pass; this lands that realignment so the tree stops
dirtying itself after each gate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The EXIT trap handed out four fixed budgets, 1500 + 900 + 900 + 900, against
the 900s reserve the governor sets aside for it. Its first step alone was
allowed to run longer than the whole reserve, so the trap regularly outlived
the deadline and the outer `timeout -k 60` escalated to SIGKILL. Nextcloud#3
and Nextcloud#11 in run 33574056480 died that way, both at 20460s to the
second, with the diagnostics still running in their own process group and the
teardown never reached.
Each step now takes the smaller of its nominal budget and the time actually
left, recomputed before it runs, and leaves behind what every later step needs.
The worst case where each one exhausts its cap fits the reserve exactly: 480 +
120 + 30 + 120 + 30 + 120. Two steps that carried no timeout at all join the
accounting, because one unbounded step defeats the whole arrangement.
The cap only engages near the deadline. A deploy that fails on its own with
hours of budget left sees the nominal budgets unchanged, so the depth is
traded away only in the case where the exit code already names the cause.
What that case gains is a truthful ending. The job now reports the deploy's own
rc rather than 137, the cluster is torn down instead of leaked into the next
distro, and the reports are collected. A step cut short by its cap says so on
stdout and returns 124 to the caller, so a half-written rescue bundle can no
longer read as a complete one.
The drill is not made faster by any of this. A round too slow for
INFINITO_CI_DISTRO_BUDGET_SECONDS still fails, now as a clean 124.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b8a6205396 put INFINITO_SWARM_TEARDOWN_RESERVE_SECONDS into default.env and
made 00_one.sh require it with a :? guard, but wired it into no handler.
default.env is only the source of the value; a key reaches the generated .env
solely through a handler's setdefault, so make dotenv dropped it and the
swarm routine aborted on its own guard.
The two nocheck markers cover the neighbouring key on those lines:
INFINITO_CI_DISTRO_DEADLINE_EPOCH is computed by distros.sh from the sweep
budget at run time, so a static default would be a stale timestamp. They have
to sit on the same line; above the statement the marker does not apply.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moodle#0 of run 33445831564 was cancelled by the 355-minute job cap with almost no
artifacts. The cap did not end a deploy; it reaped a zombie.
00_one.sh wrapped `timeout` around `sudo`, so the guard ran as the unprivileged
runner and the payload as root. An unprivileged SIGKILL cannot reach a root process,
and the same holds for distros.sh's own `timeout -k` one level up. Probed both
shapes: the payload lands in a pgid the outer governor never addresses either way, so
--foreground would not have helped -- the privilege wall is what carries.
The log shows the consequence directly. The governor fired on schedule:
06:18:27.884 distros.sh: line 103: 15914 Killed timeout -k 60 "${remaining}s" "$@"
06:18:27.894 [ERROR] Run failed for distro=ubuntu: rc=137 after 20460s
and then ansible output kept arriving in the same tee'd log -- a TASK at 06:18:27.929,
another at 06:26:00.189, elapsed still advancing -- until
`##[error]The operation was canceled.` The orphan held the step's pipe open, so tee
never saw EOF and the step never returned.
954b9f1ae6 added the TERM trap precisely so a timeout kill would still collect
diagnostics, and its reasoning ("GNU timeout signals the whole process group") is
correct. It simply never gets signalled on the swarm path, because sudo stands
between. That is why this cell carries only inventory- and role-runtimes- while its
compose twin also has rescue-diagnostics- and playwright-. This completes that fix
rather than replacing it.
The ladder was also inverted: INFINITO_CI_DISTRO_BUDGET_SECONDS=20400 (340 min)
against INFINITO_SWARM_STEP_TIMEOUT_MINUTES=350, with default.env:97 asserting the
order as if correct. For the trap to run, the step guard must fire BEFORE the sweep
governor. Rather than re-tune two constants that start their clocks at different
points -- matrix start trails drill start by 1m49s to 2m04s, measured -- 00_one.sh now
derives its ceiling from the governor's deadline minus a teardown reserve, and keeps
its own configured value when that is smaller.
Not fixed here, and it must be said: Moodle#0 does not fit on a hosted runner at all.
Its pass-2 pacing tracks pass-1 within 5%, putting the pass end at roughly 362 minutes
after job start, past both the 355 cap and GitHub's own 360 ceiling. Raising the cap
would be masking and is arithmetically impossible anyway. What this buys is an honest
red cell at ~323 minutes WITH its diagnostics, instead of a silent cancellation with
none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both Workspace jobs of run 33445831564 die at
roles/sys-svc-container/handlers/main.yml:9, on the first registry pull a nested
node's dockerd performs after a restart:
docker: Error response from daemon: Get "https://registry-1.docker.io/v2/":
tls: failed to verify certificate: x509: certificate signed by unknown authority
The image name is a red herring. rpardini/docker-registry-proxy SSL-bumps, and the
certificate it presents for registry-1.docker.io and for ghcr.io carries the same
issuer, CN=DockerMirrorBox Intermediate IA -- measured against the live proxy through
the make helper, with three controls: direct fetch on public roots returns 401, the
same fetch through the proxy on public roots only returns "self-signed certificate in
certificate chain", and --cacert /dev/null errors, proving curl honours the flag.
That last control mattered: curl consults its compiled-in CApath as well, where
update-ca-certificates has hash-linked the proxy CA, so without --capath /nonexistent
the negative control passes silently.
So a mirror-name fix would have MOVED the error from docker.io to ghcr.io rather than
removing it. What is missing is trust, and the trust anchor came from
INFINITO_CACHE_REGISTRY_CA_HOST_PATH -- a variable act does not forward, and whose
`make dotenv` rewrites the shared .env back to default.env. It can therefore name a
directory the proxy that actually answers never signed with.
The bootstrap now reads the anchor off that proxy's own /ca bind. And it stops
limping: with the frontend up and no /ca bind exposed, the run aborts naming the
reason instead of continuing untrusting and surfacing three minutes later as a
cryptic x509.
Locally the patch is a measured no-op -- the inspected value is byte-identical to the
.env value, and the render with and without it is identical -- so it changes behaviour
only where the variable is wrong, which is act.
Not proven: that /mnt/cache/registry/ca/ca.crt on the hosted runner is the root the
live proxy presents. It follows from rpardini's design and from the sibling leg that
works, but both host directories are absent from every artifact and that runner is
gone. The line "==> registry-cache CA source: /mnt/cache/registry/ca" in the next
workspace run settles it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every one of these call sites ended in `|| true`, so a failure left no trace
and the surrounding script reported a clean run.
service_converged.sh needed the `|| true` for a reason that had nothing to do
with the command: under `set -o pipefail`, `head -10` closes the pipe and awk
dies of SIGPIPE, which pipefail then reports as the pipeline's status. That
made a genuine `container service ps` failure indistinguishable from a normal
truncation. Capping inside awk (`n++ < 10`) removes the SIGPIPE, so the command
status can be tested on its own and a real failure now prints why. give_up and
report_failed_tasks carried a byte-identical copy of that block, so the fix
lifts it into one report_tasks helper rather than applying it twice.
The swarm-clean scripts keep going after a failed step on purpose, since the
host layer below them clears what survives. They now say what did not work
before continuing, instead of continuing silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The coherence step wrapped a blocking 'systemctl try-restart' in a 120s
budget, one second short of the wedge-recovery cycle the unit performs
when the liburcu-bp startup wedge (f10a4a181) fires - observed in run
32657210518, where the Akaunting drill died on rc=124 while ganesha came
back healthy 4s later. 240s covers even the uncapped 123s cycle; with the
abort grace capped at 10s (64b39e227) one cycle is ~43s.
The '|| systemctl try-restart nfs-server' fallback also masked exactly
this failure: nfs-server.service is 'inactive dead' on a ganesha node,
try-restart on an inactive unit is a documented no-op exiting 0, so a
larger budget alone would have turned a SIGKILLed ganesha into a green
drill. Dispatching on the installed unit file keeps the kernel-NFS branch
and lets a real restart failure surface: a wedge leg now fails loud after
one ~43s cycle instead of passing on self-healing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
03_wait_converge built the DB service name from the engine type:
"${DB_DEP}_${DB_DEP}" -> mariadb_mariadb. That name is only correct when
the database is central. force_shared_db deliberately leaves a
manager-pinned role carrying a literal `shared: false` alone, so
web-app-magento keeps its embedded engine, and sys-svc-rdbms deploys it as
magento_database inside magento's own stack.
The name came from the engine, which is global, and was looked up with
`docker service ls` across the whole swarm instead of the role's stack, so
the `shared` flag never entered the check at any step. Both polarities were
wrong:
variant #1 - no consumer pulls the central provider in, so
mariadb_mariadb does not exist. Run 32630684501 job 97175522113 polled it
ninety times and failed the drill, while the rescue artifact shows
magento_database at 1/1 Running and the play at failed=0.
variant #0 - matomo and mailu declare a Jinja `shared`, get force-shared
and drag svc-db-mariadb in. Job 95662862942 then reported
`db(mariadb): 1/1 Running 2 hours ago` beside `magento: Running 21
minutes ago` - green on a database magento never touches.
resolve_db_service asks the deployed swarm which service carries this
role's database: <entity>_database first, else <dep>_<dep>. The two cannot
collide, because sys-svc-rdbms renders the _database sidecar only under
`lookup('database', application_id, 'local')`. Exact-line matching, not
docker's substring name filter, per 246558a13.
Only web-app-magento changes behaviour: it is the one role that is both
pinned by force_shared_db and seen by the DB_DEP grep. matrix and
bigbluebutton never reach the DB half - their entity key carries no image,
so skip_if_no_swarm_service exits first - and discourse resolves
postgres_postgres exactly as before, in every variant.
A role that declares a DB dep no service serves now says so and gates the
app half alone, instead of polling a name nobody deployed.
diagnostics.sh carried the same guess in its container filter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
01_bootstrap.sh ran apt-get update with no timeout, so a stalled mirror
connection hung until the drill's global budget killed it - twice in run
32284541284 (jobs 96178507396 funkwhale, 96178507443 friendica), each
after hours of silence. The journal pins it exactly: sudo[462047] opened
19:51:36 and closed 00:07:30, 4h15m54s in a single apt-get update, and
the session only ended because timeout fired. Healthy runs of the same
line in the same jobs took 3s, 21s and 41s.
The cost is not one red job. distros.sh:130 is the only ceiling and it
wraps the whole per-distro sequence in the remaining global budget, so
one stalled fetch also takes the untried distros with it: funkwhale ran
2 of 5, friendica 3 of 5, both ending at budget remaining=-19s/-20s.
-qq hid the stall and || true cannot catch a hang.
Same failure class and same remedy as ba8a72309, which bounded
scripts/install/apt.sh; this closes the sibling call site it left open.
Constants are taken verbatim from there rather than re-derived. timeout
runs under sudo so SIGTERM reaches apt-get instead of the sudo wrapper -
the 4h session above is what the other order looks like. -E stays so the
package-cache proxy variables survive.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were left by earlier commits and are collapsed by markdownlint-cli2 and shfmt respectively, so every make quality run left the tree dirty and the change would eventually have ridden along in an unrelated commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The converge gate polls STACK_NAME_PRIMARY_SERVICE_KEY. Four roles never
create that name and each was given a static `workload: node-local` to make
the gate skip itself:
21fe9e984 web-svc-cdn ships only cdn_build, never cdn_cdn
2c6c8b763 web-app-matrix names its services matrix_synapse and friends
76508c946 svc-runner deploys node-local compose projects per node
cb52a72f2 bigbluebutton phantom service, and the DR drill must still run
Four hand-written claims for one observable fact. The claim is also static
where the truth is not: web-app-discourse renders `services: {}` only while
its sidecars are shared, and job 94965456921 of run 31864448027 polled
discourse_discourse ninety times for a stack that had none. Declaring the
role node-local would have blinded the gate in the variants where its
sidecars are local and a stack does exist.
has_swarm_service asks the deployed stack for the exact name instead. Every
caller runs behind `if rc == 0`, so an absent service is the role's shape and
not a broken deploy. Exact-line matching, not docker's name filter, which
matches substrings.
No coverage is lost: 03_wait_converge cannot pass without that service, and
05/06/07 plus the backup drill only run once it has. The four roles keep the
skip they were given; discourse and every future case get it without a
declaration. The teardown branch in the drill keeps its meaning - stack rm
where the service exists, compose stop per node where it does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
call-test-deploy-{compose,host,swarm,single-node}.yml differed only in
which mode they pinned and which artifact prefix they wrote; compose and
host already ran the identical drill. They become one call-test-deploy.yml
whose matrix rows carry the mode, so the swarm-specific steps are guarded
on it and everything else is shared.
The orchestrator loses its eight deploy jobs and the sequencing job for
three serial chunk blocks. GitHub Actions cannot generate a variable
number of jobs, so the blocks are written out and INFINITO_CI_MAX_CHUNKS
tells the budget how many exist; blocks whose slice is empty skip
themselves. chunk_gate stops the chain at the first red chunk,
resume_from_chunk re-enters after a fix instead of re-running the green
chunks, and modes collapses to a single mode choice.
apps.sh now emits matrix entries for one chunk rather than a token list
for one mode, and no longer sorts them: the discovery order is what the
chunker slices on. split_runner_apps.sh reads its runner quota from
INFINITO_CI_CONCURRENCY instead of carrying a second copy of the 20.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill starts the backup units itself, but the deploy arms their calendar
timers first, and a job that runs long enough crosses every backup window in
group_vars/all/07_schedule.yml. web-app-decidim ran 23:05-02:24 and hit exactly
that: the drill's own volume backup succeeded at 01:00:49, then the timer fired
a second, unattended run at 01:03:18 while step [8b/9] was lazily unmounting
/var/lib/infinito underneath it.
The failure is silent by construction. baudolo writes the dump in two
processes - 'pg_dump > x.tmp', then 'mv x.tmp x'. The redirect shell exits 0
because a shell never reports a failed writeback on a soft NFS mount, but the
unmount discarded that mount instance together with the unflushed file, so the
mv resolved through the fresh instance and got ENOENT. The unit died at
01:04:04, the same second the drill finished remounting, and 80 minutes later
the CLI test read 'failed' and aborted the job before running its own backup.
Disarming the timers makes the drill the only thing driving those units.
Stopping a timer does not stop its service, and both the drill and
roles/svc-bkp-volume-2-local/files/test/test.sh start their units explicitly,
so nothing under test loses coverage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker's name filter matches substrings, so every stack whose sidecar is
named after its app answered with two containers and two services. The
drill took the first, which after a reschedule was the MCP sidecar: it
declares no healthcheck and does not listen on the app port, so the step
waited sixty seconds on the wrong process. Selection now goes through the
swarm service label and an exact name match, both behind one helper.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill probed for a fixed 60s. A container without a shell can only be
observed through its docker health status, and mattermost declares
start_period 30s with interval 1m, so that status cannot report healthy
before 90s. The step was unreachable for such an app however fast it
started. The budget now comes from the container's own healthcheck and
stays at 60s for every container that declares none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The quiesce guard spelled the dev container as the literal
infinito_nexus_debian, so it only ever matched a checkout whose .env was
generated for debian without a project prefix. It missed the self-hosted
runner form infinito_<project>_nexus_<distro>, any custom.env pin, and any
.env generated for another distro — and it missed them silently, because a
non-matching grep just skips the block and leaves the competing onion
publisher running.
INFINITO_CONTAINER is the SPOT for that name: compose.yml uses it as
container_name, and scripts/meta/env/load.sh (already sourced at line 19)
exports it. Proven against the container running on the author's host: the
literal does not match it, the variable does.
The outer `docker ps | grep -qx` pre-check is gone with it — `docker exec`
on an absent container already exits non-zero with stderr suppressed, and
the `if` absorbs that even under `set -euo pipefail`.
The variable joins the line-21 assertion so a broken env fails by name
instead of dying inside a pipeline subshell, which would reproduce exactly
the silent-skip class this commit removes. compose.yml already requires the
same variable with `:?`, so this is not a new demand on the environment.
tests/lint/repository/test_no_hardcoded_container_name.py pins it: run
against the pre-fix file it flags all three lines, and it stays green on
the distro-agnostic spellings in stack_refresh.sh and worktree/up.sh.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dump listed the state path itself, so the volume directories showed up but
nothing inside them. Ownership questions about a specific file were therefore
unanswerable from the artifact alone, which is how a permission failure on a
database file cost two investigations without ever being settled.
Records mode, numeric owner and size one level in. Numeric because the
container uids have no names on the nfs server, and those uids are what the
questions are about.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Docker hands out the lowest free address, so a node attached without one lands
in the same band where the cache services pin themselves and can take an
address its owner then cannot claim. That happened: a node held the
registry-cache address and every swarm cycle died at the image pull.
Each node now takes a fixed address, offset per instance because worktrees
share the primary's cache network while clearing only their own cluster, so a
surviving cluster elsewhere no longer collides with a fresh one.
A failed attach is classified rather than swallowed. The nodes already carry
extra_hosts pointing at the cache addresses, so a node that silently fails to
attach reaches neither apt nor the registry and reports it as network
flakiness. Only the markers the docker_network_noop filter already treats as
benign are tolerated; anything else stops the bootstrap.
The subnet is asserted rather than assumed, since the address arithmetic only
holds for a x.y.z.0/24.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The routine only carried an EXIT trap, so a run terminated by the enclosing
`timeout` in distros.sh:128 left no rescue diagnostics, no playwright reports
and no teardown — the exact case where the artifacts matter most.
Add a TERM trap and make the handler take the status as an argument, so EXIT
keeps using `$?` while TERM passes 143 explicitly. The `_teardown_done` latch
is required because the EXIT trap still fires after the TERM trap; without it
the whole collect-and-teardown block, including its 1500s and 900s timeouts,
would run twice.
This works because GNU timeout signals the whole process group: bash defers a
trap while it waits on a foreground child, so a bare kill to the script PID
alone would not surface it until that child exited. Reproduced under the real
shape (timeout -k): TEARDOWN rc=143, exactly once, after 2.01s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm DR stage aborted with "no backup unit installed on <mgr> or
<nfs-server>" on all five workspace distros. routine/backup/base.sh triggers
svc-bkp-volume-2-local / svc-bkp-secrets-2-local on the manager and
svc-bkp-nfs-2-local on the export host; all three trigger_units calls returned
rc=2 (no unit matches), so _triggered stayed 0.
The units were never installed because the roles were never in the closure.
utils/tests/swarm/README.md pins the contract: derive_includes pulls
svc-bkp-volume-2-local in via the `container_backup` consumer and
svc-bkp-nfs-2-local via `nfs_backup`. derive_includes._disabled_provider_roles
maps those same keys back to those roles and `found -= disabled_roles` prunes
them, so any caller that lists them in `disable` silently removes what the
drill then requires.
scripts/tests/environment/12_swarm.sh and 13_roundtrip.sh both do exactly that:
`variable_services <app> node nfs-server` keeps three keys and disables every
other group-conditional one, which sweeps up container_backup and nfs_backup as
collateral. Reproduced offline byte-identically to the CI line:
$ python3 -m cli.meta.roles.applications.dynamic_services \
--exclude mariadb node nfs-server
asset,...,container_backup,...,nfs_backup,...
Filtering in those two callers would leave `make swarm-zombie app=X disable=...`
and any future caller exposed, so the filter goes to the single entry point every
swarm drill passes through: the CI matrix reaches routine/00_one.sh via
distros.sh, and swarm-zombie reaches the same script through act. Both `disable`
consumers -- the provision-time services_disabler and derive_includes in the
matrix subprocess -- read the env var this script re-exports.
Measured before/after on the derive_includes closure:
APP_ID=svc-db-mariadb -> {} / {svc-bkp-secrets-2-local, svc-bkp-volume-2-local}
APP_ID=svc-storage-nfs-server-> {} / {svc-bkp-nfs-2-local}
node and nfs-server join the set for the same reason db87a1a1c added them to the
callers: they build the lab cluster itself.
Token matching is exact, so nfs_backup2 and mynode survive; `IFS=', ' read -a`
avoids the pathname expansion an unquoted split would have allowed. The CI swarm
workflow passes no `disable` at all, so the filter is a no-op there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workflow directory mixed entry points, reusable stages and scheduled jobs into one flat list sorted by topic, so the first question about any file - what starts it - was the one the name did not answer. Every workflow now carries one of three prefixes, derived from its on: block: cron- for a schedule trigger, call- for one only reachable through workflow_call, entry- for everything a repository event or a human starts. Scheduled workflows additionally carry an alarm clock as the first character of their name.
Two lints keep it that way. One derives the expected prefix from the triggers and compares it against the file name; it flagged all 17 offenders before the rename and none after. The other resolves every .github/workflows/<file> string in the repository against the directory, which is what a rename silently breaks in Markdown links, Python constants and shell API URLs alike.
Renaming pulled 69 pre-existing comments into the comment lint's changed-file scope. The vendored CodeQL template got the nocheck marker its scorecard sibling already carried, two workflow headers moved above name: where they count as file headers, and the remaining banners were deleted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role README claimed the backup machinery "operates transparently against the mount
paths" and demoted the source-of-truth rule to advice in a prose box - both written
before 2efc73d3b made that rule the code's behaviour. It now states what is enforced:
this role skips export-backed volumes, svc-bkp-nfs-2-local is their single capture, and
manager-pinned roles, roles forcing another compose_mode_force and nfs: false opt-outs
stay covered here. It also names the limit the code actually has - the exclusion checks
that some host is in the svc-bkp-nfs-2-local inventory group, not that the role ever ran
successfully - because a README promising more than the code delivers is worse than one
promising less.
Its Schema section carried the role's timer flow; it now carries the mechanism the
change set is about: meta/volumes.yml -> swarm_nfs_backed() as the single predicate ->
its two consumers (the compose rewrite and the backup exclusion) -> the inventory-group
gate -> the export host's single capture -> the restore path with its mountpoint probe
-> the drill's [3b] guard against over-exclusion.
The two swarm READMEs drew the drill with eight steps and pre-773a25ab6 semantics. Both
now draw the nine that exist, including what makes the proof a proof: the pulled tree is
wiped before the device recover, so the device holds the only copy left; the live export
is wiped with the marker proven gone before it is restored; and [9] deletes the marker
server-side before restoring the volume through its bound backing store.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three steps of the DR drill, one theme: every step now proves what it claims.
[9] restored into the volume's mountpoint while the stack was down - a bare directory,
because docker mounts a volume declaring its own backing store only while a container
holds it. The payload landed on the node disk and the real mount shadowed it on the next
deploy: success reported, nothing delivered, and roughly a gigabyte added to the very
device the drill later runs out of. It now deletes the marker server-side, binds the
backing store at the mountpoint, runs the recover CLI through it, and asserts the marker
on the NFS server's own disk. That also exercises the positive branch of the guard from
773a25ab6 - its refusal path is unit-tested, its pass-through was tested nowhere. When
the manager exposes no backing store the branch skips with a reason: docker volume
inspect is node-local, and with replicas: 1 the volume object usually lives elsewhere.
[3b] replaces a proof that used to happen by accident. The double capture 2efc73d3b
removed also showed, incidentally, that the volume unit really writes - the drill found
its marker in the volume repo too. The new step checks it directly: the newest
backup-docker-to-local generation must be at least as new as DRILL_START_TS (taken on
the manager, the same clock that names generations) and hold a non-empty file under a
volume subtree. Its failure message names the suspicion it exists for: an exclusion that
swallowed every volume. With no volume unit deployed it skips, tracked by
VOLUME_TRIGGERED alongside the existing SECRETS_TRIGGERED.
[5] prints du -sm per machine and repo when the space precondition fails, so the next red
run names what fills the pulled tree instead of leaving it to arithmetic.
Four adversarial review rounds found five defects in this block and confirmed each fix:
an NFS mount reconstructed from driver options the codebase never emits (it is a bind),
the recover CLI's second positional passed a volume name where it takes a host, a missing
volume object aborting the drill instead of skipping, and an umount ordered before the
return-code check, which under set -e would have killed the drill on a busy mount with no
diagnostic and left the bind behind.
None of this has run in a swarm yet. Every failure path is loud and names its cause.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
