compose names the swarm-test volume after its project, so every cluster
left a <cluster>_nfs-export behind that no teardown path touched. At
roughly 5GB each they filled the host disk after ~30 runs, which showed
up as unexplained playwright timeouts once writes started stalling.
Remove it per cluster in teardown, per worktree subnet in lab-reclaim,
and host-wide in swarm-clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make quality rejected the new reclaim script for missing the executable
bit, and the S310 suppression in the Home Assistant token probe for
carrying no reason. A security suppression must state why it is safe, so
the loopback justification goes back on the urlopen call; ruff dropped
the second marker on Request(), where S310 never fires.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lab subnet is pinned per worktree slot while lab network names carry
the cluster id, so a cluster kept from one app blocked the next run of
another with 'invalid pool request: Pool overlaps'. The pre-run teardown
only cleared the same cluster id, which never matched. Matching on the
IPAM subnet instead clears exactly the blocker and leaves a sibling
worktree's cluster alone; swarm-clean stays the host-wide tool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CI deploy bootstrap hand-listed two packages and let the ubuntu-latest runner image supply the rest, so a new host-side import surfaced as a red deploy job instead of a failed install. Size [project.dependencies] to what runs outside the project image and move the deployment stack into a deploy extra, then have every install site read that instead of naming packages itself: the image build and make install-python take the deploy extra, the CI host and the mirror and instructions workflows take the base.
Declare cryptography, which utils/manager/value_generator.py imports directly and which only reached the environment transitively through ansible. Drop numpy, which nothing in the repository imports. Guard the boundary with a lint test that blocks every deploy package and asserts the runner-host CLI still builds its parser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The storage-constrained work put utils.storage.constrained into the host-side import chain of cli.administration.deploy.development, which pulls humanfriendly. The CI dependency line installed only ruamel.yaml and PyYAML, so every compose and host deploy job aborted at parser build time with ModuleNotFoundError.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The topology SPOT read default.env only, so every lab took the same subnet no matter what the generated .env said. Two labs on one host therefore collided with "invalid pool request: Pool overlaps", which is what stopped a worktree lab from running beside the primary checkout's.
Collect the generated .env first, then default.env, and emit each assignment as := so precedence becomes exported environment, then .env, then default.env. default.env stays the fallback because the workflow sources this file before make dotenv has run, which was the original reason for reading it directly.
A line that does not parse as a balanced assignment now aborts naming the line. Sourced as-is it would have become an unconditional assignment and silently defeated the precedence this change exists to establish.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An aborted Playwright run - watchdog SIGTERM, task timeout - returns a result dict with neither stdout nor rc. The reporting task dereferenced both, so it became the fatal error itself: the log ended on "object of type 'dict' has no attribute 'stdout'" while the real cause, a disk-exhaustion abort with rc 75, sat eight thousand lines earlier. Guard all three reads, fall back to msg for stderr because that is what Ansible fills on an abort, and treat an absent rc as a failure so the run is recorded and named by the aggregate check rather than vanishing behind a template error.
The distro sweep's summary also mislabelled the distros it never started: they kept the initial "not reached within the budget" note even when the sweep aborted on a failure with 12214 seconds of budget left, pointing at a budget problem that did not exist. Say what happened instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
act merges its --env-file, which defaults to .env, after the workflow-level env block (pkg/runner/run_context.go:84 at v0.2.89) and interpolates only afterwards, so INFINITO_DISTROS: ${{ inputs.distros }} was replaced by the repository's own five-distro list before the expression was ever evaluated. Every workspace leg therefore swept all five distros instead of its own: 25 distro deploys where 5 were intended, which is what walked the five legs into their 345-minute cap. The drill step's name kept rendering the single distro because it is interpolated from the workflow model rather than from the env map, which is why the logs read as a contradiction.
Pass INFINITO_DISTROS as an --env pair instead, the layer act itself wins with, where --env beats the env-file because the file only fills keys that are absent (cmd/root.go:364-366). The expression was duplicated across ACT_ENV and ACT_INPUTS, so name it once.
Wrapping the assignment for readability required the splitter to tolerate whitespace: inside double quotes the shell removes the backslash-newline but keeps the continuation line's indentation, so every pair after the first arrived with a leading tab.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge of the swarm-nfs work moved utils/unmount_nfs_mounts.sh to utils/unmount/nfs_mounts.sh and updated teardown.sh and the bootstrap, but clean/all.sh kept the old path. make swarm-clean died at the unmount step with "No such file or directory" before it reached container removal, which is exactly the state that needs cleaning up. The host-state release is not added here because all.sh already unmounts DIR_VAR_LIB itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f660dd6bd wrote the placeholder into a parameter expansion, `${INFINITO_IMAGE:-<none, bootstrap will build it>}`, which tests/lint/filesystem/shell/test_no_infinito_defaults.py rejects: an INFINITO_* default in a shell file is a second source next to default.env that drifts silently, and only the empty form is allowed. It was the sole failure of test-lint.
Split the message into the resolved and the unresolved case instead of suppressing the lint; the output is also no longer self-contradictory, since 'published <none>' could not both be true.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f660dd6bd introduced `${INFINITO_IMAGE_TAG:?}` in the image resolution, a hard requirement the replaced call deliberately did not have -- ci.sh exits 0 before its own assert when no registry owner is in scope. The check sat after the EXIT trap was armed and before the env layer is sourced, so a missing tag ran collect_and_teardown against a cluster that was never created, and the variable appeared in neither the # Param header nor the strict guard block that carries the other four required inputs.
Move it into that guard block, which runs before mktemp and before the trap, and drop the now-redundant inline assert.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every workspace leg pinned INFINITO_DISTRO=debian and drilled all five distros sequentially, pulling its swarm nodes from ghcr <distro>:latest -- an image from 2026-03-21 whose nfs-ganesha 7.2 segfaults on any accepted TCP connection on 2049. The controller mount then failed with ECONNREFUSED for 26 minutes and took run 30446813110 down after 5h13m, with 3 of 5 distros never reached.
Derive the distro from the leg's own matrix entry, build that image, and drill only it: swarm-zombie takes an optional distros= that defaults to INFINITO_DISTRO, and 00_one.sh prefers the locally built image over the published one. 12_swarm.sh refuses to start when that image is missing, so a broken build fails loudly instead of being masked by a stale registry pull.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test-environment.yml and test-guide.yml become test-workspace.yml and
test-instructions.yml. Orchestrator job ids, the distros-SPOT matrix
lint, the act example, the diagnostics slug and the rescue artefact
names follow.
Replace the roles_only input with per-job workspace/instructions
tri-state inputs (auto|true|false). 'auto' derives the former
roles_only skip straight from whitelist: both jobs run on a global
scope (whitelist empty or __ALL__) and are skipped on a role-scoped
run, which keeps the PR behaviour identical because the diff resolver
never emits an empty whitelist. entry-manual exposes both as choice
dropdowns defaulting to false, so a manual run opts in explicitly.
The now redundant roles_only output is dropped from
pr_affected_roles.sh and subset_roles.py.
Branch-protection required-status-check names change along with the
job renames.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The artifact of the wedged run carried journal, unit status and mounts but no process data, so the hang could be localized to the export config commit and not pinned to a syscall. wchan and syscall per thread close that gap.
MainPID 0 is its own likely state given Restart=on-failure with the start-rate limit disabled, so that branch reports the unit counters instead. Every read keeps 2>&1 because /proc stack and syscall are EPERM on hardened kernels.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The LUKS step of the DR drill died on `The file /dev/mapper/usbdrill does
not exist and no size was specified.` -- from mkfs, not from cryptsetup,
because luksOpen had exited 0. The node did have a udev, and it had
created the mapper entry: as a symlink to ../dm-1. Modern udev no longer
mknods, it only manages symlinks, and the /dev/dm-N target it points at is
a devtmpfs node that a container's tmpfs /dev never receives. Without a
udev libdevmapper creates the node itself, which is why the debian nodes
passed -- their image ships no udevd at all, while arch cannot install
systemd without one.
Masking udevd in the node image removes the whole class rather than one
call site. DM_DISABLE_UDEV rides on the node containers instead of the
script because the drill opens the volume a second time through the
recover CLI in a separate docker exec, which would not inherit an export.
The block-device check then fails loudly with both prerequisites named,
and closes the mapping first so an abort cannot leak it onto the runner --
device-mapper is not namespaced and every distro round shares the VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ganesha unregisters its RPC programs from inside nfs_Init_svc(), before
the sockets are ever put into listen(). That unregister reaches
libntirpc's rpcb_unset -> CLNT_DESTROY -> svc_vc_destroy_it, which spins
on nanosleep({0,0}) with no timeout -- and it only takes that path when
the AF_LOCAL connect to /var/run/rpcbind.sock succeeds. So a reachable
rpcbind is what wedges the server, not a missing one.
The wedged instance burned 5m30s of CPU while serving nothing, never
logged NFS SERVER INITIALIZED, and left the controller mount retrying
into ECONNREFUSED for 1706s before the deploy failed 20 minutes further
downstream. Masking and stopping both rpcbind units before ganesha starts
makes the spin unreachable: without a local socket rpcb_unset returns
false immediately. Wants=rpcbind.service is not Requires=, so a masked
unit is logged and ignored, and v4 registration is explicitly optional in
ganesha -- which is why this holds only while the export set stays
NFSv4-only.
The unit list is probed first so distros without rpcbind are skipped
rather than failing the play, and the probe distinguishes "no such unit"
from an unreachable manager so the mask can never be skipped silently.
The Type=forking unit reports started while the daemon is still spinning,
so a wait_for on 2049 turns a 28-minute mystery into a named failure, and
the diagnostics now carry rpcbind's journal plus what actually listens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The controller mount of DIR_VAR_LIB survived the teardown, so the next
distro cycle inherited an NFS session whose server container was already
gone: the kernel logged 'server not responding, timed out' and every
access under /var/lib/infinito wedged, taking the deploy down with it.
All five swarm jobs of the last manual run died this way, each on a
different distro but always on the third lab cycle.
Both halves of the old teardown were unable to clear it. The guard
`mountpoint -q` blocks on exactly the dead mount it is asked about, so
the timeout dropped the unmount that was needed; and `umount -lf` ran
without sudo on a non-root runner, so the EPERM went into `|| true`.
Detection now reads /proc/self/mountinfo, which touches no filesystem,
the unmount elevates when the caller is not root, and a mount that
survives all of it fails loudly instead of silently poisoning the next
cycle. The bootstrap runs the same release before bringing a lab up, so
a cycle whose teardown never ran cannot hand the mount on either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The loop may drop a distro when the budget runs out, and both skip paths fell through to an implicit exit 0, so a run that never touched two distros still reported success. That hid the coverage gap: arch, fedora and centos get no package cache (cache.override.yml is apt-only), are therefore the slowest, and are the first the remaining < max_seen heuristic drops.
Skipping stays non-fatal and the order stays randomised, but every distro now shows up in a GITHUB_STEP_SUMMARY table in execution order, as passed, failed with its rc, or skipped with the reason. The table is also rendered before an aborting exit, since that is the run whose table matters. Two contract tests cover it; the order assertion checks the logged shuf order rather than the input list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d9a4498b3 collapsed eleven CI steps into one, which broke the assumption every handover in the drill rested on: a $GITHUB_ENV write only comes back in the NEXT step. Inside one step nobody sees it. matrix.py reads os.environ["MGR_IP"], so every swarm job died with KeyError before the first deploy round; 05 -> 06 -> 07 passed INITIAL_NODE/NEW_NODE the same way and would have died next.
The node IPs now come from the topology SPOT: compose.yml already pins every node to those INFINITO_SWARM_*_IP keys, so the docker-inspect capture only read back what compose had set. It goes, together with the $GITHUB_ENV publish and lab_net. lab_domain_target reads the env instead, guarded by an assert task because lookup('env') yields '' for a missing key and an empty dnsmasq `address=/domain/` is valid syntax meaning NODATA -- the failure would have surfaced 30 retries later as a DNS problem.
INITIAL_NODE/NEW_NODE move to a SWARM_DRILL_ENV file, created fresh per distro so the previous distro's node cannot leak into the next drain. The out-of-drill callers of _context.sh keep working through a visible guard; both writers stay strict via :?.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The discover job was named after the full distro list and the plan summary printed all five, but test-deploy-swarm.yml never set INFINITO_DISTROS, so every run fell back to the default.env distro and only debian was ever exercised. The job name has been advertising coverage that did not exist.
The workflow now passes the distros input into the env and hands the drill to the shared loop from 1bbe7a0eb. Bootstrap, setup, matrix deploy, seed, drain and assert move out of eleven separate steps into routine/00_one.sh, which is what the loop runs once per distro.
Collection and teardown move with them, into an EXIT trap: it replaces the removed 'if: always()' and 'if: failure()' step conditions and returns the original exit code so cleanup cannot mask the deploy failure. The always() teardown step stays as a second net for a step that times out as a whole.
Playwright reports and rescue diagnostics gain a distro segment, matching the layout the compose path already writes, because a second distro would otherwise overwrite the first one's evidence. The structure summary carries the distro in its heading for the same reason.
The README described the old shape and is corrected: the loop entries are listed, topology_summary.sh was missing, and 01_bootstrap.sh is no longer 'one CI step'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The serial distro loop (shuffle, time budget, fast-fail skip, exit on the first failing distro) lived inline in scripts/tests/deploy/ci/all.sh and was therefore reachable only from the compose and host deploys. Swarm and guide have to iterate distros with the same semantics, so the loop moves into scripts/tests/deploy/distros.sh and takes the per-distro command as "$@".
ci/all.sh keeps the app-level preamble and execs the loop with the new ci/one.sh, which resolves the CI image and dispatches to the per-app override or dedicated.sh.
Image resolution leaves the loop as well: scripts/meta/resolve/image/ci.sh renders the reference through cli.meta.ci.image_ref instead of respelling ghcr.io/... locally, which meta/distros.yml declares as the single source.
The two CI time knobs are declared in default.env instead of being repeated as literals, so no caller has to invent a budget.
Three paths are dropped as dead: the empty budget that disabled the limit, the failed counter that could only ever print 0, and the branch for an unset deadline.
Both new scripts are pinned by contract tests. Those tests clear BASH_ENV because the test container exports it as the env loader, which would otherwise re-inject every stripped key and make the assertions pass for the wrong reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
moby/buildkit was hardcoded in four places across two workflows and catthehacker/ubuntu in two act scripts, none of them reachable by the image mirror: utils/docker/image/discovery.py only scans roles/**/meta/services.yml. So the one image the CI build cannot proceed without was pulled straight from Docker Hub, unmirrored, which is what broke the fedora build in run 30160077828.
Declare both as mirror-only services on svc-runner, following the test-e2e-playwright precedent, so the existing mirror pipeline picks them up unchanged. New scripts/meta/resolve/image/service_ref.sh reads image+version back out of the declaration and is the single source for consumers outside Ansible; its mode argument is mandatory (upstream never touches the network, mirror prefers the GHCR tag and falls back to upstream). The fallback is why the mirror job can keep running in parallel with the build instead of gating it.
ACT_PLATFORM_IMAGE moves to an env handler so act consumers read the declared value via BASH_ENV, and workflow.sh now demands it instead of defaulting to a literal. Operator overrides still win through eb.setdefault.
Verified: service_ref.sh in all four paths (upstream, mirror hit, missing mode, invalid mode); the mirror path matches GHCRProvider.image_base() character for character and resolves against the live registry anonymously; ACT_PLATFORM_IMAGE resolves through the real BASH_ENV mechanism; make quality green on all four targets.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill configured svc-bkp-remote-2-local with a fixed manager plus NFS provider list for every app, while the manager only runs a backup repository when the app's include closure pulls svc-bkp-volume-2-local or svc-bkp-secrets-2-local in. For an app whose container_backup consumer is statically off, such as svc-dns-unbound, user-backup never ran on the manager, so the pull was rejected with 'Permission denied (publickey,password)' and the fail-if-any wrapper took the whole unit down at drill step 4/9. Four roles reach that step today: svc-dns-unbound, svc-db-redis, svc-registry-cache and svc-registry-docker.
backup_repos.py expresses the placement once and matrix derives the provider list from it per round. derive_includes accepts the round's variant map because INFINITO_APP_VARIANTS only reaches the subprocess callers. The extras file keeps the device paths the drill reads back and loses the provider list nothing read from it, and the drill banners stop claiming backup legs that were never installed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill configured svc-bkp-remote-2-local with a fixed manager plus NFS provider list for every app, while the manager only runs a backup repository when the app's include closure pulls svc-bkp-volume-2-local or svc-bkp-secrets-2-local in. For an app whose container_backup consumer is statically off, such as svc-dns-unbound, user-backup never ran on the manager, so the pull was rejected with 'Permission denied (publickey,password)' and the fail-if-any wrapper took the whole unit down at drill step 4/9. Four roles reach that step today: svc-dns-unbound, svc-db-redis, svc-registry-cache and svc-registry-docker.
backup_repos.py expresses the placement once and matrix derives the provider list from it per round. derive_includes accepts the round's variant map because INFINITO_APP_VARIANTS only reaches the subprocess callers. The extras file keeps the device paths the drill reads back and loses the provider list nothing read from it, and the drill banners stop claiming backup legs that were never installed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Restarting docker does not clear a container whose shim never sent an exit
event: that shim belongs to containerd, a separate unit that survives the
restart. Measured on a host carrying two such remnants - docker.service had
been restarted hours earlier while containerd.service had been up for three
days, and `rm -f`, plain `rm` and `stop -t 1` all still failed afterwards.
The hint also has to say what the restart costs. It wipes every exec instance,
so following it during a deploy kills the run: an act deploy died after 1h48m
with `failed to inspect exec: No such exec instance`, and the log carried no
ansible error at all, only post-steps failing in milliseconds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every teardown left one node container behind, always the same one, and the
warning blamed a kernel D-state. The cause was a budget: the client-unmount
step gave all four containers one shared 120s allowance, the unmount chain
costs up to three attempts per mount, and the third container in the list never
got its turn. That container was then killed with its mount still attached.
Killing was the second half of it. The node image sets STOPSIGNAL SIGRTMIN+3 so
that `docker stop` triggers a clean systemd poweroff, but the teardown only ever
ran `docker kill`, which skips it and leaves the nested engine and its task
containers to die mid-flight. Those task containers hold binds of the NFS
superblock inside their own mount namespaces, where an unmount in the node
namespace cannot reach them, so the nested engine has to go down first.
Under act the ansible controller is the act job container itself, so its own
NFS mount is invisible to the host mountpoint check and survived every cleanup.
Three more paths reached the same containers without any of this: the
cross-cluster reaper removed them with no unmount at all and then reported the
wedges it had just manufactured, the bootstrap pre-clean detached only inside
the NFS server, and the escape hatch filtered its candidates down to
`name=nfs-server` so it could never see the worker that actually survives.
The removal verdict itself was wrong: a single inspect fired immediately after
`rm -f` reports containers the daemon is still removing, and two of the three
warnings in the observed run were false alarms.
Verified by tearing a live five-node cluster down: all five containers removed,
no remnant, no warning.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The failover-drill assert polled service convergence for 60 iterations (120s).
OpenProject web boots in ~140s, so after 06_drain_worker rescheduled it the
replicas turned healthy ~20s after the poll gave up, and 07_assert_state.sh
reported 'Replicas not fully converged: 1/3' and exited 1 even though the stack
did converge moments later. The loop breaks early once converged, so raising the
ceiling to 120 iterations (240s) costs fast apps nothing and only widens the
window for heavy apps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
INFINITO_RESCUE_DIAGNOSTICS_BASE joins default.env as the only literal
of the runner-side collection root and is wired through the passthrough
handler into the generated .env. The four deploy workflows load it with
one grep bootstrap step after checkout instead of hardcoding the path in
their env blocks; upload paths and per-app DIR derivations reference the
variable. dedicated.sh sources it from .env alongside the existing
diagnostics keys and hard-requires it, no script-level default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every DinD node is attached to a second network by the bootstrap cache connect, so the embedded DNS returns two A records per node name in undefined order; workers that resolved the manager onto the cache network could not reach the registry bound to the lab IP and rejected locally built images with 'No such image', pushing all replicas onto the manager. Pin a fixed lab ipv4_address per node and every node name via extra_hosts (files-first resolution wins over DNS); the subnet and IPs live in default.env as the INFINITO_SWARM_* SPOT consumed by compose interpolation and the topology exports. Proven by A/B: the identical friendica variant-2 swarm job went from worker rejects to fully green including the drain test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Back to the pre-swarm-nfs-fix baseline:
- 5c2c8f531 + 985ec7eab: the swarm writer-quiesce DR drill. Deep-dive
attribution of CI run 29715516546 found it the dominant failure source
(17 of 23 jobs): [9b/9] handed-off generation mismatch, [6/9] quiesce
stack-rm network race, [4/9] remote-2-local pull stall, compose discourse
restore duplicate key.
- d6f980574: openproject swarm ingress to web:8080. Carried a leaking
proxy_internal_port lookup (no per-app default) that broke the seaweedfs
dependency front-proxy ('Key web not found'); the bypass-the-Apache-proxy
approach was never verified against downloads/X-Sendfile.
- 1dd153bad: xwiki replicas:1 + mem 3g. Did not resolve the xwiki superadmin
failure (still red in run 29715516546), so it is unverified.
Re-approach each from a clean baseline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up corrections to b449f8157 (swarm DR drill):
- export test.env in 06/07 via set -a so the child db restore/probe
scripts inherit BKP_TEST_DATABASES_CSV and BKP_TEST_RESTORE_BIN
instead of aborting at their :? guards before any restore runs
- move db_probe.sh/db_restore.sh into a db/ subfolder (probe.sh,
restore.sh) and update all callers, so they stop tripping the
prefix-grouped-files structure lint
- define TEST_E2E_SWARM_DRILL in group_vars/all and drop the now
redundant default() from test.env.j2 (satisfies the undefined-var
and spot-default lints together)
- fix a mermaid parse error in the drill README: a ; in a
sequenceDiagram note was read as a statement separator
make quality-high green (test + lint phases, 15 targets, 0 failures).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Defer Swarm SQL replay to the disposable DR drill so shared writers such as Discourse are stopped before the destructive restore. Verify the selected generation with before/after database probes while preserving the Compose restore path.
swarm-exec reaches a DinD node; reaching a swarm service container (moodle,
openldap) needed a nested docker exec. Add swarm-app-exec, which resolves one
running replica on the node and runs a base64-encoded command inside it, so
live in-container inspection stays on a sanctioned make helper. Adding a
second exec_* script tripped the prefix-group lint, so exec_node.sh moves to
exec/node.sh and the new one lands at exec/service.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
07_assert_state.sh sources _context.sh under set -o pipefail, and the printf | head-n1 that picked the first NFS volume raised a broken-pipe write error the moment head closed the pipe after line one. Any role with two or more NFS volumes (erpnext ships erpnext_sites and erpnext_logs) crashed the reschedule assertion before it could check the surviving marker. Plain parameter expansion returns the same first line with no pipe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The swarm deploy job now renders the same summary page as compose:
role runtimes, a swarm structure section (topology roles plus live node
list and service placement from the manager) and the Playwright table.
The matrix deploy tees its ansible output and prints the compose
matrix-deploy round/PASS markers, so the runtime parser segments swarm
runs per variant round too.
The runtime table unifies tasks and roles with a Type column (tasks
first), appends the recap total row and flags a mismatch between the
parsed sum and the recap total; rows are read from ROLES RECAP sections
only, which removes the double-counted TASKS RECAP task rows. Each role
row carries one column per host: executed, skipped or failed, with
ignored failures downgraded. The Playwright variant column drops the
redundant variant- prefix. Verified against the logs of runs
29656268771 (swarm) and 29620158441 (compose): every segment's parsed
sum matches the recap total within 0.05s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The context derived PRIMARY_NFS_VOLUME from the first declared volume, but gitlab's first volume is repositories with nfs: false (gitaly cannot run on NFS), so the seed step wrote its pre-drain marker into a state directory that never exists on the export and killed a job whose deploy had already converged. Volumes declaring nfs: false are now skipped, making the drain check target a volume that actually lives on the export (gitlab_shared).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The discovery SPOT (cli.meta.ci.query) now runs one query shape per mode:
compose and host stay whole-role with variants packed into bundles, swarm
ranks and budget-cuts on per-variant rows and emits role#variant tokens
through apps.sh into variant_bundles, so the cut can pick a subset of a
role's variants and the --matrix view is the selection priority.
Replace the --unique flag with visible dna/clone columns (heaviest role
per dna cluster stays ahead of the cut), add the integrated column and
sort, count host bundles like compose, and give the deploy plan an id
counter, per-variant weights and display-width-aware alignment matching
the matrix renderer. Add i8ma*/i8pa* inspection aliases per mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dedicated.sh streams rescue diagnostics via docker exec tar since docker cp cannot see the systemd container's mount namespace and silently extracted nothing. recover_drill.sh guards on docker availability: non-stack role deploys now drill the nfs and secrets legs instead of hard-failing on the impossible volume leg; stack jobs still drill all three. 04_stack_rm_wait tolerates an already-removed stack and re-issues the rm once services are gone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In compose/host mode the whole nested app stack lives inside INFINITO_CONTAINER, which dedicated.sh's EXIT-trap cleanup removes. The separate workflow-level rescue step ran container.py on the runner AFTER that teardown, so it saw an emptied Docker and captured only 12 host files (unlike swarm, whose DiD nodes outlive the deploy step and get recursed into).
Run the recursive container.py capture inside INFINITO_CONTAINER during cleanup, before the existing copy-out and the teardown, so it descends into the app containers. Drop the now-redundant, post-teardown 'Collect rescue diagnostics' step from the compose and host workflows (runner state stays covered by dump_state.sh; the upload step keeps uploading what dedicated.sh writes). container.py is now called once per mode: inside dedicated.sh for compose/host, at runner level for swarm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The drill entry gate treated 'deploys no swarm service' as 'has nothing
to restore', but the backup chain operates on NFS volumes and deployed
backup units, not on scheduler objects; bigbluebutton's seven
NFS-backed volumes went entirely unproven. Gate the drill solely on the
NFS-flagged volume and make only the full-disaster step
workload-aware: stack roles keep docker stack rm, node-local roles get
their compose project (label com.docker.compose.project=<entity>, the
compose handler's --project convention) stopped on every node so no
writer races the export wipe and restore. Declare
workload: node-local on bigbluebutton so the phantom-service
converge/chaos gates skip while play, e2e and now the drill keep
running; svc-runner stays skipped via its missing NFS volume. Document
the nine drill steps host-by-host in the README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The marker discovery matched any .dr-drill-marker under files/, so
SRC_REL could resolve to a parent tree and step 8 restored the wrong
subtree into the live export as root, leaving n8n_data owned root:root
and n8n (uid 1000) crash-looping on EACCES. Scope both repo layouts to
the primary volume's own marker and declare uid/gid 1000 on the n8n
data volume so nfs_prep enforces ownership on every deploy instead of
relying on Docker's copy-on-first-use.
The drill also stacked four copies of the backup tree on shared space
(live export, pulled tree, fully-allocated LUKS image, final restore
into a tmpfs /tmp) and ENOSPCed on large apps. Restore to /var/tmp,
drop the pulled tree before the device recover, make the loop image
sparse and free it after the marker verify, and stage only the volume
generation instead of the whole machine tree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The swarm convergence gate derived runner_runner-1 from the services
metadata and polled a stack service the role never creates: the runner
fleet deploys as node-local compose projects on every node because each
runner needs its own node's docker socket and identity. Declare
workload: node-local on the runner services entry and let the gate
context map it to HAS_SWARM_SERVICE=false, so the stack-service gates
skip themselves while the swarm deploy, the registry distribution chain
and the CLI e2e keep running. Roles without the attribute keep the
image/template heuristic; workload: stack forces the gates on. Document
the runner architecture for both deploy modes in the role README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
