f660dd6bd introduced `${INFINITO_IMAGE_TAG:?}` in the image resolution, a hard requirement the replaced call deliberately did not have -- ci.sh exits 0 before its own assert when no registry owner is in scope. The check sat after the EXIT trap was armed and before the env layer is sourced, so a missing tag ran collect_and_teardown against a cluster that was never created, and the variable appeared in neither the # Param header nor the strict guard block that carries the other four required inputs.
Move it into that guard block, which runs before mktemp and before the trap, and drop the now-redundant inline assert.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every workspace leg pinned INFINITO_DISTRO=debian and drilled all five distros sequentially, pulling its swarm nodes from ghcr <distro>:latest -- an image from 2026-03-21 whose nfs-ganesha 7.2 segfaults on any accepted TCP connection on 2049. The controller mount then failed with ECONNREFUSED for 26 minutes and took run 30446813110 down after 5h13m, with 3 of 5 distros never reached.
Derive the distro from the leg's own matrix entry, build that image, and drill only it: swarm-zombie takes an optional distros= that defaults to INFINITO_DISTRO, and 00_one.sh prefers the locally built image over the published one. 12_swarm.sh refuses to start when that image is missing, so a broken build fails loudly instead of being masked by a stale registry pull.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f660dd6bd wrote the placeholder into a parameter expansion, `${INFINITO_IMAGE:-<none, bootstrap will build it>}`, which tests/lint/filesystem/shell/test_no_infinito_defaults.py rejects: an INFINITO_* default in a shell file is a second source next to default.env that drifts silently, and only the empty form is allowed. It was the sole failure of test-lint.
Split the message into the resolved and the unresolved case instead of suppressing the lint; the output is also no longer self-contradictory, since 'published <none>' could not both be true.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f660dd6bd introduced `${INFINITO_IMAGE_TAG:?}` in the image resolution, a hard requirement the replaced call deliberately did not have -- ci.sh exits 0 before its own assert when no registry owner is in scope. The check sat after the EXIT trap was armed and before the env layer is sourced, so a missing tag ran collect_and_teardown against a cluster that was never created, and the variable appeared in neither the # Param header nor the strict guard block that carries the other four required inputs.
Move it into that guard block, which runs before mktemp and before the trap, and drop the now-redundant inline assert.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every workspace leg pinned INFINITO_DISTRO=debian and drilled all five distros sequentially, pulling its swarm nodes from ghcr <distro>:latest -- an image from 2026-03-21 whose nfs-ganesha 7.2 segfaults on any accepted TCP connection on 2049. The controller mount then failed with ECONNREFUSED for 26 minutes and took run 30446813110 down after 5h13m, with 3 of 5 distros never reached.
Derive the distro from the leg's own matrix entry, build that image, and drill only it: swarm-zombie takes an optional distros= that defaults to INFINITO_DISTRO, and 00_one.sh prefers the locally built image over the published one. 12_swarm.sh refuses to start when that image is missing, so a broken build fails loudly instead of being masked by a stale registry pull.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm branch predates the shell-or-true lint, so its EXIT-trap collectors
arrived unmarked. They are the legitimate case: best-effort diagnostics and
teardown that must not mask the original exit code, which the trailing
return "${rc}" preserves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mapper name is host-global and nothing closes it once a drill succeeds:
the only teardowns sit inside the recover step and in the post-luksOpen
error path. A preserved cluster therefore hands the next round a live
'usbdrill' and cryptsetup refuses with 'Device usbdrill already exists'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test-environment.yml and test-guide.yml become test-workspace.yml and
test-instructions.yml. Orchestrator job ids, the distros-SPOT matrix
lint, the act example, the diagnostics slug and the rescue artefact
names follow.
Replace the roles_only input with per-job workspace/instructions
tri-state inputs (auto|true|false). 'auto' derives the former
roles_only skip straight from whitelist: both jobs run on a global
scope (whitelist empty or __ALL__) and are skipped on a role-scoped
run, which keeps the PR behaviour identical because the diff resolver
never emits an empty whitelist. entry-manual exposes both as choice
dropdowns defaulting to false, so a manual run opts in explicitly.
The now redundant roles_only output is dropped from
pr_affected_roles.sh and subset_roles.py.
Branch-protection required-status-check names change along with the
job renames.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The artifact of the wedged run carried journal, unit status and mounts but no process data, so the hang could be localized to the export config commit and not pinned to a syscall. wchan and syscall per thread close that gap.
MainPID 0 is its own likely state given Restart=on-failure with the start-rate limit disabled, so that branch reports the unit counters instead. Every read keeps 2>&1 because /proc stack and syscall are EPERM on hardened kernels.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The LUKS step of the DR drill died on `The file /dev/mapper/usbdrill does
not exist and no size was specified.` -- from mkfs, not from cryptsetup,
because luksOpen had exited 0. The node did have a udev, and it had
created the mapper entry: as a symlink to ../dm-1. Modern udev no longer
mknods, it only manages symlinks, and the /dev/dm-N target it points at is
a devtmpfs node that a container's tmpfs /dev never receives. Without a
udev libdevmapper creates the node itself, which is why the debian nodes
passed -- their image ships no udevd at all, while arch cannot install
systemd without one.
Masking udevd in the node image removes the whole class rather than one
call site. DM_DISABLE_UDEV rides on the node containers instead of the
script because the drill opens the volume a second time through the
recover CLI in a separate docker exec, which would not inherit an export.
The block-device check then fails loudly with both prerequisites named,
and closes the mapping first so an abort cannot leak it onto the runner --
device-mapper is not namespaced and every distro round shares the VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ganesha unregisters its RPC programs from inside nfs_Init_svc(), before
the sockets are ever put into listen(). That unregister reaches
libntirpc's rpcb_unset -> CLNT_DESTROY -> svc_vc_destroy_it, which spins
on nanosleep({0,0}) with no timeout -- and it only takes that path when
the AF_LOCAL connect to /var/run/rpcbind.sock succeeds. So a reachable
rpcbind is what wedges the server, not a missing one.
The wedged instance burned 5m30s of CPU while serving nothing, never
logged NFS SERVER INITIALIZED, and left the controller mount retrying
into ECONNREFUSED for 1706s before the deploy failed 20 minutes further
downstream. Masking and stopping both rpcbind units before ganesha starts
makes the spin unreachable: without a local socket rpcb_unset returns
false immediately. Wants=rpcbind.service is not Requires=, so a masked
unit is logged and ignored, and v4 registration is explicitly optional in
ganesha -- which is why this holds only while the export set stays
NFSv4-only.
The unit list is probed first so distros without rpcbind are skipped
rather than failing the play, and the probe distinguishes "no such unit"
from an unreachable manager so the mask can never be skipped silently.
The Type=forking unit reports started while the daemon is still spinning,
so a wait_for on 2049 turns a 28-minute mystery into a named failure, and
the diagnostics now carry rpcbind's journal plus what actually listens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node container has no udev, so LOOP_CTL_GET_FREE hands out a minor whose
/dev node never appears; it also cannot load kernel modules, so those nodes
stay dead unless the host carries the loop driver. Materialize the nodes,
verify one can actually be attached, and expose the host prerequisite as a
bootstrap step instead of failing three hours in with a cryptsetup message.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The controller mount of DIR_VAR_LIB survived the teardown, so the next
distro cycle inherited an NFS session whose server container was already
gone: the kernel logged 'server not responding, timed out' and every
access under /var/lib/infinito wedged, taking the deploy down with it.
All five swarm jobs of the last manual run died this way, each on a
different distro but always on the third lab cycle.
Both halves of the old teardown were unable to clear it. The guard
`mountpoint -q` blocks on exactly the dead mount it is asked about, so
the timeout dropped the unmount that was needed; and `umount -lf` ran
without sudo on a non-root runner, so the EPERM went into `|| true`.
Detection now reads /proc/self/mountinfo, which touches no filesystem,
the unmount elevates when the caller is not root, and a mount that
survives all of it fails loudly instead of silently poisoning the next
cycle. The bootstrap runs the same release before bringing a lab up, so
a cycle whose teardown never ran cannot hand the mount on either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The loop may drop a distro when the budget runs out, and both skip paths fell through to an implicit exit 0, so a run that never touched two distros still reported success. That hid the coverage gap: arch, fedora and centos get no package cache (cache.override.yml is apt-only), are therefore the slowest, and are the first the remaining < max_seen heuristic drops.
Skipping stays non-fatal and the order stays randomised, but every distro now shows up in a GITHUB_STEP_SUMMARY table in execution order, as passed, failed with its rc, or skipped with the reason. The table is also rendered before an aborting exit, since that is the run whose table matters. Two contract tests cover it; the order assertion checks the logged shuf order rather than the input list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d9a4498b3 collapsed eleven CI steps into one, which broke the assumption every handover in the drill rested on: a $GITHUB_ENV write only comes back in the NEXT step. Inside one step nobody sees it. matrix.py reads os.environ["MGR_IP"], so every swarm job died with KeyError before the first deploy round; 05 -> 06 -> 07 passed INITIAL_NODE/NEW_NODE the same way and would have died next.
The node IPs now come from the topology SPOT: compose.yml already pins every node to those INFINITO_SWARM_*_IP keys, so the docker-inspect capture only read back what compose had set. It goes, together with the $GITHUB_ENV publish and lab_net. lab_domain_target reads the env instead, guarded by an assert task because lookup('env') yields '' for a missing key and an empty dnsmasq `address=/domain/` is valid syntax meaning NODATA -- the failure would have surfaced 30 retries later as a DNS problem.
INITIAL_NODE/NEW_NODE move to a SWARM_DRILL_ENV file, created fresh per distro so the previous distro's node cannot leak into the next drain. The out-of-drill callers of _context.sh keep working through a visible guard; both writers stay strict via :?.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The discover job was named after the full distro list and the plan summary printed all five, but test-deploy-swarm.yml never set INFINITO_DISTROS, so every run fell back to the default.env distro and only debian was ever exercised. The job name has been advertising coverage that did not exist.
The workflow now passes the distros input into the env and hands the drill to the shared loop from 1bbe7a0eb. Bootstrap, setup, matrix deploy, seed, drain and assert move out of eleven separate steps into routine/00_one.sh, which is what the loop runs once per distro.
Collection and teardown move with them, into an EXIT trap: it replaces the removed 'if: always()' and 'if: failure()' step conditions and returns the original exit code so cleanup cannot mask the deploy failure. The always() teardown step stays as a second net for a step that times out as a whole.
Playwright reports and rescue diagnostics gain a distro segment, matching the layout the compose path already writes, because a second distro would otherwise overwrite the first one's evidence. The structure summary carries the distro in its heading for the same reason.
The README described the old shape and is corrected: the loop entries are listed, topology_summary.sh was missing, and 01_bootstrap.sh is no longer 'one CI step'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The serial distro loop (shuffle, time budget, fast-fail skip, exit on the first failing distro) lived inline in scripts/tests/deploy/ci/all.sh and was therefore reachable only from the compose and host deploys. Swarm and guide have to iterate distros with the same semantics, so the loop moves into scripts/tests/deploy/distros.sh and takes the per-distro command as "$@".
ci/all.sh keeps the app-level preamble and execs the loop with the new ci/one.sh, which resolves the CI image and dispatches to the per-app override or dedicated.sh.
Image resolution leaves the loop as well: scripts/meta/resolve/image/ci.sh renders the reference through cli.meta.ci.image_ref instead of respelling ghcr.io/... locally, which meta/distros.yml declares as the single source.
The two CI time knobs are declared in default.env instead of being repeated as literals, so no caller has to invent a budget.
Three paths are dropped as dead: the empty budget that disabled the limit, the failed counter that could only ever print 0, and the branch for an unset deadline.
Both new scripts are pinned by contract tests. Those tests clear BASH_ENV because the test container exports it as the env loader, which would otherwise re-inject every stripped key and make the assertions pass for the wrong reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
moby/buildkit was hardcoded in four places across two workflows and catthehacker/ubuntu in two act scripts, none of them reachable by the image mirror: utils/docker/image/discovery.py only scans roles/**/meta/services.yml. So the one image the CI build cannot proceed without was pulled straight from Docker Hub, unmirrored, which is what broke the fedora build in run 30160077828.
Declare both as mirror-only services on svc-runner, following the test-e2e-playwright precedent, so the existing mirror pipeline picks them up unchanged. New scripts/meta/resolve/image/service_ref.sh reads image+version back out of the declaration and is the single source for consumers outside Ansible; its mode argument is mandatory (upstream never touches the network, mirror prefers the GHCR tag and falls back to upstream). The fallback is why the mirror job can keep running in parallel with the build instead of gating it.
ACT_PLATFORM_IMAGE moves to an env handler so act consumers read the declared value via BASH_ENV, and workflow.sh now demands it instead of defaulting to a literal. Operator overrides still win through eb.setdefault.
Verified: service_ref.sh in all four paths (upstream, mirror hit, missing mode, invalid mode); the mirror path matches GHCRProvider.image_base() character for character and resolves against the live registry anonymously; ACT_PLATFORM_IMAGE resolves through the real BASH_ENV mechanism; make quality green on all four targets.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill configured svc-bkp-remote-2-local with a fixed manager plus NFS provider list for every app, while the manager only runs a backup repository when the app's include closure pulls svc-bkp-volume-2-local or svc-bkp-secrets-2-local in. For an app whose container_backup consumer is statically off, such as svc-dns-unbound, user-backup never ran on the manager, so the pull was rejected with 'Permission denied (publickey,password)' and the fail-if-any wrapper took the whole unit down at drill step 4/9. Four roles reach that step today: svc-dns-unbound, svc-db-redis, svc-registry-cache and svc-registry-docker.
backup_repos.py expresses the placement once and matrix derives the provider list from it per round. derive_includes accepts the round's variant map because INFINITO_APP_VARIANTS only reaches the subprocess callers. The extras file keeps the device paths the drill reads back and loses the provider list nothing read from it, and the drill banners stop claiming backup legs that were never installed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill configured svc-bkp-remote-2-local with a fixed manager plus NFS provider list for every app, while the manager only runs a backup repository when the app's include closure pulls svc-bkp-volume-2-local or svc-bkp-secrets-2-local in. For an app whose container_backup consumer is statically off, such as svc-dns-unbound, user-backup never ran on the manager, so the pull was rejected with 'Permission denied (publickey,password)' and the fail-if-any wrapper took the whole unit down at drill step 4/9. Four roles reach that step today: svc-dns-unbound, svc-db-redis, svc-registry-cache and svc-registry-docker.
backup_repos.py expresses the placement once and matrix derives the provider list from it per round. derive_includes accepts the round's variant map because INFINITO_APP_VARIANTS only reaches the subprocess callers. The extras file keeps the device paths the drill reads back and loses the provider list nothing read from it, and the drill banners stop claiming backup legs that were never installed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Restarting docker does not clear a container whose shim never sent an exit
event: that shim belongs to containerd, a separate unit that survives the
restart. Measured on a host carrying two such remnants - docker.service had
been restarted hours earlier while containerd.service had been up for three
days, and `rm -f`, plain `rm` and `stop -t 1` all still failed afterwards.
The hint also has to say what the restart costs. It wipes every exec instance,
so following it during a deploy kills the run: an act deploy died after 1h48m
with `failed to inspect exec: No such exec instance`, and the log carried no
ansible error at all, only post-steps failing in milliseconds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every teardown left one node container behind, always the same one, and the
warning blamed a kernel D-state. The cause was a budget: the client-unmount
step gave all four containers one shared 120s allowance, the unmount chain
costs up to three attempts per mount, and the third container in the list never
got its turn. That container was then killed with its mount still attached.
Killing was the second half of it. The node image sets STOPSIGNAL SIGRTMIN+3 so
that `docker stop` triggers a clean systemd poweroff, but the teardown only ever
ran `docker kill`, which skips it and leaves the nested engine and its task
containers to die mid-flight. Those task containers hold binds of the NFS
superblock inside their own mount namespaces, where an unmount in the node
namespace cannot reach them, so the nested engine has to go down first.
Under act the ansible controller is the act job container itself, so its own
NFS mount is invisible to the host mountpoint check and survived every cleanup.
Three more paths reached the same containers without any of this: the
cross-cluster reaper removed them with no unmount at all and then reported the
wedges it had just manufactured, the bootstrap pre-clean detached only inside
the NFS server, and the escape hatch filtered its candidates down to
`name=nfs-server` so it could never see the worker that actually survives.
The removal verdict itself was wrong: a single inspect fired immediately after
`rm -f` reports containers the daemon is still removing, and two of the three
warnings in the observed run were false alarms.
Verified by tearing a live five-node cluster down: all five containers removed,
no remnant, no warning.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
moby/buildkit was hardcoded in four places across two workflows and catthehacker/ubuntu in two act scripts, none of them reachable by the image mirror: utils/docker/image/discovery.py only scans roles/**/meta/services.yml. So the one image the CI build cannot proceed without was pulled straight from Docker Hub, unmirrored, which is what broke the fedora build in run 30160077828.
Declare both as mirror-only services on svc-runner, following the test-e2e-playwright precedent, so the existing mirror pipeline picks them up unchanged. New scripts/meta/resolve/image/service_ref.sh reads image+version back out of the declaration and is the single source for consumers outside Ansible; its mode argument is mandatory (upstream never touches the network, mirror prefers the GHCR tag and falls back to upstream). The fallback is why the mirror job can keep running in parallel with the build instead of gating it.
ACT_PLATFORM_IMAGE moves to an env handler so act consumers read the declared value via BASH_ENV, and workflow.sh now demands it instead of defaulting to a literal. Operator overrides still win through eb.setdefault.
Verified: service_ref.sh in all four paths (upstream, mirror hit, missing mode, invalid mode); the mirror path matches GHCRProvider.image_base() character for character and resolves against the live registry anonymously; ACT_PLATFORM_IMAGE resolves through the real BASH_ENV mechanism; make quality green on all four targets.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The failover-drill assert polled service convergence for 60 iterations (120s).
OpenProject web boots in ~140s, so after 06_drain_worker rescheduled it the
replicas turned healthy ~20s after the poll gave up, and 07_assert_state.sh
reported 'Replicas not fully converged: 1/3' and exited 1 even though the stack
did converge moments later. The loop breaks early once converged, so raising the
ceiling to 120 iterations (240s) costs fast apps nothing and only widens the
window for heavy apps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
INFINITO_RESCUE_DIAGNOSTICS_BASE joins default.env as the only literal
of the runner-side collection root and is wired through the passthrough
handler into the generated .env. The four deploy workflows load it with
one grep bootstrap step after checkout instead of hardcoding the path in
their env blocks; upload paths and per-app DIR derivations reference the
variable. dedicated.sh sources it from .env alongside the existing
diagnostics keys and hard-requires it, no script-level default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conflicts: keep both new passthrough env vars; adopt the swarm-nfs
ollama manager-ops split while porting our model-pull retry hardening
into it. Strip three narration comments the stricter comment lint
rejects.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every DinD node is attached to a second network by the bootstrap cache connect, so the embedded DNS returns two A records per node name in undefined order; workers that resolved the manager onto the cache network could not reach the registry bound to the lab IP and rejected locally built images with 'No such image', pushing all replicas onto the manager. Pin a fixed lab ipv4_address per node and every node name via extra_hosts (files-first resolution wins over DNS); the subnet and IPs live in default.env as the INFINITO_SWARM_* SPOT consumed by compose interpolation and the topology exports. Proven by A/B: the identical friendica variant-2 swarm job went from worker rejects to fully green including the drain test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolves the runner Dockerfile to the lint-standard Acquire::Retries=3
and keeps the collabora TODO wording; hardens the merged tree for the
stricter lints (apt retries in the tor Dockerfile, shell-or-true
markers, comment cleanup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin node hostnames to their lab-net IPs (dual-homed nodes let Docker
DNS hand workers the cache-net IP while the registry binds only the
lab IP), quiesce the CI compose tor before bootstrap (it publishes the
same onion identity as the cluster), and reap wedged nodes with a
retried SIGKILL so leftovers cannot starve later phases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Back to the pre-swarm-nfs-fix baseline:
- 5c2c8f531 + 985ec7eab: the swarm writer-quiesce DR drill. Deep-dive
attribution of CI run 29715516546 found it the dominant failure source
(17 of 23 jobs): [9b/9] handed-off generation mismatch, [6/9] quiesce
stack-rm network race, [4/9] remote-2-local pull stall, compose discourse
restore duplicate key.
- d6f980574: openproject swarm ingress to web:8080. Carried a leaking
proxy_internal_port lookup (no per-app default) that broke the seaweedfs
dependency front-proxy ('Key web not found'); the bypass-the-Apache-proxy
approach was never verified against downloads/X-Sendfile.
- 1dd153bad: xwiki replicas:1 + mem 3g. Did not resolve the xwiki superadmin
failure (still red in run 29715516546), so it is unverified.
Re-approach each from a clean baseline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up corrections to b449f8157 (swarm DR drill):
- export test.env in 06/07 via set -a so the child db restore/probe
scripts inherit BKP_TEST_DATABASES_CSV and BKP_TEST_RESTORE_BIN
instead of aborting at their :? guards before any restore runs
- move db_probe.sh/db_restore.sh into a db/ subfolder (probe.sh,
restore.sh) and update all callers, so they stop tripping the
prefix-grouped-files structure lint
- define TEST_E2E_SWARM_DRILL in group_vars/all and drop the now
redundant default() from test.env.j2 (satisfies the undefined-var
and spot-default lints together)
- fix a mermaid parse error in the drill README: a ; in a
sequenceDiagram note was read as a statement separator
make quality-high green (test + lint phases, 15 targets, 0 failures).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Defer Swarm SQL replay to the disposable DR drill so shared writers such as Discourse are stopped before the destructive restore. Verify the selected generation with before/after database probes while preserving the Compose restore path.
swarm-exec reaches a DinD node; reaching a swarm service container (moodle,
openldap) needed a nested docker exec. Add swarm-app-exec, which resolves one
running replica on the node and runs a base64-encoded command inside it, so
live in-container inspection stays on a sanctioned make helper. Adding a
second exec_* script tripped the prefix-group lint, so exec_node.sh moves to
exec/node.sh and the new one lands at exec/service.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
07_assert_state.sh sources _context.sh under set -o pipefail, and the printf | head-n1 that picked the first NFS volume raised a broken-pipe write error the moment head closed the pipe after line one. Any role with two or more NFS volumes (erpnext ships erpnext_sites and erpnext_logs) crashed the reschedule assertion before it could check the surviving marker. Plain parameter expansion returns the same first line with no pipe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The swarm deploy job now renders the same summary page as compose:
role runtimes, a swarm structure section (topology roles plus live node
list and service placement from the manager) and the Playwright table.
The matrix deploy tees its ansible output and prints the compose
matrix-deploy round/PASS markers, so the runtime parser segments swarm
runs per variant round too.
The runtime table unifies tasks and roles with a Type column (tasks
first), appends the recap total row and flags a mismatch between the
parsed sum and the recap total; rows are read from ROLES RECAP sections
only, which removes the double-counted TASKS RECAP task rows. Each role
row carries one column per host: executed, skipped or failed, with
ignored failures downgraded. The Playwright variant column drops the
redundant variant- prefix. Verified against the logs of runs
29656268771 (swarm) and 29620158441 (compose): every segment's parsed
sum matches the recap total within 0.05s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The context derived PRIMARY_NFS_VOLUME from the first declared volume, but gitlab's first volume is repositories with nfs: false (gitaly cannot run on NFS), so the seed step wrote its pre-drain marker into a state directory that never exists on the export and killed a job whose deploy had already converged. Volumes declaring nfs: false are now skipped, making the drain check target a volume that actually lives on the export (gitlab_shared).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
