Every swarm job on ubuntu-26.04 runners (kernel 7.0) died at the NFS
probe write with `[Errno 121] Remote I/O error`. ganesha's NFSv4 debug log
in the rescue bundles shows the cause on every client, the same second as
the failure:
Request 2: opcode 46 is OP_GET_DIR_DELEGATION
Status of OP_GET_DIR_DELEGATION in position 2 = NFS4ERR_OP_ILLEGAL
Linux 6.19 added directory delegations: after ACCESS, CREATE, REMOVE or
RENAME the client adds GET_DIR_DELEGATION to the directory's next GETATTR
(`should_request_dir_deleg`, gated per request by the nfsv4 parameter
`directory_delegations`). nfs-ganesha before 9.11 routes op 46 to
`nfs4_op_illegal`, whose reply carries opcode 10044; the client decoder
expects 46 and fails the call with EREMOTEIO. Upstream fixed it in
nfs-ganesha#1385 (98eb77c4, first in V9.11), which answers NOTSUPP; the
client then drops the capability. The distro packages the role installs
predate it: Debian trixie 6.5-5, Ubuntu 26.04 6.5-8.
Every client and the controller now read the server's installed
nfs-ganesha release and, below 9.11, set `directory_delegations=N` right
after the mount and before the first access (the mount itself never sends
op 46), plus `options nfsv4 directory_delegations=N` in /etc/modprobe.d for
later module loads. No ganesha release grants directory delegations, so
the end state equals a fixed server. The parameter is kernel-wide and is
only ever set to N, so clients sharing one kernel cannot flip it back on
each other. Servers on 9.11 or later and the kernel nfsd flavor keep the
client default.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It was there to name the cause when the guard fired on a healthy host. The
cause is settled -- the whole-stream comparison, not the mount -- so the probe
goes back to reporting state alone.
Ansible allocates a pty whenever become needs a password, and a pty merges
stderr into stdout, so a caller matching the whole stream reads login noise as
the answer. The state is tagged so a match is a substring rather than the
whole stream.
The local-data branch also lists what it found, because 'migrate or remove the
local tree manually' is not actionable without it. Temporary: it comes out
once the transport question is settled.
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The controller mount of DIR_VAR_LIB survived the teardown, so the next
distro cycle inherited an NFS session whose server container was already
gone: the kernel logged 'server not responding, timed out' and every
access under /var/lib/infinito wedged, taking the deploy down with it.
All five swarm jobs of the last manual run died this way, each on a
different distro but always on the third lab cycle.
Both halves of the old teardown were unable to clear it. The guard
`mountpoint -q` blocks on exactly the dead mount it is asked about, so
the timeout dropped the unmount that was needed; and `umount -lf` ran
without sudo on a non-root runner, so the EPERM went into `|| true`.
Detection now reads /proc/self/mountinfo, which touches no filesystem,
the unmount elevates when the caller is not root, and a mount that
survives all of it fails loudly instead of silently poisoning the next
cycle. The bootstrap runs the same release before bringing a lab up, so
a cycle whose teardown never ran cannot hand the mount on either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolves CI swarm rc=69 ("msmtp: needs a password"): controller-side
Ansible lookups (lookup('email').password etc.) couldn't see the
tokens.yml that Mailu writes target-side, because the controller has
its own filesystem in swarm mode.
Mount NFS-shared /var/lib/infinito on every swarm host AND on the
controller during deploy (in-play delegate_to: localhost task at end
of svc-storage-nfs-server role). State subdir exported separately
with no_root_squash + wildcard CIDRs so root-owned writes succeed
and the controller can mount from outside the svc-swarm IP set.
Single-writer guarantee on FILE_TOKENS via delegate_to: STACK_HOST +
run_once on sys-token-store/{01_core,write}.yml. Backup roles
(svc-bkp-container-2-local, sys-ctl-cln-bkps, sys-ctl-cln-faild-bkps,
user-backup/03_permissions_folders, three direct callers of
04_seed-database-to-backup.yml) gated on IS_STACK_HOST to prevent
N-host write races on the shared NFS backup tree.
Fail-fast assertion in svc-storage-nfs-client + 05_controller_mount
when pre-existing local /var/lib/infinito would be hidden by mount-
over. Operator must migrate or wipe before re-deploying.
SPOTs: SWARM_LAB_NETWORK in workflow env block, storage.nfs.state_subdir
in group_vars/all/15_storage.yml, DIR_VAR_LIB unchanged. UPPER_CASE
constants in NFS roles. Cleanup script umounts controller view before
container teardown to prevent stale-mount leak between matrix jobs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5.5
evangelostsak