On ubuntu-26.04 runners the swarm deploy fails at svc-storage-nfs-client's
probe-write, right after a successful probe-mount, on every client node:
`Destination directory /run/svc-storage-nfs-client/probe does not exist`.
The rescue bundles cannot say why: ganesha writes to
/var/log/ganesha/ganesha.log, which the swarm diagnostics never collected,
and its NFS_V4 component logged nothing below EVENT.
Debug runs now log NFS_V4 at DEBUG, and the diagnostics append the tail
of ganesha.log to the nfs-boundary section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SOFTWARE_NAME, SOFTWARE_DOMAIN, SOFTWARE_URL and ORGANIZATION are declared once in group_vars/all/00_general.yml and derive from each other. Eleven templates wrote the value instead, so a rename would reach the four declarations and every consumer that reads them, and silently miss the literal.
The lint reads the name from the declaration rather than carrying it, and searches case-insensitively: the domain is the name lowered, the URL is https:// plus the domain and the organization is the name again, so one literal finds the value of all four. A hit is reported with the variable it should have been, derived from the scheme in front of it and its case. The escape is nocheck: software-identity, same line or the one above.
Every migrated template renders the same bytes as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role runs ganesha at -N NIV_EVENT, and roughly eighteen init steps between the CLIENT commits and the listener log below that level, so the silence in run 30699990912 marked the edge of visibility rather than the fault. Raise INIT and DISPATCH under MODE_DEBUG: nfs_Init_svc(), which binds 2049, logs under COMPONENT_DISPATCH, so INIT alone would have missed exactly the step in question.
Add a userspace backtrace beside the kernel stack dump, which only ever says futex_do_wait. eu-stack names the function holding the lock; elfutils goes into the lab image rather than the role, since a production NFS server should not carry a debugger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ganesha template used two non-existent option names (NLM_Param, Enable_RPCSTATD) that ganesha ignores as 'Unknown parameter', so RQUOTA stayed enabled and bound port 875. When rpc.statd took 875 first (PID-deterministic) ganesha crash-looped with a FATAL bind error (exit 2), so NFS 2049 never served and controller mounts failed with 'Connection refused'. Use the real NFS_CORE_PARAM booleans Enable_NLM/Enable_RQUOTA (both default true) to actually turn the sidebands off; the stack is NFSv4-only so neither is used. Verified on nfs-ganesha 4.3-8ubuntu1: corrected config starts, binds only 2049, serves a v4 mount, and stays up under the exact 875-occupied condition that crashes the old config.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
storage.nfs shrinks to the deployment-specific server address: export_base moves into svc-storage-nfs-server's services.yml (the server dictates where exports root) and the mount protocol version into svc-storage-nfs-client's services.yml (the client owns how it mounts). Both are read through utils.storage.nfs loaders by the nfs lookups, the env handler and 00_topology, eliminating the unrendered-variables class for these values; the group_vars assert shrinks accordingly and the swarm test extras stop restating role defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ganesha (dev/act) test backend sent name@domain owner attributes, but the swarm DinD clients mount with nfs4_disable_idmapping=Y (AUTH_SYS numeric passthrough) and decode names to nobody (65534). A named non-root service (matomo runs as www-data/33) then read its own files as 65534, so PHP session_start aborted with a 500 and matomo crashlooped, stalling the swarm deploy at converge. Set Only_Numeric_Owners so ganesha emits numeric ids matching the client; scoped to the ganesha flavor, so the production kernel-nfsd path is untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of the kernel-flavor swarm ENOENT: no export carried fsid=0, so the NFSv4 mount of the nested /srv/nfs/infinito-state sub-export had no pseudo-root anchor and the server returned NFS4ERR_NOENT on the PUTROOTFH+LOOKUP walk. crossmnt could not fix it (it only crosses a boundary mid-walk, never sets the starting filehandle). Kernel now exports the state dir itself at fsid=0 and clients mount the bare v4 root ':/' (zero LOOKUPs). ganesha keeps Pseudo=Path and v3 mounts the full path. The self-bind is kept so fsid=0 rides a real mount boundary (stable st_dev) under overlayfs DinD.
The NFS derivations move to per-value lookup plugins (nfs_flavor, nfs_client_src, nfs_fstype, nfs_mount_opts, nfs_state_path) backed by pure helpers in utils/storage/nfs.py with unit tests; group_vars STORAGE_NFS_* become thin lookup wrappers. The dev/act->ganesha flavor rule stays the single SPOT in the role config; nfs_flavor reads the merged value so client and server agree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The controller mount of the nested state export failed with server 'No such file or directory': over NFSv4 the controller must LOOKUP-traverse the base /srv/nfs export (real export, fsid=1) to reach the nested /srv/nfs/infinito-state (fsid=2), but a controller outside svc-swarm-node is absent from the base client list, so traversal is denied. Re-export the base read-only to SVC_STORAGE_NFS_SERVER_CONTROLLER_CIDRS (empty by default -> no extra exposure; in-node controllers are already listed; the swarm test sets the lab subnet). Worker nodes keep base rw via their explicit-IP lines (exports(5) host-specificity wins).
Add nfs_export_all_rw seboolean for the kernel flavor so RedHat SELinux-enforcing hosts (fedora/centos) serve the export; install python3-libselinux and re-gather the selinux fact first so the boolean is not silently skipped when the binding is absent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Real GitHub Actions and production use the kernel NFS flavor (RUNTIME not in dev/act); only local act uses ganesha. The kernel exports.j2 set no fsid, so the export base and its nested state subdir (one device, a bind-mount under test) get colliding auto-derived NFSv4 fsids; the controller's mount of /srv/nfs/infinito-state then fails with the server returning 'No such file or directory'. Add fsid=1 (base) and fsid=2 (state subdir), mirroring the ganesha flavor's Export_Id 1/2.
Extend the swarm test 12_collect_diagnostics.sh to dump the nfs-server /etc/exports, exportfs -v, ganesha.conf, export tree, nfsd status, and the controller's NFS reachability, so an export failure is diagnosable from the job log instead of only the client-side ENOENT.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
svc-storage-nfs-server gains a backend flavor selected by services.nfs-server.flavor (default "{{ 'ganesha' if RUNTIME in ['dev','act'] else 'kernel' }}"). The ganesha flavor installs nfs-ganesha + FSAL_VFS and renders /etc/ganesha/ganesha.conf (NFSv4, Pseudo == Path so clients mount the real paths unchanged, Root_Squash on the export base and No_Root_Squash on the state subdir), reloaded via systemctl restart. It runs in the dev/act DinD swarm-test, where the host kernel nfsd is not network-namespaced and a privileged node driving exportfs would wedge it into a D-state. The kernel flavor keeps nfs-kernel-server + exportfs -ra for production and every non-dev/act runtime. 01_core dispatches on the flavor, 03_prepare holds the shared export preparation, flavor/{kernel,ganesha}/main.yml hold the per-backend install/render/service steps, and handlers gains the ganesha restart.
docs/agents/action/iteration: role.md renamed to compose.md (Compose Loop); new swarm.md for the swarm deploy loop split out of workflow.md; workflow.md trimmed to GitHub Actions workflow iteration; all cross-references updated; the duplicated README gotcha removed; the NFS gotcha rewritten to the ganesha design.
Also drops a retired-flag history comment in group_vars/all/01_modes.yml and a redundant comment in the init selection script, and allows WebFetch(repology.org) used during ganesha packaging research.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The swarm-test rig provisions storage via utils/tests/swarm/write_extras.py
which emits storage.nfs as the complete {server, export_base, version}
triple, overwriting (not merging) the group_vars default. Any key nested
under storage.nfs that is NOT re-emitted by the provisioner is lost on
the test rig, which broke svc-storage-nfs-server with
"object of type 'dict' has no attribute 'state_subdir'" on the
e415f97b0 swarm baserow CI job (run 27226937629).
Move state_subdir out of the shadowable storage.nfs sub-dict to a
top-level constant STORAGE_NFS_STATE_SUBDIR, and DRY the four call
sites that previously hand-built `[export_base, state_subdir]|path_join`
into a single derived constant STORAGE_NFS_STATE_PATH. The provisioner
needs no change because the constants live outside its override target.
Migrates four consumers (svc-storage-nfs-server/tasks/03_exports.yml,
05_controller_mount.yml, templates/exports.j2 and
svc-storage-nfs-client/tasks/main.yml). Fixes Cluster A and the
cascading Cluster B (workers cannot probe-mount because
nfs-server never wrote /etc/exports).
Resolves CI swarm rc=69 ("msmtp: needs a password"): controller-side
Ansible lookups (lookup('email').password etc.) couldn't see the
tokens.yml that Mailu writes target-side, because the controller has
its own filesystem in swarm mode.
Mount NFS-shared /var/lib/infinito on every swarm host AND on the
controller during deploy (in-play delegate_to: localhost task at end
of svc-storage-nfs-server role). State subdir exported separately
with no_root_squash + wildcard CIDRs so root-owned writes succeed
and the controller can mount from outside the svc-swarm IP set.
Single-writer guarantee on FILE_TOKENS via delegate_to: STACK_HOST +
run_once on sys-token-store/{01_core,write}.yml. Backup roles
(svc-bkp-container-2-local, sys-ctl-cln-bkps, sys-ctl-cln-faild-bkps,
user-backup/03_permissions_folders, three direct callers of
04_seed-database-to-backup.yml) gated on IS_STACK_HOST to prevent
N-host write races on the shared NFS backup tree.
Fail-fast assertion in svc-storage-nfs-client + 05_controller_mount
when pre-existing local /var/lib/infinito would be hidden by mount-
over. Operator must migrate or wipe before re-deploying.
SPOTs: SWARM_LAB_NETWORK in workflow env block, storage.nfs.state_subdir
in group_vars/all/15_storage.yml, DIR_VAR_LIB unchanged. UPPER_CASE
constants in NFS roles. Cleanup script umounts controller view before
container teardown to prevent stale-mount leak between matrix jobs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the svc-docker-swarm, svc-docker-swarm-manager, svc-storage-nfs-server
and svc-storage-nfs-client roles, the compose_volumes filter NFS support,
the test-deploy-swarm-nfs CI workflow with its swarm-nfs-pilot script set,
and the deploy_mode-aware compose/swarm wiring across mediawiki, mariadb
and sys-svc-* helpers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
