Every swarm job on ubuntu-26.04 runners (kernel 7.0) died at the NFS
probe write with `[Errno 121] Remote I/O error`. ganesha's NFSv4 debug log
in the rescue bundles shows the cause on every client, the same second as
the failure:
Request 2: opcode 46 is OP_GET_DIR_DELEGATION
Status of OP_GET_DIR_DELEGATION in position 2 = NFS4ERR_OP_ILLEGAL
Linux 6.19 added directory delegations: after ACCESS, CREATE, REMOVE or
RENAME the client adds GET_DIR_DELEGATION to the directory's next GETATTR
(`should_request_dir_deleg`, gated per request by the nfsv4 parameter
`directory_delegations`). nfs-ganesha before 9.11 routes op 46 to
`nfs4_op_illegal`, whose reply carries opcode 10044; the client decoder
expects 46 and fails the call with EREMOTEIO. Upstream fixed it in
nfs-ganesha#1385 (98eb77c4, first in V9.11), which answers NOTSUPP; the
client then drops the capability. The distro packages the role installs
predate it: Debian trixie 6.5-5, Ubuntu 26.04 6.5-8.
Every client and the controller now read the server's installed
nfs-ganesha release and, below 9.11, set `directory_delegations=N` right
after the mount and before the first access (the mount itself never sends
op 46), plus `options nfsv4 directory_delegations=N` in /etc/modprobe.d for
later module loads. No ganesha release grants directory delegations, so
the end state equals a fixed server. The parameter is kernel-wide and is
only ever set to N, so clients sharing one kernel cannot flip it back on
each other. Servers on 9.11 or later and the kernel nfsd flavor keep the
client default.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The guarantees were asserted against the declared swarm and data pair,
so they proved two meshes work rather than that the planner works. A
deployment declares its own, and the review asks for any number of them with
the nodes arranged however the client wants.
The same invariants now run over generated topologies: one to eight meshes,
three arrangements each, hosts dealt out so some belong to several meshes and
some to none. 2247 subtests, four seconds, and the planner needed no change to
pass them.
`WIREGUARD_ENABLED` is `DEPLOYMENT_MODE == 'swarm'`, so it is true on
both arms of the vpn axis and the export ACL carried the mesh pools even on
a run that deploys no mesh at all.
The NFS host joins `svc-net-wireguard` only when the mesh is on, so its own
membership is the discriminator the ACL needs.
Both readers move together: the assert and the when that unmounts a foreign
source. The controller runs locally today, so this keeps the two call sites
reading one contract rather than fixing only the one that broke.
The export ACL was built from each swarm node's default_ipv4 fact, which is its
underlay address. A client mounting over WireGuard arrives from its mesh
address instead, so the server refused every mount from the cluster: 'mount
system call failed', not a timeout, because the packets reached it and were
rejected.
The pool is allowed as a range rather than the members enumerated, because a
worker is not a data-plane member and reaches the export through the hub, so
which address it presents depends on that hop. The pool is the VPN itself, and
only its members can source from it.
The delegated controller mount used the cluster source, so once the NFS server
was addressed on the mesh it timed out: 'mount.nfs4: Connection timed out for
10.100.1.11'. The controller is deliberately not a mesh member, so that address
was never reachable from there.
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A handler name was doing two jobs at once: the line an operator reads in the
play output, and the identifier a `notify:` binds to. The two pull in opposite
directions -- the log wants "restart the SSH service, with diagnostics", the
binding wants a token that survives quoting, YAML flow style and a shell
round-trip -- and a name that changes for readability silently changes which
handler runs.
Every handler that a notify reached by name now carries a `listen:` with that
exact wording, so the binding is explicit before anything is renamed. The
topics themselves become identifiers: `compose up` reads `compose-up`,
`Restart NFS ganesha` reads `restart-nfs-ganesha`, 22 of them across both
sides of every notify. Where the topic is assembled at runtime, both branches
moved together: the conditional in sys-service carries two literals, and the
LDAP renderer interpolates `import-{{ folder }}-ldif-files`, which is what
makes those handlers reachable at all.
Freed from the binding, the names take the emoji prefix every other task line
already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make quality-high stopped at autoformat: ruff reported PERF401 in the addon
variant-pinning lint, whose third check collected findings through a loop whose
entire body was findings.append(). It hands a generator to findings.extend() now.
The two sibling checks were not flagged -- their `continue` guard makes them
something other than a pure transformation.
Rewriting a finding collector can silence a rule while every test still passes,
so the rule was broken again afterwards: a dynamic addon pinned in variant 1
still reports all eight variants, the same set as before the change.
The rest is what the formatters rewrote by themselves -- a trailing blank line in
the ganesha task file and two over-long lines wrapped in the volume spot lints.
make quality-high is green end to end after this: four test suites, ansible-lint
0 failures over 3632 files at the production profile, eleven lint targets.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Penpot 0 of run 31658241728 died on the restart handler while the node was
healing itself. Its journal, in order:
20:46:47 nfs-ganesha.service: State 'stop-sigterm' timed out. Aborting.
20:46:58 nfs-ganesha.service: State 'stop-watchdog' timed out. Killing.
20:46:58 Main process exited, code=killed, status=9/KILL
20:46:58 Failed to start nfs-ganesha.service
20:47:00 [main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
20:47:01 systemd[1]: Started nfs-ganesha.service - NFS-Ganesha file server.
Ansible failed at 20:46:58, three seconds before the replacement was serving.
The rescue bundle confirms the end state independently: 17 threads, RPC workers
in svc_rqst_epoll_loop's epoll_wait, fridge threads idle -- a healthy server.
9b52a966c built exactly this recovery (ExecStartPost turns a wedge into a unit
failure, Restart=on-failure replaces the process, a wait_for on 2049 absorbs one
cycle). The defect was that the recovery sat downstream of the handler that
died on the event it exists for: `systemd: state=restarted` reports the restart
job's result, and the play never reached the gate.
A shell script now owns the whole statement -- restart or start, then wait for
2049, exit 0 only once the port answers, exit 1 with the unit's status and
journal. systemctl's own result is deliberately not the verdict, because a
recovering unit and a broken one both read 'failed' at the moment it returns;
the difference only exists two seconds later, which is what the wait measures.
The handler is that SPOT's restart entry point and the task after it is the
start one; vars carries unit, port and budget once.
That removes the tolerance instead of relocating it. Gone with it: a
`failed_when: false`, a dead `register`, a handler that set a fact instead of
restarting, three `|| true` against this repo's own no-or-true rule (unnecessary
anyway -- the script sets -u, not -e), and the now-duplicated wait_for gate.
`changed_when` on both call sites reads the script's CHANGED marker, so a start
that had to act reports changed and one that found the unit already active does
not; previously the task claimed `changed_when: false` even when it had started
the service itself.
Measured, and in the image it will run in (infinito-swarm-node, the same one the
rescue bundle came from):
- all four combinations: start+inactive -> CHANGED, start+active -> no change,
restart -> CHANGED either way
- systemctl failing while the port answers -> rc 0; port never opening -> rc 1
with systemctl's own rc and message, status and journal on stderr
- bash 5.2.37, /usr/bin/systemctl, /usr/bin/journalctl present; /dev/tcp works
- the script run inside that image with no systemd at all still judges
correctly, because its verdict is the port
TimeoutStopSec stays at 10. The first attempt at this raised it to 90, which
9b52a966c rules out in writing -- the wedged process blocks SIGTERM, so a longer
budget only burns the recovery cycle the 120s gate has to hold. The merge-fix
search caught that before it was committed.
Not verified: that ansible.builtin.script's temp directory is executable on
every node (it was exercised here through docker run, not through the module),
that 120s covers a real recovery beyond the single three-second data point, and
the three sibling node images.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-penpot 2 failed with 'Port 2049 never opened'. ganesha had not
crashed: the unit was active, 5 threads, 18ms CPU, and the journal stopped
after the export CLIENT warnings without ever reaching NFS SERVER
INITIALIZED. All five threads sat on one futex word (FUTEX_WAIT_PRIVATE,
val 2) with no live thread holding it - an orphaned mutex in liburcu-bp,
entered from ReadExports -> fsal_cfg_commit -> urcu_bp_register. Bind_sockets()
is strictly downstream of ReadExports(), so nothing was ever bound.
This is not the Fedora 43 / ganesha 7.2 SEGV: that one reached listen state
and died on accept, 100% reproducible. This one never listens, alive and
idle, and hits roughly 1-3% of fedora legs - lam 2 started the same build
twice in the same run.
The recovery that already existed could not fire. Type=forking plus an
ExecStart parent that exited 0 means systemd considers the unit started, so
Restart=on-failure never saw a failure. ExecStartPost runs before the start
job completes, so a failing probe there is the event it has been waiting for.
Verified rather than assumed, with a throwaway transient unit carrying the
same properties: a Type=forking unit whose ExecStartPost fails ends in
'failed' with the same message this class produces in CI, Restart=on-failure
then schedules restarts 1, 2, 3, and StartLimitBurst stops it with 'Start
request repeated too quickly'. StartLimitIntervalSec was 0, i.e. unbounded -
without the burst cap this fix would trade a rare wedge for a silent loop.
The 20s budget is measured, not guessed: a healthy ganesha binds 2049 in
0.32-0.38s across eight successful gates in lam 2, so the probe sits ~57x
above normal. /usr/bin/timeout and /bin/bash were checked on all four node
images before hardcoding them.
TimeoutStopSec=10 is required, not cosmetic: the wedged process blocks
SIGTERM in main() and its sigwait consumer is created downstream of the
wedge, so every recovery cycle would otherwise burn ~45s to SIGABRT.
The Ansible gate goes 60s -> 120s because one recovery cycle is ~32s and 60
expires before the first one finishes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role started ganesha and then let the queued handler restart it a second later, so the instance the wait_for validated was never the one that had to survive. Enabling before the flush keeps the unit enabled at boot even when the handler fails, and the start after it degrades to a no-op.
Both unit-file writes now notify: changing the drop-in's OPTIONS was silently inert before. The handler carries no daemon_reload of its own, hence the dedicated reload task ahead of it.
The wait_for blamed nfs_Init_svc(). The journal of the wedged instance stops between the third and fourth client_commit, which is inside ReadExports().
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ganesha unregisters its RPC programs from inside nfs_Init_svc(), before
the sockets are ever put into listen(). That unregister reaches
libntirpc's rpcb_unset -> CLNT_DESTROY -> svc_vc_destroy_it, which spins
on nanosleep({0,0}) with no timeout -- and it only takes that path when
the AF_LOCAL connect to /var/run/rpcbind.sock succeeds. So a reachable
rpcbind is what wedges the server, not a missing one.
The wedged instance burned 5m30s of CPU while serving nothing, never
logged NFS SERVER INITIALIZED, and left the controller mount retrying
into ECONNREFUSED for 1706s before the deploy failed 20 minutes further
downstream. Masking and stopping both rpcbind units before ganesha starts
makes the spin unreachable: without a local socket rpcb_unset returns
false immediately. Wants=rpcbind.service is not Requires=, so a masked
unit is logged and ignored, and v4 registration is explicitly optional in
ganesha -- which is why this holds only while the export set stays
NFSv4-only.
The unit list is probed first so distros without rpcbind are skipped
rather than failing the play, and the probe distinguishes "no such unit"
from an unreachable manager so the mask can never be skipped silently.
The Type=forking unit reports started while the daemon is still spinning,
so a wait_for on 2049 turns a 28-minute mystery into a named failure, and
the diagnostics now carry rpcbind's journal plus what actually listens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the raw package modules at 76 call sites with package_install, moves
each role's distribution mapping into its own meta/packages.yml, and turns the
inventory PACKAGES variable from source-keyed literal names into a list of
declared package ids. The lint that forbids the raw modules is enabled here,
because the migration burns it down from 76 offenders to zero.
Removes sys-aur, sys-aur-install and sys-package. The AUR build user, the
passwordless pacman rule and the inventory-driven install are part of the plan
the action plugin executes, so the roles no longer carry them.
Declaring the mappings surfaced breakage the raw modules had hidden: on Debian
the `which` binary comes from debianutils, jrnl is in no Debian or Ubuntu
suite, chromium-browser is absent from Debian stable, JDK 11 is unobtainable on
RedHat, wireguard had no RedHat branch at all, qbittorrent was built from the
AUR although Arch ships it in extra, and desk-docker installed a Docker variant
that conflicts with the one sys-svc-container installs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ganesha package and service maps were indexed by
ansible_facts['os_family'] but keyed with distribution names, so the
Fedora and Ubuntu entries were unreachable and os_family RedHat matched
nothing. Host deploys on Fedora aborted with "object of type 'dict' has
no attribute 'RedHat'".
Key both maps by os_family and cover the whole default distro matrix:
- RedHat: Fedora ships nfs-ganesha in its base repo. Its 9.16 build is
linked against NTIRPC_7.2 symbols the current libntirpc does not
export, so ganesha.nfsd exits 127; pull libntirpc from updates-testing
to restore a runnable pair.
- RedHat on EL: nfs-ganesha is in no base, AppStream, CRB or EPEL repo,
only in the CentOS Storage SIG, so enable that repository first.
- Archlinux: no official package exists at all. Build it from the AUR
and ship the systemd unit, because the AUR package installs binaries
only.
Drop the unreachable distribution keys here and in the client role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_task_file_numbering.py: numbered task
files use exactly two leading digits, a numbered *_core.yml must be
00_core.yml, per directory the numbers are unique/consecutive and start
at 01 (00 with a core file), numbered includes appear in ascending
order, and shared helpers live in tasks/utils/ instead of the numbered
chain.
Apply it across every role: rename NN_core.yml to 00_core.yml with a
consecutive down-shift of the siblings, move letter-suffix children into
a subdirectory named after their parent task, relocate helpers included
from more than one place into tasks/utils/, and update every reference
(include_tasks, path_absolute, .ansible-lint-ignore, requirement docs).
Legacy narration comments in the touched files are deleted; the upstream
xwiki.properties reference block opts out via nocheck. The unused
MAILU_IP6_PUBLIC var (only ever named in now-removed comments) is dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ganesha FSAL_VFS cannot initialise export roots on overlayfs (name_to_handle_at returns EOPNOTSUPP in init_export_root) and cannot claim tmpfs either (resolve_posix_filesystem ENOENT), so on CI/DinD nodes whose /srv/nfs sits on overlay every client mount fails with 'No such file or directory'. On overlay hosts inside the dev/act/github runtime, mount a 2G ext4 loopback image on the export base before rendering exports; refuse to mount over a non-empty base. Real hosts (RUNTIME=host) are never touched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
storage.nfs.server defaults to an empty string, so on a single-node or
self-mount deploy nfs_client_src builds the source as ":/srv/nfs/..." and
mount.nfs4 fails with "Failed to resolve server :". Default the server to
this node's own address when unset; the exporter is also the mount target
in that topology. The guard fires only on an empty server, so an explicitly
configured server is never overridden.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A prior deploy on a reused host leaves DIR_VAR_LIB ephemerally mounted (no fstab, but live). When the recomputed client src differs (flavor path or server IP), state: ephemeral refuses to override it and retries to failure. Read the live source with findmnt and unmount only when it is a non-empty differing src, then mount; a correctly-mounted src is left for the module to remount.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
allowed_ips was derived only from svc-swarm-node member IPs, so single-node/host mode (empty group) produced an empty list and the 'at least one allowed IP' assert hard-failed on localhost. Fall back to the host's own default_ipv4 (or 127.0.0.1) when the swarm-node list is empty; swarm mode keeps the member IPs unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
container.py now captures, per container, systemctl status and journalctl, plus host-level systemctl list-units. The SPOT rescue diagnostic reaches the top-level systemd nfs-server container, so a nfs-ganesha startup failure is saved without a role-local dump.
nfs-ganesha now logs to syslog via an OPTIONS override (-L SYSLOG) so late runtime failures reach journald and are captured by the generalized diagnostic. LOGFILE= was a no-op because the packaged unit expands ${OPTIONS}, never $LOGFILE. The bespoke ganesha probe/dump block is removed, superseded by the generalized capture.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
storage.nfs shrinks to the deployment-specific server address: export_base moves into svc-storage-nfs-server's services.yml (the server dictates where exports root) and the mount protocol version into svc-storage-nfs-client's services.yml (the client owns how it mounts). Both are read through utils.storage.nfs loaders by the nfs lookups, the env handler and 00_topology, eliminating the unrendered-variables class for these values; the group_vars assert shrinks accordingly and the swarm test extras stop restating role defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ganesha flavor (dev/act swarm test only; prod uses kernel nfsd) had no restart policy. Ganesha FSAL_VFS can SEGV under DinD load, and a single crash took down every client NFS mount and failed the swarm deploy. Install a systemd drop-in (Restart=on-failure, RestartSec=2, StartLimitIntervalSec=0) and daemon_reload on start. Validated live: kill -s SEGV then the service self-restarts (nrestarts=1) and clients recover. Compose is unaffected (no NFS).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NFSv4 without a matching idmap Domain squashes non-root owners (www-data uid 33) to nobody (65534) on the client, breaking PHP ownership checks. Pin Domain=localdomain on the ganesha server flavor and the dev/act client, and drop the stale idmap cache. Scoped to dev/act so the production kernel flavor stays untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the [DEPRECATION WARNING] entries the compose/swarm deploys emitted:
- apt_repository (removed in ansible-core 2.25): write docker.list via copy +
an explicit apt update. deb822_repository is unusable here because
apt_repo.sh's sanitize_sources deletes .sources every run.
- INJECT_FACTS_AS_VARS (default deprecated in 2.24): use ansible_facts['<name>']
instead of the auto-injected top-level ansible_os_family / virtualization_* /
default_ipv4 vars.
- community.mysql.* (removed in community.mysql 6.0.0): switch to ansible.mysql
and pin the collection in requirements.galaxy.yml.
Add a compose deploy gate (assert_no_deprecation_warnings.sh + workflow step +
test) that fails on [DEPRECATION WARNING] in the deploy log, and a lint test
(test_no_deprecated_ansible_patterns) that forbids the three patterns above so
they cannot return.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce plugins/lookup/path_absolute.py as the single source for the
repo-root anchor: lookup('path_absolute', 'roles/...') returns an absolute
path under playbook_dir, joining every term. Replace all hand-built
playbook_dir path constructions across .yml/.yaml and .j2 templates with it:
[playbook_dir, '<rel>'] | path_join -> lookup('path_absolute', '<rel>')
playbook_dir ~ '/<rel>' -> lookup('path_absolute', '<rel>')
{{ playbook_dir }}/<rel> -> lookup('path_absolute', '<rel>')
Unrelated repo_path symbols in cli/meta/roles/applications/resolution and the
single bare discover_playwright_roles(playbook_dir) call are left untouched.
Add tests/lint/ansible/test_playbook_dir_via_lookup.py: fails the build if any
of the three construction forms reappears in .yml/.yaml/.j2, pointing callers
at the lookup. Unit test covers the plugin.
Touching these files pulled their pre-existing narration comments into
test_comments_valid scope; clean them (delete restatement/banners, keep real
trip-wires as Workaround/Upstream/Invariant markers). Drop caution, important
and pitfall from the comment-lint marker vocabulary and remove the comments
that used them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consumer lookup plugins now reach the merged config through the Ansible
lookup loader instead of importing get_merged_applications /
get_merged_domains from utils.cache. The merge stays one cached
implementation surfaced via the applications/config/domains/users lookups;
consumers call lookup_loader.get(...).run() (the compose_networks idiom).
- get_merged_applications walks the whole meta/ tree recursively
(meta/**/*.yml to applications.<app>.<path>, addons and any depth) and
folds the raw meta/volumes.yml subtree onto applications.<app>.volumes
after the templar render so the Jinja source: strings stay unrendered.
- 31 consumer lookups rerouted; providers (applications/config/
applications_current_play/users/domains) and the raw-volume accessors
(volume/nginx) keep the cache import behind # nocheck: lookup-cache-import.
- New lint test_no_cache_imports_in_lookups enforces the contract.
- nfs_flavor is now the flavor SPOT (svc-storage-nfs-server + nfs_client_src
resolve via lookup('nfs_flavor')); removed the unused STORAGE_NFS_FLAVOR
group_var. Extracted extract_default_credentials to
utils/cache/schema_credentials.py to keep applications.py under the cap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of the kernel-flavor swarm ENOENT: no export carried fsid=0, so the NFSv4 mount of the nested /srv/nfs/infinito-state sub-export had no pseudo-root anchor and the server returned NFS4ERR_NOENT on the PUTROOTFH+LOOKUP walk. crossmnt could not fix it (it only crosses a boundary mid-walk, never sets the starting filehandle). Kernel now exports the state dir itself at fsid=0 and clients mount the bare v4 root ':/' (zero LOOKUPs). ganesha keeps Pseudo=Path and v3 mounts the full path. The self-bind is kept so fsid=0 rides a real mount boundary (stable st_dev) under overlayfs DinD.
The NFS derivations move to per-value lookup plugins (nfs_flavor, nfs_client_src, nfs_fstype, nfs_mount_opts, nfs_state_path) backed by pure helpers in utils/storage/nfs.py with unit tests; group_vars STORAGE_NFS_* become thin lookup wrappers. The dev/act->ganesha flavor rule stays the single SPOT in the role config; nfs_flavor reads the merged value so client and server agree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the inline "nfs4 if version>=4 else nfs" fstype expression (repeated across svc-storage-nfs-client and svc-storage-nfs-server) with a single STORAGE_NFS_FSTYPE in group_vars/all/15_storage.yml, next to the other STORAGE_NFS_* values it derives from.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
NFSv4 has no MOUNT protocol; the kernel nfsd only crosses from the base export into the nested distinct-fsid state export at a REAL mount boundary (nfsd_lookup_dentry -> nfsd_cross_mnt, gated on nfsd_mountpoint()). /srv/nfs/infinito-state is a plain same-filesystem subdirectory, not a mountpoint, so the cross is never attempted, the fsid=2 filehandle is never built, and the controller's direct mount returns NFS4ERR_NOENT. crossmnt/nohide cannot help (only consulted inside the cross, never reached); exportfs/showmount only show the export table, not the v4 pseudo-walk, which is why the child looked exported yet was unreachable. Bind-mount the state subdir onto itself (kernel flavor, before exportfs) so it becomes a real boundary; the existing fsid=2/no_root_squash child export then resolves. Ganesha (userspace, per-Export handle) was never affected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The controller mount of the nested state export failed with server 'No such file or directory': over NFSv4 the controller must LOOKUP-traverse the base /srv/nfs export (real export, fsid=1) to reach the nested /srv/nfs/infinito-state (fsid=2), but a controller outside svc-swarm-node is absent from the base client list, so traversal is denied. Re-export the base read-only to SVC_STORAGE_NFS_SERVER_CONTROLLER_CIDRS (empty by default -> no extra exposure; in-node controllers are already listed; the swarm test sets the lab subnet). Worker nodes keep base rw via their explicit-IP lines (exports(5) host-specificity wins).
Add nfs_export_all_rw seboolean for the kernel flavor so RedHat SELinux-enforcing hosts (fedora/centos) serve the export; install python3-libselinux and re-gather the selinux fact first so the boolean is not silently skipped when the binding is absent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dev/act ganesha server is NFSv4-only, but the swarm-test runtime extras forced storage.nfs.version: 3, so every mount failed 'Protocol not supported'. Set version 4 (write_extras + dev swarm inventory). The client probe mounted the Root_Squash export_base and a root probe-write failed; point it at the No_Root_Squash state path. Both client mounts hardcoded 'hard', so a torn-down test server wedged nodes into D-state; gate them to soft in dev/act (reusing the controller mount's gate). Hoist the shared mount-opts string into the STORAGE_NFS_MOUNT_OPTS SPOT (group_vars/all/15_storage.yml), used by all three mounts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
svc-storage-nfs-server gains a backend flavor selected by services.nfs-server.flavor (default "{{ 'ganesha' if RUNTIME in ['dev','act'] else 'kernel' }}"). The ganesha flavor installs nfs-ganesha + FSAL_VFS and renders /etc/ganesha/ganesha.conf (NFSv4, Pseudo == Path so clients mount the real paths unchanged, Root_Squash on the export base and No_Root_Squash on the state subdir), reloaded via systemctl restart. It runs in the dev/act DinD swarm-test, where the host kernel nfsd is not network-namespaced and a privileged node driving exportfs would wedge it into a D-state. The kernel flavor keeps nfs-kernel-server + exportfs -ra for production and every non-dev/act runtime. 01_core dispatches on the flavor, 03_prepare holds the shared export preparation, flavor/{kernel,ganesha}/main.yml hold the per-backend install/render/service steps, and handlers gains the ganesha restart.
docs/agents/action/iteration: role.md renamed to compose.md (Compose Loop); new swarm.md for the swarm deploy loop split out of workflow.md; workflow.md trimmed to GitHub Actions workflow iteration; all cross-references updated; the duplicated README gotcha removed; the NFS gotcha rewritten to the ganesha design.
Also drops a retired-flag history comment in group_vars/all/01_modes.yml and a redundant comment in the init selection script, and allows WebFetch(repology.org) used during ganesha packaging research.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add tests/lint/ansible/tasks/test_include_uniform_apply_keyword.py: when a file pulled in via a dynamic include_tasks carries the same hoistable task keyword on every top-level task, that keyword should be set once via apply: on the include site instead of being repeated per task. The rule is blacklist-based: it flags any uniform Ansible task keyword except the per-task-binding ones (name, register, args, vars, when, loop), and never flags the module itself.
Hoist every existing occurrence to apply: across the affected roles (delegate_to, run_once, become, notify, no_log, changed_when, delay, async, poll), dropping the per-task duplication. Behaviour is identical because apply: propagates each keyword to every included task; no_log masking via apply: was empirically verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Install mount.nfs helper on the controller (delegated, run_once) so the
ansible.builtin.mount task can actually invoke /sbin/mount.nfs when the
controller container ships without nfs-common (jrei/systemd-ubuntu in
the act-CI swarm rig).
Add nolock to controller and client mount opts. NFSv4 has built-in
lease-based locking so nolock is a no-op there. For the v3 fallback it
bypasses rpc.statd, which is not running in systemd-DinD containers and
otherwise blocks the mount handshake. Production NFSv4 deployments are
unaffected; v3 environments fall back to local file-locking which is
safe for the single-writer-per-volume access pattern of the cluster
state directory.
Adds a `swarm:` subcategory under `svc:` in roles/categories.yml so the
shared-cluster role is properly invokable, and renames the existing
`svc-swarm` role + inventory group to `svc-swarm-node` (with
`get_entity_name`-aligned primary key `node:` in meta/services.yml).
`svc-swarm-manager` stays as the manager-marker role, becoming invokable
via the same new category and re-keyed to `manager:`.
Consumers of the cluster-orchestration service (svc-registry-cache,
svc-registry-docker, svc-swarm-manager) now reference the service-key
`node:` (was `swarm:`), matching the renamed primary entry. The planner
empirically resolves `service_registry['node']` -> `svc-swarm-node` and
emits it into --include for stack-host deploys, so the CI failure on
`universal-svc-registry-docker` (`svc-swarm: not present in inventory`)
no longer reaches `ValidDeployId`.
Breaking change for operators: inventory groups named `svc-swarm:` must
be renamed to `svc-swarm-node:`. `DEPLOYMENT_MODE` reads
`groups['svc-swarm-node']`, so a stale `svc-swarm:` group silently falls
back to compose mode.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`ansible.builtin.script` preserves the trailing newline that `echo`
emits in `premount_state.sh`, so the assert compared `"empty\n"` /
`"mounted\n"` to `["mounted", "empty"]` and always failed on a fresh
runner, blocking all swarm-CI deploys before any role could run.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The swarm-test rig provisions storage via utils/tests/swarm/write_extras.py
which emits storage.nfs as the complete {server, export_base, version}
triple, overwriting (not merging) the group_vars default. Any key nested
under storage.nfs that is NOT re-emitted by the provisioner is lost on
the test rig, which broke svc-storage-nfs-server with
"object of type 'dict' has no attribute 'state_subdir'" on the
e415f97b0 swarm baserow CI job (run 27226937629).
Move state_subdir out of the shadowable storage.nfs sub-dict to a
top-level constant STORAGE_NFS_STATE_SUBDIR, and DRY the four call
sites that previously hand-built `[export_base, state_subdir]|path_join`
into a single derived constant STORAGE_NFS_STATE_PATH. The provisioner
needs no change because the constants live outside its override target.
Migrates four consumers (svc-storage-nfs-server/tasks/03_exports.yml,
05_controller_mount.yml, templates/exports.j2 and
svc-storage-nfs-client/tasks/main.yml). Fixes Cluster A and the
cascading Cluster B (workers cannot probe-mount because
nfs-server never wrote /etc/exports).
Resolves CI swarm rc=69 ("msmtp: needs a password"): controller-side
Ansible lookups (lookup('email').password etc.) couldn't see the
tokens.yml that Mailu writes target-side, because the controller has
its own filesystem in swarm mode.
Mount NFS-shared /var/lib/infinito on every swarm host AND on the
controller during deploy (in-play delegate_to: localhost task at end
of svc-storage-nfs-server role). State subdir exported separately
with no_root_squash + wildcard CIDRs so root-owned writes succeed
and the controller can mount from outside the svc-swarm IP set.
Single-writer guarantee on FILE_TOKENS via delegate_to: STACK_HOST +
run_once on sys-token-store/{01_core,write}.yml. Backup roles
(svc-bkp-container-2-local, sys-ctl-cln-bkps, sys-ctl-cln-faild-bkps,
user-backup/03_permissions_folders, three direct callers of
04_seed-database-to-backup.yml) gated on IS_STACK_HOST to prevent
N-host write races on the shared NFS backup tree.
Fail-fast assertion in svc-storage-nfs-client + 05_controller_mount
when pre-existing local /var/lib/infinito would be hidden by mount-
over. Operator must migrate or wipe before re-deploying.
SPOTs: SWARM_LAB_NETWORK in workflow env block, storage.nfs.state_subdir
in group_vars/all/15_storage.yml, DIR_VAR_LIB unchanged. UPPER_CASE
constants in NFS roles. Cleanup script umounts controller view before
container teardown to prevent stale-mount leak between matrix jobs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
svc-cache-registry -> svc-registry-cache
svc-docker-registry -> svc-registry-docker
svc-docker-swarm -> svc-swarm
svc-docker-swarm-manager -> svc-swarm-manager
categories.yml: drop svc.docker, add svc.registry. Deliberately no svc.swarm
category so get_entity_name('svc-swarm') stays 'swarm' instead of collapsing
to '' (which would break discover_role_services).
Entity keys updated to match the new svc.registry prefix:
services.cache-registry.* -> services.cache.*
services.registry.* -> services.docker.*
01_constructor: replace svc-docker bootstrap-loop entry with explicit
include_role of svc-swarm + bootstrap loop over [svc-registry, svc-storage].
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the svc-docker-swarm, svc-docker-swarm-manager, svc-storage-nfs-server
and svc-storage-nfs-client roles, the compose_volumes filter NFS support,
the test-deploy-swarm-nfs CI workflow with its swarm-nfs-pilot script set,
and the deploy_mode-aware compose/swarm wiring across mediawiki, mariadb
and sys-svc-* helpers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
