A handler name was doing two jobs at once: the line an operator reads in the
play output, and the identifier a `notify:` binds to. The two pull in opposite
directions -- the log wants "restart the SSH service, with diagnostics", the
binding wants a token that survives quoting, YAML flow style and a shell
round-trip -- and a name that changes for readability silently changes which
handler runs.
Every handler that a notify reached by name now carries a `listen:` with that
exact wording, so the binding is explicit before anything is renamed. The
topics themselves become identifiers: `compose up` reads `compose-up`,
`Restart NFS ganesha` reads `restart-nfs-ganesha`, 22 of them across both
sides of every notify. Where the topic is assembled at runtime, both branches
moved together: the conditional in sys-service carries two literals, and the
LDAP renderer interpolates `import-{{ folder }}-ldif-files`, which is what
makes those handlers reachable at all.
Freed from the binding, the names take the emoji prefix every other task line
already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make quality-high stopped at autoformat: ruff reported PERF401 in the addon
variant-pinning lint, whose third check collected findings through a loop whose
entire body was findings.append(). It hands a generator to findings.extend() now.
The two sibling checks were not flagged -- their `continue` guard makes them
something other than a pure transformation.
Rewriting a finding collector can silence a rule while every test still passes,
so the rule was broken again afterwards: a dynamic addon pinned in variant 1
still reports all eight variants, the same set as before the change.
The rest is what the formatters rewrote by themselves -- a trailing blank line in
the ganesha task file and two over-long lines wrapped in the volume spot lints.
make quality-high is green end to end after this: four test suites, ansible-lint
0 failures over 3632 files at the production profile, eleven lint targets.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Penpot 0 of run 31658241728 died on the restart handler while the node was
healing itself. Its journal, in order:
20:46:47 nfs-ganesha.service: State 'stop-sigterm' timed out. Aborting.
20:46:58 nfs-ganesha.service: State 'stop-watchdog' timed out. Killing.
20:46:58 Main process exited, code=killed, status=9/KILL
20:46:58 Failed to start nfs-ganesha.service
20:47:00 [main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
20:47:01 systemd[1]: Started nfs-ganesha.service - NFS-Ganesha file server.
Ansible failed at 20:46:58, three seconds before the replacement was serving.
The rescue bundle confirms the end state independently: 17 threads, RPC workers
in svc_rqst_epoll_loop's epoll_wait, fridge threads idle -- a healthy server.
9b52a966c built exactly this recovery (ExecStartPost turns a wedge into a unit
failure, Restart=on-failure replaces the process, a wait_for on 2049 absorbs one
cycle). The defect was that the recovery sat downstream of the handler that
died on the event it exists for: `systemd: state=restarted` reports the restart
job's result, and the play never reached the gate.
A shell script now owns the whole statement -- restart or start, then wait for
2049, exit 0 only once the port answers, exit 1 with the unit's status and
journal. systemctl's own result is deliberately not the verdict, because a
recovering unit and a broken one both read 'failed' at the moment it returns;
the difference only exists two seconds later, which is what the wait measures.
The handler is that SPOT's restart entry point and the task after it is the
start one; vars carries unit, port and budget once.
That removes the tolerance instead of relocating it. Gone with it: a
`failed_when: false`, a dead `register`, a handler that set a fact instead of
restarting, three `|| true` against this repo's own no-or-true rule (unnecessary
anyway -- the script sets -u, not -e), and the now-duplicated wait_for gate.
`changed_when` on both call sites reads the script's CHANGED marker, so a start
that had to act reports changed and one that found the unit already active does
not; previously the task claimed `changed_when: false` even when it had started
the service itself.
Measured, and in the image it will run in (infinito-swarm-node, the same one the
rescue bundle came from):
- all four combinations: start+inactive -> CHANGED, start+active -> no change,
restart -> CHANGED either way
- systemctl failing while the port answers -> rc 0; port never opening -> rc 1
with systemctl's own rc and message, status and journal on stderr
- bash 5.2.37, /usr/bin/systemctl, /usr/bin/journalctl present; /dev/tcp works
- the script run inside that image with no systemd at all still judges
correctly, because its verdict is the port
TimeoutStopSec stays at 10. The first attempt at this raised it to 90, which
9b52a966c rules out in writing -- the wedged process blocks SIGTERM, so a longer
budget only burns the recovery cycle the 120s gate has to hold. The merge-fix
search caught that before it was committed.
Not verified: that ansible.builtin.script's temp directory is executable on
every node (it was exercised here through docker run, not through the module),
that 120s covers a real recovery beyond the single three-second data point, and
the three sibling node images.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-penpot 2 failed with 'Port 2049 never opened'. ganesha had not
crashed: the unit was active, 5 threads, 18ms CPU, and the journal stopped
after the export CLIENT warnings without ever reaching NFS SERVER
INITIALIZED. All five threads sat on one futex word (FUTEX_WAIT_PRIVATE,
val 2) with no live thread holding it - an orphaned mutex in liburcu-bp,
entered from ReadExports -> fsal_cfg_commit -> urcu_bp_register. Bind_sockets()
is strictly downstream of ReadExports(), so nothing was ever bound.
This is not the Fedora 43 / ganesha 7.2 SEGV: that one reached listen state
and died on accept, 100% reproducible. This one never listens, alive and
idle, and hits roughly 1-3% of fedora legs - lam 2 started the same build
twice in the same run.
The recovery that already existed could not fire. Type=forking plus an
ExecStart parent that exited 0 means systemd considers the unit started, so
Restart=on-failure never saw a failure. ExecStartPost runs before the start
job completes, so a failing probe there is the event it has been waiting for.
Verified rather than assumed, with a throwaway transient unit carrying the
same properties: a Type=forking unit whose ExecStartPost fails ends in
'failed' with the same message this class produces in CI, Restart=on-failure
then schedules restarts 1, 2, 3, and StartLimitBurst stops it with 'Start
request repeated too quickly'. StartLimitIntervalSec was 0, i.e. unbounded -
without the burst cap this fix would trade a rare wedge for a silent loop.
The 20s budget is measured, not guessed: a healthy ganesha binds 2049 in
0.32-0.38s across eight successful gates in lam 2, so the probe sits ~57x
above normal. /usr/bin/timeout and /bin/bash were checked on all four node
images before hardcoding them.
TimeoutStopSec=10 is required, not cosmetic: the wedged process blocks
SIGTERM in main() and its sigwait consumer is created downstream of the
wedge, so every recovery cycle would otherwise burn ~45s to SIGABRT.
The Ansible gate goes 60s -> 120s because one recovery cycle is ~32s and 60
expires before the first one finishes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role started ganesha and then let the queued handler restart it a second later, so the instance the wait_for validated was never the one that had to survive. Enabling before the flush keeps the unit enabled at boot even when the handler fails, and the start after it degrades to a no-op.
Both unit-file writes now notify: changing the drop-in's OPTIONS was silently inert before. The handler carries no daemon_reload of its own, hence the dedicated reload task ahead of it.
The wait_for blamed nfs_Init_svc(). The journal of the wedged instance stops between the third and fourth client_commit, which is inside ReadExports().
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ganesha unregisters its RPC programs from inside nfs_Init_svc(), before
the sockets are ever put into listen(). That unregister reaches
libntirpc's rpcb_unset -> CLNT_DESTROY -> svc_vc_destroy_it, which spins
on nanosleep({0,0}) with no timeout -- and it only takes that path when
the AF_LOCAL connect to /var/run/rpcbind.sock succeeds. So a reachable
rpcbind is what wedges the server, not a missing one.
The wedged instance burned 5m30s of CPU while serving nothing, never
logged NFS SERVER INITIALIZED, and left the controller mount retrying
into ECONNREFUSED for 1706s before the deploy failed 20 minutes further
downstream. Masking and stopping both rpcbind units before ganesha starts
makes the spin unreachable: without a local socket rpcb_unset returns
false immediately. Wants=rpcbind.service is not Requires=, so a masked
unit is logged and ignored, and v4 registration is explicitly optional in
ganesha -- which is why this holds only while the export set stays
NFSv4-only.
The unit list is probed first so distros without rpcbind are skipped
rather than failing the play, and the probe distinguishes "no such unit"
from an unreachable manager so the mask can never be skipped silently.
The Type=forking unit reports started while the daemon is still spinning,
so a wait_for on 2049 turns a 28-minute mystery into a named failure, and
the diagnostics now carry rpcbind's journal plus what actually listens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the raw package modules at 76 call sites with package_install, moves
each role's distribution mapping into its own meta/packages.yml, and turns the
inventory PACKAGES variable from source-keyed literal names into a list of
declared package ids. The lint that forbids the raw modules is enabled here,
because the migration burns it down from 76 offenders to zero.
Removes sys-aur, sys-aur-install and sys-package. The AUR build user, the
passwordless pacman rule and the inventory-driven install are part of the plan
the action plugin executes, so the roles no longer carry them.
Declaring the mappings surfaced breakage the raw modules had hidden: on Debian
the `which` binary comes from debianutils, jrnl is in no Debian or Ubuntu
suite, chromium-browser is absent from Debian stable, JDK 11 is unobtainable on
RedHat, wireguard had no RedHat branch at all, qbittorrent was built from the
AUR although Arch ships it in extra, and desk-docker installed a Docker variant
that conflicts with the one sys-svc-container installs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ganesha package and service maps were indexed by
ansible_facts['os_family'] but keyed with distribution names, so the
Fedora and Ubuntu entries were unreachable and os_family RedHat matched
nothing. Host deploys on Fedora aborted with "object of type 'dict' has
no attribute 'RedHat'".
Key both maps by os_family and cover the whole default distro matrix:
- RedHat: Fedora ships nfs-ganesha in its base repo. Its 9.16 build is
linked against NTIRPC_7.2 symbols the current libntirpc does not
export, so ganesha.nfsd exits 127; pull libntirpc from updates-testing
to restore a runnable pair.
- RedHat on EL: nfs-ganesha is in no base, AppStream, CRB or EPEL repo,
only in the CentOS Storage SIG, so enable that repository first.
- Archlinux: no official package exists at all. Build it from the AUR
and ship the systemd unit, because the AUR package installs binaries
only.
Drop the unreachable distribution keys here and in the client role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
container.py now captures, per container, systemctl status and journalctl, plus host-level systemctl list-units. The SPOT rescue diagnostic reaches the top-level systemd nfs-server container, so a nfs-ganesha startup failure is saved without a role-local dump.
nfs-ganesha now logs to syslog via an OPTIONS override (-L SYSLOG) so late runtime failures reach journald and are captured by the generalized diagnostic. LOGFILE= was a no-op because the packaged unit expands ${OPTIONS}, never $LOGFILE. The bespoke ganesha probe/dump block is removed, superseded by the generalized capture.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ganesha flavor (dev/act swarm test only; prod uses kernel nfsd) had no restart policy. Ganesha FSAL_VFS can SEGV under DinD load, and a single crash took down every client NFS mount and failed the swarm deploy. Install a systemd drop-in (Restart=on-failure, RestartSec=2, StartLimitIntervalSec=0) and daemon_reload on start. Validated live: kill -s SEGV then the service self-restarts (nrestarts=1) and clients recover. Compose is unaffected (no NFS).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NFSv4 without a matching idmap Domain squashes non-root owners (www-data uid 33) to nobody (65534) on the client, breaking PHP ownership checks. Pin Domain=localdomain on the ganesha server flavor and the dev/act client, and drop the stale idmap cache. Scoped to dev/act so the production kernel flavor stays untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the [DEPRECATION WARNING] entries the compose/swarm deploys emitted:
- apt_repository (removed in ansible-core 2.25): write docker.list via copy +
an explicit apt update. deb822_repository is unusable here because
apt_repo.sh's sanitize_sources deletes .sources every run.
- INJECT_FACTS_AS_VARS (default deprecated in 2.24): use ansible_facts['<name>']
instead of the auto-injected top-level ansible_os_family / virtualization_* /
default_ipv4 vars.
- community.mysql.* (removed in community.mysql 6.0.0): switch to ansible.mysql
and pin the collection in requirements.galaxy.yml.
Add a compose deploy gate (assert_no_deprecation_warnings.sh + workflow step +
test) that fails on [DEPRECATION WARNING] in the deploy log, and a lint test
(test_no_deprecated_ansible_patterns) that forbids the three patterns above so
they cannot return.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The controller mount of the nested state export failed with server 'No such file or directory': over NFSv4 the controller must LOOKUP-traverse the base /srv/nfs export (real export, fsid=1) to reach the nested /srv/nfs/infinito-state (fsid=2), but a controller outside svc-swarm-node is absent from the base client list, so traversal is denied. Re-export the base read-only to SVC_STORAGE_NFS_SERVER_CONTROLLER_CIDRS (empty by default -> no extra exposure; in-node controllers are already listed; the swarm test sets the lab subnet). Worker nodes keep base rw via their explicit-IP lines (exports(5) host-specificity wins).
Add nfs_export_all_rw seboolean for the kernel flavor so RedHat SELinux-enforcing hosts (fedora/centos) serve the export; install python3-libselinux and re-gather the selinux fact first so the boolean is not silently skipped when the binding is absent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
svc-storage-nfs-server gains a backend flavor selected by services.nfs-server.flavor (default "{{ 'ganesha' if RUNTIME in ['dev','act'] else 'kernel' }}"). The ganesha flavor installs nfs-ganesha + FSAL_VFS and renders /etc/ganesha/ganesha.conf (NFSv4, Pseudo == Path so clients mount the real paths unchanged, Root_Squash on the export base and No_Root_Squash on the state subdir), reloaded via systemctl restart. It runs in the dev/act DinD swarm-test, where the host kernel nfsd is not network-namespaced and a privileged node driving exportfs would wedge it into a D-state. The kernel flavor keeps nfs-kernel-server + exportfs -ra for production and every non-dev/act runtime. 01_core dispatches on the flavor, 03_prepare holds the shared export preparation, flavor/{kernel,ganesha}/main.yml hold the per-backend install/render/service steps, and handlers gains the ganesha restart.
docs/agents/action/iteration: role.md renamed to compose.md (Compose Loop); new swarm.md for the swarm deploy loop split out of workflow.md; workflow.md trimmed to GitHub Actions workflow iteration; all cross-references updated; the duplicated README gotcha removed; the NFS gotcha rewritten to the ganesha design.
Also drops a retired-flag history comment in group_vars/all/01_modes.yml and a redundant comment in the init selection script, and allows WebFetch(repology.org) used during ganesha packaging research.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
