sys-daemon sets DefaultTimeoutAbortSec=90s host-wide, and it governs the
phase after the SIGABRT that a wedged ganesha (the liburcu-bp futex wedge
from f10a4a181) blocks. One probe-fail -> SIGTERM -> SIGABRT -> SIGKILL ->
restart cycle therefore took 20+10+90+2+1 = ~123s; f10a4a181 sized its
gates on ~32s per cycle, assuming the SIGABRT kills. Observed in run
32657210518 (Akaunting swarm drill, fedora): the drill's 120s try-restart
budget expired one second before the SIGKILL landed.
TimeoutAbortSec=10 cuts the cycle to ~43s. The wedged process never dumps
core, so the shorter grace costs nothing there; a genuinely aborting
ganesha may truncate its core dump - the documented trade-off of a short
TimeoutAbortSec. Without sys-daemon the abort phase already fell back to
TimeoutStopSec=10, so standalone behavior is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Penpot 0 of run 31658241728 died on the restart handler while the node was
healing itself. Its journal, in order:
20:46:47 nfs-ganesha.service: State 'stop-sigterm' timed out. Aborting.
20:46:58 nfs-ganesha.service: State 'stop-watchdog' timed out. Killing.
20:46:58 Main process exited, code=killed, status=9/KILL
20:46:58 Failed to start nfs-ganesha.service
20:47:00 [main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
20:47:01 systemd[1]: Started nfs-ganesha.service - NFS-Ganesha file server.
Ansible failed at 20:46:58, three seconds before the replacement was serving.
The rescue bundle confirms the end state independently: 17 threads, RPC workers
in svc_rqst_epoll_loop's epoll_wait, fridge threads idle -- a healthy server.
9b52a966c built exactly this recovery (ExecStartPost turns a wedge into a unit
failure, Restart=on-failure replaces the process, a wait_for on 2049 absorbs one
cycle). The defect was that the recovery sat downstream of the handler that
died on the event it exists for: `systemd: state=restarted` reports the restart
job's result, and the play never reached the gate.
A shell script now owns the whole statement -- restart or start, then wait for
2049, exit 0 only once the port answers, exit 1 with the unit's status and
journal. systemctl's own result is deliberately not the verdict, because a
recovering unit and a broken one both read 'failed' at the moment it returns;
the difference only exists two seconds later, which is what the wait measures.
The handler is that SPOT's restart entry point and the task after it is the
start one; vars carries unit, port and budget once.
That removes the tolerance instead of relocating it. Gone with it: a
`failed_when: false`, a dead `register`, a handler that set a fact instead of
restarting, three `|| true` against this repo's own no-or-true rule (unnecessary
anyway -- the script sets -u, not -e), and the now-duplicated wait_for gate.
`changed_when` on both call sites reads the script's CHANGED marker, so a start
that had to act reports changed and one that found the unit already active does
not; previously the task claimed `changed_when: false` even when it had started
the service itself.
Measured, and in the image it will run in (infinito-swarm-node, the same one the
rescue bundle came from):
- all four combinations: start+inactive -> CHANGED, start+active -> no change,
restart -> CHANGED either way
- systemctl failing while the port answers -> rc 0; port never opening -> rc 1
with systemctl's own rc and message, status and journal on stderr
- bash 5.2.37, /usr/bin/systemctl, /usr/bin/journalctl present; /dev/tcp works
- the script run inside that image with no systemd at all still judges
correctly, because its verdict is the port
TimeoutStopSec stays at 10. The first attempt at this raised it to 90, which
9b52a966c rules out in writing -- the wedged process blocks SIGTERM, so a longer
budget only burns the recovery cycle the 120s gate has to hold. The merge-fix
search caught that before it was committed.
Not verified: that ansible.builtin.script's temp directory is executable on
every node (it was exercised here through docker run, not through the module),
that 120s covers a real recovery beyond the single three-second data point, and
the three sibling node images.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-penpot 2 failed with 'Port 2049 never opened'. ganesha had not
crashed: the unit was active, 5 threads, 18ms CPU, and the journal stopped
after the export CLIENT warnings without ever reaching NFS SERVER
INITIALIZED. All five threads sat on one futex word (FUTEX_WAIT_PRIVATE,
val 2) with no live thread holding it - an orphaned mutex in liburcu-bp,
entered from ReadExports -> fsal_cfg_commit -> urcu_bp_register. Bind_sockets()
is strictly downstream of ReadExports(), so nothing was ever bound.
This is not the Fedora 43 / ganesha 7.2 SEGV: that one reached listen state
and died on accept, 100% reproducible. This one never listens, alive and
idle, and hits roughly 1-3% of fedora legs - lam 2 started the same build
twice in the same run.
The recovery that already existed could not fire. Type=forking plus an
ExecStart parent that exited 0 means systemd considers the unit started, so
Restart=on-failure never saw a failure. ExecStartPost runs before the start
job completes, so a failing probe there is the event it has been waiting for.
Verified rather than assumed, with a throwaway transient unit carrying the
same properties: a Type=forking unit whose ExecStartPost fails ends in
'failed' with the same message this class produces in CI, Restart=on-failure
then schedules restarts 1, 2, 3, and StartLimitBurst stops it with 'Start
request repeated too quickly'. StartLimitIntervalSec was 0, i.e. unbounded -
without the burst cap this fix would trade a rare wedge for a silent loop.
The 20s budget is measured, not guessed: a healthy ganesha binds 2049 in
0.32-0.38s across eight successful gates in lam 2, so the probe sits ~57x
above normal. /usr/bin/timeout and /bin/bash were checked on all four node
images before hardcoding them.
TimeoutStopSec=10 is required, not cosmetic: the wedged process blocks
SIGTERM in main() and its sigwait consumer is created downstream of the
wedge, so every recovery cycle would otherwise burn ~45s to SIGABRT.
The Ansible gate goes 60s -> 120s because one recovery cycle is ~32s and 60
expires before the first one finishes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ganesha package and service maps were indexed by
ansible_facts['os_family'] but keyed with distribution names, so the
Fedora and Ubuntu entries were unreachable and os_family RedHat matched
nothing. Host deploys on Fedora aborted with "object of type 'dict' has
no attribute 'RedHat'".
Key both maps by os_family and cover the whole default distro matrix:
- RedHat: Fedora ships nfs-ganesha in its base repo. Its 9.16 build is
linked against NTIRPC_7.2 symbols the current libntirpc does not
export, so ganesha.nfsd exits 127; pull libntirpc from updates-testing
to restore a runnable pair.
- RedHat on EL: nfs-ganesha is in no base, AppStream, CRB or EPEL repo,
only in the CentOS Storage SIG, so enable that repository first.
- Archlinux: no official package exists at all. Build it from the AUR
and ship the systemd unit, because the AUR package installs binaries
only.
Drop the unreachable distribution keys here and in the client role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
container.py now captures, per container, systemctl status and journalctl, plus host-level systemctl list-units. The SPOT rescue diagnostic reaches the top-level systemd nfs-server container, so a nfs-ganesha startup failure is saved without a role-local dump.
nfs-ganesha now logs to syslog via an OPTIONS override (-L SYSLOG) so late runtime failures reach journald and are captured by the generalized diagnostic. LOGFILE= was a no-op because the packaged unit expands ${OPTIONS}, never $LOGFILE. The bespoke ganesha probe/dump block is removed, superseded by the generalized capture.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ganesha flavor (dev/act swarm test only; prod uses kernel nfsd) had no restart policy. Ganesha FSAL_VFS can SEGV under DinD load, and a single crash took down every client NFS mount and failed the swarm deploy. Install a systemd drop-in (Restart=on-failure, RestartSec=2, StartLimitIntervalSec=0) and daemon_reload on start. Validated live: kill -s SEGV then the service self-restarts (nrestarts=1) and clients recover. Compose is unaffected (no NFS).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5