fix(nfs-server): turn a wedged ganesha into a unit failure

web-app-penpot 2 failed with 'Port 2049 never opened'. ganesha had not
crashed: the unit was active, 5 threads, 18ms CPU, and the journal stopped
after the export CLIENT warnings without ever reaching NFS SERVER
INITIALIZED. All five threads sat on one futex word (FUTEX_WAIT_PRIVATE,
val 2) with no live thread holding it - an orphaned mutex in liburcu-bp,
entered from ReadExports -> fsal_cfg_commit -> urcu_bp_register. Bind_sockets()
is strictly downstream of ReadExports(), so nothing was ever bound.

This is not the Fedora 43 / ganesha 7.2 SEGV: that one reached listen state
and died on accept, 100% reproducible. This one never listens, alive and
idle, and hits roughly 1-3% of fedora legs - lam 2 started the same build
twice in the same run.

The recovery that already existed could not fire. Type=forking plus an
ExecStart parent that exited 0 means systemd considers the unit started, so
Restart=on-failure never saw a failure. ExecStartPost runs before the start
job completes, so a failing probe there is the event it has been waiting for.

Verified rather than assumed, with a throwaway transient unit carrying the
same properties: a Type=forking unit whose ExecStartPost fails ends in
'failed' with the same message this class produces in CI, Restart=on-failure
then schedules restarts 1, 2, 3, and StartLimitBurst stops it with 'Start
request repeated too quickly'. StartLimitIntervalSec was 0, i.e. unbounded -
without the burst cap this fix would trade a rare wedge for a silent loop.

The 20s budget is measured, not guessed: a healthy ganesha binds 2049 in
0.32-0.38s across eight successful gates in lam 2, so the probe sits ~57x
above normal. /usr/bin/timeout and /bin/bash were checked on all four node
images before hardcoding them.

TimeoutStopSec=10 is required, not cosmetic: the wedged process blocks
SIGTERM in main() and its sigwait consumer is created downstream of the
wedge, so every recovery cycle would otherwise burn ~45s to SIGABRT.

The Ansible gate goes 60s -> 120s because one recovery cycle is ~32s and 60
expires before the first one finishes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

roles/svc-storage-nfs-server/files/nfs-ganesha-restart.conf

@@ -1,7 +1,10 @@
 [Unit]
-StartLimitIntervalSec=0
+StartLimitIntervalSec=300
+StartLimitBurst=3
 
 [Service]
 Restart=on-failure
 RestartSec=2
 Environment="OPTIONS=-L SYSLOG -f /etc/ganesha/ganesha.conf -N NIV_EVENT"
+ExecStartPost=/usr/bin/timeout 20 /bin/bash -c 'until (exec 3<>/dev/tcp/127.0.0.1/2049) 2>/dev/null; do sleep 1; done'
+TimeoutStopSec=10

roles/svc-storage-nfs-server/tasks/flavor/ganesha/main.yml

@@ -108,9 +108,11 @@
   ansible.builtin.wait_for:
     port: 2049
     host: 127.0.0.1
-    timeout: "{{ lookup('timeout', 60) | int }}"
+    timeout: "{{ lookup('timeout', 120) | int }}"
     msg: >-
       Port 2049 never opened within the timeout. Check 'systemctl status
       nfs-ganesha' and 'journalctl -u nfs-ganesha': a unit that stays active
       but logs nothing after the export CLIENT warnings has wedged before
-      'NFS SERVER INITIALIZED' and systemd will not restart it.
+      'NFS SERVER INITIALIZED'. ExecStartPost now turns that wedge into a
+      unit failure so Restart=on-failure can recover it; this budget covers
+      one full recovery cycle.