Files
kevinveenbirkenbachandClaude Opus 5.5 9b7f25f2fe fix(svc-storage-nfs-client): stop requesting directory delegations from ganesha < 9.11
Every swarm job on ubuntu-26.04 runners (kernel 7.0) died at the NFS
probe write with `[Errno 121] Remote I/O error`. ganesha's NFSv4 debug log
in the rescue bundles shows the cause on every client, the same second as
the failure:

  Request 2: opcode 46 is OP_GET_DIR_DELEGATION
  Status of OP_GET_DIR_DELEGATION in position 2 = NFS4ERR_OP_ILLEGAL

Linux 6.19 added directory delegations: after ACCESS, CREATE, REMOVE or
RENAME the client adds GET_DIR_DELEGATION to the directory's next GETATTR
(`should_request_dir_deleg`, gated per request by the nfsv4 parameter
`directory_delegations`). nfs-ganesha before 9.11 routes op 46 to
`nfs4_op_illegal`, whose reply carries opcode 10044; the client decoder
expects 46 and fails the call with EREMOTEIO. Upstream fixed it in
nfs-ganesha#1385 (98eb77c4, first in V9.11), which answers NOTSUPP; the
client then drops the capability. The distro packages the role installs
predate it: Debian trixie 6.5-5, Ubuntu 26.04 6.5-8.

Every client and the controller now read the server's installed
nfs-ganesha release and, below 9.11, set `directory_delegations=N` right
after the mount and before the first access (the mount itself never sends
op 46), plus `options nfsv4 directory_delegations=N` in /etc/modprobe.d for
later module loads. No ganesha release grants directory delegations, so
the end state equals a fixed server. The parameter is kernel-wide and is
only ever set to N, so clients sharing one kernel cannot flip it back on
each other. Servers on 9.11 or later and the kernel nfsd flavor keep the
client default.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 16:28:52 +02:00

137 lines
4.4 KiB
YAML

---
- name: "✅ Assert the NFS server is configured"
ansible.builtin.assert:
that:
- storage.nfs.server is defined
- storage.nfs.server | length > 0
fail_msg: |
svc-storage-nfs-client requires `storage.nfs.server` to be set in
the inventory (see group_vars/all/15_storage.yml). A reachable NFS
server IP/hostname is mandatory.
- name: "📦 Install the NFS client"
package_install:
id: nfs-client
- name: "🔧 Match NFSv4 idmap to the ganesha test server"
when: RUNTIME in ['dev', 'act']
block:
- name: "🔧 Set idmapd.conf Domain = localdomain"
ansible.builtin.lineinfile:
path: /etc/idmapd.conf
regexp: "^#?\\s*Domain\\s*="
line: "Domain = localdomain"
insertafter: "^\\[General\\]"
create: true
owner: root
group: root
mode: "0644"
- name: "🧹 Drop stale NFSv4 idmap cache entries"
ansible.builtin.command: nfsidmap -c
changed_when: false
failed_when: false
- name: "✅ Ensure probe mount point exists"
ansible.builtin.file:
path: "{{ SVC_STORAGE_NFS_CLIENT_PROBE_DIR }}"
state: directory
owner: root
group: root
mode: "0700"
- name: "🔍 Probe-mount the NFS export"
ansible.posix.mount:
path: "{{ SVC_STORAGE_NFS_CLIENT_PROBE_DIR }}"
src: "{{ STORAGE_NFS_CLIENT_SRC }}"
fstype: "{{ STORAGE_NFS_FSTYPE }}"
opts: "{{ STORAGE_NFS_MOUNT_OPTS }},_netdev,noauto"
state: ephemeral
register: svc_storage_nfs_client_probe
failed_when:
- svc_storage_nfs_client_probe is failed
- name: "🔧 Align the NFS client's directory delegations with the ganesha server"
ansible.builtin.include_tasks: directory_delegations.yml
when: lookup('nfs_flavor') == 'ganesha'
- name: "🔍 Probe-write a marker to confirm the mount is RW"
ansible.builtin.copy:
dest: "{{ SVC_STORAGE_NFS_CLIENT_PROBE_DIR }}/.svc-storage-nfs-client-probe.{{ inventory_hostname }}"
content: |
probed-by={{ inventory_hostname }}
probed-at={{ ansible_facts.date_time.iso8601 }}
mode: "0644"
- name: "🔗 Unmount probe mount"
ansible.posix.mount:
path: "{{ SVC_STORAGE_NFS_CLIENT_PROBE_DIR }}"
state: unmounted
- name: "🔍 Detect {{ DIR_VAR_LIB }} pre-mount state"
ansible.builtin.script:
cmd: "{{ role_path }}/files/shell/premount_state.sh"
executable: /bin/bash
environment:
DIR_VAR_LIB: "{{ DIR_VAR_LIB }}"
register: _svc_storage_nfs_client_premount_state
changed_when: false
- name: "✅ Assert no pre-existing local data at {{ DIR_VAR_LIB }} (mount would hide it)"
ansible.builtin.assert:
that:
- >-
'STATE=mounted' in _svc_storage_nfs_client_premount_state.stdout
or 'STATE=empty' in _svc_storage_nfs_client_premount_state.stdout
fail_msg: >-
Pre-existing local data found at {{ DIR_VAR_LIB }}; NFS mount-over
would silently hide it. Migrate or remove the local tree manually,
then re-run the deploy. Probe said:
{{ _svc_storage_nfs_client_premount_state.stdout | trim }}
- name: "✅ Ensure infinito state mount point exists"
ansible.builtin.file:
path: "{{ DIR_VAR_LIB }}"
state: directory
owner: root
group: root
mode: "0755"
- name: "🔍 Detect a pending change of the {{ DIR_VAR_LIB }} mount"
ansible.posix.mount:
path: "{{ DIR_VAR_LIB }}"
src: "{{ STORAGE_NFS_CLIENT_SRC }}"
fstype: "{{ STORAGE_NFS_FSTYPE }}"
opts: "{{ STORAGE_NFS_MOUNT_OPTS }},_netdev"
state: mounted
check_mode: true
register: svc_storage_nfs_client_state_pending
- name: "🔌 Release {{ DIR_BACKUPS }} so the state mount can be remounted"
ansible.posix.mount:
path: "{{ DIR_BACKUPS }}"
state: unmounted
when: svc_storage_nfs_client_state_pending is changed
- name: "🔗 Persistently mount cluster-shared infinito state from NFS"
ansible.posix.mount:
path: "{{ DIR_VAR_LIB }}"
src: "{{ STORAGE_NFS_CLIENT_SRC }}"
fstype: "{{ STORAGE_NFS_FSTYPE }}"
opts: "{{ STORAGE_NFS_MOUNT_OPTS }},_netdev"
state: mounted
- name: "✅ Ensure the node-local backup store exists"
ansible.builtin.file:
path: "{{ SVC_STORAGE_NFS_CLIENT_BACKUPS_DIR }}"
state: directory
mode: "0755"
- name: "🔗 Keep {{ DIR_BACKUPS }} node-local over the shared state"
ansible.posix.mount:
path: "{{ DIR_BACKUPS }}"
src: "{{ SVC_STORAGE_NFS_CLIENT_BACKUPS_DIR }}"
fstype: none
opts: bind,_netdev
state: mounted