Files
computer-playbook/utils/storage/nfs.py
T
kevinveenbirkenbachandClaude Opus 5 f46c4d0bab fix(storage): keep the NFS client off the ports the platform publishes
A swarm job lost every published port of the mail stack without anything
reporting it. dockerd aborted the whole ingress program:

  Failed to add ingress: failed to program ingress ports:
  failed to bind host port 0.0.0.0:995/tcp: address already in use

/proc/net/tcp named the squatter: 10.100.1.1:995 -> 10.100.1.11:2049,
the node's own NFS client. The mount carries no noresvport, so the
kernel drew a source port from the 665-1023 reserved window and landed
on pop3s. Afterwards none of 25/465/993/995/4190/8095 was bound while a
sibling service's port was, and stalwart still reported 1/1 healthy
because its healthcheck runs inside the container. The e2e CLI probe
found it two hours later as ECONNREFUSED on SMTP 25.

993 and 995 are the only published ports repository-wide inside that
window, so this is luck of the draw on any NFS-backed swarm host that
runs the mail role.

insecure on the kernel export is not separate hygiene, it is the other
half: kernel nfsd defaults to secure and answers a mount from an
unprivileged source port with EACCES, so noresvport without it would
break every kernel-flavor mount. It reaches the kernel flavor only,
since templates/exports.j2 is reached from tasks/flavor/kernel/main.yml
alone. The ganesha flavor needs nothing: ganesha.conf.j2 sets no
Privileged_Port and the default already accepts unprivileged clients.

What it costs: the next deploy of every existing NFS client sees a
pending opts change and remounts the shared state. The role already
anticipates that, releasing DIR_BACKUPS first when the check-mode probe
reports a change. Nothing bind-mounts that path into a container, so
the exposure is a host-side job holding it at that instant, which fails
loudly rather than silently.

Not yet proven in effect: a source port is chosen when the RPC
connection is established, so a plain remount can leave a running mount
on its old reserved port until the next real mount. Reading
/proc/net/tcp after the first deploy on an existing client is what
confirms it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-20 19:38:01 +02:00

128 lines
4.2 KiB
Python

from __future__ import annotations
import re
from collections.abc import Mapping
from pathlib import PurePosixPath
from utils import PROJECT_ROOT
from utils.cache.files import read_text
from utils.roles.mapping import ROLE_FILE_META_SERVICES
STATE_SUBDIR = "infinito-state"
NFS_SERVER_SERVICES_YML = str(
PROJECT_ROOT / "roles" / "svc-storage-nfs-server" / ROLE_FILE_META_SERVICES
)
NFS_CLIENT_SERVICES_YML = str(
PROJECT_ROOT / "roles" / "svc-storage-nfs-client" / ROLE_FILE_META_SERVICES
)
def _read_spot_value(services_yml: str, key: str) -> str:
"""Read a scalar entity value via line parse.
Exception: stdlib-only on purpose - these SPOTs feed the .env
generation, which bootstraps fresh hosts before PyYAML exists.
Args:
services_yml: repo-relative services.yml path.
key: two-space-indented scalar key to read.
"""
pattern = re.compile(rf"^ {re.escape(key)}:\s*(\S+)\s*$")
for line in read_text(services_yml).splitlines():
match = pattern.match(line)
if match:
return match.group(1)
raise KeyError(f"{key} not found in {services_yml}")
def get_export_base() -> str:
"""NFS export base from the provider's services.yml SPOT."""
return _read_spot_value(NFS_SERVER_SERVICES_YML, "export_base")
def get_client_version() -> int:
"""NFS mount protocol version from the client's services.yml SPOT."""
return int(_read_spot_value(NFS_CLIENT_SERVICES_YML, "nfs_version"))
def state_path(export_base, subdir):
return str(PurePosixPath(str(export_base)) / str(subdir))
def fstype(version):
return "nfs4" if int(version) >= 4 else "nfs"
def mount_opts(version, runtime):
reliability = (
"soft,timeo=50,retrans=3"
if runtime in ("dev", "act", "github")
else "hard,timeo=600"
)
locking = "" if int(version) >= 4 else ",nolock"
return f"vers={version},rw,noresvport,{reliability}{locking}"
def client_src(server, version, flavor, state_path_value):
use_root = flavor == "kernel" and int(version) >= 4
return f"{server}:{'/' if use_root else state_path_value}"
def _role_forces_other_mode(application_id):
"""True unless the role certainly runs at the cluster's deployment mode.
``compose_mode_force`` overrides the cluster mode for one role only
(roles/sys-svc-compose/defaults/main.yml). A value this function cannot
resolve statically - a Jinja expression - counts as forcing, so an
unknown mode keeps a volume backed up rather than silently dropping it
from both capture paths.
Args:
application_id: role whose ``vars/main.yml`` is read.
"""
from utils import PROJECT_ROOT
from utils.cache.yaml import load_yaml_any
from utils.roles.mapping import ROLE_FILE_VARS_MAIN
path = PROJECT_ROOT / "roles" / str(application_id) / ROLE_FILE_VARS_MAIN
if not path.is_file():
return False
loaded = load_yaml_any(str(path))
if not isinstance(loaded, Mapping):
return False
forced = loaded.get("compose_mode_force")
if forced is None:
return False
forced = str(forced).strip()
if not forced:
return False
if "{{" in forced or "{%" in forced:
return True
return forced.lower() != "swarm"
def swarm_nfs_backed(entry, *, application_id, deployment_mode, storage_backend):
"""True when the swarm NFS rewrite backs this ``meta/volumes.yml`` entry.
Args:
entry: volume entry mapping; ``nfs: false`` opts it out of the rewrite.
application_id: role the volume belongs to; a manager-pinned role or one
forcing a non-swarm mode keeps its volumes node-local.
deployment_mode: cluster-wide ``compose`` or ``swarm``.
storage_backend: the ``storage.backend`` value, e.g. ``local`` or ``nfs``.
"""
if str(deployment_mode).strip().lower() != "swarm":
return False
if str(storage_backend).strip().lower() != "nfs":
return False
if not isinstance(entry, Mapping):
return False
if entry.get("nfs") is False:
return False
from utils.roles.meta_lookup import get_role_placement
if str(get_role_placement(application_id) or "").strip().lower() == "manager":
return False
return not _role_forces_other_mode(application_id)