Main CI run 40976399687, socialhome#0 (swarm, tor, centos): the CLI tests
of svc-net-tor (onion:25 "socks CONNECT failed with reply 0x01") and
web-app-stalwart (probe_mail relay: ConnectionRefusedError on port 25)
failed. dockerd on the manager had aborted the whole ingress program of
the mail stack:
Failed to add ingress: failed to program ingress ports:
failed to bind host port 0.0.0.0:993/tcp: address already in use
/proc/net/tcp named the squatter: 192.168.244.10:993 -> :2049, the
node's own NFS client. The mount carries no noresvport, so the kernel
drew a source port from the 665-1023 reserved window and landed on
imaps. None of 25/465/993/995/4190/8095 got a DNAT rule or a
docker-proxy listener while every other published port did, and
stalwart still reported healthy because its healthcheck runs inside the
container. 993 and 995 are the only published ports inside that window,
so this is luck of the draw on any NFS-backed swarm host that runs the
mail role.
mount_opts now emits noresvport. insecure on the kernel-flavor export is
the other half: kernel nfsd defaults to secure and answers a mount from
an unprivileged source port with EACCES. It reaches the kernel flavor
only; ganesha already accepts unprivileged clients.
What it costs: the next deploy of every existing NFS client sees a
pending opts change and remounts the shared state; a host-side job
holding that path at that instant fails loudly. A plain remount may keep
a running mount on its old reserved port until the next real mount.
Port of f46c4d0bab from feature/store-i18n, applied identically so the
later store-i18n -> main merge stays conflict-free.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the swarm-NFS migration two chains copied the same bytes: svc-bkp-nfs-2-local
reads the export from the NFS server's own filesystem, while baudolo on the manager
copied the same files again THROUGH the NFS mount. That was a migration leftover -
svc-bkp-nfs-2-local arrived in adb8bcf69 (2026-07-10) and nobody taught the older role
to step back - and the role README already forbade it as a rule nobody enforced. In the
DR drill the duplicate costs three times over: the pulled tree, the device copy and the
restore root.
swarm_nfs_backed() in utils/storage/nfs.py is now the single answer to "does the swarm
NFS rewrite back this volume". plugins/filter/compose_volumes.py (which performs the
rewrite) and this role's backup_volume lookup (which feeds baudolo's
--volumes-no-backup-required) both call it, so the two cannot drift. On the current tree
that excludes 83 of 158 type: volume entries.
Four conditions keep the exclusion from ever dropping a volume nobody else captures:
* a manager-pinned role's volumes are not on the export (web-app-seaweedfs holds
Nextcloud's object store, deliberately node-local per its meta/services.yml)
* a role forcing another mode via compose_mode_force is not on the export either.
That override is per role (roles/sys-svc-compose/defaults/main.yml:3), so it is
resolved per role inside the predicate from roles/<id>/vars/main.yml, not read once
from ambient scope. web-app-bigbluebutton pins compose statically;
web-app-matrix computes it, and a value the predicate cannot resolve counts as
forcing - unknown mode keeps the volume backed up
* an entry opting out with nfs: false stays node-local
* the exclusion applies at all only when the svc-bkp-nfs-2-local inventory group is
populated. That role enters a play by group membership alone
(tasks/groups/svc-bkp-roles.yml:19-22) and svc-storage-nfs-server ships a variant
with nfs_backup: {enabled: false}, so without the gate an export whose repository
nobody deployed would lose 83 volumes from both capture paths
Every condition fails toward the old double capture: wasted space, never lost data.
Compose mode is untouched - the exclusion cannot fire without swarm and an nfs backend.
DEPLOYMENT_MODE and storage are templated through the lookup's templar, because
group_vars/all/18_swarm.yml:2 defines the mode as a Jinja expression; comparing the raw
source string would silently disable the whole exclusion.
Twelve new tests pin the contract, including the two failure modes that would be
invisible in production: an untemplated mode, and an exclusion without an export
repository. Both were verified by mutation - removing either guard makes its test fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nfs(5) lists both 'nolock' and 'local_lock=' under "Options for NFS versions 2 and 3 only". NFSv4 carries locking in the protocol and has no NLM sideband to switch off, so the kernel accepts the token and reports local_lock=none. mount_opts had the condition inverted: the branch that can work sat on v3, the inert one on v4 - and svc-storage-nfs-server pins nfs_version: 4, so the effective locking configuration was always the dead branch.
The artifact proves it landed nowhere: the mount module ran with local_lock=flock in opts on every node, every other token from that same string is present in /proc/mounts (soft, timeo=50, retrans=3, _netdev), and only local_lock reads 'none'.
The false premise had spread. docker-entrypoint-suitecrm.sh justified its mkdir mutex with 'the swarm NFS mount forces local_lock=flock, so flock(2) never crosses nodes'; that comment is removed. The mkdir mutex stays correct for a different reason: against this ganesha export a non-blocking flock works while a blocking one returns EIO.
Nothing about the effective mount changes - the option was already being discarded. On an existing host the first run rewrites fstab and remounts /var/lib/infinito once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker info reports DockerRootDir as the DAEMON sees it, and
host_storage_constrained fed that string straight into shutil.disk_usage,
which statvfs()es it in the CALLING process's mount namespace. The two
coincide only when the caller is the daemon's host. In the nested act
container of the workspace jobs they do not, so os.statvfs('/var/lib/docker')
raised FileNotFoundError, matrix.py died, and because 00_one.sh invokes it
bare under set -euo pipefail the whole swarm drill went down with it. All
five workspace jobs of run 30560683608 failed this way, byte-identically,
across five base images and five nested distros.
docker_root_free_bytes() now takes the vantage point as a required keyword:
it measures the daemon's data root when that path resolves here, and the
caller's stated local path when it does not. matrix.py passes "/", which is
the same vantage it already trusts for its ENOSPC abort 270 lines earlier;
overlayfs passes statfs through to the upper filesystem, so a container of a
daemon measures that daemon's data-root filesystem through its own rootfs.
Nothing is caught and nothing is guessed. An unreachable daemon still raises
RuntimeError out of docker_data_root(), which is the guard this module exists
for, and local_vantage carries no default because only the caller knows where
it is running.
The compose-side twin loses its || true and its hardcoded /var/lib/docker
substitution, so an unresolvable root no longer measures a guessed filesystem.
Its compose.exec locality was already correct and is unchanged.
Verified against the pre-fix module under the CI condition: with the data root
stubbed to a path absent from this namespace, the old code raises
FileNotFoundError and the new code measures the local vantage; with the daemon
stubbed unreachable, the new code still raises RuntimeError.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The flag was decided by a fixed free-space threshold (100 GiB by default), which fired on roomy hosts and stayed silent on a 145 GiB CI runner that a single deploy round could fill. It now compares free space on Docker's data root against what the deploy declares it needs: the summed min_storage of the selected applications and their transitive dependencies.
The need is variant-aware. utils/storage/constrained.py overlays each app's meta/variants.yml entry before walking the service registry, so a round that trims services down is measured at its own size instead of the base configuration's.
Both entry paths pass their per-round variant map: cli/administration/deploy/development/init.py decides inside the round loop (compose and host share this via ci/one.sh -> dedicated.sh), and utils/tests/swarm/matrix.py feeds the decision into the swarm host_vars payload. --threshold-gib is gone; --force-storage-constrained still overrides.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils.storage.nfs read the nfs-server/client services.yml SPOTs via a CWD-relative path. The recover drill imports these inside the deploy container where the CWD is not the repo root, so get_export_base() raised FileNotFoundError. Anchor both paths to utils.PROJECT_ROOT (matching utils.paths), making the read CWD-independent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The swarm labs run NFS server and clients on one shared kernel; with
hard mounts a writeback stall wedges containers in D-state and takes
the whole DinD node with it (the bbb/matomo and nextcloud wedges).
dev/act already mount soft for exactly this reason; github gets the
same lab profile. Production (host) keeps hard mounts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The nfs SPOT loaders pulled PyYAML into the .env generation, which bootstraps fresh hosts BEFORE python dependencies exist: the import crashed cli.meta.env, load.sh produced an empty environment and make install-system-python died on the required-var guard (environment/manjaro and pkgmgr/centos CI jobs). The loaders now line-parse the services.yml SPOTs via the stdlib-backed cached reader and the mapping path constant. The client's protocol version field becomes nfs_version, since services.yml reserves version for image tags (schema lint), and the image discovery casts versions defensively.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
storage.nfs shrinks to the deployment-specific server address: export_base moves into svc-storage-nfs-server's services.yml (the server dictates where exports root) and the mount protocol version into svc-storage-nfs-client's services.yml (the client owns how it mounts). Both are read through utils.storage.nfs loaders by the nfs lookups, the env handler and 00_topology, eliminating the unrendered-variables class for these values; the group_vars assert shrinks accordingly and the swarm test extras stop restating role defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A wedged DinD dockerd hung a single diagnostics docker exec until the 360-minute job timeout cancelled the job together with its artifact uploads (job 86013153099 sat 5h in service logs). Route every diagnostics exec through a timeout wrapper, tail-limit the log dumps, bound showmount, cap the collect step at 30 minutes, and dump the pending services' log tails once on the final converge failure. The NFS state path moves into a single source of truth: STATE_SUBDIR in utils/storage/nfs.py feeds the nfs_state_subdir lookup (deploy side) and the derived INFINITO_SWARM_NFS_STATE_PATH env var (harness side); the diagnostics script requires both path vars instead of defaulting, and the swarm inventory drops its redundant export_base restatement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Checkmk's OMD site crashes at boot with fcntl.flock raising OSError
[Errno 9] Bad file descriptor: NFSv4 ignores the nolock mount option and
routes flock() to the server. Force flock() client-local on v4 so
single-writer apps (checkmk, openldap) boot; POSIX locks stay
server-side. NFSv3 keeps nolock (statd-free, flock already local).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of the kernel-flavor swarm ENOENT: no export carried fsid=0, so the NFSv4 mount of the nested /srv/nfs/infinito-state sub-export had no pseudo-root anchor and the server returned NFS4ERR_NOENT on the PUTROOTFH+LOOKUP walk. crossmnt could not fix it (it only crosses a boundary mid-walk, never sets the starting filehandle). Kernel now exports the state dir itself at fsid=0 and clients mount the bare v4 root ':/' (zero LOOKUPs). ganesha keeps Pseudo=Path and v3 mounts the full path. The self-bind is kept so fsid=0 rides a real mount boundary (stable st_dev) under overlayfs DinD.
The NFS derivations move to per-value lookup plugins (nfs_flavor, nfs_client_src, nfs_fstype, nfs_mount_opts, nfs_state_path) backed by pure helpers in utils/storage/nfs.py with unit tests; group_vars STORAGE_NFS_* become thin lookup wrappers. The dev/act->ganesha flavor rule stays the single SPOT in the role config; nfs_flavor reads the merged value so client and server agree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5.5