Main CI run 40976399687, socialhome#0 (swarm, tor, centos): the CLI tests
of svc-net-tor (onion:25 "socks CONNECT failed with reply 0x01") and
web-app-stalwart (probe_mail relay: ConnectionRefusedError on port 25)
failed. dockerd on the manager had aborted the whole ingress program of
the mail stack:
Failed to add ingress: failed to program ingress ports:
failed to bind host port 0.0.0.0:993/tcp: address already in use
/proc/net/tcp named the squatter: 192.168.244.10:993 -> :2049, the
node's own NFS client. The mount carries no noresvport, so the kernel
drew a source port from the 665-1023 reserved window and landed on
imaps. None of 25/465/993/995/4190/8095 got a DNAT rule or a
docker-proxy listener while every other published port did, and
stalwart still reported healthy because its healthcheck runs inside the
container. 993 and 995 are the only published ports inside that window,
so this is luck of the draw on any NFS-backed swarm host that runs the
mail role.
mount_opts now emits noresvport. insecure on the kernel-flavor export is
the other half: kernel nfsd defaults to secure and answers a mount from
an unprivileged source port with EACCES. It reaches the kernel flavor
only; ganesha already accepts unprivileged clients.
What it costs: the next deploy of every existing NFS client sees a
pending opts change and remounts the shared state; a host-side job
holding that path at that instant fails loudly. A plain remount may keep
a running mount on its old reserved port until the next real mount.
Port of f46c4d0bab from feature/store-i18n, applied identically so the
later store-i18n -> main merge stays conflict-free.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Penpot 0 of run 31658241728 died on the restart handler while the node was
healing itself. Its journal, in order:
20:46:47 nfs-ganesha.service: State 'stop-sigterm' timed out. Aborting.
20:46:58 nfs-ganesha.service: State 'stop-watchdog' timed out. Killing.
20:46:58 Main process exited, code=killed, status=9/KILL
20:46:58 Failed to start nfs-ganesha.service
20:47:00 [main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
20:47:01 systemd[1]: Started nfs-ganesha.service - NFS-Ganesha file server.
Ansible failed at 20:46:58, three seconds before the replacement was serving.
The rescue bundle confirms the end state independently: 17 threads, RPC workers
in svc_rqst_epoll_loop's epoll_wait, fridge threads idle -- a healthy server.
9b52a966c built exactly this recovery (ExecStartPost turns a wedge into a unit
failure, Restart=on-failure replaces the process, a wait_for on 2049 absorbs one
cycle). The defect was that the recovery sat downstream of the handler that
died on the event it exists for: `systemd: state=restarted` reports the restart
job's result, and the play never reached the gate.
A shell script now owns the whole statement -- restart or start, then wait for
2049, exit 0 only once the port answers, exit 1 with the unit's status and
journal. systemctl's own result is deliberately not the verdict, because a
recovering unit and a broken one both read 'failed' at the moment it returns;
the difference only exists two seconds later, which is what the wait measures.
The handler is that SPOT's restart entry point and the task after it is the
start one; vars carries unit, port and budget once.
That removes the tolerance instead of relocating it. Gone with it: a
`failed_when: false`, a dead `register`, a handler that set a fact instead of
restarting, three `|| true` against this repo's own no-or-true rule (unnecessary
anyway -- the script sets -u, not -e), and the now-duplicated wait_for gate.
`changed_when` on both call sites reads the script's CHANGED marker, so a start
that had to act reports changed and one that found the unit already active does
not; previously the task claimed `changed_when: false` even when it had started
the service itself.
Measured, and in the image it will run in (infinito-swarm-node, the same one the
rescue bundle came from):
- all four combinations: start+inactive -> CHANGED, start+active -> no change,
restart -> CHANGED either way
- systemctl failing while the port answers -> rc 0; port never opening -> rc 1
with systemctl's own rc and message, status and journal on stderr
- bash 5.2.37, /usr/bin/systemctl, /usr/bin/journalctl present; /dev/tcp works
- the script run inside that image with no systemd at all still judges
correctly, because its verdict is the port
TimeoutStopSec stays at 10. The first attempt at this raised it to 90, which
9b52a966c rules out in writing -- the wedged process blocks SIGTERM, so a longer
budget only burns the recovery cycle the 120s gate has to hold. The merge-fix
search caught that before it was committed.
Not verified: that ansible.builtin.script's temp directory is executable on
every node (it was exercised here through docker run, not through the module),
that 120s covers a real recovery beyond the single three-second data point, and
the three sibling node images.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the raw package modules at 76 call sites with package_install, moves
each role's distribution mapping into its own meta/packages.yml, and turns the
inventory PACKAGES variable from source-keyed literal names into a list of
declared package ids. The lint that forbids the raw modules is enabled here,
because the migration burns it down from 76 offenders to zero.
Removes sys-aur, sys-aur-install and sys-package. The AUR build user, the
passwordless pacman rule and the inventory-driven install are part of the plan
the action plugin executes, so the roles no longer carry them.
Declaring the mappings surfaced breakage the raw modules had hidden: on Debian
the `which` binary comes from debianutils, jrnl is in no Debian or Ubuntu
suite, chromium-browser is absent from Debian stable, JDK 11 is unobtainable on
RedHat, wireguard had no RedHat branch at all, qbittorrent was built from the
AUR although Arch ships it in extra, and desk-docker installed a Docker variant
that conflicts with the one sys-svc-container installs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ganesha package and service maps were indexed by
ansible_facts['os_family'] but keyed with distribution names, so the
Fedora and Ubuntu entries were unreachable and os_family RedHat matched
nothing. Host deploys on Fedora aborted with "object of type 'dict' has
no attribute 'RedHat'".
Key both maps by os_family and cover the whole default distro matrix:
- RedHat: Fedora ships nfs-ganesha in its base repo. Its 9.16 build is
linked against NTIRPC_7.2 symbols the current libntirpc does not
export, so ganesha.nfsd exits 127; pull libntirpc from updates-testing
to restore a runnable pair.
- RedHat on EL: nfs-ganesha is in no base, AppStream, CRB or EPEL repo,
only in the CentOS Storage SIG, so enable that repository first.
- Archlinux: no official package exists at all. Build it from the AUR
and ship the systemd unit, because the AUR package installs binaries
only.
Drop the unreachable distribution keys here and in the client role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Root cause of the kernel-flavor swarm ENOENT: no export carried fsid=0, so the NFSv4 mount of the nested /srv/nfs/infinito-state sub-export had no pseudo-root anchor and the server returned NFS4ERR_NOENT on the PUTROOTFH+LOOKUP walk. crossmnt could not fix it (it only crosses a boundary mid-walk, never sets the starting filehandle). Kernel now exports the state dir itself at fsid=0 and clients mount the bare v4 root ':/' (zero LOOKUPs). ganesha keeps Pseudo=Path and v3 mounts the full path. The self-bind is kept so fsid=0 rides a real mount boundary (stable st_dev) under overlayfs DinD.
The NFS derivations move to per-value lookup plugins (nfs_flavor, nfs_client_src, nfs_fstype, nfs_mount_opts, nfs_state_path) backed by pure helpers in utils/storage/nfs.py with unit tests; group_vars STORAGE_NFS_* become thin lookup wrappers. The dev/act->ganesha flavor rule stays the single SPOT in the role config; nfs_flavor reads the merged value so client and server agree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Kernel nfsd only crosses the self-bind mountpoint into the distinct-fsid state export when the parent export carries NFSEXP_CROSSMOUNT (nfsd_mountpoint()). Without crossmnt the v4 controller/node mount of the state path returns ENOENT even though the export is present. Add crossmnt to the node and controller-traversal parent exports.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The github swarm legs picked the kernel NFS flavor inside the privileged DinD nfs-server, where kernel nfsd is not network-namespaced and wedges exportfs. Move flavor selection out of the role meta default and into the dev inventory: meta/services.yml now defaults to kernel (production); the development inventory overrides it to ganesha for the containerized runtimes (dev/act/github) and keeps kernel for real hosts (RUNTIME=host).
Add meta/variants.yml (kernel + ganesha) and a TODO to test both flavors via files/test.sh, plus server-side kernel-nfsd diagnostics in 12_collect_diagnostics.sh.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The controller mount of the nested state export failed with server 'No such file or directory': over NFSv4 the controller must LOOKUP-traverse the base /srv/nfs export (real export, fsid=1) to reach the nested /srv/nfs/infinito-state (fsid=2), but a controller outside svc-swarm-node is absent from the base client list, so traversal is denied. Re-export the base read-only to SVC_STORAGE_NFS_SERVER_CONTROLLER_CIDRS (empty by default -> no extra exposure; in-node controllers are already listed; the swarm test sets the lab subnet). Worker nodes keep base rw via their explicit-IP lines (exports(5) host-specificity wins).
Add nfs_export_all_rw seboolean for the kernel flavor so RedHat SELinux-enforcing hosts (fedora/centos) serve the export; install python3-libselinux and re-gather the selinux fact first so the boolean is not silently skipped when the binding is absent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
svc-storage-nfs-server gains a backend flavor selected by services.nfs-server.flavor (default "{{ 'ganesha' if RUNTIME in ['dev','act'] else 'kernel' }}"). The ganesha flavor installs nfs-ganesha + FSAL_VFS and renders /etc/ganesha/ganesha.conf (NFSv4, Pseudo == Path so clients mount the real paths unchanged, Root_Squash on the export base and No_Root_Squash on the state subdir), reloaded via systemctl restart. It runs in the dev/act DinD swarm-test, where the host kernel nfsd is not network-namespaced and a privileged node driving exportfs would wedge it into a D-state. The kernel flavor keeps nfs-kernel-server + exportfs -ra for production and every non-dev/act runtime. 01_core dispatches on the flavor, 03_prepare holds the shared export preparation, flavor/{kernel,ganesha}/main.yml hold the per-backend install/render/service steps, and handlers gains the ganesha restart.
docs/agents/action/iteration: role.md renamed to compose.md (Compose Loop); new swarm.md for the swarm deploy loop split out of workflow.md; workflow.md trimmed to GitHub Actions workflow iteration; all cross-references updated; the duplicated README gotcha removed; the NFS gotcha rewritten to the ganesha design.
Also drops a retired-flag history comment in group_vars/all/01_modes.yml and a redundant comment in the init selection script, and allows WebFetch(repology.org) used during ganesha packaging research.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolves CI swarm rc=69 ("msmtp: needs a password"): controller-side
Ansible lookups (lookup('email').password etc.) couldn't see the
tokens.yml that Mailu writes target-side, because the controller has
its own filesystem in swarm mode.
Mount NFS-shared /var/lib/infinito on every swarm host AND on the
controller during deploy (in-play delegate_to: localhost task at end
of svc-storage-nfs-server role). State subdir exported separately
with no_root_squash + wildcard CIDRs so root-owned writes succeed
and the controller can mount from outside the svc-swarm IP set.
Single-writer guarantee on FILE_TOKENS via delegate_to: STACK_HOST +
run_once on sys-token-store/{01_core,write}.yml. Backup roles
(svc-bkp-container-2-local, sys-ctl-cln-bkps, sys-ctl-cln-faild-bkps,
user-backup/03_permissions_folders, three direct callers of
04_seed-database-to-backup.yml) gated on IS_STACK_HOST to prevent
N-host write races on the shared NFS backup tree.
Fail-fast assertion in svc-storage-nfs-client + 05_controller_mount
when pre-existing local /var/lib/infinito would be hidden by mount-
over. Operator must migrate or wipe before re-deploying.
SPOTs: SWARM_LAB_NETWORK in workflow env block, storage.nfs.state_subdir
in group_vars/all/15_storage.yml, DIR_VAR_LIB unchanged. UPPER_CASE
constants in NFS roles. Cleanup script umounts controller view before
container teardown to prevent stale-mount leak between matrix jobs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the svc-docker-swarm, svc-docker-swarm-manager, svc-storage-nfs-server
and svc-storage-nfs-client roles, the compose_volumes filter NFS support,
the test-deploy-swarm-nfs CI workflow with its swarm-nfs-pilot script set,
and the deploy_mode-aware compose/swarm wiring across mediawiki, mariadb
and sys-svc-* helpers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5.5