Since the inner mdad dockerd actually starts (storage-driver fix), its
default docker0 claims 172.17.0.1 inside the container netns and
shadows exactly the resolver address the swarm inventory hands to every
container via networks.internet.dns, so the bootstrap's ansible-galaxy
died on 'Could not resolve host: github.com' one second after boot.
Declare the nested-docker bridge ranges as a network SPOT
(NETWORK_NESTED_DOCKER_BIP / _ADDRESS_POOLS, deliberately outside
172.16.0.0/12 and away from the 192.168.244.0/24 lab net) and render
the runner's daemon.json from it at staging time instead of shipping a
static file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PHP's libcurl and stream layer ignore CURL_CA_BUNDLE/SSL_CERT_FILE, so
swarm containers - which get env-and-mount CA injection only, the
entrypoint wrapper being stack-deploy-incompatible - fail TLS against
self-signed in-cluster services from PHP code (verified live: curl CLI
reaches keycloak, PHP curl_exec returns 'unable to get local issuer
certificate'; nextcloud's /apps/oidc_login/oidc answered 500 on every
swarm variant). Ship a ca-trust-php.ini pointing openssl.cafile and
curl.cainfo at the mounted CA and bind it into the official-image PHP
conf.d path from the compose CA override in both deploy modes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the constructor's pre_deploy_snapshot.sh loop with the loader
itself: volume/nfs/secrets instant-start their oneshot in the preload
pass under MODE_BACKUP (force_flush_instant + state started), so the
live pre-update state is snapshotted before any app mutation; nfs keeps
run_after svc-storage-nfs-server, volume pulls sys-svc-container for a
live dockerd and pre-creates /opt/compose. remote pulls at end of play
via the terminator (force_flush_final) once every provider snapshot is
done. Dropping suppress_flush arms the result assert: a failed
pre-state backup aborts the deploy. Role READMEs carry the new trigger
schema as mermaid diagrams.
Known residual: on a host whose pull identity is not yet provisioned
the terminator-forced remote pull fails and turns the deploy red; the
swarm lab installs the trust only in the DR drill after round 1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The provider ssh-wrapper enumerated only the docker and nfs repo types,
so the first secrets generation made every remote pull fail: the puller
discovers types dynamically, hit backup-secrets-to-local, and the ls
fell through to 'This command is not supported' (exit 1), killing the
DR drill at step 4/9. BACKUP_REPOSITORIES in group_vars now drives both
the wrapper whitelist and its type discovery, so only whitelisted repos
are returned and only whitelisted repos are pullable. A lint pins the
SPOT against the svc-bkp-* repo names in both directions, so a fourth
repo type cannot bypass the wrapper again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New svc-bkp-secrets-2-local role: a differential rsync --link-dest snapshot
of the host-generated secret material that lives in no docker volume, NFS
export or operator inventory - infinito secrets + tokens, the self-signed
CA, the Let's Encrypt account/certs, the ACME DNS credentials - plus the
node identity (ssh host keys + machine-id). files/recover.py restores each
present subtree to its fixed system path and gates the node-identity
restore behind --restore-node-identity. Wires the systemd unit + nightly
schedule (06_daemons, 07_schedule). Unit tests cover the domain/target map
and the main() subtree-selection + node-identity gating.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two wrapper behaviours move onto single sources of truth:
Exec timeout: SYS_TIMEOUT_CONTAINER_EXEC (07_schedule.yml, default 7d so a
hard kill never beats a production migration; the dev/CI inventory
tightens it to 60min) reaches every task host via the play environment as
CONTAINER_EXEC_TIMEOUT. The wrapper prefixes only the exec subcommand with
coreutils timeout --kill-after=15. Ansible task timeouts are SIGALRM-based
and slept through the last wedged docker socket for 4.5 hours; the OS
level kill still fires there and returns 124, which until/failed_when and
the rescue diagnostics react to. Unset or 0 keeps todays behaviour.
CA container paths: the bind-mount targets for the injected root CA and
wrapper script lived as string literals in container.py, compose_ca.py
and the filter plugin. They now live once in the CA_TRUST group_vars SPOT
(inject_cert_container, inject_wrapper_container) and reach container.py
as env vars (play environment plus CA_TRUST_SYSTEMD_ENV) and compose_ca.py
as CLI arguments built by the compose_ca_inject_cmd lookup. A caller that
provides the host CA env but not the container paths is half-migrated and
fails loudly instead of silently dropping CA injection; a host without any
CA env keeps the soft fallback. All tests read the paths from the SPOT,
and a regression test asserts the runtime scripts carry no literal copy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every Ansible rescue: block used to carry ad-hoc per-role failure dumps that
drifted apart and captured different subsets. Replace them all with one
diagnostics script (scripts/system/diagnostics/rescue.sh) that snapshots
containers (full logs plus inspect per container), swarm services (ps plus
logs per service), journal, host resources, runtime info and stats into a
per-run folder and always exits 1 so the deploy still fails. A new lint
enforces the shape: every rescue ends with the script, at most one
nocheck-justified extra task before it (postgres pg_hba dump, xwiki
extension-repo probe), and a nocheck rescue-recovery exemption for the one
block that must genuinely recover: keycloak object updates, where a
cross-node reschedule needs a swarm-wide node re-resolve plus a delegate_to
change that only a rescue can express; its recovery is kept and now also
captures diagnostics before the terminal failure.
The snapshot folder path is a group_vars SPOT: DIR_RESCUE_DIAGNOSTICS and
DIR_VAR_LIB live in group_vars/all/05_paths.yml as plain strings; new env
handlers derive INFINITO_RESCUE_DIAGNOSTICS_DIR and INFINITO_DIR_VAR_LIB
from it for the generated .env, replacing the default.env entries, and the
play-level environment forwards the value to every task host so rescue.sh
needs no second default. CI uploads the folder like the inventory and
playwright artifacts: compose copies it out of the deploy container in
dedicated.sh, swarm captures per node (manager and both workers) into
node-named subfolders.
Deleting the ad-hoc dumps orphaned helper files (postgres inspect_container,
terminator docker_diagnostics, peertube diagnostics, matomo bootstrap debug,
xwiki and openresty rescue includes) and 14 role vars; all removed. Generic
pieces of those dumps (runtime version and info, stats, dmesg oom evidence,
swapon) moved into the SPOT script instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the role, its test tree, the primary entity key
(volume-2-local), the derived vars (BKP_VOLUME_2_LOCAL_*,
SYS_SERVICE_BACKUP_VOLUME_2_LOCAL), the run-once fact, and every
group_names consumer flag, include path, doc, and lint reference. The
container_backup service key and the on-disk backup-docker-to-local
tree keep their names: both are contracts with existing consumer
entries, snapshot trees, and the baudolo pull tooling.
Operators must rename the svc-bkp-container-2-local inventory group and
disable the previous versioned units on already-provisioned hosts; the
pre-deploy snapshot glob derives its base from the new role name and
will not find them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a systemd-driven rsync --link-dest backup of the NFS export into the
baudolo-compatible local snapshot tree, plus a pre-deploy snapshot hook in
the constructor. A missing export fails the nightly unit loudly, a failed
rsync removes its incomplete generation so it cannot poison the next
--link-dest, and rsync exit 24 from the live export is tolerated.
Both pre-deploy snapshot scripts (container and nfs) now resolve the
previous deploy's unit through a version glob, so the snapshot still runs
after a project version bump. The user-backup ssh wrapper whitelists the
new backup-nfs-to-local type for downstream pulls. The nfs-server role
declares the backup as a dynamic consumer with true/false pinning
variants; svc-bkp-container-2-local drops its variants.yml because
non-subject closure members resolve foreign round indices against it and
leak its companion pins into other rounds.
Unit tests execute the real script against tempdirs and prove hardlink
deduplication by inode, history isolation, delete semantics, loud failure
on a missing source, and cleanup of incomplete generations.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
storage.nfs shrinks to the deployment-specific server address: export_base moves into svc-storage-nfs-server's services.yml (the server dictates where exports root) and the mount protocol version into svc-storage-nfs-client's services.yml (the client owns how it mounts). Both are read through utils.storage.nfs loaders by the nfs lookups, the env handler and 00_topology, eliminating the unrendered-variables class for these values; the group_vars assert shrinks accordingly and the swarm test extras stop restating role defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lookup plugins receive ansible variables unrendered, so routing the state subdir through a group_vars jinja value handed state_path the literal '{{ lookup(...) }}' string and the controller mount asked the server for /srv/nfs/{{ lookup('nfs_state_subdir') }} (Connection refused, nextcloud swarm roundtrip). Both nfs lookups now import STATE_SUBDIR directly; the intermediate lookup, the group_vars indirection and the unit-test fixture var that masked the gap are gone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A wedged DinD dockerd hung a single diagnostics docker exec until the 360-minute job timeout cancelled the job together with its artifact uploads (job 86013153099 sat 5h in service logs). Route every diagnostics exec through a timeout wrapper, tail-limit the log dumps, bound showmount, cap the collect step at 30 minutes, and dump the pending services' log tails once on the final converge failure. The NFS state path moves into a single source of truth: STATE_SUBDIR in utils/storage/nfs.py feeds the nfs_state_subdir lookup (deploy side) and the derived INFINITO_SWARM_NFS_STATE_PATH env var (harness side); the diagnostics script requires both path vars instead of defaulting, and the swarm inventory drops its redundant export_base restatement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the global TIMEOUT_FACTOR and a shared timeouts.js helper: resolveTimeout(base) scales by TIMEOUT_FACTOR and, over a .onion target, an extra multiplier, so Tor's per-request latency no longer trips hard-coded timeouts. Migrate every raw timeout in the specs (timeout:/timeout=/setTimeout/waitForTimeout) to resolveTimeout, stage the helper in all three staging paths, and add test_no_raw_timeouts to forbid raw numeric timeouts. Clearnet behaviour is unchanged (factor 1: resolveTimeout(N) === N).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The healthcheck fetched GET / which is auth-gated: Laravel answers with an absolute redirect to the public domain (URL::forceRootUrl), GNU wget follows it and fails DNS inside the swarm lab, so the task never turned healthy and converge expired. Probe the unauthenticated /health endpoint instead. Also replace four default-placement justifications with the audit-verified reasons (bluesky sqlite-WAL over NFS, mailu mail-edge IP binding plus NFS-hostile postfix/dovecot state, opencloud JetStream/bleve/boltdb engine state, seaweedfs leveldb/raft/needle volumes) and drop dead general vars whose only consumers were commented-out lines.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
make autoformat output across role meta files, a task file and 19_tor.yml (single->double quotes on mode values, Jinja line reflow). No semantic changes. Kept separate from the onion CSP / oauth2-proxy fixes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The egress-router trans (9040) and dns (9053) ports had no PORT_BANDS entry and failed the port-band-membership lint. Add the local.trans and local.dns bands next to socks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove the inline/trailing comments added across the Tor egress-router, onion-domain, mailu-notls and vault round-trip changes, per the repo no-comments convention. Comment-only: no code behaviour changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route any outbound TCP to a *.onion address through Tor, protocol-agnostically (HTTP, SMTP, IMAP, ...), while clearnet traffic resolves normally and goes direct. Implemented on the target host and gated on TOR_EGRESS_ROUTER_ENABLED: torrc gains AutomapHostsOnResolve + AutomapHostsSuffixes .onion + VirtualAddrNetworkIPv4 + TransPort/DNSPort; a dnsmasq drop-in split-resolves *.onion via the DNSPort (clearnet upstreams embedded inline, since an AppArmor-confined dnsmasq may only read /etc/dnsmasq.d/**) and the host resolver points at it; an iptables nat OUTPUT rule redirects the automapped virtual range to TransPort. Verified in-container: http://<app>.<node-onion>/ returns 200 with no SOCKS wrapper, clearnet DNS + HTTPS stay direct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New env handler reads svc-net-tor services.tor.ports.local.socks (via ROLE_FILE_META_SERVICES) and emits INFINITO_TOR_SOCKS_PORT; registered in the handler registry + passthrough static keys + default.env, and forwarded to the container by compose.yml — so shell consumers (the Playwright rerunner) get the port without a literal. 08_networks.yml adds the local.socks PORT_BANDS band (9050).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per-app PLAYWRIGHT_PROXY: an app whose canonical URL is a .onion (services.tor primary/exclusive) has its browser routed through the node Tor SOCKS proxy (socks5://host.docker.internal:9050); clearnet apps stay direct. A SOCKS readiness gate probes the hidden service and skips (not fails) the app's run when Tor cannot reach it. TEST_E2E_PLAYWRIGHT_PROXY is removed since the per-app logic supersedes the global knob. Non-overridable tuning constants moved defaults->vars (UPPERCASE); only genuinely overridable knobs (only_roles/skip_roles/keep/VARIANT_INDEX) remain in defaults.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a `tor_ports` lookup that collects every single-int `ports.public` entry
from the deployed roles' meta/services.yml and returns HiddenServicePort
mappings (`127.0.0.1:<port>`). UDP-only categories (relay, media) are excluded
since Tor forwards TCP streams only. Wire it into TOR_ONION_EXTRA_PORTS
(19_tor.yml) alongside the SSH port so mail (25/143/465/587/993), gitea/gitlab
SSH, ldaps, matrix federation and STUN/TURN control ports are reachable at
<node>.onion. Register Mailu's mail endpoints under ports.public and add the
matching PORT_BANDS entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- svc-net-tor: expose SSH (22) + Mailu SMTP/submission/IMAP (25/465/587/143/993)
as HiddenServicePorts on the node onion (TOR_ONION_SSH_ENABLED, 19_tor.yml).
- email lookup: relay plaintext (tls off, port 25) for .onion mail domains —
no public TLS cert for a .onion; onion transport is already authenticated.
- CA root CN: drop the " Root CA" suffix when a 62-char v3 onion + suffix would
overflow the X.509 commonName 64-char cap (02_tls.yml).
- Playwright onion compat: HSTS is HTTPS-only (RFC 6797), absent on the http
onion vhost. New shared helper expectHstsWhenTls() gates the assertion on the
reached scheme; wired into web-svc-cdn spec.
- rerun-spec.sh: route the inner-loop browser through the Tor SOCKS proxy when
the staged env targets a .onion, mirroring the deploy-time runner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Turn a NAT/edge node into a full Tor v3 onion node: DOMAIN_PRIMARY
becomes the node's .onion, so the whole stack (web, SSO, LDAP, CA)
resolves onion domains and is reachable over Tor without a public IP,
port forwarding, VPS or DynDNS.
- utils/tor_onion.py: offline v3 key mint (address + HiddenServiceDir
key files), verified against a known key->address test vector.
- cli .../inventory/onion: `init-env` pre-mints the node onion, writes
INFINITO_DOMAIN(+_RE) into .env and authoritative key files into
.onion-identity/hs (idempotent). The onion is a deploy-time input;
no in-deploy random minting (that would not match the pre-set domain).
- roles/svc-net-tor: Tor sidecar (Debian-built, network_mode: host),
installs the minted keys, torrc with one node HiddenService + a SOCKS
proxy restricted to loopback/DiD ranges. files/test.sh probes every
served onion domain over the Tor SOCKS.
- tls: .onion -> TLS off (onion v3 crypto is the transport auth; LE
cannot sign .onion) + app-level flavor override.
- harness/DNS: coredns resolves the onion zone via INFINITO_DOMAIN;
skip the clearnet DNS sanity check for onion; raise nginx
server_names_hash_bucket_size to 128 for long onion server_names.
- test-e2e-playwright: gated browser SOCKS proxy (PLAYWRIGHT_PROXY) +
navigation/action timeouts so specs reach .onion over Tor and fail
fast; the onion node points it at the Tor SOCKS.
- docs/requirements/031: full design.
Verified: dashboard + SSO (Keycloak) + 9 apps reachable over Tor
(HTTP 200 through the node hidden service). make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ca_cert_host derives the host cert path from SOFTWARE_DOMAIN; ca_cert_container carries the fixed in-container bind-mount target. CA_ROOT.cert_host and the test fixtures consume the filter; the standalone runtime scripts (compose_ca.py, container.py) cannot import it and are pinned to the SPOT by the unit test instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a 'timeout' lookup plugin that returns round(base * TIMEOUT_FACTOR)
(group_vars default 1, so value-identical to the old literals). One knob
in group_vars stretches every wait in the stack for a slow uplink:
TIMEOUT_FACTOR: 2 doubles them all.
- plugins/lookup/timeout.py + unit tests (identity, factor scaling,
string coercion, error paths).
- group_vars/all/17_resource.yml: TIMEOUT_FACTOR: 1.
- Convert all 34 literal 'timeout:' task keywords / module args across 25
task+handler files to "{{ lookup('timeout', <base>) | int }}".
- tests/lint/repository/yaml/test_timeout_uses_lookup.py enforces the
lookup in every task/handler timeout (opt-out: # nocheck: timeout-lookup).
- swarm.yml pre-deploy build handler: add the missing timeout (same
network-build hang vector as compose.yml's Build compose).
- Drop 13 pre-existing narration comments in the touched files (comment
lint enforces whole-file on any edited file).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce plugins/lookup/path_absolute.py as the single source for the
repo-root anchor: lookup('path_absolute', 'roles/...') returns an absolute
path under playbook_dir, joining every term. Replace all hand-built
playbook_dir path constructions across .yml/.yaml and .j2 templates with it:
[playbook_dir, '<rel>'] | path_join -> lookup('path_absolute', '<rel>')
playbook_dir ~ '/<rel>' -> lookup('path_absolute', '<rel>')
{{ playbook_dir }}/<rel> -> lookup('path_absolute', '<rel>')
Unrelated repo_path symbols in cli/meta/roles/applications/resolution and the
single bare discover_playwright_roles(playbook_dir) call are left untouched.
Add tests/lint/ansible/test_playbook_dir_via_lookup.py: fails the build if any
of the three construction forms reappears in .yml/.yaml/.j2, pointing callers
at the lookup. Unit test covers the plugin.
Touching these files pulled their pre-existing narration comments into
test_comments_valid scope; clean them (delete restatement/banners, keep real
trip-wires as Workaround/Upstream/Invariant markers). Drop caution, important
and pitfall from the comment-lint marker vocabulary and remove the comments
that used them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements requirement 029: web-app-checkmk deploying checkmk/check-mk-raw
2.4.0p32 (Raw/Community) behind sys-svc-proxy. No native OIDC, so SSO uses the
platform oauth2-proxy gate + Checkmk header-auth (auth_by_http_header =
X-Remote-User, injected by the role proxy.conf.j2 from $authrequest_username =
Keycloak preferred_username), plus a native LDAP connection for user records.
No external DB. Agent receiver (8000) host-port mapped via a new
PORT_BANDS.local.agent band.
Auth/LDAP/proxy mechanisms source-verified against Checkmk v2.4.0p32
(cmk/gui/auth.py, userdb/ldap_connector.py, userdb/_connections.py) and the
infinito sso_proxy nginx chain: fixed the LDAP file path (user_connections.mk)
and directory_type ("openldap"). LDAP sync timing and classic-GUI Playwright
selectors remain live-deploy-validation items (env-blocked).
Validation: make autoformat clean; make test green (lint, test-lint, test-unit,
test-integration 190, test-external).
Refs: docs/requirements/029-web-app-checkmk.md
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consumer lookup plugins now reach the merged config through the Ansible
lookup loader instead of importing get_merged_applications /
get_merged_domains from utils.cache. The merge stays one cached
implementation surfaced via the applications/config/domains/users lookups;
consumers call lookup_loader.get(...).run() (the compose_networks idiom).
- get_merged_applications walks the whole meta/ tree recursively
(meta/**/*.yml to applications.<app>.<path>, addons and any depth) and
folds the raw meta/volumes.yml subtree onto applications.<app>.volumes
after the templar render so the Jinja source: strings stay unrendered.
- 31 consumer lookups rerouted; providers (applications/config/
applications_current_play/users/domains) and the raw-volume accessors
(volume/nginx) keep the cache import behind # nocheck: lookup-cache-import.
- New lint test_no_cache_imports_in_lookups enforces the contract.
- nfs_flavor is now the flavor SPOT (svc-storage-nfs-server + nfs_client_src
resolve via lookup('nfs_flavor')); removed the unused STORAGE_NFS_FLAVOR
group_var. Extracted extract_default_credentials to
utils/cache/schema_credentials.py to keep applications.py under the cap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of the kernel-flavor swarm ENOENT: no export carried fsid=0, so the NFSv4 mount of the nested /srv/nfs/infinito-state sub-export had no pseudo-root anchor and the server returned NFS4ERR_NOENT on the PUTROOTFH+LOOKUP walk. crossmnt could not fix it (it only crosses a boundary mid-walk, never sets the starting filehandle). Kernel now exports the state dir itself at fsid=0 and clients mount the bare v4 root ':/' (zero LOOKUPs). ganesha keeps Pseudo=Path and v3 mounts the full path. The self-bind is kept so fsid=0 rides a real mount boundary (stable st_dev) under overlayfs DinD.
The NFS derivations move to per-value lookup plugins (nfs_flavor, nfs_client_src, nfs_fstype, nfs_mount_opts, nfs_state_path) backed by pure helpers in utils/storage/nfs.py with unit tests; group_vars STORAGE_NFS_* become thin lookup wrappers. The dev/act->ganesha flavor rule stays the single SPOT in the role config; nfs_flavor reads the merged value so client and server agree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the inline "nfs4 if version>=4 else nfs" fstype expression (repeated across svc-storage-nfs-client and svc-storage-nfs-server) with a single STORAGE_NFS_FSTYPE in group_vars/all/15_storage.yml, next to the other STORAGE_NFS_* values it derives from.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dev/act ganesha server is NFSv4-only, but the swarm-test runtime extras forced storage.nfs.version: 3, so every mount failed 'Protocol not supported'. Set version 4 (write_extras + dev swarm inventory). The client probe mounted the Root_Squash export_base and a root probe-write failed; point it at the No_Root_Squash state path. Both client mounts hardcoded 'hard', so a torn-down test server wedged nodes into D-state; gate them to soft in dev/act (reusing the controller mount's gate). Hoist the shared mount-opts string into the STORAGE_NFS_MOUNT_OPTS SPOT (group_vars/all/15_storage.yml), used by all three mounts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
svc-storage-nfs-server gains a backend flavor selected by services.nfs-server.flavor (default "{{ 'ganesha' if RUNTIME in ['dev','act'] else 'kernel' }}"). The ganesha flavor installs nfs-ganesha + FSAL_VFS and renders /etc/ganesha/ganesha.conf (NFSv4, Pseudo == Path so clients mount the real paths unchanged, Root_Squash on the export base and No_Root_Squash on the state subdir), reloaded via systemctl restart. It runs in the dev/act DinD swarm-test, where the host kernel nfsd is not network-namespaced and a privileged node driving exportfs would wedge it into a D-state. The kernel flavor keeps nfs-kernel-server + exportfs -ra for production and every non-dev/act runtime. 01_core dispatches on the flavor, 03_prepare holds the shared export preparation, flavor/{kernel,ganesha}/main.yml hold the per-backend install/render/service steps, and handlers gains the ganesha restart.
docs/agents/action/iteration: role.md renamed to compose.md (Compose Loop); new swarm.md for the swarm deploy loop split out of workflow.md; workflow.md trimmed to GitHub Actions workflow iteration; all cross-references updated; the duplicated README gotcha removed; the NFS gotcha rewritten to the ganesha design.
Also drops a retired-flag history comment in group_vars/all/01_modes.yml and a redundant comment in the init selection script, and allows WebFetch(repology.org) used during ganesha packaging research.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The container_ports migration left make test red; reconcile it:
- Delete 10 port vars orphaned when their compose mappings moved to the
container_ports lookup (OPENLDAP_DOCKER_PORT_OPEN, RABBITMQ_INTERNAL_PORT,
REGISTRY_CACHE_PORT, ERPNEXT_FRONTEND_PORT, FLOWISE_PORT_PUBLIC,
JITSI_WEB_HTTP_PORT, KEYCLOAK_SERVER_HOST, MINIO_API_PORT_INTERNAL,
OPENCLOUD_HTTP_PORT, OPENWEBUI_PORT_PUBLIC).
- Add PORT_BANDS.public entries for the fixed protocol ports the migration
declared (smtp/smtps/submission/pop3/pop3s/imap/imaps/sieve, rtmp, c2s/s2s).
- Route the container_ports internal-declared lint through the SPOT constants
(ROLE_FILE_TEMPL_COMPOSE / ROLE_FILE_META_SERVICES), cached read_text, and
load_yaml_any, per repo conventions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DEPLOYMENT_MODE is binary (swarm / compose), so a negated `!= 'swarm'` gate
always means `== 'compose'`. Replace every negated gate across roles/ and
group_vars with the positive form for readability, and add a lint
(tests/lint/ansible/variables/test_no_deployment_mode_negation.py) that forbids
`DEPLOYMENT_MODE !=` in .yml/.yaml/.j2 with a `# nocheck:
deployment-mode-negation` escape.
The compose-verb / chdir / jinja-gate lint guidance messages now suggest the
positive form too; the gate matcher (utils/annotations/task_gate.py) and its
test inputs keep recognising the legacy `!= 'swarm'` so a nocheck-marked
exception is still detected as compose-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Foundation for the central svc-db-*/svc-dns-* engine roles: the engine lookup and engines descriptor, per-consumer engine credentials in the inventory manager, shared-net overlay rendering, container_depends_on, the sys-svc-engine provisioning dispatcher, and the compose_volumes NFS-placement split. base.yml.j2 gates the embedded redis sidecar on lookup('engine','redis',application_id,'local').
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add group_vars/all/18_api.yml defining the global API mapping (cloudflare, telegram, github, google, jira) and plugins/lookup/api.py — the single sanctioned reader that resolves a dotted 'provider.key' path against it and raises on unknown keys. Routing every proprietary/external credential access through one lookup keeps a single point to change how credentials are sourced. Includes the unit test covering nested resolution, subtree access, error paths, and templar handling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make web-app-keycloak deployable in swarm without the manager-only
assumptions of the original compose-only shape.
Manager-side delegate scaffolding:
* tasks/02_manager_ops.yml: switch the `include_tasks` of
02a_manager_ops_on_host.yml to the dict form so the `apply:
delegate_to: KEYCLOAK_HOST_NODE` block actually attaches (free
form raised `conflicting action statements: include_tasks, apply`
in ansible 2.19).
Swarm bind sources that exist only on the manager:
* tasks/05b_realm_import_swarm.yml: pin the one-shot
`<stack>_<service>_import` service to `node.role == manager` so
the realm-import bind mount `/opt/compose/keycloak/volumes/import`
resolves.
* templates/sso_proxy/container.yml.j2: same manager pin for the
sso-proxy sidecar (its oauth2-proxy cfg bind source also lives
only on the manager). Also add `extra_hosts` for the keycloak
auth domain pointing at `host-gateway` so OIDC discovery reaches
the host's openresty, and mount the self-signed CA when TLS_MODE
is `self_signed`.
* templates/sso_proxy/oauth2-proxy-keycloak.cfg.j2: add
`provider_ca_files` so oauth2-proxy validates the self-signed
cert instead of failing OIDC discovery with TLS EOF.
CA mount path SPOT:
* group_vars/all/02_tls.yml: add `CA_TRUST.cert_container` for the
in-container CA file path so the literal `/etc/ssl/certs/...` is
defined once.
* roles/web-app-matrix/vars/main.yml: consume the new SPOT.
Container ID resolution in delegated tasks:
* vars/main.yml: add `KEYCLOAK_TARGET_CONTAINER` (= LOCAL_CID with
fallback to KEYCLOAK_CONTAINER_ADDRESS) and recompose
`KEYCLOAK_EXEC_CONTAINER` from it, replacing the inline default.
* tasks/update/05c_per_app_mappers.yml and
tasks/update/ldap/per_app_mapper_one.yml: `KC_CONTAINER` env now
resolves via the new SPOT, so the per-app-mapper helper scripts
hit the actual swarm container ID on the delegated host node
(compose mode keeps using the bare container_name via the
fallback).
Delegated script paths:
* tasks/update/08_users_email_verified.yml and
tasks/update/08_users_realm.yml: drop
`chdir: "{{ role_path }}/files"` from `ansible.builtin.script`.
role_path is a controller-side absolute path; on a delegated
DinD swarm node that path doesn't exist and the script fails to
start. Both scripts only consume env vars.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Renumber the tasks/ tree to a flat NN_ scheme (drop the 02b / 03a / 03b
suffixes) and update all include_tasks references; main.yml -> 01_core
stays unchanged. Lift the hardcoded matomo bootstrap timezone label
into a global HOST_TIMEZONE_LABEL constant in group_vars/all so other
apps can share it; default to "United Kingdom" because GB has a single
IANA timezone and matomo's getTimezoneName collapses to the country
name in that case. Drop unconditional double quotes from env.j2 and
bootstrap.env.j2 since docker stack deploy preserves env-file quotes
literally and was feeding matomo "matomo" instead of matomo as the
DB name. Add MATOMO_ADMIN_EMAIL via the users lookup so the bootstrap
template can reference a typed variable instead of constructing the
address inline. Switch 08_patch_trusted_hosts to ansible.builtin.shell
with the script body piped in via stdin from lookup('file', ...) and
shrink the script file to just the inner sh body. Strip section-banner
and restate-the-code comments from vars/main.yml, env.j2 and the
extracted script; keep the two genuine trip-wire comments (matrix
deploy round-2 DB creds, include_role-chain role_vars pollution) in a
shorter Exception form.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a `swarm:` subcategory under `svc:` in roles/categories.yml so the
shared-cluster role is properly invokable, and renames the existing
`svc-swarm` role + inventory group to `svc-swarm-node` (with
`get_entity_name`-aligned primary key `node:` in meta/services.yml).
`svc-swarm-manager` stays as the manager-marker role, becoming invokable
via the same new category and re-keyed to `manager:`.
Consumers of the cluster-orchestration service (svc-registry-cache,
svc-registry-docker, svc-swarm-manager) now reference the service-key
`node:` (was `swarm:`), matching the renamed primary entry. The planner
empirically resolves `service_registry['node']` -> `svc-swarm-node` and
emits it into --include for stack-host deploys, so the CI failure on
`universal-svc-registry-docker` (`svc-swarm: not present in inventory`)
no longer reaches `ValidDeployId`.
Breaking change for operators: inventory groups named `svc-swarm:` must
be renamed to `svc-swarm-node:`. `DEPLOYMENT_MODE` reads
`groups['svc-swarm-node']`, so a stale `svc-swarm:` group silently falls
back to compose mode.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a mode-aware `container_address` lookup plugin and a
`resolve-container-id` helper script that resolve swarm task IDs at
exec time (compose mode returns bare names unchanged). Migrates ~25
roles + ~28 task files from `command:` to `shell:` so the lookup's
`$(...)` subshell evaluates correctly, and extends `database` lookup
with an `address` key for cross-role DB exec.
Three new lint tests enforce the contract: every container exec
target goes through the resolver, lives behind an IS_STACK_HOST gate
(walks the include graph one hop deep), and uses `shell:` rather than
`command:` when the target is a Jinja reference. Existing nocheck
suppressions on bare-name DB exec sites (nextcloud redis, listmonk,
moodle OIDC, mailu) are replaced with the lookup-based form.
Fixes the systemd Environment= injection in sys-ctl-cln-disc-space
(it never expanded $(...)) by passing STACK / SERVICE_KEY parts and
resolving inside the script. Extracts shopware admin and mediawiki
extension install bodies into dedicated .sh files invoked via env.
Rewrites password-bearing shell tasks to either `| quote` directly or
the `-e VAR=... | quote` + `sh -c '... ${VAR} ...'` env-bridge pattern.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The swarm-test rig provisions storage via utils/tests/swarm/write_extras.py
which emits storage.nfs as the complete {server, export_base, version}
triple, overwriting (not merging) the group_vars default. Any key nested
under storage.nfs that is NOT re-emitted by the provisioner is lost on
the test rig, which broke svc-storage-nfs-server with
"object of type 'dict' has no attribute 'state_subdir'" on the
e415f97b0 swarm baserow CI job (run 27226937629).
Move state_subdir out of the shadowable storage.nfs sub-dict to a
top-level constant STORAGE_NFS_STATE_SUBDIR, and DRY the four call
sites that previously hand-built `[export_base, state_subdir]|path_join`
into a single derived constant STORAGE_NFS_STATE_PATH. The provisioner
needs no change because the constants live outside its override target.
Migrates four consumers (svc-storage-nfs-server/tasks/03_exports.yml,
05_controller_mount.yml, templates/exports.j2 and
svc-storage-nfs-client/tasks/main.yml). Fixes Cluster A and the
cascading Cluster B (workers cannot probe-mount because
nfs-server never wrote /etc/exports).
Resolves Cluster C (12 universal jobs) + Cluster F (4 swarm jobs)
from CI run 27063963773.
Cluster C: `groups[application_id]` raised "'application_id' is
undefined" in contexts without an active role (universal-deploy for
sys-ctl-alm-compose etc.), aborting the play at group_vars eval time.
Cluster F: per-role overrides of application_id (e.g. web-opt-rdr-
domains setting it to a domain string) silently flipped DEPLOYMENT_MODE
to 'compose' on every host. IS_STACK_HOST then evaluated to True on
swarm workers, defeating the stack_host_template skip-on-workers gate
and causing "Destination directory /etc/nginx/conf.d/servers/http
does not exist" failures on swarm-wrk-*.
`groups['svc-swarm']` is host-stable: depends only on inventory
membership, never on per-role context.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Resolves CI swarm rc=69 ("msmtp: needs a password"): controller-side
Ansible lookups (lookup('email').password etc.) couldn't see the
tokens.yml that Mailu writes target-side, because the controller has
its own filesystem in swarm mode.
Mount NFS-shared /var/lib/infinito on every swarm host AND on the
controller during deploy (in-play delegate_to: localhost task at end
of svc-storage-nfs-server role). State subdir exported separately
with no_root_squash + wildcard CIDRs so root-owned writes succeed
and the controller can mount from outside the svc-swarm IP set.
Single-writer guarantee on FILE_TOKENS via delegate_to: STACK_HOST +
run_once on sys-token-store/{01_core,write}.yml. Backup roles
(svc-bkp-container-2-local, sys-ctl-cln-bkps, sys-ctl-cln-faild-bkps,
user-backup/03_permissions_folders, three direct callers of
04_seed-database-to-backup.yml) gated on IS_STACK_HOST to prevent
N-host write races on the shared NFS backup tree.
Fail-fast assertion in svc-storage-nfs-client + 05_controller_mount
when pre-existing local /var/lib/infinito would be hidden by mount-
over. Operator must migrate or wipe before re-deploying.
SPOTs: SWARM_LAB_NETWORK in workflow env block, storage.nfs.state_subdir
in group_vars/all/15_storage.yml, DIR_VAR_LIB unchanged. UPPER_CASE
constants in NFS roles. Cleanup script umounts controller view before
container teardown to prevent stale-mount leak between matrix jobs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
