Lookup plugins receive ansible variables unrendered, so routing the state subdir through a group_vars jinja value handed state_path the literal '{{ lookup(...) }}' string and the controller mount asked the server for /srv/nfs/{{ lookup('nfs_state_subdir') }} (Connection refused, nextcloud swarm roundtrip). Both nfs lookups now import STATE_SUBDIR directly; the intermediate lookup, the group_vars indirection and the unit-test fixture var that masked the gap are gone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
04_bootstrap_nodes.sh consumed INFINITO_PACKAGE_INSTALL_SCRIPT and INFINITO_NODE_SRC_DIR from the ambient environment; local make targets provide them via BASH_ENV, but plain workflow steps do not, so every swarm CI job died at the required-var guard. Source load.sh explicitly like copy_worktree_to_node.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The archlinux-keyring upgrade hook now signs the local keyring and aborts with 'There is no secret key available to sign with' when pacman-key --init never ran (fresh arch containers in the CI image build). Initialize the keyring, idempotently, before every pacman -Syu.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test-e2e-cli discovers roles via group_names, not per stage, so the three per-stage invocations ran the same test set up to three times per play. A single invocation after sys-service-terminator also guarantees the final service flushes (including the backup generation) happened before the tests inspect their results.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
role_is_stack in the role-mapping SPOT recognises compose.yml.j2, *.compose.yml.j2 siblings and compose.*.yml.j2 variants (bigbluebutton's compose.override.yml.j2). The new lint walks every invokable stack role whose meta/volumes.yml declares a named volume and requires the container_backup consumer flag in meta/services.yml, so no stack role's data can silently stay outside the backup plan. Opt out per role with '# nocheck: backup-service-flag <reason>' on the first line of meta/volumes.yml.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Variant 0 pins the backup consumer flag enabled so the matrix baseline exercises the in-group branch; every other variant pins it disabled (shared: false carries the static-override suppression since the base value stays literally true). The matrix and wordpress variant entries move their services mapping to the list-item head so the static-override line parser resolves the override paths; yaml semantics are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every invokable stack role that declares named volumes in meta/volumes.yml now carries the standard container_backup consumer entry: enabled resolves via 'svc-bkp-container-2-local' in group_names, shared stays literally true since there is no per-consumer backup instance. Playwright/dynamic-flag/bond-baseline suppressions ride on the block per the established idioms; stale explanatory comments in the touched files are dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Register the role in the service registry under the abstract container_backup name (provides + enabled/shared/bond on the primary entry) so consumer roles can declare the backup dependency without naming the implementation. The run-once lint learns to find the 01_core.yml include inside a gated block, since this role wraps its core load in the IS_STACK_HOST block instead of duplicating the condition per task.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The run-once guard lint required the when to be exactly the guard string, rejecting roles that legitimately combine it with further conditions (svc-bkp-container-2-local gates its whole core on IS_STACK_HOST). A when-list containing the guard now passes. The variants coverage test learns the same block-above '# nocheck: dynamic-flag' key suppression the services dynamic-flags test already honors, so infrastructure consumer flags can opt out of two-polarity variant coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
INFINITO_PACKAGE_INSTALL_SCRIPT in default.env replaces the scripts/install/package.sh literal in the Dockerfile (new required build arg), the environment install step, and both swarm bootstrap call sites. Shell consumers hard-require the variable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First boot on loaded fork runners exceeded the healthcheck windows: swarm reaped moodle's fpm container after 60s+5x1m (exit 137 mid-install) and nextcloud's occ-status containers right at the 10m+3x1m boundary. Extend to 15m/20m following the joomla/espocrm/suitecrm precedent; container exits still reap immediately, only hanging unhealthy checks wait longer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
All three swarm nodes probed the shared export with the same destination file; ansible's atomic tmp-write + rename raced across NFS clients and failed with 'Destination ... does not exist' (bridgy-fed). A per-host suffix removes the collision by construction.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Matomo can answer SitesManager queries with HTTP 200 carrying an error dict ({"result":"error",...}) instead of the expected site array; indexing site_check.json[0] then crashed the deploy of every app with tracking enabled (decidim). Route the mapping shape into the existing API-not-ready deferral, keeping the array path untouched. A dict passes Jinja's sequence test, so is mapping is the only reliable discriminator.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A bare input[type=file] .first() can bind the dashboard's .penpot/.zip import uploader during the SPA transition into the workspace, pushing the PNG through the import pipeline (media object not found dialog) instead of the toolbar image upload. Target #image-upload explicitly and scope the layer assertion to the Layers tabpanel, since the import-error dialog lists the filename too and a page-wide match would false-pass on the exact failure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Variant 1 runs redis without a password; the unconditional env line handed the image an empty REDIS_HOST_PASSWORD, phpredis sent AUTH to a passwordless redis (ERR Client sent AUTH), the session backend died and every login bounced in a CSRF loop. Emit the line only when the engine actually has a password; the rendered env is byte-identical whenever one is set.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A wedged DinD dockerd hung a single diagnostics docker exec until the 360-minute job timeout cancelled the job together with its artifact uploads (job 86013153099 sat 5h in service logs). Route every diagnostics exec through a timeout wrapper, tail-limit the log dumps, bound showmount, cap the collect step at 30 minutes, and dump the pending services' log tails once on the final converge failure. The NFS state path moves into a single source of truth: STATE_SUBDIR in utils/storage/nfs.py feeds the nfs_state_subdir lookup (deploy side) and the derived INFINITO_SWARM_NFS_STATE_PATH env var (harness side); the diagnostics script requires both path vars instead of defaulting, and the swarm inventory drops its redundant export_base restatement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-role existence asserts and the test script path were controller paths evaluated on the remote swarm node (docker connection, node checkout under /opt/infinito-nexus), so every svc-runner swarm job failed on a stat that could never succeed. Assert both required assets in one localhost-delegated loop, stage roles/<app>/files/test onto the node, run the staged script, and delegate the optional defaults/vars probes to the controller (they silently skipped before). svc-runner's test reaches its sources via RUNNER_SRC_DIR (compose: INFINITO_SRC_DIR dev-container mount; swarm: node checkout). The node checkout path itself moves into a single source of truth: INFINITO_NODE_SRC_DIR in default.env, consumed by the bootstrap/copy scripts, the swarm-playwright target, and the swarm inventory extras.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In swarm mode the container_address lookup returns a $(/usr/bin/resolve-container-id ...) fragment that must expand at exec time; piping it through Jinja quote hands docker the literal string and every exec fails with 'No such container' (broke all three snipe-it CI variants). Remove the quote at the seven snipe-it sites and the live matrix/openproject sites, convert openproject's argv exec to shell so the fragment can expand, split the matrix bridge restart per deployment mode with a manager gate, and mark the compose-only or eval-contract sites with nocheck. A new lint (container-address-quote) bans the pattern repo-wide.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bare wget -q exited 8 on raw.githubusercontent.com rate limits under the parallel CI fan-out, failing the image build and with it whole deploy jobs (kix, suitecrm, akaunting). Retry with backoff on 429/5xx and log the actual http status.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The controller crash-looped on a missing S3 bucket and the recorder on an unprovisioned OIDC client while the swarm converge gate blocked before any post-flush task could create either. The embedded seaweedfs sidecar now bootstraps its consumer bucket itself at startup (new seaweedfs_sidecar_script filter: entrypoint-wrapped server, bounded readiness poll, idempotent weed shell bucket create, TERM/INT forwarded to the server), and the Keycloak clients are provisioned before the handler flush. Post-flush bucket tasks stay as an idempotent backstop; the obsolete restart tasks are dropped. Live-validated against chrislusf/seaweedfs:4.38: bucket created, stop behavior identical to the stock entrypoint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The override declared prometheus-exporter unconditionally with only volumes, while the upstream-generated base compose emits the service (with its image) only when ENABLE_PROMETHEUS_EXPORTER is set. Deploying without web-app-prometheus in the group produced a merged project with a service that has neither image nor build, which compose config rejects. The guard mirrors the env.j2 gate, keeping override and base in lockstep like the greenlight and recording blocks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
n8n CE caches isInstanceOwnerSetUp per process without cross-replica invalidation, so the swarm topology default of 3 replicas left stale replicas bouncing logged-in sessions to /setup (variant 1) and caused session-crash churn during the reschedule test (variant 0). Pin to 1 replica; multi-main needs licensed queue mode. Add the conventional /healthz wget healthcheck so swarm convergence and the chaos-harness Health.Status probe carry a real readiness signal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The single-file cert bind only exists on the rendering node, so swarm rejected every coturn task scheduled onto a worker (bind source path does not exist) and the nextcloud/bbb/opentalk stacks never converged. A config is raft-distributed to all nodes; the key stays a secret.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tighten verbose comments to one line each (no-long-comments) and remove
leftovers from the old mesh design: unused NODE_OCTET / WG_PORT and the
dead WIREGUARD_IMAGE/VERSION env asserts.
Reuse the proven client tunnel mechanics but restructure the e2e into 3
server/client pairs: the 3 debian nodes deploy in [server] flavor
(linuxserver generates each peer config), and each workstation joins via
that config in [client] flavor (wsdebian as [client, nat]). Verify each
client<->server tunnel, the NAT masquerade rule on the behind-NAT node,
and a sane auto-derived wg0 MTU.
Replace the hardcoded 1400 client MTU with a pkgmgr automtu probe (path
MTU minus WireGuard overhead), falling back to the configured value when
probing is unavailable. Runs before the compose env is rendered so WG_MTU
reflects the derived value (decision 11).
Endpoints used the outer node name (wg-e2e-<m>), which the wireguard
container cannot resolve from inside the peer node's inner dockerd, so no
handshakes formed (all 30 pairs down). Resolve each node's WGNET IP and
use it as the peer Endpoint, reached via the published relay port. Also
dump `wg show` per node before the connectivity check for debugging.
With the default server flavor, env.j2 sets PEERS/SERVERURL so the
linuxserver image runs in server mode and regenerates wg0.conf on
restart, wiping the rendered full-mesh peer set -> no handshakes (all 30
pairs unreachable). Deploy the mesh with WIREGUARD_FLAVOR=[client] so the
container honors the provided wg0.conf.
The DinD daemon.json hardcoded fuse-overlayfs, which fails copy-up on
images shipping an /etc/mtab symlink (e.g. linuxserver/wireguard) under
deeply-nested CI: "symlink /proc/mounts .../etc/mtab: file exists".
Expose SYS_SVC_CONTAINER_STORAGE_DRIVER (default fuse-overlayfs,
unchanged) and let the wireguard e2e deploy override it to vfs.
make install only sets up the venv + collections; the systemd dependency
comes from the distro package (built at image-build, not run here), so
/sbin/init never existed and the systemd re-create failed. Install
systemd (+ systemd-sysv / systemd-sysvcompat for /sbin/init) as a
bootstrap prerequisite on every node.
The pkgmgr bases have no systemd (no /sbin/init), so booting init failed;
systemd only arrives as an infinito package dependency during make
install. Two-phase: boot the bare base, run the per-node make install
(fresh environment), mask container-hostile first-boot units + seed a
machine-id, then commit and re-create the node with --entrypoint=/sbin/init
so systemd is PID 1 and the deploy can start docker.service like a host.
The pkgmgr base image's ENTRYPOINT runs the pkgmgr CLI, so passing
/sbin/init as a command arg never made systemd PID 1 (systemd never
became ready). Override with --entrypoint=/sbin/init so systemd (already
present in the base) boots as PID 1, and dump container logs on a
readiness timeout.
The role starts docker via systemd (docker.service), which raw debian/
manjaro/centos containers (PID1=sleep) cannot do. Boot the nodes from
Infinito's own fresh pkgmgr base images (systemd + pkgmgr, no infinito
baked) as systemd containers (cgroupns=host, cgroupfs rw, tmpfs /run),
so each node still runs its own make install + inventory + deploy and
the deploy can install/start docker exactly like a real host. Images are
pinned to 1.15.2 (never a moving tag); manjaro workstation -> arch base.
sys-svc-container configures the docker-ce apt repo via apt_repository,
which needs a gpg binary to handle the signing key; the minimal images
lack it (Either apt-key or gpg binary is required). Add gnupg/gnupg2 to
the per-distro prerequisites.
The journald cleanup ran under set -e but journalctl --vacuum-size lacked
a || true guard, so on minimal containers without journald the deploy
hard-failed (journalctl: command not found, rc 127). Skip the block when
journalctl is absent and make every call best-effort.
The -e container=docker env did not flip DOCKER_IN_CONTAINER: fact
gathering runs under become/sudo, which env_resets and drops `container`
before the setup module reads it. Force DOCKER_IN_CONTAINER=true as an
ansible extra-var (highest precedence) so the deploy takes the
container/DinD path and skips the host-only timezone task.
Infinito's DOCKER_IN_CONTAINER derives from the `container` env var,
which plain Docker (unlike systemd/podman) does not set, so the nodes
misdetected as bare metal: the constructor ran the host-only timezone
task (needs hwclock) and sys-svc-container would take the non-DinD path.
Boot nodes with -e container=docker so autodetect is correct.
The deploy runs on the target node itself, but ansible defaulted to SSH
and failed UNREACHABLE (Connection refused, no sshd). Force
ansible_connection=local via extra-var so the role applies locally.
provision auto-creates a random vault password at <inv>/.password and
encrypts ansible_become_password with it. deploy dedicated does not
default to it, so rendering become_pass failed. Pass --password-file
<inv>/.password (node-local, written by provision) to the deploy.
The shared bind-mount let make install state (.env, venv stamps) leak
across nodes, so only the first nodes ran the real install and the rest
short-circuited with no venv (no infinito venv python). Mount the repo
read-only as -src and tar-copy it (excluding .git) into a node-local
/opt/src/infinito per node, so every node runs a full, independent
install. Aligns with the per-node-inventory design.
server1 resolved /opt/venvs/infinito/bin/python but a later node did
not, so the venv lands at a different path/name there. Glob the common
venv parents (/opt/venvs/*, /root/.venvs/*, $HOME/.venvs/*) instead of
two fixed paths, and on a miss dump the existing venv dirs + .env venv
lines so the real location is visible in CI.
load.sh's PYTHON resolver silently falls back to the system python3
(missing install deps -> ModuleNotFoundError: ruamel) when it cannot
confirm the venv, which happened on later nodes once server1 generated
the shared .env. Pin /opt/venvs/infinito/bin/python (or the HOME venv)
explicitly and fail loudly if absent, for both provision and deploy.
The `infinito` console script was not reliably on PATH across distros
(found on debian, missing on centos). Call the venv "$PYTHON" -m cli
from the repo root instead — deterministic, no console-script/PATH
dependency, and the venv python carries the install deps.
Bare `python3 -m cli` used the system interpreter, missing make install's
venv deps (ModuleNotFoundError: ruamel). Source scripts/meta/env/load.sh
to put /opt/venvs/infinito/bin on PATH and call the `infinito` console
script for provision + deploy.
