The nginx sidecar resolved the moodle container at config load, so it
died or pinned a stale IP whenever the fpm container was not up yet or
got recreated (hlth-webserver failures in compose moodle 0,1,2).
Server-level resolver via NETWORK_DOCKER_DNS_NGINX_DIRECTIVE plus a
variable-based fastcgi_pass defers the DNS lookup to request time; a
literal host would keep ignoring the resolver directive entirely (same
pattern as web-app-bluesky). Verified with nginx -t on the rendered
vhost in the stock nginx image, including startup with an unresolvable
upstream.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
deploy.yml.j2 keyed on the cluster-wide DEPLOYMENT_MODE, so forced-compose
roles (web-app-matrix ansible flavor, web-app-bigbluebutton) rendered a
swarm deploy block into their node-local compose files; docker compose v2
honors deploy.resources.limits, capping Synapse at 512m and resetting
connections in the swarm nextcloud-3 job. Use the same
compose_mode_force-or-DEPLOYMENT_MODE resolution the compose_* and
container_* lookup plugins apply; the default() is required because
renders like the xwiki manager-ops re-render run outside the
sys-svc-compose scope where compose_mode/compose_mode_force are defined.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Upstream actionInstall() runs create-admin-user "$ESPOCRM_ADMIN_USERNAME"
but then pipes the password to a hardcoded 'set-password admin' (line 203,
verified in the 10.0.2 image); with our admin named 'administrator' the
silenced failure aborts the install under set -euo pipefail and loops the
container. Patch the line at image build time; the grep guard fails the
build if a future image reshapes it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New lint rejects literal port numbers in roles/*/vars/main.yml: the port
belongs in meta/services.yml (services.<svc>.ports.<scope>.<name>) where
band governance and stack rendering read it; vars resolve it via
lookup('config', ...). Migrated the 16 remaining literals across
elasticsearch, registry-docker, erpnext, flowise, funkwhale, magento,
moodle, nextcloud, penpot and shopware.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace scripts/system/diagnostics/rescue.sh with utils/diagnostics/
container.py: one workflow call on failure captures the current level
(containers with logs+inspect, swarm services, journal, host resources,
in-play role dumps) and recurses itself through every DiD boundary via
self-copy, from the runner down to the innermost runtime (RESCUE_MAX_DEPTH).
compose and swarm now share an identical failure step; the swarm per-node
collector wrapper is gone. All 17 script-only Ansible rescue: blocks are
deleted and their bare block: wrappers inlined (plus a wordpress leftover);
the three justified blocks (postgres pg_hba, xwiki extensions, keycloak
cross-node recovery) keep their evidence tasks and now end in a fail task.
Two lints enforce the state: rescue: is forbidden without a
'# nocheck: rescue-<reason>' first-task marker, and a block without
rescue/always/when or another scoping key must be inlined.
sys-service-terminator hoists become to the include apply. Also move
routine/08_cleanup.sh to utils/clean/teardown.sh and topology/ to
utils/topology/ so routine/ holds only the sequenced flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extend_inventory maps svc-bkp-remote-2-local + svc-bkp-local-2-device onto
the backup host and write_extras configures them (backup_providers =
manager + NFS server IPs; device mount/target/source), so the matrix
deploy installs the real systemd units there. The drill now exercises
them: 01_ssh_trust.sh only installs the pull identity, the remote-2-local
unit pulls every provider (replacing both manual pull calls), and after
02_luks_device.sh simulates the USB plug the local-2-device unit performs
the device sync; the device recover passes the configured target subpath
via --device-target. Unit absence on the backup host is now a hard
failure since the roles deploy with the round.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
svc-bkp-volume-2-local now declares secrets_backup (enabled+shared true):
host secrets are captured unconditionally wherever container backups run.
svc-bkp-remote-2-local declares local-2-device gated on group membership
(operator flag, no variants) so the pull host syncs to the encrypted
device when the role group is assigned; local-2-device registers as a
shared provider and gains the 01_core run-once schema. local-2-device
recover.py learns --device-target: snapshots live under <mount>/<target>,
so recovering without the configured target subpath found nothing.
secrets-2-local memory shrinks to 128m/256m (a oneshot rsync) to keep
variant stacks under the host memory budget.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add routine/backup/: base.sh drives a marker seeded into the live NFS
volume and the manager secrets through the whole chain (volume+secrets+nfs
backup units -> remote-2-local pull -> local-2-device LUKS 'USB') and back
via every recover.py (device -> local root -> nfs export, docker volume and
host secrets), asserting each marker survives. Each per-host step is one
in-node routine run over docker exec. extend_inventory maps the volume/
secrets/nfs backup roles onto their hosts so the units install; write_extras
generates the pull keypair; the INFINITO_SWARM_BACKUP_* keys are added to
default.env + the passthrough handler.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the per-node docker-run bring-up with a single node image
(compose/swarm/Dockerfile bakes python3/dnsmasq/openssh/rsync/cryptsetup)
declared in compose/swarm.yml, one compose up, and one ansible play
(compose/swarm/playbook.yml over ansible_connection: docker) that waits for
systemd, captures IPs, wires lab DNS and installs the .deb per node. Reshape
the harness into topology/ (naming SPOT), routine/ (01_bootstrap + the
sequenced 02-08 steps) and utils/ (clean/, collect/, shared helpers); the
workflow, matrix.py and Makefile follow the new paths. swarm init/join was
already ansible-owned (svc-swarm-node), corrected in the README diagram.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New svc-bkp-secrets-2-local role: a differential rsync --link-dest snapshot
of the host-generated secret material that lives in no docker volume, NFS
export or operator inventory - infinito secrets + tokens, the self-signed
CA, the Let's Encrypt account/certs, the ACME DNS credentials - plus the
node identity (ssh host keys + machine-id). files/recover.py restores each
present subtree to its fixed system path and gates the node-identity
restore behind --restore-node-identity. Wires the systemd unit + nightly
schedule (06_daemons, 07_schedule). Unit tests cover the domain/target map
and the main() subtree-selection + node-identity gating.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add utils/recovery/base.py DirectoryRecovery: a service-backed pre-recover
(start the role's backup unit first) + rsync -a --delete mirror. Each
svc-bkp-* role gets a files/recover.py subclass (volume resolves the docker
mountpoint, nfs mirrors the export, local-2-device opens the newest LUKS
snapshot across machine hashes) or a documented one-way opt-out
(remote-2-local recover.py.nocheck). A lint guard enforces every svc-bkp-*
role ships a recover.py and a ## Recover README section; docs/architecture/
backup.md documents the end-to-end schema. Unit tests cover the base flow,
the per-role class logic and every recover.py main() CLI wiring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Env handlers feed the .env generation, which bootstraps fresh hosts
before PyYAML is initialised, so the YAML cache is unavailable at that
point. The guard fails on any utils.cache.yaml use under
utils/env/handlers/ and points to utils.cache.files.read_text with a
stdlib line parse (canonical pattern: utils/storage/nfs.py); per-line
nocheck suppression stays possible.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A default like ${BACKUP_KEY_PATH:-/tmp/swarm-nfs-backup.key} duplicates
a path contract that belongs in a SPOT (default.env or a role vars
file) and drifts silently. The guard covers every ALL-CAPS variable
whose inline default starts with a slash, allows per-line nocheck
suppression, and its failure message forbids dodging the syntax via
conditional assignments. Legacy offenders carry explicit nocheck
reasons (act tooling runs before default.env exists, POSIX TMPDIR
convention).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New handler reads TEST_E2E_PLAYWRIGHT_STAGE_BASE_DIR from
roles/test-e2e-playwright/vars/main.yml (via ROLE_FILE_VARS_MAIN, plain
read_text line parse because env handlers bootstrap fresh hosts before
PyYAML exists) and publishes it as INFINITO_PLAYWRIGHT_STAGE_BASE_DIR.
rerun-spec.sh consumes the handler value and falls back to parsing the
same SPOT on DiD nodes without a generated .env, dropping its /tmp path
copy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The v10 image entrypoint silences several install commands
(create-admin-user, set-password run under set -e with all output
redirected), so a failing init crash-loops the swarm service with
nothing in docker logs. Trace the init to a file, mirror the upstream
info/warning/error lines, and dump the trace tail plus the newest app
log to stderr when init exits non-zero.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Controller v0.34 restructured its configuration: the top-level livekit,
turn, stun and log keys are gone, livekit now lives under the mandatory
roomserver section (run internally via kind=internal) and
http.service_api_keys became required for the roomserver integration
(new service_api_secret credential). TURN moves to the livekit embedded
server on the stun_turn band port 3479 since the controller no longer
brokers coturn to clients. Verified by booting controller v0.34.1 with
the rendered template (settings load, startup reaches the DB connect)
and livekit v1.13.3 with the TURN block (TURN server starts). LiveKit
and controller ports resolve from the services.yml ports SPOT instead
of hardcoded values.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The runner Dockerfile adds the download.docker.com apt repo per
VERSION_CODENAME, which does not exist yet for the 26.10 interim release
(stonking), so the image build failed with a missing Release file. Hold
the LTS pin via nocheck until docker publishes the codename repo.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Agent sandboxes run in a network namespace with an empty /proc/net/route
while /sys/class/net still exposes the host NICs, so dotenv generated the
static 1500 fallback although the host uplink is 1400 and every TLS
handshake from the DinD stack blackholed. Take the minimum MTU across UP
physical interfaces (device symlink present) before resorting to the
static default.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The DiD deploy container boots systemd, whose persistent apt-daily timers
fire a catch-up run at a randomized delay after boot. apt fails hard when
another process holds the dpkg lock, and the apt module's python3-apt
bootstrap only waits 60 seconds, so the play's first apt task died with
'Could not get lock /var/lib/dpkg/lock-frontend' whenever the catch-up
happened to be running (two of the last three sweeps).
One Debian-gated block early in the constructor: stop and disable the
apt-daily timers inside an ephemeral deploy container (production hosts
keep their nightly update timers), then wait for the dpkg lock natively
via apt-get check with a 600s DPkg::Lock::Timeout so a catch-up that is
already holding the lock finishes instead of failing the play.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The job-level 360min limit was the only backstop, so a wedged deploy ran
5.5 hours as a zombie before the cancel. The step cap cuts that while
leaving full multi-hour deploys (nextcloud, bigbluebutton, multi-variant
rounds) enough room; the in-container exec timeout catches wedges far
earlier and with diagnostics.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two wrapper behaviours move onto single sources of truth:
Exec timeout: SYS_TIMEOUT_CONTAINER_EXEC (07_schedule.yml, default 7d so a
hard kill never beats a production migration; the dev/CI inventory
tightens it to 60min) reaches every task host via the play environment as
CONTAINER_EXEC_TIMEOUT. The wrapper prefixes only the exec subcommand with
coreutils timeout --kill-after=15. Ansible task timeouts are SIGALRM-based
and slept through the last wedged docker socket for 4.5 hours; the OS
level kill still fires there and returns 124, which until/failed_when and
the rescue diagnostics react to. Unset or 0 keeps todays behaviour.
CA container paths: the bind-mount targets for the injected root CA and
wrapper script lived as string literals in container.py, compose_ca.py
and the filter plugin. They now live once in the CA_TRUST group_vars SPOT
(inject_cert_container, inject_wrapper_container) and reach container.py
as env vars (play environment plus CA_TRUST_SYSTEMD_ENV) and compose_ca.py
as CLI arguments built by the compose_ca_inject_cmd lookup. A caller that
provides the host CA env but not the container paths is half-migrated and
fails loudly instead of silently dropping CA injection; a host without any
CA env keeps the soft fallback. All tests read the paths from the SPOT,
and a regression test asserts the runtime scripts carry no literal copy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1.8.0 ships the restore fixes the svc-bkp e2e drill needs: the mariadb
empty mode drops FK-linked tables in one session, the postgres replay
skips superuser-only dump lines without touching COPY data, dumps are
produced app-user-replayable, and swarm task containers are never
stopped around volume backups.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
