The swarm inventory carried the manager's docker bridge gateway as a literal under networks.internet.dns, so one field meant two things at once and the value went stale the moment default-address-pools moved the bridge. The daemon kept handing out the new address while the inventory kept naming the old one, and the DinD diagnostic probed that same literal, so it failed in step with the defect instead of catching it.
The inventory now declares only the clearnet resolver. The lookup hands containers the bridge whenever the host resolves solely through a loopback listener, which is how the simulated cluster nodes are wired and which docker strips from a container's resolv.conf. The diagnostic reads the gateway from the running daemon instead of being told it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A handler name was doing two jobs at once: the line an operator reads in the
play output, and the identifier a `notify:` binds to. The two pull in opposite
directions -- the log wants "restart the SSH service, with diagnostics", the
binding wants a token that survives quoting, YAML flow style and a shell
round-trip -- and a name that changes for readability silently changes which
handler runs.
Every handler that a notify reached by name now carries a `listen:` with that
exact wording, so the binding is explicit before anything is renamed. The
topics themselves become identifiers: `compose up` reads `compose-up`,
`Restart NFS ganesha` reads `restart-nfs-ganesha`, 22 of them across both
sides of every notify. Where the topic is assembled at runtime, both branches
moved together: the conditional in sys-service carries two literals, and the
LDAP renderer interpolates `import-{{ folder }}-ldif-files`, which is what
makes those handlers reachable at all.
Freed from the binding, the names take the emoji prefix every other task line
already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The check asserted the A record equals networks.internet.ip4, which answers a different question: under which public address the deployment answers. The two coincide in compose and diverge in the swarm lab, where dnsmasq maps the domain to the manager, so job 91358007229 failed on three healthy nodes that had resolved the name correctly.
Assert that an address came back instead. The SERVFAIL guard is unchanged, and the role no longer reads networks.internet.ip4 at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make quality wrapped the conditional NODE_IMAGE expression introduced in 823c6f4ee; the rendered value is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The check starts a container with the advertised resolver, which is exactly the path a pre-deploy build takes, but MODE_DEBUG gated it off in the swarm drill: that path loads only inventories/development/swarm.yml, which never sets it. Run 30688981182 skipped the check at 07:31 and lost the openldap_custom build at 07:44 on the one node whose dnsmasq held no UDP socket for the advertised address.
Pass NODE_IMAGE empty in swarm mode instead: node:26-alpine is 63.3 of the 65.5 MB pulled per node, and its getaddrinfo assertion is a second opinion on a SERVFAIL check the busybox half already performs. NODE_IMAGE stays mandatory to set, so omitting it remains an error while empty is a stated decision.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible's run_once inspects the first host and generalises to the whole play, so a cluster where a single node's dnsmasq never bound the advertised resolver still reads as healthy. The property is per node, and forks=25 covers them concurrently. Also require a non-empty resolver, since 'container run --dns ""' cannot work.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replace scripts/system/diagnostics/rescue.sh with utils/diagnostics/
container.py: one workflow call on failure captures the current level
(containers with logs+inspect, swarm services, journal, host resources,
in-play role dumps) and recurses itself through every DiD boundary via
self-copy, from the runner down to the innermost runtime (RESCUE_MAX_DEPTH).
compose and swarm now share an identical failure step; the swarm per-node
collector wrapper is gone. All 17 script-only Ansible rescue: blocks are
deleted and their bare block: wrappers inlined (plus a wordpress leftover);
the three justified blocks (postgres pg_hba, xwiki extensions, keycloak
cross-node recovery) keep their evidence tasks and now end in a fail task.
Two lints enforce the state: rescue: is forbidden without a
'# nocheck: rescue-<reason>' first-task marker, and a block without
rescue/always/when or another scoping key must be inlined.
sys-service-terminator hoists become to the include apply. Also move
routine/08_cleanup.sh to utils/clean/teardown.sh and topology/ to
utils/topology/ so routine/ holds only the sequenced flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every Ansible rescue: block used to carry ad-hoc per-role failure dumps that
drifted apart and captured different subsets. Replace them all with one
diagnostics script (scripts/system/diagnostics/rescue.sh) that snapshots
containers (full logs plus inspect per container), swarm services (ps plus
logs per service), journal, host resources, runtime info and stats into a
per-run folder and always exits 1 so the deploy still fails. A new lint
enforces the shape: every rescue ends with the script, at most one
nocheck-justified extra task before it (postgres pg_hba dump, xwiki
extension-repo probe), and a nocheck rescue-recovery exemption for the one
block that must genuinely recover: keycloak object updates, where a
cross-node reschedule needs a swarm-wide node re-resolve plus a delegate_to
change that only a rescue can express; its recovery is kept and now also
captures diagnostics before the terminal failure.
The snapshot folder path is a group_vars SPOT: DIR_RESCUE_DIAGNOSTICS and
DIR_VAR_LIB live in group_vars/all/05_paths.yml as plain strings; new env
handlers derive INFINITO_RESCUE_DIAGNOSTICS_DIR and INFINITO_DIR_VAR_LIB
from it for the generated .env, replacing the default.env entries, and the
play-level environment forwards the value to every task host so rescue.sh
needs no second default. CI uploads the folder like the inventory and
playwright artifacts: compose copies it out of the deploy container in
dedicated.sh, swarm captures per node (manager and both workers) into
node-named subfolders.
Deleting the ad-hoc dumps orphaned helper files (postgres inspect_container,
terminator docker_diagnostics, peertube diagnostics, matomo bootstrap debug,
xwiki and openresty rescue includes) and 14 role vars; all removed. Generic
pieces of those dumps (runtime version and info, stats, dmesg oom evidence,
swapon) moved into the SPOT script instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`lookup('image', '<role>', '<svc>', 'ref')` only consulted
`images_overrides.*` plus role defaults, which the mirror pipeline
never populates — the pipeline writes into
`applications.<role>.services.<svc>.{image,version}`. As a result,
`sys-svc-container`'s `BUSYBOX_IMAGE` / `NODE_IMAGE`,
`sys-ctl-hlth-csp`'s `HEALTH_CSP_IMAGE`, and `test-e2e-playwright`'s
`TEST_E2E_PLAYWRIGHT_IMAGE` always resolved to the raw upstream ref
and hit Docker Hub directly. In CI this surfaced as
`toomanyrequests: ...unauthenticated pull rate limit` on
`busybox:1.37`.
Switches every productive `lookup('image', ...)` to
`lookup('config', '<role>', 'services.<svc>.image|version')` — the
same path templates like `web-app-oauth2-proxy` already use, which
resolves through the merged `applications.*` map and so transparently
picks up the GHCR mirror override written by the inventory creator.
Removes the now-unused `plugins/lookup/image.py` plugin and its unit
test (zero remaining consumers across roles, scripts, and tests).
Rewrites the image-origin / image-mirror / dockerfile docs to teach
the `lookup('config', ...)` pattern, dropping references to the
deleted plugin and the dead `images_overrides.*` variable. Doc edits
follow the `documentation.md` rules (no em dashes, one sentence per
line, no historical phrasing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Renames the container-wrapper CA toggle from the negated INFINITO_CONTAINER_NO_CA
to the positive INFINITO_CONTAINER_CA_ENABLED, mirroring the BUILDKIT_ENABLED
pattern: SPOT in env/default.env (default 1), passthrough handler + .env baseline
test, compose.yml env block, the CA_CONTAINER_ENABLED ansible var in
group_vars/all/02_tls.yml, a playbook-level CA_CONTAINER_ENABLED task-env
projection, and a strict `== "1"` read in container.py. Also drops the
redundant INFINITO_ prefix from the DNS/DOMAIN/IP4 vars that the DinD-DNS
test-dns.sh handler injects, since those names are local to the script's
caller-controlled env block. Finally lets pull.py fall back to a plain
`compose build` when `compose build --pull` fails — BuildKit can then use the
local FROM image instead of requiring a registry round-trip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Deploy v6 fails at the `Ensure Docker-In-Docker DNS works` handler
because test-dns.sh still asserts legacy short names DNS_IP / DOMAIN
/ IP4 -- the INFINITO_* namespace refactor (a77a57040) only covered
the .env / compose.yml / Python-CLI layers, not this Ansible role
script. The handler passes only BUSYBOX_IMAGE + NODE_IMAGE, so the
strict asserts in test-dns.sh fail with `Missing env DNS_IP` after
7 retries.
Two coordinated changes:
1. test-dns.sh asserts and consumes INFINITO_DNS_IP / INFINITO_DOMAIN /
INFINITO_IP4 throughout (all 9 references migrated).
2. handlers/main.yml's `Ensure Docker-In-Docker DNS works` adds the
three INFINITO_* env entries sourced from the playbook variables
(networks.internet.dns/ip4 + DOMAIN_PRIMARY).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `image` lookup inferred `role_id` from `role_name` when only
`service_name` was passed. Inference broke lazy re-evaluation: a default
like `HEALTH_CSP_IMAGE: "{{ lookup('image', 'csp-checker', 'ref') }}"`
defined in sys-ctl-hlth-csp got re-evaluated inside sys-service's
systemctl.service.j2 template (via `system_service_tpl_exec_start`),
where `role_name='sys-service'`. The lookup then searched
`roles/sys-service/meta/services.yml` for `csp-checker` and aborted with
`no image mapping found for role 'sys-service' service 'csp-checker'`.
CI run 25976825584 surfaced this on six apps (web-app-{bridgy-fed,chess,
mig,hugo,mini-qr,postmarks}) whose deploy reached sys-svc-webserver-core
outside the service-loader preload context, which is what actually
triggers the sys-ctl-hlth-csp -> sys-service include chain.
Make `role_id` mandatory in the plugin signature, remove `_infer_role_id`,
and update the three callsites that relied on inference
(sys-ctl-hlth-csp, sys-svc-container, test-e2e-playwright) to pass the
owning role id explicitly. A migration guard catches the removed
`(service_name, want)` form and emits a clear error instead of the
confusing "missing services.yml" file-not-found message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switch SYS_SVC_DOCKER_DNS_TEST_BUSYBOX_IMAGE / _NODE_IMAGE to the
role-local `images:` pattern. Consumers now resolve through
`lookup('image', <service>, 'ref')`, which picks up mirrored GHCR
paths from `images_overrides` automatically. Drops the hardcoded
rate-limit workaround from inventories/dev.yml.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Extract repeated systemctl/journalctl diagnostic blocks from
sys-service, sys-service-terminator, sys-svc-container, and
sys-svc-sshd handlers into a reusable files/debug.sh SPOT
- Add tasks/utils/debug.yml as the canonical Ansible wrapper
- Wrap sys-svc-sshd restart handler in block/rescue to emit
diagnostics and sshd -t config validation on failure
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace direct docker calls with container wrapper across scripts, roles, and templates
- Add support for cp, start, stop, and restart passthrough commands
- Align handlers and notifications to container manager restart semantics
- Fix systemd ExecStartPost to use container wrapper without hardcoded paths
- Silence ShellCheck SC2016 false positive for container exec password injection
- Update tests and documentation strings to reflect container-based workflow
https://chatgpt.com/share/69866958-c88c-800f-9b4f-3a972615bf42
