834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
89fdcbda4 gated the cid retry on `retries: 0` in compose, which Ansible
folds into a single attempt (retries = 1 + max(0, n)) and then skips the
`until` entirely -- the guard was dead code, and the swarm branch
inherited the 600x2s converge budget, so one stuck resolve could burn 20
minutes per call site against a 340-minute sweep.
Use a fixed 30x2s on both queries, sized to the observed gap between task
assignment and a running container (18:47:13 -> 18:47:18 in CI run
32119014832), and short-circuit compose in the `until` itself, where the
command is a plain echo of the id the caller passed in.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The converged check and the node query are two separate service-ps
snapshots; a task killed between them leaves the replacement unassigned,
.Node comes back empty and the empty delegate_to kills the play (CI run
32119014832, swarm+tor Checkmk#2). The cid lookup has the same window
until the replacement container reaches status=running.
Retry both until the query returns a value, on the converge budget the
callers already pass in.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A paused update never completes on its own: the updater returns early on that
state, before the branch that would finish it, and refuses to start again while
an update status is set. The restart policy brings the replicas back but does
not touch the status, so the service can sit fully replicated while the gate
still refuses it.
The probe polled that for six hundred attempts and reported a timeout after
twenty minutes. It now separates a state worth another poll from one that is
final, and the wait distinguishes the two exit codes instead of retrying every
refusal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wait_converged checked only the first task line, so a rolling update with a still-running old task reported false convergence. The probe now lives in files/swarm/service_converged.sh (stack_ready.sh style: script module, SERVICE env, timeout 15 per engine call, container wrapper): it requires no update in flight (UpdateStatus completed/rollback_completed, or absent) and every desired-state=running task to be Running. Covered by a seven-scenario stub test during extraction; a never-updated service maps the engine's nil-template error to the no-update case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace scripts/system/diagnostics/rescue.sh with utils/diagnostics/
container.py: one workflow call on failure captures the current level
(containers with logs+inspect, swarm services, journal, host resources,
in-play role dumps) and recurses itself through every DiD boundary via
self-copy, from the runner down to the innermost runtime (RESCUE_MAX_DEPTH).
compose and swarm now share an identical failure step; the swarm per-node
collector wrapper is gone. All 17 script-only Ansible rescue: blocks are
deleted and their bare block: wrappers inlined (plus a wordpress leftover);
the three justified blocks (postgres pg_hba, xwiki extensions, keycloak
cross-node recovery) keep their evidence tasks and now end in a fail task.
Two lints enforce the state: rescue: is forbidden without a
'# nocheck: rescue-<reason>' first-task marker, and a block without
rescue/always/when or another scoping key must be inlined.
sys-service-terminator hoists become to the include apply. Also move
routine/08_cleanup.sh to utils/clean/teardown.sh and topology/ to
utils/topology/ so routine/ holds only the sequenced flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every Ansible rescue: block used to carry ad-hoc per-role failure dumps that
drifted apart and captured different subsets. Replace them all with one
diagnostics script (scripts/system/diagnostics/rescue.sh) that snapshots
containers (full logs plus inspect per container), swarm services (ps plus
logs per service), journal, host resources, runtime info and stats into a
per-run folder and always exits 1 so the deploy still fails. A new lint
enforces the shape: every rescue ends with the script, at most one
nocheck-justified extra task before it (postgres pg_hba dump, xwiki
extension-repo probe), and a nocheck rescue-recovery exemption for the one
block that must genuinely recover: keycloak object updates, where a
cross-node reschedule needs a swarm-wide node re-resolve plus a delegate_to
change that only a rescue can express; its recovery is kept and now also
captures diagnostics before the terminal failure.
The snapshot folder path is a group_vars SPOT: DIR_RESCUE_DIAGNOSTICS and
DIR_VAR_LIB live in group_vars/all/05_paths.yml as plain strings; new env
handlers derive INFINITO_RESCUE_DIAGNOSTICS_DIR and INFINITO_DIR_VAR_LIB
from it for the generated .env, replacing the default.env entries, and the
play-level environment forwards the value to every task host so rescue.sh
needs no second default. CI uploads the folder like the inventory and
playwright artifacts: compose copies it out of the deploy container in
dedicated.sh, swarm captures per node (manager and both workers) into
node-named subfolders.
Deleting the ad-hoc dumps orphaned helper files (postgres inspect_container,
terminator docker_diagnostics, peertube diagnostics, matomo bootstrap debug,
xwiki and openresty rescue includes) and 14 role vars; all removed. Generic
pieces of those dumps (runtime version and info, stats, dmesg oom evidence,
swapon) moved into the SPOT script instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
resolve_host_cid/wait_converged/guard_instance_dir took a run_host parameter every caller had to pass; STACK_HOST already encodes the correct host in both modes (first swarm manager, degrading to inventory_hostname in compose), so the manager-only queries now delegate to it directly and the parameter plus all caller declarations are gone. Also repairs the swarm guard include: apply/delegate_to are invalid on a HandlerTaskInclude and the previous commit carried duplicate run_once/delegate_to keys, which failed the ansible syntax-check. The keycloak handler drops its masking default; the 02b conditional collapses into the SPOT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The converge-failure diagnostics truncated service logs to --tail 50 (and head -100), but a crashing service prints its exception at the TOP of the trace, so the root cause was cut off and non-converging services could not be debugged from CI. Remove the caps in the converge-failure rescue (wait_converged.yml) and the diagnostics collector (12_collect_diagnostics.sh) so the full log, including the exception, is captured.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In swarm an app task can be scheduled on a worker, but the role exec'd
`container exec $(resolve-container-id ...)` on the manager -> "No such
container". Resolve the hosting node + node-local container id and delegate
each exec to that node.
- database.py: add a `service_name` want; resolve_host_cid/wait_converged
gain `*_service_name` for cross-stack services (mailu's shared mariadb)
- mailu wait-for-mariadb (DB probe + admin), mastodon db:migrate,
gitea migrate/admin-create, friendica manager-ops/patch/addons/bootstrap
(~13 exec sites) switch to resolve_host_cid + delegate_to
Compose-safe: resolve_host_cid degrades to HOST_NODE=inventory_hostname
(delegate is a no-op) + LOCAL_CID=bare compose container name, matching the
8 roles already using this pattern.
Note: friendica also needs service_replicas=1 (next bundle) so its resolved
container id does not churn across replicas.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce plugins/lookup/path_absolute.py as the single source for the
repo-root anchor: lookup('path_absolute', 'roles/...') returns an absolute
path under playbook_dir, joining every term. Replace all hand-built
playbook_dir path constructions across .yml/.yaml and .j2 templates with it:
[playbook_dir, '<rel>'] | path_join -> lookup('path_absolute', '<rel>')
playbook_dir ~ '/<rel>' -> lookup('path_absolute', '<rel>')
{{ playbook_dir }}/<rel> -> lookup('path_absolute', '<rel>')
Unrelated repo_path symbols in cli/meta/roles/applications/resolution and the
single bare discover_playwright_roles(playbook_dir) call are left untouched.
Add tests/lint/ansible/test_playbook_dir_via_lookup.py: fails the build if any
of the three construction forms reappears in .yml/.yaml/.j2, pointing callers
at the lookup. Unit test covers the plugin.
Touching these files pulled their pre-existing narration comments into
test_comments_valid scope; clean them (delete restatement/banners, keep real
trip-wires as Workaround/Upstream/Invariant markers). Drop caution, important
and pitfall from the comment-lint marker vocabulary and remove the comments
that used them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
wait_converged failed after 600 silent retries (rc=1, empty) with no clue why. Wrap the wait in a rescue that, on the manager, dumps 'container service ps --no-trunc' + 'container service logs --tail 50' and aborts with a root-cause message, so converge failures (e.g. collabora's coolmount permission error) surface immediately instead of after 20 minutes of silence. Swarm-gated; the success path is unchanged and compose skips the block entirely.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the per-role swarm host-node/container-id resolve duplicated across fider, matomo, decidim, keycloak (login + realm-import), collabora and listmonk with two shared task utils under roles/sys-svc-compose/tasks/utils/swarm/. resolve_host_cid sets <prefix>_HOST_NODE / <prefix>_LOCAL_CID (swarm: converge + node + cid delegated to the node; compose: inventory host + given cid); wait_converged is the bare convergence gate.
Teach test_variable_definitions to read templated set_fact keys like {{ prefix }}_HOST_NODE as a *_HOST_NODE name pattern (guarded so a pure {{ x }} key cannot become a catch-all), and add the playbook_dir-relative include load-failure exemptions to .ansible-lint-ignore.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DEPLOYMENT_MODE is now derived per role from group membership:
swarm iff len(groups[application_id]) > 1, else compose. The cluster
infrastructure flag SWARM_CLUSTER_AVAILABLE is true whenever any host
sits in svc-docker-swarm, so DB and proxy services render attachable
overlay networks even when their own group lists only one host.
All service_is_stateful flags are removed across svc-db-mariadb,
svc-db-postgres, svc-db-openldap, svc-prx-openresty, svc-docker-registry
and svc-db-mariadb's compose template; the inventory shape now expresses
the same intent. Cluster-wide handlers (swarm build/push/validate/deploy,
external network creation, DB healthcheck) gain run_once + delegate_to
the manager so they execute exactly once regardless of how many hosts
the role's play targets. CA-injection and compose validate are gated to
compose-mode only; their compose-config strictness breaks when the
deploy block coexists with container_name under swarm.
Dead per-host swarm-resolve infrastructure is removed: swarm_resolve_
container.sh, resolve_container.yml, resolve_self.yml, the MariaDB-
specific swarm_mariadb_cid.sh, the cross-host template replication
step, and the _swarm_retries / _swarm_delay defaults. With run_once +
delegate_to: manager replacing the old "render every template on every
swarm peer" pattern, every consumer becomes a noop on workers and
collapses to the manager's one-shot path.
The web-app-nextcloud memcache override mapping flips from the legacy
"single" sentinel to "compose" (semantically equivalent under the new
model). Requirement 023 is updated to reflect the new model and drops
the now-irrelevant per-role opt-out / placement-constraint clauses.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the host-scoped set_fact resolvers in sys-svc-compose/tasks/01_core.yml
and tasks/stages/01_constructor.yml with a single group_vars definition driven
by group_names. Uppercase the variable to match the SPOT convention and drop
every `| default('compose')` fallback now that the resolution is unconditional.
Role-var opt-out on a swarm host still works via Ansible's role-vars > group_vars
precedence.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three coupled fixes for the Controller≠Target topology the swarm pilot
exposed in iter12, iter15, and iter17/iter18:
1. `swarm_resolve_container.sh` now locates task containers via the
swarm state API (`docker service ps`, then `docker inspect TASK
--format '{{.Status.ContainerStatus.ContainerID}}'`) instead of a
local `docker ps --filter label=...` on the manager. A replica that
landed on a worker stayed invisible to the manager-side filter and
the role-level wait spun out after 60 retries (iter17 / iter18).
The state check accepts both `running` and `starting` so the
resolve unblocks before the app's healthcheck has had to pass —
first-deploy apps like mediawiki need their install_or_update.sh
step BEFORE they can become healthy.
2. `resolve_container.yml` invokes the script via an absolute
`{{ playbook_dir }}/roles/sys-svc-compose/files/...` path. When the
include_tasks was pulled in from another role (e.g. svc-db-mariadb's
02_init.yml), ansible's `ansible.builtin.script` file-lookup
searched the calling role's `files/` directory and aborted with
"Could not find or access 'swarm_resolve_container.sh'" (iter15).
3. `post_deploy.yml` gates the swarm task resolve on
`not service_is_stateful`. Stateful services (svc-docker-registry,
svc-db-mariadb) deploy as plain compose containers on the manager,
not as replicated swarm services — there is no `STACK_SERVICE` task
to resolve, so the role spun out for 60 retries waiting for one
that would never exist (iter12).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the svc-docker-swarm, svc-docker-swarm-manager, svc-storage-nfs-server
and svc-storage-nfs-client roles, the compose_volumes filter NFS support,
the test-deploy-swarm-nfs CI workflow with its swarm-nfs-pilot script set,
and the deploy_mode-aware compose/swarm wiring across mediawiki, mariadb
and sys-svc-* helpers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
