A deploy that renders a changed compose.yml, .env or compose override
notifies the compose-up handler. When the run fails before that handler
completes, for example because the image pull hits Docker Hub's
toomanyrequests, the notification is lost. The next run renders the same
files (changed=false), and the fallback in tasks/utils/up.yml notifies
compose-up only when no container of the stack runs, so the stack keeps
its previous images and configuration while the deploy ends with
failed=0 (reproduced with web-app-akaunting 3.1.8 -> 3.1.21).
04_files.yml now writes `.compose-up-pending` into the instance
directory whenever one of those three files changed in compose mode. The
compose-up handler removes it right after `compose up` succeeds, so a
failed pull or up leaves it in place, and up.yml notifies compose-up
whenever it finds the marker on a later run.
Comparing `compose config --hash` with the containers' config-hash
labels was ruled out: on an up-to-date stack the two differ for every
service, so it would force-recreate on every deploy.
Fixes#1169
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
In swarm mode `docker stack deploy` ignores the host IP of a short port
mapping, so every `127.0.0.1:<port>:<port>` that container_ports renders for
a `ports.local` entry is published by the ingress routing mesh on every
node, via docker-proxy on 0.0.0.0 and [::] plus DNAT in nat PREROUTING.
Databases, LDAP and application backends answered from outside the cluster.
svc-swarm-node now loads an nftables table through svc-net-firewall. Its
prerouting filter chain (priority mangle, ahead of Docker's dstnat) drops
packets addressed to the node for any ingress-published port that no
application declares under `ports.public`. Loopback, docker0,
docker_gwbridge, the user bridges and the swarm nodes' own addresses stay
admitted. Host-side provisioning over 127.0.0.1, containers that reach a
host port through host-gateway, and node-to-node consumers such as the
registry-cache mirror keep working.
Both sets are read live on the manager (`docker service inspect` for
`.Endpoint.Ports`, `docker node inspect` for `.Status.Addr`), so ephemeral
ingress ports and nodes outside `--limit` are covered. The table is loaded
on a node before it joins, reloaded on every node after the join, and
reloaded after each stack deploy by the sys-svc-compose swarm handler.
Refs #1089
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cache-less rebuild is the fallback for a build whose spaced retries
are exhausted. It ran once, so one more registry timeout turned a
transient network failure into a failed deploy. Both the compose and the
swarm handler now retry it three times, 30 seconds apart, like the build
they back up.
Closes#620
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
docker stack deploy --detach=false converges on a running replica. web-svc-cdn's build container and every migration job end Shutdown/Complete by design, so the CLI waited for a state those stacks cannot reach: nine jobs of run 33698844864 burned the runner's 20400s budget inside this one handler, six of them on web-svc-cdn. The gate that follows already knows better, is_completed_oneshot accepts exactly that task shape.
The timeout keyword could not bound it either. The stack host is reached over ansible_connection: docker, whose plugin runs the child inside a Popen context manager, and __exit__ waits the child out before the timeout exception escapes; a probe measured 90s of wall clock against a 6s timeout. The bound now sits in the shell, where it works.
Handing the wait back to the gate would reopen the race 87e4313e11 closed, so the deploy stamps its start and names the services it reported touching, and the gate treats their earlier completed update as still pending. An inspect format the daemon rejects falls back to the old one, which only disables that check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
87e4313e1 gave the blocking stack deploy a timeout computed straight from
swarm_stack_converge_retries. That bypasses lookup('timeout', <base>), which is
how every wait in this repo scales from one TIMEOUT_FACTOR, and
test_timeout_uses_lookup enforces it. The commit went out having run only
tests/lint/ansible, and the rule lives under tests/lint/repository/yaml.
At the shipped factor of 1 the value is unchanged at 1200s, so this restores the
scaling without moving the number.
It also corrects a claim that commit made. The two budgets do not hang on one
knob: the gate cannot scale, because test_swarm_converge_budget multiplies its
retries by a literal delay to prove the worst case fits
INFINITO_CI_DISTRO_BUDGET_SECONDS, and a templated delay would end that proof.
That rule exists because the two budgets drifted once already and the shorter
one killed a healthy but slow stack. The gate's budget is deliberately static
and provable; the deploy's timeout scales with the environment, and the deploy
is the authoritative wait.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker stack deploy defaults to --detach, so it returns before the daemon has
registered the rolling update. In that window a converged stack and one about to
be replaced look identical: UpdateStatus still reads completed from the previous
round, the outgoing task is still desired-state running and Running, and the
replica column still reads 1/1. The gate answers correctly and answers the wrong
question.
Moodle in run 33574056480 shows the whole shape. The stack deploy reported
changed at 02:26:30.797, the convergence gate went green 1.14s later, cfg.php
failed at 02:26:36 with
Access denied for user 'moodle'@'192.168.201.4'
and the task carrying the new password did not start until 02:27:15, 38 seconds
after the failure. There was one task at the moment of the failure and it was the
old one, so this is not a resolver picking the wrong task of two.
--detach=false hands the waiting to the daemon, which is the only party that
knows an update has been created. The task after it starts once the update is
through, and the window is gone.
This sits next to 4b9e191200 rather than replacing it: that one stops an outgoing
task from satisfying the replica count while an update is in flight. This one
covers the moment before the update exists at all.
The deploy timeout derives from swarm_stack_converge_retries, the same knob the
gate polls on, so the two budgets cannot drift apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
roles/sys-svc-mail-msmtp/tasks/ca_bundle.yml stat'ed candidate bundle paths on
the Ansible target and wrote the winner into msmtprc. Moodle and WordSpress then
copied that file into a container, where the named path is a different file - or
none. Only update-ca-certificates populates a container's system bundle, and
that runs solely from the with-ca-trust.sh entrypoint, which compose_ca.py sets
only when 'docker image inspect' succeeded. On a fresh host the image is built
minutes later, so it never ran: job 94965457602 of run 31864448027 shows
Config.Entrypoint still moodle-entrypoint.sh while the CA sits mounted beside it,
and msmtp reporting 'The certificate issuer is unknown'.
An explicit tls_trust_file also outranks SSL_CERT_FILE, so the env injection the
override always performs could not compensate.
Name CA_TRUST.inject_cert_container when the CA is actually injected, and emit
no line at all otherwise - msmtp's own 'system' default is the right answer for
a letsencrypt relay, and it is what the container's stock bundle verifies. This
also removes a latent defect on letsencrypt: a RedHat host would have written
/etc/pki/... into a Debian container.
The removal of the probing apparatus is taken from 9a72d01eb on
fork/feature/ai-agent-employees, which diagnosed the wrong-machine measurement
correctly. Its own conclusion - drop the line entirely and rely on the trust
store - does not hold where the wrapper never runs, and in swarm it never does:
handlers/swarm.yml passes wrapper=False by design.
The three-term predicate behind all of this was hand-copied into four places.
lookup('ca_injected', application_id) is now its single source; the swarm and
compose gates and sys-svc-compose-ca's own var read it too.
compose_ca.py finally says when it degrades to mount-plus-env, naming the service
and the image it could not inspect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A handler name was doing two jobs at once: the line an operator reads in the
play output, and the identifier a `notify:` binds to. The two pull in opposite
directions -- the log wants "restart the SSH service, with diagnostics", the
binding wants a token that survives quoting, YAML flow style and a shell
round-trip -- and a name that changes for readability silently changes which
handler runs.
Every handler that a notify reached by name now carries a `listen:` with that
exact wording, so the binding is explicit before anything is renamed. The
topics themselves become identifiers: `compose up` reads `compose-up`,
`Restart NFS ganesha` reads `restart-nfs-ganesha`, 22 of them across both
sides of every notify. Where the topic is assembled at runtime, both branches
moved together: the conditional in sys-service carries two literals, and the
LDAP renderer interpolates `import-{{ folder }}-ldif-files`, which is what
makes those handlers reachable at all.
Freed from the binding, the names take the emoji prefix every other task line
already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
swarmkit's default failure_action is pause, and the updater returns early
on a paused state without ever reaching its completion branch, so a
service that latches never leaves that state on its own. The gate every
role runs read replica counts and nothing else, so a latched update looked
exactly like a slow one and burned the full retry budget before failing
without a cause. That is what turned one corrupted openclaw task into a
25-minute silent timeout.
Read UpdateStatus before comparing replicas. The order carries the fix: a
rolled-back service reports N/N of the PREVIOUS spec, so a replica check
alone reports it as converged. Guard the format against a service with no
update in flight, since a bare .UpdateStatus.State nil-pointers and would
fail the whole multi-service inspect.
The handler has to honour the give-up or the gate keeps polling. It stops
only on the explicit give-up and still retries every other exit code, so a
timeout returning 124 does not turn into a failed deploy. That deviates
from wait_converged.yml, which stops on anything but 1; the deviation is
deliberate.
Stop treating rollback_completed as success in the per-service probe. It
is inert today because nothing here issues a rollback, and it is the guard
that keeps a silent revert from returning as a green deploy.
failure_action is deliberately left at the default. Rolling back would
revert a failed image bump while reporting success, and a daily cron opens
those bumps; it would also place the old binary against a schema ~28 roles
migrate on start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two callers polled the same stack_ready.sh with different budgets: the
handler read swarm_stack_converge_retries while post_deploy hardcoded
150. Eleven roles had set the override and were unprotected at the
second caller anyway. Both now read one SPOT in group_vars.
The gate also had no way to give up. A crash-looping espocrm burned 600
polls (20 min) before failing, while flowise was killed 45 s short of a
409 s image pull. has_fatal_task now exits 2 - and both callers stop at
once via 'until: rc in [0, 2]' - when a slot's newest attempt is
Rejected or Pending, or when a slot burned three attempts after the
service churned past swarm_stack_fatal_grace. Patience only grows where
nothing is broken.
report_tasks loses its 'awk !/error=$/' filter, which printed only rows
that HAVE an error and so stayed silent in exactly the no-error case
that occurred.
Measured on a live single-node swarm: a service failing 3x and
converging on the 4th returned rc 2 at t+15s before this change and rc 0
at t+32s after; a 3-replica stop-first rolling update gave 68/68 samples
rc 0; a permanent crash loop still reaches rc 2. The deploy epoch comes
from 'docker service inspect .UpdatedAt.Unix' rather than a /tmp marker,
which was reproduced contaminating a later deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`files/swarm/stack_ready.sh` has exactly two callers and they disagreed: the
handler defaulted to 150 polls at delay 2, and post_deploy.yml hardcoded the
same 150 with no way to override it. 389 seconds is not enough for a stack that
carries a large image.
Four jobs across two runs died on it, every one starting at "149 retries left":
Nextcloud 2 (run 31340965955) plus Workspace centos, debian and manjaro (run
31311996621), the last three all waiting on svc-db-mariadb, a role that carries
no per-role override. Nextcloud 2 is the measured case — the budget expired at
12:59:54 and the task reached Running at ~13:00:32, 38 seconds later. Nothing
was broken; the deadline was.
600 is not a new number: tasks/utils/swarm/wait_converged.yml already waits that
long at the identical delay, and eleven roles had each hand-raised the handler to
exactly 600. 150 was the outlier.
The declaration goes in group_vars/all, not in the role defaults. A role default
is invisible to the by-path include in roles/svc-db-openldap/tasks/01_docker.yml,
which pulls post_deploy.yml in after include_role has finished — verified with a
scratch play, where the same variable resolves inside the role and raises
"is undefined" one task later. post_deploy.yml already encodes that distinction:
it reads DEPLOYMENT_MODE bare and guards compose_mode_force with `| default('')`.
The lint gains a check that the gate's callers never drift apart again. It
discovers them by scanning for the script rather than listing them, so a third
caller is covered, and it asserts it found at least two so an empty scan cannot
pass silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible's `timeout:` keyword is a local signal.alarm in the worker. Over the
docker connection community.docker runs the child inside `with Popen(...)`,
whose `__exit__` ends in an unbounded `self.wait()` — the TaskTimeoutError is
raised but cannot leave the with-block until the remote child exits. A hung
`compose build` therefore ran to the 3h59m job limit instead of the declared
budget. Reproduced locally: with-Popen 20.00s vs bare Popen 2.00s for the same
2s alarm.
Wrap both build bodies in GNU timeout so the bound is enforced remotely, where
the child actually lives. `-k 60` guarantees SIGKILL if the build ignores
SIGTERM, which is the precondition for `wait()` to return at all; measured
client response to SIGTERM is 0.12s, so the grace window is pure reserve. Same
flag pairing as scripts/tests/deploy/distros.sh:128.
The `timeout:` keywords stay — the retry-budget lint reads task["timeout"].
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three of the failures triaged today were a single unretried external fetch:
deb.debian.org went dark for 76 seconds on one node while three other distro
rounds of the same job built the same image; packagist answered 502 for two
metadata files while the identical build passed thirteen minutes later; and
ghcr.io timed out on a manifest HEAD for six minutes.
Both build tasks were the only network-bound tasks in their file without a
retry, and their in-shell fallback made a transient fault worse: it re-ran
immediately, with no wait, and threw away the cache it had just filled. The
ladder now waits 30 seconds between three attempts, and the cache-discarding
rebuild moved out into its own task that runs only once those are exhausted -
at which point the transient explanation is spent and a corrupt cache is the
next candidate. Three attempts rather than ten because ten would run 36270s
against a 21000s step cap.
svc-registry-docker is excluded from the registry sync - it cannot mirror
itself into its own registry - which left it the one role fetching from a public
registry through the swarm agent, single-shot, where every other role gets ten
tries. It now pre-pulls through a retried task, gated on the sync task having
been skipped so the exclusion keeps one source.
The accompanying check computes the ladder's worst-case wall clock from the
handlers and measures it against default.env and the workflow step cap rather
than restating the numbers, and it fails if an ignore_errors ever loses its
paired fallback - which would turn a failed build into a silent pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
compose_networks renders a foreign provider network as external: true, leaving open who creates it. The swarm handler has answered that since it was written, looping compose_external_networks through the network create task so a shared provider overlay exists even when the provider role did not run in the same play. Compose had no counterpart: it only worked while consumer and provider deployed together, because then the provider role created the network itself.
Deploying roles one at a time, as the roundtrip does, breaks that assumption and compose up aborts with "network <provider> declared as external, but could not be found". The compose handler now mirrors the swarm block: same lookup, same create task, triggered before the stack comes up. No cluster gate, since that condition is swarm-only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The new handler had neither retries nor an assertion. update_config carries no failure_action, so Docker's default is pause: one flapping healthcheck inside the monitor window pauses the rolling update, docker service update exits non-zero, pipefail propagates it and any_errors_fatal kills the whole deploy - where the neighbouring converge handler would have retried. Give it five attempts fifteen seconds apart, matching how the registry-sync handler beside it already treats a transient action.
The registration probe only notified; nothing checked that the recreation actually registered the plugin. A plugin that can never register - corrupt pnpm tree, engine mismatch, invalid library code - therefore reported changed on every run, recreated the stack every time and still left the play green. Flush the recreation, re-resolve the container id the recreation invalidated, and assert the registration with a bounded wait, so that case fails loudly in the run where it happens instead of churning forever in silence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker stack deploy leaves tasks running when the service spec is unchanged, so a change made inside a container never reaches the process. PeerTube's CLI installs plugins with register:false, so the OIDC plugin sat installed and enabled in the database while the running server had loaded nothing: /login rendered only the local form and both OIDC specs timed out through all three attempts. Compose never showed it because its handler force-recreates.
Add a container recreate notify. In swarm it forces a task recreation per stack service and waits for convergence; in compose it maps onto the existing --force-recreate, so a role states the intent once and both modes honour it.
In the peertube role the install probe now tests node_modules/<npm-name>, where pnpm puts the plugin code; the old path was the data directory that registerPlugin creates at boot, so it never existed and the plugin was reinstalled on every run. The settings upsert derives changed from the rowcount via RETURNING, because database_query reports changed from the kind of statement rather than from what it touched. A registration probe notifies when the running process has not loaded the plugin, so an already-broken instance heals itself instead of walking past both earlier probes. The plugin:enable task is gone: npm has no such script, the call always failed and failed_when swallowed it.
The spec now polls the button lookup instead of counting once twenty milliseconds after goto, while still asserting on the URL so a failure keeps printing the address the browser actually sat on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolve every 3+-file same-prefix group flagged by the new prefix lint
by moving the files into a subfolder named after the shared prefix and
dropping the prefix from the file names, updating all references:
- utils/env/handlers/infinito_*.py -> infinito/ (dir/, package_cache/
nested a second level); registry imports and the four handler-dir
globs switched to iter_project_files
- utils/install/lint/ansible_*.py -> ansible/ (aliased imports)
- roles/sys-svc-compose/files/swarm_*.{py,sh} -> files/swarm/
- web-app-erpnext/web-app-zammad files/scripts/apply_*.{py,rb} -> apply/
- web-app-odoo files/sql/{select,update}-* and sql/oidc/{grant,insert,
select,update}-* -> matching subfolders
- web-app-keycloak tasks/update/05_ldap*.yml -> 05_ldap/ (internal
includes made role_path-absolute)
- plugins/filter/get_*.py -> plugins/filter/get/ (filter names are the
filters() dict keys, so Jinja usage is unaffected; ansible discovers
the one-level-deep subdir)
Ansible lookup dirs, addon dirs and .github/workflows keep their flat
layout (the filename is the public id / GitHub only discovers flat).
Unit tests mirror the new source layout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The controller-side template cannot read file-backed secret sources
that only exist on the stack host, so the content-hash suffix silently
fell back to hashing the path string and never rotated. A regenerated
source (e.g. a self-signed key re-minted on SAN drift between deploy
passes) then collided with the immutable docker secret of the previous
deploy and stack deploy failed with 'only updates to Labels are
allowed'. Recompute the suffix from the actual file bytes on the stack
host right before the deploy; service references stay untouched since
they point at the yaml key, not the name.
Retention follow-up (pruning superseded generations) tracked in
https://github.com/infinito-nexus/core/issues/415.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
resolve_host_cid/wait_converged/guard_instance_dir took a run_host parameter every caller had to pass; STACK_HOST already encodes the correct host in both modes (first swarm manager, degrading to inventory_hostname in compose), so the manager-only queries now delegate to it directly and the parameter plus all caller declarations are gone. Also repairs the swarm guard include: apply/delegate_to are invalid on a HandlerTaskInclude and the previous commit carried duplicate run_once/delegate_to keys, which failed the ansible syntax-check. The keycloak handler drops its masking default; the 02b conditional collapses into the SPOT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every chdir-dependent compose/swarm handler assumed the instance dir of the flush-time application_id exists; a drifted notification then died with a cryptic 'Unable to change directory'. Guard the flush with a shared stat+fail (tasks/utils/guard_instance_dir.yml, included first in both handler files) that names the app, the missing dir and the drift cause, and instructs to fix the notifying role.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a 'timeout' lookup plugin that returns round(base * TIMEOUT_FACTOR)
(group_vars default 1, so value-identical to the old literals). One knob
in group_vars stretches every wait in the stack for a slow uplink:
TIMEOUT_FACTOR: 2 doubles them all.
- plugins/lookup/timeout.py + unit tests (identity, factor scaling,
string coercion, error paths).
- group_vars/all/17_resource.yml: TIMEOUT_FACTOR: 1.
- Convert all 34 literal 'timeout:' task keywords / module args across 25
task+handler files to "{{ lookup('timeout', <base>) | int }}".
- tests/lint/repository/yaml/test_timeout_uses_lookup.py enforces the
lookup in every task/handler timeout (opt-out: # nocheck: timeout-lookup).
- swarm.yml pre-deploy build handler: add the missing timeout (same
network-build hang vector as compose.yml's Build compose).
- Drop 13 pre-existing narration comments in the touched files (comment
lint enforces whole-file on any edited file).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A network pull with no time bound can hang indefinitely: run10 sat 6h
in 'docker compose build --pull' for bbb with no new output, and the
streaming monitor stayed silent because a hang emits no log lines.
Add timeout: 3600 to both network-pull handlers (compose pull, Build
compose) so a stuck pull self-aborts at 60min and the deploy fails
loudly instead of hanging forever.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The KIX backend runs a FRESH_INSTALL schema bootstrap that outlives the default
150x2s converge budget on loaded CI runners (db reaches 1/1 while the backend
stays 0/1 Starting), so the swarm converge gate timed out. Make the gate's
retries overridable (default unchanged at 150) and raise it to 600 for kix.
Swarm-only gate; compose is unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BBB runs the external bigbluebutton/docker project (host networking,
node-local stateful stack) and cannot be a docker stack deploy service.
Add a per-role compose_mode override so it deploys via node-local
compose up on STACK_HOST even inside a swarm cluster:
- sys-svc-compose: compose_mode_force (per-role override) + computed
compose_mode (SPOT) in defaults; up.yml + handlers route on
compose_mode instead of DEPLOYMENT_MODE.
- container_address: honour compose_mode_force so a node-local role's
exec targets its compose container (bare name), not a swarm service.
- web-app-bigbluebutton: compose_mode_force=compose, ungate compose
generation for swarm, flush handlers in-scope, and add
services.bigbluebutton.ports.internal.http for the swarm proxy upstream.
- web-app-matrix: add services.element.ports.internal.http (swarm proxy).
- test_variant_bundles: pin compose mode in the main test so it is not
polluted by another test leaking INFINITO_DEPLOY_MODE=swarm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a handler that blocks on swarm_stack_ready.sh (replica running == desired) on every swarm deploy, not only when compose_handlers_flush is set, so the app stack has converged before the health/e2e gates probe the proxy upstream. Without it sys-ctl-hlth-webserver probed the vhost while tasks were still starting, got 502, and sys-service-terminator aborted the play. Swarm-only (import-gated); stack name matches the deploy handler; no job/global services exist so replica-count parity is a valid convergence signal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In swarm, OmniAuth/OpenSSL (decidim OIDC discovery) and Node (bluesky-broker -> PDS) could not trust Keycloak's internal self-signed CA: docker stack deploy never received compose.ca.override.yml, and the override's $$-escaped entrypoint wrapper is swarm-incompatible (stack deploy does not un-escape it). Add a --no-wrapper override mode (env + CA mount only: SSL_CERT_FILE/CURL_CA_BUNDLE/NODE_EXTRA_CA_CERTS) generated on the stack host for self-signed apps, and build the stack-config/stack-deploy --compose-file list via compose_file_args (new --compose-file flag kwarg) instead of hardcoding compose.yml -- which also stops swarm silently dropping a role's own compose.override.yml. Compose untouched: self-signed-gated, no-wrapper is swarm-only, lookup defaults to -f.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DEPLOYMENT_MODE is binary (swarm / compose), so a negated `!= 'swarm'` gate
always means `== 'compose'`. Replace every negated gate across roles/ and
group_vars with the positive form for readability, and add a lint
(tests/lint/ansible/variables/test_no_deployment_mode_negation.py) that forbids
`DEPLOYMENT_MODE !=` in .yml/.yaml/.j2 with a `# nocheck:
deployment-mode-negation` escape.
The compose-verb / chdir / jinja-gate lint guidance messages now suggest the
positive form too; the gate matcher (utils/annotations/task_gate.py) and its
test inputs keep recognising the legacy `!= 'swarm'` so a nocheck-marked
exception is still detected as compose-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
compose_ca.py ran ~3 docker calls per service sequentially; on a DiD
daemon saturated by a many-service app's compose up (e.g. BigBlueButton,
16 services) the inject blew the CA-trust handler's timeout and failed
the whole deploy.
- run(): Popen + kill the whole process group on timeout so a wedged
docker/buildx grandchild cannot hold the captured pipe open and defeat
the timeout (subprocess.run's own timeout was inert in that case).
- /bin/sh probe: DEVNULL + short bound, so a wedged probe has no pipe to
block on.
- gather_image_meta(): read per-image (inspect + probe) CONCURRENTLY
(8 workers) up front; the render loop then only consults the cache.
- unreadable image (missing, or inspect stalled under load) -> env-only
injection (mount CA + set SSL_*/CURL_CA_BUNDLE env, skip the wrapper);
never build or hard-fail, so the inject always completes.
- handlers/main.yml: timeout backstop on the "Generate CA trust override"
handler.
- test_compose_ca.py: updated to the new run() signature + gather_image_meta.
Proven on the heaviest round: nextcloud variant-0 full_cycle went green
(failed=0, sync + async) with BigBlueButton + xwiki + all bonded partners.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Workers in the DiD swarm cluster cannot reach upstream registries
(Docker Hub, ghcr) from the overlay network. Image pulls hang
indefinitely in "Preparing", which used to abort with a task-converge
timeout much later in the deploy. Until now the existing
`swarm_registry_push.py` only retagged + pushed services that had a
`build:` key (locally-built images); plain upstream images were left
for each worker to pull on its own, which never works in the DiD path.
This change extends the script to also cache upstream images:
* `docker pull <image>` on the manager,
* `docker tag <image> <prefix><image>`,
* `docker push <prefix><image>` to the on-cluster registry
(`svc-registry-docker`),
* rewrite the on-disk compose.yml in place so the swarm scheduler
pulls the prefixed name from the local registry over the overlay
network.
Idempotency: a `docker manifest inspect <prefix><image>` probe runs
before each push; on re-deploys with no upstream version bump the
push is a fast no-op. Hard-fails on any non-zero docker
pull/tag/push so the standard "fail loud when upstream broken" policy
holds; no `failed_when: false`.
Chicken-and-egg guard: when the registry service itself
(`svc-registry-docker`) is being deployed, the new branch would try to
push `registry:2` to its own not-yet-running registry endpoint and
loop on `connection refused`. The handler now skips the sync step
when `application_id == 'svc-registry-docker'`. The registry's own
image is pulled directly by the manager (default_placement: manager)
and never needs to be cached into itself.
Mechanics:
* `swarm_registry_push.py` renamed to `swarm_registry_sync.py` to
reflect the broadened scope (pull + tag + push, not just push).
* `roles/sys-svc-compose/handlers/swarm.yml`: updated script
reference, updated task title, added the registry-self-skip
conjunction in the `when:` guard.
Validated via act-swarm-zombie web-app-baserow: deploy progressed
from ok=145 (aborted at the chicken-and-egg push) to ok=806 once the
guard landed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extract the mode-specific handlers from a single main.yml with per-task
`when: DEPLOYMENT_MODE ==/\!= 'swarm'` conditions into two sibling files
(compose.yml, swarm.yml) imported via `import_tasks` with the mode guard
on the wrapper. The mode condition is set once at the import site
instead of being repeated on every handler.
main.yml keeps the shared prune handler (storage-constrained, fires for
both compose-up and compose-build listens regardless of mode).
Also: the handler-invocation lint at
tests/integration/ansible/handlers/test_invoked.py now follows
import_tasks references inside handlers/main.yml so it picks up handler
definitions in sibling files. Without this, the new layout fails the
"every notify maps to a handler" check because the scanner previously
only looked at main.yml.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- roles/sys-svc-container/templates/deploy.yml.j2: render
`placement.constraints: node.role == manager` in the swarm
`deploy:` block whenever the application_id is one of the roles
whose `meta/services.yml` declares `default_placement: manager`
(resolved via the `roles_with_default_placement` lookup). Pins
state-sensitive services (mariadb, postgres, openldap, openresty,
registries) to the manager so their bind-source / local volume
expectations hold; app roles without the marker stay free to land
on workers.
- roles/sys-svc-compose/handlers/main.yml: add `retries: 10 / delay: 6
/ until: succeeded` to the `swarm: retag built images for local
registry and push` handler. On a cold deploy the local registry task
may still be settling when the push fires; retrying for up to 60s
lets the push outlast the registry's warmup instead of aborting the
whole stack deploy on a single transient connect-refused.
Adds MODE_ASSERT_LOCAL to group_vars/all/01_modes.yml:
MODE_ASSERT_LOCAL: "{{ (MODE_ASSERT | bool) and (IS_STACK_HOST | bool) }}"
The mailu preload-assertion in sys-svc-mail/tasks/01_core.yml previously
fired on every host in `'web-app-mailu' in group_names` with no
MODE_ASSERT gate at all, so swarm workers ran into the assertion every
single CI run even though the preload state lives only on the stack
host. Same shape applies to a dozen other assertions: sys-svc-rdbms
(already double-gated, now collapsed), keycloak's kcadm-driven LDAP /
RBAC / reCAPTCHA / groups-path asserts, the sys-svc-compose handler
validators for compose/swarm config, svc-prx-openresty's 02_assert,
sys-svc-webserver-core's 05_assert, web-app-matomo's bootstrap token
check, web-app-joomla's 05_assert, and the sys-ctl-alm-compose notifier
self-start.
Each asserted state (run_once_<role> flags, kcadm output, openresty
container probes, compose-tree files) lives only on IS_STACK_HOST, so
the new SPOT keeps the assertion semantics intact while skipping the
spurious worker-side failures.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The IS_STACK_HOST gate in main.yml never blocked workers because
test-e2e-playwright does not set its own application_id, so DEPLOYMENT_MODE
fell through to 'compose' and IS_STACK_HOST evaluated true on every host.
Workers then ran 02_run_one.yml and crashed on the CA cert check because
the cert only exists on the manager.
Replace with block: + delegate_to: STACK_HOST + run_once: true so the
orchestration runs once on the primary manager regardless of which hosts
are in the play.
Also rename SWARM_PRIMARY_MANAGER -> STACK_HOST repo-wide to match the
existing IS_STACK_HOST symmetry and drop the swarm-specific terminology
(the variable falls back to inventory_hostname in compose mode).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Collapse two repeated swarm-vs-compose expressions into single-source-of-truth
group vars and apply across sys-svc-compose, web-app-mediawiki:
- SWARM_MANAGER_ACTIVE: true when DEPLOYMENT_MODE \!= 'swarm' or this host is
in svc-docker-swarm-manager. Used as the gate for "execute manager-only
deploy work here" (dir creation, file render, compose up, handler tasks).
- SWARM_PRIMARY_MANAGER: first svc-docker-swarm-manager or self under
compose-mode fallback. Used as delegate_to: for cluster-wide ops
(compose pull/build/up, swarm stack deploy, post-deploy DB health probe,
network create, MediaWiki one-shot install/admin).
Pure refactor — no semantic change. Subsequent edits to either expression
now live in one file.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DEPLOYMENT_MODE is now derived per role from group membership:
swarm iff len(groups[application_id]) > 1, else compose. The cluster
infrastructure flag SWARM_CLUSTER_AVAILABLE is true whenever any host
sits in svc-docker-swarm, so DB and proxy services render attachable
overlay networks even when their own group lists only one host.
All service_is_stateful flags are removed across svc-db-mariadb,
svc-db-postgres, svc-db-openldap, svc-prx-openresty, svc-docker-registry
and svc-db-mariadb's compose template; the inventory shape now expresses
the same intent. Cluster-wide handlers (swarm build/push/validate/deploy,
external network creation, DB healthcheck) gain run_once + delegate_to
the manager so they execute exactly once regardless of how many hosts
the role's play targets. CA-injection and compose validate are gated to
compose-mode only; their compose-config strictness breaks when the
deploy block coexists with container_name under swarm.
Dead per-host swarm-resolve infrastructure is removed: swarm_resolve_
container.sh, resolve_container.yml, resolve_self.yml, the MariaDB-
specific swarm_mariadb_cid.sh, the cross-host template replication
step, and the _swarm_retries / _swarm_delay defaults. With run_once +
delegate_to: manager replacing the old "render every template on every
swarm peer" pattern, every consumer becomes a noop on workers and
collapses to the manager's one-shot path.
The web-app-nextcloud memcache override mapping flips from the legacy
"single" sentinel to "compose" (semantically equivalent under the new
model). Requirement 023 is updated to reflect the new model and drops
the now-irrelevant per-role opt-out / placement-constraint clauses.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`service_is_stateful` is set unconditionally to `false` in
group_vars/all/18_swarm.yml; per-role overrides in vars/main.yml win by
Ansible precedence. The `| default(false)` fallback is no longer reachable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the host-scoped set_fact resolvers in sys-svc-compose/tasks/01_core.yml
and tasks/stages/01_constructor.yml with a single group_vars definition driven
by group_names. Uppercase the variable to match the SPOT convention and drop
every `| default('compose')` fallback now that the resolution is unconditional.
Role-var opt-out on a swarm host still works via Ansible's role-vars > group_vars
precedence.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a manager-pinned, NFS-backed `registry:2` service plus a transparent
retag/push/rewrite step so swarm workers can pull custom-built images from a
local registry, replacing the `docker save | docker load` pilot crutch.
Mechanism: after `compose build`, the new `swarm_registry_push.py` reads the
rendered compose.yml, retags each service that declares both `build:` and
`image:` with `<manager>:5000/<image>`, pushes the retag, and rewrites the
`image:` field in place. Subsequent `docker stack deploy --resolve-image
never` then references the registry-qualified name, so workers pull from the
cluster-local registry over HTTP via `insecure-registries`.
No per-role changes required — the rewrite is centralized in
sys-svc-compose handlers and skips services without a `build:` directive.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the svc-docker-swarm, svc-docker-swarm-manager, svc-storage-nfs-server
and svc-storage-nfs-client roles, the compose_volumes filter NFS support,
the test-deploy-swarm-nfs CI workflow with its swarm-nfs-pilot script set,
and the deploy_mode-aware compose/swarm wiring across mediawiki, mariadb
and sys-svc-* helpers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Without a timeout, `docker compose up -d` blocks indefinitely on
`depends_on.condition: service_completed_successfully` when an upstream
container is stuck in a restart loop (observed: zammad-init crashing
at boot on the broken telegram-bot-ruby gem in Zammad 6.5.0; the play
hung 5+ hours before manual kill). Failing fast surfaces the broken
upstream and unblocks iteration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move `COMPOSE_HTTP_TIMEOUT`, `DOCKER_CLIENT_TIMEOUT`, `DOCKER_BUILDKIT`
and `COMPOSE_DOCKER_CLI_BUILD` from per-task `environment:` blocks to
the play-level `environment:` directive in playbook.yml so every Ansible
task that shells out to docker/compose picks them up automatically.
Removes three duplicate blocks (sys-svc-compose handlers `compose pull`
+ `Build compose`, web-app-matrix `compose up`, web-app-bigbluebutton
`create compose.yml`). The `compose up` handler keeps its
`COMPOSE_PARALLEL_LIMIT` (not in the centralised set).
Lets operators force the legacy docker builder via INFINITO_BUILDKIT_ENABLED=0
(propagated through env/default.env, the compose env block, the BUILDKIT_ENABLED
ansible var and both the compose wrapper and the Build-compose handler), and
makes sys-svc-compose pull.py treat a failed `docker compose pull` as success
when every required image is already present locally — both needed to deploy
in sandboxed environments where docker.io is unreachable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
