docker stack deploy --detach=false converges on a running replica. web-svc-cdn's build container and every migration job end Shutdown/Complete by design, so the CLI waited for a state those stacks cannot reach: nine jobs of run 33698844864 burned the runner's 20400s budget inside this one handler, six of them on web-svc-cdn. The gate that follows already knows better, is_completed_oneshot accepts exactly that task shape.
The timeout keyword could not bound it either. The stack host is reached over ansible_connection: docker, whose plugin runs the child inside a Popen context manager, and __exit__ waits the child out before the timeout exception escapes; a probe measured 90s of wall clock against a 6s timeout. The bound now sits in the shell, where it works.
Handing the wait back to the gate would reopen the race 87e4313e11 closed, so the deploy stamps its start and names the services it reported touching, and the gate treats their earlier completed update as still pending. An inspect format the daemon rejects falls back to the old one, which only disables that check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gate read UpdateStatus but latched only on the states an update cannot
leave by itself, so an in-flight "updating" passed. It then accepted the stack
replica column, whose current count is satisfied by old-spec tasks that are
still running. Between the two, a rolling update reported ready before it had
begun.
Moodle is where that surfaced. A rotation between rounds changes the database
password, the role alters the MariaDB user and renders the new one into the
service environment, and the post-deploy tasks then run inside a container
from the previous spec:
Access denied for user 'moodle'@'192.168.201.4'
with MariaDB healthy throughout and that address being the manager node's
overlay load-balancer sandbox.
Name "updating" and block on it, and collect any state the case does not know
into its own bucket that reports itself before being treated as in flight, so
a state Docker adds later is diagnosable rather than a silent repository-wide
freeze. Count replicas from the tasks whose desired state is running, which a
rolling update sets to shutdown on the outgoing spec.
The cheap replica comparison stays the fast path and the per-service query
runs only when it passes, which is the only case the old count got wrong. This
gate runs on every swarm deploy, and a converging stack must not pay a
subprocess per service per poll. The service inspect gets one retry before it
fails the gate, so a timeout does not end a deploy on its own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
House rule: a comment is a nocheck directive, a parameter doc, or a marker for a
concrete trip-wire. Neither of these is.
realign_admin_password.sh explained that staleness is measured against a marker of
the password the script last wrote rather than against the account. That is design
rationale, it belongs in a commit message, and 5793512ffb already carries it. It
sat directly under the legitimate `# Exit:` doc, which is what let it pass every
earlier read: the banned label hidden inside the allowed kind of comment.
stack_ready.sh explained that `docker service ps` renders CurrentState as prose.
Pre-existing, not mine, but 455a73378 touched the file, and touching a file means
purging the narration it inherited too.
Neither diff removes anything but comment lines; both files re-parse under bash -n
and make quality-high is green over this tree, lint-shellcheck included.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
stack_ready.sh reports on both of its unhappy exits -- the latched branch dumps
the offending services, the not-converged branch dumps task rows and state -- and
falls off the end of the file in silence when it succeeds.
That silence cost a whole investigation. Moodle#1 of run 33414875521 died with
rc 137 and empty stdout/stderr 1.6s after this gate returned ok, and the entire
44422-line job contains no docker service ps, no task-state dump and no
State.OOMKilled for moodle. Which task generation was live at the moment of the
exec had to be reconstructed from the file NAMES in the rescue tree, and the
proof only arrived from a separately downloaded bundle:
moodle_moodle.3.mwdxfc7z6m3q877dr3ow1f5qr finished = 20:05:17.418099896Z
install task end = 20:05:17.427801
The task ids and Status.Timestamp this prints are exactly what was missing.
Cost is one run per convergence, not per poll: ansible's until stops at the first
rc 0, so the success path executes once per deploy. Both docker calls inside
report_task_states are already timeout 15 and swallow their own failure, so a
slow manager delays nothing and breaks nothing.
The array guard mirrors the existing one at :90 -- under set -u, "${arr[@]}" on
an empty array aborts in bash before 4.4.
Deliberately NOT included: a gate for UpdateStatus.State == updating. The gate is
demonstrably blind to in-place updates -- it returned ok in 0.630s after a stack
deploy that reported changed, and across all nine deploys of PASS 2 it passed on
the first poll, while in PASS 1 (service creation) it retried 56 times over
2:25.116. But UpdateStatus is written by swarmkit's updater goroutine after the
raft commit, which is the same lag that makes the replica count stale, and the
old container still reported healthy 0.6s AFTER the gate returned. A previous
update also leaves the status at completed, which an early poll reads as fine.
That fix would narrow a window of unmeasured width across 40+ swarm roles rather
than close it; it needs the timing measured on a throwaway cluster first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every one of these call sites ended in `|| true`, so a failure left no trace
and the surrounding script reported a clean run.
service_converged.sh needed the `|| true` for a reason that had nothing to do
with the command: under `set -o pipefail`, `head -10` closes the pipe and awk
dies of SIGPIPE, which pipefail then reports as the pipeline's status. That
made a genuine `container service ps` failure indistinguishable from a normal
truncation. Capping inside awk (`n++ < 10`) removes the SIGPIPE, so the command
status can be tested on its own and a real failure now prints why. give_up and
report_failed_tasks carried a byte-identical copy of that block, so the fix
lifts it into one report_tasks helper rather than applying it twice.
The swarm-clean scripts keep going after a failed step on purpose, since the
host layer below them clears what survives. They now say what did not work
before continuing, instead of continuing silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merged branch moved every role-local script out of files/ into a folder
named for its language (files/python/, files/php/, files/shell/), but the
moves lost their staging and never landed. HEAD still carried the old paths
while the worktree had the new ones.
Move the 24 remaining scripts and update the six places that resolve them:
the mcp-adapter instance/isolation/probe tasks, the litellm key task, the
adapter README and the endpoint-path lint. Every changed line in those
files is a path.
Also drop roles/sys-svc-compose/files/swarm/stack_ready.sh, a stale
duplicate of the copy already tracked under files/shell/swarm/.
The added comments are the branch's own `nocheck: mirrored-unit-test`
directives, which state why a script whose measurement only exists inside
its container cannot have a mirrored unit test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b384f4825 put two of them above churned_past_grace and has_fatal_task, and
e41508f48 put one in the force_shared_db module docstring. The prefix is not
a licence to keep explanatory prose; the comment rule wants it deleted.
The docstring paragraph keeps its content and loses only the prefix: it
records that Magento hard-fails setup:install against the central MariaDB
major, which is the reason the manager-pinned carve-out exists at all.
No executable line changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
swarmkit's default failure_action is pause, and the updater returns early
on a paused state without ever reaching its completion branch, so a
service that latches never leaves that state on its own. The gate every
role runs read replica counts and nothing else, so a latched update looked
exactly like a slow one and burned the full retry budget before failing
without a cause. That is what turned one corrupted openclaw task into a
25-minute silent timeout.
Read UpdateStatus before comparing replicas. The order carries the fix: a
rolled-back service reports N/N of the PREVIOUS spec, so a replica check
alone reports it as converged. Guard the format against a service with no
update in flight, since a bare .UpdateStatus.State nil-pointers and would
fail the whole multi-service inspect.
The handler has to honour the give-up or the gate keeps polling. It stops
only on the explicit give-up and still retries every other exit code, so a
timeout returning 124 does not turn into a failed deploy. That deviates
from wait_converged.yml, which stops on anything but 1; the deviation is
deliberate.
Stop treating rollback_completed as success in the per-service probe. It
is inert today because nothing here issues a rollback, and it is the guard
that keeps a silent revert from returning as a green deploy.
failure_action is deliberately left at the default. Rolling back would
revert a failed image bump while reporting success, and a daily cron opens
those bumps; it would also place the old binary against a schema ~28 roles
migrate on start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two callers polled the same stack_ready.sh with different budgets: the
handler read swarm_stack_converge_retries while post_deploy hardcoded
150. Eleven roles had set the override and were unprotected at the
second caller anyway. Both now read one SPOT in group_vars.
The gate also had no way to give up. A crash-looping espocrm burned 600
polls (20 min) before failing, while flowise was killed 45 s short of a
409 s image pull. has_fatal_task now exits 2 - and both callers stop at
once via 'until: rc in [0, 2]' - when a slot's newest attempt is
Rejected or Pending, or when a slot burned three attempts after the
service churned past swarm_stack_fatal_grace. Patience only grows where
nothing is broken.
report_tasks loses its 'awk !/error=$/' filter, which printed only rows
that HAVE an error and so stayed silent in exactly the no-error case
that occurred.
Measured on a live single-node swarm: a service failing 3x and
converging on the 4th returned rc 2 at t+15s before this change and rc 0
at t+32s after; a 3-replica stop-first rolling update gave 68/68 samples
rc 0; a permanent crash loop still reaches rc 2. The deploy epoch comes
from 'docker service inspect .UpdatedAt.Unix' rather than a /tmp marker,
which was reproduced contaminating a later deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A paused update never completes on its own: the updater returns early on that
state, before the branch that would finish it, and refuses to start again while
an update status is set. The restart policy brings the replicas back but does
not touch the status, so the service can sit fully replicated while the gate
still refuses it.
The probe polled that for six hundred attempts and reported a timeout after
twenty minutes. It now separates a state worth another poll from one that is
final, and the wait distinguishes the two exit codes instead of retrying every
refusal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Driven by an until loop of 600 polls, the probe exited 1 without a word on
every refusal, so a service that never converged produced a twenty minute wait
and then a fatal carrying an empty stdout and an empty stderr. The reason had
to be reconstructed from a rescue artifact, and the update-status branch is not
recorded there at all.
Each refusal now names itself on stderr. Only the branch with task rows to show
prints them, filtered to the rows carrying an error and capped, which is the
shape the neighbouring stack_ready.sh already follows; the update-in-progress
branch stays a single line because it is the normal state of a healthy rollout
and fires on every poll.
A timed out service ps used to abort the script with its own code and no
message, reproducing the very symptom this addresses. It is now named too.
Adds the unit tests the script never had, mirroring the sibling's, including
the assertion that error-free task rows stay out of the output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`docker service ps` renders CurrentState as prose — "Preparing 3 minutes ago" —
which cannot distinguish an image still extracting from a task idling on
something else. That gap cost a whole investigation: Nextcloud 2's flowise task
spent ~412s somewhere between pull and running, and the artifact could only
bound the window, never attribute it.
report_task_states adds the machine-readable half:
zqimhnngo3my state=preparing since=2026-08-10T12:53:40Z msg=preparing
Verified against a real swarm, read-only, not just the stub: `--type task` is a
valid type (the daemon answers "task ... not found", not a type rejection), all
three field paths resolve on live tasks, and ten inspects cost 0.07s against the
15s timeout. The type pin matters here because this repo names a role's overlay
network after the same entity as its main container, so an untyped inspect can
resolve the wrong object — tests/lint/ansible/shell/container caught that.
The test file also corrects a stale contract. 3ebd6e950 dropped the
`awk '!/error=$/'` filter precisely so a healthy-but-slow task stays visible, but
test_error_free_task_rows_are_not_echoed still asserted the old behaviour and had
been failing since. It now asserts the row IS echoed, and a second case pins that
an empty inspect result does not print a bare indented line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
report_tasks filtered with awk '!/error=$/', keeping only rows whose ERROR
column is non-empty. That discards exactly the states a converge timeout
needs reported -- Pending, Assigned, Preparing, Starting, and any
unhealthy-kill whose image STOPSIGNAL yields exit 0 -- and keeps only the
states that already explain themselves.
Friendica hit precisely that hole: all six task rows sat in Starting or
Shutdown/Complete with a blank ERROR (its apache image uses SIGWINCH, so a
health-kill exits 0 and swarm records Complete, not Failed). Every row was
dropped, the `[ -n "${rows}" ] || return 0` guard returned silently, and the
operator saw a bare "not converged: friendica_friendica" with no diagnosis.
Decidim's rows printed only because Failed carries an error string.
Drop the filter. report_tasks runs per non-converged service, and
docker service ps lists each slot's current task before its history, so the
head -10 cap still shows the rows that matter. Strictly additive: more rows,
never fewer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three of the six failures triaged today were explainable only after downloading
a rescue artifact, and two not even then. The job log carried "not converged:
docker_docker", a bare "rc=1", and "failed after 3 attempt(s)".
The convergence gate now prints the task rows of a service that will not
converge - but only those whose error field is filled. A stack coming up
normally stays silent, while the one that killed svc-db-postgres would have
shown `error=Head "https://ghcr.io/...": request canceled` in the job log
instead of hiding it in an artifact. awk rather than grep: grep exits 1 when
nothing matches, which under set -e would take the script with it.
The remote pull decodes its exit status. os.system returns rc << 8, so the
message said 5888 where rsync said 23, and every attempt now names its code.
The sleep after the final attempt is gone - five wasted minutes per run.
The matomo poller hands its last 40 lines to stderr on deadline, redacted at
the source, and the failing task interpolates them the way the compose flavor
already does. No no_log: MASK_CREDENTIALS_IN_LOGS is force-set anyway and would
have censored exactly the payload this exists to deliver.
The `-p warning` filter added to trigger_units.sh a few commits ago is removed.
systemd journals a unit's stdout and stderr at info, so that filter would have
swallowed the per-provider attribution the same commit introduced.
stack_ready.sh gets its first tests: replica comparison, the completed-oneshot
branch, and both halves of the new reporting - the error reaches the caller, the
error-free rows do not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
generate_cache_override pinned deb.debian.org/archive.ubuntu.com/security.ubuntu.com
to INFINITO_CACHE_PACKAGE_FRONTEND_IP for every buildable service whenever that
env var was non-empty. The IP is a baked default.env value, so the hijack fired
even where no cache frontend exists: the guide job sources .env, replays a role's
compose build inside the DiD stack, and the moodle apt build hit 172.30.0.4 with
'No route to host' because no frontend is provisioned in CI.
Gate the override additionally on the frontend CA file existing non-empty, the
same signal ca.sh uses for an active cache profile and that registry_cache_active
produces. When the cache profile is inactive (CI/guide) the CA is absent, the
override is skipped, and the build resolves the mirrors via real DNS. compose.py
is copy-deployed and cannot import Profile, so a filesystem signal is required.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docker manifest inspect is a pure client command and ignores the daemon's insecure-registries list, so against the swarm-local HTTP registry it reported every existing manifest as missing. The push skip therefore never triggered: each deploy re-pulled every external image over the slow lab NAT, re-pushed it, and the final rmi dropped the engine copy again. With the flag the second and every later sync recognizes cached manifests and completes in seconds, proven live on the kept matrix cluster. The test fake_run learns the timeout keyword the pull sites now pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The act NAT path delivers large upstream images at roughly 1.4 MB/s, so an 836 MB pull legitimately exceeds the shared 600s subprocess timeout; killing the client discards the in-flight layer and every retry restarts from zero, wedging the cache-externals handler in an endless loop. Pulls now get a 1800s budget, three times the measured cold duration of the largest offender, while tag and push keep the tight limit because they only ever talk to the node-local registry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The install and realm-import waits accepted Shutdown as success, so a crashed job passed silently: akaunting went green without ever installing, and a keycloak import crash left synapse against a realm-less provider, surfacing as matrix Playwright OIDC timeouts. The state probe moves to a shared sys-svc-compose script where only Complete passes, Shutdown fails loudly with the task error, and Failed/Rejected count as pending because the jobs now restart on-failure with three bounded attempts, turning the observed import race from a coin flip into self-healing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
wait_converged checked only the first task line, so a rolling update with a still-running old task reported false convergence. The probe now lives in files/swarm/service_converged.sh (stack_ready.sh style: script module, SERVICE env, timeout 15 per engine call, container wrapper): it requires no update in flight (UpdateStatus completed/rollback_completed, or absent) and every desired-state=running task to be Running. Covered by a seven-scenario stub test during extraction; a never-updated service maps the engine's nil-template error to the no-update case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run_cmd merged stderr into stdout, so compose warnings ('variable is
not set', ...) leaked into everything pull.py parses: the yaml config
for pull targets, the config --images fallback list (turning warning
lines into phantom image names), and the pull --help probe. A project
with an unset optional variable then failed the pull step even though
every image was present locally. Capture the streams separately, parse
stdout only, and forward stderr to the log.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The manager-only convergence check classified a service by the
CurrentState of every task docker service ps returns, so a completed
run-once job whose first attempt failed before the database was ready
kept tripping the health gate on historical Failed entries, failing the
terminator for the full retry window while workers passed. Classify by
DesiredState and CurrentState of only the newest task per slot: stale
history no longer votes, a newest failed task still flags, and live
non-converged services keep flagging. stack_ready.sh gets the same
classifier so the deploy gate and the health gate agree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve every 3+-file same-prefix group flagged by the new prefix lint
by moving the files into a subfolder named after the shared prefix and
dropping the prefix from the file names, updating all references:
- utils/env/handlers/infinito_*.py -> infinito/ (dir/, package_cache/
nested a second level); registry imports and the four handler-dir
globs switched to iter_project_files
- utils/install/lint/ansible_*.py -> ansible/ (aliased imports)
- roles/sys-svc-compose/files/swarm_*.{py,sh} -> files/swarm/
- web-app-erpnext/web-app-zammad files/scripts/apply_*.{py,rb} -> apply/
- web-app-odoo files/sql/{select,update}-* and sql/oidc/{grant,insert,
select,update}-* -> matching subfolders
- web-app-keycloak tasks/update/05_ldap*.yml -> 05_ldap/ (internal
includes made role_path-absolute)
- plugins/filter/get_*.py -> plugins/filter/get/ (filter names are the
filters() dict keys, so Jinja usage is unaffected; ansible discovers
the one-level-deep subdir)
Ansible lookup dirs, addon dirs and .github/workflows keep their flat
layout (the filename is the public id / GitHub only discovers flat).
Unit tests mirror the new source layout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A docker pull against ghcr wedged for 8+ minutes with zero progress
during the update pass and only the 1h task timeout would have
reaped it; give every pull/tag/push subprocess a 600s bound so the
handler's own retry loop takes over instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
--ignore-buildable only skips services that declare build:, so a
service that merely reuses an image another service builds (e.g. a cron
service on the app's custom tag) made docker compose pull abort the
whole run with 'pull access denied', taking the legitimate mirror pulls
down with it. Compute the locally-built image set from the compose
config and pass only genuinely pullable services to pull; skip the pull
entirely when every image is built locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The controller-side template cannot read file-backed secret sources
that only exist on the stack host, so the content-hash suffix silently
fell back to hashing the path string and never rotated. A regenerated
source (e.g. a self-signed key re-minted on SAN drift between deploy
passes) then collided with the immutable docker secret of the previous
deploy and stack deploy failed with 'only updates to Labels are
allowed'. Recompute the suffix from the actual file bytes on the stack
host right before the deploy; service references stay untouched since
they point at the yaml key, not the name.
Retention follow-up (pruning superseded generations) tracked in
https://github.com/infinito-nexus/core/issues/415.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
swarm_stack_ready.sh piped 'docker stack services' through a process
substitution, so the timeout wrapper added in 8516e3167 turned a slow or
wedged manager into an empty listing, an empty not_running set and exit
0: the convergence gate would have gone green with zero running replicas.
Capture the listing, fail on a non-zero rc.
That exposed the second half: post_deploy gated the swarm wait on the
cluster-wide DEPLOYMENT_MODE, so forced-compose roles (matrix ansible
flavor, bigbluebutton) ran it against a stack that does not exist and
only passed because the old code swallowed the error. Gate it on
compose_mode, like the handlers already do.
teardown.sh: the NFS mountpoint and umount calls, the likeliest hang
sites, were the only ones left unwrapped, and the timeout around docker
inspect inverted the D-state warning (a hung inspect returned 124, the
if went false and the script reported a clean teardown). Warn on rc 0
and 124, and shrink the per-node budget so the script fits inside its
step timeout.
Swarm step budget: the bounded steps alone summed to 365 over a 360-min
cap, so cleanup was still starved. Deploy 240, diagnostics 15, playwright
15, rescue 25, cleanup 15.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
27 jobs of run 29147179243 burned their full 360-min budget. Three
systemic layers, all addressed here:
- The CLI e2e task ran the staged test.sh unbounded, and test.sh blocks
in 'systemctl start' on a oneshot backup unit with no JobTimeout; on a
wedged dockerd or a stalled NFS mount both freeze silently for hours.
The task now runs the script under 'timeout TEST_E2E_CLI_TIMEOUT'
(60 min via the timeout lookup, so TIMEOUT_FACTOR scales it) and the
unit starts are bounded by BKP_TEST_HEALTH_TIMEOUT.
- The converge probe re-ran the full per-service diagnostics (service ps,
service logs, inspect, journalctl) on EVERY retry, so 600 nominal
2-second attempts cost 25-33s each and silently outlived the step
timeout. The block is gone (the workflow-level container.py rescue
snapshot already collects it recursively on failure) and every docker
call is timeout-bounded. Same removal in the swarm collect script,
which keeps only the NFS and render evidence container.py does not see.
- Playwright-collect, rescue diagnostics and Cleanup had no
timeout-minutes, and teardown.sh calls docker against D-state nodes,
so a failed job hung there for hours until the job cap killed it
without artifacts. Every step is bounded now, every teardown call is
timeout-wrapped, and the deploy steps (swarm 270, compose 300) leave
room for diagnostics and cleanup below the 360-min cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A wedged DinD dockerd hung a single diagnostics docker exec until the 360-minute job timeout cancelled the job together with its artifact uploads (job 86013153099 sat 5h in service logs). Route every diagnostics exec through a timeout wrapper, tail-limit the log dumps, bound showmount, cap the collect step at 30 minutes, and dump the pending services' log tails once on the final converge failure. The NFS state path moves into a single source of truth: STATE_SUBDIR in utils/storage/nfs.py feeds the nfs_state_subdir lookup (deploy side) and the derived INFINITO_SWARM_NFS_STATE_PATH env var (harness side); the diagnostics script requires both path vars instead of defaulting, and the swarm inventory drops its redundant export_base restatement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On disk-constrained sync runners (act/CI) a mirrored external image was double-stored: once in the docker engine and once in the local registry. After pushing, docker rmi the upstream and prefixed tags so only the registry copy remains; swarm re-pulls from the registry at deploy time. Locally-built images are never removed (recovery and base images depend on them staying in the engine store). Also skip the re-pull when the prefixed image already has a registry manifest. The compose image rewrite and registry push are unchanged; rmi is best-effort.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dump docker service inspect (UpdateStatus plus the resolved TaskTemplate)
and docker node ls, and widen the docker journal window to 10 minutes, so
a non-converging stack reveals rolling-update loops, node flaps, and the
desired-state changes that the previous 3-minute service-ps-only dump did
not capture.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The docker journal/events dump added for convergence-failure debugging
tripped the repo's raw-docker lint: 'docker events' is not in its
allowlist, and the lint also matches the literal string in the echo
messages. Call the container wrapper ('container events', which passes
through to docker) and drop 'docker events' from the surrounding echoes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The convergence probe only echoed the non-converged service name, so a
healthy-but-churning task (rescheduled with no error) could not be
diagnosed from the CI log. On the failing path emit 'docker service ps
--no-trunc' (reschedule ERROR column), 'journalctl -u docker' and
'docker events' for the last 3 min. The converged path exits before this,
so successful deploys are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The convergence gate flagged any service where running \!= desired replicas as not converged. A run-once job (restart_policy: none) such as matomo_bootstrap or erpnext_configurator runs, exits 0 and then permanently reports 0/1, so the gate false-failed the deploy even though the job succeeded. Now, for a service short of its desired count, consult docker service ps and treat it as converged only when a task reached DesiredState=Shutdown/CurrentState=Complete with no task still wanting to run; genuinely crashing (Failed) or still-starting (Running/Ready) services stay flagged. Since matomo_bootstrap is bundled into most swarm groups, this was a broad false-negative. Swarm-only; compose never invokes this script.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A launcher-managed app (discourse) renders an empty services map, so 'docker compose up' fails with 'no service selected'. The wrapper now returns no command for an 'up' against a zero-service project (main() exits 0); a compose that fails to parse is not treated as serviceless, so real errors still surface.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A service that builds an image locally produces a <prefix>X image; sibling services may reference that same already-prefixed image without their own build:. swarm_registry_sync compared the de-prefixed upstream against a locally_built set holding the PREFIXED form, so the siblings missed the membership check and fell through to docker pull X for a local-only image with no registry upstream (espocrm/bookwyrm/shopware 'pull access denied'). Store locally_built de-prefixed so it matches upstream.
Add test_swarm_registry_sync.py covering the build-sibling skip, the real-upstream pull, the already-in-registry skip and the empty-prefix short-circuit; the first test fails on the pre-fix logic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- web-app-keycloak: KEYCLOAK_CID_RESOLVE_CMD filtered status=running AND
health=healthy, so during swarm converge (no replica healthy yet) it returned
empty; the kcadm dead-container recovery then looped against the same dead cid,
burned its retries and fail_json'd. Drop health=healthy so a freshly
rescheduled running replica is usable.
- sys-svc-compose/swarm_registry_sync.py: a service sharing a sibling's locally
built <entity>_custom image (no build: of its own, already registry-prefixed)
fell into the prefix branch and ran `docker pull <entity>_custom` against the
registry -> 404. Collect the locally-built refs first and skip the upstream
pull when the prefix-stripped ref is one of them (it is retagged and pushed by
the building service).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DEPLOYMENT_MODE is binary (swarm / compose), so a negated `!= 'swarm'` gate
always means `== 'compose'`. Replace every negated gate across roles/ and
group_vars with the positive form for readability, and add a lint
(tests/lint/ansible/variables/test_no_deployment_mode_negation.py) that forbids
`DEPLOYMENT_MODE !=` in .yml/.yaml/.j2 with a `# nocheck:
deployment-mode-negation` escape.
The compose-verb / chdir / jinja-gate lint guidance messages now suggest the
positive form too; the gate matcher (utils/annotations/task_gate.py) and its
test inputs keep recognising the legacy `!= 'swarm'` so a nocheck-marked
exception is still detected as compose-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`/usr/bin/compose` (roles/sys-svc-compose/files/compose.py) auto-adds
`--env-file` and compose.yml carries service `env_file:`. `docker compose
build`/`config` parse+interpolate those env_file values and reject
$-bearing secrets (e.g. the dev keycloak password), whereas `docker stack
deploy` reads env verbatim and is unaffected. build/config need no runtime
env, so route them through env_file-stripped compose copies with no
--env-file; all other subcommands keep the original files + --env-file.
Adds `_subcommand`, `strip_env_files`, `_ENV_PARSING_SUBCOMMANDS` and unit
coverage for the new build/config path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`resolve-container-id` previously used the first running task from the
cluster-wide `container service ps` listing and emitted that task's
container ID. When the picked task lived on a worker but the caller
ran on the manager (every `delegate_to: STACK_HOST` exec path), the
follow-up `docker exec <id>` failed with
`Error response from daemon: No such container: <id>`.
Add a fast path that queries the local docker daemon via
`label=com.docker.swarm.service.name=<svc>` and returns a locally-
running task's container ID directly. The original cluster-wide
task lookup stays as a fallback for the case where no replica is
local — same behaviour as before, no regression possible.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The container_image lookup now returns a registry-prefixed image
string in swarm mode unconditionally. When the resulting compose.yml
references e.g. `swarm-mgr-01:5000/registry:2` but the local docker
daemon has not yet been told to tag that name, the previous
push-only path failed with "An image does not exist locally".
Add a backfill branch: if the image is already prefix-form and
neither has a sibling `build:` block nor an existing manifest in
the registry, strip the prefix to derive the upstream ref, pull it,
retag locally with the prefixed name, then continue to the push
step.
This fixes the svc-registry-cache bootstrap step where the cache
service uses `registry:2` upstream but the swarm scheduler needs
the prefixed `<host>/registry:2` tag in the swarm registry before
it can place the service. Previously this only worked accidentally
for roles whose build pass produced the local tag.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Workers in the DiD swarm cluster cannot reach upstream registries
(Docker Hub, ghcr) from the overlay network. Image pulls hang
indefinitely in "Preparing", which used to abort with a task-converge
timeout much later in the deploy. Until now the existing
`swarm_registry_push.py` only retagged + pushed services that had a
`build:` key (locally-built images); plain upstream images were left
for each worker to pull on its own, which never works in the DiD path.
This change extends the script to also cache upstream images:
* `docker pull <image>` on the manager,
* `docker tag <image> <prefix><image>`,
* `docker push <prefix><image>` to the on-cluster registry
(`svc-registry-docker`),
* rewrite the on-disk compose.yml in place so the swarm scheduler
pulls the prefixed name from the local registry over the overlay
network.
Idempotency: a `docker manifest inspect <prefix><image>` probe runs
before each push; on re-deploys with no upstream version bump the
push is a fast no-op. Hard-fails on any non-zero docker
pull/tag/push so the standard "fail loud when upstream broken" policy
holds; no `failed_when: false`.
Chicken-and-egg guard: when the registry service itself
(`svc-registry-docker`) is being deployed, the new branch would try to
push `registry:2` to its own not-yet-running registry endpoint and
loop on `connection refused`. The handler now skips the sync step
when `application_id == 'svc-registry-docker'`. The registry's own
image is pulled directly by the manager (default_placement: manager)
and never needs to be cached into itself.
Mechanics:
* `swarm_registry_push.py` renamed to `swarm_registry_sync.py` to
reflect the broadened scope (pull + tag + push, not just push).
* `roles/sys-svc-compose/handlers/swarm.yml`: updated script
reference, updated task title, added the registry-self-skip
conjunction in the `when:` guard.
Validated via act-swarm-zombie web-app-baserow: deploy progressed
from ok=145 (aborted at the chicken-and-egg push) to ok=806 once the
guard landed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Tier-1 changes verified live in act-12 (mariadb stack converged, postgres
stack started, swarm-readiness gate fired correctly for docker/mariadb/
postgres/openresty).
* `plugins/filter/compose_volumes.py` + `plugins/lookup/compose_volumes.py`:
thread `dir_var_lib` through filter (required keyword, no default)
and lookup (reads from `vars_["DIR_VAR_LIB"]`, raises if missing). Replace
native NFS driver with a bind-mount from the host's already-NFS-mounted
`DIR_VAR_LIB` because native `type: nfs` re-handshakes portmap, which is
unreachable in nested DinD. Drop the FilterModule wrapper so the deprecated
pipe form keeps failing fast per the lookup-only SPOT contract.
* `tests/unit/plugins/{filter,lookup}/test_compose_volumes.py`:
thread `DIR_VAR_LIB` through every fixture (no default-fallbacks).
* `roles/sys-svc-compose/tasks/utils/post_deploy.yml`:
compose-mode db-health probe gated to `DEPLOYMENT_MODE \!= 'swarm'`; new
swarm-mode convergence gate runs the extracted `swarm_stack_ready.sh`
via `ansible.builtin.script` with `STACK` env, addressed by absolute
`playbook_dir`-rooted path so the include resolves from caller roles.
* `roles/sys-svc-compose/files/swarm_stack_ready.sh` (new): probe
`docker stack services` for a single `STACK` and exit non-zero until
every service's running/desired replica count matches.
* `roles/svc-db-mariadb/vars/main.yml`: new `MARIADB_CONTAINER_ADDRESS`
constant wraps the `container_address` lookup so callers reference one
SPOT instead of duplicating the lookup.
* `roles/svc-db-mariadb/tasks/01_core.yml` + `02_init.yml`:
switch root-password probes and bootstrap SQL calls to
`container exec -i {{ MARIADB_CONTAINER_ADDRESS }} sh -c 'mariadb -p"$MARIADB_ROOT_PASSWORD"'`.
Using the container's own `MARIADB_ROOT_PASSWORD` env var mirrors the
healthcheck and matches whatever quoting Docker swarm preserves from
the env_file, sidestepping the compose-vs-swarm env_file quote-strip
divergence.
* `roles/svc-db-postgres/files/set_postgres_superuser_password.sh`:
`eval "container=$container_raw"` so the swarm-mode lookup payload
`"$(/usr/bin/resolve-container-id <stack> <svc>)"` is re-evaluated by
bash and resolves to the live task id at exec time.
* `roles/svc-db-postgres/tasks/01_core.yml`: rename arg to use the new
swarm-aware `POSTGRES_CONTAINER_ADDRESS`.
* `utils/tests/swarm/extend_inventory.py`: when the seed app's primary
service declares `default_placement: manager`, do not add the app to
the worker group, so the swarm test inventory matches the
single-replica placement-pinned semantics.
