A deploy that renders a changed compose.yml, .env or compose override
notifies the compose-up handler. When the run fails before that handler
completes, for example because the image pull hits Docker Hub's
toomanyrequests, the notification is lost. The next run renders the same
files (changed=false), and the fallback in tasks/utils/up.yml notifies
compose-up only when no container of the stack runs, so the stack keeps
its previous images and configuration while the deploy ends with
failed=0 (reproduced with web-app-akaunting 3.1.8 -> 3.1.21).
04_files.yml now writes `.compose-up-pending` into the instance
directory whenever one of those three files changed in compose mode. The
compose-up handler removes it right after `compose up` succeeds, so a
failed pull or up leaves it in place, and up.yml notifies compose-up
whenever it finds the marker on a later run.
Comparing `compose config --hash` with the containers' config-hash
labels was ruled out: on an up-to-date stack the two differ for every
service, so it would force-recreate on every deploy.
Fixes#1169
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cache-less rebuild is the fallback for a build whose spaced retries
are exhausted. It ran once, so one more registry timeout turned a
transient network failure into a failed deploy. Both the compose and the
swarm handler now retry it three times, 30 seconds apart, like the build
they back up.
Closes#620
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
roles/sys-svc-mail-msmtp/tasks/ca_bundle.yml stat'ed candidate bundle paths on
the Ansible target and wrote the winner into msmtprc. Moodle and WordSpress then
copied that file into a container, where the named path is a different file - or
none. Only update-ca-certificates populates a container's system bundle, and
that runs solely from the with-ca-trust.sh entrypoint, which compose_ca.py sets
only when 'docker image inspect' succeeded. On a fresh host the image is built
minutes later, so it never ran: job 94965457602 of run 31864448027 shows
Config.Entrypoint still moodle-entrypoint.sh while the CA sits mounted beside it,
and msmtp reporting 'The certificate issuer is unknown'.
An explicit tls_trust_file also outranks SSL_CERT_FILE, so the env injection the
override always performs could not compensate.
Name CA_TRUST.inject_cert_container when the CA is actually injected, and emit
no line at all otherwise - msmtp's own 'system' default is the right answer for
a letsencrypt relay, and it is what the container's stock bundle verifies. This
also removes a latent defect on letsencrypt: a RedHat host would have written
/etc/pki/... into a Debian container.
The removal of the probing apparatus is taken from 9a72d01eb on
fork/feature/ai-agent-employees, which diagnosed the wrong-machine measurement
correctly. Its own conclusion - drop the line entirely and rely on the trust
store - does not hold where the wrapper never runs, and in swarm it never does:
handlers/swarm.yml passes wrapper=False by design.
The three-term predicate behind all of this was hand-copied into four places.
lookup('ca_injected', application_id) is now its single source; the swarm and
compose gates and sys-svc-compose-ca's own var read it too.
compose_ca.py finally says when it degrades to mount-plus-env, naming the service
and the image it could not inspect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A handler name was doing two jobs at once: the line an operator reads in the
play output, and the identifier a `notify:` binds to. The two pull in opposite
directions -- the log wants "restart the SSH service, with diagnostics", the
binding wants a token that survives quoting, YAML flow style and a shell
round-trip -- and a name that changes for readability silently changes which
handler runs.
Every handler that a notify reached by name now carries a `listen:` with that
exact wording, so the binding is explicit before anything is renamed. The
topics themselves become identifiers: `compose up` reads `compose-up`,
`Restart NFS ganesha` reads `restart-nfs-ganesha`, 22 of them across both
sides of every notify. Where the topic is assembled at runtime, both branches
moved together: the conditional in sys-service carries two literals, and the
LDAP renderer interpolates `import-{{ folder }}-ldif-files`, which is what
makes those handlers reachable at all.
Freed from the binding, the names take the emoji prefix every other task line
already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three of the failures triaged today were a single unretried external fetch:
deb.debian.org went dark for 76 seconds on one node while three other distro
rounds of the same job built the same image; packagist answered 502 for two
metadata files while the identical build passed thirteen minutes later; and
ghcr.io timed out on a manifest HEAD for six minutes.
Both build tasks were the only network-bound tasks in their file without a
retry, and their in-shell fallback made a transient fault worse: it re-ran
immediately, with no wait, and threw away the cache it had just filled. The
ladder now waits 30 seconds between three attempts, and the cache-discarding
rebuild moved out into its own task that runs only once those are exhausted -
at which point the transient explanation is spent and a corrupt cache is the
next candidate. Three attempts rather than ten because ten would run 36270s
against a 21000s step cap.
svc-registry-docker is excluded from the registry sync - it cannot mirror
itself into its own registry - which left it the one role fetching from a public
registry through the swarm agent, single-shot, where every other role gets ten
tries. It now pre-pulls through a retried task, gated on the sync task having
been skipped so the exclusion keeps one source.
The accompanying check computes the ladder's worst-case wall clock from the
handlers and measures it against default.env and the workflow step cap rather
than restating the numbers, and it fails if an ignore_errors ever loses its
paired fallback - which would turn a failed build into a silent pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
compose_networks renders a foreign provider network as external: true, leaving open who creates it. The swarm handler has answered that since it was written, looping compose_external_networks through the network create task so a shared provider overlay exists even when the provider role did not run in the same play. Compose had no counterpart: it only worked while consumer and provider deployed together, because then the provider role created the network itself.
Deploying roles one at a time, as the roundtrip does, breaks that assumption and compose up aborts with "network <provider> declared as external, but could not be found". The compose handler now mirrors the swarm block: same lookup, same create task, triggered before the stack comes up. No cluster gate, since that condition is swarm-only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker stack deploy leaves tasks running when the service spec is unchanged, so a change made inside a container never reaches the process. PeerTube's CLI installs plugins with register:false, so the OIDC plugin sat installed and enabled in the database while the running server had loaded nothing: /login rendered only the local form and both OIDC specs timed out through all three attempts. Compose never showed it because its handler force-recreates.
Add a container recreate notify. In swarm it forces a task recreation per stack service and waits for convergence; in compose it maps onto the existing --force-recreate, so a role states the intent once and both modes honour it.
In the peertube role the install probe now tests node_modules/<npm-name>, where pnpm puts the plugin code; the old path was the data directory that registerPlugin creates at boot, so it never existed and the plugin was reinstalled on every run. The settings upsert derives changed from the rowcount via RETURNING, because database_query reports changed from the kind of statement rather than from what it touched. A registration probe notifies when the running process has not loaded the plugin, so an already-broken instance heals itself instead of walking past both earlier probes. The plugin:enable task is gone: npm has no such script, the call always failed and failed_when swallowed it.
The spec now polls the button lookup instead of counting once twenty milliseconds after goto, while still asserting on the URL so a failure keeps printing the address the browser actually sat on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every chdir-dependent compose/swarm handler assumed the instance dir of the flush-time application_id exists; a drifted notification then died with a cryptic 'Unable to change directory'. Guard the flush with a shared stat+fail (tasks/utils/guard_instance_dir.yml, included first in both handler files) that names the app, the missing dir and the drift cause, and instructs to fix the notifying role.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a 'timeout' lookup plugin that returns round(base * TIMEOUT_FACTOR)
(group_vars default 1, so value-identical to the old literals). One knob
in group_vars stretches every wait in the stack for a slow uplink:
TIMEOUT_FACTOR: 2 doubles them all.
- plugins/lookup/timeout.py + unit tests (identity, factor scaling,
string coercion, error paths).
- group_vars/all/17_resource.yml: TIMEOUT_FACTOR: 1.
- Convert all 34 literal 'timeout:' task keywords / module args across 25
task+handler files to "{{ lookup('timeout', <base>) | int }}".
- tests/lint/repository/yaml/test_timeout_uses_lookup.py enforces the
lookup in every task/handler timeout (opt-out: # nocheck: timeout-lookup).
- swarm.yml pre-deploy build handler: add the missing timeout (same
network-build hang vector as compose.yml's Build compose).
- Drop 13 pre-existing narration comments in the touched files (comment
lint enforces whole-file on any edited file).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A network pull with no time bound can hang indefinitely: run10 sat 6h
in 'docker compose build --pull' for bbb with no new output, and the
streaming monitor stayed silent because a hang emits no log lines.
Add timeout: 3600 to both network-pull handlers (compose pull, Build
compose) so a stuck pull self-aborts at 60min and the deploy fails
loudly instead of hanging forever.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the mode-specific handlers from a single main.yml with per-task
`when: DEPLOYMENT_MODE ==/\!= 'swarm'` conditions into two sibling files
(compose.yml, swarm.yml) imported via `import_tasks` with the mode guard
on the wrapper. The mode condition is set once at the import site
instead of being repeated on every handler.
main.yml keeps the shared prune handler (storage-constrained, fires for
both compose-up and compose-build listens regardless of mode).
Also: the handler-invocation lint at
tests/integration/ansible/handlers/test_invoked.py now follows
import_tasks references inside handlers/main.yml so it picks up handler
definitions in sibling files. Without this, the new layout fails the
"every notify maps to a handler" check because the scanner previously
only looked at main.yml.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
