A handler name was doing two jobs at once: the line an operator reads in the
play output, and the identifier a `notify:` binds to. The two pull in opposite
directions -- the log wants "restart the SSH service, with diagnostics", the
binding wants a token that survives quoting, YAML flow style and a shell
round-trip -- and a name that changes for readability silently changes which
handler runs.
Every handler that a notify reached by name now carries a `listen:` with that
exact wording, so the binding is explicit before anything is renamed. The
topics themselves become identifiers: `compose up` reads `compose-up`,
`Restart NFS ganesha` reads `restart-nfs-ganesha`, 22 of them across both
sides of every notify. Where the topic is assembled at runtime, both branches
moved together: the conditional in sys-service carries two literals, and the
LDAP renderer interpolates `import-{{ folder }}-ldif-files`, which is what
makes those handlers reachable at all.
Freed from the binding, the names take the emoji prefix every other task line
already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The body_filter lua render and the redirect-domain config both notified a full restart. Every accumulation window in the odoo swarm job carried one of them, so converting only the vhost render in 641925331 removed no restart at all.
Both write under directories.configuration.base, the same conf.d DIRECTORY bind the vhost uses, so SIGHUP is enough. Measured on job 93953781117: 27 of 30 restart windows become reloads, 27 x 39.9s = 18 min per swarm job. The remaining three are openresty's own base/default config and the prometheus http-block, which stay restarts.
Verified in openresty/openresty:alpine that a reload picks up a rewritten body_filter_by_lua_file across an inode swap -- lua_code_cache defaults to on, and SIGHUP forks workers with a fresh LuaJIT VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adding a domain redeployed the whole OpenResty task. Measured in the green onion job odoo 0: 26 redeploys at 41 to 43 seconds each, 18.1 minutes of a 4h44m job. A reload costs one to two seconds.
It is safe for a vhost because of where the file lands. nginx.py builds files.domain as servers_dir + protocol + domain.conf, and servers_dir is /etc/nginx/conf.d/servers/ - inside the conf.d DIRECTORY bind. A directory keeps its inode across the atomic write-and-rename that stack_host_template performs, so the running master already sees the new file. Only nginx.conf is bound as a FILE, whose inode the task pins at create time; that path keeps the redeploy, as do the default server config and the site-is-down case.
History matters here: a9ca2ae68 once made the handler reload for everything, including nginx.conf, and f9dea42c6 replaced it with the force-recreate. That replacement rode along in a commit about apt mirrors and explains itself nowhere, so the only recorded reason is the file header - the file bind and its pinned inode. That reason does not apply to a vhost in the directory bind, which is why this reload is confined to that case.
The reload cannot prove it took, so it does not have to. After the flush the domain is probed once, and anything other than 200, 301 or 302 notifies the full redeploy. The worst case is therefore one HTTP probe on top of the behaviour this replaces, never a domain left unserved behind a green deploy.
OpenResty runs replicated 1/1 (services.txt of a real deployment), so the exec reaches the only instance and no replica can keep stale config.
Not verified: the reload itself was never executed against a running stack.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The MODE_DEBUG block of the restart handler carries no host gate, unlike its two
siblings in the same file which are pinned to a deployment mode and, in swarm,
to IS_STACK_HOST. A handler runs on the host that notified it, and a worker
reaches this notify: sys-stk-front-proxy/tasks/main.yml:112-118 is an
always-changed `command: true` whose only guard is `not nginx_conf.changed`, and
the stack-host template action forces changed:False on workers, so the guard is
always satisfied there.
On a worker the container_address subshell then exits 64 - resolve-container-id
refuses to run off a manager - the substitution collapses to empty,
`container logs --tail=200` fails, and failed_when:false plus ignore_errors:true
swallow it. The debug task below prints an empty string, so the diagnostic
reports "no logs" on exactly the node someone enabled MODE_DEBUG to inspect.
container_address returns a string to embed, not a delegation; it decides what
is addressed, not from where. delegate_to supplies the second half.
STACK_HOST rather than the resolve_host_cid include, because svc-prx-openresty
declares placement: manager (meta/services.yml:9), so the task can only run on a
manager and the full host resolution buys nothing. In compose STACK_HOST is
inventory_hostname, so this is a no-op there. The sibling `container ps -a` stays
undelegated: a per-node container listing is legitimately node-local.
Diagnostics only - nothing gates on openresty_logs, and MODE_DEBUG defaults to
false. Not executed: no swarm worker run reproduces the empty dump.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add a 'timeout' lookup plugin that returns round(base * TIMEOUT_FACTOR)
(group_vars default 1, so value-identical to the old literals). One knob
in group_vars stretches every wait in the stack for a slow uplink:
TIMEOUT_FACTOR: 2 doubles them all.
- plugins/lookup/timeout.py + unit tests (identity, factor scaling,
string coercion, error paths).
- group_vars/all/17_resource.yml: TIMEOUT_FACTOR: 1.
- Convert all 34 literal 'timeout:' task keywords / module args across 25
task+handler files to "{{ lookup('timeout', <base>) | int }}".
- tests/lint/repository/yaml/test_timeout_uses_lookup.py enforces the
lookup in every task/handler timeout (opt-out: # nocheck: timeout-lookup).
- swarm.yml pre-deploy build handler: add the missing timeout (same
network-build hang vector as compose.yml's Build compose).
- Drop 13 pre-existing narration comments in the touched files (comment
lint enforces whole-file on any edited file).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the forced OpenResty rollout handler fires twice in one play, the second
'service update --force' can submit a stale object version and the daemon
rejects it with 'update out of sequence', failing the deploy. Retry until the
CLI reports success; any other stderr fails on the first attempt. Swarm-only
and stack-host gated, so compose is byte-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The swarm openresty restart used 'container service update --force --detach' (fire-and-forget), so the deploy + e2e proceeded before openresty reconverged with prometheus.conf. The per-vhost log_by_lua then saw a nil metric_requests ('attempt to index global metric_requests') and the prometheus metricz/scrape e2e failed. Drop --detach so the update waits for convergence (bounded by the existing timeout 600). Also guard the per-vhost log_by_lua against a nil metric so any residual pre-init window can't throw.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DEPLOYMENT_MODE is binary (swarm / compose), so a negated `!= 'swarm'` gate
always means `== 'compose'`. Replace every negated gate across roles/ and
group_vars with the positive form for readability, and add a lint
(tests/lint/ansible/variables/test_no_deployment_mode_negation.py) that forbids
`DEPLOYMENT_MODE !=` in .yml/.yaml/.j2 with a `# nocheck:
deployment-mode-negation` escape.
The compose-verb / chdir / jinja-gate lint guidance messages now suggest the
positive form too; the gate matcher (utils/annotations/task_gate.py) and its
test inputs keep recognising the legacy `!= 'swarm'` so a nocheck-marked
exception is still detected as compose-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The OpenResty redeploy handler ('remounts bind to live inode') fires once per app; without --detach each call blocked until the full rolling convergence, so the per-app blocks accumulated into an apparent hang. openresty actually converges (clean Shutdown->Running, old replica keeps serving during the rolling update, readiness is gated separately by the converge-wait), so add --detach to return immediately. Also add timeout: 600 (10 min) to that handler and to the three single-fire app service-update-force tasks (baserow post-migration restart, matomo install restart + bootstrap re-run) as a hard cap: a genuinely stuck redeploy now fails cleanly instead of hanging forever. 600s comfortably exceeds the normal rolling convergence (~2-5 min, matomo healthcheck start_period 60s x replicas) and matches the existing timeout: 600 already used elsewhere in baserow; the three keep their blocking so the next step still gets the service ready.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two compose-mode-only gaps that blocked compose deploys while swarm was unaffected.
svc-prx-openresty: the compose restart handler did 'openresty -s reload', which re-reads the file bind-mount but keeps the stale pinned inode after the real nginx.conf is rendered atomically (write-temp + rename). openresty stayed on the bootstrap 'listen 8080' config and the :80 default-vhost assert got connection refused. Force-recreate the container (like swarm's 'service update --force') so the bind remounts the live inode, plus a post-flight 'openresty -t'.
web-app-keycloak: KEYCLOAK_LOCAL_CID was resolved only in the swarm branch (via the swarm service label), so in compose it stayed undefined and the container health check fatalled. Set it from the compose container address when DEPLOYMENT_MODE \!= 'swarm'.
Both changes are gated to the compose path; the swarm deploy is untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace bare-name docker restart with container_address-resolved
openresty -s reload so the handler works on swarm tasks (named
<stack>_<svc>.<n>.<id>) without dropping listeners on 80/443. Apply
the same container_address fix to the MODE_DEBUG logs collection.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a mode-aware `container_address` lookup plugin and a
`resolve-container-id` helper script that resolve swarm task IDs at
exec time (compose mode returns bare names unchanged). Migrates ~25
roles + ~28 task files from `command:` to `shell:` so the lookup's
`$(...)` subshell evaluates correctly, and extends `database` lookup
with an `address` key for cross-role DB exec.
Three new lint tests enforce the contract: every container exec
target goes through the resolver, lives behind an IS_STACK_HOST gate
(walks the include graph one hop deep), and uses `shell:` rather than
`command:` when the target is a Jinja reference. Existing nocheck
suppressions on bare-name DB exec sites (nextcloud redis, listmonk,
moodle OIDC, mailu) are replaced with the lookup-based form.
Fixes the systemd Environment= injection in sys-ctl-cln-disc-space
(it never expanded $(...)) by passing STACK / SERVICE_KEY parts and
resolving inside the script. Extracts shopware admin and mediawiki
extension install bodies into dedicated .sh files invoked via env.
Rewrites password-bearing shell tasks to either `| quote` directly or
the `-e VAR=... | quote` + `sh -c '... ${VAR} ...'` env-bridge pattern.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh: rename AUTOFORMAT_PARALLEL to PARALLEL,
declare it at file start with default `true`; still env-overridable.
Reword the worker comment so the literal `# shellcheck` token no
longer appears at start-of-line and trips SC1072/SC1073.
* scripts/lint/galaxy.sh: drop the GALAXY_FATAL / GALAXY_NAMESPACE /
GALAXY_PARALLEL env vars (over-engineered for a single repo). Failure
is always fatal. NAMESPACE is an internal constant. Add the same
PARALLEL env var (default `true`, env-overridable); falsy demotes the
xargs fan-out to one worker.
* docs/contributing/environment/variables.md: collapse the four
AUTOFORMAT_*/GALAXY_* rows into one `PARALLEL` row covering both
consumers.
* docs/contributing/tools/make.md: refer to `PARALLEL=0` everywhere;
add the lint-galaxy parallelism note.
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Replace direct docker calls with container wrapper across scripts, roles, and templates
- Add support for cp, start, stop, and restart passthrough commands
- Align handlers and notifications to container manager restart semantics
- Fix systemd ExecStartPost to use container wrapper without hardcoded paths
- Silence ShellCheck SC2016 false positive for container exec password injection
- Update tests and documentation strings to reflect container-based workflow
https://chatgpt.com/share/69866958-c88c-800f-9b4f-3a972615bf42
- Move OpenResty restart logic into included handler tasks
- Use block/rescue to dump container logs on failure
- Tolerate restarting state during pre-flight validation
- Wait up to 2 minutes for container to reach running state
- Add explicit change detection for docker restart
- Enable assert mode in container deploy CLI
https://chatgpt.com/share/6948fa05-1970-800f-a010-59c8c3588713
- Replace separate handlers with atomic block/rescue handler
- Add pre- and post-flight OpenResty config validation
- Wait for container to reach running state before post-check
- Dump container logs automatically on failure for easier CI debugging
- Skip Let’s Encrypt tasks when SSL is disabled
https://chatgpt.com/share/6948fa05-1970-800f-a010-59c8c3588713
- Improved get_service_name filter plugin (clearer suffix handling, consistent var names).
- Added MODE_ASSERT flag to optionally execute validation/assertion tasks.
- Fixed systemd unit handling: consistent use of %I instead of %i, correct escaping of instance names.
- Unified on_failure behavior and alarm composer scripts.
- Cleaned up redundant logging, handlers, and debug config.
- Strengthened sys-service template resolution with assert (only active when MODE_ASSERT).
- Simplified timer and suffix handling with get_service_name filter.
- Hardened sensitive tasks with no_log.
- Added conditional asserts across roles (Keycloak, DNS, Mailu, Discourse, etc.).
These changes improve consistency, safety, and validation across the automation stack.
Conversation: https://chatgpt.com/share/68a4ae28-483c-800f-b2f7-f64c7124c274
