The compose flush recreates a dedicated postgres moments before this probe
runs, and its healthcheck sits on a 10s interval, so health=healthy cannot have
reported yet. The probe returned empty, the realign task below is gated on that
output being non-empty, and it skipped silently. The rotated POSTGRES_PASSWORD
therefore never reached the surviving data volume, and ./launcher rebuild
discourse died minutes later:
PG::ConnectionBad: connection to server at "192.168.9.2", port 5432 failed:
FATAL: password authentication failed for user "discourse"
Measured in the failing job: the engine started 01:29:44.757, postgres logged
"database system is ready to accept connections" at 01:29:46.737, and the probe
ran at 01:29:48.419 - 1.68s after the socket was already live, and 3.66s after
a start whose first healthcheck could not fire before ~10s.
Running is the predicate the operation needs, not a loosening.
set_postgres_superuser_password.sh reaches the cluster over the container local
socket by peer auth, which is exactly what lets a rotated credential be written
into an engine nobody can log into any more, and it carries its own readiness
loops: 30 x 30 x 2s around SHOW hba_file and 15 x 2s of TCP auth retry. Against
that 60s budget the observed start-to-ready times were 1.98s, 4.05s and 5.51s
across three engines, an 11x margin.
The other call site, sys-svc-compose post_deploy, already waits for healthy
before reaching this file, so running matches there too and its behaviour is
unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
da729cecec reads .Options.device when a volume declares a bind, on the premise that only compose volumes carry no .Options. A swarm node can hold the volume object without them anyway: in run 33923190220 the DR drill probe reported 'exposes no backing store for kix_kix_config' on the node that held it. The script then falls back to .Mountpoint, the node-local _data that is empty while the stack is down, finds no config there and exits 0.
Pass the inventory's own path as PATCH_DATA_DIR, the same DIR_VAR_LIB/<volume> that compose_volumes.py bakes into driver_opts.device, and try it when the volume-derived path holds nothing.
The second half is the asymmetry that hid this: a config present but unmatched fails loudly since 2c4e0e18b1, while a config found nowhere passed silently. The silent case is the dangerous one. KIX kept the pre-rotation password, its backend crash-looped on exit 255, and the converge gate only gave up 44 retries later. A total miss now prints SKIPPED with both paths it tried, and stays non-fatal so a first install still works.
The suite covered three of the seven consuming roles; it now covers all seven, plus the fallback and the named miss.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two branch-only commits each armed a defect in the compose-up handler
topic, and both failed in run 33785198875.
63f7f962bf registered the topic owner at the top of sys-svc-rdbms'
dedicated.yml so the notify two tasks below stopped aborting with
ERROR_ON_MISSING_HANDLER. The notification then survived to the
unconditional meta: flush_handlers at sys-svc-container/tasks/00_core.yml:53,
reached from sys-svc-compose's own 00_core before that role installs
BIN_COMPOSE, and long before 02_stack_host_prep installs BIN_CA_INJECT.
Matrix#1 died there on ca-inject with rc 2 after seven retries. That play
carries exactly one compose notification (job 100756596907, log line
11046), so closing this window is sufficient, not merely necessary.
dedicated.yml now enters 00_core.yml under the role's run-once flag
before it notifies, so the flush drains an empty queue and the flag then
makes main.yml skip the body on the real entry. The registration itself
stays ungated: gating it would skip the owner from the second app on.
95186163aa hoisted the sys-svc-compose include to sys-stk-backend
position 1. The consumer therefore rendered and notified compose-up
before sys-svc-objstore lazily loaded the provider app role, whose own
unconditional flush consumed the notification and ran the single
registered handler in the provider's chdir. mastodon and peertube never
got a stack, in compose and in swarm alike; the victim's own guard task
appears zero times in all four logs. Restoring the include below rdbms,
objstore and engine puts the notify after every foreign crossing. Two
pre-hoist runs in exactly that order are green on record.
The test keeps its cases and gains one. It selects the render include by
the absence of tasks_from instead of by first occurrence, so an early
registration-only include stays legal while a hoisted render still fails.
The new case pins the bootstrap: entry point, position before the first
notify, and the run-once guard whose removal would let 01_reset.yml tear
down DIR_COMPOSITIONS once per dedicated-DB app.
Same class, pre-existing, not fixed here: svc-db-qdrant notifies and
flushes three tasks before its first compose entry; web-app-magento pairs
a dedicated database with a shared object store; web-app-seaweedfs still
flushes a topic it does not own; svc-runner enters 00_core.yml unguarded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The name added in 63f7f962b was 112 characters and failed
tests/lint/repository/test_name_emoji_and_length.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dedicated.yml notifies compose-up but registers no handler for it, so a
caller that reaches sys-svc-rdbms before any compose consumer has loaded
the sys-svc-compose block dies with "The requested handler 'compose-up'
was not found" and no PLAY RECAP. 95186163aa guarded only the
sys-stk-backend caller; web-app-matrix's ansible flavor includes
sys-svc-rdbms directly in 03_database.yml and still hit it as the first
compose consumer of its play (run 33751742070, job 100643400267).
Register the owner as the first task of the notifying file. In the
sys-stk-backend path this lands before the pre-flush rebind, so the app's
copy stays the last registration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generic realign ran for every app whose database resolves to local
postgres and handed set_postgres_superuser_password.sh the app's entity name
as the superuser role. That pairing only holds for an engine sys-svc-rdbms
initdb'd itself, because templates/env/postgres.env.j2 fills POSTGRES_USER
from the same lookup.
BigBlueButton does not go down that path. Its postgres comes from the vendored
upstream compose and is initialised as superuser postgres, so
`psql -U bigbluebutton -d postgres -Atc 'SHOW hba_file;'` fails on every
attempt, the script's nested 30x30 retry burns its whole budget, and the task
ends in [FAIL]. In run 33574056480 that cost 32m59s of a Nextcloud variant-11
round, which reaches BBB as a dependency, and took the deploy with it.
main.yml already gates dedicated.yml on the app rendering its own
templates/compose.yml.j2. The realign now carries the same condition. It has to
live in this file rather than in main.yml, because sys-svc-compose's post_deploy
and sys-stk-backend include the task directly and never see main.yml's facts.
Measured against the current role set: 32 postgres roles can be local in some
round, 31 keep the realign, and BigBlueButton alone loses it - the only one
with a static shared: false and no compose template of its own, and the only
one carrying its own tasks/01_compose/postgres_password_sync.yml. No coverage
is lost, only a retry storm against a role that does not exist.
The stat carries the same compose-mode condition as the block, so a swarm round
runs exactly what it ran before, and it is run_once because the path does not
vary by host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
536b12e174 gated the realign on compose_handlers_flush, copying the flag that
decides who flushes rather than tracking whether the database needs realigning.
Those are different questions. `compose_handlers_flush: false` means "I flush
later myself" -- web-app-gitlab/tasks/00_core.yml:22 and web-app-matrix set it and
then flush on their own -- so for them the realign never ran at all.
Not a deadlock, not an ordering problem: the task was simply never scheduled.
Negative control from CI job 99670614257: `included: .../realign_dedicated_db.yml`
appears exactly three times, for openresty and flowise, and never for gitlab. Both
gitlab call sites report `false_condition: compose_handlers_flush | bool`.
What that costs, measured at the container in two independent jobs -- the tor one
(99670614346) and the compose one:
PostgreSQL Database directory appears to contain a database; Skipping initialization
2026-09-01 03:55:17.471 UTC [40] FATAL: password authentication failed for user "gitlab"
65 of them, 30s apart. gitlab-migrations never leaves /scripts/wait-for-deps,
webservice/sidekiq/workhorse never leave `created` because they depend on it with
service_completed_successfully, and `compose up` burns its full 1800s. The health
gate cannot see any of it: the engine's healthcheck is `pg_isready -U gitlab`,
which does not authenticate, so it reports healthy throughout.
17 roles pair a dedicated postgres with compose_handlers_flush: false and were
latently broken under rotation: baserow, bookwyrm, decidim, gitlab, kix, listmonk,
mastodon, matrix, mobilizon, n8n, odoo, openproject, opentalk, penpot, taiga,
xwiki, zammad.
The engine probe is required, not decoration. Un-gated, this task also runs on a
FIRST deploy where no engine exists yet, and set_postgres_superuser_password.sh
would spend ~30 minutes retrying before reporting [FAIL]. It is a no-op for the
other call site, post_deploy.yml:16, which already waits for healthy two lines
above it.
Verified by deploy, twice: `[CHANGED] New password did not work before, and works
now.` A skeptic's objection -- that the un-gate only helps when the previous pass
left the engine running -- was measured and does not hold: in both affected jobs
the engine was healthy 12-14 seconds BEFORE the task that got skipped.
Break risk measured rather than estimated. One deploy drove this across nine roles
and 18 invocations: 1 CHANGED, 1 UNCHANGED, 16 correctly skipped, 0 FAIL.
Still open: a dedicated postgres with no healthcheck yields an empty probe and
stays a silent no-op. Not a regression -- the task did not run there before either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Discourse#1 of run 33414875521 failed on a rebuild that could not log in:
PostgreSQL Database directory appears to contain a database; Skipping initialization
FATAL: password authentication failed for user "discourse"
Eight lines in the sidecar's log, all of that one class and nothing else. The
volume carried a cluster from an earlier run, so docker-entrypoint-initdb.d was
skipped and the role kept its pre-rotation password; the job log shows the launcher
being handed 8ilTTI... in the first pass and hgBFXE... in the second, exactly one
rotation apart.
The realignment for that exists and never ran. sys-stk-backend includes
sys-svc-compose with compose_handlers_flush hardcoded to false -- deliberately, so
the SSO-proxy sidecar can be rendered before anything flushes -- and then flushes
itself without the post-deploy step that up.yml pairs with its own flush. Counting
the rendered task names across the job: post_deploy ran for svc-net-tor and
svc-prx-openresty, both of which include sys-svc-compose directly, and for nobody
else. Discourse's two invocations both report
`false_condition: compose_handlers_flush | bool`.
Only the realignment is added back, not all of post_deploy. A first attempt included
the whole file and would have armed a 120s database-health wait and a swarm
convergence wait for the roles that reach compose through this wrapper; 46 roles set
compose_handlers_flush: true. The realignment itself is gated on compose + local +
postgres, which is 21 roles with a dedicated engine.
The task moves to sys-svc-rdbms/tasks/utils/, next to patch_persisted_password.yml.
Both halves of the same rotation work now live in the database role: one rewrites what
the application persisted, the other what the engine kept. sys-stk-backend calls it
after its flush because that is the only point where both facts hold -- the database
is this role's, and the stack is up. sys-svc-rdbms cannot call it itself: it runs
before sys-svc-compose and only notifies compose-up, so no container exists yet.
MariaDB needs none of this, because --init-file realigns it declaratively at start;
postgres has no equivalent for a cluster that already exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1f870b1e5 gave the patch one call site and put it in post_deploy.yml, which runs
after the deploy has waited for the stack. 3831c8f804 had named the constraint in
its own subject line: before the health gate. An app whose health check
authenticates against its own database fails that check with the password it
persisted, compose gives up with "dependency failed to start: container matomo is
unhealthy", and nothing behind the gate ever runs -- including the patch that
would have fixed it.
Measured, not argued: with the call in post_deploy.yml a local compose deploy of
five roles passed its first pass at ok=4571 failed=0 and died in the second, the
one that carries the rotated credential. With the call back in front, all three
passes report failed=0 and the Playwright specs pass in both.
sys-svc-rdbms/tasks/main.yml is where it goes: the stack include reaches it
before anything comes up, it is per-consumer rather than per-role-once, and it
still resolves the declaration through the same path, so there is still exactly
one call site. The patch reads the volume from the host and needs no running
container, which is what makes the earlier position possible at all; before the
first deploy has created the volume it exits without doing anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The body was a Jinja-interpolated shell string inside the task. As a file it
can be read, shellchecked and run, and the three parameters travel through the
environment the way the password already did.
That forces one thing into the open. The old expression carried
${PATCH_PASSWORD} and the shell expanded it while assembling the command; a
value arriving in an environment variable is not expanded a second time, so the
substitution has to be explicit. The consumers now write @PASSWORD@ and the
script replaces it, which is also the version a reader can follow without
knowing that rule.
The script refuses to run without its parameters rather than treating an unset
one as empty, because an empty volume name would look exactly like a node that
does not hold the volume: a silent success on every host.
Tests move with it, to the files/ segment every other test of a shipped script
uses. They run it against a stubbed `container` in three states per consumer -
no volume, volume without the config, both present - and assert the marker the
task reads as a change. Five mutations of the script were checked against them:
dropping the substitution, the marker, the existence test, the parameter guards
and the volume filter each turn the suite red.
Layout follows what the linters already require: files/shell/ is enforced by
role-files-layout, and 65e34746fc names ansible.builtin.script as the sanctioned
way to run one from a task.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b067d21ef shipped two ways to kill a dedicated MariaDB, both reproduced.
The file was rendered 0640 under a play that runs as root, while
stack_host_template otherwise defaults to 0644. Compose emulates a config as a
plain host bind and ignores uid, gid and mode on the reference, so the engine,
which does not run as root, could not open the file. mariadbd treats an
unopenable init file as fatal and aborts every start. Dropping the explicit mode
puts it back on 0644, next to the env file that carries the identical password.
On a fresh data directory the entrypoint starts a temporary server with the same
arguments, so the init file ran before the database was ever set up. Giving root
a password there breaks the entrypoint's own readiness probe, which connects
without one, and the schema and grants are then never created. The CREATE USER
had the same problem from the other side: the entrypoint's own CREATE USER
carries no IF NOT EXISTS and the script runs under set -e.
What remains is the one statement the failure actually needs: the application
user, guarded by IF EXISTS so it is a no-op on the first start and a
realignment on every later one. That is the identity the health check
authenticates as.
Root keeps the password its data directory was initialised with. It did so
before this feature existed, nothing in the repository authenticates as root
against a dedicated engine, and the account is reachable only from inside the
app's own stack network. Realigning it needs a gate that can tell the
entrypoint's temporary server apart from the real one, which this does not have.
The GRANT is gone with it: rotating a credential changes passwords, not
privileges.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An engine of its own keeps its passwords in its data directory, written once
when that directory was initialised. When the deploy rotates the credential the
new value reaches the container's environment and never the engine, so the
health check authenticates with a password nothing has ever accepted. The
application user and root come from the same value, so nothing is left that
could log in and change it either.
mariadbd reads --init-file on every start, before it opens the socket, and runs
the statements with full privileges. That is the one point where a password can
be reset without knowing the previous one.
The SQL is rendered next to the env file that already holds the same secret in
plaintext, so it adds no new exposure. It reaches the container as a config
rather than a bind mount because a swarm task may be scheduled on a node the
rendered path does not exist on; the config name carries a hash of the file
content, so a rotated password produces a new name and swarm replaces the
immutable old one.
The guard is mariadb-and-not-shared, which matches the `local` guard the service
template already renders on, so the block and the top-level entry appear
together or not at all.
3831c8f804 fixed the other half of this failure: matomo, moodle and nextcloud
keep the same password in a config file on their data volume. Same rotation,
opposite end of the connection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
path_absolute builds from playbook_dir, a controller-side value, but the task
carried no delegate_to and therefore ran on the managed node. Anywhere the
controller is not the target - every swarm node, every remote deploy - the
path does not exist, stat.exists is false and the dedicated branch is skipped
for every role, taking the env file, the initdb setup and the backup seed with
it. Silently, while the rendered service still points env_file at a file
nobody wrote.
A single-host compose run cannot see this: there the controller and the target
are the same machine, which is why 47c01259e looked green.
Delegating alone is enough. run_once would additionally hand one host's answer
to every other host of a `hosts: all` play whose include is gated per host,
making the registered result depend on host ordering rather than on the host
being configured - for a filesystem check that is identical for all of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
47c01259e named the task 87 characters wide; the convention allows 80 so the
name stays readable in a playbook run. The lint that caught it also says where
a longer explanation belongs, and it is not the task name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A dedicated database is not the same as one this role provides.
web-app-bigbluebutton declares postgres enabled and unshared, so the lookup
reports it local and the dedicated branch ran - rendering an env file, an
initdb setup and a backup row for an engine bbb-docker brings itself. Nothing
of that output is read: bigbluebutton never calls the database lookup, ships
its own env.j2 and vendors its own initdb hook.
The backup row was the one that hurt. 7d044c265 taught the dedicated branch to
seed, and it seeded credentials derived from the entity name into a postgres
that has no such role:
pg_dump -U bigbluebutton -d bigbluebutton
FATAL: role "bigbluebutton" does not exist
bigbluebutton seeds its own row correctly - the '*' cluster row with the
superuser its image actually creates - so the drill now has one working row
and one impossible one for the same instance.
Whether this role provides the engine is already written down: its service
template is reached through <role>/templates/compose.yml.j2 -> base.yml.j2, so
a role without that template never gets one. Across the 53 roles with a
database service exactly one lacks it, and none has it without including
base.yml.j2, which makes the condition exact rather than a stand-in for "is
this bigbluebutton". Reading it beats declaring it: a flag would be a second
copy of a fact the tree already states, free to drift from it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The task name broke the 80-char convention; naming the consumer alone says as much, since the enclosing role already names the application.
The manifest read opts out of the shared read cache rather than joining it. utils.cache.files pulls in PROJECT_ROOT and the gitignore reader, which would break the import-free rule utils/recovery/manifest.py exists under, and a process-wide cache is wrong for this file in any case: a backup run writes a new generation that the same process then reads back, so the cached copy would be the stale one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baudolo dumps only what databases.csv names, and the entry was seeded from the central branch alone. An application running its own engine - Discourse with svc-db-postgres absent from the group - therefore never reached the seed: databases.csv stayed empty, baudolo warned that it was continuing without dumps, and the volume was stored as a raw PGDATA tree that restores a torn cluster.
Whether a database must be backed up is a property of there being one, not of it being central, so both branches now include the same helper with their own gate: centrally the host from the provider group seeds, dedicated the stack host itself. The coordinates come from the same lookup either way.
Verified without a deploy that the row this writes is the row baudolo looks for: the seed records the consumer entity as the instance, and baudolo derives the same key from the container name (discourse-database -> discourse). A mismatch there would have filled databases.csv and still produced no dump.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A handler name was doing two jobs at once: the line an operator reads in the
play output, and the identifier a `notify:` binds to. The two pull in opposite
directions -- the log wants "restart the SSH service, with diagnostics", the
binding wants a token that survives quoting, YAML flow style and a shell
round-trip -- and a name that changes for readability silently changes which
handler runs.
Every handler that a notify reached by name now carries a `listen:` with that
exact wording, so the binding is explicit before anything is renamed. The
topics themselves become identifiers: `compose up` reads `compose-up`,
`Restart NFS ganesha` reads `restart-nfs-ganesha`, 22 of them across both
sides of every notify. Where the topic is assembled at runtime, both branches
moved together: the conditional in sys-service carries two literals, and the
LDAP renderer interpolates `import-{{ folder }}-ldif-files`, which is what
makes those handlers reachable at all.
Freed from the binding, the names take the emoji prefix every other task line
already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_task_file_numbering.py: numbered task
files use exactly two leading digits, a numbered *_core.yml must be
00_core.yml, per directory the numbers are unique/consecutive and start
at 01 (00 with a core file), numbered includes appear in ascending
order, and shared helpers live in tasks/utils/ instead of the numbered
chain.
Apply it across every role: rename NN_core.yml to 00_core.yml with a
consecutive down-shift of the siblings, move letter-suffix children into
a subdirectory named after their parent task, relocate helpers included
from more than one place into tasks/utils/, and update every reference
(include_tasks, path_absolute, .ansible-lint-ignore, requirement docs).
Legacy narration comments in the touched files are deleted; the upstream
xwiki.properties reference block opts out via nocheck. The unused
MAILU_IP6_PUBLIC var (only ever named in now-removed comments) is dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename the role, its test tree, the primary entity key
(volume-2-local), the derived vars (BKP_VOLUME_2_LOCAL_*,
SYS_SERVICE_BACKUP_VOLUME_2_LOCAL), the run-once fact, and every
group_names consumer flag, include path, doc, and lint reference. The
container_backup service key and the on-disk backup-docker-to-local
tree keep their names: both are contracts with existing consumer
entries, snapshot trees, and the baudolo pull tooling.
Operators must rename the svc-bkp-container-2-local inventory group and
disable the previous versioned units on already-provisioned hosts; the
pre-deploy snapshot glob derives its base from the new role name and
will not find them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A consumer that forces a per-role deploy mode (web-app-matrix ansible-flavor and web-app-bigbluebutton set compose_mode_force=compose) leaked that value through the include_role chain into the shared database provider (svc-db-postgres/mariadb). In a swarm cluster the provider then ran via 'docker compose up' (bridge) instead of 'docker stack deploy' (overlay), breaking the shared DB. Reset compose_mode_force to empty on the provider include, alongside the existing compose_repository_pull override, so the shared provider always follows the cluster DEPLOYMENT_MODE. Covers every consumer, not just matrix/bbb.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce plugins/lookup/path_absolute.py as the single source for the
repo-root anchor: lookup('path_absolute', 'roles/...') returns an absolute
path under playbook_dir, joining every term. Replace all hand-built
playbook_dir path constructions across .yml/.yaml and .j2 templates with it:
[playbook_dir, '<rel>'] | path_join -> lookup('path_absolute', '<rel>')
playbook_dir ~ '/<rel>' -> lookup('path_absolute', '<rel>')
{{ playbook_dir }}/<rel> -> lookup('path_absolute', '<rel>')
Unrelated repo_path symbols in cli/meta/roles/applications/resolution and the
single bare discover_playwright_roles(playbook_dir) call are left untouched.
Add tests/lint/ansible/test_playbook_dir_via_lookup.py: fails the build if any
of the three construction forms reappears in .yml/.yaml/.j2, pointing callers
at the lookup. Unit test covers the plugin.
Touching these files pulled their pre-existing narration comments into
test_comments_valid scope; clean them (delete restatement/banners, keep real
trip-wires as Workaround/Upstream/Invariant markers). Drop caution, important
and pitfall from the comment-lint marker vocabulary and remove the comments
that used them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolves CI swarm rc=69 ("msmtp: needs a password"): controller-side
Ansible lookups (lookup('email').password etc.) couldn't see the
tokens.yml that Mailu writes target-side, because the controller has
its own filesystem in swarm mode.
Mount NFS-shared /var/lib/infinito on every swarm host AND on the
controller during deploy (in-play delegate_to: localhost task at end
of svc-storage-nfs-server role). State subdir exported separately
with no_root_squash + wildcard CIDRs so root-owned writes succeed
and the controller can mount from outside the svc-swarm IP set.
Single-writer guarantee on FILE_TOKENS via delegate_to: STACK_HOST +
run_once on sys-token-store/{01_core,write}.yml. Backup roles
(svc-bkp-container-2-local, sys-ctl-cln-bkps, sys-ctl-cln-faild-bkps,
user-backup/03_permissions_folders, three direct callers of
04_seed-database-to-backup.yml) gated on IS_STACK_HOST to prevent
N-host write races on the shared NFS backup tree.
Fail-fast assertion in svc-storage-nfs-client + 05_controller_mount
when pre-existing local /var/lib/infinito would be hidden by mount-
over. Operator must migrate or wipe before re-deploying.
SPOTs: SWARM_LAB_NETWORK in workflow env block, storage.nfs.state_subdir
in group_vars/all/15_storage.yml, DIR_VAR_LIB unchanged. UPPER_CASE
constants in NFS roles. Cleanup script umounts controller view before
container teardown to prevent stale-mount leak between matrix jobs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds MODE_ASSERT_LOCAL to group_vars/all/01_modes.yml:
MODE_ASSERT_LOCAL: "{{ (MODE_ASSERT | bool) and (IS_STACK_HOST | bool) }}"
The mailu preload-assertion in sys-svc-mail/tasks/01_core.yml previously
fired on every host in `'web-app-mailu' in group_names` with no
MODE_ASSERT gate at all, so swarm workers ran into the assertion every
single CI run even though the preload state lives only on the stack
host. Same shape applies to a dozen other assertions: sys-svc-rdbms
(already double-gated, now collapsed), keycloak's kcadm-driven LDAP /
RBAC / reCAPTCHA / groups-path asserts, the sys-svc-compose handler
validators for compose/swarm config, svc-prx-openresty's 02_assert,
sys-svc-webserver-core's 05_assert, web-app-matomo's bootstrap token
check, web-app-joomla's 05_assert, and the sys-ctl-alm-compose notifier
self-start.
Each asserted state (run_once_<role> flags, kcadm output, openresty
container probes, compose-tree files) lives only on IS_STACK_HOST, so
the new SPOT keeps the assertion semantics intact while skipping the
spurious worker-side failures.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The unconditional run_once_<provider> assertion fires on every swarm
deploy where sys-service-loader's loop iteration order does not strictly
honour bucket order (a known race when include_tasks runs across mixed
host scopes). MODE_ASSERT keeps the check available in debug runs to
surface the underlying loader race, while IS_STACK_HOST scopes it to
the host where the provider would actually have been preloaded. In
production the immediately following Initialize task lets svc-db-<x>
self-heal via its own run_once gate, so the assertion is no longer
load-bearing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
sys-svc-rdbms now decides central-DB vs dedicated-DB mode from the provider
group's presence in the inventory (groups[provider_id] | length > 0)
instead of the per-host lookup('database', _, 'shared'). The old form
returned different values on different hosts (e.g. mariadb.shared =
'svc-db-mariadb' in group_names), so workers would silently disagree with
the manager about which mode is active and trip the central-init handlers
on the wrong host. The central-init block now also gates on the host
actually being in the provider group.
sys-stk-full guards the sys-stk-front-proxy include behind
"inventory_hostname in groups['svc-prx-openresty']", so under swarm only
the openresty host renders the per-app vhost. Combined with the
compose-mode auto-place from the previous commit, single-host compose and
swarm both pick exactly one host.
tasks/utils/load_app.yml skips loading a role for hosts that are not in
its target group when a group exists, matching the new gating semantic
upstream.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Renames the shared-services loader role from sys-utils-service-loader to
sys-service-loader (shorter, no orphaned "utils" prefix; entity name in
meta/services.yml realigns to service-loader per get_entity_name).
Introduces an explicit two-phase contract in the loader: after the
ordered preload pass, flush a `_service_loader_post_run` queue that
service roles register into during preload via the new helper
`list_or_shoot.yml` (list during preload, shoot directly outside it).
Replaces the `when: not _service_loader_preloading` guards in
sys-svc-compose and sys-svc-webserver-core with helper calls. Closes
the gap where non-web deploys would never load these post-load
dependencies; in normal web deploys behavior stays the same thanks to
each sub-role's `run_once_<role>` gate.
Docs, comments, test docstring, and constructor-stage invocation
follow the rename. Historical changelogs (CHANGELOG.md, packaging/*)
intentionally left at the original name.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidates the backup-role topology around four directional roles
under the existing svc.bkp category:
sys-ctl-bkp-docker-2-loc → svc-bkp-container-2-local
svc-bkp-loc-2-usb → svc-bkp-local-2-device
svc-bkp-rmt-2-loc → svc-bkp-remote-2-local
sys-bkp-provider → deleted (purely a bundler with no own logic;
the two pieces it pulled in — user-backup
and sys-timer-cln-bkps — are now declared
directly by callers that need them)
Every renamed role moves into svc.bkp (already invokable per
roles/categories.yml), gets an `application_id` in vars/main.yml, and
declares the per-app primary-entity service flags + resource limits the
invokable-role lint requires.
Companion changes:
- svc-bkp-container-2-local: meta/variants.yml declares three matrix
rounds (gitea+keycloak, postgres+mariadb, openldap) that pin the
companion topology via service-key flags the CombinedResolver
translates into provider-role includes.
- sys-svc-compose: container-2-local is no longer in the docker-mgmt
include loop; it joins via a separate `'svc-bkp-container-2-local'
in group_names` gate so the 3 backup roles are deployable
independently (each addresses its own failure mode).
- The three `04_seed-database-to-backup.yml` includes (web-app-matrix,
web-app-bigbluebutton, sys-svc-rdbms) are gated by the same
group_names check so consumer apps only seed when the container
backup is actually in scope.
- web-app-gitea picks up `enabled: true, shared: true, provides:
gitea` on its primary entity so the service registry can pull it in
via a `services.gitea.enabled+shared` flag (needed for the
container-backup variant 0).
- group_vars constants and the pyproject ruff per-file-ignore + the
.ansible-lint-ignore paths follow the role-id rename.
- The remote-2-local script picks up the snake_case rename
(`pull-specific-host.py` → `pull_specific_host.py`) and the
`os.makedirs` → `Path.mkdir` conversion so the role no longer needs
the PTH/N999 ruff escape hatch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Generalises the postgres image build to be driven by a declarative
extension→library registry instead of a hardcoded POSTGRES_VECTOR_ENABLED
flag. Same Dockerfile.j2 now serves both the central svc-db-postgres
image (libraries aggregated across all deployed consumers) and per-
consumer images for shared:false postgres consumers whose extensions
need libraries not in the postgis/postgis base (e.g. pgvector).
postgres.yml.j2 switches between image: and build: based on whether
custom libraries are needed. New postgres_libraries lookup filters
base-image extensions out and returns recipes for the rest. Dedicated
task path extracted into dedicated.yml + dedicated_postgres.yml so the
block-with-when size limit stays satisfied and the postgres-type check
happens once.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
sys-svc-rdbms now renders an extensions.sql into a per-consumer
.initdb.d/ directory and bind-mounts it into the dedicated postgres
container at /docker-entrypoint-initdb.d/. CREATE EXTENSION runs as
POSTGRES_USER (= bootstrap superuser) on first initdb so non-trusted
extensions succeed. Same declarative source (service_extensions
lookup, services.yml extensions field) drives both shared:true
(02_init.yml loop) and shared:false (initdb.d) paths. database lookup
exposes a new initdb_dir field; service_extensions lookup loses its
shared:true guard (shared:false is now a real path). Refactored
repeated lookups in sys-svc-rdbms tasks into vars/main.yml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure 1:1 rename (60 insertions / 60 deletions across 30 files) to
align the variable names with the role that owns them and to drop the
misleading ``docker_git_`` prefix — these knobs configure the
compose-stack's git clone step, not the docker CLI:
- docker_git_repository_address → compose_repository_url
- docker_git_repository_version → compose_repository_ref
- docker_git_repository_pull → compose_pull_repository
- docker_repository_path → compose_repository_path
``ref`` better captures git refs (branches/tags/SHAs are not
"versions"); ``url`` is the idiomatic name for the clone source.
Touches every consumer (web-app-akaunting/bigbluebutton/dashboard/
fediwall/hugo/littlejs/mini-qr/minio/openproject/openwebui/
roulette-wheel/taiga + svc-db-openldap/svc-prx-openresty/sys-svc-rdbms
+ web-svc-collabora/coturn/logout/onlyoffice + sys-front-inj-all), the
role's own defaults/tasks, the bigbluebutton compose_mods filter and
its unit test, the declared-in-services lint, and the hugo
requirements doc.
Breaking change for any external override that still sets the old
names — no alias is provided.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduce a single grammar for per-line, per-block, and per-file
suppression markers across the test suite, replacing seven ad-hoc
syntaxes that had drifted apart.
Helper
utils/annotations/suppress.py exposes is_suppressed_at,
is_suppressed_in_head, is_suppressed_anywhere, line_has_rule, and
suppressed_line_numbers. Markers may use either `noqa` or `nocheck`
(case-insensitive synonyms), live inside `#`, `//`, `{# ... #}`, or
`<\!-- ... -->` comments, and combine multiple kebab-case rules
comma-separated. 17 unit tests cover the grammar in isolation.
Refactored consumers
- tests/external/repository/test_urls_reachable.py (rule: url)
- tests/external/docker/test_image_versions.py via
utils/docker/version_updater.py (rule: docker-version)
- tests/lint/repository/test_no_direct_yaml_calls.py (rule:
direct-yaml)
- tests/lint/ansible/test_service_shared_consistency.py (rule:
shared)
- tests/lint/ansible/roles/test_web_app_email_integration.py (rule:
email)
- tests/lint/repository/test_python_file_size.py (rule: file-size)
- tests/lint/ansible/test_run_once_tags.py +
tests/integration/roles/run_once/test_schema.py (rules: run-once,
run-once-suffix)
Marker migration
12 role tasks/main.yml files: `# run_once_<key>: deactivated`
-> `# nocheck: run-once`. roles/web-app-suitecrm/meta/services.yml
drops a malformed inline `noqa: intentional legacy pin` from a TODO
comment so the new parser does not pick it up as a stray rule.
Caching wins
Test files that previously loaded YAML / read files via raw
yaml.safe_load + Path.read_text now route through utils.cache.yaml
(load_yaml_any / load_yaml_str) and utils.cache.files.read_text:
- test_service_shared_consistency.py
- test_web_app_email_integration.py
- tests/integration/roles/run_once/test_schema.py
- utils/docker/version_updater.py
This both speeds up repeated reads and stays consistent with the
direct-yaml lint policy.
Documentation
New SPOT page at docs/contributing/actions/testing/suppression.md
with grammar, position semantics, the catalog of rules, examples,
and an "adding a new rule" checklist. lint.md and external.md drop
their per-rule tables and link to the new page. All touched docs
conform to docs/contributing/documentation.md (no em-dash thought
breaks, no SPOT self-labels, RFC-2119 keywords used consistently).
Settings sort tooling
scripts/lint/sort_claude_settings.sh wraps utils/claude/settings_sort.py
and is wired into scripts/lint/autoformat.sh. The tooling fixes the
ASCII-ascending requirement enforced by
tests/lint/repository/test_claude_settings_sorted.py; the test's
failure message now points operators at the script. .claude/settings.json
is committed in sorted form.
- Delete group_vars/all/20_services.yml; all service metadata now lives
in each role's config/main.yml under compose.services.<name>
- Rename generic 'database' service key to explicit 'mariadb'/'postgres'
across all app roles; remove legacy type: field
- Add utils/service_registry.py and plugins/lookup/service_registry.py
as the single source of truth for service discovery
- Add utils/database_service.py to centralize DB service resolution
- Delete service_load_enabled filter and service_should_load lookup;
replaced by service lookup plugin
- Update sys-utils-service-loader to run in constructor stage and use
generic load_service.yml; remove load_frontend_service.yml
- Add _service_loader_preloading guard to sys-svc-compose,
sys-svc-webserver-core and sys-ctl-alm-compose to prevent
health-checks/alarms from firing during preload phase
- Replace manual load_app.yml calls for shared DB/Mailu with assert-
based preload validation in sys-svc-rdbms and sys-svc-mail
- Update networks.yml.j2 templates to resolve LDAP/AI networks via
service_registry instead of hardcoded role names
- Migrate all tests to new service discovery API; delete obsolete tests
- Update docs/contributing/code/services.md to reflect role-local model
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename lookup plugin from `docker` to `container`
- Switch all lookups from `lookup('docker', ...)` to `lookup('container', ...)`
- Normalize compose file keys:
- docker_compose → compose
- docker_compose_override → compose_override
- docker_compose_ca_override → compose_ca_override
- Centralize compose path resolution via docker_paths_utils
- Update compose_file_args to always derive paths from DIR_COMPOSITIONS
- Align all roles, templates, and vars with container/compose naming
- Update and rename unit tests accordingly
This prepares the codebase for engine-agnostic container backends
(e.g. Docker ↔ Podman) and removes legacy docker-specific assumptions.
https://chatgpt.com/share/6987a7cc-d374-800f-99ad-bb4c8f92c59e
- Replace lookup('docker', ..., want='...') with lookup('docker', ..., 'path')
- Allow both top-level keys (directories/files) and dotted paths
- Simplify docker lookup API and reduce lazy dict propagation
- Update unit tests to reflect new lookup semantics
https://chatgpt.com/share/6984ccea-8290-800f-a6d7-ec0e94df95a0
Remove implicit docker_compose variable usage and resolve all docker paths
(files, directories, env) via lookup('docker', application_id).
This prevents scope leakage, centralizes path resolution, and makes docker
context deterministic across roles and handlers.
https://chatgpt.com/share/6984ccea-8290-800f-a6d7-ec0e94df95a0
