The compose flush recreates a dedicated postgres moments before this probe
runs, and its healthcheck sits on a 10s interval, so health=healthy cannot have
reported yet. The probe returned empty, the realign task below is gated on that
output being non-empty, and it skipped silently. The rotated POSTGRES_PASSWORD
therefore never reached the surviving data volume, and ./launcher rebuild
discourse died minutes later:
PG::ConnectionBad: connection to server at "192.168.9.2", port 5432 failed:
FATAL: password authentication failed for user "discourse"
Measured in the failing job: the engine started 01:29:44.757, postgres logged
"database system is ready to accept connections" at 01:29:46.737, and the probe
ran at 01:29:48.419 - 1.68s after the socket was already live, and 3.66s after
a start whose first healthcheck could not fire before ~10s.
Running is the predicate the operation needs, not a loosening.
set_postgres_superuser_password.sh reaches the cluster over the container local
socket by peer auth, which is exactly what lets a rotated credential be written
into an engine nobody can log into any more, and it carries its own readiness
loops: 30 x 30 x 2s around SHOW hba_file and 15 x 2s of TCP auth retry. Against
that 60s budget the observed start-to-ready times were 1.98s, 4.05s and 5.51s
across three engines, an 11x margin.
The other call site, sys-svc-compose post_deploy, already waits for healthy
before reaching this file, so running matches there too and its behaviour is
unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
da729cecec reads .Options.device when a volume declares a bind, on the premise that only compose volumes carry no .Options. A swarm node can hold the volume object without them anyway: in run 33923190220 the DR drill probe reported 'exposes no backing store for kix_kix_config' on the node that held it. The script then falls back to .Mountpoint, the node-local _data that is empty while the stack is down, finds no config there and exits 0.
Pass the inventory's own path as PATCH_DATA_DIR, the same DIR_VAR_LIB/<volume> that compose_volumes.py bakes into driver_opts.device, and try it when the volume-derived path holds nothing.
The second half is the asymmetry that hid this: a config present but unmatched fails loudly since 2c4e0e18b1, while a config found nowhere passed silently. The silent case is the dangerous one. KIX kept the pre-rotation password, its backend crash-looped on exit 255, and the converge gate only gave up 44 retries later. A total miss now prints SKIPPED with both paths it tried, and stays non-fatal so a first install still works.
The suite covered three of the seven consuming roles; it now covers all seven, plus the fallback and the named miss.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generic realign ran for every app whose database resolves to local
postgres and handed set_postgres_superuser_password.sh the app's entity name
as the superuser role. That pairing only holds for an engine sys-svc-rdbms
initdb'd itself, because templates/env/postgres.env.j2 fills POSTGRES_USER
from the same lookup.
BigBlueButton does not go down that path. Its postgres comes from the vendored
upstream compose and is initialised as superuser postgres, so
`psql -U bigbluebutton -d postgres -Atc 'SHOW hba_file;'` fails on every
attempt, the script's nested 30x30 retry burns its whole budget, and the task
ends in [FAIL]. In run 33574056480 that cost 32m59s of a Nextcloud variant-11
round, which reaches BBB as a dependency, and took the deploy with it.
main.yml already gates dedicated.yml on the app rendering its own
templates/compose.yml.j2. The realign now carries the same condition. It has to
live in this file rather than in main.yml, because sys-svc-compose's post_deploy
and sys-stk-backend include the task directly and never see main.yml's facts.
Measured against the current role set: 32 postgres roles can be local in some
round, 31 keep the realign, and BigBlueButton alone loses it - the only one
with a static shared: false and no compose template of its own, and the only
one carrying its own tasks/01_compose/postgres_password_sync.yml. No coverage
is lost, only a retry storm against a role that does not exist.
The stat carries the same compose-mode condition as the block, so a swarm round
runs exactly what it ran before, and it is run_once because the path does not
vary by host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
536b12e174 gated the realign on compose_handlers_flush, copying the flag that
decides who flushes rather than tracking whether the database needs realigning.
Those are different questions. `compose_handlers_flush: false` means "I flush
later myself" -- web-app-gitlab/tasks/00_core.yml:22 and web-app-matrix set it and
then flush on their own -- so for them the realign never ran at all.
Not a deadlock, not an ordering problem: the task was simply never scheduled.
Negative control from CI job 99670614257: `included: .../realign_dedicated_db.yml`
appears exactly three times, for openresty and flowise, and never for gitlab. Both
gitlab call sites report `false_condition: compose_handlers_flush | bool`.
What that costs, measured at the container in two independent jobs -- the tor one
(99670614346) and the compose one:
PostgreSQL Database directory appears to contain a database; Skipping initialization
2026-09-01 03:55:17.471 UTC [40] FATAL: password authentication failed for user "gitlab"
65 of them, 30s apart. gitlab-migrations never leaves /scripts/wait-for-deps,
webservice/sidekiq/workhorse never leave `created` because they depend on it with
service_completed_successfully, and `compose up` burns its full 1800s. The health
gate cannot see any of it: the engine's healthcheck is `pg_isready -U gitlab`,
which does not authenticate, so it reports healthy throughout.
17 roles pair a dedicated postgres with compose_handlers_flush: false and were
latently broken under rotation: baserow, bookwyrm, decidim, gitlab, kix, listmonk,
mastodon, matrix, mobilizon, n8n, odoo, openproject, opentalk, penpot, taiga,
xwiki, zammad.
The engine probe is required, not decoration. Un-gated, this task also runs on a
FIRST deploy where no engine exists yet, and set_postgres_superuser_password.sh
would spend ~30 minutes retrying before reporting [FAIL]. It is a no-op for the
other call site, post_deploy.yml:16, which already waits for healthy two lines
above it.
Verified by deploy, twice: `[CHANGED] New password did not work before, and works
now.` A skeptic's objection -- that the un-gate only helps when the previous pass
left the engine running -- was measured and does not hold: in both affected jobs
the engine was healthy 12-14 seconds BEFORE the task that got skipped.
Break risk measured rather than estimated. One deploy drove this across nine roles
and 18 invocations: 1 CHANGED, 1 UNCHANGED, 16 correctly skipped, 0 FAIL.
Still open: a dedicated postgres with no healthcheck yields an empty probe and
stays a silent no-op. Not a regression -- the task did not run there before either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Discourse#1 of run 33414875521 failed on a rebuild that could not log in:
PostgreSQL Database directory appears to contain a database; Skipping initialization
FATAL: password authentication failed for user "discourse"
Eight lines in the sidecar's log, all of that one class and nothing else. The
volume carried a cluster from an earlier run, so docker-entrypoint-initdb.d was
skipped and the role kept its pre-rotation password; the job log shows the launcher
being handed 8ilTTI... in the first pass and hgBFXE... in the second, exactly one
rotation apart.
The realignment for that exists and never ran. sys-stk-backend includes
sys-svc-compose with compose_handlers_flush hardcoded to false -- deliberately, so
the SSO-proxy sidecar can be rendered before anything flushes -- and then flushes
itself without the post-deploy step that up.yml pairs with its own flush. Counting
the rendered task names across the job: post_deploy ran for svc-net-tor and
svc-prx-openresty, both of which include sys-svc-compose directly, and for nobody
else. Discourse's two invocations both report
`false_condition: compose_handlers_flush | bool`.
Only the realignment is added back, not all of post_deploy. A first attempt included
the whole file and would have armed a 120s database-health wait and a swarm
convergence wait for the roles that reach compose through this wrapper; 46 roles set
compose_handlers_flush: true. The realignment itself is gated on compose + local +
postgres, which is 21 roles with a dedicated engine.
The task moves to sys-svc-rdbms/tasks/utils/, next to patch_persisted_password.yml.
Both halves of the same rotation work now live in the database role: one rewrites what
the application persisted, the other what the engine kept. sys-stk-backend calls it
after its flush because that is the only point where both facts hold -- the database
is this role's, and the stack is up. sys-svc-rdbms cannot call it itself: it runs
before sys-svc-compose and only notifies compose-up, so no container exists yet.
MariaDB needs none of this, because --init-file realigns it declaratively at start;
postgres has no equivalent for a cluster that already exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The body was a Jinja-interpolated shell string inside the task. As a file it
can be read, shellchecked and run, and the three parameters travel through the
environment the way the password already did.
That forces one thing into the open. The old expression carried
${PATCH_PASSWORD} and the shell expanded it while assembling the command; a
value arriving in an environment variable is not expanded a second time, so the
substitution has to be explicit. The consumers now write @PASSWORD@ and the
script replaces it, which is also the version a reader can follow without
knowing that rule.
The script refuses to run without its parameters rather than treating an unset
one as empty, because an empty volume name would look exactly like a node that
does not hold the volume: a silent success on every host.
Tests move with it, to the files/ segment every other test of a shipped script
uses. They run it against a stubbed `container` in three states per consumer -
no volume, volume without the config, both present - and assert the marker the
task reads as a change. Five mutations of the script were checked against them:
dropping the substitution, the marker, the existence test, the parameter guards
and the volume filter each turn the suite red.
Layout follows what the linters already require: files/shell/ is enforced by
role-files-layout, and 65e34746fc names ansible.builtin.script as the sanctioned
way to run one from a task.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The task name broke the 80-char convention; naming the consumer alone says as much, since the enclosing role already names the application.
The manifest read opts out of the shared read cache rather than joining it. utils.cache.files pulls in PROJECT_ROOT and the gitignore reader, which would break the import-free rule utils/recovery/manifest.py exists under, and a process-wide cache is wrong for this file in any case: a backup run writes a new generation that the same process then reads back, so the cached copy would be the stale one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baudolo dumps only what databases.csv names, and the entry was seeded from the central branch alone. An application running its own engine - Discourse with svc-db-postgres absent from the group - therefore never reached the seed: databases.csv stayed empty, baudolo warned that it was continuing without dumps, and the volume was stored as a raw PGDATA tree that restores a torn cluster.
Whether a database must be backed up is a property of there being one, not of it being central, so both branches now include the same helper with their own gate: centrally the host from the provider group seeds, dedicated the stack host itself. The coordinates come from the same lookup either way.
Verified without a deploy that the row this writes is the row baudolo looks for: the seed records the consumer entity as the instance, and baudolo derives the same key from the container name (discourse-database -> discourse). A mismatch there would have filled databases.csv and still produced no dump.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5