The compose flush recreates a dedicated postgres moments before this probe
runs, and its healthcheck sits on a 10s interval, so health=healthy cannot have
reported yet. The probe returned empty, the realign task below is gated on that
output being non-empty, and it skipped silently. The rotated POSTGRES_PASSWORD
therefore never reached the surviving data volume, and ./launcher rebuild
discourse died minutes later:
PG::ConnectionBad: connection to server at "192.168.9.2", port 5432 failed:
FATAL: password authentication failed for user "discourse"
Measured in the failing job: the engine started 01:29:44.757, postgres logged
"database system is ready to accept connections" at 01:29:46.737, and the probe
ran at 01:29:48.419 - 1.68s after the socket was already live, and 3.66s after
a start whose first healthcheck could not fire before ~10s.
Running is the predicate the operation needs, not a loosening.
set_postgres_superuser_password.sh reaches the cluster over the container local
socket by peer auth, which is exactly what lets a rotated credential be written
into an engine nobody can log into any more, and it carries its own readiness
loops: 30 x 30 x 2s around SHOW hba_file and 15 x 2s of TCP auth retry. Against
that 60s budget the observed start-to-ready times were 1.98s, 4.05s and 5.51s
across three engines, an 11x margin.
The other call site, sys-svc-compose post_deploy, already waits for healthy
before reaching this file, so running matches there too and its behaviour is
unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generic realign ran for every app whose database resolves to local
postgres and handed set_postgres_superuser_password.sh the app's entity name
as the superuser role. That pairing only holds for an engine sys-svc-rdbms
initdb'd itself, because templates/env/postgres.env.j2 fills POSTGRES_USER
from the same lookup.
BigBlueButton does not go down that path. Its postgres comes from the vendored
upstream compose and is initialised as superuser postgres, so
`psql -U bigbluebutton -d postgres -Atc 'SHOW hba_file;'` fails on every
attempt, the script's nested 30x30 retry burns its whole budget, and the task
ends in [FAIL]. In run 33574056480 that cost 32m59s of a Nextcloud variant-11
round, which reaches BBB as a dependency, and took the deploy with it.
main.yml already gates dedicated.yml on the app rendering its own
templates/compose.yml.j2. The realign now carries the same condition. It has to
live in this file rather than in main.yml, because sys-svc-compose's post_deploy
and sys-stk-backend include the task directly and never see main.yml's facts.
Measured against the current role set: 32 postgres roles can be local in some
round, 31 keep the realign, and BigBlueButton alone loses it - the only one
with a static shared: false and no compose template of its own, and the only
one carrying its own tasks/01_compose/postgres_password_sync.yml. No coverage
is lost, only a retry storm against a role that does not exist.
The stat carries the same compose-mode condition as the block, so a swarm round
runs exactly what it ran before, and it is run_once because the path does not
vary by host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
536b12e174 gated the realign on compose_handlers_flush, copying the flag that
decides who flushes rather than tracking whether the database needs realigning.
Those are different questions. `compose_handlers_flush: false` means "I flush
later myself" -- web-app-gitlab/tasks/00_core.yml:22 and web-app-matrix set it and
then flush on their own -- so for them the realign never ran at all.
Not a deadlock, not an ordering problem: the task was simply never scheduled.
Negative control from CI job 99670614257: `included: .../realign_dedicated_db.yml`
appears exactly three times, for openresty and flowise, and never for gitlab. Both
gitlab call sites report `false_condition: compose_handlers_flush | bool`.
What that costs, measured at the container in two independent jobs -- the tor one
(99670614346) and the compose one:
PostgreSQL Database directory appears to contain a database; Skipping initialization
2026-09-01 03:55:17.471 UTC [40] FATAL: password authentication failed for user "gitlab"
65 of them, 30s apart. gitlab-migrations never leaves /scripts/wait-for-deps,
webservice/sidekiq/workhorse never leave `created` because they depend on it with
service_completed_successfully, and `compose up` burns its full 1800s. The health
gate cannot see any of it: the engine's healthcheck is `pg_isready -U gitlab`,
which does not authenticate, so it reports healthy throughout.
17 roles pair a dedicated postgres with compose_handlers_flush: false and were
latently broken under rotation: baserow, bookwyrm, decidim, gitlab, kix, listmonk,
mastodon, matrix, mobilizon, n8n, odoo, openproject, opentalk, penpot, taiga,
xwiki, zammad.
The engine probe is required, not decoration. Un-gated, this task also runs on a
FIRST deploy where no engine exists yet, and set_postgres_superuser_password.sh
would spend ~30 minutes retrying before reporting [FAIL]. It is a no-op for the
other call site, post_deploy.yml:16, which already waits for healthy two lines
above it.
Verified by deploy, twice: `[CHANGED] New password did not work before, and works
now.` A skeptic's objection -- that the un-gate only helps when the previous pass
left the engine running -- was measured and does not hold: in both affected jobs
the engine was healthy 12-14 seconds BEFORE the task that got skipped.
Break risk measured rather than estimated. One deploy drove this across nine roles
and 18 invocations: 1 CHANGED, 1 UNCHANGED, 16 correctly skipped, 0 FAIL.
Still open: a dedicated postgres with no healthcheck yields an empty probe and
stays a silent no-op. Not a regression -- the task did not run there before either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Discourse#1 of run 33414875521 failed on a rebuild that could not log in:
PostgreSQL Database directory appears to contain a database; Skipping initialization
FATAL: password authentication failed for user "discourse"
Eight lines in the sidecar's log, all of that one class and nothing else. The
volume carried a cluster from an earlier run, so docker-entrypoint-initdb.d was
skipped and the role kept its pre-rotation password; the job log shows the launcher
being handed 8ilTTI... in the first pass and hgBFXE... in the second, exactly one
rotation apart.
The realignment for that exists and never ran. sys-stk-backend includes
sys-svc-compose with compose_handlers_flush hardcoded to false -- deliberately, so
the SSO-proxy sidecar can be rendered before anything flushes -- and then flushes
itself without the post-deploy step that up.yml pairs with its own flush. Counting
the rendered task names across the job: post_deploy ran for svc-net-tor and
svc-prx-openresty, both of which include sys-svc-compose directly, and for nobody
else. Discourse's two invocations both report
`false_condition: compose_handlers_flush | bool`.
Only the realignment is added back, not all of post_deploy. A first attempt included
the whole file and would have armed a 120s database-health wait and a swarm
convergence wait for the roles that reach compose through this wrapper; 46 roles set
compose_handlers_flush: true. The realignment itself is gated on compose + local +
postgres, which is 21 roles with a dedicated engine.
The task moves to sys-svc-rdbms/tasks/utils/, next to patch_persisted_password.yml.
Both halves of the same rotation work now live in the database role: one rewrites what
the application persisted, the other what the engine kept. sys-stk-backend calls it
after its flush because that is the only point where both facts hold -- the database
is this role's, and the stack is up. sys-svc-rdbms cannot call it itself: it runs
before sys-svc-compose and only notifies compose-up, so no container exists yet.
MariaDB needs none of this, because --init-file realigns it declaratively at start;
postgres has no equivalent for a cluster that already exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5