GitLab's migration on its dedicated postgres sidecar failed with "ERROR: out of shared memory" and the hint to raise max_locks_per_transaction. The default of 64 locks per transaction cannot cover a schema that locks thousands of tables and indexes in one transaction: a reproduction on postgres:18 failed at table 7453 with the default and held 20000 tables at 256.
The value sits next to the other postgres tuning constants in group_vars/all/17_resource.yml and reaches both the central svc-db-postgres command and the per-app sidecar. The sidecar's new command keeps the image entrypoint and only adds the flag. The shared lock table grows to about 256 x max_connections entries, roughly 30 MB at the 400-connection cap.
Ported from 7690d2ca6d on feature/store-i18n, which kept the constant in the resource lookup that main no longer has.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
postgis/postgis publishes amd64 only, so every postgres consumer failed
on arm64. The official postgres image serves both architectures and
already ships the contrib extensions bloom, pg_trgm and unaccent.
PostGIS moves into the postgres_libraries registry and is installed from
PGDG (postgresql-18-postgis-3) only for a consumer that declares it as
an extension, which is mobilizon alone. apt update reads every source
list again, because the PostGIS package depends on GDAL from Debian
main, which a PGDG-only update left without an install candidate.
Postgres 18 keeps its cluster in /var/lib/postgresql/18/docker and its
entrypoint treats a volume at the former /var/lib/postgresql/data as an
old database, so the central and the dedicated data volume both mount at
/var/lib/postgresql. Existing 17 clusters are not migrated: their data
directory trips the entrypoint's old-data check and the container
refuses to start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f78ebe46a4)
start_interval is the cadence while start_period runs, and only while it runs.
Once the window closes the probe falls back to interval, so a service that
becomes ready just after it pays a whole tick anyway and the cadence never
reached it. Six of the seven flavors were shaped that way, Curl worst: a thirty
second window against a one minute interval.
Measured across 151 converge waits in four job logs, 63 sit exactly on a tick
boundary, 36 in the thirty second step and 27 in the sixty second one.
Attributed by application, web-app-seaweedfs is the largest at 1501s over
sixteen waits with a median of 57.7s: it probes with curl and fell straight into
the gap. web-app-zammad probes with connect, whose window is fifteen minutes,
and its 154s median is real boot time with nothing to recover.
So the window moves to ten minutes and the steady state interval to two, and the
flavor overrides that sat below them go: Curl, Wget and Connect lose interval
1m, Http its twenty second window, Nc its ten. Connect keeps fifteen minutes.
Neither number costs anything while a container works. start_period delays
nothing, it only stops failures counting, and interval no longer touches
convergence at all since that hangs on the window alone; it is steady state
load, which drops from 120 probes an hour per service to 30.
Forty six role files follow. Twenty seven declarations go that repeat the new
default or undercut it, an undercut window being the defect itself. Eight
windows longer than ten minutes keep their value and now say why. The literal
blocks and the three Dockerfiles this repository builds spell every timing out,
because outside the lookup an omission takes docker's value and not the
platform's.
A fourth lint rule keeps the shape: start_period must outlast one interval,
checked against the effective values, so raising interval without the window
cannot quietly bring the cliff back.
Two faults in the lint surfaced while building it and are fixed here. It
compared timings as strings, so 600s and 10m read as different values, and it
matched the healthcheck key exactly, so a line carrying a trailing nocheck was
skipped whole, which is how the two swarm-init partials went unexamined.
One regression, bounded. A container that starts and never turns healthy is
neither rejected nor errored, so stack_ready.sh only calls it fatal after three
killed tasks, and each cycle now spans the longer window. That reaches twenty
minutes rather than six, capped there by swarm_stack_converge_retries at 600
with a two second delay, and it lands only on runs that were failing anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A healthcheck written outside the container_healthcheck lookup inherits nothing
from utils/docker/healthcheck, and docker offers nowhere to put a replacement.
daemon.json carries storage, logging, DNS, registries and runtimes, and no
health key at all, so an omitted timing lands on docker's own value rather than
the platform's. The two disagree where it matters most: docker starts a
container with no start period, which leaves start_interval no window to act in.
That is why the hand-written blocks never gained the cadence.
The three images this repository builds itself now carry it. openldap and
postgres also drop to the platform interval, and bluesky gains the start window
it never had:
svc-db-openldap/files/Dockerfile --interval=30s ... --start-interval=5s
svc-db-postgres/templates/Dockerfile.j2 --interval=30s ... --start-interval=5s
web-app-bluesky/.../Dockerfile --start-period=30s --start-interval=5s
Six literal blocks move to the platform interval as well: magento, redis,
elasticsearch, matrix and both sys-svc-rdbms partials. Since start_interval
exists, interval no longer touches convergence and is steady state load alone.
Sixteen values that stay where they are now name the mechanism that keeps them
there. Two are worth reading. web-app-kix overrides start_period and nothing
else on purpose, letting the image's own HEALTHCHECK supply test, interval,
timeout and retries; filling those in would have overwritten what upstream tuned.
The memcached and redis swarm-init partials probe every second and deliberately
run with no start window, which is why start_interval is absent rather than
forgotten.
The rest name their probe: occ boots the whole Nextcloud stack, tor re-parses
its torrc instead of asking a running service, opensearch is still forming a
cluster, matrix waits on a file another step writes to a share.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
42ebd9ea71 gave the healthcheck SPOT start_interval, the cadence docker uses
while start_period is still running. A block a template writes out itself
bypasses the SPOT and never gained it. Measured across the repository: 46 such
blocks, 40 of them already carrying start_period, not one carrying
start_interval.
Without it a probe fires only every interval, so a service ready after two
seconds is reported healthy at the next tick and a stack converges no faster
than its coarsest quantisation. That lands in sys-svc-compose's swarm converge
wait, 1551.7s per CI job over 38 stack deploys.
Twenty five of the blocks move into meta/services.yml and render through
container_healthcheck. Each keeps an explicit test rather than picking a flavor:
a flavor would replace the command instead of carrying it across.
The move is lossless by construction. probes.py block() writes
overrides.get(key, default) for interval, timeout, retries, start_period and
start_interval, so a declaration repeating the literal values renders the same
block plus the missing cadence. Checked per block against the old values: the
only difference is start_interval 5s, nothing changed, nothing lost.
Twenty one stay literal. Seventeen interpolate a template variable, whose value
resolves in a scope meta/services.yml does not share, so moving them would be a
rewrite rather than a transcription. Four interpolate nothing and still cannot
move: svc-db-memcached and svc-db-redis probe every second in their swarm
service partial, where a 5s start cadence would report healthy later than the
block already does, and the sys-svc-rdbms and keycloak sso_proxy partials render
for the consuming role's application_id, so their declaration has no single
meta/services.yml to live in. Each carries a nocheck naming its reason.
All twenty one gain start_interval. Three also gain start_period, without which
the key does nothing at all, since it only applies while the start period runs.
That widens their failure tolerance rather than narrowing it: bluesky from 90 to
120 seconds, magento from 50 to 80, the mariadb partial from 180 to 210.
pixelfed spelled its probe as a bare string and now spells it CMD-SHELL. Docker
reads both as a shell command; only the YAML shape changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The compose flush recreates a dedicated postgres moments before this probe
runs, and its healthcheck sits on a 10s interval, so health=healthy cannot have
reported yet. The probe returned empty, the realign task below is gated on that
output being non-empty, and it skipped silently. The rotated POSTGRES_PASSWORD
therefore never reached the surviving data volume, and ./launcher rebuild
discourse died minutes later:
PG::ConnectionBad: connection to server at "192.168.9.2", port 5432 failed:
FATAL: password authentication failed for user "discourse"
Measured in the failing job: the engine started 01:29:44.757, postgres logged
"database system is ready to accept connections" at 01:29:46.737, and the probe
ran at 01:29:48.419 - 1.68s after the socket was already live, and 3.66s after
a start whose first healthcheck could not fire before ~10s.
Running is the predicate the operation needs, not a loosening.
set_postgres_superuser_password.sh reaches the cluster over the container local
socket by peer auth, which is exactly what lets a rotated credential be written
into an engine nobody can log into any more, and it carries its own readiness
loops: 30 x 30 x 2s around SHOW hba_file and 15 x 2s of TCP auth retry. Against
that 60s budget the observed start-to-ready times were 1.98s, 4.05s and 5.51s
across three engines, an 11x margin.
The other call site, sys-svc-compose post_deploy, already waits for healthy
before reaching this file, so running matches there too and its behaviour is
unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
da729cecec reads .Options.device when a volume declares a bind, on the premise that only compose volumes carry no .Options. A swarm node can hold the volume object without them anyway: in run 33923190220 the DR drill probe reported 'exposes no backing store for kix_kix_config' on the node that held it. The script then falls back to .Mountpoint, the node-local _data that is empty while the stack is down, finds no config there and exits 0.
Pass the inventory's own path as PATCH_DATA_DIR, the same DIR_VAR_LIB/<volume> that compose_volumes.py bakes into driver_opts.device, and try it when the volume-derived path holds nothing.
The second half is the asymmetry that hid this: a config present but unmatched fails loudly since 2c4e0e18b1, while a config found nowhere passed silently. The silent case is the dangerous one. KIX kept the pre-rotation password, its backend crash-looped on exit 255, and the converge gate only gave up 44 retries later. A total miss now prints SKIPPED with both paths it tried, and stays non-fatal so a first install still works.
The suite covered three of the seven consuming roles; it now covers all seven, plus the fallback and the named miss.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker mounts a bind-backed volume's device only while a container holds it, so .Mountpoint is an empty _data whenever the stack is down - which is exactly when the credential realign runs. The script then failed its 'test -f' and exited 0, reporting ok instead of changed, so a rotated password never reached the app's persisted config.
Read .Options.device when .Options.o declares a bind; compose volumes carry no .Options and keep the .Mountpoint path. Affects every role shipping tasks/utils/patch/db_credentials.yml: friendica, joomla, kix, matomo, moodle, nextcloud, suitecrm.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two branch-only commits each armed a defect in the compose-up handler
topic, and both failed in run 33785198875.
63f7f962bf registered the topic owner at the top of sys-svc-rdbms'
dedicated.yml so the notify two tasks below stopped aborting with
ERROR_ON_MISSING_HANDLER. The notification then survived to the
unconditional meta: flush_handlers at sys-svc-container/tasks/00_core.yml:53,
reached from sys-svc-compose's own 00_core before that role installs
BIN_COMPOSE, and long before 02_stack_host_prep installs BIN_CA_INJECT.
Matrix#1 died there on ca-inject with rc 2 after seven retries. That play
carries exactly one compose notification (job 100756596907, log line
11046), so closing this window is sufficient, not merely necessary.
dedicated.yml now enters 00_core.yml under the role's run-once flag
before it notifies, so the flush drains an empty queue and the flag then
makes main.yml skip the body on the real entry. The registration itself
stays ungated: gating it would skip the owner from the second app on.
95186163aa hoisted the sys-svc-compose include to sys-stk-backend
position 1. The consumer therefore rendered and notified compose-up
before sys-svc-objstore lazily loaded the provider app role, whose own
unconditional flush consumed the notification and ran the single
registered handler in the provider's chdir. mastodon and peertube never
got a stack, in compose and in swarm alike; the victim's own guard task
appears zero times in all four logs. Restoring the include below rdbms,
objstore and engine puts the notify after every foreign crossing. Two
pre-hoist runs in exactly that order are green on record.
The test keeps its cases and gains one. It selects the render include by
the absence of tasks_from instead of by first occurrence, so an early
registration-only include stays legal while a hoisted render still fails.
The new case pins the bootstrap: entry point, position before the first
notify, and the run-once guard whose removal would let 01_reset.yml tear
down DIR_COMPOSITIONS once per dedicated-DB app.
Same class, pre-existing, not fixed here: svc-db-qdrant notifies and
flushes three tasks before its first compose entry; web-app-magento pairs
a dedicated database with a shared object store; web-app-seaweedfs still
flushes a topic it does not own; svc-runner enters 00_core.yml unguarded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The name added in 63f7f962b was 112 characters and failed
tests/lint/repository/test_name_emoji_and_length.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dedicated.yml notifies compose-up but registers no handler for it, so a
caller that reaches sys-svc-rdbms before any compose consumer has loaded
the sys-svc-compose block dies with "The requested handler 'compose-up'
was not found" and no PLAY RECAP. 95186163aa guarded only the
sys-stk-backend caller; web-app-matrix's ansible flavor includes
sys-svc-rdbms directly in 03_database.yml and still hit it as the first
compose consumer of its play (run 33751742070, job 100643400267).
Register the owner as the first task of the notifying file. In the
sys-stk-backend path this lands before the pre-flush rebind, so the app's
copy stays the last registration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generic realign ran for every app whose database resolves to local
postgres and handed set_postgres_superuser_password.sh the app's entity name
as the superuser role. That pairing only holds for an engine sys-svc-rdbms
initdb'd itself, because templates/env/postgres.env.j2 fills POSTGRES_USER
from the same lookup.
BigBlueButton does not go down that path. Its postgres comes from the vendored
upstream compose and is initialised as superuser postgres, so
`psql -U bigbluebutton -d postgres -Atc 'SHOW hba_file;'` fails on every
attempt, the script's nested 30x30 retry burns its whole budget, and the task
ends in [FAIL]. In run 33574056480 that cost 32m59s of a Nextcloud variant-11
round, which reaches BBB as a dependency, and took the deploy with it.
main.yml already gates dedicated.yml on the app rendering its own
templates/compose.yml.j2. The realign now carries the same condition. It has to
live in this file rather than in main.yml, because sys-svc-compose's post_deploy
and sys-stk-backend include the task directly and never see main.yml's facts.
Measured against the current role set: 32 postgres roles can be local in some
round, 31 keep the realign, and BigBlueButton alone loses it - the only one
with a static shared: false and no compose template of its own, and the only
one carrying its own tasks/01_compose/postgres_password_sync.yml. No coverage
is lost, only a retry storm against a role that does not exist.
The stat carries the same compose-mode condition as the block, so a swarm round
runs exactly what it ran before, and it is run_once because the path does not
vary by host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
536b12e174 gated the realign on compose_handlers_flush, copying the flag that
decides who flushes rather than tracking whether the database needs realigning.
Those are different questions. `compose_handlers_flush: false` means "I flush
later myself" -- web-app-gitlab/tasks/00_core.yml:22 and web-app-matrix set it and
then flush on their own -- so for them the realign never ran at all.
Not a deadlock, not an ordering problem: the task was simply never scheduled.
Negative control from CI job 99670614257: `included: .../realign_dedicated_db.yml`
appears exactly three times, for openresty and flowise, and never for gitlab. Both
gitlab call sites report `false_condition: compose_handlers_flush | bool`.
What that costs, measured at the container in two independent jobs -- the tor one
(99670614346) and the compose one:
PostgreSQL Database directory appears to contain a database; Skipping initialization
2026-09-01 03:55:17.471 UTC [40] FATAL: password authentication failed for user "gitlab"
65 of them, 30s apart. gitlab-migrations never leaves /scripts/wait-for-deps,
webservice/sidekiq/workhorse never leave `created` because they depend on it with
service_completed_successfully, and `compose up` burns its full 1800s. The health
gate cannot see any of it: the engine's healthcheck is `pg_isready -U gitlab`,
which does not authenticate, so it reports healthy throughout.
17 roles pair a dedicated postgres with compose_handlers_flush: false and were
latently broken under rotation: baserow, bookwyrm, decidim, gitlab, kix, listmonk,
mastodon, matrix, mobilizon, n8n, odoo, openproject, opentalk, penpot, taiga,
xwiki, zammad.
The engine probe is required, not decoration. Un-gated, this task also runs on a
FIRST deploy where no engine exists yet, and set_postgres_superuser_password.sh
would spend ~30 minutes retrying before reporting [FAIL]. It is a no-op for the
other call site, post_deploy.yml:16, which already waits for healthy two lines
above it.
Verified by deploy, twice: `[CHANGED] New password did not work before, and works
now.` A skeptic's objection -- that the un-gate only helps when the previous pass
left the engine running -- was measured and does not hold: in both affected jobs
the engine was healthy 12-14 seconds BEFORE the task that got skipped.
Break risk measured rather than estimated. One deploy drove this across nine roles
and 18 invocations: 1 CHANGED, 1 UNCHANGED, 16 correctly skipped, 0 FAIL.
Still open: a dedicated postgres with no healthcheck yields an empty probe and
stays a silent no-op. Not a regression -- the task did not run there before either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Discourse#1 of run 33414875521 failed on a rebuild that could not log in:
PostgreSQL Database directory appears to contain a database; Skipping initialization
FATAL: password authentication failed for user "discourse"
Eight lines in the sidecar's log, all of that one class and nothing else. The
volume carried a cluster from an earlier run, so docker-entrypoint-initdb.d was
skipped and the role kept its pre-rotation password; the job log shows the launcher
being handed 8ilTTI... in the first pass and hgBFXE... in the second, exactly one
rotation apart.
The realignment for that exists and never ran. sys-stk-backend includes
sys-svc-compose with compose_handlers_flush hardcoded to false -- deliberately, so
the SSO-proxy sidecar can be rendered before anything flushes -- and then flushes
itself without the post-deploy step that up.yml pairs with its own flush. Counting
the rendered task names across the job: post_deploy ran for svc-net-tor and
svc-prx-openresty, both of which include sys-svc-compose directly, and for nobody
else. Discourse's two invocations both report
`false_condition: compose_handlers_flush | bool`.
Only the realignment is added back, not all of post_deploy. A first attempt included
the whole file and would have armed a 120s database-health wait and a swarm
convergence wait for the roles that reach compose through this wrapper; 46 roles set
compose_handlers_flush: true. The realignment itself is gated on compose + local +
postgres, which is 21 roles with a dedicated engine.
The task moves to sys-svc-rdbms/tasks/utils/, next to patch_persisted_password.yml.
Both halves of the same rotation work now live in the database role: one rewrites what
the application persisted, the other what the engine kept. sys-stk-backend calls it
after its flush because that is the only point where both facts hold -- the database
is this role's, and the stack is up. sys-svc-rdbms cannot call it itself: it runs
before sys-svc-compose and only notifies compose-up, so no container exists yet.
MariaDB needs none of this, because --init-file realigns it declaratively at start;
postgres has no equivalent for a cluster that already exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The script ran sed and printed PATCHED unconditionally, so an expression that
matched no line reported success and the deploy believed a rotated password had
reached the application's own config. Nothing downstream could tell the difference.
Adding the check found two expressions that could never have worked:
friendica sed: -e expression #1, char 44: invalid reference \2 on 's' command
kix sed: -e expression #1, char 59: Invalid content of \{\}
Both spell their groups as (...) with \1 \2, which is extended-regex syntax, while
the script called basic-regex sed. Under sed -E all five declarations match a sample
of the line they target -- friendica, matomo, moodle, nextcloud and kix -- so the fix
is the flag, not the expressions.
That makes 06f5905d64 incomplete: it moved friendica's patch in front of the health
gate, but the patch itself aborted sed on every run and said PATCHED anyway.
The dry pass uses the same expression the write will use, so a matching line is
proven before the file is touched rather than after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1f870b1e5 gave the patch one call site and put it in post_deploy.yml, which runs
after the deploy has waited for the stack. 3831c8f804 had named the constraint in
its own subject line: before the health gate. An app whose health check
authenticates against its own database fails that check with the password it
persisted, compose gives up with "dependency failed to start: container matomo is
unhealthy", and nothing behind the gate ever runs -- including the patch that
would have fixed it.
Measured, not argued: with the call in post_deploy.yml a local compose deploy of
five roles passed its first pass at ok=4571 failed=0 and died in the second, the
one that carries the rotated credential. With the call back in front, all three
passes report failed=0 and the Playwright specs pass in both.
sys-svc-rdbms/tasks/main.yml is where it goes: the stack include reaches it
before anything comes up, it is per-consumer rather than per-role-once, and it
still resolves the declaration through the same path, so there is still exactly
one call site. The patch reads the volume from the host and needs no running
container, which is what makes the earlier position possible at all; before the
first deploy has created the volume it exits without doing anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The body was a Jinja-interpolated shell string inside the task. As a file it
can be read, shellchecked and run, and the three parameters travel through the
environment the way the password already did.
That forces one thing into the open. The old expression carried
${PATCH_PASSWORD} and the shell expanded it while assembling the command; a
value arriving in an environment variable is not expanded a second time, so the
substitution has to be explicit. The consumers now write @PASSWORD@ and the
script replaces it, which is also the version a reader can follow without
knowing that rule.
The script refuses to run without its parameters rather than treating an unset
one as empty, because an empty volume name would look exactly like a node that
does not hold the volume: a silent success on every host.
Tests move with it, to the files/ segment every other test of a shipped script
uses. They run it against a stubbed `container` in three states per consumer -
no volume, volume without the config, both present - and assert the marker the
task reads as a change. Five mutations of the script were checked against them:
dropping the substitution, the marker, the existence test, the parameter guards
and the volume filter each turn the suite red.
Layout follows what the linters already require: files/shell/ is enforced by
role-files-layout, and 65e34746fc names ansible.builtin.script as the sanctioned
way to run one from a task.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b067d21ef shipped two ways to kill a dedicated MariaDB, both reproduced.
The file was rendered 0640 under a play that runs as root, while
stack_host_template otherwise defaults to 0644. Compose emulates a config as a
plain host bind and ignores uid, gid and mode on the reference, so the engine,
which does not run as root, could not open the file. mariadbd treats an
unopenable init file as fatal and aborts every start. Dropping the explicit mode
puts it back on 0644, next to the env file that carries the identical password.
On a fresh data directory the entrypoint starts a temporary server with the same
arguments, so the init file ran before the database was ever set up. Giving root
a password there breaks the entrypoint's own readiness probe, which connects
without one, and the schema and grants are then never created. The CREATE USER
had the same problem from the other side: the entrypoint's own CREATE USER
carries no IF NOT EXISTS and the script runs under set -e.
What remains is the one statement the failure actually needs: the application
user, guarded by IF EXISTS so it is a no-op on the first start and a
realignment on every later one. That is the identity the health check
authenticates as.
Root keeps the password its data directory was initialised with. It did so
before this feature existed, nothing in the repository authenticates as root
against a dedicated engine, and the account is reachable only from inside the
app's own stack network. Realigning it needs a gate that can tell the
entrypoint's temporary server apart from the real one, which this does not have.
The GRANT is gone with it: rotating a credential changes passwords, not
privileges.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An engine of its own keeps its passwords in its data directory, written once
when that directory was initialised. When the deploy rotates the credential the
new value reaches the container's environment and never the engine, so the
health check authenticates with a password nothing has ever accepted. The
application user and root come from the same value, so nothing is left that
could log in and change it either.
mariadbd reads --init-file on every start, before it opens the socket, and runs
the statements with full privileges. That is the one point where a password can
be reset without knowing the previous one.
The SQL is rendered next to the env file that already holds the same secret in
plaintext, so it adds no new exposure. It reaches the container as a config
rather than a bind mount because a swarm task may be scheduled on a node the
rendered path does not exist on; the config name carries a hash of the file
content, so a rotated password produces a new name and swarm replaces the
immutable old one.
The guard is mariadb-and-not-shared, which matches the `local` guard the service
template already renders on, so the block and the top-level entry appear
together or not at all.
3831c8f804 fixed the other half of this failure: matomo, moodle and nextcloud
keep the same password in a config file on their data volume. Same rotation,
opposite end of the connection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
path_absolute builds from playbook_dir, a controller-side value, but the task
carried no delegate_to and therefore ran on the managed node. Anywhere the
controller is not the target - every swarm node, every remote deploy - the
path does not exist, stat.exists is false and the dedicated branch is skipped
for every role, taking the env file, the initdb setup and the backup seed with
it. Silently, while the rendered service still points env_file at a file
nobody wrote.
A single-host compose run cannot see this: there the controller and the target
are the same machine, which is why 47c01259e looked green.
Delegating alone is enough. run_once would additionally hand one host's answer
to every other host of a `hosts: all` play whose include is gated per host,
making the registered result depend on host ordering rather than on the host
being configured - for a filesystem check that is identical for all of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
47c01259e named the task 87 characters wide; the convention allows 80 so the
name stays readable in a playbook run. The lint that caught it also says where
a longer explanation belongs, and it is not the task name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A dedicated database is not the same as one this role provides.
web-app-bigbluebutton declares postgres enabled and unshared, so the lookup
reports it local and the dedicated branch ran - rendering an env file, an
initdb setup and a backup row for an engine bbb-docker brings itself. Nothing
of that output is read: bigbluebutton never calls the database lookup, ships
its own env.j2 and vendors its own initdb hook.
The backup row was the one that hurt. 7d044c265 taught the dedicated branch to
seed, and it seeded credentials derived from the entity name into a postgres
that has no such role:
pg_dump -U bigbluebutton -d bigbluebutton
FATAL: role "bigbluebutton" does not exist
bigbluebutton seeds its own row correctly - the '*' cluster row with the
superuser its image actually creates - so the drill now has one working row
and one impossible one for the same instance.
Whether this role provides the engine is already written down: its service
template is reached through <role>/templates/compose.yml.j2 -> base.yml.j2, so
a role without that template never gets one. Across the 53 roles with a
database service exactly one lacks it, and none has it without including
base.yml.j2, which makes the condition exact rather than a stand-in for "is
this bigbluebutton". Reading it beats declaring it: a flag would be a second
copy of a fact the tree already states, free to drift from it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The task name broke the 80-char convention; naming the consumer alone says as much, since the enclosing role already names the application.
The manifest read opts out of the shared read cache rather than joining it. utils.cache.files pulls in PROJECT_ROOT and the gitignore reader, which would break the import-free rule utils/recovery/manifest.py exists under, and a process-wide cache is wrong for this file in any case: a backup run writes a new generation that the same process then reads back, so the cached copy would be the stale one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baudolo dumps only what databases.csv names, and the entry was seeded from the central branch alone. An application running its own engine - Discourse with svc-db-postgres absent from the group - therefore never reached the seed: databases.csv stayed empty, baudolo warned that it was continuing without dumps, and the volume was stored as a raw PGDATA tree that restores a torn cluster.
Whether a database must be backed up is a property of there being one, not of it being central, so both branches now include the same helper with their own gate: centrally the host from the provider group seeds, dedicated the stack host itself. The coordinates come from the same lookup either way.
Verified without a deploy that the row this writes is the row baudolo looks for: the seed records the consumer entity as the instance, and baudolo derives the same key from the container name (discourse-database -> discourse). A mismatch there would have filled databases.csv and still produced no dump.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cosmos generator linked a consumer to its provider only through an
'<role>' in group_names reference or an explicit provides: name. A hard
dependency has neither -- pinning the provider on unconditionally is
exactly what removes the Jinja reference the generator was reading -- so
web-svc-asset's bond to web-svc-file vanished from the diagram the moment
it became binding. Match a service key against the provider's own primary
key from the registry, in both directions.
Services the role turns off drew a solid 1:1 edge, claiming a mandatory
relationship for something that carries nothing at deploy time. They now
render as a red 0..0 edge; edge merging gained a real precedence so a
provider consumed through two keys stays live when only one is off.
Named colour, not hex: a '#' opens an entity code in the mermaid lexer and
turns the whole diagram into a parse error on GitHub.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A handler name was doing two jobs at once: the line an operator reads in the
play output, and the identifier a `notify:` binds to. The two pull in opposite
directions -- the log wants "restart the SSH service, with diagnostics", the
binding wants a token that survives quoting, YAML flow style and a shell
round-trip -- and a name that changes for readability silently changes which
handler runs.
Every handler that a notify reached by name now carries a `listen:` with that
exact wording, so the binding is explicit before anything is renamed. The
topics themselves become identifiers: `compose up` reads `compose-up`,
`Restart NFS ganesha` reads `restart-nfs-ganesha`, 22 of them across both
sides of every notify. Where the topic is assembled at runtime, both branches
moved together: the conditional in sys-service carries two literals, and the
LDAP renderer interpolates `import-{{ folder }}-ldif-files`, which is what
makes those handlers reachable at all.
Freed from the binding, the names take the emoji prefix every other task line
already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
force_shared_db flipped every DB service to shared with no exemption, so
web-app-magento's deliberate embedded MariaDB 11.4 - declared shared:
false with a comment warning that no bot may move the major underneath
it - became dead config and setup:install hit the central engine's 12.3.2,
outside Magento's supported range. The run log names the override:
'swarm force-shared: web-app-magento.services.mariadb -> shared=true'.
Exempts a DB service whose role meta declares a literal shared: false AND
whose role is manager-pinned. The second condition is not decoration:
utils/storage/nfs.py keeps a manager-pinned role's volumes node-local, and
InnoDB over NFS is why the harness force-shares in the first place. Of the
two roles with a literal local DB, magento qualifies and bigbluebutton
(unpinned) does not, so its behaviour is unchanged.
derive_includes._force_shared_db_view documents itself as mirroring
force_shared_db, so it takes the same exemption - without it the dep walk
would compute magento's include closure against the central engine while
the deploy runs a local sidecar.
The sidecar carried no deploy block at all, so an exempted service would
have been free-floating with one task per node over a single-writer
datadir. The block now comes from a container_deploy lookup shared with
sys-svc-container/templates/deploy.yml.j2, which replaces the replicas
ternary and the duplicated placement constraint there. Its unit test
caught a real defect while being written: with compose_mode_force=swarm
over DEPLOYMENT_MODE=compose the delegated compose_replicas returned
empty, emitting a deploy: key with no child.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The entrypoint initialises on a server that listens on the unix socket only, so pg_isready without -h reports healthy while the database is still being built -- visible in run 30699990912 as socket-only at 14:07:37.425 against socket+TCP at 14:07:42.174. Probe 127.0.0.1 instead, which the bootstrap server does not serve.
pg_isready keeps using PQping, so the check stays immune to pg_hba changes as a psql probe would not be. The compose healthcheck gains the start_period it never had, so the stricter probe cannot eat into its 60s retry budget on a cold start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_task_file_numbering.py: numbered task
files use exactly two leading digits, a numbered *_core.yml must be
00_core.yml, per directory the numbers are unique/consecutive and start
at 01 (00 with a core file), numbered includes appear in ascending
order, and shared helpers live in tasks/utils/ instead of the numbered
chain.
Apply it across every role: rename NN_core.yml to 00_core.yml with a
consecutive down-shift of the siblings, move letter-suffix children into
a subdirectory named after their parent task, relocate helpers included
from more than one place into tasks/utils/, and update every reference
(include_tasks, path_absolute, .ansible-lint-ignore, requirement docs).
Legacy narration comments in the touched files are deleted; the upstream
xwiki.properties reference block opts out via nocheck. The unused
MAILU_IP6_PUBLIC var (only ever named in now-removed comments) is dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rewrite all 254 role descriptions to a uniform noun-phrase schema
(function-first, no leading verb, no trailing period, <=120 chars) and
enforce it via the extended description lint. Rename 12 role README H1
titles for clarity (e.g. Backup Container to Local -> Backup Docker
Volumes, CSS Service -> Corporate Design, NGINX Legal -> Imprint) and
regenerate their Cosmos and Quick Setup sections. Overview table
regenerated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Roles declared under meta/main.yml dependencies now appear on both the
Dependencies and Dependents side, labelled with the glossary gear
symbol and connected by a fixed 1:1 edge; the legend explains it. All
role READMEs regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Edges now carry their relation: solid 1:1 for fixed consumption, dashed
0..1 for conditional (group-membership gated) consumption. Below every
diagram the generator renders a reading key for the edge kinds and the
node markers, with the glyphs drawn from the shared symbol glossary.
All role READMEs regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add cli/build/readme: derives each role README's managed sections from
templates/roles/README.md.j2.tmpl (the Cosmos mermaid diagram from
meta/services.yml, an invokable-gated Quick Setup, and Credits).
The Cosmos diagram shows dependencies, every declared service, and
dependents (roles that consume this one, matched by role name and by
provides provision-name in both directions). Role nodes carry deploy-mode
emoji markers (compose/swarm/host); disabled and shared services get a
stop mark; dependents are capped at 12.
test_readme enforces the Quick Setup body is byte-exact with the template
render. `make readme-generate` and `make cosmos` regenerate; `make cosmos`
also runs inside `make quality`.
Refactor: utils/symbol_glossary.py is the single source of truth for the
word/emoji mapping (unique project-wide), consumed by the complexity
matrix and the Cosmos builder. utils/roles/deploy.py holds role_has_stack
and role_deploy_modes, shared by both so the logic never drifts.
Add `make lint-mermaid` and its CI workflow, rendering every diagram via
mermaid-cli; the render step provisions chrome-headless-shell and unzip
on demand so it works in the minimal container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dedicated database branch reused the provider's <db>_custom tag,
which only exists where the central db stack happened to build it; a
cold runner deploying a central-db-disabled variant (nextcloud 1) hit
'pull access denied for mariadb_custom' and the aborted pull dragged
the remaining service pulls down with it. Pin the sidecar to the
upstream reference via the image= override: the CI mirror overrides
rewrite exactly that config path, so CI pulls the ghcr mirror and local
deploys pull the hub. mariadb's custom image is a bare passthrough and
postgres' baked healthcheck is shadowed by the compose-level
healthcheck, so the sidecar behavior is unchanged; the postgres
extensions branch keeps its local build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the role, its test tree, the primary entity key
(volume-2-local), the derived vars (BKP_VOLUME_2_LOCAL_*,
SYS_SERVICE_BACKUP_VOLUME_2_LOCAL), the run-once fact, and every
group_names consumer flag, include path, doc, and lint reference. The
container_backup service key and the on-disk backup-docker-to-local
tree keep their names: both are contracts with existing consumer
entries, snapshot trees, and the baudolo pull tooling.
Operators must rename the svc-bkp-container-2-local inventory group and
disable the previous versioned units on already-provisioned hosts; the
pre-deploy snapshot glob derives its base from the new role name and
will not find them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A consumer that forces a per-role deploy mode (web-app-matrix ansible-flavor and web-app-bigbluebutton set compose_mode_force=compose) leaked that value through the include_role chain into the shared database provider (svc-db-postgres/mariadb). In a swarm cluster the provider then ran via 'docker compose up' (bridge) instead of 'docker stack deploy' (overlay), breaking the shared DB. Reset compose_mode_force to empty on the provider include, alongside the existing compose_repository_pull override, so the shared provider always follows the cluster DEPLOYMENT_MODE. Covers every consumer, not just matrix/bbb.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Role README Credits now name galaxy_info.author (plain bold or wrapped
in a Markdown link), matched byte-for-byte against the role's own
meta/main.yml, instead of a hard-coded maintainer name. Regenerate all
245 role README Credits blocks to the new 'Implemented by ...' format.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce plugins/lookup/path_absolute.py as the single source for the
repo-root anchor: lookup('path_absolute', 'roles/...') returns an absolute
path under playbook_dir, joining every term. Replace all hand-built
playbook_dir path constructions across .yml/.yaml and .j2 templates with it:
[playbook_dir, '<rel>'] | path_join -> lookup('path_absolute', '<rel>')
playbook_dir ~ '/<rel>' -> lookup('path_absolute', '<rel>')
{{ playbook_dir }}/<rel> -> lookup('path_absolute', '<rel>')
Unrelated repo_path symbols in cli/meta/roles/applications/resolution and the
single bare discover_playwright_roles(playbook_dir) call are left untouched.
Add tests/lint/ansible/test_playbook_dir_via_lookup.py: fails the build if any
of the three construction forms reappears in .yml/.yaml/.j2, pointing callers
at the lookup. Unit test covers the plugin.
Touching these files pulled their pre-existing narration comments into
test_comments_valid scope; clean them (delete restatement/banners, keep real
trip-wires as Workaround/Upstream/Invariant markers). Drop caution, important
and pitfall from the comment-lint marker vocabulary and remove the comments
that used them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Convert the two registry-driven networks templates into pure-Python
rendering functions plus thin Ansible lookup plugins:
roles/sys-svc-compose/templates/networks.yml.j2 -> {{ lookup('compose_networks') }}
roles/sys-svc-container/templates/networks.yml.j2 -> {{ lookup('container_networks') | indent(4) }}
The rendering logic lives in utils/networks/render.py as pure-Python
functions that take a registry + closure-style config/database lookups,
making the logic directly unit-testable without a Jinja stub harness.
The lookup plugins are thin wrappers that resolve application_id,
DEPLOYMENT_MODE, and swarm.network.encryption from the templar, build
the service_registry, and forward to the render functions.
Migrate 233 include lines across 85 production templates. Convert
web-app-matrix's hand-coded MDAD networks block (default + conditional
postgres + conditional openldap) to the generic lookup; this adds an
openresty attachment when services.sso.enabled (harmless extra
interface, pinned in test_matrix_mdad_consumer_attaches_postgres_ldap_sso).
Mark local-DB / redis sidecars (sys-svc-rdbms/templates/services/,
svc-db-redis/templates/service.yml.j2) with `# nocheck: networks-literal`
because they intentionally attach to default-only.
Extend the lint to scan ALL roles/*/templates/**/*.yml.j2 (not only
compose.yml.j2) and to forbid the legacy include form; the marker
above is the documented escape hatch.
Coverage:
- tests/unit/utils/networks/test_render.py (30 cases incl. encrypted=false literal pin)
- tests/unit/plugins/lookup/test_compose_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/unit/plugins/lookup/test_container_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/integration/infrastructure/compose/test_networks_render.py (10 snapshot scenarios)
- tests/integration/infrastructure/compose/test_network_includes.py (3 lint rules)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).
* 25 unit tests for the lookup cover compose vs swarm prefix, custom
flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
fails on any raw `image:` line in roles/*/templates/*.j2 unless
nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
`{{ lookup('container_image', application_id, '<service_key>') }}`
form. 26 roles use `custom=True` (mariadb, postgres, openldap,
jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
into consuming roles where application_id evaluates to the outer
role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
test_build_requires_image) recognise the standalone lookup as an
implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
the registry-prefixed image string because the custom var is now
bare.
Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
