8986b21270 wrote the patch expression from the OTRS convention, looking for
`$Self->{'DatabasePw'} = '...';`. The file does not use that form. Now that
kix_kix_config persists, the line can be read instead of guessed:
01main:13: 'DatabasePw' => 'oo4ca3...',
A perl hash entry, the same shape friendica's php array uses, so the expression
becomes friendica's with a different key and terminator.
The wrong expression surfaced because of 2c4e0e18b, on its first run: the deploy
stopped with
FAILED: no line in 01main matches the patch expression
Before that commit the same deploy would have printed PATCHED and gone green while
01main kept its pre-rotation password -- which is exactly how friendica looked fixed
for months.
Checked against the real line rather than a sample: sed -E with the new expression
returns `'DatabasePw' => 'NEWPW',` and rc 0, and grep finds DatabasePw on that
line only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Discourse#1 of run 33414875521 failed on a rebuild that could not log in:
PostgreSQL Database directory appears to contain a database; Skipping initialization
FATAL: password authentication failed for user "discourse"
Eight lines in the sidecar's log, all of that one class and nothing else. The
volume carried a cluster from an earlier run, so docker-entrypoint-initdb.d was
skipped and the role kept its pre-rotation password; the job log shows the launcher
being handed 8ilTTI... in the first pass and hgBFXE... in the second, exactly one
rotation apart.
The realignment for that exists and never ran. sys-stk-backend includes
sys-svc-compose with compose_handlers_flush hardcoded to false -- deliberately, so
the SSO-proxy sidecar can be rendered before anything flushes -- and then flushes
itself without the post-deploy step that up.yml pairs with its own flush. Counting
the rendered task names across the job: post_deploy ran for svc-net-tor and
svc-prx-openresty, both of which include sys-svc-compose directly, and for nobody
else. Discourse's two invocations both report
`false_condition: compose_handlers_flush | bool`.
Only the realignment is added back, not all of post_deploy. A first attempt included
the whole file and would have armed a 120s database-health wait and a swarm
convergence wait for the roles that reach compose through this wrapper; 46 roles set
compose_handlers_flush: true. The realignment itself is gated on compose + local +
postgres, which is 21 roles with a dedicated engine.
The task moves to sys-svc-rdbms/tasks/utils/, next to patch_persisted_password.yml.
Both halves of the same rotation work now live in the database role: one rewrites what
the application persisted, the other what the engine kept. sys-stk-backend calls it
after its flush because that is the only point where both facts hold -- the database
is this role's, and the stack is up. sys-svc-rdbms cannot call it itself: it runs
before sys-svc-compose and only notifies compose-up, so no container exists yet.
MariaDB needs none of this, because --init-file realigns it declaratively at start;
postgres has no equivalent for a cluster that already exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The script carried its procedure twice. A retry loop was added around the original
linear body, and the body below it stayed: a second pg_hba backup, a second
`trap ... EXIT`, a second prepended scram line, a second reload, two more auth_test
rounds and a second ALTER USER. Every invocation therefore rewrote the host-based
auth file twice and set the password twice.
It worked, because each step is idempotent, and it cost every call an extra pass
with two auth probes that wait up to thirty seconds each.
The subtle part is the output. The loop set `result` and never printed it; what the
caller read came from the second body, which echoed [CHANGED] or [UNCHANGED] from its
own pre_ok/post_ok. Deleting the duplicate alone would have left the script silent and
`changed_when: '[CHANGED]' in stdout` permanently false. The verdict now comes from
the pass that actually measured, through `echo "$result"`.
164 lines to 110. The contract is unchanged: [CHANGED] / [UNCHANGED] on stdout,
[FAIL] on stderr with exit 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two workspace jobs of run 33414875521 died the same way, on debian and on fedora:
RuntimeError: envsubst not found. Install gettext-base (Ubuntu/Debian) or gettext (Arch).
make: *** [Makefile:252: compose-up] Error 1
make: *** [Makefile:778: swarm-zombie] Error 1
swarm-zombie runs the deploy with INFINITO_APP_DISCOVERY_RUNNER=host, so the README
replay step calls compose-up on the act runner, and that image ships no gettext. A
two-variable substitution took the whole deploy down with it, and the failure reads
like a broken host rather than a missing package.
string.Template.safe_substitute handles ${VAR} exactly as envsubst does for this
template, including the four bare `$` that end the match lines -- both leave them
alone. The rendered Corefile is byte-identical, checked over three re-renders, and
getent resolves inside the deploy container afterwards.
The concurrency property from f3826589f2 is untouched: the render still writes into
the descriptor from tempfile.mkstemp and lands through an atomic replace. The read
goes through utils.cache.files.read_text, as the uncached-read lint requires.
Nothing else in the replay step failed -- one traceback, one raise, one message.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
allowed_consumers was twenty hand-kept lists of which fourteen were identical, so every new client meant an edit in every provider. A client now declares services.<own-entity>.mcp_consumer: true once and a provider refuses one with mcp_consumer: false, which is also what tells a deliberate exclusion apart from a forgotten entry.
The group_names-gated consumer entries give the inventory resolver the co-deploy edge meta/mcp.yml never had: one of sixty-nine matrix rounds carried an MCP client before, twenty-three role/round pairs do now. bond 0.5 keeps the clients out of round 0 so the host memory budget still holds.
BREAKING CHANGE: meta/mcp.yml no longer accepts allowed_consumers; declare services.<entity>.mcp_consumer on the client instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The direct Path.read_text bypassed the shared reader, so the uncached-file-read lint failed and blocked the full quality gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The instructions test held a job of its own that picked one random role per
run, so a README's Production block was proven roughly once per 265 roles.
The replay now rides on a deploy row instead: no extra runner, and every role
the sweep touches gets its documented one-command deploy exercised.
The row is the role's smallest variant -- fewest services enabled -- because
that is the shape the README describes, and it is chosen among the variants
the sweep actually deploys, so a row cut as redundant coverage cannot silently
drop the role from the test. A role qualifies only when its README carries a
Production block and the guide's own mode is one the role runs and tests.
The marked entry carries the guide mode in `instructions` and a 📖 in its job
title, between the variant number and a priority ⭐. The guide mode is
independent of the mode the rotation gave the row: a swarm row still replays
in compose. The step runs last in the deploy job, after the swarm teardown,
because the replay deploys the role a second time and needs the runner's
ports.
call-test-instructions.yml, cli.meta.ci.guide_select and the `instructions`
dispatch input go with it. `redundant()` moves next to the marker so the
chunker, the plan table and the marker share one definition.
The plan table trades its 📊 Weight column for 📖 Instructions; nothing read
the weight any more, so it leaves the matrix entries too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The offset a retrigger carries skipped every row the source run merely
started. A row that failed or was cancelled therefore retired its place in
the ranking: the regular line resumed past it, and the row came back only on
the priority line, pinned to the one combination it was recorded in. Every
other combination of that row was silently dropped from the sweep.
resume_offset now stops at the first row that is not green, so only proven
coverage is skipped. passed_selections supplies it; the three token
comprehensions that only differed in their state filter collapse into one
helper. deployed_selections and settled_selections keep feeding
--include-unrun unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The collector took `logs` from every container, which is stdout and stderr and
nothing else. An application that logs to a file left the bundle with a status code
and no trace of why.
That is what stopped the snipe-it investigation in run 33317483326. Its MCP adapter
reported `UNREACHABLE read call snipeit_get_license_summary: the provider answered
500`, and Laravel had written the exception to storage/logs inside the container,
where `logs` never looks. Three separate root causes were proposed from static
reading and all three were refuted -- the relation exists, inherited from SnipeModel;
the group permissions are stored exactly as decodePermissions expects; the controller
is identical at the deployed tag -- and the job is still unexplained.
The collector already execs into every container for resolv.conf, systemctl and
journalctl, so the gap was the application's own files, not the ability to reach them.
find drives the capture through -exec rather than a loop over $(find ...): the first
version split on whitespace and broke on paths containing spaces, which a probe
against a directory with two of them showed immediately.
The per-container and per-service captures fold into tables because the file sits at
the 500-line limit. Splitting it is not an option: this script copies itself into
every container it inspects and runs there standalone, so an import of a sibling
module resolves to nothing -- an attempt at that broke seventeen tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The convergence check compared the stored service element to the declared one with
jsonb equality, and the deploy failed on "The Agents plugin still stores a service or
bot that differs from the deployed gateway contract".
The sequence is write via SQL, cycle the plugin, read back. Cycling is exactly when
the plugin rewrites that structure, and it owns it. Any field it adds of its own makes
byte-identical equality impossible, permanently, while the contract itself holds.
Containment says what the task means. Its own name asks whether the stored config
still matches the gateway contract, not whether it is the same object: the deployed
keys must be present with the deployed values, and what the plugin keeps besides is
its business.
Only that one condition changes. The bot check, the legacy-bot check, the FROM clause
and the per-active-row semantics stay as they were -- a first attempt restructured the
whole query into CTEs, which would have aggregated across active rows instead of
evaluating each, and reported converged as soon as any one of them matched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
compose gave up on `dependency failed to start: container kix-backend is unhealthy`,
and the backend's own log names the reason:
psql -q -h postgres -p 5432 -U kix kix -c 'CREATE EXTENSION ...' -> succeeds
/opt/kix/bin/kix.Console.pl Maint::Config::Rebuild --allow-root -> FATAL:
password authentication failed for user "kix"
Two calls in a row, one working and one not, with the same database. psql takes its
password from PGPASSWORD; the perl layer takes it from /opt/kix/config/01main, and the
entrypoint's own `[ ! -f /opt/kix/config/01main ]` test reports that file missing on
every start.
/opt/kix/config was on no volume. Every recreated container therefore loses 01main,
re-enters the first-install path, and runs it against a database that is already
installed. The credential rotation between the sync and the async pass is only what
made the container get recreated; any redeploy does this.
Everything else was ruled out from the run rather than assumed: the ALTER USER landed
(changed: true, 22:18:44), the rendered env carried the same value (22:18:57), there is
exactly one postgres container and 192.168.200.2 is its address, and it stayed up and
healthy across the whole window.
Persisting the directory alone would trade one failure for another: the entrypoint runs
`ln -s '/opt/kix/conf.d/*' /opt/kix/config/` on every start, unquoted-glob-style but with
the glob quoted, so it creates a single dangling link named `*` -- and the second start
would find it there. The link points at a path with a literal asterisk and nothing reads
it, so clear_stale_conf_link.yml removes it on every node before the stack comes up.
The password patch declaration follows the friendica pattern and needs 2c4e0e18b to work
at all: its groups are extended-regex, which basic-regex sed rejected outright.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The script ran sed and printed PATCHED unconditionally, so an expression that
matched no line reported success and the deploy believed a rotated password had
reached the application's own config. Nothing downstream could tell the difference.
Adding the check found two expressions that could never have worked:
friendica sed: -e expression #1, char 44: invalid reference \2 on 's' command
kix sed: -e expression #1, char 59: Invalid content of \{\}
Both spell their groups as (...) with \1 \2, which is extended-regex syntax, while
the script called basic-regex sed. Under sed -E all five declarations match a sample
of the line they target -- friendica, matomo, moodle, nextcloud and kix -- so the fix
is the flag, not the expressions.
That makes 06f5905d64 incomplete: it moved friendica's patch in front of the health
gate, but the patch itself aborted sed on every run and said PATCHED anyway.
The dry pass uses the same expression the write will use, so a matching line is
proven before the file is touched rather than after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
espocrm_daemon and espocrm_websocket exited 0 every six seconds while the main
container sat healthy for thirty-nine minutes, so the deploy waited out its retries
on
not converged: espocrm_daemon espocrm_websocket
espocrm_daemon.1 desired=Shutdown current=Complete
The service log shows our own gate working -- "[entrypoint] Exec: docker-websocket.sh"
-- and upstream's failing right after it. docker-*.sh call exitIfNotReady, which runs
`bin/command app-check` and exits 0 when that fails. CommandManager resolves the name
to an AppCheck class, and espocrm/espocrm carries no AppCheck.php at 10.0.6, so the
gate can never pass. isLegacy does not spare us either: it returns true only when
/var/www/html is itself a mount point, and we mount three subdirectories.
Only the broken half is replaced. applyConfigEnv stays, because the websocket service
is the only carrier of ESPOCRM_CONFIG_WEB_SOCKET_*, and those reach the shared
config.php nowhere else. Readiness is the caller's job already: --wait-ready blocks on
the marker the main container writes once its install has finished, which is a
strictly better-informed signal than app-check.
The logic lives in files/shell/sidecar_boot.sh rather than a bash -c string, following
the route this role already uses for docker-entrypoint-custom.sh and seed_config.php:
copied into the build context, COPYed and chmod'ed in the Dockerfile, addressed
through ESPOCRM_SCRIPT_SIDECAR_DOCKER. That buys shellcheck and bash -n coverage the
embedded string never had, drops one layer of quoting, and turns a missing argument
into a named error instead of a php invocation with no script.
ESPOCRM_APP_DIR replaces the second hardcoded /var/www/html in vars.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
After minting the MCP identity the role restarted Listmonk with `container restart`
on the container id. In swarm that id belongs to a task the orchestrator owns, and
the daemon refuses:
Error response from daemon: Cannot restart container 21bda395f531: failed
Thirty attempts, the same refusal each time, and the job ended there.
The compose path keeps `container restart`; swarm gets `container service update
--force` on the service name, which is what the orchestrator offers for this and
what web-app-jellyfin already does to reload its plugins. The swarm branch needs no
delegate_to -- a service update is the manager's business, not that of whichever node
happens to hold the task -- and carries a timeout, because forcing an update waits
for convergence.
Both branches sit in one block under the shared `when: listmonk_mcp_identity.changed`
so the condition cannot drift apart, and run_once moves up with it. Only the mode
test stays per branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The MCP registry reconcile retried thirty times and failed with Flowise's own words:
FAILED calling qdrant_list_collections on svc-db-qdrant: 500
{"message":"Error: predictionsServices.buildChatflow - inputParams is not iterable"}
fixture_flow_data hand-builds an Agentflow v2 definition, and both of its nodes
carried inputs, outputAnchors and outputs but no inputParams. buildChatflow iterates
that key, finds undefined and throws, so the fixture flow could be created and was
dead on its first invocation.
Empty is the right value rather than a placation: the fixture puts its arguments in
inputs literally, so there is nothing for Flowise to resolve. inputAnchors comes
along because a Flowise node carries both and the next node written here would
otherwise trip over the same edge.
The new test asserts the three keys on every node the fixture emits, not on the two
that exist today, so a third node cannot reintroduce the bug.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The install one-shot seeds the administrator from ADMIN_PASSWORD and runs only while
the first-run marker is absent, so after a credential rotation the account keeps the
secret the deployment no longer holds. The playwright trace of the run shows the
consequence: the native login never leaves /auth/login.
Akaunting ships no first-party command for this. user:seed only invokes a seeder and
sets no password, so the realignment goes through artisan tinker, which composer.json
lists under require rather than require-dev and is therefore present in the served
image. ADMIN_EMAIL and ADMIN_PASSWORD reach it through the environment, never through
an argument.
The password is assigned in clear text deliberately: App\Models\Auth\User defines
setPasswordAttribute, which bcrypts on save, so hashing before assignment would store
a hash of a hash. The fourth test case pins that down by asserting the raw value on
the model after save.
The role had no host/cid resolution at all -- setup/swarm.yml sidesteps it by scaling
to zero and running a one-shot service, which is too coarse for a per-run task -- so
it gains resolve_host_cid and addresses the container directly.
The snippet is not standalone PHP: it carries no opening tag because tinker evaluates
what it is given. The mirrored test therefore stubs App\Models\Auth\User and
Illuminate\Support\Facades\Hash and evaluates the file, covering the missing user, the
missing email, the already-matching password (no save) and the differing one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bench new-site --admin-password lives in create_site.yml, which runs only
`when: erpnext_site_probe.rc != 0`. Once the site exists nothing touches the
Administrator again, so a credential rotated between the sync and the async pass
leaves the instance authenticating against the secret the deployment has forgotten.
The playwright trace of the run says it plainly: the native login form comes back
with "Invalid credentials, try again." for Administrator.
set-admin-password is frappe's own command and idempotent, so it runs
unconditionally, placed after the conditional bootstrap and before anything that
uses the site. The password reaches the container through `-e ADMIN_PASSWORD` with
no value on the flag, the same way the neighbouring create_site.yml passes it, so it
never appears in an argument list.
ERPNEXT_BACKEND_LOCAL_CID is available here regardless of whether the site was just
created: 01_manager_ops.yml resolves it in its first task and includes this file
twenty lines later.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
seed_admin_and_get_token only ever drove the startup wizard, and that wizard works
exactly once: on a server that is not yet provisioned. After the first deploy the
/Startup/* endpoints are closed and the administrator carries whatever password the
inventory held at install time, so a rotated credential leaves the account
unreachable. The CI rotates every credential between the sync and the async pass,
which is why job 99275699298 spent its 40 attempts on
curl: (22) The requested URL returned error: 401
and then failed with the script's own diagnosis: "the administrator cannot
authenticate after wizard retries; the deployed password does not match the running
server". The script named the cause correctly and had no way back.
realign_admin_password runs only after the wizard loop is exhausted and uses
Jellyfin's documented local recovery. Read from the server's own source rather than
assumed:
UserController.ForgotPassword writes the pin file only when the request
is local (IsLocal || IsInLocalNetwork)
DefaultPasswordResetProvider:100 ProgramDataPath/passwordreset<md5>.json,
serialized without options, so key "Pin"
DefaultPasswordResetProvider:77 ChangePassword(resetUser.Id, pin)
That last line is the one that matters: redeeming the pin sets the password TO the
pin, it does not clear it. The pin is therefore both the credential to authenticate
with afterwards and the CurrentPw of the change that follows. A first draft logged
in with an empty password and would have failed on every run while passing every
lint.
Two shell corrections came out of the same pass. get_token now takes the password as
an argument, because reading it from the environment invited
`JELLYFIN_ADMIN_PASSWORD="" get_token`, where an assignment prefixed to a function
call can outlive the call. And the three captured values now guard with
`x="$(cmd)" || x=""` instead of one of them carrying `|| true` and the others none:
under set -euo pipefail the unguarded assignments aborted the script before the
emptiness check below them could report anything.
Worst case is the status quo: nothing here runs unless the deploy was already lost.
Same shape as 82452789c9 did for the homeassistant owner.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2383311b7c declared the extension as org.xwiki.contrib.llm:application-ai-llm-ui.
That artifact does not exist at version 0.7, so the extension manager fails while
resolving it and nothing installs.
Measured against the registry the role reads:
200 .../org/xwiki/contrib/llm/application-ai-llm-models-ui/0.7/
404 .../org/xwiki/contrib/llm/application-ai-llm-ui/0.7/
The 404 body names the missing path itself, so this is not a redirect or a
substring artefact.
The version stays at 0.7; only the artifact id was wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The container_address lookup does not return a container name in swarm. It
returns a shell fragment, "$(/usr/bin/resolve-container-id <stack> <service>)",
which is only a name once an interpreter evaluates the substitution. The shell
module provides one. The command module does not -- it execs the binary directly,
so the fragment arrives at container exec as literal text and the call dies with
"No such container: $(...)".
Compose hides the entire class: there the same lookup yields a bare name and
command: works. The failure exists only in swarm, and no swarm run had touched
these paths -- both were added by feature commits (b23653adf8, 0421ebeb16) that
wrote the same argv shape independently, which is the argument for a rule rather
than six edits.
Six call sites moved to the shell module: the moodle egress task, the three
moodle AI-gateway tasks, and the two wordpress AI-Engine tasks. The wordpress
pair keeps its stdin, so it uses the dict form with cmd:. Moving off argv exposes
the surrounding values to word splitting, so they now go through quote -- the
address itself must NOT, because single quotes would suppress exactly the
substitution this depends on.
test_argv_exec_needs_shell.py closes the gap that let this through.
container-exec-resolver checks that the resolver is used; it has nothing to say
about whether anything can evaluate the result. The new rule checks that other
half. It reuses the sibling's argv parser, which recognises only container/docker
in first position, so an argv list that does start an interpreter
(argv: [bash, -lc, ...], as joomla does) is correctly not matched.
Run against the tree the rule reported exactly the five sites found by hand plus
bigbluebutton, and passed over joomla and the discourse task, whose address is a
config lookup rather than the resolver. Bigbluebutton is pinned to compose via
compose_mode_force and can only ever see a bare name, so it takes a nocheck.
Five container-exec-resolver markers disappear with the argv form: under shell
the resolver is the ordinary path, not an exception.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nextcloud seeds the administrator once, at install time, from NEXTCLOUD_ADMIN_USER
and NEXTCLOUD_ADMIN_PASSWORD in the environment file, and no later run touches the
account again. A rotated credential therefore leaves the instance authenticating
against the old secret while every task in the role still reports success -- which
is exactly what the CI proves against, rotating every inventory credential between
the sync and the async pass.
The reset runs only when the declared password differs from the one this script
last applied, compared through a sha256 marker kept in the config volume. An
unconditional reset would have worked too, but it reports changed on every single
run and turns a rotation check into noise. The marker's cost is stated in the
script header: a password changed outside this script is not noticed.
The comparison cannot be an authentication probe. Nextcloud has no occ command
that verifies a password, and the HTTP endpoints that would are not reachable from
inside an fpm image, where no web server listens in the same container.
OC_PASS reaches occ through the environment and never through an argument, so the
secret stays out of the process list on both the host and the container, the same
way dc32574873 took the redis password off the exec command line.
The account this touches is the local administrator. The LDAP resolution in
07_admin_identities.yml only collects additional identities for the same person,
and the MCP service account reset added in e3caa074b2 targets a different user.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both renderers build their output as a list of lines seeded with the "networks:"
header and append one line per attachment. With no attachment to append they
still returned the bare header, and a mapping key with no children parses as
null, not as an empty mapping. Compose then rejects the whole file instead of
ignoring an empty section.
The two functions do not share their non-empty paths and disagree on where the
newline goes -- render_compose_networks terminates with one, render_container_networks
leads with one -- so the guard is written out in each rather than factored into a
helper that would have to know both shapes.
Both callers already treat an empty string as "insert nothing", so returning it
needs no change on their side.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1f870b1e5 gave the patch one call site and put it in post_deploy.yml, which runs
after the deploy has waited for the stack. 3831c8f804 had named the constraint in
its own subject line: before the health gate. An app whose health check
authenticates against its own database fails that check with the password it
persisted, compose gives up with "dependency failed to start: container matomo is
unhealthy", and nothing behind the gate ever runs -- including the patch that
would have fixed it.
Measured, not argued: with the call in post_deploy.yml a local compose deploy of
five roles passed its first pass at ok=4571 failed=0 and died in the second, the
one that carries the rotated credential. With the call back in front, all three
passes report failed=0 and the Playwright specs pass in both.
sys-svc-rdbms/tasks/main.yml is where it goes: the stack include reaches it
before anything comes up, it is per-consumer rather than per-role-once, and it
still resolves the declaration through the same path, so there is still exactly
one call site. The patch reads the volume from the host and needs no running
container, which is what makes the earlier position possible at all; before the
first deploy has created the volume it exits without doing anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1f870b1e5 left two marks the layout rules reject. friendica's utils/ held
patch_config.yml beside patch_db_credentials.yml, which is a shared
delimiter-bounded prefix, and its numbered files ran 00, 01, 02, 04, 05 with
nothing at 03 after the config patch moved out of the sequence.
The prefix becomes the folder it was standing in for, the way 8e8dbbd08c did it
for the AI files. All four roles move, not only the one the lint could see: the
path post_deploy.yml resolves a role's declaration through has to read the same
everywhere, so a folder in one role and a prefix in the other three would be no
domain at all.
utils/patch_db_credentials.yml -> utils/patch/db_credentials.yml (4 roles)
utils/patch_config.yml -> utils/patch/config.yml (friendica)
The numbers close up behind the file that left: 04_addons and 05_user_bootstrap
become 03 and 04.
Four references follow: the include in 01_manager_ops.yml, the lookup in
post_deploy.yml, the caller path in the unit test that reads each role's
declaration, and a requirement doc that linked the old addon file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f85996b43a turned the tmpfs options into a dict so a declaration could carry a
mode as well as a size, and coerced the mode to the int docker expects. The size
sitting next to it kept being copied through verbatim. The short form accepts
"512m" and so does a human writing the declaration; the long form this renders
takes an integer and refuses the string rather than parsing it:
services.checkmk.volumes.1.tmpfs.size must be a integer
That is the whole of Checkmk#0 in run 33317483326, and it stops the deploy before
the role does anything.
The suffix is resolved the way the mode already was, in a helper beside it, and
a value that is neither an int nor a byte count is refused instead of passed on
-- passing it on is what let "512m" reach compose in the first place.
The existing test asserted {"size": "64m"}, so it pinned the defect: the suite
stayed green while CI failed on it. It now asserts the byte count, every unit a
declaration may write resolves, and three malformed sizes raise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3831c8f804 gave matomo, moodle and nextcloud a patch each, called from their
own 00_core.yml. That file only runs while the role's run-once flag is unset,
so the patch runs on the deploy's first pass and never on the second -- which is
the only pass where a rotated credential exists. The app keeps the password it
was installed with while the engine moves on, and every request answers 500.
Run 33317483326 shows it on friendica, which had the same shape: svc-db-mariadb
altered the user at 15:39 and again at 16:26, either side of the 15:58 rotation,
while the app's config was patched once. The engine said what followed:
[Warning] Access denied for user 'friendica'@'192.168.201.4' (using password: YES)
Five failed healthchecks in a row, swarm replaced the task, and the convergence
gate reported "tasks not running yet" for a service that started correctly every
time.
The call moves to post_deploy.yml, which runs after every deploy and sits behind
no role's flag. It includes roles/<app>/tasks/utils/patch_db_credentials.yml when
that file exists, so a role declares what to patch and nothing else: friendica
joins the other three with its volume, its config path and its expression.
It cannot live in the role. 00_core.yml has to set the run-once flag as its very
first task or load/app.yml re-includes the role and recurses, which
tests/lint/ansible/roles/test_service_core_first_task_run_once.py enforces, and
everything after that flag is closed to a second invocation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
383a59e9a1 gave the shared engine a way to be told a rotated superuser
password: swap pg_hba to scram, ALTER USER over the local socket, verify the
new password over TCP, put pg_hba back. An engine an app owns alone never got
that, so after the deploy rotates a credential the app holds the new password
and its own postgres still holds the old one, and every request answers 500
until someone recreates the volume.
The script is now told which role to alter instead of assuming postgres. Both
engines are reached the same way -- the container's local socket authenticates
by peer, which is the whole reason a password can be written into an engine
nobody can log into any more -- so one implementation covers both. The role
travels as an environment variable and is applied through psql's identifier
quoting rather than string interpolation.
MariaDB solves this at engine start with --init-file (b067d21ef5). Postgres has
no equivalent and /docker-entrypoint-initdb.d only runs on an empty PGDATA, so
this has to be an ansible step after the container is up. post_deploy.yml is
where it goes, right behind the health wait, and that gate is passable here for
a reason worth stating: the dedicated healthcheck is pg_isready, which does not
authenticate, so the container reports healthy on a stale password and leaves
the window this task needs.
Seven jobs of run 33279942812 fail this way -- mattermost, pretix, xwiki,
gitlab, opentalk, flowise, discourse -- each traced to the same rotation gate
that exists to prove a changed secret propagates.
Exercised against a stubbed container for both roles: postgres and an app role
behave identically, the role reaches all nine calls, ALTER USER fires. It still
refuses to run without a role, and still reports FAIL when authentication never
succeeds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
db66e01020 gave chunk_gate the job of stopping the chain at the first red
chunk, and each gate reads its immediate predecessor to do it. A blocked chunk
reports skipped, and skipped is what the gate accepts -- for good reason, since
that commit also lets a block whose slice is empty skip itself. So the chunk
after a blocked one sees a pass and runs.
Exactly one chunk falls out, the chain carries on, and the rows that chunk
owned get no verdict at all while nothing says so. Run 33279942812: chunk 0 red
with 13 of 44, chunk 1 skipped by the gate, chunk 2 deployed 81 jobs behind it.
Chunk 2 now reads chunk 0 as well as chunk 1, and needs what it reads. Chunk 1
already had only one predecessor and is unchanged. The allow-list keeps
skipped: an empty slice still has to pass.
The finish job was already right and stays untouched. It fails on failure or
cancelled and not on skipped, which is correct there for the same reason it is
wrong in the gate: a chunk this skips was skipped because an earlier one failed,
and that failure is what it sees.
The test pins both, because a fourth chunk written from the third's template
would repeat this: every gate names every chunk before it, needs what it names,
and the finish job needs them all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
68adb944ff made the constructor the sole opener of the preload context so
list_or_shoot would queue instead of shoot during the Bootstrap block, and the
loader's flush would run the queue once the shared services were up. Nothing
closes it again. After the flush the flag is still true, so every later
list_or_shoot appends to a queue nobody drains.
Where a shared service gets preloaded the leak is masked: preload_service.yml
clears the flag around each service, and the flush had items by then. It is
masked, not absent -- in three healthy jobs of run 33279942812 the last of the
43 mark/clear events is a mark, so the flag is true when the loader returns
there too, and the 43 to 59 appends that follow are discarded in silence. The
first of them is sys-ctl-alm-email's "Include mail dependency", which is why
the mail dependency never loads.
Where no shared service is preloaded there is nothing to mask it. The flush ran
once at 08:48:23 against an empty queue and all six appends came after it, so
eight sys-ctl roles never executed, the play still reported failed=0, and the
post-run check in cli/administration/deploy/dedicated/runner.py caught it with
exit 2. Twelve jobs across eight apps in that run failed exactly there.
The opener stays where it is. Two appends per healthy job land before the
loader opens its own first context, so preload_service.yml alone would not
cover them -- which is the case 68adb944ff was written for.
10f320df98 already assumed this flag flips: it pulled two roles out of their
once-gates so work would "re-evaluate after the preload-context flag flipped".
After the loader it never did. This is the condition that commit needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two switches and an alias for the case f4c5a7cc36 was not written for: a run
where most of the ranking is red and much of it never ran.
--roles-only collapses the failed selections to their role ids. f4c5a7cc36
replaced exactly that form with pinned tokens, and its reason holds: the
rotation may not draw the combination that broke. It holds for a handful of red
rows. At thirty roles the question stops being which combination broke and
becomes whether the role comes up at all, and a token per combination spends
the priority budget proving the same role red twice. The default is unchanged;
this is the trade, named, behind a flag.
--include-unrun puts the rows the source run holds no job for on the priority
line, with their axes. resume_offset covers a run that walked a contiguous
prefix; it stops at the first gap, and a run whose priority line consumed the
budget before the head of the ranking leaves a hole at row one. Then the offset
is empty, the retrigger starts over, and the rows nobody judged queue behind
the ones already settled.
With --strict the two agree on what settles a row. Only a success or a failure
does. A cancelled row, or one still running when the run was read, was never
given the chance to be either, so it comes back as itself rather than as a role
name that would let the rotation draw some other combination instead. On the
run this was written against that moves 24 rows: 123 started, 99 judged.
Membership is decided on role#variant. Mode, onion state and distro are
assigned by the sweep rotation, so comparing full tokens between a source run
and a freshly computed ranking reports almost every row as unrun -- measured,
266 of 269, against 177 once the identity is right. Each ranking row is a
distinct role#variant, which is what makes it the identity to compare on.
i8ciall carries the combination, next to i8cifail which reproduces the exact
red rows and i8cifo which takes only the hard failures.
resume_offset and the new code now build their tokens through one helper
rather than two copies of the same five lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
880bec33bb wrote the regex across three lines; it fits on one under the line
limit, so `make quality` rewrites it on every run. The pattern is unchanged -
identical once whitespace is stripped from both revisions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four tasks in this file talk to the gateway, and each carries its own
tolerance for it not being up: 60 seconds for the keys, 50 for the closure
proof, 50 for the model warm-up. The first one to run is the keys, so a gateway
that takes longer than a minute fails there and the later budgets never apply.
54a6f3b533 already named this when it added the warm-up: "Five retries cover a
gateway that is not listening yet." It gave the last task 50 seconds and left
the first one at the 60 it already had.
One wait in front of all four, on the signal the failure actually reports.
CI shows ConnectionRefusedError from urllib after eleven attempts, which is the
TCP layer, so the probe opens a socket rather than asking an HTTP endpoint whose
path would be a guess. python3 is what the failing task itself runs in that
container, so nothing new has to exist there.
A healthcheck would have been the obvious shape and would have changed nothing:
sys-svc-compose/tasks/utils/post_deploy.yml waits on the database container's
health in compose and on swarm tasks being Running in swarm, and reads the
application container's health in neither.
This does not explain every job in that cluster. One role's rescue artifact
shows all three replicas reaching "Proxy initialized with Config" while
postgres resets and then terminates their connections (SqlState 57P01), which
is a different failure wearing the same stack trace. Where that is the cause,
this makes the deploy fail at a task that names the gateway instead of at the
one provisioning keys.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The body was a Jinja-interpolated shell string inside the task. As a file it
can be read, shellchecked and run, and the three parameters travel through the
environment the way the password already did.
That forces one thing into the open. The old expression carried
${PATCH_PASSWORD} and the shell expanded it while assembling the command; a
value arriving in an environment variable is not expanded a second time, so the
substitution has to be explicit. The consumers now write @PASSWORD@ and the
script replaces it, which is also the version a reader can follow without
knowing that rule.
The script refuses to run without its parameters rather than treating an unset
one as empty, because an empty volume name would look exactly like a node that
does not hold the volume: a silent success on every host.
Tests move with it, to the files/ segment every other test of a shipped script
uses. They run it against a stubbed `container` in three states per consumer -
no volume, volume without the config, both present - and assert the marker the
task reads as a change. Five mutations of the script were checked against them:
dropping the substitution, the marker, the existence test, the parameter guards
and the volume filter each turn the suite red.
Layout follows what the linters already require: files/shell/ is enforced by
role-files-layout, and 65e34746fc names ansible.builtin.script as the sanctioned
way to run one from a task.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule read one of the two ways a fact is written, and it read only
templates. Both gaps hid the same kind of value.
Bracket subscripts were invisible: ansible_facts.default_ipv4 was matched,
ansible_facts['default_ipv4'] was not. Both forms are now generated from one
key list rather than spelled out.
Role vars and defaults were out of scan. A fact assigned to a variable there
reaches the template exactly as if it had been written in the template, and the
only real violation in the tree was sitting in that blind spot, which is how it
survived the rule that exists to catch it. The scan covers vars/main.yml and
defaults/main.yml through the mapping constants, 839 files where it saw only
templates before.
Interface addresses were not a pattern at all, only the named default_ipv4. A
resolver reading the docker bridge off ansible_facts['docker0'] cost thirteen
jobs in one run and was removed by hand; the rule that should have caught it did
not know the shape. Any ansible_facts[<iface>]['ipv4'] is now matched, and the
legacy ansible_<iface>.ipv4 with it.
What stays permitted is the distinction the rule turns on. A fact that ADDRESSES
a host travels wrongly. A fact that describes its PLATFORM does not: os_family,
architecture and processor_vcpus choose behaviour for the node the rendered file
is consumed on, which is the node it was rendered for. All seven ansible_facts
reads in the tree were read individually; five are of that second kind, and a
rule needing a suppression on five correct lines out of seven is not a rule.
svc-net-tor is the sixth and seventh. It reads both the bridge address and
default_ipv4, and meta/services.yml pins the role to the manager, so the fact
describes the node the hidden service forwards on. Declared rather than left
looking like an oversight.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Home Assistant writes the owner password into the auth store on its volume
during onboarding and never reads the environment again. When the deploy
rotates credentials the container gets the new value and the store keeps the
old one, so the login that every later step needs is refused.
The provisioning script returned that refusal as an ordinary step, and the
/auth/token call that followed failed on a missing result key - a message about
the transport for a problem in the credentials. It now exits on any login flow
that does not reach create_entry, naming the account.
That turns a five-minute wait into an immediate branch: the retry loop stopped
only on success, so a password that will never be accepted was tried thirty
times. The loop now also breaks on the refusal, and only that one outcome is
allowed past failed_when - a crash or a traceback still fails the task, which
matters because the step after this reads a token out of stdout.
hass --script auth change_password reaches the store through the filesystem, so
it needs no working login. The hub holds the store in memory, hence the restart
before the same mint runs again.
e3caa074b2 fixed the account one layer in: the MCP service account, through
admin_change_password, which needs an admin token. This is the login that
issues that token, so without it that fix cannot be reached at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b067d21ef added the realignment helper as an argument to compose_volumes(),
so it now runs before that function reaches its own guarded call. A role with
both engines enabled therefore raised the bare ValueError through the lookup,
and the message that names the collision and says to pick one dbtype per role
had become unreachable. The helper catches it and defers.
No role can trigger this today: over all 265 meta/services.yml, none declares
both keys at all, and `enabled` has a single shape across the tree, the literal
True in 54 places. The guard protects the diagnostic for whoever adds the
second engine, which is exactly when a bare ValueError would be least useful.
The tests pin what the two startup failures cost to find. The SQL must stay one
ALTER USER IF EXISTS: root must not appear, because giving it a password during
the entrypoint's temporary server breaks a readiness probe that connects
without one; CREATE USER must not appear, because the entrypoint's own carries
no IF NOT EXISTS and the script runs under set -e; and the GRANT must not come
back, because rotating a credential changes passwords, not privileges.
Verified by rendering the version those tests were written against: it puts
root, CREATE USER and GRANT in the file and four statements in place of one, so
every assertion fails on it. The deferral was checked the same way, against the
module as it stands in HEAD.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The database seed, the websocket map and the manager operations each
repeated the same IS_STACK_HOST condition, and the third of them was
already a task file holding everything else the stack host does. Moving the
other two into it leaves one gate at the include site.
32c33ee5d4 left these two where they were when it created that file. Both
already carried the gate, so they never ran on a worker; what that commit
needed ungated is the run-once flag, which still sits outside.
The backup condition stays on the seed task, since it asks a different
question than which host owns the stack. Order is unchanged: the two are the
first tasks in the file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
