Four MCP clients each carried a services entry for homeassistant, and
openwebui and flowise one for baserow, with matching run_after lines. Those
predate the admission refactor: since then the co-deploy edge lives on the
provider, where mcp-consumer-services-entry enforces it, and homeassistant
and baserow already carry an entry for every client they admit. The client
copies stated the same relation a second time.
The asymmetry gives them away. Open WebUI consumes twenty MCP servers and
declared three of them; the other seventeen work because the bridge is built
from MCP_DISCOVERED_SERVERS, never from the client's own services map.
Drop the run_after lines in the same move: a run_after target must be backed
by a services entry, so one cannot go without the other. Hermes and OpenClaw
lose the whole key, homeassistant being its only entry, because a bare
run_after parses as null rather than an empty list.
Keep qdrant on flowise, which renders QDRANT_URL and QDRANT_API_KEY into the
container and is a real dependency, and keep prometheus everywhere as the
ordinary metrics edge.
Open WebUI stops dragging both providers into every round it joins, which
brings moodle variant 2 from 64.75 GB back to 62.75 GB, under the budget the
preceding commit pushed it over.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mcp-consumer-variant guard demanded three things of a provider's MCP
round: every admitted consumer enabled, each of them pinned `services: null`
so it dragged no dependencies along, and no other service left enabled. Two
of those were the rule legislating a round's contents rather than checking
that the pairing gets proven at all.
Keep the first and drop the rest. What else the round carries is the role's
own call, so the tor/litellm exemption, the datastore exemption and the
`allowed` set they fed have no reader left and go with them.
Strip the 63 `services: null` pins the rule forced across 20 variants, all
written bare as `services:`. The topic-emptying capability behind them
stays: apply_topic still normalises None to {} for the inventory payload,
and the unit tests still cover it. Only the uses and the mandate are gone.
That leaves overridden_providers scanning a repository with nothing left to
find, so its three assertions passed over an empty dict and would have
stayed green however the function broke. Rebuild them on a synthetic role
tree, which also covers the case the repository scan never held: a variant
naming a plain flag is not an override.
Removing the pins changes what deploys. Measured across all 20 variants: 18
identical, prometheus#1 grows 23 to 31 roles, moodle#2 grows 31 to 32 and
crosses the memory budget until the following commit shrinks it back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A bond lives on the consumer's side of a service relation, so seeing how two
roles couple means opening both roles' meta/services.yml and comparing, and
retuning one means editing a scalar buried among dozens of unrelated keys.
Add `infinito meta roles applications bond` (make bond / i8bond): it resolves
every services.<key>.bond to a role-to-role edge through the service registry
and serves the result as a matrix, each cell carrying both directions because
the two need not agree. Brightness is the bond, black for none and white for
1, so the coupling of the whole repository reads at a glance.
Clicking a cell writes the new value back to the declaring role. The write is
line surgery rather than a YAML round-trip: these files carry `# nocheck:`
directives and hand-tuned ordering that re-dumping would flatten, so only the
bond scalar moves and a trailing comment on that line survives. The browser
sends a role pair, never a path, and the server resolves the service key from
its own edge map, so a pair it does not know is refused before any write.
`--out FILE` exports a read-only page and `--format json` the raw edges, for
callers that want to post-process rather than edit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
register_new_matrix_user only creates. On PASS 2 of a matrix-deploy round the administrator and the ChatGPT bridge bot already exist carrying their PASS-1 password, the call returns 'User ID already taken', and both tasks tolerate that. chatgpt_bridge_token.yml then logs in with the rotated password against an account that still holds the old one and burns ten retries, which is where job 99670614406 of run 33445831564 ended.
A new realign_synapse_password.yml hashes the declared password with Synapse's own hash_password and writes it into the users table, addressing the engine through lookup('database', application_id, 'container') the way realign_dedicated_db.yml already does. It runs for both accounts and only when the registration found them, so a first deploy never reaches it.
The hash is handed to psql as a variable and used as :'hash', so it never reaches the SQL parser as text, and it is passed through Ansible's quote filter, so the bcrypt dollar segments survive the shell. Verified by running the rendered command against a stub that prints its argv: hash=$2b$12$abc/def$ghi arrives as a single unsplit argument.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4e46858202 closed the auth gate: meta/services.yml admits only the administrator and mcp groups. biber is provisioned with no roles, so the oauth2-proxy rejects its OIDC callback and serves a 403. The spec still asserted the opposite, and its comment still described the era when the hook auto-provisioned any Keycloak identity. Both n8n jobs of run 33445831564 burned three Playwright attempts against that contradiction.
The journey is unchanged: same env assertions, same signInViaN8nOidc pass through Keycloak. Only the expectation is inverted, and the load-bearing half is the absence of n8n's surface, carrying all five markers the old positive assertion used. The wording check is secondary and broad, so a reworded proxy page does not decide the verdict.
n8nLogout leaves the spec without losing coverage; it is exercised twice in test-login-administrator.js and once in test-litellm-gateway.js, and a persona that never signs in has nothing to sign out of.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The credential was baked into the image: files/Dockerfile appended xwiki.superadminpassword to WEB-INF/xwiki.cfg from a build arg. The swarm matrix rotates it between passes, so the manager rebuilds the image with the new value, but 'docker stack deploy --resolve-image never' leaves the mutable tag untouched and a worker that already holds it never re-pulls. The task keeps serving the pre-rotation password and answers 401 to every authenticated call the deploy makes next, which is where both XWiki jobs of run 33445831564 died: 03_administrator.yml:7 and 04_extensions.yml:25, after 31 attempts each.
The entrypoint writes it at start instead, from the environment the role already renders into templates/env.j2 and that sys-svc-container/templates/base.yml.j2 attaches as env_file. Both copies of xwiki.cfg are written, because the stock entrypoint restores the data volume over WEB-INF on every start but the first; writing only WEB-INF would survive one boot and be reverted by the next. The keys are deleted and re-appended rather than substituted, so a generated password never reaches a sed replacement.
Read from xwiki/xwiki-docker at master to settle three assumptions: other_starts() does restore xwiki.cfg, the stock entrypoint never touches the superadmin keys, and CONTEXT_PATH defaults to ROOT. The delete pattern was exercised against all four key spellings xwiki_replace accepts, and a password carrying sed metacharacters was verified to arrive byte for byte.
Dropping the build arg also keeps the credential out of the image history.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The replay is a second deploy on the same runner, and mark() only ever checked the variant number of a row, never its mode. So the marker landed on whichever row of the smallest variant came first, which for 38 of 98 marked rows was a swarm one: that runner would have had to bring a compose stack up beside a swarm stack.
Two places needed it. The row now has to run in the mode guide_deployable names, and only rows in that mode count as deployed when the search looks for the smallest variant - without the second half the search settles on a variant this sweep runs in swarm alone and the marker falls away silently.
Measured over the full matrix: 98 marked rows become 71, swarm drops from 38 to 0, compose rises from 47 to 59. The 27 roles that lose the replay entirely are the ones this sweep deploys in no compose row of an eligible variant; getting them back means forcing such a row into the sweep, which is a change to matrix selection rather than to this marker.
mark() had no tests. It has eight now, including the two failure modes the old code showed under a negative control: a swarm row carrying instructions=compose, and a compose row left unmarked because a swarm-only variant won the search.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GET /api/v1/tools/ builds its answer for an MCP server out of stored config alone - routers/tools.py:125-126 reads info off the connection row and never touches the network. So the three specs that read it pass unchanged against a provider that is down, refuses the deployed bearer, or serves nothing. What they establish is registration and the OIDC group to access-grant chain, not that the bridge carries.
POST /api/v1/configs/tool_servers/verify is the one route in 0.11.0 that does a real MCP initialize plus tools/list without an LLM, and it authenticates with the key in the body it is handed. The spec reads the stored connection and echoes it back, so the handshake runs over the deployed URL with the deployed credential through Open WebUI's own client, and asserts three things per server: it answers, it serves at least one tool, and the tool set contains the read probe its meta/mcp.yml declares.
Only bearer connections are verified. A system_oauth one resolves the caller's own upstream token per request and the administrator holds none, so it would fail on the caller rather than on the bridge; no provider declares auth: oidc today, but the filter keeps the spec honest when one does. The per-test budget scales with the number of registered servers instead of a fixed guess.
MCP_EXPECTED_TOOLS carries the declared read probe per server. Its rendering is unit-tested by extracting the block from the template itself rather than restating it, and decoded the way a spec decodes it - dotenv_quote emits an escaped JSON string that decodeDotenvQuotedValue unwraps before the spec parses the payload, so comparing after one decode would test a value no spec ever sees.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The greedy set-cover retires a role's variant-0 row when an earlier row of another role embeds it, on the premise that embedding pulls the provider at variant 0. The premise is a property of the pair, not of the coverer: a job's round index is the primary role's own variant index, and plan_dev_inventory_matrix hands each pulled-in dependency 'round_index if round_index < its variant count else 0'. A variant 3 coverer therefore pulls a 3-variant provider at 0 and a 14-variant provider at 3.
Coverage now asks that question per pair instead of assuming the answer. Nothing changes on the current tree, where 53 rows are covered before and after: the discovery sort ends on 'asc variant', so a role's variant-0 row is always green before its higher ones and gets to cover first. That is a property of a sort key in default.env, not of this code, and 2633 embed pairs would fall the wrong way without it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A round deploys a provider once, so the role that pulls it in is the only place that can say how lean it has to be. A variant entry may now carry any application config topic under a service it depends on: a mapping deep-merges, so naming one key changes that key alone, and null empties the topic. Empty was the case that needed a marker of its own, because merging an empty mapping is a no-op and would leave exactly the services the round was trying to shed.
The topic vocabulary and the merge semantics live in utils.roles.applications.topics, which is also where resources.py now gets them rather than spelling the same null-or-merge out a second time. Collecting who demands what stays in the inventory tree, refuses two roles demanding different values for one provider in the same round, and skips a role's own entity entry so the existing services.<own>.users declarations are not read as overrides.
The CI set-cover learns the same fact. Its greedy cover retires a role's variant-0 row when an earlier row of another role embeds it, on the stated premise that embedding pulls the provider's variant 0. A variant that dictates that provider's config breaks the premise from the other side: whatever ran, it was not the provider as the provider declares it. Such a pair now covers nothing, so the retired row keeps its job, its artefact and its verdict.
Measured on the current tree: 20 override pairs, 41 covered rows before and after, so nothing is revoked today. The rule bites the moment an override lands in a row that covers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The contract check ran as an Ansible task in the middle of every provider deploy, and the two client checks ran the same way. A test that is part of the deploy cannot be run without deploying, cannot be re-run on its own, and reports its verdict as a deploy failure. Twenty-seven task files lose 491 lines to this.
It is now what it always was. Each provider ships files/test/test.sh, templates/test.env.j2 and meta/tests.yml, and test-e2e-cli stages one shared harness beside it: run.sh drives contract.sh in a throwaway container on the provider's own network, then isolation.sh for adapter-fronted providers. The read probe each provider offers moved from the calling task into meta/mcp.yml next to the allowlist it must satisfy, and the schema now rejects a read probe naming a tool outside that allowlist.
The bearer the test presents is the one a client resolves through mcp.credential, via the new mcp_client_bearer lookup over resolve_credential, which moved to utils.roles.applications.mcp so it is the same function mcp_servers renders every client connection with. The probe presented whatever the calling role passed, which for the fifteen adapter-fronted providers is a different secret than any client holds.
Sidecar isolation is read from the engine rather than measured inside the container, so it no longer needs a delegation to the node holding the task: attachment is compared by network name, which drops the gwbridge special case, and a swarm worker says it cannot measure instead of skipping silently.
Retrying is bounded by wall clock against MCP_DEADLINE_SECONDS, which scales through the same lookup('timeout') as the harness budget it has to stay under. A flag that is neither true nor false aborts rather than reading as off.
mcp-provider-probe becomes mcp-provider-cli-test: an enabled provider must ship the test rather than include the task. mcp-probe-bearer becomes mcp-adapter-bearer, tying ADAPTER_BEARER to the declared credential key now that no probe include carries it. A new integration test renders the shared env template for every provider through Ansible's own templar, so a later change to it breaks visibly instead of poisoning twenty deploys quietly.
Hermes and OpenClaw keep their client-side proof as CLI tests of their own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
32dd66099 gave web-app-jenkins the third consumer its neighbours already
had, and the report reads that list from the same mcp_consumer edges the
derivation does. One cell.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
00478b8ead moved the Corefile substitution in-process, to string.Template,
and left this file describing envsubst. That is the page a reader consults
before wiring gettext back into CI, which is what b3686a977 just removed.
grep for envsubst across the tree now returns one hit, the docstring saying
why it is not used.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
83c1c729de translated jenkins' hand-kept allowed_consumers verbatim, and
that list had never carried web-app-openwebui: git log -S over the role
finds no commit removing it. Its own message names that shape as the
problem — twenty lists of which fourteen were identical — and jenkins was
one of the six that diverged.
Nothing in the role justifies the divergence. Against gitlab, which admits
Open WebUI, jenkins matches on auth, transport, exposure, implementation,
read_only_default and mutating_tools_enabled, and bearer_token is in
BEARER_PRESENTABLE_AUTHS, so tool_server_connections registers it with a
credential it can present. The reason that carries wordpress's refusal,
basic_auth, does not exist here.
The entry takes the shape hermes and openclaw already have and the variants
mirror them, so the pairing is planned into the same round as the other two
clients rather than a new one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6ac48b7b8 gave talk_recording an enabled, which is what makes
test_env_services_match look for a TALK_RECORDING_SERVICE_ENABLED line: an
entry without the key is invisible to that rule, one with it is not.
The container is reachable only from the Talk signaling backend and serves
no browser surface, so it takes the marker context_agent already carries
rather than a flag no spec could gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A compose.yml.j2 that wraps a service key in {% if VAR | bool %} has to reach
that service's own meta/services.yml entry, or the template and the
declaration become two switches for one container: the resource model, the
README graph and the network layer read the entry while the container follows
something else.
The reference may sit anywhere in the chain. The resolver walks the role's
vars/ and defaults/ and follows config paths into the resolved application
config, because nextcloud reaches its Talk entry over
addons.spreed.enabled and a resolver that stopped at role variables called
that a bypass. Only guards indented by at most two spaces count: deeper ones
select content inside a service, not the service.
The rule found one real bypass. NEXTCLOUD_MCP_ENABLED read mcp.enabled while
switching the context_agent container, whose entry admitted two peers where
mcp.enabled admits three, so a round carrying only web-app-openwebui ran a
container its own entry declared off. mcp.endpoint.path routes through
app_api/proxy/context_agent/mcp/, so the container is the MCP surface and the
entry was the stale side. Both ends now meet at the entry.
talk_recording carried no enabled at all and fell back to
_looks_like_container, so the resource model billed it unconditionally. It
now reads services.talk.enabled, which is what its guard already computed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Thirteen of sixteen MCP sidecar entries carried no enabled at all, so
collect_role_resources fell back to _looks_like_container and billed every
one of them as running. Three roles already gated theirs on mcp.enabled;
the rest now do the same, gitea twice because it is the only role with a
vendor container behind the adapter.
The fifteen adapter roles then read services.<key>.enabled instead of
mcp.enabled, so the compose block and the task gates follow the services
entry rather than running beside it on a second path. The six plugin and
native roles keep reading the topic: they ship no sidecar entry to point at.
The three-level resolution is not new. KEYCLOAK_LOGOUT_PANEL_ENABLED has
the same shape and depth and gates a task the same way, and the same value
reaches JAVASCRIPT_SERVICE_ENABLED, where service-gating.js throws on
anything but "true" or "false" — an unresolved chain would have been
killing keycloak's Playwright run since 575a5a87ac.
This does not change what the resource model counts: _is_enabled reads the
unrendered Jinja and still treats it as on. What it buys is one switch
instead of two, and a declaration that matches it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
83c1c729de gave every provider a group_names-gated services entry per MCP
client, including the clients it refuses with mcp_consumer: false. The gate
is what makes the inventory resolver plan provider and client into one round,
so 19 of the repo's 82 co-deploy pairs started a client for a pairing the
provider had already declined. flowise is a live MCP client of qdrant,
homeassistant and prometheus; the other 17 providers were bringing up its
node stack for nothing.
The entries now read enabled: false, shared: false, mcp_consumer: false, and
the variants that pinned them to true are gone. Setting those variants to
false instead is not an option: test_variants_no_static_override forbids any
variant override of a literal base value in either direction.
bond leaves with them, and a new lint holds that line: collect_role_resources
skips a literally disabled entry before it reads any resource key, so a bond
there describes a cost nobody pays. Thirteen entries unrelated to MCP carried
one and are cleared too.
test_mcp_consumer_services_entry demanded the group_names shape for every
client entry, refused ones included, and now demands the opposite there.
dynamic-flag keeps its rule and takes its own documented per-key exemption.
make docs redraws twenty graphs: the edges go, the nodes stay as
'openwebui ❌' and 'flowise ❌', so the refusal is visible rather than read as
an integration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
509cd19985 added gettext-base because the Act runner image lacks what
GitHub's ubuntu-latest ships, and coredns.py shelled out to envsubst. That
only ever covered the outer runner: the nested deploy containers it also
failed in are not reachable from an apt call in the job.
00478b8ead removed the dependency instead, substituting the Corefile through
string.Template in-process. git grep envsubst at HEAD now finds four hits and
none of them is a call: this step's name, the docstring that explains the
change, and two stale README lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Exception tag labelled prose that restates why the names carry hyphens.
The reason belongs in the history of ffdef28da, not beside the value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b8a6205396 put INFINITO_SWARM_TEARDOWN_RESERVE_SECONDS into default.env and
made 00_one.sh require it with a :? guard, but wired it into no handler.
default.env is only the source of the value; a key reaches the generated .env
solely through a handler's setdefault, so make dotenv dropped it and the
swarm routine aborted on its own guard.
The two nocheck markers cover the neighbouring key on those lines:
INFINITO_CI_DISTRO_DEADLINE_EPOCH is computed by distros.sh from the sweep
budget at run time, so a static default would be a stale timestamp. They have
to sit on the same line; above the statement the marker does not apply.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
WordPress mints an application password, writes it to the token store and
every MCP client is configured with it, but nothing ever presented it to the
endpoint. credential.owner and auth_subject stayed declarations, and the
role's Playwright specs cannot close that: a browser holds no bearer, and a
run that did would capture it in its trace.
The role is also the only one with exposure: public, and its implementation
is plugin, so no adapter sidecar filters the served surface. That makes the
probe's tools/list comparison the only runtime evidence behind
mutating_tools_enabled: false; an extra registered ability or an adapter
release that grows a tool would otherwise ship silently past metadata that
still promises three read tools.
The include sits after the plugin loop rather than in tasks/utils/mcp.yml
where the credential is issued: 05_enable_plugin.yml runs the addon task
before utils/install/plugin.yml, which holds the only wp plugin activate, so
a probe run from the addon reaches an unregistered REST route and a fresh
site fails on a 404 that a second deploy makes disappear.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
enabled and shared were hand-maintained in 24 roles and wrong in both
directions. Both now resolve from the mcp_consumer edges already declared
in meta/services.yml: a server takes the consumers it admits, a client
takes the servers that admit it, and the pair renders as "'peer' in
group_names or ...", or False when no peer is admitted at all. setdefault
keeps meta/mcp.yml and meta/variants.yml as the override channel, so
pinning either key still wins.
The resolved block replaces the mapping instead of being written into it.
_build_role_base_config parks the YAML cache's own object under mcp, so an
in-place setdefault injected the derived keys into every later raw read and
made test_mcp_schema pass alone but fail in a directory run.
The derivation keys on peer presence, never on a peer's own value: a
re-entrant read under _RENDER_GUARD returns the unrendered dict.
Four lints now ask the resolver instead of reading meta/mcp.yml, which they
could not do while the key was hand-written. That is what exposed the eleven
missing hostile-path specs the previous commit closed.
Carries one change that belongs to the WordPress probe but shares a file
with this one: the tool allowlist moves to the hyphenated names the adapter
actually serves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine roles enabled an MCP surface without a browser-side spec, so nothing
ever asked what the endpoint answers a caller with no credential at all.
Lift gitea's assertion into two shared registrars and register them per
role, and give each role the MCP_SERVICE_ENABLED flag they gate on.
mcpEndpointUrl now takes the base URL rather than reading APP_BASE_URL
unconditionally: jellyfin and zammad export their own base variable, and
declaring APP_BASE_URL for them would un-skip the shared guest persona,
which collapses itself on that variable being absent.
The env-gate scanner follows require("./x") into the shared harness so a
gate declared there counts as consumed, and strips block comments first
so service-gating.js's own doc examples do not credit EMAIL, SSO and OICD
to every role that imports it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A root-owned, empty .ansible/ (collections/modules/roles) directory
skeleton can appear in the repo root, from some root-run containerized
process's Ansible invocation resolving its collections_paths default
against the wrong $HOME. Confirmed this is not the project's own
convention: every reference in the codebase (Makefile, scripts/install/
all.sh, .github/workflows/lint-ansible.yml) consistently targets
~/.ansible/collections outside the repo, and .ansible/ has never been
tracked in this repository's history. Regardless of the exact trigger,
this directory is untracked and root-owned, and being previously
unlisted caused 'git stash push -u' to abort mid-cleanup on it
(EACCES), silently leaving other unrelated working-tree changes
unreset.
Moodle#0 of run 33445831564 was cancelled by the 355-minute job cap with almost no
artifacts. The cap did not end a deploy; it reaped a zombie.
00_one.sh wrapped `timeout` around `sudo`, so the guard ran as the unprivileged
runner and the payload as root. An unprivileged SIGKILL cannot reach a root process,
and the same holds for distros.sh's own `timeout -k` one level up. Probed both
shapes: the payload lands in a pgid the outer governor never addresses either way, so
--foreground would not have helped -- the privilege wall is what carries.
The log shows the consequence directly. The governor fired on schedule:
06:18:27.884 distros.sh: line 103: 15914 Killed timeout -k 60 "${remaining}s" "$@"
06:18:27.894 [ERROR] Run failed for distro=ubuntu: rc=137 after 20460s
and then ansible output kept arriving in the same tee'd log -- a TASK at 06:18:27.929,
another at 06:26:00.189, elapsed still advancing -- until
`##[error]The operation was canceled.` The orphan held the step's pipe open, so tee
never saw EOF and the step never returned.
954b9f1ae6 added the TERM trap precisely so a timeout kill would still collect
diagnostics, and its reasoning ("GNU timeout signals the whole process group") is
correct. It simply never gets signalled on the swarm path, because sudo stands
between. That is why this cell carries only inventory- and role-runtimes- while its
compose twin also has rescue-diagnostics- and playwright-. This completes that fix
rather than replacing it.
The ladder was also inverted: INFINITO_CI_DISTRO_BUDGET_SECONDS=20400 (340 min)
against INFINITO_SWARM_STEP_TIMEOUT_MINUTES=350, with default.env:97 asserting the
order as if correct. For the trap to run, the step guard must fire BEFORE the sweep
governor. Rather than re-tune two constants that start their clocks at different
points -- matrix start trails drill start by 1m49s to 2m04s, measured -- 00_one.sh now
derives its ceiling from the governor's deadline minus a teardown reserve, and keeps
its own configured value when that is smaller.
Not fixed here, and it must be said: Moodle#0 does not fit on a hosted runner at all.
Its pass-2 pacing tracks pass-1 within 5%, putting the pass end at roughly 362 minutes
after job start, past both the 355 cap and GitHub's own 360 ceiling. Raising the cap
would be masking and is arithmetically impossible anyway. What this buys is an honest
red cell at ~323 minutes WITH its diagnostics, instead of a silent cancellation with
none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_APP_LOG_TAIL filtered with -size -20M. That is GNU syntax; busybox rejects the
suffix and aborts the whole find, not just the test:
$ docker run --rm busybox find /etc -type f -size -20M
find: invalid number '20M'
$ docker run --rm busybox find /etc -type f -size -20971520c
/etc/localtime
/etc/group
So every alpine-based container returned an empty capture. -20971520c is the same
bound in bytes, and c is the only suffix POSIX requires; both finds take it.
The 2>/dev/null goes with it, and that is the part worth naming: capture() at
container.py:76 already merges stdout and stderr, so the redirect is exactly what
swallowed `find: invalid number '20M'`. The collector reported success while finding
nothing.
Two investigations today paid for this. The dnf cause on LittleJS#0 could not be
settled because /var/log/dnf5.log was absent, and the discourse OIDC failure in
Nextcloud#7 had to be reconstructed from the reverse-proxy log because
/var/www/discourse/log/production.log was absent -- while two vendor mkmf.log files
from the same tree were present, proving the directory was traversed. Both times the
missing file read as a property of the situation; it was a property of the tool.
Nothing depends on the constant: grep over tests/, scripts/ and .github/ finds no
reference to _APP_LOG_TAIL or app-logs.
Introduced yesterday in 8434d35d82, so it has never worked on an alpine container.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ruff wraps the ternary in _sandbox_workers (8ce66075b) and the yaml formatter
reflows db_key_base's description (e3d5ab287); both lines were written over the
limit. No behaviour changes.
autoformat runs as quality-high's first phase, so it corrects a line only on the
run AFTER the one that introduced it -- which is why git status is worth reading
after every gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nextcloud#7 of run 33445831564 fails on three discourse specs, all on one
assertion, all for one reason. The received body is Discourse's own page:
"Sorry, there was an error while trying to authorize your account with oidc."
The browser leg works. What dies is Discourse's server-side back-channel:
"GET /auth/oidc/callback?...code=..." 302 2.125 2.125
"GET /auth/failure?message=openid_connect_request_failed&...&strategy=oidc" 200
Four of those, at 2.125 / 2.015 / 2.063 / 2.014 seconds. That is not the plugin's
10s HTTP timeout; it is Discourse's DNS budget, hard-coded at 2s in
lib/final_destination/resolver.rb (connect_timeout arrives nil, so the default
applies). openid_connect_request_failed has exactly one emitter --
omniauth_open_id_connect.rb:147-155, rescue Faraday::Error -- and the alternative,
SocketError -> LookupFailedError, is not a Faraday::Error and would have produced a
500 rather than the observed 302.
The correlation leaves no room: across 18 back-channel requests, all from
192.168.9.3 with UA Faraday v2.14.3, there are exactly two complete token 200 +
userinfo 200 pairs -- and exactly two specs passed. Thirteen truncated flows,
thirteen failures. No test that got a complete exchange failed; none that failed got
one.
Why discourse alone: discourse.inspect.json:72 reads "ExtraHosts": null. Grepping the
whole rescue bundle for the pin finds nextcloud, nextcloud-proxy, mailu-admin,
prometheus and prometheus-sso-proxy. Discourse could not have it: config.yml.j2's
docker_args: carried only --network= and --name=. It is built by discourse_docker's
launcher, not by a compose file, so the lookup's rendered block has nowhere to go --
hence entries_only, which returns the strings and lets the template write
--add-host= lines. The pin itself is NOT re-derived: _sso_pins() and
_gateway_address() stay the single place that knows compose from swarm.
Its existing allowed_internal_hosts cannot help, because ssrf_detector's
lookup_and_filter_ips resolves BEFORE host_bypasses_checks?; the setting only makes a
private answer acceptable once it arrives.
Rendered through a real Templar both ways: with SSO on, the two --add-host lines land
between the networks and --name; with SSO off the output is byte-identical to today,
so clearnet and variant-1 deployments render an unchanged app.yml and trigger no
launcher rebuild. 24 lookup tests green.
Not closed: the biber persona has never passed, here or in run 33317483326. Its
failure text and code path match administrator, which did pass, so it is explained --
but a persona-specific second defect is not positively excluded. Deploying discourse
variant 0 locally, which already has sso and tor enabled, settles it.
Separately worth queueing: discourse's own meta/variants.yml variant 1 sets
sso.enabled: false and only variant 1 is scheduled, so the role that owns these specs
never runs them. Only partner rounds like Nextcloud#7 catch this class.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both Workspace jobs of run 33445831564 die at
roles/sys-svc-container/handlers/main.yml:9, on the first registry pull a nested
node's dockerd performs after a restart:
docker: Error response from daemon: Get "https://registry-1.docker.io/v2/":
tls: failed to verify certificate: x509: certificate signed by unknown authority
The image name is a red herring. rpardini/docker-registry-proxy SSL-bumps, and the
certificate it presents for registry-1.docker.io and for ghcr.io carries the same
issuer, CN=DockerMirrorBox Intermediate IA -- measured against the live proxy through
the make helper, with three controls: direct fetch on public roots returns 401, the
same fetch through the proxy on public roots only returns "self-signed certificate in
certificate chain", and --cacert /dev/null errors, proving curl honours the flag.
That last control mattered: curl consults its compiled-in CApath as well, where
update-ca-certificates has hash-linked the proxy CA, so without --capath /nonexistent
the negative control passes silently.
So a mirror-name fix would have MOVED the error from docker.io to ghcr.io rather than
removing it. What is missing is trust, and the trust anchor came from
INFINITO_CACHE_REGISTRY_CA_HOST_PATH -- a variable act does not forward, and whose
`make dotenv` rewrites the shared .env back to default.env. It can therefore name a
directory the proxy that actually answers never signed with.
The bootstrap now reads the anchor off that proxy's own /ca bind. And it stops
limping: with the frontend up and no /ca bind exposed, the run aborts naming the
reason instead of continuing untrusting and surfacing three minutes later as a
cryptic x509.
Locally the patch is a measured no-op -- the inspected value is byte-identical to the
.env value, and the render with and without it is identical -- so it changes behaviour
only where the variable is wrong, which is act.
Not proven: that /mnt/cache/registry/ca/ca.crt on the hosted runner is the root the
live proxy presents. It follows from rpardini's design and from the sibling leg that
works, but both host directories are absent from every artifact and that runner is
gone. The line "==> registry-cache CA source: /mnt/cache/registry/ca" in the next
workspace run settles it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Joomla keeps its DB credential in configuration.php on its own volume. CI rotates
credentials between the two passes, so in pass 2 the file holds a password MariaDB
no longer accepts. Job 99670614260 of run 33445831564:
mariadb_mariadb.log 1058 x Access denied for user 'joomla'@'192.168.201.7' (using password: YES)
negative control on the same file: 'wordpress' -> 0
joomla_joomla.1.*.log "resuming normal operations" 02:36:53, then the access log's
first line is "GET / HTTP/1.1" 500 216982, every ~5s
The role's own repair cannot reach that state. tasks/main.yml:21-26 flushes handlers
before including 02_manager_ops.yml, which is the only path to 05_patch.yml: the
converge gate wants a working configuration.php, and the repair wants the gate to
clear.
sys-svc-rdbms already runs a hook on the correct side of that gate, and it already
runs for joomla in both passes -- it skipped for exactly one reason (log line 103788):
false_condition: _app_persisted_credentials is exists
main.yml:29-33 opts a role in by the existence of
roles/<app>/tasks/utils/patch/db_credentials.yml. friendica, kix, matomo, moodle and
nextcloud have one. This adds the sixth.
The expression was checked against the real line read out of the running volume, not
against a sample -- `public $password = '...';` with a leading tab. sed -E with it
returns that one line and leaves dbprefix, secret and smtppass untouched; smtppass in
particular would have been caught without the `public $` anchor. The shared script
cannot silently no-op either: patch_persisted_password.sh:47 prints
"FAILED: no line in configuration.php matches the patch expression" and exits 1.
Verified by deploy, pass 2:
TASK [sys-svc-rdbms : 🔑 Realign the persisted database password wherever joomla_data lives]
changed: true
stdout: PATCHED
An earlier attempt moved the reconcile into the container entrypoint instead. It also
went green, and it was still the wrong mechanism: run against the image, that route
wipes the whole DB block when the credential resolves empty (database.py:104 returns
"password": "" for the no-database branch as a legitimate result) while php -l calls
the file clean; a password containing $0 or a trailing backslash writes a parse error
and still reports "changed" with rc 0; and a lost env var makes exec /entrypoint.sh
unreachable, so the container never starts again. Six files became one, and the path
now runs once per deploy instead of on every boot of every replica.
${PATCH_PASSWORD:?} in the shared script fires on empty as well as unset.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two Moodle#1 jobs of run 33445831564 -- tor/arch and clearnet/debian -- end with the
same line, 65 minutes apart:
FAILURE: no running moodle task on a worker node
The deploy is clean; PLAY RECAP reports failed=0 on all four hosts. Only the
post-deploy drain drill fails, and it fails because nothing is left to drain.
The chain, each link measured rather than assumed:
meta/variants.yml variant 1 turns openwebui/hermes/openclaw/flowise on;
variant 0 turns all four off
extend_inventory.py:88-89 svc-virt-kata in the closure put BOTH workers in the group
svc-virt-kata/00_core.yml:61 labels every group member kata-capable=true
(2 label tasks in each moodle#1 log, 0 in moodle#0)
deploy.yml.j2:18-23 non-kata services get node.labels.kata-capable != true
05_seed_content.sh:12-22 filters running tasks with grep -vx "${MGR}"
With both workers labelled, moodle and nginx have no eligible node but the stack
host. `docker service ps` at failure shows all three moodle_nginx replicas Running
on mgr-01. The drill then looks for a worker task and finds none.
Controls in the same run: moodle#0 carries no kata group and 0 label tasks, and its
drill was reachable; Nextcloud#10 (green, no kata) logs
"initially scheduled on: ...wrk-01" -> "Rescheduled ... to ...mgr-01" -> drill complete.
Exactly three of the run's 18 swarm inventories carry the kata group, and all three
are red.
b20a70646f widened _WORKERS[-1] to *_WORKERS and said so in its own message: "whether
the stack host holds up under it is what the next swarm run measures". The stack host
held up. What broke is the drill's precondition.
The discriminator is the app's own declaration, and it separates cleanly -- executed,
not reasoned:
web-app-moodle kata.enabled=None -> ('swarm-wrk-02',)
web-app-nextcloud kata.enabled=None -> ('swarm-wrk-02',)
web-app-openclaw kata.enabled=True -> ('swarm-wrk-01', 'swarm-wrk-02')
web-app-hermes kata.enabled=True -> ('swarm-wrk-01', 'swarm-wrk-02')
_WORKERS[-1:] restores exactly the pre-b20a70646f tuple for the non-sandboxed case, so
that commit's drain fix stays intact for the rounds it was written for.
svc-virt-kata's own assert (00_core.yml:80-88, labelled_nodes | length > 0) holds with
one labelled worker.
applications_for_round() is a thin public wrapper over the existing
_applications_for_active_variants(get_merged_applications(...)); extend_inventory would
otherwise have to import a neighbour module's private helper.
Three tests cover the branch, which had none: grep -c "kata\|_WORKERS" on
test_extend_inventory.py returned 0 before this.
Not covered: the full chain -- label, placement, drill -- has not run. One
`make act-swarm apps=web-app-moodle variant=1` closes that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1bf28bb2bf added realign_admin_password for exactly the case CI produces -- an
existing server whose credential rotated between the two passes -- and then looked
for the pin file one directory too deep. The path had never run, so nothing caught
it.
Jellyfin#0 of run 33445831564 is that path's first execution. The sync deploy at
04:33 drove the wizard green; the async deploy at 06:03 met the same persisted
/config with a new password. /Startup/* is closed on a provisioned server, so the
wizard attempts produced exactly 40 x `curl: (22) ... error: 401`, and the realign
took over.
Its first step worked. The stderr block holds exactly 40 401s and no 41st, so
POST /Users/ForgotPassword returned 200; and the client IP 192.168.215.1 sits inside
the server's own `Used LAN subnets`, so `isInNetwork` held and no
"initiated from outside the local network" warning was logged -- two other [WRN]
lines in the same log prove warnings are captured. The file was written.
It was written to ProgramDataPath. DefaultPasswordResetProvider.cs:38-39 anchors
`passwordreset` at `ApplicationPaths.ProgramDataPath`, and the failing container's
own startup line reads `Main: Program data path: /config`. /config/data is the
DataPath, the SQLite directory the same log reports as a separate `Storage path`.
The glob never matched, `cat` fell into 2>/dev/null, pin came back empty, and the
function bailed with its own "wrote no pin file" -- an accurate message pointing at
the wrong culprit.
1bf28bb2bf's commit message names the right anchor
("DefaultPasswordResetProvider:100 ProgramDataPath/passwordreset<md5>.json"). Only
the shell line disagreed with it.
Line 47 was the only /config/data in the file; :102, :104, :110, :168 and
meta/volumes.yml:6 all use /config. The filename is MD5 of the uppercased username,
so exactly one file matches and head -n1 stays harmless.
Nothing is masked: both error branches stay, and no retry count or timeout is
touched. The steps after it are already correct against v10.11.11 --
DefaultPasswordResetProvider.cs:77 `ChangePassword(resetUser.Id, pin)` makes the pin
the password, which is what get_token and the follow-up password change assume.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An earlier change in this branch marked six GitLab key bases rotatable: false after
one of them, db_key_base, was proven to break a deploy. Six was the shape of
"everything in secrets.yml.j2 that could plausibly be blamed", not the shape of
"what is lost". Measured in the shipped image (gitlab-rails-ce:v19.3.1), four of
them rotate fine:
A_new_key_only => FAIL Decryption (control: reproduces the rotation)
B_old_then_new_list => s3cret-written-before-rotation
C_new_only_reads_post_rotation_write => written-after-rotation
D_salt_rotated => FAIL Decryption
B and C prove active_record_encryption_primary_key is list-rotatable and that the
old key can later be dropped; D proves the salt is not. Note the new key goes LAST
-- key_provider.rb:21 is `@keys.last` -- so a prepend would have shipped a broken
rotation.
Unpinned, with the reason each:
otp_key_base gitlab:two_factor:rotate_key:apply exists, with rollback!
encrypted_settings_key_base protects zero bytes here: grep -c encrypted on
gitlab.yml.j2 is 0, ldap and incoming_email are off,
and upstream has rotated_encrypted_settings_key_base
active_record_encryption_primary_key list rotation measured above
active_record_encryption_deterministic_key one column, a push device token the device re-registers
Still pinned, and the descriptions now name why rather than leaving it to a commit
message: db_key_base (settings.rb raises MultipleDbKeyBaseError on a second key and
upstream ships no re-encrypt task) and active_record_encryption_key_derivation_salt
(scalar PBKDF2 input, no previous-salt slot in Rails). What db_key_base protects is
also worse than first recorded -- grep -rl returns 36 files: CI/CD variable values at
every scope, terraform state, Pages TLS private keys, webhook and integration
credentials. The two tokens that actually crashed were only the ones custom-instance-setup
happened to touch.
The second half matters more than the first. _drop_generated skipped a pinned path
with a bare `continue`, and reset/cli.py printed "Rotated 66 value(s)" with no mention
that six were refused. An operator rotating a disclosed key was told it worked. A tool
that silently declines to rotate a leaked secret and reports success inverts the
property it exists to provide, so every skip now names itself on stderr.
Not addressed here, and it is the real repair: gitlab:doctor:reset_encrypted_tokens in
the db-migrate path would let any instance with undecryptable tokens heal itself,
including after a restore under a different secrets file, which no pin can help. It
changes the migrations container's command and is not measurable without a live GitLab.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The richdocuments spec matched with a three-branch union whose last branch,
input[id*='wopi' i], is a substring search across every element id. Nextcloud
renders its initial state as hidden inputs, and one of them is
`initial-state-richdocuments-wopi_callback_url` -- it contains "wopi", so .first()
took it before the real field was ever considered.
The artifact says it outright:
119 × locator resolved to <input value="IiI=" type="hidden"
id="initial-state-richdocuments-wopi_callback_url"/>
Expected: visible Received: hidden Timeout: 60000ms
Nothing was wrong with the deployment. The same error-context a11y snapshot shows
"Collabora Online server is reachable." and the real textbox carrying
https://collabora.infinito.example.
The narrowed selector accepts both real shapes and neither matches the hidden id,
which is neither equal to wopi_url nor carries that name attribute. Dropping the
substring branch also immunises the spec against every future initial-state field.
.filter({ visible: true }) would work too and is the house idiom
(web-app-mailu/files/playwright/playwright.spec.js:136), but a selector that cannot
match the wrong element is deterministic rather than visibility-dependent.
Scope is one variant: only variant 0 enables collabora, 1-13 set
collabora.enabled: false.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
536b12e174 gated the realign on compose_handlers_flush, copying the flag that
decides who flushes rather than tracking whether the database needs realigning.
Those are different questions. `compose_handlers_flush: false` means "I flush
later myself" -- web-app-gitlab/tasks/00_core.yml:22 and web-app-matrix set it and
then flush on their own -- so for them the realign never ran at all.
Not a deadlock, not an ordering problem: the task was simply never scheduled.
Negative control from CI job 99670614257: `included: .../realign_dedicated_db.yml`
appears exactly three times, for openresty and flowise, and never for gitlab. Both
gitlab call sites report `false_condition: compose_handlers_flush | bool`.
What that costs, measured at the container in two independent jobs -- the tor one
(99670614346) and the compose one:
PostgreSQL Database directory appears to contain a database; Skipping initialization
2026-09-01 03:55:17.471 UTC [40] FATAL: password authentication failed for user "gitlab"
65 of them, 30s apart. gitlab-migrations never leaves /scripts/wait-for-deps,
webservice/sidekiq/workhorse never leave `created` because they depend on it with
service_completed_successfully, and `compose up` burns its full 1800s. The health
gate cannot see any of it: the engine's healthcheck is `pg_isready -U gitlab`,
which does not authenticate, so it reports healthy throughout.
17 roles pair a dedicated postgres with compose_handlers_flush: false and were
latently broken under rotation: baserow, bookwyrm, decidim, gitlab, kix, listmonk,
mastodon, matrix, mobilizon, n8n, odoo, openproject, opentalk, penpot, taiga,
xwiki, zammad.
The engine probe is required, not decoration. Un-gated, this task also runs on a
FIRST deploy where no engine exists yet, and set_postgres_superuser_password.sh
would spend ~30 minutes retrying before reporting [FAIL]. It is a no-op for the
other call site, post_deploy.yml:16, which already waits for healthy two lines
above it.
Verified by deploy, twice: `[CHANGED] New password did not work before, and works
now.` A skeptic's objection -- that the un-gate only helps when the previous pass
left the engine running -- was measured and does not hold: in both affected jobs
the engine was healthy 12-14 seconds BEFORE the task that got skipped.
Break risk measured rather than estimated. One deploy drove this across nine roles
and 18 invocations: 1 CHANGED, 1 UNCHANGED, 16 correctly skipped, 0 FAIL.
Still open: a dedicated postgres with no healthcheck yields an empty probe and
stays a silent no-op. Not a regression -- the task did not run there before either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f58ff5522 gave the socket-timeout shim a single point of truth by way of a
defaulted ARG. test_arg_without_default.py forbids that tree-wide:
roles/web-app-flowise/files/Dockerfile:7: ARG FLOWISE_SOCKET_TIMEOUT_SHIM carries a default
The rule's reasoning does not fit this value at all: "A defaulted ARG answers a
question the caller failed to answer. The build then succeeds against a base
image, version or path nobody chose." An ARG is a contract with the caller, and
nobody needs to answer where a temp file lives that is written on line 21 and
deleted on line 25 of the same layer.
f45df04bf1 shows the house pattern for the case where the value IS a caller
decision: it dropped PY_BASE's default from bridgy-fed and had compose.yml.j2
supply it. Doing that here would invent a build contract for a scratch file.
A shell variable keeps the single point of truth and narrows the scope on top of
it -- the name was visible across the whole build context as an ARG, and now
lives only inside this RUN. Same path, same three references, same build.
Suppression was available and not used: a nocheck marker would have kept the
wrong mechanism and hidden the reason it was wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
House rule: a comment is a nocheck directive, a parameter doc, or a marker for a
concrete trip-wire. Neither of these is.
realign_admin_password.sh explained that staleness is measured against a marker of
the password the script last wrote rather than against the account. That is design
rationale, it belongs in a commit message, and 5793512ffb already carries it. It
sat directly under the legitimate `# Exit:` doc, which is what let it pass every
earlier read: the banned label hidden inside the allowed kind of comment.
stack_ready.sh explained that `docker service ps` renders CurrentState as prose.
Pre-existing, not mine, but 455a73378 touched the file, and touching a file means
purging the narration it inherited too.
Neither diff removes anything but comment lines; both files re-parse under bash -n
and make quality-high is green over this tree, lint-shellcheck included.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitLab#1 of run 33414875521 died in the flowise image build. The obvious line is
not the cause:
npm error command sh -c prebuild-install -r napi || node-gyp rebuild
npm error prebuild-install warn install Request timed out
...
npm error gyp ERR! stack Error: not found: make
`not found: make` is the third domino. Counting the whole job settles it:
`Request timed out` 38, sqlite3 failures 6, `not found: make` 6 -- and sharp
failures 16, of 22 build attempts total. sharp emits no gyp output at all; its
fallback dies at `can-compile` because the image ships no libvips. A toolchain
would have touched 6 of 22 and left the dominant mode alone.
What actually aborts: node >=19 puts a 5s socket-inactivity timeout on
https.globalAgent (v24.19.0 lib/internal/http.js:263-268), while sharp and
prebuild-install fetch their prebuilt binaries through simple-get, which sets no
timeout of its own and never retries. Any 5s gap in the byte stream of the ~8 MB
libvips tarball drops the whole RUN layer.
The trigger was environmental, and it is worth being explicit that it was: the
same run, the same commit and the same base-image digest built flowise green
twice in another job, overlapping these failures. MTU is 1500 in both jobs, there
is no proxy configured, and svc-net-tor redirects only 10.240.0.0/12, so
github.com never enters Tor. The apt layer of the failing build itself fetched at
6178 kB/s and npm resolved all 3196 packages in the same 157s as the green one.
The fragility is ours. Eleven lines above, the apt layer already carries
-o Acquire::Retries=3; the npm layer had nothing. That asymmetry is what this
closes.
NODE_OPTIONS is attached to the install command alone rather than exported for
the layer: npm init and npm pkg set reach no network, and npm install is what
spawns the lifecycle scripts that download. --require is kAllowedInEnvvar on this
node (src/node_options.cc:1124-1127). No ENV, so the runtime image and flowise's
own HTTP behaviour are untouched. 60s is 12x headroom over a real stall while a
genuinely dead socket still fails inside a minute.
Not proven: that 60s clears this particular stall. The stall duration is nowhere
in the log, and per 850b795a51 a local quality gate cannot answer a
network-dependent question at all. Only a CI build closes that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Listmonk#0 of run 33414875521 failed the adapter probe in PASS 2:
REJECTED read call listmonk_get_stats answered 502:
{"error":{"code":-32000,"message":"upstream_error","data":{"upstream_status":403}}}
403, not a transport error: the adapter reached Listmonk and was turned away.
changed_when matched the psql tag `INSERT 0 1`, which by construction appears only
on the very first deploy -- mcp_identity.sql:14 guards the INSERT with
WHERE NOT EXISTS. On a rotation the row is already there, the statement takes the
UPDATE branch, the new sha256 lands in users.password, and changed_when evaluates
false. The restart block that 8e6fc81eff built then skips, and Listmonk keeps
answering from the API-user cache it loaded when compose recreated it two seconds
earlier -- still holding the previous hash.
The two halves have to move together. The users UPDATE carried no equality guard,
unlike the roles UPDATE two statements above it, so `UPDATE 1` fired on every run
and meant nothing. Guarding it makes the tag informative; only then may
changed_when read it. Guard alone leaves the rotation invisible exactly as today;
widened changed_when alone would restart Listmonk on every converged run.
Measured, not reasoned. sha256 recomputed locally and matched against the log:
'ZOPJ...XUNP' -> 6a8b5b5d... = PASS 1 token_hash (listmonk0.log:149940)
'aTiG...91Iz' -> a90e2a4d... = PASS 2 token_hash (:343692)
and the window is closed on both ends: compose up ended 21:08:57.93, the new hash
landed 21:08:59.70, the probe ran 21:09:02.15, nothing restarted in between. The
gate flip is verbatim in the log -- PASS 1 `INSERT 0 1 ... changed: true` then the
restart task; PASS 2 `INSERT 0 0 / UPDATE 1 ... changed: false` then
`skipping: false_condition: listmonk_mcp_identity.changed`.
Retrying could not have helped: probe.yml:115 short-circuits its 30 attempts on a
REJECTED verdict, and rightly so -- the database row was correct, the running
process was not.
Matching stdout_lines as list membership rather than a substring of stdout is
deliberate: psql prints each command tag on its own line, and 'UPDATE 1' as a
substring is also satisfied by `UPDATE 10`.
Listmonk was the only role in the tree inferring change from a psql tag --
grep -rn "INSERT 0 1" roles/ returns this one hit, while checkmk, n8n, baserow,
pretix, flowise and jellyfin all emit an explicit marker.
Not covered: the restart path is exercised against a first creation, never against
a rotation. A two-pass deploy closes that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bb975843f1 raised web-app-gitea to 1.27.3 in meta/services.yml and left the audit
table at 1.27.2, so test_mcp_audit_report.py has been failing since:
AssertionError: audit.md differs from a fresh generation.
Regenerate and commit it by running audit.gen.py from the repository root
Regenerated, nothing else. audit.gen.py:54-62 reads the version out of each role's
own meta/services.yml and makes no network call, so the file is a pure projection
of the tree and drifts only when a version changes -- this is real drift, not a
flaky check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
stack_ready.sh reports on both of its unhappy exits -- the latched branch dumps
the offending services, the not-converged branch dumps task rows and state -- and
falls off the end of the file in silence when it succeeds.
That silence cost a whole investigation. Moodle#1 of run 33414875521 died with
rc 137 and empty stdout/stderr 1.6s after this gate returned ok, and the entire
44422-line job contains no docker service ps, no task-state dump and no
State.OOMKilled for moodle. Which task generation was live at the moment of the
exec had to be reconstructed from the file NAMES in the rescue tree, and the
proof only arrived from a separately downloaded bundle:
moodle_moodle.3.mwdxfc7z6m3q877dr3ow1f5qr finished = 20:05:17.418099896Z
install task end = 20:05:17.427801
The task ids and Status.Timestamp this prints are exactly what was missing.
Cost is one run per convergence, not per poll: ansible's until stops at the first
rc 0, so the success path executes once per deploy. Both docker calls inside
report_task_states are already timeout 15 and swallow their own failure, so a
slow manager delays nothing and breaks nothing.
The array guard mirrors the existing one at :90 -- under set -u, "${arr[@]}" on
an empty array aborts in bash before 4.4.
Deliberately NOT included: a gate for UpdateStatus.State == updating. The gate is
demonstrably blind to in-place updates -- it returned ok in 0.630s after a stack
deploy that reported changed, and across all nine deploys of PASS 2 it passed on
the first poll, while in PASS 1 (service creation) it retried 56 times over
2:25.116. But UpdateStatus is written by swarmkit's updater goroutine after the
raft commit, which is the same lag that makes the replica count stale, and the
old container still reported healthy 0.6s AFTER the gate returned. A previous
update also leaves the status at completed, which an early poll reads as fine.
That fix would narrow a window of unmeasured width across 40+ swarm roles rather
than close it; it needs the timing measured on a throwaway cluster first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moodle#1 of run 33414875521 failed in PASS 2 with rc 137, no output at all:
rc: 137, delta '0:00:01.530480', start 20:05:15.897321, stdout '', stderr ''
The rescue bundle names the killer to the millisecond:
moodle_moodle.3.mwdxfc7z6m3q877dr3ow1f5qr finished = 20:05:17.418099896Z
install task end = 20:05:17.427801
Ten milliseconds. OOMKilled is false on all six containers, so the surviving
alternative -- a per-cgroup OOM against mem_limit: 2g -- is out.
PASS 2 re-rendered the env file and rotated the file-backed secret hashes, so the
stack deploy changed the moodle and cron specs and swarm rolled every replica.
The rescue tree shows two generations for each moodle_moodle slot and for
moodle_cron, and exactly one for each moodle_nginx slot; moodle and cron are the
two services carrying env_file. Slots went at 20:05:17, 20:05:36 and 20:05:55 --
19s apart, which is parallelism: 1 plus delay: 10s. Whichever slot the exec drew,
it was going to be killed: the install needs ~176s and the roll takes ~39s.
The convergence gate did not stop this and structurally cannot. It is a handler,
it returns before the task starts, and a barrier at time T cannot protect an exec
that begins at T+1.6s. The same-pass control settles what does work:
svc-db-mariadb went through the identical gate (0.517s, zero retries) and its next
in-container exec survived, because it has retries. This task was the only
in-container command in the role without them -- the health probe above it has 120
and the mail-marker probe below it has 30.
MOODLE_CONTAINER_ADDRESS expands to a literal $(resolve-container-id ...) that is
re-executed per attempt, so a retry re-resolves onto the surviving container by
itself. until gates only on the SIGKILL signature; a genuine install failure
satisfies rc != 137 immediately and the existing failed_when still decides.
Not covered: this exec drew the first slot replaced and died at 1.5s, before any
DDL. A kill at second ~100 of ~176 could leave a half-written schema that matches
neither 'Moodle is already installed' nor 'Database tables already present', and
the retry would then fail differently. Unmeasured.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moodle#0 of run 33414875521 failed its litellm spec in the login precondition.
The page snapshot names the cause rather than the symptom:
Blocked http://auth.4fc5...onion/realms/infinito.nexus/protocol/openid-connect/token
line 90 of /auth/oidc/classes/httpclient.php: call to curl->request()
line 3814 of /lib/filelib.php: call to curl->trigger_url_blocked_event()
Not an auth failure: the browser leg finished and came back with a code. What
died is the server-side token exchange.
On an onion node container_extra_hosts pins the Keycloak authority into the
container's /etc/hosts, for exactly this call -- its docstring says so: "code-to-
token exchange, JWKS, userinfo ... Pinning the provider to a reachable address
sends those calls over the local network to the proxy that already serves the
vhost." The pin points at host-gateway, which is 10.208.0.1 here because
08_networks.yml overrides docker's pool to 10.208.0.0/12.
MOODLE_EGRESS_BLOCKED_HOSTS then covered that address with 10.0.0.0/8. One half
of the deploy made Keycloak reachable and the other half blocked the address it
had just published.
b23653adf8 stated its intent as "172.16/12 and 192.168/16 become reachable so the
container network resolves, while 10.0.0.0/8, link-local and loopback stay
blocked", and README.md recorded the premise "Keycloak answers on 172.16.0.0/12".
It does not. The commit opened the two RFC1918 ranges this deployment does not
use and kept the one it does. 0f6728a4c1 is the same class of bug on a different
mechanism: an iptables range that also swallowed 10.208.0.0/12.
Measured rather than reasoned, with the exact --add-host the run emitted:
gethostbynamel("auth.4fc5....onion") => ["10.208.0.1"]
address_in_subnet("10.208.0.1", "...,10.0.0.0/8", true) => true
address_in_subnet("10.208.0.1", "...", true) => false
Two positive controls in the same job: prometheus-sso-proxy redeems a code
against the byte-identical URL and passes, and so does mailu. Moodle is the only
role in the run that puts an application-level SSRF blocklist in front of its own
HTTP client, which is why only Moodle broke.
No new security decision. README.md already carries the trade-off the earlier
commit accepted -- Moodle has no allowlist, so one internal host cannot be
permitted without its range -- and this extends it to the range the deployment
actually occupies. Loopback, 0.0.0.0, the metadata address and 0000::1 stay.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
