The pre-compose staging (social-app clone, login-broker copy, pdsadmin
download) ran before sys-stk-backend and notified compose-build and
compose-up. On a fresh host the container runtime's first flush ("Restart
and enable docker service") then ran the compose handlers before
/usr/bin/ca-inject and compose.yml existed, and "Generate CA trust
override" failed (run 34615516009, job 103327034609).
The staging tasks now only register their result. A changed-only notifier
after the sys-stk-backend include triggers compose-build and compose-up, so
no compose topic is pending at any earlier flush, and the handler-only
include that registered the owner early is gone. The ordering test asserts
both halves: the staging files notify no compose topic, and the rebuild
notifier sits after the backend include.
Verified locally in compose, variant 1: the reinstall full cycle passed
pass 1 with failed=0 and the rebuild task changed after the backend
include, pass 2 Playwright passed, and an update run finished failed=0 with
the rebuild task skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The flavor-coverage test from 214e7e29d1 read the role's vars with
yaml.safe_load(VARS_PATH.read_text()) and spelled "vars" / "main.yml"
out by hand. test_no_uncached_file_reads forbids a direct .read_text
and test_file_spot requires the ROLE_FILE_* constants, so the file now
loads through utils.cache.yaml.load_yaml_any and joins
ROLE_FILE_VARS_MAIN. What the test asserts is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34421944328 web-app-litellm#1 (README host-mode replay) answered
every health probe on llm.ai.infinito.test with 502: litellm
crash-looped on "P1000: Authentication failed against database server
at postgres" while postgres logged 'Role "litellm" does not exist'. The
README inventory has no svc-db-postgres group, so sys-svc-rdbms skipped
the central provisioning, yet svc-ai-litellm pinned postgres.shared to
true and kept consuming a shared database nobody had created its role in.
postgres.shared now reads "{{ 'svc-db-postgres' in group_names }}":
an inventory with the central group keeps the shared database, the
README inventory gets a dedicated one.
That dedicated instance, rendered by sys-svc-rdbms postgres.yml.j2,
joins only the compose default network, and the networks renderer
dropped default from every svc-db-*/svc-ai-* role. A local variant-1
deploy measured litellm-database on litellm_default alone and litellm on
the shared litellm network alone. _suppress_default now keeps default
when the role's database is local. The variants pin shared to true (0)
and false (1) so the matrix exercises both paths, and the generated
READMEs show the postgres relation as 0..1.
The three attachment helpers move from render.py to attachments.py,
which render.py re-exports, and test_render.py splits along the same
seams into five files plus _render_helpers.py with every test id kept.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34421944328 Bluesky#1 stopped at "The requested handler 'compose-up'
was not found in either the main handlers list nor in the listening
handlers list", right after cloning the social-app build context. The
pre-compose staging (01_social_app.yml, 02_login_broker.yml) and the PDS
routine (03_pds.yml) notify compose-up and compose-build, but the
handlers only exist once sys-stk-backend loads sys-svc-compose, which
happens later in the role.
00_core.yml now includes sys-svc-compose with tasks_from: none.yml and
handlers_from: main right after the once flag, the same handler-only
registration sys-stk-backend and sys-svc-rdbms dedicated.yml already use.
The notifications still run at the stack's own flush, after the compose
files exist; the role has no earlier flush. A handler-order test pins
the registration ahead of the pre-compose block.
The oauth2-proxy cfg heal leaves its inline shell for
files/shell/heal_sso_proxy_cfg.sh, which receives the cfg path and the
container name through the environment. The blanket
"container stop ... || true" becomes a precondition: the container is
stopped only when container inspect finds it, so a directory is still
removed when the container is gone, and a failing stop of an existing
container now fails the task instead of vanishing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit wrote the same proxy-aware logout expression into
four javascript.js.j2 templates. The rule it encodes - sign out of
oauth2-proxy first when the app is proxy-gated, else go straight to the
provider - belongs where proxy-gating is already decided.
utils/roles/applications/services/sso.py gains logout_url(url,
is_proxy_gated) next to get_sso_config, with OAUTH2_SIGN_OUT_PATH as the
path constant; the rd value is quoted exactly as Jinja's urlencode
filter does it. The sso lookup exposes it as the logout_url term: it
renders OIDC through the templar, takes is_proxy_gated from the same
resolution, and refuses a context without OIDC. Prometheus, Checkmk,
Baserow and Jitsi now render lookup('sso', application_id, 'logout_url').
Behaviour is unchanged: the resolver tests pin the rd encoding against
jinja2's do_urlencode, and the lookup tests cover a proxy-gated app, an
OIDC-native app, a disabled SSO and a missing OIDC.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34421944328 odoo#0 loaded assets over the node onion that the page
could not use.
build_csp_header mirrors every clearnet token that carries the primary
domain onto the node onion, but kept its scheme. Onion vhosts are served
in plaintext, so a sibling like https://cdn.<onion> never matched the
http://cdn.<onion> request it was meant to allow. Onion siblings now map
https to http and wss to ws; the clearnet token is unchanged.
The body filter rewrote external asset origins to the onion mirror only
in HTML, so stylesheets kept pulling https://<origin> directly. When an
app has tor enabled and external asset origins, the header filter now
also flags text/css responses on .onion hosts (dropping Content-Length,
since the body changes), and the body filter runs those through the same
buffer and mirror rewrite and returns before the HTML injections. The
flag is set under the same Jinja condition that renders the mirror
block, so a flagged response always has a rewrite to run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34421944328 n8n#1: Open WebUI 0.11.0 ships only a Streamable HTTP
MCP client. n8n 1.95.3's MCP Server Trigger speaks SSE only and answered
the client's POST to /webhook/infinito-mcp/sse with 404, so "every
registered MCP server answers Open WebUI's client" failed.
n8n moves to 1.100.1 and the managed trigger to typeVersion 2, which
serves Streamable HTTP at /mcp/<path>. meta/mcp.yml declares
transport: streamable_http at /mcp/infinito-mcp (the trigger path stays
infinito-mcp) and points source_url at the 1.100.1 tree. Open WebUI's
supported_transports drops sse, which its client never spoke; no
provider declares sse any more, so no pairing is lost.
n8n#0: the guest spec probed ${APP_BASE_URL}${MCP_ENDPOINT_PATH} and
expected a refusal of >= 300 without an event stream. The trigger is
exposure: internal, so the public vhost must answer 404; the spec now
asserts exactly that through mcpEndpointUrl(..., N8N_BASE_URL), and the
bearer guard itself is proven in-cluster by the CLI MCP contract.
Requirement 035 and the MCP audit table follow the version and
transport.
Upgrade path, read from upstream at both tags: the postgres migrations
are identical (45/45) and the only new common migration is
AddProjectDescriptionColumn, an additive nullable column; BREAKING-
CHANGES.md lists only 1.98.0 in range (route metric format, Form node
tag attributes, Node >= 20), none of which this deployment uses. A fresh
1.100.1 install ran all 89 migrations and passed the MCP bridge, server,
non-member and guest specs locally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every mautrix bridge generated its own registration with -g on start,
with fresh random tokens. Synapse had to read that file from a shared
directory, be restarted afterwards, and on swarm the services had to be
removed first so the registrations could be wiped (93735fd533). The
tokens changed on every redeploy and the deploy depended on a wait for
files a container writes.
Each bridge now has as_token and hs_token declared in meta/secrets.yml
and a fixed appservice_port in its addon. The bridge config reads the
tokens through lookup('config'), and registration.yaml.j2 renders the
registration from that config in the upstream -g format, so both sides
are known before anything starts and stay identical across redeploys.
The files reach the containers as secrets: matrix_bridge_mounts offers a
stack view (top-level definitions, content-hash named by compose_volumes),
a synapse view (one registration per bridge under /registrations, mode
0444) and a bridge view (the config at /config/config.yaml). The bridges
run as the synapse uid with an explicit --config and --no-update, keep
their scratch /data on a 16m tmpfs so hlth-volumes finds no anonymous
volume, and depend on a healthy synapse. Their healthcheck is
/_matrix/mau/ready, which the bridge only reports after whoami with the
as_token and an appservice ping carrying the hs_token, so a healthy
bridge proves the token pair.
The compose tasks lose the compose down, the swarm service rm block, the
registration wait, the permission fixes and both synapse restarts; this
reverts the workaround of 93735fd533.
Verified on the local compose stack: four bridges ready, synapse pings
answered, registration and config sha256 unchanged across a redeploy,
test-bridge-roster and 12 matrix Playwright specs green. A token
rotation (new pair in both files, compose up --force-recreate as the
handler runs it) came back healthy; a mismatched pair left the bridge
unhealthy. The swarm-shaped stack file passes docker stack config with
the 0444 secret modes, the 16 MiB tmpfs and content-hash secret names.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
compose_volumes copied extra_configs and extra_secrets through verbatim.
A file-backed extra without a name therefore reached swarm under a fixed
name, and swarm secrets are immutable: a new content under the same name
is rejected on the next stack deploy. Such extras now get the same
{role}_{key}_{sha8(content)} name that _config_secret_name already gives
meta-declared entries. An explicit name is kept.
container_volumes appended extras as handed over. Ansible passes one
lookup's result to another as proxy containers, and a TaggedInt mode
inside a proxy mapping made PyYAML raise "cannot represent an object".
The private _to_plain of compose_volumes moves to utils/templating as
to_plain, now coercing bool, int and float to the exact builtin types
(bool before int, since bool subclasses int), and container_volumes runs
its extras through it.
Only web-app-matrix passes extra secrets or configs today; the
extra_volumes path that web-app-nextcloud uses is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
resolveTimeout multiplies every Playwright window by 5 for a .onion
target. That default arrived in 35dee215c8 (2026-07-08), before
5399b48482 (2026-07-24) gave CI the chutney flavor, a private Tor net on
localhost, and PLAYWRIGHT_ONION_TIMEOUT_MULTIPLIER was never set. Waits
that expire by design therefore cost five times their base under chutney
too, which is where most of the Tor surcharge came from.
The Playwright run now passes the multiplier for onion targets only,
looked up from svc-net-tor's services.tor.flavor: 2 for chutney, 5 for
the public network, whose circuits really are slow. An unknown flavor
fails the lookup instead of falling back, and a unit test holds every
flavor directory of svc-net-tor to an entry.
Chutney traces of run 34421944328: 700 successful timed waits across 186
tests used at most 0.42 of their unscaled window, and the longest
client-side expect took 5.1s. Across runs 34421944328 and 34419507038 no
chutney test with its own ceiling (401 of them) used more than 53% of
the new x2 ceiling, even at today's x5-stretched duration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Over streamable_http the contract probe judged initialize, tools/list and the
read call by HTTP status alone. A server that answers 200 with a JSON-RPC
`error` object therefore passed initialize and the read call, and failed
tools/list only as "server serves []", which names the wrong cause. The SSE
path checked initialize and the read call, but not tools/list. mcp-adapter,
which fronts WordPress, maps invalid params to 200, and Moodle's
webservice_mcp answers a failing tool call with 200 and code -32603, so a read
that returned nothing could turn the check green.
rpc_error() extracts the error object of a JSON-RPC body, refused() is built
on it, and reject_error() fails initialize, tools/list and the read call on
both transports with the server's own message. A 200 whose body is not JSON
still carries no error object, so a provider that has not started keeps the
caller retrying as before.
The streamable_http initialize, and the one in the negative probes, sent no
clientInfo, which the MCP SDK requires. Home Assistant (mcp 1.26.0, stateless)
answers such an initialize with 200 and -32602 and serves the rest anyway, so
only the old status check let it pass. All three initialize calls now share one
INITIALIZE with clientInfo, as the SSE path already sent. Against a Home
Assistant-shaped endpoint on mcp 1.26.0 the SDK logs "InitializeRequest.params.
clientInfo Field required" for the old probe, and the fixed one prints OK.
Against HEAD, an error-on-200 at initialize and at the read call printed OK;
now all three steps print REJECTED naming the error. The MCP adapter in front
of sixteen of the nineteen streamable_http providers answers 200 only with a
result (svc-ai-mcp-adapter/files/python/server.py), so none of them changes
outcome.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard landed in 2acf91e18 breaking two rules it should have followed from
the start, and `make quality-high` caught both:
test_ca_trust_passthrough.py:23: `parents[N]` index expression
test_ca_trust_passthrough.py:23: `PROJECT_ROOT` assignment outside `__init__.py`
test_ca_trust_passthrough.py:44: .read_text(...) uncached
So `PROJECT_ROOT` moves into the package `__init__.py`, where every other role
test suite keeps it, and both template reads go through `read_text` from
`utils.cache.files` so a second reader of the same file pays one open.
Verified with the flags the suite actually runs under. scripts/tests/code/run.sh
passes `--import-mode=importlib` and `consider_namespace_packages=true`, which
is what makes a hyphenated `roles/<role>/` directory importable at all; a plain
`pytest <path>` fails the relative import here and does so for the existing
sys-svc-mail-msmtp suite too, which is the control that says the pattern is
right and the invocation was wrong. Under those flags both suites pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mdad stack runs its own Docker daemon inside the matrix container, so a
bind source in `matrix_synapse_container_extra_arguments` is resolved in that
container's mount namespace, not on the host. `vars.yml.j2` mounts
`CA_TRUST.bundle_host` while `services.yml.j2` passed only
`CA_TRUST.cert_host`, and those are two different files: the root CA and the
bundle beside it. The bundle therefore never existed where synapse looked.
`docker create` refuses the unit before Matrix logs anything of its own. From
the rescue artifact of job 102465251843 in run 34348821577, the nested
journal at containers/matrix/nested/.../journal.txt:5018:
matrix-synapse[5612]: Error response from daemon: invalid mount config for
type "bind": bind source path does not exist:
/etc/infinito.nexus/ca/ca-bundle.crt
Five start attempts, the same line each time, and the only error in the 872 KB
journal. matrix-heisenbridge follows it down as a dependency, `systemctl
restart` returns rc=1, the mdad bootstrap fails and the rescue task ends the
play. Not an OOM (exit 1, not 137, `OOMKilled: false`, 9.7 GB free), not disk
pressure, not a failed pull, and 16 % of the job budget spent.
b5fcd535de is where the asymmetry began: it moved this mount from `cert_host`
to `bundle_host` for a good reason -- SSL_CERT_FILE and REQUESTS_CA_BUNDLE
replace the trust store rather than extend it -- and left the passthrough on
`cert_host`. So the bundle stays; what was missing is the line that carries it
in, at the same absolute path the nested daemon asks for.
The test that comes with it reads both templates as text and requires every
CA_TRUST path synapse binds to be passed through at that same path. Rendering
would need a full Ansible context, and the invariant is about what the two
files say to each other rather than about the values. It fails against HEAD,
where `bundle_host` is bound and only `cert_host` is passed. It also fails
loudly if the regex ever stops matching how the extra arguments are built,
rather than quietly guarding nothing.
The cause is proven to the byte; the effect is not observed. `docker create`
never succeeded, so a second synapse defect behind this one would still be
invisible. A local `make compose-deploy apps=web-app-matrix mode=update` with
TLS_MODE=self_signed, grepping the nested journal for "Started
matrix-synapse.service", settles both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rows are indexed by `(name, variant)` and a variant is an int. A token that
names only a role carries no variant, so the lookup ran with `None` and could
never hit, and every such token was reported as matching no discovered row.
That is the form `collapse_to_roles` produces, so it is the form a retriggered
priority line arrives in. Run 34348821577 shows the result: all twenty-one
priority entries carried
priority: '<role>' matches no discovered row
(no row in this run's mode and lifecycle envelope)
while baserow, checkmk, decidim and the rest went on to deploy fine. The
warning was not wrong about one entry and right about twenty; it was blind. A
role that genuinely has no row in the envelope looked exactly like a role with a
full set, which is how priority work goes missing without anyone being able to
see which entry did.
A token that narrows nothing beyond the role name is now answered by existence:
it asks for the role in whatever axes the rotation picks, so any discovered row
satisfies it. `Pin.pinned` already draws that line - "whether the token narrows
anything at all beyond the role name" - and pinned tokens keep the old path,
`check_pins` included.
Three tests come with it. There was already one for a bare role that matches
nothing, and none for a bare role that matches something, which is why the
lookup could stay broken. One of the new ones uses variant 4 so a future
implementation cannot pass by quietly guessing variant 0. Checked against HEAD:
the same input returns the spurious warning there, so the tests fail without
this change.
This restores the signal, not the lost jobs. The other silent path, `priority`
being truncated to the job budget in cli/meta/ci/chunks.py, is untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
start_interval is the cadence while start_period runs, and only while it runs.
Once the window closes the probe falls back to interval, so a service that
becomes ready just after it pays a whole tick anyway and the cadence never
reached it. Six of the seven flavors were shaped that way, Curl worst: a thirty
second window against a one minute interval.
Measured across 151 converge waits in four job logs, 63 sit exactly on a tick
boundary, 36 in the thirty second step and 27 in the sixty second one.
Attributed by application, web-app-seaweedfs is the largest at 1501s over
sixteen waits with a median of 57.7s: it probes with curl and fell straight into
the gap. web-app-zammad probes with connect, whose window is fifteen minutes,
and its 154s median is real boot time with nothing to recover.
So the window moves to ten minutes and the steady state interval to two, and the
flavor overrides that sat below them go: Curl, Wget and Connect lose interval
1m, Http its twenty second window, Nc its ten. Connect keeps fifteen minutes.
Neither number costs anything while a container works. start_period delays
nothing, it only stops failures counting, and interval no longer touches
convergence at all since that hangs on the window alone; it is steady state
load, which drops from 120 probes an hour per service to 30.
Forty six role files follow. Twenty seven declarations go that repeat the new
default or undercut it, an undercut window being the defect itself. Eight
windows longer than ten minutes keep their value and now say why. The literal
blocks and the three Dockerfiles this repository builds spell every timing out,
because outside the lookup an omission takes docker's value and not the
platform's.
A fourth lint rule keeps the shape: start_period must outlast one interval,
checked against the effective values, so raising interval without the window
cannot quietly bring the cliff back.
Two faults in the lint surfaced while building it and are fixed here. It
compared timings as strings, so 600s and 10m read as different values, and it
matched the healthcheck key exactly, so a line carrying a trailing nocheck was
skipped whole, which is how the two swarm-init partials went unexamined.
One regression, bounded. A container that starts and never turns healthy is
neither rejected nor errored, so stack_ready.sh only calls it fatal after three
killed tasks, and each cycle now spans the longer window. That reaches twenty
minutes rather than six, capped there by swarm_stack_converge_retries at 600
with a two second delay, and it lands only on runs that were failing anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LITELLM_BACKEND_ROLES read lookup('deployment').groups, which deployment.py
fills from group_names. That variable is host-scoped: it lists only the groups
the host currently being processed belongs to.
On a swarm round the manager carries svc-ai-ollama, svc-ai-lmstudio and
svc-ai-litellm while the workers carry three groups and none of them is an AI
backend. Both branches of LITELLM_CHAT_MODEL therefore missed on a worker and,
with OPENROUTER_API_KEY empty, the model resolved to the empty string.
Only one task made that visible. Every stateful task in the gateway include
carries delegate_to STACK_HOST plus run_once, so it templates in the manager's
scope and wrote the correct model into the database. The assert carries
neither, so it re-templated in each worker's own scope and failed there while
the manager passed.
The manager-only membership is not a placement decision. The swarm test
inventory is provisioned with a single --host MGR, so every role of the round's
closure gets that one host, and extend_inventory then widens only
svc-swarm-node, svc-storage-nfs-client, svc-storage-nfs-server and the round's
own application. The three AI roles declare no placement at all; they are
manager-only because nothing widened them.
Add an inventory selector next to the host-scoped one, built from the play-wide
groups magic var, skipping groups without hosts and Ansible's implicit all and
ungrouped, sorted so the cache key is stable. The key takes it too, or two
hosts with equal group_names but different inventories would share an entry.
The blast radius is two readers, both in 16_ai.yml, and both ask only whether
svc-ai-ollama or svc-ai-lmstudio is present. On the manager the new value gains
exactly svc-storage-nfs-server and svc-bkp-nfs-2-local, neither of which either
reader tests, so the manager's answer does not move and only the workers change,
to match it.
This is the third step on the same line. 46a16127fc set .running; b149ad0d22
corrected that to .groups because a targeted apps= round leaves only the
command-line roles in the whitelist. .groups solves the targeted round and not
the multi-host one, because group_names stays host-scoped either way.
Measured in a two-host play with a real group_vars file and the real lookup:
the worker resolves ollama false through group_names and true through the new
selector, the manager is true both ways, and the assert that failed in CI
passes on both hosts.
Requirement 036 named the host-scoped backend selection as the thing that has
to change and deferred it. That residual is closed, so the paragraph now
describes the selector in use and why the two rejected ones are wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The account loop walked every user any role declares, 291 of them in a CI
round, and let each task discard the rest through its own when:. That still
costs a task execution per user per task: 1706 of 1746 executions reported
skipping, at 0.146-0.157s each, for 259.5s of a single job.
A role-local mailu_users lookup returns the subset instead, in the dict
shape lookup('users') yields so the call site keeps dict2items. The
predicate is the union of both gates the tasks apply: a mailbox account, or
a bot role, whose API token Mailu issues. Filtering on the mailbox alone
would quietly stop issuing tokens to a bot without one.
The per-task when: conditions stay. The returned set is a superset of what
each of them selects, so it decides how often they are evaluated, never
their outcome.
A non-list accounts declaration raises rather than being filtered away, the
way reserved_users.py already treats it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_ram_gb read mem_total_mb while mem_available_mb sat next to it, added by
3e9deab6f7 for the drill abort and never carried over here. The ceiling exists
to stop the sidecar from competing with what already runs, so measuring the
machine instead of what is free made the term inert: on a 16 GB runner it
computed five workers while 5.5 GB was actually available, and the CI cap was
the only thing that ever bound.
Reading MemAvailable makes the guard real and self-limiting, which matters now
that the onion case raises the count. Against the memory the run itself reports:
PASS 1, 9113M free clearnet 1 onion 2
PASS 2, 5550M free clearnet 1 onion 1
Also withdraws a CI CPU divisor I had added on a false premise. It cited the
matrix telemetry stall60 as evidence that reserving three cores of four buys
nothing on a CI runner - but mem_stall_pct reads /proc/pressure/memory
(utils/env/runtime.py:84), so that figure proves memory headroom and says
nothing about CPU. Raising the clearnet count needs CPU evidence this run does
not carry, and a contention flake would cost coverage, so the divisor stays at
four until something measures the CPU side.
The onion lift keeps its own justification, which never rested on that number:
a browser waiting on Tor circuits does not consume CPU, measured at 4817s over
Tor against 911s on clearnet for the same specs on the same cores.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sidecar is the dominant cost of an onion round and almost free without it.
Measured on one Nextcloud variant at equal mode and filesystem, differing only
in the onion axis, sync pass:
test-e2e-playwright 4817s over Tor 911s on clearnet +5.3x
sys-svc-compose 2294s 2174s +120s
everything else < +25s
The whole Tor surcharge sits in one role. Tor burns no browser CPU, so a 5.3x
wall clock against identical specs on identical cores is circuit latency: the
phase waits rather than computes. The CPU divisor that sizes the clearnet count
does not describe it, and on a 4-core runner it yields exactly one worker,
which leaves the CI cap of 2 unreachable.
compute_workers takes an onion flag and, when set, multiplies the count and
caps it separately. The CI ceiling is overridden rather than respected: it
describes CPU contention, which Tor does not create. ram_workers keeps its
veto, because circuits are cheap and browsers are not.
The lookup reads the transport itself, from the same source the role derives
its own flag from, lookup(tls, application_id, url.base) tested for .onion,
following the _swarm_prefix idiom in plugins/lookup/image.py: read from
variables, render through the templar, fall back to the safe answer. One truth
about the onion state, and a caller asking for a worker count cannot forget to
pass it.
Factor 2 and cap 4 rather than more: each worker holds its own circuits and the
Tor daemon is the next bound, which no measurement here reaches. Expect the
sidecar near 2.6ks per pass instead of 4.8ks, roughly 4.4ks per onion job
against a 20.4ks budget. It does not help a clearnet round, where the sidecar
costs 911s to begin with.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SSL_CERT_FILE, CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE replace the trust store;
only NODE_EXTRA_CA_CERTS extends it. compose_ca.py pointed all four at the
deployment own root CA, so every OpenSSL and rustls client in every container
was blind to every public certificate. Measured here, one variable, same URL:
unset curl https://registry.npmjs.org/ -> 200
a single unrelated CA -> 000 unable to get local issuer certificate
that CA + the system bundle -> 200
PeerTube was the one process that hit it: its plugin install fetches from
registry.npmjs.org at runtime through a client that honours SSL_CERT_FILE, and
it failed identically in compose and in swarm with
invalid peer certificate: UnknownIssuer
Everything else reaches its artefacts at image build time, or through Node,
whose NODE_EXTRA_CA_CERTS is additive and therefore kept working.
The replacement was never a posture. e7e8cf8c03 calls these variables an
"env-based CA trust fallback", f1ea036bd2 adds them "instead of falling back to
a plain docker run without CA trust", e79183f96a wires the extra CA in "so npm
traffic through the cache validates". Every commit framed them as adding trust.
sys-ca-selfsigned now builds ca-bundle.crt beside the root CA on every node,
from that node own files: the stack host mints the CA and workers fetch it, so
there is no cross-node dependency. The bundle is the root CA concatenated with
the first present CA_BUNDLE_CANDIDATES entry, the SPOT fc0881ff33 introduced.
The cat argument list is assembled in Jinja, so a host with no public bundle
emits the root CA alone, which is exactly the previous behaviour, rather than an
empty argument that would fail. bundle_host joins the existing stray-directory
sweep and the build runs after it.
compose_ca.py uses the bundle only when the file exists and is non-empty, so a
host that has not built one keeps the previous behaviour instead of binding a
missing path that Docker would materialise as a directory. The php ini, msmtprc
and the Matrix flavour may name the container path with no fallback because the
bundle is written at task 99 and the php ini at task 123: no host can carry the
ini without the bundle.
CA_TRUST_CERT and NODE_EXTRA_CA_CERTS stay on the single certificate. keytool,
occ security:certificates:import and update-ca-certificates each take one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two of them lived inside docs/ next to the pages they render, which is why both had grown a __file__ upward walk to find the repository root, a sys.path insert to import from utils, and a nocheck for each. A generator parked among documents reads as documentation and drops out of the checks that apply to code.
Moving them surfaced six real violations that the old location had hidden: yaml.safe_load instead of the shared cache, two uncached read_text calls, five hard-coded role paths instead of ROLE_FILE_META_SERVICES and ROLE_FILE_META_MCP, two make targets without an alias, and a requirement reference in code. All six are fixed at the root, none with a suppression, and both generators lose their project-root exemptions because a package supplies PROJECT_ROOT.
cli/build/readme joins them so the three generators that write into docs/ sit together. A new lint keeps docs/ to .md, .html, .rst and .nocheck; it enumerates through git ls-files so untracked __pycache__ is not its business. Checked against HEAD, it names exactly the three .py files that moved.
026-unified-addon-syntax stated that the matrix generator stays under docs/requirements/. The lint makes that impossible, so the requirement now records where the generator actually lives.
Verified: every one of the twelve modules under the new tree imports, no old module path survives anywhere in the repository, 482 lint tests and 1246 cli unit tests pass, and both generators run without drift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
da729cecec reads .Options.device when a volume declares a bind, on the premise that only compose volumes carry no .Options. A swarm node can hold the volume object without them anyway: in run 33923190220 the DR drill probe reported 'exposes no backing store for kix_kix_config' on the node that held it. The script then falls back to .Mountpoint, the node-local _data that is empty while the stack is down, finds no config there and exits 0.
Pass the inventory's own path as PATCH_DATA_DIR, the same DIR_VAR_LIB/<volume> that compose_volumes.py bakes into driver_opts.device, and try it when the volume-derived path holds nothing.
The second half is the asymmetry that hid this: a config present but unmatched fails loudly since 2c4e0e18b1, while a config found nowhere passed silently. The silent case is the dangerous one. KIX kept the pre-rotation password, its backend crash-looped on exit 255, and the converge gate only gave up 44 retries later. A total miss now prints SKIPPED with both paths it tried, and stays non-fatal so a first install still works.
The suite covered three of the seven consuming roles; it now covers all seven, plus the fallback and the named miss.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
VaultHandler.encrypt_string started one ansible-vault process for every single credential. The swarm matrix' rotation gate between PASS 1 and PASS 2 drives 113 application subprocesses over 855 credential keys, so the step cost 1078s in run 33923190220 while the green baseline never ran it at all. Measured per value: 226ms via the CLI, 3.3ms via VaultLib, so 855 keys drop from 193s to 2.8s.
The snippet layout is a contract: encrypt_leaves splits the returned lines and strips the body indent, and existing inventories were written by the CLI. Verified byte-compatible (same 'name: !vault |' header, same 10-space body indent, same $ANSIBLE_VAULT;1.1;AES256 marker), and ansible-vault decrypt reads back what the new path writes.
Reading the password file delegates to ansible's own FileVaultSecret rather than approximating it: ansible strips, then decrypts a vaulted password file, then rejects an empty secret. A hand-rolled .strip() covered only the first of those. _vault stays lazy because callers construct VaultHandler with a password file that need not exist yet.
The tests replace two subprocess.run mocks that asserted against invented output. They now pin the layout encrypt_leaves depends on, and pin the guarantee from 052cea02e2 (values starting with a dash), which landed without a test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mcp/vhost.conf.j2 wrapped its whole body in `{% if not mcp_acl_gated %}`.
That skipped the serve block, which was the intent, but it also skipped the
withdrawal loop underneath it. An ACL-gated vhost therefore rendered no
`location` for its MCP paths at all, so requests fell through to the
application's generic location and the endpoint was reachable behind
nothing but the SSO proxy ACL.
Folding the gate into the `mcp_served` expression keeps the serve block
gated while letting the withdrawal loop run in every case. With an empty
`mcp_served` both declared paths differ from it, so both are withdrawn with
`return 404`.
The behaviour is latent today: only moodle and wordpress declare
`exposure: public`, and both carry `flavor: oidc`, so `is_proxy_gated` is
false and the branch is never taken. This closes the hole before a role
combines public exposure with an oauth2 proxy flavor.
test_mcp_vhost_exposure.py could not reach the branch: its lookup stub
returned "" for every non-config kind, so mcp_acl_gated was always falsy.
It now resolves is_proxy_gated and oauth2_acl, and gained the bool filter
that the internal-exposure cases never needed because `mcp.exposure ==
'public'` short-circuits before it. Rendering the pre-fix template under
gating yields zero withdrawals; the new one yields two.
The base image postgis/postgis:17-3.5 is Debian 11, whose archive was decommissioned on 2026-08-31. The pgvector layer pulled 174 MB of build toolchain from deb.debian.org, and those pool files are now purged while the index still advertises them: linux-libc-dev 404'd on 44 of 44 attempts across four Fastly POPs. Since 2026-09-07 the suite's InRelease is also expired, so apt-get update itself exits 100.
PGDG ships postgresql-17-pgvector prebuilt and its bullseye-pgdg suite is alive, so reading only the pgdg.list the image writes itself removes deb.debian.org from the build entirely. Verified on the exact mirrored base image: the package installs, vector.so lands, zero bytes fetched from deb.debian.org. Signature verification is untouched - no Acquire::Check-Valid-Until.
Side effect worth naming: the old recipe cloned pgvector.git with no tag or ref, so every rebuild baked unpinned master into the central RDBMS. It is now pinned at 0.8.6-1.pgdg11+1. The base image is still EOL and its migration remains open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway publishes only models it has a backend for and renders model_list: [] when neither ollama nor lmstudio is deployed and no OpenRouter key is set. LITELLM_CHAT_MODEL fell back to openrouter/auto without checking that key, so consumers were configured for a model the gateway does not carry and every prompt failed with a 400.
Observed in the Mattermost#1 CI job: litellm.log carries 12 occurrences of "Invalid model name passed in model=openrouter/auto" next to 400 Bad Request, and tests/test-litellm-agent.js failed on every attempt while the rest of the suite passed.
Make the fallback conditional on the key so the name is empty when nothing is served, promote LITELLM_CHAT_MODEL_SERVED from the role vars to group_vars where consumers can see it (it was the third copy of this logic and the only correct one), and switch the 14 consumers from "litellm is deployed" to "litellm serves a model". The Playwright service flag reads the same lookup, so the spec skips together with the Ansible gate. The bool filter carries weight: the folded scalar yields the string False, which Jinja treats as truthy.
A new unit test renders both expressions against Jinja and pins the invariant that a model name exists exactly when a model is served, including the case where ollama preloads embedding models only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The front proxy hardcoded [200, 301, 302] and escalated to a full OpenResty redeploy on anything else. Apps that answer 403, 404 or 405 by design tripped it on every run: 19 escalations in one Nextcloud job at a measured median of 40.5s each.
roles/<app>/meta/server.yml already declares status_codes per canonical key, and sys-ctl-hlth-webserver already reads it. Move the resolution into utils/roles/applications/status_codes.py so both consumers share one definition of DEFAULT_OK, and expose it to templates as the accepted_status_codes filter.
The proxy gate takes the union of the default and the declaration rather than replacing it. Most declarations are narrower than the triple, so replacing would escalate vhosts that answer perfectly well. Contract checking stays with sys-ctl-hlth-webserver, which keeps replacement semantics.
_normalize_codes and _valid_http_code became dead in the health filter and are removed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The step banner announced a start and nothing else, so a phase duration had to be read off the next banner, and a phase with no output of its own was invisible. In the budget-killed Nextcloud#0 of run 33828309745 the credential rotation spent 17 minutes with nothing but the 30-second sampler between its banner and the next one, and reconstructing the split between provisioning, both passes, the DR drill and the rotation meant subtracting timestamps by hand.
The watch loop moves unchanged into _run_watched. _run now brackets it and prints the elapsed time and the exit code, so this holds for every phase, not only the rotation.
The test stubs _run_watched and pins the three things that matter: the exit code passes through, the opening banner is printed, and the closing line carries duration and rc. It is the first test over this module, whose absence was named earlier as the reason a wrong change to the pass sequence would turn nothing red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_no_uncached_file_reads flagged SCRIPT.read_text() in the extraction test. The suite shares one decoded read per file through utils.cache.files, and a bare pathlib read opts out of that.
The full gate surfaced it; the targeted run that accompanied a8469e064 executed the test in isolation and could not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ruff wrapped the set comprehension in the jellyfin auth extraction test at the 88 column boundary. No assertion changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keycloak persists only attributes the declarative user profile declares, and unmanagedAttributePolicy defaults to DISABLED in 26.x. ensure_realm_user.sh:81 writes attributes.uidNumber and kcadm returns rc 0, but the value never lands, so the mattermost-id mapper (web-app-mattermost/tasks/02_keycloak.yml:118) has nothing to emit, userinfo answers 200 without an id claim, and the Mattermost GitLab provider rejects the callback with "Could not parse auth data out of gitlab user object ... user id can be 0". Run 33828309745 fails three Mattermost jobs on it, in compose and swarm alike, and all three Playwright specs are that one failure seen at three downstream points. b0ca8784e2 added the mirroring for exactly this failure and it could not take. LDAP realms are unaffected: the federation writes the attribute, and 08_users/realm.yml runs only when LDAP is off.
The profile push must now precede the seeding that depends on it, so it is synchronous. It measured 0.9 to 1.5s, and its reap sat after the seeding, so with ASYNC_ENABLED true the write would still have hit the old profile. Its register drops out of the reap list with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
configure-auth.sh pins DeviceId="infinito-deploy" in CLIENT_HDR, and realign_admin_password authenticated twice with it: once for the access token, once for the user id. Jellyfin replaces the session bound to a device id when a new one is created, so the first token was already revoked when POST /Users/{id}/Password presented it. Run 33828309745 job 100890535194 shows the result as fifteen times "curl: (22) The requested URL returned error: 401", with the deploy otherwise complete (ok=2830 failed=1). get_user_id arrived with 1bf28bb2bf and carried the second call in from the start.
One AuthenticateByName response now yields both fields, read through token_of and user_id_of so neither expression is spelled twice.
Redeeming the reset pin clears the stored password: in that job forty authentications with the deployed password were denied, and the next one, with the pin string as the password, succeeded. Every exit after the redemption therefore states that the administrator is currently without one, and the two silent returns around ForgotPassword now log why they gave up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reachability check called showmount, which speaks the MOUNT protocol. The ganesha export sets NFS_CORE_PARAM Protocols = 4 and Enable_NLM = false, and every EXPORT block repeats Protocols = 4, so MOUNT is never registered and showmount cannot succeed against a healthy server. The repo already records this elsewhere: scripts/tests/deploy/swarm/utils/collect/diagnostics.sh:125 prints "showmount -e failed; NFSv4-only servers do not answer MOUNT".
Every failure fed status, and a nonzero exit reaches OnFailure, which starts sys-ctl-cln-anon-volumes (dockreap --no-confirmation). So on any host holding both anonymous and NFS volumes the check has been failing on every run and reaping volumes for a reason that was never true. Probing tcp/2049 is what the role does for its own readiness gate in files/nfs-ganesha-restart.conf:9, and it is correct for the kernel flavor too.
The two tests drive a whitelisted anonymous volume so the anonymous section contributes nothing and the exit status reflects only the NFS check. One asserts a listening port passes, the other that a refused one fails; the first is the regression guard, the second keeps the check from being one that always fails.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four probes returned the number of failures as the process exit code. An exit status is 8 bits wide, so exactly 256 or 512 failures arrive as 0 and the probe reports success. sys-service installs these as Type=oneshot units (roles/sys-service/vars/main.yml:28) and sys-service-terminator starts them with until: result is not failed, so that exit code is the whole health verdict.
The count is not lost: each of the four prints it to stdout a line earlier. Only the exit status stops carrying it, because it cannot.
This makes no CI job green and is not meant to. For any count other than 256 or 512 nothing changes; at those two it turns a false success into an honest failure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The retrigger turned a suite off only when the source run had been
dispatched with that suite explicitly on. b1ca39e9dd introduced that
guard together with the feature itself, and it contradicts the same
commit's message and docstring, both of which state that the job record
is the only evidence.
The guard made the feature a no-op in its most common case. An input
left on its default renders no title segment and resolves to empty, so
config.get('workspace') was never 'true' for a run that reached the
suite on the 'auto' default. call-orchestrator.yml reads 'auto' as
global scope off the whitelist alone, and a --failed retrigger puts its
selection on the priority line while leaving the whitelist empty, so
every retrigger in the chain redeployed a suite that had already gone
green. Measured on run 33785198875: six workspace jobs green, and the
dispatch still went out carrying workspace=auto.
Drop the guard. suite_passed already refuses to retire anything that
failed, was cancelled, timed out, was skipped or never ran at all, so
the job record decides on its own.
Three tests cover the paths 8600f1d9aa left open, all of which pinned
the explicit 'true': green on 'auto', green with no recorded input, and
red on 'auto' staying on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Line-length reflow in three test modules and a stray blank line in zammad variants. In router.yml the formatter also moved the nsswitch lineinfile regex from single to double quotes with the backslashes doubled to match; single-quoted YAML already passes a backslash through literally, so the string reaching lineinfile is byte-identical and the rule still changes only NOTFOUND.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two branch-only commits each armed a defect in the compose-up handler
topic, and both failed in run 33785198875.
63f7f962bf registered the topic owner at the top of sys-svc-rdbms'
dedicated.yml so the notify two tasks below stopped aborting with
ERROR_ON_MISSING_HANDLER. The notification then survived to the
unconditional meta: flush_handlers at sys-svc-container/tasks/00_core.yml:53,
reached from sys-svc-compose's own 00_core before that role installs
BIN_COMPOSE, and long before 02_stack_host_prep installs BIN_CA_INJECT.
Matrix#1 died there on ca-inject with rc 2 after seven retries. That play
carries exactly one compose notification (job 100756596907, log line
11046), so closing this window is sufficient, not merely necessary.
dedicated.yml now enters 00_core.yml under the role's run-once flag
before it notifies, so the flush drains an empty queue and the flag then
makes main.yml skip the body on the real entry. The registration itself
stays ungated: gating it would skip the owner from the second app on.
95186163aa hoisted the sys-svc-compose include to sys-stk-backend
position 1. The consumer therefore rendered and notified compose-up
before sys-svc-objstore lazily loaded the provider app role, whose own
unconditional flush consumed the notification and ran the single
registered handler in the provider's chdir. mastodon and peertube never
got a stack, in compose and in swarm alike; the victim's own guard task
appears zero times in all four logs. Restoring the include below rdbms,
objstore and engine puts the notify after every foreign crossing. Two
pre-hoist runs in exactly that order are green on record.
The test keeps its cases and gains one. It selects the render include by
the absence of tasks_from instead of by first occurrence, so an early
registration-only include stays legal while a hoisted render still fails.
The new case pins the bootstrap: entry point, position before the first
notify, and the run-once guard whose removal would let 01_reset.yml tear
down DIR_COMPOSITIONS once per dedicated-DB app.
Same class, pre-existing, not fixed here: svc-db-qdrant notifies and
flushes three tasks before its first compose entry; web-app-magento pairs
a dedicated database with a shared object store; web-app-seaweedfs still
flushes a topic it does not own; svc-runner enters 00_core.yml unguarded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The hub serves 94 tools across 23 categories, write tools included, and enabling the ExApp switches every category on. That surface was published on the public vhost and every discovered client was handed an app password for it. svc-ai-mcp-adapter now sits in front in mcp_passthrough mode: clients present credentials.mcp_bearer to contextagentmcp and reach the nine read tools of calendar, contacts and files, which files/mcp/tools.json names and the sidecar enforces per call. Those three categories are the ones whose upstream is_available is unconditional, so the served set does not vary with which optional apps a deployment installs.
Enforcing it in the sidecar rather than in each client is what makes it hold: web-app-openwebui renders no per-tool filter at all, so a client-side include list would have bounded two consumers out of three.
06c13e7f3e withdrew the declared path from the edge for exactly this reason - "that surface is what the adapter exists to narrow". Nextcloud answers the same route under a second spelling, without the /index.php front controller, which try_files rewrites back onto the route after the edge has already declined to block it. The withdrawal becomes a regex covering both, and only where stripping the front controller changes the path, so the four adapter providers without one render unchanged.
The app password is minted after the stack renders, so it reaches the sidecar through upstream.env and rebuild.yml rather than the compose environment: compose up resolves env_file while creating the container, a restart does not. The sidecar is the last service in the template because a Jinja set has no block scope and its env_file would otherwise displace the one talk, whiteboard and talk_recording each carry.
Two suites cover what nothing measured: the vhost template rendered without a container, asserting both spellings are withdrawn, and every shipped tools.json loaded through the adapter own loader, so a bad contract fails in a second instead of as a probe timing out an hour into a deploy. The guest spec probed mcp.endpoint.path, which this change makes the proxy answer 404 for its own reasons; it now probes all three public paths and requires no MCP body.
Not yet deployed: the sidecar build, the upstream.env delivery and the hub answering exactly nine tools are unverified against a running stack.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule lived only in suite_passed's body, and it inverts easily on a
casual read: ask for the suite again when it failed. Under that reading
the retrigger aliases would drop the suite in their most common case,
because they force-cancel the source run before reading it, so a shard
that was still running comes back cancelled rather than red.
Only an all-green suite is retired. Failed, cancelled, skipped, timed
out, still running and never dispatched all keep it on the retrigger,
and each of those now has a test. The two docstrings enumerate them, and
so does the i8ciall description, which is where an operator reads it.
dedicated.yml notifies compose-up but registers no handler for it, so a
caller that reaches sys-svc-rdbms before any compose consumer has loaded
the sys-svc-compose block dies with "The requested handler 'compose-up'
was not found" and no PLAY RECAP. 95186163aa guarded only the
sys-stk-backend caller; web-app-matrix's ansible flavor includes
sys-svc-rdbms directly in 03_database.yml and still hit it as the first
compose consumer of its play (run 33751742070, job 100643400267).
Register the owner as the first task of the notifying file. In the
sys-stk-backend path this lands before the pre-flush rebind, so the app's
copy stays the last registration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
65c72fc374 settled the convention: import unittest, and reach the submodule as import unittest.mock as mock, so a call site has one spelling of the name. test_onion.py and test_tor_socks_proxy.py still carried from unittest import mock; CodeQL reports them as py/import-and-import-from, the last two in the repository.
Same one-line change per file the sweep applied; mock.<attr> call sites untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
