d5fc099166 spelled out roles/web-app-matrix/vars/main.yml, which test_file_spot rejects: utils.roles.mapping is the single place that knows the role layout, and every construction of such a path has to read it from there.
The role directory is now a single name and ROLE_FILE_VARS_MAIN is appended to it, so the templates path and the vars path share one root and the role name appears once.
The violation shipped because that commit was gated by the two affected unit tests rather than by make test, which is where the lint runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mu-plugin header describes the split as 'the adapter authorises the transport, the callback authorises the operation'. The second half held, every ability carries infinito_mcp_may_read. The first did not: the adapter answered the initialize handshake to anyone, so the server and its protocol version were readable without a credential.
Measured in run 34653754175, the swarm WordPress#0: test-mcp-guest.js posts an anonymous initialize and expects at least 400, and the error context reads 'Expected: >= 400, Received: 200'. Its neighbour addons/mcp-adapter.spec.js posts an anonymous tools/list and passed, so the tool call was already refused and only the handshake was not.
The filter hangs on rest_pre_dispatch and refuses every route under the mcp namespace while the site cannot attribute the request. It leaves an already short-circuited response alone rather than overwriting it, scopes itself by route prefix so the rest of the REST API stays as open as it was, and reaches is_user_logged_in, which triggers determine_current_user and with it wp_validate_application_password, so a client presenting its application password passes. A WP_Error renders no jsonrpc key, which is the second thing the spec asserts.
The new unit test reads the guarded prefix out of the PHP and the endpoint out of the spec and requires the endpoint to sit under the prefix. Without it an adapter that moves namespace would leave the guard protecting nothing while both negative specs keep passing.
A positive spec was tried and reverted: handing the MCP credential to the Playwright env is forbidden by tests/lint/ansible/services/test_mcp_secret_not_in_playwright.py, because the trace travels in the CI artifact. No role probes its MCP endpoint with the credential at deploy time either, so the authenticated direction stays unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sys-svc-compose-ca writes the CA material for exactly lookup('ca_injected', application_id), which asks three questions per application: has a domain, TLS enabled, mode self_signed. The lookup exists against this drift and says so in its own docstring: anything that reads the mounted file must ask the same three or it points at a path that was never mounted.
The matrix templates asked the global TLS_MODE instead. On an onion deployment TLS is off per application while TLS_MODE stays self_signed, so run 34653754175 logged 'CA trust N/A for web-app-matrix (TLS off/not self-signed) mark done' and the templates rendered the binds anyway.
The rest follows from the timing in that run: the matrix container was created at 05:49:35, the bundle was first written at 05:58:45 by sys-ca-selfsigned. At create time the bind pointed at a path that did not exist, which docker materialises as a directory, and the later docker restart matrix from 02_render_inventory.yml:22 failed on 'error mounting /etc/infinito.nexus/ca/ca-bundle.crt ... not a directory'.
MATRIX_CA_INJECTED holds the predicate once in vars/main.yml, beside MATRIX_CA_MOUNT_PATH, and both templates read it: the outer container bind and the synapse arguments for the nested daemon. The test pins both links of the chain, that the constant is derived from the lookup and that both if lines use the constant, so redefining it from TLS_MODE cannot pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CI zone served A and AAAA for INFINITO_DOMAIN and let every other type fall through to the public resolvers, which return NXDOMAIN for the reserved .test TLD. Per RFC 2308 that denial covers the whole name, not the queried type, so any caching resolver in front of CoreDNS applies it to the A record the deployment depends on.
Without such a cache nobody noticed: Mastodon's EmailMxValidator asks MX, takes the NXDOMAIN, then asks A anyway and gets the address. On a Tor node svc-net-tor puts dnsmasq in front, the denial is cached, and tootctl accounts create ends on 'Failure/Error: email unreachable'. It cost Fediwall#0 and Fediwall#2 in run 34653542922 and Fediwall#2 in run 34653754175, all three compose plus tor; the same two jobs passed in run 34614940508 as compose without tor, same command, same administrator@infinito.test, rc 0.
A template with no answer returns NOERROR with an empty answer section and the SOA in authority, so the zone denies a type without denying the name. It sits after the A and AAAA templates, which would otherwise have their names answered NODATA, and before forward, so no type escapes.
Measured against 172.30.0.53 in the running stack: infinito.test MX went from NXDOMAIN with an authority record to NOERROR with zero answers and the aa flag, infinito.test A and auth.infinito.test A still return 172.30.0.10, and deb.debian.org still resolves through the forward.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The compose-up handler resolves its chdir from lookup('container', application_id, 'directories.instance') at flush time. Bluesky passes compose_handlers_flush false to sys-stk-backend and notifies compose-build and compose-up afterwards, so the topics stayed pending until the next flush. That flush belongs to svc-prx-openresty, which the front proxy include pulls in, and it ran them with openresty's application_id.
Run 34653754175 shows the effect: the only compose up in the whole job created the openresty container in 0.52s, no pds container lifecycle line appears anywhere, and the wait for PDS XRPC then retried for 69s against a stack that was never brought up, ending on 'Status code was -1 and not [200]: Request failed: <urlopen error [Errno 111] Connection refused>'.
The flush now sits directly after the rebuild notifier, before the first front proxy include. sys-stk-backend has already rendered the stack there, so this is not a return to 293c67780d, whose registration sat before the staging and let the container runtime's first flush run the compose handlers before ca-inject and compose.yml existed.
The re-own include ahead of it repeats what sys-stk-backend and sys-svc-rdbms already do: it runs no task, it re-registers the handlers, and it binds them to this application_id instead of whatever the play happens to carry. The ordering test asserts the flush precedes the proxy and that the re-own precedes the flush.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible templates delegate_to before it evaluates the task's when, so a task delegating to N8N_HOST_NODE kills the play with 'N8N_HOST_NODE' is undefined whenever the resolver that sets it was skipped, even though the delegating task would have been skipped too. 43e09ebf0d put the resolver inside the same conditional block. Run 34653754175 shows the pair directly above the failure: the include skipped on the false condition while the setup call had returned status 200, and the play ended at 01_bootstrap.yml:92.
The resolver now runs unconditionally, before the block. It costs one lookup per deploy and changes no state.
The new lint reads the class rather than this instance: a conditional resolve_host_cid include whose PREFIX_HOST_NODE a delegate_to in the same role reads, unless an unconditional resolver or set_fact defines that prefix role-wide. The walker carries the condition through block, rescue and always, so an include nested in a block does not read as unconditional. Against the pre-fix file it reports would flag: ['N8N'].
It found one more: web-app-yourls/tasks/install.yml wrapped its tasks in a block whose when repeated the IS_STACK_HOST condition that main.yml already carries on the include. The wrapper is gone and the tasks are dedented; nothing had broken there because the condition always held.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A workflow the entries call declares its own concurrency group, and its jobs then leave the entry's group. The entry's cancel-in-progress no longer reaches them, so a superseded run keeps the runners its remaining jobs hold. Run 34614940508 held its runners after its successor arrived and only a manual force cancel ended it; its check runs carry no 'Canceling since a higher priority waiting request' annotation, only the forced cancel.
call-orchestrator.yml and call-test-runner-smoke.yml drop their groups. The smoke group was keyed on the caller's ref as well, so a manually dispatched smoke run cancelled the smoke job of an orchestrator run on the same ref. call-images-build-ci.yml and call-images-mirror-missing.yml keep theirs: their group deduplicates image builds across runs rather than scoping a single run, and it does not cancel.
The test asserts both halves, that the orchestrator declares no group and that every entry calling it carries group plus cancel-in-progress, and adds the chain below it. It finds the callers by their uses: line rather than by an entry-* filename, so a caller named otherwise cannot skip the requirement. It fails against HEAD, where call-test-runner-smoke.yml declares a group.
The cause is established negatively: no concurrency annotation exists on the superseded run. What would settle it is a run that supersedes another and cancels it, and that needs a push.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
csp-checker navigates every URL with a 20000 ms budget until
domcontentloaded. Over the Tor SOCKS proxy that expires before a page of
module scripts finishes: run 34614940508 lost Mastodon#1 and OpenProject#1
to it, 26 of 27 attempts logging "Unable to reach http://<onion>/
(Navigation timeout of 20000 ms exceeded)" while sys-service-terminator
retried the unit for 35 minutes. No policy was ever read; the checker never
got that far.
csp-checker 2.3.0 takes --timeout, so script.py takes --onion-timeout and
hands it to the onion batch alone, before the -- separator that would make
it a URL. Clearnet keeps the image default. The budget is
HEALTH_CSP_TIMEOUT_MS (20000, the image default) times the tor flavor's
multiplier, 100000 ms on the public network.
The unit's TimeoutStartSec had to follow. It was 30 + 25 s per domain --
130 s for the two onion vhosts Mastodon checks, while a 100 s budget needs
up to 200 s, so systemd would have killed the check mid-run. The filter now
takes onion_per_domain_seconds and counts .onion domains at that rate;
without the argument it computes exactly what it did before, so
sys-ctl-hlth-webserver is unchanged. The Mastodon case goes from 130 s to
450 s, a clearnet-only play stays at 130 s.
Tests pin that --timeout is appended only when set and stays ahead of the
separator, that main budgets the onion batch alone, that the role passes
both --onion-timeout and the onion rate for the start budget (both guards
fail against the previous tasks file), and the filter's split rate.
Not covered: the filter still caps at 3600 s, which binds beyond 34 onion
domains in one round.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-flavor multipliers lived in test-e2e-playwright's vars, where only
that role could read them. The CSP health check needs the same number: it
probes onion vhosts through the Tor proxy, where the checker's navigation
budget has to grow by the same factor as a Playwright wait.
They now sit in svc-net-tor's meta/services.yml next to services.tor.flavor,
the key they are indexed by, and the Playwright var reads them through
lookup('config', 'svc-net-tor', 'services.tor.timeout_multipliers'). That
lookup resolves from the role defaults, so it answers even where svc-net-tor
is not part of the round, which is how the flavor was already read.
The flavor-coverage test moves to the tor role and reads the new location;
the values (chutney 2, public 5) and what the Playwright run computes from
them are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sweep may deploy 210 rows (available) in chunks of 80. Three declared
blocks hold exactly that when there is no priority line, but priority
rows get a chunk of their own that is never topped up with regular rows.
Nine priority rows therefore occupied a whole block, left the regular
rows 2 x 80 = 160, and 42 rows of budget went unused.
Adding a block to the YAML alone would not have changed that:
matrix.chunks_of handed the split slots.chunk_count(), which is
min(ceil(available / chunk size), INFINITO_CI_MAX_CHUNKS) and ignores
the priority seam, so the plan still stopped at three blocks.
call-orchestrator.yml now declares test-deploy-chunk-3, gated on chunks
0 to 2 like every chunk gates on all before it, and report-main-failures
and done need it. INFINITO_CI_MAX_CHUNKS is 4, one above what the budget
fills, and slots.chunk_blocks() returns it. chunks_of hands the split
every declared block; available still caps the rows, and a chunk whose
slice comes back empty skips its deploy. chunk_count() stays the "chunks
filled" figure of the budget report.
Tests: the split spends 9 + 80 + 80 + 41 = 210 rows with a spare block;
chunks_of plans three chunks from chunk_blocks=3 while chunk_count=2
(the previous wiring yields [1, 10]); an integration guard requires the
declared chunk jobs to equal INFINITO_CI_MAX_CHUNKS and to exceed
ceil(available / chunk size) by one, and fails on the stale value 3.
A shell that inherited INFINITO_CI_MAX_CHUNKS=3 keeps failing that guard
until it is restarted; make dotenv-force rewrites .env.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Snipe-IT#1 (run 34615516009, job 103327034508) failed every login in the
async pass with an oauth2-proxy 500. The sidecar log shows 24x "error
saving redis session: WRONGPASS invalid username-password pair". The swarm
matrix rotated 54 credentials before that pass, svc-db-redis re-rendered
its requirepass and redis_redis was recreated ("about an hour ago"), while
snipe-it_sso-proxy kept running from the first pass ("3 hours ago").
oauth2-proxy reads its config, redis password included, from a bind mount
at start. Its render task notifies compose-up, which in compose is
"compose up -d --force-recreate" and so recreates the sidecar, but in swarm
is "docker stack deploy", which only restarts a service whose spec changed.
A new file behind an unchanged bind mount is no spec change, so the sidecar
never read the rotated password. pihole#1 in the same run shows the compose
side: shared redis, 47 rotated values, the config re-rendered, compose up
recreated the sidecar, and the job passed.
The sidecar now carries a container label with the sha256 of its config,
rendered from the same template sso_proxy.yml writes, with
sso_proxy_application_id supplied through template_vars. A changed config
changes the label and with it the service spec, so stack deploy restarts
only the sidecar; an unchanged config keeps the hash and restarts nothing.
Compose behaviour is unchanged.
A guard test pins the label and requires it to hash the template
sso_proxy.yml renders; it fails against the previous template. The hash is
stable for equal input and changes with a rotated secret, checked with the
template lookup on ansible-core.
Not covered: other bind-mounted configs that embed rotated secrets have the
same swarm exposure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
At -vvv Ansible prints each module argument and result, so a password in
a command line, a request body, a header or a fact lands in the deploy
log. Nothing enforced masking outside the MCP task files, and no check
looked at parents at all.
A new lint, test_credential_tasks_are_masked, finds tasks whose printed
parts (module arguments, environment, loop) reference an identifier ending
in password, secret, token, api_key or private_key, or
secrets.credentials. A task passes when it or an enclosing block sets
no_log, when every file that pulls it in does so through include_tasks
with apply.no_log or import_tasks with no_log (transitively), when the
secret only reaches a parameter the module masks itself, or when the
module masks its values. no_log on an include_tasks task itself does not
count, because it does not reach the included tasks. The rule can be
suppressed per task with `# nocheck: credential-masking`, now listed in
the suppression catalog, and the lint tests its parent and suppression
logic on synthetic cases.
It found 39 unmasked tasks in 19 roles, all real: join tokens, client
secrets, API tokens in Cloudflare headers, database passwords in
command lines, the ChatGPT bot login body and more. Each now sets
no_log from MASK_CREDENTIALS_IN_LOGS. The sys-dns-hetzner-rdns alias
hetzner_no_log, which only restated that switch, is gone.
Not covered yet: template and copy tasks whose rendered file carries a
secret print it as a diff under --diff.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
304144ce67 probed each bridge's healthcheck at /_matrix/mau/ready. mautrix
sets Ready only after Synapse has pinged the appservice (connector.go
starts the listener, then ensureConnection, then AS.Ready = true), and a
swarm task with a healthcheck stays "starting" and receives no VIP traffic
until it is healthy. The ping could never arrive: Synapse logged 150
ConnectionRefused per bridge, each bridge exited FTL 13, and the stack
never converged (run 34615516009, Matrix#2, job 103327034630).
The healthcheck now probes /_matrix/mau/live, which answers once the
listener is up, before the ping. A deploy task after the Synapse wait asks
every bridge for /_matrix/mau/ready from inside Synapse, over the same
name Synapse pings, so a bridge that never becomes ready still fails the
deploy without gating swarm routing.
The ChatGPT bot is rendered only once its access token exists. Its account
and token are created by stack_host_provisioning after the first stack
deploy, so an earlier bot task failed its login with M_FORBIDDEN until the
converge wait called it stuck. The re-render after provisioning adds it
with a valid token.
Verified in a compose run of variant 2: the readiness task got 200 from
meta, signal, slack and whatsapp on the first attempt. Tests pin the live
probe and the token guard and fail against the previous template.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sociallogin creates a Nextcloud user as "<provider name>-<keycloak sub>",
and the provider name was the Keycloak domain. On an onion variant that
domain alone is 67 chars, so every OIDC callback died with
InvalidArgumentException "Username is too long" and HTTP 500 at
/apps/sociallogin/custom_oidc/auth.<onion> (run 34615516009,
Nextcloud#1, 204 occurrences). The whiteboard and admin OIDC specs failed
on the same login.
This is the second derived identifier an onion domain overflows; the
kernel hostname is the first, clamped to 63 bytes by container_hostname
since the pretix and bookwyrm fixes. The domain-or-entity rule now lives in
one lookup, bounded_name(application_id, max_length): the primary domain
when it fits, otherwise the entity name. container_hostname delegates to it
at 63 and keeps its eight callers and its behaviour.
The provider name is bounded_name('web-app-keycloak', 64 - 1 - 36), the uid
limit minus the separator and the UUID sub. A short clearnet domain stays
the name, so existing users keep their uids; an onion deploy gets
"keycloak", which, unlike a hash of the domain, survives a rotated onion
address.
Tests cover the lookup's fit, boundary and fallback cases and its argument
validation, container_hostname's delegation at 63, and the provider name's
room for the sub.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Discourse latest now validates openid_connect_enabled and refuses it
until the discovery document, client ID and client secret are set. The
bootstrap enabled OIDC first, so a fresh database aborted the launcher
rebuild at `rails r "SiteSetting.openid_connect_enabled = true"` with
Discourse::InvalidHTMLParameters (run 34615516009, WordPress#0, job
103327034654). Existing databases hid it because the values were already
stored.
The enable step now follows the three settings it depends on. A test
reads the real template and requires every prerequisite line to precede
the enable line; against the previous template it fails.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
INFINITO_DOMAIN_RE carried an already escaped regex through the env
chain: EnvBuilder.setdefault prefers a value already in the process env,
quote_for_dotenv escapes every backslash, and load.sh unescapes it again.
One inherited value with a doubled backslash was enough to write
"infinito\\\\.test" into .env, the rendered Corefile then matched a
literal backslash, and getent hosts test.infinito.test failed with rc 2.
The renderer now substitutes DOMAIN_RE = re.escape(INFINITO_DOMAIN) at
render time, so no backslash travels through the env or .env at all. The
template placeholder is renamed because it is a derived render value, not
an env key, and INFINITO_DOMAIN_RE is removed from default.env and the
passthrough keys, leaving INFINITO_DOMAIN as the single source.
A renderer test renders the real template and requires a single-escaped
match with no doubled backslash and no leftover placeholder. Verified
live: the Corefile renders ^infinito\.test\.$, test.infinito.test
resolves to 172.30.0.10 and test.infinitoXtest does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pre-compose staging (social-app clone, login-broker copy, pdsadmin
download) ran before sys-stk-backend and notified compose-build and
compose-up. On a fresh host the container runtime's first flush ("Restart
and enable docker service") then ran the compose handlers before
/usr/bin/ca-inject and compose.yml existed, and "Generate CA trust
override" failed (run 34615516009, job 103327034609).
The staging tasks now only register their result. A changed-only notifier
after the sys-stk-backend include triggers compose-build and compose-up, so
no compose topic is pending at any earlier flush, and the handler-only
include that registered the owner early is gone. The ordering test asserts
both halves: the staging files notify no compose topic, and the rebuild
notifier sits after the backend include.
Verified locally in compose, variant 1: the reinstall full cycle passed
pass 1 with failed=0 and the rebuild task changed after the backend
include, pass 2 Playwright passed, and an update run finished failed=0 with
the rebuild task skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ubuntu CI image build (job 103277644636) hung on Canonical's
archive: apt fetched 26 MB at 417 kB/s, then 61.6 kB in 29 s, then sat
on the package download until the job was cancelled after 23 minutes,
while download.docker.com delivered 42.8 MB in under a second on the
same runner. Locally archive.ubuntu.com and security.ubuntu.com took
20-26 s to the first byte against 2 s for a university mirror.
INFINITO_APT_UBUNTU_MIRRORS in default.env is now the one list of
Ubuntu mirrors, in failover order: de.archive.ubuntu.com, ftp.fau.de,
ftp.halifax.rwth-aachen.de, archive.ubuntu.com.
- Image builds: the Dockerfile sources the list alongside the install
scripts and runs scripts/install/apt-mirrors.sh first. In an image
with ubuntu.sources it points both stanzas at apt's mirror+file
method, priorities in list order, and sets Acquire::http::Timeout to
10 s, so apt moves on from a mirror that fails or goes silent; other
distros are untouched. In ubuntu:latest apt-get update took 8 s
against 110 s on the stock sources, a dead first mirror was left
behind within 25 s against 62 s on apt's defaults, and the swarm node
packages (80 debs) came in 17 s. The Dockerfile's own RUN lines ran
clean on ubuntu, debian, fedora and arch.
- Package cache: package.sh proxies the first entry as the primary
Ubuntu repos and the second as their fallback. Nexus revalidates
metadata on every request here (metadataMaxAge 0), which costs a
full download from a mirror without an ETag; both de.archive and
ftp.fau.de send one, and the 15 MB universe index now comes back
from the cache in 0.02 s.
- tests/external/repository/test_apt_ubuntu_mirrors.py keeps the list
honest: the first mirror must answer InRelease within 5 s (median of
three) or the test names a faster later one, no mirror's -updates may
lag the freshest by more than 24 h, and every entry must be the
primary archive, a Canonical country alias or an enabled official
mirror in Launchpad's registry. It caught the first draft of the
list: mirrors.edge.kernel.org is not an official Ubuntu mirror.
After pulling, an existing checkout needs `make dotenv-force`: plain
`make dotenv` re-reads the old .env through BASH_ENV and keeps its
values.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The flavor-coverage test from 214e7e29d1 read the role's vars with
yaml.safe_load(VARS_PATH.read_text()) and spelled "vars" / "main.yml"
out by hand. test_no_uncached_file_reads forbids a direct .read_text
and test_file_spot requires the ROLE_FILE_* constants, so the file now
loads through utils.cache.yaml.load_yaml_any and joins
ROLE_FILE_VARS_MAIN. What the test asserts is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34421944328 web-app-litellm#1 (README host-mode replay) answered
every health probe on llm.ai.infinito.test with 502: litellm
crash-looped on "P1000: Authentication failed against database server
at postgres" while postgres logged 'Role "litellm" does not exist'. The
README inventory has no svc-db-postgres group, so sys-svc-rdbms skipped
the central provisioning, yet svc-ai-litellm pinned postgres.shared to
true and kept consuming a shared database nobody had created its role in.
postgres.shared now reads "{{ 'svc-db-postgres' in group_names }}":
an inventory with the central group keeps the shared database, the
README inventory gets a dedicated one.
That dedicated instance, rendered by sys-svc-rdbms postgres.yml.j2,
joins only the compose default network, and the networks renderer
dropped default from every svc-db-*/svc-ai-* role. A local variant-1
deploy measured litellm-database on litellm_default alone and litellm on
the shared litellm network alone. _suppress_default now keeps default
when the role's database is local. The variants pin shared to true (0)
and false (1) so the matrix exercises both paths, and the generated
READMEs show the postgres relation as 0..1.
The three attachment helpers move from render.py to attachments.py,
which render.py re-exports, and test_render.py splits along the same
seams into five files plus _render_helpers.py with every test id kept.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34421944328 Bluesky#1 stopped at "The requested handler 'compose-up'
was not found in either the main handlers list nor in the listening
handlers list", right after cloning the social-app build context. The
pre-compose staging (01_social_app.yml, 02_login_broker.yml) and the PDS
routine (03_pds.yml) notify compose-up and compose-build, but the
handlers only exist once sys-stk-backend loads sys-svc-compose, which
happens later in the role.
00_core.yml now includes sys-svc-compose with tasks_from: none.yml and
handlers_from: main right after the once flag, the same handler-only
registration sys-stk-backend and sys-svc-rdbms dedicated.yml already use.
The notifications still run at the stack's own flush, after the compose
files exist; the role has no earlier flush. A handler-order test pins
the registration ahead of the pre-compose block.
The oauth2-proxy cfg heal leaves its inline shell for
files/shell/heal_sso_proxy_cfg.sh, which receives the cfg path and the
container name through the environment. The blanket
"container stop ... || true" becomes a precondition: the container is
stopped only when container inspect finds it, so a directory is still
removed when the container is gone, and a failing stop of an existing
container now fails the task instead of vanishing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit wrote the same proxy-aware logout expression into
four javascript.js.j2 templates. The rule it encodes - sign out of
oauth2-proxy first when the app is proxy-gated, else go straight to the
provider - belongs where proxy-gating is already decided.
utils/roles/applications/services/sso.py gains logout_url(url,
is_proxy_gated) next to get_sso_config, with OAUTH2_SIGN_OUT_PATH as the
path constant; the rd value is quoted exactly as Jinja's urlencode
filter does it. The sso lookup exposes it as the logout_url term: it
renders OIDC through the templar, takes is_proxy_gated from the same
resolution, and refuses a context without OIDC. Prometheus, Checkmk,
Baserow and Jitsi now render lookup('sso', application_id, 'logout_url').
Behaviour is unchanged: the resolver tests pin the rd encoding against
jinja2's do_urlencode, and the lookup tests cover a proxy-gated app, an
OIDC-native app, a disabled SSO and a missing OIDC.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34421944328 odoo#0 loaded assets over the node onion that the page
could not use.
build_csp_header mirrors every clearnet token that carries the primary
domain onto the node onion, but kept its scheme. Onion vhosts are served
in plaintext, so a sibling like https://cdn.<onion> never matched the
http://cdn.<onion> request it was meant to allow. Onion siblings now map
https to http and wss to ws; the clearnet token is unchanged.
The body filter rewrote external asset origins to the onion mirror only
in HTML, so stylesheets kept pulling https://<origin> directly. When an
app has tor enabled and external asset origins, the header filter now
also flags text/css responses on .onion hosts (dropping Content-Length,
since the body changes), and the body filter runs those through the same
buffer and mirror rewrite and returns before the HTML injections. The
flag is set under the same Jinja condition that renders the mirror
block, so a flagged response always has a rewrite to run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34421944328 n8n#1: Open WebUI 0.11.0 ships only a Streamable HTTP
MCP client. n8n 1.95.3's MCP Server Trigger speaks SSE only and answered
the client's POST to /webhook/infinito-mcp/sse with 404, so "every
registered MCP server answers Open WebUI's client" failed.
n8n moves to 1.100.1 and the managed trigger to typeVersion 2, which
serves Streamable HTTP at /mcp/<path>. meta/mcp.yml declares
transport: streamable_http at /mcp/infinito-mcp (the trigger path stays
infinito-mcp) and points source_url at the 1.100.1 tree. Open WebUI's
supported_transports drops sse, which its client never spoke; no
provider declares sse any more, so no pairing is lost.
n8n#0: the guest spec probed ${APP_BASE_URL}${MCP_ENDPOINT_PATH} and
expected a refusal of >= 300 without an event stream. The trigger is
exposure: internal, so the public vhost must answer 404; the spec now
asserts exactly that through mcpEndpointUrl(..., N8N_BASE_URL), and the
bearer guard itself is proven in-cluster by the CLI MCP contract.
Requirement 035 and the MCP audit table follow the version and
transport.
Upgrade path, read from upstream at both tags: the postgres migrations
are identical (45/45) and the only new common migration is
AddProjectDescriptionColumn, an additive nullable column; BREAKING-
CHANGES.md lists only 1.98.0 in range (route metric format, Form node
tag attributes, Node >= 20), none of which this deployment uses. A fresh
1.100.1 install ran all 89 migrations and passed the MCP bridge, server,
non-member and guest specs locally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0a52433dcb gave the standalone request fixture a Tor-resilient GET,
because the bundled socks client caps the CONNECT at 30s and a cold
onion circuit fails a request regardless of its timeout. POSTs and calls
made through context.request still went out once, and a circuit Tor
rejected ("Socks5 proxy rejected connection") was not recognised as
transient at all.
apiFetchOnion now takes the verb from opts.method and calls
request.fetch, so it serves both APIRequestContexts; apiGetOnion becomes
its GET shorthand. The rejection joins the transient patterns. Retries
stay onion-only and transport-only: clearnet keeps a single attempt and
HTTP or other errors re-throw at once.
The Mattermost LiteLLM spec routes its login, users/me, users/username
and channels/direct calls through it. The /posts call that sends the
prompt stays a single attempt, because a replayed post would message the
bot twice. The Hermes administrator spec reads /v1/models through
apiGetOnion.
env.test.js pins a POST surviving a rejected circuit, the single
clearnet attempt, no retry of a non-transport error over Tor, and the
GET shorthand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34421944328 Nextcloud#13 lost the dashboard's Matomo tracker. The
dns53 sampler on mgr-01 shows no :53 socket from 01:39:52 to 01:48:44
while the dnsmasq PID stayed the same, six seconds after a short-lived
container's veth on docker0. In that window the manager could not
resolve its own onion vhosts, so sys-front-inj-matomo's site checks for
web-svc-logout (01:45:35) and web-app-dashboard (01:47:18) never reached
OpenResty, the injection was disabled as "API not ready", and the
Playwright check found no matomo.js. The other four swarm managers of
the run that carried the same listener kept it throughout.
Under bind-dynamic, dnsmasq 2.91 re-enumerates interfaces on every
netlink address event and then garbage-collects every listener it did
not re-find (network.c enumerate_interfaces), also when the enumeration
failed; release_listener has no other caller. 5aa7506e36 reproduced this
on the lab nodes and 11c98825aa already moved the bridge-less case to a
static bind. The bridge case now binds statically as well.
This supersedes two earlier rationales, both measured on the local stack:
- bb0e098a9f kept bind-dynamic because dnsmasq starts at boot before
docker0 exists. A unit drop-in (files/dnsmasq-restart.conf,
Restart=on-failure, RestartSec=5) now retries the start: with a missing
listen-address the start exits 2, and dnsmasq came up 4s after the
address appeared.
- 4739b2de99 called bind-dynamic load-bearing because docker0 can
disappear across a docker restart. Only a start without the address is
fatal; a bound socket survived the address being removed and re-added
and the interface being recreated with a new ifindex, answering on the
same socket inode each time.
The docker0 fact is re-gathered before torrc and the dnsmasq drop-in are
rendered. The play-start fact predates default-address-pools moving the
bridge (285ba1e76c), and a static bind on a stale address would make
dnsmasq exit. container_dns reads only the pool, so daemon.json is not
affected by the refresh.
The host resolv.conf moves from an inline copy content to
templates/resolv.conf.j2; both it and the drop-in render byte-identical
to what the local stack already runs.
A veth churn of 13.8k cycles did not reproduce the loss locally, under
either bind mode.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every mautrix bridge generated its own registration with -g on start,
with fresh random tokens. Synapse had to read that file from a shared
directory, be restarted afterwards, and on swarm the services had to be
removed first so the registrations could be wiped (93735fd533). The
tokens changed on every redeploy and the deploy depended on a wait for
files a container writes.
Each bridge now has as_token and hs_token declared in meta/secrets.yml
and a fixed appservice_port in its addon. The bridge config reads the
tokens through lookup('config'), and registration.yaml.j2 renders the
registration from that config in the upstream -g format, so both sides
are known before anything starts and stay identical across redeploys.
The files reach the containers as secrets: matrix_bridge_mounts offers a
stack view (top-level definitions, content-hash named by compose_volumes),
a synapse view (one registration per bridge under /registrations, mode
0444) and a bridge view (the config at /config/config.yaml). The bridges
run as the synapse uid with an explicit --config and --no-update, keep
their scratch /data on a 16m tmpfs so hlth-volumes finds no anonymous
volume, and depend on a healthy synapse. Their healthcheck is
/_matrix/mau/ready, which the bridge only reports after whoami with the
as_token and an appservice ping carrying the hs_token, so a healthy
bridge proves the token pair.
The compose tasks lose the compose down, the swarm service rm block, the
registration wait, the permission fixes and both synapse restarts; this
reverts the workaround of 93735fd533.
Verified on the local compose stack: four bridges ready, synapse pings
answered, registration and config sha256 unchanged across a redeploy,
test-bridge-roster and 12 matrix Playwright specs green. A token
rotation (new pair in both files, compose up --force-recreate as the
handler runs it) came back healthy; a mismatched pair left the bridge
unhealthy. The swarm-shaped stack file passes docker stack config with
the 0444 secret modes, the 16 MiB tmpfs and content-hash secret names.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
compose_volumes copied extra_configs and extra_secrets through verbatim.
A file-backed extra without a name therefore reached swarm under a fixed
name, and swarm secrets are immutable: a new content under the same name
is rejected on the next stack deploy. Such extras now get the same
{role}_{key}_{sha8(content)} name that _config_secret_name already gives
meta-declared entries. An explicit name is kept.
container_volumes appended extras as handed over. Ansible passes one
lookup's result to another as proxy containers, and a TaggedInt mode
inside a proxy mapping made PyYAML raise "cannot represent an object".
The private _to_plain of compose_volumes moves to utils/templating as
to_plain, now coercing bool, int and float to the exact builtin types
(bool before int, since bool subclasses int), and container_volumes runs
its extras through it.
Only web-app-matrix passes extra secrets or configs today; the
extra_volumes path that web-app-nextcloud uses is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
performKeycloakLogin polled the page URL with toContain(canonicalDomain).
On the Keycloak auth page the app domain already sits in redirect_uri,
so the poll was satisfied before the login had completed. The denial
path and the biber/admin iframe checks used the same substring test.
All of them now compare hostnameOf(url) with the canonical domain, which
is a bare hostname (env.js reads it with /\.onion$/).
LOGIN_CONTROL_NAME never matched a login link that carries an
icon-font glyph: Playwright's accessible name includes the private-use
character, and the pattern only allowed whitespace before the text. The
post-login waitFor({state: "hidden"}) therefore passed at once, even
before oidc.js had swapped the link. The pattern now admits \p{Co}
before the text, so that wait really waits for the swap.
dom.test.js pins both: glyph-prefixed login controls match while Logout
and Accessories do not, an IdP URL carrying the app domain in
redirect_uri is rejected, a look-alike suffix is rejected, and
about:blank yields an empty host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
resolveTimeout multiplies every Playwright window by 5 for a .onion
target. That default arrived in 35dee215c8 (2026-07-08), before
5399b48482 (2026-07-24) gave CI the chutney flavor, a private Tor net on
localhost, and PLAYWRIGHT_ONION_TIMEOUT_MULTIPLIER was never set. Waits
that expire by design therefore cost five times their base under chutney
too, which is where most of the Tor surcharge came from.
The Playwright run now passes the multiplier for onion targets only,
looked up from svc-net-tor's services.tor.flavor: 2 for chutney, 5 for
the public network, whose circuits really are slow. An unknown flavor
fails the lookup instead of falling back, and a unit test holds every
flavor directory of svc-net-tor to an entry.
Chutney traces of run 34421944328: 700 successful timed waits across 186
tests used at most 0.42 of their unscaled window, and the longest
client-side expect took 5.1s. Across runs 34421944328 and 34419507038 no
chutney test with its own ceiling (401 of them) used more than 53% of
the new x2 ceiling, even at today's x5-stretched duration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
confirmKeycloakLogoutIfPrompted waited the full 2s times the onion
multiplier for Keycloak's confirmation button even when Keycloak had
already redirected away because the end-session request carried an
id_token_hint. It now races the button against the page leaving the
logout endpoint (waitUntil: commit) and still clicks the button when
Keycloak renders the prompt. The dashboard's own confirmLogoutIfNeeded
stops the same way, and its poll interval no longer scales, so a slow
target is noticed sooner rather than later.
inAppLogout's first candidate poll ignored a visible settings link, so a
surface without a recognisable logout control or menu trigger, such as
the Mastodon home timeline, burned 30s times the multiplier before the
existing settings fallback ran. A visible settings link now ends that
poll as well; logout controls and menu triggers are still checked first
in every round. The settings selector moves to settingsLinksOn so both
callers share it.
Traces of run 34421944328: the dashboard's confirm probe expired after
2.03s on clearnet and 10.07s over Tor, and Mastodon's biber logout
polled 165s over Tor before reaching the settings page. Replayed in
Chromium, a Mastodon-shaped surface logs out in 3.24s instead of 33.45s
and a redirecting Keycloak logout returns in 0.20s instead of 2.00s,
while a prompting one is still confirmed. In the traced persona logouts
of six artifacts every other poll found a candidate in its first round,
so their path does not change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Over streamable_http the contract probe judged initialize, tools/list and the
read call by HTTP status alone. A server that answers 200 with a JSON-RPC
`error` object therefore passed initialize and the read call, and failed
tools/list only as "server serves []", which names the wrong cause. The SSE
path checked initialize and the read call, but not tools/list. mcp-adapter,
which fronts WordPress, maps invalid params to 200, and Moodle's
webservice_mcp answers a failing tool call with 200 and code -32603, so a read
that returned nothing could turn the check green.
rpc_error() extracts the error object of a JSON-RPC body, refused() is built
on it, and reject_error() fails initialize, tools/list and the read call on
both transports with the server's own message. A 200 whose body is not JSON
still carries no error object, so a provider that has not started keeps the
caller retrying as before.
The streamable_http initialize, and the one in the negative probes, sent no
clientInfo, which the MCP SDK requires. Home Assistant (mcp 1.26.0, stateless)
answers such an initialize with 200 and -32602 and serves the rest anyway, so
only the old status check let it pass. All three initialize calls now share one
INITIALIZE with clientInfo, as the SSE path already sent. Against a Home
Assistant-shaped endpoint on mcp 1.26.0 the SDK logs "InitializeRequest.params.
clientInfo Field required" for the old probe, and the fixed one prints OK.
Against HEAD, an error-on-200 at initialize and at the read call printed OK;
now all three steps print REJECTED naming the error. The MCP adapter in front
of sixteen of the nineteen streamable_http providers answers 200 only with a
result (svc-ai-mcp-adapter/files/python/server.py), so none of them changes
outcome.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard landed in 2acf91e18 breaking two rules it should have followed from
the start, and `make quality-high` caught both:
test_ca_trust_passthrough.py:23: `parents[N]` index expression
test_ca_trust_passthrough.py:23: `PROJECT_ROOT` assignment outside `__init__.py`
test_ca_trust_passthrough.py:44: .read_text(...) uncached
So `PROJECT_ROOT` moves into the package `__init__.py`, where every other role
test suite keeps it, and both template reads go through `read_text` from
`utils.cache.files` so a second reader of the same file pays one open.
Verified with the flags the suite actually runs under. scripts/tests/code/run.sh
passes `--import-mode=importlib` and `consider_namespace_packages=true`, which
is what makes a hyphenated `roles/<role>/` directory importable at all; a plain
`pytest <path>` fails the relative import here and does so for the existing
sys-svc-mail-msmtp suite too, which is the control that says the pattern is
right and the invocation was wrong. Under those flags both suites pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mdad stack runs its own Docker daemon inside the matrix container, so a
bind source in `matrix_synapse_container_extra_arguments` is resolved in that
container's mount namespace, not on the host. `vars.yml.j2` mounts
`CA_TRUST.bundle_host` while `services.yml.j2` passed only
`CA_TRUST.cert_host`, and those are two different files: the root CA and the
bundle beside it. The bundle therefore never existed where synapse looked.
`docker create` refuses the unit before Matrix logs anything of its own. From
the rescue artifact of job 102465251843 in run 34348821577, the nested
journal at containers/matrix/nested/.../journal.txt:5018:
matrix-synapse[5612]: Error response from daemon: invalid mount config for
type "bind": bind source path does not exist:
/etc/infinito.nexus/ca/ca-bundle.crt
Five start attempts, the same line each time, and the only error in the 872 KB
journal. matrix-heisenbridge follows it down as a dependency, `systemctl
restart` returns rc=1, the mdad bootstrap fails and the rescue task ends the
play. Not an OOM (exit 1, not 137, `OOMKilled: false`, 9.7 GB free), not disk
pressure, not a failed pull, and 16 % of the job budget spent.
b5fcd535de is where the asymmetry began: it moved this mount from `cert_host`
to `bundle_host` for a good reason -- SSL_CERT_FILE and REQUESTS_CA_BUNDLE
replace the trust store rather than extend it -- and left the passthrough on
`cert_host`. So the bundle stays; what was missing is the line that carries it
in, at the same absolute path the nested daemon asks for.
The test that comes with it reads both templates as text and requires every
CA_TRUST path synapse binds to be passed through at that same path. Rendering
would need a full Ansible context, and the invariant is about what the two
files say to each other rather than about the values. It fails against HEAD,
where `bundle_host` is bound and only `cert_host` is passed. It also fails
loudly if the regex ever stops matching how the extra arguments are built,
rather than quietly guarding nothing.
The cause is proven to the byte; the effect is not observed. `docker create`
never succeeded, so a second synapse defect behind this one would still be
invisible. A local `make compose-deploy apps=web-app-matrix mode=update` with
TLS_MODE=self_signed, grepping the nested journal for "Started
matrix-synapse.service", settles both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rows are indexed by `(name, variant)` and a variant is an int. A token that
names only a role carries no variant, so the lookup ran with `None` and could
never hit, and every such token was reported as matching no discovered row.
That is the form `collapse_to_roles` produces, so it is the form a retriggered
priority line arrives in. Run 34348821577 shows the result: all twenty-one
priority entries carried
priority: '<role>' matches no discovered row
(no row in this run's mode and lifecycle envelope)
while baserow, checkmk, decidim and the rest went on to deploy fine. The
warning was not wrong about one entry and right about twenty; it was blind. A
role that genuinely has no row in the envelope looked exactly like a role with a
full set, which is how priority work goes missing without anyone being able to
see which entry did.
A token that narrows nothing beyond the role name is now answered by existence:
it asks for the role in whatever axes the rotation picks, so any discovered row
satisfies it. `Pin.pinned` already draws that line - "whether the token narrows
anything at all beyond the role name" - and pinned tokens keep the old path,
`check_pins` included.
Three tests come with it. There was already one for a bare role that matches
nothing, and none for a bare role that matches something, which is why the
lookup could stay broken. One of the new ones uses variant 4 so a future
implementation cannot pass by quietly guessing variant 0. Checked against HEAD:
the same input returns the spurious warning there, so the tests fail without
this change.
This restores the signal, not the lost jobs. The other silent path, `priority`
being truncated to the job budget in cli/meta/ci/chunks.py, is untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
What `make quality-high` rewrote on its own, kept apart from the changes it ran
against so neither has to be read through the other.
The matomo healthcheck test moves from a flow sequence to a block one, which is
the formatter's threshold for a long line and not a change to the command: both
revisions parse to the same two-element list, escapes included. The healthcheck
timing lint loses two blank lines between methods.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule read the character after `timeout:` and objected to a digit, so a
duration bypassed the helper the moment it was given a name. That was already
reachable before, and it became easy to reach now that a poll interval is
allowed to stay unscaled: the two look identical at the call site and only one
of them is correct.
They are opposites, and the rule now says so. An upper bound must scale, because
the answer over a Tor circuit arrives later and a fixed cap expires before it
does. A poll interval must not, because scaling one makes a slow target notice
its element later rather than sooner, which is the reverse of the tolerance the
scaling exists to buy. So `timeout:` rejects a CONSTANT_CASE name as well as a
digit, while `waitForTimeout` and `setTimeout` keep matching on a leading digit
only and accept a named interval.
Across the 1307 `timeout:` sites in scope the sharpened rule finds exactly one,
and it is a true positive that wanted saying out loud. The n8n LiteLLM spec
passes a completion timeout into an `n8n-nodes-base.httpRequest` node: not a
Playwright wait at all, but a parameter for a request n8n makes to LiteLLM
inside the deployment, which never crosses a circuit. Leaving it unscaled is
right and was invisible; it now carries the sanctioned marker and its reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three waits in the logout flow were blind sleeps scaled by the onion multiplier.
A timeout is an upper bound and shrinks to nothing when the answer arrives early,
but page.waitForTimeout always burns its full duration, so multiplying it by five
spends the time whether the target is slow or not. The settle before the logout
sweep cost three seconds on clearnet and fifteen on an onion, every persona, every
role; the dropdown settle cost that again per menu trigger tried; and the poll
interval of waitForAnyLogoutCandidate was multiplied too, which made detection
five times coarser exactly where the target is slowest.
So the sleeps become waitForLogoutControl, a bounded poll that keeps the scaled
deadline and drops the scaling from the interval. It returns the moment a
logout-shaped element is visible, which is the condition the click that follows
needs anyway. Worst case is unchanged: with nothing to find it runs the deadline
out, exactly what the sleep did unconditionally. The first of the two
waitForAnyLogoutCandidate calls around the settle went with it; its result was
discarded, and the second still carries the thirty second deadline that covers a
navbar arriving late.
Four unit tests pin the contract, since the argument for the change is entirely
about when it stops waiting: it returns without sleeping when the control is
already there, returns as soon as one appears, costs at most the deadline when
none ever does, and polls on a fixed 250ms interval no multiplier touches.
Measured context, from run 34348821577: the same dashboard logout test takes
161.0 to 164.0s over Tor and 34.1 to 34.8s on clearnet across five jobs spanning
two distros, both deploy modes and three filesystems. A 1.9% spread across five
runner VMs is not network latency, it is a constant being multiplied, and the
ratio 4.73 is the multiplier itself. Solving the pair puts 32.0s of that test
inside scaled waiting against 2.3s of work. This commit removes the part of that
which is a sleep; the larger part is timeout caps on waits that expire and are
swallowed by a catch, which is a separate defect and stays for now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The info-media marker moved to meta/info.yml one commit ago, and this rule is
where the reason it could not move earlier was written down: a comment-only
info.yml parses to an empty document, which the schema lint used to reject. That
obstacle is gone, so the docstring now states a justification that no longer
holds, and the two video-related exemptions sit in different files for no reason
a reader can reconstruct.
So the marker is read from info.yml, the failure message names info.yml, and the
docstring says why the marker lives there rather than why it cannot. No role
opts out of this rule today, so nothing under roles/ changes and the move cannot
break in either direction.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The marker names the reason a role has no upstream homepage or demo video, and
it sat in meta/main.yml because a comment-only info.yml parses to an empty
document, which the schema lint rejected outright. So the exemption lived one
file away from the two fields it explains, and a reader looking for the video
found neither the field nor the reason it was missing.
The schema lint now accepts an empty document exactly when it carries the
marker, and that is what allows the move. The media lint reads the marker from
info.yml rather than main.yml, and its failure message names that same file, so
the fix it suggests lands where the absent field would have gone.
A hundred and two roles follow. Every occurrence was a standalone comment on
the second line of main.yml, directly under the document start, so the move is
a line delete and an insert at the same offset. Seventy-three roles already had
an info.yml and gain the line; twenty-nine get one whose whole content is the
document start and the marker. All hundred and two reasons carry over verbatim,
including the thirty-five bespoke ones.
Nothing downstream sees a change. Of the four readers of meta/info.yml, three
coerce a null parse through `or {}` behind an isinstance guard, and
utils/cache/applications.py skips a falsy parse before assignment, so
applications.<role>.info stays absent for the twenty-nine exactly as it was
while the file did not exist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
start_interval is the cadence while start_period runs, and only while it runs.
Once the window closes the probe falls back to interval, so a service that
becomes ready just after it pays a whole tick anyway and the cadence never
reached it. Six of the seven flavors were shaped that way, Curl worst: a thirty
second window against a one minute interval.
Measured across 151 converge waits in four job logs, 63 sit exactly on a tick
boundary, 36 in the thirty second step and 27 in the sixty second one.
Attributed by application, web-app-seaweedfs is the largest at 1501s over
sixteen waits with a median of 57.7s: it probes with curl and fell straight into
the gap. web-app-zammad probes with connect, whose window is fifteen minutes,
and its 154s median is real boot time with nothing to recover.
So the window moves to ten minutes and the steady state interval to two, and the
flavor overrides that sat below them go: Curl, Wget and Connect lose interval
1m, Http its twenty second window, Nc its ten. Connect keeps fifteen minutes.
Neither number costs anything while a container works. start_period delays
nothing, it only stops failures counting, and interval no longer touches
convergence at all since that hangs on the window alone; it is steady state
load, which drops from 120 probes an hour per service to 30.
Forty six role files follow. Twenty seven declarations go that repeat the new
default or undercut it, an undercut window being the defect itself. Eight
windows longer than ten minutes keep their value and now say why. The literal
blocks and the three Dockerfiles this repository builds spell every timing out,
because outside the lookup an omission takes docker's value and not the
platform's.
A fourth lint rule keeps the shape: start_period must outlast one interval,
checked against the effective values, so raising interval without the window
cannot quietly bring the cliff back.
Two faults in the lint surfaced while building it and are fixed here. It
compared timings as strings, so 600s and 10m read as different values, and it
matched the healthcheck key exactly, so a line carrying a trailing nocheck was
skipped whole, which is how the two swarm-init partials went unexamined.
One regression, bounded. A container that starts and never turns healthy is
neither rejected nor errored, so stack_ready.sh only calls it fatal after three
killed tasks, and each cycle now spans the longer window. That reaches twenty
minutes rather than six, capped there by swarm_stack_converge_retries at 600
with a two second delay, and it lands only on runs that were failing anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three checks over the two places a timing can be written.
In meta/services.yml the lookup supplies the defaults, so a declaration is
either noise or a decision. A value equal to what the service would have
inherited is deleted rather than justified: it changes nothing and hides the day
the default moves. A value that differs is kept and carries its reason.
That reason is required for interval, timeout and start_interval only. Those
three cost something: steady state load for the life of the deployment, the
threshold at which a slow probe counts as failed, and the only timing that moves
the moment a stack reports healthy. start_period and retries buy patience with a
container that is already broken and cost nothing while it works. Demanding a
defence for them would have produced sixty one sentences nobody had evidence
for.
Outside the lookup, in a literal healthcheck block and in a role's own
Dockerfile, an omission is a finding too. Nothing there inherits from the SPOT,
and docker has no daemon-wide default to fall back on, so a missing key lands on
docker's value rather than the platform's. They disagree on start_period, zero
against thirty seconds, which is exactly what left start_interval without a
window in every hand-written block.
Comparison is against the effective default, resolved through the same compose
the lookup uses. Curl probes every minute, Connect grants a fifteen minute start
window, and a prefix raises timeout, retries and start_period further; measuring
against Probe alone would have called out every service that simply agrees with
its flavor.
Negative control for each check. Before the three commits that precede this one
the rule named 47 restatements, 20 unexplained deviations and 34 definitions
outside the lookup. It names none now.
Two limits worth knowing. The line number for a meta timing is found by
indentation and service name rather than by a parser, so an unusually formatted
file makes the rule ask for a reason it cannot anchor; it errs towards asking,
never towards silence. And the Dockerfile reader follows a backslash
continuation but not a comment inside one, which no Dockerfile in the tree uses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils/docker/healthcheck owns the probe timings, and among them start_interval,
the cadence docker uses while start_period is still running. A block written out
in a template bypasses that owner and never gains the key. Nothing stopped it,
which is how forty six blocks ended up without the cadence and a stack converged
no faster than its coarsest interval.
The rule walks every .j2 under roles/, takes each healthcheck: key with the
lines nested under it, and reports the block when its body contains neither {{
nor {%. Such a block can move verbatim, because the lookup writes
overrides.get(key, default) for every timing key and a declaration repeating the
literal values renders the same block plus the cadence.
A block that interpolates is exempt rather than tolerated. Its value resolves in
the template's own scope, which meta/services.yml does not share, so moving it
would be a rewrite and the check would be asking for a different probe.
The opt-out is a nocheck on the healthcheck line or the one above it, resolved
through is_suppressed_at in same-or-above mode, the idiom both sibling rules
already use. It is meant for a block that interpolates nothing yet still cannot
move: a partial rendered for another role's application_id, or timings the
flavor defaults would make worse.
With the two existing rules this closes the triangle. One requires every call to
find a declaration, the second requires every declaration to reach a call, and
this one requires a declaration to exist wherever one is possible.
Negative control: run before the four nocheck markers were added, it named
exactly those four blocks and nothing else; run after, it is green.
One known limit. The body ends at the first blank line, so a block a future
author splits with one would be read only as far as the gap. No block in the
tree is written that way today, checked across all twenty one that remain
literal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LITELLM_BACKEND_ROLES read lookup('deployment').groups, which deployment.py
fills from group_names. That variable is host-scoped: it lists only the groups
the host currently being processed belongs to.
On a swarm round the manager carries svc-ai-ollama, svc-ai-lmstudio and
svc-ai-litellm while the workers carry three groups and none of them is an AI
backend. Both branches of LITELLM_CHAT_MODEL therefore missed on a worker and,
with OPENROUTER_API_KEY empty, the model resolved to the empty string.
Only one task made that visible. Every stateful task in the gateway include
carries delegate_to STACK_HOST plus run_once, so it templates in the manager's
scope and wrote the correct model into the database. The assert carries
neither, so it re-templated in each worker's own scope and failed there while
the manager passed.
The manager-only membership is not a placement decision. The swarm test
inventory is provisioned with a single --host MGR, so every role of the round's
closure gets that one host, and extend_inventory then widens only
svc-swarm-node, svc-storage-nfs-client, svc-storage-nfs-server and the round's
own application. The three AI roles declare no placement at all; they are
manager-only because nothing widened them.
Add an inventory selector next to the host-scoped one, built from the play-wide
groups magic var, skipping groups without hosts and Ansible's implicit all and
ungrouped, sorted so the cache key is stable. The key takes it too, or two
hosts with equal group_names but different inventories would share an entry.
The blast radius is two readers, both in 16_ai.yml, and both ask only whether
svc-ai-ollama or svc-ai-lmstudio is present. On the manager the new value gains
exactly svc-storage-nfs-server and svc-bkp-nfs-2-local, neither of which either
reader tests, so the manager's answer does not move and only the workers change,
to match it.
This is the third step on the same line. 46a16127fc set .running; b149ad0d22
corrected that to .groups because a targeted apps= round leaves only the
command-line roles in the whitelist. .groups solves the targeted round and not
the multi-host one, because group_names stays host-scoped either way.
Measured in a two-host play with a real group_vars file and the real lookup:
the worker resolves ollama false through group_names and true through the new
selector, the manager is true both ways, and the assert that failed in CI
passes on both hosts.
Requirement 036 named the host-scoped backend selection as the thing that has
to change and deferred it. That residual is closed, so the paragraph now
describes the selector in use and why the two rejected ones are wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The account loop walked every user any role declares, 291 of them in a CI
round, and let each task discard the rest through its own when:. That still
costs a task execution per user per task: 1706 of 1746 executions reported
skipping, at 0.146-0.157s each, for 259.5s of a single job.
A role-local mailu_users lookup returns the subset instead, in the dict
shape lookup('users') yields so the call site keeps dict2items. The
predicate is the union of both gates the tasks apply: a mailbox account, or
a bot role, whose API token Mailu issues. Filtering on the mailbox alone
would quietly stop issuing tokens to a bot without one.
The per-task when: conditions stay. The returned set is a superset of what
each of them selects, so it decides how often they are evaluated, never
their outcome.
A non-list accounts declaration raises rather than being filtered away, the
way reserved_users.py already treats it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PASSWORD_TOKEN_RE matches the token anywhere inside a Jinja expression, so a
path or a filename carrying it demands "| quote" as loudly as a credential
does. It fired on Moodle upstream CLI script:
roles/web-app-moodle/tasks/01_manager_ops.yml:55: In shell tasks, password
expressions must include quote: {{ [MOODLE_CLI_DIR, reset_password.php] | path_join }}
The credential on the next line already carries the filter. Listmonk hit the
same shape earlier and was resolved by renaming the file, which is not open
here: the name belongs to Moodle.
The rule had no suppression at all - it never called is_suppressed_at - so a
marker in the role would have been silently ignored. Teach it the marker, then
use it.
Suppression is honoured at the finding line OR at the shell: key that opens the
block, and the second placement is the one that matters. A marker beside the
finding necessarily sits inside the scalar, and this block is folded: lines
join with spaces, so a hash would comment out the rest of the command and
swallow --username, --password and --ignore-password-policy. On the shell: key
it is a YAML comment instead. The failure message now says so, so the next
false positive explains itself rather than repeating the discovery.
Verified the command is untouched by loading the task and reading the string
Ansible would hand the shell: no hash, no marker, byte-identical.
Note the cost: the marker grants the whole block, not one expression, because
a per-expression marker cannot be placed. This block loses the guard against a
future unquoted addition.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_ram_gb read mem_total_mb while mem_available_mb sat next to it, added by
3e9deab6f7 for the drill abort and never carried over here. The ceiling exists
to stop the sidecar from competing with what already runs, so measuring the
machine instead of what is free made the term inert: on a 16 GB runner it
computed five workers while 5.5 GB was actually available, and the CI cap was
the only thing that ever bound.
Reading MemAvailable makes the guard real and self-limiting, which matters now
that the onion case raises the count. Against the memory the run itself reports:
PASS 1, 9113M free clearnet 1 onion 2
PASS 2, 5550M free clearnet 1 onion 1
Also withdraws a CI CPU divisor I had added on a false premise. It cited the
matrix telemetry stall60 as evidence that reserving three cores of four buys
nothing on a CI runner - but mem_stall_pct reads /proc/pressure/memory
(utils/env/runtime.py:84), so that figure proves memory headroom and says
nothing about CPU. Raising the clearnet count needs CPU evidence this run does
not carry, and a contention flake would cost coverage, so the divisor stays at
four until something measures the CPU side.
The onion lift keeps its own justification, which never rested on that number:
a browser waiting on Tor circuits does not consume CPU, measured at 4817s over
Tor against 911s on clearnet for the same specs on the same cores.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sidecar is the dominant cost of an onion round and almost free without it.
Measured on one Nextcloud variant at equal mode and filesystem, differing only
in the onion axis, sync pass:
test-e2e-playwright 4817s over Tor 911s on clearnet +5.3x
sys-svc-compose 2294s 2174s +120s
everything else < +25s
The whole Tor surcharge sits in one role. Tor burns no browser CPU, so a 5.3x
wall clock against identical specs on identical cores is circuit latency: the
phase waits rather than computes. The CPU divisor that sizes the clearnet count
does not describe it, and on a 4-core runner it yields exactly one worker,
which leaves the CI cap of 2 unreachable.
compute_workers takes an onion flag and, when set, multiplies the count and
caps it separately. The CI ceiling is overridden rather than respected: it
describes CPU contention, which Tor does not create. ram_workers keeps its
veto, because circuits are cheap and browsers are not.
The lookup reads the transport itself, from the same source the role derives
its own flag from, lookup(tls, application_id, url.base) tested for .onion,
following the _swarm_prefix idiom in plugins/lookup/image.py: read from
variables, render through the templar, fall back to the safe answer. One truth
about the onion state, and a caller asking for a worker count cannot forget to
pass it.
Factor 2 and cap 4 rather than more: each worker holds its own circuits and the
Tor daemon is the next bound, which no measurement here reaches. Expect the
sidecar near 2.6ks per pass instead of 4.8ks, roughly 4.4ks per onion job
against a 20.4ks budget. It does not help a clearnet round, where the sidecar
costs 911s to begin with.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
