The guard landed in 2acf91e18 breaking two rules it should have followed from
the start, and `make quality-high` caught both:
test_ca_trust_passthrough.py:23: `parents[N]` index expression
test_ca_trust_passthrough.py:23: `PROJECT_ROOT` assignment outside `__init__.py`
test_ca_trust_passthrough.py:44: .read_text(...) uncached
So `PROJECT_ROOT` moves into the package `__init__.py`, where every other role
test suite keeps it, and both template reads go through `read_text` from
`utils.cache.files` so a second reader of the same file pays one open.
Verified with the flags the suite actually runs under. scripts/tests/code/run.sh
passes `--import-mode=importlib` and `consider_namespace_packages=true`, which
is what makes a hyphenated `roles/<role>/` directory importable at all; a plain
`pytest <path>` fails the relative import here and does so for the existing
sys-svc-mail-msmtp suite too, which is the control that says the pattern is
right and the invocation was wrong. Under those flags both suites pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mdad stack runs its own Docker daemon inside the matrix container, so a
bind source in `matrix_synapse_container_extra_arguments` is resolved in that
container's mount namespace, not on the host. `vars.yml.j2` mounts
`CA_TRUST.bundle_host` while `services.yml.j2` passed only
`CA_TRUST.cert_host`, and those are two different files: the root CA and the
bundle beside it. The bundle therefore never existed where synapse looked.
`docker create` refuses the unit before Matrix logs anything of its own. From
the rescue artifact of job 102465251843 in run 34348821577, the nested
journal at containers/matrix/nested/.../journal.txt:5018:
matrix-synapse[5612]: Error response from daemon: invalid mount config for
type "bind": bind source path does not exist:
/etc/infinito.nexus/ca/ca-bundle.crt
Five start attempts, the same line each time, and the only error in the 872 KB
journal. matrix-heisenbridge follows it down as a dependency, `systemctl
restart` returns rc=1, the mdad bootstrap fails and the rescue task ends the
play. Not an OOM (exit 1, not 137, `OOMKilled: false`, 9.7 GB free), not disk
pressure, not a failed pull, and 16 % of the job budget spent.
b5fcd535de is where the asymmetry began: it moved this mount from `cert_host`
to `bundle_host` for a good reason -- SSL_CERT_FILE and REQUESTS_CA_BUNDLE
replace the trust store rather than extend it -- and left the passthrough on
`cert_host`. So the bundle stays; what was missing is the line that carries it
in, at the same absolute path the nested daemon asks for.
The test that comes with it reads both templates as text and requires every
CA_TRUST path synapse binds to be passed through at that same path. Rendering
would need a full Ansible context, and the invariant is about what the two
files say to each other rather than about the values. It fails against HEAD,
where `bundle_host` is bound and only `cert_host` is passed. It also fails
loudly if the regex ever stops matching how the extra arguments are built,
rather than quietly guarding nothing.
The cause is proven to the byte; the effect is not observed. `docker create`
never succeeded, so a second synapse defect behind this one would still be
invisible. A local `make compose-deploy apps=web-app-matrix mode=update` with
TLS_MODE=self_signed, grepping the nested journal for "Started
matrix-synapse.service", settles both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rows are indexed by `(name, variant)` and a variant is an int. A token that
names only a role carries no variant, so the lookup ran with `None` and could
never hit, and every such token was reported as matching no discovered row.
That is the form `collapse_to_roles` produces, so it is the form a retriggered
priority line arrives in. Run 34348821577 shows the result: all twenty-one
priority entries carried
priority: '<role>' matches no discovered row
(no row in this run's mode and lifecycle envelope)
while baserow, checkmk, decidim and the rest went on to deploy fine. The
warning was not wrong about one entry and right about twenty; it was blind. A
role that genuinely has no row in the envelope looked exactly like a role with a
full set, which is how priority work goes missing without anyone being able to
see which entry did.
A token that narrows nothing beyond the role name is now answered by existence:
it asks for the role in whatever axes the rotation picks, so any discovered row
satisfies it. `Pin.pinned` already draws that line - "whether the token narrows
anything at all beyond the role name" - and pinned tokens keep the old path,
`check_pins` included.
Three tests come with it. There was already one for a bare role that matches
nothing, and none for a bare role that matches something, which is why the
lookup could stay broken. One of the new ones uses variant 4 so a future
implementation cannot pass by quietly guessing variant 0. Checked against HEAD:
the same input returns the spurious warning there, so the tests fail without
this change.
This restores the signal, not the lost jobs. The other silent path, `priority`
being truncated to the job budget in cli/meta/ci/chunks.py, is untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three waits in the logout flow were blind sleeps scaled by the onion multiplier.
A timeout is an upper bound and shrinks to nothing when the answer arrives early,
but page.waitForTimeout always burns its full duration, so multiplying it by five
spends the time whether the target is slow or not. The settle before the logout
sweep cost three seconds on clearnet and fifteen on an onion, every persona, every
role; the dropdown settle cost that again per menu trigger tried; and the poll
interval of waitForAnyLogoutCandidate was multiplied too, which made detection
five times coarser exactly where the target is slowest.
So the sleeps become waitForLogoutControl, a bounded poll that keeps the scaled
deadline and drops the scaling from the interval. It returns the moment a
logout-shaped element is visible, which is the condition the click that follows
needs anyway. Worst case is unchanged: with nothing to find it runs the deadline
out, exactly what the sleep did unconditionally. The first of the two
waitForAnyLogoutCandidate calls around the settle went with it; its result was
discarded, and the second still carries the thirty second deadline that covers a
navbar arriving late.
Four unit tests pin the contract, since the argument for the change is entirely
about when it stops waiting: it returns without sleeping when the control is
already there, returns as soon as one appears, costs at most the deadline when
none ever does, and polls on a fixed 250ms interval no multiplier touches.
Measured context, from run 34348821577: the same dashboard logout test takes
161.0 to 164.0s over Tor and 34.1 to 34.8s on clearnet across five jobs spanning
two distros, both deploy modes and three filesystems. A 1.9% spread across five
runner VMs is not network latency, it is a constant being multiplied, and the
ratio 4.73 is the multiplier itself. Solving the pair puts 32.0s of that test
inside scaled waiting against 2.3s of work. This commit removes the part of that
which is a sleep; the larger part is timeout caps on waits that expire and are
swallowed by a catch, which is a separate defect and stays for now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
start_interval is the cadence while start_period runs, and only while it runs.
Once the window closes the probe falls back to interval, so a service that
becomes ready just after it pays a whole tick anyway and the cadence never
reached it. Six of the seven flavors were shaped that way, Curl worst: a thirty
second window against a one minute interval.
Measured across 151 converge waits in four job logs, 63 sit exactly on a tick
boundary, 36 in the thirty second step and 27 in the sixty second one.
Attributed by application, web-app-seaweedfs is the largest at 1501s over
sixteen waits with a median of 57.7s: it probes with curl and fell straight into
the gap. web-app-zammad probes with connect, whose window is fifteen minutes,
and its 154s median is real boot time with nothing to recover.
So the window moves to ten minutes and the steady state interval to two, and the
flavor overrides that sat below them go: Curl, Wget and Connect lose interval
1m, Http its twenty second window, Nc its ten. Connect keeps fifteen minutes.
Neither number costs anything while a container works. start_period delays
nothing, it only stops failures counting, and interval no longer touches
convergence at all since that hangs on the window alone; it is steady state
load, which drops from 120 probes an hour per service to 30.
Forty six role files follow. Twenty seven declarations go that repeat the new
default or undercut it, an undercut window being the defect itself. Eight
windows longer than ten minutes keep their value and now say why. The literal
blocks and the three Dockerfiles this repository builds spell every timing out,
because outside the lookup an omission takes docker's value and not the
platform's.
A fourth lint rule keeps the shape: start_period must outlast one interval,
checked against the effective values, so raising interval without the window
cannot quietly bring the cliff back.
Two faults in the lint surfaced while building it and are fixed here. It
compared timings as strings, so 600s and 10m read as different values, and it
matched the healthcheck key exactly, so a line carrying a trailing nocheck was
skipped whole, which is how the two swarm-init partials went unexamined.
One regression, bounded. A container that starts and never turns healthy is
neither rejected nor errored, so stack_ready.sh only calls it fatal after three
killed tasks, and each cycle now spans the longer window. That reaches twenty
minutes rather than six, capped there by swarm_stack_converge_retries at 600
with a two second delay, and it lands only on runs that were failing anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LITELLM_BACKEND_ROLES read lookup('deployment').groups, which deployment.py
fills from group_names. That variable is host-scoped: it lists only the groups
the host currently being processed belongs to.
On a swarm round the manager carries svc-ai-ollama, svc-ai-lmstudio and
svc-ai-litellm while the workers carry three groups and none of them is an AI
backend. Both branches of LITELLM_CHAT_MODEL therefore missed on a worker and,
with OPENROUTER_API_KEY empty, the model resolved to the empty string.
Only one task made that visible. Every stateful task in the gateway include
carries delegate_to STACK_HOST plus run_once, so it templates in the manager's
scope and wrote the correct model into the database. The assert carries
neither, so it re-templated in each worker's own scope and failed there while
the manager passed.
The manager-only membership is not a placement decision. The swarm test
inventory is provisioned with a single --host MGR, so every role of the round's
closure gets that one host, and extend_inventory then widens only
svc-swarm-node, svc-storage-nfs-client, svc-storage-nfs-server and the round's
own application. The three AI roles declare no placement at all; they are
manager-only because nothing widened them.
Add an inventory selector next to the host-scoped one, built from the play-wide
groups magic var, skipping groups without hosts and Ansible's implicit all and
ungrouped, sorted so the cache key is stable. The key takes it too, or two
hosts with equal group_names but different inventories would share an entry.
The blast radius is two readers, both in 16_ai.yml, and both ask only whether
svc-ai-ollama or svc-ai-lmstudio is present. On the manager the new value gains
exactly svc-storage-nfs-server and svc-bkp-nfs-2-local, neither of which either
reader tests, so the manager's answer does not move and only the workers change,
to match it.
This is the third step on the same line. 46a16127fc set .running; b149ad0d22
corrected that to .groups because a targeted apps= round leaves only the
command-line roles in the whitelist. .groups solves the targeted round and not
the multi-host one, because group_names stays host-scoped either way.
Measured in a two-host play with a real group_vars file and the real lookup:
the worker resolves ollama false through group_names and true through the new
selector, the manager is true both ways, and the assert that failed in CI
passes on both hosts.
Requirement 036 named the host-scoped backend selection as the thing that has
to change and deferred it. That residual is closed, so the paragraph now
describes the selector in use and why the two rejected ones are wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The account loop walked every user any role declares, 291 of them in a CI
round, and let each task discard the rest through its own when:. That still
costs a task execution per user per task: 1706 of 1746 executions reported
skipping, at 0.146-0.157s each, for 259.5s of a single job.
A role-local mailu_users lookup returns the subset instead, in the dict
shape lookup('users') yields so the call site keeps dict2items. The
predicate is the union of both gates the tasks apply: a mailbox account, or
a bot role, whose API token Mailu issues. Filtering on the mailbox alone
would quietly stop issuing tokens to a bot without one.
The per-task when: conditions stay. The returned set is a superset of what
each of them selects, so it decides how often they are evaluated, never
their outcome.
A non-list accounts declaration raises rather than being filtered away, the
way reserved_users.py already treats it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_ram_gb read mem_total_mb while mem_available_mb sat next to it, added by
3e9deab6f7 for the drill abort and never carried over here. The ceiling exists
to stop the sidecar from competing with what already runs, so measuring the
machine instead of what is free made the term inert: on a 16 GB runner it
computed five workers while 5.5 GB was actually available, and the CI cap was
the only thing that ever bound.
Reading MemAvailable makes the guard real and self-limiting, which matters now
that the onion case raises the count. Against the memory the run itself reports:
PASS 1, 9113M free clearnet 1 onion 2
PASS 2, 5550M free clearnet 1 onion 1
Also withdraws a CI CPU divisor I had added on a false premise. It cited the
matrix telemetry stall60 as evidence that reserving three cores of four buys
nothing on a CI runner - but mem_stall_pct reads /proc/pressure/memory
(utils/env/runtime.py:84), so that figure proves memory headroom and says
nothing about CPU. Raising the clearnet count needs CPU evidence this run does
not carry, and a contention flake would cost coverage, so the divisor stays at
four until something measures the CPU side.
The onion lift keeps its own justification, which never rested on that number:
a browser waiting on Tor circuits does not consume CPU, measured at 4817s over
Tor against 911s on clearnet for the same specs on the same cores.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sidecar is the dominant cost of an onion round and almost free without it.
Measured on one Nextcloud variant at equal mode and filesystem, differing only
in the onion axis, sync pass:
test-e2e-playwright 4817s over Tor 911s on clearnet +5.3x
sys-svc-compose 2294s 2174s +120s
everything else < +25s
The whole Tor surcharge sits in one role. Tor burns no browser CPU, so a 5.3x
wall clock against identical specs on identical cores is circuit latency: the
phase waits rather than computes. The CPU divisor that sizes the clearnet count
does not describe it, and on a 4-core runner it yields exactly one worker,
which leaves the CI cap of 2 unreachable.
compute_workers takes an onion flag and, when set, multiplies the count and
caps it separately. The CI ceiling is overridden rather than respected: it
describes CPU contention, which Tor does not create. ram_workers keeps its
veto, because circuits are cheap and browsers are not.
The lookup reads the transport itself, from the same source the role derives
its own flag from, lookup(tls, application_id, url.base) tested for .onion,
following the _swarm_prefix idiom in plugins/lookup/image.py: read from
variables, render through the templar, fall back to the safe answer. One truth
about the onion state, and a caller asking for a worker count cannot forget to
pass it.
Factor 2 and cap 4 rather than more: each worker holds its own circuits and the
Tor daemon is the next bound, which no measurement here reaches. Expect the
sidecar near 2.6ks per pass instead of 4.8ks, roughly 4.4ks per onion job
against a 20.4ks budget. It does not help a clearnet round, where the sidecar
costs 911s to begin with.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SSL_CERT_FILE, CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE replace the trust store;
only NODE_EXTRA_CA_CERTS extends it. compose_ca.py pointed all four at the
deployment own root CA, so every OpenSSL and rustls client in every container
was blind to every public certificate. Measured here, one variable, same URL:
unset curl https://registry.npmjs.org/ -> 200
a single unrelated CA -> 000 unable to get local issuer certificate
that CA + the system bundle -> 200
PeerTube was the one process that hit it: its plugin install fetches from
registry.npmjs.org at runtime through a client that honours SSL_CERT_FILE, and
it failed identically in compose and in swarm with
invalid peer certificate: UnknownIssuer
Everything else reaches its artefacts at image build time, or through Node,
whose NODE_EXTRA_CA_CERTS is additive and therefore kept working.
The replacement was never a posture. e7e8cf8c03 calls these variables an
"env-based CA trust fallback", f1ea036bd2 adds them "instead of falling back to
a plain docker run without CA trust", e79183f96a wires the extra CA in "so npm
traffic through the cache validates". Every commit framed them as adding trust.
sys-ca-selfsigned now builds ca-bundle.crt beside the root CA on every node,
from that node own files: the stack host mints the CA and workers fetch it, so
there is no cross-node dependency. The bundle is the root CA concatenated with
the first present CA_BUNDLE_CANDIDATES entry, the SPOT fc0881ff33 introduced.
The cat argument list is assembled in Jinja, so a host with no public bundle
emits the root CA alone, which is exactly the previous behaviour, rather than an
empty argument that would fail. bundle_host joins the existing stray-directory
sweep and the build runs after it.
compose_ca.py uses the bundle only when the file exists and is non-empty, so a
host that has not built one keeps the previous behaviour instead of binding a
missing path that Docker would materialise as a directory. The php ini, msmtprc
and the Matrix flavour may name the container path with no fallback because the
bundle is written at task 99 and the php ini at task 123: no host can carry the
ini without the bundle.
CA_TRUST_CERT and NODE_EXTRA_CA_CERTS stay on the single certificate. keytool,
occ security:certificates:import and update-ca-certificates each take one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two of them lived inside docs/ next to the pages they render, which is why both had grown a __file__ upward walk to find the repository root, a sys.path insert to import from utils, and a nocheck for each. A generator parked among documents reads as documentation and drops out of the checks that apply to code.
Moving them surfaced six real violations that the old location had hidden: yaml.safe_load instead of the shared cache, two uncached read_text calls, five hard-coded role paths instead of ROLE_FILE_META_SERVICES and ROLE_FILE_META_MCP, two make targets without an alias, and a requirement reference in code. All six are fixed at the root, none with a suppression, and both generators lose their project-root exemptions because a package supplies PROJECT_ROOT.
cli/build/readme joins them so the three generators that write into docs/ sit together. A new lint keeps docs/ to .md, .html, .rst and .nocheck; it enumerates through git ls-files so untracked __pycache__ is not its business. Checked against HEAD, it names exactly the three .py files that moved.
026-unified-addon-syntax stated that the matrix generator stays under docs/requirements/. The lint makes that impossible, so the requirement now records where the generator actually lives.
Verified: every one of the twelve modules under the new tree imports, no old module path survives anywhere in the repository, 482 lint tests and 1246 cli unit tests pass, and both generators run without drift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
da729cecec reads .Options.device when a volume declares a bind, on the premise that only compose volumes carry no .Options. A swarm node can hold the volume object without them anyway: in run 33923190220 the DR drill probe reported 'exposes no backing store for kix_kix_config' on the node that held it. The script then falls back to .Mountpoint, the node-local _data that is empty while the stack is down, finds no config there and exits 0.
Pass the inventory's own path as PATCH_DATA_DIR, the same DIR_VAR_LIB/<volume> that compose_volumes.py bakes into driver_opts.device, and try it when the volume-derived path holds nothing.
The second half is the asymmetry that hid this: a config present but unmatched fails loudly since 2c4e0e18b1, while a config found nowhere passed silently. The silent case is the dangerous one. KIX kept the pre-rotation password, its backend crash-looped on exit 255, and the converge gate only gave up 44 retries later. A total miss now prints SKIPPED with both paths it tried, and stays non-fatal so a first install still works.
The suite covered three of the seven consuming roles; it now covers all seven, plus the fallback and the named miss.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
VaultHandler.encrypt_string started one ansible-vault process for every single credential. The swarm matrix' rotation gate between PASS 1 and PASS 2 drives 113 application subprocesses over 855 credential keys, so the step cost 1078s in run 33923190220 while the green baseline never ran it at all. Measured per value: 226ms via the CLI, 3.3ms via VaultLib, so 855 keys drop from 193s to 2.8s.
The snippet layout is a contract: encrypt_leaves splits the returned lines and strips the body indent, and existing inventories were written by the CLI. Verified byte-compatible (same 'name: !vault |' header, same 10-space body indent, same $ANSIBLE_VAULT;1.1;AES256 marker), and ansible-vault decrypt reads back what the new path writes.
Reading the password file delegates to ansible's own FileVaultSecret rather than approximating it: ansible strips, then decrypts a vaulted password file, then rejects an empty secret. A hand-rolled .strip() covered only the first of those. _vault stays lazy because callers construct VaultHandler with a password file that need not exist yet.
The tests replace two subprocess.run mocks that asserted against invented output. They now pin the layout encrypt_leaves depends on, and pin the guarantee from 052cea02e2 (values starting with a dash), which landed without a test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mcp/vhost.conf.j2 wrapped its whole body in `{% if not mcp_acl_gated %}`.
That skipped the serve block, which was the intent, but it also skipped the
withdrawal loop underneath it. An ACL-gated vhost therefore rendered no
`location` for its MCP paths at all, so requests fell through to the
application's generic location and the endpoint was reachable behind
nothing but the SSO proxy ACL.
Folding the gate into the `mcp_served` expression keeps the serve block
gated while letting the withdrawal loop run in every case. With an empty
`mcp_served` both declared paths differ from it, so both are withdrawn with
`return 404`.
The behaviour is latent today: only moodle and wordpress declare
`exposure: public`, and both carry `flavor: oidc`, so `is_proxy_gated` is
false and the branch is never taken. This closes the hole before a role
combines public exposure with an oauth2 proxy flavor.
test_mcp_vhost_exposure.py could not reach the branch: its lookup stub
returned "" for every non-config kind, so mcp_acl_gated was always falsy.
It now resolves is_proxy_gated and oauth2_acl, and gained the bool filter
that the internal-exposure cases never needed because `mcp.exposure ==
'public'` short-circuits before it. Rendering the pre-fix template under
gating yields zero withdrawals; the new one yields two.
The base image postgis/postgis:17-3.5 is Debian 11, whose archive was decommissioned on 2026-08-31. The pgvector layer pulled 174 MB of build toolchain from deb.debian.org, and those pool files are now purged while the index still advertises them: linux-libc-dev 404'd on 44 of 44 attempts across four Fastly POPs. Since 2026-09-07 the suite's InRelease is also expired, so apt-get update itself exits 100.
PGDG ships postgresql-17-pgvector prebuilt and its bullseye-pgdg suite is alive, so reading only the pgdg.list the image writes itself removes deb.debian.org from the build entirely. Verified on the exact mirrored base image: the package installs, vector.so lands, zero bytes fetched from deb.debian.org. Signature verification is untouched - no Acquire::Check-Valid-Until.
Side effect worth naming: the old recipe cloned pgvector.git with no tag or ref, so every rebuild baked unpinned master into the central RDBMS. It is now pinned at 0.8.6-1.pgdg11+1. The base image is still EOL and its migration remains open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway publishes only models it has a backend for and renders model_list: [] when neither ollama nor lmstudio is deployed and no OpenRouter key is set. LITELLM_CHAT_MODEL fell back to openrouter/auto without checking that key, so consumers were configured for a model the gateway does not carry and every prompt failed with a 400.
Observed in the Mattermost#1 CI job: litellm.log carries 12 occurrences of "Invalid model name passed in model=openrouter/auto" next to 400 Bad Request, and tests/test-litellm-agent.js failed on every attempt while the rest of the suite passed.
Make the fallback conditional on the key so the name is empty when nothing is served, promote LITELLM_CHAT_MODEL_SERVED from the role vars to group_vars where consumers can see it (it was the third copy of this logic and the only correct one), and switch the 14 consumers from "litellm is deployed" to "litellm serves a model". The Playwright service flag reads the same lookup, so the spec skips together with the Ansible gate. The bool filter carries weight: the folded scalar yields the string False, which Jinja treats as truthy.
A new unit test renders both expressions against Jinja and pins the invariant that a model name exists exactly when a model is served, including the case where ollama preloads embedding models only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The front proxy hardcoded [200, 301, 302] and escalated to a full OpenResty redeploy on anything else. Apps that answer 403, 404 or 405 by design tripped it on every run: 19 escalations in one Nextcloud job at a measured median of 40.5s each.
roles/<app>/meta/server.yml already declares status_codes per canonical key, and sys-ctl-hlth-webserver already reads it. Move the resolution into utils/roles/applications/status_codes.py so both consumers share one definition of DEFAULT_OK, and expose it to templates as the accepted_status_codes filter.
The proxy gate takes the union of the default and the declaration rather than replacing it. Most declarations are narrower than the triple, so replacing would escalate vhosts that answer perfectly well. Contract checking stays with sys-ctl-hlth-webserver, which keeps replacement semantics.
_normalize_codes and _valid_http_code became dead in the health filter and are removed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The step banner announced a start and nothing else, so a phase duration had to be read off the next banner, and a phase with no output of its own was invisible. In the budget-killed Nextcloud#0 of run 33828309745 the credential rotation spent 17 minutes with nothing but the 30-second sampler between its banner and the next one, and reconstructing the split between provisioning, both passes, the DR drill and the rotation meant subtracting timestamps by hand.
The watch loop moves unchanged into _run_watched. _run now brackets it and prints the elapsed time and the exit code, so this holds for every phase, not only the rotation.
The test stubs _run_watched and pins the three things that matter: the exit code passes through, the opening banner is printed, and the closing line carries duration and rc. It is the first test over this module, whose absence was named earlier as the reason a wrong change to the pass sequence would turn nothing red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_no_uncached_file_reads flagged SCRIPT.read_text() in the extraction test. The suite shares one decoded read per file through utils.cache.files, and a bare pathlib read opts out of that.
The full gate surfaced it; the targeted run that accompanied a8469e064 executed the test in isolation and could not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ruff wrapped the set comprehension in the jellyfin auth extraction test at the 88 column boundary. No assertion changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keycloak persists only attributes the declarative user profile declares, and unmanagedAttributePolicy defaults to DISABLED in 26.x. ensure_realm_user.sh:81 writes attributes.uidNumber and kcadm returns rc 0, but the value never lands, so the mattermost-id mapper (web-app-mattermost/tasks/02_keycloak.yml:118) has nothing to emit, userinfo answers 200 without an id claim, and the Mattermost GitLab provider rejects the callback with "Could not parse auth data out of gitlab user object ... user id can be 0". Run 33828309745 fails three Mattermost jobs on it, in compose and swarm alike, and all three Playwright specs are that one failure seen at three downstream points. b0ca8784e2 added the mirroring for exactly this failure and it could not take. LDAP realms are unaffected: the federation writes the attribute, and 08_users/realm.yml runs only when LDAP is off.
The profile push must now precede the seeding that depends on it, so it is synchronous. It measured 0.9 to 1.5s, and its reap sat after the seeding, so with ASYNC_ENABLED true the write would still have hit the old profile. Its register drops out of the reap list with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
configure-auth.sh pins DeviceId="infinito-deploy" in CLIENT_HDR, and realign_admin_password authenticated twice with it: once for the access token, once for the user id. Jellyfin replaces the session bound to a device id when a new one is created, so the first token was already revoked when POST /Users/{id}/Password presented it. Run 33828309745 job 100890535194 shows the result as fifteen times "curl: (22) The requested URL returned error: 401", with the deploy otherwise complete (ok=2830 failed=1). get_user_id arrived with 1bf28bb2bf and carried the second call in from the start.
One AuthenticateByName response now yields both fields, read through token_of and user_id_of so neither expression is spelled twice.
Redeeming the reset pin clears the stored password: in that job forty authentications with the deployed password were denied, and the next one, with the pin string as the password, succeeded. Every exit after the redemption therefore states that the administrator is currently without one, and the two silent returns around ForgotPassword now log why they gave up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reachability check called showmount, which speaks the MOUNT protocol. The ganesha export sets NFS_CORE_PARAM Protocols = 4 and Enable_NLM = false, and every EXPORT block repeats Protocols = 4, so MOUNT is never registered and showmount cannot succeed against a healthy server. The repo already records this elsewhere: scripts/tests/deploy/swarm/utils/collect/diagnostics.sh:125 prints "showmount -e failed; NFSv4-only servers do not answer MOUNT".
Every failure fed status, and a nonzero exit reaches OnFailure, which starts sys-ctl-cln-anon-volumes (dockreap --no-confirmation). So on any host holding both anonymous and NFS volumes the check has been failing on every run and reaping volumes for a reason that was never true. Probing tcp/2049 is what the role does for its own readiness gate in files/nfs-ganesha-restart.conf:9, and it is correct for the kernel flavor too.
The two tests drive a whitelisted anonymous volume so the anonymous section contributes nothing and the exit status reflects only the NFS check. One asserts a listening port passes, the other that a refused one fails; the first is the regression guard, the second keeps the check from being one that always fails.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four probes returned the number of failures as the process exit code. An exit status is 8 bits wide, so exactly 256 or 512 failures arrive as 0 and the probe reports success. sys-service installs these as Type=oneshot units (roles/sys-service/vars/main.yml:28) and sys-service-terminator starts them with until: result is not failed, so that exit code is the whole health verdict.
The count is not lost: each of the four prints it to stdout a line earlier. Only the exit status stops carrying it, because it cannot.
This makes no CI job green and is not meant to. For any count other than 256 or 512 nothing changes; at those two it turns a false success into an honest failure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The retrigger turned a suite off only when the source run had been
dispatched with that suite explicitly on. b1ca39e9dd introduced that
guard together with the feature itself, and it contradicts the same
commit's message and docstring, both of which state that the job record
is the only evidence.
The guard made the feature a no-op in its most common case. An input
left on its default renders no title segment and resolves to empty, so
config.get('workspace') was never 'true' for a run that reached the
suite on the 'auto' default. call-orchestrator.yml reads 'auto' as
global scope off the whitelist alone, and a --failed retrigger puts its
selection on the priority line while leaving the whitelist empty, so
every retrigger in the chain redeployed a suite that had already gone
green. Measured on run 33785198875: six workspace jobs green, and the
dispatch still went out carrying workspace=auto.
Drop the guard. suite_passed already refuses to retire anything that
failed, was cancelled, timed out, was skipped or never ran at all, so
the job record decides on its own.
Three tests cover the paths 8600f1d9aa left open, all of which pinned
the explicit 'true': green on 'auto', green with no recorded input, and
red on 'auto' staying on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Line-length reflow in three test modules and a stray blank line in zammad variants. In router.yml the formatter also moved the nsswitch lineinfile regex from single to double quotes with the backslashes doubled to match; single-quoted YAML already passes a backslash through literally, so the string reaching lineinfile is byte-identical and the rule still changes only NOTFOUND.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two branch-only commits each armed a defect in the compose-up handler
topic, and both failed in run 33785198875.
63f7f962bf registered the topic owner at the top of sys-svc-rdbms'
dedicated.yml so the notify two tasks below stopped aborting with
ERROR_ON_MISSING_HANDLER. The notification then survived to the
unconditional meta: flush_handlers at sys-svc-container/tasks/00_core.yml:53,
reached from sys-svc-compose's own 00_core before that role installs
BIN_COMPOSE, and long before 02_stack_host_prep installs BIN_CA_INJECT.
Matrix#1 died there on ca-inject with rc 2 after seven retries. That play
carries exactly one compose notification (job 100756596907, log line
11046), so closing this window is sufficient, not merely necessary.
dedicated.yml now enters 00_core.yml under the role's run-once flag
before it notifies, so the flush drains an empty queue and the flag then
makes main.yml skip the body on the real entry. The registration itself
stays ungated: gating it would skip the owner from the second app on.
95186163aa hoisted the sys-svc-compose include to sys-stk-backend
position 1. The consumer therefore rendered and notified compose-up
before sys-svc-objstore lazily loaded the provider app role, whose own
unconditional flush consumed the notification and ran the single
registered handler in the provider's chdir. mastodon and peertube never
got a stack, in compose and in swarm alike; the victim's own guard task
appears zero times in all four logs. Restoring the include below rdbms,
objstore and engine puts the notify after every foreign crossing. Two
pre-hoist runs in exactly that order are green on record.
The test keeps its cases and gains one. It selects the render include by
the absence of tasks_from instead of by first occurrence, so an early
registration-only include stays legal while a hoisted render still fails.
The new case pins the bootstrap: entry point, position before the first
notify, and the run-once guard whose removal would let 01_reset.yml tear
down DIR_COMPOSITIONS once per dedicated-DB app.
Same class, pre-existing, not fixed here: svc-db-qdrant notifies and
flushes three tasks before its first compose entry; web-app-magento pairs
a dedicated database with a shared object store; web-app-seaweedfs still
flushes a topic it does not own; svc-runner enters 00_core.yml unguarded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The hub serves 94 tools across 23 categories, write tools included, and enabling the ExApp switches every category on. That surface was published on the public vhost and every discovered client was handed an app password for it. svc-ai-mcp-adapter now sits in front in mcp_passthrough mode: clients present credentials.mcp_bearer to contextagentmcp and reach the nine read tools of calendar, contacts and files, which files/mcp/tools.json names and the sidecar enforces per call. Those three categories are the ones whose upstream is_available is unconditional, so the served set does not vary with which optional apps a deployment installs.
Enforcing it in the sidecar rather than in each client is what makes it hold: web-app-openwebui renders no per-tool filter at all, so a client-side include list would have bounded two consumers out of three.
06c13e7f3e withdrew the declared path from the edge for exactly this reason - "that surface is what the adapter exists to narrow". Nextcloud answers the same route under a second spelling, without the /index.php front controller, which try_files rewrites back onto the route after the edge has already declined to block it. The withdrawal becomes a regex covering both, and only where stripping the front controller changes the path, so the four adapter providers without one render unchanged.
The app password is minted after the stack renders, so it reaches the sidecar through upstream.env and rebuild.yml rather than the compose environment: compose up resolves env_file while creating the container, a restart does not. The sidecar is the last service in the template because a Jinja set has no block scope and its env_file would otherwise displace the one talk, whiteboard and talk_recording each carry.
Two suites cover what nothing measured: the vhost template rendered without a container, asserting both spellings are withdrawn, and every shipped tools.json loaded through the adapter own loader, so a bad contract fails in a second instead of as a probe timing out an hour into a deploy. The guest spec probed mcp.endpoint.path, which this change makes the proxy answer 404 for its own reasons; it now probes all three public paths and requires no MCP body.
Not yet deployed: the sidecar build, the upstream.env delivery and the hub answering exactly nine tools are unverified against a running stack.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule lived only in suite_passed's body, and it inverts easily on a
casual read: ask for the suite again when it failed. Under that reading
the retrigger aliases would drop the suite in their most common case,
because they force-cancel the source run before reading it, so a shard
that was still running comes back cancelled rather than red.
Only an all-green suite is retired. Failed, cancelled, skipped, timed
out, still running and never dispatched all keep it on the retrigger,
and each of those now has a test. The two docstrings enumerate them, and
so does the i8ciall description, which is where an operator reads it.
dedicated.yml notifies compose-up but registers no handler for it, so a
caller that reaches sys-svc-rdbms before any compose consumer has loaded
the sys-svc-compose block dies with "The requested handler 'compose-up'
was not found" and no PLAY RECAP. 95186163aa guarded only the
sys-stk-backend caller; web-app-matrix's ansible flavor includes
sys-svc-rdbms directly in 03_database.yml and still hit it as the first
compose consumer of its play (run 33751742070, job 100643400267).
Register the owner as the first task of the notifying file. In the
sys-stk-backend path this lands before the pre-flush rebind, so the app's
copy stays the last registration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
65c72fc374 settled the convention: import unittest, and reach the submodule as import unittest.mock as mock, so a call site has one spelling of the name. test_onion.py and test_tor_socks_proxy.py still carried from unittest import mock; CodeQL reports them as py/import-and-import-from, the last two in the repository.
Same one-line change per file the sweep applied; mock.<attr> call sites untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible resolves a notification to the last handler block registered
under that name and templates application_id in that block's context.
Since 95186163aa sys-svc-compose is loaded first, so the provider roles
included after it (svc-db-postgres via sys-svc-rdbms, objstore, engine)
leave their own copy as the last one, and the trailing flush deployed
the dedicated postgres instead of the app. Run 33751742070: six jobs
waited on a keycloak or mattermost container that was never created.
A dynamic include_role directly before the flush makes the app's copy
the last registration again. import_role would not do: it expands when
main.yml is parsed, before the providers run. The first include stays,
so the missing-handler error stays fixed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Hermes and OpenClaw wrapped the tool allowlist in a guard that dropped the key when the callable set was empty. Both upstreams read an absent filter as "register every tool the server serves", so a provider resolving to no tools got the widest surface instead of none. Nextcloud keeps its contract in tools.categories with no allowlist and admits both clients, so it was the live case.
The templates now render the filter unconditionally, the way the n8n client node already pins include to selected; the two tests that asserted the omission are inverted. mcp_servers additionally rejects such a provider with the fatal code tools_unenforceable: a client filters by tool name, so a runtime-composed surface reaches it as no surface at all, and a server the agent can never call is worse wired in than refused with a reason.
A deployment pairing Nextcloud with either agent now aborts naming the provider, where it previously handed the agent every Context Agent tool unfiltered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The regular line restarted at the head on every retrigger. resume_offset
compared the source run's deploy tokens against a freshly computed
ranking, but every axis rotates with the sweep number and a retrigger is
a new run with a new one: measured against the current 277-row ranking,
2 of 277 tokens matched. The walk stopped on its first row every time,
and the 66 rows past the job budget never deployed at all.
Membership now compares role#variant, the identity the rotation cannot
move, and a row counts as proven only when every job it had was green.
A priority row deploys its whole cross-product, so one role#variant can
carry several verdicts, and a green compose job must not carry a red
swarm one past the line. The offset also inherits the source run's own
offset as its floor rather than falling back to the head.
The priority line drops --include-unrun with it: only the hard failures
and the priority entries the run never deployed belong there. The rows
it never reached are the regular line's turn now that the offset walks.
sys-svc-rdbms notifies compose-up from dedicated.yml, but sys-svc-compose, the only role that owns that topic, was included after it. With ERROR_ON_MISSING_HANDLER on, the strategy raises before the task result is emitted, so the play dies with '[ERROR]: The requested handler compose-up was not found' and no PLAY RECAP. Rounds that happened to deploy another compose consumer first masked it; a round that deploys the app alone does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d14354e892 added both probes reading only /etc/dnsmasq.conf, but svc-net-tor writes its onion zone to /etc/dnsmasq.d/tor-onion.conf and leaves the main file with nothing but a conf-dir pointer. Every Tor host of run 33698844864 therefore captured resolve-inzone.txt as '[no output, exit 2]' and all 471 sampler lines as inzone=unconfigured: during the resolver outage that ended Gitea#0, nobody ever asked dnsmasq for an onion name, and two investigations went without the one measurement that would have settled them.
Both probes now follow the conf-dir pointer, main file first so a lab node with its zone in the main config answers as before. Each is exercised by a test that runs it against real files rather than comparing its text, which the rescue constant had no test for at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils/roles/applications/config.py turns a missing key with a None default into False, so the provider-side probe in attachments.py read an omitted mcp_consumer flag as an explicit refusal. No MCP client ever joined a provider's shared_net: open-webui rendered no gitlab network, http://gitlabmcp:8080/mcp did not resolve from its namespace, and Open WebUI's verify call answered 400 in both GitLab#1 jobs of run 33698844864, clearnet and tor alike. Measured on that run's own inventory: without the change open-webui does not join the gitlab overlay, with it it does, and a sentinel probe shows the provider carries no mcp_consumer key at all.
utils/roles/applications/mcp.py:295 already states the rule this restores, that an omitted flag inherits the client's own declaration and only an explicit false excludes.
The stub in the render tests answered an unset path with the caller's default, which is why the regression passed them; it now coerces the way the accessor does, and the new case fails without the fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker stack deploy --detach=false converges on a running replica. web-svc-cdn's build container and every migration job end Shutdown/Complete by design, so the CLI waited for a state those stacks cannot reach: nine jobs of run 33698844864 burned the runner's 20400s budget inside this one handler, six of them on web-svc-cdn. The gate that follows already knows better, is_completed_oneshot accepts exactly that task shape.
The timeout keyword could not bound it either. The stack host is reached over ansible_connection: docker, whose plugin runs the child inside a Popen context manager, and __exit__ waits the child out before the timeout exception escapes; a probe measured 90s of wall clock against a 6s timeout. The bound now sits in the shell, where it works.
Handing the wait back to the gate would reopen the race 87e4313e11 closed, so the deploy stamps its start and names the services it reported touching, and the gate treats their earlier completed update as still pending. An inspect format the daemon rejects falls back to the old one, which only disables that check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
26 roles resolve a domain in their metadata, so lookup('tls') and lookup('domains') run inside the applications render. Under the re-entry guard those calls receive the unrendered tree, whose service flags are still Jinja strings, and _as_bool reads "{{ 'svc-net-tor' in group_names }}" as false. The resulting map skipped onion injection for every app that does not enable tor literally, and it was cached, so it answered every later read in that process.
Before the applications carrier each worker rendered and poisoned its own cache the same way, which is why the shared Keycloak client registered 5 onion origins then and 65 now. The render state joins the cache key, so the render's own call sites still share one map while a later read builds the finished one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The SSO gate handed X-Forwarded-Access-Token to every oauth2-fronted upstream, so each of the 28 gated roles received a full-scope bearer for the signed-in user while exactly one, n8n, decodes it. The token also grows with the realm's registered origins, and LAM's Apache rejects a header field above 8190 bytes (Moodle#0, run 33689701895).
Consumers now declare services.sso.oauth2.pass_access_token. Without it oauth2-proxy already answers the auth request without the token, and the gate blanks the upstream header rather than dropping the directive, because nginx inherits proxy_set_header as an array and an unset header would let a client-supplied one through. The new lint keeps declaration and use in step in both directions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lookup('redirect_uris') walked every app in the domains map whose services.sso.enabled resolves truthy, which every role spells as "'web-app-keycloak' in group_names". A round deploying 21 apps therefore registered 69 redirect URIs and web origins on the shared client, 65 of them onion, and an app that is not running has no reason to be a redirect target. group_names is now required and scopes the result, matching KEYCLOAK_SAML_APPS which already selects on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The web-origins scope mapper listed every registered origin in each access token. openresty forwards that token to SSO-gated upstreams, so on a Tor deployment the claim reached 5457 of the token's 8650 characters and LAM's Apache refused the request with 400 (Moodle#0, run 33689701895).
The realm template carries the change, so a fresh import and every CI round get it directly. The reconcile task exists because 04_realm_import.yml only imports when the realm is missing, which would leave the old mapper in place on running installations.
Keycloak resolves CORS from client.webOrigins in DefaultCors.checkAllowedOrigins whenever the token carries no origins, so enforcement is unchanged; that fallback exists since 26.7 and the role pins 26.7.3. Introspection keeps the claim, and nothing in this repository reads it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
