Both roles mount their model cache as a `type: bind` volume, and swarm
rejects a task whose bind source is missing on the node it lands on. Neither
declares a `placement`, so the scheduler may put the container on any node,
while the directory was created by a single task that reached one host. CI
showed the consequence directly:
s1_s1.1 desired=Ready current=Rejected error="invalid mount config for
type "bind": bind source path does not exist:
/var/cache/infinito/core/cache/models/s1"
with the creating task logging exactly one result line, for the manager.
`nfs_prep.yml` already fans directories out across the cluster but gates on
`driver_opts is defined`, which a bind volume never carries, so these two
volumes were never covered by it.
The host list moves into a `node_hosts` lookup rather than an inline
expression, because two roles already spelled the same idea by hand and a
third and fourth were about to. It answers one question - the hosts a
node-local path must exist on - and keeps the swarm group name in one place
per plugin. The call sites use `query()`, since `lookup()` joins several
results into one string and a loop needs the list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lookup takes the role from its term and falls back to reading
application_id out of the play's variables. That read was raw, which is the
trap that took down three jobs from webserver_resource, so it now goes through
resolve_var like every other lookup.
Prevention rather than a fix. The fallback is the else of a term that the only
caller always passes, and where it would fire web-app-docs sets application_id
literally; the roles carrying a templated id reach it through sys-stk-full,
which web-app-docs does not. The consequence would also have differed: an
unrendered string never matches a peer, so the replay would have asked the
site for a role named after the template and failed on the page rather than in
the lookup.
Measured: with the resolver's body stripped back to the raw read this test and
the webserver_resource one both fail, and both pass with it restored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moving the effective-cpus expression into a lookup changed where the role id
is read from. Inside a Jinja expression the templar resolves a variable
recursively, so a value that is itself a template arrives fully rendered.
Reading it out of the play's variables does not. sys-stk-full sets
application_id from sys_stk_full_application_id, so the lookup asked the
applications dict for a role literally named
{{ sys_stk_full_application_id }}
and three jobs of one run died on it: dsk-gnt-claude, svc-ai-agent-broker and
svc-ai-searxng, every one of them a role that goes through that stack.
The lookup now templates application_id and service_name when they still
carry a template, and returns them untouched otherwise, so the ordinary role
never reaches the templar at all.
Measured: with the resolution removed the new test fails with the run's
message character for character, path_trace and config_path included, and
passes with it restored.
The same raw read survives as a fallback in guide_role, unreachable while its
only caller passes the id as a term.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A container cap had to be an absolute number, and docker refuses one above
the host's core count, so "use this machine" could only be written as a
number that breaks on a smaller machine. cpus now also takes a percentage,
read against the host's raw count: 100% is every CPU and lands exactly on the
ceiling docker accepts, 50% is half of it.
The share is read against that raw count rather than against the fair-share
default, because the two CPUs the resource lookup reserves exist to bound
what a service gets when it declares nothing. A service writing a percentage
overrides that default on purpose, and subtracting the reserve a second time
would count it twice and leave no way to name the whole machine.
A bare cpus: 0 is refused from here on. Docker reads a zero as uncapped
rather than as none, so the value most likely to be typed by mistake was also
the one that removed the limit; the error names the fix. No role declared
one: all 315 cpus values in the tree are plain numbers, the smallest 0.01.
Two lookups carry the resolution so that no caller repeats it or has to know
what a percentage is read against. lookup('cpus', app, service) replaces six
copies of lookup('config', ...) piped through the resolver; those six size
worker pools and thread counts from their own cpus, and left alone a
percentage would have given libretranslate a single thread and handed the
agent broker an uncapped container. lookup('webserver_resource', key)
replaces the two blocks that took the smaller of the proxy's value and the
app's, which were the same eight lines written twice for cpus and for
pids_limit.
resolve_cpus is no longer registered as a jinja filter. Nothing referenced it
from a template once the lookups existed, and keeping it would have left two
public ways to resolve one value.
Measured: 96 tests across the filter, both lookups and the container render
templates, including that a percentage resolves before the smaller of the two
is chosen, and that pids_limit stays an integer because nginx renders it
straight into worker_connections.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The instructions test brings up a machine per role, so it can only ever
replay one guide. Which one it picked was a side effect of ordering: the
lookup returned the first invokable peer the round listed, and on a CI row
that is the role under test only because the row's apps happen to read
<role>,web-app-docs.
It now reads APP_ID, which the deploy workflow already sets to the row's app
and which the swarm routine already documents as the app under test. A round
that carries no APP_ID, or one whose app publishes no Production block, draws
an invokable peer at random instead, so a local round covers a different
guide each time rather than always the first one the inventory lists.
Ansible runs inside the infinito container rather than on the runner, and
that exec passes an explicit allowlist, so the name arrives only where it is
forwarded. Two seams carry it: _FORWARDED_ENV_KEYS, which covers CI and the
local initialize and reinstall verbs, and the update verb's own docker exec,
which assembles its arguments without that CLI.
The pin is tolerant rather than strict. APP_ID is set on every deploy row and
not only on documented ones, so refusing an app that publishes no guide would
let a generic variable abort a deploy.
Measured: the draw is observed over 200 rounds rather than assumed, and the
forwarding is negative-controlled, removing the key and watching the new test
fail on that key alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b64e4a0558 gave web-svc-mirror the shared monitoring include so it would
appear in the scrape set. That include ends in two locations proxying to
the role's own backend, and this role runs no container: it renders a
vhost and nothing else. Every deploy that reached its vhost then died in
resolve_upstream, twenty-one jobs of one CI run, in both the compose and
the swarm phrasing of the same message.
What followed was worse than the original gap. 514a1fe74b guarded the
probe inside healthz.conf.j2, which ninety-six roles render, so a
one-role want became a change to shared infrastructure. Three further
attempts sharpened that guard, none of them fixed the deploy, and the
renders I offered as proof were vacuous: the outer integration guard
short-circuited in the context I tested, so the block I was measuring had
never rendered.
main does not carry the include, cannot reach that code path, and does
not fail. The complaint that started this, that mirror carries no
app="web-svc-mirror" label, does not appear in the run preceding the
include either. It was not a regression and not urgent, and it did not
justify touching a template every monitored vhost depends on.
So the tree returns to main for both files and the lookup written to
support them is deleted. Mirror is unscraped again, which is how it has
always been, and now a decision to take deliberately rather than a defect
invented from a passing remark in a job log.
Measured: both templates are byte-identical to main, nothing references
the deleted lookup, and the ansible lint suite passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The monitoring block a vhost includes ends in two locations that proxy to
the application's own backend, and resolve_upstream raises rather than
returning empty when no port exists to build that upstream from. A role
that renders a vhost without running a container of its own has none, so
including the block aborted its play. b64e4a0558 made web-svc-mirror the
first such role, four months after the assumption was written down.
The probe halves are now conditional. /healthz/live stays: it answers from
openresty, which is what liveness means for a role whose only artifact is
a vhost. /healthz/ready and /_live_probe go, because both proxy to a
backend that is not there. Guarding only the second one left the first
referencing a variable the guard no longer set, which is its own failure
and the reason this took two passes.
The condition asks a lookup rather than rebuilding the rule in the
template. Hand-derived, it tested the local port alone, which is correct
outside swarm and wrong inside it, where the upstream comes from the
internal port; web-app-matrix declares only that one and would have lost
its probe silently. The lookup reads both scopes and the port a role
resolved for itself, which is what resolve_upstream consumes.
No working endpoint is removed. A vhost rendering the block today must
already carry a port, or the lookup would be raising for it as well, so
the condition can only skip what could never have rendered. Not yet
rendered: the deploy that proves the template resolves has not run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The entry carried two fields naming the same host by two routes.
canonical_url came from the tls lookup, which substitutes a role's onion
address on a Tor deployment; canonical_domain came straight from
domains.canonical, which does not. On clearnet the two agree and nobody
notices. Under Tor they part ways, and the field a consumer compares
against a rendered page is the one that stayed clearnet.
Every dashboard tile assertion failed on an onion swarm job for that
reason: the page offered
http://prometheus.toxlaup2sbprl6zklgcsofj7g4xim6uxjd4zjiiqtk3qzffykqwda2qd.onion
and the locator looked for a[href*="prometheus.main.infinito.test"]. The
matomo tracker spec failed the same way, comparing registered site URLs
against clearnet hosts. Deriving the domain from the URL leaves one
source instead of two that only agree off Tor.
The bug shipped with 6fa7bd2f57 in May, when canonical_url was added
beside the existing field without reconciling how each is resolved. It
stayed unreachable on swarm because the dashboard container failed to
start there at all, until 0e4ed57c3c let it start and the tiles rendered.
The stubbed tls in the tests answered with a host that no clearnet deploy
can produce, which is why the old expectations looked right. It now
answers with the role's own domain through the module's resolver, and
takes url_host to model a deployment whose URL host differs.
Consumers are the five playwright specs that read these fields; no task
reads them. Prometheus keeps matching on the role id, so its needle is
unaffected. css and logout carried the same latent failure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ruff reflows the guide_role signature and its two call sites, the same for
three test modules, and the YAML formatter prepends the document marker to
the machine's compose file. These files were created one commit too late to
be caught by the previous formatting pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The replay of a role's "### Production" block moves out of the CI guide
scripts and into the documentation role itself. The block is read off the
rendered page over HTTP, so what runs is what a reader copies, and the role
that publishes the page is also the role that proves it.
scripts/github/guide/{one,compose_deploy,host_deploy}.sh and
cli/build/docs/readme/production.py are gone with the workflow step that
called them. The instructions marker now only adds web-app-docs to the row's
apps; test-e2e-cli discovers the role through its new templates/test.env.j2
and runs files/test/test.sh.
The block runs in a throwaway machine, never on the stack host. compose.yml
starts one service built from files/test/Dockerfile, whose base is
INFINITO_PARENT_IMAGE out of the mounted checkout's .env, so the machine is
the stack host's own distribution and the replay exercises the installation
instead of a prebuilt image. files/test/init.sh adds an init when the base
carries none and keeps the full -Syu that ef0f03ea98 established for arch.
scripts/tests/workspace/base/01_install.sh then prepares the machine before
the block runs.
Nothing is skipped any more. The previous host-install branch exited 0 and
passed silently over 65 of the 178 invokable roles; a block without a
"docker run" is now replayed against systemd in that same machine, with the
git clone dropped in favour of the mounted tree under test.
The target role comes from the new guide_role lookup, which reads
list_invokable_app_ids() rather than matching role-name prefixes, and falls
back to the asking role when the round deploys nothing else. deployment.py
grows running_apps() so both lookups share one definition of what the round
deploys. required_by_coverage.role_is_invokable drops its own walk over
categories.yml and delegates to the same SPOT; measured equal across all 280
roles.
Carried in the same index, because the refactor pass ran over this change
set: library.py splits into sites.py and build_queue.py mixins, builder.py
hands translated_languages to catalogs.py and folds three repeated log-tail
and state writes, and placeholders.py becomes a facade over spans.py,
repairs.py and damage.py. No test changed for those splits, and the two
ref-resolution dicts builder.py carried are replaced by _wanted_ref, which
also stops hashing the deployed tree on every tag build.
Verified: 7132 unit and lint tests pass over the whole tree, the guide_role
lookup renders through a real Templar, the base image resolves from .env,
and compose config validates the service. The replay itself has never been
executed; that needs a deploy of web-app-docs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DEPLOYMENT_MODE was compared against a quoted literal 219 times across 110
files. IS_COMPOSE_MODE and IS_SWARM_MODE say the same thing once, and
test_deployment_mode_booleans keeps it that way: only the file that defines
them may still compare.
Both are real booleans under this ansible, measured before the swap because
the alternative is silent: a variable that templates to the string "False" is
truthy in Jinja, which is why IS_STACK_HOST is written `| bool` at every use.
These need no filter.
What had to learn the names:
* utils/annotations/task_gate.py, which several lints ask whether a task is
compose-only gated;
* four compose-template lints that recognise their gate by matching the
expression;
* the fixtures that stand in for the play's variables. They derive the
booleans from the mode now instead of carrying them, or a test that
overrides DEPLOYMENT_MODE keeps the old pair and asserts the wrong
branch.
Each replacement was checked rather than trusted: the 206 conditions that
changed in place were evaluated old against new through the Templar under
compose, swarm and host, with no disagreement, and no changed line turned out
to be anything but a condition swap. `!= 'swarm'` became `not IS_SWARM_MODE`
rather than the shorter IS_COMPOSE_MODE, because only the literal translation
stays correct should the mode ever be host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sys-svc-compose writes compose.ca.override.yml for exactly lookup('ca_injected', application_id), which since 1254e27c2d also answers true for a service that declares ca_client and only calls the self-signed endpoint. compose_file_args rebuilt that predicate instead of asking it, and its copy still demanded a domain, so for a domainless ca_client the file was written and never reached --compose-file. Swarm takes its file list from that lookup; compose finds the file on disk by itself, which is why only the swarm jobs of run 36064589578 failed while the compose job passed.
The broker's container in that run carries no SSL_CERT_FILE and no CA mount, while openwebui, same job and same node, carries seven CA env vars and four binds. Every outbound call to Keycloak then fails the handshake, which the CLI test reports as 503 and Open WebUI relays as HTTP 400.
Two corrections in ca_injected itself. Its templar.template() call was dead: as_str strips the TrustedAsTemplate tag and ansible-core 2.19+ refuses to render an untrusted string, so the line could not resolve what it advertised, and tls.py and cert.py never template TLS_MODE either. And the ca_client branch short-circuited on the global TLS_MODE, skipping resolve_mode's per-app server.tls.mode override; it now applies only where there is no domain and so no per-app answer to get.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every desk- role becomes dsk-, and the three coding agents move to dsk-gnt- under a category of their own. The category carries a title, hanzi and icon but no invokable flag: an invokable child under an invokable parent is rejected outright, which is also why web itself is not invokable.
Three things a search and replace would have broken without a sound. get_entity_name now strips dsk-gnt-, so the primary entity keys agnt-claude, agnt-cursor and agnt-pi no longer resolved and all three roles dropped out of the CI plan. Eighteen run_once guards carried the old name against a convention its own test enforces. DESKTOP_ROLE_PREFIX in the addon spec lint is a literal, so the desktop exemption it grants would have matched nothing and the lint would have moved to the wrong rule in silence.
docs/requirements/035 is not a historical document but a register a lint holds to every role with an application_id, so its list carries the new ids. In 026 only the links move; the prose stays as a statement about the tree at the time it was written.
Measured against the plan with the rename normalised, the deployed role#variant set is unchanged. A stale id in an inventory is caught before any deploy starts: the pre-flight validator reports Invalid group 'desk-firefox' (not in application_ids).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ca_injected asked three questions, the first being whether the application
has a domain. That decides whether it is SERVED over the self-signed CA. An
application that only CALLS one has no domain of its own, so the predicate
answered false, sys-svc-compose-ca skipped its stack, and the container ran
with an empty trust store.
svc-ai-agent-broker is such a caller: it authorizes every request against
Keycloak group paths over the public HTTPS base. Without the CA every
outbound call died in the TLS handshake, the broker answered 503 "access
check unavailable", Open WebUI turned that into HTTP 400, and the spec
failed on a model it had just listed. Its container was the only one in the
rescue artifact without SSL_CERT_FILE, while litellm and open-webui carried
it.
A service may now declare ca_client: true, and the predicate then answers on
the deployment's TLS mode alone, because the domain question is meaningless
for a client. The three served-side conditions are untouched, so no
application that already received the CA stops receiving it.
The alternative of pointing the broker at an internal Keycloak URL was
rejected: the role declares no keycloak service, so the URL would render
without a route.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_passthrough_swarm quoted and escaped any value containing whitespace,
which contradicts the contract stated at the top of the same module:
docker stack deploy does not strip outer quotes, so the value reaches the
container with them.
Ansible's to_json emits the default ", " separators, so AGENT_PLATFORMS
always carried spaces and arrived as "{\"hermes\": ...}". json.loads then
returned a str rather than a mapping, and sorted(PLATFORMS) in the broker
iterated its characters: the broker advertised one model per character and
never hermes or openclaw, which is why Open WebUI's model list held neither
and the agent-broker specs failed before reaching a completion. The same
encoding mangled AGENT_CONSTRAINT, whose placement expression would have
failed once an agent was reachable.
The branch existed for the opposite consumer: test.env is sourced by bash,
where KEY=one two leaves the variable unset under set -u. Quoting is the
wrong instrument for that, because one filter cannot serve both contracts
keyed on DEPLOYMENT_MODE alone. Both values that join on a space now join
on a comma and their consumers split on one, so no rendered env value needs
quoting to survive the shell.
Two unit tests asserted the quoting while a sibling in the same class
asserted that swarm does not escape a double quote; the pair could not both
hold. They are replaced by one test that renders real JSON and checks the
container receives a mapping. The lint that flags a space-joined value no
longer accepts dotenv_quote as the remedy, since in swarm it no longer
quotes, and its message names the comma instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every stage looped over every role of its group and skipped the ones this host
does not deploy, one task at a time, plus an unconditional handler flush behind
each. Measured on a one-app deploy on 2026-09-23: 132 of 139 entries skipped,
278 task deltas, 41.8 seconds of nothing.
The lookup now takes the `groups` magic variable and drops the entries no host
in the inventory carries, so they never reach the loop.
Which roles get deployed does not change, and that is provable rather than
hoped for: the body deploys a role exactly when application_allowed holds,
which requires the app to be in that host's group_names; if it is, that host is
in groups[app], so groups[app] is non-empty and the lookup keeps the entry. The
filter is therefore strictly weaker than the condition it precedes and can drop
nothing the body would have run.
The filter deliberately reads `groups` and not `group_names`. 8aea5a9b35
measured that a per-host condition above the loop makes ansible insert the
included blocks in host order instead of loop order, after which
svc-storage-nfs-client mounted an export svc-storage-nfs-server had not
deployed yet and the play died on 'mount.nfs4: Connection refused'
(run 31645153977). `groups` is the same mapping on every host, so every host
still walks one identical list, and tasks/utils/setup/role.yml keeps its `when`
exactly where that commit put it. Two of the new cases pin this: a role only
one host carries still enters the loop, and a group_names passed to the lookup
changes nothing.
First written the other way, with group_names in the lookup and the `when`
removed. A single-host deploy cannot show that defect, and it came back green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A deploy refused at ok=771 failed=1 with 'services.litellm.remote_models names provider(s) mock; known providers are openai, anthropic, openrouter'. Declaring the mock among the remote models, which is what makes it an ordinary model rather than a config key of its own, runs into a guard that demands a key slot for every provider. A mock needs none: it answers from its own declared response.
The guard keeps its teeth for real providers and now names the exception and its reason, instead of tolerating it silently.
AI_MOCK_PROVIDER holds the name once. The guard, AI_REMOTE_ALIASES and AI_MOCK_ALIASES read it, and the template hands it to the route builder, which lost its own constant. Two literals spelling the same thing is how the LM Studio branch came to drop the context window its README still advertised; a parameter cannot drift.
Static checking never touched this line. It took a real deploy, 772 steps in.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The proxy config grew a loop per model source, so the rule for how an alias is called lived in four places - and they had drifted: the LM Studio branch published no model_info.max_input_tokens although the role's README says a model entry carrying context publishes exactly that. Serving anything new meant a fifth branch.
The sources now meet in one filter that returns {alias, params, context}, and the template renders a single loop. Every value goes through to_json, so a response carrying a colon, a quote or a newline cannot break the YAML it is written into.
Publication order is a contract, not an accident of how the loops were stacked: LITELLM_CHAT_MODEL takes the first alias, so a locally served model has to outrank a declared one. That is now stated in the module and asserted, instead of being implied.
The gateway lint had to widen with it. Its canary checks that some gateway template still matches the provider-URL pattern, because without one the rule would pass vacuously on every consumer; the pattern demanded {{ … }} while the template now sets the URL in {% set … %}. Widening it to {{ or {% makes the rule stricter, since a provider URL in a set block is just as direct an address and would have slipped through anywhere else.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gui, mig and the portfolio read core's role data and translations from one
place instead of cloning core themselves. web-svc-api serves them at
api.<domain> as a FastAPI service.
The container keeps a bare git mirror of core and, with forks: auto, of
every fork the GitHub forks API lists, fetched every fetch_interval seconds
under a file lock. The deployed working tree is committed as the ref
`deployed` from a snapshot baked into the image with fixed dates, so every
replica commits the same SHA. A ref is `deployed`, a SHA or prefix, a
branch or tag, or `owner:name` for a fork.
Endpoints: /v1/repositories, /v1/log, /v1/roles, /v1/roles/<role>,
/v1/categories, /v1/bundles, /v1/tree, /v1/file, /v1/todos, /v1/languages
and /v1/catalogs/core/<code>. Descriptions and category titles follow `lang`
or Accept-Language and the response names the language it used. Every
response carries Access-Control-Allow-Origin: *, an invalid ref or path is
a 400 and an unknown one a 404.
A role that enables services.api gets the API origin in its connect-src.
The Playwright spec covers every endpoint against the live stack.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A version bump that only changes a build argument never reached a running
container. In compose mode a changed compose.yml only notified compose-up,
and `compose up` reused the existing image under the unchanged tag. In swarm
mode registry_sync skipped every push whose tag already existed in the
registry, so a rebuilt image never reached the nodes.
Compose mode now reads the rendered compose file back and records the
docker_build_digest of its build sections (context, dockerfile, args) in
.build-inputs; only a change of that digest notifies compose-build, so a
port or env change still costs no build. registry_sync records the local
image ID it pushed for every locally built image in .pushed-images.json and
skips a push only when the tag is in the registry and the ID is unchanged.
External images keep the plain registry skip.
Comparing against the registry manifest instead would depend on the image
store: with the containerd store or BuildKit attestations the local ID is an
index digest, not the config digest the manifest carries.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
roles_with_service drops every role without a canonical domain, which
suits consumers that need a URL but made LiteLLM skip headless
consumers: svc-ai-agent-broker never got its key registered and every
relayed call came back 401 "Unable to find token".
The lookup gains include_headless, which keeps such roles with an empty
canonical_domain and canonical_url, and LiteLLM's consumer key list
passes it. Callers that do not pass it behave as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
databases.csv rows come only from consumers, through sys-svc-rdbms. A
database provider deployed without one, like the lean svc-db-postgres#1
variant (mirror, tor, backup and postgres, nothing else), therefore has no
row. baudolo copies its volume live and records it undumped, and the drill
failed with "postgres_data (postgres) ... no sql dump" in fork run
35731501845, on a cluster that holds only what the engine creates itself.
The new database_consumerless_volumes lookup names the data volume of every
deployed provider that no deployed role consumes as a shared database; a
dedicated database brings its own volume and does not count. test.env hands
the list to verify/backup.sh, which notes those volumes instead of failing.
Every other undumped database volume still fails, above all one whose
consumer forgot to seed its row, since that consumer is deployed.
Only the drill's verdict changes; what gets backed up does not, and the CLI
tests run on dev, act and github runtimes only. The one case it now lets
pass is a dev host still carrying the database of an app its inventory no
longer lists: that data sits undumped in the provider volume and reads as
NOTE instead of FAIL. A CI job always starts on a fresh host.
Counting the databases in the copied data directory was rejected: the
bootstrap set depends on the image, and postgis/postgis adds
template_postgis to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
postgis/postgis publishes amd64 only, so every postgres consumer failed
on arm64. The official postgres image serves both architectures and
already ships the contrib extensions bloom, pg_trgm and unaccent.
PostGIS moves into the postgres_libraries registry and is installed from
PGDG (postgresql-18-postgis-3) only for a consumer that declares it as
an extension, which is mobilizon alone. apt update reads every source
list again, because the PostGIS package depends on GDAL from Debian
main, which a PGDG-only update left without an install candidate.
Postgres 18 keeps its cluster in /var/lib/postgresql/18/docker and its
entrypoint treats a volume at the former /var/lib/postgresql/data as an
old database, so the central and the dedicated data volume both mount at
/var/lib/postgresql. Existing 17 clusters are not migrated: their data
directory trips the entrypoint's old-data check and the container
refuses to start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Stubs native_metrics_apps and asserts the intersection, including the case the
refactor exists for: a role that ships a fragment and is deployed but is not
scraped must not reach the rule file.
Brings in Kevin's AI-agent work, the svc-net-firewall role and the update
tooling. One conflict, in the MCP proxy audit: both sides appended a role to
privileged_control_plane, ours web-app-openbao and upstream svc-net-firewall.
Taken as a union at 25, which is what test_mcp_audit_completeness requires -
neither role declares meta/mcp.yml, so the audit is the only place they are
classified.
Nothing here was typed by hand. The docs generator rebuilt the MCP audit
table against pins that had already landed, so fider reads v0.36.0 instead
of stable, jenkins 2.568.3-lts instead of lts, pretix 2026.7.0 instead of
stable, and flowise 24-slim instead of 24-bookworm-slim. That is the table
catching up with da665ab69 and 10c5b35e8, reading the same declaration the
updater reads.
Autoformat wrapped three test files from the same series: one condition
pulled onto a single line, three assertEqual calls broken across lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mirror proxies /<origin>/<path> to https://<origin>/<path> and relayed the upstream Location header untouched. A root-relative Location then resolved against the mirror instead of the origin, the prefix that names the origin was lost, the follow-up request matched no location block and fell into the catch-all 404. Chromium discards a 404 served for a script request, so the page lost the asset with net::ERR_BLOCKED_BY_ORB.
MIG job 105068912392 shows both halves, 28 times each: GET /unpkg.com/3d-force-graph/dist/3d-force-graph.min.js answered 302, then GET /3d-force-graph@1.80.0/dist/3d-force-graph.min.js answered 404. unpkg redirects an unversioned path to the resolved version and expresses the target as an absolute path.
The forms move into a redirect_sources filter so the template states the intent once and the list is unit-tested. Measured against a two-container nginx fixture: an absolute https Location, an absolute http one and a root-relative one all come back as /<origin>/target. A Location naming a third host stays untouched, which is unchanged behaviour and still leaks past the mirror.
A network-path form was written first and then dropped: nginx normalises such a target in return, and routing one through add_header emits two Location headers of which the proxy relays only the first, so the case could not be produced and the rule stayed unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The list form could publish a host:container pair and nothing else, so thirteen blocks stayed hand-written: the ones needing a protocol suffix, a port range, the swarm long syntax with mode host, or a container-port-only publish. Each of those duplicated numbers the role already declares in meta/services.yml, which is the drift the lookup exists to remove.
The lookup keeps the list form unchanged for its 93 existing call sites and gains a dict form: transport emits one item per protocol, container 'same' serves the services that cannot remap (STUN/TURN, media relays), mode 'host' emits the long block an edge role needs in swarm so the client address survives the hop, publish 'ephemeral' emits the container port alone, and key names the block for Discourse's launcher file, whose expose: list is what becomes docker run -p there. A declared {start, end} mapping renders as start-end on both sides, so a relay range needs no term of its own.
svc-prx-openresty had no ports block at all and bound 80 and 443 as literals in its template, while torrc.j2 carried a third literal 80 for the hidden service. None of the three referenced the others. openresty now declares the ports, the compose template reads them, and the onion's port 80 comes from ports.onion.http through TOR_ONION_EXTRA_PORTS.
Three blocks stay raw, each with the reason it cannot route through one call: keycloak's sso_proxy maps a published port owned by the consuming role to a container port owned by web-app-keycloak, and matrix's ansible flavor takes element on 8080 and federation on 8048 from the upstream playbook rather than from this repo's ports.internal.
The enforcing lint scanned only files with compose in the name and only double-quoted list items, so nine of the thirteen blocks were invisible to it. It now reads every role template that can carry a ports block, catches unquoted items, protocol suffixes, ranges, flow style and the long syntax, and tracks the block by the indentation of its key: keying on the next line closed the block on anything shaped like a key, which target, published, protocol and mode all are. Under expose: only a real mapping counts, because there the single-segment form documents a container port and publishes nothing.
Verified by rendering all 110 container_ports call sites through the real templar, the real lookup and the real merged applications view.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The public-port sweep never forwarded anything. It read tor.enabled from meta/services.yml on disk, where 104 of 105 roles declare it as the Jinja string "{{ 'svc-net-tor' in group_names }}", which _as_bool answers False for. Running HEAD's collect_public_ports the way HEAD's run() calls it, against the real roles directory with every role deployed, returns []. Only svc-net-tor reads True on disk, and it declares no public ports. The node onion therefore carried port 80 and port 22 and nothing else.
Reviving the sweep by reading the merged view instead would have been worse than leaving it dead: it publishes 16 ports across 7 roles, and declared is not published. openldap's ldaps 636 sits next to network.public false, Mailu's implicit-TLS ports are dropped by container_ports whenever TLS is off (which an onion deployment always is), and svc-net-tor's own SOCKS port would become an open proxy. So the source and the selection rule change together: read the variant-merged applications view, and forward only the categories a service names under ports.onion, resolving the number from ports.local when declared there and ports.public otherwise, which is the precedence container_ports publishes by.
collect_public_ports had no consumer outside this module and its own test. UDP_ONLY_CATEGORIES becomes public so the accompanying lint imports the list rather than restating it, and gains stun_turn and stun_turn_tls: those speak TCP, but the allocation TURN hands back is a UDP relay address an onion client can never receive on.
No role declares ports.onion yet, so the rendered HiddenServicePort set is unchanged by this commit alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/addons/oidc_login.yml gates on lookup('sso_oidc_plugin'), which the applications render evaluates while it is still in flight. Under that re-entry guard the merged view hands out service flags as their own Jinja text, and bool("{{ ... }}") is True. The lookup read services.sso.enabled and services.ldap.enabled as enabled on a deployment with neither Keycloak nor OpenLDAP and announced oidc_login, so Nextcloud advertised a login hand-off to a provider that is not there.
lookup('config', ...) renders the value it returns, which is why the roughly ninety '| bool' sites in meta/ do not have this problem. 238bc285c7 established reaching the merged config through the lookup loader; this one had not been converted. utils.roles.applications.config.get is the raw accessor and does not render.
Measured against the real render: with only web-app-nextcloud deployed the addon set is now empty; adding Keycloak yields sociallogin; adding OpenLDAP as well yields oidc_login. That is the resolution order the lookup's own docstring documents. All 14 nextcloud variants pin both flags as literal booleans, so no CI-deployed variant changes.
The stub next to the plugin answered every lookup with the applications payload; it now mirrors lookup('config', app, path, default). A stub resolves a path but does not render it, so it cannot see this defect at all: the regression test renders for real.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
15a6251d7 introduced the merge but tripped over three of the repository's own rules and over the lookup's existing contract test.
_templated called templar.template(value, fail_on_undefined=False). A templar whose signature is template(self, value) raises TypeError on that kwarg, the except swallowed it, and the value came back unrendered: the lookup returned ['{{ vault_cf }}'] where ['secret-cf'] was due. It now falls back to the positional call, the way utils/cache/base.py has always done it.
The SPOT read goes through load_yaml_any instead of yaml.safe_load, which tests/lint/repository/yaml/test_no_direct_calls.py requires so the file is parsed once per process. The module docstring drops the inventories/development/default.yml literal, which tests/unit/python/cli/administration/deploy/development/test_inventory_vars_file_spot.py whitelists by path.
test_missing_api_global_raises_unknown_key asserted exactly the behaviour that broke CI: a scope without API had to raise. It now asserts the fallback, and two new cases pin the edges. A provider the scope omits still resolves; a provider the declaration does not know still raises.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
113113e553 parks the carrier after the token store, 165 lines below the merge
block, so each of the block's six tasks still renders the applications SPOT
cold in its own forked worker. Measured in CI run wp-swarm-tor2 they cost
323.51s in the PASS 1 main play, 180.39s in the backup play and 479.79s in
PASS 2.
The cost is the render, not the work. Two tasks in that block share the exact
construct `defaults_X | combine(X | default({}, true), recursive=True)`: design
takes 0.247s and service_provider 75.94s. The difference is that
defaults_service_provider is dense with lookup('config'/'users'/'domains'/'tls')
and every one of them lands in get_merged_applications.
The carrier cannot simply move up. get_merged_applications serves the carrier
fact whenever the key matches, and the carrier action goes through that same
path, so a second carrier task would be handed the first one's payload instead
of rendering. merged_applications_cache_key reads neither group_names nor the
token file, and with a real templar 71 of 265 applications differ by
group_names alone, so a carrier left standing across add_host or the token
store would be served where a fresh render belongs.
So the block opens its own carrier and drops it again at the end. Nothing
outside the block changes: every task from sys-version to the stage-wide
carrier renders exactly as before.
Placement follows measurement. A per-key perturbation test over the whole
265-application payload, with the credential regeneration noise floor
subtracted and every role in group_names, finds exactly two reads of the seven
facts the block sets:
.web-app-matomo.services.matomo.excluded_ips[0] <- networks
.web-app-wordpress.services.wordpress.title <- service_provider
networks is the block's first task and service_provider its last, which is why
the carrier opens after the one and is dropped before the other takes effect.
The block's tasks already render before service_provider is merged today, so
the frozen payload is what they would have rendered anyway.
The action gains a `clear` argument so the playbook never spells the private
fact name. Parking the carrier serves a payload in 0.000s against a 3.417s cold
render; a wrong key and a cleared fact both force the render again, which is
what makes the drop task work.
Worth roughly 806s of the 20199s job, against a floor of 673.6s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only two files were touched by both sides. env.js merged automatically: the
incoming side owns recordVideoOptions and this branch had cherry-picked that
same function from it, so the result carries one definition. README.md is the
generated roles index and both sides append rows; resolved as a union, taking
the incoming side for the three rows it edits (OnlyOffice, Open WebUI, OpenClaw)
and keeping ours only for OpenBao, then re-sorted.
The incoming side adds lint rules and resources that this role then violated.
Five fixes, each the remedy the failing rule names:
* services.openbao.ports.local.http 8081 -> 8093; svc-ai-litellm now holds 8081.
* meta/networks.yml subnet 192.168.221.0/24 -> 192.168.222.0/24; svc-ai-mcp-adapter
now holds 221, and the suggester only sub-allocates inside occupied /24s, so
the next free block was taken directly.
* services.email bond 1 -> 0; the entry is disabled and the resource model skips
it before reading the weight.
* README Quick Setup regenerated from the template it had drifted from.
* Credits link the implementer, matching how they are credited elsewhere.
* docs/requirements/035-mcp-proxy-expansion.md disposes of web-app-openbao as
privileged_control_plane, which is what a secrets manager is, and the list
count follows.
Verified against a worktree of #454 alone so the baseline is measured rather
than assumed: lint 501 passed with only the branch-name rule failing, which
fails identically on that baseline because the worktree branch is named
merge/*; unit 5665 passed with the same three host-environment failures the
baseline has; integration 324 passed, 0 failed.
Output of `make quality-high`, which rewrites the tree before it tests it.
Formatting only: the YAML null goes from `~` to an empty value, which parses
the same, and the Python changes are line reflow plus the parenthesized `with`
groups.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible's Linux hardware collector reads /sys/block/<dev>/size without a None
guard, and that read runs before the CPU facts in the same collector. One
unreadable block device therefore drops every hardware fact at once and without
a warning. Seven expressions across four files dereferenced processor_vcpus or
memtotal_mb directly and died on the first render. Measured on the interpreter
the deploy actually uses, gather_subset=all returns none of those facts.
plugins/lookup/resource.py now measures and computes every host-resource
constant. The two measured inputs resolve as an inventory override, then the
host's own hardware facts, then a probe of the machine the plugin runs on. The
probe applies only to a target reached over the local connection, because a
lookup executes on the Ansible controller and would otherwise report the
controller's hardware for a remote host.
No value is ever substituted. What that abort may interrupt depends on context.
A play carries inventory_hostname and has to stop, while the inventory
generator and the lint suite render the same applications payload with no host
and must not fail over a value they never read. The first case raises
UnresolvableValueError, the one class the best-effort render layers re-raise
instead of swallowing; the second raises an ordinary AnsibleError that they
absorb, exactly as the missing fact did before.
The meta files of web-app-nextcloud and svc-opt-swapfile call the lookup
directly rather than through a group_var. _render_with_templar replaces the
templar's variables with a flat dict, which drops the lazy resolution a play
uses to expand a variable that itself holds a lookup, so the indirection
renders 0M there while the same expression yields 2133M in a play.
The container count is computed only for the keys that divide by it. Reading
applications pulls the payload back through the templar while it is still being
produced, and those meta files sit in that payload.
utils/cache/errors.py lives in the cache package rather than at the top of
utils/: ansible.cfg exposes ./utils as module_utils, so a top-level errors.py
becomes ansible.module_utils.errors and shadows the real one inside every
module payload zip, which broke uri and get_url with an ImportError pointing
into a temporary zip. A lint now guards that class of collision.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
csp-checker navigates every URL with a 20000 ms budget until
domcontentloaded. Over the Tor SOCKS proxy that expires before a page of
module scripts finishes: run 34614940508 lost Mastodon#1 and OpenProject#1
to it, 26 of 27 attempts logging "Unable to reach http://<onion>/
(Navigation timeout of 20000 ms exceeded)" while sys-service-terminator
retried the unit for 35 minutes. No policy was ever read; the checker never
got that far.
csp-checker 2.3.0 takes --timeout, so script.py takes --onion-timeout and
hands it to the onion batch alone, before the -- separator that would make
it a URL. Clearnet keeps the image default. The budget is
HEALTH_CSP_TIMEOUT_MS (20000, the image default) times the tor flavor's
multiplier, 100000 ms on the public network.
The unit's TimeoutStartSec had to follow. It was 30 + 25 s per domain --
130 s for the two onion vhosts Mastodon checks, while a 100 s budget needs
up to 200 s, so systemd would have killed the check mid-run. The filter now
takes onion_per_domain_seconds and counts .onion domains at that rate;
without the argument it computes exactly what it did before, so
sys-ctl-hlth-webserver is unchanged. The Mastodon case goes from 130 s to
450 s, a clearnet-only play stays at 130 s.
Tests pin that --timeout is appended only when set and stays ahead of the
separator, that main budgets the onion batch alone, that the role passes
both --onion-timeout and the onion rate for the start budget (both guards
fail against the previous tasks file), and the filter's split rate.
Not covered: the filter still caps at 3600 s, which binds beyond 34 onion
domains in one round.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sociallogin creates a Nextcloud user as "<provider name>-<keycloak sub>",
and the provider name was the Keycloak domain. On an onion variant that
domain alone is 67 chars, so every OIDC callback died with
InvalidArgumentException "Username is too long" and HTTP 500 at
/apps/sociallogin/custom_oidc/auth.<onion> (run 34615516009,
Nextcloud#1, 204 occurrences). The whiteboard and admin OIDC specs failed
on the same login.
This is the second derived identifier an onion domain overflows; the
kernel hostname is the first, clamped to 63 bytes by container_hostname
since the pretix and bookwyrm fixes. The domain-or-entity rule now lives in
one lookup, bounded_name(application_id, max_length): the primary domain
when it fits, otherwise the entity name. container_hostname delegates to it
at 63 and keeps its eight callers and its behaviour.
The provider name is bounded_name('web-app-keycloak', 64 - 1 - 36), the uid
limit minus the separator and the UUID sub. A short clearnet domain stays
the name, so existing users keep their uids; an onion deploy gets
"keycloak", which, unlike a hash of the domain, survives a rotated onion
address.
Tests cover the lookup's fit, boundary and fallback cases and its argument
validation, container_hostname's delegation at 63, and the provider name's
room for the sub.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit wrote the same proxy-aware logout expression into
four javascript.js.j2 templates. The rule it encodes - sign out of
oauth2-proxy first when the app is proxy-gated, else go straight to the
provider - belongs where proxy-gating is already decided.
utils/roles/applications/services/sso.py gains logout_url(url,
is_proxy_gated) next to get_sso_config, with OAUTH2_SIGN_OUT_PATH as the
path constant; the rd value is quoted exactly as Jinja's urlencode
filter does it. The sso lookup exposes it as the logout_url term: it
renders OIDC through the templar, takes is_proxy_gated from the same
resolution, and refuses a context without OIDC. Prometheus, Checkmk,
Baserow and Jitsi now render lookup('sso', application_id, 'logout_url').
Behaviour is unchanged: the resolver tests pin the rd encoding against
jinja2's do_urlencode, and the lookup tests cover a proxy-gated app, an
OIDC-native app, a disabled SSO and a missing OIDC.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34421944328 odoo#0 loaded assets over the node onion that the page
could not use.
build_csp_header mirrors every clearnet token that carries the primary
domain onto the node onion, but kept its scheme. Onion vhosts are served
in plaintext, so a sibling like https://cdn.<onion> never matched the
http://cdn.<onion> request it was meant to allow. Onion siblings now map
https to http and wss to ws; the clearnet token is unchanged.
The body filter rewrote external asset origins to the onion mirror only
in HTML, so stylesheets kept pulling https://<origin> directly. When an
app has tor enabled and external asset origins, the header filter now
also flags text/css responses on .onion hosts (dropping Content-Length,
since the body changes), and the body filter runs those through the same
buffer and mirror rewrite and returns before the HTML injections. The
flag is set under the same Jinja condition that renders the mirror
block, so a flagged response always has a rewrite to run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
compose_volumes copied extra_configs and extra_secrets through verbatim.
A file-backed extra without a name therefore reached swarm under a fixed
name, and swarm secrets are immutable: a new content under the same name
is rejected on the next stack deploy. Such extras now get the same
{role}_{key}_{sha8(content)} name that _config_secret_name already gives
meta-declared entries. An explicit name is kept.
container_volumes appended extras as handed over. Ansible passes one
lookup's result to another as proxy containers, and a TaggedInt mode
inside a proxy mapping made PyYAML raise "cannot represent an object".
The private _to_plain of compose_volumes moves to utils/templating as
to_plain, now coercing bool, int and float to the exact builtin types
(bool before int, since bool subclasses int), and container_volumes runs
its extras through it.
Only web-app-matrix passes extra secrets or configs today; the
extra_volumes path that web-app-nextcloud uses is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LITELLM_BACKEND_ROLES read lookup('deployment').groups, which deployment.py
fills from group_names. That variable is host-scoped: it lists only the groups
the host currently being processed belongs to.
On a swarm round the manager carries svc-ai-ollama, svc-ai-lmstudio and
svc-ai-litellm while the workers carry three groups and none of them is an AI
backend. Both branches of LITELLM_CHAT_MODEL therefore missed on a worker and,
with OPENROUTER_API_KEY empty, the model resolved to the empty string.
Only one task made that visible. Every stateful task in the gateway include
carries delegate_to STACK_HOST plus run_once, so it templates in the manager's
scope and wrote the correct model into the database. The assert carries
neither, so it re-templated in each worker's own scope and failed there while
the manager passed.
The manager-only membership is not a placement decision. The swarm test
inventory is provisioned with a single --host MGR, so every role of the round's
closure gets that one host, and extend_inventory then widens only
svc-swarm-node, svc-storage-nfs-client, svc-storage-nfs-server and the round's
own application. The three AI roles declare no placement at all; they are
manager-only because nothing widened them.
Add an inventory selector next to the host-scoped one, built from the play-wide
groups magic var, skipping groups without hosts and Ansible's implicit all and
ungrouped, sorted so the cache key is stable. The key takes it too, or two
hosts with equal group_names but different inventories would share an entry.
The blast radius is two readers, both in 16_ai.yml, and both ask only whether
svc-ai-ollama or svc-ai-lmstudio is present. On the manager the new value gains
exactly svc-storage-nfs-server and svc-bkp-nfs-2-local, neither of which either
reader tests, so the manager's answer does not move and only the workers change,
to match it.
This is the third step on the same line. 46a16127fc set .running; b149ad0d22
corrected that to .groups because a targeted apps= round leaves only the
command-line roles in the whitelist. .groups solves the targeted round and not
the multi-host one, because group_names stays host-scoped either way.
Measured in a two-host play with a real group_vars file and the real lookup:
the worker resolves ollama false through group_names and true through the new
selector, the manager is true both ways, and the assert that failed in CI
passes on both hosts.
Requirement 036 named the host-scoped backend selection as the thing that has
to change and deferred it. That residual is closed, so the paragraph now
describes the selector in use and why the two rejected ones are wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
