The tor extra-ports test carried '# noqa: SLF001' on the three lines that set lookup._templar and lookup._loader, which the test must do because the plugin reads the variables it is handed from them. Ruff reports no violation there any more, so RUF100 removed the directives as unused during make autoformat.
A suppression that suppresses nothing asserts a conflict that does not exist and invites the next reader to treat the access as a problem.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The skip list gated its whole loop body on the group_names selection, so a role reaching a node through a runtime include_role never contributed its server.status_codes.default. web-svc-mirror arrives that way on every Tor node and declares 404 as healthy; on a node provisioned from the svc-net-tor README the probe read that 404 as unreachable and hlth-csp could not start.
Measured in CI job 104622514568: the unit rendered without --skip-domain and the checker reported ERR_SOCKS_CONNECTION_FAILED on the mirror onion 26 times. Only the service-disabled half depends on the selection now. The widened branch can contribute at most 14 of 266 applications, and naming a domain no vhost serves is inert because script.py filters an nginx-enumerated list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
group_vars/all/README.md forbids role-specific configuration in that directory, and 19_tor.yml held five variables no role but svc-net-tor ever read. They now live where the README points: the values an operator may override in meta/services.yml, where the inventory's applications block is deep-merged on top, and the derived ones in vars/main.yml.
The exception is the egress virtual range, which moves the other way into 08_networks.yml. It is not a property of the role but a claim on shared address space, and 0f6728a4c1 records what a collision costs: tor's upstream default 10.192.0.0/10 swallowed the docker pool at 10.208.0.0/12, and every connection to a container address came back as 'errno 104 Connection reset by peer'. The two numbers that must not overlap now sit one line apart.
The onion's HTTP forward becomes a declaration rather than a derivation. lookup('tor_ports') walks group_names and collects what those roles declare under ports.onion; port 80 belongs to svc-prx-openresty, which a node acquires through sys-stk-front-proxy and which therefore appears in no dependency list an inventory carries. A node provisioned with --include svc-net-tor published an onion forwarding SSH and nothing else, leaving every app it serves unreachable - measured as ERR_SOCKS_CONNECTION_FAILED in the CI instructions replay. The new role-local tor_extra_ports lookup composes the flagged forwards with the derived ones and collapses a port named twice, since torrc writes one HiddenServicePort line per entry.
Not moved: tor_ports stays in plugins/lookup because tor_reachable imports from it and three database roles use that; and the flags sit outside ports.onion because every key there mirrors a port the role declares under ports.local, which neither sshd nor the front proxy is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dnsmasq binds the docker bridge address with the clearnet upstreams configured and the node's own .onion records loaded, and no rule covered port 53. Any source that could route a packet to that address got an open resolver plus the split-DNS records. c89ca42c46 claimed local-service closed this; abb6f37297 showed the keyword never took effect.
The guard now carries two named sets instead of one plus a special case. 'containers' is the docker pool list, for tor's own transparent and DNS ports. 'internal' is NETWORK_INTERNAL_CIDRS, for the resolver and the SOCKS port. Every guarded entry names its set, which folds the hand-written SOCKS rule and its manual append into the same list the reverse-path sets are built from.
The resolver takes the wider set deliberately: 'internal' is the list torrc.j2 already spends on SocksPolicy accept. The narrow set carries only 10.208.0.0/12 and 10.7.7.0/24 out of 10.0.0.0/8, which leaves out swarm's ingress network, every overlay docker allocates without an ipam block, and a node whose own address sits elsewhere in 10/8. A query bound to such an address was measured timing out against the narrow set. Guarding the resolver more strictly than the proxy beside it would refuse a name lookup to a source that is handed a full SOCKS proxy six lines earlier.
NETWORK_DNS_PORT lives next to the other resolver constants rather than in meta/services.yml: ports.local describes host-bound container ports, where 53 collides with svc-dns-unbound and falls outside every PORT_BANDS band.
Measured on the rendered artifact: the table loads and reloads with a byte-identical listing, every compose source of port 53 resolves inside 'internal' and every source of 9040/9053 inside 'containers', and a mistyped set name, an empty CIDR list or a missing admitted key each fail loudly at validate time. Port 53 newly passes a strict reverse-path check, so an asymmetrically routed client loses name resolution.
test_fragments built its guard list from a fixture of three entries while the role shipped six, so it validated the template against a port list nobody deploys. It now takes the shape from the declaration and fills only the numbers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The line that mocks the role_subnets dependency exceeded the line length; the project formatter broke it across three. No behaviour changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard drops every source outside the list tor_egress_clients returns, and containers reach the TransPort and the DNSPort through exactly those rules. Nothing held that relationship: narrowing a pool, renaming a key or dropping an entry from the lookup would have taken the resolver and the transparent proxy away from every container at once, in production.
The test runs the real lookup over the repository's real network declarations rather than a fixture, so it fails where the mistake is made. Its templar resolves the cross references those declarations carry, because without it the lookup would compare unrendered template text and the check would pass on nothing.
It also pins the decision that the admitted list is deliberately narrower than SocksPolicy: 10.0.0.0/8 stays out, because a transparent proxy and a resolver with no access control of their own do not belong to the whole LAN. Left unwritten, that reads like an oversight and gets repaired on the next pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An audit of what the host actually filters found a single input chain in the whole ruleset, covering three tor ports. ss showed two more binds with nothing in front of them: the SOCKS port on 0.0.0.0, and dnsmasq on the bridge address.
Each gets the layer it belongs in. SOCKS gets a packet filter rule against the same CIDRs SocksPolicy already accepts, so no client loses access while the filter closes if tor's own config ever stops enforcing. Reusing the narrower guard set was deliberately avoided: it excludes the LAN blocks SocksPolicy admits, so it would have changed the policy rather than hardened it.
dnsmasq gets local-service, which answers only directly connected networks. A packet filter rule there would have sat on the container DNS path, which daemon.json points at that same bridge address, in exchange for closing an exposure that needs an extra route onto the bridge.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role advertised holding a forwarding or masquerade rule for a tunnel. A fragment cannot do that: it lives in its own table, and an accept there terminates only its own base chain, so a DROP policy another component set still executes. svc-net-wireguard-firewalled was measured losing every forwarded packet that way.
The use case is replaced by one the design does serve, and the rule that decides it is written down where the next author will look rather than left in a commit body.
The fragment test no longer seeds the two wireguard interface names; the fragment they described is gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-net-tor rendered its own guard script, its own unit and its own handler, and hooked the rules into DOCKER-USER with an insert, because a rule appended to a chain docker owns lands behind the RETURN docker puts there. The script flushed its chain and refilled it rule by rule, so the boundary was gone for the duration of the refill.
A role that needs packet filter rules now writes templates/nftables.conf.j2 declaring its own table and includes svc-net-firewall, which renders it to /etc/nftables.d/<role>.conf, rejects it with nft -c, loads it, reads the table back and refuses one that carries no chain. Deployed on its own the role installs nftables and the ruleset that loads every fragment at boot. That ruleset is a distribution decision, so its path follows os_family: RedHat reads /etc/sysconfig/nftables.conf, the others /etc/nftables.conf. The unit is only enabled, never started, because on Debian its ExecStop flushes the whole ruleset, docker's tables included.
One table per role removes the shared path: a fragment replaces its own table and touches no other, and docker never touches either. nft -f on an existing table merges, so each fragment creates, deletes and redeclares its table in one transaction. The tests load every fragment twice into a throwaway network namespace and require the listing to match, because a merging fragment loads twice without error and shows up only as a duplicated rule.
The tor guard now applies in both deploy modes. It was pinned to swarm while torrc binds TransPort and DNSPort on the bridge address in compose too, so on a single-host install those ports answered every on-link neighbour. The guard drops rather than admits, so nothing short-circuits another table, and ip saddr carries an implicit ipv4 dependency, which leaves the ipv6 exposure exactly where the iptables rules left it. svc-net-tor reads its own chain back and refuses a guard that drops nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lint-python runs ruff, which the pytest lint suite does not cover, so these
slipped through locally: an unnecessary lambda around bool, and a Jinja
Environment left on the implicit autoescape default. The suppression has to sit
on the autoescape argument itself, which is where ruff anchors the violation.
Sending to a .onion needs two things of the node: a resolver that answers the suffix, and a route from the container to Tor's transparent port. Neither existed. Compose pointed Mailu at unbound, which knows nothing about .onion, and in swarm Tor's DNSPort and TransPort listen inside the container where the host's dnsmasq and nat rules cannot reach them, so the forwarding rules that were already there pointed at a closed port on every swarm node.
Compose resolves through unbound, which now forwards the onion zone to Tor's DNSPort on the bridge it already owns. The forward alone would have answered SERVFAIL: the signed root proves that onion does not exist, so with auto-trust-anchor-file set every answer below it validates as bogus, and Tor's DNSPort can supply no DS. The zone is therefore declared insecure. Swarm resolves through the node dnsmasq instead, which every container already uses because container_dns puts the bridge resolver into daemon.json wherever svc-net-tor is deployed, and which now reaches Tor because its ports are published host-mode next to the SOCKS port that was published there already.
The transport rule changed target rather than mechanism. REDIRECT rewrites the destination to the primary address of the interface the packet arrived on, which is a different project bridge for every role, so Tor would have had to listen on all of them. A DNAT naming the bridge keeps one listener, and in swarm the same address reaches the published port through the proxy docker runs there.
That leaves the exposure. SocksPort has SocksPolicy; DNSPort and TransPort have no equivalent, and a swarm-published port carries no host address to bind to, so in swarm the packet filter is their only boundary. Compose needs none, because there Tor binds loopback and the bridge and nothing else. The swarm guard admits this deployment's own address space and drops the rest, and it is installed as a unit ordered after docker rather than as ansible-applied rules: nothing here persists netfilter state, so docker recreating its chains at boot would have left Tor back on the wildcard with the guard gone. It rebuilds its chain instead of appending to it, because the appending version could only ever be loosened. It hooks INPUT and DOCKER-USER both, since host-local traffic reaches a published port through docker's proxy and external traffic through FORWARD.
The deploy now proves all of it back and fails when a rule is missing, the router no longer runs on swarm nodes that carry no Tor, and the CLI test probes the forwarded ports over SOCKS instead of assuming a HiddenServicePort line means a reachable service.
Measured so far: Tor accepts the non-loopback binds and reports circuit-established, and the chain, its order and the DOCKER-USER hooks read back correctly from a deployed host. The swarm path is not yet exercised.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The list form could publish a host:container pair and nothing else, so thirteen blocks stayed hand-written: the ones needing a protocol suffix, a port range, the swarm long syntax with mode host, or a container-port-only publish. Each of those duplicated numbers the role already declares in meta/services.yml, which is the drift the lookup exists to remove.
The lookup keeps the list form unchanged for its 93 existing call sites and gains a dict form: transport emits one item per protocol, container 'same' serves the services that cannot remap (STUN/TURN, media relays), mode 'host' emits the long block an edge role needs in swarm so the client address survives the hop, publish 'ephemeral' emits the container port alone, and key names the block for Discourse's launcher file, whose expose: list is what becomes docker run -p there. A declared {start, end} mapping renders as start-end on both sides, so a relay range needs no term of its own.
svc-prx-openresty had no ports block at all and bound 80 and 443 as literals in its template, while torrc.j2 carried a third literal 80 for the hidden service. None of the three referenced the others. openresty now declares the ports, the compose template reads them, and the onion's port 80 comes from ports.onion.http through TOR_ONION_EXTRA_PORTS.
Three blocks stay raw, each with the reason it cannot route through one call: keycloak's sso_proxy maps a published port owned by the consuming role to a container port owned by web-app-keycloak, and matrix's ansible flavor takes element on 8080 and federation on 8048 from the upstream playbook rather than from this repo's ports.internal.
The enforcing lint scanned only files with compose in the name and only double-quoted list items, so nine of the thirteen blocks were invisible to it. It now reads every role template that can carry a ports block, catches unquoted items, protocol suffixes, ranges, flow style and the long syntax, and tracks the block by the indentation of its key: keying on the next line closed the block on anything shaped like a key, which target, published, protocol and mode all are. Under expose: only a real mapping counts, because there the single-segment form documents a container port and publishes nothing.
Verified by rendering all 110 container_ports call sites through the real templar, the real lookup and the real merged applications view.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The public-port sweep never forwarded anything. It read tor.enabled from meta/services.yml on disk, where 104 of 105 roles declare it as the Jinja string "{{ 'svc-net-tor' in group_names }}", which _as_bool answers False for. Running HEAD's collect_public_ports the way HEAD's run() calls it, against the real roles directory with every role deployed, returns []. Only svc-net-tor reads True on disk, and it declares no public ports. The node onion therefore carried port 80 and port 22 and nothing else.
Reviving the sweep by reading the merged view instead would have been worse than leaving it dead: it publishes 16 ports across 7 roles, and declared is not published. openldap's ldaps 636 sits next to network.public false, Mailu's implicit-TLS ports are dropped by container_ports whenever TLS is off (which an onion deployment always is), and svc-net-tor's own SOCKS port would become an open proxy. So the source and the selection rule change together: read the variant-merged applications view, and forward only the categories a service names under ports.onion, resolving the number from ports.local when declared there and ports.public otherwise, which is the precedence container_ports publishes by.
collect_public_ports had no consumer outside this module and its own test. UDP_ONLY_CATEGORIES becomes public so the accompanying lint imports the list rather than restating it, and gains stun_turn and stun_turn_tls: those speak TCP, but the allocation TURN hands back is a UDP relay address an onion client can never receive on.
No role declares ports.onion yet, so the rendered HiddenServicePort set is unchanged by this commit alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/addons/oidc_login.yml gates on lookup('sso_oidc_plugin'), which the applications render evaluates while it is still in flight. Under that re-entry guard the merged view hands out service flags as their own Jinja text, and bool("{{ ... }}") is True. The lookup read services.sso.enabled and services.ldap.enabled as enabled on a deployment with neither Keycloak nor OpenLDAP and announced oidc_login, so Nextcloud advertised a login hand-off to a provider that is not there.
lookup('config', ...) renders the value it returns, which is why the roughly ninety '| bool' sites in meta/ do not have this problem. 238bc285c7 established reaching the merged config through the lookup loader; this one had not been converted. utils.roles.applications.config.get is the raw accessor and does not render.
Measured against the real render: with only web-app-nextcloud deployed the addon set is now empty; adding Keycloak yields sociallogin; adding OpenLDAP as well yields oidc_login. That is the resolution order the lookup's own docstring documents. All 14 nextcloud variants pin both flags as literal booleans, so no CI-deployed variant changes.
The stub next to the plugin answered every lookup with the applications payload; it now mirrors lookup('config', app, path, default). A stub resolves a path but does not render it, so it cannot see this defect at all: the regression test renders for real.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
services.tor.enabled is Jinja in 99 of the 105 roles that declare it. A lookup('tls') or lookup('domains') inside a meta/*.yml value runs while the applications render is in flight, the re-entry guard hands it the unrendered tree, and _as_bool read the flag's own source text as False. The onion injection then did nothing for any app and the clearnet domain was returned.
fa3797540a already described this and quarantined it: the render state joined the domains cache key so the poisoned map answers no later read. The value stayed wrong, which is enough until a call site bakes it into applications. roles/web-svc-asset/meta/services.yml:15 is the only one in the repo that does, and in run 34922994823 it froze https://file.infinito.test/assets into the payload while every vhost of that round was published under the node onion. web-app-dashboard then waited 60 times on a host openresty answers with ssl_reject_handshake, and the deploy died on TLSV1_UNRECOGNIZED_NAME.
_resolved_flag renders enabled, exclusive and primary and raises when template text survives, the shape plugins/lookup/proxy_pass.py already uses. plugins/lookup/config.py and plugins/lookup/email.py render before coercing for the same reason; this module was the one that did not.
Measured across the whole map: 44 of 12819 leaves change, none added, none removed, 17 roles. 42 are clearnet to onion. The two others are repairs as well: keycloak's csp frame-ancestors moves from https to http, which is what an onion node serves, and the nextcloud sociallogin provider name stops being a hostname and becomes the bounded name that test_sociallogin_provider_name.py documents.
The new tests drive the same function with Jinja instead of the literal booleans the existing ones pass, which is why those never saw this. Three of the four fail without the fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
15a6251d7 introduced the merge but tripped over three of the repository's own rules and over the lookup's existing contract test.
_templated called templar.template(value, fail_on_undefined=False). A templar whose signature is template(self, value) raises TypeError on that kwarg, the except swallowed it, and the value came back unrendered: the lookup returned ['{{ vault_cf }}'] where ['secret-cf'] was due. It now falls back to the positional call, the way utils/cache/base.py has always done it.
The SPOT read goes through load_yaml_any instead of yaml.safe_load, which tests/lint/repository/yaml/test_no_direct_calls.py requires so the file is parsed once per process. The module docstring drops the inventories/development/default.yml literal, which tests/unit/python/cli/administration/deploy/development/test_inventory_vars_file_spot.py whitelists by path.
test_missing_api_global_raises_unknown_key asserted exactly the behaviour that broke CI: a scope without API had to raise. It now asserts the fallback, and two new cases pin the edges. A provider the scope omits still resolves; a provider the declaration does not know still raises.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
API is a plain mapping, so an inventory that configures three providers replaces the whole declaration from group_vars/all/18_api.yml instead of merging into it. inventories/development/default.yml declares openai, anthropic and openrouter, so github, google and jira did not exist in CI at all and lookup('api', 'github.client_id') raised.
A raising lookup inside a value the applications renderer is templating does not fail the render: _templar_render_best_effort returns the string untouched and, for a value carrying a non-env lookup(, refuses the fallback as well. The raw template text reached plugin_value.enabled | bool, which coerced it to False and emitted the deprecation warning that fails the deploy gate. Run 34893363518, job 104153194109 lost web-app-nextcloud to exactly that, for three integrations at once. The configvalue entries would have stored template text as the client id.
utils/api.py merges the scope's override over the declaration, so an override narrows values and never the provider set, and holds the enable rule: every credential a provider declares must carry a value. The api lookup reads through it; the new api_enabled lookup gates on it and returns a real boolean instead of a string a filter has to coerce.
The same class sat in group_vars/all/16_ai.yml and roles/svc-ai-litellm/meta/secrets.yml as bare API.<provider>.<key> dereferences; both now go through the lookups.
Two lints keep it shut. test_addon_enabled_renders_to_bool renders the whole applications map against a scope with API blanked, the most hostile override an inventory may legally write, and requires every addon gate to come out a boolean. test_api_accessed_via_lookup forbids a bare API dereference in Jinja under roles/, group_vars/ and inventories/.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every one of these breaks into something that looks like ordinary drift rather
than a defect: a state read moved after the backend include leaves the node
sealed, a token helper fed the first login's stdout gets an empty string on
every rotated deploy and fails later at the first write, an ungated SecretID
swap answers 500 on every deploy that did not rotate, and a swap that never
destroys the old SecretID revokes nothing at all.
Job 104155149606 died at the warm request with
litellm.APIConnectionError: OllamaException - Cannot connect to host
ollama:11434 [Connect call failed ('192.168.244.10', 11434)]
192.168.244.10 is not ollama. It is the manager node on the lab bridge: the
name never resolved through the overlay, because the litellm task was attached
to ingress, litellm and postgres and to nothing else. The lmstudio round is the
control, where the same replica does carry the lmstudio network.
The attachment comes from _is_consumer, which reads svc-ai-litellm's own
services.<backend>.enabled and .shared. Variant 1 pins both false. The config
template meanwhile asked `'svc-ai-ollama' in group_names`, which answers a
different question: is ollama deployed anywhere in the round. It is, six edges
away, because litellm pulls postgres, postgres pulls tor, tor pulls the mirror,
the mirror pulls prometheus and prometheus declares flowise, which declares
ollama. So the gateway advertised a route it had no network for.
The template and LITELLM_CHAT_MODEL_SERVED now read the same two flags the
network attachment reads, through one constant each. Publication and routing
cannot diverge any more, and in a variant that declines a deployed backend the
warm task no longer runs at all.
LITELLM_CHAT_MODEL and LITELLM_BACKEND_ROLES are deliberately left alone.
5324eadd8b made them inventory-scoped because web-app-mattermost's assert, the
one task in its file carrying neither delegate_to nor run_once, re-templates in
each worker's own scope; web-app-discourse's addon tasks carry no delegate_to
at all. A host-scoped model name fails there again. Both templates here render
delegate_to STACK_HOST plus run_once, where group_names is authoritative.
meta/services.yml keeps its `in group_names` flags, so the five separate
substring detectors that decide whether a flag counts as dynamic keep seeing
svc-ai-litellm.
Measured over every (deployed, pinned) combination against the real
expressions and the real template: the model consumers are told to ask for is
published whenever the warm task runs, and nothing is published without its
flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34883491022 lost all 15 of its jobs and run 34883296966 four more, every
one of them on the same line and before any role deployed:
ERROR: the value supplied for 'credentials.openai_api_key' does not match
the pattern its schema declares: \S{20,}
786899700f gave the three provider keys a regex and wired validation onto the
value that sits beside a `default:`. That value is the default itself. The
credential leaf writes meta["default"] into the inventory verbatim, so the next
read hands `{{ API.openai.api_key }}` back as if an operator had typed it. The
literal is 24 characters and carries spaces, so \S{20,} cannot match it, and
svc-ai-litellm sits in every closure that holds a litellm consumer, which is
why roles as distant as Baserow and Memcached died with it.
A pattern describes the rendered secret. Template text is not that value, so
validate_supplied_value returns on it, next to the guard that already returns
on an empty one. One guard covers all three call sites. It can only skip a
check, never add one, and the three roles carrying a Jinja default for far
longer (listmonk, keycloak, espocrm) declare no regex and keep their behaviour.
The test class asserted that `<your-key-here>` is rejected but never that the
schema's own default survives, so the gap shipped green.
test_every_declared_default_survives_its_own_schema closes it for every default
the file declares, which is why _schema() widened into _credentials().
Reproduced at the call site that failed: the credentials subprocess taken from
the job log exits 0, and the value it emits decrypts back to the template.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make quality-high pass 75 rewrote it: a concatenation whose right operand is a
multi-line literal gets parenthesised, with the operator leading its own line.
The string itself is unchanged, so
test_narrowing_the_credentials_still_rotates_every_user keeps the declaration
that separates web-app-keycloak from web-app-a and still fails when the user
half of the rotation is narrowed along with the credentials.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The matrix gate rotates credentials between the two deploy passes so PASS 2
has to carry fresh values. It rotated every application block in the host_vars
file, and on a matrix inventory that is not the round. Provision applies its
--include as a strict intersection (filters.py:57-62), so it generates
credentials for at most derive_includes, while mirror_overrides.py adds a block
per mirror artefact carrying only image and version. Rotating those costs one
credential subprocess each for values provision never wrote.
matrix.py now passes --include derive_includes(app_id, variants=round_variants),
the same call 02_provision_inventory.sh:11 feeds provision, from the same
round_variants the provision step got one call earlier. The rotation set is
therefore a superset of what provision generated: every database_password still
rotates, because utils/manager/inventory.py:236-242 writes it under an id from
that same filtered list. Measured for web-app-wordpress: 35 ids.
The scope stops at the credentials. reset_credentials takes credential_ids
beside application_ids because the two halves shared one list, and narrowing it
narrowed _drop_user_passwords too: that resolves through
required_user_policies(roles_dir, application_ids), so 60 of 66 declared users
would have stopped rotating, root, backup and every mcp-* bot among them. The
user half keeps the full document list and is byte-identical to before.
The test fixture declared the same users in both roles, which made the
regression test pass against the regression. web-app-keycloak now declares a
user web-app-a does not, so the assertion has something to bite on.
The CLI aborts when --include matches no block instead of rotating nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway could reach exactly one remote provider, through a bare
OPENROUTER_API_KEY group_var with an empty-string default and an `{% if %}`
around a single hardcoded model entry. OpenAI and Anthropic had no path at all.
All three are now declared in roles/svc-ai-litellm/meta/secrets.yml, which
mapping.py already calls the application config schema, so a key lands at
applications.svc-ai-litellm.secrets.credentials.<provider>_api_key like every
other credential. Each carries the `type` and `regex` its value must satisfy and
defaults to the central API.<provider>.api_key block, which keeps one
operator-facing place to fill a provider in while the role owns the shape.
type and regex had to become real to be worth declaring. 192 `validation:`
blocks sit in the role schemas today and not one line of Python enforces them,
so a `regex` would have joined a pile of decoration. validate_supplied_value
now runs on every path a value arrives from outside — the --set override, a
value already in the inventory, and the value standing next to a `default:`.
A generated value stays exempt because the algorithm that produced it is named
by the same schema and generate_value takes no schema argument. min_length is
deliberately left unenforced: the generators ignore it, so holding them to it
would fail where nothing is wrong.
The pattern is written for the paste, not for the vendor. OpenAI moved issuance
from sk- to sk-proj- without notice and Anthropic issues sk-ant-api03-; a schema
pinning a prefix would reject a valid key and abort inventory creation. `\S{20,}`
accepts every form those three issue today and one nobody has announced, and
rejects a trailing newline, an embedded space, a truncated copy and a
<placeholder>. A key pasted into the wrong provider's slot is caught at deploy
instead, where the route it publishes answers nothing.
The gateway iterates one LITELLM_REMOTE_BACKENDS list instead of three `{% if %}`
blocks; config.yaml.j2 and env.j2 read the same list and selectattr('key')
decides. LITELLM_CHAT_MODEL falls back to the first configured remote alias
rather than to a hardcoded openrouter/auto.
For CI the keys travel as INFINITO_API_{OPENAI,ANTHROPIC,OPENROUTER}_KEY through
default.env, the passthrough allowlist, the infinito service in compose.yml, the
workflow env block and the development inventory. The compose leg is not
optional: the env lint rejected the change without it, because a key the
inventory reads but compose does not forward resolves to empty inside the
container with no error. The workflow reads `secrets.X || vars.X` so a variable
works as asked while a secret wins, which matters because a variable is not
masked and the development inventory sets MASK_CREDENTIALS_IN_LOGS: false.
An unset key is not an error anywhere: the provider's routes stay unpublished
and the CLI test asserts their absence rather than skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway publishing a model proves nothing about a backend answering it: LM
Studio shipped a published lmstudio/default route for as long as it had no
model to load. This CLI test closes that gap by asking the running gateway what
it serves and comparing it against what the deployment enabled.
templates/test.env.j2 is both the input and the entry point — discover_cli_roles
finds a role by that file and filters on group_names, so the test runs only
where the gateway is deployed. meta/tests.yml names the container service key so
run_one.yml resolves the node holding it in swarm, and allows 600s for a cold
model load.
probe.py runs inside the gateway container, where the API answers on loopback
and the backends resolve by name, and uses the standard library only because
the image ships the litellm app rather than a test toolchain. It asserts that
every declared model is served, that the LM-Studio-only aliases are present
exactly when that backend is deployed, that no Ollama-backed model is published
without Ollama, that LITELLM_CHAT_MODEL is routable, that an empty deployment
publishes nothing, and that a real completion comes back non-empty.
The fourth assertion is the one worth having: LITELLM_CHAT_MODEL is built from
lookup('deployment').inventory while the config is built from this host's
group_names. Two sources for one fact, and this is where they are made to
agree.
The alias list is rendered whether or not LM Studio is deployed. Gating it
would empty the set in exactly the case the negative direction needs, turning
"a route must not stand without its backend" into a vacuous check.
The verdict logic sits in a pure evaluate() so it is exercised without mocks:
four deployment shapes, each with its positive and negative cases, including a
both-backends case that asserts a shared alias proves nothing about LM Studio
and only the exclusive one counts.
files/test/.nocheck is required rather than convenient. role-files-layout sorts
files/ by language while test-e2e-cli stages files/test/ under that fixed path;
the other 29 CLI test directories carry the same marker.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
328113e7c added a guard that reads every group_vars/all file to confirm each
GROUP_GATED_CONSTANTS entry is really defined, and read them with
path.read_text(). test_no_uncached_file_reads rejects that under tests/: the
cached read_text shares one decoded read per file across the whole session, and
these files are read by several other tests already.
The nocheck escape does not apply. It covers reads outside the project tree,
and group_vars/all is inside it.
What the guard asserts is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fourteen roles repeated the same two lines to say they use the gateway: a
`shared` flag gated on group_names, and an `enabled` flag conjoining that gate
with LITELLM_CHAT_MODEL_SERVED. Both are now single-sourced.
`shared` becomes literal true. The flag means the provider is a shared instance
rather than one deployed per consumer, and a central gateway is always that;
the dynamic form only said "shared if deployed", which `enabled` already
decides. The 50 shared pins under `litellm:` leave meta/variants.yml with it.
The tree had already noticed: 20 of those 50 lines carried
`# nocheck: variants-static-override`, two of them with copy-pasted text about
container_backup. This narrows e5de4e085a for one service key rather than
reversing it; every other dynamic flag keeps the group_names form.
`enabled` becomes `{{ LITELLM_USABLE }}`, defined in group_vars/all/16_ai.yml.
That shorthand is only possible because is_explicit_truth learns about it.
The resolver reads a dependency out of the flag TEXT — "in group_names" in
value — so a constant hides the role id and the provider silently leaves the
static closure of every role referencing it. Measured: the old expression
resolves True, `{{ LITELLM_USABLE }}` resolves False. GROUP_GATED_CONSTANTS
names the constants whose own definition carries the gate, and the docstring
states the obligation: a SPOT of this kind that is not listed there drops its
provider without a word.
The wider fix, accepting any Jinja as dynamic, was measured and rejected: 35
other flags carry a Jinja value without the marker (MCP sidecars, javascript
and SSO gates, recaptcha), and they would all enter closures they are not in
today.
A literal shared flag is no longer dynamic, so test_dynamic_flags wants the
`# nocheck: dynamic-flag` its own message documents for legitimate exceptions;
where a marker already existed the rule list is extended rather than a second
comment added.
Behaviour is unchanged where it counts. The closure keeps litellm with the same
app counts (wordpress 26, openwebui 23, hermes 23, mediawiki 22, flowise 23),
unrelated Jinja still resolves False, and the truth table of the new enabled
form matches the old one in all four served/deployed combinations. Three roles
resolve without litellm, and `git show HEAD:` proves that predates this change:
zammad pins enabled false in variant 0 at line 39, nextcloud at line 97.
Six unit tests cover the resolver, including one asserting that every
registered constant is actually defined in group_vars, so an entry cannot
become a dead promise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-ai-lmstudio had no model-load step at all: tasks/00_core.yml set the
once-flag and the container network, nothing else. The gateway published
lmstudio/default anyway, so the route existed and nothing could answer it.
wp-compose-tor.log:48994 shows the route written into the config, :51770 shows
the warm step reaching qwen2.5:0.5b on Ollama instead, and the green ai-engine
spec ran against that. LM Studio was deployed, published and never prompted.
The pre-pull mirrors the one that works. tasks/01_manager_ops.yml resolves the
hosting node and container id, loops the declared models through
`lms get --yes --gguf`, overlaps them with async/poll and reaps before
playwright, exactly like svc-ai-ollama. That `lms` exists is measured, not
assumed: /app/daemon-run.sh:67 in the image starts ./.bundle/lms server start,
and `lms get --help` documents -y for scripting and --gguf for the format.
Synonymity needs both namespaces in one place. Ollama knows qwen2.5:0.5b, LM
Studio knows qwen2.5-0.5b-instruct, so AI_LOCAL_MODELS carries both per entry:
`alias` is what the gateway publishes and what Ollama pulls, `name` is what
`lms get` resolves. Both roles now reference that one list, and config.yaml.j2
publishes each backend under the alias, routing Ollama to ollama/<alias> and LM
Studio to openai/<name>. The lmstudio loop skips an alias Ollama already
publishes, so two backends never produce a duplicate model_name and Ollama
keeps precedence. lmstudio/default is gone; the model name is now the same
whichever backend answers it.
Ollama's preload_models changes shape from a list of strings to the {alias,
name} entries. Every consumer moves with it: the pull command, the template
loop, and both expressions in 16_ai.yml. An inventory that overrides
`applications.svc-ai-ollama.services.ollama.preload_models` with the old string
list will break at map(attribute='alias'); this is a deliberate shape change,
in the spirit of 7b15c61dfc.
The CI model list is not a property of an installation, so AI_LOCAL_MODELS_CI
lives in inventories/development/default.yml, whose own header scopes that file
to the cross-cutting environment knobs. test_variable_definitions only knew
group_vars, so it reported the constant as undefined; the shipped inventories
are a definition source too, because the deploy CLI bakes them into host_vars,
and the guard's own docstring already claimed "defined somewhere in the
repository".
A latent crash came out of this: `| list | first` raises on an empty sequence,
so a backend deployed without preload models blew up LITELLM_CHAT_MODEL. Ollama
with only embedding models could already hit it; the existing test dodged it by
rendering only _SERVED. `| default('')` now yields the empty string that
_SERVED already reports as false.
Tests: config.yaml.j2 is rendered rather than read, over ollama-only,
lmstudio-only, both and neither, asserting the published model_name set, that
both backends name the same alias, that a shared alias is published once and
routed to Ollama, and that the render is valid YAML. test_litellm_chat_model
gains the lmstudio branch, the no-preload case and the same-model assertion.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
113113e553 parks the carrier after the token store, 165 lines below the merge
block, so each of the block's six tasks still renders the applications SPOT
cold in its own forked worker. Measured in CI run wp-swarm-tor2 they cost
323.51s in the PASS 1 main play, 180.39s in the backup play and 479.79s in
PASS 2.
The cost is the render, not the work. Two tasks in that block share the exact
construct `defaults_X | combine(X | default({}, true), recursive=True)`: design
takes 0.247s and service_provider 75.94s. The difference is that
defaults_service_provider is dense with lookup('config'/'users'/'domains'/'tls')
and every one of them lands in get_merged_applications.
The carrier cannot simply move up. get_merged_applications serves the carrier
fact whenever the key matches, and the carrier action goes through that same
path, so a second carrier task would be handed the first one's payload instead
of rendering. merged_applications_cache_key reads neither group_names nor the
token file, and with a real templar 71 of 265 applications differ by
group_names alone, so a carrier left standing across add_host or the token
store would be served where a fresh render belongs.
So the block opens its own carrier and drops it again at the end. Nothing
outside the block changes: every task from sys-version to the stage-wide
carrier renders exactly as before.
Placement follows measurement. A per-key perturbation test over the whole
265-application payload, with the credential regeneration noise floor
subtracted and every role in group_names, finds exactly two reads of the seven
facts the block sets:
.web-app-matomo.services.matomo.excluded_ips[0] <- networks
.web-app-wordpress.services.wordpress.title <- service_provider
networks is the block's first task and service_provider its last, which is why
the carrier opens after the one and is dropped before the other takes effect.
The block's tasks already render before service_provider is merged today, so
the frozen payload is what they would have rendered anyway.
The action gains a `clear` argument so the playbook never spells the private
fact name. Parking the carrier serves a payload in 0.000s against a 3.417s cold
render; a wrong key and a cleared fact both force the render again, which is
what makes the drop task work.
Worth roughly 806s of the 20199s job, against a floor of 673.6s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_file_spot failed the gate on the pathlib chain that spelled
meta/services.yml out by hand. Both paths now come from utils.roles.mapping as
ROLE_FILE_META_SERVICES and ROLE_DIR_META_ADDONS, following ea37027f49 and
8243aa571b.
All four chains move, not only the reported one. The lint reads the role-file
shape and lets a "meta" / "addons" directory chain through, so leaving those
behind would have split one fixture across two sources of truth, which is the
split the lint exists to prevent. The constants are slash-bearing strings that
Path accepts as a single segment, so the paths are unchanged.
Only the throwaway role built under tempfile in TestCollectEntriesCoversAddons
is affected; its assertions on entity_path and config_path.name are untouched.
autoformat then reflowed the two write_text concatenations, which fit on one
line once the names shortened, and the generator in test_oidc_setting_order.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only two files were touched by both sides. env.js merged automatically: the
incoming side owns recordVideoOptions and this branch had cherry-picked that
same function from it, so the result carries one definition. README.md is the
generated roles index and both sides append rows; resolved as a union, taking
the incoming side for the three rows it edits (OnlyOffice, Open WebUI, OpenClaw)
and keeping ours only for OpenBao, then re-sorted.
The incoming side adds lint rules and resources that this role then violated.
Five fixes, each the remedy the failing rule names:
* services.openbao.ports.local.http 8081 -> 8093; svc-ai-litellm now holds 8081.
* meta/networks.yml subnet 192.168.221.0/24 -> 192.168.222.0/24; svc-ai-mcp-adapter
now holds 221, and the suggester only sub-allocates inside occupied /24s, so
the next free block was taken directly.
* services.email bond 1 -> 0; the entry is disabled and the resource model skips
it before reading the weight.
* README Quick Setup regenerated from the template it had drifted from.
* Credits link the implementer, matching how they are credited elsewhere.
* docs/requirements/035-mcp-proxy-expansion.md disposes of web-app-openbao as
privileged_control_plane, which is what a secrets manager is, and the list
count follows.
Verified against a worktree of #454 alone so the baseline is measured rather
than assumed: lint 501 passed with only the branch-name rule failing, which
fails identically on that baseline because the worktree branch is named
merge/*; unit 5665 passed with the same three host-environment failures the
baseline has; integration 324 passed, 0 failed.
collect_entries walked roles/*/meta/services.yml only, so a repository and ref
declared in an addon file was invisible to it. That mattered once 84a4b04f0
made meta/addons the one place a Discourse plugin is pinned: the pins would
have left the updater's view entirely. The other updater cannot stand in,
utils/update/docker.py reaches version: through iter_role_images and skips any
service without an image:.
e317f08ac9 built the addon contract with its own catalog-driven SPOT in
utils/update/addons.py and left repository.py untouched without saying why, so
this widens the ref path rather than replacing the catalog one; an addon can
be maintained by either.
The per-file work moves into _entries_of_file so services.yml and every
meta/addons/*.yml share it. Addon entries carry an ("addons", <stem>) prefix
on their entity path, which keeps them distinguishable from a service entry
holding the same ref. The line index, the semver filter and the per-file
"# nocheck: repository-version" suppression apply unchanged.
Two tests cover it: an addon ref is collected beside a service ref with its
own path and config_path, and a suppressed addon ref yields nothing. Against
this repository the extension currently collects nothing, because every addon
ref is a branch and is_semver rejects those; the tests are what prove the walk
rather than the absence of output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All 22 run: hooks in app.yml were rails and rake calls against the database,
yet the launcher executes them only while building the image. Two of them
carried rotating values, the OIDC client secret and the administrator
password, so every rotation forced a full rebuild with bundle install and
asset precompile: 961.82s in one pass and 13m16s in the next, on a job that
then missed its wall-clock budget by 201s.
They now run against the running instance through container exec, following
the pattern web-app-openproject already uses: real .rb files under files/ruby,
values through ENV.fetch, a readback that prints booleans rather than values,
and an assert that fails the deploy when a rotated credential did not arrive.
ensure_admin_user.rb mirrors the Y/Y path of admin:create using upstream's own
grant_admin!, change_trust_level! and activate; setting the flags by hand had
skipped auto_approve_user, the staff welcome message, the notification levels,
trust level 1 and the email confirmation.
What stays in run: is the CA trust call, which writes into the image and would
not survive a recreate. With the rest gone, the bootstrap digest f47ea0e5bb
records covers templates, expose, params, volumes, after_code and docker_args,
and nothing in that set rotates.
00_core.yml carried the same IS_STACK_HOST condition three times; it now
includes 04_stack_host.yml once, which holds the three with their own
conditions, and their implementations moved under tasks/stack_host/.
The OIDC values are read only when OIDC is enabled. Ansible resolves vars
lazily, so referencing DISCOURSE_OIDC_LOGOUT_REDIRECT_URL unconditionally
would have run its lookup against web-svc-logout on a deployment that has no
SSO, failing where the template did not.
test_oidc_setting_order.py follows the settings into apply_settings.rb. The
invariant 88218fc740 introduced is unchanged: openid_connect_enabled comes
after the discovery document, the client id and the client secret.
Four explanatory "# Exception:" comments are removed, two of them from
f47ea0e5bb. They described mechanisms rather than naming a trip-wire, which
belongs in a commit body.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both template tests listed the filters they hand to jinja by name, which made
the FilterModule a second source: adding a filter to it left these two behind,
and the templates that used it failed to compile with "No filter named …" in
test-unit and test-integration at once. The list is derived from the module now,
so a filter reaches the tests the moment the role can use it.
The two entries that are not filters at all, to_json and bool, stay spelled out
because jinja needs them and the module does not ship them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The config held literals for values vars/main.yml already owned, and the two
drifted without a sound. actionTimeout carried `|| 30_000` for its whole life
while nothing ever set PLAYWRIGHT_ACTION_TIMEOUT, so every run used the
unscaled 30 s. expect had no entry at all, leaving Playwright's built-in 5 s to
govern every assertion over Tor while its neighbours were scaled by the onion
multiplier. That is what killed the second Nextcloud login attempt: an
expect(usernameField).toBeVisible() that gave up after 5 s inside a 150 s
budget.
vars/main.yml now owns the values in TEST_E2E_PLAYWRIGHT_CONFIG_TIMEOUTS, the
harness renders that map into the staged .env both the deploy and
scripts/tests/e2e/rerun-spec.sh already pass, and the config demands each name
through requiredTimeout() instead of inventing a fallback. The two `-e` flags
that fed the old pair are gone with it.
scripts/lint/playwright.sh stages the same config and runs `playwright test
--list`, which loads it, but built its stub env only from each role's
playwright.env.j2 and no such template carries a timeout. Without this it would
have aborted all 87 specs on a missing variable. It now takes the names from
the same map; the value is a placeholder because --list consumes none of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Hugo image build in run 34653542922 died on a Debian mirror mid-sync: 'Err:13 libruby ... File has unexpected size (6300780 != 5440). Mirror sync in progress?', and apt closed with its own advice, 'maybe run apt-get update or try with --fix-missing?'. The index the build had fetched disagreed with the pool it then read.
Acquire::Retries=3 does not cover that. It re-fetches the same URL against the same stale index and reads the same wrong size, which is why three retries produced three identical failures.
install_package_candidates now runs its candidate loop through a helper and, when every candidate failed and the manager is apt-get, fetches a fresh index and runs the loop once more. The helper returns two facts rather than one because the two existing error paths distinguish them: whether the caller's command became callable, and whether anything installed at all.
The retry is bounded so it cannot mask a real absence: apt-get only, once, only when nothing installed, and only behind a new index. A package that does not exist fails the second pass the same way and raises the same RuntimeError. The tests pin both directions, the exact update, install, update, install sequence for apt-get and two calls only for a manager without an index, so the retry cannot spread to pacman or dnf.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d5fc099166 spelled out roles/web-app-matrix/vars/main.yml, which test_file_spot rejects: utils.roles.mapping is the single place that knows the role layout, and every construction of such a path has to read it from there.
The role directory is now a single name and ROLE_FILE_VARS_MAIN is appended to it, so the templates path and the vars path share one root and the role name appears once.
The violation shipped because that commit was gated by the two affected unit tests rather than by make test, which is where the lint runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mu-plugin header describes the split as 'the adapter authorises the transport, the callback authorises the operation'. The second half held, every ability carries infinito_mcp_may_read. The first did not: the adapter answered the initialize handshake to anyone, so the server and its protocol version were readable without a credential.
Measured in run 34653754175, the swarm WordPress#0: test-mcp-guest.js posts an anonymous initialize and expects at least 400, and the error context reads 'Expected: >= 400, Received: 200'. Its neighbour addons/mcp-adapter.spec.js posts an anonymous tools/list and passed, so the tool call was already refused and only the handshake was not.
The filter hangs on rest_pre_dispatch and refuses every route under the mcp namespace while the site cannot attribute the request. It leaves an already short-circuited response alone rather than overwriting it, scopes itself by route prefix so the rest of the REST API stays as open as it was, and reaches is_user_logged_in, which triggers determine_current_user and with it wp_validate_application_password, so a client presenting its application password passes. A WP_Error renders no jsonrpc key, which is the second thing the spec asserts.
The new unit test reads the guarded prefix out of the PHP and the endpoint out of the spec and requires the endpoint to sit under the prefix. Without it an adapter that moves namespace would leave the guard protecting nothing while both negative specs keep passing.
A positive spec was tried and reverted: handing the MCP credential to the Playwright env is forbidden by tests/lint/ansible/services/test_mcp_secret_not_in_playwright.py, because the trace travels in the CI artifact. No role probes its MCP endpoint with the credential at deploy time either, so the authenticated direction stays unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sys-svc-compose-ca writes the CA material for exactly lookup('ca_injected', application_id), which asks three questions per application: has a domain, TLS enabled, mode self_signed. The lookup exists against this drift and says so in its own docstring: anything that reads the mounted file must ask the same three or it points at a path that was never mounted.
The matrix templates asked the global TLS_MODE instead. On an onion deployment TLS is off per application while TLS_MODE stays self_signed, so run 34653754175 logged 'CA trust N/A for web-app-matrix (TLS off/not self-signed) mark done' and the templates rendered the binds anyway.
The rest follows from the timing in that run: the matrix container was created at 05:49:35, the bundle was first written at 05:58:45 by sys-ca-selfsigned. At create time the bind pointed at a path that did not exist, which docker materialises as a directory, and the later docker restart matrix from 02_render_inventory.yml:22 failed on 'error mounting /etc/infinito.nexus/ca/ca-bundle.crt ... not a directory'.
MATRIX_CA_INJECTED holds the predicate once in vars/main.yml, beside MATRIX_CA_MOUNT_PATH, and both templates read it: the outer container bind and the synapse arguments for the nested daemon. The test pins both links of the chain, that the constant is derived from the lookup and that both if lines use the constant, so redefining it from TLS_MODE cannot pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CI zone served A and AAAA for INFINITO_DOMAIN and let every other type fall through to the public resolvers, which return NXDOMAIN for the reserved .test TLD. Per RFC 2308 that denial covers the whole name, not the queried type, so any caching resolver in front of CoreDNS applies it to the A record the deployment depends on.
Without such a cache nobody noticed: Mastodon's EmailMxValidator asks MX, takes the NXDOMAIN, then asks A anyway and gets the address. On a Tor node svc-net-tor puts dnsmasq in front, the denial is cached, and tootctl accounts create ends on 'Failure/Error: email unreachable'. It cost Fediwall#0 and Fediwall#2 in run 34653542922 and Fediwall#2 in run 34653754175, all three compose plus tor; the same two jobs passed in run 34614940508 as compose without tor, same command, same administrator@infinito.test, rc 0.
A template with no answer returns NOERROR with an empty answer section and the SOA in authority, so the zone denies a type without denying the name. It sits after the A and AAAA templates, which would otherwise have their names answered NODATA, and before forward, so no type escapes.
Measured against 172.30.0.53 in the running stack: infinito.test MX went from NXDOMAIN with an authority record to NOERROR with zero answers and the aa flag, infinito.test A and auth.infinito.test A still return 172.30.0.10, and deb.debian.org still resolves through the forward.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The compose-up handler resolves its chdir from lookup('container', application_id, 'directories.instance') at flush time. Bluesky passes compose_handlers_flush false to sys-stk-backend and notifies compose-build and compose-up afterwards, so the topics stayed pending until the next flush. That flush belongs to svc-prx-openresty, which the front proxy include pulls in, and it ran them with openresty's application_id.
Run 34653754175 shows the effect: the only compose up in the whole job created the openresty container in 0.52s, no pds container lifecycle line appears anywhere, and the wait for PDS XRPC then retried for 69s against a stack that was never brought up, ending on 'Status code was -1 and not [200]: Request failed: <urlopen error [Errno 111] Connection refused>'.
The flush now sits directly after the rebuild notifier, before the first front proxy include. sys-stk-backend has already rendered the stack there, so this is not a return to 293c67780d, whose registration sat before the staging and let the container runtime's first flush run the compose handlers before ca-inject and compose.yml existed.
The re-own include ahead of it repeats what sys-stk-backend and sys-svc-rdbms already do: it runs no task, it re-registers the handlers, and it binds them to this application_id instead of whatever the play happens to carry. The ordering test asserts the flush precedes the proxy and that the re-own precedes the flush.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
csp-checker navigates every URL with a 20000 ms budget until
domcontentloaded. Over the Tor SOCKS proxy that expires before a page of
module scripts finishes: run 34614940508 lost Mastodon#1 and OpenProject#1
to it, 26 of 27 attempts logging "Unable to reach http://<onion>/
(Navigation timeout of 20000 ms exceeded)" while sys-service-terminator
retried the unit for 35 minutes. No policy was ever read; the checker never
got that far.
csp-checker 2.3.0 takes --timeout, so script.py takes --onion-timeout and
hands it to the onion batch alone, before the -- separator that would make
it a URL. Clearnet keeps the image default. The budget is
HEALTH_CSP_TIMEOUT_MS (20000, the image default) times the tor flavor's
multiplier, 100000 ms on the public network.
The unit's TimeoutStartSec had to follow. It was 30 + 25 s per domain --
130 s for the two onion vhosts Mastodon checks, while a 100 s budget needs
up to 200 s, so systemd would have killed the check mid-run. The filter now
takes onion_per_domain_seconds and counts .onion domains at that rate;
without the argument it computes exactly what it did before, so
sys-ctl-hlth-webserver is unchanged. The Mastodon case goes from 130 s to
450 s, a clearnet-only play stays at 130 s.
Tests pin that --timeout is appended only when set and stays ahead of the
separator, that main budgets the onion batch alone, that the role passes
both --onion-timeout and the onion rate for the start budget (both guards
fail against the previous tasks file), and the filter's split rate.
Not covered: the filter still caps at 3600 s, which binds beyond 34 onion
domains in one round.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-flavor multipliers lived in test-e2e-playwright's vars, where only
that role could read them. The CSP health check needs the same number: it
probes onion vhosts through the Tor proxy, where the checker's navigation
budget has to grow by the same factor as a Playwright wait.
They now sit in svc-net-tor's meta/services.yml next to services.tor.flavor,
the key they are indexed by, and the Playwright var reads them through
lookup('config', 'svc-net-tor', 'services.tor.timeout_multipliers'). That
lookup resolves from the role defaults, so it answers even where svc-net-tor
is not part of the round, which is how the flavor was already read.
The flavor-coverage test moves to the tor role and reads the new location;
the values (chutney 2, public 5) and what the Playwright run computes from
them are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sweep may deploy 210 rows (available) in chunks of 80. Three declared
blocks hold exactly that when there is no priority line, but priority
rows get a chunk of their own that is never topped up with regular rows.
Nine priority rows therefore occupied a whole block, left the regular
rows 2 x 80 = 160, and 42 rows of budget went unused.
Adding a block to the YAML alone would not have changed that:
matrix.chunks_of handed the split slots.chunk_count(), which is
min(ceil(available / chunk size), INFINITO_CI_MAX_CHUNKS) and ignores
the priority seam, so the plan still stopped at three blocks.
call-orchestrator.yml now declares test-deploy-chunk-3, gated on chunks
0 to 2 like every chunk gates on all before it, and report-main-failures
and done need it. INFINITO_CI_MAX_CHUNKS is 4, one above what the budget
fills, and slots.chunk_blocks() returns it. chunks_of hands the split
every declared block; available still caps the rows, and a chunk whose
slice comes back empty skips its deploy. chunk_count() stays the "chunks
filled" figure of the budget report.
Tests: the split spends 9 + 80 + 80 + 41 = 210 rows with a spare block;
chunks_of plans three chunks from chunk_blocks=3 while chunk_count=2
(the previous wiring yields [1, 10]); an integration guard requires the
declared chunk jobs to equal INFINITO_CI_MAX_CHUNKS and to exceed
ceil(available / chunk size) by one, and fails on the stale value 3.
A shell that inherited INFINITO_CI_MAX_CHUNKS=3 keeps failing that guard
until it is restarted; make dotenv-force rewrites .env.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
304144ce67 probed each bridge's healthcheck at /_matrix/mau/ready. mautrix
sets Ready only after Synapse has pinged the appservice (connector.go
starts the listener, then ensureConnection, then AS.Ready = true), and a
swarm task with a healthcheck stays "starting" and receives no VIP traffic
until it is healthy. The ping could never arrive: Synapse logged 150
ConnectionRefused per bridge, each bridge exited FTL 13, and the stack
never converged (run 34615516009, Matrix#2, job 103327034630).
The healthcheck now probes /_matrix/mau/live, which answers once the
listener is up, before the ping. A deploy task after the Synapse wait asks
every bridge for /_matrix/mau/ready from inside Synapse, over the same
name Synapse pings, so a bridge that never becomes ready still fails the
deploy without gating swarm routing.
The ChatGPT bot is rendered only once its access token exists. Its account
and token are created by stack_host_provisioning after the first stack
deploy, so an earlier bot task failed its login with M_FORBIDDEN until the
converge wait called it stuck. The re-render after provisioning adds it
with a valid token.
Verified in a compose run of variant 2: the readiness task got 200 from
meta, signal, slack and whatsapp on the first attempt. Tests pin the live
probe and the token guard and fail against the previous template.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sociallogin creates a Nextcloud user as "<provider name>-<keycloak sub>",
and the provider name was the Keycloak domain. On an onion variant that
domain alone is 67 chars, so every OIDC callback died with
InvalidArgumentException "Username is too long" and HTTP 500 at
/apps/sociallogin/custom_oidc/auth.<onion> (run 34615516009,
Nextcloud#1, 204 occurrences). The whiteboard and admin OIDC specs failed
on the same login.
This is the second derived identifier an onion domain overflows; the
kernel hostname is the first, clamped to 63 bytes by container_hostname
since the pretix and bookwyrm fixes. The domain-or-entity rule now lives in
one lookup, bounded_name(application_id, max_length): the primary domain
when it fits, otherwise the entity name. container_hostname delegates to it
at 63 and keeps its eight callers and its behaviour.
The provider name is bounded_name('web-app-keycloak', 64 - 1 - 36), the uid
limit minus the separator and the UUID sub. A short clearnet domain stays
the name, so existing users keep their uids; an onion deploy gets
"keycloak", which, unlike a hash of the domain, survives a rotated onion
address.
Tests cover the lookup's fit, boundary and fallback cases and its argument
validation, container_hostname's delegation at 63, and the provider name's
room for the sub.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Discourse latest now validates openid_connect_enabled and refuses it
until the discovery document, client ID and client secret are set. The
bootstrap enabled OIDC first, so a fresh database aborted the launcher
rebuild at `rails r "SiteSetting.openid_connect_enabled = true"` with
Discourse::InvalidHTMLParameters (run 34615516009, WordPress#0, job
103327034654). Existing databases hid it because the values were already
stored.
The enable step now follows the three settings it depends on. A test
reads the real template and requires every prerequisite line to precede
the enable line; against the previous template it fails.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
INFINITO_DOMAIN_RE carried an already escaped regex through the env
chain: EnvBuilder.setdefault prefers a value already in the process env,
quote_for_dotenv escapes every backslash, and load.sh unescapes it again.
One inherited value with a doubled backslash was enough to write
"infinito\\\\.test" into .env, the rendered Corefile then matched a
literal backslash, and getent hosts test.infinito.test failed with rc 2.
The renderer now substitutes DOMAIN_RE = re.escape(INFINITO_DOMAIN) at
render time, so no backslash travels through the env or .env at all. The
template placeholder is renamed because it is a derived render value, not
an env key, and INFINITO_DOMAIN_RE is removed from default.env and the
passthrough keys, leaving INFINITO_DOMAIN as the single source.
A renderer test renders the real template and requires a single-escaped
match with no doubled backslash and no leftover placeholder. Verified
live: the Corefile renders ^infinito\.test\.$, test.infinito.test
resolves to 172.30.0.10 and test.infinitoXtest does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
