A pin outside semver was given its own 'docker-ref' marker on the theory
that every class needs one. That holds for a digest and not for a tag. A
digest pin declares something about the registry, that bytes move under a
standing tag, which no other choice says. A tag outside semver declares
that upstream publishes nothing orderable, which is exactly what
'docker-version' already means and what minio and the act runner already
carry on their version lines.
Keeping both would have put '# nocheck: docker-version, docker-ref' on
every such line: two keys, two checks, one sentence said twice. 'docker-ref'
is dropped instead. It lived in this one mapping, was never entered in the
suppression catalog, and no check ever read it.
The semver entry goes with it, because a semver pin never needs a
declaration. The test that asserted three distinct rules is replaced by two
that pin the separation where it is real: a digest marker differs from a
tag marker, and a semver pin has no marker at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven roles pin several images at one shared version, because those images
are one product: the four Jitsi components, six GitLab CNG images, eight
BigBlueButton services, three Penpot, plus Mastodon, Funkwhale and
Nextcloud Talk. The bump job walks entries one by one and takes each
image's own newest tag, so a run landing between upstream's first and last
push raises the components already there and leaves the rest behind. What
deploys afterwards is a mixed set, and for Jitsi a mixed set breaks video.
The window is real rather than theoretical. Measured on Docker Hub,
'stable-9457' exists for jitsi/prosody, jitsi/jicofo and jitsi/jvb but not
for jitsi/web, one build out of sixty.
Candidates are now grouped by role and current version, and the group is
held to the newest tag the intersection of all their tag lists still
offers. A group with nothing newer in common yields no update at all rather
than a partial one. A lone entry passes through untouched, so nothing
changes for the large majority.
The key is deliberately the role and the version, not the image namespace.
A namespace rule does not survive the data: prom/* carries four components
at four different versions, opentalk/* three, and minio, taigaio and
markoshust each two. Grouping by namespace would hold back pins that are
correctly independent.
Five of the seven roles have carried this exposure all along; the channel
grammar in the previous commit added the sixth by making Jitsi orderable at
all. This closes all of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Upstreams that ship several channels in parallel put the channel first:
LiteLLM publishes 'main-v1.77.7-stable' next to 'main-v1.77.7-nightly',
Jitsi ships 'stable-9646'. The grammar only allowed a leading 'v', so both
shapes parsed as nothing at all, sat at depth 0 next to 'latest', and the
bump job never looked at them.
The numeric core now takes an optional leading channel, and the channel
folds into the flavor the same way the patch letter already does.
Everything that is not the number is family; only the number is order. That
is what keeps '-stable' away from '-nightly' and 'stable-9646' away from a
bare '9646', which would leave the channel and change what the role
deploys.
The change is additive. Only a tag starting with letters followed by a dash
can match the new group, and every such tag failed the grammar outright
before, so no tag that parses today parses differently. The regression case
pins the four shapes that already worked, and the new cases pin the two
directions that must not happen along with the tags that must stay refused:
'act-latest', 'bookworm-slim', 'main-latest' and 'buildx-stable-1' carry a
channel but no version.
Two pins in this repository become orderable as a result: the four Jitsi
components and the LiteLLM gateway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A codename carries no ordering, so 'bookworm' and 'trixie' are two opaque
strings and no updater can tell that one supersedes the other. A pin that
names its Debian release that way is frozen on that release for as long as
it stands, and the freeze is invisible because the tag keeps resolving.
Numeric tags are comparable and stay inside the freshness pipeline, so the
rule is worth holding for the whole repository rather than fixing case by
case.
The check reads two shapes, because a codename reaches the tree in two
spellings. A 'name:tag' reference is recognised by the absence of a space
after the colon, which is what separates 'debian:bookworm' from the YAML
'image: python' and covers Dockerfile FROM lines and heredocs without a
second parser. A 'version:' value is read as a key, because '3.11-bookworm'
carries no colon at all.
False positives are kept out by two independent gates: a word-boundary
prefilter, so 'residual' and 'inside' are not read as 'sid', and a split of
the tag on '-', '.' and '_' that requires a whole component to be a
codename. That is what lets 'cache buster', '.stretched-link',
'fa-bullseye' and a tags list containing 'stretch' through. Every codename
occurrence in the repository was checked against the matcher by hand, and
the check was negative-controlled in both shapes, including an untracked
file.
Four references had to move for the rule to hold. meta/distros.yml takes
'debian:13' rather than '13.6': at depth 1 the updater can raise it across
a major, while depth 2 pins it inside the 13.x line forever. The whole
'make install' path was run in a debian:13 container to back that: all
twenty-four Depends and Recommends resolve on trixie, the docker repository
bootstrap derives the right suite and installs docker-ce-cli, and the
package builds and installs.
The workspace DiD probe moves to 'debian:13-slim'. An earlier version of
this change carried a suppression claiming the probe had to follow the
suite the apt proxy mirrors. That was wrong twice over: it cited a function
that has nothing to do with the probe, and the coupling does not exist,
because the cache frontend rewrites by path prefix and the suite never
enters a routing decision. Replaying the probe verbatim against
debian:13-slim fetches all three trixie suites through the
bookworm-configured proxies, so the suppression is gone rather than
reworded.
The rule ships with an escape hatch and no uses of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two tag shapes in this repository name a fixed release yet failed the
semver gate, so the bump job skipped them and nothing could tell whether a
newer one existed: the Checkmk counter '2.4.0p32' and the Collabora
five-component '26.04.2.1.1'. Both returned depth 0 and fell into the
unorderable residue next to 'latest' and 'alpine'.
The numeric core now takes up to five components, and a counter written as
a letter plus a number is parsed into its own group. The letter joins the
flavor and the number joins the ordering key, which is what keeps
'2.4.0p32' in a different family from the four-component '2.4.0.32': a
bump between them would silently change what the role deploys. The number
in the key also makes p9 order below p10 instead of above it.
version_flavor reads the parse result positionally, so the wider tuple
leaves it untouched. Its contract does change for counter tags, which now
report the letter as their flavor; every shape the grammar already handled
keeps its classification, pinned by a regression case in the new test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A digest pin was emitted as `name:sha256:...`, which no registry accepts:
image_source('ghcr.io/.../csp-checker', 'sha256:d559...') returned
'ghcr.io/.../csp-checker:3.0.0'. The separator depends on the pin class, so
the class has to be decided before the reference is built.
utils/docker/image/pin.py names the three classes in preference order
(semver, then digest, then a general ref that no registry can order) and
maps each to the nocheck marker that declares it. Both discovery entry
points now end in pull_reference(), so '@' versus ':' is decided in one
place instead of at every call site.
The semver pin lint points a digest at 'docker-digest' instead of
'docker-version', so the two fallbacks stay distinguishable in the
suppression catalog. It stays warn-only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mirror proxies /<origin>/<path> to https://<origin>/<path> and relayed the upstream Location header untouched. A root-relative Location then resolved against the mirror instead of the origin, the prefix that names the origin was lost, the follow-up request matched no location block and fell into the catch-all 404. Chromium discards a 404 served for a script request, so the page lost the asset with net::ERR_BLOCKED_BY_ORB.
MIG job 105068912392 shows both halves, 28 times each: GET /unpkg.com/3d-force-graph/dist/3d-force-graph.min.js answered 302, then GET /3d-force-graph@1.80.0/dist/3d-force-graph.min.js answered 404. unpkg redirects an unversioned path to the resolved version and expresses the target as an absolute path.
The forms move into a redirect_sources filter so the template states the intent once and the list is unit-tested. Measured against a two-container nginx fixture: an absolute https Location, an absolute http one and a root-relative one all come back as /<origin>/target. A Location naming a third host stays untouched, which is unchanged behaviour and still leaks past the mirror.
A network-path form was written first and then dropped: nginx normalises such a target in return, and routing one through add_header emits two Location headers of which the proxy relays only the first, so the case could not be produced and the rule stayed unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The checker matches accepted codes by hostname, and a clearnet vhost
shares only its leftmost label with its onion twin, so --accept-status
mirror.<primary>=404 never applied to mirror.<host>.onion. That vhost
serves the same by-design 404, so the probe called it unreachable and
systemd failed the service -- with the identical message the missing
flag produced, while the status had already moved from 2/INVALIDARGUMENT
to 1/FAILURE.
--skip-domain carried this derivation all along, in is_skipped_domain;
replacing it with --accept-status dropped it. The expansion now runs on
the same label rule, in the wrapper rather than the lookup, because the
onion address does not exist yet when the unit is rendered -- only the
wrapper sees both names, and it reads them from the nginx configs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
suppressed_line_numbers matches the marker on the flagged line itself, and
ruff had split two of the calls so the comment landed behind the closing
paren, one line too low. The reads that can be cached now go through
utils.cache.files, which the bare interpreter imports fine; the two that
cannot keep a marker on a line short enough to stay whole.
The parser also gains the two guards utils/install/lint/pinned.py already
carries for the same bootstrap problem: its reading is cross-checked
against PyYAML on both requirements files, and a subprocess with yaml
made unimportable proves the module still works without it. That turns
the manual venv check into a standing test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gate flagged six of my own changes. INFINITO_ANSIBLE_COLLECTIONS_SOURCE
reached default.env without a handler, so the generator dropped it
silently; ansible.sh then carried a shell-side default for the same key,
a second source that drifts. Both are now one declaration, read as
required, so a missing key fails loudly instead of falling back.
File reads route through the session cache except where the same run
rewrites the file, which is the documented opt-out and now says so per
call. The updater reads yaml through utils.cache.yaml rather than calling
safe_load itself.
restore_collection_history.sh was committed without its executable bit,
and the new workflow's name was 31 characters where the dispatch menu
truncates at 25; the explanatory wording lives in run-name instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lint installer imports this module, and it runs on the runner's bare
interpreter before a single dependency exists. Sharing the version check
with it therefore handed the bootstrap a PyYAML import, which killed it:
'ModuleNotFoundError: No module named yaml' raised through
utils/install/lint/__main__.py:35.
Routing the read through utils.cache.yaml would not have helped, that
imports yaml too. The requirements files are this repository's own and
hold one shape, so the pins are read with two regexes and the standard
library. A line that does not match the shape leaves the entry unpinned,
and unpinned already counts as unsatisfied, so the parser errs towards
installing rather than towards skipping.
Quoted versions are unwrapped: yaml accepts version: "13.4.0" and a
formatter may introduce it, and carrying the quotes into the comparison
would silently defeat the check instead of failing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Which of the two sources is faster is not a constant. galaxy.ansible.com
degraded from 207s to 687s over one day and then served 502s outright,
while the pinned git fallback carried the same install in 31s; a week
later the order may be the other way round. The daily job times both and
swaps the default only when the preferred one's median reaches three
times the other's over twelve samples.
A failed install is recorded as None and ranks worse than any duration,
so an outage counts instead of dropping out of the statistic. Below a
full window nothing changes, which keeps a single bad day from flipping
the default.
History lives in an artifact rather than the Actions cache: a schedule
run resolves to the default branch and has read-only cache access there,
so it could not carry the window forward.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pinning the collections froze them, and nothing thaws a pin: dependabot
reads pyproject.toml, the docker updater reads image tags, and neither
sees a version inside requirements/. The new job resolves upstream
releases with git ls-remote --tags and opens a PR the same way the pip,
image and repository-ref updaters do.
Both requirement files are bumped together, because the git fallback has
to install what the Galaxy file declares; moving only one would break
that. A collection whose two files disagree, or whose version is not a
semver, is left alone -- that covers the branch ref and the commit pin,
which state an intent a bumper must not guess.
The job calls pr.sh rather than role_prs.sh: the latter splits per
sub-directory and would open no PR at all for flat paths like
requirements/*.yml.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The requirements named seven collections without versions, so every
install re-resolved them against galaxy.ansible.com and --force-with-deps
redownloaded all 37 MB even when nothing had changed. Exact pins make the
installed set checkable, and the check short-circuits the whole roundtrip:
make install-ansible drops from 8m24s to 0.077s, and a Galaxy outage stops
mattering once the pinned set is present.
The git fallback is pinned to the same versions so it installs what the
Galaxy file declares rather than whatever the branches currently hold.
kewlfft.aur pins a commit because its tags carry no galaxy.yml and master
has moved past the released version.
The lint installer kept a second, weaker truth about the same directory,
guessing from directory presence for three hardcoded collections; it now
shares the version-exact check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The systemd unit has passed --accept-status since f247b29fd0, but the
wrapper script never learned the flag. argparse rejected it, so the
service failed with status=2/INVALIDARGUMENT on every host that deploys
the role, in compose and in swarm alike.
The checker consumes every token after a list flag that does not start
with "--", and a URL does not, so both lists now share a single
separator placed before the URLs. The previous placement sat directly
behind --ignore-network-blocks-from and would have fed a second list
straight into the URL parser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
08b4d0f2ff was committed while the gate was still running, so ruff format wrapped the dict.fromkeys call after the commit had already been written.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2f533eb279 read TOR_ONION_SSH_ENABLED straight out of the lookup's variables, but group vars reach a lookup unrendered, so the flag arrived as its own template text and boolean() refused it. The torrc render aborted, and with it every tor row: 14 jobs across both runs on 095ea5e882 failed with the same line, Baserow#0 in compose and Nextcloud#1 in swarm among them.
Quoted from job 104843161609: The lookup plugin 'tor_extra_ports' failed: The value '{{ lookup('config', application_id, 'services.tor.forwards.ssh') | bool }}' is not a valid boolean. Reproduced locally: boolean() on that literal raises with the identical message.
tor_egress_clients.py:43 in this same role has resolved its own references through templar.template(trust_as_template(...)) since bf32f57b85. _flag now does the same and names the variable when the value is still not a boolean afterwards.
The unit tests passed real booleans, which is why they stayed green through seven gate runs; the new case sends template text through a templar, the shape production delivers. No lint or unit test could have caught this, because it only appears once ansible hands the variable over unrendered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The probe dropped every canonical and alias domain of a role whose server.status_codes.default named anything at or above 400, so a single by-design 4xx removed the whole vhost from CSP inspection. The checker now takes --accept-status <domain>=<code>, which keeps the page navigated and its policy evaluated, so the declaration buys coverage instead of costing it.
csp_accept_status reads the existing SPOT in utils/roles/applications/status_codes.py rather than restating the rule. That resolves per vhost, because server.status_codes is keyed like domains.canonical, so two domains of one role can differ; the old filter looked only at default and was blind to seaweedfs declaring its api, filer and master keys separately. The threshold is the checker's own: anything below 300 passes unflagged, so a declared redirect is named too.
csp_skip_domains keeps only the half that has no vhost to probe at all, a canonical domain owned by a disabled service, and gains the tests that half never had.
Measured: 14 domains move from skipped to probed, among them jenkins, mattermost, n8n, prometheus, mirror and xmpp. 28 domains carry an entry in total. Across the CI diagnostics of this run 473 of 476 observed responses were 200, the single infinito vhost answering 403 is api.seaweedfs, and it declares that code, so it stays green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tor extra-ports test carried '# noqa: SLF001' on the three lines that set lookup._templar and lookup._loader, which the test must do because the plugin reads the variables it is handed from them. Ruff reports no violation there any more, so RUF100 removed the directives as unused during make autoformat.
A suppression that suppresses nothing asserts a conflict that does not exist and invites the next reader to treat the access as a problem.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The skip list gated its whole loop body on the group_names selection, so a role reaching a node through a runtime include_role never contributed its server.status_codes.default. web-svc-mirror arrives that way on every Tor node and declares 404 as healthy; on a node provisioned from the svc-net-tor README the probe read that 404 as unreachable and hlth-csp could not start.
Measured in CI job 104622514568: the unit rendered without --skip-domain and the checker reported ERR_SOCKS_CONNECTION_FAILED on the mirror onion 26 times. Only the service-disabled half depends on the selection now. The widened branch can contribute at most 14 of 266 applications, and naming a domain no vhost serves is inert because script.py filters an nginx-enumerated list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
group_vars/all/README.md forbids role-specific configuration in that directory, and 19_tor.yml held five variables no role but svc-net-tor ever read. They now live where the README points: the values an operator may override in meta/services.yml, where the inventory's applications block is deep-merged on top, and the derived ones in vars/main.yml.
The exception is the egress virtual range, which moves the other way into 08_networks.yml. It is not a property of the role but a claim on shared address space, and 0f6728a4c1 records what a collision costs: tor's upstream default 10.192.0.0/10 swallowed the docker pool at 10.208.0.0/12, and every connection to a container address came back as 'errno 104 Connection reset by peer'. The two numbers that must not overlap now sit one line apart.
The onion's HTTP forward becomes a declaration rather than a derivation. lookup('tor_ports') walks group_names and collects what those roles declare under ports.onion; port 80 belongs to svc-prx-openresty, which a node acquires through sys-stk-front-proxy and which therefore appears in no dependency list an inventory carries. A node provisioned with --include svc-net-tor published an onion forwarding SSH and nothing else, leaving every app it serves unreachable - measured as ERR_SOCKS_CONNECTION_FAILED in the CI instructions replay. The new role-local tor_extra_ports lookup composes the flagged forwards with the derived ones and collapses a port named twice, since torrc writes one HiddenServicePort line per entry.
Not moved: tor_ports stays in plugins/lookup because tor_reachable imports from it and three database roles use that; and the flags sit outside ports.onion because every key there mirrors a port the role declares under ports.local, which neither sshd nor the front proxy is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dnsmasq binds the docker bridge address with the clearnet upstreams configured and the node's own .onion records loaded, and no rule covered port 53. Any source that could route a packet to that address got an open resolver plus the split-DNS records. c89ca42c46 claimed local-service closed this; abb6f37297 showed the keyword never took effect.
The guard now carries two named sets instead of one plus a special case. 'containers' is the docker pool list, for tor's own transparent and DNS ports. 'internal' is NETWORK_INTERNAL_CIDRS, for the resolver and the SOCKS port. Every guarded entry names its set, which folds the hand-written SOCKS rule and its manual append into the same list the reverse-path sets are built from.
The resolver takes the wider set deliberately: 'internal' is the list torrc.j2 already spends on SocksPolicy accept. The narrow set carries only 10.208.0.0/12 and 10.7.7.0/24 out of 10.0.0.0/8, which leaves out swarm's ingress network, every overlay docker allocates without an ipam block, and a node whose own address sits elsewhere in 10/8. A query bound to such an address was measured timing out against the narrow set. Guarding the resolver more strictly than the proxy beside it would refuse a name lookup to a source that is handed a full SOCKS proxy six lines earlier.
NETWORK_DNS_PORT lives next to the other resolver constants rather than in meta/services.yml: ports.local describes host-bound container ports, where 53 collides with svc-dns-unbound and falls outside every PORT_BANDS band.
Measured on the rendered artifact: the table loads and reloads with a byte-identical listing, every compose source of port 53 resolves inside 'internal' and every source of 9040/9053 inside 'containers', and a mistyped set name, an empty CIDR list or a missing admitted key each fail loudly at validate time. Port 53 newly passes a strict reverse-path check, so an asymmetrically routed client loses name resolution.
test_fragments built its guard list from a fixture of three entries while the role shipped six, so it validated the template against a port list nobody deploys. It now takes the shape from the declaration and fills only the numbers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The line that mocks the role_subnets dependency exceeded the line length; the project formatter broke it across three. No behaviour changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard drops every source outside the list tor_egress_clients returns, and containers reach the TransPort and the DNSPort through exactly those rules. Nothing held that relationship: narrowing a pool, renaming a key or dropping an entry from the lookup would have taken the resolver and the transparent proxy away from every container at once, in production.
The test runs the real lookup over the repository's real network declarations rather than a fixture, so it fails where the mistake is made. Its templar resolves the cross references those declarations carry, because without it the lookup would compare unrendered template text and the check would pass on nothing.
It also pins the decision that the admitted list is deliberately narrower than SocksPolicy: 10.0.0.0/8 stays out, because a transparent proxy and a resolver with no access control of their own do not belong to the whole LAN. Left unwritten, that reads like an oversight and gets repaired on the next pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An audit of what the host actually filters found a single input chain in the whole ruleset, covering three tor ports. ss showed two more binds with nothing in front of them: the SOCKS port on 0.0.0.0, and dnsmasq on the bridge address.
Each gets the layer it belongs in. SOCKS gets a packet filter rule against the same CIDRs SocksPolicy already accepts, so no client loses access while the filter closes if tor's own config ever stops enforcing. Reusing the narrower guard set was deliberately avoided: it excludes the LAN blocks SocksPolicy admits, so it would have changed the policy rather than hardened it.
dnsmasq gets local-service, which answers only directly connected networks. A packet filter rule there would have sat on the container DNS path, which daemon.json points at that same bridge address, in exchange for closing an exposure that needs an extra route onto the bridge.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role advertised holding a forwarding or masquerade rule for a tunnel. A fragment cannot do that: it lives in its own table, and an accept there terminates only its own base chain, so a DROP policy another component set still executes. svc-net-wireguard-firewalled was measured losing every forwarded packet that way.
The use case is replaced by one the design does serve, and the rule that decides it is written down where the next author will look rather than left in a commit body.
The fragment test no longer seeds the two wireguard interface names; the fragment they described is gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-net-tor rendered its own guard script, its own unit and its own handler, and hooked the rules into DOCKER-USER with an insert, because a rule appended to a chain docker owns lands behind the RETURN docker puts there. The script flushed its chain and refilled it rule by rule, so the boundary was gone for the duration of the refill.
A role that needs packet filter rules now writes templates/nftables.conf.j2 declaring its own table and includes svc-net-firewall, which renders it to /etc/nftables.d/<role>.conf, rejects it with nft -c, loads it, reads the table back and refuses one that carries no chain. Deployed on its own the role installs nftables and the ruleset that loads every fragment at boot. That ruleset is a distribution decision, so its path follows os_family: RedHat reads /etc/sysconfig/nftables.conf, the others /etc/nftables.conf. The unit is only enabled, never started, because on Debian its ExecStop flushes the whole ruleset, docker's tables included.
One table per role removes the shared path: a fragment replaces its own table and touches no other, and docker never touches either. nft -f on an existing table merges, so each fragment creates, deletes and redeclares its table in one transaction. The tests load every fragment twice into a throwaway network namespace and require the listing to match, because a merging fragment loads twice without error and shows up only as a duplicated rule.
The tor guard now applies in both deploy modes. It was pinned to swarm while torrc binds TransPort and DNSPort on the bridge address in compose too, so on a single-host install those ports answered every on-link neighbour. The guard drops rather than admits, so nothing short-circuits another table, and ip saddr carries an implicit ipv4 dependency, which leaves the ipv6 exposure exactly where the iptables rules left it. svc-net-tor reads its own chain back and refuses a guard that drops nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sending to a .onion needs two things of the node: a resolver that answers the suffix, and a route from the container to Tor's transparent port. Neither existed. Compose pointed Mailu at unbound, which knows nothing about .onion, and in swarm Tor's DNSPort and TransPort listen inside the container where the host's dnsmasq and nat rules cannot reach them, so the forwarding rules that were already there pointed at a closed port on every swarm node.
Compose resolves through unbound, which now forwards the onion zone to Tor's DNSPort on the bridge it already owns. The forward alone would have answered SERVFAIL: the signed root proves that onion does not exist, so with auto-trust-anchor-file set every answer below it validates as bogus, and Tor's DNSPort can supply no DS. The zone is therefore declared insecure. Swarm resolves through the node dnsmasq instead, which every container already uses because container_dns puts the bridge resolver into daemon.json wherever svc-net-tor is deployed, and which now reaches Tor because its ports are published host-mode next to the SOCKS port that was published there already.
The transport rule changed target rather than mechanism. REDIRECT rewrites the destination to the primary address of the interface the packet arrived on, which is a different project bridge for every role, so Tor would have had to listen on all of them. A DNAT naming the bridge keeps one listener, and in swarm the same address reaches the published port through the proxy docker runs there.
That leaves the exposure. SocksPort has SocksPolicy; DNSPort and TransPort have no equivalent, and a swarm-published port carries no host address to bind to, so in swarm the packet filter is their only boundary. Compose needs none, because there Tor binds loopback and the bridge and nothing else. The swarm guard admits this deployment's own address space and drops the rest, and it is installed as a unit ordered after docker rather than as ansible-applied rules: nothing here persists netfilter state, so docker recreating its chains at boot would have left Tor back on the wildcard with the guard gone. It rebuilds its chain instead of appending to it, because the appending version could only ever be loosened. It hooks INPUT and DOCKER-USER both, since host-local traffic reaches a published port through docker's proxy and external traffic through FORWARD.
The deploy now proves all of it back and fails when a rule is missing, the router no longer runs on swarm nodes that carry no Tor, and the CLI test probes the forwarded ports over SOCKS instead of assuming a HiddenServicePort line means a reachable service.
Measured so far: Tor accepts the non-loopback binds and reports circuit-established, and the chain, its order and the DOCKER-USER hooks read back correctly from a deployed host. The swarm path is not yet exercised.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The list form could publish a host:container pair and nothing else, so thirteen blocks stayed hand-written: the ones needing a protocol suffix, a port range, the swarm long syntax with mode host, or a container-port-only publish. Each of those duplicated numbers the role already declares in meta/services.yml, which is the drift the lookup exists to remove.
The lookup keeps the list form unchanged for its 93 existing call sites and gains a dict form: transport emits one item per protocol, container 'same' serves the services that cannot remap (STUN/TURN, media relays), mode 'host' emits the long block an edge role needs in swarm so the client address survives the hop, publish 'ephemeral' emits the container port alone, and key names the block for Discourse's launcher file, whose expose: list is what becomes docker run -p there. A declared {start, end} mapping renders as start-end on both sides, so a relay range needs no term of its own.
svc-prx-openresty had no ports block at all and bound 80 and 443 as literals in its template, while torrc.j2 carried a third literal 80 for the hidden service. None of the three referenced the others. openresty now declares the ports, the compose template reads them, and the onion's port 80 comes from ports.onion.http through TOR_ONION_EXTRA_PORTS.
Three blocks stay raw, each with the reason it cannot route through one call: keycloak's sso_proxy maps a published port owned by the consuming role to a container port owned by web-app-keycloak, and matrix's ansible flavor takes element on 8080 and federation on 8048 from the upstream playbook rather than from this repo's ports.internal.
The enforcing lint scanned only files with compose in the name and only double-quoted list items, so nine of the thirteen blocks were invisible to it. It now reads every role template that can carry a ports block, catches unquoted items, protocol suffixes, ranges, flow style and the long syntax, and tracks the block by the indentation of its key: keying on the next line closed the block on anything shaped like a key, which target, published, protocol and mode all are. Under expose: only a real mapping counts, because there the single-segment form documents a container port and publishes nothing.
Verified by rendering all 110 container_ports call sites through the real templar, the real lookup and the real merged applications view.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The public-port sweep never forwarded anything. It read tor.enabled from meta/services.yml on disk, where 104 of 105 roles declare it as the Jinja string "{{ 'svc-net-tor' in group_names }}", which _as_bool answers False for. Running HEAD's collect_public_ports the way HEAD's run() calls it, against the real roles directory with every role deployed, returns []. Only svc-net-tor reads True on disk, and it declares no public ports. The node onion therefore carried port 80 and port 22 and nothing else.
Reviving the sweep by reading the merged view instead would have been worse than leaving it dead: it publishes 16 ports across 7 roles, and declared is not published. openldap's ldaps 636 sits next to network.public false, Mailu's implicit-TLS ports are dropped by container_ports whenever TLS is off (which an onion deployment always is), and svc-net-tor's own SOCKS port would become an open proxy. So the source and the selection rule change together: read the variant-merged applications view, and forward only the categories a service names under ports.onion, resolving the number from ports.local when declared there and ports.public otherwise, which is the precedence container_ports publishes by.
collect_public_ports had no consumer outside this module and its own test. UDP_ONLY_CATEGORIES becomes public so the accompanying lint imports the list rather than restating it, and gains stun_turn and stun_turn_tls: those speak TCP, but the allocation TURN hands back is a UDP relay address an onion client can never receive on.
No role declares ports.onion yet, so the rendered HiddenServicePort set is unchanged by this commit alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/addons/oidc_login.yml gates on lookup('sso_oidc_plugin'), which the applications render evaluates while it is still in flight. Under that re-entry guard the merged view hands out service flags as their own Jinja text, and bool("{{ ... }}") is True. The lookup read services.sso.enabled and services.ldap.enabled as enabled on a deployment with neither Keycloak nor OpenLDAP and announced oidc_login, so Nextcloud advertised a login hand-off to a provider that is not there.
lookup('config', ...) renders the value it returns, which is why the roughly ninety '| bool' sites in meta/ do not have this problem. 238bc285c7 established reaching the merged config through the lookup loader; this one had not been converted. utils.roles.applications.config.get is the raw accessor and does not render.
Measured against the real render: with only web-app-nextcloud deployed the addon set is now empty; adding Keycloak yields sociallogin; adding OpenLDAP as well yields oidc_login. That is the resolution order the lookup's own docstring documents. All 14 nextcloud variants pin both flags as literal booleans, so no CI-deployed variant changes.
The stub next to the plugin answered every lookup with the applications payload; it now mirrors lookup('config', app, path, default). A stub resolves a path but does not render it, so it cannot see this defect at all: the regression test renders for real.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
services.tor.enabled is Jinja in 99 of the 105 roles that declare it. A lookup('tls') or lookup('domains') inside a meta/*.yml value runs while the applications render is in flight, the re-entry guard hands it the unrendered tree, and _as_bool read the flag's own source text as False. The onion injection then did nothing for any app and the clearnet domain was returned.
fa3797540a already described this and quarantined it: the render state joined the domains cache key so the poisoned map answers no later read. The value stayed wrong, which is enough until a call site bakes it into applications. roles/web-svc-asset/meta/services.yml:15 is the only one in the repo that does, and in run 34922994823 it froze https://file.infinito.test/assets into the payload while every vhost of that round was published under the node onion. web-app-dashboard then waited 60 times on a host openresty answers with ssl_reject_handshake, and the deploy died on TLSV1_UNRECOGNIZED_NAME.
_resolved_flag renders enabled, exclusive and primary and raises when template text survives, the shape plugins/lookup/proxy_pass.py already uses. plugins/lookup/config.py and plugins/lookup/email.py render before coercing for the same reason; this module was the one that did not.
Measured across the whole map: 44 of 12819 leaves change, none added, none removed, 17 roles. 42 are clearnet to onion. The two others are repairs as well: keycloak's csp frame-ancestors moves from https to http, which is what an onion node serves, and the nextcloud sociallogin provider name stops being a hostname and becomes the bounded name that test_sociallogin_provider_name.py documents.
The new tests drive the same function with Jinja instead of the literal booleans the existing ones pass, which is why those never saw this. Three of the four fail without the fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
15a6251d7 introduced the merge but tripped over three of the repository's own rules and over the lookup's existing contract test.
_templated called templar.template(value, fail_on_undefined=False). A templar whose signature is template(self, value) raises TypeError on that kwarg, the except swallowed it, and the value came back unrendered: the lookup returned ['{{ vault_cf }}'] where ['secret-cf'] was due. It now falls back to the positional call, the way utils/cache/base.py has always done it.
The SPOT read goes through load_yaml_any instead of yaml.safe_load, which tests/lint/repository/yaml/test_no_direct_calls.py requires so the file is parsed once per process. The module docstring drops the inventories/development/default.yml literal, which tests/unit/python/cli/administration/deploy/development/test_inventory_vars_file_spot.py whitelists by path.
test_missing_api_global_raises_unknown_key asserted exactly the behaviour that broke CI: a scope without API had to raise. It now asserts the fallback, and two new cases pin the edges. A provider the scope omits still resolves; a provider the declaration does not know still raises.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
API is a plain mapping, so an inventory that configures three providers replaces the whole declaration from group_vars/all/18_api.yml instead of merging into it. inventories/development/default.yml declares openai, anthropic and openrouter, so github, google and jira did not exist in CI at all and lookup('api', 'github.client_id') raised.
A raising lookup inside a value the applications renderer is templating does not fail the render: _templar_render_best_effort returns the string untouched and, for a value carrying a non-env lookup(, refuses the fallback as well. The raw template text reached plugin_value.enabled | bool, which coerced it to False and emitted the deprecation warning that fails the deploy gate. Run 34893363518, job 104153194109 lost web-app-nextcloud to exactly that, for three integrations at once. The configvalue entries would have stored template text as the client id.
utils/api.py merges the scope's override over the declaration, so an override narrows values and never the provider set, and holds the enable rule: every credential a provider declares must carry a value. The api lookup reads through it; the new api_enabled lookup gates on it and returns a real boolean instead of a string a filter has to coerce.
The same class sat in group_vars/all/16_ai.yml and roles/svc-ai-litellm/meta/secrets.yml as bare API.<provider>.<key> dereferences; both now go through the lookups.
Two lints keep it shut. test_addon_enabled_renders_to_bool renders the whole applications map against a scope with API blanked, the most hostile override an inventory may legally write, and requires every addon gate to come out a boolean. test_api_accessed_via_lookup forbids a bare API dereference in Jinja under roles/, group_vars/ and inventories/.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Job 104155149606 died at the warm request with
litellm.APIConnectionError: OllamaException - Cannot connect to host
ollama:11434 [Connect call failed ('192.168.244.10', 11434)]
192.168.244.10 is not ollama. It is the manager node on the lab bridge: the
name never resolved through the overlay, because the litellm task was attached
to ingress, litellm and postgres and to nothing else. The lmstudio round is the
control, where the same replica does carry the lmstudio network.
The attachment comes from _is_consumer, which reads svc-ai-litellm's own
services.<backend>.enabled and .shared. Variant 1 pins both false. The config
template meanwhile asked `'svc-ai-ollama' in group_names`, which answers a
different question: is ollama deployed anywhere in the round. It is, six edges
away, because litellm pulls postgres, postgres pulls tor, tor pulls the mirror,
the mirror pulls prometheus and prometheus declares flowise, which declares
ollama. So the gateway advertised a route it had no network for.
The template and LITELLM_CHAT_MODEL_SERVED now read the same two flags the
network attachment reads, through one constant each. Publication and routing
cannot diverge any more, and in a variant that declines a deployed backend the
warm task no longer runs at all.
LITELLM_CHAT_MODEL and LITELLM_BACKEND_ROLES are deliberately left alone.
5324eadd8b made them inventory-scoped because web-app-mattermost's assert, the
one task in its file carrying neither delegate_to nor run_once, re-templates in
each worker's own scope; web-app-discourse's addon tasks carry no delegate_to
at all. A host-scoped model name fails there again. Both templates here render
delegate_to STACK_HOST plus run_once, where group_names is authoritative.
meta/services.yml keeps its `in group_names` flags, so the five separate
substring detectors that decide whether a flag counts as dynamic keep seeing
svc-ai-litellm.
Measured over every (deployed, pinned) combination against the real
expressions and the real template: the model consumers are told to ask for is
published whenever the warm task runs, and nothing is published without its
flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34883491022 lost all 15 of its jobs and run 34883296966 four more, every
one of them on the same line and before any role deployed:
ERROR: the value supplied for 'credentials.openai_api_key' does not match
the pattern its schema declares: \S{20,}
786899700f gave the three provider keys a regex and wired validation onto the
value that sits beside a `default:`. That value is the default itself. The
credential leaf writes meta["default"] into the inventory verbatim, so the next
read hands `{{ API.openai.api_key }}` back as if an operator had typed it. The
literal is 24 characters and carries spaces, so \S{20,} cannot match it, and
svc-ai-litellm sits in every closure that holds a litellm consumer, which is
why roles as distant as Baserow and Memcached died with it.
A pattern describes the rendered secret. Template text is not that value, so
validate_supplied_value returns on it, next to the guard that already returns
on an empty one. One guard covers all three call sites. It can only skip a
check, never add one, and the three roles carrying a Jinja default for far
longer (listmonk, keycloak, espocrm) declare no regex and keep their behaviour.
The test class asserted that `<your-key-here>` is rejected but never that the
schema's own default survives, so the gap shipped green.
test_every_declared_default_survives_its_own_schema closes it for every default
the file declares, which is why _schema() widened into _credentials().
Reproduced at the call site that failed: the credentials subprocess taken from
the job log exits 0, and the value it emits decrypts back to the template.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make quality-high pass 75 rewrote it: a concatenation whose right operand is a
multi-line literal gets parenthesised, with the operator leading its own line.
The string itself is unchanged, so
test_narrowing_the_credentials_still_rotates_every_user keeps the declaration
that separates web-app-keycloak from web-app-a and still fails when the user
half of the rotation is narrowed along with the credentials.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The matrix gate rotates credentials between the two deploy passes so PASS 2
has to carry fresh values. It rotated every application block in the host_vars
file, and on a matrix inventory that is not the round. Provision applies its
--include as a strict intersection (filters.py:57-62), so it generates
credentials for at most derive_includes, while mirror_overrides.py adds a block
per mirror artefact carrying only image and version. Rotating those costs one
credential subprocess each for values provision never wrote.
matrix.py now passes --include derive_includes(app_id, variants=round_variants),
the same call 02_provision_inventory.sh:11 feeds provision, from the same
round_variants the provision step got one call earlier. The rotation set is
therefore a superset of what provision generated: every database_password still
rotates, because utils/manager/inventory.py:236-242 writes it under an id from
that same filtered list. Measured for web-app-wordpress: 35 ids.
The scope stops at the credentials. reset_credentials takes credential_ids
beside application_ids because the two halves shared one list, and narrowing it
narrowed _drop_user_passwords too: that resolves through
required_user_policies(roles_dir, application_ids), so 60 of 66 declared users
would have stopped rotating, root, backup and every mcp-* bot among them. The
user half keeps the full document list and is byte-identical to before.
The test fixture declared the same users in both roles, which made the
regression test pass against the regression. web-app-keycloak now declares a
user web-app-a does not, so the assertion has something to bite on.
The CLI aborts when --include matches no block instead of rotating nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway could reach exactly one remote provider, through a bare
OPENROUTER_API_KEY group_var with an empty-string default and an `{% if %}`
around a single hardcoded model entry. OpenAI and Anthropic had no path at all.
All three are now declared in roles/svc-ai-litellm/meta/secrets.yml, which
mapping.py already calls the application config schema, so a key lands at
applications.svc-ai-litellm.secrets.credentials.<provider>_api_key like every
other credential. Each carries the `type` and `regex` its value must satisfy and
defaults to the central API.<provider>.api_key block, which keeps one
operator-facing place to fill a provider in while the role owns the shape.
type and regex had to become real to be worth declaring. 192 `validation:`
blocks sit in the role schemas today and not one line of Python enforces them,
so a `regex` would have joined a pile of decoration. validate_supplied_value
now runs on every path a value arrives from outside — the --set override, a
value already in the inventory, and the value standing next to a `default:`.
A generated value stays exempt because the algorithm that produced it is named
by the same schema and generate_value takes no schema argument. min_length is
deliberately left unenforced: the generators ignore it, so holding them to it
would fail where nothing is wrong.
The pattern is written for the paste, not for the vendor. OpenAI moved issuance
from sk- to sk-proj- without notice and Anthropic issues sk-ant-api03-; a schema
pinning a prefix would reject a valid key and abort inventory creation. `\S{20,}`
accepts every form those three issue today and one nobody has announced, and
rejects a trailing newline, an embedded space, a truncated copy and a
<placeholder>. A key pasted into the wrong provider's slot is caught at deploy
instead, where the route it publishes answers nothing.
The gateway iterates one LITELLM_REMOTE_BACKENDS list instead of three `{% if %}`
blocks; config.yaml.j2 and env.j2 read the same list and selectattr('key')
decides. LITELLM_CHAT_MODEL falls back to the first configured remote alias
rather than to a hardcoded openrouter/auto.
For CI the keys travel as INFINITO_API_{OPENAI,ANTHROPIC,OPENROUTER}_KEY through
default.env, the passthrough allowlist, the infinito service in compose.yml, the
workflow env block and the development inventory. The compose leg is not
optional: the env lint rejected the change without it, because a key the
inventory reads but compose does not forward resolves to empty inside the
container with no error. The workflow reads `secrets.X || vars.X` so a variable
works as asked while a secret wins, which matters because a variable is not
masked and the development inventory sets MASK_CREDENTIALS_IN_LOGS: false.
An unset key is not an error anywhere: the provider's routes stay unpublished
and the CLI test asserts their absence rather than skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway publishing a model proves nothing about a backend answering it: LM
Studio shipped a published lmstudio/default route for as long as it had no
model to load. This CLI test closes that gap by asking the running gateway what
it serves and comparing it against what the deployment enabled.
templates/test.env.j2 is both the input and the entry point — discover_cli_roles
finds a role by that file and filters on group_names, so the test runs only
where the gateway is deployed. meta/tests.yml names the container service key so
run_one.yml resolves the node holding it in swarm, and allows 600s for a cold
model load.
probe.py runs inside the gateway container, where the API answers on loopback
and the backends resolve by name, and uses the standard library only because
the image ships the litellm app rather than a test toolchain. It asserts that
every declared model is served, that the LM-Studio-only aliases are present
exactly when that backend is deployed, that no Ollama-backed model is published
without Ollama, that LITELLM_CHAT_MODEL is routable, that an empty deployment
publishes nothing, and that a real completion comes back non-empty.
The fourth assertion is the one worth having: LITELLM_CHAT_MODEL is built from
lookup('deployment').inventory while the config is built from this host's
group_names. Two sources for one fact, and this is where they are made to
agree.
The alias list is rendered whether or not LM Studio is deployed. Gating it
would empty the set in exactly the case the negative direction needs, turning
"a route must not stand without its backend" into a vacuous check.
The verdict logic sits in a pure evaluate() so it is exercised without mocks:
four deployment shapes, each with its positive and negative cases, including a
both-backends case that asserts a shared alias proves nothing about LM Studio
and only the exclusive one counts.
files/test/.nocheck is required rather than convenient. role-files-layout sorts
files/ by language while test-e2e-cli stages files/test/ under that fixed path;
the other 29 CLI test directories carry the same marker.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
328113e7c added a guard that reads every group_vars/all file to confirm each
GROUP_GATED_CONSTANTS entry is really defined, and read them with
path.read_text(). test_no_uncached_file_reads rejects that under tests/: the
cached read_text shares one decoded read per file across the whole session, and
these files are read by several other tests already.
The nocheck escape does not apply. It covers reads outside the project tree,
and group_vars/all is inside it.
What the guard asserts is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fourteen roles repeated the same two lines to say they use the gateway: a
`shared` flag gated on group_names, and an `enabled` flag conjoining that gate
with LITELLM_CHAT_MODEL_SERVED. Both are now single-sourced.
`shared` becomes literal true. The flag means the provider is a shared instance
rather than one deployed per consumer, and a central gateway is always that;
the dynamic form only said "shared if deployed", which `enabled` already
decides. The 50 shared pins under `litellm:` leave meta/variants.yml with it.
The tree had already noticed: 20 of those 50 lines carried
`# nocheck: variants-static-override`, two of them with copy-pasted text about
container_backup. This narrows e5de4e085a for one service key rather than
reversing it; every other dynamic flag keeps the group_names form.
`enabled` becomes `{{ LITELLM_USABLE }}`, defined in group_vars/all/16_ai.yml.
That shorthand is only possible because is_explicit_truth learns about it.
The resolver reads a dependency out of the flag TEXT — "in group_names" in
value — so a constant hides the role id and the provider silently leaves the
static closure of every role referencing it. Measured: the old expression
resolves True, `{{ LITELLM_USABLE }}` resolves False. GROUP_GATED_CONSTANTS
names the constants whose own definition carries the gate, and the docstring
states the obligation: a SPOT of this kind that is not listed there drops its
provider without a word.
The wider fix, accepting any Jinja as dynamic, was measured and rejected: 35
other flags carry a Jinja value without the marker (MCP sidecars, javascript
and SSO gates, recaptcha), and they would all enter closures they are not in
today.
A literal shared flag is no longer dynamic, so test_dynamic_flags wants the
`# nocheck: dynamic-flag` its own message documents for legitimate exceptions;
where a marker already existed the rule list is extended rather than a second
comment added.
Behaviour is unchanged where it counts. The closure keeps litellm with the same
app counts (wordpress 26, openwebui 23, hermes 23, mediawiki 22, flowise 23),
unrelated Jinja still resolves False, and the truth table of the new enabled
form matches the old one in all four served/deployed combinations. Three roles
resolve without litellm, and `git show HEAD:` proves that predates this change:
zammad pins enabled false in variant 0 at line 39, nextcloud at line 97.
Six unit tests cover the resolver, including one asserting that every
registered constant is actually defined in group_vars, so an entry cannot
become a dead promise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-ai-lmstudio had no model-load step at all: tasks/00_core.yml set the
once-flag and the container network, nothing else. The gateway published
lmstudio/default anyway, so the route existed and nothing could answer it.
wp-compose-tor.log:48994 shows the route written into the config, :51770 shows
the warm step reaching qwen2.5:0.5b on Ollama instead, and the green ai-engine
spec ran against that. LM Studio was deployed, published and never prompted.
The pre-pull mirrors the one that works. tasks/01_manager_ops.yml resolves the
hosting node and container id, loops the declared models through
`lms get --yes --gguf`, overlaps them with async/poll and reaps before
playwright, exactly like svc-ai-ollama. That `lms` exists is measured, not
assumed: /app/daemon-run.sh:67 in the image starts ./.bundle/lms server start,
and `lms get --help` documents -y for scripting and --gguf for the format.
Synonymity needs both namespaces in one place. Ollama knows qwen2.5:0.5b, LM
Studio knows qwen2.5-0.5b-instruct, so AI_LOCAL_MODELS carries both per entry:
`alias` is what the gateway publishes and what Ollama pulls, `name` is what
`lms get` resolves. Both roles now reference that one list, and config.yaml.j2
publishes each backend under the alias, routing Ollama to ollama/<alias> and LM
Studio to openai/<name>. The lmstudio loop skips an alias Ollama already
publishes, so two backends never produce a duplicate model_name and Ollama
keeps precedence. lmstudio/default is gone; the model name is now the same
whichever backend answers it.
Ollama's preload_models changes shape from a list of strings to the {alias,
name} entries. Every consumer moves with it: the pull command, the template
loop, and both expressions in 16_ai.yml. An inventory that overrides
`applications.svc-ai-ollama.services.ollama.preload_models` with the old string
list will break at map(attribute='alias'); this is a deliberate shape change,
in the spirit of 7b15c61dfc.
The CI model list is not a property of an installation, so AI_LOCAL_MODELS_CI
lives in inventories/development/default.yml, whose own header scopes that file
to the cross-cutting environment knobs. test_variable_definitions only knew
group_vars, so it reported the constant as undefined; the shipped inventories
are a definition source too, because the deploy CLI bakes them into host_vars,
and the guard's own docstring already claimed "defined somewhere in the
repository".
A latent crash came out of this: `| list | first` raises on an empty sequence,
so a backend deployed without preload models blew up LITELLM_CHAT_MODEL. Ollama
with only embedding models could already hit it; the existing test dodged it by
rendering only _SERVED. `| default('')` now yields the empty string that
_SERVED already reports as false.
Tests: config.yaml.j2 is rendered rather than read, over ollama-only,
lmstudio-only, both and neither, asserting the published model_name set, that
both backends name the same alias, that a shared alias is published once and
routed to Ollama, and that the render is valid YAML. test_litellm_chat_model
gains the lmstudio branch, the no-preload case and the same-model assertion.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
113113e553 parks the carrier after the token store, 165 lines below the merge
block, so each of the block's six tasks still renders the applications SPOT
cold in its own forked worker. Measured in CI run wp-swarm-tor2 they cost
323.51s in the PASS 1 main play, 180.39s in the backup play and 479.79s in
PASS 2.
The cost is the render, not the work. Two tasks in that block share the exact
construct `defaults_X | combine(X | default({}, true), recursive=True)`: design
takes 0.247s and service_provider 75.94s. The difference is that
defaults_service_provider is dense with lookup('config'/'users'/'domains'/'tls')
and every one of them lands in get_merged_applications.
The carrier cannot simply move up. get_merged_applications serves the carrier
fact whenever the key matches, and the carrier action goes through that same
path, so a second carrier task would be handed the first one's payload instead
of rendering. merged_applications_cache_key reads neither group_names nor the
token file, and with a real templar 71 of 265 applications differ by
group_names alone, so a carrier left standing across add_host or the token
store would be served where a fresh render belongs.
So the block opens its own carrier and drops it again at the end. Nothing
outside the block changes: every task from sys-version to the stage-wide
carrier renders exactly as before.
Placement follows measurement. A per-key perturbation test over the whole
265-application payload, with the credential regeneration noise floor
subtracted and every role in group_names, finds exactly two reads of the seven
facts the block sets:
.web-app-matomo.services.matomo.excluded_ips[0] <- networks
.web-app-wordpress.services.wordpress.title <- service_provider
networks is the block's first task and service_provider its last, which is why
the carrier opens after the one and is dropped before the other takes effect.
The block's tasks already render before service_provider is merged today, so
the frozen payload is what they would have rendered anyway.
The action gains a `clear` argument so the playbook never spells the private
fact name. Parking the carrier serves a payload in 0.000s against a 3.417s cold
render; a wrong key and a cleared fact both force the render again, which is
what makes the drop task work.
Worth roughly 806s of the 20199s job, against a floor of 673.6s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_file_spot failed the gate on the pathlib chain that spelled
meta/services.yml out by hand. Both paths now come from utils.roles.mapping as
ROLE_FILE_META_SERVICES and ROLE_DIR_META_ADDONS, following ea37027f49 and
8243aa571b.
All four chains move, not only the reported one. The lint reads the role-file
shape and lets a "meta" / "addons" directory chain through, so leaving those
behind would have split one fixture across two sources of truth, which is the
split the lint exists to prevent. The constants are slash-bearing strings that
Path accepts as a single segment, so the paths are unchanged.
Only the throwaway role built under tempfile in TestCollectEntriesCoversAddons
is affected; its assertions on entity_path and config_path.name are untouched.
autoformat then reflowed the two write_text concatenations, which fit on one
line once the names shortened, and the generator in test_oidc_setting_order.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
collect_entries walked roles/*/meta/services.yml only, so a repository and ref
declared in an addon file was invisible to it. That mattered once 84a4b04f0
made meta/addons the one place a Discourse plugin is pinned: the pins would
have left the updater's view entirely. The other updater cannot stand in,
utils/update/docker.py reaches version: through iter_role_images and skips any
service without an image:.
e317f08ac9 built the addon contract with its own catalog-driven SPOT in
utils/update/addons.py and left repository.py untouched without saying why, so
this widens the ref path rather than replacing the catalog one; an addon can
be maintained by either.
The per-file work moves into _entries_of_file so services.yml and every
meta/addons/*.yml share it. Addon entries carry an ("addons", <stem>) prefix
on their entity path, which keeps them distinguishable from a service entry
holding the same ref. The line index, the semver filter and the per-file
"# nocheck: repository-version" suppression apply unchanged.
Two tests cover it: an addon ref is collected beside a service ref with its
own path and config_path, and a suppressed addon ref yields nothing. Against
this repository the extension currently collects nothing, because every addon
ref is a branch and is_semver rejects those; the tests are what prove the walk
rather than the absence of output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All 22 run: hooks in app.yml were rails and rake calls against the database,
yet the launcher executes them only while building the image. Two of them
carried rotating values, the OIDC client secret and the administrator
password, so every rotation forced a full rebuild with bundle install and
asset precompile: 961.82s in one pass and 13m16s in the next, on a job that
then missed its wall-clock budget by 201s.
They now run against the running instance through container exec, following
the pattern web-app-openproject already uses: real .rb files under files/ruby,
values through ENV.fetch, a readback that prints booleans rather than values,
and an assert that fails the deploy when a rotated credential did not arrive.
ensure_admin_user.rb mirrors the Y/Y path of admin:create using upstream's own
grant_admin!, change_trust_level! and activate; setting the flags by hand had
skipped auto_approve_user, the staff welcome message, the notification levels,
trust level 1 and the email confirmation.
What stays in run: is the CA trust call, which writes into the image and would
not survive a recreate. With the rest gone, the bootstrap digest f47ea0e5bb
records covers templates, expose, params, volumes, after_code and docker_args,
and nothing in that set rotates.
00_core.yml carried the same IS_STACK_HOST condition three times; it now
includes 04_stack_host.yml once, which holds the three with their own
conditions, and their implementations moved under tasks/stack_host/.
The OIDC values are read only when OIDC is enabled. Ansible resolves vars
lazily, so referencing DISCOURSE_OIDC_LOGOUT_REDIRECT_URL unconditionally
would have run its lookup against web-svc-logout on a deployment that has no
SSO, failing where the template did not.
test_oidc_setting_order.py follows the settings into apply_settings.rb. The
invariant 88218fc740 introduced is unchanged: openid_connect_enabled comes
after the discovery document, the client id and the client secret.
Four explanatory "# Exception:" comments are removed, two of them from
f47ea0e5bb. They described mechanisms rather than naming a trip-wire, which
belongs in a commit body.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both template tests listed the filters they hand to jinja by name, which made
the FilterModule a second source: adding a filter to it left these two behind,
and the templates that used it failed to compile with "No filter named …" in
test-unit and test-integration at once. The list is derived from the module now,
so a filter reaches the tests the moment the role can use it.
The two entries that are not filters at all, to_json and bool, stay spelled out
because jinja needs them and the module does not ship them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The config held literals for values vars/main.yml already owned, and the two
drifted without a sound. actionTimeout carried `|| 30_000` for its whole life
while nothing ever set PLAYWRIGHT_ACTION_TIMEOUT, so every run used the
unscaled 30 s. expect had no entry at all, leaving Playwright's built-in 5 s to
govern every assertion over Tor while its neighbours were scaled by the onion
multiplier. That is what killed the second Nextcloud login attempt: an
expect(usernameField).toBeVisible() that gave up after 5 s inside a 150 s
budget.
vars/main.yml now owns the values in TEST_E2E_PLAYWRIGHT_CONFIG_TIMEOUTS, the
harness renders that map into the staged .env both the deploy and
scripts/tests/e2e/rerun-spec.sh already pass, and the config demands each name
through requiredTimeout() instead of inventing a fallback. The two `-e` flags
that fed the old pair are gone with it.
scripts/lint/playwright.sh stages the same config and runs `playwright test
--list`, which loads it, but built its stub env only from each role's
playwright.env.j2 and no such template carries a timeout. Without this it would
have aborted all 87 specs on a missing variable. It now takes the names from
the same map; the value is a placeholder because --list consumes none of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
