The nocheck markers were written as long single lines; ruff splits the
call across three lines and leaves the marker behind the closing paren.
Committing the formatter's own output keeps the tree and the history in
step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Computing the distro list no longer justifies a runner of its own: it is
two steps, and the job that consumes them already checks the repository
out. The workflow now occupies one runner instead of seven.
The step also went silent for minutes at a time, printing five pids and
then nothing until every build had finished. Each build's output is piped
through a [distro][HH:MM:SS] stamp as it arrives, so the log shows what is
happening while it happens; tee still files the per-distro logs for the
artifact, and the replay block at the end is gone as duplication.
A pipeline hands $! the pid of tee rather than of the build, which would
have swallowed every failure, so each build runs in a subshell under
pipefail and wait reads that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gate flagged six of my own changes. INFINITO_ANSIBLE_COLLECTIONS_SOURCE
reached default.env without a handler, so the generator dropped it
silently; ansible.sh then carried a shell-side default for the same key,
a second source that drifts. Both are now one declaration, read as
required, so a missing key fails loudly instead of falling back.
File reads route through the session cache except where the same run
rewrites the file, which is the documented opt-out and now says so per
call. The updater reads yaml through utils.cache.yaml rather than calling
safe_load itself.
restore_collection_history.sh was committed without its executable bit,
and the new workflow's name was 31 characters where the dispatch menu
truncates at 25; the explanatory wording lives in run-name instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lint installer imports this module, and it runs on the runner's bare
interpreter before a single dependency exists. Sharing the version check
with it therefore handed the bootstrap a PyYAML import, which killed it:
'ModuleNotFoundError: No module named yaml' raised through
utils/install/lint/__main__.py:35.
Routing the read through utils.cache.yaml would not have helped, that
imports yaml too. The requirements files are this repository's own and
hold one shape, so the pins are read with two regexes and the standard
library. A line that does not match the shape leaves the entry unpinned,
and unpinned already counts as unsatisfied, so the parser errs towards
installing rather than towards skipping.
Quoted versions are unwrapped: yaml accepts version: "13.4.0" and a
formatter may introduce it, and carrying the quotes into the comparison
would silently defeat the check instead of failing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The parent directory was created and chowned on every run, before the
script looked at whether the virtualenv exists. Where the parent is not
writable that reached for sudo, which a sandboxed or no-new-privileges
container refuses -- so an install that had nothing to do failed with
'The "no new privileges" flag is set'.
Nothing needs preparing once the interpreter is there, so the check moves
ahead of the preparation and returns.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The matrix spent five runners on work that is almost entirely network
wait: the collection install measured 0% CPU for its whole duration, and
the parent-image pull, the package managers and the push are no
different. One runner carries all five concurrently and frees the other
four.
Losing the matrix loses fail-fast: false with it, so push_all.sh awaits
every build and reports each one by name before exiting non-zero. A
broken distro no longer hides the other four, which is what made today's
triage find three causes instead of one.
Starts are spread by fibonacci seconds because each build opens with a
parent-image pull of several hundred MB. Per-distro logs are grouped in
the job output and uploaded as an artifact, and the pressure report now
carries free -h: memory is the one resource this consolidation contends
for, and an OOM kill would otherwise read as a build error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ignore-error traded a loud fault for a silent one. The cache writes to
the same GHCR repository as the image and under the same packages:write
token, so every shared failure mode -- permissions, outage, rate limit --
already fails --push. What remains is buildkit's own cache manifest type,
which a registry can reject while accepting an ordinary image push.
That case is exactly the one worth hearing about. Swallowed, it leaves CI
green while the cache never fills and every build silently pays the full
855-899s again; raised, the first run answers it once and for good.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Which of the two sources is faster is not a constant. galaxy.ansible.com
degraded from 207s to 687s over one day and then served 502s outright,
while the pinned git fallback carried the same install in 31s; a week
later the order may be the other way round. The daily job times both and
swaps the default only when the preferred one's median reaches three
times the other's over twelve samples.
A failed install is recorded as None and ranks worse than any duration,
so an outage counts instead of dropping out of the statistic. Below a
full window nothing changes, which keeps a single bad day from flipping
the default.
History lives in an artifact rather than the Actions cache: a schedule
run resolves to the default branch and has read-only cache access there,
so it could not carry the window forward.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pinning the collections froze them, and nothing thaws a pin: dependabot
reads pyproject.toml, the docker updater reads image tags, and neither
sees a version inside requirements/. The new job resolves upstream
releases with git ls-remote --tags and opens a PR the same way the pip,
image and repository-ref updaters do.
Both requirement files are bumped together, because the git fallback has
to install what the Galaxy file declares; moving only one would break
that. A collection whose two files disagree, or whose version is not a
semver, is left alone -- that covers the branch ref and the commit pin,
which state an intent a bumper must not guess.
The job calls pr.sh rather than role_prs.sh: the latter splits per
sub-directory and would open no PR at all for flat paths like
requirements/*.yml.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
COPY . preceded every expensive RUN, so each commit rebuilt the whole
chain and ansible-galaxy ran on every build of every distro. The
collection install now sits in its own layer fed only by default.env,
scripts/install, the dev-python shell scripts, requirements/ and the
three files the version check imports. Everything that needs the full
tree, the project's own distro package included, stays behind COPY ..
Measured on a role-only change: the collection layer stays CACHED and
the build drops to 3m10s against a CI baseline of 855-899s. An
unchanged rebuild is 15 cached layers in 1.5s.
The gha cache imported and exported nothing in a plain run step, so the
export moves to the registry the job already authenticates against.
ignore-error keeps a cache write failure from taking the build with it,
which it otherwise does: proven against an unreachable registry, where
the build fails hard without it and exits 0 with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The requirements named seven collections without versions, so every
install re-resolved them against galaxy.ansible.com and --force-with-deps
redownloaded all 37 MB even when nothing had changed. Exact pins make the
installed set checkable, and the check short-circuits the whole roundtrip:
make install-ansible drops from 8m24s to 0.077s, and a Galaxy outage stops
mattering once the pinned set is present.
The git fallback is pinned to the same versions so it installs what the
Galaxy file declares rather than whatever the branches currently hold.
kewlfft.aur pins a commit because its tags carry no galaxy.yml and master
has moved past the released version.
The lint installer kept a second, weaker truth about the same directory,
guessing from directory presence for three hardcoded collections; it now
shares the version-exact check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The systemd unit has passed --accept-status since f247b29fd0, but the
wrapper script never learned the flag. argparse rejected it, so the
service failed with status=2/INVALIDARGUMENT on every host that deploys
the role, in compose and in swarm alike.
The checker consumes every token after a list flag that does not start
with "--", and a URL does not, so both lists now share a single
separator placed before the URLs. The previous placement sat directly
behind --ignore-network-blocks-from and would have fed a second list
straight into the URL parser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
08b4d0f2ff was committed while the gate was still running, so ruff format wrapped the dict.fromkeys call after the commit had already been written.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2f533eb279 read TOR_ONION_SSH_ENABLED straight out of the lookup's variables, but group vars reach a lookup unrendered, so the flag arrived as its own template text and boolean() refused it. The torrc render aborted, and with it every tor row: 14 jobs across both runs on 095ea5e882 failed with the same line, Baserow#0 in compose and Nextcloud#1 in swarm among them.
Quoted from job 104843161609: The lookup plugin 'tor_extra_ports' failed: The value '{{ lookup('config', application_id, 'services.tor.forwards.ssh') | bool }}' is not a valid boolean. Reproduced locally: boolean() on that literal raises with the identical message.
tor_egress_clients.py:43 in this same role has resolved its own references through templar.template(trust_as_template(...)) since bf32f57b85. _flag now does the same and names the variable when the value is still not a boolean afterwards.
The unit tests passed real booleans, which is why they stayed green through seven gate runs; the new case sends template text through a templar, the shape production delivers. No lint or unit test could have caught this, because it only appears once ansible hands the variable over unrendered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The catalogue held three models and the development inventory carried a second, CI-only list that both AI roles pulled in through their variant 0. That made the model a variant concern and split one fact across a group_var, an inventory file and two meta files.
AI_LOCAL_MODELS now holds the single entry the CI list held, qwen2.5 0.5b instruct at 531 MB in Q8_0, which lms get is already proven to fetch in this stack. AI_LOCAL_MODELS_CI and both variant overrides are gone, so variant 0 states only container_backup and the memory limits.
The dropped entries have no consumer: llama3, mistral and nomic appear solely in test fixtures that stub the lookups, and nothing requests an embedding model, the only embed matches being the filters that exclude one when picking the chat model. LITELLM_CHAT_MODEL takes the first non-embed alias, so it resolves and LITELLM_USABLE stays true.
Note this changes production, not only CI: a deployment no longer pulls a 7B model or an embedding model. It also shrinks the pre-pull that burned 18796s in job 104622514450 from gigabytes to 531 MB.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The probe dropped every canonical and alias domain of a role whose server.status_codes.default named anything at or above 400, so a single by-design 4xx removed the whole vhost from CSP inspection. The checker now takes --accept-status <domain>=<code>, which keeps the page navigated and its policy evaluated, so the declaration buys coverage instead of costing it.
csp_accept_status reads the existing SPOT in utils/roles/applications/status_codes.py rather than restating the rule. That resolves per vhost, because server.status_codes is keyed like domains.canonical, so two domains of one role can differ; the old filter looked only at default and was blind to seaweedfs declaring its api, filer and master keys separately. The threshold is the checker's own: anything below 300 passes unflagged, so a declared redirect is named too.
csp_skip_domains keeps only the half that has no vhost to probe at all, a canonical domain owned by a disabled service, and gains the tests that half never had.
Measured: 14 domains move from skipped to probed, among them jenkins, mattermost, n8n, prometheus, mirror and xmpp. 28 domains carry an entry in total. Across the CI diagnostics of this run 473 of 476 observed responses were 200, the single infinito vhost answering 403 is api.seaweedfs, and it declares that code, so it stays green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
e1d513a3c0 replaced include=round_include with variant_scope in the payload resolve and left the loop variable without a reader, which ruff B007 rejects: 'Loop control variable round_include not used within loop body'. grep confirms the unpacking at matrix.py:388 is now its only occurrence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
round_variants was built by walking round_include. When a primary's variant switches its couplings off the planner resolves no prerequisites, round_include collapses to the primary, and no dependency gets an index at all. derive_includes then evaluates them at base config, where an unrendered Jinja coupling counts as enabled, so the closure inflates to everything reachable instead of what the variant declares.
Measured: svc-ai-litellm round 1 deploys 10 apps instead of 32, web-app-bigbluebutton round 1 and web-app-peertube round 1 drop from 34 to 10 and 11. Every other round of every role measured is unchanged, including all fourteen nextcloud rounds. The apps that leave are not dependencies of the round: svc-ai-litellm declares litellm, lmstudio, ollama and postgres and no sso at all, while bigbluebutton and peertube declare sso but switch it off in variant 1 together with dashboard, matomo, css, logout and prometheus.
complexity/cli.py:300 already documents the rule this makes true: the planner hands each pulled-in dependency round_index if round_index is below its variant count else 0, and pulls_at_zero implements exactly that. The coverage model was describing behaviour the planner only delivered for apps inside the include.
init.py filters the printed variant map to the apps actually shown, because round_variants now carries 266 entries and round 1 would otherwise print 112 of them on one line while ten apps deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two guards over four primaries. svc-ai-litellm, web-app-bigbluebutton and web-app-peertube are the three whose round-1 include collapses to the primary alone, which is the only shape under which a deployed dependency can lose its payload; web-app-nextcloud is the ordinary control. One planner call costs 1.4s, so sweeping all 117 variant-carrying roles would cost 160s and buy nothing the three collapsing rounds do not already cover.
The first attempt asserted round_include as a subset of the payload scope. That invariant only holds without disable: a clearnet row prunes the tor provider and everything reachable only through it, so the scope legitimately shrinks below the planner's include. The assertion now runs the other way, that every deployed app with a non-empty override receives it, which holds in both network modes.
Both guards were watched failing before being kept: cutting web-app-keycloak and svc-net-tor out of variant_scope turns them red, and the suite passes with disable set as well as unset.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm matrix resolved variant payloads over the planner's round_include while provisioning the closure derive_includes returns. The two agree until a variant switches its couplings off: the planner then resolves no prerequisites, round_include collapses to the primary, and every deployed dependency silently keeps its base config. For svc-ai-litellm round 1 that meant web-app-keycloak ran with the base realm.totp_enabled true, so CONFIGURE_TOTP became a default required action and every SSO persona sat in a 300s predicate timeout until the job hit its wall clock at 20086s.
payload.py states the intent this restores: the role pulling a dependency in is what decides how lean the dependency runs. variant_scope keeps the decision next to the closure it filters and excludes the three lab infrastructure roles, whose payload would replace group-membership expressions such as {{ 'svc-swarm-node' in group_names }} with an unconditional true in every round.
Measured: round 1 goes from 1 to 29 payload apps and keycloak from absent to totp_enabled false, while the deployed closure stays 32. Repo-wide 62 of 280 rounds gain a real setting, dominated by postgres exposed in 48 and mariadb in 22; no round loses one, and no gained payload enables a service whose provider is not deployed. With disable set the scope shrinks instead, which is correct because a pruned provider is not deployed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
playbook.yml:4 runs hosts: all under any_errors_fatal with no serial and no max_fail_percentage. When a task fails there, ansible/plugins/strategy/linear.py:329-333 marks every host that did not itself fail, while strategy/__init__.py:575-576 adds the failing host to _failed_hosts only when its next state is COMPLETE, which a rescue prevents. A rescued preload therefore leaves the stack host running alone through every later role while the workers and the nfs server are already marked failed.
The recovery is kept where it was measured: the stall came from a compose row, container exec -i lmstudio by name rather than a swarm task id. Swarm now re-raises the original failure as the first rescue task, the shape roles/web-app-keycloak/tasks/03_manager_ops/updates.yml:28-31 already uses in mirror image, and the nocheck marker moves with it because test_no_rescue_blocks.py:82 reads only the first rescue task's line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tor extra-ports test carried '# noqa: SLF001' on the three lines that set lookup._templar and lookup._loader, which the test must do because the plugin reads the variables it is handed from them. Ruff reports no violation there any more, so RUF100 removed the directives as unused during make autoformat.
A suppression that suppresses nothing asserts a conflict that does not exist and invites the next reader to treat the access as a problem.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tests/lint/ansible/roles/test_async_reason_documented.py requires a one-line '# rationale: async; ...' directly above every async key and offers no nocheck escape, because fire-and-forget is a deployment-wide decision that has to hold in swarm and stay playwright compatible. A comment audit removed the line while moving the task into utils/preload.yml, and make test went red on it.
The restored line answers the question the lint asks, naming the reader rather than the reason for the async: the reap two tasks below reads every job back before the play continues, so no model is still downloading when playwright starts. The marker is not narration; the comment lint accepts rationale: by design.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tests/lint/ansible/structure/test_no_rescue_blocks.py forbids every rescue block in roles/ and tasks/ unless its first task carries a same-line '# nocheck: rescue-<reason>' comment, and make test runs it. The preload recovery added in df2fc5084a had none, so the gate failed with exactly one offender.
The block meets the rule the lint documents rather than only its regex: it performs real in-play recovery, since a timed-out transfer leaves the daemon reporting 0.00% for every further pull and only restarting the container that owns it clears that, which the workflow-level capture cannot do; and it ends by preloading again, so a failure that recovery does not fix still re-raises.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A preload that fails leaves the daemon unable to move bytes: measured in CI job 104622514450, attempt 1 reached 437.37 MB of 531.07 MB and died with 'Download failed: Timed-out', then all five retries sat at 0.00% for a full hour each until CONTAINER_EXEC_TIMEOUT killed the client. Six executions, 18796s, no progress.
Killing the client does not touch the daemon that owns the transfer, so a plain retry cannot help. The rescue restarts the container, which a local probe showed restores a working transfer, then re-resolves the CID because swarm may reschedule the restarted task under a fresh id, and preloads once more. Pull and reap moved into one included unit so the rescue also covers the async path, where the failure surfaces at the reap rather than at the pull. retries drops from 5 to 1, since retries N yields N+1 executions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The skip list gated its whole loop body on the group_names selection, so a role reaching a node through a runtime include_role never contributed its server.status_codes.default. web-svc-mirror arrives that way on every Tor node and declares 404 as healthy; on a node provisioned from the svc-net-tor README the probe read that 404 as unreachable and hlth-csp could not start.
Measured in CI job 104622514568: the unit rendered without --skip-domain and the checker reported ERR_SOCKS_CONNECTION_FAILED on the mirror onion 26 times. Only the service-disabled half depends on the selection now. The widened branch can contribute at most 14 of 266 applications, and naming a domain no vhost serves is inert because script.py filters an nginx-enumerated list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule named roles/<role>/config/main.yml, which no role has: f65f4d60fb migrated the last one, web-app-pihole, to meta/services.yml and did not carry the README with it. A reader following the sentence found nowhere to put the value.
It now names the two real homes and the property that decides between them. meta/services.yml is overridable, because the inventory's applications block is deep-merged on top of it and lookup('config', ...) reads the result; vars/main.yml is not, because role vars outrank inventory variables. The clause about a value only one role reads today is the case that just went wrong: group_vars/all/19_tor.yml looked harmless for exactly that reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
group_vars/all/README.md forbids role-specific configuration in that directory, and 19_tor.yml held five variables no role but svc-net-tor ever read. They now live where the README points: the values an operator may override in meta/services.yml, where the inventory's applications block is deep-merged on top, and the derived ones in vars/main.yml.
The exception is the egress virtual range, which moves the other way into 08_networks.yml. It is not a property of the role but a claim on shared address space, and 0f6728a4c1 records what a collision costs: tor's upstream default 10.192.0.0/10 swallowed the docker pool at 10.208.0.0/12, and every connection to a container address came back as 'errno 104 Connection reset by peer'. The two numbers that must not overlap now sit one line apart.
The onion's HTTP forward becomes a declaration rather than a derivation. lookup('tor_ports') walks group_names and collects what those roles declare under ports.onion; port 80 belongs to svc-prx-openresty, which a node acquires through sys-stk-front-proxy and which therefore appears in no dependency list an inventory carries. A node provisioned with --include svc-net-tor published an onion forwarding SSH and nothing else, leaving every app it serves unreachable - measured as ERR_SOCKS_CONNECTION_FAILED in the CI instructions replay. The new role-local tor_extra_ports lookup composes the flagged forwards with the derived ones and collapses a port named twice, since torrc writes one HiddenServicePort line per entry.
Not moved: tor_ports stays in plugins/lookup because tor_reachable imports from it and three database roles use that; and the flags sit outside ports.onion because every key there mirrors a port the role declares under ports.local, which neither sshd nor the front proxy is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-net-tor aborts at roles/svc-net-tor/tasks/router.yml:2 when networks.internet.dns is empty, and rightly so: dnsmasq runs with no-resolv, so an empty upstream leaves the node answering .onion and nothing else. That value is set in exactly two places, both under inventories/development/, and the documented production path reaches neither - the provisioner writes no networks key at all. A reader following the role README verbatim therefore cannot deploy a Tor node.
CI has been proving this since 2026-08-31 without anyone seeing it: the README replay rides on the role's smallest variant and is skipped when the deploy is red, so the only row that ever reached it is one where svc-net-tor itself went green. That happened first on Ubuntu, where the local-service defect abb6f37297 fixes never fired.
The template now emits the DNS assignment, the -e passthrough and the networks entry in --vars for a role whose tasks name networks.internet.dns, which today is svc-net-tor alone; the generator derives that rather than taking a declaration, so a second role asking for the value documents it by doing so. scripts/github/guide/compose_deploy.sh substitutes the placeholder from INFINITO_DNS_IP, the same SPOT inventories/development/default.yml reads, next to the existing domain substitution.
This follows 972596a758, which added the administrator key to the same block for the same reason. No fallback inside the role: defaulting to NETWORK_PUBLIC_DNS_RESOLVERS would route a Tor node's clearnet queries to Cloudflare without the operator choosing it, which is what the assert exists to prevent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The target decides its matrix from two ends. ACT_MATRIX picks from the finished rows, ACT_INPUTS decides which rows exist at all. 86033b63f1 drew that distinction for the mode axis and pinned modes=swarm; the variant axis stayed open, so the whitelist reached discovery unpinned and only variant 0 was ever produced.
With variant=1 the intersection was therefore empty: act found no matrix entry for the deploy job, ran discovery alone and exited 0 - the same green-while-drilling-nothing shape 9e06f4c9a6 set out to refuse. Its guard did not catch this one because it validates that the selection token names a real row, not that discovery will emit it under the inputs the recipe passes.
Measured: before the change, 'make swarm-zombie app=svc-net-tor variant=1' ended after the discovery job with a plan naming variant 0 and no deploy; after it, the same invocation runs the deploy job for that row.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The systemd drop-in is written inside a block gated on TOR_DNSMASQ_OWNS_LISTENER, which turns false as soon as a foreign file sits in /etc/dnsmasq.d. The start task read that block's register unconditionally, so the expression's correctness depended on ansible defining a register for a task it skipped. Short-circuiting on the same condition means the register is never reached when the block did not run, and the question stops mattering.
The test asserted the literal declaration string, which exercises nothing. It now renders the expression read from the role with ansible's own bool filter and a changed test, in both states: owning the listener with a rewritten drop-in yields True, not owning it yields False without touching the register. The previous form raises UndefinedError under the second case.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dnsmasq binds the docker bridge address with the clearnet upstreams configured and the node's own .onion records loaded, and no rule covered port 53. Any source that could route a packet to that address got an open resolver plus the split-DNS records. c89ca42c46 claimed local-service closed this; abb6f37297 showed the keyword never took effect.
The guard now carries two named sets instead of one plus a special case. 'containers' is the docker pool list, for tor's own transparent and DNS ports. 'internal' is NETWORK_INTERNAL_CIDRS, for the resolver and the SOCKS port. Every guarded entry names its set, which folds the hand-written SOCKS rule and its manual append into the same list the reverse-path sets are built from.
The resolver takes the wider set deliberately: 'internal' is the list torrc.j2 already spends on SocksPolicy accept. The narrow set carries only 10.208.0.0/12 and 10.7.7.0/24 out of 10.0.0.0/8, which leaves out swarm's ingress network, every overlay docker allocates without an ipam block, and a node whose own address sits elsewhere in 10/8. A query bound to such an address was measured timing out against the narrow set. Guarding the resolver more strictly than the proxy beside it would refuse a name lookup to a source that is handed a full SOCKS proxy six lines earlier.
NETWORK_DNS_PORT lives next to the other resolver constants rather than in meta/services.yml: ports.local describes host-bound container ports, where 53 collides with svc-dns-unbound and falls outside every PORT_BANDS band.
Measured on the rendered artifact: the table loads and reloads with a byte-identical listing, every compose source of port 53 resolves inside 'internal' and every source of 9040/9053 inside 'containers', and a mistyped set name, an empty CIDR list or a missing admitted key each fail loudly at validate time. Port 53 newly passes a strict reverse-path check, so an asymmetrically routed client loses name resolution.
test_fragments built its guard list from a fixture of three entries while the role shipped six, so it validated the template against a port list nobody deploys. It now takes the shape from the declaration and fills only the numbers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
EL10 and Fedora ship an active local-service=host in /etc/dnsmasq.conf. The drop-in repeated the keyword, and dnsmasq refuses a repeated ARG_ONE option: 'illegal repeated keyword at line N of /etc/dnsmasq.d//tor-onion.conf', then 'FAILED to start up'. Eight CI rows died at roles/svc-net-tor/tasks/router.yml:67, all of them on centos or fedora; arch and ubuntu start dnsmasq fine because their stock config leaves the keyword commented.
The line never took effect on any distribution. dnsmasq resets OPT_LOCAL_SERVICE as soon as an access-control option is present, and listen-address fills exactly that list; the template emits listen-address=127.0.0.1 unconditionally in the same block. A query from a source outside every directly attached subnet is answered identically with and without the line.
Port 53 therefore carries no source restriction. That is unchanged here and belongs in the nftables guard, which currently covers only the trans, DNS and SOCKS ports of tor itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SOFTWARE_NAME, SOFTWARE_DOMAIN, SOFTWARE_URL and ORGANIZATION are declared once in group_vars/all/00_general.yml and derive from each other. Eleven templates wrote the value instead, so a rename would reach the four declarations and every consumer that reads them, and silently miss the literal.
The lint reads the name from the declaration rather than carrying it, and searches case-insensitively: the domain is the name lowered, the URL is https:// plus the domain and the organization is the name again, so one literal finds the value of all four. A hit is reported with the variable it should have been, derived from the scheme in front of it and its case. The escape is nocheck: software-identity, same line or the one above.
Every migrated template renders the same bytes as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The line that mocks the role_subnets dependency exceeded the line length; the project formatter broke it across three. No behaviour changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible warns that the name is already in use and carries on. Carrying on is the problem: inside the loop the original value is gone, so every lookup, template and nested include reading that name sees the loop item instead, and the task still reports success. The damage lands in a log nobody greps.
The lint collects every key declared in group_vars and in each role's vars or defaults, then reports any loop_var matching one. A denylist of known-bad names would have found application_id and missed domain, which is the same class in four more places.
All seven existing sites are marked, each with the reason read at the source rather than inferred. The three application_id loops pass the role id as the item, and the include_vars that follows loads that role's own application_id with the same value, so both meanings coincide by construction. The four domain loops hand the vhost to sys-stk-front-proxy, whose tasks read front_proxy_domain from exactly that name, so the shadowing is the interface.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A fragment declares its own nftables table. An accept there terminates that table's base chain and nothing else: the packet still traverses every other base chain at the same hook, and a drop in any of them wins. Docker sets the filter FORWARD policy to drop by default, so a fragment that accepts is not permitting anything, it is declining to decide.
Measured rather than argued. Converting a role's appended iptables FORWARD accept into a fragment rule at priority -10 delivered 0 of 3 forwarded packets under that policy where the rule it replaced delivered 3 of 3, and the fragment's own counter showed the accept had matched all three. Both versions deployed green, which is why this belongs in a lint and not in a paragraph.
The message names both ways out: express the intent as a drop of what must not pass, or put the rule in the chain that carries the policy it has to overrule.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard drops every source outside the list tor_egress_clients returns, and containers reach the TransPort and the DNSPort through exactly those rules. Nothing held that relationship: narrowing a pool, renaming a key or dropping an entry from the lookup would have taken the resolver and the transparent proxy away from every container at once, in production.
The test runs the real lookup over the repository's real network declarations rather than a fixture, so it fails where the mistake is made. Its templar resolves the cross references those declarations carry, because without it the lookup would compare unrendered template text and the check would pass on nothing.
It also pins the decision that the admitted list is deliberately narrower than SocksPolicy: 10.0.0.0/8 stays out, because a transparent proxy and a resolver with no access control of their own do not belong to the whole LAN. Left unwritten, that reads like an oversight and gets repaired on the next pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard trusted an address. rp_filter ships at all=0 and default=2 on this host and the repository sets it nowhere, so loose mode accepts a packet whose source is routable via any interface at all. A datagram claiming a docker pool address and arriving on the uplink passed both the kernel check and the admitted set, which for udp needs one forged packet and no handshake.
A strict reverse-path check on the guarded ports closes it. Measured in a namespace rig: a forged source routable via another interface reaches the input hook, because loose rp_filter lets it through, and the new rule counts it; a source routable nowhere never gets that far, the kernel drops it first. A loopback datagram to the same port does not match, so the host's own .onion resolution through 127.0.0.1 to the DNSPort is untouched.
Scoped to the four guarded ports rather than the whole chain. Strict reverse-path filtering on everything traversing input and forward breaks asymmetric routing, and this role runs on hosts that terminate wireguard tunnels.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An audit of what the host actually filters found a single input chain in the whole ruleset, covering three tor ports. ss showed two more binds with nothing in front of them: the SOCKS port on 0.0.0.0, and dnsmasq on the bridge address.
Each gets the layer it belongs in. SOCKS gets a packet filter rule against the same CIDRs SocksPolicy already accepts, so no client loses access while the filter closes if tor's own config ever stops enforcing. Reusing the narrower guard set was deliberately avoided: it excludes the LAN blocks SocksPolicy admits, so it would have changed the policy rather than hardened it.
dnsmasq gets local-service, which answers only directly connected networks. A packet filter rule there would have sat on the container DNS path, which daemon.json points at that same bridge address, in exchange for closing an exposure that needs an extra route onto the bridge.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A templated script is opaque to every tool that reads shell. shellcheck parses Jinja as syntax errors, the repository's own .sh guards skip the file because it is not a .sh, and nothing can execute it to test it: what runs on the host never exists in the repository, so a defect in the rendered half is found by deploying it.
The lint reports any *.sh.j2 under roles and points at the static slot sys-service already resolves, files/shell/script.sh, with the values passed as ExecStart arguments or Environment entries the way this repository's python scripts already take them.
None of the seven templates present meets the bar the marker is for, which is a script whose shape is generated rather than its values. Each is grandfathered with a marker naming what it interpolates, because converting a deployed script to argv changes what runs on the host. The guard exists to stop the next one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sys-service resolves the template, writes it into the unit directory, reloads the daemon, enables and starts the unit, and carries the timer, async and flush behaviour. A role that renders its own service.j2 with a bare template task gets the file and none of the rest, and every one of those omissions is silent: a unit that exists but was never enabled looks installed until the host reboots.
The lint reports any src pointing at a service.j2 inside a role's tasks. desk-ssh was the one finding left after sys-svc-firewall moved over, and it migrates here: its ssh-agent unit now comes from sys-service in the user scope, with the type, ordering, environment and Install section supplied as template fields. RuntimeMaxSec is pinned empty on purpose, because the default applies it to anything that is not oneshot and would kill a long-running agent after a day.
The unversioned unit is retired first. sys-service names units by version, so the old file would otherwise stay behind enabled and a second agent would bind the same socket.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role advertised holding a forwarding or masquerade rule for a tunnel. A fragment cannot do that: it lives in its own table, and an accept there terminates only its own base chain, so a DROP policy another component set still executes. svc-net-wireguard-firewalled was measured losing every forwarded packet that way.
The use case is replaced by one the design does serve, and the rule that decides it is written down where the next author will look rather than left in a commit body.
The fragment test no longer seeds the two wireguard interface names; the fragment they described is gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role ran one shell line, iptables -A FORWARD -i wg0-client -j ACCEPT followed by iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE. An append checks nothing, so every deploy added the same two rules again and both chains grew for the life of the host.
The iptables module probes with -C before it inserts, so the rules converge instead of accumulating, and the two interface names move out of the shell string into variables.
The mechanism stays iptables deliberately. An nftables fragment in its own table cannot do this job: an accept there terminates its own base chain and nothing more, so docker's DROP policy on the filter FORWARD chain still executes and the forwarded packet dies. Measured in a namespace with a real forwarding rig: the appended rule delivers 3 of 3 packets under that policy, the fragment delivers 0 of 3, with the fragment's own counter showing the accept matched all three. The inet family would also have started masquerading IPv6, which iptables -t nat never touched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-net-tor rendered its own guard script, its own unit and its own handler, and hooked the rules into DOCKER-USER with an insert, because a rule appended to a chain docker owns lands behind the RETURN docker puts there. The script flushed its chain and refilled it rule by rule, so the boundary was gone for the duration of the refill.
A role that needs packet filter rules now writes templates/nftables.conf.j2 declaring its own table and includes svc-net-firewall, which renders it to /etc/nftables.d/<role>.conf, rejects it with nft -c, loads it, reads the table back and refuses one that carries no chain. Deployed on its own the role installs nftables and the ruleset that loads every fragment at boot. That ruleset is a distribution decision, so its path follows os_family: RedHat reads /etc/sysconfig/nftables.conf, the others /etc/nftables.conf. The unit is only enabled, never started, because on Debian its ExecStop flushes the whole ruleset, docker's tables included.
One table per role removes the shared path: a fragment replaces its own table and touches no other, and docker never touches either. nft -f on an existing table merges, so each fragment creates, deletes and redeclares its table in one transaction. The tests load every fragment twice into a throwaway network namespace and require the listing to match, because a merging fragment loads twice without error and shows up only as a duplicated rule.
The tor guard now applies in both deploy modes. It was pinned to swarm while torrc binds TransPort and DNSPort on the bridge address in compose too, so on a single-host install those ports answered every on-link neighbour. The guard drops rather than admits, so nothing short-circuits another table, and ip saddr carries an implicit ipv4 dependency, which leaves the ipv6 exposure exactly where the iptables rules left it. svc-net-tor reads its own chain back and refuses a guard that drops nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lint-python runs ruff, which the pytest lint suite does not cover, so these
slipped through locally: an unnecessary lambda around bool, and a Jinja
Environment left on the implicit autoescape default. The suppression has to sit
on the autoescape argument itself, which is where ruff anchors the violation.
The generic unit template could express neither an ordering dependency nor an Install section, so a role needing either shipped its own templates/systemctl.service.j2 and re-answered the daemon reload, the enable and the flush behaviour by hand.
Add After, Before, Wants, RemainAfterExit and a WantedBy driven Install section, all inert at their defaults, and route the unit path, its owner, become and the systemctl probes through a system_service_scope so a per-login unit lands under the user's own systemd directory.
00_core is gated on the system scope rather than only its prune and lockdown steps: it sets run_once_sys_service, so a user-scope caller running first would otherwise skip the lockdown for every system-scope role behind it in the same play.
Rendering the previous template against the same inputs drops no directive and adds none; the only difference is the blank line an empty directive used to leave behind.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sending to a .onion needs two things of the node: a resolver that answers the suffix, and a route from the container to Tor's transparent port. Neither existed. Compose pointed Mailu at unbound, which knows nothing about .onion, and in swarm Tor's DNSPort and TransPort listen inside the container where the host's dnsmasq and nat rules cannot reach them, so the forwarding rules that were already there pointed at a closed port on every swarm node.
Compose resolves through unbound, which now forwards the onion zone to Tor's DNSPort on the bridge it already owns. The forward alone would have answered SERVFAIL: the signed root proves that onion does not exist, so with auto-trust-anchor-file set every answer below it validates as bogus, and Tor's DNSPort can supply no DS. The zone is therefore declared insecure. Swarm resolves through the node dnsmasq instead, which every container already uses because container_dns puts the bridge resolver into daemon.json wherever svc-net-tor is deployed, and which now reaches Tor because its ports are published host-mode next to the SOCKS port that was published there already.
The transport rule changed target rather than mechanism. REDIRECT rewrites the destination to the primary address of the interface the packet arrived on, which is a different project bridge for every role, so Tor would have had to listen on all of them. A DNAT naming the bridge keeps one listener, and in swarm the same address reaches the published port through the proxy docker runs there.
That leaves the exposure. SocksPort has SocksPolicy; DNSPort and TransPort have no equivalent, and a swarm-published port carries no host address to bind to, so in swarm the packet filter is their only boundary. Compose needs none, because there Tor binds loopback and the bridge and nothing else. The swarm guard admits this deployment's own address space and drops the rest, and it is installed as a unit ordered after docker rather than as ansible-applied rules: nothing here persists netfilter state, so docker recreating its chains at boot would have left Tor back on the wildcard with the guard gone. It rebuilds its chain instead of appending to it, because the appending version could only ever be loosened. It hooks INPUT and DOCKER-USER both, since host-local traffic reaches a published port through docker's proxy and external traffic through FORWARD.
The deploy now proves all of it back and fails when a rule is missing, the router no longer runs on swarm nodes that carry no Tor, and the CLI test probes the forwarded ports over SOCKS instead of assuming a HiddenServicePort line means a reachable service.
Measured so far: Tor accepts the non-loopback binds and reports circuit-established, and the chain, its order and the DOCKER-USER hooks read back correctly from a deployed host. The swarm path is not yet exercised.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b036f6003 gave svc-prx-openresty the ports it had been binding as literals, including ports.public.http and the onion flag that turns the node onion's port 80 into a declaration instead of a hardcoded torrc line. The matching lint change stayed in the working tree, so that commit does not pass its own gate: the onion-flag lint excludes http wholesale, never finds the category among the declared ones, and reports the flag as out of scope.
The exclusion was right for the 128 roles that declare http, sso or websocket on the loopback and are reached through the proxy on the onion's port 80. It was wrong for the single role that publishes http on the host, because that role is the proxy. The lint now splits on scope rather than on category: exempt under local, decision required under public.
The companion lint gets the same treatment it needed for the dict-form call sites: the ports block is tracked by the indentation of its key rather than by the next line that looks like one, since target, published, protocol and mode are all valid keys and closed the block while it was still being read. Under expose: only a real host:container mapping counts, because there the single-segment form documents a container port and publishes nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make lint-shellcheck walks files on disk, so a .sh.j2 is invisible to it. The thing that actually runs on the host is the render, and nothing looked at the render until it failed on a node. That is an unpleasant place to discover a quoting mistake in a script which runs as root and rewrites the host packet filter.
Every roles/*/templates/*.sh.j2 is now rendered and handed to bash -n, plus shellcheck where it is installed. A value the check does not know renders as a placeholder token, an unknown collection renders as empty, an unknown filter passes its value through, and lookup returns a token, so a template written for ansible still renders to whole bash and the check judges the shape of the script rather than the values one deployment happens to supply.
SC2034 is excluded with its reason: emptying a conditional branch orphans the variables that branch used, so a partial render cannot tell a genuinely unused variable from one whose only reader was not rendered. Everything else stays on.
First run, first finding: the backup ssh wrapper substituted with backticks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
