The onion port probe gave every SOCKS connect 60s, half of Tor's
SocksTimeout (120s by default; torrc does not set it), although its own
docstring argued that abandoning an attempt before SocksTimeout is
destructive. Every attempt authenticates with fresh credentials, so Tor
builds a whole new rendezvous for it and keeps working on it until
SocksTimeout. The probe walked away first and called the port dead. The
other raw onion connects in the repository, the svc-db-* CLI tests and
test-e2e-playwright's wait_onion_reachable.sh, wait 120s per attempt.
Run 40128751852 shows how often an attempt needs more than that. Of 178
svc-net-tor CLI runs in its 89 compose onion jobs, three lost their
first attempt to the 60s timeout and connected on the second within
seconds (Shopware#1, Hugo#1, n8n#1; the CLI test took 77-79s against
the usual 1-16s), and Hugo#1's other run took 40s on a single attempt. When
the stall outlasts the 70s until the second attempt, the probe fails
while the HTTP probe reaches the same onion port seconds earlier:
OpenBao#1 (run 39166850273, job 118050491920, the original failure of
#1048) and Socialhome#1 (run 40088473634, job 121019758461). In the
Socialhome job tor.log holds no line for either attempt, because Tor
never reached its own verdict.
ONION_PORT_TIMEOUT is now 130: SocksTimeout plus room for Tor's
once-per-second expiry sweep, so an attempt ends with Tor's own answer
(reply 0x06, and a "Giving up" notice in tor.log, which the rescue
artifact captures) instead of the probe's. The second attempt now starts
at 140s and ends at 270s, so the probe rides out every stall the old
budget did, and any stall shorter than about 260s instead of about 120s.
A green run still costs one connect per port, and a refused or unmapped
port still fails at once. A unit test pins the window above Tor's
SocksTimeout and fails as soon as torrc sets its own.
That doubles what a dead port costs, and meta/tests.yml caps the whole
CLI test. The largest compose onion deploy in run 40128751852 probes 19
domains and 5 ports; with everything dead that is 19 x 350 + 5 x 260 +
10 = 7960s, beyond the 7200s that 069df39027 set for the same reason.
9000 keeps the full failure output for up to 22 domains.
Closes#1214
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The check listed the torrcs without the limit through "grep -L ...; true".
With no torrc under /opt/chutney/net/nodes, or with one that cannot be
read, grep printed nothing, the trailing true turned its error into
success, and the test reported that every node runs one worker.
The check now counts the limit per file with "grep -cH" and reads the exit
status. A status above 1 or an empty answer fails the test, and a file
with a count of 0 is named as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Tor starts its worker threads by the number of CPU cores it detects, and
the cpus limit of the container does not change that number. On a host
with 20 cores every node of the private chutney network ran 21 threads and
grew to about 0.6 GB. The seven relays and authorities filled the 2g
limit and its swap within ten minutes, the kernel killed them one by one,
and no circuit could be built any more.
The entrypoint now writes NumCPUs 1 into the torrc of every chutney node
before it starts the network. Measured on that host over ten minutes: the
nodes run 3 threads and stay at 65 to 86 MB, the container at 0.47 GB
without swap, where it had filled 2 GB plus 2 GB of swap before.
The CLI test of the role fails when a chutney node runs without the limit.
Closes#824
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Tor attaches a stream to an onion to the rendezvous circuit that matches its
isolation and keeps it there until SocksTimeout; it never moves the stream to a
fresh circuit. The probe connected without SOCKS authentication and gave up
after 60s, so a dead rendezvous circuit would absorb both passes. That is the
most likely reading of run 35661520542 (web-app-mastodon#1, public Tor), where
<onion>:80 timed out twice right after curl had reached the same onion port.
Authenticate every connect with fresh random username/password credentials.
Tor accepts any and, under its default IsolateSOCKSAuth, builds a separate
rendezvous circuit per credential set, so a retry can no longer land on the
circuit that failed the attempt before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-net-tor rendered its own guard script, its own unit and its own handler, and hooked the rules into DOCKER-USER with an insert, because a rule appended to a chain docker owns lands behind the RETURN docker puts there. The script flushed its chain and refilled it rule by rule, so the boundary was gone for the duration of the refill.
A role that needs packet filter rules now writes templates/nftables.conf.j2 declaring its own table and includes svc-net-firewall, which renders it to /etc/nftables.d/<role>.conf, rejects it with nft -c, loads it, reads the table back and refuses one that carries no chain. Deployed on its own the role installs nftables and the ruleset that loads every fragment at boot. That ruleset is a distribution decision, so its path follows os_family: RedHat reads /etc/sysconfig/nftables.conf, the others /etc/nftables.conf. The unit is only enabled, never started, because on Debian its ExecStop flushes the whole ruleset, docker's tables included.
One table per role removes the shared path: a fragment replaces its own table and touches no other, and docker never touches either. nft -f on an existing table merges, so each fragment creates, deletes and redeclares its table in one transaction. The tests load every fragment twice into a throwaway network namespace and require the listing to match, because a merging fragment loads twice without error and shows up only as a duplicated rule.
The tor guard now applies in both deploy modes. It was pinned to swarm while torrc binds TransPort and DNSPort on the bridge address in compose too, so on a single-host install those ports answered every on-link neighbour. The guard drops rather than admits, so nothing short-circuits another table, and ip saddr carries an implicit ipv4 dependency, which leaves the ipv6 exposure exactly where the iptables rules left it. svc-net-tor reads its own chain back and refuses a guard that drops nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sending to a .onion needs two things of the node: a resolver that answers the suffix, and a route from the container to Tor's transparent port. Neither existed. Compose pointed Mailu at unbound, which knows nothing about .onion, and in swarm Tor's DNSPort and TransPort listen inside the container where the host's dnsmasq and nat rules cannot reach them, so the forwarding rules that were already there pointed at a closed port on every swarm node.
Compose resolves through unbound, which now forwards the onion zone to Tor's DNSPort on the bridge it already owns. The forward alone would have answered SERVFAIL: the signed root proves that onion does not exist, so with auto-trust-anchor-file set every answer below it validates as bogus, and Tor's DNSPort can supply no DS. The zone is therefore declared insecure. Swarm resolves through the node dnsmasq instead, which every container already uses because container_dns puts the bridge resolver into daemon.json wherever svc-net-tor is deployed, and which now reaches Tor because its ports are published host-mode next to the SOCKS port that was published there already.
The transport rule changed target rather than mechanism. REDIRECT rewrites the destination to the primary address of the interface the packet arrived on, which is a different project bridge for every role, so Tor would have had to listen on all of them. A DNAT naming the bridge keeps one listener, and in swarm the same address reaches the published port through the proxy docker runs there.
That leaves the exposure. SocksPort has SocksPolicy; DNSPort and TransPort have no equivalent, and a swarm-published port carries no host address to bind to, so in swarm the packet filter is their only boundary. Compose needs none, because there Tor binds loopback and the bridge and nothing else. The swarm guard admits this deployment's own address space and drops the rest, and it is installed as a unit ordered after docker rather than as ansible-applied rules: nothing here persists netfilter state, so docker recreating its chains at boot would have left Tor back on the wildcard with the guard gone. It rebuilds its chain instead of appending to it, because the appending version could only ever be loosened. It hooks INPUT and DOCKER-USER both, since host-local traffic reaches a published port through docker's proxy and external traffic through FORWARD.
The deploy now proves all of it back and fails when a rule is missing, the router no longer runs on swarm nodes that carry no Tor, and the CLI test probes the forwarded ports over SOCKS instead of assuming a HiddenServicePort line means a reachable service.
Measured so far: Tor accepts the non-loopback binds and reports circuit-established, and the chain, its order and the DOCKER-USER hooks read back correctly from a deployed host. The swarm path is not yet exercised.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The probe abandoned each attempt at --max-time 30 while Tor's SocksTimeout
sits at its 120s default (no SocksTimeout anywhere in the repo; positive
control SocksPort hits templates/torrc.j2:1). It gave up at a quarter of
Tor's working window, and destructively: every retry opens a fresh SOCKS
connection, discarding the descriptor fetches and rendezvous retries Tor
was still working through.
Run 32284541284 job 96178512465 shows the cost. Ansible recorded start
07:22:17.260, end 07:26:33.366, delta 0:04:16.105994. The first domain
burned its deterministic 6x30 + 5x10 = 230s and failed; the second then
succeeded on its first attempt but needed ~23s of that 30s window - 77%
of the limit. The transient had cleared at t~233s. The first domain's
budget ran out three seconds short of it.
Two round numbers, one behaviour: 5 attempts at 60s is 340s of patience
against today's 230s, so the same transient now has 107s of margin
instead of -3s, and a 23s connect sits at 38% of the window instead of
77%. LAM#1 (job 96179137200) is the near-miss that corroborates it - four
dead attempts, then a ~4s success on the fifth.
meta/tests.yml is required, not optional. roles/test-e2e-cli/tasks/run_one.yml:74
runs the script under `timeout {{ _cli_timeout }}`, and with no meta/tests.yml
this role fell through to the 3600s default in roles/test-e2e-cli/defaults/main.yml:3.
Widening the window moves the all-domains-fail ceiling from 16 domains to
11, and a 13-domain role demonstrably exists (Jellyfin#0, four passes of
"13 probed, 0 failed"). 7200 restores the headroom.
The `|| true` goes with it: the script runs under `set -uo pipefail` with
no -e, so the command substitution's exit code was already discarded and
line 63 branches on ${code}. tests/lint/filesystem/shell/test_no_or_true.py
never saw it because its regex wants #, ; or EOL after `true` and a `)`
follows here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Job 94849355639 of run 31823845576 failed on exactly one name: the bare apex
infinito.example, HTTP 000, "Could not resolve host", while
espo.crm.infinito.example answered 200 in the same second through the same
resolver. The probe printed dns= and stopped there, so the log named the
symptom and no artifact carried the cause. There is no dnsmasq log
(log-queries is off, the journal has zero dnsmasq lines), no CoreDNS log in
the compose artifact set, and no query output from the host at failure time.
Reproducing the resolver stack locally did not close it either. CoreDNS on the
CI Corefile, dnsmasq on the drop-in pulled byte-identical from that job's own
--diff at 20:58:35, and the full dnsmasq -> 127.0.0.11 -> CoreDNS chain with a
live forward path all answer the apex with 172.30.0.10. The configuration is
exonerated; what is missing is the transaction itself.
So the failure branch now prints the active resolver, any /etc/hosts match,
what the local dnsmasq answers, and what each upstream answers. The last two
are the split that matters: they separate "dnsmasq dropped it" from "upstream
dropped it", which is the fork the bench could not settle. Upstreams are read
from the drop-in directory rather than one file, so a shared dnsmasq carrying
foreign drop-ins is covered, and matching only bare-IP server= lines skips
server=/onion/127.0.0.1#9053.
TOR_DNSMASQ_CONF arrives through test.env instead of a hardcoded
/etc/dnsmasq.d, using the same :? idiom 8d7fff4d2 gave the other four values,
so the path keeps its single source in vars/main.yml. dig is opportunistic:
without bind-tools the resolver and hosts lines still land.
Verified: bash -n and shellcheck clean, the failure path exercised end to end
against a CI-shaped drop-in with 127.0.0.11 extracted and the /onion/ line
correctly skipped, test_comments_valid passed.
Two things this does not close. dig is probably absent on the Arch target that
failed, since the bind9-dnsutils lines in that job come from a Debian build
stage, not the target; adding bind-tools to the role would fix that and is a
separate concern. And the trigger is now rare, because pinning tor true across
the espocrm variants removed the mixed round that put the clearnet apex into
the vhost list at all. This is insurance, not a plan that pays out next run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
roles/svc-net-tor/templates/test.env.j2 sets RETRIES=6 and SLEEP_SECONDS=10, and
test-e2e-cli/tasks/run_one.yml renders that file unconditionally at :66 before
sourcing it at :74. The ${RETRIES:-20} / ${SLEEP_SECONDS:-15} fallbacks could
therefore never be reached -- a second source of truth that additionally
documented the wrong numbers: the header promised "default 20 / default 15"
while every run used 6 and 10.
They now take the :? form the two variables above them already use, so
test.env.j2 is the only place the values live, and a missing one aborts loudly
instead of silently probing three times longer than intended. The header stops
claiming defaults.
The sibling probes in svc-db-mariadb, svc-db-openldap and svc-db-postgres keep
${RETRIES:-20}: their test.env.j2 does not set the pair, so there the fallback is
live, not dead. Converting them would mean adding the values to those templates
first, and is deliberately left out.
Measured: a missing RETRIES exits 1 with "line 24: RETRIES: pass RETRIES as env
(attempts per domain, from svc-net-tor test.env)"; a complete env leaves the
probe output unchanged. shellcheck is clean against the staged content, and the
comment, shell and cli-asset-pairing lints pass.
The message wording avoids an apostrophe on purpose: inside ${VAR:?...} bash
still treats one as a quote character, which splits the abort message across the
next source line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HTTP 000 cannot tell a name that never resolved from a listener that never
answered. Both leave curl without a status and openresty without an access-log
entry, which is exactly the signature the apex infinito.example has produced in
every compose job of run 31543714113.
2a60ddb3c already stopped discarding curl's stderr, so the message now names the
layer -- (6) could not resolve, (7) failed to connect, (35) handshake. It never
names the address, and that is the half that decides between "the resolver
answered wrong" and "the resolver did not answer".
getent hosts goes through the same NSS path curl uses, so the verdict carries
what the node itself resolved:
[FAIL] localhost HTTP 000 dns=::1
[FAIL] no-such-host-xyz.invalid HTTP 000 dns=
Empty means resolution failed. An address together with a connect or TLS error
means the resolver answered and the far side did not. An address other than the
node's own means the resolver answered wrongly. Read next to curl's line, those
three branches separate.
Only the failure path changes; [OK] keeps its format, and getent's own noise
goes to /dev/null so a probe never fails on the diagnostic.
Measured: both branches above; getent and head present in all four node images
(fedora, arch, centos, debian) via docker run; shellcheck clean against the
staged content; tests/lint/repository/test_comments_valid.py,
tests/lint/filesystem/shell/ and test_cli_test_assets_paired.py pass.
This is not the fix for the apex failure. Nine mechanisms were eliminated by
measurement today -- CoreDNS, dnsmasq and Docker's 127.0.0.11 each resolve both
the apex and its sibling to 172.30.0.10; the vhost exists in http and https;
nginx validated six times; no server-name conflict, no proxy env, no conntrack
pressure. What remains needs the one fact the log has never carried, and this
line carries it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The apex infinito.example has failed six compose jobs with HTTP 000 and no
explanation, because the probe throws curl's stderr away:
curl ... "${scheme}://${domain}/" 2>/dev/null || true
--show-error was already set, so curl had the answer the whole time and the
redirect discarded it. Command substitution captures stdout only, so
%{http_code} and the accept regex at :61 are untouched, and || true still
swallows the exit code. test-e2e-cli/tasks/run_one.yml:85-92 prints stdout and
stderr when rc != 0, so the text reaches the job log.
This is NOT the fix for that failure. Three rounds of reasoning about it rested
on the premise that the apex answers 301 and curl follows it off-instance;
openresty.log:126 refutes it. That line logs the successful clearnet probe
"GET / HTTP/2.0" 200 with "Scheme: https" and User-Agent curl/8.18.0, so https
requests do appear in that log -- and the apex left no entry at all. curl never
completed hop 1, and there never was a 301 to follow. Whether the cause is
NXDOMAIN, a refused connect, or the 443 default_server's ssl_reject_handshake
(sys-svc-webserver-core/templates/default_server.conf.j2:17-20, which produces
exactly this signature: rejected handshake, 000 at curl, no access-log entry)
is what one curl error line will now say.
Costs up to six extra stderr lines per failing domain, RETRIES=6.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role shipped a complete probe script at files/test.sh and no
templates/test.env.j2. discover_cli_roles globs the env template alone,
so the role was never discovered and the script had never run — and the
script hard-requires TOR_SOCKS and NGINX_SERVERS_DIR via ${VAR:?}, so it
could not have run without one either. Its own usage hint pointed at a
path that does not exist.
Moving the script to the discovered path and adding the env template
makes both halves line up. Verified against the live stack: the onion
answers through the Tor SOCKS proxy, 1 probed, 0 failed.
The new lint keeps the pair together and rejects a script left at
files/test.sh. The two halves fail differently and only one is loud: an
env template without a script aborts the play, a script without an env
template is never discovered and its silence reads like a pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5.5