The onion port probe gave every SOCKS connect 60s, half of Tor's
SocksTimeout (120s by default; torrc does not set it), although its own
docstring argued that abandoning an attempt before SocksTimeout is
destructive. Every attempt authenticates with fresh credentials, so Tor
builds a whole new rendezvous for it and keeps working on it until
SocksTimeout. The probe walked away first and called the port dead. The
other raw onion connects in the repository, the svc-db-* CLI tests and
test-e2e-playwright's wait_onion_reachable.sh, wait 120s per attempt.
Run 40128751852 shows how often an attempt needs more than that. Of 178
svc-net-tor CLI runs in its 89 compose onion jobs, three lost their
first attempt to the 60s timeout and connected on the second within
seconds (Shopware#1, Hugo#1, n8n#1; the CLI test took 77-79s against
the usual 1-16s), and Hugo#1's other run took 40s on a single attempt. When
the stall outlasts the 70s until the second attempt, the probe fails
while the HTTP probe reaches the same onion port seconds earlier:
OpenBao#1 (run 39166850273, job 118050491920, the original failure of
#1048) and Socialhome#1 (run 40088473634, job 121019758461). In the
Socialhome job tor.log holds no line for either attempt, because Tor
never reached its own verdict.
ONION_PORT_TIMEOUT is now 130: SocksTimeout plus room for Tor's
once-per-second expiry sweep, so an attempt ends with Tor's own answer
(reply 0x06, and a "Giving up" notice in tor.log, which the rescue
artifact captures) instead of the probe's. The second attempt now starts
at 140s and ends at 270s, so the probe rides out every stall the old
budget did, and any stall shorter than about 260s instead of about 120s.
A green run still costs one connect per port, and a refused or unmapped
port still fails at once. A unit test pins the window above Tor's
SocksTimeout and fails as soon as torrc sets its own.
That doubles what a dead port costs, and meta/tests.yml caps the whole
CLI test. The largest compose onion deploy in run 40128751852 probes 19
domains and 5 ports; with everything dead that is 19 x 350 + 5 x 260 +
10 = 7960s, beyond the 7200s that 069df39027 set for the same reason.
9000 keeps the full failure output for up to 22 domains.
Closes#1214
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tor starts its worker threads by the number of CPU cores it detects, and
the cpus limit of the container does not change that number. On a host
with 20 cores every node of the private chutney network ran 21 threads and
grew to about 0.6 GB. The seven relays and authorities filled the 2g
limit and its swap within ten minutes, the kernel killed them one by one,
and no circuit could be built any more.
The entrypoint now writes NumCPUs 1 into the torrc of every chutney node
before it starts the network. Measured on that host over ten minutes: the
nodes run 3 threads and stay at 65 to 86 MB, the container at 0.47 GB
without swap, where it had filled 2 GB plus 2 GB of swap before.
The CLI test of the role fails when a chutney node runs without the limit.
Closes#824
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test-e2e-cli loads test.env with `set -a && . test.env && set +a`. The
unquoted `ONION_PORTS=25 80 110 ...` is not an assignment there but the
command `80` with a one-word env prefix: bash prints `command not found`,
leaves ONION_PORTS unset, and onion_ports.py reports "no onion-forwarded
ports declared; nothing to probe". The port check passed vacuously in every
tor job with more than one forwarded port (OpenResty, Listmonk, Pixelfed in
run 35661520542) and only ran where a single port was declared.
Pass the value through dotenv_quote, which quotes a space-separated value in
both compose and swarm and keeps the env-file lint satisfied. A unit test
renders the template in both modes and sources it the way test-e2e-cli does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sending to a .onion needs two things of the node: a resolver that answers the suffix, and a route from the container to Tor's transparent port. Neither existed. Compose pointed Mailu at unbound, which knows nothing about .onion, and in swarm Tor's DNSPort and TransPort listen inside the container where the host's dnsmasq and nat rules cannot reach them, so the forwarding rules that were already there pointed at a closed port on every swarm node.
Compose resolves through unbound, which now forwards the onion zone to Tor's DNSPort on the bridge it already owns. The forward alone would have answered SERVFAIL: the signed root proves that onion does not exist, so with auto-trust-anchor-file set every answer below it validates as bogus, and Tor's DNSPort can supply no DS. The zone is therefore declared insecure. Swarm resolves through the node dnsmasq instead, which every container already uses because container_dns puts the bridge resolver into daemon.json wherever svc-net-tor is deployed, and which now reaches Tor because its ports are published host-mode next to the SOCKS port that was published there already.
The transport rule changed target rather than mechanism. REDIRECT rewrites the destination to the primary address of the interface the packet arrived on, which is a different project bridge for every role, so Tor would have had to listen on all of them. A DNAT naming the bridge keeps one listener, and in swarm the same address reaches the published port through the proxy docker runs there.
That leaves the exposure. SocksPort has SocksPolicy; DNSPort and TransPort have no equivalent, and a swarm-published port carries no host address to bind to, so in swarm the packet filter is their only boundary. Compose needs none, because there Tor binds loopback and the bridge and nothing else. The swarm guard admits this deployment's own address space and drops the rest, and it is installed as a unit ordered after docker rather than as ansible-applied rules: nothing here persists netfilter state, so docker recreating its chains at boot would have left Tor back on the wildcard with the guard gone. It rebuilds its chain instead of appending to it, because the appending version could only ever be loosened. It hooks INPUT and DOCKER-USER both, since host-local traffic reaches a published port through docker's proxy and external traffic through FORWARD.
The deploy now proves all of it back and fails when a rule is missing, the router no longer runs on swarm nodes that carry no Tor, and the CLI test probes the forwarded ports over SOCKS instead of assuming a HiddenServicePort line means a reachable service.
Measured so far: Tor accepts the non-loopback binds and reports circuit-established, and the chain, its order and the DOCKER-USER hooks read back correctly from a deployed host. The swarm path is not yet exercised.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The probe abandoned each attempt at --max-time 30 while Tor's SocksTimeout
sits at its 120s default (no SocksTimeout anywhere in the repo; positive
control SocksPort hits templates/torrc.j2:1). It gave up at a quarter of
Tor's working window, and destructively: every retry opens a fresh SOCKS
connection, discarding the descriptor fetches and rendezvous retries Tor
was still working through.
Run 32284541284 job 96178512465 shows the cost. Ansible recorded start
07:22:17.260, end 07:26:33.366, delta 0:04:16.105994. The first domain
burned its deterministic 6x30 + 5x10 = 230s and failed; the second then
succeeded on its first attempt but needed ~23s of that 30s window - 77%
of the limit. The transient had cleared at t~233s. The first domain's
budget ran out three seconds short of it.
Two round numbers, one behaviour: 5 attempts at 60s is 340s of patience
against today's 230s, so the same transient now has 107s of margin
instead of -3s, and a 23s connect sits at 38% of the window instead of
77%. LAM#1 (job 96179137200) is the near-miss that corroborates it - four
dead attempts, then a ~4s success on the fifth.
meta/tests.yml is required, not optional. roles/test-e2e-cli/tasks/run_one.yml:74
runs the script under `timeout {{ _cli_timeout }}`, and with no meta/tests.yml
this role fell through to the 3600s default in roles/test-e2e-cli/defaults/main.yml:3.
Widening the window moves the all-domains-fail ceiling from 16 domains to
11, and a 13-domain role demonstrably exists (Jellyfin#0, four passes of
"13 probed, 0 failed"). 7200 restores the headroom.
The `|| true` goes with it: the script runs under `set -uo pipefail` with
no -e, so the command substitution's exit code was already discarded and
line 63 branches on ${code}. tests/lint/filesystem/shell/test_no_or_true.py
never saw it because its regex wants #, ; or EOL after `true` and a `)`
follows here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repository env lint requires ascending keys in every env file, and
test.env.j2 listed TOR_SOCKS ahead of TOR_DNSMASQ_CONF, so make quality
stopped there. Order carries no meaning in a dotenv template: Jinja
resolves the values from the role vars, not from preceding lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Job 94849355639 of run 31823845576 failed on exactly one name: the bare apex
infinito.example, HTTP 000, "Could not resolve host", while
espo.crm.infinito.example answered 200 in the same second through the same
resolver. The probe printed dns= and stopped there, so the log named the
symptom and no artifact carried the cause. There is no dnsmasq log
(log-queries is off, the journal has zero dnsmasq lines), no CoreDNS log in
the compose artifact set, and no query output from the host at failure time.
Reproducing the resolver stack locally did not close it either. CoreDNS on the
CI Corefile, dnsmasq on the drop-in pulled byte-identical from that job's own
--diff at 20:58:35, and the full dnsmasq -> 127.0.0.11 -> CoreDNS chain with a
live forward path all answer the apex with 172.30.0.10. The configuration is
exonerated; what is missing is the transaction itself.
So the failure branch now prints the active resolver, any /etc/hosts match,
what the local dnsmasq answers, and what each upstream answers. The last two
are the split that matters: they separate "dnsmasq dropped it" from "upstream
dropped it", which is the fork the bench could not settle. Upstreams are read
from the drop-in directory rather than one file, so a shared dnsmasq carrying
foreign drop-ins is covered, and matching only bare-IP server= lines skips
server=/onion/127.0.0.1#9053.
TOR_DNSMASQ_CONF arrives through test.env instead of a hardcoded
/etc/dnsmasq.d, using the same :? idiom 8d7fff4d2 gave the other four values,
so the path keeps its single source in vars/main.yml. dig is opportunistic:
without bind-tools the resolver and hosts lines still land.
Verified: bash -n and shellcheck clean, the failure path exercised end to end
against a CI-shaped drop-in with 127.0.0.11 extracted and the /onion/ line
correctly skipped, test_comments_valid passed.
Two things this does not close. dig is probably absent on the Arch target that
failed, since the bind9-dnsutils lines in that job come from a Debian build
stage, not the target; adding bind-tools to the role would fix that and is a
separate concern. And the trigger is now rare, because pinning tor true across
the espocrm variants removed the mixed round that put the clearnet apex into
the vhost list at all. This is insurance, not a plan that pays out next run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deleting the external onion probe left check_external, _CURL_REACHED_RCS,
the internal|external|both dispatch and the optional socks argument with
no caller, while the docstring still promised 'Exit 0 only when both
succeed' and that the external path 'must actually work end to end' —
the exact contract that deletion removed.
The script now takes one argument. Real-network reachability stays gated
where it blocks something: test-e2e-playwright's wait_onion_reachable.sh.
Wiring the CLI test into e2e-cli also armed its default retry budget:
20 attempts x curl --max-time 30 plus 19 x 15s sleep is 885s per failing
domain, serial. TEST_E2E_CLI_TIMEOUT is 3600s, so four failing domains
exhaust it and a tor deploy serves about twelve vhosts. test.env.j2 now
pins RETRIES=6 and SLEEP_SECONDS=10 — 230s per domain.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role shipped a complete probe script at files/test.sh and no
templates/test.env.j2. discover_cli_roles globs the env template alone,
so the role was never discovered and the script had never run — and the
script hard-requires TOR_SOCKS and NGINX_SERVERS_DIR via ${VAR:?}, so it
could not have run without one either. Its own usage hint pointed at a
path that does not exist.
Moving the script to the discovered path and adding the env template
makes both halves line up. Verified against the live stack: the onion
answers through the Tor SOCKS proxy, 1 probed, 0 failed.
The new lint keeps the pair together and rejects a script left at
files/test.sh. The two halves fail differently and only one is loud: an
env template without a script aborts the play, a script without an env
template is never discovered and its silence reads like a pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5.5