The check listed the torrcs without the limit through "grep -L ...; true".
With no torrc under /opt/chutney/net/nodes, or with one that cannot be
read, grep printed nothing, the trailing true turned its error into
success, and the test reported that every node runs one worker.
The check now counts the limit per file with "grep -cH" and reads the exit
status. A status above 1 or an empty answer fails the test, and a file
with a count of 0 is named as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Tor starts its worker threads by the number of CPU cores it detects, and
the cpus limit of the container does not change that number. On a host
with 20 cores every node of the private chutney network ran 21 threads and
grew to about 0.6 GB. The seven relays and authorities filled the 2g
limit and its swap within ten minutes, the kernel killed them one by one,
and no circuit could be built any more.
The entrypoint now writes NumCPUs 1 into the torrc of every chutney node
before it starts the network. Measured on that host over ten minutes: the
nodes run 3 threads and stay at 65 to 86 MB, the container at 0.47 GB
without swap, where it had filled 2 GB plus 2 GB of swap before.
The CLI test of the role fails when a chutney node runs without the limit.
Closes#824
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sending to a .onion needs two things of the node: a resolver that answers the suffix, and a route from the container to Tor's transparent port. Neither existed. Compose pointed Mailu at unbound, which knows nothing about .onion, and in swarm Tor's DNSPort and TransPort listen inside the container where the host's dnsmasq and nat rules cannot reach them, so the forwarding rules that were already there pointed at a closed port on every swarm node.
Compose resolves through unbound, which now forwards the onion zone to Tor's DNSPort on the bridge it already owns. The forward alone would have answered SERVFAIL: the signed root proves that onion does not exist, so with auto-trust-anchor-file set every answer below it validates as bogus, and Tor's DNSPort can supply no DS. The zone is therefore declared insecure. Swarm resolves through the node dnsmasq instead, which every container already uses because container_dns puts the bridge resolver into daemon.json wherever svc-net-tor is deployed, and which now reaches Tor because its ports are published host-mode next to the SOCKS port that was published there already.
The transport rule changed target rather than mechanism. REDIRECT rewrites the destination to the primary address of the interface the packet arrived on, which is a different project bridge for every role, so Tor would have had to listen on all of them. A DNAT naming the bridge keeps one listener, and in swarm the same address reaches the published port through the proxy docker runs there.
That leaves the exposure. SocksPort has SocksPolicy; DNSPort and TransPort have no equivalent, and a swarm-published port carries no host address to bind to, so in swarm the packet filter is their only boundary. Compose needs none, because there Tor binds loopback and the bridge and nothing else. The swarm guard admits this deployment's own address space and drops the rest, and it is installed as a unit ordered after docker rather than as ansible-applied rules: nothing here persists netfilter state, so docker recreating its chains at boot would have left Tor back on the wildcard with the guard gone. It rebuilds its chain instead of appending to it, because the appending version could only ever be loosened. It hooks INPUT and DOCKER-USER both, since host-local traffic reaches a published port through docker's proxy and external traffic through FORWARD.
The deploy now proves all of it back and fails when a rule is missing, the router no longer runs on swarm nodes that carry no Tor, and the CLI test probes the forwarded ports over SOCKS instead of assuming a HiddenServicePort line means a reachable service.
Measured so far: Tor accepts the non-loopback binds and reports circuit-established, and the chain, its order and the DOCKER-USER hooks read back correctly from a deployed host. The swarm path is not yet exercised.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The probe abandoned each attempt at --max-time 30 while Tor's SocksTimeout
sits at its 120s default (no SocksTimeout anywhere in the repo; positive
control SocksPort hits templates/torrc.j2:1). It gave up at a quarter of
Tor's working window, and destructively: every retry opens a fresh SOCKS
connection, discarding the descriptor fetches and rendezvous retries Tor
was still working through.
Run 32284541284 job 96178512465 shows the cost. Ansible recorded start
07:22:17.260, end 07:26:33.366, delta 0:04:16.105994. The first domain
burned its deterministic 6x30 + 5x10 = 230s and failed; the second then
succeeded on its first attempt but needed ~23s of that 30s window - 77%
of the limit. The transient had cleared at t~233s. The first domain's
budget ran out three seconds short of it.
Two round numbers, one behaviour: 5 attempts at 60s is 340s of patience
against today's 230s, so the same transient now has 107s of margin
instead of -3s, and a 23s connect sits at 38% of the window instead of
77%. LAM#1 (job 96179137200) is the near-miss that corroborates it - four
dead attempts, then a ~4s success on the fifth.
meta/tests.yml is required, not optional. roles/test-e2e-cli/tasks/run_one.yml:74
runs the script under `timeout {{ _cli_timeout }}`, and with no meta/tests.yml
this role fell through to the 3600s default in roles/test-e2e-cli/defaults/main.yml:3.
Widening the window moves the all-domains-fail ceiling from 16 domains to
11, and a 13-domain role demonstrably exists (Jellyfin#0, four passes of
"13 probed, 0 failed"). 7200 restores the headroom.
The `|| true` goes with it: the script runs under `set -uo pipefail` with
no -e, so the command substitution's exit code was already discarded and
line 63 branches on ${code}. tests/lint/filesystem/shell/test_no_or_true.py
never saw it because its regex wants #, ; or EOL after `true` and a `)`
follows here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Job 94849355639 of run 31823845576 failed on exactly one name: the bare apex
infinito.example, HTTP 000, "Could not resolve host", while
espo.crm.infinito.example answered 200 in the same second through the same
resolver. The probe printed dns= and stopped there, so the log named the
symptom and no artifact carried the cause. There is no dnsmasq log
(log-queries is off, the journal has zero dnsmasq lines), no CoreDNS log in
the compose artifact set, and no query output from the host at failure time.
Reproducing the resolver stack locally did not close it either. CoreDNS on the
CI Corefile, dnsmasq on the drop-in pulled byte-identical from that job's own
--diff at 20:58:35, and the full dnsmasq -> 127.0.0.11 -> CoreDNS chain with a
live forward path all answer the apex with 172.30.0.10. The configuration is
exonerated; what is missing is the transaction itself.
So the failure branch now prints the active resolver, any /etc/hosts match,
what the local dnsmasq answers, and what each upstream answers. The last two
are the split that matters: they separate "dnsmasq dropped it" from "upstream
dropped it", which is the fork the bench could not settle. Upstreams are read
from the drop-in directory rather than one file, so a shared dnsmasq carrying
foreign drop-ins is covered, and matching only bare-IP server= lines skips
server=/onion/127.0.0.1#9053.
TOR_DNSMASQ_CONF arrives through test.env instead of a hardcoded
/etc/dnsmasq.d, using the same :? idiom 8d7fff4d2 gave the other four values,
so the path keeps its single source in vars/main.yml. dig is opportunistic:
without bind-tools the resolver and hosts lines still land.
Verified: bash -n and shellcheck clean, the failure path exercised end to end
against a CI-shaped drop-in with 127.0.0.11 extracted and the /onion/ line
correctly skipped, test_comments_valid passed.
Two things this does not close. dig is probably absent on the Arch target that
failed, since the bind9-dnsutils lines in that job come from a Debian build
stage, not the target; adding bind-tools to the role would fix that and is a
separate concern. And the trigger is now rare, because pinning tor true across
the espocrm variants removed the mixed round that put the clearnet apex into
the vhost list at all. This is insurance, not a plan that pays out next run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
roles/svc-net-tor/templates/test.env.j2 sets RETRIES=6 and SLEEP_SECONDS=10, and
test-e2e-cli/tasks/run_one.yml renders that file unconditionally at :66 before
sourcing it at :74. The ${RETRIES:-20} / ${SLEEP_SECONDS:-15} fallbacks could
therefore never be reached -- a second source of truth that additionally
documented the wrong numbers: the header promised "default 20 / default 15"
while every run used 6 and 10.
They now take the :? form the two variables above them already use, so
test.env.j2 is the only place the values live, and a missing one aborts loudly
instead of silently probing three times longer than intended. The header stops
claiming defaults.
The sibling probes in svc-db-mariadb, svc-db-openldap and svc-db-postgres keep
${RETRIES:-20}: their test.env.j2 does not set the pair, so there the fallback is
live, not dead. Converting them would mean adding the values to those templates
first, and is deliberately left out.
Measured: a missing RETRIES exits 1 with "line 24: RETRIES: pass RETRIES as env
(attempts per domain, from svc-net-tor test.env)"; a complete env leaves the
probe output unchanged. shellcheck is clean against the staged content, and the
comment, shell and cli-asset-pairing lints pass.
The message wording avoids an apostrophe on purpose: inside ${VAR:?...} bash
still treats one as a quote character, which splits the abort message across the
next source line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HTTP 000 cannot tell a name that never resolved from a listener that never
answered. Both leave curl without a status and openresty without an access-log
entry, which is exactly the signature the apex infinito.example has produced in
every compose job of run 31543714113.
2a60ddb3c already stopped discarding curl's stderr, so the message now names the
layer -- (6) could not resolve, (7) failed to connect, (35) handshake. It never
names the address, and that is the half that decides between "the resolver
answered wrong" and "the resolver did not answer".
getent hosts goes through the same NSS path curl uses, so the verdict carries
what the node itself resolved:
[FAIL] localhost HTTP 000 dns=::1
[FAIL] no-such-host-xyz.invalid HTTP 000 dns=
Empty means resolution failed. An address together with a connect or TLS error
means the resolver answered and the far side did not. An address other than the
node's own means the resolver answered wrongly. Read next to curl's line, those
three branches separate.
Only the failure path changes; [OK] keeps its format, and getent's own noise
goes to /dev/null so a probe never fails on the diagnostic.
Measured: both branches above; getent and head present in all four node images
(fedora, arch, centos, debian) via docker run; shellcheck clean against the
staged content; tests/lint/repository/test_comments_valid.py,
tests/lint/filesystem/shell/ and test_cli_test_assets_paired.py pass.
This is not the fix for the apex failure. Nine mechanisms were eliminated by
measurement today -- CoreDNS, dnsmasq and Docker's 127.0.0.11 each resolve both
the apex and its sibling to 172.30.0.10; the vhost exists in http and https;
nginx validated six times; no server-name conflict, no proxy env, no conntrack
pressure. What remains needs the one fact the log has never carried, and this
line carries it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The apex infinito.example has failed six compose jobs with HTTP 000 and no
explanation, because the probe throws curl's stderr away:
curl ... "${scheme}://${domain}/" 2>/dev/null || true
--show-error was already set, so curl had the answer the whole time and the
redirect discarded it. Command substitution captures stdout only, so
%{http_code} and the accept regex at :61 are untouched, and || true still
swallows the exit code. test-e2e-cli/tasks/run_one.yml:85-92 prints stdout and
stderr when rc != 0, so the text reaches the job log.
This is NOT the fix for that failure. Three rounds of reasoning about it rested
on the premise that the apex answers 301 and curl follows it off-instance;
openresty.log:126 refutes it. That line logs the successful clearnet probe
"GET / HTTP/2.0" 200 with "Scheme: https" and User-Agent curl/8.18.0, so https
requests do appear in that log -- and the apex left no entry at all. curl never
completed hop 1, and there never was a 301 to follow. Whether the cause is
NXDOMAIN, a refused connect, or the 443 default_server's ssl_reject_handshake
(sys-svc-webserver-core/templates/default_server.conf.j2:17-20, which produces
exactly this signature: rejected handshake, 000 at curl, no access-log entry)
is what one curl error line will now say.
Costs up to six extra stderr lines per failing domain, RETRIES=6.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role shipped a complete probe script at files/test.sh and no
templates/test.env.j2. discover_cli_roles globs the env template alone,
so the role was never discovered and the script had never run — and the
script hard-requires TOR_SOCKS and NGINX_SERVERS_DIR via ${VAR:?}, so it
could not have run without one either. Its own usage hint pointed at a
path that does not exist.
Moving the script to the discovered path and adding the env template
makes both halves line up. Verified against the live stack: the onion
answers through the Tor SOCKS proxy, 1 probed, 0 failed.
The new lint keeps the pair together and rejects a script left at
files/test.sh. The two halves fail differently and only one is loud: an
env template without a script aborts the play, a script without an env
template is never discovered and its silence reads like a pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5.5