Every blackbox probe of a Tor deployment answered 404, so AppDown fired for
all deployed apps although they were up. The job built its targets from
canonical_domains_map(DOMAIN_PRIMARY), which knows no onion domain. In a Tor
deployment the vhosts answer under <sub>.<onion> only, and the front proxy
served the clearnet names from its default server.
The targets now come from lookup('domains'), the map the vhosts are built
from, and each target takes its scheme from the domain it probes instead of
from the Prometheus vhost.
Which roles are probed follows lookup('roles_with_service', 'prometheus'),
the list the role's own spec expects metrics for. It leaves out the database
roles, which have a domain in the map and no vhost, and the roles that
declare scrape: false because their vhost carries no Prometheus block.
Two roles declared more than their vhost delivers. web-svc-mirror renders a
vhost of its own with no backend behind it, and on a live stack its probe
answered 404 under the onion name as well, so it declares scrape: false now.
web-app-peertube has a backend, but its vhost template never included the
Prometheus locations. It includes them now.
The spec could not see any of this in a compose deployment. dotenv_quote
wraps PROMETHEUS_TARGET_ROLES_JSON in quotes there, JSON.parse returned a
string, the role list came out empty and the per-role test was skipped. The
value is decoded now, and the test also requires probe_success == 1 for
every role in the list.
Closes#792
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
72fd9c870 required the onion in every variant but exempted variant 1, on the
argument that the minimal round is the one place the stack runs without it. That
argument does not hold: the onion is part of the deployment under test, not a
feature bolted onto the full shape. A round that skips it is not a smaller
deployment, it is a different one.
So the exemption is gone and 95 variant-1 entries pin the onion on. Two
exclusions remain, both read from data rather than a comment: a role with no tor
block has no onion to pin, and a role whose meta/services.yml pins tor.enabled to
a literal false has already declared it has none.
That put the rule in conflict with test_coverage, which requires both polarities
of every dynamic flag: with tor on everywhere, the false side has no variant left
to live in. 94 roles take the exemption that contract already provides for
exactly this - "flags whose group-derived value is a hard invariant that variants
must never override", the same one 4e6ef2542 used for the seaweedfs frontend.
web-app-nextcloud needs none: its variant 8 pairs the clearnet-only PeerTube and
keeps the false polarity.
Measured, because the declaration is not what CI runs. assign_tor gives the onion
to every second capable entry with the run number as the parity switch, so more
capable variants do not mean proportionally more onion jobs: across five roles
and 21 entries the count goes 8 -> 10 on an even run and 9 on an odd one. The
clearnet path also survives - checkmk reports v1=false on run 0 and v1=true on
run 1, so both states are still covered across two consecutive runs. What the
94 markers give up is a declarative guarantee in variants.yml, not the coverage
itself.
The real cost is weight, not job count. web-app-nextcloud variant 1 goes from
3.09GB reservation / 7.27GB limit to 18.26GB / 43.01GB once svc-net-tor and its
closure join, so the cheap round is no longer cheap. Every variant stays under
the 32/64GB budget, and the same effect applies to the other 94 unmeasured.
Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles/meta/
variants plus the role-meta lints 39 passed. The rule was checked in both
directions - it reported the variant-1 offenders before the flips and none after.
Not measured: what the heavier rounds do to wall-clock, against swarm jobs capped
at 355 minutes and one that has hit that ceiling before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every variants.yml header states the rule the bond feeds: variant 0 carries each
tightly-coupled (bond>=1, same-host) service, while loosely-coupled (bond<1,
separate-host) partners are excluded from the closure and distributed across
later variants to keep every host under budget. tor was declared bond: 1 in all
101 consumers, so it had to sit in every variant-0 closure.
It does not belong there. The onion is a separate-host provider reached over the
node's hidden service, the same shape as the bridged partners that already carry
0.5, and f1dff50fb established both the reasoning and the precedent when it
dropped xwiki and bigbluebutton from 1 to 0.5: bridged partners with their own
role and their own spec, whose bond declaration was the outlier rather than
their placement.
This also unblocks the variant-level tor contract. Requiring a non-baseline
variant to pin the onion on while tor still counted as tightly coupled would
have pulled variant 0's closure requirements along with it.
Verified: tests/lint 336 passed / 307 subtests and tests/integration/roles 107
passed / 610 subtests, including test_resource_budget over every variant's
deduplicated footprint and the bond-aware test_order closure.
Not measured, and the honest cost: f1dff50fb's own caveat applies unchanged
here. Whether redistributing a service out of the baseline closure buys back
wall-clock is not established; role-runtimes has the per-role seconds and no run
has been made with this shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
New web-svc-mirror role: an nginx vhost that proxies and caches exactly
the external asset hosts aggregated from all deployed applications' CSP
whitelists (external_asset_origins filter) under first-party paths
(/<origin-host>/...), refusing every other upstream. svc-net-tor
consumes the service, so any tor deployment automatically serves the
mirror on its onion.
For apps with services.tor.enabled the shared body filter rewrites
external asset URLs in HTML to the onion mirror (onion sessions only;
clearnet untouched) and the CSP builder allows the mirror origin in the
asset directives. The dashboard's keycloak-js vendoring is reverted:
onion sessions now load the adapter through the mirror, clearnet loads
jsdelivr directly as before.
Verified: variant=0 deploy green (failed=0), dashboard 14/14 over onion
with keycloak-js served through the mirror, mirror spec green, health
check accepts the 404 vhost root.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5.5