Adding stylelint left package-lock.json out of sync, and I regenerated it with
`npm install --package-lock-only` inside the compose container. That container
resolves npm through package-cache:8081, so npm rewrote 110 `resolved` URLs to
`registry.npmjs.org/repository/npm-proxy/…` — a path that exists only behind
that proxy. The integrity hashes were never wrong; only the place to fetch from.
It took down two CI jobs from one mistake. The test job probes external URLs
and got 108 HTTP 404s. The lint job never linted at all: install-lint died
fetching write-file-atomic from the same dead path, so every lint target behind
it was skipped rather than run.
A local `make quality-high` cannot catch this class. The sandbox has no route to
the registry, so the reachability check degrades to a warning and the run goes
green on an unanswered question. That is worth remembering beyond this fix:
green here is not evidence for anything that needs the network.
Verified this time: all 184 resolved URLs answer HTTP 200, and a structural
diff against the pre-stylelint lockfile shows 110 packages added, none removed,
and no existing package altered in version or integrity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two installers write the same database and neither reconciles the admin
credential, so `/administrator` answered every login with "Username and
password do not match" — six POSTs across two orchestrator passes, all 303, in
the only CI row that exercises the local-login hatch. The green sibling row
proves nothing: its JUnit report lists that test as skipped.
The image entrypoint auto-installs Joomla on first boot. The role waited for
`plugins/actionlog/joomla/joomla.xml`, a file copied early in that boot, then
removed configuration.php and ran its own CLI install against the same
database. Whichever finished last owned the admin row, and the run's artifacts
show the entrypoint finishing one second after the play ended.
So the role now waits for the entrypoint to be done before deciding anything.
The marker it already writes, `.infinito-init.ready`, is that signal; the
`|| test -e configuration.php` arm covers re-deploys, where the entrypoint's
setup block never runs and the marker is therefore never touched.
With the state settled, the password reset no longer needs a gate. It was
skipped on fresh installs on the assumption that the CLI installer had set the
password — an assumption this run falsifies, since that installer exits 1 and
is swallowed as a soft success. The race it was written to avoid ("The user
does not exist!" right after install) is absorbed inside the reset itself,
which retries six times at five-second intervals and whose header documents
exactly that symptom.
The marker's name and path now live only in vars/main.yml and reach the
container through the env file. The entrypoint requires it without a default,
matching svc-runner and svc-db-openldap: a fallback would be the second
definition this change exists to remove.
Verified: the guarded expansion executed in both env states (127 with a legible
message, 0 with the path), the retry-exhaustion path executed against localhost
to confirm `failed_when: false` lets the play continue, plus shellcheck,
ansible-lint on the production profile, and 201 Jinja/ansible lint tests.
Not verified: no deploy was run. Whether the barrier holds on a real fresh
install is unmeasured.
Still open, both separate defects: 03_install.yml swallows the installer's
rc 1 as success, and test-oidc-login.js passes on the plugin's catch path
because it only asserts the URL prefix and a visible body.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role tracked `latest`, which was not a choice: until today the upstream
repository had never published a versioned tag at all, so `latest` was the only
thing that existed. Its release pipeline is fixed now and v1.3.1 is in the
registry, so the pin can say what it means.
Two things follow. Deploys stop depending on when a host last pulled, and the
role joins the daily update-image-versions job, which only bumps entries whose
current value is semver-shaped — `latest` kept it out of the freshness pipeline
entirely.
The `v` prefix is the actual tag: the upstream build takes `github.ref_name`
verbatim, and the registry lists exactly `latest` and `v1.3.1`.
Verified against the live registry rather than assumed. `make test-external`
passes with the pin, and a control run with a deliberately absent tag fails and
names this very line — so the green run tested this entry rather than skipping
it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`roles/<role>/files/<lang>/<rel>` is tested by
`tests/unit/<lang>/roles/<role>/files/<rel>`, renamed to that language's
convention. The language appears once, in `tests/unit/`, because the suite
already says what the file is. Without a fixed mapping a shipped script is
tested wherever its author happened to look, and nobody can tell an untested
file from one whose test lives somewhere unexpected.
Most of what a role ships is a script against a live console -- it boots a
Laravel kernel, imports frappe, or writes through an ActiveRecord model -- so
the escape is a `nocheck: mirrored-unit-test` comment in the file, with a
rationale of its own. A file carrying both a test and an exemption fails: the
exemption outlived its reason and reads as "untested" to everyone after you.
Five scripts turned out to have real units behind the console wrapper, so they
get tests rather than markers (62 cases):
* svc-net-tor/onion_self_reach.py -- pins that an HTTP error status, a closed
connection and a refused connection all count as reached, because each
proves the loopback fast path resolved; only a timeout or an unresolved
name is a failure.
* sys-lock/sys-lock.py -- `parse_time_to_seconds` unit precedence, and that a
service still activating holds the lock (otherwise the gate races it).
* svc-opt-keyboard-color/script.py -- colour interpolation and the
zero-length transition window that would otherwise divide by zero.
* sys-svc-compose/swarm/secret_rotate.py -- including the binary fallback a
TLS key needs, and that a missing source leaves the name alone rather than
pinning the hash of a file that is not there.
* sys-front-inj-logout/logout.js -- pins that `href="/logout"` alone does not
match and a non-interactive tag is never patched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four parse checks join the `lint-*` family, each using the language's own
parser rather than a reimplementation: `php -l`, `ruby -c`, `sqlfluff parse`,
and stylelint with an empty rule set (it raises CssSyntaxError whatever the
rules say, so the check stays syntax-only and grows no style opinion). Three
unit runners join `test-unit`: node:test, PHPUnit, minitest -- the first and
third ship with their runtime, so only PHPUnit adds a dependency.
Provisioning is best effort and the lint script decides whether it can run. A
machine without PHP must not fail the gate, but it must not look like success
either: the skip names how many files went unchecked. The first version of
these scripts read `git ls-files`, which inside the compose container returns
nothing (git refuses the bind-mounted repo as "dubious ownership") and reported
"No SQL files found." over 52 files -- green, and a lie. They use `find` now,
as the surrounding lint scripts already did.
sqlfluff needed two corrections before it parsed anything real. `%(name)s` is a
psycopg parameter marker, not SQL, so the placeholder templater is configured
rather than the files exempted; and the dialect follows each role's declared
engine (`meta/services.yml`), which is postgres for six roles and mysql for
svc-db-mariadb and pixelfed.
Recipes stay in scripts, not in the Makefile: the targets delegate, matching
every other target in the file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ca40eee68 made the admin test wait for bearerAuth to disappear before leaving
Keycloak's logout page. Nothing proves that wait is load-bearing. Delete it and
the suite stays green on clearnet, because there the sweep wins the race
anyway; it turns red only over Tor, where a red job reads as flake. The
property the wait protects is therefore untested exactly where it matters.
Racing it deliberately cannot fix that: a test that leaves early and asserts
the session survived would pass over Tor and fail on clearnet - the same
latency dependency, mirrored. 801923eb3 measured that asymmetry on Funkwhale:
"Only swarm+tor is slow enough to lose the race."
So this does not race. page.route aborts every request whose pathname is
/logout - each app host the conductor sweeps - and the test then asserts the
admin cookie SURVIVES. What is pinned is the causal link, not a timing: the
session outlives a logout exactly when the sweep does not land. That stays
true however fast the sweep becomes, and breaks only if logout stops depending
on it, which is a change worth failing on.
The pathname filter is deliberate. A glob on **/logout* would also swallow
Keycloak's own /realms/x/protocol/openid-connect/logout and break the flow
under test; url.pathname === "/logout" matches the sweep targets and neither
Keycloak's end-session endpoint nor the conductor page at /.
Two ways this test could pass while proving nothing, both closed. A parallel
page.on("request") counter separates them: no request to /logout at all means
the conductor never reached its sweep, the precondition is absent and the test
skips - making the sweep land is the sibling test's assertion. Requests seen
but nothing intercepted means the abort is a no-op, which fails loudly rather
than skipping forever if page.route ever stops matching.
Skipping rather than failing on the absent precondition also keeps the cost
bounded: playwright.config.js runs workers: 1 with retries: 2, so a failure
here would spend three full Tor login cycles proving something the sibling
already reports.
Interception was measured, not assumed: with three origins, a cross-origin
iframe and keepalive: true, the handler fires and the target server sees no
request. signIn/dismissFirstRunWizard/clickLogout are extracted so the second
test costs helpers rather than a copy of the first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-svc-logout already reported every domain it swept - into a table inside
the invisible front-channel iframe Keycloak loads. The status existed and
nobody could see it, so the visitor read Keycloak's "you are logged out" and
left while the sweep was still running.
The injected script listens for the conductor's postMessage reports and
renders them on Keycloak's own page: one row per service, a counter, and a
terminal state that names an action rather than describing one - "You can
close this page now" is the question the visitor actually has.
Silence must not read as success. The panel paints itself on the logout path
before any message arrives, and a conductor that never loads leaves a visible
"could not check your other sessions" plus a link, instead of a page that
looks finished. The unload guard is armed from that same moment and released
on done, on a 60s stall, or when the grace period expires without a start -
a guard that outlives its sweep would prompt on every later navigation, and a
stall that only releases silently would leave a spinner that never resolves,
which reads worse than an error. Neither timeout claims the logout finished.
A ❌ carries the manual <host>/logout?manual=1 link the conductor's own table
had; the panel builds it from the reported host, so universal-logout is
untouched. Rows show "Shop" with the full host on hover, state text sits
beside every glyph so aria-live has something to announce, and the border
colour carries the state for anyone who does not read the heading.
The trusted origin comes from lookup('tls', 'web-svc-logout', 'url.base') -
the same expression vars/main.yml:85 uses for KEYCLOAK_FRONTCHANNEL_LOGOUT_URL,
so the framed origin and the accepted origin cannot drift apart. A test pins
that, because they live in different files and the coupling was accidental.
frontchannel.logout.session.required is "true" in the realm import, so
Keycloak must append ?iss= per the OIDC front-channel spec; document.referrer
is the fallback.
Eight mutations were checked against the collapsed script - dropping the
pessimistic render, either timer, the origin check, the manual link, or making
either terminal state claim success all fail the suite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sys-front-inj-javascript collapses a role's javascript.js(.j2) through
to_one_liner, body_filter.lua.j2:5 embeds the result as a Lua [=[ ]=] long
string, and add_csp_hash takes the hash over that same collapsed text.
Nothing checked that the text survives any of those three stages.
Both failure modes are silent. A script that parses only before the collapse
is served with a matching hash, so the browser rejects nothing and executes
nothing - the page loses its injection without an error anywhere in the
deploy. An inner ]=] closes the Lua long string early, after which the
OpenResty config stops loading for every site on the host.
The test walks all three source shapes (templates/javascript.js.j2,
templates/javascript.js, files/javascript.js), substitutes Jinja instead of
rendering it - a neutral literal for expressions, nothing for statements,
enough to expose an ASI break without standing up an inventory - and runs
node --check over the collapsed result. Eleven scripts are covered today and
all eleven pass.
Neither guard existed anywhere: git log --all finds no `node --check` and no
`]=]` assertion in tests on any ref.
node is gated with skipUnless, matching test_service_gating_helper.py and
test_timeouts_helper.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
e59ea21c6 waited on document.cookie, which is origin-scoped. By the time
it runs the page sits on logout.<domain>, where Shopware's bearerAuth -
host-scoped to shop.<domain> - is not visible. The predicate was true on
the first poll, so the wait did nothing and three of the six jobs were
unchanged.
page.context().cookies() reads the browser context's jar regardless of
which document is displayed, so the same post-condition can be observed
from Keycloak's logout page.
The wait matters because the thing that clears that cookie is not
Shopware and not the injected patch - it is web-svc-logout. Keycloak
loads its conductor as the front-channel logout URL in an iframe; the
conductor fetches <host>/logout for every domain with
credentials: "include" (universal-logout templates/conductor.html.j2),
each /logout deletes the cookies visible on that host and sends
Clear-Site-Data (app.py), and roles/web-svc-logout/templates/logout-proxy.conf.j2
supplies the matching Access-Control-Allow-Origin plus
Allow-Credentials. Every link of that chain is in place.
What was missing was time. In run 32284541284 the trace shows the
logout-confirm click at 16543.805 and the navigation to /admin at
16902.227 - 358 ms, against 220-350 ms per Tor hop. The conductor's
document had not finished loading, so its sweep never ran:
grep -c -a "onion/logout" on the trace network log returns 0. On clearnet
the same chain fits inside those 358 ms, which is why variant 0 passes
there and fails over Tor.
A failure now reports the cookie that survived instead of the missing
password field two lines below, which said nothing about the cause.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test set INFINITO_CI_DISTRO_BUDGET_SECONDS=1 and asserted the loop
returns 124. distros.sh:45 fixes the deadline at start+1s and only checks
it at :105, so whenever the shell needs longer than that one second to
reach the loop, `remaining <= 0` breaks out before anything runs, the
script falls through to render_summary and exits 0. The assertion then
reads 0 != 124.
That is what CI hit in run 32369164168 job 96426381092, under pytest-xdist
([gw2] in the failure header) where a second of wall clock is not the
script's to spend. A retry passed on the same code, which is the flake
stated plainly.
5s keeps every property the test is there for - sleep 30 still exceeds it,
timeout still kills, the assertion still demands 124 - while putting the
fixture's own startup well inside the budget. 67b6072bb made a budget kill
fail loudly on purpose rather than booking it as skipped; this only makes
the test reach that kill reliably.
The message argument moves from proc.stderr to proc.stdout because
stderr is empty here: the "[WARN] Global budget exhausted" line the script
prints goes to stdout, so the old failure said nothing about its cause.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-ai-litellm serves an OpenAI-compatible gateway and web-app-nextcloud is
already bound to it, but that binding exists once and is written down
nowhere. A sweep over all 85 application roles finds provider references in
nine, five of which are AI-native, so every other application with a native
AI feature still defaults to a vendor.
Record which roles have a surface with a configurable endpoint, which are
excluded and why, and which vendor claims still need a probe against a
running instance. Two criteria close classes that cost a full session to
diagnose: a deploy must fail when an enabled AI surface carries an empty
key, and no credential schema may encode a vendor key format —
web-app-matrix validates ^sk-[a-zA-Z0-9]{40,}$ today and would reject a
gateway-issued key by construction.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workflow was created deactivated and never activated, so the declared
endpoint answered 404 and every client configured against it failed its
reachability check. The intent was that an operator opts in before an MCP
server answers, but mcp.enabled already is that opt-in: a second one hidden
in the provisioning script made the declared state unenforceable, and the
mismatch surfaced three layers away as a failing consumer.
Activate after create and after update. Keep the protection that matters:
an already-active workflow is left untouched rather than rewritten under
callers bound to its endpoint, and that case now returns instead of ending
the deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gitea-mcp holds no credential and expects a token per request, so anything
on the provider's network reached it directly and skipped the adapter's
allowlist and limits. A private internal network fixes that, but the
isolation guard admitted exactly one subnet and rejected the adapter for
holding a second, reading a narrower attachment as a wider one.
Teach the guard the concept instead of working around it: a provider may
declare adapter.upstream_network, and the guard resolves that network and
adds its subnet to the expected set. Providers without one are unchanged.
web-app-gitea is the first consumer: the sidecar now joins the private net
and nothing else, bridge under compose and overlay under swarm, while gitea
and the adapter join it alongside their own attachments.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0d8ed0ba8 bounded these call sites with one ceiling for both operations.
The two are not equally safe to cut. A SIGTERM landing mid-postinst
leaves dpkg interrupted, and in docker_dataroot.sh that fails every later
filesystem candidate and sends the run through decline() - one truncated
package install becomes a dead job. An interrupted update has no such
cascade; it just repeats.
Measured on ubuntu:latest with the exact commands: update 10s, install
zfsutils-linux 20s. 1200s is a 60x margin on the operation that carries
the cascade, and update keeps the 600s it cannot plausibly need more than.
This breaks the single-number invariant with scripts/install/apt.sh:19,
which is already broken in fact - that call site carries a different
Acquire option set than these three do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six findings from a full audit of the roles this branch touched. None was
behind a failing job; all of them lie about what the code does.
Two defaults that can never fire, and would do harm if they did.
vars/main.yml gave services.tor.flavor a 'public' fallback, but
meta/services.yml:28 always defines the key - so the fallback is
unreachable, and it defeats the assert at tasks/00_core.yml:4-9: remove
the key and the role silently deploys public instead of failing loudly.
templates/dnsmasq-tor-onion.conf.j2 guarded twice on
`TOR_DNSMASQ_OWNS_LISTENER | default(true)`, while tasks/router.yml:35-37
sets that fact unconditionally two tasks earlier; had it ever gone
missing, `true` would have reshaped a dnsmasq this role does not own.
Two tasks with no readers. sys-front-inj-logout/tasks/01_deploy.yml
stat'ed logout.js to build INJ_LOGOUT_JS_VERSION; neither that fact nor
INJ_LOGOUT_JS_STAT is referenced anywhere under roles/ or group_vars/.
Cache busting already runs independently through lookup('local_mtime_qs')
in templates/head_sub.j2:1.
One README claim inverted. svc-net-tor's feature list named
TOR_EGRESS_ENABLED and called it off by default. That variable exists only
in docs/requirements/031-*.md - the implementation settled on
TOR_EGRESS_ROUTER_ENABLED, which group_vars/all/19_tor.yml:4 sets to true.
The README stated the opposite of its own default under a name nothing
reads.
Three files nothing can load, in web-app-prometheus. tasks/reset.yml is a
single comment and is included nowhere; templates/compose.override.yml is
one byte, and sys-svc-compose/tasks/04_files.yml:52-54 searches only
templates/compose.override.yml.j2 and files/compose.override.yml;
templates/style.css is a single comment, and sys-front-inj-css/tasks/main.yml:10
searches only templates/style.css.j2 and files/style.css.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The injected logout built `?redirect_uri=<web_protocol>://<DOMAIN_PRIMARY>`
and appended it to the end-session endpoint. Keycloak 26.7.1
(roles/web-app-keycloak/meta/services.yml:49-50) does not accept
redirect_uri there - it takes post_logout_redirect_uri plus id_token_hint
or client_id - so the parameter was dropped on arrival. Run 32284541284
job 96178507650 shows the proof: Keycloak answered with its logout-confirm
interstitial carrying client_id=account, i.e. it resolved no client from
the request at all.
The value was wrong as well as inert. DOMAIN_PRIMARY is claimed only by
roles/svc-prx-openresty, which declares no sso block, so it is never a
registered redirect URI - the parameter would have been rejected on
clearnet too the moment it became effective. Under tor it also glued the
app's own scheme to the portal's host.
Removing rather than correcting is what keeps this safe: every way this
could break requires a redirect target to be validated, and after the diff
there is no target. The emitted request is a strict prefix of today's.
The same three lines were duplicated verbatim in four web-app roles;
they collapse to the bare endpoint the shared role already passed. The
caller graph is closed - initLogoutPatch appears exactly three times
(definition, export, call site) - so the arity change leaves no orphan.
Fixes nothing that was failing; no CI job was red because of this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
54aeba7e0 pinned /etc/resolv.conf at 127.0.0.1 so onion names route
through the role's dnsmasq. That pin has no fallback, so every way
dnsmasq can stop answering takes the whole host's name resolution with
it.
It did. In run 32284541284 job 96178507650 the rescue bundle shows
sockets-udp.txt with zero :53 entries, `dig @127.0.0.1` returning
"connection refused", the dnsmasq process alive (state S, PPID 1) and
systemctl reporting `loaded active running` - 1201 resolution failures
from 21:58:17 onward, never recovering, while CoreDNS on the configured
upstream answered correctly the whole time.
The second entry is the same address dnsmasq already forwards to
(networks.internet.dns, the upstream 54aeba7e0 introduced), so this adds
no new destination: with dnsmasq up, glibc never reaches it; with dnsmasq
down, the host reaches the upstream directly instead of nothing. It is
the instinct plugins/lookup/container_dns.py already documents for
containers - "the clearnet resolver stays as a second entry so that
losing dnsmasq costs onion names rather than all name resolution" - and
daemon.json has carried both entries all along. Only the host was left
without one.
Unconditional, not guarded: router.yml:2-7 asserts networks.internet.dns
is non-empty before the file does anything else, so a guard would be dead
code.
Deliberately not fixing this by dropping bind-dynamic from
templates/dnsmasq-tor-onion.conf.j2. Without the directive dnsmasq binds
0.0.0.0:53, which on Ubuntu collides with systemd-resolved on
127.0.0.53:53 - reproduced on dnsmasq 2.93: "failed to create listening
socket for port ...: Address already in use", exit 2. bind-interfaces is
worse: TOR_CONTAINER_DNS_HOST is a deploy-time docker0 fact, and dnsmasq
starts at boot before docker0 exists. A fallback resolver covers the
listener loss, the bind collision and the boot-order miss at once.
Known trade: while dnsmasq is down, .onion lookups fall through to the
clearnet resolver and return NXDOMAIN instead of timing out. RFC 7686
says .onion must not reach DNS, and the role's own test cannot see this
because it resolves onions through --socks5-hostname.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The probe abandoned each attempt at --max-time 30 while Tor's SocksTimeout
sits at its 120s default (no SocksTimeout anywhere in the repo; positive
control SocksPort hits templates/torrc.j2:1). It gave up at a quarter of
Tor's working window, and destructively: every retry opens a fresh SOCKS
connection, discarding the descriptor fetches and rendezvous retries Tor
was still working through.
Run 32284541284 job 96178512465 shows the cost. Ansible recorded start
07:22:17.260, end 07:26:33.366, delta 0:04:16.105994. The first domain
burned its deterministic 6x30 + 5x10 = 230s and failed; the second then
succeeded on its first attempt but needed ~23s of that 30s window - 77%
of the limit. The transient had cleared at t~233s. The first domain's
budget ran out three seconds short of it.
Two round numbers, one behaviour: 5 attempts at 60s is 340s of patience
against today's 230s, so the same transient now has 107s of margin
instead of -3s, and a 23s connect sits at 38% of the window instead of
77%. LAM#1 (job 96179137200) is the near-miss that corroborates it - four
dead attempts, then a ~4s success on the fifth.
meta/tests.yml is required, not optional. roles/test-e2e-cli/tasks/run_one.yml:74
runs the script under `timeout {{ _cli_timeout }}`, and with no meta/tests.yml
this role fell through to the 3600s default in roles/test-e2e-cli/defaults/main.yml:3.
Widening the window moves the all-domains-fail ceiling from 16 domains to
11, and a 13-domain role demonstrably exists (Jellyfin#0, four passes of
"13 probed, 0 failed"). 7200 restores the headroom.
The `|| true` goes with it: the script runs under `set -uo pipefail` with
no -e, so the command substitution's exit code was already discarded and
line 63 branches on ${code}. tests/lint/filesystem/shell/test_no_or_true.py
never saw it because its regex wants #, ; or EOL after `true` and a `)`
follows here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4a543b3e1 moved the assertion back onto /admin so a surviving session
would fail where it means something. It does fail there - and the reason
is that nothing waits for the logout to finish before the page is torn
down.
waitForLoadState only observes a navigation that is already committed; it
never waits for one a click is about to start. Shopware's SPA logout
triggers no document load at all, so the call returned in 0.7ms and the
gotoOnion below fired 3.4ms after the click, killing the in-flight
teardown. Measured in run 32284541284 job 96178512465's trace: click at
10363.718, waitForLoadState returns 10420.992, goto starts 10422.365.
Proof the teardown never completed: POST /api/oauth/token with
grant_type=refresh_token returned 200 five minutes later, and
_action/user/logout appears zero times in the network log (positive
control _action: 6).
Waiting on the credential instead of on a load states the real
precondition, and it is faster on the happy path than any fixed settle
would be - it ends the moment the cookie goes. The catch keeps a timeout
from masking the assertion below it with a different error.
Six jobs across both runs, and the grid rules out the two obvious
culprits: variant 2 carries no injected logout script and still fails,
and variant 2 swarm+clearnet fails while variant 2 swarm+tor passes. It
is a race, not a tor defect and not an injector defect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tcp flavor renders ["CMD","bash","-c","exec 3<>/dev/tcp/..."]
(utils/docker/healthcheck/probes.py:159-177), but ejabberd/ecs:26.07 is
assembled FROM scratch and carries no bash, so every probe returned
'exec: "bash": executable file not found in $PATH' and the container
stayed Health=unhealthy forever. Confirmed from two rescue bundles in
run 32284644282 (swarm+clearnet snapshot 20260820040348Z, swarm+tor
snapshot 20260820035204Z).
That surfaced three different ways and cost ten jobs across both runs:
compose waited out its 1200s gate, swarm's routing mesh never admitted
the unhealthy task so the proxy answered 502, and the stack convergence
wait exhausted 600 attempts. Every cell of the matrix was red - both
variants, both deploy modes, both exposures - with no green counterexample.
Declaring `test` without `flavor` routes through class Custom
(probes.py:99-105), so probes.py is untouched and the twenty other roles
on flavor tcp/http_status keep byte-identical argv; svc-db-postgres and
web-app-mastodon already use this escape hatch. The probe is exec form,
so it needs no shell at all. retries 20 at the 30s default interval
matches the image's own 10x60s tolerance, and timeout 10s covers the case
where the ejabberd_conf volume masks CTL_OVER_HTTP and ejabberdctl falls
back to spawning an Erlang ctl node.
port_key goes because its only consumer defaults to http anyway
(plugins/lookup/container_healthcheck.py:111-114); ports.internal.http
stays in use through container_ports.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
