Under swarm an unpinned role's volumes are rewritten onto the shared NFS
export while its services run one task per node. A role that declares no
RDBMS keeps its state in-process, so several replicas then write the same
SQLite file, Mnesia store or search index over NFS and corrupt it. This
already cost a deploy: openclaw ran three replicas against one shared
export and its state directory answered "file is not a database".
The existing unpinned-no-engine-volume lint states the right rule but
detects state through a hardcoded path list, so app-specific paths slip
through and each case gets pinned reactively after it fails. Add a lint
that joins three facts the roles already declare: the volume is NFS
backed (asked of the production swarm_nfs_backed, not re-derived), it is
mounted read-write, and the role runs no RDBMS. Share-safe payloads carry
a justification marker.
Replicas resolve service-first and fall back to the primary entity, and
a role whose swarm mode is disabled is skipped; both cases exist in tree.
Also correct the requirement and the design doc, which still described
the per-volume opt-in model that b09598dc8 replaced with placement
derived backing. Requirement 023 contradicted itself: line 76 already
described the implemented behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both sidecar templates rendered a swarm service with no deploy: block at
all, so the kata placement constraint never applied and a non-kata
workload could land on a kata node under runsc. They also ran without
resource limits under swarm.
Include deploy.yml.j2 from both templates. Guard the replica count with
an explicit override, because deploy.yml.j2 otherwise falls back to
compose_replicas (the group size): these caches have no clustering, so a
replica per node would leave a consumer's session missing from the other
instances. Swarm already defaults to one task without a deploy block, so
the forced value preserves the current behaviour.
The central path is untouched: svc-db-redis/compose.yml.j2 reaches
deploy.yml.j2 via sys-svc-container/base.yml.j2, where the override is
undefined and the original resource_filter expression applies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The docstring promised the narrow form "{{ '<role>' in group_names }}" and
pointed at tests/integration/roles/meta/test_services_dynamic_flags.py, a path
that does not exist. The helper substring-matches "in group_names", so a
negated predicate matches too -- and that is deliberate: such a flag is true on
some node shape, and the static closure is the co-deploy superset, so the
dependency must resolve at tooling time either way.
tests/integration/roles/meta/services/test_dynamic_flags.py:194 asserts the same
substring rule, which is the real contract.
Reported during triage as a latent defect (seaweedfs' frontend is gated on
"not in group_names" and reads as enabled on a tor node). It is not one; the
description was. Tightening the matcher would have changed behaviour against the
test that pins it, so that is filed as won't-fix and only the wording changes
here. The README table gains the row it was missing.
No executable line in this diff.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
taiga-back sets SESSION_COOKIE_SECURE and CSRF_COOKIE_SECURE to True in
settings/common.py:198-200, and the image re-reads them at docker/config.py:83-84
via os.getenv(..., 'True') == 'True'. With neither variable set, Django stamped
Secure on the session cookie even over a plaintext http://<onion> origin.
Chromium does not treat .onion as trustworthy and drops it.
Taiga 0 of run 31340965955: the trace carries
'Set-Cookie: sessionid=370jln...; HttpOnly; Path=/; SameSite=Lax; Secure', and
grep -c sessionid over the whole network log is 1 -- set once, never sent back.
Without a session mozilla_django_oidc finds no oidc_states at the callback and
takes login_failure(), whose redirect carries a literal error=None, so Keycloak
had returned no error at all.
The filter chain is load-bearing: Django compares literally against 'True', so
the value must render True/False, not true/false. Copying the pixelfed line
(correct for Laravel) would render 'true', resolve to False, and strip Secure
from every CLEARNET Taiga session -- turning a fix into a security defect. The
expression therefore lives once in vars/main.yml as TAIGA_COOKIE_SECURE rather
than twice in the template.
env.j2 rather than the settings append: the image on hand still carries the
getenv pair, and env.j2 avoids un-grandfathering six unrelated comments in
config.append.py.j2. If a future image drops the hook, the same Playwright job
goes red again and says so.
Third instance of this class in this run, after pixelfed (Laravel) and joomla's
fallback cookie. Everything behind the cookie -- verify_token, get_userinfo,
user creation, taiga-front consuming the auth_token -- has never executed in
this stack, so the job is not expected to go green on this alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
daggerhart-openid-connect-generic calls wp_remote_post() against
http://auth.<onion>/.../token, which bottoms out in libcurl -- and libcurl
refuses a .onion host at name resolution (RFC 7686) before /etc/hosts,
extra_hosts or DNS is consulted. The result is cURL error 6, which WordPress
turns into WP_Error('http_request_failed') and a bounce to
/wp-login.php?login-error=http_request_failed. Five specs of WordPress 0 in run
31340965955 failed on that single cause, 60 occurrences in the log.
Only a configured proxy makes curl skip the check, and socks5h:// hands the name
to Tor. The mu-plugin hooks http_api_curl because http_request_args cannot
express a proxy and WP core's WP_HTTP_Proxy hardcodes CURLPROXY_HTTP, so SOCKS5
is reachable only on the raw handle. env.j2 sets WORDPRESS_OIDC_SOCKS_PROXY only
when the issuer is an onion, so a clearnet deployment never routes through Tor.
Deliberately not adding the onion_oidc_socks include: 238b8b41f established it
is a no-op for this client, since libcurl never reads /etc/hosts here.
Depends on 1bb31fba4 -- without it swarm renders socks5h://host.docker.internal,
which is unreachable from inside the overlay.
The plugin's behaviour is unexercised: syntax checked, never run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pixelfed's config/session.php reads env('SESSION_SECURE_COOKIE', true), and the
role never set the variable, so Laravel stamped Secure on the pxfs cookie even
over a plaintext http://<onion> origin. Chromium does not treat .onion as a
trustworthy origin and drops it: the Pixelfed 0 trace of run 31340965955 shows
four distinct pxfs values inside a single login while the only cookie coming
back is Matomo's, which carries no Secure.
With no session the OIDC callback has no stored state and
RemoteOidcController.php:49 calls hash_equals(null, ) -> TypeError -> 500,
on all three replicas (7 on wrk-02, 3 on wrk-01, 2 on mgr-01). The manager
replica failing too is what rules out the node-locality class here.
The line mirrors FORCE_HTTPS_URLS two rows up, same lookup and same filter
chain, so a clearnet deployment renders true -- Laravel's current effective
value -- and only the onion variant changes. roles/web-app-snipe-it does the
same for the identical Laravel problem.
Everything behind the login is untested rather than green: only the guest test
passed, and it passes precisely because nothing authenticates.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each of the seven addon specs opened a fresh context and ran the full SSO
round-trip. Over Tor that is roughly twelve minutes apiece, and odoo 0 of run
31340965955 died on 'Test timeout of 600000ms exceeded' inside
gotoOnion -> loginToOdoo -> contacts.spec.js:13.
authenticatedContext caches the storage state after the first login and restores
it for later callers. A restored state that no longer renders the web client is
discarded and the full login runs, so an expired session degrades to today's
behaviour rather than failing the spec.
Coverage is not reduced: the SSO flow keeps its own dedicated tests in
playwright.spec.js:98 and :160 (admin and biber, log in / verify ui / log out),
which this change does not touch. The seven addon logins were repetition of an
already-covered path, and no addon spec asserts anything about the login or
needs an anonymous session. What does change is isolation -- the specs now share
one restored session instead of each starting clean.
The TEST_E2E_PLAYWRIGHT_ONION_TEST_TIMEOUT bump belongs in this commit: four
specs lost their own test.setTimeout(resolveTimeout(180_000)), which resolved to
900s on an onion. Without the bump those four would regress from 900s to 600s.
Cause B is untouched: the global ONION_RUN_TIMEOUT of 5400s left eight tests
unstarted. The cache should relieve it, but that is arithmetic on measured
single-login times, not an observed run, and raising a timeout that spans every
onion role needs its own measurement.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The plugin stored the fallback sticky bit in a cookie set with 'secure' => true
unconditionally. Over a plaintext http://<onion> origin Chromium does not treat
the origin as trustworthy and drops it, so ?fallback=local set the bit, the
browser discarded it, the next request found nothing and the guard bounced the
user back to Keycloak -- a loop with no error anywhere.
Proven from the Joomla 0 trace of run 31340965955, both cookies set by the same
origin in the same request chain:
1227964e...=dd529f27...; path=/; HttpOnly <- session, no secure
kc_fallback_local=1; ...; path=/; secure; HttpOnly; SameSite=Lax
Across the 50 request Cookie headers in that trace the session comes back 3
times and kc_fallback_local never. The session is therefore the transport that
demonstrably works on this origin.
Moving the flag there also lets the expiry be checked server-side instead of
delegated to the cookie, and an elapsed flag is cleared rather than lingering
for the rest of the session.
Third instance of the same class in this run: pixelfed's Laravel
SESSION_SECURE_COOKIE and taiga's Django SESSION_COOKIE_SECURE default to true
and fail the same way over an onion.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The service declared no replica count, so it took the topology default of
three and spread them over two nodes while its state directory is one NFS
export shared by all of them. The agent keeps that state in SQLite, and the
swarm run shows what concurrent writers do to it: one replica exits with the
database reported as not a database, another cannot even create the state
directory.
Those exits are what started the restart storm the run then failed on, and
they are runtime independent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A paused update never completes on its own: the updater returns early on that
state, before the branch that would finish it, and refuses to start again while
an update status is set. The restart policy brings the replicas back but does
not touch the status, so the service can sit fully replicated while the gate
still refuses it.
The probe polled that for six hundred attempts and reported a timeout after
twenty minutes. It now separates a state worth another poll from one that is
final, and the wait distinguishes the two exit codes instead of retrying every
refusal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The override was assembled inline in compose.yml.j2 and handed to
container_volumes as an extra bind mount. A bind requires the source file on
every node, and in swarm it exists only where the template rendered: Nextcloud 4
and Fediwall 2 of run 31340965955 both died on 'bind source path does not exist:
/opt/compose/mastodon/overrides/zzz_disable_force_ssl.rb', followed by
mastodon_migrate.1 exiting non-zero.
Declaring it in meta/volumes.yml next to force_ssl_override, with type: config,
lets the manager distribute the content instead. It also reaches the migrate
service, which the macro-scoped extra_volumes never did.
MASTODON_CONTAINER_OIDC_HTTP_DISCOVERY_PATH is gone; the target now lives in the
declaration and the variable has no remaining reference in the repo.
Not proven: that the migrate exit-1 was caused by the missing mounts. The bind
rejection is byte-for-byte in the log; the causation would need rails db:prepare
against a live postgres. Both jobs also aborted inside the mastodon dependency,
so their own target roles remain untested rather than green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BOOKWYRM_HOSTNAME fed both the container's hostname: field and Django's
ALLOWED_HOSTS/DOMAIN, but it was sourced from container_hostname, which is
computed for sethostname(2) and clamped to 63 characters. BookWyrm's onion
domain is longer, so the lookup degraded to the bare entity name and Django was
told to accept 'bookwyrm'.
The rescue artifact of BookWyrm 0 in run 31340965955 carries the contradiction
verbatim: ALLOWED_HOSTS=bookwyrm,127.0.0.1,localhost next to
BOOKWYRM_URL=http://book.mt2j65....onion/. Every request under the onion host
was a DisallowedHost, so the app answered 400 and the OIDC spec never got past
the first page.
The two concerns are now separate: hostname: keeps container_hostname, and
BOOKWYRM_HOSTNAME resolves the real domain for the HTTP layer.
Same root as the pretix Host-header failure, different manifestation: pretix
hardcodes ALLOWED_HOSTS = ['*'] and gates in its own middleware instead.
Downstream of the 400 nothing has ever run -- test-oidc-security.js in
particular is untested, not green, until a rerun.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extra_hosts mapped auth.<onion> to host-gateway in both deploy modes, but
host-gateway is a per-node address while openresty publishes :80 with mode: host
under node.role == manager. Every replica scheduled on a worker therefore dialled
its own gateway, where nothing listens, and the server-side OIDC token exchange
died with 'Connection refused'. Penpot 0 of run 31340965955 measured it: 60 of
the ConnectException lines on wrk-02, 20 on wrk-01, zero on mgr-01.
In swarm the pin now resolves to the stack host's routable address -- the same
destination svc-net-tor already sends all hidden-service traffic to via
TOR_HS_BACKEND_HOST. That address reaches every task, because
plugins/action/stack_host_template.py skips any host whose IS_STACK_HOST is
false and IS_STACK_HOST is true only on a swarm manager: the compose file is
rendered once with the manager as inventory host, so ansible_facts carries the
manager's IP and swarm applies the same extra_hosts to every task.
Retargeting rather than dropping the pin, because a manager-pinned consumer
(the keycloak sso-proxy) is measurably working through this very line today.
The guard fa2f6eb7e removed would have deleted it for them too.
Keyed on compose_mode_force or DEPLOYMENT_MODE, not the bare cluster mode:
container_networks resolves attachments through the same effective mode, so a
role with an override must not lose its pin and its overlay at once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The spec looked for a link whose accessible name matched /administration|admin/i
and only navigated to /admin when none was visible. Mailu's admin entry point is
a Roundcube task-bar BUTTON, so a link locator could never match it -- the
branch fired exclusively on false positives.
In PhpMyAdmin 0 of run 31340965955 it resolved to an inbox row: the trace shows
the click landing on <a href="/webmail/?_task=mail&_mbox=INBOX&_uid=20&_action=show">,
whose subject is 'web-app-phpmyadmin-swarm-mgr-01 is alive' -- mail the job's own
health mailer had just delivered. Only that job is affected, because only there
does the inventory hostname contain 'admin'.
Everywhere else the else-branch already ran and passed: the mailu junit archived
by the green onion job Akaunting 0 reports tests=5 failures=0 with 'mailu: sso
login, open admin interface, logout' at 17.98s. Navigation is also the Tor-safe
form -- clickThroughMailuSsoPage already avoids clicks over Tor because Matomo
link-tracking intercepts them and the navigation never commits.
The added visible filter keeps the heading assertion off Roundcube's sr-only
<h1 class="voice">Mail</h1>, which reported a misleading 'Received: hidden'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
container_hostname is computed for sethostname(2) and clamped to 63 chars
(plugins/lookup/container_hostname.py). pretix's onion domain is 74, so the
lookup degrades to the bare entity name 'pretix' -- and container_healthcheck
forwards that value into the curl probe's Host header.
pretix gates on Host in MultiDomainMiddleware, not in ALLOWED_HOSTS (settings.py
hardcodes ALLOWED_HOSTS = ['*']). It accepts the site domain, a KnownDomain, or
one of LOCAL_HOST_NAMES = ('testserver', 'localhost'); everything else gets a
400. The probe could therefore never pass, swarm killed the task with exit 137
and the stack never converged -- Pretix 0 of run 31340965955.
A/B against pretix/standalone, same volume and same PRETIX_PRETIX_URL, probe run
by Docker itself, only the header differing:
Host: pretix -> curl (22) 400 x4, unhealthy streak=6
Host: localhost -> exit 0 x3, healthy
The Tcp flavor already hardcodes Host: localhost for the same reason; curl was
the outlier. Reusing a kernel hostname as an HTTP Host is the underlying design
bug, but defaulting the curl probe would change six roles' probes at once, so it
stays a separate change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Driven by an until loop of 600 polls, the probe exited 1 without a word on
every refusal, so a service that never converged produced a twenty minute wait
and then a fatal carrying an empty stdout and an empty stderr. The reason had
to be reconstructed from a rescue artifact, and the update-status branch is not
recorded there at all.
Each refusal now names itself on stderr. Only the branch with task rows to show
prints them, filtered to the rows carrying an error and capped, which is the
shape the neighbouring stack_ready.sh already follows; the update-in-progress
branch stays a single line because it is the normal state of a healthy rollout
and fires on every poll.
A timed out service ps used to abort the script with its own code and no
message, reproducing the very symptom this addresses. It is now named too.
Adds the unit tests the script never had, mirroring the sibling's, including
the assertion that error-free task rows stay out of the output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role declares an oauth2 proxy, so an unauthenticated request to the root
answers 302 towards the identity provider. Its status codes still described the
shape from before the gate existed, where the api server mapped only /health
and /v1 and the bare root answered 404 by design. The webserver health check
therefore refused the redirect and the service failed to start, in compose and
in swarm alike.
Keeps 404 alongside: the variant that disables sso is not proxied, and there
the root still answers 404. Both variants share one runner and one declaration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gates that assert no sign-in affordance survived the OIDC round-trip
matched /log\s*in|sign\s*in|sso/i unanchored, so the pattern hit ordinary
navigation entries of an already authenticated page as substrings:
Accessories, Login Enabled, Blog inbox, Asset Login History. Any of them made
the gate report 'OIDC login did NOT establish a session' while the login had in
fact succeeded. Snipe-IT 2 of run 31340965955 failed exactly that way.
LOGIN_CONTROL_NAME anchors both ends and still admits the 'with ...' suffix, so
'Sign in with Keycloak' keeps matching and the pattern is not narrowed into
matching nothing.
The pre-login fallbacks in admin.js:58 and biber.js:79 stay deliberately broad:
they hunt for a sign-in control on the UNAUTHENTICATED landing page after the
anchored attempt misses, where no authenticated menu exists to collide with.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role has shipped complete mail plumbing all along: the Dockerfile installs
msmtp and msmtp-mta (so PHP mail() relays through it), vars/main.yml:23 renders
sys-svc-mail-msmtp's msmtprc template, meta/volumes.yml mounts it at
/etc/msmtprc, and tasks/main.yml:26 transfers it on every deploy, ungated.
What was missing is the services.email declaration. container_healthcheck only
emits the msmtp half of the msmtp_curl probe when services.email.enabled is
true, so the flavor the role declares has been rendering as a bare curl and the
relay path was never exercised. WordPress mail could have been broken for
months without a single red job.
Declaring email restores the smoke test, and the assertion added after
convergence reports the relay's refusal in the clear instead of leaving it in a
health log nobody reads.
The flag is reactive ('web-app-mailu' in group_names), so it does not pull the
provider in: the derived swarm closure is 24 members with mailu both before and
after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The msmtp_curl probe joined its mail branch to the curl probe with '&&', so a
relay that refuses the startup message short-circuited the whole command: curl
never ran, the container stayed unhealthy forever, and the swarm converge gate
burned all 150 retries before aborting the deploy. Friendica 0 and Fediwall 0
of run 31340965955 died that way, reporting only 'not converged:
friendica_friendica' with the real cause buried in a rescue artifact.
That contradicted the probe's own docstring, which promises an unrelated SMTP
outage cannot flap the container into unhealthy -- it only delivered that for
email_enabled=False. sys-ctl-hlth-msmtp/files/script.sh:23 has always tolerated
exactly these exit codes; the container probe was the outlier.
Liveness is now curl's verdict alone. The '&&' before touch stays, so a refused
mail leaves no marker and the next probe retries it rather than turning the
smoke test one-shot.
Detection does not disappear with the coupling: web-app-friendica now asserts
the delivery marker after convergence and fails with the relay's reason instead
of a timeout three layers away. sys-ctl-hlth-msmtp is a systemd timer and the
Mailu Playwright roundtrip submits through Roundcube, so neither covers a
container relaying over port 25 -- this assertion is what does.
The marker path moves into utils.docker.healthcheck.MAIL_MARKER and reaches
Ansible through lookup('container_mail_marker'), so the probe and the assertion
cannot drift apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A lookup plugin receives group_vars through variables.get() untemplated, and
group_vars/all/18_swarm.yml:2 defines DEPLOYMENT_MODE as a Jinja expression.
The comparison against 'swarm' could therefore never match, so every swarm
deploy fell through to _COMPOSE_HOST and baked
socks5h://host.docker.internal:9050 into Nextcloud's proxy system value.
svc-net-tor publishes SOCKS with mode: host, so that listener exists on the
manager alone. With three Nextcloud replicas against one Tor node the failure
is node-determined, not intermittent: the 104 'cURL error 7 ... over proxy
host.docker.internal' lines in run 31340965955 split 56 on wrk-02, 48 on
wrk-01 and 0 on mgr-01, which is what made the OIDC callback 500 on 2 of 3
replicas and read as a 39% flake.
Templating first yields socks5h://tor:9050 in swarm, which every replica
reaches over the overlay it already joins through the onion_sso consumer rule.
Compose is unchanged: a single-node inventory renders the expression to
'compose', and a suppressed template error falls through to the same branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The quiesce guard spelled the dev container as the literal
infinito_nexus_debian, so it only ever matched a checkout whose .env was
generated for debian without a project prefix. It missed the self-hosted
runner form infinito_<project>_nexus_<distro>, any custom.env pin, and any
.env generated for another distro — and it missed them silently, because a
non-matching grep just skips the block and leaves the competing onion
publisher running.
INFINITO_CONTAINER is the SPOT for that name: compose.yml uses it as
container_name, and scripts/meta/env/load.sh (already sourced at line 19)
exports it. Proven against the container running on the author's host: the
literal does not match it, the variable does.
The outer `docker ps | grep -qx` pre-check is gone with it — `docker exec`
on an absent container already exits non-zero with stderr suppressed, and
the `if` absorbs that even under `set -euo pipefail`.
The variable joins the line-21 assertion so a broken env fails by name
instead of dying inside a pipeline subshell, which would reproduce exactly
the silent-skip class this commit removes. compose.yml already requires the
same variable with `:?`, so this is not a new demand on the environment.
tests/lint/repository/test_no_hardcoded_container_name.py pins it: run
against the pre-fix file it flags all three lines, and it stays green on
the distro-agnostic spellings in stack_refresh.sh and worktree/up.sh.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The split left the swarm track without any teardown: a `base swarm` run
started the development stack in base/04_bootstrap.sh and never brought it
down, so it leaked AppArmor, dev DNS and IPv6 state onto the host and ran
the swarm cluster alongside a live dev stack the old suite had already
released.
The old single job ran `make compose-down` + `make environment-teardown`
as step 10, immediately before 12_swarm.sh and 13_roundtrip.sh. The swarm
track now reproduces exactly that: 01_teardown releases the stack before
02_zombie and 03_roundtrip (which are byte-identical to the old steps apart
from their source path), and 04_teardown leaves the host clean — something
the old suite never did, since it ended on the roundtrip's restarted stack.
Both closing steps and the new leading one share utils/teardown.sh so the
teardown cannot drift between tracks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scripts/tests/environment/ becomes scripts/tests/workspace/ with one
subdirectory per track. base/ prepares the workspace and is a prerequisite;
compose/ and swarm/ are independent of each other, so CI can run them as
separate jobs with a budget each. 00_orchestrator.sh runs the tracks named
by INFINITO_WORKSPACE_TRACKS and aborts on an unknown one.
The job count stays at five: cli.meta.ci.dev_runtime matrix assigns exactly
one track per distro and flips the split with the run number, so two
consecutive runs cover both tracks on every distro without doubling the
matrix. The workspace_track input (rotate|compose|swarm) pins it from manual
steer through the orchestrator for a targeted re-run.
Job names carry the track glyph from utils/symbol_glossary and the image
shortened to its last two segments, so quay.io/centos/centos:latest reads
as centos:latest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils/tls_common.py:275-283 returns enabled=False for any onion primary domain,
so lookup('tls', <app>, 'protocols.web') is "http" there and "https" everywhere
else. Every OIDC consumer inherits that through group_vars/all/11_oidc.yml, which
is where PeerTube's OAUTH_HTTP_REQUEST_FORBIDDEN comes from — 52 roles hang off
this one derivation and nothing covered it.
test_app_override_enabled_false already asserts protocols.web == "http", but it
gets there through an explicit server.tls.enabled: false, not through the onion
rule. These two cases exercise the rule itself, in both directions.
The mid-code comment above the sys.modules shim goes because the comment lint
flags it once the file is touched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Joomla's Keycloak plugin builds a jumbojett OpenIDConnectClient, which bottoms
out in libcurl — and libcurl refuses a .onion host at name resolution, before it
ever consults /etc/hosts, extra_hosts or DNS (RFC 7686). 20ddda8c1 measured that
verbatim: `curl --noproxy '*'` exits 6, and even
`--resolve auth.<onion>:80:127.0.0.1` exits 6. Only a configured proxy makes curl
skip the check.
So the `onion_oidc_socks.yml.j2` include the role already carries is a no-op for
this client: its extra_hosts entry can never be reached. Server-side discovery
against an onion IdP could not work, whatever the scheme — this class is
unaffected by TLS on the onion, which is why it needs its own fix.
buildOidcClient now calls setHttpProxy when JOOMLA_OIDC_SOCKS_PROXY is set, and
env.j2 sets it only when the issuer is an .onion, so a clearnet deployment is
untouched and never routes discovery through Tor. web-app-nextcloud is the
working reference for the same construction. The swarm side needs nothing: the
onion_sso overlay is declared provider-side in svc-net-tor/meta/networks.yml and
attaches to every app with sso enabled.
The six narration comments that fell out of Keycloak.php were pre-existing and
lost their grandfathering when the file was touched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
openid-client v6 refuses a non-HTTPS issuer, and an onion-exclusive Keycloak has
none: the onion publishes only HiddenServicePort 80, and https there is blocked
on four separate things (see TODO.md). PeerTube's plugin therefore threw
OAUTH_HTTP_REQUEST_FORBIDDEN during discovery, never built a client, and answered
2326 logins with "OpenID provider unavailable" — while `test -d` on the plugin
directory reported the deploy as ok.
Two changes, one honest and one not:
The gate at the end of enable.yml now fails when the container log carries
"Initial OpenID discovery failed", so a plugin that registered without building a
client stops the deploy instead of surfacing hours later in the browser suite.
The sed over the installed dist is a hack and is marked as one in TODO.md. It
widens the trigger of the escape hatch upstream already implements — the plugin
gates `openid.allowInsecureRequests` on NODE_ENV alone, and setting NODE_ENV=dev
in production would swap PeerTube's whole runtime profile. The `grep -q` after
the substitution is what keeps it tolerable: an upstream release that moves the
line fails the deploy loudly rather than silently doing nothing.
Both are driven by one variable, PEERTUBE_OIDC_INSECURE_ISSUER, which requires
SSO enabled AND tor enabled for peertube AND an .onion issuer, so no deploy
without Tor can execute the patch or ever fail on it. Verified with Ansible's own
templar across seven cases, including https://onionsoup.example.com/ to catch a
too-greedy match. The sed itself was run against the real 2.0.1 tarball: exactly
one hit, `node --check` clean, and the patched expression exercised in four
environment combinations.
Exit path in TODO.md: the plugin is an official Framasoft one (AGPL-3.0 in the
repo root, the npm package merely ships no license field), and the proper fix is
a `registerSetting({ name: 'allow-insecure-issuer' })` OR-ed into the same
condition. Issue text and patch are written; upstream filing is waiting on a
framagit account.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
keycloak-js dropped the UMD build in 26.0.0. The package now ships only
lib/keycloak.js with "type": "module" and a default export, so the CDN
path dist/keycloak.js resolves to a file that no longer exists, and a
classic script tag could not parse the file that does. The adapter never
loaded, window.__oidcLoginReady was never set, and every app job that
co-deploys the dashboard failed its Playwright run on the resulting
console error.
The loader now uses a dynamic import and takes the constructor from the
module's default export. That also retires the window.Keycloak check,
which had nothing left to check once the global stopped existing.
Module fetches always run in CORS mode, unlike classic script loads. The
internal CDN emitted no Access-Control-Allow-Origin at all, so it could
not have served an ES module to the dashboard's origin. jsdelivr sends
"*", which is why the external flavor would have hidden the gap.
Every init option the template passes carries the same default in 24.0.5
and 26.2.4, and the silent-SSO postMessage contract is unchanged, so the
adapter's behaviour is untouched beyond the packaging.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mkstemp creates its file 0600 and Path.replace carries that mode into the target, so every freshly rendered Corefile was owner-only while coredns runs as nonroot. The container exits with 'open /Corefile: permission denied', name resolution dies for the whole stack, and deploys then fail at whatever they try to fetch first.
Long-running stacks hid it: coredns reads its config once at startup, so a container created before a re-render keeps working. Only new worktree stacks broke.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit efc281e327)
A branch with no tracking remote fell straight through to origin, so on a fork checkout every run lookup asked the upstream repository and answered HTTP 404. i8cifail could not reach a single run of this branch.
Consult remote.pushDefault between the tracking remote and origin. That is git's own answer to where the branch is pushed, hence where its runs live, and the i8puma alias already reads exactly this key — the two resolution paths now agree instead of contradicting each other.
The fallback chain had no test at all: every existing case mocks resolve_repo away. Four cases now pin it, including a pushDefault naming a remote that no longer exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 123794c7f4)
Retriggering leaves the original run alive, so its remaining jobs keep runner
slots while the newly triggered ones queue behind them. Force-cancelling first
frees those slots.
The cancel is not chained with &&: the usual case is a run that already
finished, where force-cancel reports an error, and chaining would block the
trigger exactly when it is wanted. A failed cancel prints a line and the
trigger proceeds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bfe7741c6 anchored both credential fields to form:has(input[type=password]) to keep the username out of a header search box. Funkwhale renders no form element at all: the login chunk of funkwhale/front:2.0.6 emits four div and two span tags and nothing else, the inputs coming from a shared component. The locator therefore matched nothing and the spec would have failed on its first assertion, blaming the application for the author's mistake.
Return to the visible-scoped selectors the shopware spec uses, which did run green against a live stack. The search box that motivated the scoping was never evidenced: the failure artifact shows navigation links on the landing page, no visible text input. The URL poll added by the same commit is unaffected and stays.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The hourly rerun bot looked only at the newest run's conclusion. When that run was cancelled while an older one was still pending, reviving it added a second waiter to the ref's global-ci concurrency group — and cancel-in-progress evicts only in_progress runs, so neither could start and every later dispatch queued behind them.
Query the branch once with status included and leave it alone while anything on it is still active. The selection itself is unchanged; it now reads the same response instead of issuing a second one.
Scope, measured rather than assumed: this does NOT explain today's deadlock. The blocking revival happened at 14:51 on an idle branch, which the guard permits, and the later revival at 15:51 never produced a second attempt (run 31405424413 is still attempt 1). The guard covers a narrower case than the incident that motivated it.
scripts/github/rerun had no tests at all. The harness added here runs the script against a stubbed gh, following the pattern of tests/unit/scripts/tests/deploy/utils/test_rescue_index.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A branch with no tracking remote fell straight through to origin, so on a fork checkout every run lookup asked the upstream repository and answered HTTP 404. i8cifail could not reach a single run of this branch.
Consult remote.pushDefault between the tracking remote and origin. That is git's own answer to where the branch is pushed, hence where its runs live, and the i8puma alias already reads exactly this key — the two resolution paths now agree instead of contradicting each other.
The fallback chain had no test at all: every existing case mocks resolve_repo away. Four cases now pin it, including a pushDefault naming a remote that no longer exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Handlers are play-global and deduplicated by name, the last registration
wins, and a notification resolves in the handler's own context rather than
the notifier's. Every app normally re-registers the compose handlers in its
own context through the backend stack right before it notifies, which is what
keeps that resolution correct.
Reconciling entered each client through tasks_from, so the client's core never
ran and no re-registration happened. The last registration in the play then
came from a role imported without vars, which lets the handler-role literal
through, and the guard caught a flush bound to a role that owns no instance
directory. Had the last registration belonged to a real app the guard would
have passed and the restart would have gone to the wrong one, unnoticed.
Rebinding through an include gives the client id as an include parameter,
which outranks the role vars the handler import sets. Flushing inside the loop
keeps a client that does not flush on its own from having its notification
collapsed into the following client's context at the trailing flush.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
c98260afc moved force_ssl_override from `type: bind` to `type: config` so the
override reaches every node, but left `read_only: true` on both mounts. The
volumes schema allows that key only for bind and volume mounts:
(force_ssl_override).mounts[0]: 'read_only' is only valid for type
bind/volume (got type='config')
A swarm config is read-only by construction, and `mode: "0444"` one level up
already says so, making the flag meaningless as well as invalid.
This shipped red: the schema lint has been failing since c98260afc, and it only
surfaced now because that commit landed without running the ansible role-meta
lints.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The convergence-gate guard added in 5555d9530 discovered its callers with
`ROLE.rglob("*.yml")`, a raw directory walk. tests/lint/filesystem/python
forbids exactly that in test code: the suite shares one filesystem walk and one
read per file through `utils.cache.files`, and a private rglob opts out of both.
A guard against drift that itself broke a rule is not much of a guard.
Behaviour is unchanged — the guard still finds both callers of stack_ready.sh by
scanning rather than by a hardcoded list, and still asserts it found at least two
so an empty scan cannot pass silently. Re-proved after the migration by putting
post_deploy.yml back to a hardcoded 150: the guard fails and names both sides.
`iter_project_files` yields strings, hence the `Path(...)` for the finding label.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Tor-path credential tests tunnel their client through a local SOCKS5 shim.
When the CONNECT fails, handle() closed the client socket and returned, so the
client library saw nothing but an empty socket and reported whatever its own
generic read-EOF error is.
For MySQL that is `(2013, 'Lost connection to MySQL server during query')` —
which claims a query was in flight when none ever was. Fediwall 1 printed that
twenty times and it sent the investigation looking for a server that had hung
up, when in truth no HiddenServicePort existed behind the dialled onion port.
Postgres and ldap3 are less misleading ("server closed the connection
unexpectedly", "Connection reset by peer") but equally silent about the cause.
Reproduced for all three against a SOCKS5 proxy that completes the no-auth
greeting and rejects CONNECT with rep=5, running the real python block out of
each test.sh: every one now prints `socks connect failed: SOCKS5 CONNECT failed
rep=5` ahead of its library error.
This does not change any verdict — the root cause is fixed by 2ba3a14b9, where
tor_reachable requires both `services.tor.enabled` and an `exposed: true` that
actually earns a forward. This only stops the next occurrence from lying.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`docker service ps` renders CurrentState as prose — "Preparing 3 minutes ago" —
which cannot distinguish an image still extracting from a task idling on
something else. That gap cost a whole investigation: Nextcloud 2's flowise task
spent ~412s somewhere between pull and running, and the artifact could only
bound the window, never attribute it.
report_task_states adds the machine-readable half:
zqimhnngo3my state=preparing since=2026-08-10T12:53:40Z msg=preparing
Verified against a real swarm, read-only, not just the stub: `--type task` is a
valid type (the daemon answers "task ... not found", not a type rejection), all
three field paths resolve on live tasks, and ten inspects cost 0.07s against the
15s timeout. The type pin matters here because this repo names a role's overlay
network after the same entity as its main container, so an untyped inspect can
resolve the wrong object — tests/lint/ansible/shell/container caught that.
The test file also corrects a stale contract. 3ebd6e950 dropped the
`awk '!/error=$/'` filter precisely so a healthy-but-slow task stays visible, but
test_error_free_task_rows_are_not_echoed still asserted the old behaviour and had
been failing since. It now asserts the row IS echoed, and a second case pins that
an empty inspect result does not print a bare indented line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
