The marker claimed concurrent replicas serialise into a lock error, a symptom
that never appeared in the logs, and then that NFS carries POSIX advisory
locks unreliably, which the captured mount options contradict: the export is
mounted with local_lock=none over NFSv4.2, so locks do reach the server.
The reason that holds is the one the application states itself. It disables
WAL on a network filesystem because the shared-memory index WAL coordinates
writers through is per host and not coherent between nodes, and the board is a
single SQLite file every replica shares.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dump listed the state path itself, so the volume directories showed up but
nothing inside them. Ownership questions about a specific file were therefore
unanswerable from the artifact alone, which is how a permission failure on a
database file cost two investigations without ever being settled.
Records mode, numeric owner and size one level in. Numeric because the
container uids have no names on the nfs server, and those uids are what the
questions are about.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Docker hands out the lowest free address, so a node attached without one lands
in the same band where the cache services pin themselves and can take an
address its owner then cannot claim. That happened: a node held the
registry-cache address and every swarm cycle died at the image pull.
Each node now takes a fixed address, offset per instance because worktrees
share the primary's cache network while clearing only their own cluster, so a
surviving cluster elsewhere no longer collides with a fresh one.
A failed attach is classified rather than swallowed. The nodes already carry
extra_hosts pointing at the cache addresses, so a node that silently fails to
attach reaches neither apt nor the registry and reports it as network
flakiness. Only the markers the docker_network_noop filter already treats as
benign are tolerated; anything else stops the bootstrap.
The subnet is asserted rather than assumed, since the address arithmetic only
holds for a x.y.z.0/24.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`files/swarm/stack_ready.sh` has exactly two callers and they disagreed: the
handler defaulted to 150 polls at delay 2, and post_deploy.yml hardcoded the
same 150 with no way to override it. 389 seconds is not enough for a stack that
carries a large image.
Four jobs across two runs died on it, every one starting at "149 retries left":
Nextcloud 2 (run 31340965955) plus Workspace centos, debian and manjaro (run
31311996621), the last three all waiting on svc-db-mariadb, a role that carries
no per-role override. Nextcloud 2 is the measured case — the budget expired at
12:59:54 and the task reached Running at ~13:00:32, 38 seconds later. Nothing
was broken; the deadline was.
600 is not a new number: tasks/utils/swarm/wait_converged.yml already waits that
long at the identical delay, and eleven roles had each hand-raised the handler to
exactly 600. 150 was the outlier.
The declaration goes in group_vars/all, not in the role defaults. A role default
is invisible to the by-path include in roles/svc-db-openldap/tasks/01_docker.yml,
which pulls post_deploy.yml in after include_role has finished — verified with a
scratch play, where the same variable resolves inside the role and raises
"is undefined" one task later. post_deploy.yml already encodes that distinction:
it reads DEPLOYMENT_MODE bare and guards compose_mode_force with `| default('')`.
The lint gains a check that the gate's callers never drift apart again. It
discovers them by scanning for the script rather than listing them, so a third
caller is covered, and it asserts it found at least two so an empty scan cannot
pass silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The frontend proxies to the package-cache by name, and nginx resolves a
literal proxy_pass hostname once while parsing the config and keeps it for the
worker's lifetime; the resolver only re-resolves targets that contain a
variable. The package-cache was the one cache service without a pinned
address, so recreating it left the frontend dialling a dead one, answering 502
to everything and reporting unhealthy.
That surfaces far from its cause: pip.conf deliberately keeps the real index
URL and extra_hosts points pypi.org at the frontend, so a pip install reports
no versions at all and looks like a network fault.
Pins it the way the registry-cache and the frontend are already pinned, and
carries the address into the runner slots.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`timeout-minutes` reports the job as cancelled, not as failed, so a step gated
on `if: failure()` never runs. The upload beside it is gated on `always()`, runs,
and finds an empty directory — the workspace centos job logged exactly that:
No files were found with the provided path: /tmp/rescue-diagnostics/.
That the upload did run also proves the runner still executes steps after the
cancellation, so widening the gate is enough. Every one of these four steps sits
on a job with a 345-355 minute budget, and three workspace distros burned that
budget in full on 2026-08-10 and produced no diagnostics at all.
call-test-deploy-swarm.yml needs no change: its routine carries an EXIT/TERM
trap (954b9f1ae) that covers the same case. There is no .github/actions/, and no
other `if: failure()` exists outside .github/workflows/.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
clickOidcLoginLink assumed the element matching the persona's anchored login pattern is the provider trigger. On mobilizon it is not: the navbar carries a plain Login router link, which the anchored pattern matches, while the actual provider button sits on the page that link opens and is labelled with the instance name, SSO Login (INFINITO.EXAMPLE). The helper clicked the navbar link, landed on /login, waited fifteen seconds for openid-connect/auth, returned false, and the persona then failed its authenticated-surface assertion with the login page as its final URL.
Retrying with the loose pattern would not have helped. On the landing page the provider button does not exist at all, and on /login the navbar link stays visible and sits ahead of the button in DOM order, so both patterns resolve to it a second time. What is missing is a second resolution on the page the click opened.
Run the existing resolution again, but only when the click navigated somewhere new without reaching the auth endpoint. That second pass additionally accepts a provider control by accessible name, which is the only thing that separates the button from the navbar link. Pass one keeps its locator chain, its timeouts and its order, so a role whose strict candidate already is the provider trigger returns on the first pass and never pays for the retry; guest.js does not call this helper at all, and the remaining callers are biber.js, admin.js and the mastodon seaweedfs spec.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The overview claimed a single kernel server with one set of export options
and root squashing. The role dispatches to either the kernel server or
nfs-ganesha, only ganesha exports the export base, and only the kernel flavor
reads the options variable. It also described access as restricted to the
swarm node group while both templates additionally export to a CIDR list.
Names the sources instead of repeating their values, so the paths and the
option string stay defined in one place.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The readiness gate ahead of it only polls the login page, which the site
serves before the administrator hash is written, so a single non-zero return
against a still-initialising site failed the whole deploy. Retrying matches
the neighbouring gate and is idempotent, since the task sets the same password
each time.
Corrects the two places that still described the break-glass credential as
seeded once on first run. The requirement notes stay as they are: they
describe the upstream CMK_PASSWORD variable, which is still first-run only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
config.yaml reaches the container as a docker config, which swarm mounts
read-only. The image runs its schema migration on every boot, and because the
rendered file carries no _config_version it counts as legacy, so the migrator
backs the file up and writes it back, failing with EACCES every start. The
image offers HERMES_SKIP_CONFIG_MIGRATION for exactly this case, and the
config is rendered fresh on every deploy, so there is no persisted legacy
shape left to migrate.
Drops the volume ownership block that a wrong reading of the same failure
introduced. The rescue artifact of that run shows the directory was already
owned by the runtime user, and the log shows that user creating directories in
it, so the block only widened the mode and fixed nothing.
Keeps the single replica but states the reason the evidence supports: SQLite
on an NFS-backed shared volume, where advisory locks do not carry reliably.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guide selector asked only meta/tests.yml whether a role opts out of its
deploy mode, so a role that switches the mode off in meta/services.yml stayed
in the random candidate pool and could be deployed by an orchestrator run that
passes no role, priority or whitelist.
Consulting get_role_mode_enabled covers both modes the selector can pick, host
included, which get_role_skip does not reach.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image seeds app.ini from the flat DB_* variables only while the file is
absent, so a persisted install kept whatever credentials it was first given and
the role patched the current ones back in with sed. environment-to-ini runs
unconditionally on every start and overwrites the [database] keys in place, so
declaring them as GITEA__database__* converges the config on its own.
The restart that makes the new values take effect already comes from rendering
env.j2, whose handler the backend stack flushes before the manager tasks run,
so the patch task loses its last reason to exist.
Drops 02_database.yml and renumbers the following task files to keep the
sequence gapless.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The spec picked its username field with a global input[type=text]:visible .first(). Funkwhale's header carries a search field, so whenever that renders before the login input in DOM order the credentials would go into the search box and the run would fail later, at the password field, with a misleading message. Anchor both fields to form:has(input[type=password]) instead, letting the password field define which form is meant.
Also wait for the URL to leave the canonical domain before driving the Keycloak form. Without it a proxy that fails to intercept /login surfaces as a bare 60 second timeout inside the shared Keycloak helper, with nothing pointing at the missing redirect. The pattern is taken from roles/web-app-n8n/files/playwright/_shared.js, which runs the same chain and is green in CI.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The routine only carried an EXIT trap, so a run terminated by the enclosing
`timeout` in distros.sh:128 left no rescue diagnostics, no playwright reports
and no teardown — the exact case where the artifacts matter most.
Add a TERM trap and make the handler take the status as an argument, so EXIT
keeps using `$?` while TERM passes 143 explicitly. The `_teardown_done` latch
is required because the EXIT trap still fires after the TERM trap; without it
the whole collect-and-teardown block, including its 1500s and 900s timeouts,
would run twice.
This works because GNU timeout signals the whole process group: bash defers a
trap while it waits on a foreground child, so a bare kill to the script PID
alone would not surface it until that child exited. Reproduced under the real
shape (timeout -k): TEARDOWN rc=143, exactly once, after 2.01s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The budget formula read `retries: 3` as three attempts with two gaps. Ansible
counts the first run separately — task_executor.py:606-608 sets `retries = 1`
and adds the user value on top, so the ladder makes 4 attempts, and the sleep
guard at :734 (`attempt < retries`) yields 3 gaps, not 2. Both terms were
understated at once.
Worst case for both builds (timeout 3600, retries 3, delay 30, plus a 3600s
fallback) is therefore 18090s, not 14460s. Still inside the 20400s sweep budget
and the 21300s deploy step cap, but the reserve drops to 2310s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible's `timeout:` keyword is a local signal.alarm in the worker. Over the
docker connection community.docker runs the child inside `with Popen(...)`,
whose `__exit__` ends in an unbounded `self.wait()` — the TaskTimeoutError is
raised but cannot leave the with-block until the remote child exits. A hung
`compose build` therefore ran to the 3h59m job limit instead of the declared
budget. Reproduced locally: with-Popen 20.00s vs bare Popen 2.00s for the same
2s alarm.
Wrap both build bodies in GNU timeout so the bound is enforced remotely, where
the child actually lives. `-k 60` guarantees SIGKILL if the build ignores
SIGTERM, which is the precondition for `wait()` to return at all; measured
client response to SIGTERM is 0.12s, so the grace window is pure reserve. Same
flag pairing as scripts/tests/deploy/distros.sh:128.
The `timeout:` keywords stay — the retry-budget lint reads task["timeout"].
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shared persona helper cannot drive this role's sign-in in any matrix variant. The oauth2 ACL blacklists only /login, and sys-svc-proxy renders a blacklist as expose-everything-then-protect-the-listed-paths, so the helper's landing on / is public and the proxy never intercepts it. Funkwhale renders its own sign-in control as a bare div, which the helper's link and button role query cannot select. And application-level authentication is Funkwhale's LDAP form rather than OIDC, so even a completed Keycloak round trip at the proxy leaves the Funkwhale session anonymous.
The previous gate derived both blocks from services.sso.enabled, which unblocked the personas in exactly the variant where the chain does not work. The sibling jobs of run 31341444049 show it without interpretation: Funkwhale 0 with sso on fails, Funkwhale 2 with sso off passes, and it passes only because the block fires there.
The journey is not dropped. files/playwright/test-native-login.js drives both personas through Funkwhale's own chain: /login, the Keycloak round trip whenever the proxy gates that route, then Funkwhale's own sign-in form, an authenticated-surface assertion and sign-out back to the anonymous state. It skips when services.ldap.enabled is false, because no persona holds an account in that variant. Stage one reuses performKeycloakLoginForm from the shared helper without modifying it.
The selectors are structural rather than Funkwhale-specific: the upstream front-end source is not reachable from this environment, and the CI traces only ever captured the landing page, so the spec keys on input types the way the shared Keycloak helper does. The anonymous-state assertion is evidence-backed: the failure artifact shows the sidebar offering sign-in while unauthenticated. The spec has not yet run against a live stack.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The staging task listed three files by hand while the runner Dockerfile
COPYs five. 0a581f662 added hostpin.sh and hostpin.service together with
their two COPY lines and did not extend the list, so the build failed
deterministically:
#18 ERROR: failed to calculate checksum of ref ...: "/hostpin.sh": not found
failed to solve: failed to compute cache key
and compose pull retried it ten times -- a missing file no retry can heal.
Both files and the defect are branch-local: neither exists on origin/main.
Copy the directory instead of enumerating it. The per-file modes the list
carried were fiction anyway: every source is 100644 in git, including
bootstrap.sh which the list published as 0755, and the Dockerfile chmods
0755 on both scripts itself, so the staged mode never reached the image.
The point is not the two missing entries. A list that must be updated
whenever a file is added is silent exactly when it is forgotten -- here it
went wrong inside a single commit that touched the Dockerfile and the
directory at the same time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four readiness probes waited on lookup('domain', <app>) port 80 -- a
publication identity used as a machine address. With svc-net-tor deployed
that name is onion-only, because exclusive: true replaces the clearnet
domain rather than adding to it (utils/cache/domains.py).
Nextcloud 5 then burned its full 600s on zammad.helpdesk.<onion>:80. The task
carries no delegate_to but inherits apply: delegate_to: NEXTCLOUD_HOST_NODE
from 02_manager_ops.yml:46-50, which propagates through nested includes, so it
ran on a worker. svc-net-tor is placement: manager, so that worker had no
.onion split-DNS, no TransPort redirect and no :80 listener. wait_for swallows
socket.gaierror in a bare except and retries, so an unresolvable name is
indistinguishable from a slow service -- it just burns the timeout.
Point the probes at DOCKER_REACH_HOST and the service's own
ports.local.http instead. No name resolution, so the onion replacement cannot
reach it; node-local, so the inherited delegation stops mattering; and it hits
the service's own listener rather than the shared front proxy, which is what
the probe was supposed to measure in the first place.
Verified on the failing run's artifacts that this is node-independent in
swarm: zammad_railsserver.1 ran only on mgr-01, yet dockerd listens on *:8077
on all three nodes -- the ingress mesh. Only svc-prx-openresty and svc-net-tor
publish with mode: host; every other role, these four included, goes through
ingress.
Same swap for openproject, suitecrm and wordpress->discourse, which had the
identical construct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six enabled providers carry a non-empty allowlist, declare
mutating_tools_enabled: false and have no adapter in front of them. That
flag is enforced only inside the adapter, so for those six the exact
match is the only runtime evidence that nothing beyond the declared
read-only surface is served. An upstream release adding a mutating tool
would otherwise deploy green against metadata promising none.
The shipped frontend bundle calls eval() from its runtime config chunk,
which script-src blocks, so the CSP health unit failed twenty-five times
in a row and aborted the deploy. The application itself answers 200 and
loads every asset.
This widens the policy rather than fixing the cause, which sits in the
upstream bundle; four other roles carry the same allowance for the same
reason. The failure was masked while the adapter build failed earlier in
the same job.
Treating 401 as transient let the surrounding task burn its full budget
of thirty retries, each one a failed cmkadmin login. Checkmk locks a user
after ten, and the password tool only writes the htpasswd hash, so
nothing in the role can clear the lock again: every later run repeats the
thirty attempts and reports them as "still starting".
The distinction it bought is no longer needed, because the manager-ops
pass now writes the declared password onto the running site before this
script authenticates.
The agent runs as uid 10000 and keeps its kanban state in embedded
SQLite. Without an nfs block the export directory is created root:root,
no_root_squash helps only root, and uid 10000 gets EACCES, which SQLite
reports as "attempt to write a readonly database". Concurrent writers
would report "database is locked" instead, which is how the two apart.
Compose has no NFS bind, which is why only swarm failed.
An earlier series gave five other non-root roles the same block; this
role was the one left out.
replicas: 1 stays as hardening, not as the fix: once the writes land,
three replicas contend for one SQLite writer lock.
composer install resolved against php 8.5.6 and failed with two problems, both of which name the same ceiling: magento/product-community-edition 2.4.8 requires php ~8.2.0 or ~8.3.0 or ~8.4.0, and magento/magento-allure-phpunit 3.0.2 requires ~8.1.0 through ~8.4.0. Both constraints admit 8.4, so the single pin resolves the whole failure rather than half of it.
This is a regression, not a new incompatibility: dc08bc621 pinned the image down from 8.5 to 8.4 on 5 August, and the image update bot moved it back to 8.5-fpm on 8 August in 1e4f82128. Without a suppression the next bot run repeats that. The added nocheck marker is what prevents it; utils.update.docker.suppressed_services reports the line as suppressed, so the guard is effective rather than decorative.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The earlier attempt used meta/tests.yml, which the deploy-modes lint
records as replaced: it reached CI discovery but left get_role_skip
empty, so the roundtrip script, the swarm context and the deployment
lookup all still treated the adapter as a standalone application.
modes.<mode>.enabled is the SPOT, and the role has nothing to deploy on
its own: its vars read mcp.adapter.digest against the including
provider, which does not exist when the adapter itself is the
application, so its standalone job failed while rendering the template.
The digest shipped with the digest-pinning change was a valid but
superseded python:3.13-slim index that no tag points at any more. CI
pulls through a mirror that only copies manifests reachable from the
tags it syncs, so that digest was absent there and every adapter build
failed to resolve its base image.
The replacement is the digest the tag carries today, which resolves both
upstream and through the mirror. Removing the digest instead would trip
the schema lint that requires one.
The loop condition is the usage of the whole filesystem, which the script cannot move once every remaining version is the last one of its application: is_directory_deletable always skips those. It then repeats an identical, empty pass. The journal of a real unit shows 132192 iterations in two minutes, and systemd kills the unit on its start timeout long before the script's own hour-long limit, so the failure surfaces as 'Active: failed (Result: timeout)'. The sys-service handler turns that into a failed play, nine minutes before the first application role runs.
b694f9335 already fixed this busy-loop for the shape where no backup versions exist at all, by breaking when average_version_directories reaches zero. With one version per application the average is one, so that guard never fires. Key the exit on the outcome instead: a pass that deleted nothing ends the loop. While anything is deletable the behaviour is unchanged, and the last version of an application stays untouched as before.
The regression test needs a running clock, because the shared helper freezes time.time() at zero and a regression would therefore hang rather than fail, plus a Path.is_dir mock that delegates outside the backup tree so runpy keeps working.
Touching the file pulled it into the comment lint, which flagged five pre-existing comments. Four were narration and are deleted; the fifth justified the noqa S602 marker and moves onto that marker's line, where the security lint expects it, so the file leaves that lint's baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two independent defects kept mastodon_mastodon from converging; either alone
leaves the stack red.
meta/volumes.yml declared force_ssl_override as type: bind pointing at a host
path, but tasks/01_overrides.yml writes that file with plain file/copy and no
delegate_to, so it exists only where the play ran. Every worker placement was
refused before container create:
Shutdown Rejected invalid mount config for type "bind": bind source path
does not exist: .../overrides/zzz_disable_force_ssl.rb
type: config is the swarm-distributable form and what svc-dns-unbound,
web-app-listmonk, web-app-n8n and web-app-decidim already use for small
mounted files, each with mode 0444.
Separately, compose.yml.j2 rendered the migrate service as the only one
without with_volumes=true -- mastodon and sidekiq both pass it. So migrate
never received 00_oidc_http_discovery.rb, ran OIDC discovery over HTTPS at
boot and died with "tlsv1 unrecognized name". Fixing only the mount would
have unblocked placement and still left the stack short of converged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
env.j2 emitted SMTP_TLS and SMTP_STARTTLS but never SMTP_AUTH, while always
emitting SMTP_AUTH_USER. The image turns that into "auth on":
/setup_msmtp.sh:9 smtp_auth="${SMTP_AUTH:=on}"
/setup_msmtp.sh:32 if [ -n "${SMTP_AUTH_USER+x}" ]; then echo "auth $smtp_auth"; fi
Against an onion relay that offers no authentication, msmtp then aborts with
"the server does not support authentication ... tls=off auth=on ...
exitcode=EX_UNAVAILABLE". The role was contradicting its own resolver:
plugins/lookup/email.py:175-179 returns auth = tls, i.e. False for an onion.
That single failure took the whole stack down, because the healthcheck chains
the mail smoke test ahead of the web probe with && and only touches its
marker on success -- so curl never ran, the container was health-killed every
cycle, and the swarm stack never converged. Friendica 0 and Fediwall 0 both
died this way; Friendica 1 and 2, which have email disabled, converge fine.
Emit SMTP_AUTH from email.auth and gate the credentials on it, matching
sys-svc-mail-msmtp/templates/msmtprc.conf.j2:6,15-18. In the auth-off case the
effective mechanism is dropping SMTP_AUTH_USER -- line 32 only writes an auth
line when that variable exists, so msmtp falls back to its own default of off.
The explicit SMTP_AUTH stays because it states the intent and covers a future
caller that sets the user independently.
Not fixed here, deliberately: the healthcheck's && chaining. That is the
amplifier, not the cause, and decoupling it would let friendica report healthy
while unable to send a single mail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
report_tasks filtered with awk '!/error=$/', keeping only rows whose ERROR
column is non-empty. That discards exactly the states a converge timeout
needs reported -- Pending, Assigned, Preparing, Starting, and any
unhealthy-kill whose image STOPSIGNAL yields exit 0 -- and keeps only the
states that already explain themselves.
Friendica hit precisely that hole: all six task rows sat in Starting or
Shutdown/Complete with a blank ERROR (its apache image uses SIGWINCH, so a
health-kill exits 0 and swarm records Complete, not Failed). Every row was
dropped, the `[ -n "${rows}" ] || return 0` guard returned silently, and the
operator saw a bare "not converged: friendica_friendica" with no diagnosis.
Decidim's rows printed only because Failed carries an error string.
Drop the filter. report_tasks runs per non-converged service, and
docker service ps lists each slot's current task before its history, so the
head -10 cap still shows the rows that matter. Strictly additive: more rows,
never fewer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The task name referenced the loop variable but is rendered outside the
loop, so every run logged the task as "Reconcile << error 1 -
'mcp_client' is undefined >>". The loop itself was fine; the damage was
diagnostic, which is the worst kind to leave in a log someone reads
while hunting an unrelated failure. The per-iteration label already
carries the client id.
The role stages a build context and creates a network; it never starts a
container, because the adapter runs as a sidecar inside each provider's
stack. Invokability follows the category path alone, so the svc-ai
prefix put a library role into the deployable matrix, where its job can
only fail. It is exercised in every provider stack that carries it, and
its contract probe is what reports a drifted upstream.
The group map is built by looping over the discovered MCP servers, so an
empty list left the variable undefined and the next task aborted. That
list became reachable-empty once discovery narrowed to the providers a
run actually deploys.
The consuming tasks move into their own file behind one condition rather
than taking a default, because an empty result is a different case and
not a missing value: defaulting it would reconcile the existing grants
against nothing and drop them.
CMK_PASSWORD only applies while the site is being created. The site
volume outlives an inventory re-initialization, so the site kept the
password it was born with while the deployment minted a new one, and
every API call answered 401. The password is now written onto the
running site each deploy, which is what the shipped cmk-passwd tool
exists for.
401 becomes a transient status for the identity script, because the site
serves its API before it writes the admin htpasswd: from outside, a
starting site and a wrong password answer identically, and only the
retry budget tells them apart. That distinction is what showed the
password was wrong rather than late.
4693a8195 put `# nocheck: cache-read` on a line that exceeded the length
limit, so ruff format wrapped the call and left the marker sitting on the
closing paren. The lint matches per call line, so the exemption stopped
applying and test_no_uncached_file_reads failed again -- the commit undid
its own fix the moment the formatter ran.
Bind the read to a local first: short enough that no wrap is possible, and
the marker stays on the call it exempts. Verified formatter-stable with
`ruff format --diff` reporting no change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both are fallout from earlier commits in this branch.
4b10d6b93 moved redirect_uris.py from the keycloak role's filter_plugins to
plugins/lookup, but the requirements doc still linked the old path, which
test_relative_links flagged.
test_primitives.py reads back a file that download_file just wrote into the
test's own TemporaryDirectory. test_no_uncached_file_reads asks for
utils.cache.files.read_text, but its own message names this case as the
opt-out -- "tempdirs, synthetic fixtures written inside the same test" -- and
read_text is wrong here anyway: the payload is bytes, and a session-shared
cache would defeat the point of asserting what was just written. Same marker
as tests/integration/meta/workflows/test_subset_roles.py:40.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
08_users/realm.yml seeded the realm from `lookup('users') | non_reserved_users`
while 08_rbac_groups_realm.yml built group memberships from the raw
`lookup('users')`. The membership builder has its own guards, but both are dead
against real data: across all 269 default users, none carries a `reserved` key
and none has an empty password. So membership was effectively unfiltered while
seeding filtered on `accounts`.
helpdesk and no-reply (accounts: [mailbox], roles: [bot]) were therefore
demanded as members of roles/web-app-mailu/bot but never created, and
ensure_group_path_members.sh exits 1 on the first miss. Four jobs -- Mattermost 1,
Friendica 1, Matrix 1, OpenProject 1 -- each burned five cross-node recovery
attempts on a data failure no reschedule could fix.
Rather than adding the filter at the site that forgot it, remove the ability to
forget: lookup('users_non_reserved') resolves users and applies the predicate in
one place, and all four consumers now read it -- the two Keycloak user tasks, the
RBAC group builder, and OPENLDAP_USERS, which had it right all along. The filter
stays as the pure predicate the lookup calls.
Reproduced offline against the real user model, byte-identical to the CI member
list: before, bot demanded helpdesk and no-reply; after, members are exactly the
four seeded accounts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shared persona helper cannot log the administrator out of Shopware. The only logout control lives in .sw-admin-menu__user-actions, which the administration stylesheet keeps at display:none until its toggle is clicked, and that toggle is a bare div carrying no role, no aria-haspopup and no dropdown class, so nothing in the helper's trigger set matches it. Its second fallback, following a same-origin settings link, misses as well because the administration is hash-routed and every settings target is #/sw/settings/...
Declare PERSONA_ADMINISTRATOR_BLOCKED and move the journey into a role-local spec that drives Shopware's own selectors: native admin login, an authenticated backend assertion, logout through the user-actions toggle, and back to the login form. It skips when SSO_SERVICE_ENABLED=true, because the oauth2-proxy then owns /admin and the native form is unreachable.
The spec leaves the first-run wizard by navigating away from its route rather than closing it: the wizard renders with :closable=false on its first step, so it has neither a close button nor an escape handler, while its footer intercepts the pointer events aimed at the admin menu behind it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The wrapper calls the image entrypoint purely for its side effect --
install and migrate -- and then starts its own server, so it passed
/bin/true as the command to mean "do the init, start nothing". But
upstream gates that init on the command name:
if [[ "${1-}" == apache2* ]] || [ "${1-}" = 'php-fpm' ]; then
start
applyConfigEnv
fi
exec "$@"
/bin/true matches neither arm. The whole block was skipped and
exec /bin/true returned 0, so a complete no-op looked like a successful
init. No install ran, data/config.php was never written, and the failure
surfaced two steps later as the misleading "seed_config.php execution
failed".
Pass 'apache2 -v' instead: it matches apache2*, so start runs, and the
trailing exec "$@" only prints the version banner and exits 0 -- hence
the added >/dev/null, which keeps that banner out of the wrapper's
stdout.
Fail loudly when data/config.php is still missing afterwards, so a
regression of this shape is reported at its source, and surface the
throwable from $writer->save() with class, message, file and line
instead of a bare fatal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 6db7c4ad6f)
Host and container share this checkout through a bind mount but run as
different uids -- 1000 and 0, measured. Whichever writes .ruff_cache or
.import_linter_cache first owns it, and the other is then locked out:
ruff aborts with "Failed to create temporary file: Permission denied" and
test_import_contracts fails the same way, reporting a broken contract that
is in fact perfectly intact.
Point both at build/<tool>-cache-$(id -u). build/ is already 0777 and
gitignored, and this is the mechanism the lint stamp
(build/install-lint-<hash>.stamp) already uses to keep the two environments
apart. Passing --no-cache would have silenced the symptom at the cost of the
cache on every single run.
The pre-existing .ruff_cache and .import_linter_cache are nobody-owned and
cannot be removed without root; they are simply unused from now on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PYI034. The earlier attempt did not land -- the string replace missed
because the formatter had already rewritten the line, leaving an unused
Self import that ruff then removed while the finding stayed open.
The import belongs under TYPE_CHECKING: the module has
`from __future__ import annotations`, so the annotation is a string and is
never evaluated at runtime.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A single anonymous GET for the pinned hadolint binary completed its TLS
handshake and its request, then got a clean FIN with no status line after
3.6s. download_file was a bare urlopen with no retry, and make install-lint
is a prerequisite of autoformat, lint and every lint-* target -- so one shed
socket took down the whole quality gate. The same asset downloaded fine five
times elsewhere in that run, 15s after the failure on another distro.
Retry four times with exponential backoff, catching (OSError, HTTPException).
Two names, not the obvious four: ssl.SSLError is an OSError but NOT a
URLError, so the narrow tuple would have missed it, while OSError subsumes
URLError, ConnectionError, TimeoutError and gaierror in one token.
HTTPException is needed separately because IncompleteRead is not an OSError.
HTTPError is caught FIRST and re-raised immediately for any status outside
{408,429,500,502,503,504}. That ordering is load-bearing -- HTTPError is
itself an OSError, so a broader catch first would swallow it -- and it is
what keeps this robustness rather than masking: a 404 is exactly what a
stale version pin looks like after roles/sys-lint is bumped, and it still
fails on the first attempt with the URL. The final attempt re-raises the
original with its traceback, and write_bytes stays in the else branch so a
full disk is never retried.
apt was the other uncached, unretried input in the same target (its default
Acquire::Retries is 0), so give both invocations -o Acquire::Retries=3,
matching what eight Dockerfiles in this repo already do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bind-dynamic sat outside the {% if TOR_DNSMASQ_OWNS_LISTENER %} guard that
gates the listen-address lines, and router.yml:72 forces that flag false in
swarm. So on a stack host the drop-in declared it does not own the listener
and then dictated its bind mode anyway -- naming no address at all.
That converts a wildcard-bound instance's sockets into enumeration-derived
ones. When an IPv6 netlink dump aborts, dnsmasq skips the AF_INET pass and
its GC closes every IPv4 listener, silently: the message is LOG_DEBUG and
bind-dynamic bind failures are non-fatal. The node was left with 10 IPv6
listeners and zero IPv4 while four IPv4 addresses existed, and router.yml
had already pointed /etc/resolv.conf at 127.0.0.1 -- so dockerd got ICMP
port-unreachable and could not resolve ghcr.io. 33 failures in 11 groups,
none before the drop-in landed. The two worker nodes, which never received
it, kept 0.0.0.0:53 bound throughout.
Ownership is the right axis, not deployment mode. Where the role owns the
listener, bind-dynamic is load-bearing: a listen-address naming docker0 can
disappear across a docker restart, and without OPT_CLEVERBIND that bind is
fatal. Where it does not own the instance, the directive is pure collateral
-- and next to a base config carrying bind-interfaces it aborts startup
outright.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
KEYCLOAK_REDIRECT_URIS derived the scheme from Keycloak's OWN TLS state and
stamped that single scalar onto every SSO consumer. With svc-net-tor
deployed Keycloak is onion-primary, so resolve_enabled returns False and
lookup('tls', ...) yields "http" -- which then went onto clearnet apps too.
Keycloak registered http://matrix.infinito.example/* while Synapse sends
https://.../_synapse/client/oidc/callback, and its redirect-URI match is
scheme-exact: 90 invalid_redirect_uri events, an error page instead of a
login form, and a 60s selector timeout that looked like a Playwright bug.
jitsi hit the same wall on https://meet.infinito.example/oauth2/callback.
Resolve per consumer instead, via resolve_enabled: an .onion is always
plaintext, a clearnet app follows its own server.tls.enabled, and only
then the global TLS_ENABLED. That override existed and was silently
discarded -- passing a pre-computed scheme string threw it away, which is
the same "what I am" instead of "what the consumer is" mistake one layer
up. resolve_enabled handles the onion case itself, so the filter's own
is_onion_domain branch is gone rather than duplicated.
Move it from a filter to a lookup while at it: the thing needs domains,
applications and TLS_ENABLED anyway, so resolving them inside collapses
the call site to lookup('redirect_uris') and drops
KEYCLOAK_REDIRECT_FEATURES, which listed the same path twice. A lookup
under plugins/lookup/ is also plainly importable, so the test loses the
sys.modules stubbing apparatus a role-local filter needed -- 60 lines net
smaller.
KEYCLOAK_WEB_ORIGINS is regex-derived from these URIs and inherits the
fix. Verified against the real applications view: 69 URIs, all https with
TLS_ENABLED true and all http with it false -- the same 69 the failing run
registered as plaintext.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Decidim ran RAILS_ENV=development against a database that was created but
never migrated. Only development.rb carries
config.active_record.migration_error = :page_load, which inserts a
pre-routing guard that raises ActiveRecord::PendingMigrationError on every
request -- so the healthcheck's `curl -f /up` got a 151 kB debug page and
swarm reaped the tasks.
The role could never recover, because its ordering is a deadlock:
02_manager_ops.yml:26 `meta: flush_handlers` runs the converge gate, and
only afterwards does 01_setup.yml migrate -- behind its own wait for a
healthy container. Health needs migrations; migrations need health.
Clearnet runs hid this for months. DECIDIM_FORCE_SSL was true there, so
ActionDispatch::SSL answered every probe with a 301 before any app code
ran, and `curl -f` treats 301 as success -- it reported healthy even for
/health_check, a path this image does not route. Giving Decidim an .onion
primary domain turns TLS off (utils/tls_common.py resolve_enabled), the
301 disappears, and the pre-existing defect becomes visible.
Let the container migrate itself, so "healthy" means "migrated and
serving" in compose and swarm alike. An `until` loop rather than `&&`:
Rails takes a pg advisory lock and raises ConcurrentMigrationError on
contention, which would push two of three fresh replicas into a restart
loop. Measured: migrate takes ~37s on an empty database, both concurrent
replicas reach healthy at 133s/123s, the loser recovers after retrying.
The probe therefore needs an initialization window (start_period 180s,
interval 10s; the old 30s/1m/3 kills the task at ~211s), and the converge
gate needs the same 600-retry knob ten other slow roles already set.
Keep the Ansible migrate task as an idempotent safety net, and keep
`path: up` -- /health_check does not exist in this image and only ever
passed because of the 301.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The admin returned HTTP 500 with 'TwigFeaturesWithInheritanceExtension::
include(): Return value must be of type string, Twig\Markup returned'.
Twig 3.28 dropped the return type from CoreExtension::include(); 6.7.8.2
still declares ': string'. The template ships no composer.lock, so the
build resolves twig freely and picks up 3.28.
Upstream fixed the signature to ': string|Markup' and shipped it from
6.7.12.1 on, so this is a version bump rather than a transitive pin the
repo cannot see. Untested beyond the signature check: five patch releases
of delta, and 6.7.13.0 adds a shopware.github.io composer repository the
package cache does not proxy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
display_errors=On makes PHP emit the warning before the fatal, so the
headers are already sent and the response is 200 instead of 500. The
healthcheck is a bare curl -f against index.php, so a dead PHP stack
reports healthy and the deploy goes green -- job 92930113419 logged
'GET / 200 674' next to 'Failed opening required autoload_runtime.php'
for fourteen minutes.
CI runs with ENVIRONMENT=development, which is exactly the branch that
turned display_errors on. Off everywhere; log_errors keeps the trace in
the container log, error_reporting keeps its previous non-development
value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nextcloud's SSRF guard answers every server-side call to an RFC1918
address with 401 'violates local access rules'. integration_mastodon
registers its OAuth app dynamically against the partner, so in the swarm
lab -- where every partner resolves inside 192.168.244.0/24 -- the
integration never connects and the addon spec never leaves Nextcloud.
The switch is a service config entry rather than a hardcoded value, so a
deployment whose partners really do sit on a private network can turn it
on without the role stomping the setting on every deploy. Default is on
in development only; production keeps the guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two violations that tests/lint cannot see, because ruff runs under the
autoformat target rather than the lint suite:
TC003 test_db_tor_transport_gate.py imported pathlib.Path at runtime
for an annotation only. The module has `from __future__ import
annotations` and never constructs a Path -- the values come out
of ROLES_DIR.glob() -- so the import moves into TYPE_CHECKING.
RUF012 test_render.py's ENTRY is a mutable class attribute; annotate it
ClassVar[dict].
Plus three reflows from ruff format. No runtime behaviour changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
