n8n's playwright specs arrived un-migrated via a branch update; wrap their raw numeric timeouts in resolveTimeout so they satisfy the no-raw-timeouts lint and scale over onion. Clearnet-identical (resolveTimeout(N) === N).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace page.goto with gotoOnion in the Keycloak account + admin-console login/logout tests so a transient Tor-circuit failure on a navigation is retried on a fresh circuit; combined with the resolveTimeout budgets the journeys now complete over .onion (verified ~33-41s each), so they no longer carry an onion skip. Coverage unchanged — the tests still run over onion, now green instead of flaky.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Use gotoOnion for the landing navigation so a transient Tor-circuit failure (ERR_SOCKS_CONNECTION_FAILED / ERR_TIMED_OUT) is retried on a fresh circuit instead of failing the test — a longer timeout alone cannot recover a failed circuit. Budget the test timeout via resolveTimeout. The never-authenticated assertion is unchanged; clearnet behaviour is identical (single goto attempt, resolveTimeout is identity).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Over Tor, Matomo link-tracking intercepts the SSO-link click to fire a beacon that stalls, so the click never commits the navigation. Under a .onion canonical, read the link href and goto it directly (same state/nonce, skips the interceptor); clearnet keeps the original click path unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ignore Keycloak's third-party-cookie probe (/3p-cookies/, requestStorageAccess) in the console-error assertion — that probe errors by design when 3p cookies are blocked (always over onion) and is Keycloak feature-detection, not a dashboard fault; real dashboard console errors still fail. Fix the iframe-sync race by re-dispatching inside the poll until the CDN-loaded listener is live (over Tor it loads after DOMContentLoaded).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The client-side iframe.js postMessage handler validated message origins and redirect targets against a single .DOMAIN_PRIMARY suffix, which rejected the onion mirror domain and broke iframe sync over Tor. Allow the svc-net-tor node suffix alongside clearnet when tor is deployed; both the origin and redirect checks stay restrictive.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
env.js gains gotoOnion (retries transient Tor-circuit errors on page.goto, clearnet unchanged) and isOnionCanonical (detect a .onion canonical). playwright.config.js marks .onion origins as a secure context via --unsafely-treat-insecure-origin-as-secure, emulating real Tor Browser (RFC 7686) so Clear-Site-Data logout and the Storage Access API behave as in production over onion.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the global TIMEOUT_FACTOR and a shared timeouts.js helper: resolveTimeout(base) scales by TIMEOUT_FACTOR and, over a .onion target, an extra multiplier, so Tor's per-request latency no longer trips hard-coded timeouts. Migrate every raw timeout in the specs (timeout:/timeout=/setTimeout/waitForTimeout) to resolveTimeout, stage the helper in all three staging paths, and add test_no_raw_timeouts to forbid raw numeric timeouts. Clearnet behaviour is unchanged (factor 1: resolveTimeout(N) === N).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PyMySQL, psycopg2-binary and python-ldap are imported by the controller-side svc-db-*/files/test/test.sh scripts that authenticate to the databases over the Tor SOCKS forward.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
web_health_expectations swaps (exclusive) or extends (dual) each services.tor.enabled app's clearnet domains with their onion domains via _inject_onion_domains, so the probe checks exactly what the deploy serves; script.py probes .onion over http (plaintext-only) with a longer Tor timeout. Adds onion unit tests for both.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
logout_domains takes an optional resolved domains map (lookup('domains')) and prefers it over the raw canonical config, so the logout sweep targets the domains the browser is actually served at — over Tor the clearnet host is unreachable via SOCKS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Emit cookie_domains only when a clearnet domain exists; onion-only deployments then fall back to host-only session/CSRF cookies, so the oauth2-proxy CSRF check works over .onion.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Normalize YAML formatting flagged by make autoformat: document-start
markers and long-description line wrapping. No behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a `stack` column to the complexity matrix: true when a role renders
its own container stack (ships templates/*compose*.yml.j2), false for
host-only roles (backup crons, wireguard, swapfile), the swarm
infrastructure roles themselves, and central-webserver content
providers. Exposed as a sort key and filter field.
Make `swarm` imply `stack`: a role with no compose template of its own
is never a swarm stack-deploy target, so swarm is forced false when
stack is false. Because scripts/meta/resolve/apps.sh selects the swarm
deploy jobs via `--filter "swarm == true"`, this drops 17 stackless
roles (106 -> 89) from the swarm job list; all keep compose testing and
their swarm overlay coverage still runs through the webserver/proxy
role, so no real deploy coverage is lost.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The curl-timeout lint scanned tasks, templates and .sh scripts but not
Dockerfiles, so four build-time downloads (composer for mediawiki and
joomla, wp-cli for wordpress, the Docker GPG key for svc-runner) could
still hang a build against a silent mirror. Pin --connect-timeout and
--max-time inline (Dockerfiles cannot render the filter) and extend the
lint scope to roles/*/files/**/Dockerfile so future ones are caught.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A curl against an accepted-but-silent peer hangs forever: task retries
never fire because the first attempt never returns, and the CI job dies
at the 6h runner cut (proven by the mailu admin-API hang that took the
suitecrm/peertube swarm legs down). The new `curl` filter renders the
hardened base command ("{{ 30 | curl }}" -> curl -s --connect-timeout 5
--max-time 30); scaling composes with lookup('timeout').
test_curl_via_filter enforces it: bare curl invocations in role tasks,
templates and shipped scripts fail the lint with a pointer to the
filter. Plain .sh files, which cannot render Jinja, pass by pinning
--max-time inline; Docker healthchecks are exempt (their timeout field
caps them); `# nocheck: curl-timeout` opts out (catalogued in
suppression.md).
Convert all 34 existing invocations: mailu (replacing the short-lived
MAILU_CURL var), elasticsearch, checkmk, yourls (300s for the
schema-creating installer POST), the telegram alert script, and the
plain scripts of jellyfin, svc-runner, apt_repo, moodle (600s tarball)
and the mattermost provisioner. Mailu's user/token tasks additionally
carry task-level timeouts so a wedged container exec cannot stall the
play.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add registry.gitlab.com to the mirrorable registries so discovery
yields the GitLab CNG images to the version updater and mirror sync.
Seed the generic v2 tag scan with the OCI last cursor for v-prefixed
pins and raise the page size to 1000: CNG repos carry tens of
thousands of commit-sha tags that sort before the release tags, so an
unseeded scan exhausted the page cap without reaching any semver tag.
Pin every gitlab service version in meta/services.yml and drop the
GITLAB_VERSION template override so services.yml is the single source
for per-service versions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The container_address lookup emits $(...) shell syntax in swarm and
documents that callers must use the shell module; the 22 occ tasks
still used command:, so the resolution reached docker literally and
every plugin/LDAP/integration task died on all swarm variants. Convert
them to shell: (args stay single-quoted or [A-Za-z0-9] slugs, so
compose semantics are unchanged).
NFS mounts use local_lock=flock, which makes the stock entrypoint's
init flock node-local: concurrent replicas each won "exclusive" locks
and shredded the rsync install (occ 255, converge livelock). A small
init gate now fronts the entrypoint: only swarm task slot 1 runs the
init, other slots wait for version parity plus config.php; without
TASK_SLOT (compose) it degenerates to a plain exec. The gate ships as
files/nextcloud-init-gate.sh, copied into the build context on the
stack host and referenced through the NEXTCLOUD_INIT_GATE_FILE SPOT.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ref pin and the build-strip condition carry their intent in code;
the guard task already fails loudly when a service still builds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keycloak's realm passwordPolicy requires a digit and a special char;
the LDAP-disabled seeding path died on set-password with
invalidPasswordMinSpecialCharsMessage. All consumers read the value
through the users lookup, so tests and seeding stay in sync.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keycloak's realm import enforces the ^[a-z0-9]+$ username pattern, so
the LDAP-disabled seeding path (Ensure non-reserved users exist) died
on every hyphenated bot: "Username is reserved or contains invalid
characters". These are API/basic-auth accounts that never need an SSO
identity; accounts: [] takes them out of the seeding loop (semaphore's
breakglass-administrator convention), and no-reply keeps only its
mailbox account.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Compose created the manager and taiga containers in the same parallel
wave; both mount the fresh static/media volumes and the concurrent
copy-up died with "mkdir .../emails: file exists". Depending on the
backend (service_started, as async already does) lets taiga populate
the volumes alone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Variant 1 (no auth stack) died because jellyfin_auth_env eagerly
resolved the LDAP bind and Keycloak client lookups even when neither
service is deployed; the env is now composed from a base dict plus
LDAP/SSO parts combined only when the respective service is enabled.
Jellyfin keeps all state in embedded SQLite under the shared /config
volume; concurrent replicas corrupt it via node-local NFS flocks
(SQLite disk I/O error in CI), and upstream has no multi-instance
support. One floating replica keeps reschedules safe: the NFS volume
follows the task to any node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Upstream's web service concatenates ThemeServer with
'/themes/opencloud/theme.json' without trimming, so a trailing slash
yields '//themes/...'; the swarm front proxy forwards the raw request
URI and the SPA fallback answers with index.html instead of the theme
JSON, breaking the UI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The freeswitch build fetched libopusenc from launchpad.net, whose
librarian backend rate-limits CI (429s across 11 attempts); three more
build-time fetches carried the same risk. The maintainer publishes an
alangecker/* image for every buildable service, so compose_mods drops
build: wherever the service carries an image and compose pulls instead;
a guard task fails loudly if a future upstream service still builds.
Pin ref to the v3.0.23 release branch (the packaging repo has no tags)
and clone it without submodules via the new compose_repository_recursive
toggle: submodules only feed build contexts and DEV_MODE mounts, and
generate-compose falls back to the committed repos/tags file.
Declare the pulled images in meta/services.yml (updater-managed SPOT,
mirror pre-pull) and the named data volumes in meta/volumes.yml; the
override renders both through the standard container_image /
container_volumes / compose_volumes lookups with the service_name
pattern. Compose merges volumes by target, so the override's named
volumes verifiably replace the clone-local bind mounts (docker compose
config proven). This also heals a split-brain mount: apps-akka,
periodic and prometheus-exporter previously kept bind mounts while
their peers used the named volumes.
Teach container_image and compose_volumes to honour compose_mode_force
(the SPOT the other mode-aware lookups already read): bbb deploys as a
node-local compose stack in swarm, so it must get hub image refs and
plain local volumes there, not registry-prefixed refs and NFS
driver_opts. The unpinned-engine-volume lint learns the same two safe
conditions. Read the embedded Collabora port from web-svc-collabora's
services SPOT instead of hardcoding 9980.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CNG's set-config copies the rendered configs into CONFIG_DIRECTORY on
every container start; the first cp creates the destination with the
source mode, so 0440/0400/0444 sources left the copy owner-unwritable
and the next start's truncating cp died with EACCES (permanent
crash-loop after any restart, including the deploy's own recreate).
Grant owner-write to exactly the files set-config copies; files bind-
mounted read-only at their final destination stay tight.
Mount the CNG images' declared VOLUME paths (/var/log*, /var/opt/gitlab)
as named node-local volumes so containers stop spawning anonymous
volumes that fail the hlth-volumes check.
Verified live: HTTP 200, all services healthy across recreate,
hlth-volumes Result=success.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The role rendered its vhost directly via stack_host_template, bypassing
sys-svc-webserver-https and sys-front-tls entirely, so no HTTPS server
block or certificate ever existed for peertube domains. Swap the loop
body to include_role: sys-stk-front-proxy per domain (mastodon pattern)
and rename the template to proxy.conf.j2 so the front proxy's fileglob
discovery picks it up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The image installer itself calls session_start (install/cli.php); with the conf.d save_path pointing at data/.sessions, which the entrypoint only created after init, the installer lost its session state on every invocation and aborted with 'No database name specified in config'. Create and chown the session dir inside the lock before running the image entrypoint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploy GitLab from the official CNG CE images as six services: webservice (puma) and sidekiq scale to the topology default, workhorse is the sole HTTP entry, gitaly and a one-shot migrations job are pinned singletons, shell runs gitlab-sshd. The Omnibus container, its Dockerfile and the GITLAB_OMNIBUS_* config groups are removed. All rails/workhorse/shell/gitaly configs and shared secrets are pre-rendered by Ansible and mounted read-only; eleven credentials land in meta/schema.yml including an rsa_pem_2048 OIDC signing key (new value_generator algorithm). Postgres, redis and S3 object storage ride the existing lookups; OIDC, SMTP, the public SSH port and the root password are preserved. compose_volumes gains a per-volume nfs:false opt-out so the gitaly repositories volume stays node-local in swarm (upstream forbids git repositories on NFS); nfs_prep skips volumes without driver_opts accordingly. The Nextcloud gitlab integration execs the CNG rails runner. ruby_single_line dies with its only consumer; ruby_dq keeps test coverage. Ports (8181/8080/2222/8075/3808/9122) and service names resolve from single SPOT declarations in meta/services.yml via vars constants; the version pin sits on the webservice image pair so the updater can bump it. Live-validated in compose up to all six services healthy and migrations completed; fresh-install-only semantics for existing Omnibus data are documented.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The healthcheck fetched GET / which is auth-gated: Laravel answers with an absolute redirect to the public domain (URL::forceRootUrl), GNU wget follows it and fails DNS inside the swarm lab, so the task never turned healthy and converge expired. Probe the unauthenticated /health endpoint instead. Also replace four default-placement justifications with the audit-verified reasons (bluesky sqlite-WAL over NFS, mailu mail-edge IP binding plus NFS-hostile postfix/dovecot state, opencloud JetStream/bleve/boltdb engine state, seaweedfs leveldb/raft/needle volumes) and drop dead general vars whose only consumers were commented-out lines.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dockerfiles have no suffix, so the comments lint never scanned them. Match the Dockerfile basename in the changed-targets collector and dispatch it to the hash-comment parser; changed-files-only semantics stay, so the legacy tree is unaffected until a Dockerfile is touched. Drop the remaining prose comment from the espocrm Dockerfile it immediately flagged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The earlier attempt routed sessions through framework.session.handler_id with a redis DSN, which lazy-loads symfony/cache's RedisProxy whose dump() declaration is incompatible with the image's php-redis - a PHP fatal in every php container under swarm. Route sessions below Symfony instead: bake a conf.d ini registering phpredis's native session save handler with session.save_path taken from the REDIS_SESSION_TARGET env at runtime (php.ini env substitution, no secret in the image), emit that env with phpredis auth[user]/auth[pass] url-encoded query syntax from the engine lookup, and set framework.session.handler_id to null so Symfony defers to php.ini. Sessions are shared across replicas; cache and messenger paths stay untouched. All Dockerfile prose comments dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Upstream Fider renders the header sign-in trigger as a button; the specs waited on getByRole('link') and timed out although the app was healthy and fully rendered (proven by the run artifact page snapshot). Use an .or() locator covering both roles in both spec files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The root-run auto-install leaves administrator/cache, administrator/logs, cache and tmp root-owned on the shared NFS docroot; apache (www-data) then wedges every request in a cache-write permission loop, each healthcheck probe hangs its full 10s timeout, and the swarm converge wait expires - the earlier start_period raise addressed a misdiagnosis and could not help. The role-side chown runs only after converge, a circular dependency. Chown the four dirs in the entrypoint leader branch before touching the ready marker; the role-side chown becomes an idempotent no-op in compose.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Worker play-hosts fataled with 'CHECKMK_HOST_NODE is undefined': the resolve_host_cid include was IS_STACK_HOST-gated, but the consuming wait/configure tasks sat at the top level of 01_core.yml and their delegate_to was evaluated on every host despite the task-level when. Move the three manager-side tasks verbatim into 02_manager_ops.yml included behind when: IS_STACK_HOST (the gitea/espocrm pattern), so workers never instantiate them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
odoo -u all writes every ir_cron row while loading base data; live cron workers (N replicas x max_cron_threads) hold FOR NO KEY UPDATE row locks for the whole job, so the NOWAIT probe aborted with 'Record cannot be modified right now'. Snapshot active crons into infinito_ir_cron_pause and deactivate them (the UPDATE drains in-flight jobs), run all module operations inside a block whose always leg resumes from the snapshot (crash-safe: a killed run leaves the table and the next pause reuses it), and drop the redundant -u core update that ran outside the window.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The manager pin existed for the embedded sqlite database and the embedded qdrant sidecar. Switch DATABASE_* to the database lookup (central svc-db-postgres or the auto sidecar), consume qdrant central-only via svc-db-qdrant group membership, and pin the five secrets Flowise otherwise generates per replica (secretkey overwrite, JWT auth/refresh, express session, token hash) so N replicas share crypto state. The app scales to the topology default; variants pin the new dynamic flags. Existing sqlite data is not migrated (fresh-install semantics); pre-existing inventories need the schema apply to materialize the five new credentials.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The manager pin existed for the embedded kix-db container (private registry image with node-local pgdata). Remove that container and wire KIXDB_* through the database lookup: central svc-db-postgres (pg_trgm pre-activated via services.postgres.extensions) or the auto-rendered sidecar when no central provider is in the inventory. The proxy service floats and scales to the topology default; backend stays a pinned singleton (it bundles the kix.Daemon scheduler) and frontend too (websocket notifications without cross-instance pub/sub). Variants pin postgres.shared once true/once false. Existing embedded-postgres data is not migrated (fresh-install semantics for the DB move).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The manager pin existed only because the coolconfig setup execed the task via the manager-only resolve-container-id helper. Replace default_placement: manager with replicas: 1 (coolwsd holds open-document sessions in kit-process memory and the ingress has no WOPISrc affinity, so one replica stays correct) and run the coolconfig tasks through resolve_host_cid: LOCAL_CID exec delegated to the hosting node, so the task may land on any node. The dead COLLABORA_DOCKER_CONF_EXEC var is removed; compose behavior is unchanged (HOST_NODE=self, bare container name).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the replicas:1 pin and make the web tier genuinely scalable. The custom entrypoint now runs the image entrypoint's install/upgrade/config-seed exactly once cluster-wide: an atomic mkdir mutex on the shared volume root (flock is node-local on this NFS mount), the image init invoked with /bin/true so its final exec returns control, a ready marker for followers, stale-lock takeover past the orchestrator kill ceiling and TERM/INT release; every replica then starts apache itself. Previously the image entrypoint execed apache directly, so the seed/clear-cache block was dead code - it now runs as intended. PHP sessions move to the NFS-shared data/.sessions via a conf.d ini so OIDC/SAML handshakes survive replica hops. The cron daemon and the websocket service become explicitly pinned singletons (they only inherited the entity pin before). A missing image entrypoint after a version bump now fails loudly (upstream master restructured the entrypoint; 9.3.10 is verified). Converge retries 600, healthcheck start_period 15m, seeder debug env renamed out of the ESPOCRM_SEED_ config prefix scan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
