The workspace swarm roundtrip for svc-db-postgres failed on every host
with "The lookup plugin 'tls' failed: NETWORK_MODE 'multi' needs
svc-net-tor with a node onion on this node" (CI run 40979252521, job
Workspace debian:13). roundtrip.sh (b2469ff948) pinned the swarm step to
multi whenever the validator accepts it and never looked at disable=,
while 03_roundtrip.sh passes a disable list that removes tor. The
hard-fail is the documented contract (network_modes.md: tor or multi on
a node without the provider fails the deploy); the request was wrong.
roundtrip.sh now pins clearnet when disable= holds the Tor key (read
from utils.github.variant.network.TOR_SERVICE), else multi when the role
takes it, else the rotation. 02_zombie.sh, which also disables tor,
passes network=clearnet instead of relying on the rotation drawing it.
TestRoundtripSwarmNetwork runs the real script against stubs; the old
pin logic fails three of its cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI run 40979252521 Taiga#0 picked the onion sibling and failed five specs
(test-login-oidc, test-themed-routes, network-leak, design:242/263).
- Login: TAIGA_COOKIE_SECURE renders True once clearnet has TLS, so
taiga-back marks sessionid and csrftoken Secure. Chromium drops them on
the plaintext onion, the /oidc/callback/ arrives without a session and
mozilla-django-oidc falls into login_failure() (gateway log: the 302 to
/login?type=oidc&error=None six times, no sessionid on the callback in
the trace). The plaintext sibling vhost now strips Secure from both
cookies via proxy_cookie_flags, wired as webserver_location_extra_
configuration like Semaphore; TAIGA_COOKIE_SECURE is pinned to the
canonical domain so the per-vhost tls lookup cannot flip it.
- Logo: the design sheet embedded absolute clearnet CDN URLs, rendered
once and served to both siblings. It now uses design.relative_urls,
served same-origin under /_infinito/cdn/; the spec resolves the logo
URLs against page.url().
A local compose deploy that picked the onion passed all five specs (19
passed); the onion vhost carries both proxy_cookie_flags lines, the
clearnet vhost none. test_onion_sibling.py renders the real vars through
Ansible's Templar; reverting TAIGA_COOKIE_SECURE fails two of its cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test from feature/updater renders the real daemon.json.j2, which on
store-i18n also reads IS_SWARM_MODE, the NVIDIA runtime shim and the
daemon MTU. Its context lacked them, so both cases died with
"'IS_SWARM_MODE' is undefined". The context now carries them and derives
the mode booleans from DEPLOYMENT_MODE like test_sandbox_runtime does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
make autoformat output for three lines the feature/updater merge brought
in over 88 characters; matches feature/updater-lockdown-probe 88d138ba1c.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
make autoformat rewraps the firewall-backend entry in test_userland_proxy.py, the _direct_media call in test_talk_direct_media.py and the _git listing context in test_files.py, which ran past 88 characters. The text is ruff's own output, the same store-i18n commits, so the next merge between the lines resolves cleanly.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Brings in feature/updater accb22da75, which carries main 5995766f58.
Conflict resolutions:
- web-app-fider: main's tenant bootstrap (01_tenant.yml) runs first on
the stack host, OIDC setup moves to 02_setup_oidc.yml, design to
03_design.yml; flat includes so main's gate test reads them.
- sys-front-tls test_nginx_ssl_header: the tls fake maps the domain term
store-i18n passes back to its app.
- group_vars 04_webserver.yml, sys-svc-proxy CSP header: store-i18n's
webserver_resource lookup plus #1256's WEBSERVER_SECURITY_HEADERS and
the security_headers.lua include after the CSP block.
- web-app-nextcloud: updater's decoupled Talk stack (Janus, signaling,
NATS, recording builds) replaces aio-talk; store-i18n's per-network
Talk backends move from the wrapper script into signaling.conf.j2 urls
(NEXTCLOUD_HPB_SIGNALING_BACKEND_URLS from network_urls, canonical
first); IS_COMPOSE_MODE/IS_SWARM_MODE booleans kept; the integration
OpenProject digest stays delegated to STACK_HOST.
- users lookup docs, token-store write_many, cache README, meta info
lint: both sides combined.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a clearnet deploy with self-signed TLS, test-talk-recording.js failed three times: the call recording stayed at state 4 (starting audio) because the recording server could not notify Nextcloud. Its log showed CERTIFICATE_VERIFY_FAILED on POST /ocs/v2.php/apps/spreed/api/v1/recording/backend, although compose-ca sets REQUESTS_CA_BUNDLE to the mounted bundle. nextcloud-talk-recording's BackendNotifier sends through `session.send(prepared, verify=True)`, and Session.send() skips the environment merge that only Session.request() does, so requests falls back to certifi's bundle, which does not know the injected root CA. A probe in the recording container's network namespace verified Nextcloud with the mounted bundle (200) and failed without it.
When the CA is injected, the recording image now points certifi's cacert.pem at the mounted CA bundle (the system CAs plus the injected root), in the same build step as the Firefox certificate policy. Verification stays on. Without an injected CA, as with real certificates, the build arg is empty and the image is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Conflict in roles/web-app-gitea/tasks/01_manager_ops.yml: main (c59e2838d6, c2577560f8) moved the readiness probes inside the Gitea container on its hosting node, because a host port lands on the swarm ingress mesh; this line (381c3cd5bc) had reduced them to one wait for /api/v1/version = 200 and dropped the explicit `gitea migrate`, which raced Gitea's own migration on galaxyserver. The resolution keeps main's in-container probe as that single wait with `curl -fS` (200 required), after the resolve and with this line's 600 s budget scaled by lookup('timeout'), and drops main's app.ini wait, its migrate task and its post-admin readiness wait.
main's new tests/unit/python/roles/sys-front-tls/templates/test_nginx_ssl_header.py renders nginx_ssl_header.j2 without `domain`, which ssl_credentials.j2 needs on this line since 47160d755e; its render() now passes the vhost's domain.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
00_core.yml included 01_setup_oidc.yml only when services.sso.enabled
was true, but that file carried the whole tenant bootstrap: wait for
HTTP, count tenants, POST /_api/tenants, create the admin user and
activate the tenant. Only its last task configured the Keycloak
provider. With SSO off no tenant was ever created.
Fider's RequireTenant middleware then answers every tenant route,
/api/v1/tags included, with 307 to /signup, and /signup is open to any
visitor, who can claim the fresh installation and become its
administrator. The MCP read fider_list_tags got that 307 in the swarm
SSO-off matrix row.
The tenant bootstrap moves unchanged into tasks/01_tenant.yml, which
00_core.yml includes on the stack host regardless of SSO. The OIDC file
becomes tasks/02_setup_oidc.yml and keeps only the provider upsert,
still gated on services.sso.enabled. The README states which bootstrap
steps run without SSO.
A unit test evaluates the include and task guards for every SSO,
stack-host and deploy-mode state: a fresh stack host posts exactly one
tenant with and without SSO, an existing tenant is never posted again,
the Keycloak provider is written only with SSO, and a non-stack host
runs none of it. The compose matrix row with SSO off stayed green
because nothing there read a tenant route.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nextcloud/aio-talk bundled signaling, Janus, NATS and an onboard TURN server behind a start script that rewrote their configs, and AIO publishes only build stamps, so the updaters could not raise it and the role had to guard Janus against config drift. The backend now runs as talk (Janus built from a meetecho/janus-gateway tag, files/Dockerfiles/Janus), talk_signaling (upstream strukturag/nextcloud-spreed-signaling image), talk_nats (official nats image) and talk_recording (built from a nextcloud-talk-recording release tag, files/Dockerfiles/Recording), each semver-pinned, single-replica, healthchecked and resource-limited. The Dockerfiles take every value as a build arg from compose, and Janus and the recording server get their config path through `command:`.
The role renders janus.jcfg, the signaling server.conf and the recording server.conf; they match what AIO stamp 20260409_094910 generated apart from the service addresses and the session keys, which are now derived from the signaling secret. Secrets reach the services only through those files. Janus may join the host network in compose mode, which keeps direct media available there, and an assert refuses host mode in swarm. talk and talk_recording keep their keys, so one deploy replaces the AIO containers in place.
The onboard TURN server is gone with its ports, vars and credential, so Talk uses web-svc-coturn only, and the deploy now stops when Talk is enabled without it instead of handing out no STUN or TURN server. The recording image installs the injected root CA as a Firefox policy when the CA is mounted, so recordings work against self-signed TLS, and the recording scratch volume stays off NFS. The Janus drift guard and the janus_config_drift filter are removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
services.talk.direct_media was an expression in meta/services.yml that read ansible_facts['all_ipv4_addresses'] and nested config lookups. lookup('config') renders it where the applications are merged without host facts; the render fails there, the lookup hands back the raw expression and `| bool` coerces it to False with a deprecation warning. Direct media therefore never switched on: both the first requirement-061 deploy and CI run 40877408319 logged the coercion, and a render in the development container returned the raw string without facts and true with them.
meta/services.yml keeps only the inventory switch, now `direct_media: false`, so every Talk stream goes through web-svc-coturn unless an operator opts a host in. That is the behaviour the bug produced everywhere, so no host changes on deploy, and it keeps Janus off the public network by default. vars/main.yml decides in task context from the switch, Talk, standalone TURN, the bridged network, compose mode and NEXTCLOUD_HPB_HOST_ADDRESSES, which reads the facts the way web-svc-coturn does. The flag tests render that chain against the merged applications: off by default, on for an opted-in compose host with coturn and an owned address, off for host networking, a foreign or loopback address, swarm, onboard TURN and missing facts. Administration.md documents the opt-in.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With the userland proxy on, Docker starts one docker-proxy process for every published port. A published range pays that per port: Janus's direct-media range showed 74 proxies for 64 UDP ports plus the other published ports in the development stack, at about 1.7 MB unique memory (PSS) each, which projects to roughly 1.7 GB for the default 1000-port range and the same again for OpenTalk's.
daemon.json now sets "userland-proxy": false, so iptables/nftables forwards every published port, which the CIS Docker Benchmark (2.16) also asks for. Traffic from outside already took that path, so client addresses are unchanged. Host-local traffic to ports published on 127.0.0.1 now runs through route_localnet and the loopback DNAT Docker installs. From Docker 28.0 on, Docker keeps hosts on the same L2 segment off those loopback ports; every install path of this role ships 29.x.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
repository_files() listed every path `git ls-files` returns, including tracked files the worktree has already deleted. Every content scan that reads the listing then reported "[Errno 2] No such file or directory" as a finding: in a worktree that removed roles/web-app-nextcloud/filter_plugins/nextcloud_janus.py and its test but had not committed the removal yet, test_no_rsync_backup_flag failed on exactly those two paths and turned the whole lint target red. No scan can read such a file until the deletion is committed.
The listing now subtracts `git ls-files --deleted`. A clean checkout, as in CI, has no deleted entries, so the listing there is unchanged. The rsync lint is green again on the same worktree. The unit tests stub the extra git call and cover a file deleted in the worktree.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Validation step "Assert the base branch is fully merged into the PR
head" ran main_merged.sh, which fetches the live tip of main when the job
starts. PR runs wait in the queue for hours while image-update PRs land
on main several times an hour, so nearly every queued run failed
Validation on start and had to be re-pushed after another main merge
(PR #1265: behind_by=0 at push, behind_by=9 while its run still queued).
main_merged.sh takes a new INFINITO_MAIN_COMMIT override: when set, it
fetches that full commit SHA from the remote and requires HEAD to
contain it instead of the live tip; a set but malformed or empty value
exits 2. The Validation step passes github.event.pull_request.base.sha,
the main tip at event time, which the step's pull_request-only guard
always provides. The run tests the merge against that base, so the gate
still guarantees the tested tree contains it.
The local pre-push hook (make test-main-merged) leaves the override
unset and keeps checking the live tip: a push happens right now, so
lagging the current main is the real defect there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
services.dashboard.enabled already says whether a role takes part in the dashboard: roles_with_service lists only roles with enabled and shared as tile-reachability targets, the CSP filter adds the dashboard to frame-ancestors only when it is set, and docker_cards uses it as the card's iframe flag. A separate display key duplicated that switch and left a role with enabled false still rendering a card. docker_cards now skips the card when services.dashboard.enabled is false (a role without a dashboard binding keeps its card, and a value that is not a boolean fails loudly), so display, its validator exception and its roles_with_service filter go again; base.md documents enabled as the per-host switch.
An inventory hides a tile with applications.<role>.services.dashboard.enabled: false, which also drops the dashboard framing of that role. Unit tests cover false, true, a missing binding, a rendered string and a non-boolean; forcing the helper to always show the card fails four of them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The WebUI OAuth client Stalwart serves itself registered `<origin>/<prefix>/oauth/callback` only for the canonical WebUI URL. A browser that opens the WebUI on the onion sibling sends `redirect_uri=http://mail.<onion>/account/oauth/callback`, which Stalwart rejects with "The redirect_uri parameter is missing or does not match the client registration". CI run 40877408319 failed test-login-native.js:36 that way in Meta#0, Nextcloud#0, Shopware#0, Bluesky#0 and Moodle#0, every row whose Playwright pass picked the onion sibling.
The client now takes its origins from `lookup('network_urls', STALWART_WEBUI_BASE_URL)` and registers the callback and post-logout URIs for every origin and URL prefix. A host without a sibling still yields the single canonical origin, so clearnet-only deployments register the same client as before. The Align task updates an already registered client once.
The README states the per-network origins and that Stalwart asks for consent after the password form, as the local WebUI v2.0.2 run showed.
Verified with the unit test (20 passed; registering only the first origin fails the new two-sibling test) and a local update deploy whose Playwright pass picked the onion sibling: test-login-native.js:35 passed, recap failed=0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard read `display` from the role's meta/info.yml, which no inventory can override, so a host could not hide a tile. docker_cards now skips a card when `services.dashboard.display` in the merged applications is false, and absent still means shown; a non-bool value fails loudly. roles_with_service drops the same role from the dashboard tile-reachability targets so the Playwright spec does not wait for a hidden tile, and the inventory validator accepts `services.dashboard.display` for every role that declares `services.dashboard`, since no role default carries it. `display` leaves the meta/info.yml allowlist, web-app-jellyfin drops its redundant `display: true`, and base.md and layout.md document the new switch.
Real lookups with web-app-gitea set to display false render the Dashboard and Jellyfin cards and only jellyfin as a tile target, and all three without the override; the nine tests for the new behaviour fail against the old code.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The pinned onlyoffice archive was unpacked with tar over custom_apps/onlyoffice on every deploy, so files that a newer release no longer ships stayed behind. On cloud.veen.world that left the 52 files of onlyoffice 9.15.1 (NC32 line) inside the 10.2.1 tree, and the integrity check reported them as EXTRA_FILE; the 52 equal exactly the 9.15.1 files absent from 10.2.1. replace_pinned_app.sh now unpacks the release beside the app, keeps the app when diff -rq finds it identical, and otherwise swaps the whole directory, so the next deploy heals the drifted tree and later deploys report ok. A broken archive fails before the live app is touched.
The real install task, run through ansible-playbook against a reproduced 9.15.1 plus 10.2.1 tree, changed it once to a tree identical to a clean 10.2.1 and stayed unchanged on the rerun; eleven unit cases cover identical, drifted, missing and broken inputs under sh and dash.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The admin overview reported that no server identifier is configured. Nextcloud 34 accepts 1 to 511, treats values below 1 as unset and asks for one NC_serverid per server when config.php is shared, which it is between nextcloud, cron, the task processing worker and the swarm replicas. A single serverid in config.php would let them mint identical Snowflake IDs in the same millisecond. The nextcloud service now carries NC_serverid 1 in compose and its task slot in swarm, so the check passes and replicas never share an id; the other services keep the hostname fallback.
A unit test renders the service environment in both modes and fails when the compose id leaves the accepted range.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nextcloud 34 flags "Task Processing worker status: never run" because only occ taskprocessing:worker records core/ai.taskprocessing_worker_last_iteration, and the role ran background jobs through the cron container alone; since 32.0.7 the worker command is taskprocessing:worker, so background-job:worker would never clear the check. Add a taskprocessing service that reuses the cron image, mounts, healthcheck, restart policy and networks and runs occ taskprocessing:worker -t 60 in an in-container restart loop that backs off while Nextcloud is not yet installed, so swarm neither replaces the task every minute nor marks the stack fatal on a fresh install. It scales with the swarm topology, since task claims use FOR UPDATE SKIP LOCKED.
Rendered compose and swarm stacks only gain the taskprocessing service and validate against the compose schema; the new unit tests fail when the worker mount is dropped or background-job:worker is used instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
trusted_proxies held the host's public addresses, but OpenResty reaches Nextcloud through the published port, which Docker forwards from the gateway of one of the container's networks (192.168.201.1, the central MariaDB network, on production). Nextcloud therefore ignored X-Forwarded-For, attributed every request to that gateway and brute-force throttled it. The TRUSTED_PROXIES env var was joined with commas while the image's reverse-proxy.config.php splits it on spaces, which produced the single malformed entry the admin overview rejects, and the occ value reached config:system:set as a Python repr instead of JSON, so config.php got null.
Add the proxy_hops lookup next to resolve_upstream in utils/networks/proxy.py: under compose it returns the gateway of every network the application joins, under swarm the subnets of the overlays it shares with svc-prx-openresty, both derived from the service registry and role_subnets. NEXTCLOUD_TRUSTED_PROXIES now uses it, env.j2 joins the list with spaces and system.yml passes it through to_json.
Rendered with the real roles and group_vars for the galaxyserver layout: compose trusts 192.168.41.1, 192.168.201.1 and 192.168.202.1, swarm 192.168.107.0/24; unit and integration tests cover both modes, and two mutations of proxy_hops fail them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A token store write re-rendered the whole applications SPOT: merged_applications_cache_key carried the token store digest, so the applications_carrier park in sys-token-store/tasks/write_many.yml missed its key and rendered all 269 applications again (stalwart compose roundtrip 18.3 s and 10.2 s per pass, swarm pass 1 12.2 s and 12.6 s).
Rendered from the real roles tree under an empty, a realistic and an exhaustive token store (298 users x 269 applications), the applications tree differs only under <app>.users.<key>.tokens (28783 leaves in 46 applications, 0 elsewhere); the 69 password leaves that also differ change between two renders of the same store. No code reads a token through the applications tree: every reader goes through lookup('users'), lookup('config', <app>, 'users.<key>...') (served from get_merged_users), lookup('email') or the sys_token_store_users fact, and the users carrier keys on the store.
The token digest therefore moves from merged_applications_cache_key into merged_users_cache_key, whose value stays the same. A store write keeps the applications carrier and the write_many park re-renders the users alone; the task is renamed to match. applications.md now says that user records inside the payload can predate every store write and that application config must not derive a value from a token.
Real ansible-playbook harness on 2.20.6 with the real roles, group_vars and write_many.yml: the re-park after a first mint and after a rotation drops from 4.0 s to 0.55 s on one host (applications renders 1 -> 0), and every token probe (users, config users path, email password, stalwart_users) equals the old code in compose and swarm mode. Dropping the digest from the users key as well leaves those probes stale, which the harness and two unit tests catch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Ansible forks a fresh worker per task, so every lookup('users') in a forked worker re-rendered the whole user catalogue (298 users, 0.3 to 0.5 s locally and about 1 s on the runner, 1 s locally once a users password is vaulted): _MERGED_USERS_CACHE lives in the process and had no carrier fact.
applications_carrier now also parks lookup('users', carrier=True) in _INFINITO_USERS_RENDERED at the same park points, and clear drops both facts. get_merged_users serves the fact to templated reads when its key matches: the applications carrier key (roles dir, applications, users, DOMAIN_PRIMARY, SYSTEM_EMAIL_DOMAIN, group_names, token store digest) plus inventory_hostname, so a carrier a run_once park broadcasts never answers another host with its become password. A park whose users carrier still matches leaves it in place instead of returning 119 KB again, so parks cost what they cost before.
Real ansible-playbook harness on 2.20.6 and 2.21.5 with the real roles tree: users renders per users-reading task after a park 1 -> 0, task 0.56 -> 0.08 s (vaulted 1.21 -> 0.17 s); a rotated token, a rotated inventory password, a cleared carrier and another host's carrier all re-render; 32 of 32 probe digests equal HEAD's. Removing the carrier check fails three unit tests. The stalwart swarm roundtrip pass 1 and its DR drill ran green with it on all nodes, with no 401 from JMAP, msmtp or LDAP.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A Dockerfile change notifies compose-up and compose-build in one flush. The compose pull handler (pull.py) builds every buildable service with `compose build --pull`, and the Build compose handler right after it built again with plain `compose build`. BuildKit keys a --pull build and a plain build differently, so the second build missed the first build's cache and ran every RUN layer again; in the stalwart compose roundtrip that was 233 s for tor, 129 s for openldap and 62 s for simpleicons, about 420 s per fresh compose deploy. It also overwrote pull.py's image, built on the freshly resolved base, with one from the plain-build cache: in pass 2 the --pull build of openldap ran its RUN layer again (34.5 s) while the plain build right after it was fully CACHED.
Both build handlers now run `compose build --pull` and fall back to a plain `compose build` when that fails, as pull.py does, so the second build of a flush is a cache hit and a lone-service rebuild keys like the flush builds. Without a reachable registry the fallback builds from the local base exactly as before.
A scratch compose project in the stack's daemon showed the cache behaviour: --pull after --pull CACHED, plain after --pull ran the RUN layer again. The new handler test fails on the old plain `compose build`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test_sso.py, test_sso_oidc_plugin.py and test_sso_proxy_consumers.py evict utils.cache.applications, utils.cache and the roles packages from sys.modules to import their plugin against fresh modules, and never put the originals back. A later test module in the same worker then held two copies of utils.cache.applications: plugins.lookup.applications, imported earlier, kept calling the old one, while test_email patched and reset the new one, so its carrier tests counted 0 renders and failed depending only on which modules xdist placed in one worker.
_load_module now runs inside mock.patch.dict(sys.modules): the plugin keeps the fresh modules it was built against, and sys.modules returns to its prior state afterwards. `pytest test_sso.py test_email.py` failed test_worker_reads_the_provider_view_from_the_provider_carrier and test_worker_renders_when_the_provider_carrier_key_differs before; the three SSO modules followed by test_email.py now pass 66/66. Same patch-id as feature/store-i18n ec0fde6561.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
91f25a35f2 drove the budget-kill drills through `timeout -k GRACE 0` and sent it SIGALRM at the deadline, relying on GNU timeout treating that signal as its own expiry: TERM to the stand-in's process group, KILL GRACE seconds later. The ubuntu-26.04 runner ships coreutils from uutils (`9.5-1ubuntu2+0.0.0~ubuntu25`), whose timeout does not handle SIGALRM that way, so the stand-in's `sleep 1000` outlived the driver, kept the captured pipes open and both drills hit their 120 s subprocess timeout (CI run 40869769501, test_window.py:191 and :204). Production code uses neither `kill -ALRM` nor `timeout … 0`; only the drill did.
The driver now starts the stand-in as its own process group (`set -m`), keeps the armed FIFO from 91f25a35f2, sends TERM to the group at the deadline and KILL after GRACE seconds if it is still alive, and exits 124 or 137 like GNU timeout did. It no longer depends on any timeout implementation; all eight drills pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_sso.py, test_sso_oidc_plugin.py and test_sso_proxy_consumers.py evict utils.cache.applications, utils.cache and the roles packages from sys.modules to import their plugin against fresh modules, and never put the originals back. A later test module in the same worker then held two copies of utils.cache.applications: plugins.lookup.applications, imported earlier, kept calling the old one, while test_email patched and reset the new one. Its carrier tests counted 0 renders and failed (CI run 40869769501, test_email.py:556 and :570, and the IAM merge gate), depending only on which modules xdist placed in one worker.
_load_module now runs inside mock.patch.dict(sys.modules): the plugin keeps the fresh modules it was built against, and sys.modules returns to its prior state, so the next module sees exactly what it would have without these tests. `pytest test_sso.py test_email.py` failed the two carrier tests before and passes 67/67 with the three SSO modules first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings feature/updater 3966abdb16, including updater-lockdown-probe: the deferred probe host guard (aa88429c25), the per-host post-load queue (70ed8a2e98), the NFS probe noauto (0ece4fcbc4), the E2E prepull gate (ef20262813), the worktree swarm lab subnet fixes, the Tor port window and restart skip, the Prometheus Docker poller, the Stalwart JMAP client cache and the FusionDirectory PHP hardening mount.
Conflict resolutions:
- plugins/lookup/email.py: keep store-i18n's NETWORK_MODE forwarding into the provider view and add updater's provider carrier.
- roles/svc-net-tor/templates/test.env.j2: updater's ONION_PORT_TIMEOUT=130, store-i18n's ONION_SUFFIX.
- roles/svc-swarm-node/README.md: updater's ingress guard wording, without the generated Quick Setup and Credits sections.
- roles/test-e2e-cli/tasks/main.yml: updater's cli_role_id loop variable with store-i18n's async cli.once condition.
- roles/web-app-prometheus/templates/nginx/locations.conf.j2: store-i18n's entity_name filter with updater's container_service lookup.
- tasks/stages/01_constructor.yml: updater's order (svc-swarm-node after the preload context opens) under store-i18n's task name, whose loop also covers svc-cache.
Integration fixes:
- roles/sys-stk-front-proxy: git applied updater's IS_STACK_HOST guard to the "Freeze front proxy context" task, which would have left workers without front_proxy_* facts. main.yml stays as on store-i18n; the guard moves into the per-sibling vhost.yml split, where the probe facts and the queue sit behind IS_STACK_HOST and front_proxy_domain_conf_dst_resolved stays unguarded for the Deploy arguments. deferred_probe.yml takes updater's host guard verbatim.
- roles/svc-net-tor/tasks/00_core.yml: the restart probe tests IS_SWARM_MODE instead of a literal DEPLOYMENT_MODE comparison.
- roles/web-app-stalwart/tasks/jmap_call.yml: the cached client splits into stalwart_jmap_url and stalwart_jmap_user_password, so the exec argv references no fact that carries a credential.
- roles/sys-service-loader/tasks/list_or_shoot.yml: the two direct-shoot tasks share their preload condition in one block.
- tests: the Prometheus wiring test imports entity_name and pins the host resources its renders need; the email carrier tests use an onion-only provider, which on store-i18n is what drops TLS (a dual-stack provider keeps it).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two tests that only answer on a GPU host, the logout catalogue translation check and the complete-catalogs integration test, were gated on accelerated() alone: a contributor with an NVIDIA runtime had no way to keep them out of a local gate while the GPU translation run is paused, so every gate on such a host stayed red on them.
INFINITO_TEST_GPU is a new env key. Its handler derives the default from INFINITO_GPU_COUNT: true where the compose services reserve a GPU, false for 0 or none. custom.env is applied after every handler, so `INFINITO_TEST_GPU=false` there switches the tests off on a GPU host; an exported value wins as well. tests/utils/gpu.py reads the flag for both tests, fails on any value other than true or false, and falls back to the same GPU detection only where no .env is loaded, such as a direct pytest run.
The handler joins gpu_count in utils/env/handlers/infinito/gpu/ (count.py, only_tests.py), because two handlers sharing the gpu prefix are a folder by the prefix-grouping lint. The new module avoids a test* name so unittest discovery never collects it. customize.md lists the key among the common overrides.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every JMAP call templated STALWART_ADMIN_USER and STALWART_ADMIN_PASSWORD (lookup('users')) and STALWART_JMAP_URL (config lookup) again. Ansible forks a fresh worker per task, so get_merged_users re-rendered the whole user catalogue in every call: about 0.4 to 0.6 s of controller time per call, 37 to 39 calls per deploy pass.
jmap_call.yml now sets the host fact stalwart_jmap_client (url, user_password) on the first call of a run, masked like the call itself, and every call reads it. STALWART_CONTAINER_ADDRESS stays in the command, so swarm keeps resolving the running container per call; 11_oidc_restart.yml replaces that container mid-run. The include interface is unchanged: stalwart_jmap_body in, stalwart_jmap out.
A scratch harness with a fake container measured the shell task at 1.01 to 1.10 s before and 0.46 to 0.49 s after, with argv and stdin sha256 identical in compose, swarm, vaulted-password and two-host runs. The admin password only rotates between runs, so a per-run fact stays current.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The nested carrier in tasks/utils/load/app.yml inherited run_once from the apply of sys-stk-full's front-proxy include, so ansible-core wrote the stack host's carrier, keyed with its group_names, to every play host. Workers and the NFS server then missed their own key and re-rendered the whole applications SPOT on the next carrier and on every task that read it. run_once: false keeps each host's carrier.
The email lookup's onion check renders the provider host's group view, a key no carrier on a worker holds, so every lookup('email') there rendered the whole SPOT: 8-11 s per task, 17 tasks per swarm pass in the stalwart roundtrip. Hand the provider host's carrier, read untemplated via hostvars.raw_get, to that view; get_merged_applications still serves it only when its key matches and renders as before otherwise.
Each bug hid the other: fixing either alone only moves the misses. Together they remove about 142 s per swarm pass. With real ansible-playbook and HostVars the worker email task drops from 3.97 s to 0.77 s on ansible-core 2.20.6 and from 4.22 s to 0.70 s on 2.21.5 with the same result; the new tests fail with the reuse disabled.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
c7fb6e60c5 narrowed every matrix row's services to the closure its
variant deploys (matrix.with_deployed_services) before axes.assign
derives the network states and the architecture pool. The selection
validator kept reading the raw discovery row, so it judged pins against
a different closure than the matrix it is meant to protect.
Manual run 40808080548 hit it: since 91022ad37a, svc-ai-litellm#1 no
longer deploys the guide host and with it the Tor provider, so `multi`
is impossible for that row. The validator still reported "Selection
inputs are valid for this branch", and priority_batches.sh then aborted
with "pinned network mode 'multi' is impossible here".
validate.problems now passes the same narrowed row to row_pulls_tor and
row_architectures. On the run's priority line it refuses exactly the
token the matrix refuses, before anything deploys.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two store-i18n-only code paths predate updater's custom roles
(web-app-cst-*, d92375922d), which the merge c9e7f950d9 brought in:
- utils/i18n/libretranslate.py deploys the translation tools lane with
cli.administration.deploy.dedicated but without --skip-pull. The lane
bind-mounts the checkout and its generated inventory declares no custom
role, so `make i18n-translate` would have run pull_custom_roles and
removed every pulled web-app-cst-* from the host's roles/. It now
passes --skip-pull, as every other generated deploy does.
- utils/i18n/extract.py role_messages globbed roles/*/meta/main.yml,
including the gitignored custom roles, so a checkout with a pulled
custom role failed test_core_catalogs_hold_exactly_the_current_sources
and its remedy would have written the custom role's description into
every tracked core catalog, against updater's rule that core never
tracks custom roles. It now skips is_custom(role), like the README
generators.
test_a_pulled_custom_role_stays_out_of_the_core_catalogs pins the second
point; without the skip it fails with the custom role in the list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
updater's Stalwart v1 port (store_guard.yml, migrate.env.j2) arrived in
the merge (c9e7f950d9) with three constructs store-i18n's lints reject:
- `DEPLOYMENT_MODE == 'swarm'` in store_guard.yml and migrate.env.j2
(test_deployment_mode_booleans): now IS_SWARM_MODE;
- `get_entity_name` in the store-engine fail_msg, a filter store-i18n
renamed to entity_name (a05071fe78), so TestAllUsedFiltersAreDefined
failed: now entity_name;
- two pairs of sibling tasks repeating the same `when`
(test_no_repeated_when_runs): the shared-database and dedicated-volume
lookups now sit in one block gated on `not (stalwart_store_marked |
bool)`, the migrate include and the pre-1.0 assert in one block gated
on `stalwart_store_layout is not skipped`; each task keeps its own
remaining condition. Ansible evaluates a block's `when` per task, so
the assert still sees the layout migrate.yml registers again.
updater's test_store_guard.py walked only the top-level tasks and failed
with KeyError: 'register' on the block wrappers. It now walks
block/rescue/always (utils.roles.invocations.NESTED_KEYS), hands a
block's `when` down to its children and checks the block condition
before the task's own, as Ansible does; the assertions are unchanged.
With either block's `when` removed from a copy of store_guard.yml, two
respectively four of its tests fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BudgetKillDrillTests drove the stand-in deploy with `timeout -k GRACE 3`.
Under load (gate pass 76 ran unit, integration and lint in parallel)
timeout's TERM arrived while bash was still starting, before the
stand-in's `trap cleanup EXIT` existed, so bash died on TERM without the
rescue: test_without_the_window_the_kill_lands_before_the_copy failed
with "124 != 137" and only "kill_epoch=…" on stdout. A 4 s stall before
the trap reproduces it on the old driver; nothing in the merge touched
the rescue code.
The stand-in now writes to an `armed` FIFO right after it installs the
trap. The driver runs `timeout -k GRACE 0 …` (duration 0 disables
timeout's own clock, its TERM-then-KILL handling stays), waits for
`armed`, sleeps until the deadline epoch and sends timeout the same ALRM
its own expiry raises, so TERM still lands at the deadline and KILL
GRACE seconds later, with the same 124/137 exit codes. `wait -n -p`
lets the failure-before-the-deadline drill return as soon as its deploy
ends. No timeout was raised and nothing retries; with the 4 s stall all
three drills pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
updater's TestWriteMany (86e7c844f3) looked up the persist, mirror and
hand-over tasks at the top level of write_many.yml and evaluated only
each task's own `when`. The merge (c9e7f950d9) moved those three tasks
into store-i18n's change-gated block ("💾 Write the token store when the
merge changed it", `when: sys_token_store_users_merged !=
sys_token_store_users`) and switched the mirror to IS_SWARM_MODE, so the
lookups raised StopIteration.
The test now walks into block/rescue/always, hands a block's `when` down
to its children and ANDs it with a task's own condition (a task without
any condition counts as true, as in Ansible), renders with
IS_SWARM_MODE, and counts the re-render inside blocks too. The
assertions are unchanged; with the block's `when` removed from a copy of
write_many.yml, "an unchanged store skips the write and the re-render"
fails again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
updater's TestSwarmDeployCommands (d92375922d) patched
utils.tests.swarm.matrix.run_step, which store-i18n's matrix no longer
imports: every step goes through the module's _run(cmd, *, env, label),
which wraps the watchdog. After the merge (c9e7f950d9) both tests failed
with "does not have the attribute 'run_step'".
They now patch matrix._run, as test_run_reporting and
test_extend_inventory already do, and keep their assertions: the round
deploy (cli.administration.deploy.swarm) and the backup-host deploy
(cli.administration.deploy.dedicated) both pass --skip-pull, so the
custom roles placed before the round survive. The backup-host test hands
_deploy_backup_host a temporary inventory directory, because store-i18n
reads and may rewrite host_vars there (drop_backup_network_mode) before
it deploys.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d92375922d (feature/updater) added TestCreditsLicence, which exercises
_validate_credits in tests/lint/ansible/roles/test_readme.py so that a
custom role (web-app-cst-*) may state its own licence in a committed
Credits section. On feature/store-i18n Credits is a generated section
the docs build renders and the README lint forbids committing it, so
the merge (c9e7f950d9) kept store-i18n's test_readme.py without
_validate_credits, and the five tests failed with AttributeError.
The operator decided to remove the tests now and to give custom roles
their own licence statement later as a separate feature. The three
package __init__.py files under tests/unit/python/tests/lint/ only
existed for this module and go with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Semaphore v2.19.17 builds its post-login redirect from its one WebHost
(url.JoinPath(WebHost, return)), marks the session cookie Secure when
WebHost is https, and registered a single OIDC provider whose
redirect_url is clearnet. A login started on the onion sibling therefore
lost the oauthstate cookie at the clearnet callback and the session on
the plain-http onion origin: semaphore#0 of run 40625679270 failed five
login and design specs in PASS 2, where Playwright picked the .onion
sibling ("http: named cookie not present" 24 times in the container
log, while LDAP accepted the rotated credentials six times).
Semaphore keys OIDC providers by id from the URL path and gives each its
own redirect_url, so when the role has an onion sibling and SSO is on:
- a second provider keycloak_onion, same issuer, client and claims, with
the onion sibling's plaintext redirect_url; Keycloak's redirect_uris
lookup already registers http://<onion>/*;
- the onion vhost rewrites the absolute WebHost redirects to relative
ones (proxy_redirect) and drops Secure from the semaphore cookie
(proxy_cookie_flags), next to the existing base-href rewrite;
- location = /api/auth/login filters the provider list per vhost with
sub_filter, so clearnet offers keycloak and the onion vhost
keycloak_onion, both labelled as before; each dropped provider has a
pattern for either list position because Semaphore does not fix the
order.
Without an onion sibling the provider JSON and both proxy hooks render
exactly as before. test_onion_login renders the vars through Ansible's
templar for both cases; 16 of its 18 tests fail against the previous
files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
68 conflicts, resolved per hunk; resolutions agreed with the Updater
session were applied as agreed:
- svc-db-postgres (superuser password script and core) and
sys-svc-rdbms realign_dedicated_db: updater's argv/stdin redesign with
expand_argument_vars: false, nested into store-i18n's block structure.
- svc-db-redis 01_init: store-i18n's ACL password on stdin.
- sys-svc-proxy forwarded.conf.j2: store-i18n's handover-aware file,
without the forwarded_base include (Host and X-Forwarded-Proto would
be sent twice); forwarded_base serves the inline consumers, e.g.
web-app-seaweedfs nginx.conf.j2.
- basic.conf.j2 and web-app-openproject proxy.conf.j2: issuer redirect,
then handover, else updater's location/sso_acl.conf.j2; the
server-level webserver_extra_configuration hook stays.
- web-app-keycloak: updater's admin-REST settings step and renumbered
update steps; update/01_client.yml removed; store-i18n's design step
and LDAP block kept; a73435cc9d arrived through rename detection.
- web-app-nextcloud OpenProject provisioning: updater's rewrite, all
credentials on the Ruby runner's stdin.
- web-app-semaphore: store-i18n's canonical web root (a1d7835863).
- Role READMEs: no generated Cosmos/Architecture/Quick Setup/Credits
sections; desk-* paths resolved to the dsk-* renames.
Integration fixes for what merged cleanly but did not fit together:
- web-app-matrix ansible vars: drop updater's second
matrix_client_element_configuration_extension_json, which silently
overrode store-i18n's design-aware value.
- url.origin instead of a stripped url.base in 19 places store-i18n
added since 44aaa68924, same lookup targets (the double-slash lint).
- Package batch (18ea75049e, cb4d340de9): installs the deploy-start
batch covers are removed (caffeine, cursor, blender, npm in dsk-gnt-*,
with shared references), the rest carry batch: false and a nocheck
with the verified reason (PyPI fallback, before the batch, vendor
repository, configured flavor only, addon gate); dev-npm references
the root npm id.
- sys-token-store write_many.yml: store-i18n's change-gated block and
IS_SWARM_MODE.
- web-app-openproject: IS_COMPOSE_MODE in 02_migrate/run.yml; the
seeded-admin lock (ccbf8f168d) ported into 04_admin/provision.yml and
covered by the admin-state digest.
- Bucket policy (3b4ac40df5 and follow-ups) moved to web-svc-seaweedfs,
where store-i18n's object store lives; the S3 admin secret travels on
stdin instead of a task-level environment.
- act workflow.sh: no Exception-labelled comment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The update started one kcadm JVM per step: the client and both realms' mail settings as async jobs with their reaps, then each declared client in turn, about 27-31 s in a Nextcloud deploy. realm_settings.py reads the client scopes, clients and both realms once over the admin REST API, creates what is missing and writes a client or realm only when it drifted; secrets reach it on stdin through the new rest_script_stdin parameter. The remaining update steps are renumbered and every reference follows.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`make autoformat` rewrites roles/web-app-seaweedfs/vars/main.yml and
tests/unit/python/roles/web-app-stalwart/test_store_guard.py on every
checkout of main, so every quality gate on every branch leaves them as
unstaged changes, and the gate lints a tree the commit does not carry.
The YAML change folds one line of the `>-` scalar
SEAWEEDFS_S3_ADMIN_CURL; the parsed value is identical. The Python
change is ruff's line wrapping only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
