A filesystem stall killed a Cloudflare record job before async_wrapper saw its start notice (Broken pipe), and the shared reap masked it. Each record job now has a request timeout and an attempt deadline; judge.yml reaps every job of an attempt and async_retry_split sorts the failures: hangs, timeouts, connection errors, 429 and 5xx get up to cloudflare_attempts tries, anything else fails at once and names its record. An answer that says the record already stands counts as done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On swarm workers and the nfs-server, every full lookup('email') rendered the whole applications tree again (about 20 s per task, 8 tasks per CI run, Nextcloud#7 of run 41261531965), because the provider host's carrier is keyed on the full applications inventory and each mesh member carries its own wireguard interface there. The stage-wide applications_carrier park now also parks the provider view (this host's variables under the provider host's group_names and NETWORK_MODE) on hosts that are not the onion-serving provider host, keyed like the applications carrier plus [provider host, NETWORK_MODE]; lookup('email') serves it only on an exact match and renders fresh otherwise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every network instance (dashboard, dashboard-onion) carried the full build block, so compose built the same dashboard_custom tag twice in parallel. In CI run 41261531965 (odoo#0) dockerd 29.9.2 died with SIGSEGV inside BuildKit during that double build and stopped the stack. Now only the first instance builds; its siblings run the same tag with pull_policy never in compose, the pattern bookwyrm, espocrm and openproject use. Swarm builds, labels and pushes the image once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_openproject_redirect_uris builds the vars path from ROLE_FILE_VARS_MAIN, and test_store_migration_job reads its tempdir stub log under a cache-read nocheck, because the log is rewritten between reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stalwart-migrate 1.0.0 failed the final verify on echoserver (6 rows) and galaxyserver (2 rows), all x:BlockedIp entries that expire when the tool starts 1.x, so a final run could never pass; 1.0.0 to 1.0.3 also skip accounts without a change counter. The pin moves to 1.0.4, final runs with --verify-full instead of a sample of 16 accounts, and a failed final restarts 0.16 and prints the error and verify rows of the new report phase, since final-report.txt is only written by a passing run. The README describes how to read the rows and when to start over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every lookup('email') resolved all 15 settings. On swarm workers and the nfs-server of Tor rows that took 22-26 s per call, even for tasks that only read external or timeout and then skip (CI run 41261531965, Nextcloud#7: ~600 s over both passes, the job ran out of its budget). The lookup takes key= now and stops after that entry; the four call sites and mail_recipient that need only external or timeout use it. Without key= the result is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Users could change their Keycloak email without verification, and every deploy stamped any address an inventory user held as verified. An address no Keycloak account holds (for example erpnext-api-bot@) could thus be claimed and reach apps that link SSO logins by email. The user profile now lets only admins edit the email, and email_verified.py stamps only an address that equals the inventory address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI run 41261531965 Nextcloud#3 (multi network) tested Nextcloud on its onion address. integration_openproject builds redirect_uri from the request host, so it sent http://next.cloud.<onion>.onion/.../oauth-redirect, but ensure_nextcloud_storage.rb registered only the URIs built from the canonical NEXTCLOUD_URL; Doorkeeper answered every /oauth/authorize with 400 invalid_redirect_uri (proxy log: 6x 400, OpenProject web log status=400 user=6, trace page 'The requested redirect uri ... doesn't match client redirect URI').
NEXTCLOUD_NETWORK_BASE_URLS comes from lookup('network_urls', NEXTCLOUD_URL), like the Zammad and GitLab base URLs, and reaches the provisioning input as nextcloud_base_urls; the script registers every base URL with the apps/ and index.php/apps/ paths, canonical first, and the base URLs feed the converge digest so an install switched to multi network re-provisions. Clearnet-only and tor-only rows register the same string as before. Zammad, GitLab and Mattermost integrations still register only the canonical URI.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
galaxyserver deploy -6 ran the migration as one plain command for 28:30 and then lost its SSH channel (ServerAlive 15x3 ends it after 45 s of silence): the deploy ended UNREACHABLE while the script kept running on the host unobserved, possibly through final, which stops Stalwart and swaps the databases.
The task now runs migrate.sh as an async job polled every STALWART_MIGRATE_ASYNC_POLL (60 s); each poll is a short new SSH call and a failed poll is retried, so a dropped connection no longer ends the deploy. The budget STALWART_MIGRATE_ASYNC_TIME is lookup('timeout', 21600) (6 h, scaled by TIMEOUT_FACTOR); past it Ansible kills the job and the task fails loudly. Every phase but status takes a non-blocking flock on migrate-v016/migrate.lock and stops while another run holds it or a stalwart-migrate container of an earlier run still runs, so two migrations never touch the same store and dump. A kill inside final leaves Stalwart stopped and the next run refused while the orphan container lives.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gate pass 23 failed tests/lint/filesystem/shell/test_no_or_true.py on init-gate.sh: occ_status swallowed every exit code with || true. It now returns 0 with the status on success and maps 1 (occ error), 124 (timeout) and 255 (php fatal while the tree is copied or upgraded) to an empty status, the not-ready state both slots already handle. Any other code, such as 127 from a missing su, timeout or php, ends the gate with that code and a message instead of waiting forever. Both slots now call occ_status directly so that exit reaches the gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In swarm CI run 41206003651 (Nextcloud#11) the first occ maintenance:install failed on a config.php lock after the upstream entrypoint had already copied version.php. Upstream skips the install whenever version.php exists and never retries, so the replacement task served an uninstalled Nextcloud that occ status still reported as healthy, and slots 2 and 3 waited for installed: true until converge gave up after about 25 minutes.
Slot 1, and compose where TASK_SLOT is unset, now exits 1 with the cause and the recovery when version.php and occ exist, the init lock is free and occ status prints installed: false. The task crash-loops and the stack readiness check ends the deploy within minutes with that message in the service log. occ status runs with a 120 s timeout in both branches. Why the lock failed on NFS is not addressed here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Matomo, SuiteCRM, WordPress, MediaWiki, Joomla and Listmonk only realigned an existing admin login, so existing installs without platformadmin stopped the deploy. Each admin step now creates the account next to the old admin when it is missing and realigns it when it is an admin.
A platformadmin that exists without admin rights fails with its cause and remedy. MediaWiki alone promotes it, and only while SSO is the sole login and its confirmed email matches: there the account can only come from Keycloak's preferred_username. SuiteCRM keeps native login under SAML and lets users rename themselves, Listmonk's OIDC never creates that login, and Joomla's Keycloak plugin rewrites groups on every sign-in, so none of them promote.
Matomo, SuiteCRM and WordPress write through php://stdout and php://stderr: PHP 8.2 defines no STDOUT/STDERR for code fed on stdin. MediaWiki's account probe lives in files/sql/admin_state.sql and receives name and email as hex literals. A lint checks that each role's admin step creates a missing admin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The users cache no longer demands a mailbox on every host (a workstation
inventory without mail broke every users lookup) and skips forwards that
are still unrendered Jinja; a non-mapping target names itself. Stalwart now
enforces the mailbox where it consumes the forward. Under SSO 16_forwards
skips ReloadSettings, which would open the OIDC issuer before Keycloak runs;
the OIDC restart applies the limits. The routing probe treats transient
IMAP/SMTP errors as not yet within the same budget and only checks forwards
in Stalwart's own rounds, so the 67 roles that deploy it as mail provider
keep skipping it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
System mail now goes to platformadmin@, a mailbox nobody reads. The users
cache already carries a per-user forward and Stalwart already writes a
redirect :copy Sieve script per forwarding account; stalwart_user_forwards
feeds the first into the second for every user Stalwart has an account
for, merged with inventory forwards. platform_administrator forwards to
administrator, keeping its own copy. The users cache now rejects forwards
to unknown users, to users without a mailbox, and forward loops.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A user the host's roles need but the inventory gives no password falls back
to a generated password that renders anew on every read, so password
converges set values nobody knows (echoserver and galaxyserver stopped at
web-app-matomo for that after passing validation). The validator now asks
provision's required_user_policies for each host's groups, merges the
host's users with Ansible's replace precedence and fails before the deploy,
naming the users and the provision command. Presence only: vaulted values
count, nothing is decrypted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A role's playwright.run_after names the roles its pass runs after; the new
playwright_order lookup reads it through role_tests (which now renders
Jinja with the queried role as application_id) and orders the round with
a stable topological sort, so only constrained roles move. web-app-docs
runs after every other invokable role: in CI run 41206003651 (Discourse#2)
its pass waited 3126 s for the docs server's Sphinx build while stalwart,
bigbluebutton, jitsi and prometheus (~400 s) queued behind it. Cycles,
self references, unknown roles and non-list values fail loudly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gate pass 15 rejected rbac_reconcile.py and rbac_realm_groups.py as a prefix group. They are now rbac/reconcile.py and rbac/realm_groups.py with mirrored tests; the shared rest_script runner copies a script from its subpath and stages and runs it under its file name next to keycloak_admin.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gate pass 14 rejected the parent-relative imports (ruff TID252) the moved weblate_network tests used for _stand_ins; the helper now sits in the same test package and test_settings_override imports it downward.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The realm RBAC step ran one kcadm round trip per group path and member through ensure_group_path_members.sh, about 609 s per XWiki job. A Python helper now reads the realm's groups and memberships once, computes the missing paths and members, and applies only the difference in one run; each failing item still fails loud and names itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Magento 2.4.8 wipes generated/ on every module:enable or module:disable, decided by the command name alone, and the role ran the toggle on every deploy. On CI Magento#0's update pass a Prometheus probe rendered a page during that wipe and wrote new generated classes into it, so rmdir failed and the deploy died. The role now reads module:status --enabled first and only flips the modules in the wrong state; on a fresh install the toggle runs before nginx starts routing to PHP, and a real flip also reloads nginx.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The access-log test from main (#1322) rendered nginx.conf.j2 with DEPLOYMENT_MODE, while store-i18n's template reads IS_SWARM_MODE, so every case failed with an undefined variable after the main merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gate passes 12 and 13 rejected the prefix-grouped weblate_network_*.py modules, the stand_ins.py test helper and unmirrored test paths. The modules are now weblate_network/strategy.py and middleware.py, mounted as a namespace package under /app/infinito and referenced by their dotted package names; the helper is _stand_ins.py and the module tests mirror the package.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Publishing tag v1.2.3 globbed prepared/v1.* and deleted every other v1.x tag's prepared tree, including one a parallel slot could be cloning from. With the exact glob each opened tag would keep its ~2.3 GB site and ~1 GB prepared tree forever, which 66 tags and a crawler following the version switcher turn into a full disk. Requests now stamp opened/<tag>, and after every build the builder moves out, under the slot lock, every tag beyond the three opened last that is neither queued nor building; latest and deployed always stay, and a dropped tag rebuilds on its next visit. The builder also clears scratch a killed predecessor left.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The onion Nextcloud page frames the clearnet Collabora editor, but the proxy's frame-ancestors only allowed 'self' and *.DOMAIN_PRIMARY, which matches neither the onion host nor its http scheme, so Chrome blocked the iframe and the editor websocket never opened. Collabora now adds every Nextcloud network origin, each with its own scheme, to its frame-ancestors through webserver_csp_whitelist_extra_by_app before the proxy stack renders.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Weblate built every absolute URL, the social-auth redirect and the CSRF origin from WEBLATE_SITE_DOMAIN, so the onion sibling either redirected to clearnet or failed the OIDC round trip. A settings override swaps in a network-aware social strategy and proxy middleware that take scheme and host from the request, drops the forced HTTPS redirect the http onion vhost cannot satisfy, and ALLOWED_HOSTS now lists every network sibling plus the container name and loopback. The two modules are separate because social_django loads the strategy at app-ready before defining load_strategy, which weblate.middleware imports back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cadvisor ran with its defaults: every series of every cgroup carried the union of all containers' label keys (97 on echoserver) and every cgroup was re-read each second, so each scrape grew with containers times images and the 0.25-CPU cadvisor was OOM-killed in its 1g cgroup about every 13 minutes. It now stores only the compose and swarm identity labels and keeps housekeeping at 10s, below the 30s scrape interval; limits and metric groups stay unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gate pass 9 on the merged store-i18n stopped at ruff N818 on the local Stop exception that the builder-timeout test defined next to the module's StopBuilderError from the build-before-clone tests, plus SIM117 on the nested with in the healthy-mirror test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
