The entrypoint refills the shared code volume only when the release changes, so a volume bootstrapped at 5.3.0 by an image without auth_oidc never received the plugin from a later same-release image; galaxyserver's production deploy failed at the presence check although the image ships it (probe: /opt/moodle-source/public/auth/oidc exists). 04_oidc.yml now asserts the image ships the plugin, naming the image when it does not, and syncs it into the volume as the runtime user through a staging directory, only when diff -rq finds a difference, before the existing presence check. The OIDC upgrade.php run no longer hides a failed plugin install behind failed_when: false, matching the MCP and objectfs plugin tasks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Production hosts carry objects from before the compose project was named after the entity: galaxyserver's mariadb, openldap and postgres networks have no project label, and echoserver's openldap_data volume is labelled svc-db-openldap, the role id older deploys passed as -p. The pre-flight stopped both deploys although compose reuses such networks and volumes with a warning. A network named after its entity now passes without a label or with a declaring role id, still checked against its subnet; a volume passes with a declaring role id as well. A container does not, because compose cannot see a container of another project and fails on its name. A label naming another project still stops the deploy, and the volume message now says compose would hand the role another project's data instead of claiming it refuses the volume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ingress guard read ran two remote shells per swarm deploy, each asking container info again (WordPress#0 run 41303726270: 66 s + 61 s over 64 calls each). One shell now prints 'port <json>' and 'node <address>' lines into swarm_ingress_read; the new swarm_ingress_sections filter splits them by tag and raises on any untagged line, and SWARM_INGRESS_GUARDED and SWARM_INGRESS_PEERS read their section, so the guarded set, the peers, the firewall digest and the ruleset stay the same. A test ties the --format tags in read.yml to the sections the filter knows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
render_override gathered the metadata of every image (docker image inspect, a pull when missing, a docker run probe) before it checked wrap, and the swarm handler's --no-wrapper path threw the result away, because image_meta is read only behind 'if not wrap: continue'. The gather now runs only when wrap is true, so the swarm override is unchanged byte for byte. It pulled again the externals that registry_sync had just removed (WordPress#0 run 41303726270: 275 s over 64 calls).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pre-flight stopped every deploy on a filesystem above SIZE_PERCENT_MAXIMUM_DISK_USAGE, which blocked a 1.8 TB workstation at 99 % with 30 GB free while a 145 GB runner was fine. A filesystem now fails only when it is above the percentage AND offers less than SIZE_GB_MINIMUM_DISK_FREE (20 GB): disk_usage_errors reads df's available column, the preflight_disk_usage filter passes the floor, and disk.yml runs df -P -k so the block unit stays KiB even under POSIXLY_CORRECT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts resolved by combining both sides: utils/tests/swarm/run.py keeps store-i18n's re-export of run_watched from watchdog.py, and the #1330 storage roots (lowest_free_mb over / and /var/lib/containerd, the fullest root in the abort banner and the sample line, df over every root) move into watchdog.py; test_run_storage_roots.py now targets watchdog and expects the process-group kill. The seaweedfs 4.52 bump lands in roles/web-svc-seaweedfs, where store-i18n moved the engine; roles/web-app-seaweedfs keeps only the console.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The constructor now runs tasks/utils/constructor/preflight.yml before the package install and fails with a named remedy when: the running kernel has no module index on disk or the clock is not synchronised (chrony leap status, else timedatectl, outside containers); / or the docker root is fuller than SIZE_PERCENT_MAXIMUM_DISK_USAGE (90); a docker network contradicts the entity that declares it (name, compose project label, subnet outside its block, overlap with another declared subnet); a declared container lacks or a declared volume carries a foreign entity label; a docker default address pool overlaps the mesh, Tor or a declared subnet; a host process outside docker holds a public port of the deploy; or, in assert mode, a clearnet domain of the deploy does not resolve.
A volume without any label passes, since compose adopts it and installs before this change created the Synapse volume with docker run. web-app-discourse declares its launcher-started containers through networks.local.launcher, and web-app-matrix labels a new Synapse volume with its compose project.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
swarmkit sleeps update_config.delay after the last slot too and then monitors for delay + 1s, so a one-replica service paid about 21 s per rollout for nothing to stagger (database stacks converged in 32 s instead of 8 s in WordPress#0's update pass of run 41303726270, about 400 s per row). A service whose replicas line is 'replicas: 1' now rolls with delay 0s; several replicas keep service_update_delay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BBB's upstream compose template publishes nginx's 48081/48082/48087 on 127.0.0.1 and on 10.7.7.1. On Arch, Docker's nftables backend with the userland proxy off writes one identical 'MASQ TO OWN PORT' rule per binding and the second fails with 'rule exists' (moby 60109b06fb, PR #54031, first released in 29.10.0), so compose up failed in Moodle#3 of run 41303726270. compose_mods keeps only bindings on services.bigbluebutton.upstream_host or without a host address; nothing in the platform uses the loopback ones.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The builder handed every queued marker straight to its worker pool, whose own FIFO let a background latest build run ahead of the deployed:de build a visitor waited on (40-81 min in docs#1 of run 41303726270). A marker is now claimed only while a slot is free. build_language dequeues a translation that is already current instead of rebuilding it for 2-3 h on a duplicate request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the swarm NFS volume a docs replica keeps a failed lookup until the parent's cached attributes expire, so a site another replica just renamed into place stayed invisible to it and it served the 202 'Building' page (lang en) after a sibling had answered 200 (docs#1 of run 41303726270). recheck_servable() lists the parent to force close-to-open revalidation and is called only on the path that would answer 202; ordinary page requests pay nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 2g the vulnerability-detection feed update thrashed the manager cgroup until the kernel OOM-killed wazuh-modulesd; Wazuh restarts no daemon, so the API answered every call with HTTP 500 (error 1017) and the API administrator grant failed in the update pass of run 41303726270 (Wazuh#1, swarm). The manager gets mem_limit 4g, and api_administrators.py raises '<METHOD> <path> answered HTTP <code>: <body>' instead of a bare 500 so a dead daemon shows up by name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mesh is IPv4 only (60-byte header, IPv4 pools), yet wg.conf handed
wg the endpoint name, and wg dials the first getaddrinfo answer, an AAAA
on a dual-stack name. A host with broken IPv6 then never handshook
(alphaserver, 2026-11-01). Resolve every dialled endpoint with getent
ahostsv4 on the dialling host, render that literal, measure the path MTU
against it, and fail before rendering when a name has no IPv4 address.
Also set the tunnel MTU on a live interface, since wg syncconf drops it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deploy bypasses ERPNext's setup wizard and with it install_defaults, so a site without the store's GUI fixtures had no Standard Selling or Standard Buying price list and every Item Price insert failed with "Could not find Price List: Standard Selling". company.py now creates both after the Company, insert-if-missing, with the wizard's selling and buying flags and the Company's own currency; Price List.on_update fills the Selling and Buying Settings defaults.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ansible's copy action runs one stat and one file module per file of a
directory source even when nothing changed, about 0.6 s each over SSH.
The echoserver deploy paid it on 31 branding renders and the docs
tooling tree (~770 s and 169.5 s, deploy -9). Before falling through to
the stock copy, hash the tree locally and compare it with one
file_checksums call on the target; when every file matches in type,
content and mode, return the stock copy's unchanged result. Any
difference, symlink, empty directory, owner or symbolic mode falls
through to the stock copy unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gate counted any stamp above zero as handshaken. wg syncconf keeps
the old stamp while it points the endpoint at a dead path, so a deploy
passed the gate over a broken tunnel (alphaserver, 2026-11-01: the
endpoint resolved to an AAAA the host could not reach). Read the host's
epoch in the same shell call as latest-handshakes and accept a stamp
only within WireGuard's REJECT_AFTER_TIME of 180 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deploy bypasses ERPNext's setup wizard and with it install_fixtures, so a site without the store's GUI fixtures had no root Item Group and every consumer creating one failed with "Could not find Parent Item Group: All Item Groups". company.py now seeds All Item Groups, All Customer Groups and All Supplier Groups as root groups, insert-if-missing like All Territories.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ansible-playbook splits each -e value through parse_kv on whitespace, so json.dumps' default ', ' separator broke PRIMARY_APPS=["a", "b"] into a raw parameter and the list never reached the play. Dump extra vars with separators=(',', ':') in the development deploy and the swarm matrix deploy, and test that a list extra var survives parse_kv.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Consumers that build on the ERPNext site from their own repositories (web-app-cst-store, web-app-cst-security-check) now declare the roles they need on erpnext-api-bot in services.erpnext.api_bot.extra_roles instead of granting them themselves. Every deploy grants them beside System Manager and the Server Scripts role, creating an empty Role first when the site does not hold it yet, so this role stays the only writer of the bot's roles and keys. Port of the api-bot part of 067 from feature/iam-profile-rbac (3e5acbd104).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The small GitHub runner shape (AWS, 72G root) ran out of disk in swarm and compose rows (#1329: WordPress#2 of run 41261531965, Nextcloud#6 and #7 of run 41262286962) while an empty instance-store disk nvme1n1 sat unused. mount_spare_disk.sh formats and mounts it at /mnt, but only on a GitHub-hosted runner outside act, and only a whole unpartitioned disk without any signature, mount or swap use of at least 75G; every decline leaves a notice. relocate_docker_dataroot.sh then bind-mounts /var/lib/containerd from the spare disk and moves the Docker data-root to /mnt/docker-host (not /mnt/docker, which is the nested daemon's volume that dedicated.sh wipes). It refuses before stopping anything when images, volumes, build cache, foreign containerd namespaces or shims exist or daemon.json is unreadable, and rolls back the bind mount and daemon.json on any later failure or cancel. Swap goes on / once containerd lives on /mnt, dump_state.sh logs disk size, model and free space per storage root, and the swarm watchdog samples / and /var/lib/containerd every 10 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under SSO the OIDC directory is the authentication directory, and Stalwart refuses any password credential for its accounts ("Cannot set credentials for accounts in an external directory"). Every create carried one, so an account added after SSO was switched on never came into existence: the echoserver deploy failed on the keycloakbootstrap bot right after the 0.16 to 1.x migration, and a new person would have been skipped silently. Under SSO the create now carries no credentials and the password update is skipped; existing accounts come back as primaryKeyViolation with their id, as without SSO. Any remaining refusal fails the deploy. Closes the accounts entry of the role's TODO.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Cloudflare request timeout, attempt deadline, collect deadline and retry delay now scale through lookup('timeout', ...) together, so a raised TIMEOUT_FACTOR keeps them consistent; the deadline task name fits 80 characters; test_async_retry_split reads the role defaults through PROJECT_ROOT, ROLE_FILE_DEFAULTS_MAIN and load_yaml. The provider-view helpers move from plugins/lookup/email.py (531 lines) to utils/mail/provider_view.py; provider_view_carrier stays in the lookup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A filesystem stall killed a Cloudflare record job before async_wrapper saw its start notice (Broken pipe), and the shared reap masked it. Each record job now has a request timeout and an attempt deadline; judge.yml reaps every job of an attempt and async_retry_split sorts the failures: hangs, timeouts, connection errors, 429 and 5xx get up to cloudflare_attempts tries, anything else fails at once and names its record. An answer that says the record already stands counts as done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On swarm workers and the nfs-server, every full lookup('email') rendered the whole applications tree again (about 20 s per task, 8 tasks per CI run, Nextcloud#7 of run 41261531965), because the provider host's carrier is keyed on the full applications inventory and each mesh member carries its own wireguard interface there. The stage-wide applications_carrier park now also parks the provider view (this host's variables under the provider host's group_names and NETWORK_MODE) on hosts that are not the onion-serving provider host, keyed like the applications carrier plus [provider host, NETWORK_MODE]; lookup('email') serves it only on an exact match and renders fresh otherwise.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every network instance (dashboard, dashboard-onion) carried the full build block, so compose built the same dashboard_custom tag twice in parallel. In CI run 41261531965 (odoo#0) dockerd 29.9.2 died with SIGSEGV inside BuildKit during that double build and stopped the stack. Now only the first instance builds; its siblings run the same tag with pull_policy never in compose, the pattern bookwyrm, espocrm and openproject use. Swarm builds, labels and pushes the image once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_openproject_redirect_uris builds the vars path from ROLE_FILE_VARS_MAIN, and test_store_migration_job reads its tempdir stub log under a cache-read nocheck, because the log is rewritten between reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stalwart-migrate 1.0.0 failed the final verify on echoserver (6 rows) and galaxyserver (2 rows), all x:BlockedIp entries that expire when the tool starts 1.x, so a final run could never pass; 1.0.0 to 1.0.3 also skip accounts without a change counter. The pin moves to 1.0.4, final runs with --verify-full instead of a sample of 16 accounts, and a failed final restarts 0.16 and prints the error and verify rows of the new report phase, since final-report.txt is only written by a passing run. The README describes how to read the rows and when to start over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every lookup('email') resolved all 15 settings. On swarm workers and the nfs-server of Tor rows that took 22-26 s per call, even for tasks that only read external or timeout and then skip (CI run 41261531965, Nextcloud#7: ~600 s over both passes, the job ran out of its budget). The lookup takes key= now and stops after that entry; the four call sites and mail_recipient that need only external or timeout use it. Without key= the result is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Users could change their Keycloak email without verification, and every deploy stamped any address an inventory user held as verified. An address no Keycloak account holds (for example erpnext-api-bot@) could thus be claimed and reach apps that link SSO logins by email. The user profile now lets only admins edit the email, and email_verified.py stamps only an address that equals the inventory address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI run 41261531965 Nextcloud#3 (multi network) tested Nextcloud on its onion address. integration_openproject builds redirect_uri from the request host, so it sent http://next.cloud.<onion>.onion/.../oauth-redirect, but ensure_nextcloud_storage.rb registered only the URIs built from the canonical NEXTCLOUD_URL; Doorkeeper answered every /oauth/authorize with 400 invalid_redirect_uri (proxy log: 6x 400, OpenProject web log status=400 user=6, trace page 'The requested redirect uri ... doesn't match client redirect URI').
NEXTCLOUD_NETWORK_BASE_URLS comes from lookup('network_urls', NEXTCLOUD_URL), like the Zammad and GitLab base URLs, and reaches the provisioning input as nextcloud_base_urls; the script registers every base URL with the apps/ and index.php/apps/ paths, canonical first, and the base URLs feed the converge digest so an install switched to multi network re-provisions. Clearnet-only and tor-only rows register the same string as before. Zammad, GitLab and Mattermost integrations still register only the canonical URI.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
galaxyserver deploy -6 ran the migration as one plain command for 28:30 and then lost its SSH channel (ServerAlive 15x3 ends it after 45 s of silence): the deploy ended UNREACHABLE while the script kept running on the host unobserved, possibly through final, which stops Stalwart and swaps the databases.
The task now runs migrate.sh as an async job polled every STALWART_MIGRATE_ASYNC_POLL (60 s); each poll is a short new SSH call and a failed poll is retried, so a dropped connection no longer ends the deploy. The budget STALWART_MIGRATE_ASYNC_TIME is lookup('timeout', 21600) (6 h, scaled by TIMEOUT_FACTOR); past it Ansible kills the job and the task fails loudly. Every phase but status takes a non-blocking flock on migrate-v016/migrate.lock and stops while another run holds it or a stalwart-migrate container of an earlier run still runs, so two migrations never touch the same store and dump. A kill inside final leaves Stalwart stopped and the next run refused while the orphan container lives.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gate pass 23 failed tests/lint/filesystem/shell/test_no_or_true.py on init-gate.sh: occ_status swallowed every exit code with || true. It now returns 0 with the status on success and maps 1 (occ error), 124 (timeout) and 255 (php fatal while the tree is copied or upgraded) to an empty status, the not-ready state both slots already handle. Any other code, such as 127 from a missing su, timeout or php, ends the gate with that code and a message instead of waiting forever. Both slots now call occ_status directly so that exit reaches the gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In swarm CI run 41206003651 (Nextcloud#11) the first occ maintenance:install failed on a config.php lock after the upstream entrypoint had already copied version.php. Upstream skips the install whenever version.php exists and never retries, so the replacement task served an uninstalled Nextcloud that occ status still reported as healthy, and slots 2 and 3 waited for installed: true until converge gave up after about 25 minutes.
Slot 1, and compose where TASK_SLOT is unset, now exits 1 with the cause and the recovery when version.php and occ exist, the init lock is free and occ status prints installed: false. The task crash-loops and the stack readiness check ends the deploy within minutes with that message in the service log. occ status runs with a 120 s timeout in both branches. Why the lock failed on NFS is not addressed here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
