test_sso.py, test_sso_oidc_plugin.py and test_sso_proxy_consumers.py evict utils.cache.applications, utils.cache and the roles packages from sys.modules to import their plugin against fresh modules, and never put the originals back. A later test module in the same worker then held two copies of utils.cache.applications: plugins.lookup.applications, imported earlier, kept calling the old one, while test_email patched and reset the new one. Its carrier tests counted 0 renders and failed (CI run 40869769501, test_email.py:556 and :570, and the IAM merge gate), depending only on which modules xdist placed in one worker.
_load_module now runs inside mock.patch.dict(sys.modules): the plugin keeps the fresh modules it was built against, and sys.modules returns to its prior state, so the next module sees exactly what it would have without these tests. `pytest test_sso.py test_email.py` failed the two carrier tests before and passes 67/67 with the three SSO modules first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings feature/updater 3966abdb16, including updater-lockdown-probe: the deferred probe host guard (aa88429c25), the per-host post-load queue (70ed8a2e98), the NFS probe noauto (0ece4fcbc4), the E2E prepull gate (ef20262813), the worktree swarm lab subnet fixes, the Tor port window and restart skip, the Prometheus Docker poller, the Stalwart JMAP client cache and the FusionDirectory PHP hardening mount.
Conflict resolutions:
- plugins/lookup/email.py: keep store-i18n's NETWORK_MODE forwarding into the provider view and add updater's provider carrier.
- roles/svc-net-tor/templates/test.env.j2: updater's ONION_PORT_TIMEOUT=130, store-i18n's ONION_SUFFIX.
- roles/svc-swarm-node/README.md: updater's ingress guard wording, without the generated Quick Setup and Credits sections.
- roles/test-e2e-cli/tasks/main.yml: updater's cli_role_id loop variable with store-i18n's async cli.once condition.
- roles/web-app-prometheus/templates/nginx/locations.conf.j2: store-i18n's entity_name filter with updater's container_service lookup.
- tasks/stages/01_constructor.yml: updater's order (svc-swarm-node after the preload context opens) under store-i18n's task name, whose loop also covers svc-cache.
Integration fixes:
- roles/sys-stk-front-proxy: git applied updater's IS_STACK_HOST guard to the "Freeze front proxy context" task, which would have left workers without front_proxy_* facts. main.yml stays as on store-i18n; the guard moves into the per-sibling vhost.yml split, where the probe facts and the queue sit behind IS_STACK_HOST and front_proxy_domain_conf_dst_resolved stays unguarded for the Deploy arguments. deferred_probe.yml takes updater's host guard verbatim.
- roles/svc-net-tor/tasks/00_core.yml: the restart probe tests IS_SWARM_MODE instead of a literal DEPLOYMENT_MODE comparison.
- roles/web-app-stalwart/tasks/jmap_call.yml: the cached client splits into stalwart_jmap_url and stalwart_jmap_user_password, so the exec argv references no fact that carries a credential.
- roles/sys-service-loader/tasks/list_or_shoot.yml: the two direct-shoot tasks share their preload condition in one block.
- tests: the Prometheus wiring test imports entity_name and pins the host resources its renders need; the email carrier tests use an onion-only provider, which on store-i18n is what drops TLS (a dual-stack provider keeps it).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two tests that only answer on a GPU host, the logout catalogue translation check and the complete-catalogs integration test, were gated on accelerated() alone: a contributor with an NVIDIA runtime had no way to keep them out of a local gate while the GPU translation run is paused, so every gate on such a host stayed red on them.
INFINITO_TEST_GPU is a new env key. Its handler derives the default from INFINITO_GPU_COUNT: true where the compose services reserve a GPU, false for 0 or none. custom.env is applied after every handler, so `INFINITO_TEST_GPU=false` there switches the tests off on a GPU host; an exported value wins as well. tests/utils/gpu.py reads the flag for both tests, fails on any value other than true or false, and falls back to the same GPU detection only where no .env is loaded, such as a direct pytest run.
The handler joins gpu_count in utils/env/handlers/infinito/gpu/ (count.py, only_tests.py), because two handlers sharing the gpu prefix are a folder by the prefix-grouping lint. The new module avoids a test* name so unittest discovery never collects it. customize.md lists the key among the common overrides.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every JMAP call templated STALWART_ADMIN_USER and STALWART_ADMIN_PASSWORD (lookup('users')) and STALWART_JMAP_URL (config lookup) again. Ansible forks a fresh worker per task, so get_merged_users re-rendered the whole user catalogue in every call: about 0.4 to 0.6 s of controller time per call, 37 to 39 calls per deploy pass.
jmap_call.yml now sets the host fact stalwart_jmap_client (url, user_password) on the first call of a run, masked like the call itself, and every call reads it. STALWART_CONTAINER_ADDRESS stays in the command, so swarm keeps resolving the running container per call; 11_oidc_restart.yml replaces that container mid-run. The include interface is unchanged: stalwart_jmap_body in, stalwart_jmap out.
A scratch harness with a fake container measured the shell task at 1.01 to 1.10 s before and 0.46 to 0.49 s after, with argv and stdin sha256 identical in compose, swarm, vaulted-password and two-host runs. The admin password only rotates between runs, so a per-run fact stays current.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The nested carrier in tasks/utils/load/app.yml inherited run_once from the apply of sys-stk-full's front-proxy include, so ansible-core wrote the stack host's carrier, keyed with its group_names, to every play host. Workers and the NFS server then missed their own key and re-rendered the whole applications SPOT on the next carrier and on every task that read it. run_once: false keeps each host's carrier.
The email lookup's onion check renders the provider host's group view, a key no carrier on a worker holds, so every lookup('email') there rendered the whole SPOT: 8-11 s per task, 17 tasks per swarm pass in the stalwart roundtrip. Hand the provider host's carrier, read untemplated via hostvars.raw_get, to that view; get_merged_applications still serves it only when its key matches and renders as before otherwise.
Each bug hid the other: fixing either alone only moves the misses. Together they remove about 142 s per swarm pass. With real ansible-playbook and HostVars the worker email task drops from 3.97 s to 0.77 s on ansible-core 2.20.6 and from 4.22 s to 0.70 s on 2.21.5 with the same result; the new tests fail with the reuse disabled.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
c7fb6e60c5 narrowed every matrix row's services to the closure its
variant deploys (matrix.with_deployed_services) before axes.assign
derives the network states and the architecture pool. The selection
validator kept reading the raw discovery row, so it judged pins against
a different closure than the matrix it is meant to protect.
Manual run 40808080548 hit it: since 91022ad37a, svc-ai-litellm#1 no
longer deploys the guide host and with it the Tor provider, so `multi`
is impossible for that row. The validator still reported "Selection
inputs are valid for this branch", and priority_batches.sh then aborted
with "pinned network mode 'multi' is impossible here".
validate.problems now passes the same narrowed row to row_pulls_tor and
row_architectures. On the run's priority line it refuses exactly the
token the matrix refuses, before anything deploys.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two store-i18n-only code paths predate updater's custom roles
(web-app-cst-*, d92375922d), which the merge c9e7f950d9 brought in:
- utils/i18n/libretranslate.py deploys the translation tools lane with
cli.administration.deploy.dedicated but without --skip-pull. The lane
bind-mounts the checkout and its generated inventory declares no custom
role, so `make i18n-translate` would have run pull_custom_roles and
removed every pulled web-app-cst-* from the host's roles/. It now
passes --skip-pull, as every other generated deploy does.
- utils/i18n/extract.py role_messages globbed roles/*/meta/main.yml,
including the gitignored custom roles, so a checkout with a pulled
custom role failed test_core_catalogs_hold_exactly_the_current_sources
and its remedy would have written the custom role's description into
every tracked core catalog, against updater's rule that core never
tracks custom roles. It now skips is_custom(role), like the README
generators.
test_a_pulled_custom_role_stays_out_of_the_core_catalogs pins the second
point; without the skip it fails with the custom role in the list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
updater's Stalwart v1 port (store_guard.yml, migrate.env.j2) arrived in
the merge (c9e7f950d9) with three constructs store-i18n's lints reject:
- `DEPLOYMENT_MODE == 'swarm'` in store_guard.yml and migrate.env.j2
(test_deployment_mode_booleans): now IS_SWARM_MODE;
- `get_entity_name` in the store-engine fail_msg, a filter store-i18n
renamed to entity_name (a05071fe78), so TestAllUsedFiltersAreDefined
failed: now entity_name;
- two pairs of sibling tasks repeating the same `when`
(test_no_repeated_when_runs): the shared-database and dedicated-volume
lookups now sit in one block gated on `not (stalwart_store_marked |
bool)`, the migrate include and the pre-1.0 assert in one block gated
on `stalwart_store_layout is not skipped`; each task keeps its own
remaining condition. Ansible evaluates a block's `when` per task, so
the assert still sees the layout migrate.yml registers again.
updater's test_store_guard.py walked only the top-level tasks and failed
with KeyError: 'register' on the block wrappers. It now walks
block/rescue/always (utils.roles.invocations.NESTED_KEYS), hands a
block's `when` down to its children and checks the block condition
before the task's own, as Ansible does; the assertions are unchanged.
With either block's `when` removed from a copy of store_guard.yml, two
respectively four of its tests fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BudgetKillDrillTests drove the stand-in deploy with `timeout -k GRACE 3`.
Under load (gate pass 76 ran unit, integration and lint in parallel)
timeout's TERM arrived while bash was still starting, before the
stand-in's `trap cleanup EXIT` existed, so bash died on TERM without the
rescue: test_without_the_window_the_kill_lands_before_the_copy failed
with "124 != 137" and only "kill_epoch=…" on stdout. A 4 s stall before
the trap reproduces it on the old driver; nothing in the merge touched
the rescue code.
The stand-in now writes to an `armed` FIFO right after it installs the
trap. The driver runs `timeout -k GRACE 0 …` (duration 0 disables
timeout's own clock, its TERM-then-KILL handling stays), waits for
`armed`, sleeps until the deadline epoch and sends timeout the same ALRM
its own expiry raises, so TERM still lands at the deadline and KILL
GRACE seconds later, with the same 124/137 exit codes. `wait -n -p`
lets the failure-before-the-deadline drill return as soon as its deploy
ends. No timeout was raised and nothing retries; with the 4 s stall all
three drills pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
updater's TestWriteMany (86e7c844f3) looked up the persist, mirror and
hand-over tasks at the top level of write_many.yml and evaluated only
each task's own `when`. The merge (c9e7f950d9) moved those three tasks
into store-i18n's change-gated block ("💾 Write the token store when the
merge changed it", `when: sys_token_store_users_merged !=
sys_token_store_users`) and switched the mirror to IS_SWARM_MODE, so the
lookups raised StopIteration.
The test now walks into block/rescue/always, hands a block's `when` down
to its children and ANDs it with a task's own condition (a task without
any condition counts as true, as in Ansible), renders with
IS_SWARM_MODE, and counts the re-render inside blocks too. The
assertions are unchanged; with the block's `when` removed from a copy of
write_many.yml, "an unchanged store skips the write and the re-render"
fails again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
updater's TestSwarmDeployCommands (d92375922d) patched
utils.tests.swarm.matrix.run_step, which store-i18n's matrix no longer
imports: every step goes through the module's _run(cmd, *, env, label),
which wraps the watchdog. After the merge (c9e7f950d9) both tests failed
with "does not have the attribute 'run_step'".
They now patch matrix._run, as test_run_reporting and
test_extend_inventory already do, and keep their assertions: the round
deploy (cli.administration.deploy.swarm) and the backup-host deploy
(cli.administration.deploy.dedicated) both pass --skip-pull, so the
custom roles placed before the round survive. The backup-host test hands
_deploy_backup_host a temporary inventory directory, because store-i18n
reads and may rewrite host_vars there (drop_backup_network_mode) before
it deploys.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d92375922d (feature/updater) added TestCreditsLicence, which exercises
_validate_credits in tests/lint/ansible/roles/test_readme.py so that a
custom role (web-app-cst-*) may state its own licence in a committed
Credits section. On feature/store-i18n Credits is a generated section
the docs build renders and the README lint forbids committing it, so
the merge (c9e7f950d9) kept store-i18n's test_readme.py without
_validate_credits, and the five tests failed with AttributeError.
The operator decided to remove the tests now and to give custom roles
their own licence statement later as a separate feature. The three
package __init__.py files under tests/unit/python/tests/lint/ only
existed for this module and go with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Semaphore v2.19.17 builds its post-login redirect from its one WebHost
(url.JoinPath(WebHost, return)), marks the session cookie Secure when
WebHost is https, and registered a single OIDC provider whose
redirect_url is clearnet. A login started on the onion sibling therefore
lost the oauthstate cookie at the clearnet callback and the session on
the plain-http onion origin: semaphore#0 of run 40625679270 failed five
login and design specs in PASS 2, where Playwright picked the .onion
sibling ("http: named cookie not present" 24 times in the container
log, while LDAP accepted the rotated credentials six times).
Semaphore keys OIDC providers by id from the URL path and gives each its
own redirect_url, so when the role has an onion sibling and SSO is on:
- a second provider keycloak_onion, same issuer, client and claims, with
the onion sibling's plaintext redirect_url; Keycloak's redirect_uris
lookup already registers http://<onion>/*;
- the onion vhost rewrites the absolute WebHost redirects to relative
ones (proxy_redirect) and drops Secure from the semaphore cookie
(proxy_cookie_flags), next to the existing base-href rewrite;
- location = /api/auth/login filters the provider list per vhost with
sub_filter, so clearnet offers keycloak and the onion vhost
keycloak_onion, both labelled as before; each dropped provider has a
pattern for either list position because Semaphore does not fix the
order.
Without an onion sibling the provider JSON and both proxy hooks render
exactly as before. test_onion_login renders the vars through Ansible's
templar for both cases; 16 of its 18 tests fail against the previous
files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
68 conflicts, resolved per hunk; resolutions agreed with the Updater
session were applied as agreed:
- svc-db-postgres (superuser password script and core) and
sys-svc-rdbms realign_dedicated_db: updater's argv/stdin redesign with
expand_argument_vars: false, nested into store-i18n's block structure.
- svc-db-redis 01_init: store-i18n's ACL password on stdin.
- sys-svc-proxy forwarded.conf.j2: store-i18n's handover-aware file,
without the forwarded_base include (Host and X-Forwarded-Proto would
be sent twice); forwarded_base serves the inline consumers, e.g.
web-app-seaweedfs nginx.conf.j2.
- basic.conf.j2 and web-app-openproject proxy.conf.j2: issuer redirect,
then handover, else updater's location/sso_acl.conf.j2; the
server-level webserver_extra_configuration hook stays.
- web-app-keycloak: updater's admin-REST settings step and renumbered
update steps; update/01_client.yml removed; store-i18n's design step
and LDAP block kept; a73435cc9d arrived through rename detection.
- web-app-nextcloud OpenProject provisioning: updater's rewrite, all
credentials on the Ruby runner's stdin.
- web-app-semaphore: store-i18n's canonical web root (a1d7835863).
- Role READMEs: no generated Cosmos/Architecture/Quick Setup/Credits
sections; desk-* paths resolved to the dsk-* renames.
Integration fixes for what merged cleanly but did not fit together:
- web-app-matrix ansible vars: drop updater's second
matrix_client_element_configuration_extension_json, which silently
overrode store-i18n's design-aware value.
- url.origin instead of a stripped url.base in 19 places store-i18n
added since 44aaa68924, same lookup targets (the double-slash lint).
- Package batch (18ea75049e, cb4d340de9): installs the deploy-start
batch covers are removed (caffeine, cursor, blender, npm in dsk-gnt-*,
with shared references), the rest carry batch: false and a nocheck
with the verified reason (PyPI fallback, before the batch, vendor
repository, configured flavor only, addon gate); dev-npm references
the root npm id.
- sys-token-store write_many.yml: store-i18n's change-gated block and
IS_SWARM_MODE.
- web-app-openproject: IS_COMPOSE_MODE in 02_migrate/run.yml; the
seeded-admin lock (ccbf8f168d) ported into 04_admin/provision.yml and
covered by the admin-state digest.
- Bucket policy (3b4ac40df5 and follow-ups) moved to web-svc-seaweedfs,
where store-i18n's object store lives; the S3 admin secret travels on
stdin instead of a task-level environment.
- act workflow.sh: no Exception-labelled comment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The update started one kcadm JVM per step: the client and both realms' mail settings as async jobs with their reaps, then each declared client in turn, about 27-31 s in a Nextcloud deploy. realm_settings.py reads the client scopes, clients and both realms once over the admin REST API, creates what is missing and writes a client or realm only when it drifted; secrets reach it on stdin through the new rest_script_stdin parameter. The remaining update steps are renumbered and every reference follows.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`make autoformat` rewrites roles/web-app-seaweedfs/vars/main.yml and
tests/unit/python/roles/web-app-stalwart/test_store_guard.py on every
checkout of main, so every quality gate on every branch leaves them as
unstaged changes, and the gate lints a tree the commit does not carry.
The YAML change folds one line of the `>-` scalar
SEAWEEDFS_S3_ADMIN_CURL; the parsed value is identical. The Python
change is ruff's line wrapping only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stalwart v1.0 refuses an existing account with primaryKeyViolation; the batched provisioning from 1fbd4758e3 only knew alreadyExists, so every update deploy skipped the password sync and the listed-id fallback hid the refusal (msmtp then failed with 535). An account now counts as existing when the refusal's objectId is the listed account of that name, any other refusal fails the assert, and the update patches credentials/<k>/secret instead of replacing the whole credentials list, which would have dropped every app password, API key and second factor. sys-token-store gains write_many.yml, so all bot tokens land in one store write and one applications re-render; write.yml delegates a one-entry batch to it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stalwart, Mailu and LDAP provision every bot and account holder of the whole users catalogue, but the generator pinned only the users of the resolved roles. The bots of undeployed roles (no-reply, blackhole, bounce, newsletter, helpdesk, contact) fell back to strong_password, got a new secret on every deploy, rewrote the token store six times per run and left email.password consumers on a stale value. The generator now also pins every catalogue user holding an account or the bot role unless a role states its password outright, and reads policies independent of declaration order; existing pins are never overwritten.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The apt module loads its whole cache before it finds that every name is installed, about 2.4 s per call; an update deploy spent ~28 s on twelve such no-op calls from roles outside the batch. package_install now asks dpkg-query, pacman --query or rpm --query under LC_ALL=C first and returns unchanged only when every plain name is proven to be in the requested state; anything it cannot read in full runs the module as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ruff format reflows the package batch, registry, plan and invocation code and tests and the Stalwart and Nextcloud tests, ansible-lint rewraps long Jinja lines in folded scalars of sys-service, Nextcloud and SeaweedFS, mbake restores the Makefile's final newline, and the Nextcloud OAuth test merges its nested with statements (SIM117).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
select-admin-state.sql joined select-ldap-auth-source-exists.sql under the shared 'select' prefix, which the prefix-grouping lint rejects. Both now live in files/sql/select/ and their three readers point there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The admin step ran its Rails runner on every deploy. It now digests its sources, its inputs and the administrator and settings state OpenProject holds, compares that with the admin-provision.sha256 stamp in the instance directory, and runs only on a mismatch or when services.openproject.admin.force is set.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
db:migrate took about 24 s on every deploy. After a successful run the
deploy stamps the web container's image id as the comment of
public.schema_migrations; a deploy whose image matches the stamp skips the
migration, a new image, a failed run or a recreated database migrates. The
stamp is read unmasked because masking short config values would corrupt a
hex image id, and written through a DO/format() block because sqlfluff cannot
parse COMMENT ON ... IS with a placeholder.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
OpenBao keeps no record of who read which secret without an audit device,
and 2.7 refuses to enable one over the API, so the server config declares a
file device writing to stdout, which cannot block and lands in the container
log with secret values only as salted HMACs. Every deploy fails unless the
device is active, a probe secret read through the AppRole session shows up
in the log with path, operation, display_name and time, and the probe value
does not appear in plaintext. openbao becomes a regular shared provider so
consumers such as the landing page role can declare services.openbao.
Requirement 047; taken over from the feature/openbao-audit worktree.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An inventory group web-app-cst-<name> with applications.<id>.custom: true and
a pinned source {url, version, path} is pulled into roles/ before the play,
read raw (Jinja, Ansible tags and credential URLs refused), placed through a
staging directory under an exclusive lock, and owned by a marker that names
its own directory, so a pull or roles-clean never replaces or removes a
directory it did not place and a failed pull leaves roles/ as it was.
Directory inventories, host lists and non-UTF-8 files behave as Ansible
reads them; generated test deploys pass --skip-pull. make roles-pull,
roles-clean and roles list, INFINITO_EXTRA_MOUNTS for local sources, the
web-app-cst category, the reserved prefix lint and a central
repository_files() that also yields the gitignored custom roles let every
lint scan them like in-tree roles. Supersedes the uncommitted 045 External
Roles draft and reuses its fetch, sync and mount design.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The nfs-ganesha drop-in sets Environment="OPTIONS=-L SYSLOG ...", but the
Debian and Ubuntu unit (upstream nfs-ganesha.service.debian8) also reads
EnvironmentFile=-/etc/default/nfs-ganesha, whose OPTIONS name
-L /var/log/ganesha/ganesha.log, and systemd lets EnvironmentFile=
override Environment=. So on those distros ganesha logged to a file with
no bound, while CentOS, Fedora and Arch already logged to the journal
(SystemMaxUse=256M on the lab node).
Under MODE_DEBUG ganesha.conf raised NFS_V4 to DEBUG since a085427236,
which writes a dozen lines per NFS compound. In odoo#1 of run
40625679270 (debian) the nfs-server node grew 12.6 GB above the control
node and the step aborted at "DISK EXHAUSTION IMMINENT (<6144M free on
/)". In the same run, nodes logging to the journal grew 0.8 to 1.6 GB
in four hours, nodes logging to the file 4.4 to 12.6 GB.
- The drop-in resets EnvironmentFile= before setting OPTIONS. The
Debian unit's ExecStart reads nothing else from that file (its journal
reports EPOCH, GNODEID, NUMACTL and NUMAOPTS as referenced but unset).
- NFS_V4 goes back to the default level: it was raised to chase the
probe-write bug that 9b7f25f2fe fixed. DISPATCH stays at DEBUG, since
nfs_Init_svc(), which binds 2049, logs under it (1aea8a472e).
- The swarm diagnostics read the last 600 journal lines of nfs-ganesha
instead of the file tail.
test_ganesha_log_levels and test_ganesha_unit_options pin the debug
components and the drop-in's OPTIONS source; against the previous
template and drop-in both fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
e5f3c52119 lets a redeploy skip the bootstrap restarts when the running
wiki already serves the pinned core and add-ons. Its probe GETs
/rest/wikis/xwiki/spaces with the superadmin's basic auth and accepted
any 200. XWiki answers a rejected basic auth as guest with 200 (see
b1dfff370a, tasks/03_administrator.yml reads the XWiki-User header for
exactly that reason).
Between PASS 1 and PASS 2 the harness rotates the credentials
("=== matrix-deploy: rotating credentials between passes ===",
xwiki2.log:257549 of run 40625679270). PASS 2 then probed the container
still running with PASS 1's password: XWiki logged "Authentication
failure with login [superadmin]" 34 times, answered as guest with 200,
and the add-on check PUT XWiki.CheckExtension as guest until it died on
401 after 31 attempts (XWiki#2, LDAP variant; the probe only runs with
OIDC or LDAP on).
The core-version read now also requires the XWiki-User response header
to name the superadmin. A guest answer skips it, so the role takes the
bootstrap path that recreates the container with the rotated password
before any write.
test_extensions_ready_probe evaluates the task's when list for a guest
200, a superadmin 200 and a skipped probe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docker_dataroot.sh backs each node's ZFS pool with a sparse loop image
and created the pool without autotrim. ZFS never discarded freed blocks
to the loop device, so the image only grew: in WordPress#0 of run
40625679270 the runner still held 118G after the teardown had removed
every container ("Containers: 0") while only 47G of files remained, and
the pools held ~38G of live data. The step had already aborted at
"DISK EXHAUSTION IMMINENT (<6144M free on /)" with 139G of 145G used
(rc=75). The other ZFS swarm jobs of the run show the same df-minus-du
gap (wordpress1 73G, odoo1 73G, magento0 82G, weblate1 55G, gitlab2
44G); ext4 and btrfs jobs show none.
The pool is now created with -o autotrim=on, so ZFS discards freed
ranges to the loop device, which punches holes into the image file.
test_docker_dataroot runs the script's zfs path against stubs of every
host tool and asserts the single zpool create carries -o autotrim=on;
against the previous script that assertion fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A constructor task after the update role and the swarm auto-placement installs,
in one package-manager call, every package id of every role the host's groups
may run: meta dependencies, service edges and every role invocation readable
from tasks and handlers, conditions ignored. Ids that must stay with their role
(a conditional install, a repository added first, a need before update) carry
batch: false in meta/packages.yml and keep their task; roles reference shared
root ids with shared: true. 42 per-role install tasks go, and a lint keeps the
split: no batched id in a role task, none in a role run ahead of the batch.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The onion port probe gave every SOCKS connect 60s, half of Tor's
SocksTimeout (120s by default; torrc does not set it), although its own
docstring argued that abandoning an attempt before SocksTimeout is
destructive. Every attempt authenticates with fresh credentials, so Tor
builds a whole new rendezvous for it and keeps working on it until
SocksTimeout. The probe walked away first and called the port dead. The
other raw onion connects in the repository, the svc-db-* CLI tests and
test-e2e-playwright's wait_onion_reachable.sh, wait 120s per attempt.
Run 40128751852 shows how often an attempt needs more than that. Of 178
svc-net-tor CLI runs in its 89 compose onion jobs, three lost their
first attempt to the 60s timeout and connected on the second within
seconds (Shopware#1, Hugo#1, n8n#1; the CLI test took 77-79s against
the usual 1-16s), and Hugo#1's other run took 40s on a single attempt. When
the stall outlasts the 70s until the second attempt, the probe fails
while the HTTP probe reaches the same onion port seconds earlier:
OpenBao#1 (run 39166850273, job 118050491920, the original failure of
#1048) and Socialhome#1 (run 40088473634, job 121019758461). In the
Socialhome job tor.log holds no line for either attempt, because Tor
never reached its own verdict.
ONION_PORT_TIMEOUT is now 130: SocksTimeout plus room for Tor's
once-per-second expiry sweep, so an attempt ends with Tor's own answer
(reply 0x06, and a "Giving up" notice in tor.log, which the rescue
artifact captures) instead of the probe's. The second attempt now starts
at 140s and ends at 270s, so the probe rides out every stall the old
budget did, and any stall shorter than about 260s instead of about 120s.
A green run still costs one connect per port, and a refused or unmapped
port still fails at once. A unit test pins the window above Tor's
SocksTimeout and fails as soon as torrc sets its own.
That doubles what a dead port costs, and meta/tests.yml caps the whole
CLI test. The largest compose onion deploy in run 40128751852 probes 19
domains and 5 ports; with everything dead that is 19 x 350 + 5 x 260 +
10 = 7960s, beyond the 7200s that 069df39027 set for the same reason.
9000 keeps the full failure output for up to 22 domains.
Closes#1214
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every Weblate container (weblate, beat, worker) runs collectstatic into
STATIC_ROOT=/app/cache/static on start. /app/cache is the `cache` volume
the three services share; in swarm it lives on NFS and every replica of
every service mounts the same directory. Concurrent collectstatic runs
delete and rewrite each other's hashed files, so the tasks crash-loop:
weblate.1, .2, .3 and worker.1 on both swarm nodes died with
"FileNotFoundError: [Errno 2] No such file or directory:
'/app/cache/static/admin/js/vendor/select2/i18n/az.270c257daf81.js'"
(and logo-32.png, dark_mode.a364bd93cdcc.css, widgets.8f8bf7f533a4.js),
which failed Weblate#1 in runs 40625679270 and 40243434327.
meta/volumes.yml mounts a tmpfs at /app/cache/static into every service
that shares the cache volume, so each container collects and serves its
own copy, consistent with its own manifest; the rest of /app/cache stays
shared. container_volumes renders it as a long-form tmpfs mount for
compose and swarm alike, as it already does for the Matrix bridges.
test_static_root_per_container asserts that every service mounting the
shared cache also gets the tmpfs at the static root.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Listing, reading, uploading and activating the Sieve forward scripts each run
as one request for every forwarding account, with one method call per account
addressed through its accountId by the recovery admin, at most
STALWART_JMAP_MAX_METHOD_CALLS per request, instead of one request per account
under act-as. The account lookup joins the limits read, a missing account
stops before the limits are raised, and only changed scripts are uploaded.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The carrier key switched from the tokens file's path and hash to the
sys_token_store_users fingerprint once the token store loaded, so the
stage-wide park re-rendered the applications SPOT (about 11 s) even when the
tokens were identical. Both sources now sign as a digest over exactly the
tokens the users hydration reads; the store helpers move to users/store.py
to keep the package under the file length cap.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test_role_files_are_referenced_via_spot flagged
test_multisite_usernames.py:13 for spelling the role file path
"roles/web-app-wordpress/meta/users.yml" literally. The test now joins
the role directory with ROLE_FILE_META_USERS from utils.roles.mapping,
which resolves to the same path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a multisite network WP-CLI's `user create` runs
wpmu_validate_user_signup(), which only accepts lowercase letters and
digits. The MCP subscriber was declared as `mcp-wordpress`, so every
variant with `multisite.enabled: true` (variant 1) stopped at "Create the
subscriber the MCP adapter authenticates as" with "Usernames can only
contain lowercase letters (a-z) and numbers." (WordPress#1 in runs
40625679270 and 40243434327).
meta/users.yml now declares `mcpwordpress`. The users dict is the one
place the login lives: the role tasks, the MCP basic-auth login
(mcp_client_login) and the MCP client configs (mcp_servers) all read
users[owner].username, and the owner key, hence the token-store key,
stays mcp-web-app-wordpress. The email follows the login.
On an existing single-site install the next deploy creates mcpwordpress,
mints its application password and replaces the stored token; the old
mcp-wordpress subscriber is left in place with its now untracked "mcp"
application password. Remove it with `wp user delete mcp-wordpress`.
test_multisite_usernames mirrors the multisite signup checks (charset,
reserved names, 4-60 characters, not all digits): every account the role
declares passes, and mcp-wordpress, `_`, `.`, uppercase, too short, all
digits, admin and too long are refused. With the old login the test
fails with WordPress's own message.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
forwarded.conf.j2 includes headers/forwarded_base.conf.j2 by its path from
the repository root, the way Ansible searches templates, so the test's loader
searches the root next to the headers directory.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The guard now includes store_guard/migrate.yml for a store from before 1.0
and re-reads its layout, so the walk feeds that include's result into the
final assert: a migrated store passes, one left before 1.0 still stops.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sphinx's i18n transform parses every msgstr into a temporary document
whose settings carry the whole BuildEnvironment (sphinx/transforms/
i18n.py, Sphinx 9.2). MyST nodes patched into the page keep pointing at
that document, and write_doctree only detaches the environment from the
page's own settings, so every translated Markdown page was pickled with
a full environment copy: a German doctree of drv-lid-switch grew from
21 KB to 5.5 MB, the whole doctree tree from 95 MB to 2.5 GB. The write
phase loads them all back and the forked -j worker copies the parent,
so the `deployed:de` build reached 5+ GB in the parent and was
OOM-killed inside the docs container in CI run 40625679270 (Jenkins#1,
Nextcloud#11), drove shared 16 GB swarm runners into swap and the
watchdog's memory and I/O-wedge aborts (GitLab#2, Nextcloud#7/#9/#10),
and left the daemon answering 500 to the queue task for the failed
build.
AdoptTranslatedNodes runs right after the Locale transform and points
every node whose document is not the page back at the page; conf.py
loads it for every build the image runs. No size, limit or budget
changes, and docs stays deployed wherever a variant declares it.
Measured locally on the full German build at -j 2: parent peak 685 MB
(was 5.3 GB and climbing when stopped at about a quarter of the write
phase), total 2.4 GB (was 8.35 GB), doctrees 92 MB (was 2.5 GB). The
English HTML of all 5,373 pages is byte-identical before and after. A
real German MyST build in the new test finds no foreign nodes and no
BuildEnvironment in the saved page; without the transform it fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2bf329cd3b put rescue_window.sh next to rescue_index.sh, and
test_prefix_grouped_files rejects two siblings sharing the `rescue`
prefix (quality-high pass 67). Both move to utils/rescue/ as index.sh and
window.sh, their tests to the mirrored tests/.../utils/rescue/, and every
caller follows: dedicated.sh, the swarm teardown in 00_one.sh, the
workspace workflow, and the swarm and diagnostics READMEs.
The budget-kill drill gave the stand-in a 1 s budget; under the full
parallel unit run the SIGTERM arrived before its EXIT trap was installed,
so nothing was captured and the test failed. The two kill drills now use
KILL_BUDGET = 3 s; their assertions are unchanged. A full `make test-unit`
passes them under load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
distros.sh stops a distro at its budget with `timeout -k 60`: SIGTERM at
the deadline, SIGKILL 60 s later. dedicated.sh's cleanup trap then ran
the in-container capture under `timeout 1500` (266 s in Baserow#0 of run
40625679270) and the host capture under `timeout 900`, so the SIGKILL
always landed mid-capture, the snapshot was never copied out and the
rescue-diagnostics artifact was lost. The inner `timeout` also sat in its
own process group and outlived the kill, holding the step open.
- distros.sh keeps the grace in one place, KILL_GRACE_SECONDS, and
exports INFINITO_CI_DISTRO_KILL_EPOCH next to the deadline epoch.
- scripts/tests/deploy/utils/rescue_window.sh decides per trap step:
past the deadline, the step gets half of the time left until the
SIGKILL (at least 1 s) and an end epoch; before it, the nominal
timeout and no end epoch. Inside the trap `$?` is 0 on a budget kill,
so the deadline epoch is the only reliable signal.
- dedicated.sh runs both captures through `timeout --foreground` with
that window, keeping them in the trap's process group, and passes the
end epoch as RESCUE_DEADLINE.
- utils/diagnostics/container.py caps every command at RESCUE_DEADLINE,
writes `[skipped: past RESCUE_DEADLINE]` for what no longer fits and
hands the deadline to nested captures. Without the variable nothing
changes, so the swarm and workspace callers behave as before.
-k 60, the distro budget and every timeout-minutes are unchanged. Drill
tests replay the trap with the real helper and container.py: a budget
kill ends with rc 124 and the snapshot copied out, the old shape ends
with rc 137 and nothing copied, an early failure keeps the full capture.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The file was named after the application, so an app served on several domains (SeaweedFS filer/master, Stalwart mail/webmail) overwrote it once per domain on every deploy. The last domain won its dashboard iframe host and Matomo site id for all others, and every deploy reloaded OpenResty for the flip-flop. The file is now <domain>.lua, the naming store-i18n already uses, and its path comes from lookup('nginx', 'files.lua', domain) for writer, reader and the Matomo reinjection. The reinjection stats the live domain's file instead of globbing leftovers.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Takes main's Stalwart v1.0.0 port (server v1.0.0, WebUI v2.0.2, 0.16 store guard). manager_ops.yml keeps the updater's per-account forwards ahead of main's store marker, which stays the last task. The root README drops Akaunting with main.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test_backup_nfs_excludes.py (4fcf097e5b) computed the repository root
with Path(__file__).resolve().parents[6], which
test_project_root_import rejects outside a package __init__.py
(quality-high pass 66). The package now exports PROJECT_ROOT exactly as
its sibling files/__init__.py does, and the test imports it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
svc-bkp-nfs-2-local snapshots the export base (/srv/nfs), so a
generation's files/ root already holds infinito-state/. Without an
explicit target, `recover nfs` and `recover full` restored it onto
/srv/nfs/infinito-state: the tree came out as
infinito-state/infinito-state/<volume>, and `rsync --delete` wiped every
live volume directory and the backup root under the old target.
- paths.NFS_EXPORT_BASE = get_export_base(), the function behind
lookup('nfs_export_base') that also feeds the backup unit's source,
replaces NFS_EXPORT_STATE. NfsRecoverer defaults to it; an explicit
`snapshot:/abs` target still wins, a remote target is still prefixed
with its host. `recover full` inherits the default.
- recover_drill.sh seeds its marker at files/infinito-state/<volume>/
and asserts it under the export base, so the drill can no longer pass
on a nested layout.
- The help examples, schema.yml and the CLI README show the export base;
the documented subtree example would have nested too and now names the
matching snapshot subtree.
Reproduced on a temp tree with the role's real script.sh and recover.py
and the CLI's argv: the old default loses every live volume and the
backup root, the new one restores in place. The new tests fail 6 of 6
against the old default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The NFS backup rsynced the whole export and honoured only one exclude, the
shared backup root. Volumes declared `backup: false` were copied anyway and
then pulled a second time by remote-2-local; in the swarm DR drill of
Magento#0 (run 40625679270) docs_sites alone (a full repo clone plus its
prepared and work trees) took about 15.5 GiB per copy and filled the
runner. The volume backup already skipped these through the backup_volume
lookup; only the NFS path ignored the declaration.
- utils/storage/nfs.py: volume_dir() names the state directory a volume
binds to (pinned name or key), shared with compose_volumes;
backup_disabled_dirs() returns every NFS-backed `backup: false` volume
dir that no backed-up volume also binds to.
- The role-local lookup backup_nfs_excludes feeds them into
BKP_NFS_2_LOCAL_EXCLUDES next to the backup root; script.sh takes any
number of excludes. Compose installs still pass only the backup root.
- recover.py protects every live state directory the snapshot does not
hold, so `rsync --delete` keeps the excluded volumes (model weights,
the docs tree, caches) and volumes created after the snapshot instead
of wiping them.
- The swarm DR drill restores BACKUP_NFS_VOLUME, the first NFS volume
not declared `backup: false`; PRIMARY_NFS_VOLUME stays for the drain
tests. BookWyrm now drills bookwyrm_media; decidim, fusiondirectory,
meta, coturn, svc-registry-cache and svc-registry-docker have nothing
backed up on NFS and skip the drill.
Measured: compose_volumes renders identically before and after for 291
roles across compose/swarm x local/nfs (1940 of 1940); the 15 excluded
directories are exactly the backup-disabled device dirs; the systemd unit
renders shell-safe on ansible-core 2.20 and 2.21 and stays byte-identical
in compose and swarm-local. A new unit test runs the real script.sh over
two generations and the real recover.py back into the export, and fails
without the restore protection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lookup('tls', app, 'url.base') always ends in '/', and 39 call sites in 25 files rebuilt the slash-less form by hand with regex_replace('/$', ''), regex_replace('/+$', ''), trim('/') or .rstrip('/'), directly or on a variable holding url.base. The lookup now returns that value as url.origin (scheme://host, url.base minus its final '/'), and every call site reads it; two sites that stripped the slash only to append it again now take url.base as is, and GITLAB_URL and DASHBOARD_JS_CDN_URL lost their last consumer.
test_url_base_no_double_slash gains the rule url-base-strip, so url.origin stays the only way to the slash-less URL; it flags all nine HEAD sites probed and none after the change. The same lint keeps url-base-double-slash and url-base-addon-value, which put the nine Nextcloud integration URLs, the Mattermost and SuiteCRM facts and the XWiki instance on the slash-less form, drop the stray '/' before Joomla's index.php callback, and give BBB its api.url as [base, suffix] | url_join plus the trailing '/' its client appends 'api/<call>' to; bbb.spec.js now requires exactly that form.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Job and run titles read `🐳🧅…𓏴𓏛·Nextcloud#2`; the `·` between the
hieroglyph block and the title only costs width. RoleDisplayName.encode
now joins the two directly (`𓏴𓏛Nextcloud#2`). Spaces inside a README
heading still become `·`, which keeps a display name one token in the
space-separated whitelist and priority inputs.
decode and decode_list still map the dotted form of older runs back to
the role id, so report_failures, the run tooling and selection tokens
pasted from an earlier run keep working. Old and new names cannot
collide: a glyph block holds only hieroglyphs and a title starts with a
letter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Connecting a user to OpenProject failed: the instance URL carried url.base's trailing slash, so Nextcloud redeemed the code at //oauth/token (HTTP 500), integration_openproject stayed in its fresh-setup state so isAdminConfigOk was false and no Connect button rendered, and duplicate OAuth clients piled up across deploys. The coupling now converges on each deploy: a PHP helper creates or repairs the Nextcloud OAuth2 client (numeric id, OpenProject callback, same-name duplicates removed); one rails runner call ensures the storage, its OAuth application and the Nextcloud client on it, renewing the OpenProject secret when Nextcloud holds none or a stale one, or when rotate_oauth_client forces it, and realigning the application's redirect_uri after a host change; nc_oauth_client_id is stored only after that call succeeded. openproject_instance_url has no trailing slash and fresh_project_folder_setup is 0.
No secret travels in argv or environment: on the way to OpenProject they ride the same stdin as the Ruby script, as a PROVISION_INPUT heredoc, and the credentials go into Nextcloud through occ lines on the container shell's stdin, taken verbatim from 7f72e53a58 on feature/store-i18n. A Ruby truth table over seven scenarios, with the staged state as negative control, covers fresh install, rerun, a Nextcloud that lost client and secret, a stale secret, forced rotation and a host change. A new Playwright test connects a user through the OAuth round trip and expects Nextcloud's oauth-redirect below 500 and 'Connected as'.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
