Every JMAP call templated STALWART_ADMIN_USER and STALWART_ADMIN_PASSWORD (lookup('users')) and STALWART_JMAP_URL (config lookup) again. Ansible forks a fresh worker per task, so get_merged_users re-rendered the whole user catalogue in every call: about 0.4 to 0.6 s of controller time per call, 37 to 39 calls per deploy pass.
jmap_call.yml now sets the host fact stalwart_jmap_client (url, user_password) on the first call of a run, masked like the call itself, and every call reads it. STALWART_CONTAINER_ADDRESS stays in the command, so swarm keeps resolving the running container per call; 11_oidc_restart.yml replaces that container mid-run. The include interface is unchanged: stalwart_jmap_body in, stalwart_jmap out.
A scratch harness with a fake container measured the shell task at 1.01 to 1.10 s before and 0.46 to 0.49 s after, with argv and stdin sha256 identical in compose, swarm, vaulted-password and two-host runs. The admin password only rotates between runs, so a per-run fact stays current.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The update started one kcadm JVM per step: the client and both realms' mail settings as async jobs with their reaps, then each declared client in turn, about 27-31 s in a Nextcloud deploy. realm_settings.py reads the client scopes, clients and both realms once over the admin REST API, creates what is missing and writes a client or realm only when it drifted; secrets reach it on stdin through the new rest_script_stdin parameter. The remaining update steps are renumbered and every reference follows.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`make autoformat` rewrites roles/web-app-seaweedfs/vars/main.yml and
tests/unit/python/roles/web-app-stalwart/test_store_guard.py on every
checkout of main, so every quality gate on every branch leaves them as
unstaged changes, and the gate lints a tree the commit does not carry.
The YAML change folds one line of the `>-` scalar
SEAWEEDFS_S3_ADMIN_CURL; the parsed value is identical. The Python
change is ruff's line wrapping only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stalwart v1.0 refuses an existing account with primaryKeyViolation; the batched provisioning from 1fbd4758e3 only knew alreadyExists, so every update deploy skipped the password sync and the listed-id fallback hid the refusal (msmtp then failed with 535). An account now counts as existing when the refusal's objectId is the listed account of that name, any other refusal fails the assert, and the update patches credentials/<k>/secret instead of replacing the whole credentials list, which would have dropped every app password, API key and second factor. sys-token-store gains write_many.yml, so all bot tokens land in one store write and one applications re-render; write.yml delegates a one-entry batch to it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ruff format reflows the package batch, registry, plan and invocation code and tests and the Stalwart and Nextcloud tests, ansible-lint rewraps long Jinja lines in folded scalars of sys-service, Nextcloud and SeaweedFS, mbake restores the Makefile's final newline, and the Nextcloud OAuth test merges its nested with statements (SIM117).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
select-admin-state.sql joined select-ldap-auth-source-exists.sql under the shared 'select' prefix, which the prefix-grouping lint rejects. Both now live in files/sql/select/ and their three readers point there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The admin step ran its Rails runner on every deploy. It now digests its sources, its inputs and the administrator and settings state OpenProject holds, compares that with the admin-provision.sha256 stamp in the instance directory, and runs only on a mismatch or when services.openproject.admin.force is set.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
db:migrate took about 24 s on every deploy. After a successful run the
deploy stamps the web container's image id as the comment of
public.schema_migrations; a deploy whose image matches the stamp skips the
migration, a new image, a failed run or a recreated database migrates. The
stamp is read unmasked because masking short config values would corrupt a
hex image id, and written through a DO/format() block because sqlfluff cannot
parse COMMENT ON ... IS with a placeholder.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
OpenBao keeps no record of who read which secret without an audit device,
and 2.7 refuses to enable one over the API, so the server config declares a
file device writing to stdout, which cannot block and lands in the container
log with secret values only as salted HMACs. Every deploy fails unless the
device is active, a probe secret read through the AppRole session shows up
in the log with path, operation, display_name and time, and the probe value
does not appear in plaintext. openbao becomes a regular shared provider so
consumers such as the landing page role can declare services.openbao.
Requirement 047; taken over from the feature/openbao-audit worktree.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The onion port probe gave every SOCKS connect 60s, half of Tor's
SocksTimeout (120s by default; torrc does not set it), although its own
docstring argued that abandoning an attempt before SocksTimeout is
destructive. Every attempt authenticates with fresh credentials, so Tor
builds a whole new rendezvous for it and keeps working on it until
SocksTimeout. The probe walked away first and called the port dead. The
other raw onion connects in the repository, the svc-db-* CLI tests and
test-e2e-playwright's wait_onion_reachable.sh, wait 120s per attempt.
Run 40128751852 shows how often an attempt needs more than that. Of 178
svc-net-tor CLI runs in its 89 compose onion jobs, three lost their
first attempt to the 60s timeout and connected on the second within
seconds (Shopware#1, Hugo#1, n8n#1; the CLI test took 77-79s against
the usual 1-16s), and Hugo#1's other run took 40s on a single attempt. When
the stall outlasts the 70s until the second attempt, the probe fails
while the HTTP probe reaches the same onion port seconds earlier:
OpenBao#1 (run 39166850273, job 118050491920, the original failure of
#1048) and Socialhome#1 (run 40088473634, job 121019758461). In the
Socialhome job tor.log holds no line for either attempt, because Tor
never reached its own verdict.
ONION_PORT_TIMEOUT is now 130: SocksTimeout plus room for Tor's
once-per-second expiry sweep, so an attempt ends with Tor's own answer
(reply 0x06, and a "Giving up" notice in tor.log, which the rescue
artifact captures) instead of the probe's. The second attempt now starts
at 140s and ends at 270s, so the probe rides out every stall the old
budget did, and any stall shorter than about 260s instead of about 120s.
A green run still costs one connect per port, and a refused or unmapped
port still fails at once. A unit test pins the window above Tor's
SocksTimeout and fails as soon as torrc sets its own.
That doubles what a dead port costs, and meta/tests.yml caps the whole
CLI test. The largest compose onion deploy in run 40128751852 probes 19
domains and 5 ports; with everything dead that is 19 x 350 + 5 x 260 +
10 = 7960s, beyond the 7200s that 069df39027 set for the same reason.
9000 keeps the full failure output for up to 22 domains.
Closes#1214
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Listing, reading, uploading and activating the Sieve forward scripts each run
as one request for every forwarding account, with one method call per account
addressed through its accountId by the recovery admin, at most
STALWART_JMAP_MAX_METHOD_CALLS per request, instead of one request per account
under act-as. The account lookup joins the limits read, a missing account
stops before the limits are raised, and only changed scripts are uploaded.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
forwarded.conf.j2 includes headers/forwarded_base.conf.j2 by its path from
the repository root, the way Ansible searches templates, so the test's loader
searches the root next to the headers directory.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The guard now includes store_guard/migrate.yml for a store from before 1.0
and re-reads its layout, so the walk feeds that include's result into the
final assert: a migrated store passes, one left before 1.0 still stops.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Takes main's Stalwart v1.0.0 port (server v1.0.0, WebUI v2.0.2, 0.16 store guard). manager_ops.yml keeps the updater's per-account forwards ahead of main's store marker, which stays the last task. The root README drops Akaunting with main.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
lookup('tls', app, 'url.base') always ends in '/', and 39 call sites in 25 files rebuilt the slash-less form by hand with regex_replace('/$', ''), regex_replace('/+$', ''), trim('/') or .rstrip('/'), directly or on a variable holding url.base. The lookup now returns that value as url.origin (scheme://host, url.base minus its final '/'), and every call site reads it; two sites that stripped the slash only to append it again now take url.base as is, and GITLAB_URL and DASHBOARD_JS_CDN_URL lost their last consumer.
test_url_base_no_double_slash gains the rule url-base-strip, so url.origin stays the only way to the slash-less URL; it flags all nine HEAD sites probed and none after the change. The same lint keeps url-base-double-slash and url-base-addon-value, which put the nine Nextcloud integration URLs, the Mattermost and SuiteCRM facts and the XWiki instance on the slash-less form, drop the stray '/' before Joomla's index.php callback, and give BBB its api.url as [base, suffix] | url_join plus the trailing '/' its client appends 'api/<call>' to; bbb.spec.js now requires exactly that form.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Connecting a user to OpenProject failed: the instance URL carried url.base's trailing slash, so Nextcloud redeemed the code at //oauth/token (HTTP 500), integration_openproject stayed in its fresh-setup state so isAdminConfigOk was false and no Connect button rendered, and duplicate OAuth clients piled up across deploys. The coupling now converges on each deploy: a PHP helper creates or repairs the Nextcloud OAuth2 client (numeric id, OpenProject callback, same-name duplicates removed); one rails runner call ensures the storage, its OAuth application and the Nextcloud client on it, renewing the OpenProject secret when Nextcloud holds none or a stale one, or when rotate_oauth_client forces it, and realigning the application's redirect_uri after a host change; nc_oauth_client_id is stored only after that call succeeded. openproject_instance_url has no trailing slash and fresh_project_folder_setup is 0.
No secret travels in argv or environment: on the way to OpenProject they ride the same stdin as the Ruby script, as a PROVISION_INPUT heredoc, and the credentials go into Nextcloud through occ lines on the container shell's stdin, taken verbatim from 7f72e53a58 on feature/store-i18n. A Ruby truth table over seven scenarios, with the staged state as negative control, covers fresh install, rerun, a Nextcloud that lost client and secret, a stale secret, forced rotation and a host change. A new Playwright test connects a user through the OAuth round trip and expects Nextcloud's oauth-redirect below 500 and 'Connected as'.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
render_override takes a service's entrypoint and command from the
output of `docker compose config`, which already prints a literal `$`
as `$$`, and the image's Entrypoint and Cmd from `docker image inspect`,
which are raw. It escaped the combined argv once more, so a service
command reading a variable at runtime (`$$REDIS_PASSWORD` in
compose.yml) reached the override as `$$$$` and the container as `$$`,
which the shell expands to its PID. redis then required
"<pid>REDIS_PASSWORD" and every authenticated ping failed with
WRONGPASS.
Only the image-derived parts are escaped now; the service's own argv
is written as compose config printed it. A new test keeps a service
command's `$$` intact next to an image entrypoint; the existing tests
still require an image Cmd's `$VAR` to become `$$VAR`.
Local Nextcloud update deploy: the redis override holds `$$`, the
container Cmd `$REDIS_PASSWORD`, and the authenticated ping answers
PONG on the first attempt.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
automtu searched the path MTU between its payload bounds, and a lost
echo during that search moved the result between 1400 and 1398 from one
deploy to the next. Every move changed the daemon MTU, rewrote
daemon.json and restarted dockerd.
The probe window is now a single payload: the smaller of 1500 and the
egress MTU automtu reads itself (`--print-mtu egress`), minus the 28
bytes of IP and ICMP header, with `--pmtu-policy max` so one target
answering at that size decides. registry-1.docker.io leaves the
targets because it does not answer ICMP echo. The assert now names both
the egress and the effective value.
Local repeat deploy: the probe took 0.83 s and no handler restarted
dockerd.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Handlers run in definition order, and "Generate CA trust override" was
defined before "Build compose". On a first deploy the override read the
Entrypoint and Cmd of every service image with docker image inspect
before compose build had created the locally built tags, found none,
and left those services without the with-ca-trust wrapper; on later
deploys it read the previous build's Entrypoint and Cmd instead of the
new one. The override and the compose config validation now run after
the build and rebuild handlers and before compose up; pull stays first.
The swarm handlers already build before they generate their override
and use no wrapper, so they are unchanged.
TestFirstDeployFlush replays the shell handlers of one compose-build
plus compose-up flush in the order handlers/compose.yml defines them
and asserts that compose up starts the freshly built image behind the
wrapper.
Local deploy: all six stacks with a built image carry the wrapper in
compose.ca.override.yml, nextcloud for 8 of 8 services.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
recognize's app:install downloads about 700 MB of models and held the
plugin phase for minutes. An addon may now set
config.background_install; the plan files such a plugin under the new
background list unless it is pinned through archive or mandatory.
01_background.yml starts occ app:install as an async job and holds
every later occ call at a gate until the app's schema is applied:
Nextcloud migrations diff a schema snapshot, so DDL running beside
another occ call can be reverted. The gate reads the async job file
and files/php/utils/migration_state.php, which prints the versions
oc_migrations records and the versions the app's migration folder
holds. occ migrations:status is not usable, Nextcloud registers it only
with debug on. While the app directory does not exist yet the
MigrationService constructor throws and lib/base.php's ErrorHandler
only logs it, so the probe exits 0 with nothing on stdout and the gate
waits.
The install loop then runs while the job finishes; the always section
joins every job with async_status and fails on an exit code that a
timeout or signal produced, because the php process may then still run
inside the container. 02_install.yml afterwards answers "already
installed" and enables and configures the plugin, or installs it with
its own retries when the job failed before placing the app.
The stage files are renumbered (01_install -> 02_install, 02_enable ->
03_enable, 03_configure -> 04_configure) so the numbers follow the
include order the task-numbering lint checks.
Local perf run: plugin phase 538 s -> 282 s, gate open after 30.2 s,
recognize 12.0.2 enabled with 8/8 migrations, failed=0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- test_prefix_grouped_files: files/sql/store_database.sql and
store_layout.sql become files/sql/store/database.sql and layout.sql.
- test_file_spot: test_store_guard.py builds the vars path with
ROLE_FILE_VARS_MAIN instead of a hard-coded "vars" / "main.yml" chain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodeQL flagged five notes on PR #1215: four test modules imported unittest both as a module and through from-import, and sys-front-inj-matomo's __init__ exported PROJECT_ROOT without declaring it, so the import from test_site_cache.py read as unused. The modules now bind mock through import unittest.mock as mock, and the package declares __all__, the same fixes main carries from 3610e7b9ef and 6e776a5780. The same pattern is fixed in the new Talk direct-media test and the decommission integration package, and the coturn test package drops an __init__ export nothing imports.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
janus_conf is mounted only when direct media is on, which requires compose mode, and compose ignores a config mode outside swarm and warns about it on every deploy. The file keeps the 0644 it is rendered with on the host.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Removing a role from the inventory left its compose stack running under /opt/compose, holding ports, memory and a stale instance directory, until someone stopped it by hand.
On a full compose deploy with MODE_CLEANUP (no --id), the destructor now takes every directory below DIR_COMPOSITIONS that is named after a known role, carries the full instance layout and is owned by no active role (inventory-wide groups, group_names, the dependency closure and the instances this play prepared), stops it with compose down --remove-orphans without -v, verifies through the project and working_dir labels that no container is left, and only then deletes the directory; named volumes stay. Hand-made, partial or backup directories and instances a non-compose container is named after are left in place and listed. The step is the last block of the destructor, with the timer unlock in its always section, so a failure cannot leave timers locked.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The aio-talk image runs Janus with its stock janus.jcfg: no rtp_port_range and no nat_1_1_mapping, so Janus offered only its bridge address and every stream of a call went through coturn as relay. Calls above about six participants stuttered and failed.
When talk runs bridged next to standalone coturn in compose mode on a host that owns networks.internet.ip4, a rendered janus.jcfg is mounted over the stock one with rtp_port_range 61000-61999, nat_1_1_mapping = ip4 and keep_private_host, and that UDP range is published on ip4. TURN stays the fallback. Before rendering, the deploy reads the image's launcher and stock janus.jcfg and stops when Janus no longer reads the mounted path or the stock settings drifted from the template. Variant 0 publishes 64 ports so dev and CI exercise the path; the call capacity spec asserts a non-relay selected pair on ip4 in that range. With the flag off every rendered file is byte-identical to before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With host networking and no --relay-ip, coturn put every non-loopback IPv4 of the host into its relay list, Docker bridge gateways included, and handed IPv6 and bridge clients relays on those addresses while advertising the public ip4, so their media went nowhere. It also listened on every bridge address.
In compose mode with host networking, coturn now listens on networks.internet.ip4, 127.0.0.1 for the healthcheck and networks.internet.ip6 when set, and relays on ip4. It pins only when the host owns ip4, and owns ip6 too or leaves it unset; behind 1:1 NAT, without the ip binary or with a foreign ip6 it renders the previous command unchanged, because coturn exits when it cannot bind a listening address. A unit test renders the real template and vars for every cell.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Follow-up to the v1.0.0 port, closing holes a review and a local run found:
- WebUI sign-in: WebUI v2 sends an http:// client-metadata-document id
on the onion origin, which Stalwart v1.0.0 rejects (unknown http ids
get no CIMD fallback, trusted hosts require TLS). The recovery admin
under SSO hit the same wall with the bare id 'stalwart-webui'. Register
a public first-party x:OAuthClient 'stalwart-webui' with the WebUI's
callback and login URIs on the served origin, and always inject it as
the Application's oauthClientId. Native sign-in now shows Stalwart's
consent step; test-login-native.js clicks it and also runs under SSO.
- Store guard: the previous guard only checked a running 0.16 server; a
local run showed v1 writing into a 0.16 store whose server was stopped.
The guard now reads the store whenever one exists (shared: pg_database,
dedicated: volume) and fails on the 0.16 bigint layout. A host marker
written at the end of a successful v1 deploy lets restores and
redeploys with a stopped dedicated database pass; without the marker,
a stopped dedicated database fails with an explicit message instead of
a masked psql error. Queries retry while the database recovers.
- README: WebUI sign-in, guard semantics, restore and redeploy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stalwart v1.0.0 rejects unknown JMAP properties ("Invalid property" ->
notCreated/notUpdated), validates OIDC tokens more strictly and refuses to
open a data store from before 1.0. Port the role and protect existing
installs:
- Milter: hostname/timeoutConnect/timeoutCommand/timeoutData become
host/connectTimeout/commandTimeout/dataTimeout; the existing milter is
looked up by host.
- Onion relay route: address becomes host.
- OIDC directory: requireAudience is gone. Set skipAudienceValidation and
unscopedTokens=interoperable on create and update, so Keycloak tokens
(no Stalwart audience, no mail scope) keep working for the WebUI and
Roundcube XOAUTH2.
- WebUI: send oauthClientId only with SSO enabled; without SSO the WebUI
falls back to its own client metadata document.
- Store guard: before anything changes, a running Stalwart whose table u
still has the 0.16 bigint layout stops the deploy and points to the
stalwart-migrate steps, which the README now documents under
"Upgrading from 0.16".
- Updater: pin the WebUI git_tags source to v2.* to match server 1.x.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
csp-checker 3.2.0 checks every URL in its own browser context and takes --concurrency for a worker pool. The role pins 3.2.0, sets HEALTH_CSP_CONCURRENCY to 4, and the systemd unit hands it through script.py to the container command ahead of the -- URL separator, for the clearnet and the onion batch alike. TimeoutStartSec keeps its sequential sizing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test_include_uniform_apply_keyword flagged legacy_checkout.yml after 11c2d8dcd3 moved its guard onto the include: both tasks repeated become: true. The include in 03_launcher.yml now applies become to the file and keeps the guard; the order test reads include_tasks in its mapping form too.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each plugin ran its own occ config:app:set per configuration value right after its install or enable, and a redeploy rewrote every value. Install and enable now queue the plugin; the configure stage runs once for the queue: it loads every queued addon spec, reads occ config:list once, keeps only the values Nextcloud does not hold exactly (nextcloud_plugin_config_pending; typed, masked and structured values are always kept), and runs the hooks in queue order afterwards.
Combined with store-i18n's 7f72e53a58: the pending occ config:app:set calls go on the stdin of one container exec -i ... sh -s, so no value (Talk, OnlyOffice, whiteboard, BBB secrets) reaches a host argv; a failing call names its appid/configkey.
Replay harness old vs new: identical final state, updated keys and hook order in first-deploy, converged, drift and failure scenarios (failure stops at the same key); converged config:app:set calls 19 -> 3. Merge note: resolve 03_configure.yml with this side against 7f72e53a58.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every app's compose pass ran three remote tasks before compose up: create the directories, compose ps for running containers, and stat the pending marker (53 apps x ~2.7 s in a production run). compose_instance_prepare does all three in one call from 02_stack_host_prep.yml, sharing the directory creation with directory_batch through utils/directory_batch.py; compose ps runs only in compose mode. up.yml evaluates the registered result, named compose_instance_before_files because it predates 04_files: the running check is skipped whenever the compose, env or override file changed in this run (those changes notify compose-up themselves).
Replay harness old vs new: rendered files and the compose-up, marker and swarm handlers identical in every scenario; the only difference is the skipped running check when only the override changed, where compose-up still runs. About 85 s per full production run.
Merge note: store-i18n's a4c7fda974 hoists the removed marker stat into a block; resolve up.yml with this side.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
DISCOURSE_REPOSITORY_DIR appended the repository URL to the services directory, so the launcher lived under services/https:/github.com/discourse/discourse_docker.git/. It now lives under services/discourse_docker. Before the clone, launcher/legacy_checkout.yml removes the URL-named tree (computed with relpath against the services directory) and prunes its emptied parents; a guard skips the cleanup when that path would escape the services directory or equal the new checkout. The cleanup must precede the clone: git otherwise resolves the old relative path as a URL and aborts with 'protocol ... is not supported'. Discourse data stays in the launcher's named volume and /var/discourse/shared; the first deploy rebuilds once because the bootstrap digest lived in the old containers/ directory.
relpath joins the builtin filters test_defined knows.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every redeploy ran occ maintenance:repair --include-expensive (43.5 s, maintenance mode on) and listed the appstore updates through Tor (49.7 s) although the version had not changed. 03_upgrade.yml now reads occ status and files/mimetype_version after occ upgrade; nextcloud_needs_post_upgrade decides whether the pass is due (version changed in this deploy, or the expensive repair has not run for the running version). A due deploy includes 03_upgrade/01_post_upgrade.yml with today's app update and repair; an unchanged one includes 03_upgrade/02_cached_updates.yml, which updates only the unpinned enabled apps whose release stored by the updatenotification background job is newer (nextcloud_cached_app_updates, PHP version_compare semantics). NEXTCLOUD_PLUGINS_PINNED is the single source of the pinned check.
Verified against echoserver's live occ output (status, marker, updatenotification cache, app:list, with debug log lines): unchanged redeploy is not due and finds no update. Scenario harness 24/24 incl. negative controls on the old file. About 86 s per unchanged redeploy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each vhost used to run its own sequential reachability probe. The role now queues the vhost and deferred_probe.yml checks the whole queue with one uri_probe_batch call after the shared-service preload and after the server roles. A vhost answering like a stale config restarts OpenResty once; one still unserved afterwards fails the deploy by name. uri_probe_batch takes follow_redirects explicitly so the probe sees the redirect status itself.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
5b7284c4a4 named its reset task with 81 characters, one over the limit of
test_name_emoji_and_length, and 754d35c4de built the path to
roles/web-app-keycloak/vars/main.yml with a pathlib chain that test_file_spot
requires to come from ROLE_FILE_VARS_MAIN. Both commits went in without the
lint suite. The task keeps its behaviour under a 70-character name, the test
reads the same file through the constant. make test-lint: 577 passed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SAML clients were created only by the realm import, and the import runs only
for a missing realm. echoserver keeps its legacy realm cymais.cloud and ran
SuiteCRM behind oauth2-proxy before v16, so the SuiteCRM SAML client
https://<suitecrm>/saml/metadata never existed there: the SAML hop showed
Keycloak's Invalid Request page and Keycloak logged LOGIN_ERROR
error=client_not_found reason=Cannot_match_source_hash.
The existing-realm reconcile now also loops over the clientId of every SAML
client in the rendered realm dictionary, the same source the import uses,
and creates or updates it through kcadm like the application-declared OIDC
clients. The same path already converged the stalwart-webmail and
stalwart-webui clients in echoserver full-10.
Same change as 2b4db90576 on feature/store-i18n, so a later merge of the two
lines resolves without a conflict, plus a unit test that pins the SAML client
selection and its place in the reconcile loop.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On a host whose Matomo was not deployed yet, every vhost injected before
web-app-matomo asked SitesManager.getSitesWithAtLeastViewAccess again and
waited out uri_retry (30 retries x 2 s): 6 vhosts, 84 to 98 s each, 8.7
minutes in galaxyserver full-2. matomo_site_cache only kept a loaded list.
inject.yml now caches every executed load: the list on success, none on a
failed request or an error mapping. A cache of none skips the cached-id step
and getSitesIdFromSiteUrl, so the existing skip task turns the injection off
for the vhost as before, without the retry wait. web-app-matomo resets the
cache of every play host (delegate_facts, swarm workers included) after
07_config and before 08_reinject_consumers, so the re-injection and every
later vhost load the list from the now running Matomo.
Evidence: the real inject.yml with the real uri_retry against a closed port
and a stub Matomo, 11 vhosts in 4 scenarios: the first probe took 68.6 s, the
next two 0.4 and 0.3 s with matomo off; after the reset a known site came
from the cache, an unknown one went getSitesIdFromSiteUrl -> addSite and was
appended, a later vhost of that domain hit the cache. HEAD inject.yml took
68 s per vhost with the same outcome. The swarm reset reached a second host.
ansible-lint passes; tests/unit/python/roles/sys-front-inj-matomo pins the
gates and the reset order.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The reconcile task retried every failure thirty times ten seconds apart, so a permission the key lacks surfaced only after five minutes: rdr-3 spent 331 s on a 403 that no retry could change. The reconciler now ends with exit code 3 when Flowise answers 401 or 403, and the task stops retrying on that code while its failed_when still fails the run under MCP_RECONCILE_STRICT.
Only 401 and 403 count as final: authorize may answer a provider that is still starting with another 4xx, and that wait is what the retries are for. Every exit that reports an HTTP status goes through fail(), which keeps the message on the exception, so the existing tests read it unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test_mint_mcp_api_key.py computed PROJECT_ROOT itself from Path(__file__).parents[6], which test_project_root_import forbids outside an __init__.py. The files package now exports PROJECT_ROOT the way its reconcile subpackage does, and the test imports it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The reconciler prunes the managed fixture chatflows of providers it no longer probes, but the scoped API key it gets was minted without chatflows:delete, so the first prune ended in 403 Forbidden after thirty retries; on echoserver rdr-3 stopped on 'FAILED deleting stale infinito:fixture:web-app-prometheus'. Adding the permission alone would not have reached that host, because api_key() reused an existing key unchanged.
KEY_PERMISSIONS now carries chatflows:delete, and api_key() reuses the managed key only while its permissions equal KEY_PERMISSIONS; otherwise it deletes the key through DELETE /api/v1/apikey/<id> and mints a new one, which the reconciler receives in the same run. The script gains its mirrored unit test, so its mirrored-unit-test exemption goes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every addon ran through the install, enable and configure stages on its own, and each stage asked occ for the plugin's state again. The role now reads occ app:list --output=json once, and the new nextcloud_plugin_plan filter replays the addon order (an enabled entry switches its incompatible_plugins off and itself on, a disabled entry switches itself off, the later entry wins) and splits the app ids into disable, install, enable and configure. One app:disable call takes every id that has to go off; the install, enable and configure includes loop only over the ids that need that stage, so an already enabled plugin skips straight to its configuration. A plugin shipped as a pinned archive still goes through install on every run.
On echoserver the plugin phase took 608 s in full-9 for 51 addons; a repeat deploy is derived at about 195 s instead of about 550 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
quality-high's autoformat pass rewrote three files that 9301220d05,
49aa920034 and c78d5f906c committed without the hooks. ruff drops three
`noqa: E402` markers as unused (RUF100): E402 does not fire on an import
that follows sys.path.insert. The formatter wraps the long
KEYCLOAK_RBAC_GROUP_LOGIC_DIGEST line and the OpenProject admin runner's
`sh -c` line; the double-quoted scalar and the `>-` block both fold the
break back into a space, and a parse against HEAD gives equal values.
On merge: feature/store-i18n 2bcf59109e carries the same four Keycloak
hunks byte for byte; they merge without a conflict.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The emailVerified update looped over every non-reserved user, uploaded a
shell script per user and ran kcadm with a JVM login each time, and it
always reported changed: 7 users took 23.8 s on echoserver.
email_verified.py uses the shared keycloak_admin client through
tasks/utils/rest_script.yml: it looks each user up by exact username,
skips users already verified, and writes emailVerified with a GET plus a
full-representation PUT, as kcadm update did. The task reports changed
only when a user was updated, and a failed lookup now fails the task
instead of being treated as a missing user. The old lookup piped
kcadm -q username=X into head -n1, which could have hit a longer username
sharing the prefix.
Evidence (local compose, slot 104): summary updated=0 unchanged=8
missing=0 failed=0 in 0.60 s, task ok, staging dir removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The per-app RBAC groups and LDAP group mappers were reconciled by four
kcadm shell scripts that started a JVM per call and re-synced every app
on every deploy: 582 s for the mappers plus 324 s for the orphan purge on
echoserver (39 apps).
rbac_reconcile.py now talks to the admin REST API from inside the
Keycloak network namespace (nsenter --net) with the admin credentials
read from /proc/<pid>/environ, so nothing secret reaches argv or output.
It drops the shared ldap-roles mapper, purges root groups of undeployed
apps, ensures each app group and mapper, and syncs a mapper only when the
group is new, the mapper config drifted, or the fingerprint stored on the
group (mapper payload, app rbac and domains, group-logic digest) differs.
The per-app payload no longer carries groups.path.group: Keycloak 26.8.1
does not store that key, so a payload with it never matched the stored
mapper and every deploy re-synced every app.
The admin client lives in keycloak_admin.py; tasks/utils/rest_script.yml
stages it with the script in a 0700 temp dir on KEYCLOAK_HOST_NODE, runs
it and removes the dir in an always block.
Evidence (local compose, slot 104): matrix iteration green in both
variants and passes; update runs show synced=1 in 1.1 s, then
synced=0 unchanged=1 in 0.81 s through the final delivery path, no
staging dir left behind.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The parallel-sessions test asserted that more than one of four sessions
appended something, which a loaded host where one worker drains the
shared iterator fails with nothing broken. The sessions now meet at a
barrier before their first append, so a serialised pool still fails the
assertion while a concurrent one always passes.
(cherry picked from commit 657015852a)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With the Keycloak OIDC directory active, Stalwart refuses every account create with 'forbidden: Cannot set credentials for accounts in an external directory', and full-4 failed the new batch assert on all accounts. The create request now lists the accounts in the same call and takes each id from the listing of the provisioned domain. Under SSO a forbidden create is tolerated for people, who may not exist before their first login; every bot still needs an id because its app password is addressed by account.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
