updater's Stalwart v1 port (store_guard.yml, migrate.env.j2) arrived in
the merge (c9e7f950d9) with three constructs store-i18n's lints reject:
- `DEPLOYMENT_MODE == 'swarm'` in store_guard.yml and migrate.env.j2
(test_deployment_mode_booleans): now IS_SWARM_MODE;
- `get_entity_name` in the store-engine fail_msg, a filter store-i18n
renamed to entity_name (a05071fe78), so TestAllUsedFiltersAreDefined
failed: now entity_name;
- two pairs of sibling tasks repeating the same `when`
(test_no_repeated_when_runs): the shared-database and dedicated-volume
lookups now sit in one block gated on `not (stalwart_store_marked |
bool)`, the migrate include and the pre-1.0 assert in one block gated
on `stalwart_store_layout is not skipped`; each task keeps its own
remaining condition. Ansible evaluates a block's `when` per task, so
the assert still sees the layout migrate.yml registers again.
updater's test_store_guard.py walked only the top-level tasks and failed
with KeyError: 'register' on the block wrappers. It now walks
block/rescue/always (utils.roles.invocations.NESTED_KEYS), hands a
block's `when` down to its children and checks the block condition
before the task's own, as Ansible does; the assertions are unchanged.
With either block's `when` removed from a copy of store_guard.yml, two
respectively four of its tests fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
updater's TestWriteMany (86e7c844f3) looked up the persist, mirror and
hand-over tasks at the top level of write_many.yml and evaluated only
each task's own `when`. The merge (c9e7f950d9) moved those three tasks
into store-i18n's change-gated block ("💾 Write the token store when the
merge changed it", `when: sys_token_store_users_merged !=
sys_token_store_users`) and switched the mirror to IS_SWARM_MODE, so the
lookups raised StopIteration.
The test now walks into block/rescue/always, hands a block's `when` down
to its children and ANDs it with a task's own condition (a task without
any condition counts as true, as in Ansible), renders with
IS_SWARM_MODE, and counts the re-render inside blocks too. The
assertions are unchanged; with the block's `when` removed from a copy of
write_many.yml, "an unchanged store skips the write and the re-render"
fails again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Semaphore v2.19.17 builds its post-login redirect from its one WebHost
(url.JoinPath(WebHost, return)), marks the session cookie Secure when
WebHost is https, and registered a single OIDC provider whose
redirect_url is clearnet. A login started on the onion sibling therefore
lost the oauthstate cookie at the clearnet callback and the session on
the plain-http onion origin: semaphore#0 of run 40625679270 failed five
login and design specs in PASS 2, where Playwright picked the .onion
sibling ("http: named cookie not present" 24 times in the container
log, while LDAP accepted the rotated credentials six times).
Semaphore keys OIDC providers by id from the URL path and gives each its
own redirect_url, so when the role has an onion sibling and SSO is on:
- a second provider keycloak_onion, same issuer, client and claims, with
the onion sibling's plaintext redirect_url; Keycloak's redirect_uris
lookup already registers http://<onion>/*;
- the onion vhost rewrites the absolute WebHost redirects to relative
ones (proxy_redirect) and drops Secure from the semaphore cookie
(proxy_cookie_flags), next to the existing base-href rewrite;
- location = /api/auth/login filters the provider list per vhost with
sub_filter, so clearnet offers keycloak and the onion vhost
keycloak_onion, both labelled as before; each dropped provider has a
pattern for either list position because Semaphore does not fix the
order.
Without an onion sibling the provider JSON and both proxy hooks render
exactly as before. test_onion_login renders the vars through Ansible's
templar for both cases; 16 of its 18 tests fail against the previous
files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
68 conflicts, resolved per hunk; resolutions agreed with the Updater
session were applied as agreed:
- svc-db-postgres (superuser password script and core) and
sys-svc-rdbms realign_dedicated_db: updater's argv/stdin redesign with
expand_argument_vars: false, nested into store-i18n's block structure.
- svc-db-redis 01_init: store-i18n's ACL password on stdin.
- sys-svc-proxy forwarded.conf.j2: store-i18n's handover-aware file,
without the forwarded_base include (Host and X-Forwarded-Proto would
be sent twice); forwarded_base serves the inline consumers, e.g.
web-app-seaweedfs nginx.conf.j2.
- basic.conf.j2 and web-app-openproject proxy.conf.j2: issuer redirect,
then handover, else updater's location/sso_acl.conf.j2; the
server-level webserver_extra_configuration hook stays.
- web-app-keycloak: updater's admin-REST settings step and renumbered
update steps; update/01_client.yml removed; store-i18n's design step
and LDAP block kept; a73435cc9d arrived through rename detection.
- web-app-nextcloud OpenProject provisioning: updater's rewrite, all
credentials on the Ruby runner's stdin.
- web-app-semaphore: store-i18n's canonical web root (a1d7835863).
- Role READMEs: no generated Cosmos/Architecture/Quick Setup/Credits
sections; desk-* paths resolved to the dsk-* renames.
Integration fixes for what merged cleanly but did not fit together:
- web-app-matrix ansible vars: drop updater's second
matrix_client_element_configuration_extension_json, which silently
overrode store-i18n's design-aware value.
- url.origin instead of a stripped url.base in 19 places store-i18n
added since 44aaa68924, same lookup targets (the double-slash lint).
- Package batch (18ea75049e, cb4d340de9): installs the deploy-start
batch covers are removed (caffeine, cursor, blender, npm in dsk-gnt-*,
with shared references), the rest carry batch: false and a nocheck
with the verified reason (PyPI fallback, before the batch, vendor
repository, configured flavor only, addon gate); dev-npm references
the root npm id.
- sys-token-store write_many.yml: store-i18n's change-gated block and
IS_SWARM_MODE.
- web-app-openproject: IS_COMPOSE_MODE in 02_migrate/run.yml; the
seeded-admin lock (ccbf8f168d) ported into 04_admin/provision.yml and
covered by the admin-state digest.
- Bucket policy (3b4ac40df5 and follow-ups) moved to web-svc-seaweedfs,
where store-i18n's object store lives; the S3 admin secret travels on
stdin instead of a task-level environment.
- act workflow.sh: no Exception-labelled comment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The update started one kcadm JVM per step: the client and both realms' mail settings as async jobs with their reaps, then each declared client in turn, about 27-31 s in a Nextcloud deploy. realm_settings.py reads the client scopes, clients and both realms once over the admin REST API, creates what is missing and writes a client or realm only when it drifted; secrets reach it on stdin through the new rest_script_stdin parameter. The remaining update steps are renumbered and every reference follows.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stalwart v1.0 refuses an existing account with primaryKeyViolation; the batched provisioning from 1fbd4758e3 only knew alreadyExists, so every update deploy skipped the password sync and the listed-id fallback hid the refusal (msmtp then failed with 535). An account now counts as existing when the refusal's objectId is the listed account of that name, any other refusal fails the assert, and the update patches credentials/<k>/secret instead of replacing the whole credentials list, which would have dropped every app password, API key and second factor. sys-token-store gains write_many.yml, so all bot tokens land in one store write and one applications re-render; write.yml delegates a one-entry batch to it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ruff format reflows the package batch, registry, plan and invocation code and tests and the Stalwart and Nextcloud tests, ansible-lint rewraps long Jinja lines in folded scalars of sys-service, Nextcloud and SeaweedFS, mbake restores the Makefile's final newline, and the Nextcloud OAuth test merges its nested with statements (SIM117).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
select-admin-state.sql joined select-ldap-auth-source-exists.sql under the shared 'select' prefix, which the prefix-grouping lint rejects. Both now live in files/sql/select/ and their three readers point there.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The admin step ran its Rails runner on every deploy. It now digests its sources, its inputs and the administrator and settings state OpenProject holds, compares that with the admin-provision.sha256 stamp in the instance directory, and runs only on a mismatch or when services.openproject.admin.force is set.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
db:migrate took about 24 s on every deploy. After a successful run the
deploy stamps the web container's image id as the comment of
public.schema_migrations; a deploy whose image matches the stamp skips the
migration, a new image, a failed run or a recreated database migrates. The
stamp is read unmasked because masking short config values would corrupt a
hex image id, and written through a DO/format() block because sqlfluff cannot
parse COMMENT ON ... IS with a placeholder.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
OpenBao keeps no record of who read which secret without an audit device,
and 2.7 refuses to enable one over the API, so the server config declares a
file device writing to stdout, which cannot block and lands in the container
log with secret values only as salted HMACs. Every deploy fails unless the
device is active, a probe secret read through the AppRole session shows up
in the log with path, operation, display_name and time, and the probe value
does not appear in plaintext. openbao becomes a regular shared provider so
consumers such as the landing page role can declare services.openbao.
Requirement 047; taken over from the feature/openbao-audit worktree.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The nfs-ganesha drop-in sets Environment="OPTIONS=-L SYSLOG ...", but the
Debian and Ubuntu unit (upstream nfs-ganesha.service.debian8) also reads
EnvironmentFile=-/etc/default/nfs-ganesha, whose OPTIONS name
-L /var/log/ganesha/ganesha.log, and systemd lets EnvironmentFile=
override Environment=. So on those distros ganesha logged to a file with
no bound, while CentOS, Fedora and Arch already logged to the journal
(SystemMaxUse=256M on the lab node).
Under MODE_DEBUG ganesha.conf raised NFS_V4 to DEBUG since a085427236,
which writes a dozen lines per NFS compound. In odoo#1 of run
40625679270 (debian) the nfs-server node grew 12.6 GB above the control
node and the step aborted at "DISK EXHAUSTION IMMINENT (<6144M free on
/)". In the same run, nodes logging to the journal grew 0.8 to 1.6 GB
in four hours, nodes logging to the file 4.4 to 12.6 GB.
- The drop-in resets EnvironmentFile= before setting OPTIONS. The
Debian unit's ExecStart reads nothing else from that file (its journal
reports EPOCH, GNODEID, NUMACTL and NUMAOPTS as referenced but unset).
- NFS_V4 goes back to the default level: it was raised to chase the
probe-write bug that 9b7f25f2fe fixed. DISPATCH stays at DEBUG, since
nfs_Init_svc(), which binds 2049, logs under it (1aea8a472e).
- The swarm diagnostics read the last 600 journal lines of nfs-ganesha
instead of the file tail.
test_ganesha_log_levels and test_ganesha_unit_options pin the debug
components and the drop-in's OPTIONS source; against the previous
template and drop-in both fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
e5f3c52119 lets a redeploy skip the bootstrap restarts when the running
wiki already serves the pinned core and add-ons. Its probe GETs
/rest/wikis/xwiki/spaces with the superadmin's basic auth and accepted
any 200. XWiki answers a rejected basic auth as guest with 200 (see
b1dfff370a, tasks/03_administrator.yml reads the XWiki-User header for
exactly that reason).
Between PASS 1 and PASS 2 the harness rotates the credentials
("=== matrix-deploy: rotating credentials between passes ===",
xwiki2.log:257549 of run 40625679270). PASS 2 then probed the container
still running with PASS 1's password: XWiki logged "Authentication
failure with login [superadmin]" 34 times, answered as guest with 200,
and the add-on check PUT XWiki.CheckExtension as guest until it died on
401 after 31 attempts (XWiki#2, LDAP variant; the probe only runs with
OIDC or LDAP on).
The core-version read now also requires the XWiki-User response header
to name the superadmin. A guest answer skips it, so the role takes the
bootstrap path that recreates the container with the rotated password
before any write.
test_extensions_ready_probe evaluates the task's when list for a guest
200, a superadmin 200 and a skipped probe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every Weblate container (weblate, beat, worker) runs collectstatic into
STATIC_ROOT=/app/cache/static on start. /app/cache is the `cache` volume
the three services share; in swarm it lives on NFS and every replica of
every service mounts the same directory. Concurrent collectstatic runs
delete and rewrite each other's hashed files, so the tasks crash-loop:
weblate.1, .2, .3 and worker.1 on both swarm nodes died with
"FileNotFoundError: [Errno 2] No such file or directory:
'/app/cache/static/admin/js/vendor/select2/i18n/az.270c257daf81.js'"
(and logo-32.png, dark_mode.a364bd93cdcc.css, widgets.8f8bf7f533a4.js),
which failed Weblate#1 in runs 40625679270 and 40243434327.
meta/volumes.yml mounts a tmpfs at /app/cache/static into every service
that shares the cache volume, so each container collects and serves its
own copy, consistent with its own manifest; the rest of /app/cache stays
shared. container_volumes renders it as a long-form tmpfs mount for
compose and swarm alike, as it already does for the Matrix bridges.
test_static_root_per_container asserts that every service mounting the
shared cache also gets the tmpfs at the static root.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Listing, reading, uploading and activating the Sieve forward scripts each run
as one request for every forwarding account, with one method call per account
addressed through its accountId by the recovery admin, at most
STALWART_JMAP_MAX_METHOD_CALLS per request, instead of one request per account
under act-as. The account lookup joins the limits read, a missing account
stops before the limits are raised, and only changed scripts are uploaded.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test_role_files_are_referenced_via_spot flagged
test_multisite_usernames.py:13 for spelling the role file path
"roles/web-app-wordpress/meta/users.yml" literally. The test now joins
the role directory with ROLE_FILE_META_USERS from utils.roles.mapping,
which resolves to the same path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a multisite network WP-CLI's `user create` runs
wpmu_validate_user_signup(), which only accepts lowercase letters and
digits. The MCP subscriber was declared as `mcp-wordpress`, so every
variant with `multisite.enabled: true` (variant 1) stopped at "Create the
subscriber the MCP adapter authenticates as" with "Usernames can only
contain lowercase letters (a-z) and numbers." (WordPress#1 in runs
40625679270 and 40243434327).
meta/users.yml now declares `mcpwordpress`. The users dict is the one
place the login lives: the role tasks, the MCP basic-auth login
(mcp_client_login) and the MCP client configs (mcp_servers) all read
users[owner].username, and the owner key, hence the token-store key,
stays mcp-web-app-wordpress. The email follows the login.
On an existing single-site install the next deploy creates mcpwordpress,
mints its application password and replaces the stored token; the old
mcp-wordpress subscriber is left in place with its now untracked "mcp"
application password. Remove it with `wp user delete mcp-wordpress`.
test_multisite_usernames mirrors the multisite signup checks (charset,
reserved names, 4-60 characters, not all digits): every account the role
declares passes, and mcp-wordpress, `_`, `.`, uppercase, too short, all
digits, admin and too long are refused. With the old login the test
fails with WordPress's own message.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
forwarded.conf.j2 includes headers/forwarded_base.conf.j2 by its path from
the repository root, the way Ansible searches templates, so the test's loader
searches the root next to the headers directory.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The guard now includes store_guard/migrate.yml for a store from before 1.0
and re-reads its layout, so the walk feeds that include's result into the
final assert: a migrated store passes, one left before 1.0 still stops.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sphinx's i18n transform parses every msgstr into a temporary document
whose settings carry the whole BuildEnvironment (sphinx/transforms/
i18n.py, Sphinx 9.2). MyST nodes patched into the page keep pointing at
that document, and write_doctree only detaches the environment from the
page's own settings, so every translated Markdown page was pickled with
a full environment copy: a German doctree of drv-lid-switch grew from
21 KB to 5.5 MB, the whole doctree tree from 95 MB to 2.5 GB. The write
phase loads them all back and the forked -j worker copies the parent,
so the `deployed:de` build reached 5+ GB in the parent and was
OOM-killed inside the docs container in CI run 40625679270 (Jenkins#1,
Nextcloud#11), drove shared 16 GB swarm runners into swap and the
watchdog's memory and I/O-wedge aborts (GitLab#2, Nextcloud#7/#9/#10),
and left the daemon answering 500 to the queue task for the failed
build.
AdoptTranslatedNodes runs right after the Locale transform and points
every node whose document is not the page back at the page; conf.py
loads it for every build the image runs. No size, limit or budget
changes, and docs stays deployed wherever a variant declares it.
Measured locally on the full German build at -j 2: parent peak 685 MB
(was 5.3 GB and climbing when stopped at about a quarter of the write
phase), total 2.4 GB (was 8.35 GB), doctrees 92 MB (was 2.5 GB). The
English HTML of all 5,373 pages is byte-identical before and after. A
real German MyST build in the new test finds no foreign nodes and no
BuildEnvironment in the saved page; without the transform it fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Takes main's Stalwart v1.0.0 port (server v1.0.0, WebUI v2.0.2, 0.16 store guard). manager_ops.yml keeps the updater's per-account forwards ahead of main's store marker, which stays the last task. The root README drops Akaunting with main.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test_backup_nfs_excludes.py (4fcf097e5b) computed the repository root
with Path(__file__).resolve().parents[6], which
test_project_root_import rejects outside a package __init__.py
(quality-high pass 66). The package now exports PROJECT_ROOT exactly as
its sibling files/__init__.py does, and the test imports it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The NFS backup rsynced the whole export and honoured only one exclude, the
shared backup root. Volumes declared `backup: false` were copied anyway and
then pulled a second time by remote-2-local; in the swarm DR drill of
Magento#0 (run 40625679270) docs_sites alone (a full repo clone plus its
prepared and work trees) took about 15.5 GiB per copy and filled the
runner. The volume backup already skipped these through the backup_volume
lookup; only the NFS path ignored the declaration.
- utils/storage/nfs.py: volume_dir() names the state directory a volume
binds to (pinned name or key), shared with compose_volumes;
backup_disabled_dirs() returns every NFS-backed `backup: false` volume
dir that no backed-up volume also binds to.
- The role-local lookup backup_nfs_excludes feeds them into
BKP_NFS_2_LOCAL_EXCLUDES next to the backup root; script.sh takes any
number of excludes. Compose installs still pass only the backup root.
- recover.py protects every live state directory the snapshot does not
hold, so `rsync --delete` keeps the excluded volumes (model weights,
the docs tree, caches) and volumes created after the snapshot instead
of wiping them.
- The swarm DR drill restores BACKUP_NFS_VOLUME, the first NFS volume
not declared `backup: false`; PRIMARY_NFS_VOLUME stays for the drain
tests. BookWyrm now drills bookwyrm_media; decidim, fusiondirectory,
meta, coturn, svc-registry-cache and svc-registry-docker have nothing
backed up on NFS and skip the drill.
Measured: compose_volumes renders identically before and after for 291
roles across compose/swarm x local/nfs (1940 of 1940); the 15 excluded
directories are exactly the backup-disabled device dirs; the systemd unit
renders shell-safe on ansible-core 2.20 and 2.21 and stays byte-identical
in compose and swarm-local. A new unit test runs the real script.sh over
two generations and the real recover.py back into the export, and fails
without the restore protection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lookup('tls', app, 'url.base') always ends in '/', and 39 call sites in 25 files rebuilt the slash-less form by hand with regex_replace('/$', ''), regex_replace('/+$', ''), trim('/') or .rstrip('/'), directly or on a variable holding url.base. The lookup now returns that value as url.origin (scheme://host, url.base minus its final '/'), and every call site reads it; two sites that stripped the slash only to append it again now take url.base as is, and GITLAB_URL and DASHBOARD_JS_CDN_URL lost their last consumer.
test_url_base_no_double_slash gains the rule url-base-strip, so url.origin stays the only way to the slash-less URL; it flags all nine HEAD sites probed and none after the change. The same lint keeps url-base-double-slash and url-base-addon-value, which put the nine Nextcloud integration URLs, the Mattermost and SuiteCRM facts and the XWiki instance on the slash-less form, drop the stray '/' before Joomla's index.php callback, and give BBB its api.url as [base, suffix] | url_join plus the trailing '/' its client appends 'api/<call>' to; bbb.spec.js now requires exactly that form.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Connecting a user to OpenProject failed: the instance URL carried url.base's trailing slash, so Nextcloud redeemed the code at //oauth/token (HTTP 500), integration_openproject stayed in its fresh-setup state so isAdminConfigOk was false and no Connect button rendered, and duplicate OAuth clients piled up across deploys. The coupling now converges on each deploy: a PHP helper creates or repairs the Nextcloud OAuth2 client (numeric id, OpenProject callback, same-name duplicates removed); one rails runner call ensures the storage, its OAuth application and the Nextcloud client on it, renewing the OpenProject secret when Nextcloud holds none or a stale one, or when rotate_oauth_client forces it, and realigning the application's redirect_uri after a host change; nc_oauth_client_id is stored only after that call succeeded. openproject_instance_url has no trailing slash and fresh_project_folder_setup is 0.
No secret travels in argv or environment: on the way to OpenProject they ride the same stdin as the Ruby script, as a PROVISION_INPUT heredoc, and the credentials go into Nextcloud through occ lines on the container shell's stdin, taken verbatim from 7f72e53a58 on feature/store-i18n. A Ruby truth table over seven scenarios, with the staged state as negative control, covers fresh install, rerun, a Nextcloud that lost client and secret, a stale secret, forced rotation and a host change. A new Playwright test connects a user through the OAuth round trip and expects Nextcloud's oauth-redirect below 500 and 'Connected as'.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
320929e17c read the tempdir compose.yml back with Path.read_text, which
test_no_uncached_file_reads rejects (quality-high pass 64).
utils.cache.yaml.load_yaml keys its cache on path, mtime_ns and size, so
the file sync() rewrites in place is read fresh; the label assertions
would fail on a stale read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The autoformat step of `make quality-high` reflowed three lines of
320929e17c that exceeded the line length; tokens are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Swarm deploys with `docker stack deploy --resolve-image never`, so a task
only rolls when its spec changes. Since fdf1a33c5f BookWyrm, Decidim and
the Bluesky broker build under a constant tag (bookwyrm_custom:3.11-slim
and friends): a ref bump rebuilds the image through the build-args digest
and pushes it, but the image string in compose.yml stays the same and the
running tasks keep the old image.
registry_sync now stamps every service that runs a locally built image,
siblings sharing the tag included, with the container label
`infinito.image-id=<local image ID>`. A rebuild changes the ID, the label
changes the ContainerSpec, and swarm rolls the tasks; an unchanged image
leaves the label and the deploy untouched. List and mapping label forms
are both kept intact.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
baudolo stops every container on a volume before the authoritative copy
unless its image is listed via --images-no-stop-required. Leaving the
marker out silently picked "stop": the Stalwart webmail start that the
backup cycle triggered got SIGKILLed mid `composer update` (CI run
40625679270), and 71 services that mount a named volume had never
decided either way.
- The marker is now `backup.stop_required` (true = stop first, false =
copy live), replacing the double-negated `no_stop_required`; the 50
existing declarations are inverted unchanged. backup_image takes the
marker and the wanted value explicitly:
lookup('backup_image', 'stop_required', false) still feeds
--images-no-stop-required, lookup('backup_image', 'disabled', true)
the skip list.
- The 71 undecided services carry researched values (repo files plus
upstream backup guidance: GitLab, Jenkins, Elastic, RabbitMQ, Typesense,
Qdrant, OpenLDAP, Prometheus, Moodle, ClamAV, Alertmanager, ...): engine
files, git repositories, SQLite, mnesia and multi-file state stop; image
code, write-once blob stores with metadata in the database and
atomically rewritten files copy live. Stalwart's main service stays
stopped until /var/lib/stalwart is verified to hold only logs.
- Re-obtainable data is no longer backed up at all (`backup: false` on
the volume): model weights, ClamAV signatures, SearXNG config/cache,
transient RabbitMQ queues, Funkwhale's Typesense index, the meta clone,
the agent-broker socket dir, coturn's unused data dir, and volumes no
image reads (Decidim's /app paths outside /code, BookWyrm's /app/data
which v0.9.3 never reads, FusionDirectory's measured-empty dir).
- The backup lookup only matches what actually runs once the role
declares it: Bluesky's broker (`custom: bluesky_broker`) and Decidim
(`custom: true`) declare their custom build, BookWyrm's worker/migrate,
EspoCRM's daemon/websocket, Moodle's cron and OpenProject's proxy use
their owner's image, and the templates drop their render-time
overrides. Decidim deploys decidim_custom:0.31.6 and BookWyrm
bookwyrm_custom:3.11-slim from now on; a BookWyrm ref bump still
rebuilds through the build-args digest.
- test_backup_stop_decision_declared requires the boolean on every
service with its own image that mounts a backed-up named volume;
test_stop_required replaces the old integrity test.
Measured against HEAD: 43 -> 78 services copied live, none moved the
other way; 14 -> 30 volumes skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
render_override takes a service's entrypoint and command from the
output of `docker compose config`, which already prints a literal `$`
as `$$`, and the image's Entrypoint and Cmd from `docker image inspect`,
which are raw. It escaped the combined argv once more, so a service
command reading a variable at runtime (`$$REDIS_PASSWORD` in
compose.yml) reached the override as `$$$$` and the container as `$$`,
which the shell expands to its PID. redis then required
"<pid>REDIS_PASSWORD" and every authenticated ping failed with
WRONGPASS.
Only the image-derived parts are escaped now; the service's own argv
is written as compose config printed it. A new test keeps a service
command's `$$` intact next to an image entrypoint; the existing tests
still require an image Cmd's `$VAR` to become `$$VAR`.
Local Nextcloud update deploy: the redis override holds `$$`, the
container Cmd `$REDIS_PASSWORD`, and the authenticated ping answers
PONG on the first attempt.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
automtu searched the path MTU between its payload bounds, and a lost
echo during that search moved the result between 1400 and 1398 from one
deploy to the next. Every move changed the daemon MTU, rewrote
daemon.json and restarted dockerd.
The probe window is now a single payload: the smaller of 1500 and the
egress MTU automtu reads itself (`--print-mtu egress`), minus the 28
bytes of IP and ICMP header, with `--pmtu-policy max` so one target
answering at that size decides. registry-1.docker.io leaves the
targets because it does not answer ICMP echo. The assert now names both
the egress and the effective value.
Local repeat deploy: the probe took 0.83 s and no handler restarted
dockerd.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Handlers run in definition order, and "Generate CA trust override" was
defined before "Build compose". On a first deploy the override read the
Entrypoint and Cmd of every service image with docker image inspect
before compose build had created the locally built tags, found none,
and left those services without the with-ca-trust wrapper; on later
deploys it read the previous build's Entrypoint and Cmd instead of the
new one. The override and the compose config validation now run after
the build and rebuild handlers and before compose up; pull stays first.
The swarm handlers already build before they generate their override
and use no wrapper, so they are unchanged.
TestFirstDeployFlush replays the shell handlers of one compose-build
plus compose-up flush in the order handlers/compose.yml defines them
and asserts that compose up starts the freshly built image behind the
wrapper.
Local deploy: all six stacks with a built image carry the wrapper in
compose.ca.override.yml, nextcloud for 8 of 8 services.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
recognize's app:install downloads about 700 MB of models and held the
plugin phase for minutes. An addon may now set
config.background_install; the plan files such a plugin under the new
background list unless it is pinned through archive or mandatory.
01_background.yml starts occ app:install as an async job and holds
every later occ call at a gate until the app's schema is applied:
Nextcloud migrations diff a schema snapshot, so DDL running beside
another occ call can be reverted. The gate reads the async job file
and files/php/utils/migration_state.php, which prints the versions
oc_migrations records and the versions the app's migration folder
holds. occ migrations:status is not usable, Nextcloud registers it only
with debug on. While the app directory does not exist yet the
MigrationService constructor throws and lib/base.php's ErrorHandler
only logs it, so the probe exits 0 with nothing on stdout and the gate
waits.
The install loop then runs while the job finishes; the always section
joins every job with async_status and fails on an exit code that a
timeout or signal produced, because the php process may then still run
inside the container. 02_install.yml afterwards answers "already
installed" and enables and configures the plugin, or installs it with
its own retries when the job failed before placing the app.
The stage files are renumbered (01_install -> 02_install, 02_enable ->
03_enable, 03_configure -> 04_configure) so the numbers follow the
include order the task-numbering lint checks.
Local perf run: plugin phase 538 s -> 282 s, gate open after 30.2 s,
recognize 12.0.2 enabled with 8/8 migrations, failed=0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
roles/web-app-nextcloud/tasks/01_fullstack.yml: both sides added a
replicated-template task at the same spot; store-i18n's Talk signaling
backend list and the updater's Janus config now stay as two tasks with
their own conditions.
roles/web-app-nextcloud/templates/compose.yml.j2: the updater rendered
the talk service's container_volumes a second time inside the direct-media
branch, next to store-i18n's unconditional call, which doubles the
volumes/configs keys once direct media is on. The janus_conf mount carries
its own direct_media `when` in meta/volumes.yml, so the single call
renders it; the duplicate is gone.
DEPLOYMENT_MODE == 'compose' became IS_COMPOSE_MODE | bool in
web-app-nextcloud/meta/services.yml, web-svc-coturn/vars/main.yml and
tasks/stages/03_destructor.yml, as store-i18n's deployment-mode lint
requires; the updater's coturn, Talk and decommission tests now pass
IS_COMPOSE_MODE alongside DEPLOYMENT_MODE. test_talk_direct_media.py
expects store-i18n's talk command line and mount position, and the
"host networking ignores direct media" check moved to the config level,
where direct_media already requires network_mode bridge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- test_prefix_grouped_files: files/sql/store_database.sql and
store_layout.sql become files/sql/store/database.sql and layout.sql.
- test_file_spot: test_store_guard.py builds the vars path with
ROLE_FILE_VARS_MAIN instead of a hard-coded "vars" / "main.yml" chain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodeQL flagged five notes on PR #1215: four test modules imported unittest both as a module and through from-import, and sys-front-inj-matomo's __init__ exported PROJECT_ROOT without declaring it, so the import from test_site_cache.py read as unused. The modules now bind mock through import unittest.mock as mock, and the package declares __all__, the same fixes main carries from 3610e7b9ef and 6e776a5780. The same pattern is fixed in the new Talk direct-media test and the decommission integration package, and the coturn test package drops an __init__ export nothing imports.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
janus_conf is mounted only when direct media is on, which requires compose mode, and compose ignores a config mode outside swarm and warns about it on every deploy. The file keeps the 0644 it is rendered with on the host.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
web-opt-rdr-domains renders a `return 301` vhost for each network sibling
of a redirect source (network_redirects), but the health expectations only
forced the clearnet source to [301]. A handover app's onion sibling kept
its [302] handover expectation, so Bluesky's disabled view domain failed
"view.bluesky.<onion>: ERROR: Expected [302]. Got 301." and
hlth-webserver aborted the deploy. Every expected domain whose clearnet
sibling is a redirect source now expects [301].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removing a role from the inventory left its compose stack running under /opt/compose, holding ports, memory and a stale instance directory, until someone stopped it by hand.
On a full compose deploy with MODE_CLEANUP (no --id), the destructor now takes every directory below DIR_COMPOSITIONS that is named after a known role, carries the full instance layout and is owned by no active role (inventory-wide groups, group_names, the dependency closure and the instances this play prepared), stops it with compose down --remove-orphans without -v, verifies through the project and working_dir labels that no container is left, and only then deletes the directory; named volumes stay. Hand-made, partial or backup directories and instances a non-compose container is named after are left in place and listed. The step is the last block of the destructor, with the timer unlock in its always section, so a failure cannot leave timers locked.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The aio-talk image runs Janus with its stock janus.jcfg: no rtp_port_range and no nat_1_1_mapping, so Janus offered only its bridge address and every stream of a call went through coturn as relay. Calls above about six participants stuttered and failed.
When talk runs bridged next to standalone coturn in compose mode on a host that owns networks.internet.ip4, a rendered janus.jcfg is mounted over the stock one with rtp_port_range 61000-61999, nat_1_1_mapping = ip4 and keep_private_host, and that UDP range is published on ip4. TURN stays the fallback. Before rendering, the deploy reads the image's launcher and stock janus.jcfg and stops when Janus no longer reads the mounted path or the stock settings drifted from the template. Variant 0 publishes 64 ports so dev and CI exercise the path; the call capacity spec asserts a non-relay selected pair on ip4 in that range. With the flag off every rendered file is byte-identical to before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With host networking and no --relay-ip, coturn put every non-loopback IPv4 of the host into its relay list, Docker bridge gateways included, and handed IPv6 and bridge clients relays on those addresses while advertising the public ip4, so their media went nowhere. It also listened on every bridge address.
In compose mode with host networking, coturn now listens on networks.internet.ip4, 127.0.0.1 for the healthcheck and networks.internet.ip6 when set, and relays on ip4. It pins only when the host owns ip4, and owns ip6 too or leaves it unset; behind 1:1 NAT, without the ip binary or with a foreign ip6 it renders the previous command unchanged, because coturn exits when it cannot bind a listening address. A unit test renders the real template and vars for every cell.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Follow-up to the v1.0.0 port, closing holes a review and a local run found:
- WebUI sign-in: WebUI v2 sends an http:// client-metadata-document id
on the onion origin, which Stalwart v1.0.0 rejects (unknown http ids
get no CIMD fallback, trusted hosts require TLS). The recovery admin
under SSO hit the same wall with the bare id 'stalwart-webui'. Register
a public first-party x:OAuthClient 'stalwart-webui' with the WebUI's
callback and login URIs on the served origin, and always inject it as
the Application's oauthClientId. Native sign-in now shows Stalwart's
consent step; test-login-native.js clicks it and also runs under SSO.
- Store guard: the previous guard only checked a running 0.16 server; a
local run showed v1 writing into a 0.16 store whose server was stopped.
The guard now reads the store whenever one exists (shared: pg_database,
dedicated: volume) and fails on the 0.16 bigint layout. A host marker
written at the end of a successful v1 deploy lets restores and
redeploys with a stopped dedicated database pass; without the marker,
a stopped dedicated database fails with an explicit message instead of
a masked psql error. Queries retry while the database recovers.
- README: WebUI sign-in, guard semantics, restore and redeploy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stalwart v1.0.0 rejects unknown JMAP properties ("Invalid property" ->
notCreated/notUpdated), validates OIDC tokens more strictly and refuses to
open a data store from before 1.0. Port the role and protect existing
installs:
- Milter: hostname/timeoutConnect/timeoutCommand/timeoutData become
host/connectTimeout/commandTimeout/dataTimeout; the existing milter is
looked up by host.
- Onion relay route: address becomes host.
- OIDC directory: requireAudience is gone. Set skipAudienceValidation and
unscopedTokens=interoperable on create and update, so Keycloak tokens
(no Stalwart audience, no mail scope) keep working for the WebUI and
Roundcube XOAUTH2.
- WebUI: send oauthClientId only with SSO enabled; without SSO the WebUI
falls back to its own client metadata document.
- Store guard: before anything changes, a running Stalwart whose table u
still has the 0.16 bigint layout stops the deploy and points to the
stalwart-migrate steps, which the README now documents under
"Upgrading from 0.16".
- Updater: pin the WebUI git_tags source to v2.* to match server 1.x.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
16 conflicts, resolved per hunk:
- sys-stk-front-proxy: tasks/main.yml keeps the network-sibling loop;
the single stack_host_contents write of http.conf plus vhost and the
front_proxy_vhost_changed gate move into tasks/vhost.yml, with the
http.conf decided per sibling domain so an onion vhost gets no HTTPS
redirect.
- sys-front-tls: 01_http.yml is gone as the updater intends; the
cert-renew include in main.yml gates on the vhost domain's TLS.
- sys-svc-compose: the updater's resolve_host_cid without the remote
echo and the prepared pending flag in up.yml, written with
IS_SWARM_MODE and the repeated-when block grouping.
- web-app-nextcloud: the updater's post-upgrade gate and pending-only
plugin configuration, which already carry the pinned-plugin handling.
- web-app-xwiki: one XWIKI_CORE_VERSION for the readiness probe and
style.css.j2; OIDC/LDAP follow xwiki_extensions_ready at bootstrap,
the S3 blob store stays off there as TestBlobstoreRenderOrder demands.
- roles_with_service: onion swap and dashboard card key combined, the
canonical domain is the canonical URL's host in both branches.
- sys-ctl-hlth-csp: --clearnet-timeout and --concurrency together;
run_checker forwards its options to build_docker_cmd, keeping the
script under the 500-line limit.
- web-app-matrix: Element branding and web_client_location side by side.
- web-app-stalwart/README.md: the generated Cosmos section is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
csp-checker 3.2.0 checks every URL in its own browser context and takes --concurrency for a worker pool. The role pins 3.2.0, sets HEALTH_CSP_CONCURRENCY to 4, and the systemd unit hands it through script.py to the container command ahead of the -- URL separator, for the clearnet and the onion batch alike. TimeoutStartSec keeps its sequential sizing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test_include_uniform_apply_keyword flagged legacy_checkout.yml after 11c2d8dcd3 moved its guard onto the include: both tasks repeated become: true. The include in 03_launcher.yml now applies become to the file and keeps the guard; the order test reads include_tasks in its mapping form too.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each plugin ran its own occ config:app:set per configuration value right after its install or enable, and a redeploy rewrote every value. Install and enable now queue the plugin; the configure stage runs once for the queue: it loads every queued addon spec, reads occ config:list once, keeps only the values Nextcloud does not hold exactly (nextcloud_plugin_config_pending; typed, masked and structured values are always kept), and runs the hooks in queue order afterwards.
Combined with store-i18n's 7f72e53a58: the pending occ config:app:set calls go on the stdin of one container exec -i ... sh -s, so no value (Talk, OnlyOffice, whiteboard, BBB secrets) reaches a host argv; a failing call names its appid/configkey.
Replay harness old vs new: identical final state, updated keys and hook order in first-deploy, converged, drift and failure scenarios (failure stops at the same key); converged config:app:set calls 19 -> 3. Merge note: resolve 03_configure.yml with this side against 7f72e53a58.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every app's compose pass ran three remote tasks before compose up: create the directories, compose ps for running containers, and stat the pending marker (53 apps x ~2.7 s in a production run). compose_instance_prepare does all three in one call from 02_stack_host_prep.yml, sharing the directory creation with directory_batch through utils/directory_batch.py; compose ps runs only in compose mode. up.yml evaluates the registered result, named compose_instance_before_files because it predates 04_files: the running check is skipped whenever the compose, env or override file changed in this run (those changes notify compose-up themselves).
Replay harness old vs new: rendered files and the compose-up, marker and swarm handlers identical in every scenario; the only difference is the skipped running check when only the override changed, where compose-up still runs. About 85 s per full production run.
Merge note: store-i18n's a4c7fda974 hoists the removed marker stat into a block; resolve up.yml with this side.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
DISCOURSE_REPOSITORY_DIR appended the repository URL to the services directory, so the launcher lived under services/https:/github.com/discourse/discourse_docker.git/. It now lives under services/discourse_docker. Before the clone, launcher/legacy_checkout.yml removes the URL-named tree (computed with relpath against the services directory) and prunes its emptied parents; a guard skips the cleanup when that path would escape the services directory or equal the new checkout. The cleanup must precede the clone: git otherwise resolves the old relative path as a URL and aborts with 'protocol ... is not supported'. Discourse data stays in the launcher's named volume and /var/discourse/shared; the first deploy rebuilds once because the bootstrap digest lived in the old containers/ directory.
relpath joins the builtin filters test_defined knows.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
