Commit Graph
435 Commits
Author SHA1 Message Date
kevinveenbirkenbach db17bd5d4a Merge branch 'feature/updater' into feature/store-i18n 2026-10-24 21:52:15 +02:00
kevinveenbirkenbachandClaude Opus 5.5 49aa920034 perf(web-app-keycloak): mark user emails verified in one REST call
The emailVerified update looped over every non-reserved user, uploaded a
shell script per user and ran kcadm with a JVM login each time, and it
always reported changed: 7 users took 23.8 s on echoserver.

email_verified.py uses the shared keycloak_admin client through
tasks/utils/rest_script.yml: it looks each user up by exact username,
skips users already verified, and writes emailVerified with a GET plus a
full-representation PUT, as kcadm update did. The task reports changed
only when a user was updated, and a failed lookup now fails the task
instead of being treated as a missing user. The old lookup piped
kcadm -q username=X into head -n1, which could have hit a longer username
sharing the prefix.

Evidence (local compose, slot 104): summary updated=0 unchanged=8
missing=0 failed=0 in 0.60 s, task ok, staging dir removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 19:39:53 +02:00
kevinveenbirkenbachandClaude Opus 5.5 9301220d05 perf(web-app-keycloak): reconcile per-app RBAC over the admin REST API
The per-app RBAC groups and LDAP group mappers were reconciled by four
kcadm shell scripts that started a JVM per call and re-synced every app
on every deploy: 582 s for the mappers plus 324 s for the orphan purge on
echoserver (39 apps).

rbac_reconcile.py now talks to the admin REST API from inside the
Keycloak network namespace (nsenter --net) with the admin credentials
read from /proc/<pid>/environ, so nothing secret reaches argv or output.
It drops the shared ldap-roles mapper, purges root groups of undeployed
apps, ensures each app group and mapper, and syncs a mapper only when the
group is new, the mapper config drifted, or the fingerprint stored on the
group (mapper payload, app rbac and domains, group-logic digest) differs.

The per-app payload no longer carries groups.path.group: Keycloak 26.8.1
does not store that key, so a payload with it never matched the stored
mapper and every deploy re-synced every app.

The admin client lives in keycloak_admin.py; tasks/utils/rest_script.yml
stages it with the script in a 0700 temp dir on KEYCLOAK_HOST_NODE, runs
it and removes the dir in an always block.

Evidence (local compose, slot 104): matrix iteration green in both
variants and passes; update runs show synced=1 in 1.1 s, then
synced=0 unchanged=1 in 0.81 s through the final delivery path, no
staging dir left behind.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 19:28:31 +02:00
kevinveenbirkenbachandClaude Opus 5.5 5a1761f218 fix(web-app-openclaw): admit the Control UI origin of every network the gateway serves
On a multi-network host one gateway serves the clearnet domain and its onion sibling, but allowedOrigins carried only the canonical URL. The Tor pass opens http://claw.ai.<onion>, so the gateway closed every Control UI websocket with 4008 before the pairing step and the UI reported origin-not-allowed (142 onion origins, 120 closes in the Funkwhale#1 rescue log of run 40042690056).

OPENCLAW_CONTROL_UI_ORIGINS takes lookup('network_urls', OPENCLAW_URL): one URL per network sibling with that sibling's scheme, canonical first, and the URL alone on a single-network host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 19:13:55 +02:00
kevinveenbirkenbachandClaude Opus 5.5 d30425c0ac fix(i18n): refuse a translation that adds lines or comes back in another script
A chat engine prefixed the Vietnamese catalog with its own preamble ("...Vietnamese:" followed by a blank line) and wrote Urdu in Devanagari. Neither tripped the damage predicate: vi is Latin script, so the script checks were blind, and untranslated only measures the Latin share, so a non-Latin language answered in another non-Latin script passed.

added_lines refuses a translation that breaks more lines than its source, and the gateway's _mangled carries the same check so such an answer falls through to the next engine instead of being cached. wrong_script refuses a non-Latin language when less than half of the letters it writes in another language's endonym script belong to its own; the script per language is derived from the endonym in meta/languages.yml, and letters are counted over runs of letters plus combining marks so Indic and vocalised Arabic words stay whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 19:09:00 +02:00
kevinveenbirkenbachandClaude Opus 5.5 372142848b test(web-app-stalwart): decide the IMAP parallelism instead of racing it
The parallel-sessions test asserted that more than one of four sessions
appended something, which a loaded host where one worker drains the
shared iterator fails with nothing broken. The sessions now meet at a
barrier before their first append, so a serialised pool still fails the
assertion while a concurrent one always passes.

(cherry picked from commit 657015852a)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 18:01:19 +02:00
kevinveenbirkenbach 8f22a3d488 Merge branch 'feature/updater' into feature/store-i18n
# Conflicts:
#	README.md
#	roles/svc-bkp-local-2-device/tasks/00_core.yml
#	roles/svc-bkp-nfs-2-local/tasks/00_core.yml
#	roles/svc-bkp-remote-2-local/tasks/00_core.yml
#	roles/svc-bkp-secrets-2-local/tasks/00_core.yml
#	roles/svc-bkp-volume-2-local/tasks/00_core.yml
#	roles/sys-ctl-cln-bkps/tasks/00_core.yml
#	roles/sys-ctl-cln-certs/tasks/00_core.yml
#	roles/sys-ctl-cln-disc-space/tasks/00_core.yml
#	roles/sys-ctl-cln-faild-bkps/tasks/00_core.yml
#	roles/sys-ctl-hlth-webserver/filter_plugins/web_health_expectations.py
#	roles/sys-ctl-mtn-cert-renew/tasks/00_core.yml
#	roles/sys-svc-container/README.md
#	roles/sys-svc-container/vars/main.yml
#	roles/sys-svc-proxy/templates/headers/forwarded.conf.j2
#	roles/sys-svc-webserver-https/tasks/00_core.yml
#	roles/sys-token-store/tasks/write.yml
#	roles/web-app-discourse/templates/config.yml.j2
#	roles/web-app-keycloak/files/shell/per_app_mapper/ensure_app_group.sh
#	roles/web-app-keycloak/files/shell/per_app_mapper/purge/app_subgroups.sh
#	roles/web-app-keycloak/files/shell/per_app_mapper/sync_app_mapper.sh
#	roles/web-app-keycloak/files/shell/per_app_mapper/upsert_app_mapper.sh
#	roles/web-app-matomo/tasks/03_install_path.yml
#	roles/web-app-matomo/tasks/08_reinject_consumers.yml
#	roles/web-app-matrix/README.md
#	roles/web-app-matrix/tasks/flavor/compose/04_webserver.yml
#	roles/web-app-matrix/vars/main.yml
#	roles/web-app-moodle/tasks/03_ownership.yml
#	roles/web-app-seaweedfs/meta/info.yml
#	roles/web-app-stalwart/tasks/04_manage_user.yml
#	roles/web-opt-rdr-domains/README.md
#	roles/web-opt-rdr-domains/tasks/main.yml
#	roles/web-svc-file/tasks/00_core.yml
#	roles/web-svc-html/tasks/00_core.yml
#	tests/unit/python/roles/web-app-prometheus/test_blackbox_targets.py
2026-10-24 17:37:48 +02:00
kevinveenbirkenbachandClaude Opus 5.5 6de71db3d0 fix(web-app-docs): ship remote images with the site instead of hotlinking them
The documentation index renders README.md, whose badges load from img.shields.io, and one static build is served on every network the node runs, so the onion docs page requested a clearnet third-party host and network-leak.spec.js failed on the docs+tor pass (fork run 40042690056, jobs Moodle#1, Collabora#1, Listmonk#1 and the swarm API#1 job, log line 37249).

A Sphinx extension now downloads every remote image of an HTML build into the doctree cache, names it by the hash of its URL with the suffix its Content-Type names, and points the image at the local copy so Sphinx ships it under _images. An image that cannot be fetched gives way to its alt text and keeps its link, and the gettext build, which runs with -W, is left alone. Visitors' browsers no longer contact img.shields.io at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 16:11:55 +02:00
kevinveenbirkenbachandClaude Opus 5.5 7a95245390 fix(web-app-dashboard): fetch an external branding host through the onion mirror
With the asset and design services off (dashboard variant 1) the logo falls back to service_provider.platform.logo, which without web-svc-asset is the public https://file.infinito.nexus/assets. network_url leaves a host the node does not serve unchanged and port-ui embeds it as is, so the onion dashboard requested clearnet https://file.infinito.nexus/assets/img/logo.png and network-leak.spec.js failed in four jobs of fork run 40042690056 (Gitea#1, OpenProject#1, Mastodon#1, XWiki#1). Pure tor mode loads the same URL; its leak spec only runs with more than one pass.

csp.yml now lists the asset service URL under img-src. build_csp_header does not read img-src, so the header is unchanged; external_asset_origins does, which puts the host on the web-svc-mirror allowlist and into the onion body filter's rewrite to http://mirror.<onion>/file.infinito.nexus, the path every other third-party asset of an onion page takes. A deployed web-svc-asset resolves to the node's own file domain, which external_asset_origins skips. test_branding_mirror_origin.py covers both cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 15:58:30 +02:00
kevinveenbirkenbach 9d08fda117 Merge remote-tracking branch 'origin/main' into feature/updater-lockdown-probe 2026-10-24 15:09:52 +02:00
kevinveenbirkenbachandClaude Opus 5.5 3d1480b3f7 fix(web-app-stalwart): resolve account ids when SSO owns the directory
With the Keycloak OIDC directory active, Stalwart refuses every account create with 'forbidden: Cannot set credentials for accounts in an external directory', and full-4 failed the new batch assert on all accounts. The create request now lists the accounts in the same call and takes each id from the listing of the provisioned domain. Under SSO a forbidden create is tolerated for people, who may not exist before their first login; every bot still needs an id because its app password is addressed by account.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 12:04:29 +02:00
kevinveenbirkenbachandClaude Opus 5.5 7a82a92e84 fix(web-app-prometheus): probe only the domains an app serves itself
#1180 gave web-app-matrix a canonical `jitsi` domain for its own Jitsi
and listed it under `services.jitsi.domains`. When Jitsi is shared or
disabled nothing serves that domain, and the blackbox healthz job of
web-app-prometheus probed every canonical domain, so the probe of
https://jitsi.matrix.<domain>/healthz/ready failed with "tls:
unrecognized name" and `prometheus scrape: every consumer role reports
up=1 and passes its probe` went red (main run 40017166046: Matrix#2,
Nextcloud#12).

web_health_expectations already skipped such keys. The rule now lives
in utils/domains/unserved.py: a key claimed by a service entity that is
disabled, or enabled but shared, is not served by that app. The health
filter uses it unchanged, and prometheus.yml.j2 filters its targets
through the new `served_domains` filter, which also drops the
`<key>_onion` mirror of a skipped key.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 11:44:14 +02:00
kevinveenbirkenbachandClaude Opus 5.5 b796903d21 perf(svc-db-openldap): set the user passwords over one connection
Three per-user loops ran on every deploy: an ldap_search for whether the user existed (with failed_when: false, so an LDAP error read as 'missing'), a set_fact building a map from it, and community.general.ldap_passwd per user, 18 s for 7 users on echoserver. ldap_entries state=create now registers the DNs it created, and one ldap_passwords call sets the passwords over a single admin bind: a probe bind as the user decides, INVALID_CREDENTIALS leads to the password modify operation, check mode reports without writing.

openldap_user_passwords keeps the old policy: a user whose record leaves password_update on, or whom this run created, gets its password. Errors on single entries are collected so the others still converge, and the task then fails naming every DN.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 10:54:16 +02:00
kevinveenbirkenbachandClaude Opus 5.5 1fbd4758e3 perf(web-app-stalwart): provision accounts and bot app passwords in batched JMAP calls
Every account ran about six JMAP calls of three Ansible tasks each through jmap_call.yml, 80 s for 7 accounts on echoserver. Each phase now sends one request for all accounts: one x:Account/set creates them all (ids from created or, on alreadyExists, from notCreated.objectId), one updates the passwords of the accounts that already existed, and the app-password phases (list, mint, retire, drop) carry one method call per bot, each addressing its bot through accountId, in chunks of STALWART_JMAP_MAX_METHOD_CALLS (Stalwart's default maxMethodCalls, 16).

jmap_call.yml retries and asserts on every method response of a batch, and a refusal names its owner through the call id. A create refused for a reason other than alreadyExists now fails the deploy instead of being skipped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 10:37:01 +02:00
kevinveenbirkenbachandClaude Opus 5.5 3b4727fcfe fix(web-app-stalwart): rate-limit and ban HTTP clients by their own address
With the role network's gateway on Stalwart's allowlist, every request OpenResty proxies arrives from an address that is never throttled or banned, so the anonymous HTTP limit and the scan and auth bans no longer applied to web clients at all. Stalwart can take the client address from a Forwarded header, but OpenResty passed a client-supplied one through, so trusting it would have let anyone pick the address that gets banned.

forwarded.conf.j2 gains an opt-in proxy_forwarded_client_address switch that sets Forwarded: for=$remote_addr on every location including the healthz ones, replacing any value the client sent; other vhosts render unchanged. The Stalwart vhost opts in, and the role sets useXForwarded through x:Http. The settings apply moved into security/apply.yml behind one include so the guarded block stays within the three-task limit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 10:17:42 +02:00
kevinveenbirkenbachandClaude Opus 5 657015852a test(web-app-stalwart): decide the IMAP parallelism instead of racing it
`test_parallel_sessions_append_every_message_once` asserted that more than
one of four sessions had appended something. ThreadPoolExecutor promises a
thread count, never that every worker takes an item from a shared iterator,
so a loaded host where one thread drains all 40 messages before its
siblings start fails the assertion with nothing broken. CI did exactly that
in run 40023932276; 25 local runs pinned to a single CPU did not.

The sessions now meet at a barrier before their first append. A pool that
runs them concurrently releases all four, and a serialised one reaches the
timeout and leaves the later sessions empty, which the assertion still
catches: with max_workers forced to 1 the test fails with the same
"1 not greater than 1" CI reported.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-24 00:06:52 +02:00
kevinveenbirkenbach 59c5667f1f Merge branch 'feature/updater' into feature/store-i18n 2026-10-23 22:24:51 +02:00
kevinveenbirkenbachandClaude Opus 5.5 694007c58a feat(sys-svc-cln-domains): remove the nginx configs of domains the host no longer serves
Retired apps, aliases and legacy zones left their vhosts behind, so 37 legacy cymais.cloud subdomains kept serving a certificate that does not name them after the zone moved to one wildcard redirect. With MODE_CLEANUP the role now deletes every <domain>.conf below the servers directory whose domain is not in CURRENT_PLAY_DOMAINS_ALL, the domains of every application in group_names and its dependencies including redirect sources; files not named after a domain and onion services stay.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 21:25:50 +02:00
kevinveenbirkenbach d0eaca8292 Merge remote-tracking branch 'origin/main' into feature/updater 2026-10-23 20:28:25 +02:00
kevinveenbirkenbachandClaude Opus 5.5 9875db5ff8 feat(web-app-stalwart): deliver mail of legacy alias domains to their target domain
services.stalwart.alias_domains maps a receive-only legacy domain onto the mail domain: the deploy sets the RCPT stage rewrite to one matches() rule per alias domain, reads it back, and with Cloudflare points the source's MX at the mail hostname. An empty list leaves Stalwart untouched.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:19:59 +02:00
kevinveenbirkenbachandClaude Opus 5.5 8c8e6a1242 feat(web-opt-rdr-domains): redirect a whole legacy zone with one wildcard source
redirect_domain_mappings now renders its source and target, so an inventory can map *.{{ OLD_DOMAIN }} and the apex onto a target; keep_path: false drops the request URI from the 301. Wildcards get no www. variant, sys-dns-wildcards adds the *.<zone> A/AAAA records for every zone with a wildcard source next to DOMAIN_PRIMARY, and the CSP health check skips wildcard names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:17:19 +02:00
kevinveenbirkenbachandClaude Opus 5.5 27762376f0 perf(sys-ctl-hlth-webserver): probe the domains in parallel
The checker sent one HEAD request after the other, up to 10 s per clearnet and 30 s per onion domain. A pool of 16 threads now probes them; the waits are I/O, so 16 suit hosts with any core count, and pool.map keeps the output sorted by domain.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:10:52 +02:00
kevinveenbirkenbachandClaude Opus 5.5 8e73c84b15 perf(svc-db-openldap): converge the user entries over one connection
Three loops started one async ldap_entry or ldap_attrs job per user and reaped them, about 2 minutes per deploy on echoserver. The ldap_entries module binds once and applies create, present or exact to every entry the new filters build from OPENLDAP_USERS, keeping the semantics of the community modules it replaces.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 19:59:37 +02:00
kevinveenbirkenbachandClaude Opus 5.5 5b6db5dc24 feat(web-app-stalwart): import Mailu only until one import has finished
The import script writes a done marker into the instance directory when it ends with rc 0, and the deploy skips the unit and the import block while the marker exists; deleting it imports again. The IMAP layout helpers move to imap_layout.py so the script stays under the 500-line limit, with mirrored tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 19:51:10 +02:00
kevinveenbirkenbach 0bca4bcd7f Merge remote-tracking branch 'origin/main' into fix/web-app-matrix/ansible-flavor-unwired-services/1165 2026-10-23 18:18:31 +02:00
kevinveenbirkenbach a04f2c9895 Merge branch 'feature/updater' into feature/store-i18n 2026-10-23 17:32:32 +02:00
kevinveenbirkenbachandClaude Opus 5.5 25ec8b0c59 fix(sys-ctl-hlth-webserver): skip a consumer's domains for a shared service
A service entity can list canonical domain keys under `domains`; the
web health expectations skipped them only while the service was
disabled. A consumer whose service is enabled but shared does not serve
its own domain for it either, since the provider serves the service on
its own domain, so the probe failed with an unrecognised TLS name.
Domains of an enabled and shared service are now skipped as well.

The only other entity with `domains` (web-app-seaweedfs frontend) is
never shared, so its behaviour is unchanged.

Refs #1165

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 17:16:19 +02:00
kevinveenbirkenbachandClaude Opus 5.5 462f30e46a perf(web-app-stalwart): import Mailu in parallel sessions as a background unit
On echoserver the import appended 23,045 messages one by one over a single
IMAP session, about six per second, and the deploy waited 62 minutes for
it inside the Keycloak stage, because Stalwart is a dependency of Keycloak.
galaxyserver holds 157,655 messages in one mailbox, which one session
would need 7 to 28 hours for, beyond the task's six-hour timeout.

migrate_from_mailu.py now opens several IMAP sessions per mailbox that
take messages from one queue in file name order, with the dedup set and
the counters under a lock, and logs [PROGRESS] every 1000 messages. A
session Stalwart refuses only lowers the parallelism; with one session
the import behaves exactly as before. The number of sessions is
services.stalwart.migration.workers, or one fewer than the host's vCPUs,
at least 1 and at most 8.

The deploy starts the import as the transient unit stalwart-mailu-import
and does not wait for it; a deploy that finds the unit running leaves it
alone. The unit removes the staged recovery administrator credentials when
it stops, and a failed start removes them in a justified rescue before it
re-raises. Progress and the closing [DONE] rc=<n> are in
journalctl -u stalwart-mailu-import.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 16:35:45 +02:00
kevinveenbirkenbach e4ac05dfb6 Merge remote-tracking branch 'origin/main' into feature/updater 2026-10-23 10:46:59 +02:00
kevinveenbirkenbachandClaude Opus 5 677a63931c Merge branch 'feature/network/multi-network-serving' into feature/store-i18n
Resolves 68 conflicted files. The governing rule was our structure with their
network semantics: the tor boolean becomes a network mode (clearnet | tor |
multi) throughout the variant layer, while this branch's architecture and mesh
axes, its extracted modules and its i18n content stay.

Where the two sides covered different ground, both are kept rather than one
picked:

- ca_injected answers a ca_client on the deployment's TLS mode OR on an SSO
  issuer that resolves self-signed. 3a3e9ed2c9 records why: an issuer is not
  an object store, so keeping only one side drops the other's coverage.
- get_merged_domains keeps the loud refusal when svc-net-tor is deployed
  without a provisioned node, now gated on their tor_serves_host; their
  node_onion_of returns an empty string there, which would have served every
  onion-exclusive app on clearnet in silence.
- sso_cookie_domains stays the single source for oauth2-proxy's cookie scope.
  Their per-network branch omitted the issuer domain on the onion arm, which
  breaks the callback leg.
- The clearnet CSP batch carries --clearnet-timeout again; their batch dict
  hardcoded zero while the role still passes the flag.
- The dashboard keeps its i18n content dir, image, version and design
  resolution; their side reduced the logo sources to the bare service
  provider values.

Their side wins where it is measurably better: the onion mirror rewrite is
gated per vhost rather than per app, the Playwright onion decision reads the
pass's own domain instead of the role's canonical one, and the matomo
reinjection and dashboard config render per network sibling.

Eight files were broken by the merge without ever carrying a conflict marker,
because git marks only the lines both sides touched:

- utils/cache/domains.py lost the onionize block that sat between two hunks
- roles/sys-front-inj-all/templates/body_filter.lua.j2 called it
- web-app-bluesky, keycloak sso_proxy: the get_entity_name filter was renamed
- trigger/__main__.py, playwright_coverage.py and its test: the deleted tor
  module
- cli/administration/deploy/ci/timings.py read Label.tor
- plugins/filter/csp_filters.py called _aligned_url with four of six arguments
- tests/unit/python/utils/cache/test_domains.py referenced a renamed class
- tests/.../sys-ctl-hlth-csp/files/test_script.py mocked a function main no
  longer calls

Verified: the variant layer (155 tests, 65 subtests), ca_injected (18),
the CI layer (175), the domains cache including the restored guard (32),
retrigger inputs (9), timings and the CSP health script (53). make
quality-high clears autoformat; its remaining findings are tracked separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-23 09:53:41 +02:00
kevinveenbirkenbachandClaude Opus 5.5 bd8f2b4f7f test(svc-opt-ssd-hdd): derive the registry domain in the image classification cases
The classification cases main brought in with 8567cb6992 spell the
registry host as registry.infinito.test. On this branch the
hardcoded-primary-domain lint also scans the test trees, so test-lint
failed on both lines after the main merge.

The cases now build the host from default_domain_primary(), the same
module-level DOMAIN the other domain-aware unit tests use. is_database()
reads only the last path segment of the image reference, so the expected
classification is the same for any domain.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 01:47:22 +02:00
kevinveenbirkenbach 5c989668c7 Merge remote-tracking branch 'origin/main' into feature/network/multi-network-serving 2026-10-23 01:33:24 +02:00
kevinveenbirkenbach e53751402f Merge pull request #1114 from infinito-nexus/fix/security/swarm-ingress-guard/1089
fix(svc-swarm-node): drop outside traffic to non-public routing-mesh ports
2026-10-23 01:03:04 +02:00
kevinveenbirkenbachandClaude Opus 5.5 bfdf88fae0 fix(domains): read an application's domains where the cache puts them
meta/domains.yml lands on the top-level `domains` key of the merged
application, yet three readers still looked under `server.domains`,
which no role fills any more:

- current_play_redirect_domains built every mapping from an empty
  config, so each app redirected to itself and was dropped. Neither
  `domains.aliases` nor AUTO_BUILD_ALIASES produced a single redirect,
  and echoserver's update check listed nextcloud.infinito.nexus and the
  other entity domains as dropped by the upgrade.
- csp_skip_domains never found the canonical of a disabled service.
- the logout filter's fallback returned no domains.

The unit tests fed the old layout and passed regardless. They now use
the merged one, and a new test builds the redirect from keycloak's real
role defaults.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 22:18:07 +02:00
kevinveenbirkenbachandClaude Opus 5 5b6666e70c Merge branch 'main' into feature/store-i18n
Conflicts resolved:
- Dockerfile: keep HEAD; filesystem.sh already runs at line 37 and pin_epel()
  arrived inside centos_mirror.sh, so main's hunk would have run it twice.
- peertube 02_fix-application-schema.yml: take main's task name, its
  expect_rows: true (matching the first query in the same file) and its when:
  without default(); keep HEAD's changed_when: true, which the notify:
  compose-up depends on.
- peertube check-application/defaults.sql: take main's information_schema
  query; HEAD's ::regclass cast raises instead of returning no rows when the
  table is absent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-22 21:40:06 +02:00
kevinveenbirkenbach 3cf840c72b Merge remote-tracking branch 'origin/main' into feature/updater 2026-10-22 21:14:49 +02:00
kevinveenbirkenbachandClaude Opus 5.5 8567cb6992 fix(svc-opt-ssd-hdd): recognise database images behind registry, namespace and custom builds
is_database() compared the whole repository part of the image reference
against postgres, mariadb, redis, memcached and mongo. Every reference
with a registry or namespace prefix, every derivative image and every
locally built `<engine>_custom` image therefore counted as application
data and went to the mass storage. On a local stack the central
databases report `postgres_custom:18` and `openldap_custom:13.7-slim`,
both classified False and moved to the HDD.

The check now takes the last path component of the reference, drops
digest, tag and a `_custom` suffix, and compares it against the engines
the svc-db-* providers and the app-local databases run: elasticsearch,
mariadb, memcached, mongo, mysql, openldap, opensearch, postgis,
postgres, qdrant, rabbitmq, redis, typesense, valkey.

Volumes the script already symlinked stay where they are; a database
volume misplaced on the HDD has to be moved back by hand.

Fixes #1163

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 21:02:06 +02:00
kevinveenbirkenbachandClaude Opus 5.5 558784e6ca feat(web-app-erpnext): set up the service provider's company and its accounting basics
The wizard bypass left ERPNext without a Company, so no app could create quotes, orders or invoices. The deploy now creates the Company from the service provider's name and country, with the currency from ERPNext's country data and an abbreviation that defaults to the name's initials, plus a fiscal year, five tax accounts, a cash account and the default company. It never rewrites an existing Company and fails loudly on a country ERPNext does not know. The README generator marks roles that read the service provider country, and the CI replay fills the placeholder.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 20:52:30 +02:00
kevinveenbirkenbachandClaude Opus 5.5 286004349a fix(web-app-suitecrm): upgrade the code in the data volume to the pinned release
The entrypoint stamped an installed volume with the image's version and kept the old tree, so a newer pin never reached it and the CreateUser patch stayed out. On boot it now runs SuiteCRM's own suitecrm:app:upgrade and upgrade-finalize with the image's patched tree as the package, resumes an interrupted upgrade, stamps the version the tree really has, and refuses a tree newer than the pin.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 20:44:13 +02:00
kevinveenbirkenbachandClaude Opus 5 cbb336ed7d Merge branch 'main' into feature/store-i18n
Eight conflicts, none resolved by taking a side wholesale.

compose_file_args.py: main's removal of the dead templar parameter kept, its _value_has_domain/_has_domain helpers dropped. They served only the inlined CA check that the ca_injected lookup replaces, and the call site already carried main's signature.

web-app-penpot/01_administrator.yml: main's side. PR #1170 moves the admin password from the environment and sh -lc into stdin, and the test_quote_in_shell lint it ships enforces that.

web-app-mattermost/meta/services.yml: this side. The source build exists so the image is multi-arch; main's bump to mattermost/mattermost-team-edition:12.0.1 would drop that silently. The version lag is a follow-up.

web-app-bigbluebutton: main's b83b361ea9 runs BBB v3.0.39, where upstream replaced etherpad with bbb-shared-notes-server, so the ep_redis_publisher patch, the postgres entrypoint and the initdb hook go with their files. html5_settings and its render task stay: git log -S shows main carries no commit for them, they came from 76e9441117 on this side. vars/main.yml takes main's BBB_PINNED_SERVICES with this side's entity_name filter, because get_entity_name no longer exists here. The README keeps this side's feature bullets with main's etherpad-free wording; main's Cosmos section is dropped, the branch does not carry that section.

tests/integration/iam/passwords/test_quote_in_shell.py: this side's STDIN_BLOCK_RE, which the merged file uses at line 116.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-22 19:34:17 +02:00
kevinveenbirkenbach eeb3305e52 Merge remote-tracking branch 'origin/main' into fix/security/swarm-ingress-guard/1089 2026-10-22 19:17:39 +02:00
kevinveenbirkenbach 1ad4f0ac39 Merge branch 'main' into feature/updater 2026-10-22 18:59:45 +02:00
kevinveenbirkenbach 17918992c7 Merge branch 'main' into feature/network/multi-network-serving
# Conflicts:
#	plugins/lookup/compose_file_args.py
#	roles/web-app-dashboard/tasks/02_manager_ops.yml
#	roles/web-app-dashboard/vars/main.yml
2026-10-22 18:52:10 +02:00
kevinveenbirkenbachandClaude Opus 5.5 d2ef619faa fix(web-app-matomo): install the image's release into the data volume on upgrade
The upstream entrypoint copies the release only into an empty volume, so a newer pinned image kept running the old code from matomo_data. The deploy now copies the release over the volume, prunes the files the release dropped from its own trees, runs core:update, and recreates the security files after a code change. A volume newer than the image stops the deploy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 18:51:32 +02:00
kevinveenbirkenbachandClaude Opus 5.5 b83b361ea9 fix(web-app-bigbluebutton): run BBB v3.0.39 with the shared-notes server
The version-source update in #1146 moved the bigbluebutton/docker ref from
v3.0.23 to v3.0.39. That tag changes three things the role did not
handle, and it exposed a fourth defect in compose_file_args.

1. generate-compose renders etherpad and bbb-pads only behind
   USE_INSECURE_ETHERPAD and otherwise emits bbb-shared-notes-server with
   a `build:` block and no `image:` (upstream adds the image only after
   the tag, in 9a6a9a9a4c). compose_mods drops `build` only where an image
   is set, so the guard "Fail if any service still builds locally" stopped
   every BigBlueButton deploy (main run 39749899667, Nextcloud#11:
   `rc: 0`, `stdout: '    build:'`, bootstrap_compose_project.yml:35).
   meta/services.yml now pins alangecker/bbb-shared-notes-server:v3.0.39,
   compose_mods takes `pinned` (BBB_PINNED_SERVICES, every service the
   role config pins an image for) and drops their `build`, and the
   etherpad leftovers go (ep_redis_publisher patch, etherpad postgres
   entrypoint and initdb hook, etherpad database tasks, ETHERPAD_* env,
   etherpad_api_key secret).

2. The notes server connects to postgres once at start and exits with 2
   when it is not up yet; upstream gives it no restart policy. The
   override sets the role's restart policy, so it retries until postgres
   accepts connections.

3. Greenlight moves from v3.8.0 to v3.9.0.1, which sets
   `config.hosts = URL_HOST`, and the template passes
   `URL_HOST: https://${DOMAIN}`. Rails compares hostnames, so every
   request was blocked (`Blocked hosts: bbb.talk.infinito.test`, 403).
   Greenlight documents URL_HOST as the bare hostname; the override sets
   it to the role domain.

4. compose_file_args decided whether a role ships a compose override by
   templating "{{ application_id | abs_role_path_by_application_id }}"
   through the templar. ansible-core 2.21 does not render an untrusted
   string, so the check saw the literal and always returned False: the CA
   override generation and the pull handler never read BBB's
   compose.override.yml. That was harmless while compose.yml alone was
   valid; with the notes-server image living in the override, `compose
   config` failed ("has neither an image nor a build context"). The check
   now resolves the role path for the application id it is given. BBB is
   the only role that ships an override, so no other role's file list
   changes.

Existing installations drop the etherpad and bbb-pads containers on the
next `compose up --remove-orphans`; the old `etherpad` database stays in
PostgreSQL untouched.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 16:56:48 +02:00
kevinveenbirkenbachandClaude Opus 5.5 77349c73e1 style(web-app-stalwart, web-app-joomla): apply the formatter and its lint rules
The upgrade work only ran make test, never make autoformat, so ruff now
reformats the new Stalwart and Joomla code and reports five rules it
cannot fix itself: Path.readlink instead of os.readlink (with Path moved
into the type-checking block), list.extend for a collecting loop, no
manual "Z" replacement before datetime.fromisoformat, and an S310 opt-out
for the two requests to the fixed internal JMAP origin.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 11:40:43 +02:00
kevinveenbirkenbachandClaude Opus 5.5 05135a7cee feat(svc-net-wireguard): let a compose deployment name a hub with roaming spokes
Without a swarm manager the mesh planner only built full meshes, whose
handshake gate waits for every peer, so laptops behind NAT could not join.
An inventory may now name a hub in networks.hub: the planner builds every
declared mesh around it, the hub forwards between its spokes, a spoke marked
networks.roaming gets no endpoint in the hub's peer list and dials the hub
with PersistentKeepalive, and the hub's gate skips roaming spokes while a
roaming spoke gates only on the hub. The mesh CLI takes the hub's public
address per host (--endpoint) and a vault password per host
(--host-vault-password-file). Full meshes and the swarm manager hub plan
exactly as before; NETWORK_MESH_POOL keeps the operator's subnet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 10:56:50 +02:00
kevinveenbirkenbachandClaude Opus 5.5 8701ddb769 feat(web-app-stalwart): import every Mailu mailbox and provision aliases and forwards
The Mailu import covered only inventory users and logged in with their
passwords. It now reads every <address>/ maildir of the legacy volume
(files/python/mailu_maildir.py), logs in as the recovery administrator
acting for each account, creates missing accounts under the name the OIDC
directory resolves, keeps folders, flags, keywords and internal dates,
imports and activates each mailbox's Sieve script, merges mailboxes the
inventory maps onto another account (migration.mailbox_map), skips and
reports mailboxes of domains Stalwart does not serve, and imports nothing
twice on a re-run. The recovery credentials reach the script through a
0600 file, not argv. The import task allows 6 hours like the other long
data migrations, because a production mailbox of 157k messages does not
fit into one.

services.stalwart.aliases becomes Stalwart mailing lists, with @<domain>
as the domain's catch-all and destinations outside the served domains
allowed; services.stalwart.forwards becomes a redirect :copy Sieve script
chained with the user's own. Both are provisioned on every deploy and only
touch what the role created.

The test harness groups its probes under files/test/probe/ and its
scenarios under files/test/verify/.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 10:53:11 +02:00
kevinveenbirkenbachandClaude Opus 5.5 d06fc7bd28 feat(web-app-mediawiki): refresh the extensions volume when MediaWiki changes version
Docker seeds mediawiki_extensions from the image only once, so after an
upgrade the bundled extensions of the old release kept running against the
new core. On every deploy a one-shot container of the pinned image runs
files/shell/refresh_extensions.sh against the volume: it adopts a volume
whose content matches the image, replaces the content with the image's
extensions when the recorded version differs, records the version in a
marker, and refuses when the volume records a newer version than the image.
After a refresh the role reinstalls its add-on extensions and runs
update.php.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 10:46:19 +02:00
kevinveenbirkenbachandClaude Opus 5.5 eddbccd63c feat(web-app-confluence, web-app-jira): refuse a pinned image older than the data
Atlassian applications refuse to start on data a newer build wrote, after
the old container is gone. Before the stack starts, each role compares
the build number of the data with the build of the pinned image
(files/python/build_guard.py) and stops the deploy with both builds named
when the data is newer. Confluence reads the build from the home in its
volume; Jira reads it from the database (files/sql/data_build.sql), so its
guard needs the database reachable before the stack starts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 10:38:11 +02:00