Commit Graph
555 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5.5 76b16a8abb perf(library): create a whole directory list in one module call
Directory loops over ansible.builtin.file cost one SSH round trip per path: the openresty config tree (20 paths) took up to 12.9 s, the webserver core directories (16) up to 13.2 s, every app's compose directories (5) 2.5-5 s per app, the shared CDN roots 5.8 s and the OpenLDAP LDIF dirs 2.2 s on echoserver.

directory_batch takes entries (path plus its own mode, owner, group) or paths with one shared set, and handles each path like file with state: directory: symlinks are followed, missing parents get the same attributes, an attribute left out stays as found, and it never recurses. It reports changed_paths, supports check and diff mode, and fails on a relative path, an existing non-directory or a failed mkdir. It replaces the five loops and the earlier install -d of the CDN role directories, which now report changed honestly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 10:51:18 +02:00
kevinveenbirkenbachandClaude Opus 5.5 1fbd4758e3 perf(web-app-stalwart): provision accounts and bot app passwords in batched JMAP calls
Every account ran about six JMAP calls of three Ansible tasks each through jmap_call.yml, 80 s for 7 accounts on echoserver. Each phase now sends one request for all accounts: one x:Account/set creates them all (ids from created or, on alreadyExists, from notCreated.objectId), one updates the passwords of the accounts that already existed, and the app-password phases (list, mint, retire, drop) carry one method call per bot, each addressing its bot through accountId, in chunks of STALWART_JMAP_MAX_METHOD_CALLS (Stalwart's default maxMethodCalls, 16).

jmap_call.yml retries and asserts on every method response of a batch, and a refusal names its owner through the call id. A create refused for a reason other than alreadyExists now fails the deploy instead of being skipped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 10:37:01 +02:00
kevinveenbirkenbachandClaude Opus 5.5 3b4727fcfe fix(web-app-stalwart): rate-limit and ban HTTP clients by their own address
With the role network's gateway on Stalwart's allowlist, every request OpenResty proxies arrives from an address that is never throttled or banned, so the anonymous HTTP limit and the scan and auth bans no longer applied to web clients at all. Stalwart can take the client address from a Forwarded header, but OpenResty passed a client-supplied one through, so trusting it would have let anyone pick the address that gets banned.

forwarded.conf.j2 gains an opt-in proxy_forwarded_client_address switch that sets Forwarded: for=$remote_addr on every location including the healthz ones, replacing any value the client sent; other vhosts render unchanged. The Stalwart vhost opts in, and the role sets useXForwarded through x:Http. The settings apply moved into security/apply.yml behind one include so the guarded block stays within the three-task limit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 10:17:42 +02:00
kevinveenbirkenbachandClaude Opus 5.5 57eb64e0af fix(cache): render a templated DOMAIN_PRIMARY against the play variables
get_merged_domains rendered DOMAIN_PRIMARY with no templar and an empty variable scope. A primary declared as a template, such as echoserver's "{{ SOFTWARE_NAME | lower }}", therefore came out as 'none' inside the applications SPOT, while direct lookups saw the domain Ansible had already resolved. Every SPOT value built through it named file.none, and the dashboard waited for https://file.none/assets/img/logo.png until full-2 failed. The render now gets the templar and the play variables, which also covers services.tor.node.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 00:40:32 +02:00
kevinveenbirkenbachandClaude Opus 5.5 47160d755e fix(tls): serve every vhost the certificate of its own domain
ssl_credentials.j2 resolved the certificate by application_id, so every vhost of an application got the lineage of its primary domain. echoserver serves webmail.infinito.nexus the CN=academy.infinito.nexus certificate that names mail.infinito.nexus but not webmail, and browsers refuse it with a hostname mismatch. The template and the letsencrypt existence check now resolve by the vhost's domain, so a secondary vhost points at its own live/<domain> link and a missing link triggers the issuance instead of a mismatch. Primary vhosts resolve to the same files as before in every TLS mode and flavor.

The LDAP stream passes its primary domain explicitly. Onion domains stay on the application lineage, and the app-scoped self-signed path always names the application leaf, because neither has a per-domain certificate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 23:41:33 +02:00
kevinveenbirkenbachandClaude Opus 5.5 ef246a6e03 refactor(unit_name): build unit names from SYS_SERVICE_VERSION_SUFFIX
The lookup assembled '.<version>.<software domain>' itself, a second copy of the infix the unlock and prune phases now read from the SPOT. It reads SYS_SERVICE_VERSION_SUFFIX instead and refuses one that does not start with a dot or carries an empty segment. The direct version-plugin fallback goes with it: it existed because the plugin rendered the untrusted literal "{{ lookup('version') }}", while the SPOT arrives as a trusted group_vars string. Every unit name rendered from group_vars is byte-identical before and after.

The timer description names the versioned service it starts instead of the legacy unversioned name.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 23:34:13 +02:00
kevinveenbirkenbachandClaude Opus 5.5 694007c58a feat(sys-svc-cln-domains): remove the nginx configs of domains the host no longer serves
Retired apps, aliases and legacy zones left their vhosts behind, so 37 legacy cymais.cloud subdomains kept serving a certificate that does not name them after the zone moved to one wildcard redirect. With MODE_CLEANUP the role now deletes every <domain>.conf below the servers directory whose domain is not in CURRENT_PLAY_DOMAINS_ALL, the domains of every application in group_names and its dependencies including redirect sources; files not named after a domain and onion services stay.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 21:25:50 +02:00
kevinveenbirkenbach d0eaca8292 Merge remote-tracking branch 'origin/main' into feature/updater 2026-10-23 20:28:25 +02:00
kevinveenbirkenbachandClaude Opus 5.5 9875db5ff8 feat(web-app-stalwart): deliver mail of legacy alias domains to their target domain
services.stalwart.alias_domains maps a receive-only legacy domain onto the mail domain: the deploy sets the RCPT stage rewrite to one matches() rule per alias domain, reads it back, and with Cloudflare points the source's MX at the mail hostname. An empty list leaves Stalwart untouched.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:19:59 +02:00
kevinveenbirkenbachandClaude Opus 5.5 8c8e6a1242 feat(web-opt-rdr-domains): redirect a whole legacy zone with one wildcard source
redirect_domain_mappings now renders its source and target, so an inventory can map *.{{ OLD_DOMAIN }} and the apex onto a target; keep_path: false drops the request URI from the 301. Wildcards get no www. variant, sys-dns-wildcards adds the *.<zone> A/AAAA records for every zone with a wildcard source next to DOMAIN_PRIMARY, and the CSP health check skips wildcard names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:17:19 +02:00
kevinveenbirkenbachandClaude Opus 5.5 9152c884d8 fix(tls): serve every redirect source the certificate that names it
The cert lookup handed an alias or redirect source the live directory of its app's canonical domain or of its target, so social.cymais.cloud got the infinito.nexus lineage and failed with 'Hostname mismatch'. With the san and dedicated flavors a domain term now resolves to its own live/<domain>, which cert_link_domains points at the lineage that covers it; application terms keep the canonical path. resolve_term follows redirect chains and fails on a cycle.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:14:21 +02:00
kevinveenbirkenbachandClaude Opus 5.5 27762376f0 perf(sys-ctl-hlth-webserver): probe the domains in parallel
The checker sent one HEAD request after the other, up to 10 s per clearnet and 30 s per onion domain. A pool of 16 threads now probes them; the waits are I/O, so 16 suit hosts with any core count, and pool.map keeps the output sorted by domain.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:10:52 +02:00
kevinveenbirkenbachandClaude Opus 5.5 fb18077f57 perf(utils): key the domain index by the applications' domains only
build_domain_index fingerprinted the whole applications tree on every call, 77 % of a tls lookup. The index reads only each application's domains, so the cache key now covers exactly those; a tls lookup drops from 8.2 to 3.1 ms.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:03:40 +02:00
kevinveenbirkenbachandClaude Opus 5.5 8e73c84b15 perf(svc-db-openldap): converge the user entries over one connection
Three loops started one async ldap_entry or ldap_attrs job per user and reaped them, about 2 minutes per deploy on echoserver. The ldap_entries module binds once and applies create, present or exact to every entry the new filters build from OPENLDAP_USERS, keeping the semantics of the community modules it replaces.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 19:59:37 +02:00
kevinveenbirkenbachandClaude Opus 5.5 5b6db5dc24 feat(web-app-stalwart): import Mailu only until one import has finished
The import script writes a done marker into the instance directory when it ends with rc 0, and the deploy skips the unit and the import block while the marker exists; deleting it imports again. The IMAP layout helpers move to imap_layout.py so the script stays under the 500-line limit, with mirrored tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 19:51:10 +02:00
kevinveenbirkenbachandClaude Opus 5.5 43ce74236e perf(web-app-dashboard): probe every Simple Icons slug in one parallel call
uri_retry looped over each card slug, 60 to 64 s per probe on echoserver. The new uri_probe_batch module asks all URLs through a thread pool with per-URL retries and fails listing every URL that gave no expected status.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 19:37:57 +02:00
kevinveenbirkenbach 0bca4bcd7f Merge remote-tracking branch 'origin/main' into fix/web-app-matrix/ansible-flavor-unwired-services/1165 2026-10-23 18:18:31 +02:00
kevinveenbirkenbachandClaude Opus 5.5 25ec8b0c59 fix(sys-ctl-hlth-webserver): skip a consumer's domains for a shared service
A service entity can list canonical domain keys under `domains`; the
web health expectations skipped them only while the service was
disabled. A consumer whose service is enabled but shared does not serve
its own domain for it either, since the provider serves the service on
its own domain, so the probe failed with an unrecognised TLS name.
Domains of an enabled and shared service are now skipped as well.

The only other entity with `domains` (web-app-seaweedfs frontend) is
never shared, so its behaviour is unchanged.

Refs #1165

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 17:16:19 +02:00
kevinveenbirkenbachandClaude Opus 5.5 462f30e46a perf(web-app-stalwart): import Mailu in parallel sessions as a background unit
On echoserver the import appended 23,045 messages one by one over a single
IMAP session, about six per second, and the deploy waited 62 minutes for
it inside the Keycloak stage, because Stalwart is a dependency of Keycloak.
galaxyserver holds 157,655 messages in one mailbox, which one session
would need 7 to 28 hours for, beyond the task's six-hour timeout.

migrate_from_mailu.py now opens several IMAP sessions per mailbox that
take messages from one queue in file name order, with the dedup set and
the counters under a lock, and logs [PROGRESS] every 1000 messages. A
session Stalwart refuses only lowers the parallelism; with one session
the import behaves exactly as before. The number of sessions is
services.stalwart.migration.workers, or one fewer than the host's vCPUs,
at least 1 and at most 8.

The deploy starts the import as the transient unit stalwart-mailu-import
and does not wait for it; a deploy that finds the unit running leaves it
alone. The unit removes the staged recovery administrator credentials when
it stops, and a failed start removes them in a justified rescue before it
re-raises. Progress and the closing [DONE] rc=<n> are in
journalctl -u stalwart-mailu-import.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 16:35:45 +02:00
kevinveenbirkenbachandClaude Opus 5.5 cfb2f9c70b perf(web-opt-rdr-domains): check every certificate in one task and write every file in one call
Each redirect source went through the full TLS layer on its own: cert check,
CA environment cleanup, re-check, HTTP handler and vhost template, about
nine seconds per source. echoserver has 85 sources, so the role took
thirteen minutes of every deploy.

The role now resolves the tls state and renders both files of every source
on the controller, finds the certificates of all sources in one find, and
includes sys-front-tls only for a source whose certificate is missing, so
issuance still works in every mode. library/stack_host_contents.py writes
all files in one module call, atomically and only where the content
differs, and keeps one file per source under http/ and https/. Its action
plugin carries the IS_STACK_HOST gate like stack_host_template.

The name differs from the stack_host_files module of feature/store-i18n on
purpose: that one syncs a single directory of YAML documents and prunes
what it does not write, which would delete every other vhost here.

HTTPS_REDIRECT_ENABLED is gone: nothing set it, it was true everywhere,
and the redirect role could no longer see the sys-front-tls default it
lived in. Whether a domain gets its HTTP to HTTPS handler now follows from
lookup('tls', domain, 'enabled') alone, in sys-front-tls and here.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 16:35:45 +02:00
kevinveenbirkenbachandClaude Opus 5.5 285ce910d2 feat(inventory): reject unread keys and a missing Cloudflare token before the deploy
The Keycloak stage on echoserver ran ten minutes before sys-svc-dns stopped
it with "API.cloudflare.api_token must be set": the inventory still kept the
token under the flat name CLOUDFLARE_API_TOKEN, which nothing reads any
more. The validator that runs before every deploy now catches that class.

- api.py reports a host that selects DNS_PROVIDER: cloudflare without
  API.cloudflare.api_token, with group_vars/all merged under the host_vars
- variables.py reports every top-level key of host_vars and group_vars that
  core does not read: not declared in group_vars or a role's defaults or
  vars, not tested with "is defined" or "| default(", not read by name in a
  plugin, and not a SYSTEM_EMAIL_<KEY> the email lookup builds; a flat
  credential name such as CLOUDFLARE_API_TOKEN names its API.<provider>.<key>
  path; ansible_* keys are exempt

A key that is read in a way the scan cannot see takes
"# nocheck: inventory-variable", documented in suppression.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 16:18:12 +02:00
kevinveenbirkenbachandClaude Opus 5.5 c728b4427b fix(tls): resolve a redirect source through its target
With AUTO_BUILD_ALIASES enabled, current_play_redirect_domains adds
<entity>.<DOMAIN_PRIMARY> as a redirect source for every application whose
canonical domain differs. web-opt-rdr-domains then asks the tls and cert
lookups about each source, but resolve_term only knew the domains mapping
and applications[*].domains, so the Keycloak stage on echoserver stopped
with "tls: domain 'funkwhale.infinito.nexus' not found".

redirect_targets() turns CURRENT_PLAY_REDIRECT_DOMAINS into a source to
target map, and resolve_term falls back to it, one hop, only where it
raised before, so every term that resolved keeps its answer. The tls and
cert lookups pass the map.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 16:08:18 +02:00
kevinveenbirkenbachandClaude Opus 5.5 a32f241051 fix(sys-svc-certs): link each domain to the certbundle lineage that covers it
certbundle names its lineages certbundle-<hash>-<n>, while the cert lookup
reads live/<domain>/fullchain.pem. Since 16023dbc9b removed cert_folder_find,
nothing joined the two, so every letsencrypt deploy with the SAN flavor
stopped at "Let's Encrypt certificate files not found after issuance".

library/cert_link_domains.py points live/<domain> at the newest lineage
whose SAN covers the domain, using the same CertUtils lookup as
cert_check_exists, and swaps the link atomically. A real lineage directory
is left alone, and a domain without a certificate is only reported, so the
letsencrypt re-check still fails loudly on it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 16:05:27 +02:00
kevinveenbirkenbachandClaude Opus 5 3cc0214c28 fix(deploy): stop the dedicated deploy from git-cleaning its own checkout
`MODE_CLEANUP` says what to tidy on the deployed system: prune unused Docker
resources, clear Mastodon caches, switch an LDAP user list from `present` to
`exact`. Eight roles read it that way. The dedicated runner read the same flag
and ran `make clean` in the control repository instead, which is
`git clean -fdX` over every ignored path there.

`inventories/production/` is one of those paths. The published host install
guide provisions into it and then deploys against `$INVENTORY/.password`, so
the deploy deleted the vault password it was about to use:

    Removing inventories/production/
    [ERROR] The vault password file .../inventories/production/.password was
    not found

Reproduced in CI on two unrelated rows, arm64/zfs/debian and
amd64/btrfs/centos, so it is neither architecture nor filesystem specific.

The coupling was a name collision, not a decision. 7e5990aa16 generated the
MODE_* flags from `01_modes.yml` and removed the legacy skip flags, and the
repository clean - which had had a flag of its own - was attached to the
generated `MODE_CLEANUP` because one now existed by that name. No other deploy
path does this; `container`, `development` and `swarm` never call `make` at
all.

`make clean` keeps its own entry points, `make setup-clean`, `make clean-sudo`
and `infinito --git-clean`, where a caller asks for it deliberately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-23 10:59:05 +02:00
kevinveenbirkenbach e4ac05dfb6 Merge remote-tracking branch 'origin/main' into feature/updater 2026-10-23 10:46:59 +02:00
kevinveenbirkenbach e53751402f Merge pull request #1114 from infinito-nexus/fix/security/swarm-ingress-guard/1089
fix(svc-swarm-node): drop outside traffic to non-public routing-mesh ports
2026-10-23 01:03:04 +02:00
kevinveenbirkenbachandClaude Opus 5.5 bfdf88fae0 fix(domains): read an application's domains where the cache puts them
meta/domains.yml lands on the top-level `domains` key of the merged
application, yet three readers still looked under `server.domains`,
which no role fills any more:

- current_play_redirect_domains built every mapping from an empty
  config, so each app redirected to itself and was dropped. Neither
  `domains.aliases` nor AUTO_BUILD_ALIASES produced a single redirect,
  and echoserver's update check listed nextcloud.infinito.nexus and the
  other entity domains as dropped by the upgrade.
- csp_skip_domains never found the canonical of a disabled service.
- the logout filter's fallback returned no domains.

The unit tests fed the old layout and passed regardless. They now use
the merged one, and a new test builds the redirect from keycloak's real
role defaults.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 22:18:07 +02:00
kevinveenbirkenbachandClaude Opus 5.5 96fe8bd7e6 feat(playbooks/update): pull the images of the deploy before the upgrade window
pull.yml pulls, on the target host, every pinned image of the applications the host deploys, the same closure the update check judges, and lists the images a role builds itself instead. It never touches a container, retries each pull a bounded number of times and fails loudly with the image and the registry error. The update_check filter moves next to the new update_pull filter into plugins/filter/update/, and both read the code pins from one helper.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 21:30:18 +02:00
kevinveenbirkenbach 3cf840c72b Merge remote-tracking branch 'origin/main' into feature/updater 2026-10-22 21:14:49 +02:00
kevinveenbirkenbach 4a86376aeb Merge remote-tracking branch 'origin/main' into feature/updater 2026-10-22 21:08:26 +02:00
kevinveenbirkenbachandClaude Opus 5.5 8567cb6992 fix(svc-opt-ssd-hdd): recognise database images behind registry, namespace and custom builds
is_database() compared the whole repository part of the image reference
against postgres, mariadb, redis, memcached and mongo. Every reference
with a registry or namespace prefix, every derivative image and every
locally built `<engine>_custom` image therefore counted as application
data and went to the mass storage. On a local stack the central
databases report `postgres_custom:18` and `openldap_custom:13.7-slim`,
both classified False and moved to the HDD.

The check now takes the last path component of the reference, drops
digest, tag and a `_custom` suffix, and compares it against the engines
the svc-db-* providers and the app-local databases run: elasticsearch,
mariadb, memcached, mongo, mysql, openldap, opensearch, postgis,
postgres, qdrant, rabbitmq, redis, typesense, valkey.

Volumes the script already symlinked stay where they are; a database
volume misplaced on the HDD has to be moved back by hand.

Fixes #1163

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 21:02:06 +02:00
kevinveenbirkenbachandClaude Opus 5.5 558784e6ca feat(web-app-erpnext): set up the service provider's company and its accounting basics
The wizard bypass left ERPNext without a Company, so no app could create quotes, orders or invoices. The deploy now creates the Company from the service provider's name and country, with the currency from ERPNext's country data and an abbreviation that defaults to the name's initials, plus a fiscal year, five tax accounts, a cash account and the default company. It never rewrites an existing Company and fails loudly on a country ERPNext does not know. The README generator marks roles that read the service provider country, and the CI replay fills the placeholder.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 20:52:30 +02:00
kevinveenbirkenbachandClaude Opus 5.5 286004349a fix(web-app-suitecrm): upgrade the code in the data volume to the pinned release
The entrypoint stamped an installed volume with the image's version and kept the old tree, so a newer pin never reached it and the CreateUser patch stayed out. On boot it now runs SuiteCRM's own suitecrm:app:upgrade and upgrade-finalize with the image's patched tree as the package, resumes an interrupted upgrade, stamps the version the tree really has, and refuses a tree newer than the pin.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 20:44:13 +02:00
kevinveenbirkenbach b40c05dbbf Merge pull request #1177 from infinito-nexus/fix/install/shfmt-test-docstring
fix(install): drop the shfmt docstrings that misstate ensure()
2026-10-22 20:41:38 +02:00
kevinveenbirkenbachandClaude Opus 5.5 091c58307d fix(install): drop the shfmt docstrings that misstate ensure()
Three docstrings from #1173 claimed that ensure() leaves only the pinned
release on PATH or replaces whatever version PATH finds. It does neither:
it installs the release into /usr/local/bin and fails unless PATH then
resolves to it, so a distribution copy such as /usr/bin/shfmt stays on
disk behind it.

The test module and ensure() lose their docstrings; the module docstring
of the installer returns to the form its siblings use ("Install hadolint
via GitHub-release prebuilt binary.").

Reported by CodeRabbit on #1173.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 20:37:01 +02:00
kevinveenbirkenbachandClaude Opus 5.5 628e2587b8 fix(database_query): keep psql's command tags out of the returned rows
With `-tA` psql drops headers and the row-count footer but still prints
the command tag of every write statement. A guarded
`UPDATE … RETURNING` that matched nothing therefore came back as
`rows: [['UPDATE 0']]`, and web-app-peertube's oidc/enable.yml, which
judges its change by the row count, reported `changed` and restarted
PeerTube on every deploy after the first.

The PostgreSQL rows flags are now `-qtA`. Probed against a running
postgres container:

    -tA  -> CREATE TABLE / INSERT 0 1 / UPDATE 0 / 3 / UPDATE 1
    -qtA -> 3

SELECT output and the MariaDB path (`--batch --skip-column-names`, no
status line) are unchanged.

Fixes #1164

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 20:36:32 +02:00
kevinveenbirkenbach eeb3305e52 Merge remote-tracking branch 'origin/main' into fix/security/swarm-ingress-guard/1089 2026-10-22 19:17:39 +02:00
kevinveenbirkenbachandClaude Opus 5.5 a9a12da75c fix(install): install the pinned shfmt release on every distro
`shfmt.ensure()` returned as soon as any `shfmt` was on PATH and tried
the distribution package before the release pinned in
roles/sys-lint/meta/services.yml. The Debian dev container therefore
formats with its packaged 3.8.0 (/usr/bin/shfmt), Fedora with its packaged
3.7.0, the CentOS one with the pinned v3.14.1 (/usr/local/bin/shfmt), and
they disagree on
scripts/tests/deploy/swarm/utils/unmount/nfs_mounts.sh: every gate in a
CentOS stack moved the `then` of the heredoc `if` below the terminator.

ensure() now follows ruff's installer: it reads `shfmt --version`, keeps
the binary only when it reports the pinned release, and otherwise installs
the release binary into /usr/local/bin. It fails when PATH still resolves
another version afterwards instead of formatting with it. The system
package path is gone.

nfs_mounts.sh is committed in the pinned release's format (byte-identical
to the v3.14.1 output from the CentOS container), so the first gate on any
distro leaves the tree clean.

The install-lint stamp now also depends on roles/sys-lint/meta/services.yml,
the pin file of actionlint, hadolint and shfmt, so a bumped pin reaches
environments whose stamp predates it; until now only pyproject.toml and
scripts/install/lint.sh invalidated it.

Fixes #1155

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 19:16:22 +02:00
kevinveenbirkenbach 1ad4f0ac39 Merge branch 'main' into feature/updater 2026-10-22 18:59:45 +02:00
kevinveenbirkenbachandClaude Opus 5.5 d2ef619faa fix(web-app-matomo): install the image's release into the data volume on upgrade
The upstream entrypoint copies the release only into an empty volume, so a newer pinned image kept running the old code from matomo_data. The deploy now copies the release over the volume, prunes the files the release dropped from its own trees, runs core:update, and recreates the security files after a code change. A volume newer than the image stops the deploy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 18:51:32 +02:00
kevinveenbirkenbachandClaude Opus 5.5 b83b361ea9 fix(web-app-bigbluebutton): run BBB v3.0.39 with the shared-notes server
The version-source update in #1146 moved the bigbluebutton/docker ref from
v3.0.23 to v3.0.39. That tag changes three things the role did not
handle, and it exposed a fourth defect in compose_file_args.

1. generate-compose renders etherpad and bbb-pads only behind
   USE_INSECURE_ETHERPAD and otherwise emits bbb-shared-notes-server with
   a `build:` block and no `image:` (upstream adds the image only after
   the tag, in 9a6a9a9a4c). compose_mods drops `build` only where an image
   is set, so the guard "Fail if any service still builds locally" stopped
   every BigBlueButton deploy (main run 39749899667, Nextcloud#11:
   `rc: 0`, `stdout: '    build:'`, bootstrap_compose_project.yml:35).
   meta/services.yml now pins alangecker/bbb-shared-notes-server:v3.0.39,
   compose_mods takes `pinned` (BBB_PINNED_SERVICES, every service the
   role config pins an image for) and drops their `build`, and the
   etherpad leftovers go (ep_redis_publisher patch, etherpad postgres
   entrypoint and initdb hook, etherpad database tasks, ETHERPAD_* env,
   etherpad_api_key secret).

2. The notes server connects to postgres once at start and exits with 2
   when it is not up yet; upstream gives it no restart policy. The
   override sets the role's restart policy, so it retries until postgres
   accepts connections.

3. Greenlight moves from v3.8.0 to v3.9.0.1, which sets
   `config.hosts = URL_HOST`, and the template passes
   `URL_HOST: https://${DOMAIN}`. Rails compares hostnames, so every
   request was blocked (`Blocked hosts: bbb.talk.infinito.test`, 403).
   Greenlight documents URL_HOST as the bare hostname; the override sets
   it to the role domain.

4. compose_file_args decided whether a role ships a compose override by
   templating "{{ application_id | abs_role_path_by_application_id }}"
   through the templar. ansible-core 2.21 does not render an untrusted
   string, so the check saw the literal and always returned False: the CA
   override generation and the pull handler never read BBB's
   compose.override.yml. That was harmless while compose.yml alone was
   valid; with the notes-server image living in the override, `compose
   config` failed ("has neither an image nor a build context"). The check
   now resolves the role path for the application id it is given. BBB is
   the only role that ships an override, so no other role's file list
   changes.

Existing installations drop the etherpad and bbb-pads containers on the
next `compose up --remove-orphans`; the old `etherpad` database stays in
PostgreSQL untouched.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 16:56:48 +02:00
kevinveenbirkenbachandClaude Opus 5.5 3838114696 feat(playbooks/update): judge a host and its inventory against the current code
report.yml only dumped a host's state, so every comparison with the code
of the checkout was done by hand. It now also runs state.sh on the host,
which emits tab-separated records (volumes with their resolved storage
path, containers with their port bindings, listening sockets with the
container that owns them, docker subnets, data versions, server_name
lines, certificates, failed units, newest backups, disk usage), and the
update_check filter compares them with what lookup('applications'),
application_closure, compose_volumes, current_play_redirect_domains and
the nginx include directories derive from the code and the inventory.

The verdict lists every finding as OK, WARN or FAIL per area (inventory,
volumes, data, images, ports, networks, webserver, certificates,
operations), prints the warnings and failures, keeps the full list in
/tmp/infinito-check-<host>.txt and fails the play on a red check. The
host still needs no Python.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 12:50:03 +02:00
kevinveenbirkenbachandClaude Opus 5.5 77349c73e1 style(web-app-stalwart, web-app-joomla): apply the formatter and its lint rules
The upgrade work only ran make test, never make autoformat, so ruff now
reformats the new Stalwart and Joomla code and reports five rules it
cannot fix itself: Path.readlink instead of os.readlink (with Path moved
into the type-checking block), list.extend for a collecting loop, no
manual "Z" replacement before datetime.fromisoformat, and an S310 opt-out
for the two requests to the fixed internal JMAP origin.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 11:40:43 +02:00
kevinveenbirkenbach 2d7f842613 Merge branch 'main' of github.com:infinito-nexus/core into feature/updater 2026-10-22 11:27:47 +02:00
kevinveenbirkenbach cbb29f898d Merge remote-tracking branch 'origin/main' into fix/security/swarm-ingress-guard/1089 2026-10-22 11:22:34 +02:00
kevinveenbirkenbachandClaude Opus 5.5 05135a7cee feat(svc-net-wireguard): let a compose deployment name a hub with roaming spokes
Without a swarm manager the mesh planner only built full meshes, whose
handshake gate waits for every peer, so laptops behind NAT could not join.
An inventory may now name a hub in networks.hub: the planner builds every
declared mesh around it, the hub forwards between its spokes, a spoke marked
networks.roaming gets no endpoint in the hub's peer list and dials the hub
with PersistentKeepalive, and the hub's gate skips roaming spokes while a
roaming spoke gates only on the hub. The mesh CLI takes the hub's public
address per host (--endpoint) and a vault password per host
(--host-vault-password-file). Full meshes and the swarm manager hub plan
exactly as before; NETWORK_MESH_POOL keeps the operator's subnet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 10:56:50 +02:00
kevinveenbirkenbachandClaude Opus 5.5 8701ddb769 feat(web-app-stalwart): import every Mailu mailbox and provision aliases and forwards
The Mailu import covered only inventory users and logged in with their
passwords. It now reads every <address>/ maildir of the legacy volume
(files/python/mailu_maildir.py), logs in as the recovery administrator
acting for each account, creates missing accounts under the name the OIDC
directory resolves, keeps folders, flags, keywords and internal dates,
imports and activates each mailbox's Sieve script, merges mailboxes the
inventory maps onto another account (migration.mailbox_map), skips and
reports mailboxes of domains Stalwart does not serve, and imports nothing
twice on a re-run. The recovery credentials reach the script through a
0600 file, not argv. The import task allows 6 hours like the other long
data migrations, because a production mailbox of 157k messages does not
fit into one.

services.stalwart.aliases becomes Stalwart mailing lists, with @<domain>
as the domain's catch-all and destinations outside the served domains
allowed; services.stalwart.forwards becomes a redirect :copy Sieve script
chained with the user's own. Both are provisioned on every deploy and only
touch what the role created.

The test harness groups its probes under files/test/probe/ and its
scenarios under files/test/verify/.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 10:53:11 +02:00
kevinveenbirkenbachandClaude Opus 5.5 d06fc7bd28 feat(web-app-mediawiki): refresh the extensions volume when MediaWiki changes version
Docker seeds mediawiki_extensions from the image only once, so after an
upgrade the bundled extensions of the old release kept running against the
new core. On every deploy a one-shot container of the pinned image runs
files/shell/refresh_extensions.sh against the volume: it adopts a volume
whose content matches the image, replaces the content with the image's
extensions when the recorded version differs, records the version in a
marker, and refuses when the volume records a newer version than the image.
After a refresh the role reinstalls its add-on extensions and runs
update.php.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 10:46:19 +02:00
kevinveenbirkenbachandClaude Opus 5.5 eddbccd63c feat(web-app-confluence, web-app-jira): refuse a pinned image older than the data
Atlassian applications refuse to start on data a newer build wrote, after
the old container is gone. Before the stack starts, each role compares
the build number of the data with the build of the pinned image
(files/python/build_guard.py) and stops the deploy with both builds named
when the data is newer. Confluence reads the build from the home in its
volume; Jira reads it from the database (files/sql/data_build.sql), so its
guard needs the database reachable before the stack starts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 10:38:11 +02:00
kevinveenbirkenbachandClaude Opus 5.5 c65c5ce48c fix(update): pin only image tags the registry carries
7e82ba6d71 moved web-app-bigbluebutton's bbb-pads to v1.5.11, the tag
https://raw.githubusercontent.com/bigbluebutton/docker/develop/repos/tags
names for the bbb-pads source. alangecker/bbb-docker-pads has no such
image (Docker Hub: newest v1.5.8), so every BigBlueButton deploy pulls a
missing tag and test-external fails "These pinned Docker images do not
exist in their registry (HTTP 404): web-app-bigbluebutton/bbb-pads".

bbb-pads goes back to v1.5.8. The version-source updater now knows the
image a `version` pin tags (pulled directly or as the base of a custom
build) and withholds a bump when the registry answers 404 for the new
tag; an indeterminate answer does not block it, the same line
test-external draws. Run against main, the updater now reports no
update for bbb-pads instead of v1.5.11.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 10:30:10 +02:00