Commit Graph
568 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5.5 28ce14ffe7 style(web-app-keycloak,web-app-openproject): apply autoformat to the no-verify commits
quality-high's autoformat pass rewrote three files that 9301220d05,
49aa920034 and c78d5f906c committed without the hooks. ruff drops three
`noqa: E402` markers as unused (RUF100): E402 does not fire on an import
that follows sys.path.insert. The formatter wraps the long
KEYCLOAK_RBAC_GROUP_LOGIC_DIGEST line and the OpenProject admin runner's
`sh -c` line; the double-quoted scalar and the `>-` block both fold the
break back into a space, and a parse against HEAD gives equal values.

On merge: feature/store-i18n 2bcf59109e carries the same four Keycloak
hunks byte for byte; they merge without a conflict.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 23:02:56 +02:00
kevinveenbirkenbachandClaude Opus 5.5 49aa920034 perf(web-app-keycloak): mark user emails verified in one REST call
The emailVerified update looped over every non-reserved user, uploaded a
shell script per user and ran kcadm with a JVM login each time, and it
always reported changed: 7 users took 23.8 s on echoserver.

email_verified.py uses the shared keycloak_admin client through
tasks/utils/rest_script.yml: it looks each user up by exact username,
skips users already verified, and writes emailVerified with a GET plus a
full-representation PUT, as kcadm update did. The task reports changed
only when a user was updated, and a failed lookup now fails the task
instead of being treated as a missing user. The old lookup piped
kcadm -q username=X into head -n1, which could have hit a longer username
sharing the prefix.

Evidence (local compose, slot 104): summary updated=0 unchanged=8
missing=0 failed=0 in 0.60 s, task ok, staging dir removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 19:39:53 +02:00
kevinveenbirkenbachandClaude Opus 5.5 9301220d05 perf(web-app-keycloak): reconcile per-app RBAC over the admin REST API
The per-app RBAC groups and LDAP group mappers were reconciled by four
kcadm shell scripts that started a JVM per call and re-synced every app
on every deploy: 582 s for the mappers plus 324 s for the orphan purge on
echoserver (39 apps).

rbac_reconcile.py now talks to the admin REST API from inside the
Keycloak network namespace (nsenter --net) with the admin credentials
read from /proc/<pid>/environ, so nothing secret reaches argv or output.
It drops the shared ldap-roles mapper, purges root groups of undeployed
apps, ensures each app group and mapper, and syncs a mapper only when the
group is new, the mapper config drifted, or the fingerprint stored on the
group (mapper payload, app rbac and domains, group-logic digest) differs.

The per-app payload no longer carries groups.path.group: Keycloak 26.8.1
does not store that key, so a payload with it never matched the stored
mapper and every deploy re-synced every app.

The admin client lives in keycloak_admin.py; tasks/utils/rest_script.yml
stages it with the script in a 0700 temp dir on KEYCLOAK_HOST_NODE, runs
it and removes the dir in an always block.

Evidence (local compose, slot 104): matrix iteration green in both
variants and passes; update runs show synced=1 in 1.1 s, then
synced=0 unchanged=1 in 0.81 s through the final delivery path, no
staging dir left behind.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 19:28:31 +02:00
kevinveenbirkenbachandClaude Opus 5.5 372142848b test(web-app-stalwart): decide the IMAP parallelism instead of racing it
The parallel-sessions test asserted that more than one of four sessions
appended something, which a loaded host where one worker drains the
shared iterator fails with nothing broken. The sessions now meet at a
barrier before their first append, so a serialised pool still fails the
assertion while a concurrent one always passes.

(cherry picked from commit 657015852a)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 18:01:19 +02:00
kevinveenbirkenbach 9d08fda117 Merge remote-tracking branch 'origin/main' into feature/updater-lockdown-probe 2026-10-24 15:09:52 +02:00
kevinveenbirkenbachandClaude Opus 5.5 d3b9ae59a9 fix(sys-token-store): let the users lookup read the store of the stack host
Lookups render on the controller and read FILE_TOKENS there, while sys-token-store writes the store on STACK_HOST and mirrors it to the controller only in swarm. A compose deploy run from another machine therefore resolved every token empty: on echoserver Friendica rendered an empty SMTP_AUTH_PASS (msmtp: authentication method PLAIN needs a password) and Stalwart minted new bot app passwords on every run.

get_merged_users and the applications carrier key now take the sys_token_store_users fact, which the play reads from the stack host, over the controller file; write.yml refreshes the fact after each write, and 01_load.yml slurps from STACK_HOST so every host reads the store that is written. Without the fact, as in tests and ad-hoc calls, the controller file still applies, and where controller and stack host share the file the merged users are identical.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 15:01:05 +02:00
kevinveenbirkenbach 62164178a2 Merge pull request #1204 from infinito-nexus/fix/web-app-matrix/probe-and-federation-delegation
fix(web-app-matrix): probe only served domains and serve the federation delegation
2026-10-24 13:04:32 +02:00
kevinveenbirkenbachandClaude Opus 5.5 bfd6f7c9b8 fix(install-lint): give chrome-headless-shell a font on dnf, yum and pacman
On a fresh Fedora dev container lint-mermaid failed most files with
ConnectionClosedError: the browser aborted with
`FATAL:third_party/skia/src/ports/SkFontMgr_FontConfigInterface.cpp:163]
Not implemented.` in its font fallback, because the host had no font at
all. install-lint provisions the browser's libraries by soname on dnf,
yum and pacman, and a font is not a soname; apt-get gets fonts through
puppeteer's --install-deps.

It looked load-dependent because lint-packages installs rpmlint in the
same parallel phase, which pulls fontconfig and Noto Sans: renders
crashed until that transaction landed, and every later run was green.
With every font removed, lint-mermaid alone fails 274 of 289 files the
same way.

ensure() now installs the distribution's font provide when
/usr/share/fonts holds no file (`font(:lang=en)` on dnf and yum,
`ttf-font` on pacman), and browser_installed() counts the font, so the
install-lint stamp is not fresh on a font-less host.

Closes #1154

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 12:59:42 +02:00
kevinveenbirkenbachandClaude Opus 5.5 3d1480b3f7 fix(web-app-stalwart): resolve account ids when SSO owns the directory
With the Keycloak OIDC directory active, Stalwart refuses every account create with 'forbidden: Cannot set credentials for accounts in an external directory', and full-4 failed the new batch assert on all accounts. The create request now lists the accounts in the same call and takes each id from the listing of the provisioned domain. Under SSO a forbidden create is tolerated for people, who may not exist before their first login; every bot still needs an id because its app password is addressed by account.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 12:04:29 +02:00
kevinveenbirkenbachandClaude Opus 5.5 7a82a92e84 fix(web-app-prometheus): probe only the domains an app serves itself
#1180 gave web-app-matrix a canonical `jitsi` domain for its own Jitsi
and listed it under `services.jitsi.domains`. When Jitsi is shared or
disabled nothing serves that domain, and the blackbox healthz job of
web-app-prometheus probed every canonical domain, so the probe of
https://jitsi.matrix.<domain>/healthz/ready failed with "tls:
unrecognized name" and `prometheus scrape: every consumer role reports
up=1 and passes its probe` went red (main run 40017166046: Matrix#2,
Nextcloud#12).

web_health_expectations already skipped such keys. The rule now lives
in utils/domains/unserved.py: a key claimed by a service entity that is
disabled, or enabled but shared, is not served by that app. The health
filter uses it unchanged, and prometheus.yml.j2 filters its targets
through the new `served_domains` filter, which also drops the
`<key>_onion` mirror of a skipped key.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 11:44:14 +02:00
kevinveenbirkenbachandClaude Opus 5.5 f37baef29e test(library): keep the project-walk marker on the rglob line
The reason appended to the nocheck marker pushed the line past ruff's limit, so ruff format wrapped the call and left the marker on the closing-parenthesis line, away from the rglob the lint matches. Keep the bare marker, the form every other project-walk opt-out in the repository uses.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 11:06:38 +02:00
kevinveenbirkenbachandClaude Opus 5.5 eca6dc9e50 test(library): mark the directory_batch fixture walk as outside the project tree
test_no_raw_project_walk flagged the rglob in DirectoryBatchTests.tearDown. It walks the test's own TemporaryDirectory to make every directory writable again before cleanup, after tests that leave an unwritable parent behind, so the cached project-file helpers do not apply.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 10:56:05 +02:00
kevinveenbirkenbachandClaude Opus 5.5 b796903d21 perf(svc-db-openldap): set the user passwords over one connection
Three per-user loops ran on every deploy: an ldap_search for whether the user existed (with failed_when: false, so an LDAP error read as 'missing'), a set_fact building a map from it, and community.general.ldap_passwd per user, 18 s for 7 users on echoserver. ldap_entries state=create now registers the DNs it created, and one ldap_passwords call sets the passwords over a single admin bind: a probe bind as the user decides, INVALID_CREDENTIALS leads to the password modify operation, check mode reports without writing.

openldap_user_passwords keeps the old policy: a user whose record leaves password_update on, or whom this run created, gets its password. Errors on single entries are collected so the others still converge, and the task then fails naming every DN.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 10:54:16 +02:00
kevinveenbirkenbachandClaude Opus 5.5 76b16a8abb perf(library): create a whole directory list in one module call
Directory loops over ansible.builtin.file cost one SSH round trip per path: the openresty config tree (20 paths) took up to 12.9 s, the webserver core directories (16) up to 13.2 s, every app's compose directories (5) 2.5-5 s per app, the shared CDN roots 5.8 s and the OpenLDAP LDIF dirs 2.2 s on echoserver.

directory_batch takes entries (path plus its own mode, owner, group) or paths with one shared set, and handles each path like file with state: directory: symlinks are followed, missing parents get the same attributes, an attribute left out stays as found, and it never recurses. It reports changed_paths, supports check and diff mode, and fails on a relative path, an existing non-directory or a failed mkdir. It replaces the five loops and the earlier install -d of the CDN role directories, which now report changed honestly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 10:51:18 +02:00
kevinveenbirkenbachandClaude Opus 5.5 1fbd4758e3 perf(web-app-stalwart): provision accounts and bot app passwords in batched JMAP calls
Every account ran about six JMAP calls of three Ansible tasks each through jmap_call.yml, 80 s for 7 accounts on echoserver. Each phase now sends one request for all accounts: one x:Account/set creates them all (ids from created or, on alreadyExists, from notCreated.objectId), one updates the passwords of the accounts that already existed, and the app-password phases (list, mint, retire, drop) carry one method call per bot, each addressing its bot through accountId, in chunks of STALWART_JMAP_MAX_METHOD_CALLS (Stalwart's default maxMethodCalls, 16).

jmap_call.yml retries and asserts on every method response of a batch, and a refusal names its owner through the call id. A create refused for a reason other than alreadyExists now fails the deploy instead of being skipped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 10:37:01 +02:00
kevinveenbirkenbachandClaude Opus 5.5 3b4727fcfe fix(web-app-stalwart): rate-limit and ban HTTP clients by their own address
With the role network's gateway on Stalwart's allowlist, every request OpenResty proxies arrives from an address that is never throttled or banned, so the anonymous HTTP limit and the scan and auth bans no longer applied to web clients at all. Stalwart can take the client address from a Forwarded header, but OpenResty passed a client-supplied one through, so trusting it would have let anyone pick the address that gets banned.

forwarded.conf.j2 gains an opt-in proxy_forwarded_client_address switch that sets Forwarded: for=$remote_addr on every location including the healthz ones, replacing any value the client sent; other vhosts render unchanged. The Stalwart vhost opts in, and the role sets useXForwarded through x:Http. The settings apply moved into security/apply.yml behind one include so the guarded block stays within the three-task limit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 10:17:42 +02:00
kevinveenbirkenbachandClaude Opus 5.5 57eb64e0af fix(cache): render a templated DOMAIN_PRIMARY against the play variables
get_merged_domains rendered DOMAIN_PRIMARY with no templar and an empty variable scope. A primary declared as a template, such as echoserver's "{{ SOFTWARE_NAME | lower }}", therefore came out as 'none' inside the applications SPOT, while direct lookups saw the domain Ansible had already resolved. Every SPOT value built through it named file.none, and the dashboard waited for https://file.none/assets/img/logo.png until full-2 failed. The render now gets the templar and the play variables, which also covers services.tor.node.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 00:40:32 +02:00
kevinveenbirkenbachandClaude Opus 5.5 47160d755e fix(tls): serve every vhost the certificate of its own domain
ssl_credentials.j2 resolved the certificate by application_id, so every vhost of an application got the lineage of its primary domain. echoserver serves webmail.infinito.nexus the CN=academy.infinito.nexus certificate that names mail.infinito.nexus but not webmail, and browsers refuse it with a hostname mismatch. The template and the letsencrypt existence check now resolve by the vhost's domain, so a secondary vhost points at its own live/<domain> link and a missing link triggers the issuance instead of a mismatch. Primary vhosts resolve to the same files as before in every TLS mode and flavor.

The LDAP stream passes its primary domain explicitly. Onion domains stay on the application lineage, and the app-scoped self-signed path always names the application leaf, because neither has a per-domain certificate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 23:41:33 +02:00
kevinveenbirkenbachandClaude Opus 5.5 ef246a6e03 refactor(unit_name): build unit names from SYS_SERVICE_VERSION_SUFFIX
The lookup assembled '.<version>.<software domain>' itself, a second copy of the infix the unlock and prune phases now read from the SPOT. It reads SYS_SERVICE_VERSION_SUFFIX instead and refuses one that does not start with a dot or carries an empty segment. The direct version-plugin fallback goes with it: it existed because the plugin rendered the untrusted literal "{{ lookup('version') }}", while the SPOT arrives as a trusted group_vars string. Every unit name rendered from group_vars is byte-identical before and after.

The timer description names the versioned service it starts instead of the legacy unversioned name.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 23:34:13 +02:00
kevinveenbirkenbachandClaude Opus 5.5 694007c58a feat(sys-svc-cln-domains): remove the nginx configs of domains the host no longer serves
Retired apps, aliases and legacy zones left their vhosts behind, so 37 legacy cymais.cloud subdomains kept serving a certificate that does not name them after the zone moved to one wildcard redirect. With MODE_CLEANUP the role now deletes every <domain>.conf below the servers directory whose domain is not in CURRENT_PLAY_DOMAINS_ALL, the domains of every application in group_names and its dependencies including redirect sources; files not named after a domain and onion services stay.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 21:25:50 +02:00
kevinveenbirkenbach d0eaca8292 Merge remote-tracking branch 'origin/main' into feature/updater 2026-10-23 20:28:25 +02:00
kevinveenbirkenbachandClaude Opus 5.5 9875db5ff8 feat(web-app-stalwart): deliver mail of legacy alias domains to their target domain
services.stalwart.alias_domains maps a receive-only legacy domain onto the mail domain: the deploy sets the RCPT stage rewrite to one matches() rule per alias domain, reads it back, and with Cloudflare points the source's MX at the mail hostname. An empty list leaves Stalwart untouched.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:19:59 +02:00
kevinveenbirkenbachandClaude Opus 5.5 8c8e6a1242 feat(web-opt-rdr-domains): redirect a whole legacy zone with one wildcard source
redirect_domain_mappings now renders its source and target, so an inventory can map *.{{ OLD_DOMAIN }} and the apex onto a target; keep_path: false drops the request URI from the 301. Wildcards get no www. variant, sys-dns-wildcards adds the *.<zone> A/AAAA records for every zone with a wildcard source next to DOMAIN_PRIMARY, and the CSP health check skips wildcard names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:17:19 +02:00
kevinveenbirkenbachandClaude Opus 5.5 9152c884d8 fix(tls): serve every redirect source the certificate that names it
The cert lookup handed an alias or redirect source the live directory of its app's canonical domain or of its target, so social.cymais.cloud got the infinito.nexus lineage and failed with 'Hostname mismatch'. With the san and dedicated flavors a domain term now resolves to its own live/<domain>, which cert_link_domains points at the lineage that covers it; application terms keep the canonical path. resolve_term follows redirect chains and fails on a cycle.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:14:21 +02:00
kevinveenbirkenbachandClaude Opus 5.5 27762376f0 perf(sys-ctl-hlth-webserver): probe the domains in parallel
The checker sent one HEAD request after the other, up to 10 s per clearnet and 30 s per onion domain. A pool of 16 threads now probes them; the waits are I/O, so 16 suit hosts with any core count, and pool.map keeps the output sorted by domain.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:10:52 +02:00
kevinveenbirkenbachandClaude Opus 5.5 fb18077f57 perf(utils): key the domain index by the applications' domains only
build_domain_index fingerprinted the whole applications tree on every call, 77 % of a tls lookup. The index reads only each application's domains, so the cache key now covers exactly those; a tls lookup drops from 8.2 to 3.1 ms.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:03:40 +02:00
kevinveenbirkenbachandClaude Opus 5.5 8e73c84b15 perf(svc-db-openldap): converge the user entries over one connection
Three loops started one async ldap_entry or ldap_attrs job per user and reaped them, about 2 minutes per deploy on echoserver. The ldap_entries module binds once and applies create, present or exact to every entry the new filters build from OPENLDAP_USERS, keeping the semantics of the community modules it replaces.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 19:59:37 +02:00
kevinveenbirkenbachandClaude Opus 5.5 5b6db5dc24 feat(web-app-stalwart): import Mailu only until one import has finished
The import script writes a done marker into the instance directory when it ends with rc 0, and the deploy skips the unit and the import block while the marker exists; deleting it imports again. The IMAP layout helpers move to imap_layout.py so the script stays under the 500-line limit, with mirrored tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 19:51:10 +02:00
kevinveenbirkenbachandClaude Opus 5.5 43ce74236e perf(web-app-dashboard): probe every Simple Icons slug in one parallel call
uri_retry looped over each card slug, 60 to 64 s per probe on echoserver. The new uri_probe_batch module asks all URLs through a thread pool with per-URL retries and fails listing every URL that gave no expected status.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 19:37:57 +02:00
kevinveenbirkenbach 0bca4bcd7f Merge remote-tracking branch 'origin/main' into fix/web-app-matrix/ansible-flavor-unwired-services/1165 2026-10-23 18:18:31 +02:00
kevinveenbirkenbachandClaude Opus 5.5 25ec8b0c59 fix(sys-ctl-hlth-webserver): skip a consumer's domains for a shared service
A service entity can list canonical domain keys under `domains`; the
web health expectations skipped them only while the service was
disabled. A consumer whose service is enabled but shared does not serve
its own domain for it either, since the provider serves the service on
its own domain, so the probe failed with an unrecognised TLS name.
Domains of an enabled and shared service are now skipped as well.

The only other entity with `domains` (web-app-seaweedfs frontend) is
never shared, so its behaviour is unchanged.

Refs #1165

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 17:16:19 +02:00
kevinveenbirkenbachandClaude Opus 5.5 462f30e46a perf(web-app-stalwart): import Mailu in parallel sessions as a background unit
On echoserver the import appended 23,045 messages one by one over a single
IMAP session, about six per second, and the deploy waited 62 minutes for
it inside the Keycloak stage, because Stalwart is a dependency of Keycloak.
galaxyserver holds 157,655 messages in one mailbox, which one session
would need 7 to 28 hours for, beyond the task's six-hour timeout.

migrate_from_mailu.py now opens several IMAP sessions per mailbox that
take messages from one queue in file name order, with the dedup set and
the counters under a lock, and logs [PROGRESS] every 1000 messages. A
session Stalwart refuses only lowers the parallelism; with one session
the import behaves exactly as before. The number of sessions is
services.stalwart.migration.workers, or one fewer than the host's vCPUs,
at least 1 and at most 8.

The deploy starts the import as the transient unit stalwart-mailu-import
and does not wait for it; a deploy that finds the unit running leaves it
alone. The unit removes the staged recovery administrator credentials when
it stops, and a failed start removes them in a justified rescue before it
re-raises. Progress and the closing [DONE] rc=<n> are in
journalctl -u stalwart-mailu-import.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 16:35:45 +02:00
kevinveenbirkenbachandClaude Opus 5.5 cfb2f9c70b perf(web-opt-rdr-domains): check every certificate in one task and write every file in one call
Each redirect source went through the full TLS layer on its own: cert check,
CA environment cleanup, re-check, HTTP handler and vhost template, about
nine seconds per source. echoserver has 85 sources, so the role took
thirteen minutes of every deploy.

The role now resolves the tls state and renders both files of every source
on the controller, finds the certificates of all sources in one find, and
includes sys-front-tls only for a source whose certificate is missing, so
issuance still works in every mode. library/stack_host_contents.py writes
all files in one module call, atomically and only where the content
differs, and keeps one file per source under http/ and https/. Its action
plugin carries the IS_STACK_HOST gate like stack_host_template.

The name differs from the stack_host_files module of feature/store-i18n on
purpose: that one syncs a single directory of YAML documents and prunes
what it does not write, which would delete every other vhost here.

HTTPS_REDIRECT_ENABLED is gone: nothing set it, it was true everywhere,
and the redirect role could no longer see the sys-front-tls default it
lived in. Whether a domain gets its HTTP to HTTPS handler now follows from
lookup('tls', domain, 'enabled') alone, in sys-front-tls and here.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 16:35:45 +02:00
kevinveenbirkenbachandClaude Opus 5.5 285ce910d2 feat(inventory): reject unread keys and a missing Cloudflare token before the deploy
The Keycloak stage on echoserver ran ten minutes before sys-svc-dns stopped
it with "API.cloudflare.api_token must be set": the inventory still kept the
token under the flat name CLOUDFLARE_API_TOKEN, which nothing reads any
more. The validator that runs before every deploy now catches that class.

- api.py reports a host that selects DNS_PROVIDER: cloudflare without
  API.cloudflare.api_token, with group_vars/all merged under the host_vars
- variables.py reports every top-level key of host_vars and group_vars that
  core does not read: not declared in group_vars or a role's defaults or
  vars, not tested with "is defined" or "| default(", not read by name in a
  plugin, and not a SYSTEM_EMAIL_<KEY> the email lookup builds; a flat
  credential name such as CLOUDFLARE_API_TOKEN names its API.<provider>.<key>
  path; ansible_* keys are exempt

A key that is read in a way the scan cannot see takes
"# nocheck: inventory-variable", documented in suppression.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 16:18:12 +02:00
kevinveenbirkenbachandClaude Opus 5.5 c728b4427b fix(tls): resolve a redirect source through its target
With AUTO_BUILD_ALIASES enabled, current_play_redirect_domains adds
<entity>.<DOMAIN_PRIMARY> as a redirect source for every application whose
canonical domain differs. web-opt-rdr-domains then asks the tls and cert
lookups about each source, but resolve_term only knew the domains mapping
and applications[*].domains, so the Keycloak stage on echoserver stopped
with "tls: domain 'funkwhale.infinito.nexus' not found".

redirect_targets() turns CURRENT_PLAY_REDIRECT_DOMAINS into a source to
target map, and resolve_term falls back to it, one hop, only where it
raised before, so every term that resolved keeps its answer. The tls and
cert lookups pass the map.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 16:08:18 +02:00
kevinveenbirkenbachandClaude Opus 5.5 a32f241051 fix(sys-svc-certs): link each domain to the certbundle lineage that covers it
certbundle names its lineages certbundle-<hash>-<n>, while the cert lookup
reads live/<domain>/fullchain.pem. Since 16023dbc9b removed cert_folder_find,
nothing joined the two, so every letsencrypt deploy with the SAN flavor
stopped at "Let's Encrypt certificate files not found after issuance".

library/cert_link_domains.py points live/<domain> at the newest lineage
whose SAN covers the domain, using the same CertUtils lookup as
cert_check_exists, and swaps the link atomically. A real lineage directory
is left alone, and a domain without a certificate is only reported, so the
letsencrypt re-check still fails loudly on it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 16:05:27 +02:00
kevinveenbirkenbachandClaude Opus 5 3cc0214c28 fix(deploy): stop the dedicated deploy from git-cleaning its own checkout
`MODE_CLEANUP` says what to tidy on the deployed system: prune unused Docker
resources, clear Mastodon caches, switch an LDAP user list from `present` to
`exact`. Eight roles read it that way. The dedicated runner read the same flag
and ran `make clean` in the control repository instead, which is
`git clean -fdX` over every ignored path there.

`inventories/production/` is one of those paths. The published host install
guide provisions into it and then deploys against `$INVENTORY/.password`, so
the deploy deleted the vault password it was about to use:

    Removing inventories/production/
    [ERROR] The vault password file .../inventories/production/.password was
    not found

Reproduced in CI on two unrelated rows, arm64/zfs/debian and
amd64/btrfs/centos, so it is neither architecture nor filesystem specific.

The coupling was a name collision, not a decision. 7e5990aa16 generated the
MODE_* flags from `01_modes.yml` and removed the legacy skip flags, and the
repository clean - which had had a flag of its own - was attached to the
generated `MODE_CLEANUP` because one now existed by that name. No other deploy
path does this; `container`, `development` and `swarm` never call `make` at
all.

`make clean` keeps its own entry points, `make setup-clean`, `make clean-sudo`
and `infinito --git-clean`, where a caller asks for it deliberately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-23 10:59:05 +02:00
kevinveenbirkenbach e4ac05dfb6 Merge remote-tracking branch 'origin/main' into feature/updater 2026-10-23 10:46:59 +02:00
kevinveenbirkenbach e53751402f Merge pull request #1114 from infinito-nexus/fix/security/swarm-ingress-guard/1089
fix(svc-swarm-node): drop outside traffic to non-public routing-mesh ports
2026-10-23 01:03:04 +02:00
kevinveenbirkenbachandClaude Opus 5.5 bfdf88fae0 fix(domains): read an application's domains where the cache puts them
meta/domains.yml lands on the top-level `domains` key of the merged
application, yet three readers still looked under `server.domains`,
which no role fills any more:

- current_play_redirect_domains built every mapping from an empty
  config, so each app redirected to itself and was dropped. Neither
  `domains.aliases` nor AUTO_BUILD_ALIASES produced a single redirect,
  and echoserver's update check listed nextcloud.infinito.nexus and the
  other entity domains as dropped by the upgrade.
- csp_skip_domains never found the canonical of a disabled service.
- the logout filter's fallback returned no domains.

The unit tests fed the old layout and passed regardless. They now use
the merged one, and a new test builds the redirect from keycloak's real
role defaults.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 22:18:07 +02:00
kevinveenbirkenbachandClaude Opus 5.5 96fe8bd7e6 feat(playbooks/update): pull the images of the deploy before the upgrade window
pull.yml pulls, on the target host, every pinned image of the applications the host deploys, the same closure the update check judges, and lists the images a role builds itself instead. It never touches a container, retries each pull a bounded number of times and fails loudly with the image and the registry error. The update_check filter moves next to the new update_pull filter into plugins/filter/update/, and both read the code pins from one helper.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 21:30:18 +02:00
kevinveenbirkenbach 3cf840c72b Merge remote-tracking branch 'origin/main' into feature/updater 2026-10-22 21:14:49 +02:00
kevinveenbirkenbach 4a86376aeb Merge remote-tracking branch 'origin/main' into feature/updater 2026-10-22 21:08:26 +02:00
kevinveenbirkenbachandClaude Opus 5.5 8567cb6992 fix(svc-opt-ssd-hdd): recognise database images behind registry, namespace and custom builds
is_database() compared the whole repository part of the image reference
against postgres, mariadb, redis, memcached and mongo. Every reference
with a registry or namespace prefix, every derivative image and every
locally built `<engine>_custom` image therefore counted as application
data and went to the mass storage. On a local stack the central
databases report `postgres_custom:18` and `openldap_custom:13.7-slim`,
both classified False and moved to the HDD.

The check now takes the last path component of the reference, drops
digest, tag and a `_custom` suffix, and compares it against the engines
the svc-db-* providers and the app-local databases run: elasticsearch,
mariadb, memcached, mongo, mysql, openldap, opensearch, postgis,
postgres, qdrant, rabbitmq, redis, typesense, valkey.

Volumes the script already symlinked stay where they are; a database
volume misplaced on the HDD has to be moved back by hand.

Fixes #1163

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 21:02:06 +02:00
kevinveenbirkenbachandClaude Opus 5.5 558784e6ca feat(web-app-erpnext): set up the service provider's company and its accounting basics
The wizard bypass left ERPNext without a Company, so no app could create quotes, orders or invoices. The deploy now creates the Company from the service provider's name and country, with the currency from ERPNext's country data and an abbreviation that defaults to the name's initials, plus a fiscal year, five tax accounts, a cash account and the default company. It never rewrites an existing Company and fails loudly on a country ERPNext does not know. The README generator marks roles that read the service provider country, and the CI replay fills the placeholder.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 20:52:30 +02:00
kevinveenbirkenbachandClaude Opus 5.5 286004349a fix(web-app-suitecrm): upgrade the code in the data volume to the pinned release
The entrypoint stamped an installed volume with the image's version and kept the old tree, so a newer pin never reached it and the CreateUser patch stayed out. On boot it now runs SuiteCRM's own suitecrm:app:upgrade and upgrade-finalize with the image's patched tree as the package, resumes an interrupted upgrade, stamps the version the tree really has, and refuses a tree newer than the pin.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 20:44:13 +02:00
kevinveenbirkenbach b40c05dbbf Merge pull request #1177 from infinito-nexus/fix/install/shfmt-test-docstring
fix(install): drop the shfmt docstrings that misstate ensure()
2026-10-22 20:41:38 +02:00
kevinveenbirkenbachandClaude Opus 5.5 091c58307d fix(install): drop the shfmt docstrings that misstate ensure()
Three docstrings from #1173 claimed that ensure() leaves only the pinned
release on PATH or replaces whatever version PATH finds. It does neither:
it installs the release into /usr/local/bin and fails unless PATH then
resolves to it, so a distribution copy such as /usr/bin/shfmt stays on
disk behind it.

The test module and ensure() lose their docstrings; the module docstring
of the installer returns to the form its siblings use ("Install hadolint
via GitHub-release prebuilt binary.").

Reported by CodeRabbit on #1173.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 20:37:01 +02:00
kevinveenbirkenbachandClaude Opus 5.5 628e2587b8 fix(database_query): keep psql's command tags out of the returned rows
With `-tA` psql drops headers and the row-count footer but still prints
the command tag of every write statement. A guarded
`UPDATE … RETURNING` that matched nothing therefore came back as
`rows: [['UPDATE 0']]`, and web-app-peertube's oidc/enable.yml, which
judges its change by the row count, reported `changed` and restarted
PeerTube on every deploy after the first.

The PostgreSQL rows flags are now `-qtA`. Probed against a running
postgres container:

    -tA  -> CREATE TABLE / INSERT 0 1 / UPDATE 0 / 3 / UPDATE 1
    -qtA -> 3

SELECT output and the MariaDB path (`--batch --skip-column-names`, no
status line) are unchanged.

Fixes #1164

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 20:36:32 +02:00
kevinveenbirkenbach eeb3305e52 Merge remote-tracking branch 'origin/main' into fix/security/swarm-ingress-guard/1089 2026-10-22 19:17:39 +02:00