Commit Graph
278 Commits
Author SHA1 Message Date
kevinveenbirkenbach 6c20fd4e30 Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
# Conflicts:
#	cli/administration/inventory/provision/cli.py
#	roles/sys-svc-mail-msmtp/templates/msmtprc.conf.j2
2026-07-28 16:19:34 +02:00
kevinveenbirkenbachandClaude Opus 5 cf82841be8 refactor(msmtp): resolve the SMTP transport host in a lookup
The template carried a three-way conditional across three {% set %} lines,
which was already at the edge of readability. Move the decision into
lookup('smtp_host') as plain control flow and pin all six cases in tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 16:15:59 +02:00
kevinveenbirkenbachandClaude Opus 5 ebd7d2e72b refactor(meta): centralize category/distro SPOTs
categories.yml moves from roles/ to the repository-root meta/, joining packages.yml. utils/roles/categories.py owns its location plus the tree-walking helpers that were stranded in entity/name.py, so no caller spells the path or duplicates the lookup.

meta/distros.yml becomes the SPOT for every distribution: os_family, Galaxy platform name, dev-environment base image, and the pkgmgr/pkgmgr-virgin/environment reference templates. It replaces seven scattered copies (DISTRO_FAMILY, VALID_DISTROS, _DISTROS, _CANONICAL_PLATFORMS, two shell arrays, a Makefile literal). INFINITO_DISTROS is no longer a static default but generated by utils/env/handlers/infinito/distros.py; the shell consumers read it through the env layer.

Drops dead CLI surface found along the way: --categories-file and --roles-dir on the provision CLI plus three unused parameters of generate_dynamic_inventory, and the positional roles_file of the invokable categories command.

tests/lint/repository/test_distros_spot.py pins the copies that cannot call the SPOT (workflow literals, docs table) and forbids composing a pkgmgr reference outside it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 16:15:56 +02:00
kevinveenbirkenbach 71c9f10dfd Merge branch 'feature/swarm-nfs' into feature/svc-net-tor 2026-07-28 11:22:47 +02:00
kevinveenbirkenbachandClaude Opus 5 e409674718 refactor(meta): centralize category/distro SPOTs
categories.yml moves from roles/ to the repository-root meta/, joining packages.yml. utils/roles/categories.py owns its location plus the tree-walking helpers that were stranded in entity/name.py, so no caller spells the path or duplicates the lookup.

meta/distros.yml becomes the SPOT for every distribution: os_family, Galaxy platform name, dev-environment base image, and the pkgmgr/pkgmgr-virgin/environment reference templates. It replaces seven scattered copies (DISTRO_FAMILY, VALID_DISTROS, _DISTROS, _CANONICAL_PLATFORMS, two shell arrays, a Makefile literal). INFINITO_DISTROS is no longer a static default but generated by utils/env/handlers/infinito/distros.py; the shell consumers read it through the env layer.

Drops dead CLI surface found along the way: --categories-file and --roles-dir on the provision CLI plus three unused parameters of generate_dynamic_inventory, and the positional roles_file of the invokable categories command.

tests/lint/repository/test_distros_spot.py pins the copies that cannot call the SPOT (workflow literals, docs table) and forbids composing a pkgmgr reference outside it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 11:19:30 +02:00
kevinveenbirkenbachandClaude Opus 5 52b798094a style(container_dns): satisfy ruff on the onion dns bundle
Drop a return-only temporary and state autoescape explicitly — the test
renders a dnsmasq config, where html escaping would corrupt the output.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 09:24:38 +02:00
kevinveenbirkenbach 405e29a50a Merge branch 'feature/swarm-nfs' into feature/svc-net-tor 2026-07-28 01:14:40 +02:00
kevinveenbirkenbachandClaude Opus 5 9d91fbea83 feat(svc-net-tor): route onion names through tor via dns, not per-app proxies
Reaching an onion address required each role to point its own http client
at the tor socks proxy, which then carried clearnet traffic too — fatal
in the test network, which has no exit, and the reason nextcloud needed a
per-host exclusion list.

The egress router already redirects tor's virtual range to the TransPort
and dnsmasq already forwards .onion to tor's DNSPort; the piece missing
was reachability. dnsmasq listened on loopback only, which docker
discards, so containers never saw it. It now also answers on the docker
bridge, the daemon points there with the clearnet resolver kept as
fallback, and the role's own onion resolves to a container-reachable
backend instead of loopback.

Nextcloud drops its proxy block and the exclusion list; every other role
gets the same behaviour without configuring anything.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 01:13:33 +02:00
kevinveenbirkenbachandClaude Opus 5 369ddfca52 fix(packages): install via the host's pkg_mgr module
`ansible.builtin.package` has no module implementation. It is a documentation-only stub whose logic lives in Ansible's own action plugin, so `_execute_module` shipped that stub to the target and every deploy died at the first package install with "module (ansible.builtin.package) is missing interpreter line".

Resolve the generic marker to the host's `ansible_facts.pkg_mgr` before execution.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 00:29:29 +02:00
kevinveenbirkenbach bb1b4f7f96 Merge branch 'feature/swarm-nfs' into feature/svc-net-tor 2026-07-27 23:55:03 +02:00
kevinveenbirkenbachandClaude Opus 5 f61a70c3ee feat(packages): add a distribution-independent package registry
A role states which package it needs; the registry owns every distribution
mapping and every acquisition path. Declarations live in the repository-root
meta/packages.yml when more than one consumer installs them, and in a role's
own meta/packages.yml otherwise.

schema.py owns the declaration shape, registry.py finds and indexes the
declarations, plan.py turns a resolved spec into ordered module calls, and the
package_install action plugin only executes them. Keeping the plan pure makes
every acquisition path unit-testable without an Ansible connection:
distribution repositories, the AUR with its unprivileged build user, COPR, PPA,
an EPEL bootstrap package, the pacman multilib section and source builds.

The data is guarded by lints: a package id and a concrete distribution package
name each belong to exactly one declaration, a role installs only its own or
shared ids, every id resolves for all five default distributions with an
empty mapping justified by a nocheck, declarations read in ascending order,
every plan builds and names only whitelisted modules, and an external test
queries the real package indexes of all five distributions.

This commit adds the mechanism and the shared collection; no consumer uses it
yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 23:25:50 +02:00
kevinveenbirkenbachandClaude Opus 5 a35f0a02a2 docs(svc-storage-nfs-server): correct the stale flavor description
Both iteration guides and the nfs_flavor lookup docstring still described
the flavor as RUNTIME conditional, selecting ganesha only for dev and act
and kernel everywhere else. Since the per-variant switching was dropped,
meta/services.yml pins ganesha for every runtime and kernel is a
per-inventory override.

State that, name meta/services.yml as the single point of truth, and
record which os families ship ganesha.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 23:25:49 +02:00
kevinveenbirkenbach 98b6b3db8e Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
# Conflicts:
#	plugins/lookup/application_gid.py
2026-07-25 15:44:55 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e543ca605d perf(cache): memoize redundant roles-tree rescans via functools.cache
Five helpers rescanned the whole roles/ tree on every call while their result is a pure function of the (static) tree, so callers that invoke them per-role/per-app/per-service turned the scan into O(N^2) filesystem ops. compute_application_gid was the measured hotspot: 254 rescans in one applications render (~25s of a 41s cold render). Memoize each tree-derived result with functools.cache keyed by the resolved roles_dir (the repo's existing idiom), returning tuples so the shared cache cannot be mutated; dependency_resolver uses an instance cache instead since functools.cache on a method would pin instances via self. Cold applications/domains render drops 41s -> ~8.5s (5.6x); behaviour is unchanged (GID/registry/placement/path values identical).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 14:35:35 +02:00
kevinveenbirkenbachandClaude Fable 5 f334cccf18 fix(web-app-dashboard): load keycloak-js from onion-local CDN
The asset lookup hardcoded https for the internal CDN origin, but
.onion vhosts are plaintext with no 443 listener, and the dashboard
mirror-rewrite sent even onion-local asset URLs through the mirror
(404). The adapter never loaded, so the OIDC code was never exchanged
and Login stayed visible after the round-trip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 09:03:21 +02:00
kevinveenbirkenbach 1a0af0ce09 Merge remote-tracking branch 'fork/feature/swarm-nfs' into feature/svc-net-tor
# Conflicts:
#	roles/sys-ctl-mtn-cert-deploy/tasks/main.yml
#	roles/sys-svc-compose-ca/tasks/main.yml
#	roles/web-app-dashboard/templates/javascript/oidc.js.j2
#	roles/web-app-mastodon/tasks/01_overrides.yml
#	roles/web-app-peertube/templates/compose.yml.j2
#	scripts/github/guide/compose_deploy.sh
#	scripts/github/guide/host_deploy.sh
#	scripts/make/alias.sh
#	tests/lint/ansible/roles/test_service_core_first_task_run_once.py
2026-07-24 04:30:56 +02:00
kevinveenbirkenbachandClaude Opus 4.8 d9a53a0545 chore(lint): pin ruff to 0.16.0 and resolve the rules that version enforces
The ruff toolchain was unpinned, so host venvs, the lint container and CI
installed different releases over time. That drift is why lint-python only
failed on CI: locally ruff 0.15.x preview-gated ISC004/FURB192 and (falsely)
fired S310 on urllib.request.Request, while CI's 0.16.0 has ISC004/FURB192
stable and drops the S310-on-Request false positive.

Pin ruff==0.16.0 in pyproject.toml's dev extras (Dependabot keeps it current)
and have utils.install.lint.ruff read that pin and reinstall whenever the
found version drifts, so every environment resolves the same ruff. Then fix
what 0.16.0 flags: parenthesise the implicit string concatenations (ISC004),
use min() over sorted()[0] (FURB192), drop the now-unused S310 noqa on the
Request() builders (RUF100), and delete the narration comments that the
comment lint surfaces in the touched files.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 23:08:26 +02:00
kevinveenbirkenbachandClaude Opus 4.8 4039313266 refactor(tasks): enforce and apply a single task-file numbering scheme
Add tests/lint/repository/test_task_file_numbering.py: numbered task
files use exactly two leading digits, a numbered *_core.yml must be
00_core.yml, per directory the numbers are unique/consecutive and start
at 01 (00 with a core file), numbered includes appear in ascending
order, and shared helpers live in tasks/utils/ instead of the numbered
chain.

Apply it across every role: rename NN_core.yml to 00_core.yml with a
consecutive down-shift of the siblings, move letter-suffix children into
a subdirectory named after their parent task, relocate helpers included
from more than one place into tasks/utils/, and update every reference
(include_tasks, path_absolute, .ansible-lint-ignore, requirement docs).
Legacy narration comments in the touched files are deleted; the upstream
xwiki.properties reference block opts out via nocheck. The unused
MAILU_IP6_PUBLIC var (only ever named in now-removed comments) is dropped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 20:01:31 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e99d8a8813 feat(cdn): serve role frontend dependencies from the internal CDN
Add lookup('asset', app, pkg, path) and lookup('asset_host'): both resolve
the asset origin through one shared resolve_host, returning the internal
CDN domain when web-svc-cdn is deployed with flavor internal and
cdn.jsdelivr.net otherwise, so template URLs and CSP origins can never
diverge. Versions come from the role's committed files/package-lock.json.

Roles declare frontend dependencies in files/package.json; web-svc-cdn
aggregates every role lockfile, runs one npm ci per role in a node
container and mirrors the packages under /npm/<pkg>@<version>/ following
jsdelivr's path scheme, so internal and external URLs differ only by host.

Convert web-app-dashboard (keycloak-js 24.0.5 UMD) and web-app-littlejs
(bootstrap, fontawesome) to the asset lookup, drop the vendored keycloak
blob, and add a repository lint forbidding hardcoded external CDN hosts
in templates and CSP metas; pre-existing external CDN entries are checked
out with nocheck markers and a migration TODO.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 20:01:31 +02:00
kevinveenbirkenbachandClaude Opus 4.8 07d31d9001 fix(ci): retry-harden every egress path (curl filter, shell downloads, package managers)
Transient CDN/mirror TLS resets are a recurring hard-fail class (docker.com, PyPI). Harden all three egress paths at their SPOT: the curl filter now emits --retry-all-errors (retries=0 opts a call out), so every ansible curl retries; a new lint requires --retry-all-errors on file-downloading curl and --tries on wget in verbatim files/*.sh (which cannot use the filter), fixing the moodle/alias downloads it flagged; and the update role configures native retries for every package manager - apt Acquire::Retries, dnf retries=, pacman XferCommand over retrying curl - covering all five CI distros (ubuntu shares apt, centos shares dnf).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 20:01:31 +02:00
kevinveenbirkenbachandClaude Fable 5 38fabe226c chore(lint): pathlib + safe-loader noqa for the quality gate
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 15:48:23 +02:00
kevinveenbirkenbachandClaude Fable 5 a881ec26b9 fix(mirror): read the CSP whitelist from its post-merge location
The server.yml split hoisted csp to the top level; the filter kept
reading server.csp.whitelist and silently returned no mirrorable asset
hosts, leaving the mirror vhost whitelist and body rewrite empty.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 15:43:39 +02:00
kevinveenbirkenbachandClaude Fable 5 d88f272f9f Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 15:31:08 +02:00
kevinveenbirkenbachandClaude Fable 5 26ae8e1c1b fix(fediwall): family-align wall servers to the consuming page
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 20:39:16 +02:00
kevinveenbirkenbachandClaude Opus 4.8 5936d3ba18 fix(csp): CSP token and injected asset agree on the aligned provider scheme
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 02:12:12 +02:00
kevinveenbirkenbachandClaude Opus 4.8 3585ad3be7 fix(seaweedfs): keep the embedded sidecar launcher free of shell dollars
The compose and swarm render paths launder dollars differently; the launched argv arrived with $pid collapsed to an empty string and $! doubled, and the sidecar crash-looped on a sh syntax error, so opentalk's wait for the embedded object store exhausted its retries. The bucket bootstrap now polls /status from a backgrounded subshell and exec makes the server PID 1, which needs no pid bookkeeping and receives docker stop's TERM directly; the poll loop dies with the container and readiness stays gated by the consumer's bounded wait.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 01:11:48 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0b57337032 fix(proxy): reach node-local force_bridge apps via the host gateway under swarm
A force_bridge app (matrix's MDAD flavor) deploys as a node-local compose stack and publishes its local port on its host, while openresty runs as an overlay task whose 127.0.0.1 is its own loopback; the readiness probe died on 502. The proxy_pass lookup now resolves networks.local.force_bridge (trust-tagged for the 2.19 templar, hard-failing on an unrendered template instead of silently falling back to loopback) and keys the decision on the real DEPLOYMENT_MODE, since such apps rewrite the effective mode to compose via compose_mode_force. The upstream becomes host.docker.internal:<local_port> rendered as a literal directive, because the request-time resolver cannot see /etc/hosts entries, and the swarm openresty service maps host.docker.internal to the host gateway like prometheus already does. Non-force_bridge renders stay byte-identical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 01:11:48 +02:00
kevinveenbirkenbachandClaude Opus 4.8 bce2df6225 fix(tests): use valid application_ids in family-alignment fixtures; autoformat
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 23:44:12 +02:00
kevinveenbirkenbachandClaude Opus 4.8 7d6267c3a6 feat(rdr-domains): redirect the bare onion apex to the homepage on onion nodes
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 23:44:12 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e0867fb44d feat(lookup): family-align peer refs to the consuming app (cross-family substrate)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 23:44:11 +02:00
kevinveenbirkenbachandClaude Opus 4.8 fe44e078da feat(lint): enforce category-declared mode defaults on role services
categories.yml now declares modes defaults per category (deepest match wins, parents apply as fallback): update, drv and svc-net disable compose and swarm (host-hardware and host-networking categories), svc-registry requires both enabled (CI bootstrap infrastructure). role_modes_defaults resolves a role's policy; the new category-modes lint requires every deploy-bearing service entry to declare exactly those modes.<mode>.enabled values explicitly, exempt via '# nocheck: category-modes' on the service key. update, the four drv roles and the three wireguard roles declare their values explicitly; the dict-valued modes meta key is excluded from the category-tree walkers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 19:10:00 +02:00
kevinveenbirkenbachandClaude Opus 4.8 f4172c547a fix(prometheus): exclude swarm-unreachable force_bridge apps at the native_metrics_apps SPOT
A native-metrics app on a node-local force_bridge network cannot be joined or resolved by a swarm-wide prometheus: the stack deploy fails with the network scope error and the scrape job would flap as permanently down. native_metrics_apps now drops those apps at the source (DEPLOYMENT_MODE swarm + networks.local.force_bridge), covering all four call sites (network precreate, compose service networks, external-networks block, scrape include) without per-template guards. DEPLOYMENT_MODE is templated without a default so a missing value fails loudly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 14:44:34 +02:00
kevinveenbirkenbachandClaude Opus 4.8 4dbaa60707 feat(stages): drive deploy stages from a categories.yml SPOT
roles/categories.yml now declares the ordered 'stages' and a per-category 'stage' (plus run_after and bootstrap). utils/roles/stage.py resolves a role to its stage and exposes stage_groups(); the new stage_groups lookup feeds the 01_constructor / 02_server / 02_workstation / 03_destructor group loops in place of their hardcoded lists. A static invariant test proves the lookup reproduces the exact previous loops, so the rendered plays (and deploy behavior) are unchanged.

cli.meta.roles.order.run orders its main phase by stage as a run_after tiebreak. plugins/filter/invokable_paths tolerates the new sibling 'stages' key, and the invokable-roles wiring test recognizes stage_groups-driven wiring.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 14:44:34 +02:00
kevinveenbirkenbachandClaude Fable 5 40e553069d refactor(structure): nest prefix-grouped sibling files into subfolders
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.

Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 20:00:03 +02:00
kevinveenbirkenbachandClaude Opus 4.8 fc0799883f refactor(structure): move prefix-grouped files into named subfolders
Resolve every 3+-file same-prefix group flagged by the new prefix lint
by moving the files into a subfolder named after the shared prefix and
dropping the prefix from the file names, updating all references:

- utils/env/handlers/infinito_*.py -> infinito/ (dir/, package_cache/
  nested a second level); registry imports and the four handler-dir
  globs switched to iter_project_files
- utils/install/lint/ansible_*.py -> ansible/ (aliased imports)
- roles/sys-svc-compose/files/swarm_*.{py,sh} -> files/swarm/
- web-app-erpnext/web-app-zammad files/scripts/apply_*.{py,rb} -> apply/
- web-app-odoo files/sql/{select,update}-* and sql/oidc/{grant,insert,
  select,update}-* -> matching subfolders
- web-app-keycloak tasks/update/05_ldap*.yml -> 05_ldap/ (internal
  includes made role_path-absolute)
- plugins/filter/get_*.py -> plugins/filter/get/ (filter names are the
  filters() dict keys, so Jinja usage is unaffected; ansible discovers
  the one-level-deep subdir)

Ansible lookup dirs, addon dirs and .github/workflows keep their flat
layout (the filename is the public id / GitHub only discovers flat).
Unit tests mirror the new source layout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 20:00:03 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6cc9228a10 fix(swarm-nfs): materialize export subdirs per node and honor volume nfs metadata
The export subdir was created run_once on the NFS server only; a worker
that had already readdir'd the export kept a negative dentry and swarm
rejected every replicas>1 task with 'readdirent .../_data: no such file
or directory' for the whole converge window. Materialize the bind-device
dir through each node's own mount, guarded by a per-node mountpoint
assert so an unmounted node fails the deploy loudly instead of receiving
a local mkdir that splits the brain and trips the nfs-client
has-local-data assert on the next full deploy.

compose_volumes popped the volume 'nfs' metadata before nfs_prep could
read it, so declared uid/gid/mode never took effect and every subdir
fell back to root 0755; keep the mapping under the compose-spec
extension key x-infinito-nfs (ignored by docker, verified against stack
config and compose config). KIX declares uid/gid 110010100 mode 0770
through it, fixing the backend_api_token EACCES crash on its shared
volume.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:27 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e77aae2afe perf(service-loader): single applications-load preload
Flatten a single list term in the service lookup so query(service, [ids]) does one applications load instead of one per id. Precompute the required service ids in sys-service-loader and gate the preload by set membership, collapsing about 30 applications loads into one.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 02:47:40 +02:00
kevinveenbirkenbachandClaude Opus 4.8 c8a4320478 feat(lookup): container_hostname — kernel-safe container hostname
`lookup('container_hostname', app)` returns the app's canonical primary
domain when it fits sethostname(2)'s 63-byte cap, else the short entity
name. Onion subdomains (sub.<56-char>.onion) overflow that limit and make
`docker run` fail with "sethostname: invalid argument", killing container
init on onion nodes. Replaces the seven roles that hostname'd their
container on the raw domain (bookwyrm, bridgy-fed, confluence, jira,
postmarks, pretix, xwiki) with the encapsulated lookup.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 02:34:00 +02:00
kevinveenbirkenbachandClaude Opus 4.8 32b2cf7e80 style: make autoformat pass (ruff line wraps, YAML comment spacing)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 02:33:59 +02:00
kevinveenbirkenbachandClaude Fable 5 808efad765 fix(test-e2e-cli): per-role CLI timeout via meta/tests.yml
svc-runner's CLI test runs a full nested reinstall deploy and needs
75+ minutes, so the uniform 3600s timeout killed it deterministically
in both modes. A new role_tests lookup reads roles/<id>/meta/tests.yml
(cli.timeout, scaled through the timeout lookup) with the existing
default as fallback, keeping the short hang-leash for every other CLI
test. svc-runner gets 4h and pins the nested deploy to variant 0; the
second matrix round only redeployed cdn+dashboard and added no
assertion value.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 21:13:40 +02:00
kevinveenbirkenbachandClaude Fable 5 4a23e0cb09 feat(services): declare locally-built images via services.<key>.custom
The custom flag moves from 33 hardcoded template kwargs into the merged
service config: 'custom: true' names the shared <entity>_custom build,
a string names a second distinct '<string>_custom' image inside one app
(nextcloud whiteboard). The image lookup reads the declaration as a
fallback, so config consumers (e.g. backup image matching) resolve the
exact deployed reference from one declaration point. nextcloud cron now
runs the nextcloud_custom build like the app service instead of the
upstream image.

A new services lint couples the declaration to reality in both
directions: custom requires a role Dockerfile, and a role Dockerfile
requires a custom declaration. The eight legacy roles deploying local
builds under upstream tags or ad-hoc image= overrides are ratcheted via
'# nocheck: dockerfile-custom'.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 19:48:53 +02:00
kevinveenbirkenbachandClaude Fable 5 e917f0860d refactor(lookups): split bare-ref image resolver out of container_image
New 'image' lookup is the SPOT for resolving <registry>/<image>:<version>
(swarm registry prefix in swarm mode, bare in compose; custom= accepts a
string base for a second distinct custom image inside one app).
container_image becomes a thin wrapper that forwards terms+kwargs and adds
the compose image: "<ref>" line, so all existing template call sites keep
their output byte-identical. The lookup-usage integration test now counts
lookup_loader.get() delegation as production usage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 19:48:53 +02:00
kevinveenbirkenbachandClaude Fable 5 781e787550 fix(opentalk): grant the embedded seaweedfs sidecar its consumer identity
The OpenTalk controller v0.34.1 verifies its bucket via S3 ListBuckets
membership, but SeaweedFS hides every bucket from an unconfigured
identity (confirmed identical on 4.38 and 4.39), and the embedded sidecar
started with no s3.config, so the controller's signed request got an
empty bucket list and crash-looped on 'bucket missing' although the
bucket existed. The sidecar now runs s3.configure to register the
consumer's access key with Read/Write/List/Tagging on its bucket, the
same grant the central provider gets from its mounted s3.json.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 19:48:53 +02:00
kevinveenbirkenbachandClaude Opus 4.8 be1a22661a feat(lookup): central onion-aware canonical_url resolver + enforcement
lookup('canonical_url', app_id[, key][, consumer=app]) resolves the web base
URL for an app or one of its named canonicals (element/synapse/filer),
following the onion-injected merged-domain map: .onion targets resolve to
http (Tor provides transport security), everything else follows TLS_ENABLED.
The consumer= kwarg returns '' unless the consuming app has the target's
service binding enabled, replacing the recurring
"(lookup(...) if lookup('config', app, 'services.X.enabled') else '')"
template branching.

- utils/cache/domains.py: inject the node onion into dict-form
  (named-canonical) domain declarations, not just list/str forms
- keycloak redirect_uris: onion redirect URIs are http
- 13 playwright.env.j2: manual scheme branching -> canonical_url(consumer=)
- lint: forbid manual '://' concatenation and to_onion_url in role templates
- unit tests: 10 cases (primary/named x clearnet/onion, consumer gating)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 10:56:13 +02:00
kevinveenbirkenbachandClaude Opus 4.8 bbb1220d7d feat(web-svc-mirror): first-party privacy proxy for third-party assets, coupled to tor
New web-svc-mirror role: an nginx vhost that proxies and caches exactly
the external asset hosts aggregated from all deployed applications' CSP
whitelists (external_asset_origins filter) under first-party paths
(/<origin-host>/...), refusing every other upstream. svc-net-tor
consumes the service, so any tor deployment automatically serves the
mirror on its onion.

For apps with services.tor.enabled the shared body filter rewrites
external asset URLs in HTML to the onion mirror (onion sessions only;
clearnet untouched) and the CSP builder allows the mirror origin in the
asset directives. The dashboard's keycloak-js vendoring is reverted:
onion sessions now load the adapter through the mirror, clearnet loads
jsdelivr directly as before.

Verified: variant=0 deploy green (failed=0), dashboard 14/14 over onion
with keycloak-js served through the mirror, mirror spec green, health
check accepts the 404 vhost root.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-12 10:56:13 +02:00
kevinveenbirkenbachandClaude Fable 5 e2f8d4c304 refactor(container): drive exec timeout and CA paths from group_vars SPOTs
Two wrapper behaviours move onto single sources of truth:

Exec timeout: SYS_TIMEOUT_CONTAINER_EXEC (07_schedule.yml, default 7d so a
hard kill never beats a production migration; the dev/CI inventory
tightens it to 60min) reaches every task host via the play environment as
CONTAINER_EXEC_TIMEOUT. The wrapper prefixes only the exec subcommand with
coreutils timeout --kill-after=15. Ansible task timeouts are SIGALRM-based
and slept through the last wedged docker socket for 4.5 hours; the OS
level kill still fires there and returns 124, which until/failed_when and
the rescue diagnostics react to. Unset or 0 keeps todays behaviour.

CA container paths: the bind-mount targets for the injected root CA and
wrapper script lived as string literals in container.py, compose_ca.py
and the filter plugin. They now live once in the CA_TRUST group_vars SPOT
(inject_cert_container, inject_wrapper_container) and reach container.py
as env vars (play environment plus CA_TRUST_SYSTEMD_ENV) and compose_ca.py
as CLI arguments built by the compose_ca_inject_cmd lookup. A caller that
provides the host CA env but not the container paths is half-migrated and
fails loudly instead of silently dropping CA injection; a host without any
CA env keeps the soft fallback. All tests read the paths from the SPOT,
and a regression test asserts the runtime scripts carry no literal copy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 10:58:28 +02:00
kevinveenbirkenbachandClaude Opus 4.8 8519642373 refactor(storage): SPOT the nfs export base and client version in their roles
storage.nfs shrinks to the deployment-specific server address: export_base moves into svc-storage-nfs-server's services.yml (the server dictates where exports root) and the mount protocol version into svc-storage-nfs-client's services.yml (the client owns how it mounts). Both are read through utils.storage.nfs loaders by the nfs lookups, the env handler and 00_topology, eliminating the unrendered-variables class for these values; the group_vars assert shrinks accordingly and the swarm test extras stop restating role defaults.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 23:54:32 +02:00
kevinveenbirkenbachandClaude Opus 4.8 29184e38da fix(storage): resolve the nfs state subdir from the python SPOT
Lookup plugins receive ansible variables unrendered, so routing the state subdir through a group_vars jinja value handed state_path the literal '{{ lookup(...) }}' string and the controller mount asked the server for /srv/nfs/{{ lookup('nfs_state_subdir') }} (Connection refused, nextcloud swarm roundtrip). Both nfs lookups now import STATE_SUBDIR directly; the intermediate lookup, the group_vars indirection and the unit-test fixture var that masked the gap are gone.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 23:04:40 +02:00
kevinveenbirkenbachandClaude Opus 4.8 a7fbb94972 fix(swarm-test): bound diagnostics against wedged dockerd and SPOT the nfs state path
A wedged DinD dockerd hung a single diagnostics docker exec until the 360-minute job timeout cancelled the job together with its artifact uploads (job 86013153099 sat 5h in service logs). Route every diagnostics exec through a timeout wrapper, tail-limit the log dumps, bound showmount, cap the collect step at 30 minutes, and dump the pending services' log tails once on the final converge failure. The NFS state path moves into a single source of truth: STATE_SUBDIR in utils/storage/nfs.py feeds the nfs_state_subdir lookup (deploy side) and the derived INFINITO_SWARM_NFS_STATE_PATH env var (harness side); the diagnostics script requires both path vars instead of defaulting, and the swarm inventory drops its redundant export_base restatement.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 17:30:36 +02:00
kevinveenbirkenbachandClaude Opus 4.8 cfa368ceff fix(opentalk): break the swarm bucket/client provisioning deadlock
The controller crash-looped on a missing S3 bucket and the recorder on an unprovisioned OIDC client while the swarm converge gate blocked before any post-flush task could create either. The embedded seaweedfs sidecar now bootstraps its consumer bucket itself at startup (new seaweedfs_sidecar_script filter: entrypoint-wrapped server, bounded readiness poll, idempotent weed shell bucket create, TERM/INT forwarded to the server), and the Keycloak clients are provisioned before the handler flush. Post-flush bucket tasks stay as an idempotent backstop; the obsolete restart tasks are dropped. Live-validated against chrislusf/seaweedfs:4.38: bucket created, stop behavior identical to the stock entrypoint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 17:30:36 +02:00