Commit Graph
452 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5 47467f6cce style: settle the formatter and ruff rewrites of this change set
make autoformat reflowed a resource_filter call, a generator expression and five test files it touched along the way. The new architecture test also carried the two findings the first gate run stopped on: three RUF012, where a TestCase class attribute needs ClassVar, and two PERF401, where an append loop becomes list.extend over a generator.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-06 00:23:15 +02:00
kevinveenbirkenbachandClaude Opus 5 b182c032ac fix(web-app-docs): assert the seal's mode bits instead of the kernel's refusal
The prepared tree is reused through hardlinks, so its files share inodes and a build writing a source file in place would rewrite the cache for every later reuse. _seal drops their write bits to make that a loud failure, and the test proved it by expecting a PermissionError on a write.

pytest runs as root in the test container, and root writes through missing write bits, so the test failed there while passing on a host. It now asserts st_mode & 0o222 == 0, which is what _seal actually does and holds for any user, and keeps the write attempt as an enforcement probe that is skipped under root rather than reported as a defect.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-06 00:21:53 +02:00
kevinveenbirkenbach 4b1972b165 Merge branch 'main' into feature/store-i18n 2026-10-05 22:25:40 +02:00
kevinveenbirkenbachandClaude Opus 5 0bb437d9ce feat(images): check declared architectures against the registry, and make a pin say why
Nothing held a role's architectures to reality: test_image_reachable proves a tag exists, test_arm64_images proves four named roles publish arm64, and the closure lint only proves the declarations do not intersect to nothing. An amd64-only bump became a scheduling failure on an arm64 row rather than a red test.

A new external test sweeps every pinned image and judges a role by the intersection over its services, because a role runs only where all of its images run; judging per image would demand arm64 from an amd64-only role's nginx and call a narrowing stale because nginx publishes arm64 while the application does not. A role declaring nothing must cover every architecture the matrix rotates; a role that narrows must narrow to exactly its capability, which is what retires its TODO by itself. An unread image downgrades the stale verdict to a warning, since the image a probe could not read may be the one the narrowing exists for. The probe reads the mirror first and caches on disk, because the sweep is ~160 manifests and Docker Hub counts its allowance per IP, shared.

A new lint requires every architectures declaration to carry a nocheck marker and a Reason. bigbluebutton and opentalk keep their pins with measured reasons; decidim, pretix and taiga lose theirs, because each builds its own image and their upstream Dockerfiles are arch-neutral. Their source builds are not converted yet, so until that lands the matrix may hand them an arm64 row that fails at FROM.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-05 22:11:48 +02:00
kevinveenbirkenbachandClaude Opus 5 d81af8184d fix(web-app-matrix): gate the telegram bridge in both flavors, not only in compose
mautrix-telegram cannot register without an API id and hash, and the task that dropped it when they were missing ran only in the compose flavor. The ansible flavor renders its upstream flags from MATRIX_BRIDGE_ADDONS, which carried telegram regardless, so MDAD was told to enable a bridge it could not authenticate.

The gate moves onto that variable: MATRIX_BRIDGE_ADDONS_DECLARED keeps the unfiltered selection and MATRIX_BRIDGE_ADDONS drops telegram unless MATRIX_TELEGRAM_USABLE finds both credentials non-blank. The compose bridge list built addons itself rather than reading that variable, so removing the task alone would have taken the filter off the list that drives the docker loops, the homeserver registration and the compose services; it now derives from the gated addons, and a unit test fails if it stops doing so.

MATRIX_BRIDGES deliberately stays a compose set_fact. As a role variable it would also be defined in the ansible flavor, where the playwright env would start emitting MAUTRIX_<BRIDGE>_ADDON_ENABLED and run addon specs that flavor has always skipped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-05 20:50:03 +02:00
kevinveenbirkenbachandClaude Opus 5 6cd14365b5 fix(web-app-docs): name the preload list so the playwright lint can stub it
scripts/lint/playwright.sh derives a stub value per env key by name suffix, and only *_JSON, *SLUGS and *OVERRIDE get []. Everything else gets the literal string stub, which decodeDotenvJsonList rejects at module scope, so the spec contributes no tests and playwright test --list reports none. The exported key becomes DOCS_I18N_PRELOAD_LANGUAGES_JSON on both sides; the Ansible variable keeps its own name, because only the wire name is what the stub matcher reads.

The prepared-tree fixture is walked with rglob, which the raw-walk guard flags so repo scans share one cached walk. A tempdir is the opt-out that rule names, and its marker has to sit on the call's own line, so the call moves onto one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-05 20:40:07 +02:00
kevinveenbirkenbachandClaude Opus 5 7e1c2d1273 test(gpu): make the lookup's consumers assert wiring instead of passing vacuously
Collapsing the three GPU call sites into one lookup moved the whole decision behind lookup('gpu', ...), but both consumers' tests still stubbed lookup with a constant. One test per file went red; three more kept passing for the wrong reason, because the stub's {} is falsy and every negative case was therefore true no matter what the template did.

The stubs now branch on the lookup name and record what they were asked, so each file guards what it still owns: the answer reaches the compose key or the role variable, and the lookup is asked for this service. The declared-flag-times-device matrix stays with the lookup's own tests, which is also where the boolean() over bool() rationale now lives, checked rather than written beside the call it protects.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-05 20:34:19 +02:00
kevinveenbirkenbachandClaude Opus 5 9e8c57748b feat(svc-ai-ltengine): build against CUDA only where the host has a card
The engine was a CPU-only Debian build, so a GPU host translated on the CPU. The runtime base is now nvidia/cuda and the builder its matching -devel tag, which carries no Rust, so rustup installs the pinned toolchain the rust:slim builder used to ship.

The cargo feature hangs off the gpu lookup rather than being unconditional: a CUDA runtime image ships cuBLAS but never libcuda.so.1, which the nvidia container runtime injects, so a CUDA-linked binary on a card-less host would fail to load. The -cuda suffix travels with the feature into the image tag and into cache_from, or a host that gains or loses a card would serve the image the other state built.

svc-ai-ltengine declares huggingface.co, whose weights its hf-hub dependency downloads on first start, and svc-ai-ollama declares registry.ollama.ai. The first charged svc-ai-s1 with fetching from a host it only names as its info.yml homepage, so the ownership lint now reads info.yml as prose the way it already reads a README.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-05 20:29:21 +02:00
kevinveenbirkenbachandClaude Opus 5 04d746e339 feat(web-app-docs): reuse one generated tree per ref and build several languages at once
A translated site repeated the whole checkout and generation of its version, and the builder ran one marker at a time. The prepared tree is now cached per (ref, tooling digest), hardlinked into each build's scratch and sealed read-only, and a bounded thread pool claims the markers a free slot can build now - a language only once its version is current, so a stale one is no longer claimed and bounced in a spin.

services.docs.build.jobs and services.docs.build.parallel replace build_jobs and default to half the host's CPUs, parallel capped at 2 so the two knobs cannot multiply past mem_limit. DOCS_I18N_PRELOAD_LANGUAGES replaces DOCS_I18N_LANGUAGE, defaults to the host language and warms [en, de] in both matrix variants, where docs_sites is backed up under swarm so the DR drill restores the built sites instead of rebuilding them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-05 19:58:17 +02:00
kevinveenbirkenbachandClaude Opus 5 475ab9b386 fix(images): leave a retired role out of image discovery and the pin watch
0a3da88912 closed the EOL contract at the declaration level: the four retired
roles carry `enabled: false`, an end-of-life README banner and three lints.
iter_role_images still enumerated their images, so the mirror job kept trying
to copy tags nobody can consume. MinIO's quay tags answer 401 to anonymous
pulls and exist nowhere else, which failed that job on every run while no
deploy could have reached the result: an eol role sits outside the lifecycle
envelope default.env tests.

role_lifecycle() reads the stage off the one entry that carries a `lifecycle`
field, which is the role's primary service, and iter_role_images skips the
retired ones. Measured against the tree: 202 images, with web-app-confluence,
web-app-jira, web-app-minio and web-app-phpldapadmin all absent.

test_version_pins_are_watched.py gains the same skip. utils/update/docker.py:271
builds its watched set from iter_role_images, so dropping the retired roles
there made their `version:` pins look unwatched and the lint went red on three
of them. The coherent reading is that a role no updater can reach cannot
declare a watcher either, and the lint docstring now says so rather than
claiming every pin.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-05 18:30:50 +02:00
kevinveenbirkenbachandClaude Opus 5 819d710aaa fix(gpu): decide GPU acceleration in one lookup instead of at every call site
Whether a service gets a GPU needs two facts to agree: the declaration in
applications under services.<service>.gpu, and sys-svc-container's stat of
/dev/nvidia0 on the target host. The conjunction was restated at each call
site, and web-svc-libretranslate restated only the first half: it appended the
-cuda image suffix from the declaration alone. That tag has no arm64 manifest,
so the build died with `no match for platform in manifest`, which 10 of the 79
failing jobs of run 37252453141 carry, surfacing through
sys-svc-compose/handlers/{swarm,compose}.yml.

lookup('gpu', application_id[, service]) is now the single place that answers
it. Three call sites collapse onto it: both libretranslate roles' *_GPU var
and the runtime line of sys-svc-container's resource template. A lookup runs
on the controller and cannot stat the target, so it reads the register out of
the play's variables; the applications view comes through lookup_loader
because test_no_cache_imports_in_lookups forbids a lookup the direct
utils.cache import.

The coercion is ansible's boolean(), not python's bool(). The call sites this
replaces read the flag through Jinja's `| bool`, which answers False for the
strings "false", "False", "0" and "no" where bool() answers True, and a
templated declaration renders to such a string. Measured over 13 values the
two disagree on four of them; boolean(strict=False) matches `| bool` on all
13. Two tests pin it, and with bool() restored exactly the "false" string test
fails.

Measured rather than argued: over the merged applications, 1688
(application, service) pairs resolve the predicate identically under both
coercions, 0 disagreements. The predicate turns on for 21 services, not 5,
because resource_filter falls back to services.<entity_name>.gpu and every
service of a role whose entity declares gpu inherits it.

The assert_gpu include in both roles dropped its two trailing conditions: they
re-checked the register that *_GPU now already contains.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-05 17:45:00 +02:00
kevinveenbirkenbachandClaude Opus 5 df557f7b80 feat(i18n): translate the catalogs through the gateway and re-ask a rejected entry
666312dfbd settled the catalog client on the dedicated LibreTranslate. One
engine cannot be compared against itself, so the direct path had no way to
improve on a language it translates badly: of the 2854 entries of the `ko`
catalog it produced 1 translated entry and 2704 `structure` rejections.

The client now asks the gateway, which fronts several engines and keeps a
history of which one survives a given signature. Protected spans are masked
as `<x id="N"></x>` before the request and declared per item, so the gateway
can reject an answer that altered them instead of passing it on. `_reask`
then retries each rejected entry with the engine that produced it in
`exclude`, which reaches a different backend rather than the same one. The
same catalog measured 17 translated entries with 2649 rejections once the
gateway checked the markup, and 331 with 796 once the retry could exclude.

The full picture a rejection needs is only available in the client: 2431 of
those 2649 structure rejections involve markup the masking hides from the
gateway, so the engine-switching decision belongs on this side of the call.

`server()` reuses a gateway that already answers and otherwise deploys the
bundle the gateway needs. It does not reintroduce the throwaway container
666312dfbd removed: there is no `docker run` here, the roles deploy through
the normal path and their models still come from the role images built
through the package cache. `redeploy=true` is offered because the reuse check
only asks whether a gateway answers, which cannot see that the gateway's own
code changed since it started. A run refuses outright while a deploy is in
progress, because both touch the same container stack, and leaves the deploy
alone.

`deployed()` additionally requires a non-empty catalogue, so a gateway that
answers while serving no pairs counts as not ready rather than as a backend
that refuses everything. `probe(url)` is split out of `deployed(root)` so one
readiness check serves a single engine and the gateway alike.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-05 17:13:49 +02:00
kevinveenbirkenbachandClaude Opus 5 c33d025bab fix(web-svc-translate): judge an engine on the markup it returns, not only its terms
The gateway masks protected spans as `<x id="N"></x>` before it asks an
engine, so an engine that drops or invents that markup returned an answer the
catalog client later rejected as `structure`. `_mangled` only checked that the
protected terms survived, which let a mangled skeleton count as a win and be
recorded as one. It now compares the markup characters as a multiset between
source and answer: reflowing is allowed, losing or inventing a bracket is not.

Measured on the `ko` catalog, 2854 entries, as the three parts landed: 1
translated entry with 2704 `structure` rejections on the direct engine path, 17
with 2649 once the gateway checked the skeleton, 331 with 796 once a rejected
entry could be re-asked against another engine. `candidates` therefore takes
`exclude`, so a retry reaches a different backend instead of the same one.

`_judged` is split out of `_choose` for a second reason. A decider that timed
out escaped as a 500 and threw away replies already in hand; it now degrades
to None, the fallback winner still answers, and no history row is written.
Charging the fallback engine a win the decider never awarded would let
`saturated` freeze a signature on the configured order and stop the learning
the log exists for. Every reply is still cached; only the row is conditional.

`format` becomes a parameter of `Engine.translate` instead of the hardcoded
"text" that would have destroyed every masked entry, `catalogue()` reads the
pairs a backend reports itself, and `fmt` enters the cache key so a text and an
html answer for one string cannot alias.

The canonical root answers a JSON body naming the service instead of
redirecting to /docs, and accepts HEAD. 90c448c747 introduced that redirect;
meta/domains.yml gives this role the canonical domain api.translate, and
sys-ctl-hlth-csp runs csp-checker against every deployed domain, so the
redirect sent the checker into the Swagger UI whose inline scripts violate CSP.
A HEAD against the root answered 405 and a GET 307, neither of which is a
usable probe result.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-05 17:06:55 +02:00
kevinveenbirkenbachandClaude Opus 5.5 4c955d080f feat(update): fail a declared update source whose pin is not a semver
The version-source resolver orders upstream versions by semver. A pin that
is not one, such as `latest` or a branch name, has no depth, so no candidate
ever matches it and the declaration never moves anything. Nothing reported
that: the watched-pins lint only looks at values that already are a semver.

test_version_keys now fails on an `update:` block whose pinned key does not
hold a semver. A monitored addon the resolver does not read is held to the
same rule as soon as it pins a `version`, because the catalog check skips a
non-semver pin just as silently. An addon without a `version` keeps tracking
the app default and is left alone.

The Stalwart README no longer asks for `webui_version` to be bumped by hand
next to the image tag. Its `update:` block names the upstream tags, the
server itself follows `releases/latest` of the bundle, and the resolver
plans v1.0.5 to v1.0.11 for it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 12:48:22 +02:00
kevinveenbirkenbach a306b225c3 Merge remote-tracking branch 'origin/main' into update/onlyoffice-nextcloud-connector 2026-10-05 03:13:35 +02:00
kevinveenbirkenbach 7536f489e2 Merge branch 'main' into feature/store-i18n 2026-10-05 03:12:29 +02:00
kevinveenbirkenbachandClaude Opus 5.5 69fceafb5d fix(ci): limit the fingerprint shortcut to branches of this repository
The duplicate check accepted any open PR whose head branch started with
the dedupe prefix and ended in the fingerprint of the fresh proposal. The
listing also returns PRs from forks, and the fingerprint can be derived
from public data, so a fork PR with such a branch name would have made the
bot skip its own update PR without carrying the change.

The shortcut now only applies to PRs whose head branch lives in this
repository, which only accounts with write access can create. A fork PR
is still compared by the content of its head, as before.

Raised by the CodeRabbit review of PR #781.

Refs #780

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 03:06:57 +02:00
kevinveenbirkenbachandClaude Opus 5 48db7feb5f test(i18n): guard the backslash a rejection sample must not keep
The expectation in test_a_rejected_translation_never_spills_over_its_comment_line encoded the old mechanism, an escaped newline, which is what introduced the backslash that broke docs/tr and docs/zh. The invariant its name states still holds with a space, so the expectation moves and the fix stays.

A second test asserts the property that actually prevents the defect: a sample containing [test\\_file.py] leaves no backslash in the comment, whatever babel does when it wraps it. Nothing guarded that before, which is why two unparseable catalogs only surfaced under a manual msgfmt run.

make test-unit passes in the container: 7171 passed, 9 skipped, no failures.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-05 02:51:05 +02:00
kevinveenbirkenbachandClaude Opus 5.5 739c78bad1 feat(update): bump addon pins through their declared source
The version-source updater only read meta/services.yml. An addon pinned in
meta/addons/<id>.yml stayed where it was: the ONLYOFFICE connector for
Nextcloud sat on v10.1.2, which does not install on Nextcloud 35, until
someone bumped it by hand.

The updater now reads addon files too. An addon declares its source in the
`update:` block it already has, beside monitored, catalog and upstream_id.
A monitored addon of the github-releases catalog needs nothing more: its
upstream_id names the repository whose tags are its versions. The
update-version-sources job opens the PRs, one per role.

Every pinned addon is wired:

- nextcloud onlyoffice carries its own version pin now. Its archive URL
  stays a literal that carries the pin, because
  test_pinned_addon_archives_available reads it as written
- wordpress mcp-adapter, moodle auth_oidc and jellyfin sso-authentication
  follow the tags of their repository, moodle within its v4.5 series
- the three xwiki extensions read their Maven metadata
- pretix-oidc reads the PyPI simple index. The JSON API answers 404 behind
  the package cache the test stack routes pypi.org through
- joomla's plugin is built here and jellyfin's ldap-authentication is bound
  to the md5 of its archive; both are opted out with that reason

A bump rewrites the pin and the config.archive line together, and an
archive that does not carry the pin fails test_version_keys, so the two
cannot drift apart again. test_version_pins_are_watched covers addon pins
as well: a new one without a source or a reasoned opt-out fails the lint.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 02:50:48 +02:00
kevinveenbirkenbachandClaude Opus 5 81ab6e4ffc feat(update): let a version pin declare its own upstream source
The Docker updater watches image plus version, the repository updater
repository plus ref. A dozen pins fell outside both - an app_version, a
release, a tag in a registry neither speaks - so nothing ever told anyone
they had gone stale.

Such a pin now names its upstream next to itself in an `update:` block:
git_tags, registry_tags, npm, http_regex or a role-local script, with
`match` selecting a tag flavour and `strip` removing a prefix the pin does
not carry. utils/update/source.py resolves them beside the two existing
updaters, `python -m cli.contributing.update.source` bumps them, the
update-version-sources job opens one PR per role, and
tests/external/update/source warns while a bump is pending.

Two lints hold the rule: every semver pin is watched by one of the three
updaters or opted out with a reason, and a semver only ever sits behind
`version`, `<what>_version` or `ref` - which renames moodle's release to
app_version and simpleicons' local_version to local_tag, the latter being
the tag of a locally built image rather than an upstream version. A
declaration naming a key the entity does not carry, or a type the resolver
does not know, fails the lint instead of resolving nothing in silence.

Picked from feature/store-i18n (94d607fd0a) onto current main. Two things
differ from the original: svc-ai-lmstudio no longer carries an app_version
pin, so its declaration is dropped, and web-app-stalwart's webui_version,
added since, declares the git tags of the webui repository.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 02:50:48 +02:00
kevinveenbirkenbach b5a8828980 Merge branch 'main' into feature/store-i18n 2026-10-05 01:28:08 +02:00
kevinveenbirkenbachandClaude Opus 5 4e5b3ce26f fix(web-svc-translate): drop a search field Weblate cannot parse
WeblateMemory.reviewed appended `AND source_language:{source}` whenever the caller declared a source language. Weblate has no such unit search field and answers `Could not parse query string: Unknown search field: source_language` with 400, so the memory could never return a hit. Nothing surfaced it: Gateway._reviewed swallows every exception by design, so each request fell through to the engines as if the memory simply held nothing.

The clause is removed rather than corrected, because Weblate exposes no unit-level equivalent; the exact source match plus the target language and the approved state already pin the unit. The source language stays on Gateway.translate, which the router needs.

Two unit tests asserted the broken clause and passed, because the FakeClient records the query without ever putting it to Weblate. One is now a guard that fails on any field outside the set Weblate parses. Verified against the running instance: the old query answers 400 and the new one returns the seeded approved unit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-05 01:06:49 +02:00
kevinveenbirkenbachandClaude Opus 5 73b594b004 fix(mirror): keep a variant tag out of the version a role deploys
d6d7ff8c9e made iter_role_images yield one ref per version_variants suffix so
the mirror copies the extra tags. Two of its four consumers assumed one ref per
service. The resolver writes services[service] = {image, version}, so the last
ref won and the applications override carried v1.9.6-cuda; the engine role
appends -cuda itself and asked the mirror for v1.9.6-cuda-cuda, a tag nothing
ever pushed. The version updater builds one entry per ref and writes back into
meta/services.yml, so it would have replaced the declared version with a cuda
tag once this branch reaches the bot on main.

ImageRef now carries `derived`. The resolver and the updater skip those refs;
the mirror sync keeps iterating all of them, which is what the variants exist
for, and the mirror wait already keyed on the full ref including the version.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-04 21:15:04 +02:00
kevinveenbirkenbachandClaude Opus 5 ecfefbeae9 style(tests): wrap the variant declarations the formatter reflowed
Autoformat output from a quality run, carried along since. The syntax tree is
unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-04 14:39:09 +02:00
kevinveenbirkenbachandClaude Opus 5.5 9bdf33b89d fix(ci): treat an edited update branch as a duplicate of its own bump
The update bot recognised an open PR as a duplicate by hashing the blobs
of the files its commit changed, read at the PR head. A follow-up commit
that edits one of those files changes the hash, so the next run opened a
second PR with the identical bump: PR #778 next to PR #775 for
web-svc-collabora, after a fix had edited the bumped services.yml.

The branch name of a bot PR ends in the first seven characters of the
fingerprint of the change the bot proposed. An open PR whose branch ends
in the fingerprint of the fresh proposal now counts as a duplicate before
its head is compared. The head comparison stays as the second criterion.

The test runs pr.sh in a throwaway repository against a stubbed gh. It
fails on the previous script with exit 1, because that one goes on to
push.

Closes #780

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 14:36:57 +02:00
kevinveenbirkenbachandClaude Opus 5 671dd2f192 test(lint): require a block from two repeated when runs, and read the list form
c61c68e903 set the threshold at three and matched only the inline "when: X"
form. A guard written as a list escaped it entirely, although its first element
carries the same shared condition, so a copy-pasted guard passed whenever the
task also carried a probe-dependent clause. _first_list_item reports that
element.

Lowering the threshold to two turns 72 runs across 68 role task files into
blocks, and moves the runs above three linked repetitions into include files of
their own.

test_credential_rotation.py walked the openbao task tree and compared guards as
strings. The blocks put tasks one level deeper with accumulated guards, so it
now collects them through leaf_tasks_of and guarded_blocks_of and compares
against lists.

Equivalence is measured rather than argued: expanding both the committed and
the new tree through include_tasks and diffing the effective when of every task
leaves no difference across 73 files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-04 14:36:03 +02:00
kevinveenbirkenbachandClaude Opus 5 d6d7ff8c9e feat(mirror): let a role declare the tags it derives from its version
Four libretranslate jobs in run 37173067619 failed with

  failed to solve: ghcr.io/…/mirror/docker.io/libretranslate/libretranslate:v1.9.6-cuda: not found

svc-ai-libretranslate-engine and web-svc-libretranslate append '-cuda' to the version
in vars/main.yml when the gpu flag is set. iter_role_images is the only enumeration
the mirror pipeline sees and it reads image and version alone, so the derived tag was
never a candidate for mirroring and resolved against a registry that has no such tag.
Upstream does publish it: Docker Hub answers 200 for v1.9.6-cuda, the mirror 404.

The suffixes are now declared next to the version they extend, and discovery yields a
ref per suffix. A suffix rather than a full tag keeps the version number in one place
when it is bumped.

The gpu flag is static: it is set in meta/services.yml, no inventory overrides it, and
resource_filter reads configuration rather than hardware. Every host therefore takes
the cuda path today, which is why all four jobs failed rather than some.

Rejected: mirror_policy: skip, which has no users in this repository and would route
these two images around the mirror to Docker Hub permanently, reintroducing the rate
limit the mirror exists to avoid. Also rejected: baking '-cuda' into version, which
fixes the mirror but removes the GPU switch.

The test pins both halves; without the discovery change its two variant cases fail
and its two base cases still pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-04 07:08:47 +02:00
kevinveenbirkenbachandClaude Opus 5 e2948332fd fix(merge): repair what the main merge left broken
The merge brought in main's new roles and carried this branch's cross-cutting
renames and lints past them, so files that never conflicted were left
inconsistent. make quality-high is green again: 4 test targets, 15 lint targets.

a05071fe78 renamed get_entity_name to entity_name. Three files main added still
used the old name and merged cleanly because this branch never touched them:
plugins/lookup/email.py raised ImportError on import, which also took down
sys-ctl-hlth-msmtp's mail_recipient lookup; web-app-stalwart/tasks/00_core.yml
applied `| get_entity_name`, a filter that does not exist (the registration in
plugins/filter/get/entity_name.py is `entity_name`), so a compose deploy of
stalwart would have failed while rendering sys-stk-front-proxy's vars; and
test_database.py patched a name the module no longer binds.

web-app-stalwart and web-app-mailu spell DEPLOYMENT_MODE out in 15 places. The
named booleans in group_vars/all/18_swarm.yml are the SPOT and the bare form is
the idiom: 177 existing uses, and web-app-listmonk/tasks/utils/mcp.yml:42,49
drives both as mutually exclusive `when:` conditions, which a truthy string
would break.

svc-net-tor-smtp pinned aiosmtpd and PySocks inside its Dockerfile, where
Dependabot cannot see them. The pins move to files/requirements.txt; since
sys-svc-compose stages only the Dockerfile, 00_core.yml now stages the
requirements file into the build context too.

stalwart's webui_version had no upstream declared. Its three inline
`environment:` blocks stay, with a nocheck each: the role sets no
docker_compose_env, so one templates/env.j2 would also be the env_file of the
sibling containers, and clamav would receive the recovery admin credential and
the OIDC client secret.

The 13 bindings in stalwart's playwright env.js defaulted to "" and were
consumed unconditionally. Every key is emitted unconditionally by
playwright.env.j2 from a lookup that fails rather than yields empty, and
OIDC_ISSUER_URL always concatenates the realm path, so requireDotenvValue is
the correct binding. gitlab's adminUsername gets the toBeTruthy guard its two
siblings already had.

web-svc-api and svc-cache-package collided with stalwart on local.http 8094 and
8095 and move to 8098 and 8099, which exhausts the 8001-8099 band.
svc-ai-agent-broker collided with stalwart on 192.168.223.0/24 and moves to
192.168.81.0/24; the suggester only subdivides blocks already in use at the
requested prefix, so the move was validated with --block.

web-app-weblate calls lookup('email') without emitting
lookup('container_extra_hosts'), so on an onion deploy its SMTP client drops
every message without an error.

test_main.py read the primary domain from a literal while main moved
INFINITO_DOMAIN to main.infinito.test. It now takes the value from
default_domain_primary(), in the shape e2beab9f1c uses on
feature/web-app-wazuh/add-role, so that branch merges here without a conflict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-04 03:21:09 +02:00
kevinveenbirkenbachandClaude Opus 5 f2fedfc677 Merge branch 'main' into feature/store-i18n
Fourteen real conflicts plus the 66 generated READMEs.

utils/cache/files.py and its test: both sides added a skip-dir rule, resolved
as a union so the root `build/` prune and the per-root walk key coexist.

plugins/lookup/database.py and postgres_libraries.py: main's added lookups kept,
with get_entity_name renamed to entity_name as this branch renamed it.

roles/web-app-{mailu,matrix}/meta/services.yml: main's `covers: [email]` and its
MAIL_PROVIDER-gated email entry, plus this branch's gitlab/jira/n8n/jitsi
consumers, which the merge base carried on neither side.

group_vars/all/17_resource.yml: this branch's lookup('resource', ...) refactor;
main's single added literal is already covered by _POSTGRES_MAX_LOCKS_PER_TRANSACTION.

docs/requirements/035-mcp-proxy-expansion.md: this branch moved the design into
ADR 0003, so the requirement keeps only the user story and the criteria. Main's
delta inside the moved block was two new roles, carried into the ADR audit:
web-app-stalwart as adapter_candidate, svc-net-tor-smtp as privileged_control_plane.
The no_surface total said 99 while its own subcategories summed to 112.

inventories/development/default.yml: biber keeps no explicit email. Both sides fixed
the same CI failure; this branch fixed it in _apply_account_defaults, which renders
biber@{{ DOMAIN_PRIMARY }} (utils/cache/users/__init__.py:68) — byte-identical to
the address main hardcoded, so the inventory line is a second copy of one fact.

roles/web-app-{gitlab,keycloak}/files/playwright/playwright.spec.js and
roles/web-app-gitlab/templates/playwright.env.j2: union of both import sets, with
requireDotenvValue kept over decodeDotenvQuotedValue for CANONICAL_DOMAIN because
an empty value must fail rather than match every substring.

The 66 role READMEs take this side: aceb5295ef moved Cosmos, Quick Setup and
Credits to the documentation build, and test_readme.py now rejects them in a
committed README. Main's prose edits inside them are carried over by hand (the
Mailu deprecation banner, Mailu -> Stalwart in erpnext and zammad), and the two
READMEs main added fresh are stripped of the three generated sections.
roles/web-app-sphinx/README.md stays deleted; 6eb9a59d14 merged that role into
web-app-docs. The root README's generated role table was regenerated from the
merged metadata.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-04 01:57:17 +02:00
kevinveenbirkenbachandClaude Opus 5 e583c3eeb2 test(cache-conf): isolate the env the handler reads through setdefault
0d2fadce0c simplified this test and dropped the env patch the old one carried. EnvBuilder.setdefault keeps whatever os.environ already holds, and the test container injects INFINITO_CACHE_PACKAGE_FRONTEND_CONF through BASH_ENV, so the assertion compared the worktree path against its own temp directory and only failed inside the container.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-04 00:52:29 +02:00
kevinveenbirkenbachandClaude Opus 5 8cc65b207f test(stack-host-files): read back what apply just wrote, uncached
The assertion checks the file apply() created one line above. Routed through the shared cache it would compare against the state before that write, so the read stays direct and says why.

The marker sits on a hoisted variable rather than inside the assertEqual call: the formatter wraps the call across lines, and an inline nocheck only counts within the line span of the call it belongs to, so the one-liner form strands it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-03 22:54:25 +02:00
kevinveenbirkenbachandClaude Opus 5 efaed0dbb9 refactor(images): drop the pkgmgr image reference nothing builds any more
962c3d550d made the images build FROM the distribution base, which left the pkgmgr reference in the distro matrix without a producer: utils.distros still rendered it, the image-ref CLI still offered it as a kind, and no caller asked for either.

The requirements-archive target goes with them as the last build-path use of pkgmgr; its alias follows. The pkgmgr-install role is a different thing, installs tools on the target host, and stays where the deprecation lint can keep pointing at pip.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-03 21:48:47 +02:00
kevinveenbirkenbachandClaude Opus 5 0d2fadce0c feat(cache): serve every checkout's own cache declarations
A checkout that declared a new cache host rendered the artefacts and reached nothing: the Nexus proxy repository, the frontend's leaf certificate and the nginx upstream map all live in the running stack, which only the owning checkout set up. cache-apply installs all three and reloads the frontend, and compose up runs it for a checkout that merely joins the network, so the manual call is left for a declaration changed between two bring-ups.

The map source is no longer selectable. INFINITO_CACHE_CONF_SOURCE existed to let a worktree read the primary checkout's map instead of its own, but the frontend mounts one map and the install replaces it, so the checkout that applied last wins either way: the knob chose which file to point at, never who prevailed. Dropping it also retires the git subprocess that resolved the owning checkout on every env build.

Two checkouts sharing a stack therefore overwrite each other's map, which the documentation now states instead of implying a selector could prevent it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-03 21:40:34 +02:00
kevinveenbirkenbachandClaude Opus 5 ffd82294fc feat(translate): serve translation through one gateway with memory and a glossary
Requirement 040. web-svc-translate fronts the translation engines so a consumer talks to one endpoint instead of binding an engine. Its router picks per request and drops an engine after three consecutive failures until it recovers, and the strategy follows the deployment rather than a constant: with the S1 decider present it ranks by system-one judgement, otherwise by fixed preference.

Protected glossary terms travel into the engine instead of being checked afterwards, and an answer that mangles one is refused before it reaches the cache. Weblate supplies the translation memory and the glossary, and reads its admin API token out of the running instance into the token store rather than minting a second one.

svc-ai-ltengine builds LTEngine from source as a local engine. Discourse and WordPress consume the gateway as addons under the unified contract, each deriving its enabled flag from the service flag and shipping its own Playwright spec.

The decider moves out of svc-ai-litellm into roles/svc-ai-s1/files/python/infinito_decider so the gateway and the model router share one audited copy; litellm stages and mounts it with the same play that renders the hook importing it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-03 20:45:39 +02:00
kevinveenbirkenbachandClaude Opus 5 a056c2edaa fix(web-app-seaweedfs): mount the console config on the service that renders
The role's compose template names its service after the entity, which the rename turned into seaweedfs-console, while meta/volumes.yml still pointed the config mount at a role id that no longer exists and at the service key the engine consumer occupies. A mount whose service matches nothing is dropped without an error, so the console would have come up without its vhost config and still answered the HTTP probe from the nginx default page.

The remaining files are line joins the formatter produced once the shorter symbol name let the calls fit on one line; their syntax trees are identical to the previous revision.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-03 20:14:26 +02:00
kevinveenbirkenbachandClaude Opus 5.5 52fe3c9c2c style: apply make autoformat to the merged Stalwart and Tor files
Formatter output only: the parsed YAML and the Python AST of every touched file equal their previous versions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 19:57:30 +02:00
kevinveenbirkenbachandClaude Opus 5 a05071fe78 refactor(roles): let a role declare its entity name and drop the get_ prefix
The entity name keys a role's instance directory, its compose project and its container name, and was derived only by stripping the longest matching category prefix. Two roles under different categories whose names share a suffix therefore collapse onto one entity and overwrite each other's rendered stack, which no provides: can repair.

A literal entity_name in vars/main.yml now wins over the derivation. web-app-seaweedfs-console takes its shorter name back and declares the entity seaweedfs-console, which frees the seaweedfs key in its own services map for the engine consumer and removes the provides:, covers: and alias entries that stood in for it. A new lint pairs the declaration with the primary key it has to match, because a drift between them would drop the role from the provider set in silence.

Compared across all 288 roles, exactly one entity changes, and it restores what main already keeps on disk, so no instance directory moves.

The function and the filter lose their get_ prefix. Two lints read that prefix as the discriminator between the variable and the filter that fills it; both discriminate on the pipe now. Ansible's templar resolves a filter and a same-named variable independently, verified against ansible-core 2.20.2.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-03 19:40:53 +02:00
kevinveenbirkenbachandClaude Opus 5 b83ab8e596 feat(variants): pin a partner's variant, and provision the OpenProject GitLab addon
Two concerns, staged together on request.

Variant pins
------------
A matrix round hands every role the round index and clamps to variant 0 when
the role has fewer entries, so a cross-role pairing holds only while two
independently maintained variant lists agree on an index, and it breaks
silently once the partner runs out of variants.

'# variant-pin: <role>#<index|reciprocal>' above a service entry states the
dependency instead of inheriting it. It is a comment, so it never reaches the
merged services map or host_vars the way a YAML key under the service would.
'#reciprocal' resolves to the partner's variant that switches this role back
on, so neither side tracks the other's index. Two roles pinning one partner to
different variants raises VariantPinConflictError before an inventory is
written.

web-app-openproject is the first user: a fourth variant carries the nextcloud
round, pinned reciprocally. The fourth round exists because folding it into
variant 2 next to gitlab was measured at 35.58 GB against the 32 GB
reservation cap, and it has to be the last round because a role with fewer
variants falls back to variant 0 in a later one.

Measured: the pin changes no role's index in any of the 14 rounds today, so
the mechanism sits in the deploy path without moving a single inventory.

OpenProject addons
------------------
c83c00da7d gave the role meta/addons/ and three task hooks, but the role never
included tasks/addons/ at all, so all three hooks were dead files. 05_addons.yml
now resolves the web host and container id and runs the hook of every addon
whose gate resolves true.

gitlab_integration is provisioned: a bot account, its API token and the plugin
setting carrying gitlab_user_id and webhook_secret. The GitLab-side webhook is
not created here and the hook prints the URL to register it against.

storages keeps an empty hook deliberately. web-app-nextcloud owns both halves
of that integration and runs eleven positions later in the deploy order, so at
this hook's dispatch point the storage it would touch does not exist yet.
openid_connect stays off: OIDC is Enterprise-gated at this version.

Probing the pinned openproject/openproject:17 image corrected two defects
before they could ship: the API token was minted without the name the script
fetched but never used, and the Playwright webhook probe omitted the
X-Gitlab-Event header that HookHandler dereferences before any auth check, so
the smoke request answered 500 where the spec demanded less than 500.

The addon specs now assert the integration rather than the route: storages
requires a registered storage addressing the partner Nextcloud, and
gitlab_integration requires the bot account the webhook authenticates as.

Deploying this needs a regenerated inventory: gitlab-web-app-openproject is a
new user key and users_generator pins it at inventory build time.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-03 04:30:30 +02:00
kevinveenbirkenbachandClaude Opus 5 d084aead95 perf(web-app-dashboard): write every port-ui catalog in one module run
lookup('i18n_content', …) already returns every language in one dict, and the
role took it apart again: one stack_host_template action per language, a file
task for the directory, a find, and a second loop dropping the catalog of a
language core no longer translates. Four tasks and two loops for what is one
mapping.

The template they looped over was not a template. i18n-content.yaml.j2 held a
single line, `{{ dashboard_i18n_content[item] | to_nice_yaml(width=100000) }}`
— serialisation wearing a costume — so it is deleted.

library/stack_host_files.py takes the whole mapping, serialises each entry,
writes only what differs, drops what the caller no longer wants and creates the
directory, in one module run. The IS_STACK_HOST gate moves to the task's `when`
because a module runs on the target rather than on the controller, where the
sibling stack_host_template and stack_host_copy action plugins gate it.

The per-language file layout stays: files/Dockerfile copies i18n-content/ into
the image, and port-ui reads one yaml per language from /app/i18n/content.

main() stays thin so render, survey, plan and apply can be unit-tested without
mocking AnsibleModule, the way library/database_query.py is tested. The cycle
test asserts the second pass finds nothing to do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-03 00:47:04 +02:00
kevinveenbirkenbachandClaude Opus 5 bed4b3c84f refactor(docs): derive the role integration pages instead of freezing them
cli/build/docs/integration_matrix was 1189 lines a human ran through
'make integration-matrix', and its output sat frozen as a 125-line grid in
requirement 027. Its ENTITIES and EDGES lists were hand-maintained, so the
grid could only drift: it covered 84 entities while the tree holds 98 roles,
named two roles that no longer exist, and omitted sixteen current ones -
api, checkmk, docs, hermes, homeassistant, jellyfin, meta, mirror, openbao,
openclaw, penpot, pihole, s1, seaweedfs, seaweedfs-console and semaphore.

infinito_docs/generators/integrations.py replaces it and joins the nine
sibling generators in that package, so the overview, services and addons
pages are written into generated/integrations/ on every docs build rather
than when someone remembers the target. index.rst takes the overview and the
new docs/architecture/integrations.rst references the other two.
Requirement 027 becomes prose. The make target and its i8inma alias are
removed as a pair, which keeps the alias lints satisfied.

Coverage is not strictly larger. The removed EDGES list carried a curated
upstream project URL per integration, 96 of them, and the new edges have no
such field; only 48 of 159 addon declarations hold update.upstream_id or
config.repository, so about a third is re-derivable and the rest existed only
in that list. Its fourth field is no real loss - 57 entries read 'check', so
it was a near-constant rather than an axis. In exchange the old module had
zero tests, not one file under tests/ referenced it, and the generator ships
310 lines including generate().

Residuals: the HTML build runs without -W, but i18n.py:57 runs the gettext
builder with -W and check=True, and that build has never seen
integrations.rst - no catalog mentions generated/integrations. And
docs/requirements/036:72 still carries a checked criterion asserting a table
in 027 that no longer exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-02 18:22:04 +02:00
kevinveenbirkenbach d90f1e8ed6 Merge pull request #756 from kevinveenbirkenbach/fix/svc-db-postgres/pgdg-bullseye-archived
fix(svc-db-postgres): build on postgres 18 since PGDG archived bullseye
2026-10-02 17:41:28 +02:00
kevinveenbirkenbachandClaude Opus 5 d45ed509eb fix(addons): gate the playwright addon flag on enabled alone, not on required
required is the install-failure switch, not an enablement switch:
docs/requirements/026-unified-addon-syntax.md:33 and :111 define it that way,
it is implemented only as failed_when guards in web-app-wordpress and
web-app-mattermost, and no variant and no inventory pins it anywhere. ANDing
it into the gate therefore held the flag of every 'required: false' addon at
false in every variant and every mode since 70786aa126 created the suite, so
those Playwright specs reported as skipped with the plausible reason 'addon
disabled or not deployed' while asserting nothing.

Measured against this commit's parent: 32 addon specs declare
'required: false', 24 of them in the 11 roles that call the lookup, out of the
89 addon specs those roles ship. Those 24 move from always-skipped to
executing. The flag reaches no deploy configuration, so the worst case is a red
Playwright job rather than an open surface - the old behaviour was the silent
one.

Both test files stated the old contract and are corrected with it.

Residual: the repairs for several of the specs this wakes - discourse's akismet
assertions and its missing sso gates, moodle's two addons that no install path
covers - are not in this commit. A round over those roles is red until they
land.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-02 17:10:26 +02:00
kevinveenbirkenbach 30417d2515 fix(svc-db-postgres): build on the official multi-arch postgres 18 image
postgis/postgis publishes amd64 only, so every postgres consumer failed
on arm64. The official postgres image serves both architectures and
already ships the contrib extensions bloom, pg_trgm and unaccent.

PostGIS moves into the postgres_libraries registry and is installed from
PGDG (postgresql-18-postgis-3) only for a consumer that declares it as
an extension, which is mobilizon alone. apt update reads every source
list again, because the PostGIS package depends on GDAL from Debian
main, which a PGDG-only update left without an install candidate.

Postgres 18 keeps its cluster in /var/lib/postgresql/18/docker and its
entrypoint treats a volume at the former /var/lib/postgresql/data as an
old database, so the central and the dedicated data volume both mount at
/var/lib/postgresql. Existing 17 clusters are not migrated: their data
directory trips the entrypoint's old-data check and the container
refuses to start.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f78ebe46a4)
2026-10-01 22:19:09 +02:00
kevinveenbirkenbachandClaude Opus 5 ff89a02067 fix(web-app-docs): take the builds nobody waits for off the awaited path
On a cold container the docs Playwright suite triggers four serial Sphinx builds and waits for the last one. test-languages.js polls /deployed/de/ for 200 and got 202 for 55.6 minutes in job 110003713825 of run 36745666606, after which the suite budget expired and five further specs never ran. The suite has never passed on a first attempt; it used to go green only on the harness retry, which 7127c6b96b removed for budget timeouts, so a structural overrun became a red job.

Measured cost per build, from run 36558500327 where the whole queue drained on a kept stack: container up 12:15:52, the spec green at 13:21:41 answering on its first poll, next container start only at 13:30:45. Four builds in 65 minutes 49 seconds, so at most 16.5 minutes each - not the sixty to ninety minutes 40688e4ded estimated.

Three of the four builds were none of the test's business. fetch() queued latest in the foreground although nothing waits on it at startup, and _refresh_stale_sites did the same for every servable version; both now use the background lane, which is what its own docstring describes. test-front-page.js reached GET /latest/, and reaching an unbuilt version queues its build - it now asserts the redirect from the Location header without following it, and checks the same page shape against /deployed/, which is built anyway. The release tag that test-on-demand-build.js clicks stays, and now runs behind the awaited work instead of in front of it.

The translation could only be queued after its base finished, because the 202 branch returns before _translation is reachable, so it always sorted behind whatever the sibling specs had queued meanwhile. The 202 branch now carries the language segment along, admitted by translatable(), which answers from the snapshot catalog on disk before any build; build_language still re-checks the written index.

Moving a version into the background lane exposed that status() read only the foreground one, so a waiting version would have been reported as missing rather than queued to the versions overview and to /api/versions. is_queued() now covers both lanes, mirroring _dequeue, which always removed from both. Two of the five unit failures this change first produced disappeared on that fix alone, which is what shows they pinned real behaviour; the three that pinned the lane itself were rewritten to assert the behaviour instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-01 11:32:08 +02:00
kevinveenbirkenbachandClaude Opus 5 be4074dadc refactor(objstore): build the credentialed S3 URL in the lookup
7b5833b37a assembled it in zammad's env template: urlencode both halves of the
credential by hand, patch them into the URL the lookup had already returned,
then append bucket and query. Three set-statements of string surgery on a value
the lookup builds from host and port in the first place.

The lookup now exposes credentialed_url next to url, built from the same
endpoint it already has, with urllib.parse.quote(safe='') on each half. The
template keeps only what is actually zammad's: the bucket path and the region /
force_path_style query.

This also removes the manual scheme concatenation that
test_onion_url_central_lookup refuses. That rule guards against a hand-built
app URL repeating the clearnet-vs-onion decision; the template was not doing
that, but it did name '://' and could therefore touch the scheme. Now it
cannot.

quote(x, safe='') is equivalent to the old
chain: checked over slashes, plus, equals, percent and space, no divergence.
Both branches of the lookup carry the key, the disabled one as the empty
string, so a consumer cannot hit a missing key.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-01 05:47:06 +02:00
kevinveenbirkenbachandClaude Opus 5 14950b4319 fix(sso): serve the console's onion hosts and keep one session across them
Three failures of the same deploy, each uncovered by fixing the one before it.

web-app-seaweedfs-console read its two vhost domains through
lookup('config', ..., 'domains.canonical.*'), which resolves against the raw
applications tree. The onion substitution happens elsewhere -- in
get_merged_domains, where svc-net-tor's exclusive flag REPLACES the clearnet
value. On an onion node the role therefore rendered vhosts for clearnet names
while Keycloak's redirectUris, the CSP header and LOGOUT_DOMAINS all carried
the onion names, so the Filer and Master surfaces had no vhost at all and the
default server answered. Reading through lookup('domains', app).<key> is the
shape matrix and bluesky already use for named canonicals.

With the onion names in place the console's own nginx refused to start:
a server_name of ~81 characters overflows the stock bucket size of 64 and
nginx aborts with 'could not build server_names_hash'. The central openresty
has carried server_names_hash_bucket_size 128 since forever
(sys-svc-webserver-core/templates/nginx.conf.j2); the second nginx needs the
same.

With both surfaces served, an administrator still had to re-authenticate on
the Master host: cookie_domains was emitted only when the app had clearnet
domains, so an onion-exclusive node got no cookie scope and oauth2-proxy fell
back to a host-only cookie. whitelist_domains on the very next line already
handled both address families; cookie_domains now does too.

Both lists move out of the template into lookups of their own. The template
held six set-statements and two nested inline conditionals to build two JSON
arrays; the lookups take an application_id and return the list, which also
makes the behaviour testable without rendering the template. The template
structure assertion in test_cookie_scope.py went with them -- it measured
Jinja text, not behaviour -- replaced by thirteen cases over the two lookups:
single host, multi-host collapse, onion-only, dual-stack, no domains, no tor
node, and a missing DOMAIN_PRIMARY that now raises instead of silently
scoping nothing.

Evidence: make compose-playwright role=web-app-seaweedfs-console goes from
3 failed to 3 passed against the live stack. For the single-domain case the
change newly affects, web-app-prometheus keeps all four of its SSO specs
green with the scope applied.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-01 04:32:38 +02:00
kevinveenbirkenbachandClaude Opus 5 70a4cb68cd test(seaweedfs): verify the storage engine from the node over its S3 port
The engine has no browser surface, so it is verified through the repository's
CLI harness rather than Playwright: test-e2e-cli discovers a role by its
templates/test.env.j2 and runs files/test/test.sh on the node that holds the
container.

probe.py asserts five things against the running store: an unsigned request is
refused with an S3 error, the admin identity completes a signed write, read
back and delete, a consumer identity reaches its own bucket and is refused
another's, the anonymous grant matches the declared public flag, and the filer
and master ports stayed unpublished after the split.

It creates no bucket. A bucket is a SeaweedFS collection and the engine's
-volume.max is budgeted for the declared consumers only, so a probe that
creates one can exhaust the volume slots its own consumers write through;
measured as a 500 on PUT once the slots ran out. Every write now lands under
one reserved key prefix in a bucket a consumer already owns.

SigV4 is hand-rolled because the CLI runner ships no boto3. Its vectors were
generated with botocore and are frozen in a unit test, which also pins the
Node signer the consumer specs use, so the two cannot drift apart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-01 02:24:16 +02:00
kevinveenbirkenbachandClaude Opus 5 e8e89d293d refactor(seaweedfs): split the object store from its browser consoles
web-app-seaweedfs becomes two roles. web-svc-seaweedfs keeps the storage
engine: the container, the seaweedfs_data volume, the per-consumer S3
identities and the single canonical domain api. web-app-seaweedfs-console
takes the filer and master web UIs into its own compose stack behind the
admin-only oauth2 proxy, reaching the engine over the shared object-store
network. A node can now run the store without the consoles.

The console needs a distinct role name because get_entity_name strips the
category prefix: both halves would have resolved to the entity seaweedfs and
collided on the compose project, the instance directory and the container
name.

The filer and master ports move into the engine's services.seaweedfs.ports
.internal as their single owner; 01_init_consumer.yml consumes the master
port and would otherwise have resolved it only by role-var leakage from the
console, breaking bucket provisioning on exactly the console-less node the
split makes possible.

The engine's api vhost gains the prometheus locations. It never had them: the
override satisfied the cross-cutting-include lint only because the strings sat
in the SSO branch that the api domain never took.

Consumers, the objstore lookup and the catalogs follow the role id.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-01 02:20:12 +02:00
kevinveenbirkenbachandClaude Opus 5 531bb3d4cd fix(users): reserve one username per domain label instead of the first two
`roles/user/meta/users.yml` reserved two names against the identity directory by
indexing the primary domain from the left: `sld` took label 0 and `tld` took
label 1. That is only the SLD and the TLD when the domain has exactly two
labels. On `label-a.tld.test` it reserved `label-a` and `tld` and let the real
TLD through, and on any deeper domain it drifts further.

The count is a runtime fact, so a static YAML file cannot express it. The two
entries are gone and `utils/cache/users` now emits one `domain_label_<n>` per
label:

    infinito.nexus      -> infinito, nexus          (unchanged)
    main.infinito.test  -> main, infinito, test     (was: main, infinito)
    label-a.tld.test    -> label-a, tld, test       (was: label-a, tld)

Nothing read the `sld` or `tld` keys: a grep over roles, plugins, utils and cli
finds only the generator itself. The entries exist to carry `accounts: []` into
`reserved_usernames`, which `web-app-keycloak/tasks/02_init.yml:6` joins into the
username deny pattern, so the shape of the keys is free and only the set of
usernames matters.

Worth stating so the next reader does not overestimate it: the pattern at
`UserProfileProvider.json.j2:31` allows `[a-z0-9]+`, so a label carrying a hyphen
could never have been registered anyway. What this actually wins is the third
label. What it mainly wins is that the rule no longer has to be revisited when
the domain depth changes.

The `KEYCLOAK_DOMAIN` and `domain` fallbacks go with it. Both ran through a
helper that recovered the primary domain by keeping the last two labels of a
host like `auth.<DOMAIN_PRIMARY>`, which cannot work without already knowing the
depth it is trying to find. `DOMAIN_PRIMARY` and `SYSTEM_EMAIL_DOMAIN` remain,
and every deploy path supplies one of them. The test that pinned the removed
fallback is inverted rather than deleted, so the removal stays visible.

The rest is the refactor pass the change pulls in. The function no longer
materialises aliases and is renamed to `_materialize_domain_label_reservations`;
its candidate tuple carried one extractor after the fallbacks left and is a plain
key loop now; an `or [primary_domain]` guard that would have reserved a nonsense
username for an all-dots domain is dropped. `roles/user/vars/main.yml` loses four
comments that restated the line below them, and the README documents the
reservation registry, which no longer shows its full content on disk.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 21:52:54 +02:00
kevinveenbirkenbachandClaude Opus 5 d88fe1d83a test(lint): let the legacy-path guard see the trees that still named the old layout
`config/main.yml` exists in no role: f65f4d60fb migrated the last one. A guard
against the string already exists, `TestNoLegacyPathReferences`, and it names
`config/main.yml` among its LEGACY_TOKENS. It scanned plugins, utils, cli,
roles, scripts, filter_plugins and lookup_plugins, which is every tree except
the two the stale references actually lived in.

So nine of them survived in `tests/`, and the guard reported green the whole
time. The code underneath was already correct everywhere: `test_csp_consistency`
and `test_role_wiring` iterate ROLE_FILE_META_SERVICES, and only their docstrings
and one assertion message still said `config/main.yml`. Had any of them globbed
the old path the test would have gone vacuous rather than red, which is the
failure mode worth naming here.

`tests` and `tasks` are now scanned. The targets come from
`utils/roles/mapping.py`, not from guesswork: `config/main.yml` is
`meta/services.yml`, `schema/main.yml` is `meta/secrets.yml` (confirmed at
`cli/administration/inventory/provision/reset.py:67`, which reads exactly that
file), and `users/main.yml` is `meta/users.yml`.

Scanning `tasks` exposed a second dead pointer. The migration tool became a
package on 2026-09-14, but EXEMPT_FILES still named `migrate_meta_layout.py`, a
file that no longer exists, so the exemption was inert and only invisible
because `tasks` went unscanned. Two dead pointers covering for each other. The
exemption is now a directory, so a new module in that package cannot fall out of
it. `test_no_direct_calls.py` named the same dead path in prose; that lint
allows through nocheck markers at the call site, so nothing was broken there.

`docs/administration/configuration/email.md` was the worst of them, because it
instructed operators rather than describing tests: it told them to write
`compose.services.email` into `roles/<application_id>/config/main.yml`. The
consumer, `plugins/lookup/email.py:255`, reads `entry["services"]["email"]` and
says in its own comment that there is no `compose.services` envelope. The role
snippet, the inventory snippet and the precedence list now match the reader.
`group_vars/all/TODO.md` pointed at the same dead file.

Verified by watching the guard fail, twice. Re-introducing `config/main.yml` in
a test file reports it; pointing the new directory exemption at a wrong path
reports `tasks/utils/migrate_meta_layout/credentials.py`, which proves both that
`tasks` is scanned and that the exemption is the only thing letting the
migration tool through. Restored, 11 passed.

The changelogs under `docs/changelog/` keep their mentions, and `.md` stays out
of the scanned suffixes for that reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 21:27:40 +02:00