Commit Graph
64 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5 f2fedfc677 Merge branch 'main' into feature/store-i18n
Fourteen real conflicts plus the 66 generated READMEs.

utils/cache/files.py and its test: both sides added a skip-dir rule, resolved
as a union so the root `build/` prune and the per-root walk key coexist.

plugins/lookup/database.py and postgres_libraries.py: main's added lookups kept,
with get_entity_name renamed to entity_name as this branch renamed it.

roles/web-app-{mailu,matrix}/meta/services.yml: main's `covers: [email]` and its
MAIL_PROVIDER-gated email entry, plus this branch's gitlab/jira/n8n/jitsi
consumers, which the merge base carried on neither side.

group_vars/all/17_resource.yml: this branch's lookup('resource', ...) refactor;
main's single added literal is already covered by _POSTGRES_MAX_LOCKS_PER_TRANSACTION.

docs/requirements/035-mcp-proxy-expansion.md: this branch moved the design into
ADR 0003, so the requirement keeps only the user story and the criteria. Main's
delta inside the moved block was two new roles, carried into the ADR audit:
web-app-stalwart as adapter_candidate, svc-net-tor-smtp as privileged_control_plane.
The no_surface total said 99 while its own subcategories summed to 112.

inventories/development/default.yml: biber keeps no explicit email. Both sides fixed
the same CI failure; this branch fixed it in _apply_account_defaults, which renders
biber@{{ DOMAIN_PRIMARY }} (utils/cache/users/__init__.py:68) — byte-identical to
the address main hardcoded, so the inventory line is a second copy of one fact.

roles/web-app-{gitlab,keycloak}/files/playwright/playwright.spec.js and
roles/web-app-gitlab/templates/playwright.env.j2: union of both import sets, with
requireDotenvValue kept over decodeDotenvQuotedValue for CANONICAL_DOMAIN because
an empty value must fail rather than match every substring.

The 66 role READMEs take this side: aceb5295ef moved Cosmos, Quick Setup and
Credits to the documentation build, and test_readme.py now rejects them in a
committed README. Main's prose edits inside them are carried over by hand (the
Mailu deprecation banner, Mailu -> Stalwart in erpnext and zammad), and the two
READMEs main added fresh are stripped of the three generated sections.
roles/web-app-sphinx/README.md stays deleted; 6eb9a59d14 merged that role into
web-app-docs. The root README's generated role table was regenerated from the
merged metadata.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-04 01:57:17 +02:00
kevinveenbirkenbachandClaude Opus 5 0d2fadce0c feat(cache): serve every checkout's own cache declarations
A checkout that declared a new cache host rendered the artefacts and reached nothing: the Nexus proxy repository, the frontend's leaf certificate and the nginx upstream map all live in the running stack, which only the owning checkout set up. cache-apply installs all three and reloads the frontend, and compose up runs it for a checkout that merely joins the network, so the manual call is left for a declaration changed between two bring-ups.

The map source is no longer selectable. INFINITO_CACHE_CONF_SOURCE existed to let a worktree read the primary checkout's map instead of its own, but the frontend mounts one map and the install replaces it, so the checkout that applied last wins either way: the knob chose which file to point at, never who prevailed. Dropping it also retires the git subprocess that resolved the owning checkout on every env build.

Two checkouts sharing a stack therefore overwrite each other's map, which the documentation now states instead of implying a selector could prevent it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-03 21:40:34 +02:00
kevinveenbirkenbachandClaude Opus 5 d88fe1d83a test(lint): let the legacy-path guard see the trees that still named the old layout
`config/main.yml` exists in no role: f65f4d60fb migrated the last one. A guard
against the string already exists, `TestNoLegacyPathReferences`, and it names
`config/main.yml` among its LEGACY_TOKENS. It scanned plugins, utils, cli,
roles, scripts, filter_plugins and lookup_plugins, which is every tree except
the two the stale references actually lived in.

So nine of them survived in `tests/`, and the guard reported green the whole
time. The code underneath was already correct everywhere: `test_csp_consistency`
and `test_role_wiring` iterate ROLE_FILE_META_SERVICES, and only their docstrings
and one assertion message still said `config/main.yml`. Had any of them globbed
the old path the test would have gone vacuous rather than red, which is the
failure mode worth naming here.

`tests` and `tasks` are now scanned. The targets come from
`utils/roles/mapping.py`, not from guesswork: `config/main.yml` is
`meta/services.yml`, `schema/main.yml` is `meta/secrets.yml` (confirmed at
`cli/administration/inventory/provision/reset.py:67`, which reads exactly that
file), and `users/main.yml` is `meta/users.yml`.

Scanning `tasks` exposed a second dead pointer. The migration tool became a
package on 2026-09-14, but EXEMPT_FILES still named `migrate_meta_layout.py`, a
file that no longer exists, so the exemption was inert and only invisible
because `tasks` went unscanned. Two dead pointers covering for each other. The
exemption is now a directory, so a new module in that package cannot fall out of
it. `test_no_direct_calls.py` named the same dead path in prose; that lint
allows through nocheck markers at the call site, so nothing was broken there.

`docs/administration/configuration/email.md` was the worst of them, because it
instructed operators rather than describing tests: it told them to write
`compose.services.email` into `roles/<application_id>/config/main.yml`. The
consumer, `plugins/lookup/email.py:255`, reads `entry["services"]["email"]` and
says in its own comment that there is no `compose.services` envelope. The role
snippet, the inventory snippet and the precedence list now match the reader.
`group_vars/all/TODO.md` pointed at the same dead file.

Verified by watching the guard fail, twice. Re-introducing `config/main.yml` in
a test file reports it; pointing the new directory exemption at a wrong path
reports `tasks/utils/migrate_meta_layout/credentials.py`, which proves both that
`tasks` is scanned and that the exemption is the only thing letting the
migration tool through. Restored, 11 passed.

The changelogs under `docs/changelog/` keep their mentions, and `.md` stays out
of the scanned suffixes for that reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 21:27:40 +02:00
kevinveenbirkenbachandClaude Opus 5 058e76e58c fix(test): read the two planner call sites through the shared cache
`test_no_uncached_file_reads` requires every read under tests/, utils/, cli/,
plugins/ and library/ to go through `utils.cache.files.read_text`, which keeps
one decoded read per file for the whole pytest session. The test added in
3b85e4e1e1 called `Path.read_text` directly and so read init.py and
deploy/cli.py past that cache.

The rule allows `# nocheck: cache-read` only for a read outside the project
tree, which is how bfdf3dee79 handled a tempdir. These are real source files,
so the reader is the answer and the marker would have been the convenient
wrong one; e5430ad4ca is the matching precedent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 02:34:39 +02:00
kevinveenbirkenbachandClaude Opus 5 3b85e4e1e1 fix(deploy): let init pin the guide host the way deploy already does
adb53392bd taught the deploy planner to pin the instructions host to its
smallest variant and to leave a pinned app out of the round total. It taught
only the deploy side. `init` writes the inventory and `deploy` reads it back,
and the directory carries a round suffix only when the plan holds more than
one round, so the two sides named different directories:

    >>> Inventory initialized at /home/runner/inventories/github-0
    === matrix-deploy: round 1/1 inv=/home/runner/inventories/github
    [ERROR] Some application_ids are invalid for this inventory
    [DEBUG] inventory file does not exist

Nine jobs of run 36596159856 died there, before ansible-playbook started. The
trigger is a subject with a single variant: init counted the host's two
variants and suffixed, deploy pinned the host out and did not. svc-ai-s1
declares two variants of its own and was the only instructions row to survive,
which is the shape of the bug read backwards.

`variant_select` opens by stating the invariant this broke - "keeps init and
deploy walking the exact same selection (SPOT)" - so the fix is to honour it
rather than to special-case the suffix.

The accompanying test asserts both entry points forward `pinned_variants`.
The five tests that shipped with adb53392bd all exercised the pin helper on
its own, which is why nothing caught a caller that never asked it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 01:40:55 +02:00
kevinveenbirkenbachandClaude Opus 5 adb53392bd fix(ci): deploy the instructions host in the shape its README documents
A row marked for the instructions replay gets `web-app-docs` appended to its
apps, and the planner hands every primary app the round's index. The host
therefore took the subject's index, which for a passenger is an arbitrary
shape. Measured in run 36558500327: the same appended role deployed
dashboard, matomo, prometheus and css in 141 tasks on a round-0 row and in
none at all on a round-1 row, purely because of whose round it was riding.

The README's Production block describes the bare role, so the host now takes
its smallest variant in every round. The index comes from `smallest_variant`
rather than a literal, so it follows `meta/variants.yml` instead of drifting
from it.

The row names the host in an environment variable. `--apps app#variant` would
have read better at the call site but needs a new matrix column in both
`axes.py` and `cli/meta/ci/plan.py`, while the deploy CLI already reads
`disable` from the environment and the four shell layers between the workflow
and the CLI pass it through untouched.

A pinned app no longer counts toward the round total: a passenger should not
lengthen the subject's sweep. That removes an unreachable round rather than
coverage, because `deployed_rounds` plans one app at a time and a role with
one variant therefore only ever yields `variant: 0` rows, which the workflow
pins on every deploy.

The host is left alone when it is also the subject. `web-app-docs` carries the
replay on its own rows too, where the apps list reads `web-app-docs,
web-app-docs` and the round index is the entire point of the sweep.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 18:00:45 +02:00
kevinveenbirkenbachandClaude Opus 5 16c853545a fix(deploy): stop the dedicated deploy from git-cleaning its own checkout
`MODE_CLEANUP` says what to tidy on the deployed system: prune unused Docker
resources, clear Mastodon caches, switch an LDAP user list from `present` to
`exact`. Eight roles read it that way. The dedicated runner read the same flag
and ran `make clean` in the control repository instead, which is
`git clean -fdX` over every ignored path there.

`inventories/production/` is one of those paths. The published host install
guide provisions into it and then deploys against `$INVENTORY/.password`, so
the deploy deleted the vault password it was about to use:

    Removing inventories/production/
    [ERROR] The vault password file .../inventories/production/.password was
    not found

Reproduced in CI on two unrelated rows, arm64/zfs/debian and
amd64/btrfs/centos, so it is neither architecture nor filesystem specific.

The coupling was a name collision, not a decision. 7e5990aa16 generated the
MODE_* flags from `01_modes.yml` and removed the legacy skip flags, and the
repository clean - which had had a flag of its own - was attached to the
generated `MODE_CLEANUP` because one now existed by that name. No other deploy
path does this; `container`, `development` and `swarm` never call `make` at
all.

`make clean` keeps its own entry points, `make setup-clean`, `make clean-sudo`
and `infinito --git-clean`, where a caller asks for it deliberately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 17:24:05 +02:00
kevinveenbirkenbachandClaude Opus 5 62a8f1ce2a feat(ci): retrigger a failed role whole and an unfinished one by variant
--roles-only collapsed the entire failed set, so a job that never reached a verdict came back as a bare role name and the rotation re-picked axes the run had already proved green, dropping the one combination it never got to. Nothing distinguished a verdict from an interruption.

collapse_failures() takes the failed set and the hard-failure subset: a role that failed outright collapses to its name, everything else keeps the exact role#variant@mode+tor it was given. A role with a hard failure absorbs its own unfinished rows, since the collapsed name already covers every variant of it.

With --strict the two arguments are the same list, so i8ciall, i8ciallon and i8cifo are byte-identical to before; a test asserts that agreement rather than leaving it to inspection. The new combination is exposed as i8civar, which is i8ciallon without --strict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 13:33:41 +02:00
kevinveenbirkenbachandClaude Opus 5 c782208f99 feat(tools): rename the i18n lane to tools and give the cache paths one spot
The lane is no longer LibreTranslate-only, so `i18n` is the wrong name for
the runner, its compose override, its inventory and its env keys. Renames
to `tools` throughout and points the lane at
`svc-ai-libretranslate-engine`, which is the role that actually owns the
`libretranslate-engine` service key.

The six cache host paths were each written out in full. They now derive
from `DIR_CACHE`, one plain string in `group_vars/all/05_paths.yml`, via a
`cache_paths` handler; the generated values are byte-identical. The handler
is registered directly after `cache_stack` rather than late, because
`registry_cache_max_size` reads `INFINITO_CACHE_REGISTRY_HOST_PATH` and
returns early when it is empty, which silently dropped
`INFINITO_CACHE_REGISTRY_MAX_SIZE` from the generated `.env` and broke
every `docker compose` invocation that layers `cache.override.yml`.

`INFINITO_TOOLS_MODELS_HOST_PATH` derives from the same base, so the model
bind the lane mounts follows the cache layout instead of repeating it.

The inventory pins `DOCKER_BIND_HOST=0.0.0.0`: the group_vars default keys
off `DOCKER_IN_CONTAINER`, which evaluates false for the inner deploy and
would leave the lane's services on the runner's loopback.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-28 23:50:26 +02:00
kevinveenbirkenbachandClaude Opus 5 a37039f0cb fix(web-app-docs): name the replayed role instead of taking the first one
The instructions test brings up a machine per role, so it can only ever
replay one guide. Which one it picked was a side effect of ordering: the
lookup returned the first invokable peer the round listed, and on a CI row
that is the role under test only because the row's apps happen to read
<role>,web-app-docs.

It now reads APP_ID, which the deploy workflow already sets to the row's app
and which the swarm routine already documents as the app under test. A round
that carries no APP_ID, or one whose app publishes no Production block, draws
an invokable peer at random instead, so a local round covers a different
guide each time rather than always the first one the inventory lists.

Ansible runs inside the infinito container rather than on the runner, and
that exec passes an explicit allowlist, so the name arrives only where it is
forwarded. Two seams carry it: _FORWARDED_ENV_KEYS, which covers CI and the
local initialize and reinstall verbs, and the update verb's own docker exec,
which assembles its arguments without that CLI.

The pin is tolerant rather than strict. APP_ID is set on every deploy row and
not only on documented ones, so refusing an app that publishes no guide would
let a generic variable abort a deploy.

Measured: the draw is observed over 200 rounds rather than assumed, and the
forwarding is negative-controlled, removing the key and watching the new test
fail on that key alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-28 09:16:29 +02:00
kevinveenbirkenbachandClaude Opus 5 666312dfbd refactor(i18n): translate against the dedicated LibreTranslate alone
server() preferred the deployed service and fell back to a throwaway
container when it did not answer. That container ran the upstream image
directly, with LT_UPDATE_MODELS on and neither the package-cache CA nor
the argos-net.com hijack, so it refetched gigabytes from upstream on
every run while the deployed service already carries those models in its
image, installed through the cache at build time. The fallback is gone.

It was reached by accident rather than by failure. deploy() ended in
await_service(), which polled for 300 seconds; the first start of the
service installs its models and takes far longer, so the wait expired,
deploy() reported nothing running, and the caller spawned the throwaway
beside the service it had just deployed successfully. await_service and
SERVICE_TIMEOUT_SECONDS are removed, leaving the single readiness wait
the client already performs with the budget those models need.

Step 0 of the deploy also stopped forcing a recreate. It rebuilt a
healthy lane on every invocation and took whatever model install was in
flight with it; compose recreates on a changed configuration by itself.

The wait no longer looks like a hang: it reports how many of the wanted
languages the server offers, which ones are still missing, how long it
has waited against its budget, and what the lane has pulled. i18n-tune
waits the same way and gets the same report.

service_url() and lane_container() replace the two places that each
rebuilt one of those strings, and deploy() reads the environment once.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 10:56:42 +02:00
kevinveenbirkenbachandClaude Opus 5 44473f7de1 refactor(i18n): move the dev files into their conventional homes
The top-level i18n/ directory held a compose override and a development
inventory and nothing else, so neither sat with its siblings. The override
joins compose/, whose entries are already named <name>.override.yml, and
the inventory joins inventories/development/, which is what it is. The
directory is gone.

Both moves are pure renames; only the four sites naming the old paths
change, and the -f argument in compose_file_args is the one that decides
which override every docker compose call in the i18n lane layers on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 10:37:18 +02:00
kevinveenbirkenbachandClaude Opus 5 50091194d5 fix(deploy): keep a non-invokable prerequisite out of the round's app list
Every desktop job of run 36246124624 aborted before ansible started:

    [ERROR] Some application_ids are invalid for this inventory:
      - user-workstation: not present in inventory

The include list the planner returns becomes two things at once: the
inventory provisioner's --include, and the deploy ids validate_application_ids
checks against the resulting inventory. The inventory is built from
get_all_invokable_apps, and utils/roles/stage.py loops a stage only over
invokable category paths, so a non-invokable id in that list is one no group
can hold and no stage can dispatch. `user-workstation` landed there because
the desktop-base service edge pulls it into the closure.

Measured over all 179 invokable roles, old resolution against new:

    rounds losing a member: 33
    distinct dropped roles: ['dev-java', 'user-workstation']

`user-workstation` in all 33 dsk rounds, `dev-java` in dsk-bluray-player,
which carries it as a meta dependency and would have aborted the same way
on main the day a round picked that role.

Neither stops running. A meta dependency resolves at role-include time,
independent of the inventory. `user-workstation` reaches its host through
the service loader, whose preload closure is computed from the filtered
whitelist and still contains it, and through dsk-base's own include_role:

    dsk-firefox: include=['dsk-base', 'dsk-firefox']
                 user-workstation in preload closure=True
                 workstation service required=True

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-26 23:06:58 +02:00
kevinveenbirkenbachandClaude Opus 5 c39f3ec9ee feat(web-app-docs): replay the published instructions in its CLI test
The replay of a role's "### Production" block moves out of the CI guide
scripts and into the documentation role itself. The block is read off the
rendered page over HTTP, so what runs is what a reader copies, and the role
that publishes the page is also the role that proves it.

scripts/github/guide/{one,compose_deploy,host_deploy}.sh and
cli/build/docs/readme/production.py are gone with the workflow step that
called them. The instructions marker now only adds web-app-docs to the row's
apps; test-e2e-cli discovers the role through its new templates/test.env.j2
and runs files/test/test.sh.

The block runs in a throwaway machine, never on the stack host. compose.yml
starts one service built from files/test/Dockerfile, whose base is
INFINITO_PARENT_IMAGE out of the mounted checkout's .env, so the machine is
the stack host's own distribution and the replay exercises the installation
instead of a prebuilt image. files/test/init.sh adds an init when the base
carries none and keeps the full -Syu that ef0f03ea98 established for arch.
scripts/tests/workspace/base/01_install.sh then prepares the machine before
the block runs.

Nothing is skipped any more. The previous host-install branch exited 0 and
passed silently over 65 of the 178 invokable roles; a block without a
"docker run" is now replayed against systemd in that same machine, with the
git clone dropped in favour of the mounted tree under test.

The target role comes from the new guide_role lookup, which reads
list_invokable_app_ids() rather than matching role-name prefixes, and falls
back to the asking role when the round deploys nothing else. deployment.py
grows running_apps() so both lookups share one definition of what the round
deploys. required_by_coverage.role_is_invokable drops its own walk over
categories.yml and delegates to the same SPOT; measured equal across all 280
roles.

Carried in the same index, because the refactor pass ran over this change
set: library.py splits into sites.py and build_queue.py mixins, builder.py
hands translated_languages to catalogs.py and folds three repeated log-tail
and state writes, and placeholders.py becomes a facade over spans.py,
repairs.py and damage.py. No test changed for those splits, and the two
ref-resolution dicts builder.py carried are replaced by _wanted_ref, which
also stops hashing the deployed tree on every tag build.

Verified: 7132 unit and lint tests pass over the whole tree, the guide_role
lookup renders through a real Templar, the base image resolves from .env,
and compose config validates the service. The replay itself has never been
executed; that needs a deploy of web-app-docs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-26 04:39:48 +02:00
kevinveenbirkenbachandClaude Opus 5 aceb5295ef feat(docs): let the documentation build generate what make docs generated
Cosmos, Quick Setup and Credits follow from meta/services.yml, categories.yml
and galaxy_info.author, yet 279 role READMEs carried a copy that went stale
whenever the metadata moved and nobody re-ran make docs. The documentation
build now runs the generators itself, against its own scratch checkout, and
the committed READMEs keep only the prose a person writes.

utils.cache.PROJECT_ROOT derives from the module's location, so putting the
checkout first on PYTHONPATH makes the generator operate on the scratch tree.
i18n.py and builder.py add it; the one remaining coupling, a relative_to over
PROJECT_ROOT in a log line, is gone.

CI extracted the "### Production" block from the committed README with awk.
Stripping the section made guide_deployable() return "" for every role, which
selects no deploy row and turns nothing red: a fail-open regression. It now
asks _is_role_invokable, which matches the old condition exactly (178 roles
either way, both difference sets empty), and cli/build/docs/readme/production.py
renders the block for the two guide scripts. All 178 renders are byte-identical
to what HEAD served, and both scripts' extraction pipelines produce identical
output for a web app, a service and a desktop role.

test_readme.py inverts: the three sections are now a hard failure in a
committed README. readme_md.md documents what each is derived from.

Removed: the docs, cosmos, readme-generate, readme-index and readme-check
targets, the five aliases pointing at them, and docs from make quality.

Verified by running generate_commands() in series against a copy of the
checkout: every command exits 0, no README is left without Cosmos or Credits,
and exactly the 178 invokable roles get a Quick Setup. The ansible_roles step
needs pandoc, which only the docs image carries, so that one link of the chain
is unexercised here; it is unchanged by this commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-26 01:09:22 +02:00
kevinveenbirkenbach f2cee9bb34 Merge branch 'feature/svc-ai-litellm/remote-models-config' into feature/store-i18n
# Conflicts:
#	cli/build/docs/readme/generate.py
#	docs/requirements/035-mcp-proxy-expansion.md
#	roles/svc-ai-lmstudio/meta/services.yml
2026-09-25 16:45:16 +02:00
kevinveenbirkenbachandClaude Opus 5 ac54e2c487 test(compose): pin the GPU count the file list now depends on
Both modules pin every environment variable compose_file_args() reads. Adding Profile.gpu_reservable() in 4d09286d8b added INFINITO_GPU_COUNT as a reader without pinning it, so the expectation silently followed the ambient environment: green on a shell that has no such variable, red in the test container where BASH_ENV injects .env and this host reads 'all'. That was eight of the unit failures in the gate.

Pin it to 0 in the three affected fixtures and add the two cases the gate never had: no GPU means no override, a non-zero count adds it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-25 14:53:50 +02:00
kevinveenbirkenbachandClaude Opus 5 53c120aaac refactor(dsk): rename the desktop tier and give its agents a category
Every desk- role becomes dsk-, and the three coding agents move to dsk-gnt- under a category of their own. The category carries a title, hanzi and icon but no invokable flag: an invokable child under an invokable parent is rejected outright, which is also why web itself is not invokable.

Three things a search and replace would have broken without a sound. get_entity_name now strips dsk-gnt-, so the primary entity keys agnt-claude, agnt-cursor and agnt-pi no longer resolved and all three roles dropped out of the CI plan. Eighteen run_once guards carried the old name against a convention its own test enforces. DESKTOP_ROLE_PREFIX in the addon spec lint is a literal, so the desktop exemption it grants would have matched nothing and the lint would have moved to the wrong rule in silence.

docs/requirements/035 is not a historical document but a register a lint holds to every role with an application_id, so its list carries the new ids. In 026 only the links move; the prose stays as a statement about the tree at the time it was written.

Measured against the plan with the rename normalised, the deployed role#variant set is unchanged. A stale id in an inventory is caught before any deploy starts: the pre-flight validator reports Invalid group 'desk-firefox' (not in application_ids).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-24 22:15:41 +02:00
kevinveenbirkenbachandClaude Opus 5 d6c4646b1e fix(deploy-verify): let required_by waive a role the runtime gates off
tasks/stages/03_destructor.yml includes test-e2e-cli only `when: RUNTIME in [dev, act, github]`, so a production host skips it by design. The coverage verifier demanded it anyway and failed every partial deploy there with "required role(s) did not execute". It stayed hidden because a full deploy passes no --id, which leaves allowed_applications empty and the guard unarmed.

required_by takes a `runtimes` list beside the existing compose/swarm blocks, read before the mode narrowing replaces rb. An unknown or empty runtime waives nothing, so the guard keeps its teeth wherever the runtime cannot be established; only a known runtime the list omits drops the requirement. This is 48753de136 one axis further: a role that a deploy correctly skips must not be demanded of it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-24 18:21:49 +02:00
kevinveenbirkenbachandClaude Opus 5 25dff2a780 fix(i18n): stop the translator from losing entries the server only refused
Four defects in the client surfaced in one run. LibreTranslate serves Chinese as zh-Hans and zh-Hant, never as bare zh, so the readiness wait blocked on a target that was never coming and the run stood still for an hour without printing a line. A blanket except turned every transient refusal into a permanently discarded entry, and the fallback path answered a failed batch by firing twenty single-text requests at the same overloaded server; the same catalogs that lost 130 to 295 entries came back with 0 discarded once the batch is retried with backoff first.

The per-catalog count conflated a refusal with a damaged protected span, which is measurably wrong: of 228 entries left pending in core/sw only 36 carried a span at all. Outcome now carries values, refusals, damage and the last refusal message per call rather than on the client, where twenty lanes mixed their numbers into each other. That split is what surfaced the 502 the server had been answering with. The throwaway container also ran with check=True and capture_output=True, so docker error text died inside the swallowed exception and the log showed an argv list and rc 125.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-24 09:54:10 +02:00
kevinveenbirkenbachandClaude Opus 5 a0a24e2c0e feat(i18n): deploy libretranslate into the i18n runner before translating
server() now resolves a translation endpoint in three steps: use the running
service, rebuild when the host has a GPU but no accelerated deploy left its
stamp, otherwise deploy. A plain run therefore costs nothing, and the rebuild
fires once, on the move from a CPU-built container to an accelerated one.

deploy() brings the runner up, waits for State.Running and deploys the role
inside it. The exec steps address `docker compose exec i18n` directly:
cli.administration.deploy.development exec hardcodes the `infinito` service
(compose.py:261), so every deploy through it silently ran in the main
container, where mailu already holds port 25 and postfix cannot start.
check=True stops a failing step instead of carrying on against the wrong
target.

The deploy router writes its pid to build/deploy.pid, which survives the exec
into the subscripts, and deploying() refuses a translation while that process
lives. The running deploy is left untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-24 01:53:19 +02:00
kevinveenbirkenbachandClaude Opus 5 3475b8ae6f feat(i18n): deploy libretranslate in an isolated compose runner
The i18n toolchain gets its own runner behind the `i18n` compose profile,
with its own inner docker daemon and docker state, so an i18n deploy shares
no ansible run and no container state with `infinito`. It reuses the image
that service builds (pull_policy: never), which starts it in under a second.

cache-consumer.yml holds the package-cache extra_hosts once and both runners
take them through `extends`; a YAML anchor cannot cross a compose file, an
extends can.

GPU reservation moves from a layered override file to INFINITO_GPU_COUNT on
both services. `gpus:` only accepts the literal `all`, so a variable there
invalidates the project on a CPU host, while a device reservation takes
`count: 0`. compose/gpu.override.yml and gpu_runtime_available() go with it.

test_compose_override_services_defined.py fails when an override extends a
service no compose file defines, which renders the whole project invalid.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-24 01:37:37 +02:00
kevinveenbirkenbachandClaude Opus 5 8924a06878 feat(ci-trigger): state the selection and its axes, and refuse one that cannot deploy
The trigger could only derive a selection: --failed from a source run's verdicts, --apps as a whitelist, and every axis carried over from that run. A verification run has no source run whose axes it wants to inherit; it wants the combination that answers an open question. --priority states the tokens, and five overrides state the axes.

--priority composes with --apps but collides with --failed, which builds its own line, and it skips the resume-offset computation: that only means something when a source run says where the regular line stopped.

Every dispatch is now validated first. call-orchestrator.yml already runs the same check as its first job and every other job needs it, so a bad token never reaches a chunk. What it reaches is the branch's concurrency group, which entry-manual-steer.yml declares with cancel-in-progress: a doomed dispatch cancels the run currently in flight and then fails its own validation. That cancellation is what this refuses, not the deploy minutes.

The whitelist is passed as empty when it is the all-sentinel, which is not a role name and would be reported as an unknown one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-23 11:56:24 +02:00
kevinveenbirkenbachandClaude Opus 5 bd7e0ae7fd feat(ci): measure what each deploy axis costs in runner time
The cheapest combination that still answers a question was picked by assertion, because nothing recorded the cost. Deploy job titles already carry the whole row and the API gives every job a start and an end, so the cost per axis value is read back off the runs that already happened.

Measured over six runs of this branch: compose 47m against swarm 83m, clearnet 52m against tor 79m. Choosing the axis is therefore worth minutes per row, not seconds.

Medians, not means, so one queue wait cannot outvote a hundred normal jobs, and a value below the sample minimum is dropped rather than ranked. fastest() answers None there, which means the caller leaves that axis to the rotation instead of guessing it.

The job labels are parsed with the same primitives the retrigger uses, so a title format change breaks both together rather than leaving this one silently measuring nothing. A filesystem row states the kind the matrix assigned, not the kind the deploy ran on, because a fallback is not recorded in the title.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-23 11:50:29 +02:00
kevinveenbirkenbachandClaude Opus 5 4c98624e07 feat(i18n): gettext catalogs with LibreTranslate machine translation
Core and the documentation get gettext catalogs for all 184 ISO 639-1
languages listed in meta/languages.yml, 47 of which LibreTranslate can fill.

`make i18n-extract` merges the current sources into locale/<code>/
LC_MESSAGES/{core,docs}.po: role descriptions, category and menu titles and
the logout strings for the core domain, the Sphinx gettext output for the
docs domain. A changed source keeps its previous translation as a fuzzy
entry, and catalogs are written deterministically so an unchanged merge
leaves the files untouched.

`make i18n-translate domain=<core|docs> [languages=<codes>]` fills only the
empty and fuzzy entries with a short-lived LibreTranslate container (the
image web-svc-libretranslate pins) and removes it afterwards. It starts the
container, and loads models, only for languages that still have pending
entries. Code spans, placeholders, URLs, tags and dotted names such as
Infinito.Nexus are masked for the request and a translation that damages
one is discarded. An unreachable server stops the run instead of
discarding every remaining entry, so a rerun continues where it stopped;
machine translations carry the translator comment `libretranslate`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 18:22:03 +02:00
kevinveenbirkenbachandClaude Opus 5 c7fb6e60c5 fix(ci): narrow a row's architectures by what its variant deploys
row_architectures intersected the run's pool with the declarations of
every role in the complexity closure. That closure follows every
conditional provider flag through the base graph, so tor -> web-svc-mirror
-> prometheus -> openwebui -> litellm -> lmstudio put the amd64-only
lmstudio into the closure of 93 of 128 roles, including the minimal
variants that never deploy it. A single amd64 pin would have withdrawn
arm64 from nearly the whole matrix.

The matrix now replaces a row's services with the include set that
plan_dev_inventory_matrix, the planner the deploy step itself uses,
resolves for that variant. Against CI run 35674334334 it matches every
data point: lmstudio for zammad#2, moodle#2 and xwiki#2, which pulled it
on arm64, and not for docs#0 or legal#0, which did not. With the pins of
the next commit, 126 of 128 roles keep arm64 rows instead of 34.

Without any architecture declaration the emitted matrix is byte-identical
to before under a fixed discovery seed. The planner adds about 37 s per
matrix build, and a role whose plan errors now stops the matrix instead of
only its own deploy job.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 16:01:09 +02:00
Alejandro Roman 9df3287adf Merge remote-tracking branch 'upstream/main' into feature/role-stalwart-mail-swarm 2026-09-22 12:16:44 +04:00
kevinveenbirkenbach eea14a0cfb Merge branch 'main' into feature/store-i18n 2026-09-21 23:35:55 +02:00
kevinveenbirkenbachandClaude Opus 5 5b8465260f feat(ci): deploy only the variants a change reaches
An image bump in a provider redeployed every variant of every consumer,
including the variants that switch that provider off and therefore never
deploy the bumped image at all. The diff-derived whitelist now narrows each
affected role to the variants whose own round closure still reaches a seed,
emitted as `role#0,2` selection tokens. That closure is walked on the same
variant-merged services maps the matrix planner builds its rounds from, so the
narrowing follows a parent's variant choice across every generation of the
chain. It covers every auto-derived whitelist: the weekly update PRs,
Dependabot, and any other diff-driven run.

The narrowing never shrinks the matrix on partial information. A seed role
keeps every variant, because its own files changed. A role reached only
through a run_after edge keeps every variant too, because the round closure
does not follow that edge and a role whose hits are empty is therefore
unnarrowed rather than dropped.

A pinned token naming a role the discovery query does not return now warns
instead of aborting the run. The diff-derived whitelist legitimately names
roles the lifecycle envelope filters out, and once it pins variants such a
token would abort every chunk; role names are still checked against roles/
before the run starts, so a typo is caught one layer earlier.

The round's variant merge moves out of the deploy inventory package into
utils/roles/applications/variants.py. The CI selection has to build the same
maps, and importing the deploy package for them pulls in a module that reads
INFINITO_SRC_DIR at import time, which the runner host does not set.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 22:51:03 +02:00
Alejandro Roman 4b9a834546 Merge remote-tracking branch 'upstream/main' into feature/role-stalwart-mail-swarm 2026-09-21 03:25:20 +04:00
kevinveenbirkenbach f0598470a7 Merge branch 'feature/ai-agent-employees' into feature/store-i18n 2026-09-19 07:03:43 +02:00
kevinveenbirkenbachandClaude Opus 5 2a7455bd90 feat(ci): spread the deploy matrix over amd64 and arm64
Both architectures were claimed to work and only one was ever exercised. The
deploy matrix now carries an architecture the way it carries mode, tor, distro
and filesystem, so every run proves both.

A job cannot ask for an architecture the way it asks for a distro, because the
architecture is the machine rather than something the job installs on it. The
matrix therefore carries a runner label and `runs-on` reads it, and an
architecture without a label in RUNNERS is a build error rather than a row that
lands on the wrong hardware and still reports green. assert_architecture.sh
checks the runner it actually got, because a label that stops resolving to
arm64 would otherwise go unnoticed forever.

Assignment is rotation, not chance: two architectures against five distros and
fifteen distro/filesystem positions are coprime enough that every combination
comes round. A role narrows the pool through `architectures` in its
meta/services.yml; without one it takes whatever the rotation hands it. An
empty intersection aborts instead of silently removing the role from the run.

Images are built per architecture and pushed under an architecture-suffixed
tag, then merged into the plain tag by manifest_all.sh; pushing the plain tag
from both builds would let the last one win and send half the rows to the wrong
platform. cleanup.sh now protects the `ci-<sha>-<architecture>` children of a
protected tag, because the plain tag is an index over them and deleting a child
leaves it unpullable on that platform.

label.py is split out of axes.py, which the 500-line limit no longer fits.
Writing a job title and reading one back belong together: a retrigger that
recovers an axis wrongly replays a different deploy than the one that failed,
and a drifting artifact name makes the failure reporter download nothing. The
leading `.*` in LABEL_RE is greedy on purpose so it anchors on the last mode
glyph; a reusable-workflow caller path carries a glyph of its own, and
test_selection.py guards the case.

Verified: 313 unit tests and 48 subtests across utils/github/variant,
utils/roles and cli/meta/ci. Locally, a deploy pinned to arm64 reaches ok=1444
and its image builds log aarch64-linux-gnu paths.

Not verified here: actionlint over the seven changed workflows, and a full
`make test`. The matrix doubles the row count, so runtime and runner
availability change for every CI run from here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 07:02:44 +02:00
kevinveenbirkenbachandClaude Opus 5 673de05a60 fix(inventory): write only the service toggles a role's contract declares
`disable=` wrote both `enabled: false` and `shared: false` for every role that
declares the service, but a role qualifies with any one of `enabled`, `shared`
or `provides`. compare_application_keys rejects any host_vars key the role
contract does not carry, so a role declaring only `enabled` came back as a
validation error and the deploy never reached Ansible:

    Missing default for web-svc-cdn: services.dashboard.shared
    Missing default for web-svc-mirror: services.dashboard.shared
    [ERROR] Inventory validation failed. Aborting deployment.

_declared_toggles reads the role's meta/services.yml and writes only what is
there; a role carrying neither toggle is skipped instead of gaining an empty
block. The module already held this principle: find_roles_with_service excludes
pure metadata entries so `disable` "doesn't add bogus enabled: false overrides",
and this applies the same rule per flag rather than per role.

Delta measured over the disable list the compose iteration procedure prescribes
(matomo, dashboard, prometheus, email, css): 407 of 409 role/service pairs are
untouched because they declare both toggles, exactly 2 stop receiving a
fabricated key, and 0 fall out of the disable entirely. The 2 are the pairs the
validator rejected. Across all 266 roles with a meta/services.yml, 1224 service
blocks declare `enabled`, 1190 declare `shared`, 34 declare `enabled` alone and
none declares `provides` alone, so nothing loses its disable.

Proven by the deploy sequence rather than by a test: the first local deploy
aborted at inventory validation, the second passed into the Ansible run with the
same disable list. Two regression tests cover both shapes, and the negative
control holds: hard-coding ["enabled", "shared"] fails the new test.

Without this, `disable=` is unusable in every local deploy, which the iteration
procedure requires on every run.

Residual: the generator was exercised through compose-deploy only, not through
roundtrip or swarm-zombie, which reach the same function by other call paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 06:00:56 +02:00
kevinveenbirkenbachandClaude Opus 5 66be8f6a0a test(lint): refuse a Debian codename in an image tag
A codename carries no ordering, so 'bookworm' and 'trixie' are two opaque
strings and no updater can tell that one supersedes the other. A pin that
names its Debian release that way is frozen on that release for as long as
it stands, and the freeze is invisible because the tag keeps resolving.
Numeric tags are comparable and stay inside the freshness pipeline, so the
rule is worth holding for the whole repository rather than fixing case by
case.

The check reads two shapes, because a codename reaches the tree in two
spellings. A 'name:tag' reference is recognised by the absence of a space
after the colon, which is what separates 'debian:bookworm' from the YAML
'image: python' and covers Dockerfile FROM lines and heredocs without a
second parser. A 'version:' value is read as a key, because '3.11-bookworm'
carries no colon at all.

False positives are kept out by two independent gates: a word-boundary
prefilter, so 'residual' and 'inside' are not read as 'sid', and a split of
the tag on '-', '.' and '_' that requires a whole component to be a
codename. That is what lets 'cache buster', '.stretched-link',
'fa-bullseye' and a tags list containing 'stretch' through. Every codename
occurrence in the repository was checked against the matcher by hand, and
the check was negative-controlled in both shapes, including an untracked
file.

Four references had to move for the rule to hold. meta/distros.yml takes
'debian:13' rather than '13.6': at depth 1 the updater can raise it across
a major, while depth 2 pins it inside the 13.x line forever. The whole
'make install' path was run in a debian:13 container to back that: all
twenty-four Depends and Recommends resolve on trixie, the docker repository
bootstrap derives the right suite and installs docker-ce-cli, and the
package builds and installs.

The workspace DiD probe moves to 'debian:13-slim'. An earlier version of
this change carried a suppression claiming the probe had to follow the
suite the apt proxy mirrors. That was wrong twice over: it cited a function
that has nothing to do with the probe, and the coupling does not exist,
because the cache frontend rewrites by path prefix and the suite never
enters a routing decision. Replaying the probe verbatim against
debian:13-slim fetches all three trixie suites through the
bookworm-configured proxies, so the suppression is gone rather than
reworded.

The rule ships with an escape hatch and no uses of it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 17:03:53 +02:00
kevinveenbirkenbachandClaude Opus 5 b356954d2e style(inventory): wrap the keycloak users fixture the way autoformat does
make quality-high pass 75 rewrote it: a concatenation whose right operand is a
multi-line literal gets parenthesised, with the operator leading its own line.

The string itself is unchanged, so
test_narrowing_the_credentials_still_rotates_every_user keeps the declaration
that separates web-app-keycloak from web-app-a and still fails when the user
half of the rotation is narrowed along with the credentials.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 20:50:27 +02:00
kevinveenbirkenbachandClaude Opus 5 46110f4311 perf(swarm-test): rotate the round's credentials, not every block in the file
The matrix gate rotates credentials between the two deploy passes so PASS 2
has to carry fresh values. It rotated every application block in the host_vars
file, and on a matrix inventory that is not the round. Provision applies its
--include as a strict intersection (filters.py:57-62), so it generates
credentials for at most derive_includes, while mirror_overrides.py adds a block
per mirror artefact carrying only image and version. Rotating those costs one
credential subprocess each for values provision never wrote.

matrix.py now passes --include derive_includes(app_id, variants=round_variants),
the same call 02_provision_inventory.sh:11 feeds provision, from the same
round_variants the provision step got one call earlier. The rotation set is
therefore a superset of what provision generated: every database_password still
rotates, because utils/manager/inventory.py:236-242 writes it under an id from
that same filtered list. Measured for web-app-wordpress: 35 ids.

The scope stops at the credentials. reset_credentials takes credential_ids
beside application_ids because the two halves shared one list, and narrowing it
narrowed _drop_user_passwords too: that resolves through
required_user_policies(roles_dir, application_ids), so 60 of 66 declared users
would have stopped rotating, root, backup and every mcp-* bot among them. The
user half keeps the full document list and is byte-identical to before.

The test fixture declared the same users in both roles, which made the
regression test pass against the regression. web-app-keycloak now declares a
user web-app-a does not, so the assertion has something to bite on.

The CLI aborts when --include matches no block instead of rotating nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 20:32:02 +02:00
kevinveenbirkenbachandClaude Opus 5 da09ee575b fix(coredns): answer every record type for the test zone instead of forwarding it
The CI zone served A and AAAA for INFINITO_DOMAIN and let every other type fall through to the public resolvers, which return NXDOMAIN for the reserved .test TLD. Per RFC 2308 that denial covers the whole name, not the queried type, so any caching resolver in front of CoreDNS applies it to the A record the deployment depends on.

Without such a cache nobody noticed: Mastodon's EmailMxValidator asks MX, takes the NXDOMAIN, then asks A anyway and gets the address. On a Tor node svc-net-tor puts dnsmasq in front, the denial is cached, and tootctl accounts create ends on 'Failure/Error: email unreachable'. It cost Fediwall#0 and Fediwall#2 in run 34653542922 and Fediwall#2 in run 34653754175, all three compose plus tor; the same two jobs passed in run 34614940508 as compose without tor, same command, same administrator@infinito.test, rc 0.

A template with no answer returns NOERROR with an empty answer section and the SOA in authority, so the zone denies a type without denying the name. It sits after the A and AAAA templates, which would otherwise have their names answered NODATA, and before forward, so no type escapes.

Measured against 172.30.0.53 in the running stack: infinito.test MX went from NXDOMAIN with an authority record to NOERROR with zero answers and the aa flag, infinito.test A and auth.infinito.test A still return 172.30.0.10, and deb.debian.org still resolves through the forward.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-12 09:35:58 +02:00
kevinveenbirkenbachandClaude Opus 5 c32bf320ff feat(ci): add a fourth deploy chunk so a short priority chunk strands no budget
A sweep may deploy 210 rows (available) in chunks of 80. Three declared
blocks hold exactly that when there is no priority line, but priority
rows get a chunk of their own that is never topped up with regular rows.
Nine priority rows therefore occupied a whole block, left the regular
rows 2 x 80 = 160, and 42 rows of budget went unused.

Adding a block to the YAML alone would not have changed that:
matrix.chunks_of handed the split slots.chunk_count(), which is
min(ceil(available / chunk size), INFINITO_CI_MAX_CHUNKS) and ignores
the priority seam, so the plan still stopped at three blocks.

call-orchestrator.yml now declares test-deploy-chunk-3, gated on chunks
0 to 2 like every chunk gates on all before it, and report-main-failures
and done need it. INFINITO_CI_MAX_CHUNKS is 4, one above what the budget
fills, and slots.chunk_blocks() returns it. chunks_of hands the split
every declared block; available still caps the rows, and a chunk whose
slice comes back empty skips its deploy. chunk_count() stays the "chunks
filled" figure of the budget report.

Tests: the split spends 9 + 80 + 80 + 41 = 210 rows with a spare block;
chunks_of plans three chunks from chunk_blocks=3 while chunk_count=2
(the previous wiring yields [1, 10]); an integration guard requires the
declared chunk jobs to equal INFINITO_CI_MAX_CHUNKS and to exceed
ceil(available / chunk size) by one, and fails on the stale value 3.

A shell that inherited INFINITO_CI_MAX_CHUNKS=3 keeps failing that guard
until it is restarted; make dotenv-force rewrites .env.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 23:46:48 +02:00
kevinveenbirkenbachandClaude Opus 5 c7d162fa88 fix(coredns): derive the domain regex from INFINITO_DOMAIN when rendering
INFINITO_DOMAIN_RE carried an already escaped regex through the env
chain: EnvBuilder.setdefault prefers a value already in the process env,
quote_for_dotenv escapes every backslash, and load.sh unescapes it again.
One inherited value with a doubled backslash was enough to write
"infinito\\\\.test" into .env, the rendered Corefile then matched a
literal backslash, and getent hosts test.infinito.test failed with rc 2.

The renderer now substitutes DOMAIN_RE = re.escape(INFINITO_DOMAIN) at
render time, so no backslash travels through the env or .env at all. The
template placeholder is renamed because it is a derived render value, not
an env key, and INFINITO_DOMAIN_RE is removed from default.env and the
passthrough keys, leaving INFINITO_DOMAIN as the single source.

A renderer test renders the real template and requires a single-escaped
match with no doubled backslash and no leftover placeholder. Verified
live: the Corefile renders ^infinito\.test\.$, test.infinito.test
resolves to 172.30.0.10 and test.infinitoXtest does not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 20:53:19 +02:00
kevinveenbirkenbachandClaude Opus 5 8434a31611 fix(ci): answer a role-only selection by existence, not by variant None
Rows are indexed by `(name, variant)` and a variant is an int. A token that
names only a role carries no variant, so the lookup ran with `None` and could
never hit, and every such token was reported as matching no discovered row.

That is the form `collapse_to_roles` produces, so it is the form a retriggered
priority line arrives in. Run 34348821577 shows the result: all twenty-one
priority entries carried

    priority: '<role>' matches no discovered row
              (no row in this run's mode and lifecycle envelope)

while baserow, checkmk, decidim and the rest went on to deploy fine. The
warning was not wrong about one entry and right about twenty; it was blind. A
role that genuinely has no row in the envelope looked exactly like a role with a
full set, which is how priority work goes missing without anyone being able to
see which entry did.

A token that narrows nothing beyond the role name is now answered by existence:
it asks for the role in whatever axes the rotation picks, so any discovered row
satisfies it. `Pin.pinned` already draws that line - "whether the token narrows
anything at all beyond the role name" - and pinned tokens keep the old path,
`check_pins` included.

Three tests come with it. There was already one for a bare role that matches
nothing, and none for a bare role that matches something, which is why the
lookup could stay broken. One of the new ones uses variant 4 so a future
implementation cannot pass by quietly guessing variant 0. Checked against HEAD:
the same input returns the spurious warning there, so the tests fail without
this change.

This restores the signal, not the lost jobs. The other silent path, `priority`
being truncated to the job budget in cli/meta/ci/chunks.py, is untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-10 01:44:59 +02:00
kevinveenbirkenbachandClaude Opus 5 3d6b3900e0 refactor(cli): gather the documentation generators under cli/build/docs
Two of them lived inside docs/ next to the pages they render, which is why both had grown a __file__ upward walk to find the repository root, a sys.path insert to import from utils, and a nocheck for each. A generator parked among documents reads as documentation and drops out of the checks that apply to code.

Moving them surfaced six real violations that the old location had hidden: yaml.safe_load instead of the shared cache, two uncached read_text calls, five hard-coded role paths instead of ROLE_FILE_META_SERVICES and ROLE_FILE_META_MCP, two make targets without an alias, and a requirement reference in code. All six are fixed at the root, none with a suppression, and both generators lose their project-root exemptions because a package supplies PROJECT_ROOT.

cli/build/readme joins them so the three generators that write into docs/ sit together. A new lint keeps docs/ to .md, .html, .rst and .nocheck; it enumerates through git ls-files so untracked __pycache__ is not its business. Checked against HEAD, it names exactly the three .py files that moved.

026-unified-addon-syntax stated that the matrix generator stays under docs/requirements/. The lint makes that impossible, so the requirement now records where the generator actually lives.

Verified: every one of the twelve modules under the new tree imports, no old module path survives anywhere in the repository, 482 lint tests and 1246 cli unit tests pass, and both generators run without drift.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-08 23:17:55 +02:00
kevinveenbirkenbachandClaude Opus 5 e41be38cb7 perf(vault): encrypt in-process instead of spawning ansible-vault per value
VaultHandler.encrypt_string started one ansible-vault process for every single credential. The swarm matrix' rotation gate between PASS 1 and PASS 2 drives 113 application subprocesses over 855 credential keys, so the step cost 1078s in run 33923190220 while the green baseline never ran it at all. Measured per value: 226ms via the CLI, 3.3ms via VaultLib, so 855 keys drop from 193s to 2.8s.

The snippet layout is a contract: encrypt_leaves splits the returned lines and strips the body indent, and existing inventories were written by the CLI. Verified byte-compatible (same 'name: !vault |' header, same 10-space body indent, same $ANSIBLE_VAULT;1.1;AES256 marker), and ansible-vault decrypt reads back what the new path writes.

Reading the password file delegates to ansible's own FileVaultSecret rather than approximating it: ansible strips, then decrypts a vaulted password file, then rejects an empty secret. A hand-rolled .strip() covered only the first of those. _vault stays lazy because callers construct VaultHandler with a password file that need not exist yet.

The tests replace two subprocess.run mocks that asserted against invented output. They now pin the layout encrypt_leaves depends on, and pin the guarantee from 052cea02e2 (values starting with a dash), which landed without a test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-08 21:11:19 +02:00
kevinveenbirkenbachandClaude Opus 5 f1e86eef58 fix(ci-trigger): retire a green suite on any input
The retrigger turned a suite off only when the source run had been
dispatched with that suite explicitly on. b1ca39e9dd introduced that
guard together with the feature itself, and it contradicts the same
commit's message and docstring, both of which state that the job record
is the only evidence.

The guard made the feature a no-op in its most common case. An input
left on its default renders no title segment and resolves to empty, so
config.get('workspace') was never 'true' for a run that reached the
suite on the 'auto' default. call-orchestrator.yml reads 'auto' as
global scope off the whitelist alone, and a --failed retrigger puts its
selection on the priority line while leaving the whitelist empty, so
every retrigger in the chain redeployed a suite that had already gone
green. Measured on run 33785198875: six workspace jobs green, and the
dispatch still went out carrying workspace=auto.

Drop the guard. suite_passed already refuses to retire anything that
failed, was cancelled, timed out, was skipped or never ran at all, so
the job record decides on its own.

Three tests cover the paths 8600f1d9aa left open, all of which pinned
the explicit 'true': green on 'auto', green with no recorded input, and
red on 'auto' staying on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-04 04:06:04 +02:00
Alejandro Roman f9d3a1345f fix(stalwart): fix TOr and testing properly, local pass 2026-09-04 02:43:39 +04:00
kevinveenbirkenbach 8600f1d9aa test(ci-trigger): pin the workspace suite rule
The rule lived only in suite_passed's body, and it inverts easily on a
casual read: ask for the suite again when it failed. Under that reading
the retrigger aliases would drop the suite in their most common case,
because they force-cancel the source run before reading it, so a shard
that was still running comes back cancelled rather than red.

Only an all-green suite is retired. Failed, cancelled, skipped, timed
out, still running and never dispatched all keep it on the retrigger,
and each of those now has a test. The two docstrings enumerate them, and
so does the i8ciall description, which is where an operator reads it.
2026-09-03 17:37:33 +02:00
kevinveenbirkenbachandClaude Opus 5 a703e3fc80 fix(tests): one import form, two missed modules
65c72fc374 settled the convention: import unittest, and reach the submodule as import unittest.mock as mock, so a call site has one spelling of the name. test_onion.py and test_tor_socks_proxy.py still carried from unittest import mock; CodeQL reports them as py/import-and-import-from, the last two in the repository.

Same one-line change per file the sweep applied; mock.<attr> call sites untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-03 17:19:48 +02:00
kevinveenbirkenbach 1e78a02c34 fix(ci-trigger): resume where the last run left off
The regular line restarted at the head on every retrigger. resume_offset
compared the source run's deploy tokens against a freshly computed
ranking, but every axis rotates with the sweep number and a retrigger is
a new run with a new one: measured against the current 277-row ranking,
2 of 277 tokens matched. The walk stopped on its first row every time,
and the 66 rows past the job budget never deployed at all.

Membership now compares role#variant, the identity the rotation cannot
move, and a row counts as proven only when every job it had was green.
A priority row deploys its whole cross-product, so one role#variant can
carry several verdicts, and a green compose job must not carry a red
swarm one past the line. The offset also inherits the source run's own
offset as its floor rather than falling back to the head.

The priority line drops --include-unrun with it: only the hard failures
and the priority entries the run never deployed belong there. The rows
it never reached are the regular line's turn now that the offset walks.
2026-09-03 13:38:07 +02:00
kevinveenbirkenbachandClaude Fable 5.1 eab8f5fc90 fix(inventory): give pinned user passwords the mark the realm policy demands
The shell-safe default drew from letters and digits only, and every user without declared accounts is an identity account, so kcadm set-password rejected hermes and openclaw with invalidPasswordMinSpecialCharsMessage as soon as keycloak entered the GitLab#1 round without LDAP (4164f37981) and the extras file stopped replacing the users map (03e42bae6d; run 33689701895). The generator now draws from letters, digits and -_. and guarantees upper case, lower case, a digit and one mark, never in the leading position.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-03 02:15:53 +02:00
kevinveenbirkenbachandClaude Fable 5.1 1a4ee72249 feat(ci): let a retrigger override chunk_gate, with i8ciallon as the non-stop i8ciall
A retrigger copies every dispatch input from its source run, chunk_gate
included, so a sweep that stopped at its first red chunk came back stopping
there again, and nothing short of a manual dispatch could ask the chain to
deploy every chunk and report all of it at once.

`infinito administration deploy ci trigger --chunk-gate {true,false}` now
overrides the carried value after the source run's configuration is read; it
is the one deliberate departure from "reproduce the source run exactly", and
the module docstring says so. Omitted, the carried value or the workflow
default applies as before.

`i8ciallon` is `i8ciall` with `--chunk-gate false`: force-cancel the run, put
everything without a verdict back on the priority line, and keep deploying
past a failed chunk. The alias lint caps a name at two adjacent consonants,
which is why it is not `i8ciallnonstop`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-03 00:12:56 +02:00
kevinveenbirkenbachandClaude Opus 5 af605b049f refactor(domains): give local testing a .test domain, leave .example to docs
RFC 6761 reserves .test for testing and .example for documentation, and the
project's own domain was doing both jobs under the documentation name. It is
what default.env hands every local deploy and what the test fixtures assert
against, so it is a testing name.

Rename it wherever it stands for the running machine: the environment
defaults, the development inventory, the workspace and trust scripts, the test
suites, and the four documents that describe real behaviour - the DNS mapping
to loopback, the browser CA trust, WSL2 resolution. Those documents would have
started lying the moment default.env changed.

The sandbox allowlist moves with them, since egress is matched per pattern and
nothing resolves the old name any more, and the sandbox document that mirrors
that list follows it.

Two places keep .example on purpose: the link-style sample in the
documentation guide, which is documentation about documentation, and every
generic fixture name elsewhere in the suite - a.example, foo.example and their
kin. Those are a larger sweep of their own and are not touched here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 13:10:34 +02:00