Commit Graph
33 Commits
Author SHA1 Message Date
Alejandro Roman 6144c85978 fix(stalwart): fixed secrets 2026-09-30 16:00:58 +04:00
Alejandro Roman 4b9a834546 Merge remote-tracking branch 'upstream/main' into feature/role-stalwart-mail-swarm 2026-09-21 03:25:20 +04:00
kevinveenbirkenbachandClaude Opus 5 095ea5e882 refactor(ai): preload one small chat model everywhere instead of a CI-only list
The catalogue held three models and the development inventory carried a second, CI-only list that both AI roles pulled in through their variant 0. That made the model a variant concern and split one fact across a group_var, an inventory file and two meta files.

AI_LOCAL_MODELS now holds the single entry the CI list held, qwen2.5 0.5b instruct at 531 MB in Q8_0, which lms get is already proven to fetch in this stack. AI_LOCAL_MODELS_CI and both variant overrides are gone, so variant 0 states only container_backup and the memory limits.

The dropped entries have no consumer: llama3, mistral and nomic appear solely in test fixtures that stub the lookups, and nothing requests an embedding model, the only embed matches being the filters that exclude one when picking the chat model. LITELLM_CHAT_MODEL takes the first non-embed alias, so it resolves and LITELLM_USABLE stays true.

Note this changes production, not only CI: a deployment no longer pulls a 7B model or an embedding model. It also shrinks the pre-pull that burned 18796s in job 104622514450 from gigabytes to 531 MB.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:28:24 +02:00
kevinveenbirkenbachandClaude Opus 5 f8da0fd122 fix(ai): pin each LM Studio model to its exact source, not to a search term
Run 34893750601 lost four jobs at svc-ai-litellm's warm request, every one of
them on the same upstream body:

    No models loaded. Please load a model in the developer page or use the
    `lms load` command.

The catalogue handed `lms get --yes --gguf` a bare name, which that command
treats as a hub search term. Its own help spells out the consequence: "If there
are multiple staff picks matching the search term, the first one will be used."
For `qwen2.5-0.5b-instruct` the first pick was Qwen2.5-Coder-0.5B-Instruct, so
the key LM Studio indexed never matched the `openai/{{ _model.name }}` the
gateway sends upstream, and just-in-time loading had nothing to load.

A `source:` now carries the full HuggingFace URL, the one spelling lms get
resolves deterministically; a bare owner/model pair addresses an LM Studio Hub
artifact instead and fails even for a model already on disk.

Measured against the shipped image rather than reasoned: the URL yields modelKey
qwen2.5-0.5b-instruct, `lms ps` is empty right before the request, and a plain
/v1/chat/completions for that name answers 200 and leaves the model resident. No
load step belongs here, because the server reports "Just-in-time model loading
active" and does it itself.

The embedding entry is renamed in the same breath. LM Studio indexes that model
as text-embedding-nomic-embed-text-v1.5, so the gateway was addressing a key
that never existed there either, and the first consumer to embed would have hit
the same 404.

The three fields stay separate and each has exactly one reader: Ollama pulls by
alias, LM Studio downloads by source, the gateway addresses name.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-15 00:50:45 +02:00
kevinveenbirkenbachandClaude Opus 5 786899700f feat(litellm): serve OpenAI, Anthropic and OpenRouter from declared keys
The gateway could reach exactly one remote provider, through a bare
OPENROUTER_API_KEY group_var with an empty-string default and an `{% if %}`
around a single hardcoded model entry. OpenAI and Anthropic had no path at all.

All three are now declared in roles/svc-ai-litellm/meta/secrets.yml, which
mapping.py already calls the application config schema, so a key lands at
applications.svc-ai-litellm.secrets.credentials.<provider>_api_key like every
other credential. Each carries the `type` and `regex` its value must satisfy and
defaults to the central API.<provider>.api_key block, which keeps one
operator-facing place to fill a provider in while the role owns the shape.

type and regex had to become real to be worth declaring. 192 `validation:`
blocks sit in the role schemas today and not one line of Python enforces them,
so a `regex` would have joined a pile of decoration. validate_supplied_value
now runs on every path a value arrives from outside — the --set override, a
value already in the inventory, and the value standing next to a `default:`.
A generated value stays exempt because the algorithm that produced it is named
by the same schema and generate_value takes no schema argument. min_length is
deliberately left unenforced: the generators ignore it, so holding them to it
would fail where nothing is wrong.

The pattern is written for the paste, not for the vendor. OpenAI moved issuance
from sk- to sk-proj- without notice and Anthropic issues sk-ant-api03-; a schema
pinning a prefix would reject a valid key and abort inventory creation. `\S{20,}`
accepts every form those three issue today and one nobody has announced, and
rejects a trailing newline, an embedded space, a truncated copy and a
<placeholder>. A key pasted into the wrong provider's slot is caught at deploy
instead, where the route it publishes answers nothing.

The gateway iterates one LITELLM_REMOTE_BACKENDS list instead of three `{% if %}`
blocks; config.yaml.j2 and env.j2 read the same list and selectattr('key')
decides. LITELLM_CHAT_MODEL falls back to the first configured remote alias
rather than to a hardcoded openrouter/auto.

For CI the keys travel as INFINITO_API_{OPENAI,ANTHROPIC,OPENROUTER}_KEY through
default.env, the passthrough allowlist, the infinito service in compose.yml, the
workflow env block and the development inventory. The compose leg is not
optional: the env lint rejected the change without it, because a key the
inventory reads but compose does not forward resolves to empty inside the
container with no error. The workflow reads `secrets.X || vars.X` so a variable
works as asked while a secret wins, which matters because a variable is not
masked and the development inventory sets MASK_CREDENTIALS_IN_LOGS: false.

An unset key is not an error anywhere: the provider's routes stay unpublished
and the CLI test asserts their absence rather than skipping.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 19:57:21 +02:00
kevinveenbirkenbachandClaude Opus 5 dac61edbc6 feat(ai): let LM Studio serve the models Ollama serves, under one catalogue
svc-ai-lmstudio had no model-load step at all: tasks/00_core.yml set the
once-flag and the container network, nothing else. The gateway published
lmstudio/default anyway, so the route existed and nothing could answer it.
wp-compose-tor.log:48994 shows the route written into the config, :51770 shows
the warm step reaching qwen2.5:0.5b on Ollama instead, and the green ai-engine
spec ran against that. LM Studio was deployed, published and never prompted.

The pre-pull mirrors the one that works. tasks/01_manager_ops.yml resolves the
hosting node and container id, loops the declared models through
`lms get --yes --gguf`, overlaps them with async/poll and reaps before
playwright, exactly like svc-ai-ollama. That `lms` exists is measured, not
assumed: /app/daemon-run.sh:67 in the image starts ./.bundle/lms server start,
and `lms get --help` documents -y for scripting and --gguf for the format.

Synonymity needs both namespaces in one place. Ollama knows qwen2.5:0.5b, LM
Studio knows qwen2.5-0.5b-instruct, so AI_LOCAL_MODELS carries both per entry:
`alias` is what the gateway publishes and what Ollama pulls, `name` is what
`lms get` resolves. Both roles now reference that one list, and config.yaml.j2
publishes each backend under the alias, routing Ollama to ollama/<alias> and LM
Studio to openai/<name>. The lmstudio loop skips an alias Ollama already
publishes, so two backends never produce a duplicate model_name and Ollama
keeps precedence. lmstudio/default is gone; the model name is now the same
whichever backend answers it.

Ollama's preload_models changes shape from a list of strings to the {alias,
name} entries. Every consumer moves with it: the pull command, the template
loop, and both expressions in 16_ai.yml. An inventory that overrides
`applications.svc-ai-ollama.services.ollama.preload_models` with the old string
list will break at map(attribute='alias'); this is a deliberate shape change,
in the spirit of 7b15c61dfc.

The CI model list is not a property of an installation, so AI_LOCAL_MODELS_CI
lives in inventories/development/default.yml, whose own header scopes that file
to the cross-cutting environment knobs. test_variable_definitions only knew
group_vars, so it reported the constant as undefined; the shipped inventories
are a definition source too, because the deploy CLI bakes them into host_vars,
and the guard's own docstring already claimed "defined somewhere in the
repository".

A latent crash came out of this: `| list | first` raises on an empty sequence,
so a backend deployed without preload models blew up LITELLM_CHAT_MODEL. Ollama
with only embedding models could already hit it; the existing test dodged it by
rendering only _SERVED. `| default('')` now yields the empty string that
_SERVED already reports as false.

Tests: config.yaml.j2 is rendered rather than read, over ollama-only,
lmstudio-only, both and neither, asserting the published model_name set, that
both backends name the same alias, that a shared alias is published once and
routed to Ollama, and that the render is valid YAML. test_litellm_chat_model
gains the lmstudio branch, the no-preload case and the same-model assertion.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 19:21:47 +02:00
kevinveenbirkenbachandClaude Opus 5 af605b049f refactor(domains): give local testing a .test domain, leave .example to docs
RFC 6761 reserves .test for testing and .example for documentation, and the
project's own domain was doing both jobs under the documentation name. It is
what default.env hands every local deploy and what the test fixtures assert
against, so it is a testing name.

Rename it wherever it stands for the running machine: the environment
defaults, the development inventory, the workspace and trust scripts, the test
suites, and the four documents that describe real behaviour - the DNS mapping
to loopback, the browser CA trust, WSL2 resolution. Those documents would have
started lying the moment default.env changed.

The sandbox allowlist moves with them, since egress is matched per pattern and
nothing resolves the old name any more, and the sandbox document that mirrors
that list follows it.

Two places keep .example on purpose: the link-style sample in the
documentation guide, which is documentation about documentation, and every
generic fixture name elsewhere in the suite - a.example, foo.example and their
kin. Those are a larger sweep of their own and are not touched here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 13:10:34 +02:00
Alejandro Roman e809765e23 fix(stalwart): fix all comments, fix a tor issue fouind and fix all SPOT issues 2026-09-02 15:02:46 +04:00
Alejandro Roman 333079cc2a feat(UI): fix migration, errors happening in local deploy and swarm integraton + merged main 2026-08-30 21:39:52 +04:00
Alejandro Roman 026b28f9a7 fix(swar-stalwart): migration process 2026-08-30 14:43:00 +04:00
Alejandro Roman a67334dd30 feat(swarm): fix testing 2026-08-19 20:32:43 +04:00
Alejandro Roman ac101d2efa Merge kevinveenbirkenbach/feature/swarm-nfs into stalwart integration branch 2026-07-21 10:00:17 +02:00
kevinveenbirkenbachandClaude Fable 5 d88f272f9f Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 15:31:08 +02:00
kevinveenbirkenbachandClaude Fable 5 898b6a2c4e feat(oidc): resolve OIDC lazily so the issuer family-aligns per consumer
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 20:39:17 +02:00
kevinveenbirkenbachandClaude Fable 5 e2f8d4c304 refactor(container): drive exec timeout and CA paths from group_vars SPOTs
Two wrapper behaviours move onto single sources of truth:

Exec timeout: SYS_TIMEOUT_CONTAINER_EXEC (07_schedule.yml, default 7d so a
hard kill never beats a production migration; the dev/CI inventory
tightens it to 60min) reaches every task host via the play environment as
CONTAINER_EXEC_TIMEOUT. The wrapper prefixes only the exec subcommand with
coreutils timeout --kill-after=15. Ansible task timeouts are SIGALRM-based
and slept through the last wedged docker socket for 4.5 hours; the OS
level kill still fires there and returns 124, which until/failed_when and
the rescue diagnostics react to. Unset or 0 keeps todays behaviour.

CA container paths: the bind-mount targets for the injected root CA and
wrapper script lived as string literals in container.py, compose_ca.py
and the filter plugin. They now live once in the CA_TRUST group_vars SPOT
(inject_cert_container, inject_wrapper_container) and reach container.py
as env vars (play environment plus CA_TRUST_SYSTEMD_ENV) and compose_ca.py
as CLI arguments built by the compose_ca_inject_cmd lookup. A caller that
provides the host CA env but not the container paths is half-migrated and
fails loudly instead of silently dropping CA injection; a host without any
CA env keeps the soft fallback. All tests read the paths from the SPOT,
and a regression test asserts the runtime scripts carry no literal copy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 10:58:28 +02:00
kevinveenbirkenbachandClaude Opus 4.8 094c02a69d feat(diagnostics): single SPOT rescue script with CI-uploaded snapshots
Every Ansible rescue: block used to carry ad-hoc per-role failure dumps that
drifted apart and captured different subsets. Replace them all with one
diagnostics script (scripts/system/diagnostics/rescue.sh) that snapshots
containers (full logs plus inspect per container), swarm services (ps plus
logs per service), journal, host resources, runtime info and stats into a
per-run folder and always exits 1 so the deploy still fails. A new lint
enforces the shape: every rescue ends with the script, at most one
nocheck-justified extra task before it (postgres pg_hba dump, xwiki
extension-repo probe), and a nocheck rescue-recovery exemption for the one
block that must genuinely recover: keycloak object updates, where a
cross-node reschedule needs a swarm-wide node re-resolve plus a delegate_to
change that only a rescue can express; its recovery is kept and now also
captures diagnostics before the terminal failure.

The snapshot folder path is a group_vars SPOT: DIR_RESCUE_DIAGNOSTICS and
DIR_VAR_LIB live in group_vars/all/05_paths.yml as plain strings; new env
handlers derive INFINITO_RESCUE_DIAGNOSTICS_DIR and INFINITO_DIR_VAR_LIB
from it for the generated .env, replacing the default.env entries, and the
play-level environment forwards the value to every task host so rescue.sh
needs no second default. CI uploads the folder like the inventory and
playwright artifacts: compose copies it out of the deploy container in
dedicated.sh, swarm captures per node (manager and both workers) into
node-named subfolders.

Deleting the ad-hoc dumps orphaned helper files (postgres inspect_container,
terminator docker_diagnostics, peertube diagnostics, matomo bootstrap debug,
xwiki and openresty rescue includes) and 14 role vars; all removed. Generic
pieces of those dumps (runtime version and info, stats, dmesg oom evidence,
swapon) moved into the SPOT script instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 03:04:06 +02:00
kevinveenbirkenbachandClaude Fable 5 c2b16091de fix(inventory): make biber's dev password realm-policy compliant
Keycloak's realm passwordPolicy requires a digit and a special char;
the LDAP-disabled seeding path died on set-password with
invalidPasswordMinSpecialCharsMessage. All consumers read the value
through the users lookup, so tests and seeding stay in sync.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 02:37:27 +02:00
kevinveenbirkenbachandClaude Opus 4.8 005f649498 refactor(onion): the inventory provisioner is the SPOT for the node onion
Mint the node onion during provisioning and write its address straight into applications.svc-net-tor.services.tor.node, dropping the INFINITO_TOR_NODE_ONION env indirection: ensure_node_onion (was init_env) mints/reuses the .onion-identity key files + returns the address (no env write); identity_hs_dir/IDENTITY_DIRNAME move to utils.tor_onion so cli and utils share the path SPOT; provision.apply_tor_node_onion sets services.tor.node in host_vars when svc-net-tor is deployed (run by 'inventory provision' at deploy time, and the svc-net-tor role copies the same key files into the daemon); domains.py reads services.tor.node; nginx_vhosts reads the key file instead of os.environ; INFINITO_TOR_NODE_ONION removed from default.env/passthrough/compose + the inventory env-lookup. Bonus: the address now lives in the persistent key files, surviving .env regeneration by make test/autoformat.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 23:53:18 +02:00
Alejandro Roman 48c49b8c7c Merge remote-tracking branch 'upstream/main' into feature/role-stalwart-mail 2026-07-03 07:26:30 +02:00
Alejandro Roman 564c0ab4df fix palywirght issues part 1 2026-07-01 22:33:20 +02:00
kevinveenbirkenbach 9b2f870777 Merge branch 'main' into feature/swarm-nfs 2026-07-01 05:37:36 +02:00
kevinveenbirkenbachandClaude Opus 4.8 cf3531b9e3 refactor(svc-storage-nfs-server): drive nfs flavor from meta/variants.yml
The inventory applications override in default.yml is dropped during inventory provisioning (CI ran kernel despite RUNTIME=github + a ganesha override), so select the flavor in meta/variants.yml where it reliably applies: ganesha for RUNTIME in [dev, act], kernel for github CI and production. meta/services.yml keeps kernel as the base default.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 07:41:12 +02:00
kevinveenbirkenbachandClaude Opus 4.8 539a078475 fix(svc-storage-nfs-server): select ganesha in DinD CI, kernel default
The github swarm legs picked the kernel NFS flavor inside the privileged DinD nfs-server, where kernel nfsd is not network-namespaced and wedges exportfs. Move flavor selection out of the role meta default and into the dev inventory: meta/services.yml now defaults to kernel (production); the development inventory overrides it to ganesha for the containerized runtimes (dev/act/github) and keeps kernel for real hosts (RUNTIME=host).

Add meta/variants.yml (kernel + ganesha) and a TODO to test both flavors via files/test.sh, plus server-side kernel-nfsd diagnostics in 12_collect_diagnostics.sh.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 06:22:26 +02:00
Alejandro Roman 17d6ac904f Merge remote-tracking branch 'upstream/main' into feature/self-host-runners
# Conflicts:
#	.github/workflows/test-deploy-universal.yml
#	.github/workflows/test-deploy-workstation.yml
2026-06-27 23:45:38 +02:00
kevinveenbirkenbachandClaude Opus 4.8 36ebf34b64 Merge branch 'main' into feature/swarm-nfs
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 17:13:55 +02:00
kevinveenbirkenbachandClaude Opus 4.8 cf765ee00c refactor: single /var/lib SPOT + swarm Playwright artifact upload
SPOT: declare the var-lib base once in default.env as INFINITO_DIR_VAR_LIB; the env build derives INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR from it; reference it everywhere (utils/paths.py, scripts via ${...:?}, tests, the dev inventory's DIR_VAR_LIB injection) instead of hardcoding the literal. group_vars keeps the literal default; the env-lookup lives in the dev inventory per the no-ansible-env-lookups lint rule. pull_specific_host.py --folder is now required and the role passes {{ DIR_BACKUPS }}.

Swarm e2e: collect_playwright_reports.sh docker-cp's the reports off the manager plus a workflow upload step, so swarm Playwright traces reach CI artifacts (was compose-only).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 17:01:09 +02:00
Alejandro Roman eed804b1ff fix(svc-runner): forward GH_TOKEN, IMAGE_TAG and MIRROR_PREFIX so DinD deploy test is no longer skiped 2026-06-24 18:31:11 +02:00
kevinveenbirkenbachandClaude Opus 4.8 d2f5efa7fe feat(test-e2e-playwright): parallel workers via plugin + tunable timeouts; split role vars by precedence
Add plugins/lookup/playwright_workers.py computing a conservative parallel-worker count (CPU/RAM-derived, app-under-test concurrency-capped, reduced on CI) and wire test_e2e_playwright_workers + test_e2e_playwright_fully_parallel + test_e2e_playwright_test_timeout into playwright.config.js (read from env) and 02_run_one.yml (passed via -e). fullyParallel stays false so parallelism is file-level only and intra-file ordering assumptions stay safe. Raise run_timeout to 3600 for addon-heavy roles (nextcloud ~50 specs).

Split the role's variables by Ansible precedence and intent: derived/fixed constants live in vars/main.yml (UPPERCASE, high precedence, not operator-tuned), operator-tunable knobs live in defaults/main.yml (lowercase, overridable). VARIANT_INDEX stays UPPERCASE with a single definition. Update references across tasks/templates/inventory/docs accordingly. Includes a unit test for playwright_workers and an autoformat pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 06:03:06 +02:00
kevinveenbirkenbachandClaude Opus 4.7 79faaf21e7 refactor(ansible): move INFINITO_* env lookups out of group_vars/roles into dev inventory
\`lookup('env', 'INFINITO_*')\` calls in group_vars / roles couple
deployment-agnostic Ansible code to default.env, which only exists in
dev/CI. Production deploys never source .env, so the lookups silently
fall back to their group_vars defaults and the inventory's intent gets
lost on the way (see the IP4 host_vars-overlay incident fixed in
8980b00b1).

Migrated keys (group_vars/roles -> inventory):
- DOMAIN_PRIMARY            (00_general.yml)
- BUILDKIT_ENABLED          (00_general.yml)
- CA_CONTAINER_ENABLED      (02_tls.yml)
- defaults_networks.internet.{ip4,ip6,dns}  (08_networks.yml)
- TEST_E2E_PLAYWRIGHT_KEEP  (roles/test-e2e-playwright/defaults/main.yml)

group_vars and the role default now carry literal fallback values;
the dev/CI inventory at \`inventories/development/default.yml\` pulls
the live values from .env via straight \`lookup('env', 'INFINITO_*')\`
calls without any \`| default(...)\` filter, because load.sh guarantees
the keys are set whenever .env exists.

The convention is enforced by the new lint test
\`tests/lint/filesystem/env/test_no_ansible_env_lookups.py\`, which
scans group_vars/, tasks/, roles/, and playbook.yml for any
\`lookup('env', 'INFINITO_*')\` against a default.env key and fails
with a pointer to the dev inventory as the injection site.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 17:57:13 +02:00
kevinveenbirkenbachandClaude Opus 4.7 9763cd8a4a fix(web-app-friendica): patch upstream ldapauth autocreate + variant-aware playwright
Root cause: upstream friendica-addons/ldapauth's `ldap_createaccount`
returns the raw `User::create()` result (`['user' => UserRow, 'password' => '']`),
but the `authenticate` hook only sets `authenticated = 1` when
`$user['uid']` is non-empty — never true for the autocreate path. First-
login LDAP users (e.g. biber) were inserted into `friendica.user` but
the session was never started, so /network kept serving the login form.

Encoded as an idempotent in-container patch task in `tasks/02_ldap.yml`
+ `files/patch_ldapauth_autocreate.sh` (reports PATCHED on first run,
ALREADY on subsequent runs). Runs whenever LDAP is wired, gated by
FRIENDICA_LDAP_ENABLED.

Other changes in the same stack:
- meta/services.yml: drop keycloakpassword addon (oauth2-proxy in front
  owns SSO); ldap.enabled now follows svc-db-openldap group presence so
  the variant matrix can disable it without a static override.
- tasks/01_core.yml + tasks/06_user_bootstrap.yml: pre-create the
  administrator only in LDAP-disabled variants — when LDAP is wired,
  ldapauth materialises the row on first admin login.
- files/playwright.spec.js: variant-aware spec replacing the parked
  test.skip cases. Three tests now run: guest flow (always), admin
  native login (picks double-login vs direct based on
  OAUTH2_SERVICE_ENABLED), biber + admin side-by-side in isolated
  contexts. PERSONA_* still blocked — friendica has no header-trusted
  autocreate path.
- templates/playwright.env.j2: expose BIBER_* + service-gate flags.
- inventories/development/default.yml: biber email aligned with the
  oauth2-proxy `email_domains = infinito.example` filter so biber's
  Keycloak login isn't 403'd at the proxy.
- TODO.md: SAML migration outlined as the long-term replacement for
  the double-login (currently second hop silently bypasses Keycloak 2FA).

Drive-by: trim double-space before `# nocheck` in
roles/sys-aur/meta/services.yml.

Verified on VARIANT=0 (oauth2 + ldap + oidc all enabled): three
playwright tests green, failed=0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 18:39:52 +02:00
kevinveenbirkenbachandClaude Opus 4.7 b5b0fb5331 chore: working-tree drift sync (roles/, inventories/)
Picked up by the make-test setup phase regenerating generated artefacts
during recent test runs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 01:37:56 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a26bfc4e94 refactor(cli): reorg admin/contributing tree + per-folder help system
Module moves (with imports, paths, argv literals, scripts, docs, and
parents[N] formulas updated in lockstep):

- cli/deploy             -> cli/administration/deploy
- cli/validate/inventory -> cli/administration/inventory/validate
- cli/vault              -> cli/administration/secrets/vault
- cli/administration/inventory/encrypt -> cli/administration/secrets/seal
- cli/mirror             -> cli/contributing/mirror

The new cli/administration/secrets cluster collocates the whole-file
ansible-vault wrapper next to the inventory-aware per-field sealer.
The user-facing CLI subcommand paths follow the package layout, e.g.
"infinito administration deploy dedicated <inv>" and
"infinito administration inventory validate <inv>".

Help system redesign in cli/core/{discovery,help,app}.py:

- Discovery now distinguishes runnable commands (directories with
  __main__.py) from category folders (directories that only group
  further commands). Empty support packages are filtered out.
- Each category folder MAY carry a README.md; its H1 heading is used
  as the title and its first paragraph as the one-line description in
  the help output.
- "infinito" / "infinito --help" prints only the top-level entries
  (one level deep). "infinito <category>" navigates into a folder and
  shows its sub-folders and commands one level deep, with the README
  title as the page heading and the invocation as a dim subtitle.
- New "--tree [N]" flag walks the tree under the (optional) path
  prefix recursively. N caps the depth (1-based); omit N for an
  unbounded walk. The output uses Unicode tree connectors, an emoji
  per entry (folder for categories, gear for commands), a globally
  aligned description column, and multi-line wrapped descriptions
  with tree-continuity bars on continuation lines.
- Category and command listings use the full invocation
  ("infinito meta callorder") rather than the bare leaf name, so the
  output is copy-pasteable.

Convention doc + 17 CLI READMEs:

- New docs/contributing/artefact/files/cli/readme_md.md describes the
  H1+paragraph layout the help system reads, and how it differs from
  role/docs READMEs.
- README.md added for cli/{administration, build, contributing, meta}
  and their immediate sub-folders, plus cli/meta/roles/applications
  and its resolution sub-folder. Each follows the "title with trailing
  emoji + single descriptive paragraph" convention.

Mirror resolver/sync description strings (the help system reads them
from argparse --help output) added so they no longer surface as
"options:" in the global index. cli/mirror/cleanup already had a
description and is unchanged.

Fixes uncovered by the move:

- import unittest.mock added to four dedicated-deploy tests so
  unittest.mock.patch resolves under stdlib unittest discovery
  (Python 3.13 no longer surfaces unittest.mock through "import
  unittest" alone).
- parents[N] index in the new
  cli/administration/inventory/validate/__init__.py corrected from
  3 to 4 to match the deeper layout.
- test_paths.INVENTORY_VALIDATOR_PATH assertion and test_runner
  substring assertions updated to match the new validator path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 20:17:09 +02:00
kevinveenbirkenbachandClaude Opus 4.7 d97ed7a9e8 feat(deploy): per-role matrix-deploy variants with folder-per-round model
Introduces per-role `meta/variants.yml` for declaring multi-variant
deploys (e.g. WordPress Single-Site vs. Multisite) and reshapes the
development deploy CLI to materialise one inventory folder per variant
and iterate them with co-located sync+async passes.

Why:
- The previous single-inventory + Pass-1-then-Pass-2 split deployed each
  variant twice and ran the async pass against a host left in a
  different variant's state, masking real bugs.
- Inventory and runtime concerns were tangled: variant selection lived
  as a runtime extra-var and required cross-stage coordination.

How:
- `roles/<role>/meta/variants.yml` (file format docs at
  docs/contributing/artefact/files/role/variants.md): YAML list of
  deep-merge overrides on top of `config/main.yml`. Empty entry `{}` is
  variant 0 (= legacy payload). 6 roles get their dev overrides moved
  here from inventories/dev.yml; WordPress declares 2 variants.
- `utils/runtime_data.py` gains `_build_variants` / `get_variants` and
  drops the retired `_active_variants` runtime selector. Loader always
  returns variant 0 as the default; non-zero variants reach
  `get_merged_applications` as inventory-level `applications.<app>`
  overrides baked at init time.
- New `cli/deploy/development/inventory.py`: `DevInventorySpec`,
  `plan_dev_inventory_matrix` (pure planner shared between init and
  deploy), `build_dev_inventory_matrix` (materialises one folder per
  round at `<dir>-<R>`, or `<dir>` when single-variant). Single-folder
  paths preserve full backward compat.
- `cli/deploy/development/deploy.py`: matrix loop with per-round
  cleanup (only apps whose variant changed AND that are about to
  re-deploy), `--variant <idx>` (or `VARIANT=` env) to pin a single
  round, and `--full-cycle` (or `FULL_CYCLE=true`) which interleaves
  sync + async per round so the async re-deploy targets exactly the
  host state the matching sync deploy just produced.
- `cli/deploy/development/exec.py` gains repeatable `--env KEY=VAL` so
  bash callers can inject per-run context into in-container helpers
  without inlining heredocs.
- `scripts/meta/env/inventory.sh` exports `INVENTORY_VARS_FILE`
  (Python mirror in `cli.deploy.development.common.DEV_INVENTORY_VARS_FILE`;
  test_inventory_vars_file_spot.py locks them against drift).
- `make deploy-fresh-purged-apps` and `scripts/tests/deploy/ci/dedicated.sh`
  switch to a single init + single deploy `--full-cycle`. The
  `make deploy-reuse-*` bash scripts honour `VARIANT=` by suffixing
  `INVENTORY_DIR`/`INVENTORY_FILE`.
- The 5 `scripts/tests/deploy/local/deploy/*.sh` scripts now delegate
  their in-container heredocs to dedicated helpers in
  `scripts/tests/deploy/local/utils/`, removing the bash-in-bash
  quoting hell.

Tests:
- `tests/unit/utils/test_variants.py` (renamed from
  `test_application_variants.py`): loader edge cases, variant-zero
  default, retired-`_active_variants` guard.
- `tests/unit/cli/deploy/development/test_inventory.py` (new): planner,
  variant baking, builder, `filter_plan_to_variant`, `DevInventorySpec`.
- `tests/unit/cli/deploy/development/test_matrix_deploy.py` (new):
  per-round deploy filter, cleanup decisions, `--variant` pinning,
  `--full-cycle` per-round interleave.
- `tests/unit/cli/deploy/development/test_inventory_vars_file_spot.py`
  (new): drift guard for the bash <-> Python literal pair.
- `tests/unit/cli/deploy/development/test_exec.py` extended for `--env`.

Docs:
- `docs/contributing/artefact/files/role/variants.md` (new): file format.
- `docs/contributing/design/matrix-deploy.md` (new): runtime mechanism
  (folder layout, --variant, --full-cycle, cleanup logic).
- `docs/contributing/actions/deploy.md` (new): contributor dev-deploy
  flow (make targets, --variant pinning).
- `docs/administration/deploy.md` slimmed to admin-only (was mixed
  admin+dev), cross-links to the new contributor doc.
- `docs/agents/action/iteration/role.md` keeps only agent-iteration
  recipes; mechanism explanation moved out and linked.
- `docs/contributing/documentation.md` adds the "no self-SPOT
  declaration" rule; `applications.md` renamed its SoT header for
  consistency.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 13:47:48 +02:00