Commit Graph
87 Commits
Author SHA1 Message Date
evangelostsak 81b2af9abd refactor: drop rationale and narration from inventories/examples/wireguard-three-meshes/wireguard.yml 2026-10-16 19:10:00 +02:00
evangelostsak b08dcb7dd3 refactor: drop rationale and narration from inventories/development/wireguard.yml 2026-10-16 19:10:00 +02:00
evangelostsak a87c56c569 docs(examples): show where the mesh values go in host_vars 2026-10-14 15:03:41 +02:00
evangelostsak f3e4eae919 refactor(examples): remove the separate three-mesh declarations 2026-10-14 15:03:41 +02:00
evangelostsak 1cd743532b feat(examples): the three-mesh layout as inventory values 2026-10-14 15:03:41 +02:00
evangelostsak 103ee9f7ac refactor(inventories): remove the lab's separate mesh declarations
Replaced by inventories/development/wireguard.yml, ordinary inventory values
read through the roles' own fields instead of a dedicated flag.
2026-10-14 15:03:41 +02:00
evangelostsak 033c3c5242 feat(inventories): the swarm lab's WireGuard layout as inventory values
svc-net-wireguard declares swarm and data; svc-swarm-node joins swarm, the
NFS server and the five backup roles join data. A meshed round bakes the
block into every lab host's host_vars.
2026-10-14 15:03:41 +02:00
evangelostsak a992b5d1a2 docs(examples): explain the primary mesh and how to test the example
Covers declaration order, the local and CI commands, and that the lab is
stricter than this inventory: its NFS server is a spoke of beta and gamma.
2026-10-14 11:23:22 +02:00
evangelostsak b777b65e84 test(mesh): hold the planning guarantees at any mesh count
The guarantees were asserted against the declared swarm and data pair,
so they proved two meshes work rather than that the planner works. A
deployment declares its own, and the review asks for any number of them with
the nodes arranged however the client wants.

The same invariants now run over generated topologies: one to eight meshes,
three arrangements each, hosts dealt out so some belong to several meshes and
some to none. 2247 subtests, four seconds, and the planner needed no change to
pass them.
2026-10-13 19:36:43 +02:00
Alejandro Roman 6144c85978 fix(stalwart): fixed secrets 2026-09-30 16:00:58 +04:00
Alejandro Roman 4b9a834546 Merge remote-tracking branch 'upstream/main' into feature/role-stalwart-mail-swarm 2026-09-21 03:25:20 +04:00
kevinveenbirkenbachandClaude Opus 5 095ea5e882 refactor(ai): preload one small chat model everywhere instead of a CI-only list
The catalogue held three models and the development inventory carried a second, CI-only list that both AI roles pulled in through their variant 0. That made the model a variant concern and split one fact across a group_var, an inventory file and two meta files.

AI_LOCAL_MODELS now holds the single entry the CI list held, qwen2.5 0.5b instruct at 531 MB in Q8_0, which lms get is already proven to fetch in this stack. AI_LOCAL_MODELS_CI and both variant overrides are gone, so variant 0 states only container_backup and the memory limits.

The dropped entries have no consumer: llama3, mistral and nomic appear solely in test fixtures that stub the lookups, and nothing requests an embedding model, the only embed matches being the filters that exclude one when picking the chat model. LITELLM_CHAT_MODEL takes the first non-embed alias, so it resolves and LITELLM_USABLE stays true.

Note this changes production, not only CI: a deployment no longer pulls a 7B model or an embedding model. It also shrinks the pre-pull that burned 18796s in job 104622514450 from gigabytes to 531 MB.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:28:24 +02:00
kevinveenbirkenbachandClaude Opus 5 f8da0fd122 fix(ai): pin each LM Studio model to its exact source, not to a search term
Run 34893750601 lost four jobs at svc-ai-litellm's warm request, every one of
them on the same upstream body:

    No models loaded. Please load a model in the developer page or use the
    `lms load` command.

The catalogue handed `lms get --yes --gguf` a bare name, which that command
treats as a hub search term. Its own help spells out the consequence: "If there
are multiple staff picks matching the search term, the first one will be used."
For `qwen2.5-0.5b-instruct` the first pick was Qwen2.5-Coder-0.5B-Instruct, so
the key LM Studio indexed never matched the `openai/{{ _model.name }}` the
gateway sends upstream, and just-in-time loading had nothing to load.

A `source:` now carries the full HuggingFace URL, the one spelling lms get
resolves deterministically; a bare owner/model pair addresses an LM Studio Hub
artifact instead and fails even for a model already on disk.

Measured against the shipped image rather than reasoned: the URL yields modelKey
qwen2.5-0.5b-instruct, `lms ps` is empty right before the request, and a plain
/v1/chat/completions for that name answers 200 and leaves the model resident. No
load step belongs here, because the server reports "Just-in-time model loading
active" and does it itself.

The embedding entry is renamed in the same breath. LM Studio indexes that model
as text-embedding-nomic-embed-text-v1.5, so the gateway was addressing a key
that never existed there either, and the first consumer to embed would have hit
the same 404.

The three fields stay separate and each has exactly one reader: Ollama pulls by
alias, LM Studio downloads by source, the gateway addresses name.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-15 00:50:45 +02:00
kevinveenbirkenbachandClaude Opus 5 786899700f feat(litellm): serve OpenAI, Anthropic and OpenRouter from declared keys
The gateway could reach exactly one remote provider, through a bare
OPENROUTER_API_KEY group_var with an empty-string default and an `{% if %}`
around a single hardcoded model entry. OpenAI and Anthropic had no path at all.

All three are now declared in roles/svc-ai-litellm/meta/secrets.yml, which
mapping.py already calls the application config schema, so a key lands at
applications.svc-ai-litellm.secrets.credentials.<provider>_api_key like every
other credential. Each carries the `type` and `regex` its value must satisfy and
defaults to the central API.<provider>.api_key block, which keeps one
operator-facing place to fill a provider in while the role owns the shape.

type and regex had to become real to be worth declaring. 192 `validation:`
blocks sit in the role schemas today and not one line of Python enforces them,
so a `regex` would have joined a pile of decoration. validate_supplied_value
now runs on every path a value arrives from outside — the --set override, a
value already in the inventory, and the value standing next to a `default:`.
A generated value stays exempt because the algorithm that produced it is named
by the same schema and generate_value takes no schema argument. min_length is
deliberately left unenforced: the generators ignore it, so holding them to it
would fail where nothing is wrong.

The pattern is written for the paste, not for the vendor. OpenAI moved issuance
from sk- to sk-proj- without notice and Anthropic issues sk-ant-api03-; a schema
pinning a prefix would reject a valid key and abort inventory creation. `\S{20,}`
accepts every form those three issue today and one nobody has announced, and
rejects a trailing newline, an embedded space, a truncated copy and a
<placeholder>. A key pasted into the wrong provider's slot is caught at deploy
instead, where the route it publishes answers nothing.

The gateway iterates one LITELLM_REMOTE_BACKENDS list instead of three `{% if %}`
blocks; config.yaml.j2 and env.j2 read the same list and selectattr('key')
decides. LITELLM_CHAT_MODEL falls back to the first configured remote alias
rather than to a hardcoded openrouter/auto.

For CI the keys travel as INFINITO_API_{OPENAI,ANTHROPIC,OPENROUTER}_KEY through
default.env, the passthrough allowlist, the infinito service in compose.yml, the
workflow env block and the development inventory. The compose leg is not
optional: the env lint rejected the change without it, because a key the
inventory reads but compose does not forward resolves to empty inside the
container with no error. The workflow reads `secrets.X || vars.X` so a variable
works as asked while a secret wins, which matters because a variable is not
masked and the development inventory sets MASK_CREDENTIALS_IN_LOGS: false.

An unset key is not an error anywhere: the provider's routes stay unpublished
and the CLI test asserts their absence rather than skipping.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 19:57:21 +02:00
kevinveenbirkenbachandClaude Opus 5 dac61edbc6 feat(ai): let LM Studio serve the models Ollama serves, under one catalogue
svc-ai-lmstudio had no model-load step at all: tasks/00_core.yml set the
once-flag and the container network, nothing else. The gateway published
lmstudio/default anyway, so the route existed and nothing could answer it.
wp-compose-tor.log:48994 shows the route written into the config, :51770 shows
the warm step reaching qwen2.5:0.5b on Ollama instead, and the green ai-engine
spec ran against that. LM Studio was deployed, published and never prompted.

The pre-pull mirrors the one that works. tasks/01_manager_ops.yml resolves the
hosting node and container id, loops the declared models through
`lms get --yes --gguf`, overlaps them with async/poll and reaps before
playwright, exactly like svc-ai-ollama. That `lms` exists is measured, not
assumed: /app/daemon-run.sh:67 in the image starts ./.bundle/lms server start,
and `lms get --help` documents -y for scripting and --gguf for the format.

Synonymity needs both namespaces in one place. Ollama knows qwen2.5:0.5b, LM
Studio knows qwen2.5-0.5b-instruct, so AI_LOCAL_MODELS carries both per entry:
`alias` is what the gateway publishes and what Ollama pulls, `name` is what
`lms get` resolves. Both roles now reference that one list, and config.yaml.j2
publishes each backend under the alias, routing Ollama to ollama/<alias> and LM
Studio to openai/<name>. The lmstudio loop skips an alias Ollama already
publishes, so two backends never produce a duplicate model_name and Ollama
keeps precedence. lmstudio/default is gone; the model name is now the same
whichever backend answers it.

Ollama's preload_models changes shape from a list of strings to the {alias,
name} entries. Every consumer moves with it: the pull command, the template
loop, and both expressions in 16_ai.yml. An inventory that overrides
`applications.svc-ai-ollama.services.ollama.preload_models` with the old string
list will break at map(attribute='alias'); this is a deliberate shape change,
in the spirit of 7b15c61dfc.

The CI model list is not a property of an installation, so AI_LOCAL_MODELS_CI
lives in inventories/development/default.yml, whose own header scopes that file
to the cross-cutting environment knobs. test_variable_definitions only knew
group_vars, so it reported the constant as undefined; the shipped inventories
are a definition source too, because the deploy CLI bakes them into host_vars,
and the guard's own docstring already claimed "defined somewhere in the
repository".

A latent crash came out of this: `| list | first` raises on an empty sequence,
so a backend deployed without preload models blew up LITELLM_CHAT_MODEL. Ollama
with only embedding models could already hit it; the existing test dodged it by
rendering only _SERVED. `| default('')` now yields the empty string that
_SERVED already reports as false.

Tests: config.yaml.j2 is rendered rather than read, over ollama-only,
lmstudio-only, both and neither, asserting the published model_name set, that
both backends name the same alias, that a shared alias is published once and
routed to Ollama, and that the render is valid YAML. test_litellm_chat_model
gains the lmstudio branch, the no-preload case and the same-model assertion.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 19:21:47 +02:00
kevinveenbirkenbachandClaude Opus 5 af605b049f refactor(domains): give local testing a .test domain, leave .example to docs
RFC 6761 reserves .test for testing and .example for documentation, and the
project's own domain was doing both jobs under the documentation name. It is
what default.env hands every local deploy and what the test fixtures assert
against, so it is a testing name.

Rename it wherever it stands for the running machine: the environment
defaults, the development inventory, the workspace and trust scripts, the test
suites, and the four documents that describe real behaviour - the DNS mapping
to loopback, the browser CA trust, WSL2 resolution. Those documents would have
started lying the moment default.env changed.

The sandbox allowlist moves with them, since egress is matched per pattern and
nothing resolves the old name any more, and the sandbox document that mirrors
that list follows it.

Two places keep .example on purpose: the link-style sample in the
documentation guide, which is documentation about documentation, and every
generic fixture name elsewhere in the suite - a.example, foo.example and their
kin. Those are a larger sweep of their own and are not touched here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 13:10:34 +02:00
Alejandro Roman e809765e23 fix(stalwart): fix all comments, fix a tor issue fouind and fix all SPOT issues 2026-09-02 15:02:46 +04:00
Alejandro Roman 333079cc2a feat(UI): fix migration, errors happening in local deploy and swarm integraton + merged main 2026-08-30 21:39:52 +04:00
Alejandro Roman 026b28f9a7 fix(swar-stalwart): migration process 2026-08-30 14:43:00 +04:00
kevinveenbirkenbachandClaude Opus 5 52b4504035 fix(container-dns): let the bridge be declared instead of guessed
Whether a resolver answers on the docker bridge was inferred from the host's
own resolv.conf: all entries loopback was read as "a listener of ours is there,
and containers must reach it on the bridge". A systemd-resolved stub binds
127.0.0.53, NetworkManager's dnsmasq binds 127.0.0.1, and a node dnsmasq binds
both those and the bridge. All three look identical from resolv.conf and only
the third is reachable from a container.

So ordinary hosts were handed an address with nothing behind it. The
Instructions job died there: daemon.json carried "dns": ["10.208.0.1"] alone,
and the openresty build could not resolve dl-cdn.alpinelinux.org while the
daemon, resolving through the host, pulled the same base image fine. Two netns,
two outcomes.

e308e18923 had already fixed this once, with a reachability gate whose message
names the same URL. b09f2cbd82 dropped that gate and ce43263745 put the
resolv.conf inference in its place, so the guard was gone and only the missing
docker0 fact still hid it. 285ba1e76c removed that accident and the failure
surfaced.

svc-net-tor and NETWORK_CONTAINER_BRIDGE_RESOLVER now say it instead, the
latter false by default and true in the swarm lab, which provisions the dnsmasq
its nodes resolve through. Extra-vars are global, so the lab's NFS and backup
hosts carry it although no listener is installed there; the clearnet resolver
behind the bridge keeps them working.

The docker0 fact is gone from the address too. It is not merely late, it is
wrong while it is late: facts are read before the daemon config is applied, so
docker0 still holds docker's default and writing default-address-pools moves
the bridge out from under it. A swarm node shipped "dns": ["172.17.0.1", ...]
against a live 10.208.0.1, which is why its containers asked a dead address and
fell through to the public resolver, where infinito.example does not exist. The
pool answers before the daemon exists and answers the same afterwards, and it
is read per slice rather than per supernet, which differ once a pool does not
start on a slice boundary.

A list that would carry only the bridge now gets NETWORK_PUBLIC_DNS_RESOLVERS
behind it, so a declaration that turns out to be wrong costs a wasted first
query rather than all name resolution. An empty declaration raises.

Rendered from the repository's own variable files: the Instructions host yields
[] and the template omits the key, which is what de95a95c65 emitted while that
job was green; lab and tor nodes yield the live bridge ahead of the clearnet
resolver even with a stale docker0 fact present; an ordinary host yields only
its declared resolver.

The suite asserted the two behaviours this replaces, including one case added
with 285ba1e76c demanding the stale fact win. They are inverted, and a
cross-product over nameserver shapes, tor membership, clearnet and the docker0
fact now asserts that no shape ends up holding the bridge alone - the old suite
checked each shape by itself, which is how this passed through it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-30 00:32:13 +02:00
kevinveenbirkenbachandClaude Opus 5 ce43263745 fix(container-dns): follow the live bridge, not a configured address
The swarm inventory carried the manager's docker bridge gateway as a literal under networks.internet.dns, so one field meant two things at once and the value went stale the moment default-address-pools moved the bridge. The daemon kept handing out the new address while the inventory kept naming the old one, and the DinD diagnostic probed that same literal, so it failed in step with the defect instead of catching it.

The inventory now declares only the clearnet resolver. The lookup hands containers the bridge whenever the host resolves solely through a loopback listener, which is how the simulated cluster nodes are wired and which docker strips from a container's resolv.conf. The diagnostic reads the gateway from the running daemon instead of being told it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-29 00:06:42 +02:00
Alejandro Roman a67334dd30 feat(swarm): fix testing 2026-08-19 20:32:43 +04:00
Alejandro Roman d436f994e9 feat(swarm): Add Stalwart to swarm 2026-08-19 02:36:12 +04:00
Alejandro Roman 7332bd66ae Merge upstream/main into feature/role-stalwart-mail-swarm 2026-08-18 01:42:23 +04:00
kevinveenbirkenbachandClaude Opus 5 c7ddf6eb24 Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
Three conflicts, each resolved by combining both sides rather than picking one.

compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.

daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.

The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 00:47:10 +02:00
kevinveenbirkenbachandClaude Opus 5 4c328c4fd1 feat(swarm-inventory): run the drill in debug mode, keeping async on
The swarm drill loads only this file, so MODE_DEBUG fell back to the group_vars default and the ganesha LOG block from 8f5670765 would never have rendered where the wedge happens. Turning it on also enables MODE_ASSERT and SYS_TIMER_ALL_ENABLED, which is a wider behaviour change than the diff shows.

ASYNC_ENABLED is set explicitly because it otherwise derives from 'not MODE_DEBUG': debug would have serialised every round, on a drill whose time budget two jobs of run 30699990912 already exhausted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-01 18:32:22 +02:00
kevinveenbirkenbachandClaude Opus 5 12417c889b fix(swarm-inventory): read the lab's own manager IP as the deployment address
networks.internet.ip4 claimed 127.0.0.1 while the lab's dnsmasq maps the whole domain to INFINITO_SWARM_MGR_IP (compose/swarm/playbook.yml:30). Two sources disagreed about one fact, and the DiD DNS check was the first thing that ever compared them. Read the same env SPOT from default.env so both agree by construction.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-01 14:27:13 +02:00
kevinveenbirkenbach bb1b4f7f96 Merge branch 'feature/swarm-nfs' into feature/svc-net-tor 2026-07-27 23:55:03 +02:00
kevinveenbirkenbachandClaude Opus 5 93b47a5bee refactor(packages): install every package through the registry
Replaces the raw package modules at 76 call sites with package_install, moves
each role's distribution mapping into its own meta/packages.yml, and turns the
inventory PACKAGES variable from source-keyed literal names into a list of
declared package ids. The lint that forbids the raw modules is enabled here,
because the migration burns it down from 76 offenders to zero.

Removes sys-aur, sys-aur-install and sys-package. The AUR build user, the
passwordless pacman rule and the inventory-driven install are part of the plan
the action plugin executes, so the roles no longer carry them.

Declaring the mappings surfaced breakage the raw modules had hidden: on Debian
the `which` binary comes from debianutils, jrnl is in no Debian or Ubuntu
suite, chromium-browser is absent from Debian stable, JDK 11 is unobtainable on
RedHat, wireguard had no RedHat branch at all, qbittorrent was built from the
AUR although Arch ships it in extra, and desk-docker installed a Docker variant
that conflicts with the one sys-svc-container installs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 23:25:50 +02:00
Alejandro Roman ac101d2efa Merge kevinveenbirkenbach/feature/swarm-nfs into stalwart integration branch 2026-07-21 10:00:17 +02:00
kevinveenbirkenbachandClaude Fable 5 d88f272f9f Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 15:31:08 +02:00
kevinveenbirkenbachandClaude Fable 5 898b6a2c4e feat(oidc): resolve OIDC lazily so the issuer family-aligns per consumer
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 20:39:17 +02:00
kevinveenbirkenbachandClaude Fable 5 40e553069d refactor(structure): nest prefix-grouped sibling files into subfolders
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.

Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 20:00:03 +02:00
kevinveenbirkenbachandClaude Fable 5 e2f8d4c304 refactor(container): drive exec timeout and CA paths from group_vars SPOTs
Two wrapper behaviours move onto single sources of truth:

Exec timeout: SYS_TIMEOUT_CONTAINER_EXEC (07_schedule.yml, default 7d so a
hard kill never beats a production migration; the dev/CI inventory
tightens it to 60min) reaches every task host via the play environment as
CONTAINER_EXEC_TIMEOUT. The wrapper prefixes only the exec subcommand with
coreutils timeout --kill-after=15. Ansible task timeouts are SIGALRM-based
and slept through the last wedged docker socket for 4.5 hours; the OS
level kill still fires there and returns 124, which until/failed_when and
the rescue diagnostics react to. Unset or 0 keeps todays behaviour.

CA container paths: the bind-mount targets for the injected root CA and
wrapper script lived as string literals in container.py, compose_ca.py
and the filter plugin. They now live once in the CA_TRUST group_vars SPOT
(inject_cert_container, inject_wrapper_container) and reach container.py
as env vars (play environment plus CA_TRUST_SYSTEMD_ENV) and compose_ca.py
as CLI arguments built by the compose_ca_inject_cmd lookup. A caller that
provides the host CA env but not the container paths is half-migrated and
fails loudly instead of silently dropping CA injection; a host without any
CA env keeps the soft fallback. All tests read the paths from the SPOT,
and a regression test asserts the runtime scripts carry no literal copy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 10:58:28 +02:00
kevinveenbirkenbachandClaude Opus 4.8 094c02a69d feat(diagnostics): single SPOT rescue script with CI-uploaded snapshots
Every Ansible rescue: block used to carry ad-hoc per-role failure dumps that
drifted apart and captured different subsets. Replace them all with one
diagnostics script (scripts/system/diagnostics/rescue.sh) that snapshots
containers (full logs plus inspect per container), swarm services (ps plus
logs per service), journal, host resources, runtime info and stats into a
per-run folder and always exits 1 so the deploy still fails. A new lint
enforces the shape: every rescue ends with the script, at most one
nocheck-justified extra task before it (postgres pg_hba dump, xwiki
extension-repo probe), and a nocheck rescue-recovery exemption for the one
block that must genuinely recover: keycloak object updates, where a
cross-node reschedule needs a swarm-wide node re-resolve plus a delegate_to
change that only a rescue can express; its recovery is kept and now also
captures diagnostics before the terminal failure.

The snapshot folder path is a group_vars SPOT: DIR_RESCUE_DIAGNOSTICS and
DIR_VAR_LIB live in group_vars/all/05_paths.yml as plain strings; new env
handlers derive INFINITO_RESCUE_DIAGNOSTICS_DIR and INFINITO_DIR_VAR_LIB
from it for the generated .env, replacing the default.env entries, and the
play-level environment forwards the value to every task host so rescue.sh
needs no second default. CI uploads the folder like the inventory and
playwright artifacts: compose copies it out of the deploy container in
dedicated.sh, swarm captures per node (manager and both workers) into
node-named subfolders.

Deleting the ad-hoc dumps orphaned helper files (postgres inspect_container,
terminator docker_diagnostics, peertube diagnostics, matomo bootstrap debug,
xwiki and openresty rescue includes) and 14 role vars; all removed. Generic
pieces of those dumps (runtime version and info, stats, dmesg oom evidence,
swapon) moved into the SPOT script instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 03:04:06 +02:00
kevinveenbirkenbachandClaude Opus 4.8 8519642373 refactor(storage): SPOT the nfs export base and client version in their roles
storage.nfs shrinks to the deployment-specific server address: export_base moves into svc-storage-nfs-server's services.yml (the server dictates where exports root) and the mount protocol version into svc-storage-nfs-client's services.yml (the client owns how it mounts). Both are read through utils.storage.nfs loaders by the nfs lookups, the env handler and 00_topology, eliminating the unrendered-variables class for these values; the group_vars assert shrinks accordingly and the swarm test extras stop restating role defaults.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 23:54:32 +02:00
kevinveenbirkenbachandClaude Opus 4.8 a7fbb94972 fix(swarm-test): bound diagnostics against wedged dockerd and SPOT the nfs state path
A wedged DinD dockerd hung a single diagnostics docker exec until the 360-minute job timeout cancelled the job together with its artifact uploads (job 86013153099 sat 5h in service logs). Route every diagnostics exec through a timeout wrapper, tail-limit the log dumps, bound showmount, cap the collect step at 30 minutes, and dump the pending services' log tails once on the final converge failure. The NFS state path moves into a single source of truth: STATE_SUBDIR in utils/storage/nfs.py feeds the nfs_state_subdir lookup (deploy side) and the derived INFINITO_SWARM_NFS_STATE_PATH env var (harness side); the diagnostics script requires both path vars instead of defaulting, and the swarm inventory drops its redundant export_base restatement.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 17:30:36 +02:00
kevinveenbirkenbachandClaude Opus 4.8 2b44d9f7f9 fix(test-e2e-cli): stage CLI test assets on the node and SPOT the node src dir
The per-role existence asserts and the test script path were controller paths evaluated on the remote swarm node (docker connection, node checkout under /opt/infinito-nexus), so every svc-runner swarm job failed on a stat that could never succeed. Assert both required assets in one localhost-delegated loop, stage roles/<app>/files/test onto the node, run the staged script, and delegate the optional defaults/vars probes to the controller (they silently skipped before). svc-runner's test reaches its sources via RUNNER_SRC_DIR (compose: INFINITO_SRC_DIR dev-container mount; swarm: node checkout). The node checkout path itself moves into a single source of truth: INFINITO_NODE_SRC_DIR in default.env, consumed by the bootstrap/copy scripts, the swarm-playwright target, and the swarm inventory extras.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 17:30:36 +02:00
kevinveenbirkenbachandClaude Fable 5 c2b16091de fix(inventory): make biber's dev password realm-policy compliant
Keycloak's realm passwordPolicy requires a digit and a special char;
the LDAP-disabled seeding path died on set-password with
invalidPasswordMinSpecialCharsMessage. All consumers read the value
through the users lookup, so tests and seeding stay in sync.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 02:37:27 +02:00
kevinveenbirkenbachandClaude Opus 4.8 005f649498 refactor(onion): the inventory provisioner is the SPOT for the node onion
Mint the node onion during provisioning and write its address straight into applications.svc-net-tor.services.tor.node, dropping the INFINITO_TOR_NODE_ONION env indirection: ensure_node_onion (was init_env) mints/reuses the .onion-identity key files + returns the address (no env write); identity_hs_dir/IDENTITY_DIRNAME move to utils.tor_onion so cli and utils share the path SPOT; provision.apply_tor_node_onion sets services.tor.node in host_vars when svc-net-tor is deployed (run by 'inventory provision' at deploy time, and the svc-net-tor role copies the same key files into the daemon); domains.py reads services.tor.node; nginx_vhosts reads the key file instead of os.environ; INFINITO_TOR_NODE_ONION removed from default.env/passthrough/compose + the inventory env-lookup. Bonus: the address now lives in the persistent key files, surviving .env regeneration by make test/autoformat.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 23:53:18 +02:00
Alejandro Roman 48c49b8c7c Merge remote-tracking branch 'upstream/main' into feature/role-stalwart-mail 2026-07-03 07:26:30 +02:00
Alejandro Roman 564c0ab4df fix palywirght issues part 1 2026-07-01 22:33:20 +02:00
kevinveenbirkenbach 9b2f870777 Merge branch 'main' into feature/swarm-nfs 2026-07-01 05:37:36 +02:00
kevinveenbirkenbachandClaude Opus 4.8 9942dbdbcf fix(svc-storage-nfs-server): fsid=0 v4 root + nfs_* lookup SPOT
Root cause of the kernel-flavor swarm ENOENT: no export carried fsid=0, so the NFSv4 mount of the nested /srv/nfs/infinito-state sub-export had no pseudo-root anchor and the server returned NFS4ERR_NOENT on the PUTROOTFH+LOOKUP walk. crossmnt could not fix it (it only crosses a boundary mid-walk, never sets the starting filehandle). Kernel now exports the state dir itself at fsid=0 and clients mount the bare v4 root ':/' (zero LOOKUPs). ganesha keeps Pseudo=Path and v3 mounts the full path. The self-bind is kept so fsid=0 rides a real mount boundary (stable st_dev) under overlayfs DinD.

The NFS derivations move to per-value lookup plugins (nfs_flavor, nfs_client_src, nfs_fstype, nfs_mount_opts, nfs_state_path) backed by pure helpers in utils/storage/nfs.py with unit tests; group_vars STORAGE_NFS_* become thin lookup wrappers. The dev/act->ganesha flavor rule stays the single SPOT in the role config; nfs_flavor reads the merged value so client and server agree.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 15:27:53 +02:00
kevinveenbirkenbachandClaude Opus 4.8 cf3531b9e3 refactor(svc-storage-nfs-server): drive nfs flavor from meta/variants.yml
The inventory applications override in default.yml is dropped during inventory provisioning (CI ran kernel despite RUNTIME=github + a ganesha override), so select the flavor in meta/variants.yml where it reliably applies: ganesha for RUNTIME in [dev, act], kernel for github CI and production. meta/services.yml keeps kernel as the base default.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 07:41:12 +02:00
kevinveenbirkenbachandClaude Opus 4.8 539a078475 fix(svc-storage-nfs-server): select ganesha in DinD CI, kernel default
The github swarm legs picked the kernel NFS flavor inside the privileged DinD nfs-server, where kernel nfsd is not network-namespaced and wedges exportfs. Move flavor selection out of the role meta default and into the dev inventory: meta/services.yml now defaults to kernel (production); the development inventory overrides it to ganesha for the containerized runtimes (dev/act/github) and keeps kernel for real hosts (RUNTIME=host).

Add meta/variants.yml (kernel + ganesha) and a TODO to test both flavors via files/test.sh, plus server-side kernel-nfsd diagnostics in 12_collect_diagnostics.sh.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 06:22:26 +02:00
kevinveenbirkenbachandClaude Opus 4.8 00543c4f9d fix(svc-storage-nfs-server): let the controller NFSv4-traverse the base export to mount the nested state
The controller mount of the nested state export failed with server 'No such file or directory': over NFSv4 the controller must LOOKUP-traverse the base /srv/nfs export (real export, fsid=1) to reach the nested /srv/nfs/infinito-state (fsid=2), but a controller outside svc-swarm-node is absent from the base client list, so traversal is denied. Re-export the base read-only to SVC_STORAGE_NFS_SERVER_CONTROLLER_CIDRS (empty by default -> no extra exposure; in-node controllers are already listed; the swarm test sets the lab subnet). Worker nodes keep base rw via their explicit-IP lines (exports(5) host-specificity wins).

Add nfs_export_all_rw seboolean for the kernel flavor so RedHat SELinux-enforcing hosts (fedora/centos) serve the export; install python3-libselinux and re-gather the selinux fact first so the boolean is not silently skipped when the binding is absent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 01:58:09 +02:00
Alejandro Roman 17d6ac904f Merge remote-tracking branch 'upstream/main' into feature/self-host-runners
# Conflicts:
#	.github/workflows/test-deploy-universal.yml
#	.github/workflows/test-deploy-workstation.yml
2026-06-27 23:45:38 +02:00
kevinveenbirkenbachandClaude Opus 4.8 36ebf34b64 Merge branch 'main' into feature/swarm-nfs
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 17:13:55 +02:00
kevinveenbirkenbachandClaude Opus 4.8 bb4e8bd2b8 fix(svc-storage-nfs): NFSv4 + correct probe export + soft dev/act mounts (SPOT)
The dev/act ganesha server is NFSv4-only, but the swarm-test runtime extras forced storage.nfs.version: 3, so every mount failed 'Protocol not supported'. Set version 4 (write_extras + dev swarm inventory). The client probe mounted the Root_Squash export_base and a root probe-write failed; point it at the No_Root_Squash state path. Both client mounts hardcoded 'hard', so a torn-down test server wedged nodes into D-state; gate them to soft in dev/act (reusing the controller mount's gate). Hoist the shared mount-opts string into the STORAGE_NFS_MOUNT_OPTS SPOT (group_vars/all/15_storage.yml), used by all three mounts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 20:34:46 +02:00