Job 104155149606 died at the warm request with
litellm.APIConnectionError: OllamaException - Cannot connect to host
ollama:11434 [Connect call failed ('192.168.244.10', 11434)]
192.168.244.10 is not ollama. It is the manager node on the lab bridge: the
name never resolved through the overlay, because the litellm task was attached
to ingress, litellm and postgres and to nothing else. The lmstudio round is the
control, where the same replica does carry the lmstudio network.
The attachment comes from _is_consumer, which reads svc-ai-litellm's own
services.<backend>.enabled and .shared. Variant 1 pins both false. The config
template meanwhile asked `'svc-ai-ollama' in group_names`, which answers a
different question: is ollama deployed anywhere in the round. It is, six edges
away, because litellm pulls postgres, postgres pulls tor, tor pulls the mirror,
the mirror pulls prometheus and prometheus declares flowise, which declares
ollama. So the gateway advertised a route it had no network for.
The template and LITELLM_CHAT_MODEL_SERVED now read the same two flags the
network attachment reads, through one constant each. Publication and routing
cannot diverge any more, and in a variant that declines a deployed backend the
warm task no longer runs at all.
LITELLM_CHAT_MODEL and LITELLM_BACKEND_ROLES are deliberately left alone.
5324eadd8b made them inventory-scoped because web-app-mattermost's assert, the
one task in its file carrying neither delegate_to nor run_once, re-templates in
each worker's own scope; web-app-discourse's addon tasks carry no delegate_to
at all. A host-scoped model name fails there again. Both templates here render
delegate_to STACK_HOST plus run_once, where group_names is authoritative.
meta/services.yml keeps its `in group_names` flags, so the five separate
substring detectors that decide whether a flag counts as dynamic keep seeing
svc-ai-litellm.
Measured over every (deployed, pinned) combination against the real
expressions and the real template: the model consumers are told to ask for is
published whenever the warm task runs, and nothing is published without its
flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34893750601 lost four jobs at svc-ai-litellm's warm request, every one of
them on the same upstream body:
No models loaded. Please load a model in the developer page or use the
`lms load` command.
The catalogue handed `lms get --yes --gguf` a bare name, which that command
treats as a hub search term. Its own help spells out the consequence: "If there
are multiple staff picks matching the search term, the first one will be used."
For `qwen2.5-0.5b-instruct` the first pick was Qwen2.5-Coder-0.5B-Instruct, so
the key LM Studio indexed never matched the `openai/{{ _model.name }}` the
gateway sends upstream, and just-in-time loading had nothing to load.
A `source:` now carries the full HuggingFace URL, the one spelling lms get
resolves deterministically; a bare owner/model pair addresses an LM Studio Hub
artifact instead and fails even for a model already on disk.
Measured against the shipped image rather than reasoned: the URL yields modelKey
qwen2.5-0.5b-instruct, `lms ps` is empty right before the request, and a plain
/v1/chat/completions for that name answers 200 and leaves the model resident. No
load step belongs here, because the server reports "Just-in-time model loading
active" and does it itself.
The embedding entry is renamed in the same breath. LM Studio indexes that model
as text-embedding-nomic-embed-text-v1.5, so the gateway was addressing a key
that never existed there either, and the first consumer to embed would have hit
the same 404.
The three fields stay separate and each has exactly one reader: Ollama pulls by
alias, LM Studio downloads by source, the gateway addresses name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway could reach exactly one remote provider, through a bare
OPENROUTER_API_KEY group_var with an empty-string default and an `{% if %}`
around a single hardcoded model entry. OpenAI and Anthropic had no path at all.
All three are now declared in roles/svc-ai-litellm/meta/secrets.yml, which
mapping.py already calls the application config schema, so a key lands at
applications.svc-ai-litellm.secrets.credentials.<provider>_api_key like every
other credential. Each carries the `type` and `regex` its value must satisfy and
defaults to the central API.<provider>.api_key block, which keeps one
operator-facing place to fill a provider in while the role owns the shape.
type and regex had to become real to be worth declaring. 192 `validation:`
blocks sit in the role schemas today and not one line of Python enforces them,
so a `regex` would have joined a pile of decoration. validate_supplied_value
now runs on every path a value arrives from outside — the --set override, a
value already in the inventory, and the value standing next to a `default:`.
A generated value stays exempt because the algorithm that produced it is named
by the same schema and generate_value takes no schema argument. min_length is
deliberately left unenforced: the generators ignore it, so holding them to it
would fail where nothing is wrong.
The pattern is written for the paste, not for the vendor. OpenAI moved issuance
from sk- to sk-proj- without notice and Anthropic issues sk-ant-api03-; a schema
pinning a prefix would reject a valid key and abort inventory creation. `\S{20,}`
accepts every form those three issue today and one nobody has announced, and
rejects a trailing newline, an embedded space, a truncated copy and a
<placeholder>. A key pasted into the wrong provider's slot is caught at deploy
instead, where the route it publishes answers nothing.
The gateway iterates one LITELLM_REMOTE_BACKENDS list instead of three `{% if %}`
blocks; config.yaml.j2 and env.j2 read the same list and selectattr('key')
decides. LITELLM_CHAT_MODEL falls back to the first configured remote alias
rather than to a hardcoded openrouter/auto.
For CI the keys travel as INFINITO_API_{OPENAI,ANTHROPIC,OPENROUTER}_KEY through
default.env, the passthrough allowlist, the infinito service in compose.yml, the
workflow env block and the development inventory. The compose leg is not
optional: the env lint rejected the change without it, because a key the
inventory reads but compose does not forward resolves to empty inside the
container with no error. The workflow reads `secrets.X || vars.X` so a variable
works as asked while a secret wins, which matters because a variable is not
masked and the development inventory sets MASK_CREDENTIALS_IN_LOGS: false.
An unset key is not an error anywhere: the provider's routes stay unpublished
and the CLI test asserts their absence rather than skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fourteen roles repeated the same two lines to say they use the gateway: a
`shared` flag gated on group_names, and an `enabled` flag conjoining that gate
with LITELLM_CHAT_MODEL_SERVED. Both are now single-sourced.
`shared` becomes literal true. The flag means the provider is a shared instance
rather than one deployed per consumer, and a central gateway is always that;
the dynamic form only said "shared if deployed", which `enabled` already
decides. The 50 shared pins under `litellm:` leave meta/variants.yml with it.
The tree had already noticed: 20 of those 50 lines carried
`# nocheck: variants-static-override`, two of them with copy-pasted text about
container_backup. This narrows e5de4e085a for one service key rather than
reversing it; every other dynamic flag keeps the group_names form.
`enabled` becomes `{{ LITELLM_USABLE }}`, defined in group_vars/all/16_ai.yml.
That shorthand is only possible because is_explicit_truth learns about it.
The resolver reads a dependency out of the flag TEXT — "in group_names" in
value — so a constant hides the role id and the provider silently leaves the
static closure of every role referencing it. Measured: the old expression
resolves True, `{{ LITELLM_USABLE }}` resolves False. GROUP_GATED_CONSTANTS
names the constants whose own definition carries the gate, and the docstring
states the obligation: a SPOT of this kind that is not listed there drops its
provider without a word.
The wider fix, accepting any Jinja as dynamic, was measured and rejected: 35
other flags carry a Jinja value without the marker (MCP sidecars, javascript
and SSO gates, recaptcha), and they would all enter closures they are not in
today.
A literal shared flag is no longer dynamic, so test_dynamic_flags wants the
`# nocheck: dynamic-flag` its own message documents for legitimate exceptions;
where a marker already existed the rule list is extended rather than a second
comment added.
Behaviour is unchanged where it counts. The closure keeps litellm with the same
app counts (wordpress 26, openwebui 23, hermes 23, mediawiki 22, flowise 23),
unrelated Jinja still resolves False, and the truth table of the new enabled
form matches the old one in all four served/deployed combinations. Three roles
resolve without litellm, and `git show HEAD:` proves that predates this change:
zammad pins enabled false in variant 0 at line 39, nextcloud at line 97.
Six unit tests cover the resolver, including one asserting that every
registered constant is actually defined in group_vars, so an entry cannot
become a dead promise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-ai-lmstudio had no model-load step at all: tasks/00_core.yml set the
once-flag and the container network, nothing else. The gateway published
lmstudio/default anyway, so the route existed and nothing could answer it.
wp-compose-tor.log:48994 shows the route written into the config, :51770 shows
the warm step reaching qwen2.5:0.5b on Ollama instead, and the green ai-engine
spec ran against that. LM Studio was deployed, published and never prompted.
The pre-pull mirrors the one that works. tasks/01_manager_ops.yml resolves the
hosting node and container id, loops the declared models through
`lms get --yes --gguf`, overlaps them with async/poll and reaps before
playwright, exactly like svc-ai-ollama. That `lms` exists is measured, not
assumed: /app/daemon-run.sh:67 in the image starts ./.bundle/lms server start,
and `lms get --help` documents -y for scripting and --gguf for the format.
Synonymity needs both namespaces in one place. Ollama knows qwen2.5:0.5b, LM
Studio knows qwen2.5-0.5b-instruct, so AI_LOCAL_MODELS carries both per entry:
`alias` is what the gateway publishes and what Ollama pulls, `name` is what
`lms get` resolves. Both roles now reference that one list, and config.yaml.j2
publishes each backend under the alias, routing Ollama to ollama/<alias> and LM
Studio to openai/<name>. The lmstudio loop skips an alias Ollama already
publishes, so two backends never produce a duplicate model_name and Ollama
keeps precedence. lmstudio/default is gone; the model name is now the same
whichever backend answers it.
Ollama's preload_models changes shape from a list of strings to the {alias,
name} entries. Every consumer moves with it: the pull command, the template
loop, and both expressions in 16_ai.yml. An inventory that overrides
`applications.svc-ai-ollama.services.ollama.preload_models` with the old string
list will break at map(attribute='alias'); this is a deliberate shape change,
in the spirit of 7b15c61dfc.
The CI model list is not a property of an installation, so AI_LOCAL_MODELS_CI
lives in inventories/development/default.yml, whose own header scopes that file
to the cross-cutting environment knobs. test_variable_definitions only knew
group_vars, so it reported the constant as undefined; the shipped inventories
are a definition source too, because the deploy CLI bakes them into host_vars,
and the guard's own docstring already claimed "defined somewhere in the
repository".
A latent crash came out of this: `| list | first` raises on an empty sequence,
so a backend deployed without preload models blew up LITELLM_CHAT_MODEL. Ollama
with only embedding models could already hit it; the existing test dodged it by
rendering only _SERVED. `| default('')` now yields the empty string that
_SERVED already reports as false.
Tests: config.yaml.j2 is rendered rather than read, over ollama-only,
lmstudio-only, both and neither, asserting the published model_name set, that
both backends name the same alias, that a shared alias is published once and
routed to Ollama, and that the render is valid YAML. test_litellm_chat_model
gains the lmstudio branch, the no-preload case and the same-model assertion.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LITELLM_BACKEND_ROLES read lookup('deployment').groups, which deployment.py
fills from group_names. That variable is host-scoped: it lists only the groups
the host currently being processed belongs to.
On a swarm round the manager carries svc-ai-ollama, svc-ai-lmstudio and
svc-ai-litellm while the workers carry three groups and none of them is an AI
backend. Both branches of LITELLM_CHAT_MODEL therefore missed on a worker and,
with OPENROUTER_API_KEY empty, the model resolved to the empty string.
Only one task made that visible. Every stateful task in the gateway include
carries delegate_to STACK_HOST plus run_once, so it templates in the manager's
scope and wrote the correct model into the database. The assert carries
neither, so it re-templated in each worker's own scope and failed there while
the manager passed.
The manager-only membership is not a placement decision. The swarm test
inventory is provisioned with a single --host MGR, so every role of the round's
closure gets that one host, and extend_inventory then widens only
svc-swarm-node, svc-storage-nfs-client, svc-storage-nfs-server and the round's
own application. The three AI roles declare no placement at all; they are
manager-only because nothing widened them.
Add an inventory selector next to the host-scoped one, built from the play-wide
groups magic var, skipping groups without hosts and Ansible's implicit all and
ungrouped, sorted so the cache key is stable. The key takes it too, or two
hosts with equal group_names but different inventories would share an entry.
The blast radius is two readers, both in 16_ai.yml, and both ask only whether
svc-ai-ollama or svc-ai-lmstudio is present. On the manager the new value gains
exactly svc-storage-nfs-server and svc-bkp-nfs-2-local, neither of which either
reader tests, so the manager's answer does not move and only the workers change,
to match it.
This is the third step on the same line. 46a16127fc set .running; b149ad0d22
corrected that to .groups because a targeted apps= round leaves only the
command-line roles in the whitelist. .groups solves the targeted round and not
the multi-host one, because group_names stays host-scoped either way.
Measured in a two-host play with a real group_vars file and the real lookup:
the worker resolves ollama false through group_names and true through the new
selector, the manager is true both ways, and the assert that failed in CI
passes on both hosts.
Requirement 036 named the host-scoped backend selection as the thing that has
to change and deferred it. That residual is closed, so the paragraph now
describes the selector in use and why the two rejected ones are wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SSL_CERT_FILE, CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE replace the trust store;
only NODE_EXTRA_CA_CERTS extends it. compose_ca.py pointed all four at the
deployment own root CA, so every OpenSSL and rustls client in every container
was blind to every public certificate. Measured here, one variable, same URL:
unset curl https://registry.npmjs.org/ -> 200
a single unrelated CA -> 000 unable to get local issuer certificate
that CA + the system bundle -> 200
PeerTube was the one process that hit it: its plugin install fetches from
registry.npmjs.org at runtime through a client that honours SSL_CERT_FILE, and
it failed identically in compose and in swarm with
invalid peer certificate: UnknownIssuer
Everything else reaches its artefacts at image build time, or through Node,
whose NODE_EXTRA_CA_CERTS is additive and therefore kept working.
The replacement was never a posture. e7e8cf8c03 calls these variables an
"env-based CA trust fallback", f1ea036bd2 adds them "instead of falling back to
a plain docker run without CA trust", e79183f96a wires the extra CA in "so npm
traffic through the cache validates". Every commit framed them as adding trust.
sys-ca-selfsigned now builds ca-bundle.crt beside the root CA on every node,
from that node own files: the stack host mints the CA and workers fetch it, so
there is no cross-node dependency. The bundle is the root CA concatenated with
the first present CA_BUNDLE_CANDIDATES entry, the SPOT fc0881ff33 introduced.
The cat argument list is assembled in Jinja, so a host with no public bundle
emits the root CA alone, which is exactly the previous behaviour, rather than an
empty argument that would fail. bundle_host joins the existing stray-directory
sweep and the build runs after it.
compose_ca.py uses the bundle only when the file exists and is non-empty, so a
host that has not built one keeps the previous behaviour instead of binding a
missing path that Docker would materialise as a directory. The php ini, msmtprc
and the Matrix flavour may name the container path with no fallback because the
bundle is written at task 99 and the php ini at task 123: no host can carry the
ini without the bundle.
CA_TRUST_CERT and NODE_EXTRA_CA_CERTS stay on the single certificate. keytool,
occ security:certificates:import and update-ca-certificates each take one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway publishes only models it has a backend for and renders model_list: [] when neither ollama nor lmstudio is deployed and no OpenRouter key is set. LITELLM_CHAT_MODEL fell back to openrouter/auto without checking that key, so consumers were configured for a model the gateway does not carry and every prompt failed with a 400.
Observed in the Mattermost#1 CI job: litellm.log carries 12 occurrences of "Invalid model name passed in model=openrouter/auto" next to 400 Bad Request, and tests/test-litellm-agent.js failed on every attempt while the rest of the suite passed.
Make the fallback conditional on the key so the name is empty when nothing is served, promote LITELLM_CHAT_MODEL_SERVED from the role vars to group_vars where consumers can see it (it was the third copy of this logic and the only correct one), and switch the 14 consumers from "litellm is deployed" to "litellm serves a model". The Playwright service flag reads the same lookup, so the spec skips together with the Ansible gate. The bool filter carries weight: the folded scalar yields the string False, which Jinja treats as truthy.
A new unit test renders both expressions against Jinja and pins the invariant that a model name exists exactly when a model is served, including the case where ollama preloads embedding models only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Whether a resolver answers on the docker bridge was inferred from the host's
own resolv.conf: all entries loopback was read as "a listener of ours is there,
and containers must reach it on the bridge". A systemd-resolved stub binds
127.0.0.53, NetworkManager's dnsmasq binds 127.0.0.1, and a node dnsmasq binds
both those and the bridge. All three look identical from resolv.conf and only
the third is reachable from a container.
So ordinary hosts were handed an address with nothing behind it. The
Instructions job died there: daemon.json carried "dns": ["10.208.0.1"] alone,
and the openresty build could not resolve dl-cdn.alpinelinux.org while the
daemon, resolving through the host, pulled the same base image fine. Two netns,
two outcomes.
e308e18923 had already fixed this once, with a reachability gate whose message
names the same URL. b09f2cbd82 dropped that gate and ce43263745 put the
resolv.conf inference in its place, so the guard was gone and only the missing
docker0 fact still hid it. 285ba1e76c removed that accident and the failure
surfaced.
svc-net-tor and NETWORK_CONTAINER_BRIDGE_RESOLVER now say it instead, the
latter false by default and true in the swarm lab, which provisions the dnsmasq
its nodes resolve through. Extra-vars are global, so the lab's NFS and backup
hosts carry it although no listener is installed there; the clearnet resolver
behind the bridge keeps them working.
The docker0 fact is gone from the address too. It is not merely late, it is
wrong while it is late: facts are read before the daemon config is applied, so
docker0 still holds docker's default and writing default-address-pools moves
the bridge out from under it. A swarm node shipped "dns": ["172.17.0.1", ...]
against a live 10.208.0.1, which is why its containers asked a dead address and
fell through to the public resolver, where infinito.example does not exist. The
pool answers before the daemon exists and answers the same afterwards, and it
is read per slice rather than per supernet, which differ once a pool does not
start on a slice boundary.
A list that would carry only the bridge now gets NETWORK_PUBLIC_DNS_RESOLVERS
behind it, so a declaration that turns out to be wrong costs a wasted first
query rather than all name resolution. An empty declaration raises.
Rendered from the repository's own variable files: the Instructions host yields
[] and the template omits the key, which is what de95a95c65 emitted while that
job was green; lab and tor nodes yield the live bridge ahead of the clearnet
resolver even with a stale docker0 fact present; an ordinary host yields only
its declared resolver.
The suite asserted the two behaviours this replaces, including one case added
with 285ba1e76c demanding the stale fact win. They are inverted, and a
cross-product over nameserver shapes, tor membership, clearnet and the docker0
fact now asserts that no shape ends up holding the bridge alone - the old suite
checked each shape by itself, which is how this passed through it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LITELLM_BACKEND_ROLES decides which backend branch LITELLM_CHAT_MODEL takes:
the first non-embedding ollama preload model, lmstudio/default, or the
openrouter/auto vendor fallback. It read lookup('deployment').running, which
plugins/lookup/deployment.py resolves as `whitelist or groups`.
On any targeted apps= deploy that whitelist holds only the roles named on
the command line, while backends like svc-ai-ollama arrive as dependencies
of the round. Both branches therefore missed and every consumer was
configured with openrouter/auto, a vendor model the in-cluster gateway does
not serve. The gateway answered each prompt with
400 Invalid model name passed in model=openrouter/auto
while /v1/models listed smollm2:135m and lmstudio/default, so the surface
looked correctly wired while no consumer could complete a prompt.
Read .groups instead, the round's deployed closure, which is what the
`'<role>' in group_names` form used across meta/services.yml already means.
This reverses the selector introduced in 46a16127fc, whose rationale in
requirement 036 assumed the two selectors agree on a single-host round;
they do not once apps= is set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
TOR_EGRESS_VIRTUAL_NET_IPV4 was Tor's upstream default 10.192.0.0/10, which
spans 10.192.0.0 to 10.255.255.255 and therefore contains the whole docker
pool that 08_networks.yml allocates as 10.208.0.0/12.
The value drives an iptables rule in svc-net-tor/tasks/router.yml that
REDIRECTs TCP for that range into Tor's TransPort. With the ranges
overlapping, every connection from the controller to a container IP or to
the docker bridge gateway was captured and handed to Tor, which has no
mapping for a private address and resets it. The failure surfaced as
`errno 104 Connection reset by peer` against containers that had been up
for tens of minutes without a restart, and it reached anything addressing a
container through DOCKER_REACH_HOST, not just the LDAP schema load where it
was first seen.
Move the virtual range to 10.240.0.0/12, which sits above the docker pool.
Nothing else in group_vars or roles allocates the upper 10/8 space.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A role that enables its native AI surface can reach the gateway with an empty
credential, or point its base URL at something that resolves outside the
cluster, and still deploy green. Both are silent: the surface simply answers
from somewhere else, or not at all. This adds the shared contract that makes
either state fail the deploy, so a consuming role inherits one definition of
"inside the deployment" instead of restating it.
The key assertion refuses an empty credentials.litellm_api_key while
services.litellm is enabled. Without it the gateway leg is unauthenticated and
nothing in the run says so.
The URL assertion is four independent checks, each able to fail on its own:
the scheme is plain http, the hostname carries no dot, the hostname is not the
consuming role's own domain, and the port is the gateway's internal port rather
than its published one. The undotted check is the load-bearing one. The
container dns-search suffix completes a dotted host into the public FQDN, so
`litellm.infinito.example` leaves through the ingress while looking local in
the config; only a bare service name stays on the container network. The
own-domain check catches a role that points its AI surface back at itself, and
the internal-port check catches a config that would work only via the host
bind.
An earlier revision of this file compared two values that both came from the
same lookup and therefore could never fail. Every check here is falsifiable
against the value the role actually deploys.
group_vars carries the SPOT the contract measures against.
LITELLM_OPENAI_BASE_LOCAL_URL appends the OpenAI-compatible /v1 suffix once,
rather than in each role. LITELLM_CHAT_MODEL derives the model from whichever
backend is deployed -- an ollama preload model with embedding models rejected,
else lmstudio, else openrouter -- resolved through `lookup('deployment').running`
so it reflects what is running rather than what the inventory could contain.
Contract and vars land before the roles that include them; the reverse order
would commit roles referencing a file that does not exist yet.
Verified: the contract's assertions ran green at runtime for matrix,
mattermost, moodle and zammad. Zammad's is the strongest evidence, since
`Setting.set('ai_provider_config')` triggers upstream
`AI::Provider::CustomOpenAI.ping!` and therefore a live gateway round trip.
`make quality` green on test-lint, test-unit, test-integration, test-external.
Not covered: nothing yet forces a role that declares services.litellm to
include this contract, so a future consumer can still skip it silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The package-cache frontend terminates TLS for package traffic and signs
with its own CA: probed from inside the test container, pypi.org arrives
as subject CN=pypi.org issued by CN=Infinito Package-Cache Frontend CA.
package-frontend-ca.sh installs that CA into the OS store, which is why
apt, pip, curl and urllib work — but requests reads certifi, which has
never heard of it, so test-external failed its reachability precondition
with CERTIFICATE_VERIFY_FAILED. Upgrading certifi does not help; the
issuer is not a public root.
load.sh is where the runner puts the venv on PATH, so it is also where the
interpreter carrying the pip certifi is chosen; pointing REQUESTS_CA_BUNDLE
and SSL_CERT_FILE at the store the stack deliberately fills keeps one trust
decision instead of two. A caller-set value still wins.
The bundle location was about to become a fourth copy: dev-nix and the
network diagnose CLI each carried their own candidate list, and the two had
already drifted apart in their fourth entry. All of them now read
group_vars CA_BUNDLE_CANDIDATES — Ansible directly, the shell through a
generated env key, python through read_group_paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32118850138 lost the :53 listener, so the sampler counts listeners.
Run 32538854067 lost DNS with the listener still bound: sockets-udp.txt
showed dnsmasq holding 0.0.0.0:53 while every forwarded query timed out,
and the sampler reported that hour as healthy.
Each tick now also records what a glibc caller gets back — for the zone
this resolver owns, derived from its own address= line, and for one name
it must forward — with the latency that caller paid. The rescue collector
probes the in-zone name too: probing external names alone cannot separate
"forwarding is gone" from "dnsmasq is gone".
The sampler now also runs on the swarm lab DNS nodes; it only ever ran in
the compose path, which is why the Magento artifact carried no sampler log
at all. Its location comes from group_vars FILE_DNS53_SAMPLER_LOG and is
required at every call site, so a run cannot quietly collect nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Docker's built-in pools yield about thirty networks. The stack holds
seventy-two, so every further network creation failed with 'all predefined
address pools have been fully subnetted' — which is why web-app-nextcloud
never came up: it brings its own networks and was the first role to hit the
ceiling.
Declare the pools beside the nested daemon's in group_vars, so both docker
daemons read their allocation from one place, and render them from there.
The base moves to 10.208.0.0/12 because 172.16/12 is exhausted: a probe
subnet in 172.28 was refused as overlapping while one in 10.210 was
created.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Output of 'make autoformat': ruff collapses expressions that were wrapped across lines without need, shfmt and the yaml formatters drop surplus blank lines. No semantic change - the whole diff is 10 insertions against 39 deletions, and every hunk joins lines that already said the same thing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f713ba835 rewrote the BIND_CREDENTIAL lookup onto secrets.credentials and left a doubled blank line above it. Autoformat collected it after that commit had already landed. No key, value or indentation is touched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml became meta/secrets.yml, but its content still landed at
applications.<app>.credentials — the one meta file whose payload did not sit
under a root named after the file. Every other topic obeys that rule:
meta/volumes.yml goes to volumes, meta/rbac.yml to rbac. Credentials now go to
secrets.credentials.
The path exists exactly once, in utils/manager/credential_key.py, and
OVERRIDE_SECTION is derived from its two segments rather than spelled beside
them, so the dotted form used by --set and the nested form used by the
inventory cannot drift. Six consumers read it: the loader, the generator, the
CLI override parser, the two emit helpers, and the four lookup validators.
--set now accepts exactly one key shape,
applications.<app>.secrets.credentials.<key>. The three short forms are
rejected instead of guessed at: the same short key names different credentials
on different apps. That also retired primary_app_id, which only existed to
disambiguate them.
Three defects surfaced while wiring this up. _apply_one_role_schema still wrote
to the old root, so an app's own credentials would have moved while its
providers' stayed behind. web-app-pgadmin reads through
lookup('applications', ...).credentials.<key>, an access path no regex over
lookup('config', ...) could see, and would have deployed with an empty postgres
password. And _META_NON_GENERIC_TOPICS still listed 'schema' — a bare topic
name, invisible to any path search — so the generic file-root loop assigned the
raw schema under secrets and the follow-up write landed inside the cached
document, truncating keycloak's four credentials to one for every later
reader.
tests/integration/ansible/lookups/config/test_credentials_roundtrip.py pins both
directions of the seam and the cache invariant that last defect broke.
Existing inventories still carry the pre-rename root; migrating them is
deliberately out of scope here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two callers polled the same stack_ready.sh with different budgets: the
handler read swarm_stack_converge_retries while post_deploy hardcoded
150. Eleven roles had set the override and were unprotected at the
second caller anyway. Both now read one SPOT in group_vars.
The gate also had no way to give up. A crash-looping espocrm burned 600
polls (20 min) before failing, while flowise was killed 45 s short of a
409 s image pull. has_fatal_task now exits 2 - and both callers stop at
once via 'until: rc in [0, 2]' - when a slot's newest attempt is
Rejected or Pending, or when a slot burned three attempts after the
service churned past swarm_stack_fatal_grace. Patience only grows where
nothing is broken.
report_tasks loses its 'awk !/error=$/' filter, which printed only rows
that HAVE an error and so stayed silent in exactly the no-error case
that occurred.
Measured on a live single-node swarm: a service failing 3x and
converging on the 4th returned rc 2 at t+15s before this change and rc 0
at t+32s after; a 3-replica stop-first rolling update gave 68/68 samples
rc 0; a permanent crash loop still reaches rc 2. The deploy epoch comes
from 'docker service inspect .UpdatedAt.Unix' rather than a /tmp marker,
which was reproduced contaminating a later deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`files/swarm/stack_ready.sh` has exactly two callers and they disagreed: the
handler defaulted to 150 polls at delay 2, and post_deploy.yml hardcoded the
same 150 with no way to override it. 389 seconds is not enough for a stack that
carries a large image.
Four jobs across two runs died on it, every one starting at "149 retries left":
Nextcloud 2 (run 31340965955) plus Workspace centos, debian and manjaro (run
31311996621), the last three all waiting on svc-db-mariadb, a role that carries
no per-role override. Nextcloud 2 is the measured case — the budget expired at
12:59:54 and the task reached Running at ~13:00:32, 38 seconds later. Nothing
was broken; the deadline was.
600 is not a new number: tasks/utils/swarm/wait_converged.yml already waits that
long at the identical delay, and eleven roles had each hand-raised the handler to
exactly 600. 150 was the outlier.
The declaration goes in group_vars/all, not in the role defaults. A role default
is invisible to the by-path include in roles/svc-db-openldap/tasks/01_docker.yml,
which pulls post_deploy.yml in after include_role has finished — verified with a
scratch play, where the same variable resolves inside the role and raises
"is undefined" one task later. post_deploy.yml already encodes that distinction:
it reads DEPLOYMENT_MODE bare and guards compose_mode_force with `| default('')`.
The lint gains a check that the gate's callers never drift apart again. It
discovers them by scanning for the script rather than listing them, so a third
caller is covered, and it asserts it found at least two so an empty scan cannot
pass silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every MCP role keeps its task tree under tasks/utils/ without numeric
prefixes, so a caller can reach any of them through include_role
tasks_from. utils/roles/mapping.py holds that path as the single source.
Ansible resolves a relative include against the including file's own
directory when that file was reached by path, but against tasks/ when it
was reached through tasks_from. The same file therefore loaded from its
own role and failed from a caller using tasks_from. Every nested include
now names {{ role_path }}/tasks/utils/..., which holds under both bases,
and a lint walks each tasks_from entrypoint transitively to keep it so.
The exec-gating lint learns the same form, otherwise it loses the include
chain and reports inherited IS_STACK_HOST gates as missing.
The expression resolving a provider's stored token appeared eighteen
times. plugins/lookup/mcp_credential.py replaces it, and mcp_probe_owner
disappears from eight roles: it was only ever a step towards the bearer
and no probe read it.
The Hermes and OpenClaw probes stop aborting the play while it is still
running. Roles deploy in dependency order, so a client probing during its
own role can run before a provider exists at all, and failing there
prevents that provider from ever deploying. The verdict moves to the
reconciliation pass, gated on MCP_RECONCILE_STRICT, which the reconciler
sets and group_vars declares so the readers need no default.
The sys-svc-mcp-reconcile role is added here; wiring it into a stage is a
separate change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The MCP discovery lookup answers with both the selected servers and the
rejected ones with their reason, but the global bound its whole result to
MCP_DISCOVERED_SERVERS, which every consuming role iterates as a list of
servers. The rejections are kept under their own name so the reason a provider
was dropped stays available, and consumers read the selected list.
Add basic_auth to the MCP auth schema so Jenkins can declare its
HTTP-Basic endpoint through the contract instead of around it.
Introduce the client-side renderers: mcp_authorization builds the
Authorization header per scheme, mcp_tool_server_connections builds the
Open WebUI tool-server entries and drops servers whose scheme it cannot
present, and mcp_renderable_servers filters a server list, since Jinja
select() needs a test and the repo ships no test plugins.
Replace the inline MCP_DISCOVERED_SERVERS Jinja with a mcp_servers
lookup, and carry auth and username per entry so basic_auth consumers can
build their credential.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The launcher declined on every btrfs host for a reason no operator could
guess: `/var/lib/docker` is created by mkdir, so it is an ordinary
directory inside the enclosing subvolume, and `btrfs subvolume snapshot`
needs a subvolume root as its source. Documenting that as a manual
prerequisite would have left the feature switched on and never firing.
ensure_subject_subvolume.sh converts the data root on the deploy that
introduces the mode. The copy uses `cp --reflink=always`: both trees live
on the same btrfs, so extents are shared instead of duplicated and the
conversion costs seconds and no extra space no matter how large the data
root is - where an rsync copy would have meant hours of downtime and a
free-space requirement of 110%. The original tree is kept until the daemon
has answered on the converted root, so a failed copy, a mismatched
verification pass or a daemon that will not start all put it back and
start docker on it again.
The mode itself moves from group_vars into meta/services.yml as
`snapshot_mode`, read through `lookup('config', ...)`, so a single host can
override it from the inventory. Falsey spellings normalise to `never` and
anything else fails a deploy-time assert, because YAML reads a bare
off/no/false as a boolean and `--mode False` would kill the unit nightly.
btrfs-progs is installed wherever a btrfs mount exists, with centos opted
out: centos stream ships neither the kernel module nor the package, and
demanding it there would fail the package-availability test for the whole
RedHat family.
The pin moves to 3.2.2, which is not cosmetic: that release carves the
snapshot inside its subject, and the reaper here looks for leftovers in
exactly that place. Against 3.2.1 it would search beside the data root and
never find one.
Two drills run before anything else in test.sh: subvolume_probe.sh proves
the conversion carries every path, payload and symlink across and that the
result can actually be snapshotted; snapshot_probe.sh pins the launcher's
decision, including the refusals that keep a host on the live copy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baudolo 3.2.1 can capture every volume from one atomic filesystem
snapshot, which removes both the container stops and the torn copy a
live rsync of a database produces. It states the snapshot kind rather
than probing it, so naming a kind the host cannot deliver costs a whole
backup run.
Deciding that at deploy time is not possible: the set of docker volumes
changes between deploys, and a volume created afterwards would sit in the
snapshot as an empty directory, so the copy would succeed while holding
nothing. The decision therefore moves into the run itself. ExecStart now
points at baudolo-snapshot, which probes the host, appends
--snapshot/--snapshot-subject when a snapshot will be faithful, and
becomes the baudolo process either way.
STORAGE_BACKUP_SNAPSHOT_MODE ships as `never`, so no host changes
behaviour until an operator opts in - baudolo's snapshot path has never
been driven end to end on a real deployment. `auto` probes and degrades
to today's live two-pass copy with the reason in the journal; a stated
`btrfs`/`zfs` skips only the fstype lookup and fails the unit when the
host cannot deliver, which is the point of stating it. The token is
`never` rather than `off` because YAML 1.1 reads a bare off/no/false as a
boolean, which would render `--mode False` and kill the unit on every
timer fire; falsey spellings are normalised and a deploy-time assert
rejects anything the launcher's choices would not accept.
The probe refuses unless the data root is a btrfs subvolume root whose
parent is writable and on the same filesystem with no nested subvolume
below it, and unless every docker volume uses the local driver, declares
no Options of its own, sits under the data root and has nothing mounted
inside. The volume rules read the declaration rather than the mount
table, because docker mounts an NFS-backed volume lazily and unmounts it
when the last container stops - such a volume is a separate filesystem
that a snapshot of the data root captures as an empty directory. `auto`
accepts btrfs only; zfs needs to be stated, since none of its premises
could be exercised here.
Before taking a snapshot the launcher removes the `.baudolo-*` leftovers
a killed run leaves beside the data root: baudolo removes its snapshot in
a finally, which a SIGKILL or a reboot skips, and a read-only snapshot of
the data root pins every block the live root later frees.
--hard-restart-projects is stripped when the snapshot flags are appended:
it exists for stacks whose database cannot be backed up hot, which is
what a snapshot removes.
The pin moves 3.1.3 -> 3.2.1. That version is not on PyPI yet; publish it
before deploying, or the pip task fails on every host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baudolo 3.2.1 can capture every volume from one atomic filesystem
snapshot, which removes both the container stops and the torn copy a
live rsync of a database produces. It states the snapshot kind rather
than probing it, so naming a kind the host cannot deliver costs a whole
backup run.
Deciding that at deploy time is not possible: the set of docker volumes
changes between deploys, and a volume created afterwards would sit in the
snapshot as an empty directory, so the copy would succeed while holding
nothing. The decision therefore moves into the run itself. ExecStart now
points at baudolo-snapshot, which probes the host, appends
--snapshot/--snapshot-subject when a snapshot will be faithful, and
becomes the baudolo process either way.
STORAGE_BACKUP_SNAPSHOT_MODE ships as `never`, so no host changes
behaviour until an operator opts in - baudolo's snapshot path has never
been driven end to end on a real deployment. `auto` probes and degrades
to today's live two-pass copy with the reason in the journal; a stated
`btrfs`/`zfs` skips only the fstype lookup and fails the unit when the
host cannot deliver, which is the point of stating it. The token is
`never` rather than `off` because YAML 1.1 reads a bare off/no/false as a
boolean, which would render `--mode False` and kill the unit on every
timer fire; falsey spellings are normalised and a deploy-time assert
rejects anything the launcher's choices would not accept.
The probe refuses unless the data root is a btrfs subvolume root whose
parent is writable and on the same filesystem with no nested subvolume
below it, and unless every docker volume uses the local driver, declares
no Options of its own, sits under the data root and has nothing mounted
inside. The volume rules read the declaration rather than the mount
table, because docker mounts an NFS-backed volume lazily and unmounts it
when the last container stops - such a volume is a separate filesystem
that a snapshot of the data root captures as an empty directory. `auto`
accepts btrfs only; zfs needs to be stated, since none of its premises
could be exercised here.
Before taking a snapshot the launcher removes the `.baudolo-*` leftovers
a killed run leaves beside the data root: baudolo removes its snapshot in
a finally, which a SIGKILL or a reboot skips, and a read-only snapshot of
the data root pins every block the live root later frees.
--hard-restart-projects is stripped when the snapshot flags are appended:
it exists for stacks whose database cannot be backed up hot, which is
what a snapshot removes.
The pin moves 3.1.3 -> 3.2.1. That version is not on PyPI yet; publish it
before deploying, or the pip task fails on every host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MCP_DISCOVERED_SERVERS resolves every server-direction role to its internal URL, the bearer the server role minted, and its transport, dropping any server without a token so no client renders a connection it cannot authenticate.
OpenClaw moves from 2026.3.11 to 2026.7.1 because the pinned version has no configuration surface for outbound MCP servers at all: its validator rejects mcpServers, the shipped bundle carries no bundle-mcp, and the ACP path refuses per-session servers while pointing at a gateway config that does not exist there. 2026.7.1 reads mcp.servers, which the template now renders. The gateway command gains its run subcommand because the bare gateway no longer starts anything.
Reachability was the missing half: openclaw sat on its own networks only, so the hub name resolved through the DNS wildcard to the ingress instead of the container. The hub overlay now declares an mcp consumer key, which attaches every role carrying services.mcp.
A probe task runs openclaw mcp probe after the handlers flush and fails the deploy when a configured server does not answer. Without it the previous run reported failed=0 while the client could not reach anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`docker stack deploy` drops the per-service `runtime:` key, so a swarm worker
can only carry the sandbox as its daemon-wide `default-runtime`. That was never
set: the role reached the manager only, no node was labelled, and one of three
agent replicas executed model-generated code under plain runc.
Setting it from the base render is not possible either. The container role
writes `daemon.json` before the runtime binary exists, and dockerd refuses to
start when a runtime path does not resolve ("stat /usr/local/bin/runsc: no such
file or directory"). The role's own attempt to re-render afterwards was a silent
no-op, because `include_role: sys-svc-container` hits the
`run_once_sys_svc_container` guard.
Move the probes plus the render into `daemon_config.yml` and let the role
re-enter it via `tasks_from` after installing runsc. Only runtimes whose binary
the probe found are registered, and `default-runtime` is written only for a
present shim on a non-stack host, so the stack host keeps the shared kernel for
the platform's own services.
The placement constraint compares against a fixed literal because Docker
constraints have no presence test: label the node `kata-capable=true` rather
than with the runtime name, otherwise a Kata node fails a `== runsc`
comparison. Without a worker outside the stack host no node is eligible at all,
so refuse the deploy instead of leaving the service pending forever.
`select_runtime.yml` is gone: hermes and openclaw included it by absolute path
out of a foreign role, and one probe in the container role now serves every
consumer.
Verified live on a three node cluster: worker `DefaultRuntime=runsc`, both
workers labelled, manager unlabelled, service constrained, and the running
replica reports `runtime=runsc`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Requirement 034 inverts the isolation model of the other agent roles: the
boundary is the whole machine rather than a kernel boundary inside it.
Adding this role to a device switches the agent from isolated to
embodied.
The agent then runs privileged with exactly the devices from
ROBOT_DEVICE_ALLOWLIST bound into its container, and the kernel-isolating
runtime is dropped, because a microVM between the agent and the hardware
it is meant to drive defeats the purpose. The allowlist lives in
group_vars so the agent templates can read it; a role variable is not
visible across roles.
The admission checks run inside the agent role before it deploys rather
than in this role afterwards. The agent is declared shared, so the
service loader brings it up during the constructor stage while svc-ai
roles only run at the following step; a check placed here would sit
behind the deployment it exists to prevent.
The dedicated-device check now compares against something group_names
actually carries: any other web-app role on the host. The previous list
held Docker node labels, which are a different namespace from Ansible
inventory groups, so that intersection could never match and the failure
message always blamed a multi-node swarm.
Which agent embodies the device follows from the topology: both agent
flags take the dynamic in-group_names form, Hermes Agent wins when both
are present, and the matrix pins one per round so neither path rots.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consumers of the model plane speak one OpenAI-compatible endpoint and
carry one virtual key; which backend answers is the gateway's business.
svc-ai-litellm renders its model_list from the round's actual topology,
so an alias only exists when its backend role is in group_names, and it
emits an explicit empty list when no backend is present at all because
LiteLLM rejects a bare model_list: key.
Keys reach the gateway through the per-consumer credential rather than
being minted there: 01_keys.yml probes /key/info and only POSTs
/key/generate on 404, so reruns stay idempotent. The consumer list is
built in 00_core.yml, not inside the included file, because 01_keys.yml
delegates to the hosting node and delegate_to is templated before when.
The gateway declares ollama and lmstudio as consumer flags so the
network renderer attaches it to its own backends; without them it cannot
reach them.
svc-ai-lmstudio runs the headless llmster daemon behind expose only, so
it is reachable from the gateway but never bound on the host, matching
svc-ai-ollama.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Agent workloads execute model-generated code, so they need a kernel
boundary on top of the app's own sandboxing: Kata plus Firecracker where
hardware virtualisation is available, the gVisor userspace kernel
otherwise, never plain runc.
runsc is fetched as the upstream static release binary with its published
sha512 and the host architecture in the URL, because Debian ships only a
2024 build and Arch and Fedora ship none at all; the same path serves the
arm64 Raspberry Pi target.
select_runtime.yml is a reusable task file: it picks kata only when
/dev/kvm, the kata shim and a bare host all hold. /dev/kvm is visible
inside a Docker-in-Docker daemon through host passthrough, but the microVM
still cannot start there (fuse-overlayfs, no containerd block devices), so
DinD is excluded explicitly and degrades to runsc.
daemon.json gains a runtimes block gated on group membership. Under DinD
runsc additionally needs --network=host: its netstack cannot pluck
interfaces inside a nested netns and aborts with 'cannot run with network
enabled in root network namespace'. The flag keeps the syscall sandbox and
reaches the container netns through real sockets.
The role bootstraps before the service-loader preload, alongside
svc-storage and svc-registry, so the runtime is registered and the daemon
restarted before the first workload compose file renders.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolves the runner Dockerfile to the lint-standard Acquire::Retries=3
and keeps the collabora TODO wording; hardens the merged tree for the
stricter lints (apt retries in the tor Dockerfile, shell-or-true
markers, comment cleanup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Guide-provisioned single SSO apps left OIDC.CLIENT.SECRET undefined, so the Keycloak realm client render raised "object of type 'dict' has no attribute 'SECRET'" under ansible-core 2.21. Add an oidc_client_secret credential to the Keycloak schema (auto-generated like administrator_password) and default defaults_oidc.CLIENT.SECRET to it via lookup('config', 'web-app-keycloak', 'credentials.oidc_client_secret', ''); an inventory OIDC.CLIENT.SECRET still overrides through the recursive combine at tasks/stages/01_constructor.yml.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Back to the pre-swarm-nfs-fix baseline:
- 5c2c8f531 + 985ec7eab: the swarm writer-quiesce DR drill. Deep-dive
attribution of CI run 29715516546 found it the dominant failure source
(17 of 23 jobs): [9b/9] handed-off generation mismatch, [6/9] quiesce
stack-rm network race, [4/9] remote-2-local pull stall, compose discourse
restore duplicate key.
- d6f980574: openproject swarm ingress to web:8080. Carried a leaking
proxy_internal_port lookup (no per-app default) that broke the seaweedfs
dependency front-proxy ('Key web not found'); the bypass-the-Apache-proxy
approach was never verified against downloads/X-Sendfile.
- 1dd153bad: xwiki replicas:1 + mem 3g. Did not resolve the xwiki superadmin
failure (still red in run 29715516546), so it is unverified.
Re-approach each from a clean baseline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up corrections to b449f8157 (swarm DR drill):
- export test.env in 06/07 via set -a so the child db restore/probe
scripts inherit BKP_TEST_DATABASES_CSV and BKP_TEST_RESTORE_BIN
instead of aborting at their :? guards before any restore runs
- move db_probe.sh/db_restore.sh into a db/ subfolder (probe.sh,
restore.sh) and update all callers, so they stop tripping the
prefix-grouped-files structure lint
- define TEST_E2E_SWARM_DRILL in group_vars/all and drop the now
redundant default() from test.env.j2 (satisfies the undefined-var
and spot-default lints together)
- fix a mermaid parse error in the drill README: a ; in a
sequenceDiagram note was read as a statement separator
make quality-high green (test + lint phases, 15 targets, 0 failures).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The inventory gating removed the manager's unconditional volume and secrets backup groups, which had been the only include paths that ever executed the cleanup roles; svc-bkp-nfs-2-local declares both cleanups itself but their IS_STACK_HOST gate silently skipped them on the nfs host, so the required_by validator failed the deploy. The gate becomes the group_vars constant IS_BACKUP_HOST (any svc-bkp-* group membership), which covers every include site since only the five backup roles include the cleanups, and both roles now ensure the backups root exists before installing their timer: cleanback refuses a missing root by design and the size-cleanup script walks it with os.listdir.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
