mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-09 20:26:33 +00:00
d7efffc31a6bf044b1a0b2de6143b424d0af649c
158
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  kevinveenbirkenbachandClaude Opus 5
|
d7efffc31a |
fix(tor_ports): forward only the ports a role names under ports.onion
The public-port sweep never forwarded anything. It read tor.enabled from meta/services.yml on disk, where 104 of 105 roles declare it as the Jinja string "{{ 'svc-net-tor' in group_names }}", which _as_bool answers False for. Running HEAD's collect_public_ports the way HEAD's run() calls it, against the real roles directory with every role deployed, returns []. Only svc-net-tor reads True on disk, and it declares no public ports. The node onion therefore carried port 80 and port 22 and nothing else.
Reviving the sweep by reading the merged view instead would have been worse than leaving it dead: it publishes 16 ports across 7 roles, and declared is not published. openldap's ldaps 636 sits next to network.public false, Mailu's implicit-TLS ports are dropped by container_ports whenever TLS is off (which an onion deployment always is), and svc-net-tor's own SOCKS port would become an open proxy. So the source and the selection rule change together: read the variant-merged applications view, and forward only the categories a service names under ports.onion, resolving the number from ports.local when declared there and ports.public otherwise, which is the precedence container_ports publishes by.
collect_public_ports had no consumer outside this module and its own test. UDP_ONLY_CATEGORIES becomes public so the accompanying lint imports the list rather than restating it, and gains stun_turn and stun_turn_tls: those speak TCP, but the allocation TURN hands back is a UDP relay address an onion client can never receive on.
No role declares ports.onion yet, so the rendered HiddenServicePort set is unchanged by this commit alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
9a68f0eaaf |
fix(sso): read the OIDC gate's flags through the rendering config lookup
meta/addons/oidc_login.yml gates on lookup('sso_oidc_plugin'), which the applications render evaluates while it is still in flight. Under that re-entry guard the merged view hands out service flags as their own Jinja text, and bool("{{ ... }}") is True. The lookup read services.sso.enabled and services.ldap.enabled as enabled on a deployment with neither Keycloak nor OpenLDAP and announced oidc_login, so Nextcloud advertised a login hand-off to a provider that is not there.
lookup('config', ...) renders the value it returns, which is why the roughly ninety '| bool' sites in meta/ do not have this problem.
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
5b1ac73dd9 |
fix(domains): render the tor flags before the onion map reads them
services.tor.enabled is Jinja in 99 of the 105 roles that declare it. A lookup('tls') or lookup('domains') inside a meta/*.yml value runs while the applications render is in flight, the re-entry guard hands it the unrendered tree, and _as_bool read the flag's own source text as False. The onion injection then did nothing for any app and the clearnet domain was returned.
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
182ad13717 |
fix(api): let the SPOT merge survive a templar without the kwarg
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
15a6251d72 |
fix(api): resolve credentials against the declaration an inventory override hides
API is a plain mapping, so an inventory that configures three providers replaces the whole declaration from group_vars/all/18_api.yml instead of merging into it. inventories/development/default.yml declares openai, anthropic and openrouter, so github, google and jira did not exist in CI at all and lookup('api', 'github.client_id') raised.
A raising lookup inside a value the applications renderer is templating does not fail the render: _templar_render_best_effort returns the string untouched and, for a value carrying a non-env lookup(, refuses the fallback as well. The raw template text reached plugin_value.enabled | bool, which coerced it to False and emitted the deprecation warning that fails the deploy gate. Run 34893363518, job 104153194109 lost web-app-nextcloud to exactly that, for three integrations at once. The configvalue entries would have stored template text as the client id.
utils/api.py merges the scope's override over the declaration, so an override narrows values and never the provider set, and holds the enable rule: every credential a provider declares must carry a value. The api lookup reads through it; the new api_enabled lookup gates on it and returns a real boolean instead of a string a filter has to coerce.
The same class sat in group_vars/all/16_ai.yml and roles/svc-ai-litellm/meta/secrets.yml as bare API.<provider>.<key> dereferences; both now go through the lookups.
Two lints keep it shut. test_addon_enabled_renders_to_bool renders the whole applications map against a scope with API blanked, the most hostile override an inventory may legally write, and requires every addon gate to come out a boolean. test_api_accessed_via_lookup forbids a bare API dereference in Jinja under roles/, group_vars/ and inventories/.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
b18710e188 |
fix(litellm): publish a backend only when the gateway's own flag enables it
Job 104155149606 died at the warm request with
litellm.APIConnectionError: OllamaException - Cannot connect to host
ollama:11434 [Connect call failed ('192.168.244.10', 11434)]
192.168.244.10 is not ollama. It is the manager node on the lab bridge: the
name never resolved through the overlay, because the litellm task was attached
to ingress, litellm and postgres and to nothing else. The lmstudio round is the
control, where the same replica does carry the lmstudio network.
The attachment comes from _is_consumer, which reads svc-ai-litellm's own
services.<backend>.enabled and .shared. Variant 1 pins both false. The config
template meanwhile asked `'svc-ai-ollama' in group_names`, which answers a
different question: is ollama deployed anywhere in the round. It is, six edges
away, because litellm pulls postgres, postgres pulls tor, tor pulls the mirror,
the mirror pulls prometheus and prometheus declares flowise, which declares
ollama. So the gateway advertised a route it had no network for.
The template and LITELLM_CHAT_MODEL_SERVED now read the same two flags the
network attachment reads, through one constant each. Publication and routing
cannot diverge any more, and in a variant that declines a deployed backend the
warm task no longer runs at all.
LITELLM_CHAT_MODEL and LITELLM_BACKEND_ROLES are deliberately left alone.
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
5097ccdfe3 |
fix(inventory): stop holding a default's template text to the value's pattern
Run 34883491022 lost all 15 of its jobs and run 34883296966 four more, every
one of them on the same line and before any role deployed:
ERROR: the value supplied for 'credentials.openai_api_key' does not match
the pattern its schema declares: \S{20,}
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
b356954d2e |
style(inventory): wrap the keycloak users fixture the way autoformat does
make quality-high pass 75 rewrote it: a concatenation whose right operand is a multi-line literal gets parenthesised, with the operator leading its own line. The string itself is unchanged, so test_narrowing_the_credentials_still_rotates_every_user keeps the declaration that separates web-app-keycloak from web-app-a and still fails when the user half of the rotation is narrowed along with the credentials. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
46110f4311 |
perf(swarm-test): rotate the round's credentials, not every block in the file
The matrix gate rotates credentials between the two deploy passes so PASS 2 has to carry fresh values. It rotated every application block in the host_vars file, and on a matrix inventory that is not the round. Provision applies its --include as a strict intersection (filters.py:57-62), so it generates credentials for at most derive_includes, while mirror_overrides.py adds a block per mirror artefact carrying only image and version. Rotating those costs one credential subprocess each for values provision never wrote. matrix.py now passes --include derive_includes(app_id, variants=round_variants), the same call 02_provision_inventory.sh:11 feeds provision, from the same round_variants the provision step got one call earlier. The rotation set is therefore a superset of what provision generated: every database_password still rotates, because utils/manager/inventory.py:236-242 writes it under an id from that same filtered list. Measured for web-app-wordpress: 35 ids. The scope stops at the credentials. reset_credentials takes credential_ids beside application_ids because the two halves shared one list, and narrowing it narrowed _drop_user_passwords too: that resolves through required_user_policies(roles_dir, application_ids), so 60 of 66 declared users would have stopped rotating, root, backup and every mcp-* bot among them. The user half keeps the full document list and is byte-identical to before. The test fixture declared the same users in both roles, which made the regression test pass against the regression. web-app-keycloak now declares a user web-app-a does not, so the assertion has something to bite on. The CLI aborts when --include matches no block instead of rotating nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
786899700f |
feat(litellm): serve OpenAI, Anthropic and OpenRouter from declared keys
The gateway could reach exactly one remote provider, through a bare
OPENROUTER_API_KEY group_var with an empty-string default and an `{% if %}`
around a single hardcoded model entry. OpenAI and Anthropic had no path at all.
All three are now declared in roles/svc-ai-litellm/meta/secrets.yml, which
mapping.py already calls the application config schema, so a key lands at
applications.svc-ai-litellm.secrets.credentials.<provider>_api_key like every
other credential. Each carries the `type` and `regex` its value must satisfy and
defaults to the central API.<provider>.api_key block, which keeps one
operator-facing place to fill a provider in while the role owns the shape.
type and regex had to become real to be worth declaring. 192 `validation:`
blocks sit in the role schemas today and not one line of Python enforces them,
so a `regex` would have joined a pile of decoration. validate_supplied_value
now runs on every path a value arrives from outside — the --set override, a
value already in the inventory, and the value standing next to a `default:`.
A generated value stays exempt because the algorithm that produced it is named
by the same schema and generate_value takes no schema argument. min_length is
deliberately left unenforced: the generators ignore it, so holding them to it
would fail where nothing is wrong.
The pattern is written for the paste, not for the vendor. OpenAI moved issuance
from sk- to sk-proj- without notice and Anthropic issues sk-ant-api03-; a schema
pinning a prefix would reject a valid key and abort inventory creation. `\S{20,}`
accepts every form those three issue today and one nobody has announced, and
rejects a trailing newline, an embedded space, a truncated copy and a
<placeholder>. A key pasted into the wrong provider's slot is caught at deploy
instead, where the route it publishes answers nothing.
The gateway iterates one LITELLM_REMOTE_BACKENDS list instead of three `{% if %}`
blocks; config.yaml.j2 and env.j2 read the same list and selectattr('key')
decides. LITELLM_CHAT_MODEL falls back to the first configured remote alias
rather than to a hardcoded openrouter/auto.
For CI the keys travel as INFINITO_API_{OPENAI,ANTHROPIC,OPENROUTER}_KEY through
default.env, the passthrough allowlist, the infinito service in compose.yml, the
workflow env block and the development inventory. The compose leg is not
optional: the env lint rejected the change without it, because a key the
inventory reads but compose does not forward resolves to empty inside the
container with no error. The workflow reads `secrets.X || vars.X` so a variable
works as asked while a secret wins, which matters because a variable is not
masked and the development inventory sets MASK_CREDENTIALS_IN_LOGS: false.
An unset key is not an error anywhere: the provider's routes stay unpublished
and the CLI test asserts their absence rather than skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
abd6a24b73 |
test(litellm): assert the gateway routes to the backends it actually has
The gateway publishing a model proves nothing about a backend answering it: LM
Studio shipped a published lmstudio/default route for as long as it had no
model to load. This CLI test closes that gap by asking the running gateway what
it serves and comparing it against what the deployment enabled.
templates/test.env.j2 is both the input and the entry point — discover_cli_roles
finds a role by that file and filters on group_names, so the test runs only
where the gateway is deployed. meta/tests.yml names the container service key so
run_one.yml resolves the node holding it in swarm, and allows 600s for a cold
model load.
probe.py runs inside the gateway container, where the API answers on loopback
and the backends resolve by name, and uses the standard library only because
the image ships the litellm app rather than a test toolchain. It asserts that
every declared model is served, that the LM-Studio-only aliases are present
exactly when that backend is deployed, that no Ollama-backed model is published
without Ollama, that LITELLM_CHAT_MODEL is routable, that an empty deployment
publishes nothing, and that a real completion comes back non-empty.
The fourth assertion is the one worth having: LITELLM_CHAT_MODEL is built from
lookup('deployment').inventory while the config is built from this host's
group_names. Two sources for one fact, and this is where they are made to
agree.
The alias list is rendered whether or not LM Studio is deployed. Gating it
would empty the set in exactly the case the negative direction needs, turning
"a route must not stand without its backend" into a vacuous check.
The verdict logic sits in a pure evaluate() so it is exercised without mocks:
four deployment shapes, each with its positive and negative cases, including a
both-backends case that asserts a shared alias proves nothing about LM Studio
and only the exclusive one counts.
files/test/.nocheck is required rather than convenient. role-files-layout sorts
files/ by language while test-e2e-cli stages files/test/ under that fixed path;
the other 29 CLI test directories carry the same marker.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
be01db963f |
fix(tests): read group_vars through the cache in the registry guard
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
328113e7c3 |
refactor(ai): declare the litellm consumer contract once
Fourteen roles repeated the same two lines to say they use the gateway: a
`shared` flag gated on group_names, and an `enabled` flag conjoining that gate
with LITELLM_CHAT_MODEL_SERVED. Both are now single-sourced.
`shared` becomes literal true. The flag means the provider is a shared instance
rather than one deployed per consumer, and a central gateway is always that;
the dynamic form only said "shared if deployed", which `enabled` already
decides. The 50 shared pins under `litellm:` leave meta/variants.yml with it.
The tree had already noticed: 20 of those 50 lines carried
`# nocheck: variants-static-override`, two of them with copy-pasted text about
container_backup. This narrows
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
dac61edbc6 |
feat(ai): let LM Studio serve the models Ollama serves, under one catalogue
svc-ai-lmstudio had no model-load step at all: tasks/00_core.yml set the
once-flag and the container network, nothing else. The gateway published
lmstudio/default anyway, so the route existed and nothing could answer it.
wp-compose-tor.log:48994 shows the route written into the config, :51770 shows
the warm step reaching qwen2.5:0.5b on Ollama instead, and the green ai-engine
spec ran against that. LM Studio was deployed, published and never prompted.
The pre-pull mirrors the one that works. tasks/01_manager_ops.yml resolves the
hosting node and container id, loops the declared models through
`lms get --yes --gguf`, overlaps them with async/poll and reaps before
playwright, exactly like svc-ai-ollama. That `lms` exists is measured, not
assumed: /app/daemon-run.sh:67 in the image starts ./.bundle/lms server start,
and `lms get --help` documents -y for scripting and --gguf for the format.
Synonymity needs both namespaces in one place. Ollama knows qwen2.5:0.5b, LM
Studio knows qwen2.5-0.5b-instruct, so AI_LOCAL_MODELS carries both per entry:
`alias` is what the gateway publishes and what Ollama pulls, `name` is what
`lms get` resolves. Both roles now reference that one list, and config.yaml.j2
publishes each backend under the alias, routing Ollama to ollama/<alias> and LM
Studio to openai/<name>. The lmstudio loop skips an alias Ollama already
publishes, so two backends never produce a duplicate model_name and Ollama
keeps precedence. lmstudio/default is gone; the model name is now the same
whichever backend answers it.
Ollama's preload_models changes shape from a list of strings to the {alias,
name} entries. Every consumer moves with it: the pull command, the template
loop, and both expressions in 16_ai.yml. An inventory that overrides
`applications.svc-ai-ollama.services.ollama.preload_models` with the old string
list will break at map(attribute='alias'); this is a deliberate shape change,
in the spirit of
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
ce8368019d |
perf(constructor): carry the applications SPOT through the merge block
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
4a0fca2bcb |
test(update): build the addon fixture from the role-file constants
test_file_spot failed the gate on the pathlib chain that spelled meta/services.yml out by hand. Both paths now come from utils.roles.mapping as ROLE_FILE_META_SERVICES and ROLE_DIR_META_ADDONS, following |
||
|
kevinveenbirkenbachandClaude Opus 5
|
e4141cb826 |
feat(update): let the ref updater see the addon declarations
collect_entries walked roles/*/meta/services.yml only, so a repository and ref declared in an addon file was invisible to it. That mattered once |
||
|
kevinveenbirkenbachandClaude Opus 5
|
e11c9517b2 |
fix(discourse): set runtime state at runtime, not during bootstrap
All 22 run: hooks in app.yml were rails and rake calls against the database, yet the launcher executes them only while building the image. Two of them carried rotating values, the OIDC client secret and the administrator password, so every rotation forced a full rebuild with bundle install and asset precompile: 961.82s in one pass and 13m16s in the next, on a job that then missed its wall-clock budget by 201s. They now run against the running instance through container exec, following the pattern web-app-openproject already uses: real .rb files under files/ruby, values through ENV.fetch, a readback that prints booleans rather than values, and an assert that fails the deploy when a rotated credential did not arrive. ensure_admin_user.rb mirrors the Y/Y path of admin:create using upstream's own grant_admin!, change_trust_level! and activate; setting the flags by hand had skipped auto_approve_user, the staff welcome message, the notification levels, trust level 1 and the email confirmation. What stays in run: is the CA trust call, which writes into the image and would not survive a recreate. With the rest gone, the bootstrap digest |
||
|
kevinveenbirkenbachandClaude Opus 5
|
5d0c1a442a |
test(agents): take the jinja filters from the registry that ships them
Both template tests listed the filters they hand to jinja by name, which made the FilterModule a second source: adding a filter to it left these two behind, and the templates that used it failed to compile with "No filter named …" in test-unit and test-integration at once. The list is derived from the module now, so a filter reaches the tests the moment the role can use it. The two entries that are not filters at all, to_json and bool, stay spelled out because jinja needs them and the module does not ship them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
04d4d31335 |
fix(test-e2e-playwright): give every Playwright timeout one source
The config held literals for values vars/main.yml already owned, and the two drifted without a sound. actionTimeout carried `|| 30_000` for its whole life while nothing ever set PLAYWRIGHT_ACTION_TIMEOUT, so every run used the unscaled 30 s. expect had no entry at all, leaving Playwright's built-in 5 s to govern every assertion over Tor while its neighbours were scaled by the onion multiplier. That is what killed the second Nextcloud login attempt: an expect(usernameField).toBeVisible() that gave up after 5 s inside a 150 s budget. vars/main.yml now owns the values in TEST_E2E_PLAYWRIGHT_CONFIG_TIMEOUTS, the harness renders that map into the staged .env both the deploy and scripts/tests/e2e/rerun-spec.sh already pass, and the config demands each name through requiredTimeout() instead of inventing a fallback. The two `-e` flags that fed the old pair are gone with it. scripts/lint/playwright.sh stages the same config and runs `playwright test --list`, which loads it, but built its stub env only from each role's playwright.env.j2 and no such template carries a timeout. Without this it would have aborted all 87 specs on a missing variable. It now takes the names from the same map; the value is a placeholder because --list consumes none of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbach
|
c4e9b46519 | Merge branch 'main' into feature/ai-agent-employees | ||
|
kevinveenbirkenbachandClaude Opus 5
|
2d94d1d6f9 |
fix(install): refresh the apt index once before giving up on a package
The Hugo image build in run 34653542922 died on a Debian mirror mid-sync: 'Err:13 libruby ... File has unexpected size (6300780 != 5440). Mirror sync in progress?', and apt closed with its own advice, 'maybe run apt-get update or try with --fix-missing?'. The index the build had fetched disagreed with the pool it then read. Acquire::Retries=3 does not cover that. It re-fetches the same URL against the same stale index and reads the same wrong size, which is why three retries produced three identical failures. install_package_candidates now runs its candidate loop through a helper and, when every candidate failed and the manager is apt-get, fetches a fresh index and runs the loop once more. The helper returns two facts rather than one because the two existing error paths distinguish them: whether the caller's command became callable, and whether anything installed at all. The retry is bounded so it cannot mask a real absence: apt-get only, once, only when nothing installed, and only behind a new index. A package that does not exist fails the second pass the same way and raises the same RuntimeError. The tests pin both directions, the exact update, install, update, install sequence for apt-get and two calls only for a manager without an index, so the retry cannot spread to pacman or dnf. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
7e5c613ddf |
test(matrix): build the vars path from the role-file constant
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
604843bb87 |
fix(wordpress): refuse an MCP request the site cannot attribute to a user
The mu-plugin header describes the split as 'the adapter authorises the transport, the callback authorises the operation'. The second half held, every ability carries infinito_mcp_may_read. The first did not: the adapter answered the initialize handshake to anyone, so the server and its protocol version were readable without a credential. Measured in run 34653754175, the swarm WordPress#0: test-mcp-guest.js posts an anonymous initialize and expects at least 400, and the error context reads 'Expected: >= 400, Received: 200'. Its neighbour addons/mcp-adapter.spec.js posts an anonymous tools/list and passed, so the tool call was already refused and only the handshake was not. The filter hangs on rest_pre_dispatch and refuses every route under the mcp namespace while the site cannot attribute the request. It leaves an already short-circuited response alone rather than overwriting it, scopes itself by route prefix so the rest of the REST API stays as open as it was, and reaches is_user_logged_in, which triggers determine_current_user and with it wp_validate_application_password, so a client presenting its application password passes. A WP_Error renders no jsonrpc key, which is the second thing the spec asserts. The new unit test reads the guarded prefix out of the PHP and the endpoint out of the spec and requires the endpoint to sit under the prefix. Without it an adapter that moves namespace would leave the guard protecting nothing while both negative specs keep passing. A positive spec was tried and reverted: handing the MCP credential to the Playwright env is forbidden by tests/lint/ansible/services/test_mcp_secret_not_in_playwright.py, because the trace travels in the CI artifact. No role probes its MCP endpoint with the credential at deploy time either, so the authenticated direction stays unproven. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
d5fc099166 |
fix(matrix): gate the CA binds on the predicate that provisions them
sys-svc-compose-ca writes the CA material for exactly lookup('ca_injected', application_id), which asks three questions per application: has a domain, TLS enabled, mode self_signed. The lookup exists against this drift and says so in its own docstring: anything that reads the mounted file must ask the same three or it points at a path that was never mounted.
The matrix templates asked the global TLS_MODE instead. On an onion deployment TLS is off per application while TLS_MODE stays self_signed, so run 34653754175 logged 'CA trust N/A for web-app-matrix (TLS off/not self-signed) mark done' and the templates rendered the binds anyway.
The rest follows from the timing in that run: the matrix container was created at 05:49:35, the bundle was first written at 05:58:45 by sys-ca-selfsigned. At create time the bind pointed at a path that did not exist, which docker materialises as a directory, and the later docker restart matrix from 02_render_inventory.yml:22 failed on 'error mounting /etc/infinito.nexus/ca/ca-bundle.crt ... not a directory'.
MATRIX_CA_INJECTED holds the predicate once in vars/main.yml, beside MATRIX_CA_MOUNT_PATH, and both templates read it: the outer container bind and the synapse arguments for the nested daemon. The test pins both links of the chain, that the constant is derived from the lookup and that both if lines use the constant, so redefining it from TLS_MODE cannot pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
da09ee575b |
fix(coredns): answer every record type for the test zone instead of forwarding it
The CI zone served A and AAAA for INFINITO_DOMAIN and let every other type fall through to the public resolvers, which return NXDOMAIN for the reserved .test TLD. Per RFC 2308 that denial covers the whole name, not the queried type, so any caching resolver in front of CoreDNS applies it to the A record the deployment depends on. Without such a cache nobody noticed: Mastodon's EmailMxValidator asks MX, takes the NXDOMAIN, then asks A anyway and gets the address. On a Tor node svc-net-tor puts dnsmasq in front, the denial is cached, and tootctl accounts create ends on 'Failure/Error: email unreachable'. It cost Fediwall#0 and Fediwall#2 in run 34653542922 and Fediwall#2 in run 34653754175, all three compose plus tor; the same two jobs passed in run 34614940508 as compose without tor, same command, same administrator@infinito.test, rc 0. A template with no answer returns NOERROR with an empty answer section and the SOA in authority, so the zone denies a type without denying the name. It sits after the A and AAAA templates, which would otherwise have their names answered NODATA, and before forward, so no type escapes. Measured against 172.30.0.53 in the running stack: infinito.test MX went from NXDOMAIN with an authority record to NOERROR with zero answers and the aa flag, infinito.test A and auth.infinito.test A still return 172.30.0.10, and deb.debian.org still resolves through the forward. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
263c117ff3 |
fix(bluesky): flush the compose handlers in the role that notified them
The compose-up handler resolves its chdir from lookup('container', application_id, 'directories.instance') at flush time. Bluesky passes compose_handlers_flush false to sys-stk-backend and notifies compose-build and compose-up afterwards, so the topics stayed pending until the next flush. That flush belongs to svc-prx-openresty, which the front proxy include pulls in, and it ran them with openresty's application_id.
Run 34653754175 shows the effect: the only compose up in the whole job created the openresty container in 0.52s, no pds container lifecycle line appears anywhere, and the wait for PDS XRPC then retried for 69s against a stack that was never brought up, ending on 'Status code was -1 and not [200]: Request failed: <urlopen error [Errno 111] Connection refused>'.
The flush now sits directly after the rebuild notifier, before the first front proxy include. sys-stk-backend has already rendered the stack there, so this is not a return to
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
6e98ecf66f |
fix(hlth-csp): give onion vhosts a navigation budget Tor can meet
csp-checker navigates every URL with a 20000 ms budget until domcontentloaded. Over the Tor SOCKS proxy that expires before a page of module scripts finishes: run 34614940508 lost Mastodon#1 and OpenProject#1 to it, 26 of 27 attempts logging "Unable to reach http://<onion>/ (Navigation timeout of 20000 ms exceeded)" while sys-service-terminator retried the unit for 35 minutes. No policy was ever read; the checker never got that far. csp-checker 2.3.0 takes --timeout, so script.py takes --onion-timeout and hands it to the onion batch alone, before the -- separator that would make it a URL. Clearnet keeps the image default. The budget is HEALTH_CSP_TIMEOUT_MS (20000, the image default) times the tor flavor's multiplier, 100000 ms on the public network. The unit's TimeoutStartSec had to follow. It was 30 + 25 s per domain -- 130 s for the two onion vhosts Mastodon checks, while a 100 s budget needs up to 200 s, so systemd would have killed the check mid-run. The filter now takes onion_per_domain_seconds and counts .onion domains at that rate; without the argument it computes exactly what it did before, so sys-ctl-hlth-webserver is unchanged. The Mastodon case goes from 130 s to 450 s, a clearnet-only play stays at 130 s. Tests pin that --timeout is appended only when set and stays ahead of the separator, that main budgets the onion batch alone, that the role passes both --onion-timeout and the onion rate for the start budget (both guards fail against the previous tasks file), and the filter's split rate. Not covered: the filter still caps at 3600 s, which binds beyond 34 onion domains in one round. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
ffb04da40a |
refactor(tor): keep the onion timeout multipliers with the flavor they key on
The per-flavor multipliers lived in test-e2e-playwright's vars, where only
that role could read them. The CSP health check needs the same number: it
probes onion vhosts through the Tor proxy, where the checker's navigation
budget has to grow by the same factor as a Playwright wait.
They now sit in svc-net-tor's meta/services.yml next to services.tor.flavor,
the key they are indexed by, and the Playwright var reads them through
lookup('config', 'svc-net-tor', 'services.tor.timeout_multipliers'). That
lookup resolves from the role defaults, so it answers even where svc-net-tor
is not part of the round, which is how the flavor was already read.
The flavor-coverage test moves to the tor role and reads the new location;
the values (chutney 2, public 5) and what the Playwright run computes from
them are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
c32bf320ff |
feat(ci): add a fourth deploy chunk so a short priority chunk strands no budget
A sweep may deploy 210 rows (available) in chunks of 80. Three declared blocks hold exactly that when there is no priority line, but priority rows get a chunk of their own that is never topped up with regular rows. Nine priority rows therefore occupied a whole block, left the regular rows 2 x 80 = 160, and 42 rows of budget went unused. Adding a block to the YAML alone would not have changed that: matrix.chunks_of handed the split slots.chunk_count(), which is min(ceil(available / chunk size), INFINITO_CI_MAX_CHUNKS) and ignores the priority seam, so the plan still stopped at three blocks. call-orchestrator.yml now declares test-deploy-chunk-3, gated on chunks 0 to 2 like every chunk gates on all before it, and report-main-failures and done need it. INFINITO_CI_MAX_CHUNKS is 4, one above what the budget fills, and slots.chunk_blocks() returns it. chunks_of hands the split every declared block; available still caps the rows, and a chunk whose slice comes back empty skips its deploy. chunk_count() stays the "chunks filled" figure of the budget report. Tests: the split spends 9 + 80 + 80 + 41 = 210 rows with a spare block; chunks_of plans three chunks from chunk_blocks=3 while chunk_count=2 (the previous wiring yields [1, 10]); an integration guard requires the declared chunk jobs to equal INFINITO_CI_MAX_CHUNKS and to exceed ceil(available / chunk size) by one, and fails on the stale value 3. A shell that inherited INFINITO_CI_MAX_CHUNKS=3 keeps failing that guard until it is restarted; make dotenv-force rewrites .env. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
a20883600b |
fix(matrix): gate bridge health on liveness and readiness on a deploy task
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
641f7a40dd |
fix(nextcloud): keep the sociallogin uid within 64 chars on onion deploys
Sociallogin creates a Nextcloud user as "<provider name>-<keycloak sub>",
and the provider name was the Keycloak domain. On an onion variant that
domain alone is 67 chars, so every OIDC callback died with
InvalidArgumentException "Username is too long" and HTTP 500 at
/apps/sociallogin/custom_oidc/auth.<onion> (run 34615516009,
Nextcloud#1, 204 occurrences). The whiteboard and admin OIDC specs failed
on the same login.
This is the second derived identifier an onion domain overflows; the
kernel hostname is the first, clamped to 63 bytes by container_hostname
since the pretix and bookwyrm fixes. The domain-or-entity rule now lives in
one lookup, bounded_name(application_id, max_length): the primary domain
when it fits, otherwise the entity name. container_hostname delegates to it
at 63 and keeps its eight callers and its behaviour.
The provider name is bounded_name('web-app-keycloak', 64 - 1 - 36), the uid
limit minus the separator and the UUID sub. A short clearnet domain stays
the name, so existing users keep their uids; an onion deploy gets
"keycloak", which, unlike a hash of the domain, survives a rotated onion
address.
Tests cover the lookup's fit, boundary and fallback cases and its argument
validation, container_hostname's delegation at 63, and the provider name's
room for the sub.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
88218fc740 |
fix(discourse): enable OpenID Connect only after its prerequisites are set
Discourse latest now validates openid_connect_enabled and refuses it until the discovery document, client ID and client secret are set. The bootstrap enabled OIDC first, so a fresh database aborted the launcher rebuild at `rails r "SiteSetting.openid_connect_enabled = true"` with Discourse::InvalidHTMLParameters (run 34615516009, WordPress#0, job 103327034654). Existing databases hid it because the values were already stored. The enable step now follows the three settings it depends on. A test reads the real template and requires every prerequisite line to precede the enable line; against the previous template it fails. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
c7d162fa88 |
fix(coredns): derive the domain regex from INFINITO_DOMAIN when rendering
INFINITO_DOMAIN_RE carried an already escaped regex through the env chain: EnvBuilder.setdefault prefers a value already in the process env, quote_for_dotenv escapes every backslash, and load.sh unescapes it again. One inherited value with a doubled backslash was enough to write "infinito\\\\.test" into .env, the rendered Corefile then matched a literal backslash, and getent hosts test.infinito.test failed with rc 2. The renderer now substitutes DOMAIN_RE = re.escape(INFINITO_DOMAIN) at render time, so no backslash travels through the env or .env at all. The template placeholder is renamed because it is a derived render value, not an env key, and INFINITO_DOMAIN_RE is removed from default.env and the passthrough keys, leaving INFINITO_DOMAIN as the single source. A renderer test renders the real template and requires a single-escaped match with no doubled backslash and no leftover placeholder. Verified live: the Corefile renders ^infinito\.test\.$, test.infinito.test resolves to 172.30.0.10 and test.infinitoXtest does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
128e177255 |
fix(bluesky): notify the compose rebuild only after the backend renders the stack
The pre-compose staging (social-app clone, login-broker copy, pdsadmin
download) ran before sys-stk-backend and notified compose-build and
compose-up. On a fresh host the container runtime's first flush ("Restart
and enable docker service") then ran the compose handlers before
/usr/bin/ca-inject and compose.yml existed, and "Generate CA trust
override" failed (run 34615516009, job 103327034609).
The staging tasks now only register their result. A changed-only notifier
after the sys-stk-backend include triggers compose-build and compose-up, so
no compose topic is pending at any earlier flush, and the handler-only
include that registered the owner early is gone. The ordering test asserts
both halves: the staging files notify no compose topic, and the rebuild
notifier sits after the backend include.
Verified locally in compose, variant 1: the reinstall full cycle passed
pass 1 with failed=0 and the rebuild task changed after the backend
include, pass 2 Playwright passed, and an update run finished failed=0 with
the rebuild task skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
50a13c755b |
test(test-e2e-playwright): read the onion multipliers through the cached loader and the role-file constant
The flavor-coverage test from
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
c2120395e7 |
fix(litellm): let the database follow the inventory and keep the network a dedicated one is reached on
Run 34421944328 web-app-litellm#1 (README host-mode replay) answered
every health probe on llm.ai.infinito.test with 502: litellm
crash-looped on "P1000: Authentication failed against database server
at postgres" while postgres logged 'Role "litellm" does not exist'. The
README inventory has no svc-db-postgres group, so sys-svc-rdbms skipped
the central provisioning, yet svc-ai-litellm pinned postgres.shared to
true and kept consuming a shared database nobody had created its role in.
postgres.shared now reads "{{ 'svc-db-postgres' in group_names }}":
an inventory with the central group keeps the shared database, the
README inventory gets a dedicated one.
That dedicated instance, rendered by sys-svc-rdbms postgres.yml.j2,
joins only the compose default network, and the networks renderer
dropped default from every svc-db-*/svc-ai-* role. A local variant-1
deploy measured litellm-database on litellm_default alone and litellm on
the shared litellm network alone. _suppress_default now keeps default
when the role's database is local. The variants pin shared to true (0)
and false (1) so the matrix exercises both paths, and the generated
READMEs show the postgres relation as 0..1.
The three attachment helpers move from render.py to attachments.py,
which render.py re-exports, and test_render.py splits along the same
seams into five files plus _render_helpers.py with every test id kept.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
293c67780d |
fix(bluesky): register compose handlers before the pre-compose staging and move the cfg heal into a script
Run 34421944328 Bluesky#1 stopped at "The requested handler 'compose-up' was not found in either the main handlers list nor in the listening handlers list", right after cloning the social-app build context. The pre-compose staging (01_social_app.yml, 02_login_broker.yml) and the PDS routine (03_pds.yml) notify compose-up and compose-build, but the handlers only exist once sys-stk-backend loads sys-svc-compose, which happens later in the role. 00_core.yml now includes sys-svc-compose with tasks_from: none.yml and handlers_from: main right after the once flag, the same handler-only registration sys-stk-backend and sys-svc-rdbms dedicated.yml already use. The notifications still run at the stack's own flush, after the compose files exist; the role has no earlier flush. A handler-order test pins the registration ahead of the pre-compose block. The oauth2-proxy cfg heal leaves its inline shell for files/shell/heal_sso_proxy_cfg.sh, which receives the cfg path and the container name through the environment. The blanket "container stop ... || true" becomes a precondition: the container is stopped only when container inspect finds it, so a directory is still removed when the container is gone, and a failing stop of an existing container now fails the task instead of vanishing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
9d0f618e23 |
refactor(sso): compute the app logout target once in the sso lookup
The previous commit wrote the same proxy-aware logout expression into
four javascript.js.j2 templates. The rule it encodes - sign out of
oauth2-proxy first when the app is proxy-gated, else go straight to the
provider - belongs where proxy-gating is already decided.
utils/roles/applications/services/sso.py gains logout_url(url,
is_proxy_gated) next to get_sso_config, with OAUTH2_SIGN_OUT_PATH as the
path constant; the rd value is quoted exactly as Jinja's urlencode
filter does it. The sso lookup exposes it as the logout_url term: it
renders OIDC through the templar, takes is_proxy_gated from the same
resolution, and refuses a context without OIDC. Prometheus, Checkmk,
Baserow and Jitsi now render lookup('sso', application_id, 'logout_url').
Behaviour is unchanged: the resolver tests pin the rd encoding against
jinja2's do_urlencode, and the lookup tests cover a proxy-gated app, an
OIDC-native app, a disabled SSO and a missing OIDC.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
f142f637f5 |
fix(onion): mirror CSS asset origins and emit plaintext onion CSP siblings
Run 34421944328 odoo#0 loaded assets over the node onion that the page could not use. build_csp_header mirrors every clearnet token that carries the primary domain onto the node onion, but kept its scheme. Onion vhosts are served in plaintext, so a sibling like https://cdn.<onion> never matched the http://cdn.<onion> request it was meant to allow. Onion siblings now map https to http and wss to ws; the clearnet token is unchanged. The body filter rewrote external asset origins to the onion mirror only in HTML, so stylesheets kept pulling https://<origin> directly. When an app has tor enabled and external asset origins, the header filter now also flags text/css responses on .onion hosts (dropping Content-Length, since the body changes), and the body filter runs those through the same buffer and mirror rewrite and returns before the HTML injections. The flag is set under the same Jinja condition that renders the mirror block, so a flagged response always has a rewrite to run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
3db5820b6e |
fix(n8n): serve the MCP trigger over Streamable HTTP on 1.100.1 so Open WebUI can connect
Run 34421944328 n8n#1: Open WebUI 0.11.0 ships only a Streamable HTTP
MCP client. n8n 1.95.3's MCP Server Trigger speaks SSE only and answered
the client's POST to /webhook/infinito-mcp/sse with 404, so "every
registered MCP server answers Open WebUI's client" failed.
n8n moves to 1.100.1 and the managed trigger to typeVersion 2, which
serves Streamable HTTP at /mcp/<path>. meta/mcp.yml declares
transport: streamable_http at /mcp/infinito-mcp (the trigger path stays
infinito-mcp) and points source_url at the 1.100.1 tree. Open WebUI's
supported_transports drops sse, which its client never spoke; no
provider declares sse any more, so no pairing is lost.
n8n#0: the guest spec probed ${APP_BASE_URL}${MCP_ENDPOINT_PATH} and
expected a refusal of >= 300 without an event stream. The trigger is
exposure: internal, so the public vhost must answer 404; the spec now
asserts exactly that through mcpEndpointUrl(..., N8N_BASE_URL), and the
bearer guard itself is proven in-cluster by the CLI MCP contract.
Requirement 035 and the MCP audit table follow the version and
transport.
Upgrade path, read from upstream at both tags: the postgres migrations
are identical (45/45) and the only new common migration is
AddProjectDescriptionColumn, an additive nullable column; BREAKING-
CHANGES.md lists only 1.98.0 in range (route metric format, Form node
tag attributes, Node >= 20), none of which this deployment uses. A fresh
1.100.1 install ran all 89 migrations and passed the MCP bridge, server,
non-member and guest specs locally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
304144ce67 |
fix(matrix): give each bridge declared tokens and a rendered registration, drop the swarm service removal
Every mautrix bridge generated its own registration with -g on start, with fresh random tokens. Synapse had to read that file from a shared directory, be restarted afterwards, and on swarm the services had to be removed first so the registrations could be wiped ( |
||
|
kevinveenbirkenbachandClaude Opus 5
|
a650801b35 |
fix(lookups): name file-backed extra secrets by content and hand plain types to PyYAML
compose_volumes copied extra_configs and extra_secrets through verbatim.
A file-backed extra without a name therefore reached swarm under a fixed
name, and swarm secrets are immutable: a new content under the same name
is rejected on the next stack deploy. Such extras now get the same
{role}_{key}_{sha8(content)} name that _config_secret_name already gives
meta-declared entries. An explicit name is kept.
container_volumes appended extras as handed over. Ansible passes one
lookup's result to another as proxy containers, and a TaggedInt mode
inside a proxy mapping made PyYAML raise "cannot represent an object".
The private _to_plain of compose_volumes moves to utils/templating as
to_plain, now coercing bool, int and float to the exact builtin types
(bool before int, since bool subclasses int), and container_volumes runs
its extras through it.
Only web-app-matrix passes extra secrets or configs today; the
extra_volumes path that web-app-nextcloud uses is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
214e7e29d1 |
fix(test-e2e-playwright): scale onion timeouts by the tor flavor instead of a flat five
resolveTimeout multiplies every Playwright window by 5 for a .onion target. That default arrived in |
||
|
kevinveenbirkenbachandClaude Opus 5
|
60b9006506 |
test(mcp): fail the contract on a JSON-RPC error inside an HTTP 200
Over streamable_http the contract probe judged initialize, tools/list and the read call by HTTP status alone. A server that answers 200 with a JSON-RPC `error` object therefore passed initialize and the read call, and failed tools/list only as "server serves []", which names the wrong cause. The SSE path checked initialize and the read call, but not tools/list. mcp-adapter, which fronts WordPress, maps invalid params to 200, and Moodle's webservice_mcp answers a failing tool call with 200 and code -32603, so a read that returned nothing could turn the check green. rpc_error() extracts the error object of a JSON-RPC body, refused() is built on it, and reject_error() fails initialize, tools/list and the read call on both transports with the server's own message. A 200 whose body is not JSON still carries no error object, so a provider that has not started keeps the caller retrying as before. The streamable_http initialize, and the one in the negative probes, sent no clientInfo, which the MCP SDK requires. Home Assistant (mcp 1.26.0, stateless) answers such an initialize with 200 and -32602 and serves the rest anyway, so only the old status check let it pass. All three initialize calls now share one INITIALIZE with clientInfo, as the SSE path already sent. Against a Home Assistant-shaped endpoint on mcp 1.26.0 the SDK logs "InitializeRequest.params. clientInfo Field required" for the old probe, and the fixed one prints OK. Against HEAD, an error-on-200 at initialize and at the read call printed OK; now all three steps print REJECTED naming the error. The MCP adapter in front of sixteen of the nineteen streamable_http providers answers 200 only with a result (svc-ai-mcp-adapter/files/python/server.py), so none of them changes outcome. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
98707e06f4 |
test(matrix): hold the CA passthrough guard to the repo's own conventions
The guard landed in
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
2acf91e186 |
fix(matrix): pass the CA bundle into the container synapse binds it from
The mdad stack runs its own Docker daemon inside the matrix container, so a
bind source in `matrix_synapse_container_extra_arguments` is resolved in that
container's mount namespace, not on the host. `vars.yml.j2` mounts
`CA_TRUST.bundle_host` while `services.yml.j2` passed only
`CA_TRUST.cert_host`, and those are two different files: the root CA and the
bundle beside it. The bundle therefore never existed where synapse looked.
`docker create` refuses the unit before Matrix logs anything of its own. From
the rescue artifact of job 102465251843 in run 34348821577, the nested
journal at containers/matrix/nested/.../journal.txt:5018:
matrix-synapse[5612]: Error response from daemon: invalid mount config for
type "bind": bind source path does not exist:
/etc/infinito.nexus/ca/ca-bundle.crt
Five start attempts, the same line each time, and the only error in the 872 KB
journal. matrix-heisenbridge follows it down as a dependency, `systemctl
restart` returns rc=1, the mdad bootstrap fails and the rescue task ends the
play. Not an OOM (exit 1, not 137, `OOMKilled: false`, 9.7 GB free), not disk
pressure, not a failed pull, and 16 % of the job budget spent.
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
8434a31611 |
fix(ci): answer a role-only selection by existence, not by variant None
Rows are indexed by `(name, variant)` and a variant is an int. A token that
names only a role carries no variant, so the lookup ran with `None` and could
never hit, and every such token was reported as matching no discovered row.
That is the form `collapse_to_roles` produces, so it is the form a retriggered
priority line arrives in. Run 34348821577 shows the result: all twenty-one
priority entries carried
priority: '<role>' matches no discovered row
(no row in this run's mode and lifecycle envelope)
while baserow, checkmk, decidim and the rest went on to deploy fine. The
warning was not wrong about one entry and right about twenty; it was blind. A
role that genuinely has no row in the envelope looked exactly like a role with a
full set, which is how priority work goes missing without anyone being able to
see which entry did.
A token that narrows nothing beyond the role name is now answered by existence:
it asks for the role in whatever axes the rotation picks, so any discovered row
satisfies it. `Pin.pinned` already draws that line - "whether the token narrows
anything at all beyond the role name" - and pinned tokens keep the old path,
`check_pins` included.
Three tests come with it. There was already one for a bare role that matches
nothing, and none for a bare role that matches something, which is why the
lookup could stay broken. One of the new ones uses variant 4 so a future
implementation cannot pass by quietly guessing variant 0. Checked against HEAD:
the same input returns the spurious warning there, so the tests fail without
this change.
This restores the signal, not the lost jobs. The other silent path, `priority`
being truncated to the job budget in cli/meta/ci/chunks.py, is untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
96408f23a0 |
perf(healthcheck): let the start window outlast a real boot
start_interval is the cadence while start_period runs, and only while it runs. Once the window closes the probe falls back to interval, so a service that becomes ready just after it pays a whole tick anyway and the cadence never reached it. Six of the seven flavors were shaped that way, Curl worst: a thirty second window against a one minute interval. Measured across 151 converge waits in four job logs, 63 sit exactly on a tick boundary, 36 in the thirty second step and 27 in the sixty second one. Attributed by application, web-app-seaweedfs is the largest at 1501s over sixteen waits with a median of 57.7s: it probes with curl and fell straight into the gap. web-app-zammad probes with connect, whose window is fifteen minutes, and its 154s median is real boot time with nothing to recover. So the window moves to ten minutes and the steady state interval to two, and the flavor overrides that sat below them go: Curl, Wget and Connect lose interval 1m, Http its twenty second window, Nc its ten. Connect keeps fifteen minutes. Neither number costs anything while a container works. start_period delays nothing, it only stops failures counting, and interval no longer touches convergence at all since that hangs on the window alone; it is steady state load, which drops from 120 probes an hour per service to 30. Forty six role files follow. Twenty seven declarations go that repeat the new default or undercut it, an undercut window being the defect itself. Eight windows longer than ten minutes keep their value and now say why. The literal blocks and the three Dockerfiles this repository builds spell every timing out, because outside the lookup an omission takes docker's value and not the platform's. A fourth lint rule keeps the shape: start_period must outlast one interval, checked against the effective values, so raising interval without the window cannot quietly bring the cliff back. Two faults in the lint surfaced while building it and are fixed here. It compared timings as strings, so 600s and 10m read as different values, and it matched the healthcheck key exactly, so a line carrying a trailing nocheck was skipped whole, which is how the two swarm-init partials went unexamined. One regression, bounded. A container that starts and never turns healthy is neither rejected nor errored, so stack_ready.sh only calls it fatal after three killed tasks, and each cycle now spans the longer window. That reaches twenty minutes rather than six, capped there by swarm_stack_converge_retries at 600 with a two second delay, and it lands only on runs that were failing anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
5324eadd8b |
fix(ai): resolve the gateway backend from the inventory, not from one host
LITELLM_BACKEND_ROLES read lookup('deployment').groups, which deployment.py
fills from group_names. That variable is host-scoped: it lists only the groups
the host currently being processed belongs to.
On a swarm round the manager carries svc-ai-ollama, svc-ai-lmstudio and
svc-ai-litellm while the workers carry three groups and none of them is an AI
backend. Both branches of LITELLM_CHAT_MODEL therefore missed on a worker and,
with OPENROUTER_API_KEY empty, the model resolved to the empty string.
Only one task made that visible. Every stateful task in the gateway include
carries delegate_to STACK_HOST plus run_once, so it templates in the manager's
scope and wrote the correct model into the database. The assert carries
neither, so it re-templated in each worker's own scope and failed there while
the manager passed.
The manager-only membership is not a placement decision. The swarm test
inventory is provisioned with a single --host MGR, so every role of the round's
closure gets that one host, and extend_inventory then widens only
svc-swarm-node, svc-storage-nfs-client, svc-storage-nfs-server and the round's
own application. The three AI roles declare no placement at all; they are
manager-only because nothing widened them.
Add an inventory selector next to the host-scoped one, built from the play-wide
groups magic var, skipping groups without hosts and Ansible's implicit all and
ungrouped, sorted so the cache key is stable. The key takes it too, or two
hosts with equal group_names but different inventories would share an entry.
The blast radius is two readers, both in 16_ai.yml, and both ask only whether
svc-ai-ollama or svc-ai-lmstudio is present. On the manager the new value gains
exactly svc-storage-nfs-server and svc-bkp-nfs-2-local, neither of which either
reader tests, so the manager's answer does not move and only the workers change,
to match it.
This is the third step on the same line.
|