Run 34421944328 web-app-litellm#1 (README host-mode replay) answered
every health probe on llm.ai.infinito.test with 502: litellm
crash-looped on "P1000: Authentication failed against database server
at postgres" while postgres logged 'Role "litellm" does not exist'. The
README inventory has no svc-db-postgres group, so sys-svc-rdbms skipped
the central provisioning, yet svc-ai-litellm pinned postgres.shared to
true and kept consuming a shared database nobody had created its role in.
postgres.shared now reads "{{ 'svc-db-postgres' in group_names }}":
an inventory with the central group keeps the shared database, the
README inventory gets a dedicated one.
That dedicated instance, rendered by sys-svc-rdbms postgres.yml.j2,
joins only the compose default network, and the networks renderer
dropped default from every svc-db-*/svc-ai-* role. A local variant-1
deploy measured litellm-database on litellm_default alone and litellm on
the shared litellm network alone. _suppress_default now keeps default
when the role's database is local. The variants pin shared to true (0)
and false (1) so the matrix exercises both paths, and the generated
READMEs show the postgres relation as 0..1.
The three attachment helpers move from render.py to attachments.py,
which render.py re-exports, and test_render.py splits along the same
seams into five files plus _render_helpers.py with every test id kept.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
allowed_consumers was twenty hand-kept lists of which fourteen were identical, so every new client meant an edit in every provider. A client now declares services.<own-entity>.mcp_consumer: true once and a provider refuses one with mcp_consumer: false, which is also what tells a deliberate exclusion apart from a forgotten entry.
The group_names-gated consumer entries give the inventory resolver the co-deploy edge meta/mcp.yml never had: one of sixty-nine matrix rounds carried an MCP client before, twenty-three role/round pairs do now. bond 0.5 keeps the clients out of round 0 so the host memory budget still holds.
BREAKING CHANGE: meta/mcp.yml no longer accepts allowed_consumers; declare services.<entity>.mcp_consumer on the client instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both renderers build their output as a list of lines seeded with the "networks:"
header and append one line per attachment. With no attachment to append they
still returned the bare header, and a mapping key with no children parses as
null, not as an empty mapping. Compose then rejects the whole file instead of
ignoring an empty section.
The two functions do not share their non-empty paths and disagree on where the
newline goes -- render_compose_networks terminates with one, render_container_networks
leads with one -- so the guard is written out in each rather than factored into a
helper that would have to know both shapes.
Both callers already treat an empty string as "insert nothing", so returning it
needs no change on their side.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
libcurl refuses a .onion host by name (RFC 7686, error 6) before it
consults /etc/hosts, extra_hosts or DNS, so the platform's DNS+iptables
onion egress cannot serve a server-side backchannel at all. Measured:
curl --noproxy '*' -> exit 6
curl --resolve auth.<onion>:80:127.0.0.1 … -> exit 6
--resolve fills curl's own cache strictly earlier than getaddrinfo and
is still refused, which makes sys-svc-container's onion_oidc_socks
template -- extra_hosts, despite its name -- a no-op for every libcurl
client. Only a configured proxy makes curl skip the check.
Since 9d91fbea8 emptied proxy.config.php.j2 there is no such proxy, while
svc-net-tor's exclusive/primary defaults replace Keycloak's clearnet
domain and KC_HOSTNAME pins the issuer to the onion. Nextcloud's
oidc_login discovery call therefore dies inside PHP, /apps/oidc_login/oidc
returns 500, and oidc_login_auto_redirect sends /login straight there.
Restoring the old socks5h://host.docker.internal:9050 alone would make it
worse: tor runs as a single manager-pinned replica publishing SOCKS with
mode: host, so only the replica on that node could connect and a
deterministic failure would become a 2-in-3 flake. In swarm, though, tor
is NOT host-networked -- that branch is compose-only -- and it already
binds SocksPort 0.0.0.0, so the shared overlay is a usable path from
every node.
Give svc-net-tor a shared_net overlay with alias 'tor' and dial it by
service name. Attachment is gated by a new consumer kind, onion_sso: the
consumer's services.sso.enabled AND the provider's services.tor.enabled.
Both are group-membership derived, so the conjunction is exactly "the
issuer is an onion" and stays false in clearnet deployments -- which
matters because the service registry is not filtered to deployed roles.
The existing kinds could not express it: keying on sso alone would also
hold without tor, and keying on tor would miss Nextcloud's variant 2,
which disables its own onion yet still needs the client path.
tor_socks_proxy keeps the swarm/compose split in one place so the six
roles sharing this defect need not restate it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Being an MCP client was enough to join a provider's overlay, so a single
admission anywhere reached every provider that opened one. Attachment now
requires both that the role speaks as a client and that the provider
lists it in mcp.allowed_consumers.
consumer.kind accepts a list so one overlay can carry both its
service-flag consumers and its MCP clients, which is what the vector
store needs.
The block sat under services but never behaved like a service. A service has
exactly one providing role, declared with provides, so the registry can map the
key to that role and consumers bind their flag to its presence. No role declares
provides: mcp. There are seventeen providers and the pairing is done by
allowed_consumers inside the block itself.
Every rule that enforces the service model therefore needed an exemption for it:
a shared-provider suppression in fourteen roles, a playwright-service-flag
suppression, and two freshly added exemptions in the dynamic-flag and
variants-static-override checks. Four independent rules, four escape hatches,
one cause. They are all gone now, together with a bond key that only the
resource budgeter reads and only for entries under services, and a wordpress
addon bridge that pointed at a service block that no longer exists.
The block moves to meta/mcp.yml. The applications loader maps meta/<name>.yml to
applications.<app>.<name> generically, so the move needs no wiring; the file
takes the same shape as meta/rbac.yml. Every config path follows, the variant
overrides are lifted out of their services map, and the vocabulary module leaves
the services package because it no longer describes one. roles_with_service
gains an explicit topic parameter rather than an implicit fallback, so nothing
silently resolves against the wrong container.
Files that are otherwise unrelated to this move are included where the committed
tree would otherwise keep pointing at a path that no longer resolves: thirteen
vars files, eleven playwright env templates and one task file.
A provider overlay keyed on services.mcp.enabled and .shared admitted every
MCP-capable role, servers included, because those flags carry no direction. Each
provider's network therefore collected every other provider and the result was a
mesh in which any sidecar could reach any application. The new mcp_client
consumer kind additionally requires services.mcp.direction to be client or both,
and the four provider overlays switch to it.
A sidecar rendered its networks with the same lookup as its provider and
inherited that provider's consumer attachments, which it neither needs nor
should have. own_network_only keeps it on the provider network alone. A provider
whose own topology is default_net contributes no attachment line at all, and a
bare networks: key is not a list, so the flag returns nothing in that case and
lets the sidecar fall back to the project default, which is that provider's own
network.
Leftovers from extracting lookup_context and attachments: the mock.patch target
got short enough to fit on one line again, and the file split left a spare blank
line in each module. Whitespace only, produced by autoformat during make
quality, so the 298 tests of that run already covered this state.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two places decided independently what the pre-created <entity> network is called
inside the compose project. render.py emits the key; create.yml asserted it was
always `default`. They diverged in July and nobody noticed, because compose
5.3.1 resolved networks by name and never read the claim.
Compose 5.4.0, published 2026-08-03T14:39:12Z, resolves them by label. For
web-app-seaweedfs - the one deployed role that provides its own shared_net and
therefore renders an unnamed `default` - the mislabelled `seaweedfs` network is
adopted as the project default, `seaweedfs_default` is never created, and the
containers fail to start against it. That failed web-app-nextcloud 0 and 3 in
run 30830377785.
_shared_network_key() is now the single derivation. render_compose_networks
takes its branch from it, so the rendering decision IS the key decision;
shared_network_compose_key() exports the same value, the shared_network_key
lookup hands it to Ansible, and create.yml stamps it: `seaweedfs` for a
self-provider, `default` for everyone else. Both true, both from the source that
renders the file.
Verified on one host with only the compose binary swapped:
5.4.0 network=default -> FAIL, network <project>_default not found
5.4.0 network=<entity> -> OK
5.3.1 network=default -> OK
5.3.1 network=<entity> -> OK
so this is not a fix that trades one version for the other.
test_key_matches_the_key_the_renderer_emits renders the block and asserts the
reported key appears in it, for both role shapes and both modes. The drift that
sat unnoticed for a month now breaks a test.
The move of the attachment computation into utils/networks/attachments.py rides
along because render.py would otherwise be broken between two commits: it is a
pure relocation, re-exported so no importer changes, and it brings the file from
352 to 243 lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Workers failed 'docker build: no Dockerfile': the build context, compose file, .env and runner binary were staged on the stack host only while every node deploys its own node-local runner stack. Build and push the image once on the manager (RUNNER_IMAGE_REF via the swarm registry, 04_image.yml) and let workers pull it; render compose.yml (template with lstrip_blocks, matching the replaced stack_host_template action), .env and the extracted run.sh on every node. Add a node_local kwarg to the compose_networks/container_networks/compose_restart/compose_only lookups so this node-local compose stack renders a bridge default network (workers cannot create overlays), keeps restart: unless-stopped (ephemeral runners died after one job) and keeps container_name under swarm; default is unchanged for every other role. Treat build/install.stamp as stale when the venv is missing and exclude ./build from the nested-src copy, closing the nested-e2e ruamel failure class. Regression tests for the node_local render paths; the network include tests now accept lookup kwargs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Roles whose get_entity_name(application_id) resolves to an empty string (e.g. svc-runner) rendered 'name: ' with no value in the compose networks block, producing an invalid/empty network definition. Guard both the swarm overlay and the compose bridge emissions so the name line is only written when own_entity is non-empty; docker then auto-names the network. Add regression tests for both modes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Active Storage signs URLs against AWS_ENDPOINT, so it must be the
browser-reachable public endpoint instead of the internal service url.
Compose reaches the canonical s3 api domain via extra_hosts and the
host gateway; swarm resolves it through a new proxy_aliases beacon
field that puts only the canonical domain on the front proxy, keeping
the ambiguous seaweedfs entity alias off shared overlays.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
compute_external_network_roles() + compose_external_networks lookup expose which foreign provider networks a compose project references as external, so the swarm deploy path can pre-create them. network/create.yml probes for an existing network instead of unconditional state:present (the module's delete+recreate on attached networks crashed redeploys) and hard-fails on driver drift instead of silently skipping it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extract the csp, domains, and networks blocks out of roles/*/meta/server.yml
into dedicated meta files, each mapping to its own top-level applications key
via the recursive meta-walk:
csp: -> meta/csp.yml -> applications.<app>.csp
domains: -> meta/domains.yml -> applications.<app>.domains
networks: -> meta/networks.yml -> applications.<app>.networks
Residual nginx/proxy keys (status_codes, client_max_body_size, locations,
ip6_enabled, config_upstream_url) stay in server.yml; an emptied server.yml is
deleted. 98 server.yml -> 249 new files, 74 deleted.
- Consumers repointed applications.<app>.server.{csp,domains,networks} ->
applications.<app>.{csp,domains,networks} across templates, vars, tasks,
lookups and filters.
- Infra readers migrated: canonical_domains_map, application_domain_index,
build_ldap_role_entries, service-registry overlay discovery,
roles_with_service, and the file-based readers utils/domains/list.py +
utils/meta/scan.py (read the now-deleted server.yml directly).
- Detection + mapping: _META_TOPICS gains csp/domains/networks;
ROLE_FILE_META_{CSP,DOMAINS,NETWORKS} constants + ROLE_FILES entries.
- New self-expiring lint guard (test_server_meta_no_split_fields) fails if a
server.yml still carries a split field; auto-fails from 2026-12-28 as a
delete reminder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Foundation for the central svc-db-*/svc-dns-* engine roles: the engine lookup and engines descriptor, per-consumer engine credentials in the inventory manager, shared-net overlay rendering, container_depends_on, the sys-svc-engine provisioning dispatcher, and the compose_volumes NFS-placement split. base.yml.j2 gates the embedded redis sidecar on lookup('engine','redis',application_id,'local').
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The net refactor (366dc7b70/cb27ff375) migrated every shared-net provider (postgres/mariadb/openldap/ollama/openresty/keycloak) to the overlay registry but missed seaweedfs and minio, so an object-store consumer like baserow never joined the provider's docker network and its S3 client hit ECONNREFUSED on seaweedfs:8333 (the intended direct data-plane path).
Add networks.overlay to both engines so consumers attach, and complete the multi-service shared-net-provider contract: (1) render.py emits a bare auto-IPAM project default for an own shared_net provider (no name, no subnet) so the entity-named external net created by network/create.yml is not re-requested -- this caused 'invalid pool request: Pool overlaps' on the first attempt. (2) a provider_self_alias flag lets the provider's sidecars (proxy/sso-proxy) JOIN the shared net WITHOUT publishing the <entity> alias, so the name resolves only to the main container and baserow's :8333 never round-robins onto a port-less sidecar. (3) seaweedfs/minio 01_core pre-create the external <entity> net via network/create.yml (mirroring web-app-prometheus) so compose adopts it as the project default instead of building <entity>_default.
Verified: baserow compose e2e test-seaweedfs.js passes (upload stored in the bucket, 3.9s); new TestOwnSharedNetProvider locks the contract; full make test green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the registry-driven overlay renderer so the central openresty proxy
reaches every web-facing backend over a shared swarm overlay:
- Add consumer kind `web_facing` (matches web-app-*/web-svc-*) so every
web-facing app attaches its primary service to the openresty overlay in
swarm mode; openresty resolves them by swarm service DNS.
- openresty's overlay declares `collect_proxy_resolvable`, harvesting the
keycloak beacon's `auth.<domain>` alias onto its own attachment so each
per-app sso-proxy resolves the auth host to openresty.
- Fix the default-net name collision: a shared_net provider rendering its own
compose must not also emit `name:` on its default overlay (the external net
already carries that name), which made `docker stack deploy` fail with
"network <x> already exists".
Runtime-confirmed: openresty reached keycloak over the overlay (302, no more
connection-refused) once consumers attached.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Convert the two registry-driven networks templates into pure-Python
rendering functions plus thin Ansible lookup plugins:
roles/sys-svc-compose/templates/networks.yml.j2 -> {{ lookup('compose_networks') }}
roles/sys-svc-container/templates/networks.yml.j2 -> {{ lookup('container_networks') | indent(4) }}
The rendering logic lives in utils/networks/render.py as pure-Python
functions that take a registry + closure-style config/database lookups,
making the logic directly unit-testable without a Jinja stub harness.
The lookup plugins are thin wrappers that resolve application_id,
DEPLOYMENT_MODE, and swarm.network.encryption from the templar, build
the service_registry, and forward to the render functions.
Migrate 233 include lines across 85 production templates. Convert
web-app-matrix's hand-coded MDAD networks block (default + conditional
postgres + conditional openldap) to the generic lookup; this adds an
openresty attachment when services.sso.enabled (harmless extra
interface, pinned in test_matrix_mdad_consumer_attaches_postgres_ldap_sso).
Mark local-DB / redis sidecars (sys-svc-rdbms/templates/services/,
svc-db-redis/templates/service.yml.j2) with `# nocheck: networks-literal`
because they intentionally attach to default-only.
Extend the lint to scan ALL roles/*/templates/**/*.yml.j2 (not only
compose.yml.j2) and to forbid the legacy include form; the marker
above is the documented escape hatch.
Coverage:
- tests/unit/utils/networks/test_render.py (30 cases incl. encrypted=false literal pin)
- tests/unit/plugins/lookup/test_compose_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/unit/plugins/lookup/test_container_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/integration/infrastructure/compose/test_networks_render.py (10 snapshot scenarios)
- tests/integration/infrastructure/compose/test_network_includes.py (3 lint rules)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
