The ranking the last commits built toward is now enforced: a semver tag,
else a digest, and nothing else without a written reason. Two states fail.
A pin that is neither names something the registry can move underneath us,
and nothing here can tell whether it still points where it did when it was
written. A digest on an upstream that does publish semver takes the
fallback for no gain and hides the entry from the update job.
Thirteen entries stood in the way when the check was first run. Four were
resolved by the channel grammar, which made the Jitsi components orderable
without touching their pins. Three needed nothing once a moving tag could
declare itself with 'docker-version'. Two are fixed here and four are
declared.
LiteLLM leaves 'main-v1.77.3.dynamic_rates'. That tag exists exactly once
among 2609, so it is a dead end no updater can follow, and it misreports
itself: the image inside answers 1.77.2. The pin moves to
'main-v1.77.7-stable', five patches along the same minor and onto a channel
with 134 builds. Started against this repository's config shape, 1.77.7
registers both model entry forms and accepts drop_params and the
os.environ master key.
Pixelfed leaves 'latest' for '20260917'. The upstream stamps its dates as
one eight-digit number, which is depth 1 and orderable with no tooling
change at all, and the two tags resolve to the same digest, so nothing
about the deployed image moves.
The four declarations carry the count that justifies them rather than a
claim: LM Studio publishes two tags and neither is a version, the MIG
registry carries one, and the two Nextcloud Talk images publish 174 and 104
AIO build stamps between them.
Both branches of the gate were negative-controlled. The digest branch
skips in this repository because no digest pin exists at all, so it was
exercised by introducing one and watching it fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Variant 0 pinned both local backends on, so LITELLM_CHAT_MODEL always resolved
to an Ollama model and every completion went there. The LM Studio route was
published and never travelled. A third variant turns Ollama off and LM Studio
on, which is the only shape that exercises it, and variant 0 pins LM Studio
literally off so a WordPress round stops co-deploying a backend it never
prompts: 27 apps become 26.
bond drops to 0.5 and that is what makes the variant legal. test_order's
_direct_deps filters services on bond >= 1.0 before building a closure, so at
0.5 LM Studio leaves that computation and the new variant no longer violates
"variant 0 is the maximal deploy footprint". The deploy planner does not filter
on bond — resolve_service_dependency_roles_from_config reads enabled and shared
only — so the variant still deploys it. test_bond_baseline takes its bond < 1
branch, which asks for any variant pinning the flag true rather than variant 0
specifically, and variant 2 supplies it. No suppression and no guard change.
bond < 1 states that the provider is a separate-host partner rather than a
same-host service. Reaching the gateway through LMSTUDIO_BASE_LOCAL_URL, a
container-network hostname, is consistent with that.
Measured against the planner: variant 0 resolves 21 apps without LM Studio,
variant 1 stays at 1, variant 2 resolves 8 apps with LM Studio and without
Ollama, and a WordPress round 0 drops from 27 to 26.
The variant is reachable only when svc-ai-litellm is swept as a primary row and
pinned to it: legacy_resolver clamps a non-primary role to the round index, and
a two-round WordPress job never reaches index 2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34421944328 web-app-litellm#1 (README host-mode replay) answered
every health probe on llm.ai.infinito.test with 502: litellm
crash-looped on "P1000: Authentication failed against database server
at postgres" while postgres logged 'Role "litellm" does not exist'. The
README inventory has no svc-db-postgres group, so sys-svc-rdbms skipped
the central provisioning, yet svc-ai-litellm pinned postgres.shared to
true and kept consuming a shared database nobody had created its role in.
postgres.shared now reads "{{ 'svc-db-postgres' in group_names }}":
an inventory with the central group keeps the shared database, the
README inventory gets a dedicated one.
That dedicated instance, rendered by sys-svc-rdbms postgres.yml.j2,
joins only the compose default network, and the networks renderer
dropped default from every svc-db-*/svc-ai-* role. A local variant-1
deploy measured litellm-database on litellm_default alone and litellm on
the shared litellm network alone. _suppress_default now keeps default
when the role's database is local. The variants pin shared to true (0)
and false (1) so the matrix exercises both paths, and the generated
READMEs show the postgres relation as 0..1.
The three attachment helpers move from render.py to attachments.py,
which render.py re-exports, and test_render.py splits along the same
seams into five files plus _render_helpers.py with every test id kept.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
13f5ca6d95 put upstream_timeout above every consumer budget on the reasoning that the gateway should only fire once each caller had already given up. That enumerated the consumers and missed a proxy inside one of them: Zammad ships its own nginx with a fixed proxy_read_timeout of 300 in location /, and the role mounts no override. At 420 that hop always won, so a wedged backend would reach the caller as an opaque HTML 504 rather than a gateway error carrying a message.
240 sits below the tightest hop in the path. The front proxy allows 900s and the only tighter timeouts in the repository are the 3s and 5s of the Prometheus healthz routes, which are not on the AI path. Measured on qwen2.5:0.5b, the model the matrix serves: a generation bounded at max_output_tokens 512 finishes in under five seconds, so no legitimate answer under the configured cap can approach the new ceiling.
This does NOT fix Zammad#2 of run 33923190220, and an earlier draft of this message claimed it would. The rescue artifacts say otherwise: ollama logged no generation at all during the three attempts at 09:00, 09:06 and 09:11, litellm had been up since 08:28:46 with zero restarts, and each attempt hung for exactly 300 seconds before rails raised AI::Provider::ResponseError from custom_open_ai.rb:40. The gateway never saw the request, so no gateway timeout can act on it. The same path had worked minutes earlier, warm-up 2.69s and the deploy's provider ping 10.84s from inside the rails container. Rails and litellm share the litellm overlay but sit on different swarm nodes, so that failure belongs to cross-node overlay connectivity, not to this setting.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway registered ollama and lmstudio with an rpm cap and nothing
else, so a single request could generate without an output limit and hold
a worker for as long as the backend kept producing tokens. Requests per
minute bound how often a caller may ask, not how much one answer costs.
max_output_tokens caps the answer itself. upstream_timeout is the backstop
for a wedged backend, not a latency budget, so it sits above every
consumer's own budget: the longest wait on the AI path is 360s
(discourse-ai.spec.js), with 300s and 240s behind it, and TIMEOUT_FACTOR
defaults to 1. At 420s the gateway can only fire once every caller has
already given up, which keeps a slow generation from being reported as a
gateway error instead of the slow answer it is.
openrouter/auto stays unbounded here. It never carried an rpm cap either,
and its limits are the provider's rather than ours, so giving it one is a
separate decision from closing the local-backend hole.
Consumers of the model plane speak one OpenAI-compatible endpoint and
carry one virtual key; which backend answers is the gateway's business.
svc-ai-litellm renders its model_list from the round's actual topology,
so an alias only exists when its backend role is in group_names, and it
emits an explicit empty list when no backend is present at all because
LiteLLM rejects a bare model_list: key.
Keys reach the gateway through the per-consumer credential rather than
being minted there: 01_keys.yml probes /key/info and only POSTs
/key/generate on 404, so reruns stay idempotent. The consumer list is
built in 00_core.yml, not inside the included file, because 01_keys.yml
delegates to the hosting node and delegate_to is templated before when.
The gateway declares ollama and lmstudio as consumer flags so the
network renderer attaches it to its own backends; without them it cannot
reach them.
svc-ai-lmstudio runs the headless llmster daemon behind expose only, so
it is reachable from the gateway but never bound on the host, matching
svc-ai-ollama.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5