Adds a headless LibreTranslate role for callers that reach the engine over
the container network: no vhost, no reverse proxy, no MCP adapter.
The role is named `-engine` rather than `svc-ai-libretranslate` because
`get_entity_name` strips the category prefix, so that name and
`web-svc-libretranslate` both reduce to `libretranslate`. The entity name
keys the instance directory, the compose project and the container name, so
the two roles overwrote each other's rendered stack and `sys-service-loader`
resolved the engine's own `libretranslate` service key to the web role.
`provides:` cannot fix this: it renames the registry key, not the entity.
`loading_model` selects when models arrive: `lazy` (default) on first
request, `build` into the image, `task` by a post-deploy install. Under
`lazy` and `task` the models sit on a node-local bind so a rebuild reuses
them; `build` is mounted without it, because a bind is never seeded from an
image and would hide what the build baked.
Two lints guard the class:
- entity names must be distinct among roles that render a compose stack,
which `provides:` cannot satisfy and only a rename can
- a role that reaches a cached upstream another role declares must declare
it too, so the entry is discoverable from the fetch
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A container cap had to be an absolute number, and docker refuses one above
the host's core count, so "use this machine" could only be written as a
number that breaks on a smaller machine. cpus now also takes a percentage,
read against the host's raw count: 100% is every CPU and lands exactly on the
ceiling docker accepts, 50% is half of it.
The share is read against that raw count rather than against the fair-share
default, because the two CPUs the resource lookup reserves exist to bound
what a service gets when it declares nothing. A service writing a percentage
overrides that default on purpose, and subtracting the reserve a second time
would count it twice and leave no way to name the whole machine.
A bare cpus: 0 is refused from here on. Docker reads a zero as uncapped
rather than as none, so the value most likely to be typed by mistake was also
the one that removed the limit; the error names the fix. No role declared
one: all 315 cpus values in the tree are plain numbers, the smallest 0.01.
Two lookups carry the resolution so that no caller repeats it or has to know
what a percentage is read against. lookup('cpus', app, service) replaces six
copies of lookup('config', ...) piped through the resolver; those six size
worker pools and thread counts from their own cpus, and left alone a
percentage would have given libretranslate a single thread and handed the
agent broker an uncapped container. lookup('webserver_resource', key)
replaces the two blocks that took the smaller of the proxy's value and the
app's, which were the same eight lines written twice for cpus and for
pids_limit.
resolve_cpus is no longer registered as a jinja filter. Nothing referenced it
from a template once the lookups existed, and keeping it would have left two
public ways to resolve one value.
Measured: 96 tests across the filter, both lookups and the container render
templates, including that a percentage resolves before the smaller of the two
is chosen, and that pids_limit stays an integer because nginx renders it
straight into worker_connections.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The site answers 202 while it builds a version, and 200 once a build
exists. Whether that build matches what the version currently names was
never asked at the serving gate: servable() only checks that an index.html
is on disk. So the first request after a new snapshot got the previous
build with 200, and the rebuild it triggered finished into a directory
nobody was waiting for.
For a release tag that cannot happen, and for latest it is a reasonable
trade, since a visitor would rather read the previous page than a spinner.
For deployed it is wrong by definition: the name means the working tree
this run staged, so an earlier snapshot is not a stale answer but a
different question's answer. The post-deploy test that reads the published
page therefore validated the round before it, and reported a role broken
whose page had already been fixed.
The comparison already existed. _current() weighs built_ref against
_wanted_ref, which resolves deployed to the snapshot digest, and the
builder consults it to decide whether to rebuild. Only the server did not.
outdated() exposes it for deployed alone and the serving branch now treats
a mismatch as not yet servable, so the poll waits for the build it asked
for.
Measured on the stack while the previous round was live: the served page
was written 16:44:41 from the older snapshot, its own source carried the
newer content, and the correct build landed at 17:10:56, twenty-six
minutes after the test had already read the stale one and failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The docs round ended the same way twice: seven specs passed, four never ran,
and the suite was killed at its 89-minute cap after an hour and a half. Three
separate causes, each of which hid the next.
A waiter could outlive the suite that contained it. The specs budget a poll
against the role's run timeout, but Playwright is handed that timeout minus a
60-second buffer, so the global timeout fires first. When it does, Playwright
counts the in-flight test, and every test behind it, as "did not run", which
names nothing and reads like a suite that was merely slow. resolveTimeout now
caps what it returns at the budget the suite actually has, so an exhausted
waiter reports which predicate it was waiting on.
What it was waiting on could never arrive. 59e7b0a2a7 retargeted the yaml page
from /latest to /deployed, but /deployed is built from DOCS_SNAPSHOT_PATHS and
that list does not carry compose.yml, while /latest is built from a full git
archive and does. A servable version whose file does not resolve answers 302,
not 404 and not 202, and the poll rejects redirects, so it waited 88 minutes
for a page the version could not produce.
With the suite finishing, the CLI test's own failure became reachable: it reads
the published instructions from /latest, and /latest is stale. It still serves
web-app-sphinx and desk-zoom, names this repository renamed away, so ten roles
have no page there at all and web-app-docs is one of them. It now reads the
version the run just deployed, which is what a post-deploy test should replay.
Measured on the stack: /deployed/roles/web-app-docs/README.html answers 200
with the production block, /latest answers 302, and the published index lists
275 of 282 roles. The round now passes nine specs in 17.5 minutes.
Only five of the 1512 resolveTimeout call sites across 87 roles are large
enough for the cap to reach: three here and two in web-app-openwebui, whose
suite has not been re-run.
The package cache was configured in six places at once: the nginx upstream
map, the consumer compose override, pip.conf, .npmrc, the apt sources and the
twenty ensure_proxy_repo calls that bootstrap Nexus each spelled the same
hostnames, repository names and suites by hand. A new upstream meant editing
five files and hoping none was missed, and the docs told contributors to do
exactly that.
A role now declares what it needs under cache: in its meta/networks.yml, and
everything else is derived from it. svc-cache-package holds the general
upstreams; web-svc-libretranslate holds its own. All nine derived files are
generated and gitignored, so a second spelling of a hostname cannot exist.
utils.cache.hosts reads the declarations without PyYAML, because the .env
generator reaches them before any dependency is installed; it raises rather
than truncate when a comment sits at column 0 inside the block, which would
otherwise drop every declaration below it in silence. utils.cache.records
owns the Nexus record format and the Ubuntu mirror resolution, so the dev
stack and the role provision from one rendering.
A build may only resolve a cached host it can verify: the plain-HTTP hosts
are hijacked for every image, and the TLS hosts only for an image whose
Dockerfile installs the frontend CA. Hijacking all of them would fail every
pip and npm build with CERTIFICATE_VERIFY_FAILED.
svc-cache-package deploys the cache and its TLS frontend by calling the same
shell scripts the dev stack calls, rather than reimplementing them as tasks.
The role is invokable through the new svc-cache category and inert until a
host carries it.
Verified: the twenty Nexus payloads, the fifteen nginx server blocks and all
seven package-manager configs are byte-identical to the hand-written files
they replace, and the new category adds one role to the deploy order while
preserving the relative order of the other 180.
The role has not been deployed yet.
ruff reflows the guide_role signature and its two call sites, the same for
three test modules, and the YAML formatter prepends the document marker to
the machine's compose file. These files were created one commit too late to
be caught by the previous formatting pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ruff format reflows long calls and normalises quotes, isort reorders the
imports of the documentation lint package, SIM117 folds nested `with`
statements into one parenthesised block, TC003 moves typing-only imports
under `if TYPE_CHECKING`, the JavaScript linter replaces a `var`, and the
YAML formatter rewraps five role files. None of it changes behaviour.
One entry does more than that: the formatter had reflowed two calls in
test_builder.py so their `# nocheck: cache-read` markers landed on the
closing parenthesis, where they suppress nothing and the cache-read lint
would have reported two real violations. Both calls are hoisted into a
variable so the marker sits on the reading line again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The replay of a role's "### Production" block moves out of the CI guide
scripts and into the documentation role itself. The block is read off the
rendered page over HTTP, so what runs is what a reader copies, and the role
that publishes the page is also the role that proves it.
scripts/github/guide/{one,compose_deploy,host_deploy}.sh and
cli/build/docs/readme/production.py are gone with the workflow step that
called them. The instructions marker now only adds web-app-docs to the row's
apps; test-e2e-cli discovers the role through its new templates/test.env.j2
and runs files/test/test.sh.
The block runs in a throwaway machine, never on the stack host. compose.yml
starts one service built from files/test/Dockerfile, whose base is
INFINITO_PARENT_IMAGE out of the mounted checkout's .env, so the machine is
the stack host's own distribution and the replay exercises the installation
instead of a prebuilt image. files/test/init.sh adds an init when the base
carries none and keeps the full -Syu that ef0f03ea98 established for arch.
scripts/tests/workspace/base/01_install.sh then prepares the machine before
the block runs.
Nothing is skipped any more. The previous host-install branch exited 0 and
passed silently over 65 of the 178 invokable roles; a block without a
"docker run" is now replayed against systemd in that same machine, with the
git clone dropped in favour of the mounted tree under test.
The target role comes from the new guide_role lookup, which reads
list_invokable_app_ids() rather than matching role-name prefixes, and falls
back to the asking role when the round deploys nothing else. deployment.py
grows running_apps() so both lookups share one definition of what the round
deploys. required_by_coverage.role_is_invokable drops its own walk over
categories.yml and delegates to the same SPOT; measured equal across all 280
roles.
Carried in the same index, because the refactor pass ran over this change
set: library.py splits into sites.py and build_queue.py mixins, builder.py
hands translated_languages to catalogs.py and folds three repeated log-tail
and state writes, and placeholders.py becomes a facade over spans.py,
repairs.py and damage.py. No test changed for those splits, and the two
ref-resolution dicts builder.py carried are replaced by _wanted_ref, which
also stops hashing the deployed tree on every tag build.
Verified: 7132 unit and lint tests pass over the whole tree, the guide_role
lookup renders through a real Templar, the base image resolves from .env,
and compose config validates the service. The replay itself has never been
executed; that needs a deploy of web-app-docs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DEPLOYMENT_MODE was compared against a quoted literal 219 times across 110
files. IS_COMPOSE_MODE and IS_SWARM_MODE say the same thing once, and
test_deployment_mode_booleans keeps it that way: only the file that defines
them may still compare.
Both are real booleans under this ansible, measured before the swap because
the alternative is silent: a variable that templates to the string "False" is
truthy in Jinja, which is why IS_STACK_HOST is written `| bool` at every use.
These need no filter.
What had to learn the names:
* utils/annotations/task_gate.py, which several lints ask whether a task is
compose-only gated;
* four compose-template lints that recognise their gate by matching the
expression;
* the fixtures that stand in for the play's variables. They derive the
booleans from the mode now instead of carrying them, or a test that
overrides DEPLOYMENT_MODE keeps the old pair and asserts the wrong
branch.
Each replacement was checked rather than trusted: the 206 conditions that
changed in place were evaluated old against new through the Templar under
compose, swarm and host, with no disagreement, and no changed line turned out
to be anything but a condition swap. `!= 'swarm'` became `not IS_SWARM_MODE`
rather than the shorter IS_COMPOSE_MODE, because only the literal translation
stays correct should the mode ever be host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cosmos, Quick Setup and Credits follow from meta/services.yml, categories.yml
and galaxy_info.author, yet 279 role READMEs carried a copy that went stale
whenever the metadata moved and nobody re-ran make docs. The documentation
build now runs the generators itself, against its own scratch checkout, and
the committed READMEs keep only the prose a person writes.
utils.cache.PROJECT_ROOT derives from the module's location, so putting the
checkout first on PYTHONPATH makes the generator operate on the scratch tree.
i18n.py and builder.py add it; the one remaining coupling, a relative_to over
PROJECT_ROOT in a log line, is gone.
CI extracted the "### Production" block from the committed README with awk.
Stripping the section made guide_deployable() return "" for every role, which
selects no deploy row and turns nothing red: a fail-open regression. It now
asks _is_role_invokable, which matches the old condition exactly (178 roles
either way, both difference sets empty), and cli/build/docs/readme/production.py
renders the block for the two guide scripts. All 178 renders are byte-identical
to what HEAD served, and both scripts' extraction pipelines produce identical
output for a web app, a service and a desktop role.
test_readme.py inverts: the three sections are now a hard failure in a
committed README. readme_md.md documents what each is derived from.
Removed: the docs, cosmos, readme-generate, readme-index and readme-check
targets, the five aliases pointing at them, and docs from make quality.
Verified by running generate_commands() in series against a copy of the
checkout: every command exits 0, no README is left without Cosmos or Credits,
and exactly the 178 invokable roles get a Quick Setup. The ansible_roles step
needs pandoc, which only the docs image carries, so that one link of the chain
is unexercised here; it is unchanged by this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every role README carries a ```mermaid flowchart and the documentation
showed it as source. A post-transform now replaces those literal blocks with
<pre class="mermaid"> in html builds, which the runtime picks up; every other
builder keeps the fence as it was.
sphinxcontrib-mermaid injects an inline init script, which this deployment's
CSP refuses, so the runtime and its initialisation are two linked files
instead: mermaid-init.js calls mermaid.initialize with securityLevel strict
and follows the reader's colour scheme. The diagram source is html-escaped
before it is embedded, or an & or < in a node label would break the page.
The runtime is fetched at image build time through ADD --checksum, so a
changed upstream file fails the build rather than shipping. Version and
digest are not ARG defaults, which test_arg_without_default forbids; they
travel meta/services.yml -> vars/main.yml -> compose.yml.j2 the way
DOCS_IMAGE and DOCS_VERSION already do, and an update block registers the
pin with the npm version watcher instead of exempting it.
Verified by building a page with sphinx: the diagram arrives as
<pre class="mermaid"> with its source escaped, a bash fence stays a
highlighted block, and both scripts are linked. A missing mermaid.min.js does
not fail that build, so a local run without the image stays green.
scripts/lint/mermaid.sh keeps validating diagram syntax through mmdc, which
npm installs unpinned. It therefore checks against a newer mermaid than the
11.4.1 readers get, and a diagram using syntax from in between would pass the
lint and render blank.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
untranslated_markup.py and untranslated_yaml.py shared a name prefix in one
directory, which test_prefix_grouped_files reads as a folder in disguise. They
are now extensions/untranslated/markup.py and yaml.py, beside the __init__.py
that extensions/local/ already carries for the same reason.
A module named yaml.py inside a package does not shadow PyYAML, because
imports are absolute: all nine extensions conf.py registers still import, and
the docs-role unit suite stays green.
The unit tests move with them, keeping the parents[9] root derivation their
sibling package uses at the same depth.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_brands, test_titles and test_yaml_source spelled out meta/main.yml,
meta/services.yml and meta/info.yml where utils.roles.mapping already names
them. test_file_spot reports exactly that, and it did: the three files were
its only findings in a full make test.
A hard-coded path keeps pointing at nothing once a metadata file is renamed,
and a fixture that writes to the wrong place fails somewhere else than where
the mistake is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One predicate now answers what no translator may be handed, and both ends of
the pipeline read it: the extraction withholds the message from the catalog,
the client refuses to send it. What the catalogs keep back is exactly what the
client would have declined.
Two shapes qualify. A message with no word is markup. A message that is only a
name is a label: a brand, this software, a credited person, or a bare URL.
Inside a sentence the name stays protected by the span machinery and the
sentence still needs translating.
Three details each come from a finding. The comparison runs against the prose,
because **Mastodon** is not the string Mastodon and prose() already strips the
emphasis. Case is ignored, because a page writes the name the way its sentence
needs it and **mailu** is still Mailu. The address is matched first, on the raw
text, because masking protects the address of mailto:a@b.c but leaves the
scheme behind and its prose reads as a word.
Measured over the docs domain: 22745 messages fall to 22402 with zero
warnings, and 145 of the 377 removed entries carried a corrupted brand that
was being shipped - Keycloak read Schluesselmantel, OpenProject read offenes
Projekt, micro read Mikrowelle.
SOFTWARE_NAME is read from the group_vars SPOT beside SOFTWARE_REPOSITORY. It
lived there already and no python had ever read it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A table cell holding only a link has nothing to translate, yet it became a
msgid like any sentence: 2999 of the docs domain's 25742 messages, 11.7
percent. They can never be filled, so they count as pending for good and
skew every completeness figure.
The messages come from sphinx's own extract_messages, which is what the
gettext builder collects. That is not cosmetic: sphinx takes a paragraph's
rawsource, so judging astext() would read 'e2e/' out of [`e2e/`](e2e/),
find a word in it and let the markup through. A test now pins that
difference, because the first version of the test could not see it.
The predicate is has_words, the one the translation client already applies
before it sends a batch, so extraction and translation share one definition
rather than two that drift.
Measured: 25742 messages fall to 22745, no surviving msgid loses its
translation, and 2924 of the 2948 removed entries carried their own source
as their translation.
The import is guarded because the site builds arbitrary refs and one older
than utils/i18n/ cannot supply the predicate; the transform then does
nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sphinx prefers nothing: dir/index and dir/README are two independent
documents. The preference lives wherever something maps a folder onto a
page, and this repository had two such places disagreeing.
local.file_headings has always hidden README.md from a directory carrying
an index.rst, while yesterday's resolver saw README only, so a link led to
the page the sidebar hides.
Both now try index.rst, index.md, README.md, README.rst in that order. The
extension is renamed accordingly, and the tuple lives once in the lint
package rather than a third time.
That also repairs two links this branch degraded: roles/ holds an index.rst
and no README, so a README-only rule reported every reference to it as
broken and they were turned into code spans.
Three links stay code spans on purpose. Their text names something the
target is not - a suite, a file - so resolving them would restore a
misleading reference rather than a working one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
join(' ') renders KEY=a b, which the shell reads as KEY=a prefixed to the
command b: the variable is set for that command only and unset afterwards,
so the consumer's ': "${…?}"' guard kills the run over a value the
rendered file plainly shows.
dotenv_quote does not fix it. docker stack deploy passes the value verbatim
in swarm mode, so the quotes would land inside it.
Both sides move to a comma, which the shell does not split on.
The four existing cases could not have caught this: each passes a single
name or an empty string, and one element renders the same either way. The
added case is the only one that sees the separator, and removing IFS makes
it fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
apidoc now excludes roles/ and library/ the way it already excluded tests/.
A role name carries hyphens and is therefore not an importable module path,
so automodule reported 44 invalid signatures and rendered nothing; library/
holds ansible modules whose ansible.module_utils imports exist only inside a
play. Measured: 168 generated pages drop to 151, and the 17 that go consist
of a heading and automodule directives on unresolvable paths.
The role page generator fits every section underline to its title. pandoc
counts one character fewer than docutils where a heading ends in a stray
variation selector left behind by a removed emoji. It also escapes inline
markup in role metadata, because one description reads (*.parent) and opened
an emphasis that never closed.
The index generator underlined a 21-character caption with 19 characters.
conf.py adds the tel scheme, without which the phone number in SECURITY.md
is an unresolvable reference, and the extract copies the image assets into
its conf directory the way the site builder does, without which every run
reports a missing html_static_path and favicon.
Four tests pinned values this changes. The fifth, the extension count in
test_conf, asserted nothing the loop below it does not already prove and
broke on every legitimate addition; it now asserts the list is non-empty, so
that loop cannot run over nothing.
With this, make i18n-extract domain=docs finishes with zero warnings under
the -W gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Almost every role declares galaxy_info.company as a block scalar. The
generator rendered each entry as '- **key**: value', so the second line
landed in column 0 and ended the bullet list it sat in. 280 of the build's
remaining 354 docutils messages came from generated/roles/*.rst, which is
this generator's own output; folding every value onto one line clears 277
of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both extensions landed in bd15eae9fa without the mirrored unit test the
repository lint requires, which left that lint red.
The resolver's cases include the mistake its first version made: it
filtered on refdomain == 'doc', while myst builds a document reference
only when the target is_file(), so a directory arrives with refdomain
None and the transform did nothing at all. Reintroducing that filter
fails three of the six cases.
The parser's cases pin what the build relies on: one yaml-highlighted
literal block, no block quote, a title from the filename, and the orphan
docinfo as the first child, which is the only place sphinx reads metadata
from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fetch() requested latest and run_builder() requested deployed once at startup, so a release tag was built on demand and then never revisited. A tag carrying a languages.json from before the translated-site split stays servable, nothing ever requests it, translates() stays false for every code and /v1.0.0/de/ answers a hard 404 forever.
_refresh_stale_sites() asks for every version that is already servable; request() checks _current() itself, so only a genuinely stale one is queued and an unbuilt tag stays on demand instead of triggering a mass rebuild.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
request(..., background=True) skipped a language as soon as translation_servable() found an index.html, which is an existence check with no ref in it. A language was therefore built exactly once: every later commit advanced the English site while every translated site stayed frozen on the commit it was first built from. The eager loop this replaced rebuilt them all, so the freeze arrived with f5eff500c1.
_publish_site already stamps the ref beside each translated site and nothing ever read it. translation_ref() reads it and _translation_current() compares it against _wanted_ref(), which also replaces the eager {LATEST: head, DEPLOYED: snapshot_ref()} dict so an unreadable snapshot no longer kills the builder while building an unrelated version.
Two more from the same review: build_language wrote into the version's state file, so a failing language reported the whole version as failed and leaked its sphinx log into /api/versions; it now uses its own version:code key. And the language index was written after the site was published, so a failure in between left a current site with a stale index, which _current() rejects forever and fetch() would rebuild on every tick; the index is written first.
library.py had grown past the 500-line cap, so the build machinery moves into builder.py as a Builder mixin. An AST diff proves every function, method and constant is byte-identical apart from _write_json -> write_json, and the six build-focused test cases move to test_builder.py with the count unchanged at 16.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ensure() creates the agent's network and attaches the broker to it in the next
statement. Under swarm that network is swarm scoped: the control plane holds it
the moment create returns, so the GET inside connect answers, while realizing it
on a node is the allocator's asynchronous work and connect is node local. The
attach therefore ran against a network the node did not have yet and took a 404.
Run 36107421422 shows both halves. The broker logged agent_failed with 'POST
/networks/<id>/connect -> 404: network not found' for every owner, and dockerd
on the same node logged 'Failed to find network <id> on node' once per node, for
all three. Every agent request then answered 502 and the drill reported the two
agent services missing. The compose row passed throughout, which fits: a bridge
network is local and usable as soon as it exists.
connect now retries while the engine answers 404, up to realize_timeout, and
re-raises anything else at once. EngineError carries the status for that, set at
the single raise site in call, so the decision reads a number rather than
matching the message text. The wait is scoped to the one condition known to
clear by itself: a 403 from the socket proxy still fails immediately, and a 404
that outlives the window still fails, because then the network really is gone.
Three cases pin it in both directions, and each fails against the previous
engine.py: a 404 that clears is waited out, a 404 that persists raises at the
deadline, a 403 raises on the first attempt.
Not verified: no swarm run has exercised this. realize_timeout is 120 seconds
from a single observed window of at least five, and it converts a structurally
missing network from a fast failure into a slow one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role answered one contract, POST /v1/systemone, under the name of the
single implementation that happened to serve it. Renaming it to svc-ai-s1
leaves the contract where it was and lets the flavor decide who answers:
variant 0 runs laya, variant 1 runs the GLiFormer server the role carried
before, so a round proves the contract against both rather than against
whichever one ships as the default.
Reading laya 0.3.20 rather than reasoning about it corrected two defects that
would have hit variant 0 on its first deploy. LAYA_MODELS takes the short
checkpoint names, not repository ids: router.py maps english to
convaiinnovations/laya, so the entry carries both and hf download takes the
repo while the env takes the name. And laya serves @app.get("/health") where
jeff serves healthz, so the base healthcheck moves to health and variant 1
overrides it back.
services.s1.gpu states the intent and 00_core.yml resolves what the host can
actually do, requiring /dev/nvidia0 and /usr/bin/nvidia-container-runtime
before it builds a CUDA image. Both are needed: with the device alone the
daemon has no runtime to attach it with, so a GPU machine still builds the CPU
image until the toolkit arrives. The flag therefore ships true and stays inert,
and the CPU render is byte identical to the one before this commit.
templates/Dockerfile.j2 becomes files/Dockerfile plus files/shell/build.sh and
files/shell/entrypoint.sh, in the shape svc-ai-agent-broker already uses: ARGs
supplied by build.args, the context staged through render_replicated_templates.
The branching is shell rather than Jinja, so shellcheck reads it and it runs
outside a deploy. Executed with pip, apt-get, hf and python stubbed, all four
flavor and GPU combinations emit what the template emitted.
meta/info.yml gains the video that explains what a System One model is, which
retires the marker that claimed none existed.
Verified: 519 lints with 381 subtests, 196 unit tests, the four-way build.sh
execution, and the five-row truth table for the GPU gate. Not verified: no
laya image has been built and no container has run, on either deployment mode.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
build() ran sphinx once per translated language before returning, so a deploy blocked for roughly 30 minutes per language. With 29 translated catalogs German sat eighth and its site appeared after about three and a half hours, which is why the language spec polled /deployed/de/ into a 404 for its whole run. The server already answered an unbuilt version with 202 and a build request; an unbuilt language got a bare 404.
Give languages the same treatment. request() takes a code and writes a version:code marker, build_language() builds exactly one from its own checkout via the extracted _checkout(), and the server answers a translated but unbuilt language with 202 while an untranslated one keeps its 404. No language is dropped: build() now queues every translated one into queue/background/, which next_queued() drains only after the lane a visitor is waiting on, so the requested language overtakes the backlog.
languages.json grows a translated list beside known, and _language_index() reads a pre-split file as known-only so an existing volume keeps naming its languages instead of rendering an empty switcher.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The healthcheck is http_status on healthz, which answers as soon as the HTTP server binds, while the model loads on the first real question. Everything that waits on the healthcheck therefore treats jeff as ready during a window in which it cannot answer. svc-ai-litellm's router hook allows one attempt of services.jeff.request_timeout and refuses to route on any error rather than substituting a decider, so a consumer arriving in that window gets a 400 saying the system_one strategy could not reach a decision with TimeoutError: timed out. Run 36064589578 carries that twice in litellm's container log, after which the same route answered in 6.7 seconds.
01_manager_ops.yml puts one real choice question after the handler flush and repeats until it is answered, in the shape svc-ai-litellm already uses to warm its chat model: resolve the hosting node and container id, then pass the script in through python3 -c. Raising request_timeout or retrying inside the hook would only move the window, and the refusal itself is correct and stays.
The script reads the port, the alias and the key from jeff's own environment, so nothing reaches the command line that a task echo would carry. It checks that the answers block holds the question it asked, because a 200 without an answer means the model did not decide and reporting success there hands the cold load back to the first caller. The mirrored test pins that exit and pins the budget above request_timeout, since a warm-up on the router's budget would time out exactly where the router did.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The routing probe cross-checks usage.prompt_tokens against what it sent, because that is the one number in the exchange the backend produced rather than this deployment declared. A mock route answers from mock_response and never reads the prompt, and litellm sets usage unconditionally from DEFAULT_MOCK_RESPONSE_PROMPT_TOKEN_COUNT, which is 10 in the pinned main-v1.77.7-stable. Every mock answer therefore reported 10 of 8192 and failed as a silent truncation, which is what run 36064589578 hit in both litellm rows.
The exemption sits after the declared-window check, so a mock routed a prompt its own window cannot hold is still a routing failure; only the truncation number is dropped. MOCK_ALIASES comes from AI_MOCK_ALIASES rather than a second derivation of the same list.
Pinned by three cases, including the placement: moving the exemption above the window check makes test_a_mock_is_still_held_to_its_declared_window fail. The reason string of a neighbouring case claimed the opposite of what the pinned image does and is corrected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The router settings become one mapping. Nine scalars sharing a router_ prefix were a dict spelled out in key names, and the sampling section about to arrive would have added five more. Carried along: fifteen lookups in vars, three variant overrides, four test files, and the two rejection messages that name the settings path, which would otherwise have pointed a reader at a key that no longer exists.
On a configurable trigger, seconds or requests or probability, the same request goes to several eligible routes at once. jeff is shown the replies rather than the routes that wrote them, with the aliases hidden behind option labels so the verdict rests on the text. In shadow mode the caller keeps the route already decided; in serve mode the winning reply is handed back through mock_response, so the fan-out is the answer path and no further call follows.
Each comparison writes one row per offered route to the proxy's own database, grouped by a request class of token band and capability demands. Grouping finer would spread the evidence too thin to decide, coarser would let a verdict about a vision request speak for a plain chat. A route that raises is left out rather than recorded as a loss: failing to answer is a fact about the moment.
The decider now also sees the price of this request, the measured rate, the quota and prompt caching, plus how each route fared before. A figure the catalog does not state is left out rather than defaulted, because a decider cannot tell an absent number from a zero one.
Sampling ships off. Every history and comparison path is caught wide and logged: neither may ever fail a route.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three modules reimplemented what utils.cache already offers, and each paid
for it with a suppression.
test_compose_override_services_defined carried a yaml.SafeLoader subclass
whose only job was to swallow compose's own tags, plus a noqa for the
yaml.load rule it then tripped. load_yaml_any handles !override and !reset
already, so the loader and the suppression both go.
test_compose_bind_sources_exist read default.env directly and argued in a
nocheck marker that once per session is fine. Reading it through
utils.cache.files makes the argument unnecessary.
test_gpu_gate loaded the role's vars and meta files with raw yaml over
hand-assembled paths; it now uses load_yaml with ROLE_FILE_VARS_MAIN and
ROLE_FILE_META_SERVICES, the way 50a13c755b did for the playwright
multipliers.
gpu_count drops a noqa ruff no longer raises, and the development env module
an import nothing uses, which lint-python rejects.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image pulled both directions of every Argos pair because upstream's
scripts/install_models.py matches a package when from_code AND to_code are
both wanted, so a direction cannot be expressed through
--load_only_lang_codes. prefetch_models.py now replaces that script instead
of warming its cache, and takes --directions: `both` keeps the old set,
`from_source` keeps only the pairs translating out of English, which halves
the download for a deployment that never translates back.
Replacing the script means carrying its guards. It refuses a language code
no package offers and refuses an empty selection, and it hands MiniSBD the
iso2model-converted codes rather than the raw ISO ones. Without those a typo
would have built an image with fewer models, or none, and reported success.
The build reaches the package cache over TLS, which needs the frontend CA in
the image trust store. Rather than reimplement the installer in the
Dockerfile, stage scripts/docker/cache/package-frontend/ca.sh into the build
context and run it: it already picks the trust path per distribution,
rebuilds the store and skips when no CA was staged. The trust-store path now
exists in exactly one place.
That step needs root. The base image sets USER libretranslate (uid 1032) and
RUN inherits it, so update-ca-certificates fails with
cannot create /etc/ssl/certs/ca-certificates.crt.new: Permission denied
The CA layers therefore run as root and hand the user back before the model
layers, so the models stay owned by the runtime user.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `auto` alias picked a model that could not hold the prompt, and every
layer that should have caught it was comparing this deployment's own
declarations against each other.
Three errors produced one failure. smollm2:135m was declared at 16384 while
the tag trains to 8192, so Ollama clamped num_ctx and then truncated the
prompt to half the window rather than refusing it. The token estimate ran at
three characters per token and was documented as erring high, but the
oversized prompt is built from "x ", which tokenizes at two: 16384 characters
are 8193 tokens on all three served tokenizers against an estimate of 5461.
A window too large and a demand too small cancel into "eligible", and
neither correction works alone; both ship. The third error is what fired
them: at one sample the speed measurement kept its cold load, so a 135M
model measured slower than a 0.5B one and the router ranked on the
inversion. The warm-up is now discarded at every setting.
The probe compared a declared window against an estimated demand, both ours,
so it could not observe a declaration the backend does not honour. It now
reads usage.prompt_tokens and fails when the backend evaluated materially
less than was sent. Against the live stack that check separates the two
models exactly: smollm2 reports 4098 of 8192 sent and fails, llama3.2
reports 8221 and passes, and 4098 is the figure Ollama's own truncation
logged. A 400 now carries its body, without which this failure reads as
"HTTP Error 400" and nothing more.
The strategy is selectable. Where jeff is deployed the decision comes from a
System One encoder reading the prompt; otherwise the routes are ranked on
standing preferences, and each factor carries its own weight. Both were
exercised over six prompt sizes: system_one chose qwen, then smollm2, then
the mock as the prompt grew, weighted chose llama3.2:1b throughout. The
state handed to the decider is capped at 512 characters, because the encoder
holds 512 positions and the previous 20000 came from the server's own
rejection limit, which is a different number: measured through the gateway,
a 2000 character state already exceeded the decider's timeout, and one
timeout leaves the request running so every later one queues behind it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three tasks passed the gateway master key as `container exec -e
LITELLM_MK=<key>`. Ansible echoes that command line, and the masking was
written as `no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}"`, which CI and
local deploys turn off, so every run printed the key in clear. That single
pattern is where four live master keys leaked from this branch.
The key is not masked better, it is no longer passed: the gateway container
already carries LITELLM_MASTER_KEY from its own env file, and `container
exec` inherits it, so ensure_keys.py, probe_auth.py and warm_model.py read
it from the environment they run in. Verified against the running container,
whose config env carries the variable.
The consumer keys have no such home, so they move to the shell module's
stdin: the payload for ensure_keys.py and the single key probe_auth.py
revokes and restores. That takes them out of the `cmd:` echo. It does not
take them out of `module_args`, which Ansible prints when a task fails, so
those two tasks keep their masking; a literal `no_log: true` was tried first
and rejected by the repository's own lint, which requires the masking to
stay switchable for diagnosis. The two tasks that now carry no secret at all
lose their `no_log` instead, which makes a failing speed measurement or warm
call readable again.
Exercised by unit tests only. The deploy that would run these tasks needs a
variant with consumer keys, and the litellm-only round has none, so
01_keys.yml was never entered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_passthrough_swarm quoted and escaped any value containing whitespace,
which contradicts the contract stated at the top of the same module:
docker stack deploy does not strip outer quotes, so the value reaches the
container with them.
Ansible's to_json emits the default ", " separators, so AGENT_PLATFORMS
always carried spaces and arrived as "{\"hermes\": ...}". json.loads then
returned a str rather than a mapping, and sorted(PLATFORMS) in the broker
iterated its characters: the broker advertised one model per character and
never hermes or openclaw, which is why Open WebUI's model list held neither
and the agent-broker specs failed before reaching a completion. The same
encoding mangled AGENT_CONSTRAINT, whose placement expression would have
failed once an agent was reachable.
The branch existed for the opposite consumer: test.env is sourced by bash,
where KEY=one two leaves the variable unset under set -u. Quoting is the
wrong instrument for that, because one filter cannot serve both contracts
keyed on DEPLOYMENT_MODE alone. Both values that join on a space now join
on a comma and their consumers split on one, so no rendered env value needs
quoting to survive the shell.
Two unit tests asserted the quoting while a sibling in the same class
asserted that swarm does not escape a double quote; the pair could not both
hold. They are replaced by one test that renders real JSON and checks the
container receives a mapping. The lint that flags a space-joined value no
longer accepts dotenv_quote as the remedy, since in swarm it no longer
quotes, and its message names the comma instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The check read df --output=pcent and judged every filesystem. The NVIDIA container toolkit mounts a 4 KiB tmpfs per CDI hook holding exactly one file, so it sits at 100% on every GPU host, no cleanup can ever change that, and the service failed the whole deploy with it.
A filesystem now counts only from 64 MiB up, which keeps /boot and every real partition in. Smaller ones over the threshold are still reported, as INFO rather than WARNING, so a genuinely tiny filesystem does not vanish from the log; below the threshold they stay quiet. The warning also names the mountpoint now, where it used to print bare percentages that said nothing about which filesystem had filled up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pull.py ran docker compose build and pull through subprocess.run without a timeout. The ansible task keyword eventually kills the task, but the python in between kept waiting, and the handler name says "compose pull" while pull.py also builds, so the phase the deploy sits in is not what the log claims. A first libretranslate build downloading every Argos model outlived the task timeout, and each retry restarted the RUN layer from zero.
run_cmd now passes timeout=RUN_TIMEOUT_SECONDS and turns TimeoutExpired into rc 124 plus whatever the command had already written, which every caller already treats as a failure. The matrix compose path carried no task timeout at all: its compose up has retries: 12, so a hang would never have triggered a single one of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first deploy that measured generation rates gave mock/deterministic
4148 tok/s against 70.7 for the fastest real model. That number is the
proxy's echo latency, not a rate: a mock returns its declared string
without generating anything. Being local, free and 59 times faster, it
then won every routing decision, so the alias answered pong to any
prompt and the routing probe would have gone green while proving
nothing.
The exclude list is derived rather than maintained: it is the router's
own alias plus every alias whose provider is the mock provider, so a mock
declared in services.yml or in a variant is covered without a second
place to update.
LITELLM_EXCLUDE also becomes a set. It held a single alias and was
compared with `not in`, which on a string is a substring test, so a
comma-joined list would have silently unranked any model id contained in
it.
A mock stays routable. With nothing preloaded it is the only model the
gateway serves and has to win; it simply no longer wins on a number that
is not a model's. Against the rates this deploy measured, the alias moves
from mock/deterministic back to qwen2.5:0.5b.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role declares `gpu: true`, which resource.yml.j2 turns into a nvidia
runtime, and appends `-cuda` to the image tag so the container carries the
CUDA libraries the runtime alone does not provide. Both hang off
LIBRETRANSLATE_GPU, read through resource_filter: lookup('config', ...)
returned false for the same declaration and shipped a CPU image twice.
assert_gpu.yml asks CTranslate2 for its CUDA device count after the compose
handlers flush and fails on zero. Without it the container stays healthy and
answers every request while falling back to the CPU, which is how a broken
CUDA stack survived several deploys unnoticed. It is gated on the host
carrying /dev/nvidia0 and on IS_STACK_HOST, since the exec helper exits 64 on
swarm workers.
load_only moves to the variants so CI stays slim while a production deploy
loads every language. An empty list now drops the flag instead of passing an
empty one, whose meaning is ambiguous.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
resource.yml.j2 renders `runtime: nvidia` for any service that declares
`gpu: true`, gated a second time on the host carrying /dev/nvidia0 so a
CPU-only host never gets a runtime its daemon cannot resolve. The device
fact is probed in daemon_config.yml next to the kata and gVisor shims, and
daemon.json.j2 lists nvidia alongside them. nvidia stays out of
SANDBOX_RUNTIME: it passes a device through, it does not isolate.
nvidia.yml drops rival apt sources for libnvidia-container before writing
its own. The other order leaves two sources with different Signed-By keys
for one URL, which makes apt refuse the entire source list.
test_gpu_runtime.py covers both polarities and the missing device fact; the
latter caught `x.stat.exists | default(false)`, which raises on the
undefined object before the filter runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The router already ranked routes on traits.speed, but nothing ever set
one. No catalogue carries a rate, because a rate is a property of this
machine under this load rather than of the model, and a hand-written
number in services.yml would be a guess that rots.
The deploy now measures it. After the gateway is warm, measure_speed.py
asks it which models it serves and sends router_speed_samples requests to
each model that has no rate yet, discarding the first as a cold load and
taking the median of the rest. The result is stored on the stack host and
rendered back into model_info.traits.speed on the same deploy, so the
gateway restarts once with the numbers it just produced. The stored file
is read back with slurp rather than lookup('file'), because the config
directory lives on the stack host and a lookup reads the controller.
A model that already carries a rate sends no request at all. Re-measuring
everything every deploy would produce results the tolerance discards anyway,
and that traffic is the entire live surface of this step: what can hang,
what can fail the deploy, and what the deploy waits for.
router_speed_remeasure discards the stored numbers once, for the case
where the hardware under them changed.
Two budgets bound a run. A measured request may take
router_speed_timeout; the discarded first request gets upstream_timeout
instead, because it pays the model load and holding it to the sample
budget would time out exactly the models worth measuring. Exceeding a
budget abandons that model, not the deploy, and the model keeps whatever
rate it had.
Every matrix variant overrides the sample count to 1, where the single
sample is kept rather than discarded: a CI round proves that the
measurement, the store and the re-render ran, not what the rate was on a
shared runner.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The router ranked surviving routes lexicographically: local, then price,
then speed, then alias. How much a factor counted was its position in
that tuple, so a deployment could not say that it cares about price more
than about speed.
Each factor now carries a 0..1 weight in services.litellm.router_weights.
A factor is scaled to 0..1 inside the candidate set before its weight
applies, because the raw units do not compare: a per-token price is a
millionth of a cent and a rate is tens of tokens per second. A factor
whose value is identical across the set scores every route 1 and drops
out, rather than deciding on a rounding artefact.
Confidentiality stays out of the score. local_reason removes every remote
candidate before decide() runs, so no weighting can send a request that
must stay local to a third party; a weight is a preference, and a
credential leaving the cluster is not something a preference should be
able to buy. The shipped weights additionally keep locality worth at
least cost and speed combined, and a test reads them out of
meta/services.yml rather than repeating them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
031 made the gateway the one place backends are wired, but not the place the
choice is made: every consumer still names one alias and LITELLM_CHAT_MODEL
resolves it once per deploy for all of them. That is wrong in both directions.
A prompt needing a large window reaches a model that cannot hold it, which is
how web-app-hermes answers HTTP 500 when the agent model declares fewer than
64000 tokens, and a one-line question reaches whatever the deploy picked.
This adds the System One half: fast, typed, no generation, deciding which
System II model answers. Requirement 038 records the decisions; the shape
follows prismhq/jev-router with a deterministic decision in place of a model,
and decision 7 records why no learned one is in scope.
A pre-call hook, not a semantic router. LiteLLM's own auto_router matches
embeddings against example utterances, which answers "what is this about"
rather than "which model can serve this", and it pulls in the semantic-router
extra plus an embedding backend the gateway does not otherwise need.
The hook ships as a rendered file beside config.yaml rather than a derived
image: proxy_server.py resolves a callback through get_instance_fn(value,
config_file_path), which spec_from_file_location's it out of the config's own
directory. services.litellm.image stays on its pin. It is a template because
ROUTER_ALIAS has to match what the route builder publishes.
Capabilities come from LiteLLM's own catalog, 1595 models deep in the image,
keyed on the route's litellm_params.model and then on that name without its
first segment, because ollama/llama3 is filed prefixed while openai/gpt-4o-mini
is filed bare. A traits mapping overrides it for what the catalog does not
know. Declaring what the catalog already states is forbidden: it duplicates a
fact this project does not own and rots in the dangerous direction, since a
stale supports_vision keeps a model eligible for requests it can no longer
serve. Cost is not declared either; a route with an api_base or a mock_response
is served by this deployment and is free, and a keyed route the catalog does
not price sorts behind every route it does.
It fails loud, in both halves. The hook returns a string naming the demand when
no served model can take the request, and proxy/utils.py turns that into a
RejectedRequestError carrying the text. The alias's own model_list entry exists
only so the name is listable and raises when reached. Pointing it at the first
route instead, as this first did, made a router that never ran look like one
that worked.
services.litellm.router_alias is the single switch. Empty publishes no route and
renders no callbacks entry, so the hook is not even a condition of starting the
proxy; a separate enable flag would let services.yml read as on while the
router is off.
The routing probe had to learn the alias, which is backed by no backend and was
read as a route standing without one. It now also sends a prompt the smallest
declared window cannot hold and fails when the answer names the alias itself or
a model whose window is too small. That lookup tries the prefixed and the bare
name, because litellm sets model_response.model to the resolved model: matching
only the literal name would find no window and pass every answer.
svc-ai-ollama variant 1 serves three models so the routes differ in something a
request can demand. smollm2:135m is deliberately not the smallest window:
8dbdd31b22 measured it running to n_gen=22011 with context shifts at
n_left=4091, and a route's max_tokens is a default a consumer can override
rather than a cap.
Open, and only a deploy closes them: that the proxy accepts the raising route
at startup, that the alias routes across three windows, and swarm.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The harness derived LITELLM_SERVED_PROVIDERS from the keys argument, i.e. the
providers that hold a key. vars/main.yml:16-18 splits that in two: the keyed
list feeds env.j2, which indexes LITELLM_PROVIDER_KEYS and fails the render on
a keyless entry, while the served list is the keyed one plus AI_MOCK_PROVIDER
and decides what config.yaml.j2:4 publishes. The harness carried the name of
the second and the meaning of the first, so it reproduced the very defect it
was meant to catch: a mock filtered out of model_list, an empty list, and the
gateway answering 400 for the chat model while every name-level variable still
resolves to it. That cost a deploy and every warm task in run 35875571687.
TestMockModels stayed green through it by passing keys={"mock": "none-needed"},
a key the mock provider does not have and never needs, which walked around the
broken assumption instead of asserting against it.
The harness now derives the list the way vars/main.yml does, the fake key is
gone, and a new case states the failure directly: a mock must publish a route
although no provider holds a key. Negative control: reverting
LITELLM_SERVED_PROVIDERS to LITELLM_KEYED_PROVIDERS fails
test_the_mock_is_served_without_any_key with "'mock' not found in []".
meta/services.yml declares no mock in remote_models, so the default-argument
cases keep their old outcome; TestNoBackend still renders an empty model_list.
Residual: template and vars are pinned by two separate tests against one
hand-copied derivation rather than a shared source. A drift in vars/main.yml is
caught only by test_keyed_providers.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bb78ab3fb8 moved the handler loader in front of the settings render so the
play stopped dying on "handler 'compose-up' was not found". Registering them
was necessary and not sufficient: the notification now survives, and the next
task kills the play a different way.
render_replicated_templates.yml hangs notify: compose-up on the render
(_render_one_template.yml:23, default from role_templates_expanded.py:17).
00_core.yml then enters utils/network/routine.yml, whose step 1 pulls in
sys-svc-container, and that role ends on an unconditional meta: flush_handlers
at 00_core.yml:52. Job 107238119066 of run 35875571687 shows the whole chain:
NOTIFIED HANDLER ... Validate Docker Compose configuration at log line 13787,
META: triggered running handlers at 13801, and the handler at 13923 with
rc 127, "compose: command not found". /usr/bin/compose is installed by
sys-svc-compose/tasks/00_core.yml:30, which is step 3 of that same routine and
is never reached, because a failed handler aborts the play (ok=228 failed=1).
A guarded bootstrap ahead of the render absorbs that flush against an empty
queue and installs the wrapper first, the shape 252b3e58b2 applies in
sys-svc-rdbms and 7f1025e4a7 in svc-db-qdrant. Routine step 1 then falls
through on run_once_sys_svc_container, and step 3 renders compose.yml and
flushes under searxng's own application_id. The when: guard is load-bearing:
without it 01_reset.yml tears down DIR_COMPOSITIONS again under MODE_RESET.
litellm carries the same shape and survives only because it loads
sys-svc-rdbms first, which bootstraps compose as a side effect; searxng is a
leaf with no database and is therefore the first compose consumer in its own
CI row.
The existing bootstrap case in test_handler_registration_order.py finds the
first notify through task.get("notify"), a literal key in the same file, so it
cannot see a role that notifies two include levels down. The new case anchors
on the render include itself and compares indices instead. The sweeping case
checks topic ownership, not order, which is why searxng was green here while
it failed in CI.
Same class, pre-existing, not fixed here: svc-ai-agent-broker renders at :39
and enters the routine at :61, latent because it depends on litellm and is
never the first compose consumer. 252b3e58b2 names web-app-magento,
web-app-seaweedfs and svc-runner.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A deploy warmed the chat model and got HTTP 400. The rendered config in the running container read 'model_list: []' while every name-level variable resolved to mock/deterministic: the alias list had been taught about the mock, the list that decides what is actually published had not.
One list could not answer both questions. config.yaml.j2 and test.env.j2 ask which models are served and want the mock; env.j2 asks which providers hold a key and emits <PROVIDER>_API_KEY by indexing LITELLM_PROVIDER_KEYS, so a keyless entry there kills the render. LITELLM_KEYED_PROVIDERS keeps its original meaning and LITELLM_SERVED_PROVIDERS is that set plus the mock.
The new test renders both from the role's own vars and pins the distinction from either side: the mock is served without a key, and the mock is never keyed - the second assertion carries the reason, because that is the one whose violation only shows up at deploy time.
The first attempt at this fix put the mock into the keyed list and would have broken the env render. It was caught by enumerating the variable's consumers before naming a confidence number, not by a test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A deploy refused at ok=771 failed=1 with 'services.litellm.remote_models names provider(s) mock; known providers are openai, anthropic, openrouter'. Declaring the mock among the remote models, which is what makes it an ordinary model rather than a config key of its own, runs into a guard that demands a key slot for every provider. A mock needs none: it answers from its own declared response.
The guard keeps its teeth for real providers and now names the exception and its reason, instead of tolerating it silently.
AI_MOCK_PROVIDER holds the name once. The guard, AI_REMOTE_ALIASES and AI_MOCK_ALIASES read it, and the template hands it to the route builder, which lost its own constant. Two literals spelling the same thing is how the LM Studio branch came to drop the context window its README still advertised; a parameter cannot drift.
Static checking never touched this line. It took a real deploy, 772 steps in.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The proxy config grew a loop per model source, so the rule for how an alias is called lived in four places - and they had drifted: the LM Studio branch published no model_info.max_input_tokens although the role's README says a model entry carrying context publishes exactly that. Serving anything new meant a fifth branch.
The sources now meet in one filter that returns {alias, params, context}, and the template renders a single loop. Every value goes through to_json, so a response carrying a colon, a quote or a newline cannot break the YAML it is written into.
Publication order is a contract, not an accident of how the loops were stacked: LITELLM_CHAT_MODEL takes the first alias, so a locally served model has to outrank a declared one. That is now stated in the module and asserted, instead of being implied.
The gateway lint had to widen with it. Its canary checks that some gateway template still matches the provider-URL pattern, because without one the rule would pass vacuously on every consumer; the pattern demanded {{ … }} while the template now sets the URL in {% set … %}. Widening it to {{ or {% makes the rule stricter, since a provider URL in a set block is just as direct an address and would have slipped through anywhere else.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
