On a cold container the docs Playwright suite triggers four serial Sphinx builds and waits for the last one. test-languages.js polls /deployed/de/ for 200 and got 202 for 55.6 minutes in job 110003713825 of run 36745666606, after which the suite budget expired and five further specs never ran. The suite has never passed on a first attempt; it used to go green only on the harness retry, which 7127c6b96b removed for budget timeouts, so a structural overrun became a red job.
Measured cost per build, from run 36558500327 where the whole queue drained on a kept stack: container up 12:15:52, the spec green at 13:21:41 answering on its first poll, next container start only at 13:30:45. Four builds in 65 minutes 49 seconds, so at most 16.5 minutes each - not the sixty to ninety minutes 40688e4ded estimated.
Three of the four builds were none of the test's business. fetch() queued latest in the foreground although nothing waits on it at startup, and _refresh_stale_sites did the same for every servable version; both now use the background lane, which is what its own docstring describes. test-front-page.js reached GET /latest/, and reaching an unbuilt version queues its build - it now asserts the redirect from the Location header without following it, and checks the same page shape against /deployed/, which is built anyway. The release tag that test-on-demand-build.js clicks stays, and now runs behind the awaited work instead of in front of it.
The translation could only be queued after its base finished, because the 202 branch returns before _translation is reachable, so it always sorted behind whatever the sibling specs had queued meanwhile. The 202 branch now carries the language segment along, admitted by translatable(), which answers from the snapshot catalog on disk before any build; build_language still re-checks the written index.
Moving a version into the background lane exposed that status() read only the foreground one, so a waiting version would have been reported as missing rather than queued to the versions overview and to /api/versions. is_queued() now covers both lanes, mirroring _dequeue, which always removed from both. Two of the five unit failures this change first produced disappeared on that fix alone, which is what shows they pinned real behaviour; the three that pinned the lane itself were rewritten to assert the behaviour instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The engine has no browser surface, so it is verified through the repository's
CLI harness rather than Playwright: test-e2e-cli discovers a role by its
templates/test.env.j2 and runs files/test/test.sh on the node that holds the
container.
probe.py asserts five things against the running store: an unsigned request is
refused with an S3 error, the admin identity completes a signed write, read
back and delete, a consumer identity reaches its own bucket and is refused
another's, the anonymous grant matches the declared public flag, and the filer
and master ports stayed unpublished after the split.
It creates no bucket. A bucket is a SeaweedFS collection and the engine's
-volume.max is budgeted for the declared consumers only, so a probe that
creates one can exhaust the volume slots its own consumers write through;
measured as a 500 on PUT once the slots ran out. Every write now lands under
one reserved key prefix in a bucket a consumer already owns.
SigV4 is hand-rolled because the CLI runner ships no boto3. Its vectors were
generated with botocore and are frozen in a unit test, which also pins the
Node signer the consumer specs use, so the two cannot drift apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-seaweedfs becomes two roles. web-svc-seaweedfs keeps the storage
engine: the container, the seaweedfs_data volume, the per-consumer S3
identities and the single canonical domain api. web-app-seaweedfs-console
takes the filer and master web UIs into its own compose stack behind the
admin-only oauth2 proxy, reaching the engine over the shared object-store
network. A node can now run the store without the consoles.
The console needs a distinct role name because get_entity_name strips the
category prefix: both halves would have resolved to the entity seaweedfs and
collided on the compose project, the instance directory and the container
name.
The filer and master ports move into the engine's services.seaweedfs.ports
.internal as their single owner; 01_init_consumer.yml consumes the master
port and would otherwise have resolved it only by role-var leakage from the
console, breaking bucket provisioning on exactly the console-less node the
split makes possible.
The engine's api vhost gains the prometheus locations. It never had them: the
override satisfied the cross-cutting-include lint only because the strings sat
in the SSO branch that the api domain never took.
Consumers, the objstore lookup and the catalogs follow the role id.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`make quality-high` runs autoformat before the gate, and this pass rewrote three
files the same session had just committed. Nothing here changes behaviour: two
over-long call sites in `native_metrics_target.py` wrap, the `scraped()` return
in `test_vhost_cross_cutting_includes.py` gets its parentheses, and `import ast`
moves into the stdlib block it belongs to in
`test_budget_timeout_not_retried.py`.
They ride in their own commit rather than in the changes that produced them, the
way 4c66220c50, 1e083c4963, 3dd4f0abb1 and 2ba37b55b2 did, so that a reflow never
has to be read as part of a fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects where a suite-budget kill is handled.
`SUITE_RESERVE_MS` exists so a waiter that runs out reports which predicate it
was waiting on, instead of letting the global timeout fire first and count the
in-flight test and everything behind it as "did not run". a490b28af6 set it to
60 s, but the cap it feeds is measured from the waiter's own start while the
global timeout runs from the suite's, so the reserve must also absorb every
test that ran before the waiter plus the post-failure trace and report write.
Measured prefixes in run 36651124458 were 3.5 s, 32.4 s and 86.2 s; at 86.2 s
the cap landed 26 s past the global timeout and OpenProject#1 kept only a
screenshot, no error-context. 180 s leaves roughly double the largest prefix
seen.
No waiter changes category: the newly cut band, between the old cap and the
new one, is empty on both budgets. The already-capped ones lose 120 s of about
5.28 M, and the only one that reached its cap was failing at the old value too.
The retry at orchestrate.yml exists for a Tor circuit flake. A run killed by
the global timeout fails the same way every time, so it burned a second full
budget for an identical result, up to 80% of some jobs' wall clock. Such a role
is now recorded and excluded from the retry loop.
The exclusion is only safe because the stash keeps it. That task reset
`_playwright_failures` to an empty list and only a retried role re-adds itself,
so dropping a role from the retry while leaving that reset would have removed
it from the final fail task and turned a red job green. The intersect keeps
exactly the budget timeouts in, and a test renders the task's own expressions to
hold that: the regression it guards against is a wrong set operation, not wrong
wording. Reverting the reset to `[]` makes it report `[] != ['web-app-docs']`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`/snapshot` is a tar of `DOCS_SNAPSHOT_PATHS` and the list carried 32 entries,
neither of them `Makefile` or `aliases`. Both are tracked at the repository
root and `commands.py:80-90` hands them to two generators,
`make_targets --makefile {src}/Makefile` and `aliases --aliases-file
{src}/aliases`. `generators/make_targets.py:45` reads with no existence guard,
so the build dies with FileNotFoundError before sphinx is reached.
It dies without saying so anywhere. `_failed` wrote the exception into
`states/deployed.json` and printed nothing, and `library.status` iterates
`[LATEST, *tags]`, so `deployed` never appears in `/api/versions`. No log, no
API and no test carried the failure. `/deployed/de/` therefore answered 202 for
as long as the container lived and the Playwright suite polled a dead build for
its entire budget - eight jobs of run 36651124458, three hours each.
The failing test's trace settles which build it was: the 202 body carries
`<title>Building deployed</title>`, not `deployed/de`. The English `deployed`
version is the one that never lands, so the German build is never even queued.
That also gives the discriminator: `latest` and the release tags build from a
`git archive` of the mirror, where both files are present, and they reach
`ready` in the same run.
Three parts. The list gains the two entries, in the file's ASCII order - that
is the fix. `_failed` also prints to stderr, because without it the next such
gap is just as invisible; the moodle job's docs container log was 484 access
log lines and nothing else. And a test derives the requirement instead of
restating it: it calls `generate_commands()` and requires every first path
segment under `src` to be in the snapshot, `generated` excepted, since that one
the build creates. Adding two entries by hand would have broken the same way on
the next generator.
Main is unaffected: the role does not exist there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`request()` ended in `touch(exist_ok=True)`, and on an existing file that
refreshes the mtime. `next_queued()` sorts by mtime and serves the oldest
marker, so re-requesting a build that is already queued moved it behind
everything else. The server calls `request()` on every visit, and a visitor
waiting for a page polls every thirty seconds, so waiting for a build was
what prevented it:
queued de, then a tag : ['deployed:de', 'v14.2.0'] -> builds deployed:de
after one poll on de : ['v14.2.0', 'deployed:de'] -> builds v14.2.0
One poll is enough. In the docs suite the spec before it deliberately clicks
an unbuilt release tag, so another hour-long build is always sitting there to
be overtaken by.
Observed in run 36596159856: `/deployed/de/` answered 202 and never 200 for
two hours and four minutes in one deployment and two hours fifty-seven in
another, from those jobs' own rescue artifacts, while the suite budget is
fifty-nine minutes on clearnet and eighty-nine on tor.
This removes the starvation, not the arithmetic: a full Sphinx build costs
sixty to ninety minutes here and the suite needs two of them in sequence,
English then German. Whether one round now fits is for a run to say.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every commit here went in with `--no-verify`, so the pre-commit `make test`
never ran and neither the formatter nor the linter saw the change sets. The
quality gate collects that debt now.
`test_aliases.py` built a lookup table by handing a generator to `dict()`,
which ruff C402 rejects and offers no autofix for; a dict comprehension does
the same work. `selections.py` and `test_translate.py` each wrapped a call
across three lines that fits on one, which is the formatter's own rewrite
from the run that then aborted on the C402 above.
All three edits preserve behaviour: the comprehension is equivalent to the
generator it replaces, and the rest is whitespace.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make.md names 'make help' as the authoritative target list and the site never carried one, so a reader had to run the repository to see what it offers. The same held for the i8 aliases and for the infinito command tree.
Three generators join the ones the build already runs for the YAML index and the role glossary. make_targets reads the comment block above each target, the same block make help prints, and turns 'Param x:' into a field. aliases reads the trailing comment, falling back to the target an alias wraps, because a plain make wrapper carries no comment by repository rule; an unparseable line raises instead of vanishing from the page. cli_commands asks cli.core.discovery, the rule the dispatcher itself applies, so the page cannot promise a command the CLI rejects.
cli_commands parses each __main__.py with ast rather than importing it: rendering a page would otherwise run 65 modules' import side effects, and a broken one degrades to 'Undocumented.' instead of losing the page.
literal() escapes the inline markup starters in prose taken from those files. 3b81b4de2e fixed this class once already, where a role description reading (*.parent) opened an emphasis that never closed; a Makefile comment naming two globs does the same. Measured: the unescaped form reports three docutils warnings, the escaped form none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds a headless LibreTranslate role for callers that reach the engine over
the container network: no vhost, no reverse proxy, no MCP adapter.
The role is named `-engine` rather than `svc-ai-libretranslate` because
`get_entity_name` strips the category prefix, so that name and
`web-svc-libretranslate` both reduce to `libretranslate`. The entity name
keys the instance directory, the compose project and the container name, so
the two roles overwrote each other's rendered stack and `sys-service-loader`
resolved the engine's own `libretranslate` service key to the web role.
`provides:` cannot fix this: it renames the registry key, not the entity.
`loading_model` selects when models arrive: `lazy` (default) on first
request, `build` into the image, `task` by a post-deploy install. Under
`lazy` and `task` the models sit on a node-local bind so a rebuild reuses
them; `build` is mounted without it, because a bind is never seeded from an
image and would hide what the build baked.
Two lints guard the class:
- entity names must be distinct among roles that render a compose stack,
which `provides:` cannot satisfy and only a rename can
- a role that reaches a cached upstream another role declares must declare
it too, so the entry is discoverable from the fetch
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A container cap had to be an absolute number, and docker refuses one above
the host's core count, so "use this machine" could only be written as a
number that breaks on a smaller machine. cpus now also takes a percentage,
read against the host's raw count: 100% is every CPU and lands exactly on the
ceiling docker accepts, 50% is half of it.
The share is read against that raw count rather than against the fair-share
default, because the two CPUs the resource lookup reserves exist to bound
what a service gets when it declares nothing. A service writing a percentage
overrides that default on purpose, and subtracting the reserve a second time
would count it twice and leave no way to name the whole machine.
A bare cpus: 0 is refused from here on. Docker reads a zero as uncapped
rather than as none, so the value most likely to be typed by mistake was also
the one that removed the limit; the error names the fix. No role declared
one: all 315 cpus values in the tree are plain numbers, the smallest 0.01.
Two lookups carry the resolution so that no caller repeats it or has to know
what a percentage is read against. lookup('cpus', app, service) replaces six
copies of lookup('config', ...) piped through the resolver; those six size
worker pools and thread counts from their own cpus, and left alone a
percentage would have given libretranslate a single thread and handed the
agent broker an uncapped container. lookup('webserver_resource', key)
replaces the two blocks that took the smaller of the proxy's value and the
app's, which were the same eight lines written twice for cpus and for
pids_limit.
resolve_cpus is no longer registered as a jinja filter. Nothing referenced it
from a template once the lookups existed, and keeping it would have left two
public ways to resolve one value.
Measured: 96 tests across the filter, both lookups and the container render
templates, including that a percentage resolves before the smaller of the two
is chosen, and that pids_limit stays an integer because nginx renders it
straight into worker_connections.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The site answers 202 while it builds a version, and 200 once a build
exists. Whether that build matches what the version currently names was
never asked at the serving gate: servable() only checks that an index.html
is on disk. So the first request after a new snapshot got the previous
build with 200, and the rebuild it triggered finished into a directory
nobody was waiting for.
For a release tag that cannot happen, and for latest it is a reasonable
trade, since a visitor would rather read the previous page than a spinner.
For deployed it is wrong by definition: the name means the working tree
this run staged, so an earlier snapshot is not a stale answer but a
different question's answer. The post-deploy test that reads the published
page therefore validated the round before it, and reported a role broken
whose page had already been fixed.
The comparison already existed. _current() weighs built_ref against
_wanted_ref, which resolves deployed to the snapshot digest, and the
builder consults it to decide whether to rebuild. Only the server did not.
outdated() exposes it for deployed alone and the serving branch now treats
a mismatch as not yet servable, so the poll waits for the build it asked
for.
Measured on the stack while the previous round was live: the served page
was written 16:44:41 from the older snapshot, its own source carried the
newer content, and the correct build landed at 17:10:56, twenty-six
minutes after the test had already read the stale one and failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The docs round ended the same way twice: seven specs passed, four never ran,
and the suite was killed at its 89-minute cap after an hour and a half. Three
separate causes, each of which hid the next.
A waiter could outlive the suite that contained it. The specs budget a poll
against the role's run timeout, but Playwright is handed that timeout minus a
60-second buffer, so the global timeout fires first. When it does, Playwright
counts the in-flight test, and every test behind it, as "did not run", which
names nothing and reads like a suite that was merely slow. resolveTimeout now
caps what it returns at the budget the suite actually has, so an exhausted
waiter reports which predicate it was waiting on.
What it was waiting on could never arrive. 59e7b0a2a7 retargeted the yaml page
from /latest to /deployed, but /deployed is built from DOCS_SNAPSHOT_PATHS and
that list does not carry compose.yml, while /latest is built from a full git
archive and does. A servable version whose file does not resolve answers 302,
not 404 and not 202, and the poll rejects redirects, so it waited 88 minutes
for a page the version could not produce.
With the suite finishing, the CLI test's own failure became reachable: it reads
the published instructions from /latest, and /latest is stale. It still serves
web-app-sphinx and desk-zoom, names this repository renamed away, so ten roles
have no page there at all and web-app-docs is one of them. It now reads the
version the run just deployed, which is what a post-deploy test should replay.
Measured on the stack: /deployed/roles/web-app-docs/README.html answers 200
with the production block, /latest answers 302, and the published index lists
275 of 282 roles. The round now passes nine specs in 17.5 minutes.
Only five of the 1512 resolveTimeout call sites across 87 roles are large
enough for the cap to reach: three here and two in web-app-openwebui, whose
suite has not been re-run.
The package cache was configured in six places at once: the nginx upstream
map, the consumer compose override, pip.conf, .npmrc, the apt sources and the
twenty ensure_proxy_repo calls that bootstrap Nexus each spelled the same
hostnames, repository names and suites by hand. A new upstream meant editing
five files and hoping none was missed, and the docs told contributors to do
exactly that.
A role now declares what it needs under cache: in its meta/networks.yml, and
everything else is derived from it. svc-cache-package holds the general
upstreams; web-svc-libretranslate holds its own. All nine derived files are
generated and gitignored, so a second spelling of a hostname cannot exist.
utils.cache.hosts reads the declarations without PyYAML, because the .env
generator reaches them before any dependency is installed; it raises rather
than truncate when a comment sits at column 0 inside the block, which would
otherwise drop every declaration below it in silence. utils.cache.records
owns the Nexus record format and the Ubuntu mirror resolution, so the dev
stack and the role provision from one rendering.
A build may only resolve a cached host it can verify: the plain-HTTP hosts
are hijacked for every image, and the TLS hosts only for an image whose
Dockerfile installs the frontend CA. Hijacking all of them would fail every
pip and npm build with CERTIFICATE_VERIFY_FAILED.
svc-cache-package deploys the cache and its TLS frontend by calling the same
shell scripts the dev stack calls, rather than reimplementing them as tasks.
The role is invokable through the new svc-cache category and inert until a
host carries it.
Verified: the twenty Nexus payloads, the fifteen nginx server blocks and all
seven package-manager configs are byte-identical to the hand-written files
they replace, and the new category adds one role to the deploy order while
preserving the relative order of the other 180.
The role has not been deployed yet.
ruff reflows the guide_role signature and its two call sites, the same for
three test modules, and the YAML formatter prepends the document marker to
the machine's compose file. These files were created one commit too late to
be caught by the previous formatting pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ruff format reflows long calls and normalises quotes, isort reorders the
imports of the documentation lint package, SIM117 folds nested `with`
statements into one parenthesised block, TC003 moves typing-only imports
under `if TYPE_CHECKING`, the JavaScript linter replaces a `var`, and the
YAML formatter rewraps five role files. None of it changes behaviour.
One entry does more than that: the formatter had reflowed two calls in
test_builder.py so their `# nocheck: cache-read` markers landed on the
closing parenthesis, where they suppress nothing and the cache-read lint
would have reported two real violations. Both calls are hoisted into a
variable so the marker sits on the reading line again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The replay of a role's "### Production" block moves out of the CI guide
scripts and into the documentation role itself. The block is read off the
rendered page over HTTP, so what runs is what a reader copies, and the role
that publishes the page is also the role that proves it.
scripts/github/guide/{one,compose_deploy,host_deploy}.sh and
cli/build/docs/readme/production.py are gone with the workflow step that
called them. The instructions marker now only adds web-app-docs to the row's
apps; test-e2e-cli discovers the role through its new templates/test.env.j2
and runs files/test/test.sh.
The block runs in a throwaway machine, never on the stack host. compose.yml
starts one service built from files/test/Dockerfile, whose base is
INFINITO_PARENT_IMAGE out of the mounted checkout's .env, so the machine is
the stack host's own distribution and the replay exercises the installation
instead of a prebuilt image. files/test/init.sh adds an init when the base
carries none and keeps the full -Syu that ef0f03ea98 established for arch.
scripts/tests/workspace/base/01_install.sh then prepares the machine before
the block runs.
Nothing is skipped any more. The previous host-install branch exited 0 and
passed silently over 65 of the 178 invokable roles; a block without a
"docker run" is now replayed against systemd in that same machine, with the
git clone dropped in favour of the mounted tree under test.
The target role comes from the new guide_role lookup, which reads
list_invokable_app_ids() rather than matching role-name prefixes, and falls
back to the asking role when the round deploys nothing else. deployment.py
grows running_apps() so both lookups share one definition of what the round
deploys. required_by_coverage.role_is_invokable drops its own walk over
categories.yml and delegates to the same SPOT; measured equal across all 280
roles.
Carried in the same index, because the refactor pass ran over this change
set: library.py splits into sites.py and build_queue.py mixins, builder.py
hands translated_languages to catalogs.py and folds three repeated log-tail
and state writes, and placeholders.py becomes a facade over spans.py,
repairs.py and damage.py. No test changed for those splits, and the two
ref-resolution dicts builder.py carried are replaced by _wanted_ref, which
also stops hashing the deployed tree on every tag build.
Verified: 7132 unit and lint tests pass over the whole tree, the guide_role
lookup renders through a real Templar, the base image resolves from .env,
and compose config validates the service. The replay itself has never been
executed; that needs a deploy of web-app-docs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DEPLOYMENT_MODE was compared against a quoted literal 219 times across 110
files. IS_COMPOSE_MODE and IS_SWARM_MODE say the same thing once, and
test_deployment_mode_booleans keeps it that way: only the file that defines
them may still compare.
Both are real booleans under this ansible, measured before the swap because
the alternative is silent: a variable that templates to the string "False" is
truthy in Jinja, which is why IS_STACK_HOST is written `| bool` at every use.
These need no filter.
What had to learn the names:
* utils/annotations/task_gate.py, which several lints ask whether a task is
compose-only gated;
* four compose-template lints that recognise their gate by matching the
expression;
* the fixtures that stand in for the play's variables. They derive the
booleans from the mode now instead of carrying them, or a test that
overrides DEPLOYMENT_MODE keeps the old pair and asserts the wrong
branch.
Each replacement was checked rather than trusted: the 206 conditions that
changed in place were evaluated old against new through the Templar under
compose, swarm and host, with no disagreement, and no changed line turned out
to be anything but a condition swap. `!= 'swarm'` became `not IS_SWARM_MODE`
rather than the shorter IS_COMPOSE_MODE, because only the literal translation
stays correct should the mode ever be host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cosmos, Quick Setup and Credits follow from meta/services.yml, categories.yml
and galaxy_info.author, yet 279 role READMEs carried a copy that went stale
whenever the metadata moved and nobody re-ran make docs. The documentation
build now runs the generators itself, against its own scratch checkout, and
the committed READMEs keep only the prose a person writes.
utils.cache.PROJECT_ROOT derives from the module's location, so putting the
checkout first on PYTHONPATH makes the generator operate on the scratch tree.
i18n.py and builder.py add it; the one remaining coupling, a relative_to over
PROJECT_ROOT in a log line, is gone.
CI extracted the "### Production" block from the committed README with awk.
Stripping the section made guide_deployable() return "" for every role, which
selects no deploy row and turns nothing red: a fail-open regression. It now
asks _is_role_invokable, which matches the old condition exactly (178 roles
either way, both difference sets empty), and cli/build/docs/readme/production.py
renders the block for the two guide scripts. All 178 renders are byte-identical
to what HEAD served, and both scripts' extraction pipelines produce identical
output for a web app, a service and a desktop role.
test_readme.py inverts: the three sections are now a hard failure in a
committed README. readme_md.md documents what each is derived from.
Removed: the docs, cosmos, readme-generate, readme-index and readme-check
targets, the five aliases pointing at them, and docs from make quality.
Verified by running generate_commands() in series against a copy of the
checkout: every command exits 0, no README is left without Cosmos or Credits,
and exactly the 178 invokable roles get a Quick Setup. The ansible_roles step
needs pandoc, which only the docs image carries, so that one link of the chain
is unexercised here; it is unchanged by this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every role README carries a ```mermaid flowchart and the documentation
showed it as source. A post-transform now replaces those literal blocks with
<pre class="mermaid"> in html builds, which the runtime picks up; every other
builder keeps the fence as it was.
sphinxcontrib-mermaid injects an inline init script, which this deployment's
CSP refuses, so the runtime and its initialisation are two linked files
instead: mermaid-init.js calls mermaid.initialize with securityLevel strict
and follows the reader's colour scheme. The diagram source is html-escaped
before it is embedded, or an & or < in a node label would break the page.
The runtime is fetched at image build time through ADD --checksum, so a
changed upstream file fails the build rather than shipping. Version and
digest are not ARG defaults, which test_arg_without_default forbids; they
travel meta/services.yml -> vars/main.yml -> compose.yml.j2 the way
DOCS_IMAGE and DOCS_VERSION already do, and an update block registers the
pin with the npm version watcher instead of exempting it.
Verified by building a page with sphinx: the diagram arrives as
<pre class="mermaid"> with its source escaped, a bash fence stays a
highlighted block, and both scripts are linked. A missing mermaid.min.js does
not fail that build, so a local run without the image stays green.
scripts/lint/mermaid.sh keeps validating diagram syntax through mmdc, which
npm installs unpinned. It therefore checks against a newer mermaid than the
11.4.1 readers get, and a diagram using syntax from in between would pass the
lint and render blank.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
untranslated_markup.py and untranslated_yaml.py shared a name prefix in one
directory, which test_prefix_grouped_files reads as a folder in disguise. They
are now extensions/untranslated/markup.py and yaml.py, beside the __init__.py
that extensions/local/ already carries for the same reason.
A module named yaml.py inside a package does not shadow PyYAML, because
imports are absolute: all nine extensions conf.py registers still import, and
the docs-role unit suite stays green.
The unit tests move with them, keeping the parents[9] root derivation their
sibling package uses at the same depth.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_brands, test_titles and test_yaml_source spelled out meta/main.yml,
meta/services.yml and meta/info.yml where utils.roles.mapping already names
them. test_file_spot reports exactly that, and it did: the three files were
its only findings in a full make test.
A hard-coded path keeps pointing at nothing once a metadata file is renamed,
and a fixture that writes to the wrong place fails somewhere else than where
the mistake is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One predicate now answers what no translator may be handed, and both ends of
the pipeline read it: the extraction withholds the message from the catalog,
the client refuses to send it. What the catalogs keep back is exactly what the
client would have declined.
Two shapes qualify. A message with no word is markup. A message that is only a
name is a label: a brand, this software, a credited person, or a bare URL.
Inside a sentence the name stays protected by the span machinery and the
sentence still needs translating.
Three details each come from a finding. The comparison runs against the prose,
because **Mastodon** is not the string Mastodon and prose() already strips the
emphasis. Case is ignored, because a page writes the name the way its sentence
needs it and **mailu** is still Mailu. The address is matched first, on the raw
text, because masking protects the address of mailto:a@b.c but leaves the
scheme behind and its prose reads as a word.
Measured over the docs domain: 22745 messages fall to 22402 with zero
warnings, and 145 of the 377 removed entries carried a corrupted brand that
was being shipped - Keycloak read Schluesselmantel, OpenProject read offenes
Projekt, micro read Mikrowelle.
SOFTWARE_NAME is read from the group_vars SPOT beside SOFTWARE_REPOSITORY. It
lived there already and no python had ever read it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A table cell holding only a link has nothing to translate, yet it became a
msgid like any sentence: 2999 of the docs domain's 25742 messages, 11.7
percent. They can never be filled, so they count as pending for good and
skew every completeness figure.
The messages come from sphinx's own extract_messages, which is what the
gettext builder collects. That is not cosmetic: sphinx takes a paragraph's
rawsource, so judging astext() would read 'e2e/' out of [`e2e/`](e2e/),
find a word in it and let the markup through. A test now pins that
difference, because the first version of the test could not see it.
The predicate is has_words, the one the translation client already applies
before it sends a batch, so extraction and translation share one definition
rather than two that drift.
Measured: 25742 messages fall to 22745, no surviving msgid loses its
translation, and 2924 of the 2948 removed entries carried their own source
as their translation.
The import is guarded because the site builds arbitrary refs and one older
than utils/i18n/ cannot supply the predicate; the transform then does
nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sphinx prefers nothing: dir/index and dir/README are two independent
documents. The preference lives wherever something maps a folder onto a
page, and this repository had two such places disagreeing.
local.file_headings has always hidden README.md from a directory carrying
an index.rst, while yesterday's resolver saw README only, so a link led to
the page the sidebar hides.
Both now try index.rst, index.md, README.md, README.rst in that order. The
extension is renamed accordingly, and the tuple lives once in the lint
package rather than a third time.
That also repairs two links this branch degraded: roles/ holds an index.rst
and no README, so a README-only rule reported every reference to it as
broken and they were turned into code spans.
Three links stay code spans on purpose. Their text names something the
target is not - a suite, a file - so resolving them would restore a
misleading reference rather than a working one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
join(' ') renders KEY=a b, which the shell reads as KEY=a prefixed to the
command b: the variable is set for that command only and unset afterwards,
so the consumer's ': "${…?}"' guard kills the run over a value the
rendered file plainly shows.
dotenv_quote does not fix it. docker stack deploy passes the value verbatim
in swarm mode, so the quotes would land inside it.
Both sides move to a comma, which the shell does not split on.
The four existing cases could not have caught this: each passes a single
name or an empty string, and one element renders the same either way. The
added case is the only one that sees the separator, and removing IFS makes
it fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
apidoc now excludes roles/ and library/ the way it already excluded tests/.
A role name carries hyphens and is therefore not an importable module path,
so automodule reported 44 invalid signatures and rendered nothing; library/
holds ansible modules whose ansible.module_utils imports exist only inside a
play. Measured: 168 generated pages drop to 151, and the 17 that go consist
of a heading and automodule directives on unresolvable paths.
The role page generator fits every section underline to its title. pandoc
counts one character fewer than docutils where a heading ends in a stray
variation selector left behind by a removed emoji. It also escapes inline
markup in role metadata, because one description reads (*.parent) and opened
an emphasis that never closed.
The index generator underlined a 21-character caption with 19 characters.
conf.py adds the tel scheme, without which the phone number in SECURITY.md
is an unresolvable reference, and the extract copies the image assets into
its conf directory the way the site builder does, without which every run
reports a missing html_static_path and favicon.
Four tests pinned values this changes. The fifth, the extension count in
test_conf, asserted nothing the loop below it does not already prove and
broke on every legitimate addition; it now asserts the list is non-empty, so
that loop cannot run over nothing.
With this, make i18n-extract domain=docs finishes with zero warnings under
the -W gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Almost every role declares galaxy_info.company as a block scalar. The
generator rendered each entry as '- **key**: value', so the second line
landed in column 0 and ended the bullet list it sat in. 280 of the build's
remaining 354 docutils messages came from generated/roles/*.rst, which is
this generator's own output; folding every value onto one line clears 277
of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both extensions landed in bd15eae9fa without the mirrored unit test the
repository lint requires, which left that lint red.
The resolver's cases include the mistake its first version made: it
filtered on refdomain == 'doc', while myst builds a document reference
only when the target is_file(), so a directory arrives with refdomain
None and the transform did nothing at all. Reintroducing that filter
fails three of the six cases.
The parser's cases pin what the build relies on: one yaml-highlighted
literal block, no block quote, a title from the filename, and the orphan
docinfo as the first child, which is the only place sphinx reads metadata
from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fetch() requested latest and run_builder() requested deployed once at startup, so a release tag was built on demand and then never revisited. A tag carrying a languages.json from before the translated-site split stays servable, nothing ever requests it, translates() stays false for every code and /v1.0.0/de/ answers a hard 404 forever.
_refresh_stale_sites() asks for every version that is already servable; request() checks _current() itself, so only a genuinely stale one is queued and an unbuilt tag stays on demand instead of triggering a mass rebuild.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
request(..., background=True) skipped a language as soon as translation_servable() found an index.html, which is an existence check with no ref in it. A language was therefore built exactly once: every later commit advanced the English site while every translated site stayed frozen on the commit it was first built from. The eager loop this replaced rebuilt them all, so the freeze arrived with f5eff500c1.
_publish_site already stamps the ref beside each translated site and nothing ever read it. translation_ref() reads it and _translation_current() compares it against _wanted_ref(), which also replaces the eager {LATEST: head, DEPLOYED: snapshot_ref()} dict so an unreadable snapshot no longer kills the builder while building an unrelated version.
Two more from the same review: build_language wrote into the version's state file, so a failing language reported the whole version as failed and leaked its sphinx log into /api/versions; it now uses its own version:code key. And the language index was written after the site was published, so a failure in between left a current site with a stale index, which _current() rejects forever and fetch() would rebuild on every tick; the index is written first.
library.py had grown past the 500-line cap, so the build machinery moves into builder.py as a Builder mixin. An AST diff proves every function, method and constant is byte-identical apart from _write_json -> write_json, and the six build-focused test cases move to test_builder.py with the count unchanged at 16.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ensure() creates the agent's network and attaches the broker to it in the next
statement. Under swarm that network is swarm scoped: the control plane holds it
the moment create returns, so the GET inside connect answers, while realizing it
on a node is the allocator's asynchronous work and connect is node local. The
attach therefore ran against a network the node did not have yet and took a 404.
Run 36107421422 shows both halves. The broker logged agent_failed with 'POST
/networks/<id>/connect -> 404: network not found' for every owner, and dockerd
on the same node logged 'Failed to find network <id> on node' once per node, for
all three. Every agent request then answered 502 and the drill reported the two
agent services missing. The compose row passed throughout, which fits: a bridge
network is local and usable as soon as it exists.
connect now retries while the engine answers 404, up to realize_timeout, and
re-raises anything else at once. EngineError carries the status for that, set at
the single raise site in call, so the decision reads a number rather than
matching the message text. The wait is scoped to the one condition known to
clear by itself: a 403 from the socket proxy still fails immediately, and a 404
that outlives the window still fails, because then the network really is gone.
Three cases pin it in both directions, and each fails against the previous
engine.py: a 404 that clears is waited out, a 404 that persists raises at the
deadline, a 403 raises on the first attempt.
Not verified: no swarm run has exercised this. realize_timeout is 120 seconds
from a single observed window of at least five, and it converts a structurally
missing network from a fast failure into a slow one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role answered one contract, POST /v1/systemone, under the name of the
single implementation that happened to serve it. Renaming it to svc-ai-s1
leaves the contract where it was and lets the flavor decide who answers:
variant 0 runs laya, variant 1 runs the GLiFormer server the role carried
before, so a round proves the contract against both rather than against
whichever one ships as the default.
Reading laya 0.3.20 rather than reasoning about it corrected two defects that
would have hit variant 0 on its first deploy. LAYA_MODELS takes the short
checkpoint names, not repository ids: router.py maps english to
convaiinnovations/laya, so the entry carries both and hf download takes the
repo while the env takes the name. And laya serves @app.get("/health") where
jeff serves healthz, so the base healthcheck moves to health and variant 1
overrides it back.
services.s1.gpu states the intent and 00_core.yml resolves what the host can
actually do, requiring /dev/nvidia0 and /usr/bin/nvidia-container-runtime
before it builds a CUDA image. Both are needed: with the device alone the
daemon has no runtime to attach it with, so a GPU machine still builds the CPU
image until the toolkit arrives. The flag therefore ships true and stays inert,
and the CPU render is byte identical to the one before this commit.
templates/Dockerfile.j2 becomes files/Dockerfile plus files/shell/build.sh and
files/shell/entrypoint.sh, in the shape svc-ai-agent-broker already uses: ARGs
supplied by build.args, the context staged through render_replicated_templates.
The branching is shell rather than Jinja, so shellcheck reads it and it runs
outside a deploy. Executed with pip, apt-get, hf and python stubbed, all four
flavor and GPU combinations emit what the template emitted.
meta/info.yml gains the video that explains what a System One model is, which
retires the marker that claimed none existed.
Verified: 519 lints with 381 subtests, 196 unit tests, the four-way build.sh
execution, and the five-row truth table for the GPU gate. Not verified: no
laya image has been built and no container has run, on either deployment mode.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
build() ran sphinx once per translated language before returning, so a deploy blocked for roughly 30 minutes per language. With 29 translated catalogs German sat eighth and its site appeared after about three and a half hours, which is why the language spec polled /deployed/de/ into a 404 for its whole run. The server already answered an unbuilt version with 202 and a build request; an unbuilt language got a bare 404.
Give languages the same treatment. request() takes a code and writes a version:code marker, build_language() builds exactly one from its own checkout via the extracted _checkout(), and the server answers a translated but unbuilt language with 202 while an untranslated one keeps its 404. No language is dropped: build() now queues every translated one into queue/background/, which next_queued() drains only after the lane a visitor is waiting on, so the requested language overtakes the backlog.
languages.json grows a translated list beside known, and _language_index() reads a pre-split file as known-only so an existing volume keeps naming its languages instead of rendering an empty switcher.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The healthcheck is http_status on healthz, which answers as soon as the HTTP server binds, while the model loads on the first real question. Everything that waits on the healthcheck therefore treats jeff as ready during a window in which it cannot answer. svc-ai-litellm's router hook allows one attempt of services.jeff.request_timeout and refuses to route on any error rather than substituting a decider, so a consumer arriving in that window gets a 400 saying the system_one strategy could not reach a decision with TimeoutError: timed out. Run 36064589578 carries that twice in litellm's container log, after which the same route answered in 6.7 seconds.
01_manager_ops.yml puts one real choice question after the handler flush and repeats until it is answered, in the shape svc-ai-litellm already uses to warm its chat model: resolve the hosting node and container id, then pass the script in through python3 -c. Raising request_timeout or retrying inside the hook would only move the window, and the refusal itself is correct and stays.
The script reads the port, the alias and the key from jeff's own environment, so nothing reaches the command line that a task echo would carry. It checks that the answers block holds the question it asked, because a 200 without an answer means the model did not decide and reporting success there hands the cold load back to the first caller. The mirrored test pins that exit and pins the budget above request_timeout, since a warm-up on the router's budget would time out exactly where the router did.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The routing probe cross-checks usage.prompt_tokens against what it sent, because that is the one number in the exchange the backend produced rather than this deployment declared. A mock route answers from mock_response and never reads the prompt, and litellm sets usage unconditionally from DEFAULT_MOCK_RESPONSE_PROMPT_TOKEN_COUNT, which is 10 in the pinned main-v1.77.7-stable. Every mock answer therefore reported 10 of 8192 and failed as a silent truncation, which is what run 36064589578 hit in both litellm rows.
The exemption sits after the declared-window check, so a mock routed a prompt its own window cannot hold is still a routing failure; only the truncation number is dropped. MOCK_ALIASES comes from AI_MOCK_ALIASES rather than a second derivation of the same list.
Pinned by three cases, including the placement: moving the exemption above the window check makes test_a_mock_is_still_held_to_its_declared_window fail. The reason string of a neighbouring case claimed the opposite of what the pinned image does and is corrected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The router settings become one mapping. Nine scalars sharing a router_ prefix were a dict spelled out in key names, and the sampling section about to arrive would have added five more. Carried along: fifteen lookups in vars, three variant overrides, four test files, and the two rejection messages that name the settings path, which would otherwise have pointed a reader at a key that no longer exists.
On a configurable trigger, seconds or requests or probability, the same request goes to several eligible routes at once. jeff is shown the replies rather than the routes that wrote them, with the aliases hidden behind option labels so the verdict rests on the text. In shadow mode the caller keeps the route already decided; in serve mode the winning reply is handed back through mock_response, so the fan-out is the answer path and no further call follows.
Each comparison writes one row per offered route to the proxy's own database, grouped by a request class of token band and capability demands. Grouping finer would spread the evidence too thin to decide, coarser would let a verdict about a vision request speak for a plain chat. A route that raises is left out rather than recorded as a loss: failing to answer is a fact about the moment.
The decider now also sees the price of this request, the measured rate, the quota and prompt caching, plus how each route fared before. A figure the catalog does not state is left out rather than defaulted, because a decider cannot tell an absent number from a zero one.
Sampling ships off. Every history and comparison path is caught wide and logged: neither may ever fail a route.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three modules reimplemented what utils.cache already offers, and each paid
for it with a suppression.
test_compose_override_services_defined carried a yaml.SafeLoader subclass
whose only job was to swallow compose's own tags, plus a noqa for the
yaml.load rule it then tripped. load_yaml_any handles !override and !reset
already, so the loader and the suppression both go.
test_compose_bind_sources_exist read default.env directly and argued in a
nocheck marker that once per session is fine. Reading it through
utils.cache.files makes the argument unnecessary.
test_gpu_gate loaded the role's vars and meta files with raw yaml over
hand-assembled paths; it now uses load_yaml with ROLE_FILE_VARS_MAIN and
ROLE_FILE_META_SERVICES, the way 50a13c755b did for the playwright
multipliers.
gpu_count drops a noqa ruff no longer raises, and the development env module
an import nothing uses, which lint-python rejects.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image pulled both directions of every Argos pair because upstream's
scripts/install_models.py matches a package when from_code AND to_code are
both wanted, so a direction cannot be expressed through
--load_only_lang_codes. prefetch_models.py now replaces that script instead
of warming its cache, and takes --directions: `both` keeps the old set,
`from_source` keeps only the pairs translating out of English, which halves
the download for a deployment that never translates back.
Replacing the script means carrying its guards. It refuses a language code
no package offers and refuses an empty selection, and it hands MiniSBD the
iso2model-converted codes rather than the raw ISO ones. Without those a typo
would have built an image with fewer models, or none, and reported success.
The build reaches the package cache over TLS, which needs the frontend CA in
the image trust store. Rather than reimplement the installer in the
Dockerfile, stage scripts/docker/cache/package-frontend/ca.sh into the build
context and run it: it already picks the trust path per distribution,
rebuilds the store and skips when no CA was staged. The trust-store path now
exists in exactly one place.
That step needs root. The base image sets USER libretranslate (uid 1032) and
RUN inherits it, so update-ca-certificates fails with
cannot create /etc/ssl/certs/ca-certificates.crt.new: Permission denied
The CA layers therefore run as root and hand the user back before the model
layers, so the models stay owned by the runtime user.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `auto` alias picked a model that could not hold the prompt, and every
layer that should have caught it was comparing this deployment's own
declarations against each other.
Three errors produced one failure. smollm2:135m was declared at 16384 while
the tag trains to 8192, so Ollama clamped num_ctx and then truncated the
prompt to half the window rather than refusing it. The token estimate ran at
three characters per token and was documented as erring high, but the
oversized prompt is built from "x ", which tokenizes at two: 16384 characters
are 8193 tokens on all three served tokenizers against an estimate of 5461.
A window too large and a demand too small cancel into "eligible", and
neither correction works alone; both ship. The third error is what fired
them: at one sample the speed measurement kept its cold load, so a 135M
model measured slower than a 0.5B one and the router ranked on the
inversion. The warm-up is now discarded at every setting.
The probe compared a declared window against an estimated demand, both ours,
so it could not observe a declaration the backend does not honour. It now
reads usage.prompt_tokens and fails when the backend evaluated materially
less than was sent. Against the live stack that check separates the two
models exactly: smollm2 reports 4098 of 8192 sent and fails, llama3.2
reports 8221 and passes, and 4098 is the figure Ollama's own truncation
logged. A 400 now carries its body, without which this failure reads as
"HTTP Error 400" and nothing more.
The strategy is selectable. Where jeff is deployed the decision comes from a
System One encoder reading the prompt; otherwise the routes are ranked on
standing preferences, and each factor carries its own weight. Both were
exercised over six prompt sizes: system_one chose qwen, then smollm2, then
the mock as the prompt grew, weighted chose llama3.2:1b throughout. The
state handed to the decider is capped at 512 characters, because the encoder
holds 512 positions and the previous 20000 came from the server's own
rejection limit, which is a different number: measured through the gateway,
a 2000 character state already exceeded the decider's timeout, and one
timeout leaves the request running so every later one queues behind it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three tasks passed the gateway master key as `container exec -e
LITELLM_MK=<key>`. Ansible echoes that command line, and the masking was
written as `no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}"`, which CI and
local deploys turn off, so every run printed the key in clear. That single
pattern is where four live master keys leaked from this branch.
The key is not masked better, it is no longer passed: the gateway container
already carries LITELLM_MASTER_KEY from its own env file, and `container
exec` inherits it, so ensure_keys.py, probe_auth.py and warm_model.py read
it from the environment they run in. Verified against the running container,
whose config env carries the variable.
The consumer keys have no such home, so they move to the shell module's
stdin: the payload for ensure_keys.py and the single key probe_auth.py
revokes and restores. That takes them out of the `cmd:` echo. It does not
take them out of `module_args`, which Ansible prints when a task fails, so
those two tasks keep their masking; a literal `no_log: true` was tried first
and rejected by the repository's own lint, which requires the masking to
stay switchable for diagnosis. The two tasks that now carry no secret at all
lose their `no_log` instead, which makes a failing speed measurement or warm
call readable again.
Exercised by unit tests only. The deploy that would run these tasks needs a
variant with consumer keys, and the litellm-only round has none, so
01_keys.yml was never entered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_passthrough_swarm quoted and escaped any value containing whitespace,
which contradicts the contract stated at the top of the same module:
docker stack deploy does not strip outer quotes, so the value reaches the
container with them.
Ansible's to_json emits the default ", " separators, so AGENT_PLATFORMS
always carried spaces and arrived as "{\"hermes\": ...}". json.loads then
returned a str rather than a mapping, and sorted(PLATFORMS) in the broker
iterated its characters: the broker advertised one model per character and
never hermes or openclaw, which is why Open WebUI's model list held neither
and the agent-broker specs failed before reaching a completion. The same
encoding mangled AGENT_CONSTRAINT, whose placement expression would have
failed once an agent was reachable.
The branch existed for the opposite consumer: test.env is sourced by bash,
where KEY=one two leaves the variable unset under set -u. Quoting is the
wrong instrument for that, because one filter cannot serve both contracts
keyed on DEPLOYMENT_MODE alone. Both values that join on a space now join
on a comma and their consumers split on one, so no rendered env value needs
quoting to survive the shell.
Two unit tests asserted the quoting while a sibling in the same class
asserted that swarm does not escape a double quote; the pair could not both
hold. They are replaced by one test that renders real JSON and checks the
container receives a mapping. The lint that flags a space-joined value no
longer accepts dotenv_quote as the remedy, since in swarm it no longer
quotes, and its message names the comma instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The check read df --output=pcent and judged every filesystem. The NVIDIA container toolkit mounts a 4 KiB tmpfs per CDI hook holding exactly one file, so it sits at 100% on every GPU host, no cleanup can ever change that, and the service failed the whole deploy with it.
A filesystem now counts only from 64 MiB up, which keeps /boot and every real partition in. Smaller ones over the threshold are still reported, as INFO rather than WARNING, so a genuinely tiny filesystem does not vanish from the log; below the threshold they stay quiet. The warning also names the mountpoint now, where it used to print bare percentages that said nothing about which filesystem had filled up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pull.py ran docker compose build and pull through subprocess.run without a timeout. The ansible task keyword eventually kills the task, but the python in between kept waiting, and the handler name says "compose pull" while pull.py also builds, so the phase the deploy sits in is not what the log claims. A first libretranslate build downloading every Argos model outlived the task timeout, and each retry restarted the RUN layer from zero.
run_cmd now passes timeout=RUN_TIMEOUT_SECONDS and turns TimeoutExpired into rc 124 plus whatever the command had already written, which every caller already treats as a failure. The matrix compose path carried no task timeout at all: its compose up has retries: 12, so a hang would never have triggered a single one of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first deploy that measured generation rates gave mock/deterministic
4148 tok/s against 70.7 for the fastest real model. That number is the
proxy's echo latency, not a rate: a mock returns its declared string
without generating anything. Being local, free and 59 times faster, it
then won every routing decision, so the alias answered pong to any
prompt and the routing probe would have gone green while proving
nothing.
The exclude list is derived rather than maintained: it is the router's
own alias plus every alias whose provider is the mock provider, so a mock
declared in services.yml or in a variant is covered without a second
place to update.
LITELLM_EXCLUDE also becomes a set. It held a single alias and was
compared with `not in`, which on a string is a substring test, so a
comma-joined list would have silently unranked any model id contained in
it.
A mock stays routable. With nothing preloaded it is the only model the
gateway serves and has to win; it simply no longer wins on a number that
is not a model's. Against the rates this deploy measured, the alias moves
from mock/deterministic back to qwen2.5:0.5b.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role declares `gpu: true`, which resource.yml.j2 turns into a nvidia
runtime, and appends `-cuda` to the image tag so the container carries the
CUDA libraries the runtime alone does not provide. Both hang off
LIBRETRANSLATE_GPU, read through resource_filter: lookup('config', ...)
returned false for the same declaration and shipped a CPU image twice.
assert_gpu.yml asks CTranslate2 for its CUDA device count after the compose
handlers flush and fails on zero. Without it the container stays healthy and
answers every request while falling back to the CPU, which is how a broken
CUDA stack survived several deploys unnoticed. It is gated on the host
carrying /dev/nvidia0 and on IS_STACK_HOST, since the exec helper exits 64 on
swarm workers.
load_only moves to the variants so CI stays slim while a production deploy
loads every language. An empty list now drops the flag instead of passing an
empty one, whose meaning is ambiguous.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
resource.yml.j2 renders `runtime: nvidia` for any service that declares
`gpu: true`, gated a second time on the host carrying /dev/nvidia0 so a
CPU-only host never gets a runtime its daemon cannot resolve. The device
fact is probed in daemon_config.yml next to the kata and gVisor shims, and
daemon.json.j2 lists nvidia alongside them. nvidia stays out of
SANDBOX_RUNTIME: it passes a device through, it does not isolate.
nvidia.yml drops rival apt sources for libnvidia-container before writing
its own. The other order leaves two sources with different Signed-By keys
for one URL, which makes apt refuse the entire source list.
test_gpu_runtime.py covers both polarities and the missing device fact; the
latter caught `x.stat.exists | default(false)`, which raises on the
undefined object before the filter runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The router already ranked routes on traits.speed, but nothing ever set
one. No catalogue carries a rate, because a rate is a property of this
machine under this load rather than of the model, and a hand-written
number in services.yml would be a guess that rots.
The deploy now measures it. After the gateway is warm, measure_speed.py
asks it which models it serves and sends router_speed_samples requests to
each model that has no rate yet, discarding the first as a cold load and
taking the median of the rest. The result is stored on the stack host and
rendered back into model_info.traits.speed on the same deploy, so the
gateway restarts once with the numbers it just produced. The stored file
is read back with slurp rather than lookup('file'), because the config
directory lives on the stack host and a lookup reads the controller.
A model that already carries a rate sends no request at all. Re-measuring
everything every deploy would produce results the tolerance discards anyway,
and that traffic is the entire live surface of this step: what can hang,
what can fail the deploy, and what the deploy waits for.
router_speed_remeasure discards the stored numbers once, for the case
where the hardware under them changed.
Two budgets bound a run. A measured request may take
router_speed_timeout; the discarded first request gets upstream_timeout
instead, because it pays the model load and holding it to the sample
budget would time out exactly the models worth measuring. Exceeding a
budget abandons that model, not the deploy, and the model keeps whatever
rate it had.
Every matrix variant overrides the sample count to 1, where the single
sample is kept rather than discarded: a CI round proves that the
measurement, the store and the re-render ran, not what the rate was on a
shared runner.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The router ranked surviving routes lexicographically: local, then price,
then speed, then alias. How much a factor counted was its position in
that tuple, so a deployment could not say that it cares about price more
than about speed.
Each factor now carries a 0..1 weight in services.litellm.router_weights.
A factor is scaled to 0..1 inside the candidate set before its weight
applies, because the raw units do not compare: a per-token price is a
millionth of a cent and a rate is tens of tokens per second. A factor
whose value is identical across the set scores every route 1 and drops
out, rather than deciding on a rounding artefact.
Confidentiality stays out of the score. local_reason removes every remote
candidate before decide() runs, so no weighting can send a request that
must stay local to a third party; a weight is a preference, and a
credential leaving the cluster is not something a preference should be
able to buy. The shipped weights additionally keep locality worth at
least cost and speed combined, and a test reads them out of
meta/services.yml rather than repeating them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
