meta/languages.yml carried `libretranslate: true|false` per entry, which
answered two different questions with one field: which languages the
repository translates, and which engine renders them. The first is the
table's business; the second is not. The gateway offers every backend
that serves a pair, falls through when one refuses or mangles the
answer, drops an engine whose failures pile up, and settles a request
class on the engine that wins its comparisons - and `signature()` is
"{source}:{target}:{length bucket}:{markup|plain}", so that learning is
already per target language. A static column beside it is a second
opinion that goes stale the moment the router learns something.
So the field is gone, 184 lines, and the table now holds name, endonym
and direction. Every language it names carries a catalog in every
domain.
What one engine can do is declared with that engine instead:
services.libretranslate-engine.serves, 47 codes. Not a detour - both
readers run before any engine answers. The lane prefetches one model per
code, and the readiness probe is itself the thing that would have asked.
`translatable()` split along that seam. Catalog-scoped callers take
`domain_languages()`, engine-scoped ones take the new
`served_languages()`: the CI matrix, extraction and translation the
first, the model prefetch and the tuning sweep the second. Collapsing
the two is what this change is guarding against - `baked_languages()`
renders identically before and after, verified by diffing both outputs.
`translate()` also stops dying on a language whose catalog has not been
extracted yet. It reads a missing catalog as nothing pending rather than
raising, because 136 docs catalogs do not exist until the next
extraction and refusing those languages would quietly shrink the set the
repository promises to translate.
The oracle suite follows: tests/oracle/test_translation_score.py asks
utils/i18n/judge.py about every language instead of routing a laya
checkpoint per language. It cannot run yet - the tools lane carries no
LiteLLM - and its skip message says exactly that.
test_catalogs_complete_on_gpu now measures against the table rather than
one engine's coverage. It is live on a GPU host and will report the 136
missing catalogs as the gap they are, which is the point: narrowing it
to the engine would hide exactly the languages worth watching.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The oracle suite needs a verdict in every language meta/languages.yml
names, and laya cannot give one. Measured against the suite's fixture,
its english checkpoint separates a faithful rendering from an unrelated
text in 17 of 47 languages and its multilingual checkpoint in none,
answering around 1.5 either way. One judge for all of them beats a route
per language that is noise in most of them.
The question keeps the shape the checkpoint answered: the levels are
written out and the model names an index over them, never a bare
probability. A probability carries this deployment's calibration, and a
fixed threshold on it would encode that calibration into the suite;
"below the middle level" instead means what the middle level says.
Three refusals are deliberate. temperature 0, so the same comparison
does not come out differently on a rerun. An answer naming no index
raises instead of scoring zero, which would read a refusal as "states
something else". An index outside the scale raises instead of being
clamped, because a model answering 7 has not placed the pair on this
scale at all.
The request goes to the model gateway rather than to a model server, so
it carries a consumer identity and resolves through the gateway's model
list like every other model call here.
Nothing imports this yet: the oracle suite moves to it once the tools
lane carries LiteLLM and a loaded model, which is what lets the judge be
exercised against a real one for the first time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5