write_catalog writes the rendered catalogue to .<name>.po.tmp and moves it
over the target. When the write fails, the staging file stayed behind: the
translation run that hit a full disk left eleven empty .docs.po.tmp files
untracked in locale/. Any failure between the write and the move now
deletes the staging file and re-raises, so the target keeps its old bytes
and nothing is left in the tree.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One string that every engine refused or mangled raised out of the batch route's gather, so the gateway answered the whole batch with a 422 and the client lost the other nineteen translations. Gateway.translate_each now owns the bounded fan-out; with partial=true it answers (None, None) for a string that ran into a GatewayError and keeps the rest. Non-gateway exceptions still raise, and a NoBackendError on every string still fails the batch as the outage it is.
The i18n client asks for partial batches and counts a null answer as a refused entry with its reason, so the re-ask and damage checks continue per string. A caller that does not send the flag gets the old behaviour.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PROTECTED masked identifiers with an underscore and bare numbers, so S3, OAuth2, POP3 and HTML5 went to the engines as a word plus a separately protected digit. Engines tore them ('Amazon S ilə uyğun ...' for 'S3-compatible') or glued the digit onto another word ('Distribuid3-compatible'); the masking audit over core found 159 such entries.
A letter-digit token is now one protected span, ahead of the bare-number alternative. The client masks it whole, and damage.reason() condemns a translation that lost or moved it, so prune and retranslation pick those entries up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chat engine prefixed the Vietnamese catalog with its own preamble ("...Vietnamese:" followed by a blank line) and wrote Urdu in Devanagari. Neither tripped the damage predicate: vi is Latin script, so the script checks were blind, and untranslated only measures the Latin share, so a non-Latin language answered in another non-Latin script passed.
added_lines refuses a translation that breaks more lines than its source, and the gateway's _mangled carries the same check so such an answer falls through to the next engine instead of being cached. wrong_script refuses a non-Latin language when less than half of the letters it writes in another language's endonym script belong to its own; the script per language is derived from the endonym in meta/languages.yml, and letters are counted over runs of letters plus combining marks so Indic and vocalised Arabic words stay whole.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gateway routes every request on its own, so a catalog holds the work of several engines, while the mark apply() wrote was the constant translated-by: libretranslate and attributed all of it to one backend. The catalogs on disk show the drift: 1021537 entries carry that constant and 13 carry ollama, written once the gateway began answering from more than one place.
The constant becomes machine_translation(engine), which names what the gateway reported and writes unnamed where it reported nothing, plus is_machine_translation(), which recognises any of them by prefix so every mark already on disk keeps matching. Outcome carries an engines tuple aligned with its values, merge pads it, and _reask returns the name alongside the text: a successful re-ask records the engine that answered, not the one whose rejection it replaced, because the catalog does not carry that rejected text.
apply() now strips an earlier mark before stamping the current one. clear() removes only refusal prefixes, so an entry re-translated after i18n-prune would otherwise accumulate one mark per engine and the record would be ambiguous again, which is the defect this change exists to remove.
This is the half that makes the attribution true. Recording latency in the decider history is the other half; together they let the router weigh quality against speed per engine, which a catalog claiming one engine everywhere cannot support.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A link whose label is its own target's filename is a path, not prose: [11_oidc.yml](../../group_vars/all/11_oidc.yml) comes back with a translated filename. The markup rule lets it through because the extension contributes letters, so it needs its own shape. A label that says something about the target keeps its translation, which is what separates it from [the OIDC defaults](../../group_vars/all/11_oidc.yml).
A message that is one technical token reads the same everywhere: an abbreviation in capitals, a version, or a run of command-line options. Together with the link rule this withholds 102 messages the catalogs used to send, 34 tokens and 68 changelog links, every one of them read before it landed here.
Known limit: ABBREVIATION matches any capitalised word of three or more characters, so it reaches messages that do not exist yet. MUST, TODO and TODOS are already words rather than abbreviations, and a future UI label like ERROR would be withheld the same way. The failure mode is silent, since a wrongly withheld message simply stays English without any test going red. Narrowing it to tokens the repository also uses as identifiers is the open follow-up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
untranslated() asks what share of a translation's words are Latin. WORDS matched runs of letters, and Han, Kana, Thai, Lao, Khmer and Myanmar write without separators, so a whole sentence came back as one token. Two Latin product names inside it carried that token's latin_share past a half, which made the ratio 1.0 and condemned a sound translation.
Measured on a Japanese rendering of "Runs Playwright in Docker with stable browser settings": one token, latin_share 0.516, ratio 1.00 against a threshold of 0.9. The six-word floor removed in 27d59f4bd4 had hidden this, because a separator-less script never reaches six words; dropping it as vacuous missed that it was the only guard those scripts had.
WORDS now takes one character of such a script as one word, with the single-character branch first so the run branch cannot swallow the sentence again. Ten real entries against both patterns: the old one errs on two, the new one on none, and every true positive survives, including Cyrillic welds like CI Деbugging and an English answer in a Japanese catalog.
The ranges are codepoints because a formatter rewrites an escape into the character it names, and the resulting literal blob cannot be reviewed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A deploy job name repeats the category path on every row of the matrix. One Egyptian hieroglyph costs four UTF-8 bytes where the two Chinese characters it replaces cost six. Unicode names each sign by its Gardiner code, so the opaque value stays identifiable without a comment per entry.
A label has to stay one codepoint: a sign written with a variation selector costs seven bytes and is worse than what it replaced. tests/integration/meta/categories/test_glyph.py enforces that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
27d59f4bd4 caught a non-Latin target answered in Latin letters. Its
mirror was missing, so the model answering a language it cannot write
with one it can passed every criterion:
ch 'AI Services' -> '人工智能服务' Chamorro, answered in Han
tr 'Sales' -> 'إدارة'
ae (76 entries) -> Arabic, Persian, Hebrew, Chinese
Chamorro writes in the Latin script and lives on Guam; the code 'ch'
reads as Chinese to a model that does not know the language, which is
how the whole catalog came back in Mandarin.
Measured over the 121 Latin-script catalogs before the rule was
written: 79 entries condemned, 76 of them in ae and 3 in tr. All three
Turkish ones were read individually and none is a false positive; two
are an Arabic fragment with a markdown rule and one is English plus
Bengali repeated four times.
Its own criterion rather than an arm of untranslated(): the catalogs
record the reason on the entry, and 'answered in another script' is a
different diagnosis from 'came back untranslated'. The two together
cover the script dimension for all 184 languages.
What it still cannot see: Spanish served as Corsican. Telling two
Latin-script languages apart needs a detector this repository does not
have, so that class stays open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
untranslated() is the one criterion that knows a catalog's script, and it
never fired. It returned False below LATIN_FLOOR = 6 words, while the
catalogs are mostly one- to four-word UI labels, so the guard sat above
its own corpus.
Measured across the 62 non-Latin catalogs: the floor condemns 0 entries.
Without it, 536, concentrated in the languages the model cannot write:
av 99 as 92 ba 87 be 84 ab 84 am 56 bh 13
Every entry in the low-count tail was read before the floor was removed,
because that tail is where a false positive would live. There is none:
ko 'Applications' -> a refusal sentence from the model
zh 'Updates & Package Mgmt' -> the string plus a version changelog
fa 'Cloud' -> 'Cloud' and a markdown rule
ab 'Communication' -> 'Comunicacion'
A word count was the wrong instrument. reason() consults this only after
an exact echo is ruled out, so for a non-Latin target, Latin text that
DIFFERS from the source cannot be a term the language keeps in English.
That distinction is what an earlier attempt at floor=2 lacked, which is
why it condemned 4053 entries including legitimate ones.
Scope, stated because the numbers invite a larger reading: this guards
the 62 non-Latin catalogs only. For the ~122 Latin-script languages a
wrong-language answer still passes, and catching those needs a detector
this repository does not have.
LATIN_FLOOR and its re-export are deleted rather than left at 1; nothing
else consumed them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
reason() calls itself the single source of truth for both ends of the
pipeline, and the write path honoured that: client.py:228 passes the
directory code into unmask(). Only damaged() read catalog.locale, so
for 50 catalogs the two ends judged against different languages.
44 catalogs carry no locale header, 8 of them non-Latin and therefore
silently past the script check. 6 more name a code their directory
disagrees with, so tl was judged as fil, bh as bho, tw as ak.
The language becomes a required parameter; both production callers
already held the code. Measured over every catalog, header language
against path language, one moves: core/av from 0 to 29. Those 29 hold
pseudo-Norwegian and a degeneration loop in a Cyrillic catalog, and
make i18n-prune has since cleared them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make i18n-translate addressed the engine directly, so a run could never reach more than the 47 languages Argos ships models for, while meta/languages.yml names 183. service_url now resolves web-svc-translate, which routes a pair the engine does not serve to a chat model, and the deploy brings both roles up.
The gateway advertised only what its backends declared, and the chat backend declared nothing, so /languages reported 51 targets of 183 and the readiness wait spun to its hour-long timeout without translating a line. The chat backend now declares every language the table names.
Routing offered libretranslate for every pair, because an undeclared catalogue read as any pair, so a language it holds no model for was answered with HTTP 400 instead of falling through to the next backend. serves() now reads the backend's own /languages, which catalogue() already exposed for the advertisement. Declaring the pairs in the role was the first attempt and carried a second copy that was already stale: the engine reports zh-Hans where the role's list says zh.
A small model copies the prompt markers back, echoes the source line and labels its answer with the target language. All three reached the catalog, and an entry carrying its own source scores below the oracle's middle level. clean() strips them, and only from an answer that carried scaffolding, because on its own a short word and a colon is a translation, not a label.
server() had lost its redeploy parameter while cli/build/i18n kept passing it, so every run raised TypeError before translating a line and the only symptom was an empty log.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/languages.yml carried `libretranslate: true|false` per entry, which
answered two different questions with one field: which languages the
repository translates, and which engine renders them. The first is the
table's business; the second is not. The gateway offers every backend
that serves a pair, falls through when one refuses or mangles the
answer, drops an engine whose failures pile up, and settles a request
class on the engine that wins its comparisons - and `signature()` is
"{source}:{target}:{length bucket}:{markup|plain}", so that learning is
already per target language. A static column beside it is a second
opinion that goes stale the moment the router learns something.
So the field is gone, 184 lines, and the table now holds name, endonym
and direction. Every language it names carries a catalog in every
domain.
What one engine can do is declared with that engine instead:
services.libretranslate-engine.serves, 47 codes. Not a detour - both
readers run before any engine answers. The lane prefetches one model per
code, and the readiness probe is itself the thing that would have asked.
`translatable()` split along that seam. Catalog-scoped callers take
`domain_languages()`, engine-scoped ones take the new
`served_languages()`: the CI matrix, extraction and translation the
first, the model prefetch and the tuning sweep the second. Collapsing
the two is what this change is guarding against - `baked_languages()`
renders identically before and after, verified by diffing both outputs.
`translate()` also stops dying on a language whose catalog has not been
extracted yet. It reads a missing catalog as nothing pending rather than
raising, because 136 docs catalogs do not exist until the next
extraction and refusing those languages would quietly shrink the set the
repository promises to translate.
The oracle suite follows: tests/oracle/test_translation_score.py asks
utils/i18n/judge.py about every language instead of routing a laya
checkpoint per language. It cannot run yet - the tools lane carries no
LiteLLM - and its skip message says exactly that.
test_catalogs_complete_on_gpu now measures against the table rather than
one engine's coverage. It is live on a GPU host and will report the 136
missing catalogs as the gap they are, which is the point: narrowing it
to the engine would hide exactly the languages worth watching.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The oracle suite needs a verdict in every language meta/languages.yml
names, and laya cannot give one. Measured against the suite's fixture,
its english checkpoint separates a faithful rendering from an unrelated
text in 17 of 47 languages and its multilingual checkpoint in none,
answering around 1.5 either way. One judge for all of them beats a route
per language that is noise in most of them.
The question keeps the shape the checkpoint answered: the levels are
written out and the model names an index over them, never a bare
probability. A probability carries this deployment's calibration, and a
fixed threshold on it would encode that calibration into the suite;
"below the middle level" instead means what the middle level says.
Three refusals are deliberate. temperature 0, so the same comparison
does not come out differently on a rerun. An answer naming no index
raises instead of scoring zero, which would read a refusal as "states
something else". An index outside the scale raises instead of being
clamped, because a model answering 7 has not placed the pair on this
scale at all.
The request goes to the model gateway rather than to a model server, so
it carries a consumer identity and resolves through the gateway's model
list like every other model call here.
Nothing imports this yet: the oracle suite moves to it once the tools
lane carries LiteLLM and a loaded model, which is what lets the judge be
exercised against a real one for the first time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway answered two sources with "<source>\n---\n<rendering>" and the
whole answer was stored, so twenty languages shipped the English text as
their own translation and the release gate stayed green. Every criterion
passed it: echo compares for equality and the answer is longer, the markup
comparison counts only []{}`*() so a rule of hyphens is invisible to it, and
the rendering the server appended keeps the spans, the names and the length.
echoes() reports the same "echo" criterion when the translation contains its
whole source, under the floor the equality case already uses: a short source
may legitimately survive inside a longer rendering, a sentence may not.
reason() is the single source of truth for both ends, so the client now drops
such an answer on arrival and the gate reports one that got in earlier; make
i18n-prune clears those.
Swept every core catalog: exactly the 21 damaged entries are reported, all as
echo, and nothing else in the corpus. The old predicate returns no reason at
all for the same input.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
666312dfbd settled the catalog client on the dedicated LibreTranslate. One
engine cannot be compared against itself, so the direct path had no way to
improve on a language it translates badly: of the 2854 entries of the `ko`
catalog it produced 1 translated entry and 2704 `structure` rejections.
The client now asks the gateway, which fronts several engines and keeps a
history of which one survives a given signature. Protected spans are masked
as `<x id="N"></x>` before the request and declared per item, so the gateway
can reject an answer that altered them instead of passing it on. `_reask`
then retries each rejected entry with the engine that produced it in
`exclude`, which reaches a different backend rather than the same one. The
same catalog measured 17 translated entries with 2649 rejections once the
gateway checked the markup, and 331 with 796 once the retry could exclude.
The full picture a rejection needs is only available in the client: 2431 of
those 2649 structure rejections involve markup the masking hides from the
gateway, so the engine-switching decision belongs on this side of the call.
`server()` reuses a gateway that already answers and otherwise deploys the
bundle the gateway needs. It does not reintroduce the throwaway container
666312dfbd removed: there is no `docker run` here, the roles deploy through
the normal path and their models still come from the role images built
through the package cache. `redeploy=true` is offered because the reuse check
only asks whether a gateway answers, which cannot see that the gateway's own
code changed since it started. A run refuses outright while a deploy is in
progress, because both touch the same container stack, and leaves the deploy
alone.
`deployed()` additionally requires a non-empty catalogue, so a gateway that
answers while serving no pairs counts as not ready rather than as a backend
that refuses everything. `probe(url)` is split out of `deployed(root)` so one
readiness check serves a single engine and the gateway alike.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The expectation in test_a_rejected_translation_never_spills_over_its_comment_line encoded the old mechanism, an escaped newline, which is what introduced the backslash that broke docs/tr and docs/zh. The invariant its name states still holds with a space, so the expectation moves and the fix stays.
A second test asserts the property that actually prevents the defect: a sample containing [test\\_file.py] leaves no backslash in the comment, whatever babel does when it wraps it. Nothing guarded that before, which is why two unparseable catalogs only surfaced under a manual msgfmt run.
make test-unit passes in the container: 7171 passed, 9 skipped, no failures.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The span check compared multisets, so a translation was refused whenever its
set of protected spans differed from the source's in either direction. But
`PROTECTED` ends in two very low-precision alternatives, `\d+(?:[.,]\d+)*` and
the bare markup characters, which claim every digit run and every parenthesis.
Over-claiming is right when masking, because nothing may leak to the engine
unprotected. As a comparison basis it is wrong: the translation is another
language whose ordinary prose trips those same rules, and a gained parenthesis
destroys nothing the source carried. The module's own contract says "whether a
translation broke something its source carried", which is a subset relation.
So the loss half keeps the name and the surplus half gets its own. `added_spans`
refuses only a surplus whose presence is itself damage: an injected `%s` that
`msgfmt` rejects, a decoder artefact like `♪`, a leaked `<unk>`, an invented
`{placeholder}`. Measured over representative shapes, all four of those stay
refused, a gained parenthesis stays refused through the existing `structure`
check, and what becomes admitted is a surplus numeral or an abbreviation like
the Danish `f.eks`.
The gate moves in the same commit, and it has to. fe80a23725 introduced one
predicate for both ends precisely because a writer and a gate that disagree
never converge: the run writes, the gate condemns, `i18n-prune` empties, the
next run writes the same entries back. That commit did not quite finish the job
- `test_i18n_catalogs.py` still duplicates the comparison rather than calling
the shared predicate - so the duplicate is mirrored here. Making it call
`reason()` outright would be better and is left for its own change.
No catalog entry changes here. The refusal markers are all still in place, so
the entries this releases stay empty until a translate run fills them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`truncated` compared character counts against a flat 0.5 for every language. A
logographic script carries a morpheme per character, so a complete Chinese
rendering of an English passage runs at roughly a third of its length and the
predicate read it as a dropped clause.
Measured against the deployed LibreTranslate, re-translating 15 refused sources
per language: zh 15 of 15 refused, ja 11 of 15, ko 10 of 15, while 44 of those
45 ended in terminal punctuation, so they were complete sentences. The ratios
clustered between 0.28 and 0.55; a real truncation scatters and stops
mid-sentence. 0.25 sits below every complete rendering measured and still
catches a passage cut to a quarter.
The language set is measured, not guessed: zh, ko and ja account for 5553 of
the 6135 truncation refusals, 90.5%. The next language down is at 116, small
enough to be genuine truncation, so it keeps the flat ratio.
`collapsed` already reasons this way for the same reason, its docstring naming
Chinese explicitly; `truncated` never got the same treatment. `reason` already
carried the language for `untranslated` and now passes it on.
Effect on the catalogs, from the run rather than a test: zh 4153 truncation
refusals to 20, ko 816 to 0, ja 584 to 0, and 5590 entries that had been stuck
are now translated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every commit here went in with `--no-verify`, so the pre-commit `make test`
never ran and neither the formatter nor the linter saw the change sets. The
quality gate collects that debt now.
`test_aliases.py` built a lookup table by handing a generator to `dict()`,
which ruff C402 rejects and offers no autofix for; a dict comprehension does
the same work. `selections.py` and `test_translate.py` each wrapped a call
across three lines that fits on one, which is the formatter's own rewrite
from the run that then aborted on the C402 above.
All three edits preserve behaviour: the comprehension is equivalent to the
generator it replaces, and the rest is whitespace.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
apply() already refuses a translation another entry holds, on the same floor the release gate reads, so within one process the second sharer is discarded. Four entries of docs/vi nevertheless shared one msgstr: one_catalog reads the catalog once, keeps what it has spent in memory and writes the whole file back after every chunk, so two concurrent runs each hold a private view and overwrite each other.
exclusive() takes an flock on build/i18n/<domain>-<code>.lock around the whole read-modify-write. It tries the lock non-blocking first so a second run can say it is waiting instead of looking hung. The lock files sit under build/, which .gitignore already excludes.
The damage could not be repaired either: prune ran damaged(), which reads one entry at a time, and a collapse is only visible across entries - each one is individually sound. over_shared() groups the machine translations by what they say and returns every sharer beyond MAX_SHARING_SOURCES. prune clears those first, so damaged() no longer sees them and the printed counts cannot overlap. Measured on the corpus: 10 in docs/vi, 132 overall.
MAX_SHARING_SOURCES lands in damage.py next to COLLAPSE_FLOOR and the gate imports both from there. fe80a23725 split a criterion across the two ends once already and the run never converged: every pass wrote what the gate condemned and prune emptied it again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
What is wrong with the answers the collapse check reports is not their
shape. A label is caught by its length, but `sq` shares a translation with
four sources at 0.84 of their length, `ga` at 0.73 and `he` at 0.62: full
renderings, spent twice. No length rule separates those from a language
that is simply terse, because the fault is that another entry already says
it, which a predicate reading one entry cannot see.
The writer now carries the translations a catalog already spends on a
source long enough to deserve its own, and turns down a repeat. It lives
where the whole catalog is in hand; `apply` alone sees one chunk. Both
arguments default to None, so a caller without the catalog behaves as
before.
The floor is now inclusive, matching the check it mirrors. `az` collapsed
on sources of exactly forty characters, which the check counts and the
first version of this guard skipped.
`criteria` is threaded through so the run names what it refused. A
duplicate is turned down here rather than in the client, and the summary
reported the discard with no reason for it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A server that gives up on a passage returns a word: four distinct sources
of forty characters and more all came back as `Name`, and the catalogs kept
it. Every predicate passed, because a label carries no protected span, no
name, no markup and no stutter, and `truncated` does not look below 120
characters, which is where these sources end.
`collapsed` covers that band. Its ratio is a third of the truncation one so
a language that genuinely says the same in fewer characters survives:
Chinese renders a 58-character source in 17, a fifth of it, while the label
that prompted the rule is a fifteenth. It joins `reason`, so the client
drops such an answer and the release gate reports one that got in anyway.
This does not close the whole gap the collapse check polices. An answer
that is short but not a fragment, `E-poçt ünvanı` for several distinct
sources, sits between this ratio and a legitimate Chinese rendering; no
length rule separates them, because what is wrong with it is that another
entry already has it. That needs the writer to carry per-catalog state.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
load_only reaches the image as a build ARG, and the RUN that fetches the
argos models is keyed on it. Pinning it to the languages a run happens to
translate therefore changed the ARG on every batch, invalidated that layer,
and refetched every model it names. Ten batches meant ten full rebuilds,
each one downloading what the previous had already downloaded.
Accumulating the languages instead would have been worse, not better: the
layer is all or nothing, so a growing set refetches the whole set each
time. What works is the opposite of following the batch. The lane now pins
load_only to every translatable code of both domains plus the pivot, 48 in
total, and asks for from_source, which drops the reverse pairs a lane that
only ever translates out of English never requests. The ARG is constant, so
the layer is built once and every later batch reuses it.
codes no longer reaches the rendered vars, which left it unused in deploy()
and server(); both lose it, along with the two call sites. That undoes
44260b8786, whose subject was passing it to a parameter this removes.
Correction to 4f2c9140fa: that message said an empty load_only makes the
image skip prefetch_models.py. It does not. The script always runs, and an
empty value only drops its --load_only_lang_codes filter, which is what
made it reach for every package. The consequence it described was real, the
mechanism was not.
Measured: after the one rebuild the server offers all 48 languages, and the
three batches since ran with no ansible task at all in their logs, going
straight to translating 12,518 entries across nineteen catalogs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A batch that named its languages still deployed a server that knew none of
them. The role renders load_only from the inventory, the tracked i18n.yml
leaves it empty, and an empty load_only makes the image skip
prefetch_models.py. Nothing was baked behind the package cache, so the
server reached for the whole argos set at runtime over a path no cache
serves. The readiness wait then sat for its full hour and gave up with
"does not offer", which reads like a server that refuses the language
rather than one still downloading it. The lane now renders its own host
vars with load_only pinned to the pivot plus the batch, beside the tracked
file so the container sees it, and deploy and server carry the codes that
decide it.
The span guard could not see a hyphen. Its dotted-name alternative matched
[A-Za-z]\w*(\.[A-Za-z]\w+)+, and \w excludes -, so a host like argos-net.com
was never a protected span: a translation was free to mangle it and the
comparison still balanced. Widening the head to [\w-]* brings those names
under the same guard the undotted ones already had.
A refusal named its criterion and nothing else. "protected-span" says an
entry was turned down without saying which span moved, which is the one
fact that separates a bad translation from a criterion that is too strict,
so the marker now carries lost= and gained= and the per-catalog summary
counts the criteria it saw instead of reporting every rejection as a
damaged span.
Verified in the container against babel 2.18.0: 171 unit tests pass, and
the catalog lint holds across all 218 catalogs under the widened regex,
which is the gate that would have caught a span the old pattern let through.
The rendered vars file pins de, el, en, et, hi and lv, and those catalogs
carry their translations.
make i18n-tune still calls server with one argument and is fixed separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
server() preferred the deployed service and fell back to a throwaway
container when it did not answer. That container ran the upstream image
directly, with LT_UPDATE_MODELS on and neither the package-cache CA nor
the argos-net.com hijack, so it refetched gigabytes from upstream on
every run while the deployed service already carries those models in its
image, installed through the cache at build time. The fallback is gone.
It was reached by accident rather than by failure. deploy() ended in
await_service(), which polled for 300 seconds; the first start of the
service installs its models and takes far longer, so the wait expired,
deploy() reported nothing running, and the caller spawned the throwaway
beside the service it had just deployed successfully. await_service and
SERVICE_TIMEOUT_SECONDS are removed, leaving the single readiness wait
the client already performs with the budget those models need.
Step 0 of the deploy also stopped forcing a recreate. It rebuilt a
healthy lane on every invocation and took whatever model install was in
flight with it; compose recreates on a changed configuration by itself.
The wait no longer looks like a hang: it reports how many of the wanted
languages the server offers, which ones are still missing, how long it
has waited against its budget, and what the lane has pulled. i18n-tune
waits the same way and gets the same report.
service_url() and lane_container() replace the two places that each
rebuilt one of those strings, and deploy() reads the environment once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test started a router and asked `deploying()` about it in the next
statement. `/proc/<pid>/cmdline` is empty until the child reaches execve, so
the guard could not find `deploy/main.sh` and answered, correctly, that no
deploy is running. A probe repeating exactly this sequence read an empty
command line 376 times out of 400, which is why the case fails under CI load
and passes on an idle machine.
The router now announces its first line and the test reads it before it
looks at the pid. 30 consecutive runs of the case pass.
The guard itself is unchanged: in production the pid file is written by the
router, long after its own exec, so nothing there races.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A bare mark told an operator that an entry is unanswerable without saying
what came back for it, so the verdict could not be judged and the only way
to see the criterion fire was to run the translator again. The rejection now
travels with its text: `unmask` returns a `Rejected` carrying the restored
translation and the criterion it broke, and the entry records both as
comments beside the mark.
# translation-refused: libretranslate structure
# nocheck: url
# translation-rejected: Siehe [den Leitfaden]](https://gone.invalid/a)
msgid "See [the guide](https://gone.invalid/a) first."
msgstr ""
The quote is a comment and never a msgstr, because it is exactly what the
consumers must not use.
`harms` keeps being the predicate both ends of the pipeline ask, but it is
now the boolean face of `reason`, which names the criterion instead of
folding every check into one `or` chain. The token failures `unknown-token`,
`lost-token` and `empty` join the criteria the catalog can report.
Two details are load-bearing. The `nocheck` sits on a line of its own above
the quote: babel wraps a long comment, and appended behind the text the
marker landed one line below the URL it has to cover, where the block-above
resolver no longer sees it. And the cleanup drops a `nocheck` only where a
quote follows it, because Sphinx carries a source comment into the catalog
as a user comment and removing one would let the URL probe loose on the
msgid.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A damaged protected span is a deterministic answer for an unchanged source,
so a run that asks again spends a masking pass, a request and an inference
to reach the same verdict. The refusal is now recorded as a
`translation-refused: <engine> damaged-span` annotation in the entry's
comments, naming the engine that produced it, and `pending` skips such an
entry. Gettext drops the annotation when the source changes, so the entry
comes back by itself; a change to the masking or to the damage predicate
does not, and `make i18n-retry` clears the marks so those entries are
offered again. The mark is matched by prefix, so a refusal from another
engine counts and clears too.
Measured on the `core` domain over 34 languages:
old code 266 requests, 0 translated, 266 discarded
first pass, new 266 requests, 266 annotations written
second pass, new 0 requests, nothing to translate
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every sandboxed command runs in its own pid namespace, where the low
numbers belong to that sandbox's own shell and children. Two places read a
pid as if it named the same thing everywhere, and both blocked translation.
The throwaway container was named after os.getpid(). A run whose cleanup
never ran left `infinito-i18n-libretranslate-8` behind, and the next run,
holding pid 8 in its own namespace, died on `docker run exited with 125:
Conflict. The container name is already in use`. A random token cannot
collide, so an orphan no longer blocks anything.
`deploying()` accepted any live pid as a running deploy. The router writes
build/deploy.pid and never removes it, so a file naming pid 8 refused every
translation run with "A deploy is running" until it was deleted by hand. The
recorded process now has to name the router in its command line; the marker
is verified against the invocation in Makefile:200. /proc is read uncached
because it is per-pid and vanishes with the process.
The test that guarded this asserted the bug: it wrote pytest's own pid and
expected a deploy. It now asserts the opposite for that input, and proves the
positive case with a real process whose argv carries the router path.
Neither fix addresses the root cause of the failures that exposed them: the
sandbox denies 127.0.0.6, so `deployed()` never reaches the running service
and every run falls down the throwaway path in the first place.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ruff format reflows long calls and normalises quotes, isort reorders the
imports of the documentation lint package, SIM117 folds nested `with`
statements into one parenthesised block, TC003 moves typing-only imports
under `if TYPE_CHECKING`, the JavaScript linter replaces a `var`, and the
YAML formatter rewraps five role files. None of it changes behaviour.
One entry does more than that: the formatter had reflowed two calls in
test_builder.py so their `# nocheck: cache-read` markers landed on the
closing parenthesis, where they suppress nothing and the cache-read lint
would have reported two real violations. Both calls are hoisted into a
variable so the marker sits on the reading line again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
accelerated() asked docker info directly, so the variable the project already
carries for this decision did nothing: setting it to 0 in .env left the CUDA
image and --gpus all in place, and test_catalogs_complete_on_gpu kept running.
The variable now wins where it is set, and the daemon is asked only where it
is absent, which keeps every host without the container toolkit behaving as
it did. utils/env/handlers/infinito/gpu_count.py derives it from the same
daemon query but through setdefault, so an operator value survives make
dotenv; that makes it the overridable view of the fact accelerated() used to
establish on its own.
Four call sites follow from it: the stale-container check, the stamp a
successful deploy leaves, the --gpus flag with the -cuda suffix, and the
skipUnless on the catalog completeness test. Turning it off therefore
translates on the CPU, which is what turning the GPU off means here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two domains are built apart: docs passes through sphinx, where the
transform withholds a message that is nothing but a name, while core builds
its template straight from role descriptions, categories and menu labels. A
sphinx extension never reaches it, so the same message could be withheld from
one catalog and carried by the other.
build_template now applies the same predicate.
Its effect today is nil: counted directly, none of core's 421 messages is
withheld, and the extract reports 0 catalogs changed. This is an assurance for
what gets added rather than a repair of what is there - a role description
that is only a product name would have slipped through before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
prose() is built to measure prose, and a name is not prose. Comparing
against it went wrong in both directions: n8n became nn because masking
treats the digit as a span of its own, Infinito.Nexus became empty because
the whole name reads as a domain, and Redis ``cache`` became Redis, which
withheld 131 messages that say more than a name.
_bare() removes only what a page wraps around the name: emphasis at the
edges, a link when it is the whole message, and trailing symbols. The symbol
loop walks the unicode categories rather than a character list, because an
emoji is several codepoints, and it catches the heading that writes Git 🔐.
A link counts only when nothing else is there, or 'See [Mastodon](...) and
more.' would read as a name.
Measured against the previous catalogs: 24 wrongly withheld messages return,
n8n disappears in its three forms, zero warnings. The five reported cases -
[Baserow](roles/web-app-baserow/), [API](roles/web-svc-api/), Git 🔐, Lint 🔍
and n8n - were looked up in the catalog and are gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One predicate now answers what no translator may be handed, and both ends of
the pipeline read it: the extraction withholds the message from the catalog,
the client refuses to send it. What the catalogs keep back is exactly what the
client would have declined.
Two shapes qualify. A message with no word is markup. A message that is only a
name is a label: a brand, this software, a credited person, or a bare URL.
Inside a sentence the name stays protected by the span machinery and the
sentence still needs translating.
Three details each come from a finding. The comparison runs against the prose,
because **Mastodon** is not the string Mastodon and prose() already strips the
emphasis. Case is ignored, because a page writes the name the way its sentence
needs it and **mailu** is still Mailu. The address is matched first, on the raw
text, because masking protects the address of mailto:a@b.c but leaves the
scheme behind and its prose reads as a word.
Measured over the docs domain: 22745 messages fall to 22402 with zero
warnings, and 145 of the 377 removed entries carried a corrupted brand that
was being shipped - Keycloak read Schluesselmantel, OpenProject read offenes
Projekt, micro read Mikrowelle.
SOFTWARE_NAME is read from the group_vars SPOT beside SOFTWARE_REPOSITORY. It
lived there already and no python had ever read it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
catalog.update() defaults to fuzzy matching: for every changed msgid it
searches all removed messages for a similar string and carries that
translation over as a fuzzy entry. The search is quadratic. Measured on
one catalog, it costs 11.01s for the 421-message core domain and did not
finish within 10 minutes for the 25182-message docs domain, per language,
across 20 pool workers.
The carried-over value is dead. Four places read the fuzzy flag and three
exclude it: translations(), the API reader and the docs builder. The
fourth, pending(), treats a fuzzy entry as untranslated, so the next
i18n-translate run overwrites it anyway.
With the search disabled, make i18n-extract domain=docs finishes in 2m4s
wall for 47 catalogs, of which user/real 1.56 shows almost all remaining
time is the single-threaded sphinx build. Checked msgid by msgid on two
catalogs: no surviving entry lost its translation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The server answers a failed call with its input, sometimes lightly
reworded and sometimes with a word repeated. Both shapes pass every
existing criterion: the protected spans survive, the markup count
survives, it is no echo, and truncated() only watches the short side.
stutters() reports a word the translation repeats three times in a row
and the source does not. untranslated() reports a non-Latin target whose
prose is at least 90% ASCII words, which needs the catalog language, so
harms() and unmask() now take it and client.py passes its target through.
Measured over 580079 translated entries: 3650 stutters, 611 still in
Latin script, 2 both, 4259 newly reported. None of the source messages
repeats a word three times itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The client seeds every entry with its own source and replaces only the ones that carry words, so when the server answers with its input the seed stays, gets the libretranslate marker, and is neither pending nor damaged. 542 entries across 439 msgids were plain English while counting as translated, and nothing would ever retry them.
harms() now rejects an identity mapping once the source carries 30 letters outside its protected spans. prose() is has_words' inner half, extracted so the threshold measures what the translator actually had to translate; has_words keeps its signature and behaviour.
The floor is measured, and two earlier shapes were wrong: a raw length of 30 condemned '', and counting prose characters condemned backtick lists of role names whose 75 characters were commas. Counting letters leaves 439 msgids, every one of them translated by at least one other catalog, led by ar with 327 against de with 25 - if identity were legitimate it would cluster in the languages closest to English, not the furthest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The client seeds every entry with its own source and only replaces the ones that carry words, so a word-free msgid keeps the source verbatim. That was harmless until 9679fa4221: tighten and resegment judge a translation against a rule rather than against its source, so an identity mapping became condemnable as soon as the source itself broke the rule. The gate then reported a source defect as translation damage and prune could not help, because the next run seeds the same source again.
Return False when the translation equals its source. A source that breaks a rule is what the source-side lints are for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_i18n_catalogs.py:125 and :141 report a translation whose emphasis carries a space or whose sentence boundary was swallowed, and the first one's failure message says to run prune and translate again. prune clears through damaged(), which did not know either criterion, so those entries survived every prune and the advice was impossible to follow.
Add both to harms(). The writer does not get stricter: unmask() applies tighten and resegment as repairs before it asks harms(), so a freshly written entry satisfies them by construction, and a test pins that. Measured over every checked-in catalog, 581220 translated entries: 599 condemned before, 599 after, 0 newly condemned.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
unmask() accepted a translation as soon as protected_spans matched, while damaged() also rejected it for a dropped name, a halved body or changed markup. Three of the four criteria lived only in the gate, so every run wrote entries the gate then condemned, make i18n-prune emptied them and the next run wrote them back. It never converged: ko carries 331 such entries, 3.7 percent of what it has translated, against 0.1 percent or less for Latin scripts.
Introduce harms(source, translation) in placeholders.py as the single source of truth and call it from both ends. truncated() and structure() move there with it, since translate.py already imports placeholders and the other direction would be a cycle.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2aa7325eca put role names and product titles into the spans mask() produces, and
unmask hands those same spans to resegment. That was safe for every span the
protection had before, because each ends in a delimiter, so a letter right
behind it can only be the text a lost boundary was taken from. A bare name ends
in a word character, and Bengali writes `Baserowo` where the source says
`Baserow`, so the substring search found the name inside the longer word and put
the punctuation in the middle of it:
resegment('Baserowo-r shathe', ('Baserow',), 'Adapter to Baserow: ...')
-> 'Baserow: o-r shathe'
Names are skipped. The repair still fires for the spans it was written for, and
the 85 entries the boundary lint reported across et, fi, eu, cs, ja and bn drop
to none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured over the entries no rule reported, a machine translation loses the
sentence's full stop in 12417 cases, the capital of its opening word in 2139,
and opens a run of spaces around a protected span in 610. None of them touches
the meaning, so the answer is worth keeping and worth repairing rather than
discarding.
The full stop goes missing when a closing bracket moves in front of a protected
span and takes the punctuation behind it along, which leaves the sentence
running into the next one. resegment already puts back a boundary that went
missing mid-text; terminate does the same at the end.
collapse leaves a run the source carries itself alone, because an aligned block
in a code sample means its spacing.
Truncation cannot be repaired, only reported, and joins the damaged set with a
floor of 120 characters: `Situation` becomes `Lage` and halves its length while
saying the same thing, so a shorter entry proves nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing stopped a machine translation from rewriting the identifiers the
documentation is about. Measured over the catalogs, 4836 of 25166 role name
mentions came back altered and 9131 of 33337 title mentions, among them
`(Unbound, etc.)` turning into `(Ungebunden, etc.)` for a DNS resolver and
`Snipe-IT` losing its non-breaking hyphen to an ordinary one. Emoji went the
same way.
The names come from the repository itself: the role directories, the first
heading of each README and galaxy_info.name. A hyphenated role name is never an
English phrase and holds wherever it appears. A title is protected only inside a
sentence, because at a sentence start every word is capitalised and the capital
stops telling a product from a common noun: the heading `Documentation` is meant
to be translated, `Keycloak` in the middle of a clause is not. A dictionary
cannot make that call either, since `Unbound` and `Confluence` are English words
and product names at once.
Masking and comparing had to be separated. German capitalises every noun and
compounds a name with what it qualifies, so `shell extensions` legitimately
becomes `Shell-Erweiterungen`, which a set comparison reads as a surplus span
and a word boundary ending at the hyphen reads as a loss. Names are therefore
masked out of the request but checked for presence rather than counted, and
protected_spans keeps the equality semantics it had for the structural rules.
This reports 21352 entries whose translation dropped a name or an emoji. The
finding is correct and the lints that surface it stay red until
`make i18n-prune` empties them for the next run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
xgettext counts a space as a format flag, so the prose `95% confidence` parses
as the directive `% c` and the entry is marked python-format. Every translation
moves the word behind the percent sign, msgfmt then reads `% V` and refuses the
catalog. Quoted Jinja blocks and percent-encoded URLs fall into the same trap.
A run over the documentation catalogs left 312 such entries in 19 files, none
of which could be compiled to .mo.
Removing the flag when writing does nothing: babel derives it from the msgid on
every load, so the value in the file is decoration. gettext's own switch for
this is no-python-format, which babel does not touch, and the writer now sets
it on entries the derivation flags but that carry no real placeholder. The
derived flag has to go at the same time, because gettext reads the flag line in
order and `no-python-format, python-format` cancels itself out.
What counts as a real placeholder was already spelled out in PROTECTED, so it
moves into PRINTF and is used from both places rather than written twice.
The repository lint gains the check that would have caught this. It compares
the msgid against itself first and reports only when the translation alone
fails, so a source string that mixes positional and named placeholders, which
gettext accepts, does not raise a finding.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils/i18n/libretranslate.py had grown two jobs: bringing a LibreTranslate
up, and talking to one. The HTTP half moves to utils/i18n/client.py with the
batch size, the retry constants, the SERVER_CODES map that turns the
catalog's bare ISO 639-1 codes into the script-qualified ones the server
answers to, the Outcome dataclass and merge().
The deploy module keeps the name it is imported by and re-exports the three
client symbols, so every existing caller is untouched; the exported class is
the same object, not a wrapper.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four defects in the client surfaced in one run. LibreTranslate serves Chinese as zh-Hans and zh-Hant, never as bare zh, so the readiness wait blocked on a target that was never coming and the run stood still for an hour without printing a line. A blanket except turned every transient refusal into a permanently discarded entry, and the fallback path answered a failed batch by firing twenty single-text requests at the same overloaded server; the same catalogs that lost 130 to 295 entries came back with 0 discarded once the batch is retried with backoff first.
The per-catalog count conflated a refusal with a damaged protected span, which is measurably wrong: of 228 entries left pending in core/sw only 36 carried a span at all. Outcome now carries values, refusals, damage and the last refusal message per call rather than on the client, where twenty lanes mixed their numbers into each other. That split is what surfaced the 502 the server had been answering with. The throwaway container also ran with check=True and capture_output=True, so docker error text died inside the swallowed exception and the log showed an argv list and rc 125.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
server() now resolves a translation endpoint in three steps: use the running
service, rebuild when the host has a GPU but no accelerated deploy left its
stamp, otherwise deploy. A plain run therefore costs nothing, and the rebuild
fires once, on the move from a CPU-built container to an accelerated one.
deploy() brings the runner up, waits for State.Running and deploys the role
inside it. The exec steps address `docker compose exec i18n` directly:
cli.administration.deploy.development exec hardcodes the `infinito` service
(compose.py:261), so every deploy through it silently ran in the main
container, where mailu already holds port 25 and postfix cannot start.
check=True stops a failing step instead of carrying on against the wrong
target.
The deploy router writes its pid to build/deploy.pid, which survives the exec
into the subscripts, and deploying() refuses a translation while that process
lives. The running deploy is left untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every rule so far named a construct to protect, and every run found one that
had been forgotten: paths, underscores, bracketed URLs, then the bracket
itself. 7338 sources still reached the translator carrying markup, and one in
eight came back broken.
Mask the delimiters themselves, so nothing a translator would reformat is left
in the text: emphasis, backticks, brackets, braces, parentheses, double quotes,
quoted identifiers and digits. Single quotes stay exposed on purpose, because
1419 of the 1494 sources that carry one are English contractions, and masking
those would hand over fragments instead of words.
Costs nothing: the 2606 entries that end up without translatable text were
already empty under the old rules, and the newly masked ones are galaxy_tags
lists and JSON fragments.
Two defects survive masking because they alter no protected span, so unmask
repairs them against the source: a translator reading a trailing span as
sentence-final swallows the boundary behind it, gluing the next sentence to a
code span, and it spaces out emphasis, which markdown then renders literally.
The standing checks assert the property rather than enumerate constructs, and
cover core and docs, where before only the span-parity check did, which is why
none of this was ever caught.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
