666312dfbd settled the catalog client on the dedicated LibreTranslate. One
engine cannot be compared against itself, so the direct path had no way to
improve on a language it translates badly: of the 2854 entries of the `ko`
catalog it produced 1 translated entry and 2704 `structure` rejections.
The client now asks the gateway, which fronts several engines and keeps a
history of which one survives a given signature. Protected spans are masked
as `<x id="N"></x>` before the request and declared per item, so the gateway
can reject an answer that altered them instead of passing it on. `_reask`
then retries each rejected entry with the engine that produced it in
`exclude`, which reaches a different backend rather than the same one. The
same catalog measured 17 translated entries with 2649 rejections once the
gateway checked the markup, and 331 with 796 once the retry could exclude.
The full picture a rejection needs is only available in the client: 2431 of
those 2649 structure rejections involve markup the masking hides from the
gateway, so the engine-switching decision belongs on this side of the call.
`server()` reuses a gateway that already answers and otherwise deploys the
bundle the gateway needs. It does not reintroduce the throwaway container
666312dfbd removed: there is no `docker run` here, the roles deploy through
the normal path and their models still come from the role images built
through the package cache. `redeploy=true` is offered because the reuse check
only asks whether a gateway answers, which cannot see that the gateway's own
code changed since it started. A run refuses outright while a deploy is in
progress, because both touch the same container stack, and leaves the deploy
alone.
`deployed()` additionally requires a non-empty catalogue, so a gateway that
answers while serving no pairs counts as not ready rather than as a backend
that refuses everything. `probe(url)` is split out of `deployed(root)` so one
readiness check serves a single engine and the gateway alike.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The span check compared multisets, so a translation was refused whenever its
set of protected spans differed from the source's in either direction. But
`PROTECTED` ends in two very low-precision alternatives, `\d+(?:[.,]\d+)*` and
the bare markup characters, which claim every digit run and every parenthesis.
Over-claiming is right when masking, because nothing may leak to the engine
unprotected. As a comparison basis it is wrong: the translation is another
language whose ordinary prose trips those same rules, and a gained parenthesis
destroys nothing the source carried. The module's own contract says "whether a
translation broke something its source carried", which is a subset relation.
So the loss half keeps the name and the surplus half gets its own. `added_spans`
refuses only a surplus whose presence is itself damage: an injected `%s` that
`msgfmt` rejects, a decoder artefact like `♪`, a leaked `<unk>`, an invented
`{placeholder}`. Measured over representative shapes, all four of those stay
refused, a gained parenthesis stays refused through the existing `structure`
check, and what becomes admitted is a surplus numeral or an abbreviation like
the Danish `f.eks`.
The gate moves in the same commit, and it has to. fe80a23725 introduced one
predicate for both ends precisely because a writer and a gate that disagree
never converge: the run writes, the gate condemns, `i18n-prune` empties, the
next run writes the same entries back. That commit did not quite finish the job
- `test_i18n_catalogs.py` still duplicates the comparison rather than calling
the shared predicate - so the duplicate is mirrored here. Making it call
`reason()` outright would be better and is left for its own change.
No catalog entry changes here. The refusal markers are all still in place, so
the entries this releases stay empty until a translate run fills them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A bare mark told an operator that an entry is unanswerable without saying
what came back for it, so the verdict could not be judged and the only way
to see the criterion fire was to run the translator again. The rejection now
travels with its text: `unmask` returns a `Rejected` carrying the restored
translation and the criterion it broke, and the entry records both as
comments beside the mark.
# translation-refused: libretranslate structure
# nocheck: url
# translation-rejected: Siehe [den Leitfaden]](https://gone.invalid/a)
msgid "See [the guide](https://gone.invalid/a) first."
msgstr ""
The quote is a comment and never a msgstr, because it is exactly what the
consumers must not use.
`harms` keeps being the predicate both ends of the pipeline ask, but it is
now the boolean face of `reason`, which names the criterion instead of
folding every check into one `or` chain. The token failures `unknown-token`,
`lost-token` and `empty` join the criteria the catalog can report.
Two details are load-bearing. The `nocheck` sits on a line of its own above
the quote: babel wraps a long comment, and appended behind the text the
marker landed one line below the URL it has to cover, where the block-above
resolver no longer sees it. And the cleanup drops a `nocheck` only where a
quote follows it, because Sphinx carries a source comment into the catalog
as a user comment and removing one would let the URL probe loose on the
msgid.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every rule so far named a construct to protect, and every run found one that
had been forgotten: paths, underscores, bracketed URLs, then the bracket
itself. 7338 sources still reached the translator carrying markup, and one in
eight came back broken.
Mask the delimiters themselves, so nothing a translator would reformat is left
in the text: emphasis, backticks, brackets, braces, parentheses, double quotes,
quoted identifiers and digits. Single quotes stay exposed on purpose, because
1419 of the 1494 sources that carry one are English contractions, and masking
those would hand over fragments instead of words.
Costs nothing: the 2606 entries that end up without translatable text were
already empty under the old rules, and the newly masked ones are galaxy_tags
lists and JSON fragments.
Two defects survive masking because they alter no protected span, so unmask
repairs them against the source: a translator reading a trailing span as
sentence-final swallows the boundary behind it, gluing the next sentence to a
code span, and it spaces out emphasis, which markdown then renders literally.
The standing checks assert the property rather than enumerate constructs, and
cover core and docs, where before only the span-parity check did, which is why
none of this was ever caught.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The target span started at the closing bracket, so masking [RSA 4096-bit](url)
left an unbalanced [RSA 4096-bit in the translatable text. The translator
closed it, and restoring the span produced [RSA 4096-bit]](url). Parity never
noticed, because the extra bracket sits outside every protected span, and
1248 German entries carried it.
A lookbehind now keeps the bracket in the text and masks the target alone.
The existing damage is repaired in place rather than discarded: the sources
hold no ]]( at all, so every occurrence in a translation was spurious, and
the German around it was sound.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The URL pattern stopped at the first closing parenthesis, so a Wikipedia
link such as .../Social_engineering_(security) was only masked up to the
bracket. The translator then glued the remainder to the following word and
produced .../Social_engineering_(security)auf, which the external URL check
caught as HTTP 404. Markdown targets clipped the same way, and such targets
exist in several role READMEs.
Both patterns now accept a balanced parenthesis group, so the whole link
becomes one protected span. Prose in parentheses still masks to nothing.
Pruning both domains against the stricter protection emptied nine German
entries, four in core and five in docs, including the two glued links. They
refill on the next translation run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LibreTranslate rewrote anything that read like prose: assets/img/ became
Vermoegenswerte/img/ and ](path) collapsed to )path). Mask multi-segment
repository paths, bare filenames and markdown link targets alongside the
spans that were already protected, so the translator never sees them.
The catalogs still carry the earlier damage, so add a prune subcommand. It
compares the protected spans of source and translation and empties every
entry whose translation altered them, which lets the next run refill it. A
pass over the German docs catalog cleared 85 entries without losing a single
intact translation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Core and the documentation get gettext catalogs for all 184 ISO 639-1
languages listed in meta/languages.yml, 47 of which LibreTranslate can fill.
`make i18n-extract` merges the current sources into locale/<code>/
LC_MESSAGES/{core,docs}.po: role descriptions, category and menu titles and
the logout strings for the core domain, the Sphinx gettext output for the
docs domain. A changed source keeps its previous translation as a fuzzy
entry, and catalogs are written deterministically so an unchanged merge
leaves the files untouched.
`make i18n-translate domain=<core|docs> [languages=<codes>]` fills only the
empty and fuzzy entries with a short-lived LibreTranslate container (the
image web-svc-libretranslate pins) and removes it afterwards. It starts the
container, and loads models, only for languages that still have pending
entries. Code spans, placeholders, URLs, tags and dotted names such as
Infinito.Nexus are masked for the request and a translation that damages
one is discarded. An unreachable server stops the run instead of
discarding every remaining entry, so a rerun continues where it stopped;
machine translations carry the translator comment `libretranslate`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5