Files
computer-playbook/tests/unit/python/utils
kevinveenbirkenbachandClaude Opus 5 78b557b354 fix(i18n): give a logographic target its own truncation ratio
`truncated` compared character counts against a flat 0.5 for every language. A
logographic script carries a morpheme per character, so a complete Chinese
rendering of an English passage runs at roughly a third of its length and the
predicate read it as a dropped clause.

Measured against the deployed LibreTranslate, re-translating 15 refused sources
per language: zh 15 of 15 refused, ja 11 of 15, ko 10 of 15, while 44 of those
45 ended in terminal punctuation, so they were complete sentences. The ratios
clustered between 0.28 and 0.55; a real truncation scatters and stops
mid-sentence. 0.25 sits below every complete rendering measured and still
catches a passage cut to a quarter.

The language set is measured, not guessed: zh, ko and ja account for 5553 of
the 6135 truncation refusals, 90.5%. The next language down is at 116, small
enough to be genuine truncation, so it keeps the flat ratio.

`collapsed` already reasons this way for the same reason, its docstring naming
Chinese explicitly; `truncated` never got the same treatment. `reason` already
carried the language for `untranslated` and now passes it on.

Effect on the catalogs, from the run rather than a test: zh 4153 truncation
refusals to 20, ko 816 to 0, ja 584 to 0, and 5590 entries that had been stuck
are now translated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-30 16:45:03 +02:00
..