fix(i18n): check the script of short labels, not only of sentences

untranslated() is the one criterion that knows a catalog's script, and it
never fired. It returned False below LATIN_FLOOR = 6 words, while the
catalogs are mostly one- to four-word UI labels, so the guard sat above
its own corpus.

Measured across the 62 non-Latin catalogs: the floor condemns 0 entries.
Without it, 536, concentrated in the languages the model cannot write:

  av 99   as 92   ba 87   be 84   ab 84   am 56   bh 13

Every entry in the low-count tail was read before the floor was removed,
because that tail is where a false positive would live. There is none:

  ko 'Applications'            -> a refusal sentence from the model
  zh 'Updates & Package Mgmt'  -> the string plus a version changelog
  fa 'Cloud'                   -> 'Cloud' and a markdown rule
  ab 'Communication'           -> 'Comunicacion'

A word count was the wrong instrument. reason() consults this only after
an exact echo is ruled out, so for a non-Latin target, Latin text that
DIFFERS from the source cannot be a term the language keeps in English.
That distinction is what an earlier attempt at floor=2 lacked, which is
why it condemned 4053 entries including legitimate ones.

Scope, stated because the numbers invite a larger reading: this guards
the 62 non-Latin catalogs only. For the ~122 Latin-script languages a
wrong-language answer still passes, and catching those needs a detector
this repository does not have.

LATIN_FLOOR and its re-export are deleted rather than left at 1; nothing
else consumed them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kevinveenbirkenbachandClaude Opus 5 committed 2026-10-20 17:54:15 +02:00
1 parent 8621d96f01
commit 27d59f4bd4
3 files changed
+34 -4

No files matched your search

+27 -1
View File
@@ -1,4 +1,5 @@
"""Unit tests for the criterion that condemns an answer carrying its source.
"""Unit tests for the criteria that condemn an answer carrying its source or
coming back in the wrong script.
The shape is one the gateway produced across twenty languages: the English
source, a markdown rule, then a rendering. Every other criterion passed it,
@@ -42,5 +43,30 @@ class TestEchoAndContinue(unittest.TestCase):
self.assertEqual(reason(source, "Mail-Konto des Benutzers", "de"), "")
class TestWrongScript(unittest.TestCase):
"""A language written in another script came back in Latin letters.
The catalogs are mostly one- to four-word labels, and a six-word floor
once stood in front of this criterion, so it never fired on them: a whole
run filled Belarusian with Polish and Abkhaz with Spanish while every
criterion passed.
"""
def test_a_one_word_answer_in_the_wrong_script_is_condemned(self) -> None:
self.assertEqual(reason("Applications", "Aplikacje", "be"), "untranslated")
def test_a_label_answered_in_another_language_is_condemned(self) -> None:
self.assertEqual(reason("Communication", "Comunicación", "ab"), "untranslated")
def test_an_answer_in_the_language_s_own_script_is_kept(self) -> None:
self.assertEqual(reason("Applications", "Праграмы", "be"), "")
def test_a_latin_script_target_is_not_judged_by_script(self) -> None:
self.assertEqual(reason("Applications", "Anwendungen", "de"), "")
def test_a_translation_without_a_word_is_kept(self) -> None:
self.assertEqual(reason("Status", "100 %", "be"), "")
if __name__ == "__main__":
unittest.main()
+7 -2
View File
@@ -29,7 +29,6 @@ MAX_SHARING_SOURCES = 2
ECHO_FLOOR = 30
STUTTER = re.compile(r"\b(\w+)(?:\s+\1\b){2,}", re.IGNORECASE | re.UNICODE)
WORDS = re.compile(r"[^\W\d_]{2,}", re.UNICODE)
LATIN_FLOOR = 6
LATIN_SHARE = 0.9
NON_LATIN_SCRIPTS = non_latin_languages(PROJECT_ROOT)
@@ -89,6 +88,12 @@ def untranslated(translation: str, language: str) -> bool:
English. Comparing scripts catches it wherever the target does not use the
Latin one; for a Latin-script target it would need a language detector.
There is no length floor. :func:`verdict` reaches this only once an exact
echo is ruled out, so Latin text that differs from the source cannot be a
term the language keeps in English. The catalogs are mostly one- to
four-word labels, which is where the wrong language hid for as long as a
six-word floor stood here.
Args:
translation: what came back for the entry.
language: ISO 639-1 code of the catalog being filled.
@@ -96,7 +101,7 @@ def untranslated(translation: str, language: str) -> bool:
if language not in NON_LATIN_SCRIPTS:
return False
words = WORDS.findall(prose(translation))
if len(words) < LATIN_FLOOR:
if not words:
return False
latin = sum(1 for word in words if (latin_share(word) or 0) > 0.5)
return latin / len(words) >= LATIN_SHARE
-1
View File
@@ -14,7 +14,6 @@ import html
from utils.i18n.damage import COLLAPSE_FLOOR as COLLAPSE_FLOOR
from utils.i18n.damage import COLLAPSE_RATIO as COLLAPSE_RATIO
from utils.i18n.damage import ECHO_FLOOR as ECHO_FLOOR
from utils.i18n.damage import LATIN_FLOOR as LATIN_FLOOR
from utils.i18n.damage import LATIN_SHARE as LATIN_SHARE
from utils.i18n.damage import MARKUP as MARKUP
from utils.i18n.damage import MAX_SHARING_SOURCES as MAX_SHARING_SOURCES