mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-08 11:46:35 +00:00
kevinveenbirkenbachandClaude Opus 5
ee9cbc8d5b
The oracle asks a chat model whether a translation states its source. llama3.2:1b cannot answer that: on the suite's own fixture it scored an unrelated text at the middle level instead of the bottom one, and asked as a yes-or-no question it called two identical texts different. With it the suite reached no verdict at all, reporting that the judge read none of the 47 languages the catalogs hold machine translations for. Qwen2.5-3B answers both correctly, and the same run then judged 248 samples. The pin lives in the lane inventory rather than in AI_LOCAL_MODELS, because a judge that can separate a faithful rendering from an unrelated text is a demand no deployment makes, and the default should not grow a gigabyte to satisfy a check. The litellm config builds its routes from the merged application config, so a lane override reaches the model list and its window without touching the repository default. LANE_MODEL_ALIAS repeats the one name in utils/inventory/tools.py, because ensure() asks whether the lane already serves before the deploy that writes the rendered inventory, so the probe cannot read it from there. A lane serving one model while the probe expects another never reports ready and redeploys on every call, so a test renders the template and holds the two names equal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>