mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-08 19:56:53 +00:00
kevinveenbirkenbachandClaude Opus 5
2261bbabb0
The oracle suite needs a verdict in every language meta/languages.yml names, and laya cannot give one. Measured against the suite's fixture, its english checkpoint separates a faithful rendering from an unrelated text in 17 of 47 languages and its multilingual checkpoint in none, answering around 1.5 either way. One judge for all of them beats a route per language that is noise in most of them. The question keeps the shape the checkpoint answered: the levels are written out and the model names an index over them, never a bare probability. A probability carries this deployment's calibration, and a fixed threshold on it would encode that calibration into the suite; "below the middle level" instead means what the middle level says. Three refusals are deliberate. temperature 0, so the same comparison does not come out differently on a rerun. An answer naming no index raises instead of scoring zero, which would read a refusal as "states something else". An index outside the scale raises instead of being clamped, because a model answering 7 has not placed the pair on this scale at all. The request goes to the model gateway rather than to a model server, so it carries a consumer identity and resolves through the gateway's model list like every other model call here. Nothing imports this yet: the oracle suite moves to it once the tools lane carries LiteLLM and a loaded model, which is what lets the judge be exercised against a real one for the first time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>