mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-08 11:46:35 +00:00
kevinveenbirkenbachandClaude Opus 5
ee9cbc8d5b
The oracle asks a chat model whether a translation states its source. llama3.2:1b cannot answer that: on the suite's own fixture it scored an unrelated text at the middle level instead of the bottom one, and asked as a yes-or-no question it called two identical texts different. With it the suite reached no verdict at all, reporting that the judge read none of the 47 languages the catalogs hold machine translations for. Qwen2.5-3B answers both correctly, and the same run then judged 248 samples. The pin lives in the lane inventory rather than in AI_LOCAL_MODELS, because a judge that can separate a faithful rendering from an unrelated text is a demand no deployment makes, and the default should not grow a gigabyte to satisfy a check. The litellm config builds its routes from the merged application config, so a lane override reaches the model list and its window without touching the repository default. LANE_MODEL_ALIAS repeats the one name in utils/inventory/tools.py, because ensure() asks whether the lane already serves before the deploy that writes the rendered inventory, so the probe cannot read it from there. A lane serving one model while the probe expects another never reports ready and redeploys on every call, so a test renders the template and holds the two names equal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
37 lines
1.2 KiB
Python
37 lines
1.2 KiB
Python
"""The alias the readiness probe expects is the one the lane deploys."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import unittest
|
|
|
|
import yaml
|
|
|
|
from utils import PROJECT_ROOT
|
|
from utils.inventory.tools import LANE_MODEL_ALIAS, render
|
|
|
|
OLLAMA_ROLE = "svc-ai-ollama"
|
|
|
|
|
|
class TestLaneModel(unittest.TestCase):
|
|
def test_the_rendered_inventory_pins_the_alias_the_probe_expects(self) -> None:
|
|
"""Two places name the model, and nothing else keeps them equal.
|
|
|
|
The inventory carries the entry because that is where inventory data
|
|
belongs, and the probe needs the bare alias before any render has
|
|
run, because ``ensure`` asks whether the lane serves before the
|
|
deploy that writes the rendered file. A lane serving one model while
|
|
the probe expects another never reports ready and redeploys on every
|
|
call.
|
|
"""
|
|
rendered = yaml.safe_load(render(PROJECT_ROOT, ["en"]).read_text("utf-8"))
|
|
ollama = rendered["applications"][OLLAMA_ROLE]["services"]["ollama"]
|
|
|
|
self.assertEqual(
|
|
[LANE_MODEL_ALIAS],
|
|
[model["alias"] for model in ollama["preload_models"]],
|
|
)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main()
|