mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
kevinveenbirkenbachandClaude Opus 5
abd6a24b73
The gateway publishing a model proves nothing about a backend answering it: LM
Studio shipped a published lmstudio/default route for as long as it had no
model to load. This CLI test closes that gap by asking the running gateway what
it serves and comparing it against what the deployment enabled.
templates/test.env.j2 is both the input and the entry point — discover_cli_roles
finds a role by that file and filters on group_names, so the test runs only
where the gateway is deployed. meta/tests.yml names the container service key so
run_one.yml resolves the node holding it in swarm, and allows 600s for a cold
model load.
probe.py runs inside the gateway container, where the API answers on loopback
and the backends resolve by name, and uses the standard library only because
the image ships the litellm app rather than a test toolchain. It asserts that
every declared model is served, that the LM-Studio-only aliases are present
exactly when that backend is deployed, that no Ollama-backed model is published
without Ollama, that LITELLM_CHAT_MODEL is routable, that an empty deployment
publishes nothing, and that a real completion comes back non-empty.
The fourth assertion is the one worth having: LITELLM_CHAT_MODEL is built from
lookup('deployment').inventory while the config is built from this host's
group_names. Two sources for one fact, and this is where they are made to
agree.
The alias list is rendered whether or not LM Studio is deployed. Gating it
would empty the set in exactly the case the negative direction needs, turning
"a route must not stand without its backend" into a vacuous check.
The verdict logic sits in a pure evaluate() so it is exercised without mocks:
four deployment shapes, each with its positive and negative cases, including a
both-backends case that asserts a shared alias proves nothing about LM Studio
and only the exclusive one counts.
files/test/.nocheck is required rather than convenient. role-files-layout sorts
files/ by language while test-e2e-cli stages files/test/ under that fixed path;
the other 29 CLI test directories carry the same marker.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
5 lines
57 B
YAML
5 lines
57 B
YAML
---
|
|
cli:
|
|
timeout: 600
|
|
container_service_key: litellm
|