mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-08 19:56:53 +00:00
kevinveenbirkenbachandClaude Opus 5
e7eb8ac1d2
031 made the gateway the one place backends are wired, but not the place the
choice is made: every consumer still names one alias and LITELLM_CHAT_MODEL
resolves it once per deploy for all of them. That is wrong in both directions.
A prompt needing a large window reaches a model that cannot hold it, which is
how web-app-hermes answers HTTP 500 when the agent model declares fewer than
64000 tokens, and a one-line question reaches whatever the deploy picked.
This adds the System One half: fast, typed, no generation, deciding which
System II model answers. Requirement 038 records the decisions; the shape
follows prismhq/jev-router with a deterministic decision in place of a model,
and decision 7 records why no learned one is in scope.
A pre-call hook, not a semantic router. LiteLLM's own auto_router matches
embeddings against example utterances, which answers "what is this about"
rather than "which model can serve this", and it pulls in the semantic-router
extra plus an embedding backend the gateway does not otherwise need.
The hook ships as a rendered file beside config.yaml rather than a derived
image: proxy_server.py resolves a callback through get_instance_fn(value,
config_file_path), which spec_from_file_location's it out of the config's own
directory. services.litellm.image stays on its pin. It is a template because
ROUTER_ALIAS has to match what the route builder publishes.
Capabilities come from LiteLLM's own catalog, 1595 models deep in the image,
keyed on the route's litellm_params.model and then on that name without its
first segment, because ollama/llama3 is filed prefixed while openai/gpt-4o-mini
is filed bare. A traits mapping overrides it for what the catalog does not
know. Declaring what the catalog already states is forbidden: it duplicates a
fact this project does not own and rots in the dangerous direction, since a
stale supports_vision keeps a model eligible for requests it can no longer
serve. Cost is not declared either; a route with an api_base or a mock_response
is served by this deployment and is free, and a keyed route the catalog does
not price sorts behind every route it does.
It fails loud, in both halves. The hook returns a string naming the demand when
no served model can take the request, and proxy/utils.py turns that into a
RejectedRequestError carrying the text. The alias's own model_list entry exists
only so the name is listable and raises when reached. Pointing it at the first
route instead, as this first did, made a router that never ran look like one
that worked.
services.litellm.router_alias is the single switch. Empty publishes no route and
renders no callbacks entry, so the hook is not even a condition of starting the
proxy; a separate enable flag would let services.yml read as on while the
router is off.
The routing probe had to learn the alias, which is backed by no backend and was
read as a route standing without one. It now also sends a prompt the smallest
declared window cannot hold and fails when the answer names the alias itself or
a model whose window is too small. That lookup tries the prefixed and the bare
name, because litellm sets model_response.model to the resolved model: matching
only the literal name would find no window and pass every answer.
svc-ai-ollama variant 1 serves three models so the routes differ in something a
request can demand. smollm2:135m is deliberately not the smallest window:
8dbdd31b22 measured it running to n_gen=22011 with context shifts at
n_left=4091, and a route's max_tokens is a default a consumer can override
rather than a cap.
Open, and only a deploy closes them: that the proxy accepts the raising route
at startup, that the alias routes across three windows, and swarm.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>