031 made the gateway the one place backends are wired, but not the place the
choice is made: every consumer still names one alias and LITELLM_CHAT_MODEL
resolves it once per deploy for all of them. That is wrong in both directions.
A prompt needing a large window reaches a model that cannot hold it, which is
how web-app-hermes answers HTTP 500 when the agent model declares fewer than
64000 tokens, and a one-line question reaches whatever the deploy picked.
This adds the System One half: fast, typed, no generation, deciding which
System II model answers. Requirement 038 records the decisions; the shape
follows prismhq/jev-router with a deterministic decision in place of a model,
and decision 7 records why no learned one is in scope.
A pre-call hook, not a semantic router. LiteLLM's own auto_router matches
embeddings against example utterances, which answers "what is this about"
rather than "which model can serve this", and it pulls in the semantic-router
extra plus an embedding backend the gateway does not otherwise need.
The hook ships as a rendered file beside config.yaml rather than a derived
image: proxy_server.py resolves a callback through get_instance_fn(value,
config_file_path), which spec_from_file_location's it out of the config's own
directory. services.litellm.image stays on its pin. It is a template because
ROUTER_ALIAS has to match what the route builder publishes.
Capabilities come from LiteLLM's own catalog, 1595 models deep in the image,
keyed on the route's litellm_params.model and then on that name without its
first segment, because ollama/llama3 is filed prefixed while openai/gpt-4o-mini
is filed bare. A traits mapping overrides it for what the catalog does not
know. Declaring what the catalog already states is forbidden: it duplicates a
fact this project does not own and rots in the dangerous direction, since a
stale supports_vision keeps a model eligible for requests it can no longer
serve. Cost is not declared either; a route with an api_base or a mock_response
is served by this deployment and is free, and a keyed route the catalog does
not price sorts behind every route it does.
It fails loud, in both halves. The hook returns a string naming the demand when
no served model can take the request, and proxy/utils.py turns that into a
RejectedRequestError carrying the text. The alias's own model_list entry exists
only so the name is listable and raises when reached. Pointing it at the first
route instead, as this first did, made a router that never ran look like one
that worked.
services.litellm.router_alias is the single switch. Empty publishes no route and
renders no callbacks entry, so the hook is not even a condition of starting the
proxy; a separate enable flag would let services.yml read as on while the
router is off.
The routing probe had to learn the alias, which is backed by no backend and was
read as a route standing without one. It now also sends a prompt the smallest
declared window cannot hold and fails when the answer names the alias itself or
a model whose window is too small. That lookup tries the prefixed and the bare
name, because litellm sets model_response.model to the resolved model: matching
only the literal name would find no window and pass every answer.
svc-ai-ollama variant 1 serves three models so the routes differ in something a
request can demand. smollm2:135m is deliberately not the smallest window:
8dbdd31b22 measured it running to n_gen=22011 with context shifts at
n_left=4091, and a route's max_tokens is a default a consumer can override
rather than a cap.
Open, and only a deploy closes them: that the proxy accepts the raising route
at startup, that the alias routes across three windows, and swarm.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A deploy refused at ok=771 failed=1 with 'services.litellm.remote_models names provider(s) mock; known providers are openai, anthropic, openrouter'. Declaring the mock among the remote models, which is what makes it an ordinary model rather than a config key of its own, runs into a guard that demands a key slot for every provider. A mock needs none: it answers from its own declared response.
The guard keeps its teeth for real providers and now names the exception and its reason, instead of tolerating it silently.
AI_MOCK_PROVIDER holds the name once. The guard, AI_REMOTE_ALIASES and AI_MOCK_ALIASES read it, and the template hands it to the route builder, which lost its own constant. Two literals spelling the same thing is how the LM Studio branch came to drop the context window its README still advertised; a parameter cannot drift.
Static checking never touched this line. It took a real deploy, 772 steps in.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The proxy config grew a loop per model source, so the rule for how an alias is called lived in four places - and they had drifted: the LM Studio branch published no model_info.max_input_tokens although the role's README says a model entry carrying context publishes exactly that. Serving anything new meant a fifth branch.
The sources now meet in one filter that returns {alias, params, context}, and the template renders a single loop. Every value goes through to_json, so a response carrying a colon, a quote or a newline cannot break the YAML it is written into.
Publication order is a contract, not an accident of how the loops were stacked: LITELLM_CHAT_MODEL takes the first alias, so a locally served model has to outrank a declared one. That is now stated in the module and asserted, instead of being implied.
The gateway lint had to widen with it. Its canary checks that some gateway template still matches the provider-URL pattern, because without one the rule would pass vacuously on every consumer; the pattern demanded {{ … }} while the template now sets the URL in {% set … %}. Widening it to {{ or {% makes the rule stricter, since a provider URL in a set block is just as direct an address and would have slipped through anywhere else.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5