Files
computer-playbook/tests/unit/python
kevinveenbirkenbachandClaude Opus 5 d0e01b2940 feat(svc-ai-litellm): compare routes on an interval and remember which won
The router settings become one mapping. Nine scalars sharing a router_ prefix were a dict spelled out in key names, and the sampling section about to arrive would have added five more. Carried along: fifteen lookups in vars, three variant overrides, four test files, and the two rejection messages that name the settings path, which would otherwise have pointed a reader at a key that no longer exists.

On a configurable trigger, seconds or requests or probability, the same request goes to several eligible routes at once. jeff is shown the replies rather than the routes that wrote them, with the aliases hidden behind option labels so the verdict rests on the text. In shadow mode the caller keeps the route already decided; in serve mode the winning reply is handed back through mock_response, so the fan-out is the answer path and no further call follows.

Each comparison writes one row per offered route to the proxy's own database, grouped by a request class of token band and capability demands. Grouping finer would spread the evidence too thin to decide, coarser would let a verdict about a vision request speak for a plain chat. A route that raises is left out rather than recorded as a loss: failing to answer is a fact about the moment.

The decider now also sees the price of this request, the measured rate, the quota and prompt caching, plus how each route fared before. A figure the catalog does not state is left out rather than defaulted, because a decider cannot tell an absent number from a zero one.

Sampling ships off. Every history and comparison path is caught wide and logged: neither may ever fail a route.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-24 23:35:47 +02:00
..