kevinveenbirkenbachandClaude Opus 5 54a6f3b533 feat(svc-ai-litellm): warm the chat model during the deploy ...

A backend pulls its preload model at deploy time but only loads it into
memory on the first inference. That cold load outlives a reverse proxy's
read timeout, so the first consumer request comes back as a 504 while every
later one succeeds.

Send one max_tokens=1 completion from inside the litellm container right
after the per-consumer keys exist, where the container id, master key and
port are already resolved. Five retries cover a gateway that is not
listening yet.

This pairs with the ollama keep_alive setting: the warm-up pays the load
once, keep_alive stops it decaying after five idle minutes.

Note this hardens the deploy: a gateway that cannot serve its configured
model now fails the role instead of surfacing later as a consumer error.
That makes the openrouter/auto fallback with its empty API key loud rather
than silent.

The script ships with a mirrored unit test rather than a nocheck waiver; it
patches urlopen and asserts the URL, method, authorization header, model,
max_tokens and that the timeout exceeds a proxy read timeout.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-28 12:02:17 +02:00

..

cli

…

library

…

plugins

…

roles

feat(svc-ai-litellm): warm the chat model during the deploy

2026-08-28 12:02:17 +02:00

scripts

…

utils

…

__init__.py

…