mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
kevinveenbirkenbachandClaude Opus 5
54a6f3b533
A backend pulls its preload model at deploy time but only loads it into memory on the first inference. That cold load outlives a reverse proxy's read timeout, so the first consumer request comes back as a 504 while every later one succeeds. Send one max_tokens=1 completion from inside the litellm container right after the per-consumer keys exist, where the container id, master key and port are already resolved. Five retries cover a gateway that is not listening yet. This pairs with the ollama keep_alive setting: the warm-up pays the load once, keep_alive stops it decaying after five idle minutes. Note this hardens the deploy: a gateway that cannot serve its configured model now fails the role instead of surfacing later as a consumer error. That makes the openrouter/auto fallback with its empty API key loud rather than silent. The script ships with a mirrored unit test rather than a nocheck waiver; it patches urlopen and asserts the URL, method, authorization header, model, max_tokens and that the timeout exceeds a proxy read timeout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
47 lines
1.3 KiB
Python
47 lines
1.3 KiB
Python
"""Force the gateway to load its chat model before any consumer prompts it.
|
|
|
|
A backend pulls its preload model at deploy time but only loads it into memory
|
|
on the first inference. That cold load outlives the reverse proxy's read
|
|
timeout, so the first consumer request comes back as a 504 instead of an answer
|
|
while every later one succeeds. Warming here moves the cost into the deploy.
|
|
|
|
Environment:
|
|
LITELLM_MK: master key the gateway accepts.
|
|
LITELLM_PORT: port the gateway listens on inside its own container.
|
|
LITELLM_MODEL: model name to warm.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import os
|
|
import urllib.request
|
|
|
|
MASTER_KEY = os.environ["LITELLM_MK"]
|
|
PORT = os.environ["LITELLM_PORT"]
|
|
MODEL = os.environ["LITELLM_MODEL"]
|
|
TIMEOUT = 900
|
|
|
|
payload = json.dumps(
|
|
{
|
|
"model": MODEL,
|
|
"messages": [{"role": "user", "content": "ping"}],
|
|
"max_tokens": 1,
|
|
}
|
|
).encode()
|
|
|
|
request = urllib.request.Request(
|
|
f"http://127.0.0.1:{PORT}/v1/chat/completions",
|
|
data=payload,
|
|
headers={
|
|
"Authorization": f"Bearer {MASTER_KEY}",
|
|
"Content-Type": "application/json",
|
|
},
|
|
method="POST",
|
|
)
|
|
|
|
with urllib.request.urlopen(request, timeout=TIMEOUT) as response: # noqa: S310 fixed internal http origin
|
|
response.read()
|
|
|
|
print(f"WARMED {MODEL}")
|