mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-08 19:56:53 +00:00
kevinveenbirkenbachandClaude Opus 5
973fccb59c
Ollama served qwen2.5:0.5b with its own default window of 4096 tokens and
truncated what did not fit: "truncating input prompt limit=2050
prompt=17997". The agents meanwhile assumed windows of their own, Hermes
131072 and OpenClaw 200000, so the model answered from the tail of a cut
prompt, which is where the tool-definition JSON came from. Hermes also
probed for the window it did not have, and LiteLLM answers one of those
probes, GET /v1/models/{id}, with 401 for a virtual key.
A model entry now declares its window: `context` on an AI_LOCAL_MODELS
entry or on a services.litellm.remote_models entry. The gateway sends it
to Ollama as num_ctx, so the runner serves that window instead of its
default, and publishes it as model_info.max_input_tokens. The field is
optional, because neither LM Studio's JIT load nor openrouter/auto can be
held to a declared number.
The broker gains services.agent-broker.agents.model, defaulting to the
gateway's chat model, so agents can run on a larger model than Open WebUI
serves by default. It looks the window of that alias up in the inventory
and writes it into the agent config as Hermes model.context_length and
OpenClaw contextWindow; an unknown window leaves both keys out. A deploy
refuses a model that resolved to an empty string or to an unrendered
template, because every agent would ask the gateway for it.
Measured on the local stack: a request carrying num_ctx moved the runner
from 4096 to 8192, a 14771-token prompt at 32768 answered without
truncation in 168 s, and the runner grew from 484 MB to 936 MB. Both
agents resolve the declared window from their config.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>