fix(svc-ai-ollama): preload a model that stops generating

The matrix variant preloaded smollm2:135m, which is too small to emit a
stop token reliably. Ollama's own log for the prompt "hi" shows it running
away rather than running slowly:

  n_gen = 22011, tg = 86.14 t/s
  slot context shift, n_keep = 4, n_left = 4091, n_discard = 2045
  aborting completion request due to client closing the connection

86 tokens per second is fast; 22000 tokens with repeated context shifts is
a model that never finishes. Every completion without a hard token cap ran
until the consumer's own timeout, which is why zammad reported
AI::Provider::ResponseError after exactly 300s while short warm-ups and
provider pings returned 200 immediately.

Preload qwen2.5:0.5b instead: a comparable footprint (397MB vs 270MB) that
is instruction-tuned and terminates.

web-app-openwebui asserted the served model with a hardcoded
id.startsWith("smollm2"), so it would have failed on any model change.
Render the deployed preload model into its Playwright env and compare
against that instead of a literal family name.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kevinveenbirkenbachandClaude Opus 5 committed 2026-08-28 11:54:15 +02:00
1 parent b149ad0d22
commit 8dbdd31b22
3 files changed
+6 -3

No files matched your search

+1 -1
View File
@@ -12,6 +12,6 @@
enabled: true
ollama:
preload_models:
- smollm2:135m
- qwen2.5:0.5b
mem_reservation: 1g
mem_limit: 2g
@@ -32,10 +32,11 @@ exports.register = function (shared) {
const modelsBody = await modelsResp.json();
const modelIds = (Array.isArray(modelsBody?.data) ? modelsBody.data : [])
.map((m) => String(m?.id ?? m?.name ?? ""));
const ollamaModel = modelIds.find((id) => id.startsWith("smollm2"));
const preloadedModel = String(process.env.OLLAMA_PRELOAD_MODEL || "").replace(/^"|"$/g, "");
const ollamaModel = modelIds.find((id) => id === preloadedModel || id.startsWith(preloadedModel.split(":")[0]));
expect(
ollamaModel,
`the preloaded Ollama model (smollm2) must be served by OpenWebUI, proving the Ollama backend is reachable and the model pulled (got ${JSON.stringify(modelIds)})`
`the preloaded Ollama model (${preloadedModel}) must be served by OpenWebUI, proving the Ollama backend is reachable and the model pulled (got ${JSON.stringify(modelIds)})`
).toBeTruthy();
const chatResp = await page.request.post(`${base}/api/chat/completions`, {
@@ -44,6 +44,8 @@ MCP_SERVICE_ENABLED={{ lookup('config', application_id, 'mcp.enabled') | string
OIDC_ISSUER_URL={{ OIDC.CLIENT.ISSUER_URL | dotenv_quote }}
OLLAMA_PRELOAD_MODEL={{ (lookup('config', 'svc-ai-ollama', 'services.ollama.preload_models') | reject('search', 'embed') | list | first) | dotenv_quote }}
OPENWEBUI_BASE_URL={{ lookup('tls', application_id, 'url.base') | dotenv_quote }}
SEAWEEDFS_ADMIN_PASSWORD={{ lookup('users', 'administrator').password | dotenv_quote }}