mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
refactor(openwebui): serve models through the gateway, not the model server
Open WebUI talked to Ollama directly. It now speaks the OpenAI-compatible surface of svc-ai-litellm with a per-consumer virtual key, so a second backend or an external provider becomes a gateway config change instead of a role change. A lint keeps future roles off the model servers. The gateway advertised a hardcoded model list (ollama/llama3, ollama/nomic-embed-text) while the matrix variant preloads smollm2:135m, so it offered only models the backend did not have and hid the one it did. The list is derived from the ollama preload SPOT now. The chat spec followed the rename: it gates on litellm instead of the removed ollama service flag, and picks the first non-embedding model from /api/models rather than pinning a variant-specific name, so it exercises the chain rather than a fixture. Routing openwebui through the gateway pulls both model backends into the closure of every role that consumes it, which pushed nextcloud variant 2 over the memory budget. The budget stays at 32/64 GB; instead the matrix variants of svc-ai-ollama and svc-ai-lmstudio now declare the size their test model needs (1g/2g) while the base keeps the 6g/8g that llama3 and mistral require. The budget guard only saw half of that: it replaced the evaluated role with its variant but counted every dependency from meta/services.yml, a combination no matrix round deploys. It now evaluates every variant-carrying role at the same round index, falling back to variant 0, the way the deploy planner does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kevinveenbirkenbachandClaude Opus 5 committed
1 parent
e3481b7745
commit
92c45e2471
17 files changed
+136
-36
No files matched your search
@@ -137,7 +137,7 @@ Every invokable role in the tested lifecycle envelope, with its upstream homepag
|
||||
| [Odoo ERP](roles/web-app-odoo/) | beta | Open-source ERP suite for CRM, sales, accounting, inventory, HR, and website management | [🌐](https://www.odoo.com/) [🎬](https://www.odoo.com/slides/getting-started-15) [🛠️](roles/web-app-odoo/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/), [Corporate Design](roles/web-svc-css/), [Dashboard](roles/web-app-dashboard/), [Keycloak](roles/web-app-keycloak/), [Mailu](roles/web-app-mailu/), [Matomo](roles/web-app-matomo/), [OpenLDAP](roles/svc-db-openldap/), [PostgreSQL](roles/svc-db-postgres/), [Prometheus](roles/web-app-prometheus/), [Redis](roles/svc-db-redis/), [Universal Logout](roles/web-svc-logout/) |
|
||||
| [Ollama](roles/svc-ai-ollama/) | beta | Self-hosted model server for running open LLMs locally behind a simple HTTP API | [🌐](https://ollama.com/) [🛠️](roles/svc-ai-ollama/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/) |
|
||||
| [OnlyOffice](roles/web-svc-onlyoffice/) | beta | Document server backend for real-time in-browser editing of documents, spreadsheets, and presentations | [🌐](https://www.onlyoffice.com/) [🎬](https://www.youtube.com/channel/UCNxeQm7vVujR8eFPtpVAVNg) [🛠️](roles/web-svc-onlyoffice/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/), [Corporate Design](roles/web-svc-css/), [Matomo](roles/web-app-matomo/), [PostgreSQL](roles/svc-db-postgres/), [Prometheus](roles/web-app-prometheus/), [Redis](roles/svc-db-redis/) |
|
||||
| [Open WebUI](roles/web-app-openwebui/) | beta | Self-hosted chat interface for interacting with local and private AI models via backends such as Ollama | [🌐](https://openwebui.com/) [🛠️](roles/web-app-openwebui/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/), [Corporate Design](roles/web-svc-css/), [Dashboard](roles/web-app-dashboard/), [Keycloak](roles/web-app-keycloak/), [Mailu](roles/web-app-mailu/), [Matomo](roles/web-app-matomo/), [Ollama](roles/svc-ai-ollama/), [OpenLDAP](roles/svc-db-openldap/), [Prometheus](roles/web-app-prometheus/), [SeaweedFS](roles/web-app-seaweedfs/), [Universal Logout](roles/web-svc-logout/) |
|
||||
| [Open WebUI](roles/web-app-openwebui/) | beta | Self-hosted chat interface for interacting with local and private AI models via backends such as Ollama | [🌐](https://openwebui.com/) [🛠️](roles/web-app-openwebui/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/), [Corporate Design](roles/web-svc-css/), [Dashboard](roles/web-app-dashboard/), [Keycloak](roles/web-app-keycloak/), [LiteLLM Gateway](roles/svc-ai-litellm/), [Mailu](roles/web-app-mailu/), [Matomo](roles/web-app-matomo/), [OpenLDAP](roles/svc-db-openldap/), [Prometheus](roles/web-app-prometheus/), [SeaweedFS](roles/web-app-seaweedfs/), [Universal Logout](roles/web-svc-logout/) |
|
||||
| [OpenClaw](roles/web-app-openclaw/) | beta | Self-hosted personal AI agent gateway with a browser control UI, sandboxed in a Kata microVM or gVisor runtime | [🌐](https://openclaw.ai/) [🛠️](roles/web-app-openclaw/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/), [Corporate Design](roles/web-svc-css/), [Dashboard](roles/web-app-dashboard/), [Kata & gVisor](roles/svc-virt-kata/), [Keycloak](roles/web-app-keycloak/), [LiteLLM Gateway](roles/svc-ai-litellm/), [Matomo](roles/web-app-matomo/), [Prometheus](roles/web-app-prometheus/), [Universal Logout](roles/web-svc-logout/) |
|
||||
| [OpenCloud](roles/web-app-opencloud/) | beta | Self-hosted platform for file sync, sharing, and collaboration with Keycloak SSO and central LDAP directory | [🌐](https://opencloud.eu/) [🎬](https://www.youtube.com/watch?v=Cg48rFNY0R4) [🛠️](roles/web-app-opencloud/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/), [Corporate Design](roles/web-svc-css/), [Dashboard](roles/web-app-dashboard/), [Keycloak](roles/web-app-keycloak/), [Mailu](roles/web-app-mailu/), [Matomo](roles/web-app-matomo/), [OpenLDAP](roles/svc-db-openldap/), [Prometheus](roles/web-app-prometheus/), [Universal Logout](roles/web-svc-logout/) |
|
||||
| [OpenLDAP](roles/svc-db-openldap/) | beta | Self-hosted LDAP directory service for centralized user, group, and schema management with TLS-secured access | [🌐](https://www.openldap.org/) [🛠️](roles/svc-db-openldap/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/) |
|
||||
|
||||
@@ -29,6 +29,7 @@ flowchart LR
|
||||
dpt_web_app_hermes["web-app-hermes 🐳🐝"]
|
||||
dpt_web_app_nextcloud["web-app-nextcloud 🐳🐝"]
|
||||
dpt_web_app_openclaw["web-app-openclaw 🐳🐝"]
|
||||
dpt_web_app_openwebui["web-app-openwebui 🐳🐝"]
|
||||
end
|
||||
dep_svc_ai_lmstudio -. "0..1" .-> svc_lmstudio
|
||||
dep_svc_ai_ollama -. "0..1" .-> svc_ollama
|
||||
@@ -36,6 +37,7 @@ flowchart LR
|
||||
svc_litellm -. "0..1" .-> dpt_web_app_hermes
|
||||
svc_litellm -. "0..1" .-> dpt_web_app_nextcloud
|
||||
svc_litellm -. "0..1" .-> dpt_web_app_openclaw
|
||||
svc_litellm -. "0..1" .-> dpt_web_app_openwebui
|
||||
```
|
||||
|
||||
Solid `1:1` edges are fixed relationships; dashed `0..1` edges are conditional (enabled only in matching deployments). Node markers show the role's deploy modes (💻 host, 🐳 compose, 🐝 swarm); ❌ marks a service that is explicitly turned off, and ⚙️ an Ansible role dependency declared in `meta/main.yml`.
|
||||
|
||||
@@ -1,17 +1,13 @@
|
||||
{% set has_backend = ('svc-ai-ollama' in group_names) or ('svc-ai-lmstudio' in group_names) or LITELLM_OPENROUTER_API_KEY %}
|
||||
{% set _ollama_models = lookup('config', 'svc-ai-ollama', 'services.ollama.preload_models') if 'svc-ai-ollama' in group_names else [] %}
|
||||
{% set has_backend = _ollama_models or ('svc-ai-lmstudio' in group_names) or LITELLM_OPENROUTER_API_KEY %}
|
||||
model_list:{{ ' []' if not has_backend else '' }}
|
||||
{% if 'svc-ai-ollama' in group_names %}
|
||||
- model_name: ollama/llama3
|
||||
{% for _model in _ollama_models %}
|
||||
- model_name: "{{ _model }}"
|
||||
litellm_params:
|
||||
model: ollama/llama3
|
||||
model: "ollama/{{ _model }}"
|
||||
api_base: {{ OLLAMA_BASE_LOCAL_URL }}
|
||||
rpm: 1000
|
||||
- model_name: ollama/nomic-embed-text
|
||||
litellm_params:
|
||||
model: ollama/nomic-embed-text
|
||||
api_base: {{ OLLAMA_BASE_LOCAL_URL }}
|
||||
rpm: 1000
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
{% if 'svc-ai-lmstudio' in group_names %}
|
||||
- model_name: lmstudio/default
|
||||
litellm_params:
|
||||
|
||||
@@ -5,3 +5,6 @@
|
||||
- services:
|
||||
container_backup:
|
||||
enabled: true
|
||||
lmstudio:
|
||||
mem_reservation: 1g
|
||||
mem_limit: 2g
|
||||
@@ -25,13 +25,11 @@ flowchart LR
|
||||
dpt_svc_ai_litellm["svc-ai-litellm 🐳🐝"]
|
||||
dpt_web_app_flowise["web-app-flowise 🐳🐝"]
|
||||
dpt_web_app_minio["web-app-minio 🐳🐝"]
|
||||
dpt_web_app_openwebui["web-app-openwebui 🐳🐝"]
|
||||
end
|
||||
dep_svc_bkp_volume_2_local -. "0..1" .-> svc_container_backup
|
||||
svc_ollama -. "0..1" .-> dpt_svc_ai_litellm
|
||||
svc_ollama -. "0..1" .-> dpt_web_app_flowise
|
||||
svc_ollama -. "0..1" .-> dpt_web_app_minio
|
||||
svc_ollama -. "0..1" .-> dpt_web_app_openwebui
|
||||
```
|
||||
|
||||
Solid `1:1` edges are fixed relationships; dashed `0..1` edges are conditional (enabled only in matching deployments). Node markers show the role's deploy modes (💻 host, 🐳 compose, 🐝 swarm); ❌ marks a service that is explicitly turned off, and ⚙️ an Ansible role dependency declared in `meta/main.yml`.
|
||||
|
||||
@@ -13,3 +13,5 @@
|
||||
ollama:
|
||||
preload_models:
|
||||
- smollm2:135m
|
||||
mem_reservation: 1g
|
||||
mem_limit: 2g
|
||||
@@ -15,7 +15,7 @@ The diagram places Open WebUI in the Infinito.Nexus cosmos: the components it de
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph deps [Dependencies]
|
||||
dep_svc_ai_ollama["svc-ai-ollama 🐳🐝"]
|
||||
dep_svc_ai_litellm["svc-ai-litellm 🐳🐝"]
|
||||
dep_svc_bkp_volume_2_local["svc-bkp-volume-2-local 💻"]
|
||||
dep_svc_db_openldap["svc-db-openldap 🐳🐝"]
|
||||
dep_web_app_dashboard["web-app-dashboard 🐳🐝"]
|
||||
@@ -39,7 +39,7 @@ flowchart LR
|
||||
svc_seaweedfs["seaweedfs"]
|
||||
svc_css["css"]
|
||||
svc_javascript["javascript"]
|
||||
svc_ollama["ollama"]
|
||||
svc_litellm["litellm"]
|
||||
svc_email["email"]
|
||||
svc_prometheus["prometheus"]
|
||||
svc_container_backup["container_backup"]
|
||||
@@ -47,7 +47,7 @@ flowchart LR
|
||||
subgraph dependents [Dependents]
|
||||
dpt_web_app_nextcloud["web-app-nextcloud 🐳🐝"]
|
||||
end
|
||||
dep_svc_ai_ollama -. "0..1" .-> svc_ollama
|
||||
dep_svc_ai_litellm -. "0..1" .-> svc_litellm
|
||||
dep_svc_bkp_volume_2_local -. "0..1" .-> svc_container_backup
|
||||
dep_svc_db_openldap -. "0..1" .-> svc_ldap
|
||||
dep_web_app_dashboard -. "0..1" .-> svc_dashboard
|
||||
|
||||
@@ -13,5 +13,5 @@ require("./test-admin-native-login").register(shared);
|
||||
require("./test-admin-ldap-login").register(shared);
|
||||
require("./test-biber-ldap-login").register(shared);
|
||||
require("./test-guest-persona").register(shared);
|
||||
require("./test-ollama-chat").register(shared);
|
||||
require("./test-litellm-chat").register(shared);
|
||||
require("./test-seaweedfs");
|
||||
+8
-8
@@ -2,8 +2,8 @@ const { test, expect } = require("@playwright/test");
|
||||
const { skipUnlessServiceEnabled } = require("./service-gating");
|
||||
|
||||
exports.register = function (shared) {
|
||||
test("ollama chat: the preloaded model answers a chat completion via OpenWebUI", async ({ page }) => {
|
||||
skipUnlessServiceEnabled("ollama");
|
||||
test("litellm chat: a gateway model answers a chat completion via OpenWebUI", async ({ page }) => {
|
||||
skipUnlessServiceEnabled("litellm");
|
||||
test.setTimeout(120_000);
|
||||
|
||||
await shared.signInViaDashboardOidc(
|
||||
@@ -31,30 +31,30 @@ exports.register = function (shared) {
|
||||
const modelsBody = await modelsResp.json();
|
||||
const modelIds = (Array.isArray(modelsBody?.data) ? modelsBody.data : [])
|
||||
.map((m) => String(m?.id ?? m?.name ?? ""));
|
||||
const ollamaModel = modelIds.find((id) => id.startsWith("smollm2"));
|
||||
const chatModel = modelIds.find((id) => id && !/embed/i.test(id));
|
||||
expect(
|
||||
ollamaModel,
|
||||
`the preloaded Ollama model (smollm2) must be served by OpenWebUI, proving the Ollama backend is reachable and the model pulled (got ${JSON.stringify(modelIds)})`
|
||||
chatModel,
|
||||
`a chat model must be served by OpenWebUI through the LiteLLM gateway, proving gateway and backend are reachable and the model pulled (got ${JSON.stringify(modelIds)})`
|
||||
).toBeTruthy();
|
||||
|
||||
const chatResp = await page.request.post(`${base}/api/chat/completions`, {
|
||||
headers,
|
||||
data: {
|
||||
model: ollamaModel,
|
||||
model: chatModel,
|
||||
messages: [{ role: "user", content: "Reply with exactly the word: pong" }],
|
||||
stream: false,
|
||||
},
|
||||
});
|
||||
expect(
|
||||
chatResp.ok(),
|
||||
`the ${ollamaModel} chat completion must return 200 over OpenWebUI -> Ollama (HTTP ${chatResp.status()})`
|
||||
`the ${chatModel} chat completion must return 200 over OpenWebUI -> LiteLLM (HTTP ${chatResp.status()})`
|
||||
).toBeTruthy();
|
||||
|
||||
const chatBody = await chatResp.json();
|
||||
const content = (chatBody?.choices?.[0]?.message?.content ?? "").trim();
|
||||
expect(
|
||||
content.length,
|
||||
`the model must return a non-empty assistant response over OpenWebUI -> Ollama -> ${ollamaModel} (got ${JSON.stringify(chatBody).slice(0, 300)})`
|
||||
`the model must return a non-empty assistant response over OpenWebUI -> LiteLLM -> ${chatModel} (got ${JSON.stringify(chatBody).slice(0, 300)})`
|
||||
).toBeGreaterThan(0);
|
||||
});
|
||||
};
|
||||
@@ -0,0 +1,7 @@
|
||||
---
|
||||
credentials:
|
||||
litellm_api_key:
|
||||
description: Per-consumer virtual key for the shared svc-ai-litellm gateway
|
||||
algorithm: sk_prefixed
|
||||
validation:
|
||||
min_length: 50
|
||||
@@ -81,10 +81,10 @@ javascript:
|
||||
enabled: false # nocheck: dynamic-flag
|
||||
shared: true
|
||||
# nocheck: playwright-service-flag — AI backend, infrastructural
|
||||
ollama:
|
||||
litellm:
|
||||
bond: 1
|
||||
enabled: "{{ 'svc-ai-ollama' in group_names }}"
|
||||
shared: "{{ 'svc-ai-ollama' in group_names }}"
|
||||
enabled: "{{ 'svc-ai-litellm' in group_names }}"
|
||||
shared: "{{ 'svc-ai-litellm' in group_names }}"
|
||||
email:
|
||||
bond: 1
|
||||
enabled: "{{ 'web-app-mailu' in group_names }}"
|
||||
|
||||
@@ -25,7 +25,7 @@
|
||||
sso:
|
||||
enabled: true
|
||||
shared: true
|
||||
ollama:
|
||||
litellm:
|
||||
enabled: true
|
||||
shared: true
|
||||
prometheus:
|
||||
@@ -59,7 +59,7 @@
|
||||
sso:
|
||||
enabled: false
|
||||
shared: false # nocheck: variants-static-override backup shared stays literally true
|
||||
ollama:
|
||||
litellm:
|
||||
enabled: false
|
||||
shared: false # nocheck: variants-static-override backup shared stays literally true
|
||||
prometheus:
|
||||
@@ -96,7 +96,7 @@
|
||||
matomo:
|
||||
enabled: false
|
||||
shared: false # nocheck: variants-static-override backup shared stays literally true
|
||||
ollama:
|
||||
litellm:
|
||||
enabled: false
|
||||
shared: false # nocheck: variants-static-override backup shared stays literally true
|
||||
prometheus:
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
{% include 'roles/sys-svc-container/templates/env.j2' %}
|
||||
|
||||
OLLAMA_BASE_URL={{ OLLAMA_BASE_LOCAL_URL }}
|
||||
{% if OPENWEBUI_LITELLM_ENABLED | bool %}
|
||||
OPENAI_API_BASE_URL={{ LITELLM_BASE_LOCAL_URL }}/v1
|
||||
OPENAI_API_KEY={{ OPENWEBUI_LITELLM_API_KEY | dotenv_quote }}
|
||||
{% endif %}
|
||||
OFFLINE_MODE={{ OPENWEBUI_OFFLINE_MODE | ternary(1, 0) }}
|
||||
HF_HUB_OFFLINE={{ OPENWEBUI_HF_HUB_OFFLINE | ternary(1, 0) }}
|
||||
ENABLE_PERSISTENT_CONFIG=false
|
||||
|
||||
@@ -18,10 +18,11 @@ EMAIL_SERVICE_ENABLED={{ lookup('config', application_id, 'services.email.enable
|
||||
# nocheck: playwright-service-gate — persona drives the OIDC path; LDAP federation is exercised in the LDAP-only matrix variant
|
||||
LDAP_SERVICE_ENABLED={{ lookup('config', application_id, 'services.ldap.enabled') | string | lower }}
|
||||
|
||||
LITELLM_SERVICE_ENABLED={{ lookup('config', application_id, 'services.litellm.enabled') | string | lower }}
|
||||
|
||||
# nocheck: playwright-service-gate — consumed via shared persona helpers (roles/test-e2e-playwright/files/personas/)
|
||||
LOGOUT_SERVICE_ENABLED={{ lookup('config', application_id, 'services.logout.enabled') | string | lower }}
|
||||
|
||||
# Service gate flags rendered for closure.
|
||||
MATOMO_BASE_URL={{ lookup('tls', 'web-app-matomo', 'url.base') | dotenv_quote }}
|
||||
|
||||
# nocheck: playwright-service-gate — consumed via shared persona helpers (roles/test-e2e-playwright/files/personas/)
|
||||
@@ -29,8 +30,6 @@ MATOMO_SERVICE_ENABLED={{ lookup('config', application_id, 'services.matomo.enab
|
||||
|
||||
OIDC_ISSUER_URL={{ OIDC.CLIENT.ISSUER_URL | dotenv_quote }}
|
||||
|
||||
OLLAMA_SERVICE_ENABLED={{ lookup('config', application_id, 'services.ollama.enabled') | string | lower }}
|
||||
|
||||
OPENWEBUI_BASE_URL={{ lookup('tls', application_id, 'url.base') | dotenv_quote }}
|
||||
|
||||
SEAWEEDFS_ADMIN_PASSWORD={{ lookup('users', 'administrator').password | dotenv_quote }}
|
||||
|
||||
@@ -7,6 +7,8 @@ OPENWEBUI_OFFLINE_MODE: "{{ lookup('config', application_id, 'services.openwebui
|
||||
OPENWEBUI_HF_HUB_OFFLINE: "{{ lookup('config', application_id, 'services.openwebui.hf_hub_offline') }}"
|
||||
OPENWEBUI_ENABLE_API_KEYS: "{{ lookup('config', application_id, 'services.openwebui.enable_api_keys') }}"
|
||||
OPENWEBUI_LDAP_ENABLED: "{{ lookup('config', application_id, 'services.ldap.enabled') }}"
|
||||
OPENWEBUI_LITELLM_ENABLED: "{{ lookup('config', application_id, 'services.litellm.enabled') }}"
|
||||
OPENWEBUI_LITELLM_API_KEY: "{{ lookup('config', application_id, 'credentials.litellm_api_key') }}"
|
||||
|
||||
OPENWEBUI_OIDC_ENABLED: "{{ lookup('config', application_id, 'services.sso.enabled') }}"
|
||||
OPENWEBUI_OIDC_REDIRECT_URL: "{{ [OPENWEBUI_URL,'/oauth/oidc/callback'] | url_join }}"
|
||||
|
||||
@@ -4,7 +4,11 @@ the host memory budget.
|
||||
For every role and every entry in its ``meta/variants.yml``, the deduplicated
|
||||
footprint (the role's own containers plus the shared dependencies it pulls in,
|
||||
each service counted once via the shared-service logic) is summed and checked
|
||||
against:
|
||||
against the budget below. Every role that ships variants is evaluated at the
|
||||
same round index, falling back to variant 0 when it has fewer, because that is
|
||||
the combination a matrix round actually deploys: a dependency sized for the
|
||||
production model tier is not what the round runs. ``meta/services.yml`` only
|
||||
supplies the roles that declare no variant at all.
|
||||
|
||||
- ``mem_reservation`` total ≤ 32 GB
|
||||
- ``mem_limit`` total ≤ 64 GB
|
||||
@@ -67,6 +71,11 @@ def _collect_findings(root: Path) -> list[BudgetFinding]:
|
||||
for role, variant_list in sorted(variants.items()):
|
||||
for index, variant_config in enumerate(variant_list):
|
||||
scoped = dict(applications)
|
||||
for dep_role, dep_variants in variants.items():
|
||||
if not dep_variants:
|
||||
continue
|
||||
dep_index = index if index < len(dep_variants) else 0
|
||||
scoped[dep_role] = dep_variants[dep_index] or {}
|
||||
scoped[role] = variant_config or {}
|
||||
rows: list = []
|
||||
collect_role_resources(
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
"""Keep model traffic on the gateway instead of a provider URL.
|
||||
|
||||
Rationale
|
||||
=========
|
||||
``svc-ai-litellm`` is the single entrypoint to every model backend: it owns the
|
||||
model list, the per-consumer virtual keys and the routing. An application that
|
||||
addresses a provider directly bypasses all three, so its traffic carries no
|
||||
consumer identity, ignores the routing, and keeps working after the model list
|
||||
moved on.
|
||||
|
||||
The gateway itself is the one legitimate holder of provider URLs, because
|
||||
routing to them is its job.
|
||||
|
||||
Per-line opt-out
|
||||
================
|
||||
Add ``# nocheck: model-backend-via-gateway`` on the offending line or on the
|
||||
immediately preceding non-empty line, together with the reason this consumer
|
||||
cannot go through the gateway.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from utils.annotations.suppress import is_suppressed_at
|
||||
from utils.cache.files import iter_project_files_with_content
|
||||
|
||||
from . import PROJECT_ROOT
|
||||
|
||||
_RULE = "model-backend-via-gateway"
|
||||
|
||||
_GATEWAY_ROLE = "svc-ai-litellm"
|
||||
|
||||
_PROVIDER_URL = re.compile(r"\{\{[^}]*\b(OLLAMA|LMSTUDIO)_BASE_LOCAL_URL\b")
|
||||
|
||||
|
||||
class TestModelBackendViaGateway(unittest.TestCase):
|
||||
def test_only_the_gateway_addresses_a_provider_directly(self) -> None:
|
||||
findings: list[tuple[str, int, str]] = []
|
||||
for path_str, content in iter_project_files_with_content(
|
||||
extensions=(".j2",),
|
||||
exclude_tests=True,
|
||||
):
|
||||
rel = Path(path_str).relative_to(PROJECT_ROOT).as_posix()
|
||||
parts = rel.split("/")
|
||||
if len(parts) < 2 or parts[0] != "roles" or "/templates/" not in rel:
|
||||
continue
|
||||
if parts[1] == _GATEWAY_ROLE:
|
||||
continue
|
||||
lines = content.splitlines()
|
||||
for idx, line in enumerate(lines):
|
||||
if not _PROVIDER_URL.search(line):
|
||||
continue
|
||||
if is_suppressed_at(lines, idx + 1, _RULE, mode="same-or-above"):
|
||||
continue
|
||||
findings.append((rel, idx + 1, line.strip()))
|
||||
|
||||
if findings:
|
||||
formatted = "\n".join(
|
||||
f"- {p}:{n}: {s}"
|
||||
for p, n, s in sorted(set(findings), key=lambda i: (i[0], i[1]))
|
||||
)
|
||||
self.fail(
|
||||
"These templates address a model provider directly instead of "
|
||||
f"routing through {_GATEWAY_ROLE}, so their traffic carries no "
|
||||
"consumer identity and ignores the gateway's model list.\n\n"
|
||||
"Fix: declare `services.litellm` in the role's meta/services.yml "
|
||||
"and a `litellm_api_key` credential in its meta/schema.yml, then "
|
||||
"point the application at LITELLM_BASE_LOCAL_URL with that key. "
|
||||
"The gateway provisions the virtual key for every role that "
|
||||
"declares both.\n\n"
|
||||
f"Offending lines:\n{formatted}"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user