computer-playbook/docs/requirements/031-llm-gateway-model-backends.md

031 - LLM Gateway and Model Backends (LiteLLM)

User Story

As a platform administrator of Infinito.Nexus, I want a single OpenAI-compatible LLM gateway that routes to any model backend (local Ollama, LM Studio, OpenRouter, ...), so that every AI consumer (OpenWebUI, Flowise, and the agent employees in 032) targets one /v1 endpoint and I swap or add model backends centrally without touching any consumer.

Background

AI consumers in this repository each wire their own model provider today: web-app-flowise ships a role-local LiteLLM config, others point directly at svc-ai-ollama. This duplicates provider wiring and hard-codes backends into consumers.

A single OpenAI-compatible LLM gateway (LiteLLM Proxy) fronts every backend: local svc-ai-ollama, a new svc-ai-lmstudio, OpenRouter, and any other provider LiteLLM supports. Consumers speak only the gateway /v1 with a model alias; provider selection and routing are gateway config. This requirement promotes the existing Flowise-local LiteLLM pattern to a shared svc-ai-litellm role and makes it the one place backends are declared.

This is the model plane. The tool plane (MCP) is separate (025); the agent fleet that consumes this gateway is 032.

Confirmed Decisions

These choices are settled at requirement creation time and bound the implementation. Re-opening any of them MUST be recorded in the implementing PR.

1. A svc-ai-litellm gateway is the single model entrypoint. It exposes one OpenAI-compatible /v1 endpoint and routes to backends via its model_list. No consumer addresses a provider directly.
2. svc-ai-ollama stays the default local backend behind the gateway. No new model-serving stack replaces it; it is registered as a model_list entry.
3. A svc-ai-lmstudio role is added as a routable backend, x86/GPU node only. It deploys LM Studio in headless server mode (lms server, OpenAI-compatible API) and registers behind the gateway. LM Studio has no arm64 Linux server build, so it MUST be pinned to an x86 (typically GPU) node via placement constraint and MUST NOT be scheduled onto the Raspberry Pi / arm64 agent nodes of 032. LM Studio ships primarily as a desktop GUI; the role MUST use its headless/server mode, and the README MUST document the container caveat (GPU passthrough, licensing/model-download constraints).
4. Flowise's role-local LiteLLM is refactored into the shared role. The role-local LiteLLM config and container in web-app-flowise (templates/litellm.config.yaml.j2, its compose service, related env) are removed; Flowise consumes svc-ai-litellm instead. No duplicated LiteLLM deployment remains. Flowise stays green after the refactor.
5. Consumers carry only gateway URL + model alias. Switching or adding a backend is a svc-ai-litellm config change and MUST NOT require a consumer redeploy.
6. The gateway is authenticated with per-consumer virtual keys. LiteLLM virtual keys are issued per consumer (OpenWebUI, Flowise, and each 032 agent); a master key is admin-only. This gives per-consumer rate limiting and revocation. The gateway MUST NOT be reachable unauthenticated even on the internal overlay. Keys live in each consumer role's meta/secrets.yml credentials: block.
7. svc-ai-litellm is headless and internal-only; the admin UI is a separate optional web-app-litellm role. The gateway itself exposes no public domain and no LiteLLM admin UI. A distinct web-app-litellm role serves the admin interface behind the proxy with SSO, following the standard web-app-* contract. Coupling is asymmetric: web-app-litellm REQUIRES the gateway (deploying it forces svc-ai-litellm to enabled: true + shared: true), while svc-ai-litellm only OPTIONALLY references the UI (via group_names) and runs fine without it.

Component Roles

Every consumer speaks only the gateway /v1; the gateway is the single place backends are wired.

Component	Role	Interactive?
web-app-openwebui	Human chat UI to models (and MCP client per 025)	Human, on demand
web-app-flowise	No-code builder for LLM apps / agent flows	Human, on demand
web-app-hermes, web-app-openclaw (032)	Autonomous agent employees	Autonomous
svc-ai-litellm	One OpenAI-compatible /v1, routes to any backend (headless, internal-only)	—
web-app-litellm	Optional admin UI for the gateway (proxy + SSO); requires svc-ai-litellm	Human, on demand
svc-ai-ollama	Local small models	—
svc-ai-lmstudio	Local headless models	—

flowchart TB
    human([Human])

    subgraph consumers["Model consumers"]
        owui["web-app-openwebui<br/>chat UI"]
        flowise["web-app-flowise<br/>flow builder"]
        agents["web-app-hermes / web-app-openclaw<br/>(032, autonomous)"]
    end

    gw["svc-ai-litellm<br/>LLM gateway (one /v1)"]

    subgraph back["Model backends (wired only here)"]
        ollama["svc-ai-ollama<br/>local small"]
        lms["svc-ai-lmstudio<br/>local headless"]
        openrouter["OpenRouter<br/>external / hosted"]
    end

    human -->|chat| owui
    human -->|build flows| flowise
    owui & flowise & agents -->|/v1 + model alias| gw
    gw -->|route by model| ollama
    gw -->|route by model| lms
    gw -->|route by model| openrouter

Backend blockers

lmstudio/llmster-preview:cpu publishes one linux/amd64 manifest, runs /app/daemon-run.sh as its entrypoint and exposes 1234/tcp (measured against the registry config blob on 2026-08-28). The role wires that server and the gateway routes lmstudio/default to it, but a completion additionally needs a model in the container's store on an x86 node with the memory to load it. Until such a node joins the deployment, verify the gateway against svc-ai-ollama and leave the LM Studio completion criterion unchecked.

Closing probe once the node exists: POST {gateway}/v1/chat/completions with "model": "lmstudio/default" and a consumer virtual key, expecting a non-empty choices[0].message.content.

Where each guarantee is enforced

Guarantee	Enforced by
No consumer addresses a backend directly	test_model_backend_via_gateway.py, over role templates/ and vars/
Every declared backend has a model_list branch	test_gateway_backend_registered.py
Every consumer mints its own key and runs the gateway contract	test_litellm_consumer_contract.py
Gateway is closed, keys are per consumer, revocation is isolated	probe_auth.py, run on every gateway deploy; unit-tested in test_probe_auth.py
Empty key or out-of-deployment base URL fails the deploy	consumer_contract.yml
The admin UI sits behind Keycloak	test-oidc-login.js, gated on services.sso

Decision 7's "proxy + SSO" is served by services.sso.flavor: oidc on web-app-litellm plus LiteLLM's own GENERIC_* OIDC settings, rendered into the gateway's env behind LITELLM_UI_SSO. An oauth2-proxy sidecar is not an option here: get_entity_name maps both web-app-litellm and svc-ai-litellm to litellm, so the two roles share one compose directory, one swarm stack name and one docker network name, and a second stack would overwrite the gateway's compose.yml and delete its service under docker stack deploy --prune. web-app-bigbluebutton is the precedent for a stackless role carrying flavor: oidc. The trade-off: an anonymous request reaches LiteLLM's own login page rather than being refused at the edge, and LiteLLM caps SSO at five accounts without a licence.

Re-routing an alias is a one-line edit to roles/svc-ai-litellm/templates/config.yaml.j2 followed by make compose-deploy apps=svc-ai-litellm; confirm no consumer redeploy is needed by prompting a consumer afterwards without redeploying it.

Acceptance Criteria

• A svc-ai-litellm role exposes one OpenAI-compatible /v1 endpoint; its model_list routes to at least svc-ai-ollama, svc-ai-lmstudio, and one external provider (OpenRouter).
• A svc-ai-lmstudio role deploys LM Studio in headless server mode with an OpenAI-compatible API reachable in-cluster, registered behind the gateway.
• A request to a model alias backed by svc-ai-lmstudio returns a completion through the gateway. Blocked: see Backend blockers.
• web-app-flowise is refactored to consume svc-ai-litellm; its role-local LiteLLM config, compose service, and env are removed, no duplicated LiteLLM deployment remains, and Flowise stays green.
• Switching a consumer's backend requires only a svc-ai-litellm config change, verified by re-routing a model alias with no consumer redeploy.
• The gateway rejects unauthenticated requests; each consumer authenticates with its own LiteLLM virtual key, and revoking one key blocks only that consumer.
• svc-ai-litellm is headless (no admin UI, no public domain); a separate web-app-litellm role serves the admin UI behind proxy + SSO, and deploying it forces svc-ai-litellm to enabled: true + shared: true, while the gateway deploys and runs without the UI role.
• svc-ai-lmstudio is constrained to an x86 node and is never scheduled onto an arm64/Pi node.
• The gateway comes up green in both compose and swarm modes.
• A Playwright spec (or equivalent test) verifies a completion round-trips through the gateway.

Cross-linking

• Implementing PR: to be linked.

See Also

• Agent fleet that consumes this gateway: 032-agent-employees-firecracker.md
• MCP tool plane: 025-mcp-role-integration.md
• Local model backend: svc-ai-ollama