kevinveenbirkenbachandClaude Opus 5
786899700f
The gateway could reach exactly one remote provider, through a bare
OPENROUTER_API_KEY group_var with an empty-string default and an `{% if %}`
around a single hardcoded model entry. OpenAI and Anthropic had no path at all.
All three are now declared in roles/svc-ai-litellm/meta/secrets.yml, which
mapping.py already calls the application config schema, so a key lands at
applications.svc-ai-litellm.secrets.credentials.<provider>_api_key like every
other credential. Each carries the `type` and `regex` its value must satisfy and
defaults to the central API.<provider>.api_key block, which keeps one
operator-facing place to fill a provider in while the role owns the shape.
type and regex had to become real to be worth declaring. 192 `validation:`
blocks sit in the role schemas today and not one line of Python enforces them,
so a `regex` would have joined a pile of decoration. validate_supplied_value
now runs on every path a value arrives from outside — the --set override, a
value already in the inventory, and the value standing next to a `default:`.
A generated value stays exempt because the algorithm that produced it is named
by the same schema and generate_value takes no schema argument. min_length is
deliberately left unenforced: the generators ignore it, so holding them to it
would fail where nothing is wrong.
The pattern is written for the paste, not for the vendor. OpenAI moved issuance
from sk- to sk-proj- without notice and Anthropic issues sk-ant-api03-; a schema
pinning a prefix would reject a valid key and abort inventory creation. `\S{20,}`
accepts every form those three issue today and one nobody has announced, and
rejects a trailing newline, an embedded space, a truncated copy and a
<placeholder>. A key pasted into the wrong provider's slot is caught at deploy
instead, where the route it publishes answers nothing.
The gateway iterates one LITELLM_REMOTE_BACKENDS list instead of three `{% if %}`
blocks; config.yaml.j2 and env.j2 read the same list and selectattr('key')
decides. LITELLM_CHAT_MODEL falls back to the first configured remote alias
rather than to a hardcoded openrouter/auto.
For CI the keys travel as INFINITO_API_{OPENAI,ANTHROPIC,OPENROUTER}_KEY through
default.env, the passthrough allowlist, the infinito service in compose.yml, the
workflow env block and the development inventory. The compose leg is not
optional: the env lint rejected the change without it, because a key the
inventory reads but compose does not forward resolves to empty inside the
container with no error. The workflow reads `secrets.X || vars.X` so a variable
works as asked while a secret wins, which matters because a variable is not
masked and the development inventory sets MASK_CREDENTIALS_IN_LOGS: false.
An unset key is not an error anywhere: the provider's routes stay unpublished
and the CLI test asserts their absence rather than skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6.1 KiB
LiteLLM Gateway
Description
LiteLLM is an LLM gateway that exposes an OpenAI-compatible HTTP API and forwards each request to a configured model backend, local or hosted. Clients authenticate with virtual API keys that the gateway issues and stores in its own database.
Overview
This role deploys LiteLLM as a shared gateway container in Compose and Swarm deployments, backed by the central PostgreSQL service. The gateway is headless: it binds its HTTP port on the container host and claims no domain of its own, and the browser-facing surface is published separately by the LiteLLM Admin UI. It renders the model list from the interchangeable local backends available on the host, Ollama and LM Studio, and adds the OpenAI, Anthropic and OpenRouter entries whose API key is configured. Once the gateway is up, it provisions one virtual key per consuming application through the gateway admin API.
Cosmos
The diagram places LiteLLM Gateway in the Infinito.Nexus cosmos: the components it deploys (capabilities), the central services it consumes (dependencies), and its outward reach (federation and bridged external networks).
flowchart LR
subgraph deps [Dependencies]
dep_svc_ai_lmstudio["svc-ai-lmstudio 🐳🐝"]
dep_svc_ai_ollama["svc-ai-ollama 🐳🐝"]
dep_svc_db_postgres["svc-db-postgres 🐳🐝"]
end
subgraph role [svc-ai-litellm 🐳🐝]
svc_litellm["litellm"]
svc_postgres["postgres"]
svc_ollama["ollama"]
svc_lmstudio["lmstudio"]
end
subgraph dependents [Dependents]
dpt_web_app_discourse["web-app-discourse 🐳🐝"]
dpt_web_app_flowise["web-app-flowise 🐳🐝"]
dpt_web_app_hermes["web-app-hermes 🐳🐝"]
dpt_web_app_litellm["web-app-litellm 💻"]
dpt_web_app_matrix["web-app-matrix 🐳🐝"]
dpt_web_app_mattermost["web-app-mattermost 🐳🐝"]
dpt_web_app_mediawiki["web-app-mediawiki 🐳🐝"]
dpt_web_app_moodle["web-app-moodle 🐳🐝"]
dpt_web_app_n8n["web-app-n8n 🐳🐝"]
dpt_web_app_nextcloud["web-app-nextcloud 🐳🐝"]
dpt_web_app_openclaw["web-app-openclaw 🐳🐝"]
dpt_web_app_openwebui["web-app-openwebui 🐳🐝"]
dpt_more["..."]
end
dep_svc_ai_lmstudio -. "0..1" .-> svc_lmstudio
dep_svc_ai_ollama -. "0..1" .-> svc_ollama
dep_svc_db_postgres -. "0..1" .-> svc_postgres
svc_litellm -- "1:1" --> dpt_more
svc_litellm -. "0..1" .-> dpt_web_app_discourse
svc_litellm -. "0..1" .-> dpt_web_app_flowise
svc_litellm -. "0..1" .-> dpt_web_app_hermes
svc_litellm -- "1:1" --> dpt_web_app_litellm
svc_litellm -. "0..1" .-> dpt_web_app_matrix
svc_litellm -. "0..1" .-> dpt_web_app_mattermost
svc_litellm -. "0..1" .-> dpt_web_app_mediawiki
svc_litellm -. "0..1" .-> dpt_web_app_moodle
svc_litellm -. "0..1" .-> dpt_web_app_n8n
svc_litellm -. "0..1" .-> dpt_web_app_nextcloud
svc_litellm -. "0..1" .-> dpt_web_app_openclaw
svc_litellm -. "0..1" .-> dpt_web_app_openwebui
Solid 1:1 edges are fixed relationships; dashed 0..1 edges are conditional (enabled only in matching deployments); red 0..0 edges are turned off in this role. Node markers show the role's deploy modes (💻 host, 🐳 compose, 🐝 swarm); ❌ marks a service that is explicitly turned off, and ⚙️ an Ansible role dependency declared in meta/main.yml.
Features
- OpenAI-compatible API: One HTTP endpoint serves every model listed in the gateway configuration.
- Backend routing: Model entries are generated for Ollama and LM Studio when those services run on the host, under the shared alias each model carries so a consumer names one model whichever local backend answers it.
- Remote providers: OpenAI, Anthropic and OpenRouter each add their models when a key is configured. The keys are declared in
meta/secrets.ymlwith thetypeandregextheir value must satisfy, and default to the centralAPI.<provider>.api_keyentry; an unset key leaves that provider's routes unpublished. - Per-consumer virtual keys: Each consuming application receives its own virtual key, created through the gateway admin API under an alias naming that application.
- File-based model list: The model list is mounted as a read-only config file, with database-stored model entries turned off.
- Managed credentials: The gateway master key and the admin UI password are generated and kept as role credentials, and the admin UI username is the platform administrator name.
Quick Setup
Development
Clone, set up the workstation, and deploy LiteLLM Gateway onto the local stack:
git clone https://github.com/infinito-nexus/core.git
cd core
make onboard
make compose-deploy mode=reinstall apps=svc-ai-litellm full_cycle=false
Production
Run the published image to provision the inventory and deploy LiteLLM Gateway to a managed server (the mounted volume persists the inventory):
APP=svc-ai-litellm
HOST=<your-server>
DOMAIN=<your-domain>
TLS_MODE=self_signed
SSH_PUBLIC_KEY="<your-ssh-public-key>"
docker run --rm -it \
-v "$PWD/inventories:/etc/infinito.nexus/inventories" \
-e APP="$APP" -e HOST="$HOST" -e DOMAIN="$DOMAIN" -e TLS_MODE="$TLS_MODE" -e SSH_PUBLIC_KEY="$SSH_PUBLIC_KEY" \
ghcr.io/infinito-nexus/core/debian bash -c '
INVENTORY=/etc/infinito.nexus/inventories/production
infinito administration inventory provision "$INVENTORY" \
--inventory-file "$INVENTORY/devices.yml" \
--host "$HOST" \
--include "$APP" \
--vars "{\"TLS_MODE\": \"$TLS_MODE\", \"DOMAIN_PRIMARY\": \"$DOMAIN\", \"users\": {\"administrator\": {\"authorized_keys\": [\"$SSH_PUBLIC_KEY\"]}}}" &&
infinito administration deploy dedicated "$INVENTORY/devices.yml" \
--password-file "$INVENTORY/.password" \
--diff -vv'
Credits
Implemented by Kevin Veen-Birkenbach. Part of the Infinito.Nexus Project and maintained by Kevin Veen-Birkenbach. Licensed under the Infinito.Nexus Community License (Non-Commercial).