mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 02:54:26 +00:00
fix(ai): pin each LM Studio model to its exact source, not to a search term
Run 34893750601 lost four jobs at svc-ai-litellm's warm request, every one of
them on the same upstream body:
No models loaded. Please load a model in the developer page or use the
`lms load` command.
The catalogue handed `lms get --yes --gguf` a bare name, which that command
treats as a hub search term. Its own help spells out the consequence: "If there
are multiple staff picks matching the search term, the first one will be used."
For `qwen2.5-0.5b-instruct` the first pick was Qwen2.5-Coder-0.5B-Instruct, so
the key LM Studio indexed never matched the `openai/{{ _model.name }}` the
gateway sends upstream, and just-in-time loading had nothing to load.
A `source:` now carries the full HuggingFace URL, the one spelling lms get
resolves deterministically; a bare owner/model pair addresses an LM Studio Hub
artifact instead and fails even for a model already on disk.
Measured against the shipped image rather than reasoned: the URL yields modelKey
qwen2.5-0.5b-instruct, `lms ps` is empty right before the request, and a plain
/v1/chat/completions for that name answers 200 and leaves the model resident. No
load step belongs here, because the server reports "Just-in-time model loading
active" and does it itself.
The embedding entry is renamed in the same breath. LM Studio indexes that model
as text-embedding-nomic-embed-text-v1.5, so the gateway was addressing a key
that never existed there either, and the first consumer to embed would have hit
the same 404.
The three fields stay separate and each has exactly one reader: Ollama pulls by
alias, LM Studio downloads by source, the gateway addresses name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
5097ccdfe3
commit
f8da0fd122
3 files changed
+6
-2
No files matched your search
@@ -16,10 +16,13 @@ LITELLM_BACKEND_ROLES: "{{ lookup('deployment').inventory }}"
|
||||
AI_LOCAL_MODELS:
|
||||
- alias: llama3:latest
|
||||
name: llama-3.2-3b-instruct
|
||||
source: https://huggingface.co/lmstudio-community/Llama-3.2-3B-Instruct-GGUF
|
||||
- alias: mistral:latest
|
||||
name: mistral-7b-instruct-v0.3
|
||||
source: https://huggingface.co/lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF
|
||||
- alias: nomic-embed-text:latest
|
||||
name: nomic-embed-text-v1.5
|
||||
name: text-embedding-nomic-embed-text-v1.5
|
||||
source: https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF
|
||||
|
||||
AI_REMOTE_ALIASES: >-
|
||||
{{ ['openai/gpt-4o-mini'] * ((API.openai.api_key | string | length > 0) | int)
|
||||
|
||||
@@ -38,6 +38,7 @@ design:
|
||||
AI_LOCAL_MODELS_CI:
|
||||
- alias: qwen2.5:0.5b
|
||||
name: qwen2.5-0.5b-instruct
|
||||
source: https://huggingface.co/lmstudio-community/Qwen2.5-0.5B-Instruct-GGUF
|
||||
|
||||
API:
|
||||
openai:
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
|
||||
- name: ⬇️ Pre-pull LM Studio models
|
||||
ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node
|
||||
container exec -i {{ LMSTUDIO_LOCAL_CID | quote }} lms get --yes --gguf {{ model.name | quote }}
|
||||
container exec -i {{ LMSTUDIO_LOCAL_CID | quote }} lms get --yes --gguf {{ model.source | quote }}
|
||||
register: lmstudio_pull_result
|
||||
loop: "{{ LMSTUDIO_PRELOAD_MODELS }}"
|
||||
loop_control:
|
||||
|
||||
Reference in new issue
Block a user