feat(lmstudio): fetch the model ourselves instead of trusting one client shot

The preload is the only artifact this repository pulls raw from the public
internet. Everything else, including the LM Studio image itself, arrives
through the mirror. `lms get` gets exactly one attempt under async, cannot
resume, and wedges its own daemon when a transfer dies, so one dropped socket
costs a whole job. XWiki#2 of run 35152776835 died 28 seconds into the
transfer at 3 MB/s and rising, with 3600 seconds of budget left.

The model is now fetched on the hosting node with get_url_retry, which gives
13 attempts and verifies a pinned sha256, and handed to the container with
`lms import`. Both steps are best effort: when the warm-up fails, `lms get`
runs exactly as before, so the change adds a faster path without removing the
old one. The pattern is web-app-mediawiki's extension install, down to the
`container cp` in the script.

Verified against the binary the role actually deploys, extracted from
lmstudio/llmster-preview:cpu and built 2025-08-28, not against upstream HEAD:
`lms import` takes `<file-path>`, `-y, --yes` and `--user-repo <user/repo>`;
the destination is join(modelsFolderPath, user, repo, basename(path)) under
join(lmstudioHome, "models"); the image ships no settings.json, so
downloadsFolder cannot move that target. The already-present branch of
`lms get` carries no process.exit, so it returns 0 and the task's existing
`until: rc == 0` accepts it.

The pin carries file and sha256 next to the source URL, and the digest is
quoted because an all-digit hash would otherwise parse as an integer and fail
the shape check for the wrong reason. tests/external guards the pin against
what HuggingFace publishes, comparing digests through the API without
transferring the file, and reports a network error as unverified rather than
failing a good pin on a firewalled runner.

Two limits worth stating. `lms get` still queries HuggingFace before it
reaches the download step, so this removes 531 MB of exposure rather than the
network dependency. And the tolerated substring 'already downloaded' in the
task's until, failed_when and reap list matches no output this version emits,
which says "You already have this model."; that branch is dead today and is
left untouched here because the return code carries the contract.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

group_vars/all/16_ai.yml

@@ -25,6 +25,8 @@ AI_LOCAL_MODELS:
   - alias: qwen2.5:0.5b
     name: qwen2.5-0.5b-instruct
     source: https://huggingface.co/lmstudio-community/Qwen2.5-0.5B-Instruct-GGUF
+    file: Qwen2.5-0.5B-Instruct-Q8_0.gguf
+    sha256: "1d96e30253a76970864323987e3e77eeaf170312952fd61d18d343c25cfd6e52"
 
 AI_REMOTE_ALIASES: >-
   {{ ['openai/gpt-4o-mini'] * (lookup('api_enabled', 'openai') | int)

roles/svc-ai-lmstudio/files/shell/import.sh

@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+#
+# Hand a pre-fetched GGUF to LM Studio, idempotent.
+# Runs ON THE HOSTING NODE (delegated); targets the local container via
+# `container exec` / `container cp`.
+#
+# Required env, supplied by the calling Ansible task:
+#   LMSTUDIO_LOCAL_CID  resolved container id, local to this node
+#   MODEL_FILE          node-side path to the fetched .gguf
+#   MODEL_USER_REPO     HuggingFace "user/repo" the file belongs to
+#   MODELS_DIR          models folder inside the container, under the
+#                       volume target declared in meta/volumes.yml
+set -euo pipefail
+
+ADDRESS="${LMSTUDIO_LOCAL_CID:?LMSTUDIO_LOCAL_CID env var (container id) required}"
+BASE="$(basename "${MODEL_FILE}")"
+TARGET="${MODELS_DIR}/${MODEL_USER_REPO}/${BASE}"
+
+if container exec "$ADDRESS" test -f "${TARGET}"; then
+	echo "PRESENT:${BASE}"
+	exit 0
+fi
+
+if [ ! -f "${MODEL_FILE}" ]; then
+	echo "ABSENT:${BASE}"
+	exit 0
+fi
+
+# nocheck: container-cp - MODEL_FILE is fetched on the node this script runs on
+container cp "${MODEL_FILE}" "${ADDRESS}:/tmp/${BASE}"
+
+container exec "$ADDRESS" lms import "/tmp/${BASE}" --yes --user-repo "${MODEL_USER_REPO}"
+echo "IMPORTED:${BASE}"

roles/svc-ai-lmstudio/tasks/01_manager_ops.yml

@@ -6,6 +6,12 @@
     resolve_output_prefix: LMSTUDIO
     resolve_compose_cid: "{{ LMSTUDIO_CONTAINER }}"
 
+- name: "🌱 Seed the models from the node cache before preloading"
+  ansible.builtin.include_tasks:
+    file: utils/seed.yml
+    apply:
+      delegate_to: "{{ LMSTUDIO_HOST_NODE }}"
+
 - name: "⬇️ Preload the LM Studio models"
   block:
     - name: "⬇️ Preload the LM Studio models (first pass)"

roles/svc-ai-lmstudio/tasks/utils/seed.yml

@@ -0,0 +1,33 @@
+---
+- name: "📁 Create the model cache directory on the hosting node"
+  ansible.builtin.file:
+    path: "{{ LMSTUDIO_MODEL_CACHE_DIR }}"
+    state: directory
+    mode: "0755"
+
+- name: "⬇️ Warm the model cache on the node, best effort"
+  get_url_retry:
+    url: "{{ model.source }}/resolve/main/{{ model.file }}"
+    dest: "{{ LMSTUDIO_MODEL_CACHE_DIR }}/{{ model.file }}"
+    checksum: "sha256:{{ model.sha256 }}"
+    mode: "0644"
+  loop: "{{ LMSTUDIO_PRELOAD_MODELS }}"
+  loop_control:
+    loop_var: model
+    label: "{{ model.file }}"
+  changed_when: false
+  failed_when: false
+
+- name: "📋 Hand a warmed model to LM Studio, best effort"
+  ansible.builtin.script: shell/import.sh
+  environment:
+    LMSTUDIO_LOCAL_CID: "{{ LMSTUDIO_LOCAL_CID }}"
+    MODEL_FILE: "{{ LMSTUDIO_MODEL_CACHE_DIR }}/{{ model.file }}"
+    MODEL_USER_REPO: "{{ model.source | regex_replace('^https://huggingface\\.co/', '') }}"
+    MODELS_DIR: "{{ LMSTUDIO_MODELS_DIR }}"
+  loop: "{{ LMSTUDIO_PRELOAD_MODELS }}"
+  loop_control:
+    loop_var: model
+    label: "{{ model.file }}"
+  changed_when: false
+  failed_when: false

roles/svc-ai-lmstudio/vars/main.yml

@@ -6,3 +6,7 @@ LMSTUDIO_CONTAINER: "{{ lookup('config', application_id, 'services.lmstudio.name
 LMSTUDIO_PORT: "{{ lookup('config', application_id, 'services.lmstudio.ports.internal.http') }}"
 LMSTUDIO_PRELOAD_MODELS: "{{ lookup('config', application_id, 'services.lmstudio.preload_models') }}"
 LMSTUDIO_PRELOAD_ASYNC_TIME: "{{ lookup('config', application_id, 'services.lmstudio.preload_async_time') }}"
+LMSTUDIO_CONFIG_DIR: "{{ lookup('container', application_id, 'directories.config') }}"
+LMSTUDIO_MODEL_CACHE_DIR: "{{ [ LMSTUDIO_CONFIG_DIR, 'models' ] | path_join }}"
+LMSTUDIO_HOME_DIR: "{{ lookup('volume', application_id, 'lmstudio_data').target }}"
+LMSTUDIO_MODELS_DIR: "{{ [ LMSTUDIO_HOME_DIR, 'models' ] | path_join }}"

tests/external/roles/test_ai_local_models_downloadable.py

@@ -0,0 +1,202 @@
+"""Every declared local AI model still resolves to a file upstream serves.
+
+``group_vars/all/16_ai.yml`` pins each model by repository, file name and
+sha256 so a deploy fetches it with a checksum instead of trusting whatever a
+search term resolves to. A pin that upstream no longer matches becomes a deploy
+that transfers half a gigabyte on every host and then fails on the digest, with
+nothing in the failure naming the declaration as the cause.
+
+The declaration's own shape is a repository fact and is asserted. Whether the
+file is still served is a network fact, so a timeout or a refused connection is
+reported as unverified rather than failing, the way the sibling external checks
+do: a firewalled runner must not turn a good pin red. An answer that contradicts
+the pin is a defect in the repository and does fail.
+"""
+
+from __future__ import annotations
+
+import concurrent.futures
+import re
+import unittest
+from urllib.parse import urlsplit
+
+import requests
+
+from utils.annotations.message import warning
+from utils.cache.yaml import load_yaml_any
+
+from . import PROJECT_ROOT
+
+_AI_VARS_FILE = "group_vars/all/16_ai.yml"
+_HUGGINGFACE_PREFIX = "https://huggingface.co/"
+_USER_AGENT = "infinito-nexus-model-pin-check"
+_TIMEOUT_SECONDS = 20
+_MAX_WORKERS = 4
+_SHA256 = re.compile(r"^[0-9a-f]{64}$")
+
+
+def _models() -> list[dict]:
+    data = load_yaml_any(str(PROJECT_ROOT / _AI_VARS_FILE)) or {}
+    entries = data.get("AI_LOCAL_MODELS") if isinstance(data, dict) else None
+    return [entry for entry in (entries or []) if isinstance(entry, dict)]
+
+
+def _repo_path(source: str) -> str:
+    """Return ``user/repo`` for a HuggingFace source, else the empty string."""
+    if not source.startswith(_HUGGINGFACE_PREFIX):
+        return ""
+    return source[len(_HUGGINGFACE_PREFIX) :].strip("/")
+
+
+def _download_url(entry: dict) -> str:
+    return f"{entry['source'].rstrip('/')}/resolve/main/{entry['file']}"
+
+
+def _upstream_digests(repo_path: str) -> tuple[str, object]:
+    """Return (kind, payload) with kind one of ok, warn, unverified.
+
+    On ``ok`` the payload maps each published file name to its sha256, which is
+    what the pin is compared against without transferring the file itself.
+    """
+    url = f"{_HUGGINGFACE_PREFIX}api/models/{repo_path}"
+    try:
+        response = requests.get(
+            url,
+            params={"blobs": "true"},
+            headers={"User-Agent": _USER_AGENT},
+            timeout=_TIMEOUT_SECONDS,
+        )
+    except requests.Timeout as exc:
+        return "unverified", f"Timeout: {exc}"
+    except requests.RequestException as exc:
+        return "unverified", f"{type(exc).__name__}: {exc}"
+
+    if response.status_code != 200:
+        return "warn", f"HTTP {response.status_code}"
+    try:
+        payload = response.json()
+    except ValueError as exc:
+        return "unverified", f"unparsable answer: {exc}"
+
+    digests = {}
+    for sibling in payload.get("siblings") or []:
+        if not isinstance(sibling, dict):
+            continue
+        name = sibling.get("rfilename")
+        digest = (sibling.get("lfs") or {}).get("sha256")
+        if isinstance(name, str) and isinstance(digest, str):
+            digests[name] = digest
+    return "ok", digests
+
+
+def _reachable(url: str) -> tuple[str, str]:
+    """Return (kind, detail); a redirect to the CDN counts as reachable."""
+    try:
+        response = requests.get(
+            url,
+            allow_redirects=True,
+            headers={"User-Agent": _USER_AGENT, "Range": "bytes=0-0"},
+            stream=True,
+            timeout=_TIMEOUT_SECONDS,
+        )
+        try:
+            status = response.status_code
+        finally:
+            response.close()
+    except requests.Timeout as exc:
+        return "unverified", f"Timeout: {exc}"
+    except requests.RequestException as exc:
+        return "unverified", f"{type(exc).__name__}: {exc}"
+
+    if status < 400:
+        return "ok", f"HTTP {status}"
+    return "warn", f"HTTP {status}"
+
+
+class TestAiLocalModelsDownloadable(unittest.TestCase):
+    def test_every_declared_model_is_pinned(self) -> None:
+        models = _models()
+        self.assertTrue(models, f"{_AI_VARS_FILE} declares no AI_LOCAL_MODELS")
+
+        unpinned = []
+        for entry in models:
+            alias = str(entry.get("alias") or "<no alias>")
+            source = entry.get("source")
+            file_name = entry.get("file")
+            digest = entry.get("sha256")
+            if not isinstance(source, str) or not urlsplit(source).scheme:
+                unpinned.append(f"{alias}: 'source' is not an absolute URL")
+            if not isinstance(file_name, str) or not file_name:
+                unpinned.append(f"{alias}: 'file' is missing")
+            if not isinstance(digest, str) or not _SHA256.match(digest):
+                unpinned.append(f"{alias}: 'sha256' is not 64 hex characters")
+
+        self.assertFalse(
+            unpinned,
+            f"{len(unpinned)} model declaration(s) in {_AI_VARS_FILE} cannot be "
+            f"fetched with a checksum. Each entry needs 'source', 'file' and "
+            f"'sha256':\n" + "\n".join(f"  {line}" for line in unpinned),
+        )
+
+    def test_every_pin_matches_what_upstream_serves(self) -> None:
+        models = [
+            entry
+            for entry in _models()
+            if isinstance(entry.get("source"), str)
+            and isinstance(entry.get("file"), str)
+            and isinstance(entry.get("sha256"), str)
+        ]
+        self.assertTrue(models, f"{_AI_VARS_FILE} declares no pinned model")
+
+        repo_paths = [_repo_path(entry["source"]) for entry in models]
+        with concurrent.futures.ThreadPoolExecutor(max_workers=_MAX_WORKERS) as pool:
+            api = list(pool.map(_upstream_digests, repo_paths))
+            reach = list(pool.map(_reachable, [_download_url(e) for e in models]))
+
+        mismatched = []
+        for entry, repo_path, (kind, payload), (rkind, rdetail) in zip(
+            models, repo_paths, api, reach, strict=True
+        ):
+            alias = entry["alias"]
+            url = _download_url(entry)
+            if rkind != "ok":
+                warning(
+                    f"{alias}: {url} answered {rdetail}"
+                    if rkind == "warn"
+                    else f"{alias}: {url} was not checked ({rdetail})",
+                    title="Model unreachable"
+                    if rkind == "warn"
+                    else "Model unverified",
+                    file=_AI_VARS_FILE,
+                )
+            if not repo_path:
+                continue
+            if kind != "ok":
+                warning(
+                    f"{alias}: {repo_path} was not checked ({payload})",
+                    title="Model pin unverified",
+                    file=_AI_VARS_FILE,
+                )
+                continue
+            published = payload.get(entry["file"])
+            if published is None:
+                mismatched.append(
+                    f"{alias}: {repo_path} no longer publishes {entry['file']}"
+                )
+            elif published != entry["sha256"]:
+                mismatched.append(
+                    f"{alias}: {entry['file']} is {published} upstream, "
+                    f"pinned as {entry['sha256']}"
+                )
+
+        self.assertFalse(
+            mismatched,
+            f"{len(mismatched)} model pin(s) in {_AI_VARS_FILE} no longer match "
+            f"what upstream publishes. Update 'file' and 'sha256' to the value "
+            f"the repository serves today:\n"
+            + "\n".join(f"  {line}" for line in mismatched),
+        )
+
+
+if __name__ == "__main__":  # pragma: no cover
+    unittest.main()