fix(lmstudio): bound the model fetch by size, disk and a fallback source

A wedged LM Studio daemon cost CI job 105558224377 three hours: three kills at
the global 60-minute `CONTAINER_EXEC_TIMEOUT`, for a 531 MB model that a healthy
node pulls in 27 seconds. Nothing in the path was sized against the thing being
fetched, and nothing checked whether it could land at all.

Size. Each model now declares `bytes`, taken from the value the deploy already
printed as "OK (531068224 bytes)", and the pre-pull derives its own timeout per
model: max(preload_floor_seconds, bytes / preload_min_bytes_per_second). The
floor of 600s protects a small model from a brief stall; the rate of 1 MB/s sits
twenty times below the 21.98 MB/s a healthy pull reached and seven thousand
times above the 142 B/s the wedged one managed, so it separates the two without
guessing. Evaluated against ansible-core's filters: 531 MB and 10 MB both land
on the floor, 4 GB gives 4000s.

Disk. The seed measures free space with `df -P -B1` before fetching and refuses
with a message naming both figures in GiB, the factor and the file to edit. The
factor is three because the node cache, the staging copy inside the container
and the models volume all exist at once during the import; that is measured, not
padding.

Fallback. A model may list `mirrors`, extra bases tried after the source. The
mirror task is a no-op once the source delivered, because get_url skips a dest
whose checksum already matches: measured at 145.24s for the first pass against
0.64s for the second. The list ships empty, because the obvious candidate is
not one: hf-mirror.com answers 206 but resolves to us.aws.cdn.hf.co, so it is a
front end for HuggingFace and adds no redundancy against a HuggingFace outage.
The mechanism is here; an independent copy is the operator's to name.

The external check grew with the declaration: `bytes` is compared against the
size the API publishes, and every declared mirror must still serve the pinned
file, or it is redundancy on paper only.

01_manager_ops.yml's rescue rationale claimed that restarting the container
clears a wedged transfer. Job 105558224377 shows the restart returning rc 0
without clearing it, so the claim now says what was measured instead.

Residual: the mirror path has never run in a deploy, since no mirror is
declared. Its loop expression is verified against ansible-core's subelements
over four shapes, including the zero-iteration case when nothing is mirrored.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

group_vars/all/16_ai.yml

@@ -27,6 +27,7 @@ AI_LOCAL_MODELS:
     source: https://huggingface.co/lmstudio-community/Qwen2.5-0.5B-Instruct-GGUF
     file: Qwen2.5-0.5B-Instruct-Q8_0.gguf
     sha256: "1d96e30253a76970864323987e3e77eeaf170312952fd61d18d343c25cfd6e52"
+    bytes: 531068224
 
 AI_REMOTE_ALIASES: >-
   {{ ['openai/gpt-4o-mini'] * (lookup('api_enabled', 'openai') | int)

roles/svc-ai-lmstudio/meta/services.yml

@@ -25,6 +25,8 @@ lmstudio:
       http: 1234
   preload_models: "{{ AI_LOCAL_MODELS }}"
   preload_async_time: 3600
+  preload_floor_seconds: 600
+  preload_min_bytes_per_second: 1000000
   lifecycle: beta
 
 # nocheck: playwright-service-flag, dynamic-flag

roles/svc-ai-lmstudio/tasks/01_manager_ops.yml

@@ -17,7 +17,7 @@
     - name: "⬇️ Preload the LM Studio models (first pass)"
       ansible.builtin.include_tasks: utils/preload.yml
   rescue:
-    - name: "🐝 Re-raise the preload failure unchanged in swarm" # nocheck: rescue-recovery a timed-out transfer leaves the daemon reporting 0.00% for every further pull, and only restarting the container it owns clears that, which the generic capture cannot do; swarm re-raises here instead, because under any_errors_fatal a rescued host keeps running while every other host is already marked failed, and compose ends the block by preloading again so an unrecovered failure still re-raises
+    - name: "🐝 Re-raise the preload failure unchanged in swarm" # nocheck: rescue-recovery a timed-out transfer leaves the daemon reporting 0.00% for every further pull, and restarting the container it owns is the only lever the role has, though CI job 105558224377 shows a restart returning rc 0 without clearing it; the generic capture cannot even try, so swarm re-raises here instead, because under any_errors_fatal a rescued host keeps running while every other host is already marked failed, and compose ends the block by preloading again so an unrecovered failure still re-raises
       ansible.builtin.fail:
         msg: "{{ ansible_failed_result.msg | default('LM Studio preload failed') }}"
       when: DEPLOYMENT_MODE == 'swarm'

roles/svc-ai-lmstudio/tasks/utils/preload.yml

@@ -2,6 +2,11 @@
 - name: ⬇️ Pre-pull LM Studio models
   ansible.builtin.shell: > # nocheck: container-exec-resolver  runtime-resolved CID, delegated to the hosting node
     container exec -i {{ LMSTUDIO_LOCAL_CID | quote }} lms get --yes --gguf {{ model.source | quote }}
+  environment:
+    CONTAINER_EXEC_TIMEOUT: >-
+      {{ [ LMSTUDIO_PRELOAD_FLOOR_SECONDS | int,
+           ((model.bytes | int) / (LMSTUDIO_PRELOAD_MIN_BYTES_PER_SECOND | int))
+           | round(0, 'ceil') | int ] | max }}
   register: lmstudio_pull_result
   loop: "{{ LMSTUDIO_PRELOAD_MODELS }}"
   loop_control:

roles/svc-ai-lmstudio/tasks/utils/seed.yml

@@ -5,6 +5,27 @@
     state: directory
     mode: "0755"
 
+- name: "🗄️ Measure the free space the models have to fit into"
+  ansible.builtin.command: "df -P -B1 {{ LMSTUDIO_MODEL_CACHE_DIR | quote }}"
+  register: lmstudio_cache_space
+  changed_when: false
+
+- name: "🛑 Refuse to fetch models the hosting node cannot hold"
+  vars:
+    _free: "{{ lmstudio_cache_space.stdout_lines[1].split()[3] | int }}"
+    _required: "{{ LMSTUDIO_PRELOAD_REQUIRED_BYTES | int }}"
+  ansible.builtin.fail:
+    msg: >-
+      The declared LM Studio models need {{ (_required | int / 1024 / 1024 / 1024)
+      | round(1) }} GiB on {{ LMSTUDIO_HOST_NODE }} but only {{ (_free | int / 1024
+      / 1024 / 1024) | round(1) }} GiB is free. That is
+      {{ LMSTUDIO_PRELOAD_COPIES_ON_DISK }} times the
+      {{ (LMSTUDIO_PRELOAD_BYTES | int / 1024 / 1024 / 1024) | round(1) }} GiB the
+      models declare, because the node cache, the staging copy inside the container
+      and the models volume all exist at once during the import. Shrink
+      AI_LOCAL_MODELS in group_vars/all/16_ai.yml or give the node more disk.
+  when: _free | int < _required | int
+
 - name: "⬇️ Warm the model cache on the node, best effort"
   get_url_retry:
     url: "{{ model.source }}/resolve/main/{{ model.file }}"
@@ -18,6 +39,18 @@
   changed_when: false
   failed_when: false
 
+- name: "🪞 Warm the model cache from a mirror when the source stayed silent"
+  get_url_retry:
+    url: "{{ item.1 }}/resolve/main/{{ item.0.file }}"
+    dest: "{{ LMSTUDIO_MODEL_CACHE_DIR }}/{{ item.0.file }}"
+    checksum: "sha256:{{ item.0.sha256 }}"
+    mode: "0644"
+  loop: "{{ LMSTUDIO_PRELOAD_MODELS | subelements('mirrors', skip_missing=True) }}"
+  loop_control:
+    label: "{{ item.0.file }} from {{ item.1 }}"
+  changed_when: false
+  failed_when: false
+
 - name: "📋 Hand the warmed model to LM Studio"
   ansible.builtin.script: shell/import.sh
   environment:

roles/svc-ai-lmstudio/vars/main.yml

@@ -6,6 +6,13 @@ LMSTUDIO_CONTAINER: "{{ lookup('config', application_id, 'services.lmstudio.name
 LMSTUDIO_PORT: "{{ lookup('config', application_id, 'services.lmstudio.ports.internal.http') }}"
 LMSTUDIO_PRELOAD_MODELS: "{{ lookup('config', application_id, 'services.lmstudio.preload_models') }}"
 LMSTUDIO_PRELOAD_ASYNC_TIME: "{{ lookup('config', application_id, 'services.lmstudio.preload_async_time') }}"
+LMSTUDIO_PRELOAD_FLOOR_SECONDS: "{{ lookup('config', application_id, 'services.lmstudio.preload_floor_seconds') }}"
+LMSTUDIO_PRELOAD_MIN_BYTES_PER_SECOND: "{{ lookup('config', application_id, 'services.lmstudio.preload_min_bytes_per_second') }}"
+LMSTUDIO_PRELOAD_BYTES: >-
+  {{ LMSTUDIO_PRELOAD_MODELS | map(attribute='bytes') | map('int') | sum }}
+LMSTUDIO_PRELOAD_COPIES_ON_DISK: 3
+LMSTUDIO_PRELOAD_REQUIRED_BYTES: >-
+  {{ (LMSTUDIO_PRELOAD_BYTES | int) * (LMSTUDIO_PRELOAD_COPIES_ON_DISK | int) }}
 LMSTUDIO_CONFIG_DIR: "{{ lookup('container', application_id, 'directories.config') }}"
 LMSTUDIO_MODEL_CACHE_DIR: "{{ [ LMSTUDIO_CONFIG_DIR, 'models' ] | path_join }}"
 LMSTUDIO_HOME_DIR: "{{ lookup('volume', application_id, 'lmstudio_data').target }}"

tests/external/roles/test_ai_local_models_downloadable.py

@@ -52,11 +52,22 @@ def _download_url(entry: dict) -> str:
     return f"{entry['source'].rstrip('/')}/resolve/main/{entry['file']}"
 
 
+def _mirror_urls(entry: dict) -> list[str]:
+    """Every extra base a deploy falls back to, in declaration order."""
+    mirrors = entry.get("mirrors") or []
+    return [
+        f"{str(base).rstrip('/')}/resolve/main/{entry['file']}"
+        for base in mirrors
+        if isinstance(base, str)
+    ]
+
+
 def _upstream_digests(repo_path: str) -> tuple[str, object]:
     """Return (kind, payload) with kind one of ok, warn, unverified.
 
-    On ``ok`` the payload maps each published file name to its sha256, which is
-    what the pin is compared against without transferring the file itself.
+    On ``ok`` the payload maps each published file name to its sha256 and byte
+    count, which is what the pin is compared against without transferring the
+    file itself.
     """
     url = f"{_HUGGINGFACE_PREFIX}api/models/{repo_path}"
     try:
@@ -84,8 +95,9 @@ def _upstream_digests(repo_path: str) -> tuple[str, object]:
             continue
         name = sibling.get("rfilename")
         digest = (sibling.get("lfs") or {}).get("sha256")
+        size = (sibling.get("lfs") or {}).get("size") or sibling.get("size")
         if isinstance(name, str) and isinstance(digest, str):
-            digests[name] = digest
+            digests[name] = {"sha256": digest, "size": size}
     return "ok", digests
 
 
@@ -130,6 +142,9 @@ class TestAiLocalModelsDownloadable(unittest.TestCase):
                 unpinned.append(f"{alias}: 'file' is missing")
             if not isinstance(digest, str) or not _SHA256.match(digest):
                 unpinned.append(f"{alias}: 'sha256' is not 64 hex characters")
+            size = entry.get("bytes")
+            if not isinstance(size, int) or size <= 0:
+                unpinned.append(f"{alias}: 'bytes' is not a positive integer")
 
         self.assertFalse(
             unpinned,
@@ -183,11 +198,18 @@ class TestAiLocalModelsDownloadable(unittest.TestCase):
                 mismatched.append(
                     f"{alias}: {repo_path} no longer publishes {entry['file']}"
                 )
-            elif published != entry["sha256"]:
+                continue
+            if published["sha256"] != entry["sha256"]:
                 mismatched.append(
-                    f"{alias}: {entry['file']} is {published} upstream, "
+                    f"{alias}: {entry['file']} is {published['sha256']} upstream, "
                     f"pinned as {entry['sha256']}"
                 )
+            pinned_size = entry.get("bytes")
+            if published["size"] is not None and published["size"] != pinned_size:
+                mismatched.append(
+                    f"{alias}: {entry['file']} is {published['size']} bytes "
+                    f"upstream, pinned as {pinned_size}"
+                )
 
         self.assertFalse(
             mismatched,
@@ -197,6 +219,38 @@ class TestAiLocalModelsDownloadable(unittest.TestCase):
             + "\n".join(f"  {line}" for line in mismatched),
         )
 
+    def test_every_declared_mirror_still_serves_the_file(self) -> None:
+        targets = [
+            (str(entry.get("alias") or "<no alias>"), url)
+            for entry in _models()
+            for url in _mirror_urls(entry)
+        ]
+        if not targets:
+            self.skipTest(f"{_AI_VARS_FILE} declares no model mirror")
+
+        with concurrent.futures.ThreadPoolExecutor(max_workers=_MAX_WORKERS) as pool:
+            answers = list(pool.map(_reachable, [url for _, url in targets]))
+
+        broken = []
+        for (alias, url), (kind, detail) in zip(targets, answers, strict=True):
+            if kind == "ok":
+                continue
+            if kind == "unverified":
+                warning(
+                    f"{alias}: mirror {url} was not checked ({detail})",
+                    title="Model mirror unverified",
+                    file=_AI_VARS_FILE,
+                )
+                continue
+            broken.append(f"{alias}: mirror {url} answered {detail}")
+
+        self.assertFalse(
+            broken,
+            f"{len(broken)} declared mirror(s) no longer serve the pinned file, so "
+            "they add no redundancy. Drop the entry or point it at a copy that "
+            "answers:\n" + "\n".join(f"  {line}" for line in broken),
+        )
+
 
 if __name__ == "__main__":  # pragma: no cover
     unittest.main()