The readiness wait called a bare curl, which test_curl_via_filter rejects: a peer that accepts the connection and never answers keeps curl hanging, and the task's own retries never fire. The probe now renders through the curl filter with a 10 second cap per attempt. It passes retries=0, so the task's 60 x 5 s loop stays the only retry layer instead of nesting curl's --retry inside every attempt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The preload ran lms get as soon as the container existed. On a slow runner that reaches the daemon while its delayed init is still running, and the daemon then comes up without .internal/temp. Every later model load fails, because the chat template is written with mkdtemp into that missing folder: Failed to load model "qwen2.5-0.5b-instruct". Error: ENOENT: no such file or directory, mkdtemp '/root/.lmstudio/.internal/temp/lmstudio-chat-template-...'. litellm's warm-up then returned HTTP 400 in Moodle#2 of run 35617847242, while a local deploy on faster hardware stayed green.
Reproduced on fresh llmster containers capped at 0.3 CPU: an lms get issued during startup left .internal/temp absent; the same lms get issued after /lmstudio-greeting answered left it in place. The preload now waits for that endpoint, which only answers once lms server start ran, and daemon-run.sh starts the server only after the daemon's ready file exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI job 105881436871 on 0b7a8a95d:
A container path declared in meta/volumes.yml is written out again.
roles/svc-ai-lmstudio/tasks/utils/preload/node.yml:7: /root/.lmstudio/models
(declared as 'lmstudio_models')
The disk guard typed the mount target instead of deriving it. That matters more
since the split: the role now carries two nested volumes, and which of them holds
models/ is decided solely by meta/volumes.yml. A second copy of the path drifts
the moment the mount moves, which is the drift 29015311b6 built this lint for.
LMSTUDIO_MODELS_TARGET resolves it through lookup('volume', …).target, next to
LMSTUDIO_SHARED_MODELS_PATH, which already reads .name from the same entry.
The test script held the same path a second time. The lint only reads yaml and
templates, so it would never have reported that copy, and it would have gone
stale just as quietly. It now receives the value through test.env.
Not caught locally, and the reason is worth recording: the preceding `make
quality` was red on three formalities of the new test assets. I fixed those,
re-ran only those three lints, and deployed. This finding sat a few lines below
them in the same output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
c6e8990e25 kept the daemon's auth key off the shared volume and made the red
job green, but it left multi-replica operation as broken as it found it: three
daemons still shared one /root/.lmstudio, where LM Studio rewrites
extensions/plugins with rm -rf at every start, writes settings.json in place and
resets it to defaults on a torn read, and re-copies a 102 MB bin/lms while peers
execute it.
Splitting the tree by access pattern fixes that. /root/.lmstudio is node-local,
because it is per-daemon state the daemon rewrites in place. models/ stays
shared, because a blob is written once and read forever.
That leaves one question, and it is temporal rather than mechanical: the daemon
discovers models by an unconditional scan at startup and by an inotify watcher,
and inotify never sees writes made by another NFS client. So a model seeded
after a daemon started is invisible to it, whichever tool wrote the file. The
previous design answered that with `container service update --force`, which
restarts the whole inference backend svc-ai-litellm points at, on every deploy.
The seed now runs before the stack does. 00_core.yml places the blob on the NFS
export ahead of the compose include that deploys the stack, so every daemon
finds a complete tree at its own startup scan and nothing has to be restarted.
Measured on a three-node swarm with NFS:
Fetch the models into the shared store 136.14s one 531 MB download
Pre-pull on mgr-01 0.69s
Pre-pull on wrk-01 0.50s
Pre-pull on wrk-02 0.49s
Half a second is not a download. All three daemons recognised the pre-placed
model, so the cluster now stores and transfers it once instead of three times.
The nested mount across two devices raises no EXDEV, and the per-node disk guard
correctly skipped itself four times because the pre-seed covers that check.
Vendor coupling is gone from the role. `repo` and `url` are declared fields
rather than pieces cut out of a HuggingFace URL with a regex; a model on any
host needs only those two. `lms get` still knows only HuggingFace, so `source`
stays for it, and a non-HF model simply leaves it unset: the blob is seeded and
the startup scan picks it up. The external check follows, comparing against the
HuggingFace API only for entries whose url actually points there.
The guard arithmetic was wrong and is now derived rather than guessed. Peak
usage is the missing bytes plus the largest single model, not twice the declared
total, so the read-back moved ahead of the check and subtracts what the store
already holds. Before, the requirement grew after every successful seed. Staging
moved beside the tree instead of inside it, so an aborted transfer cannot leave
orphans where the indexer scans.
A deployment test ships with it and states its findings rather than its exit
code: it fails on `authPacket` by naming the shared key, on a missing or
mismatching blob by naming path and digest, and on a model that is on disk but
absent from /v1/models by saying the startup scan did not happen.
Gone with the restart: import.sh and its container-cp suppression, seed.yml,
rescan.yml, and the disk-copy factor of three that only existed because the
import staged through the container.
Residual: compose is untested here. There the pre-seed is off, the node-local
guard is the one that runs, and `lms get` is the only fetch, so a non-HF model
is not loadable in that mode. Seeding node-local volumes is the follow-up that
closes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The preload treats "the model is already there" as success in three places, and
all three were dead. They match on the literal `already downloaded`, which the
shipped CLI never emits: `strings` over /app/.bundle/lms in the deployed image
returns zero hits. What it prints is
You already have this model. You can load it with:
One of the three was doubly dead. The `until:` reads `stdout`, but the CLI
builds its logger as
const console = new console$1.Console({
stdout: process.stderr,
stderr: process.stderr,
so every level including info goes to stderr and that clause could never fire
whatever string it carried. The other two are fine on the stream: `failed_when`
concatenates stdout and stderr, and the reap path goes through
async_failures._said(), which joins msg, stdout and stderr.
Effect until now: an already-present model was tolerated only when `lms get`
happened to exit 0. On a non-zero exit the task failed with the model sitting
on disk, and none of the three guards could catch it.
This matters more from here on, not less. The role is moving towards seeding
the model before the daemons start, which makes "already present" the normal
case rather than the exception.
Measured, not inferred: both strings and the logger construction were read out
of image 48abf3b7e061, the one the failing CI job ran.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A wedged LM Studio daemon cost CI job 105558224377 three hours: three kills at
the global 60-minute `CONTAINER_EXEC_TIMEOUT`, for a 531 MB model that a healthy
node pulls in 27 seconds. Nothing in the path was sized against the thing being
fetched, and nothing checked whether it could land at all.
Size. Each model now declares `bytes`, taken from the value the deploy already
printed as "OK (531068224 bytes)", and the pre-pull derives its own timeout per
model: max(preload_floor_seconds, bytes / preload_min_bytes_per_second). The
floor of 600s protects a small model from a brief stall; the rate of 1 MB/s sits
twenty times below the 21.98 MB/s a healthy pull reached and seven thousand
times above the 142 B/s the wedged one managed, so it separates the two without
guessing. Evaluated against ansible-core's filters: 531 MB and 10 MB both land
on the floor, 4 GB gives 4000s.
Disk. The seed measures free space with `df -P -B1` before fetching and refuses
with a message naming both figures in GiB, the factor and the file to edit. The
factor is three because the node cache, the staging copy inside the container
and the models volume all exist at once during the import; that is measured, not
padding.
Fallback. A model may list `mirrors`, extra bases tried after the source. The
mirror task is a no-op once the source delivered, because get_url skips a dest
whose checksum already matches: measured at 145.24s for the first pass against
0.64s for the second. The list ships empty, because the obvious candidate is
not one: hf-mirror.com answers 206 but resolves to us.aws.cdn.hf.co, so it is a
front end for HuggingFace and adds no redundancy against a HuggingFace outage.
The mechanism is here; an independent copy is the operator's to name.
The external check grew with the declaration: `bytes` is compared against the
size the API publishes, and every declared mirror must still serve the pinned
file, or it is redundancy on paper only.
01_manager_ops.yml's rescue rationale claimed that restarting the container
clears a wedged transfer. Job 105558224377 shows the restart returning rc 0
without clearing it, so the claim now says what was measured instead.
Residual: the mirror path has never run in a deploy, since no mirror is
declared. Its loop expression is verified against ansible-core's subelements
over four shapes, including the zero-iteration case when nothing is mirrored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The seed path added in ee582762b0 fetched the model and never handed it over.
Two independent faults sat on top of each other; the second only became visible
once the first was gone.
First, the prompt. `lms import` keeps asking "Do you wish to continue? (Y/n)"
even with `--yes`, which suppresses only the warning about moving the file.
import.sh ran it through `container exec` without `-i`, so the question met a
closed stdin and the script died with 130, printing neither PRESENT:, ABSENT:
nor IMPORTED:. The sibling call in tasks/utils/preload.yml already passes `-i`.
Second, the move. `lms import` renames by default. `container cp` stages the
file in /tmp while MODELS_DIR lives on the mounted volume, so rename(2) crosses
a device boundary and cannot work:
EXDEV: cross-device link not permitted, rename '/tmp/...gguf'
-> '/root/.lmstudio/models/lmstudio-community/...'
Hence `--copy`, and hence the `rm` that clears the staged file copying leaves
behind. That path could therefore never have succeeded, which is why every CI
job since 2026-09-17 fetched 531068224 bytes, verified their sha256 and threw
them away.
Dropping `failed_when: false` is what makes the next failure visible: it
reported the abort as `ok` in 1.40s. The task name loses "best effort" for the
same reason.
Measured in a local compose deploy: IMPORTED:Qwen2.5-0.5B-Instruct-Q8_0.gguf,
then `lms get` returns rc 0 after 0.895s instead of a 60s network abort followed
by three 3600s timeout kills, both play recaps failed=0. The short-circuits still
hold: on the second pass the node cache check costs 0.64s instead of 145.24s and
import.sh prints PRESENT: before it copies anything.
Residual: rc 130 was measured on compose only. Some of the eight matrix rows
that pull svc-ai-lmstudio take the async swarm path, where the import is
unverified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The preload is the only artifact this repository pulls raw from the public
internet. Everything else, including the LM Studio image itself, arrives
through the mirror. `lms get` gets exactly one attempt under async, cannot
resume, and wedges its own daemon when a transfer dies, so one dropped socket
costs a whole job. XWiki#2 of run 35152776835 died 28 seconds into the
transfer at 3 MB/s and rising, with 3600 seconds of budget left.
The model is now fetched on the hosting node with get_url_retry, which gives
13 attempts and verifies a pinned sha256, and handed to the container with
`lms import`. Both steps are best effort: when the warm-up fails, `lms get`
runs exactly as before, so the change adds a faster path without removing the
old one. The pattern is web-app-mediawiki's extension install, down to the
`container cp` in the script.
Verified against the binary the role actually deploys, extracted from
lmstudio/llmster-preview:cpu and built 2025-08-28, not against upstream HEAD:
`lms import` takes `<file-path>`, `-y, --yes` and `--user-repo <user/repo>`;
the destination is join(modelsFolderPath, user, repo, basename(path)) under
join(lmstudioHome, "models"); the image ships no settings.json, so
downloadsFolder cannot move that target. The already-present branch of
`lms get` carries no process.exit, so it returns 0 and the task's existing
`until: rc == 0` accepts it.
The pin carries file and sha256 next to the source URL, and the digest is
quoted because an all-digit hash would otherwise parse as an integer and fail
the shape check for the wrong reason. tests/external guards the pin against
what HuggingFace publishes, comparing digests through the API without
transferring the file, and reports a network error as unverified rather than
failing a good pin on a firewalled runner.
Two limits worth stating. `lms get` still queries HuggingFace before it
reaches the download step, so this removes 531 MB of exposure rather than the
network dependency. And the tolerated substring 'already downloaded' in the
task's until, failed_when and reap list matches no output this version emits,
which says "You already have this model."; that branch is dead today and is
left untouched here because the return code carries the contract.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
playbook.yml:4 runs hosts: all under any_errors_fatal with no serial and no max_fail_percentage. When a task fails there, ansible/plugins/strategy/linear.py:329-333 marks every host that did not itself fail, while strategy/__init__.py:575-576 adds the failing host to _failed_hosts only when its next state is COMPLETE, which a rescue prevents. A rescued preload therefore leaves the stack host running alone through every later role while the workers and the nfs server are already marked failed.
The recovery is kept where it was measured: the stall came from a compose row, container exec -i lmstudio by name rather than a swarm task id. Swarm now re-raises the original failure as the first rescue task, the shape roles/web-app-keycloak/tasks/03_manager_ops/updates.yml:28-31 already uses in mirror image, and the nocheck marker moves with it because test_no_rescue_blocks.py:82 reads only the first rescue task's line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tests/lint/ansible/roles/test_async_reason_documented.py requires a one-line '# rationale: async; ...' directly above every async key and offers no nocheck escape, because fire-and-forget is a deployment-wide decision that has to hold in swarm and stay playwright compatible. A comment audit removed the line while moving the task into utils/preload.yml, and make test went red on it.
The restored line answers the question the lint asks, naming the reader rather than the reason for the async: the reap two tasks below reads every job back before the play continues, so no model is still downloading when playwright starts. The marker is not narration; the comment lint accepts rationale: by design.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tests/lint/ansible/structure/test_no_rescue_blocks.py forbids every rescue block in roles/ and tasks/ unless its first task carries a same-line '# nocheck: rescue-<reason>' comment, and make test runs it. The preload recovery added in df2fc5084a had none, so the gate failed with exactly one offender.
The block meets the rule the lint documents rather than only its regex: it performs real in-play recovery, since a timed-out transfer leaves the daemon reporting 0.00% for every further pull and only restarting the container that owns it clears that, which the workflow-level capture cannot do; and it ends by preloading again, so a failure that recovery does not fix still re-raises.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A preload that fails leaves the daemon unable to move bytes: measured in CI job 104622514450, attempt 1 reached 437.37 MB of 531.07 MB and died with 'Download failed: Timed-out', then all five retries sat at 0.00% for a full hour each until CONTAINER_EXEC_TIMEOUT killed the client. Six executions, 18796s, no progress.
Killing the client does not touch the daemon that owns the transfer, so a plain retry cannot help. The rescue restarts the container, which a local probe showed restores a working transfer, then re-resolves the CID because swarm may reschedule the restarted task under a fresh id, and preloads once more. Pull and reap moved into one included unit so the rescue also covers the async path, where the failure surfaces at the reap rather than at the pull. retries drops from 5 to 1, since retries N yields N+1 executions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34893750601 lost four jobs at svc-ai-litellm's warm request, every one of
them on the same upstream body:
No models loaded. Please load a model in the developer page or use the
`lms load` command.
The catalogue handed `lms get --yes --gguf` a bare name, which that command
treats as a hub search term. Its own help spells out the consequence: "If there
are multiple staff picks matching the search term, the first one will be used."
For `qwen2.5-0.5b-instruct` the first pick was Qwen2.5-Coder-0.5B-Instruct, so
the key LM Studio indexed never matched the `openai/{{ _model.name }}` the
gateway sends upstream, and just-in-time loading had nothing to load.
A `source:` now carries the full HuggingFace URL, the one spelling lms get
resolves deterministically; a bare owner/model pair addresses an LM Studio Hub
artifact instead and fails even for a model already on disk.
Measured against the shipped image rather than reasoned: the URL yields modelKey
qwen2.5-0.5b-instruct, `lms ps` is empty right before the request, and a plain
/v1/chat/completions for that name answers 200 and leaves the model resident. No
load step belongs here, because the server reports "Just-in-time model loading
active" and does it itself.
The embedding entry is renamed in the same breath. LM Studio indexes that model
as text-embedding-nomic-embed-text-v1.5, so the gateway was addressing a key
that never existed there either, and the first consumer to embed would have hit
the same 404.
The three fields stay separate and each has exactly one reader: Ollama pulls by
alias, LM Studio downloads by source, the gateway addresses name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-ai-lmstudio had no model-load step at all: tasks/00_core.yml set the
once-flag and the container network, nothing else. The gateway published
lmstudio/default anyway, so the route existed and nothing could answer it.
wp-compose-tor.log:48994 shows the route written into the config, :51770 shows
the warm step reaching qwen2.5:0.5b on Ollama instead, and the green ai-engine
spec ran against that. LM Studio was deployed, published and never prompted.
The pre-pull mirrors the one that works. tasks/01_manager_ops.yml resolves the
hosting node and container id, loops the declared models through
`lms get --yes --gguf`, overlaps them with async/poll and reaps before
playwright, exactly like svc-ai-ollama. That `lms` exists is measured, not
assumed: /app/daemon-run.sh:67 in the image starts ./.bundle/lms server start,
and `lms get --help` documents -y for scripting and --gguf for the format.
Synonymity needs both namespaces in one place. Ollama knows qwen2.5:0.5b, LM
Studio knows qwen2.5-0.5b-instruct, so AI_LOCAL_MODELS carries both per entry:
`alias` is what the gateway publishes and what Ollama pulls, `name` is what
`lms get` resolves. Both roles now reference that one list, and config.yaml.j2
publishes each backend under the alias, routing Ollama to ollama/<alias> and LM
Studio to openai/<name>. The lmstudio loop skips an alias Ollama already
publishes, so two backends never produce a duplicate model_name and Ollama
keeps precedence. lmstudio/default is gone; the model name is now the same
whichever backend answers it.
Ollama's preload_models changes shape from a list of strings to the {alias,
name} entries. Every consumer moves with it: the pull command, the template
loop, and both expressions in 16_ai.yml. An inventory that overrides
`applications.svc-ai-ollama.services.ollama.preload_models` with the old string
list will break at map(attribute='alias'); this is a deliberate shape change,
in the spirit of 7b15c61dfc.
The CI model list is not a property of an installation, so AI_LOCAL_MODELS_CI
lives in inventories/development/default.yml, whose own header scopes that file
to the cross-cutting environment knobs. test_variable_definitions only knew
group_vars, so it reported the constant as undefined; the shipped inventories
are a definition source too, because the deploy CLI bakes them into host_vars,
and the guard's own docstring already claimed "defined somewhere in the
repository".
A latent crash came out of this: `| list | first` raises on an empty sequence,
so a backend deployed without preload models blew up LITELLM_CHAT_MODEL. Ollama
with only embedding models could already hit it; the existing test dodged it by
rendering only _SERVED. `| default('')` now yields the empty string that
_SERVED already reports as false.
Tests: config.yaml.j2 is rendered rather than read, over ollama-only,
lmstudio-only, both and neither, asserting the published model_name set, that
both backends name the same alias, that a shared alias is published once and
routed to Ollama, and that the render is valid YAML. test_litellm_chat_model
gains the lmstudio branch, the no-preload case and the same-model assertion.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Consumers of the model plane speak one OpenAI-compatible endpoint and
carry one virtual key; which backend answers is the gateway's business.
svc-ai-litellm renders its model_list from the round's actual topology,
so an alias only exists when its backend role is in group_names, and it
emits an explicit empty list when no backend is present at all because
LiteLLM rejects a bare model_list: key.
Keys reach the gateway through the per-consumer credential rather than
being minted there: 01_keys.yml probes /key/info and only POSTs
/key/generate on 404, so reruns stay idempotent. The consumer list is
built in 00_core.yml, not inside the included file, because 01_keys.yml
delegates to the hosting node and delegate_to is templated before when.
The gateway declares ollama and lmstudio as consumer flags so the
network renderer attaches it to its own backends; without them it cannot
reach them.
svc-ai-lmstudio runs the headless llmster daemon behind expose only, so
it is reachable from the gateway but never bound on the host, matching
svc-ai-ollama.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
