The readiness wait called a bare curl, which test_curl_via_filter rejects: a peer that accepts the connection and never answers keeps curl hanging, and the task's own retries never fire. The probe now renders through the curl filter with a 10 second cap per attempt. It passes retries=0, so the task's 60 x 5 s loop stays the only retry layer instead of nesting curl's --retry inside every attempt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The preload ran lms get as soon as the container existed. On a slow runner that reaches the daemon while its delayed init is still running, and the daemon then comes up without .internal/temp. Every later model load fails, because the chat template is written with mkdtemp into that missing folder: Failed to load model "qwen2.5-0.5b-instruct". Error: ENOENT: no such file or directory, mkdtemp '/root/.lmstudio/.internal/temp/lmstudio-chat-template-...'. litellm's warm-up then returned HTTP 400 in Moodle#2 of run 35617847242, while a local deploy on faster hardware stayed green.
Reproduced on fresh llmster containers capped at 0.3 CPU: an lms get issued during startup left .internal/temp absent; the same lms get issued after /lmstudio-greeting answered left it in place. The preload now waits for that endpoint, which only answers once lms server start ran, and daemon-run.sh starts the server only after the daemon's ready file exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI job 105881436871 on 0b7a8a95d:
A container path declared in meta/volumes.yml is written out again.
roles/svc-ai-lmstudio/tasks/utils/preload/node.yml:7: /root/.lmstudio/models
(declared as 'lmstudio_models')
The disk guard typed the mount target instead of deriving it. That matters more
since the split: the role now carries two nested volumes, and which of them holds
models/ is decided solely by meta/volumes.yml. A second copy of the path drifts
the moment the mount moves, which is the drift 29015311b6 built this lint for.
LMSTUDIO_MODELS_TARGET resolves it through lookup('volume', …).target, next to
LMSTUDIO_SHARED_MODELS_PATH, which already reads .name from the same entry.
The test script held the same path a second time. The lint only reads yaml and
templates, so it would never have reported that copy, and it would have gone
stale just as quietly. It now receives the value through test.env.
Not caught locally, and the reason is worth recording: the preceding `make
quality` was red on three formalities of the new test assets. I fixed those,
re-ran only those three lints, and deployed. This finding sat a few lines below
them in the same output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
c6e8990e25 kept the daemon's auth key off the shared volume and made the red
job green, but it left multi-replica operation as broken as it found it: three
daemons still shared one /root/.lmstudio, where LM Studio rewrites
extensions/plugins with rm -rf at every start, writes settings.json in place and
resets it to defaults on a torn read, and re-copies a 102 MB bin/lms while peers
execute it.
Splitting the tree by access pattern fixes that. /root/.lmstudio is node-local,
because it is per-daemon state the daemon rewrites in place. models/ stays
shared, because a blob is written once and read forever.
That leaves one question, and it is temporal rather than mechanical: the daemon
discovers models by an unconditional scan at startup and by an inotify watcher,
and inotify never sees writes made by another NFS client. So a model seeded
after a daemon started is invisible to it, whichever tool wrote the file. The
previous design answered that with `container service update --force`, which
restarts the whole inference backend svc-ai-litellm points at, on every deploy.
The seed now runs before the stack does. 00_core.yml places the blob on the NFS
export ahead of the compose include that deploys the stack, so every daemon
finds a complete tree at its own startup scan and nothing has to be restarted.
Measured on a three-node swarm with NFS:
Fetch the models into the shared store 136.14s one 531 MB download
Pre-pull on mgr-01 0.69s
Pre-pull on wrk-01 0.50s
Pre-pull on wrk-02 0.49s
Half a second is not a download. All three daemons recognised the pre-placed
model, so the cluster now stores and transfers it once instead of three times.
The nested mount across two devices raises no EXDEV, and the per-node disk guard
correctly skipped itself four times because the pre-seed covers that check.
Vendor coupling is gone from the role. `repo` and `url` are declared fields
rather than pieces cut out of a HuggingFace URL with a regex; a model on any
host needs only those two. `lms get` still knows only HuggingFace, so `source`
stays for it, and a non-HF model simply leaves it unset: the blob is seeded and
the startup scan picks it up. The external check follows, comparing against the
HuggingFace API only for entries whose url actually points there.
The guard arithmetic was wrong and is now derived rather than guessed. Peak
usage is the missing bytes plus the largest single model, not twice the declared
total, so the read-back moved ahead of the check and subtracts what the store
already holds. Before, the requirement grew after every successful seed. Staging
moved beside the tree instead of inside it, so an aborted transfer cannot leave
orphans where the indexer scans.
A deployment test ships with it and states its findings rather than its exit
code: it fails on `authPacket` by naming the shared key, on a missing or
mismatching blob by naming path and digest, and on a model that is on disk but
absent from /v1/models by saying the startup scan did not happen.
Gone with the restart: import.sh and its container-cp suppression, seed.yml,
rescan.yml, and the disk-copy factor of three that only existed because the
import staged through the container.
Residual: compose is untested here. There the pre-seed is off, the node-local
guard is the one that runs, and `lms get` is the only fetch, so a non-HF model
is not loadable in that mode. Seeding node-local volumes is the follow-up that
closes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5