The role test asked for /v1/models on 127.0.0.1 from inside the container, the one address that still answered while the server was bound to loopback alone. It stayed green and the failure surfaced one role later as a 500 in litellm's warm-up.
Step 3 now resolves the container's first IPv4 through hostname -I and queries that address, which a loopback-only bind refuses. It reads the interfaces directly, so it needs neither the service name nor swarm's route back through the service VIP, and compose and swarm take the same path. When the models never appear, one more request tells an unreachable server apart from an unindexed store.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI job 105881436871 on 0b7a8a95d:
A container path declared in meta/volumes.yml is written out again.
roles/svc-ai-lmstudio/tasks/utils/preload/node.yml:7: /root/.lmstudio/models
(declared as 'lmstudio_models')
The disk guard typed the mount target instead of deriving it. That matters more
since the split: the role now carries two nested volumes, and which of them holds
models/ is decided solely by meta/volumes.yml. A second copy of the path drifts
the moment the mount moves, which is the drift 29015311b6 built this lint for.
LMSTUDIO_MODELS_TARGET resolves it through lookup('volume', …).target, next to
LMSTUDIO_SHARED_MODELS_PATH, which already reads .name from the same entry.
The test script held the same path a second time. The lint only reads yaml and
templates, so it would never have reported that copy, and it would have gone
stale just as quietly. It now receives the value through test.env.
Not caught locally, and the reason is worth recording: the preceding `make
quality` was red on three formalities of the new test assets. I fixed those,
re-ran only those three lints, and deployed. This finding sat a few lines below
them in the same output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
c6e8990e25 kept the daemon's auth key off the shared volume and made the red
job green, but it left multi-replica operation as broken as it found it: three
daemons still shared one /root/.lmstudio, where LM Studio rewrites
extensions/plugins with rm -rf at every start, writes settings.json in place and
resets it to defaults on a torn read, and re-copies a 102 MB bin/lms while peers
execute it.
Splitting the tree by access pattern fixes that. /root/.lmstudio is node-local,
because it is per-daemon state the daemon rewrites in place. models/ stays
shared, because a blob is written once and read forever.
That leaves one question, and it is temporal rather than mechanical: the daemon
discovers models by an unconditional scan at startup and by an inotify watcher,
and inotify never sees writes made by another NFS client. So a model seeded
after a daemon started is invisible to it, whichever tool wrote the file. The
previous design answered that with `container service update --force`, which
restarts the whole inference backend svc-ai-litellm points at, on every deploy.
The seed now runs before the stack does. 00_core.yml places the blob on the NFS
export ahead of the compose include that deploys the stack, so every daemon
finds a complete tree at its own startup scan and nothing has to be restarted.
Measured on a three-node swarm with NFS:
Fetch the models into the shared store 136.14s one 531 MB download
Pre-pull on mgr-01 0.69s
Pre-pull on wrk-01 0.50s
Pre-pull on wrk-02 0.49s
Half a second is not a download. All three daemons recognised the pre-placed
model, so the cluster now stores and transfers it once instead of three times.
The nested mount across two devices raises no EXDEV, and the per-node disk guard
correctly skipped itself four times because the pre-seed covers that check.
Vendor coupling is gone from the role. `repo` and `url` are declared fields
rather than pieces cut out of a HuggingFace URL with a regex; a model on any
host needs only those two. `lms get` still knows only HuggingFace, so `source`
stays for it, and a non-HF model simply leaves it unset: the blob is seeded and
the startup scan picks it up. The external check follows, comparing against the
HuggingFace API only for entries whose url actually points there.
The guard arithmetic was wrong and is now derived rather than guessed. Peak
usage is the missing bytes plus the largest single model, not twice the declared
total, so the read-back moved ahead of the check and subtracts what the store
already holds. Before, the requirement grew after every successful seed. Staging
moved beside the tree instead of inside it, so an aborted transfer cannot leave
orphans where the indexer scans.
A deployment test ships with it and states its findings rather than its exit
code: it fails on `authPacket` by naming the shared key, on a missing or
mismatching blob by naming path and digest, and on a model that is on disk but
absent from /v1/models by saying the startup scan did not happen.
Gone with the restart: import.sh and its container-cp suppression, seed.yml,
rescan.yml, and the disk-copy factor of three that only existed because the
import staged through the container.
Residual: compose is untested here. There the pre-seed is off, the node-local
guard is the one that runs, and `lms get` is the only fetch, so a non-HF model
is not loadable in that mode. Seeding node-local volumes is the follow-up that
closes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5