mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
fix(lmstudio): keep the daemon's auth key off the shared volume
Job 105845259891 (svc-ai-lmstudio#0@swarm, debian/btrfs) died in the model
pre-pull, two seconds in, not on a timeout:
Error: Failed to authenticate: Provided authPacket lms-cli is not allowed.
The daemon was up and rejecting deliberately. Its own log carries the server
half of the same line, through SystemExternalAPIProvider.
Reproduced locally in compose, without swarm and without NFS, which is what
pins the cause. Two daemons sharing one /root/.lmstudio:
daemon 1 writes .internal/lms-key-2 md5 c82691a9b6cb54c0eb233d2842f9f358
daemon 2 rewrites it md5 e407c5e17d7392b88080e8ed216f641d
lms against daemon 1 Provided authPacket lms-cli is not allowed
The key is the runtime half of the CLI's passkey and it lives in .internal,
not in credentials/, which is empty. /app/daemon-run.sh starts `lms server
start` in the entrypoint, so the CLI authenticates at container start and the
loser of the race is whichever daemon started first.
Swarm only supplied the second daemon: the role pins no replicas, so
compose_replicas defaults to one task per host, and /root/.lmstudio carries no
`nfs: false`, so the swarm rewrite puts all three on one NFS export. The
diagnostics show three task slots and five .ready-<timestamp> markers in one
directory.
.internal now has its own node-local volume. The models stay shared, which is
the point: the blobs are what the nodes may share, the daemon's identity is
not. Deobfuscating the daemon bundle confirms the split lines up with the
vendor's own layout: modelIndexCache, ggufMetadataCache and modelData resolve
inside .internal, defaultModelsFolder and hubModelsFolder resolve outside it,
and the folder watcher explicitly skips .internal when looking for models. A
missing index is a supported cold start: the accessor falls back to
emptyModelIndex with neverError, and startup re-runs the full directory walk.
`name:` is pinned because backup_volume.py emits `entry.get("name") or
semantic_name`, while the volume materialises as <stack>_<key>. Without the pin
the exclusion would name lmstudio_internal, the volume would be
lmstudio_lmstudio_internal as the run's artifact shows, and `backup: false`
would silently miss.
Verified against a real three-node swarm with NFS, the same shape that was red:
the pre-pull ran `ok: [mgr-01 -> wrk-02] => (item=qwen2.5:0.5b)`, the service
reports 3/3, `lms ls` inside a container lists the model with no auth error, and
the job ends 🏁 Job succeeded with no fatal and no failed host.
Shape follows web-app-matomo, whose `tmp` is an `nfs: false` child volume nested
in an NFS-backed parent with the same nocheck; 32 such nested pairs exist across
9 roles.
Residual, and it is not small: multi-replica operation stays unsafe for other
reasons this does not touch. extensions/plugins is rm -rf'd and re-extracted at
every daemon start, settings.json is written in place and resets itself to
defaults on a torn read, bin/lms is a 102 MB copy rewritten while peers execute
it, and model downloads land on their final shared path with no temp file. All
of that was equally true before this commit; a shared .internal merely added the
auth collision on top, and the index is opened noWatch so peers would have
overwritten each other's catalogue anyway. Making that safe means either one
replica or a fully node-local lmstudio_data with a per-node preload.
Second residual, affecting an earlier commit rather than this one: on a node
whose index does not know a model whose file is already in the shared models
folder, `lms get` sits at 0.00% until it is killed. That is the signature
df2fc5084a attributed to a wedged daemon and eef3c3be71 bounded with a dynamic
timeout. The seed imports the file and the pre-pull then asks for it again, so
the wedge may be this collision rather than the daemon. Measured once, on one
node; not yet a diagnosis.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
720a1a96ef
commit
c6e8990e25
1 file changed
+10
@@ -5,3 +5,13 @@ lmstudio_data:
|
||||
mounts:
|
||||
- service: lmstudio
|
||||
target: /root/.lmstudio
|
||||
|
||||
lmstudio_internal:
|
||||
type: volume
|
||||
name: lmstudio_internal
|
||||
# nocheck: nfs-false-data-loss Reason: regenerable per-daemon cache, rebuilt in full on first start against an empty volume; shared, a second daemon overwrites the auth key lms-key-2 and the first one's CLI is rejected
|
||||
nfs: false
|
||||
backup: false
|
||||
mounts:
|
||||
- service: lmstudio
|
||||
target: /root/.lmstudio/.internal
|
||||
Reference in new issue
Block a user