The readiness wait called a bare curl, which test_curl_via_filter rejects: a peer that accepts the connection and never answers keeps curl hanging, and the task's own retries never fire. The probe now renders through the curl filter with a 10 second cap per attempt. It passes retries=0, so the task's 60 x 5 s loop stays the only retry layer instead of nesting curl's --retry inside every attempt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The preload ran lms get as soon as the container existed. On a slow runner that reaches the daemon while its delayed init is still running, and the daemon then comes up without .internal/temp. Every later model load fails, because the chat template is written with mkdtemp into that missing folder: Failed to load model "qwen2.5-0.5b-instruct". Error: ENOENT: no such file or directory, mkdtemp '/root/.lmstudio/.internal/temp/lmstudio-chat-template-...'. litellm's warm-up then returned HTTP 400 in Moodle#2 of run 35617847242, while a local deploy on faster hardware stayed green.
Reproduced on fresh llmster containers capped at 0.3 CPU: an lms get issued during startup left .internal/temp absent; the same lms get issued after /lmstudio-greeting answered left it in place. The preload now waits for that endpoint, which only answers once lms server start ran, and daemon-run.sh starts the server only after the daemon's ready file exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role test asked for /v1/models on 127.0.0.1 from inside the container, the one address that still answered while the server was bound to loopback alone. It stayed green and the failure surfaced one role later as a 500 in litellm's warm-up.
Step 3 now resolves the container's first IPv4 through hostname -I and queries that address, which a loopback-only bind refuses. It reads the interfaces directly, so it needs neither the service name nor swarm's route back through the service VIP, and compose and swarm take the same path. When the models never appear, one more request tells an unreachable server apart from an unindexed store.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Upstream re-pushed lmstudio/llmster-preview:cpu on 2026-09-18. The old build's daemon-run.sh wrote networkInterface 0.0.0.0 into .internal/http-server-config.json before starting the server. The new one drops that block, and its lms resolves the address as --bind, then LMS_SERVER_HOST, then 127.0.0.1, reading only the port from that file.
The server therefore listened on loopback alone. The image healthcheck curls localhost and every role task drives lms through container exec, so svc-ai-lmstudio stayed green, while litellm's warm-up got Connect call failed on lmstudio:1234 and failed Mattermost#2, Matrix#2, XWiki#2, Moodle#2 and Zammad#2 in fork run 35545771696.
LMS_SERVER_HOST is the documented knob and restores exactly the old bind. The port stays expose-only on the container network, so nothing new is published.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI job 105881436871 on 0b7a8a95d:
A container path declared in meta/volumes.yml is written out again.
roles/svc-ai-lmstudio/tasks/utils/preload/node.yml:7: /root/.lmstudio/models
(declared as 'lmstudio_models')
The disk guard typed the mount target instead of deriving it. That matters more
since the split: the role now carries two nested volumes, and which of them holds
models/ is decided solely by meta/volumes.yml. A second copy of the path drifts
the moment the mount moves, which is the drift 29015311b6 built this lint for.
LMSTUDIO_MODELS_TARGET resolves it through lookup('volume', …).target, next to
LMSTUDIO_SHARED_MODELS_PATH, which already reads .name from the same entry.
The test script held the same path a second time. The lint only reads yaml and
templates, so it would never have reported that copy, and it would have gone
stale just as quietly. It now receives the value through test.env.
Not caught locally, and the reason is worth recording: the preceding `make
quality` was red on three formalities of the new test assets. I fixed those,
re-ran only those three lints, and deployed. This finding sat a few lines below
them in the same output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
c6e8990e25 kept the daemon's auth key off the shared volume and made the red
job green, but it left multi-replica operation as broken as it found it: three
daemons still shared one /root/.lmstudio, where LM Studio rewrites
extensions/plugins with rm -rf at every start, writes settings.json in place and
resets it to defaults on a torn read, and re-copies a 102 MB bin/lms while peers
execute it.
Splitting the tree by access pattern fixes that. /root/.lmstudio is node-local,
because it is per-daemon state the daemon rewrites in place. models/ stays
shared, because a blob is written once and read forever.
That leaves one question, and it is temporal rather than mechanical: the daemon
discovers models by an unconditional scan at startup and by an inotify watcher,
and inotify never sees writes made by another NFS client. So a model seeded
after a daemon started is invisible to it, whichever tool wrote the file. The
previous design answered that with `container service update --force`, which
restarts the whole inference backend svc-ai-litellm points at, on every deploy.
The seed now runs before the stack does. 00_core.yml places the blob on the NFS
export ahead of the compose include that deploys the stack, so every daemon
finds a complete tree at its own startup scan and nothing has to be restarted.
Measured on a three-node swarm with NFS:
Fetch the models into the shared store 136.14s one 531 MB download
Pre-pull on mgr-01 0.69s
Pre-pull on wrk-01 0.50s
Pre-pull on wrk-02 0.49s
Half a second is not a download. All three daemons recognised the pre-placed
model, so the cluster now stores and transfers it once instead of three times.
The nested mount across two devices raises no EXDEV, and the per-node disk guard
correctly skipped itself four times because the pre-seed covers that check.
Vendor coupling is gone from the role. `repo` and `url` are declared fields
rather than pieces cut out of a HuggingFace URL with a regex; a model on any
host needs only those two. `lms get` still knows only HuggingFace, so `source`
stays for it, and a non-HF model simply leaves it unset: the blob is seeded and
the startup scan picks it up. The external check follows, comparing against the
HuggingFace API only for entries whose url actually points there.
The guard arithmetic was wrong and is now derived rather than guessed. Peak
usage is the missing bytes plus the largest single model, not twice the declared
total, so the read-back moved ahead of the check and subtracts what the store
already holds. Before, the requirement grew after every successful seed. Staging
moved beside the tree instead of inside it, so an aborted transfer cannot leave
orphans where the indexer scans.
A deployment test ships with it and states its findings rather than its exit
code: it fails on `authPacket` by naming the shared key, on a missing or
mismatching blob by naming path and digest, and on a model that is on disk but
absent from /v1/models by saying the startup scan did not happen.
Gone with the restart: import.sh and its container-cp suppression, seed.yml,
rescan.yml, and the disk-copy factor of three that only existed because the
import staged through the container.
Residual: compose is untested here. There the pre-seed is off, the node-local
guard is the one that runs, and `lms get` is the only fetch, so a non-HF model
is not loadable in that mode. Seeding node-local volumes is the follow-up that
closes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The preload treats "the model is already there" as success in three places, and
all three were dead. They match on the literal `already downloaded`, which the
shipped CLI never emits: `strings` over /app/.bundle/lms in the deployed image
returns zero hits. What it prints is
You already have this model. You can load it with:
One of the three was doubly dead. The `until:` reads `stdout`, but the CLI
builds its logger as
const console = new console$1.Console({
stdout: process.stderr,
stderr: process.stderr,
so every level including info goes to stderr and that clause could never fire
whatever string it carried. The other two are fine on the stream: `failed_when`
concatenates stdout and stderr, and the reap path goes through
async_failures._said(), which joins msg, stdout and stderr.
Effect until now: an already-present model was tolerated only when `lms get`
happened to exit 0. On a non-zero exit the task failed with the model sitting
on disk, and none of the three guards could catch it.
This matters more from here on, not less. The role is moving towards seeding
the model before the daemons start, which makes "already present" the normal
case rather than the exception.
Measured, not inferred: both strings and the logger construction were read out
of image 48abf3b7e061, the one the failing CI job ran.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Job 105845259891 (svc-ai-lmstudio#0@swarm, debian/btrfs) died in the model
pre-pull, two seconds in, not on a timeout:
Error: Failed to authenticate: Provided authPacket lms-cli is not allowed.
The daemon was up and rejecting deliberately. Its own log carries the server
half of the same line, through SystemExternalAPIProvider.
Reproduced locally in compose, without swarm and without NFS, which is what
pins the cause. Two daemons sharing one /root/.lmstudio:
daemon 1 writes .internal/lms-key-2 md5 c82691a9b6cb54c0eb233d2842f9f358
daemon 2 rewrites it md5 e407c5e17d7392b88080e8ed216f641d
lms against daemon 1 Provided authPacket lms-cli is not allowed
The key is the runtime half of the CLI's passkey and it lives in .internal,
not in credentials/, which is empty. /app/daemon-run.sh starts `lms server
start` in the entrypoint, so the CLI authenticates at container start and the
loser of the race is whichever daemon started first.
Swarm only supplied the second daemon: the role pins no replicas, so
compose_replicas defaults to one task per host, and /root/.lmstudio carries no
`nfs: false`, so the swarm rewrite puts all three on one NFS export. The
diagnostics show three task slots and five .ready-<timestamp> markers in one
directory.
.internal now has its own node-local volume. The models stay shared, which is
the point: the blobs are what the nodes may share, the daemon's identity is
not. Deobfuscating the daemon bundle confirms the split lines up with the
vendor's own layout: modelIndexCache, ggufMetadataCache and modelData resolve
inside .internal, defaultModelsFolder and hubModelsFolder resolve outside it,
and the folder watcher explicitly skips .internal when looking for models. A
missing index is a supported cold start: the accessor falls back to
emptyModelIndex with neverError, and startup re-runs the full directory walk.
`name:` is pinned because backup_volume.py emits `entry.get("name") or
semantic_name`, while the volume materialises as <stack>_<key>. Without the pin
the exclusion would name lmstudio_internal, the volume would be
lmstudio_lmstudio_internal as the run's artifact shows, and `backup: false`
would silently miss.
Verified against a real three-node swarm with NFS, the same shape that was red:
the pre-pull ran `ok: [mgr-01 -> wrk-02] => (item=qwen2.5:0.5b)`, the service
reports 3/3, `lms ls` inside a container lists the model with no auth error, and
the job ends 🏁 Job succeeded with no fatal and no failed host.
Shape follows web-app-matomo, whose `tmp` is an `nfs: false` child volume nested
in an NFS-backed parent with the same nocheck; 32 such nested pairs exist across
9 roles.
Residual, and it is not small: multi-replica operation stays unsafe for other
reasons this does not touch. extensions/plugins is rm -rf'd and re-extracted at
every daemon start, settings.json is written in place and resets itself to
defaults on a torn read, bin/lms is a 102 MB copy rewritten while peers execute
it, and model downloads land on their final shared path with no temp file. All
of that was equally true before this commit; a shared .internal merely added the
auth collision on top, and the index is opened noWatch so peers would have
overwritten each other's catalogue anyway. Making that safe means either one
replica or a fully node-local lmstudio_data with a per-node preload.
Second residual, affecting an earlier commit rather than this one: on a node
whose index does not know a model whose file is already in the shared models
folder, `lms get` sits at 0.00% until it is killed. That is the signature
df2fc5084a attributed to a wedged daemon and eef3c3be71 bounded with a dynamic
timeout. The seed imports the file and the pre-pull then asks for it again, so
the wedge may be this collision rather than the daemon. Measured once, on one
node; not yet a diagnosis.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A wedged LM Studio daemon cost CI job 105558224377 three hours: three kills at
the global 60-minute `CONTAINER_EXEC_TIMEOUT`, for a 531 MB model that a healthy
node pulls in 27 seconds. Nothing in the path was sized against the thing being
fetched, and nothing checked whether it could land at all.
Size. Each model now declares `bytes`, taken from the value the deploy already
printed as "OK (531068224 bytes)", and the pre-pull derives its own timeout per
model: max(preload_floor_seconds, bytes / preload_min_bytes_per_second). The
floor of 600s protects a small model from a brief stall; the rate of 1 MB/s sits
twenty times below the 21.98 MB/s a healthy pull reached and seven thousand
times above the 142 B/s the wedged one managed, so it separates the two without
guessing. Evaluated against ansible-core's filters: 531 MB and 10 MB both land
on the floor, 4 GB gives 4000s.
Disk. The seed measures free space with `df -P -B1` before fetching and refuses
with a message naming both figures in GiB, the factor and the file to edit. The
factor is three because the node cache, the staging copy inside the container
and the models volume all exist at once during the import; that is measured, not
padding.
Fallback. A model may list `mirrors`, extra bases tried after the source. The
mirror task is a no-op once the source delivered, because get_url skips a dest
whose checksum already matches: measured at 145.24s for the first pass against
0.64s for the second. The list ships empty, because the obvious candidate is
not one: hf-mirror.com answers 206 but resolves to us.aws.cdn.hf.co, so it is a
front end for HuggingFace and adds no redundancy against a HuggingFace outage.
The mechanism is here; an independent copy is the operator's to name.
The external check grew with the declaration: `bytes` is compared against the
size the API publishes, and every declared mirror must still serve the pinned
file, or it is redundancy on paper only.
01_manager_ops.yml's rescue rationale claimed that restarting the container
clears a wedged transfer. Job 105558224377 shows the restart returning rc 0
without clearing it, so the claim now says what was measured instead.
Residual: the mirror path has never run in a deploy, since no mirror is
declared. Its loop expression is verified against ansible-core's subelements
over four shapes, including the zero-iteration case when nothing is mirrored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The seed path added in ee582762b0 fetched the model and never handed it over.
Two independent faults sat on top of each other; the second only became visible
once the first was gone.
First, the prompt. `lms import` keeps asking "Do you wish to continue? (Y/n)"
even with `--yes`, which suppresses only the warning about moving the file.
import.sh ran it through `container exec` without `-i`, so the question met a
closed stdin and the script died with 130, printing neither PRESENT:, ABSENT:
nor IMPORTED:. The sibling call in tasks/utils/preload.yml already passes `-i`.
Second, the move. `lms import` renames by default. `container cp` stages the
file in /tmp while MODELS_DIR lives on the mounted volume, so rename(2) crosses
a device boundary and cannot work:
EXDEV: cross-device link not permitted, rename '/tmp/...gguf'
-> '/root/.lmstudio/models/lmstudio-community/...'
Hence `--copy`, and hence the `rm` that clears the staged file copying leaves
behind. That path could therefore never have succeeded, which is why every CI
job since 2026-09-17 fetched 531068224 bytes, verified their sha256 and threw
them away.
Dropping `failed_when: false` is what makes the next failure visible: it
reported the abort as `ok` in 1.40s. The task name loses "best effort" for the
same reason.
Measured in a local compose deploy: IMPORTED:Qwen2.5-0.5B-Instruct-Q8_0.gguf,
then `lms get` returns rc 0 after 0.895s instead of a 60s network abort followed
by three 3600s timeout kills, both play recaps failed=0. The short-circuits still
hold: on the second pass the node cache check costs 0.64s instead of 145.24s and
import.sh prints PRESENT: before it copies anything.
Residual: rc 130 was measured on compose only. Some of the eight matrix rows
that pull svc-ai-lmstudio take the async swarm path, where the import is
unverified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The preload is the only artifact this repository pulls raw from the public
internet. Everything else, including the LM Studio image itself, arrives
through the mirror. `lms get` gets exactly one attempt under async, cannot
resume, and wedges its own daemon when a transfer dies, so one dropped socket
costs a whole job. XWiki#2 of run 35152776835 died 28 seconds into the
transfer at 3 MB/s and rising, with 3600 seconds of budget left.
The model is now fetched on the hosting node with get_url_retry, which gives
13 attempts and verifies a pinned sha256, and handed to the container with
`lms import`. Both steps are best effort: when the warm-up fails, `lms get`
runs exactly as before, so the change adds a faster path without removing the
old one. The pattern is web-app-mediawiki's extension install, down to the
`container cp` in the script.
Verified against the binary the role actually deploys, extracted from
lmstudio/llmster-preview:cpu and built 2025-08-28, not against upstream HEAD:
`lms import` takes `<file-path>`, `-y, --yes` and `--user-repo <user/repo>`;
the destination is join(modelsFolderPath, user, repo, basename(path)) under
join(lmstudioHome, "models"); the image ships no settings.json, so
downloadsFolder cannot move that target. The already-present branch of
`lms get` carries no process.exit, so it returns 0 and the task's existing
`until: rc == 0` accepts it.
The pin carries file and sha256 next to the source URL, and the digest is
quoted because an all-digit hash would otherwise parse as an integer and fail
the shape check for the wrong reason. tests/external guards the pin against
what HuggingFace publishes, comparing digests through the API without
transferring the file, and reports a network error as unverified rather than
failing a good pin on a firewalled runner.
Two limits worth stating. `lms get` still queries HuggingFace before it
reaches the download step, so this removes 531 MB of exposure rather than the
network dependency. And the tolerated substring 'already downloaded' in the
task's until, failed_when and reap list matches no output this version emits,
which says "You already have this model."; that branch is dead today and is
left untouched here because the return code carries the contract.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ranking the last commits built toward is now enforced: a semver tag,
else a digest, and nothing else without a written reason. Two states fail.
A pin that is neither names something the registry can move underneath us,
and nothing here can tell whether it still points where it did when it was
written. A digest on an upstream that does publish semver takes the
fallback for no gain and hides the entry from the update job.
Thirteen entries stood in the way when the check was first run. Four were
resolved by the channel grammar, which made the Jitsi components orderable
without touching their pins. Three needed nothing once a moving tag could
declare itself with 'docker-version'. Two are fixed here and four are
declared.
LiteLLM leaves 'main-v1.77.3.dynamic_rates'. That tag exists exactly once
among 2609, so it is a dead end no updater can follow, and it misreports
itself: the image inside answers 1.77.2. The pin moves to
'main-v1.77.7-stable', five patches along the same minor and onto a channel
with 134 builds. Started against this repository's config shape, 1.77.7
registers both model entry forms and accepts drop_params and the
os.environ master key.
Pixelfed leaves 'latest' for '20260917'. The upstream stamps its dates as
one eight-digit number, which is depth 1 and orderable with no tooling
change at all, and the two tags resolve to the same digest, so nothing
about the deployed image moves.
The four declarations carry the count that justifies them rather than a
claim: LM Studio publishes two tags and neither is a version, the MIG
registry carries one, and the two Nextcloud Talk images publish 174 and 104
AIO build stamps between them.
Both branches of the gate were negative-controlled. The digest branch
skips in this repository because no digest pin exists at all, so it was
exercised by introducing one and watching it fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The catalogue held three models and the development inventory carried a second, CI-only list that both AI roles pulled in through their variant 0. That made the model a variant concern and split one fact across a group_var, an inventory file and two meta files.
AI_LOCAL_MODELS now holds the single entry the CI list held, qwen2.5 0.5b instruct at 531 MB in Q8_0, which lms get is already proven to fetch in this stack. AI_LOCAL_MODELS_CI and both variant overrides are gone, so variant 0 states only container_backup and the memory limits.
The dropped entries have no consumer: llama3, mistral and nomic appear solely in test fixtures that stub the lookups, and nothing requests an embedding model, the only embed matches being the filters that exclude one when picking the chat model. LITELLM_CHAT_MODEL takes the first non-embed alias, so it resolves and LITELLM_USABLE stays true.
Note this changes production, not only CI: a deployment no longer pulls a 7B model or an embedding model. It also shrinks the pre-pull that burned 18796s in job 104622514450 from gigabytes to 531 MB.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
playbook.yml:4 runs hosts: all under any_errors_fatal with no serial and no max_fail_percentage. When a task fails there, ansible/plugins/strategy/linear.py:329-333 marks every host that did not itself fail, while strategy/__init__.py:575-576 adds the failing host to _failed_hosts only when its next state is COMPLETE, which a rescue prevents. A rescued preload therefore leaves the stack host running alone through every later role while the workers and the nfs server are already marked failed.
The recovery is kept where it was measured: the stall came from a compose row, container exec -i lmstudio by name rather than a swarm task id. Swarm now re-raises the original failure as the first rescue task, the shape roles/web-app-keycloak/tasks/03_manager_ops/updates.yml:28-31 already uses in mirror image, and the nocheck marker moves with it because test_no_rescue_blocks.py:82 reads only the first rescue task's line.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tests/lint/ansible/roles/test_async_reason_documented.py requires a one-line '# rationale: async; ...' directly above every async key and offers no nocheck escape, because fire-and-forget is a deployment-wide decision that has to hold in swarm and stay playwright compatible. A comment audit removed the line while moving the task into utils/preload.yml, and make test went red on it.
The restored line answers the question the lint asks, naming the reader rather than the reason for the async: the reap two tasks below reads every job back before the play continues, so no model is still downloading when playwright starts. The marker is not narration; the comment lint accepts rationale: by design.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tests/lint/ansible/structure/test_no_rescue_blocks.py forbids every rescue block in roles/ and tasks/ unless its first task carries a same-line '# nocheck: rescue-<reason>' comment, and make test runs it. The preload recovery added in df2fc5084a had none, so the gate failed with exactly one offender.
The block meets the rule the lint documents rather than only its regex: it performs real in-play recovery, since a timed-out transfer leaves the daemon reporting 0.00% for every further pull and only restarting the container that owns it clears that, which the workflow-level capture cannot do; and it ends by preloading again, so a failure that recovery does not fix still re-raises.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A preload that fails leaves the daemon unable to move bytes: measured in CI job 104622514450, attempt 1 reached 437.37 MB of 531.07 MB and died with 'Download failed: Timed-out', then all five retries sat at 0.00% for a full hour each until CONTAINER_EXEC_TIMEOUT killed the client. Six executions, 18796s, no progress.
Killing the client does not touch the daemon that owns the transfer, so a plain retry cannot help. The rescue restarts the container, which a local probe showed restores a working transfer, then re-resolves the CID because swarm may reschedule the restarted task under a fresh id, and preloads once more. Pull and reap moved into one included unit so the rescue also covers the async path, where the failure surfaces at the reap rather than at the pull. retries drops from 5 to 1, since retries N yields N+1 executions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f8da0fd122 made the pre-pull download model.source instead of the bare model.name, but the Features list still said an entry 'carries the LM Studio name to download'. An author following that sentence writes the search term that caused the failure: lms get answers 'Multiple models found. Automatically selecting the first one due to --yes' and fetches a different model, after which the gateway addresses a name nothing serves.
Name all three fields with their roles. source is the repository that gets downloaded, name is the key LM Studio indexes it under and templates/config.yaml.j2 addresses as openai/<name>, alias is what the gateway publishes.
A second entry records that the pre-pull does not load. That is what makes the upstream 'No models loaded' body readable, and it is why svc-ai-litellm warms the gateway at deploy time rather than leaving the cold load to the first consumer.
Measured on the image CI runs (sha256:48abf3b7e061): the URL form resolves the pinned repository and indexes it as qwen2.5-0.5b-instruct, the search-term form resolves to the Coder variant and indexes it as qwen2.5-coder-0.5b-instruct, and a chat completion against an empty lms ps loads the model just in time and answers 200.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34893750601 lost four jobs at svc-ai-litellm's warm request, every one of
them on the same upstream body:
No models loaded. Please load a model in the developer page or use the
`lms load` command.
The catalogue handed `lms get --yes --gguf` a bare name, which that command
treats as a hub search term. Its own help spells out the consequence: "If there
are multiple staff picks matching the search term, the first one will be used."
For `qwen2.5-0.5b-instruct` the first pick was Qwen2.5-Coder-0.5B-Instruct, so
the key LM Studio indexed never matched the `openai/{{ _model.name }}` the
gateway sends upstream, and just-in-time loading had nothing to load.
A `source:` now carries the full HuggingFace URL, the one spelling lms get
resolves deterministically; a bare owner/model pair addresses an LM Studio Hub
artifact instead and fails even for a model already on disk.
Measured against the shipped image rather than reasoned: the URL yields modelKey
qwen2.5-0.5b-instruct, `lms ps` is empty right before the request, and a plain
/v1/chat/completions for that name answers 200 and leaves the model resident. No
load step belongs here, because the server reports "Just-in-time model loading
active" and does it itself.
The embedding entry is renamed in the same breath. LM Studio indexes that model
as text-embedding-nomic-embed-text-v1.5, so the gateway was addressing a key
that never existed there either, and the first consumer to embed would have hit
the same 404.
The three fields stay separate and each has exactly one reader: Ollama pulls by
alias, LM Studio downloads by source, the gateway addresses name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-ai-lmstudio had no model-load step at all: tasks/00_core.yml set the
once-flag and the container network, nothing else. The gateway published
lmstudio/default anyway, so the route existed and nothing could answer it.
wp-compose-tor.log:48994 shows the route written into the config, :51770 shows
the warm step reaching qwen2.5:0.5b on Ollama instead, and the green ai-engine
spec ran against that. LM Studio was deployed, published and never prompted.
The pre-pull mirrors the one that works. tasks/01_manager_ops.yml resolves the
hosting node and container id, loops the declared models through
`lms get --yes --gguf`, overlaps them with async/poll and reaps before
playwright, exactly like svc-ai-ollama. That `lms` exists is measured, not
assumed: /app/daemon-run.sh:67 in the image starts ./.bundle/lms server start,
and `lms get --help` documents -y for scripting and --gguf for the format.
Synonymity needs both namespaces in one place. Ollama knows qwen2.5:0.5b, LM
Studio knows qwen2.5-0.5b-instruct, so AI_LOCAL_MODELS carries both per entry:
`alias` is what the gateway publishes and what Ollama pulls, `name` is what
`lms get` resolves. Both roles now reference that one list, and config.yaml.j2
publishes each backend under the alias, routing Ollama to ollama/<alias> and LM
Studio to openai/<name>. The lmstudio loop skips an alias Ollama already
publishes, so two backends never produce a duplicate model_name and Ollama
keeps precedence. lmstudio/default is gone; the model name is now the same
whichever backend answers it.
Ollama's preload_models changes shape from a list of strings to the {alias,
name} entries. Every consumer moves with it: the pull command, the template
loop, and both expressions in 16_ai.yml. An inventory that overrides
`applications.svc-ai-ollama.services.ollama.preload_models` with the old string
list will break at map(attribute='alias'); this is a deliberate shape change,
in the spirit of 7b15c61dfc.
The CI model list is not a property of an installation, so AI_LOCAL_MODELS_CI
lives in inventories/development/default.yml, whose own header scopes that file
to the cross-cutting environment knobs. test_variable_definitions only knew
group_vars, so it reported the constant as undefined; the shipped inventories
are a definition source too, because the deploy CLI bakes them into host_vars,
and the guard's own docstring already claimed "defined somewhere in the
repository".
A latent crash came out of this: `| list | first` raises on an empty sequence,
so a backend deployed without preload models blew up LITELLM_CHAT_MODEL. Ollama
with only embedding models could already hit it; the existing test dodged it by
rendering only _SERVED. `| default('')` now yields the empty string that
_SERVED already reports as false.
Tests: config.yaml.j2 is rendered rather than read, over ollama-only,
lmstudio-only, both and neither, asserting the published model_name set, that
both backends name the same alias, that a shared alias is published once and
routed to Ollama, and that the render is valid YAML. test_litellm_chat_model
gains the lmstudio branch, the no-preload case and the same-model assertion.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The marker names the reason a role has no upstream homepage or demo video, and
it sat in meta/main.yml because a comment-only info.yml parses to an empty
document, which the schema lint rejected outright. So the exemption lived one
file away from the two fields it explains, and a reader looking for the video
found neither the field nor the reason it was missing.
The schema lint now accepts an empty document exactly when it carries the
marker, and that is what allows the move. The media lint reads the marker from
info.yml rather than main.yml, and its failure message names that same file, so
the fix it suggests lands where the absent field would have gone.
A hundred and two roles follow. Every occurrence was a standalone comment on
the second line of main.yml, directly under the document start, so the move is
a line delete and an insert at the same offset. Seventy-three roles already had
an info.yml and gain the line; twenty-nine get one whose whole content is the
document start and the marker. All hundred and two reasons carry over verbatim,
including the thirty-five bespoke ones.
Nothing downstream sees a change. Of the four readers of meta/info.yml, three
coerce a null parse through `or {}` behind an isinstance guard, and
utils/cache/applications.py skips a falsy parse before assignment, so
applications.<role>.info stays absent for the twenty-nine exactly as it was
while the file did not exist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Production block never set one. inventory provision writes only
ansible_connection for a local host, so DOMAIN_PRIMARY stayed at the
group_vars default of infinito.localhost and the deployment served a zone its
own resolver knows nothing about. CoreDNS templates answers for
INFINITO_DOMAIN_RE and nothing else, so every lookup of the deployment's own
name came back NXDOMAIN.
Mastodon is where that surfaced. tootctl accounts create validates the address
through EmailMxValidator, which resolves MX and then, for the domain itself,
A and AAAA. All six queries returned NXDOMAIN in run 33574056480, the
validator recorded email :unreachable, and the admin account was never created.
Only the guide rows were affected: the matrix rows write
DOMAIN_PRIMARY: "{{ lookup('env', 'INFINITO_DOMAIN') }}" into their host_vars
and resolve fine.
The template now carries DOMAIN=<your-domain> in both the host and the
container form and threads it into --vars, with the container form passing it
through the docker run boundary as well. The two guide runners substitute it
the way they already substitute the SSH key, so the replay deploys under
INFINITO_DOMAIN like every other CI deploy. compose_deploy.sh moves its env
load ahead of the substitutions to have the value; host_deploy.sh had no env
load at all.
A production reader is asked for a domain now instead of silently inheriting
infinito.localhost, which no real deployment wants either. The shipped default
keeps its value: sys-svc-nss documents that *.localhost resolves through the
myhostname NSS module without any zone, and host-side tasks lean on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The audit read as prose, so a role could sit under a classification nobody had checked against its own declaration, and "blocked" could not be told apart from "nobody looked". The sections are now named after the exact values a role declares, which makes the section titles the vocabulary: one rule holds every role under its own declared classification, a second rejects a classification outside the vocabulary, and a third requires a named blocker per blocked role while forbidding a blocked role from shipping an enabled surface.
The report itself is generated from the roles' own metadata and a drift check imports the generator and compares against the committed file, which caught a stale report twice while this work was in progress. It lives under docs/contributing rather than beside the requirements, because a generated file in that directory reads like a finished criterion.
The criteria across seven requirement files now carry what they rest on: what closed them, or what blocks them and where that was measured. Two coverage limits stay stated rather than hidden: the report is built only from roles that declare an MCP block, so a role blocked without one is invisible to it, and a ticked criterion here means implemented and tested rather than proven in operation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tor merge brought svc-net-tor in on 192.168.74.0/24 and left
web-app-socialhome on 192.168.75.0/24, both of which the AI services already
claimed. Move litellm to 192.168.80.0/24 and lmstudio to 192.168.108.0/24;
each is unique across the repository and nothing hardcodes the old values.
A different layer from 0f6728a4c: that one moved Tor's virtual address range
out of the docker pool in 10/8, this one resolves concrete docker networks in
192.168/16.
Both services ran on these subnets throughout the session; litellm answered
consumers on 192.168.80.6.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
acc9e25c0f moved the volume lookups to context-wired signatures, taking
application_id from the task context instead of an argument:
lookup('container_volumes', application_id, service_name) -> (service_name)
lookup('compose_volumes', application_id) -> ()
Six templates were left on the old form: the three AI services and hermes,
homeassistant and openclaw. One of them also picks up the matching
container_healthcheck lookup.
This is the same failure mode as the php paths: a refactor that leaves
callers behind, and the break only appears when the template renders, not in
the gate. A deploy in this session died exactly that way, on a template still
passing application_id to a lookup that no longer takes terms.
No file in the repository carries the old signature after this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Under swarm an unpinned role's volumes are rewritten onto the shared NFS
export while its services run one task per node. A role that declares no
RDBMS keeps its state in-process, so several replicas then write the same
SQLite file, Mnesia store or search index over NFS and corrupt it. This
already cost a deploy: openclaw ran three replicas against one shared
export and its state directory answered "file is not a database".
The existing unpinned-no-engine-volume lint states the right rule but
detects state through a hardcoded path list, so app-specific paths slip
through and each case gets pinned reactively after it fails. Add a lint
that joins three facts the roles already declare: the volume is NFS
backed (asked of the production swarm_nfs_backed, not re-derived), it is
mounted read-write, and the role runs no RDBMS. Share-safe payloads carry
a justification marker.
Replicas resolve service-first and fall back to the primary entity, and
a role whose swarm mode is disabled is skipped; both cases exist in tree.
Also correct the requirement and the design doc, which still described
the per-volume opt-in model that b09598dc8 replaced with placement
derived backing. Requirement 023 contradicted itself: line 76 already
described the implemented behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Read platform_arch from the role config and append a node.platform.arch
constraint where it is set, so a role whose image exists for one
architecture only cannot be scheduled onto a node that could never run
it. LM Studio ships x86_64 alone and is the first consumer.
The constraint lives with the other placement rules rather than in the
consuming role, because architecture binding is a property any role with
a platform-specific image can carry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Open WebUI talked to Ollama directly. It now speaks the OpenAI-compatible surface of svc-ai-litellm with a per-consumer virtual key, so a second backend or an external provider becomes a gateway config change instead of a role change. A lint keeps future roles off the model servers.
The gateway advertised a hardcoded model list (ollama/llama3, ollama/nomic-embed-text) while the matrix variant preloads smollm2:135m, so it offered only models the backend did not have and hid the one it did. The list is derived from the ollama preload SPOT now.
The chat spec followed the rename: it gates on litellm instead of the removed ollama service flag, and picks the first non-embedding model from /api/models rather than pinning a variant-specific name, so it exercises the chain rather than a fixture.
Routing openwebui through the gateway pulls both model backends into the closure of every role that consumes it, which pushed nextcloud variant 2 over the memory budget. The budget stays at 32/64 GB; instead the matrix variants of svc-ai-ollama and svc-ai-lmstudio now declare the size their test model needs (1g/2g) while the base keeps the 6g/8g that llama3 and mistral require. The budget guard only saw half of that: it replaced the evaluated role with its variant but counted every dependency from meta/services.yml, a combination no matrix round deploys. It now evaluates every variant-carrying role at the same round index, falling back to variant 0, the way the deploy planner does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The eight new roles shipped with the README template unfilled: every one
of them described itself as "[Name](https://example.com/) is an
application" with a single "Describe a capability" feature. They were
implemented, deployed and tested; their documentation was placeholder
text.
Each role now carries a Description, Overview and Features written
against its own services, tasks and templates, with TODO.md as the
binding list of what is not wired. That caught a false claim: OpenClaw
advertised routing through the shared gateway, but it reads model
providers from openclaw.json rather than the environment, so the gateway
only shapes its deploy topology. The claim is gone.
The overview table derives its Name column from each README H1, and all
eight had kept the humanised fallback. Worst of these, both LiteLLM roles
rendered as "Litellm" on adjacent rows, so a reference in the Integrated
with column could not say which one it meant; they are now the Gateway
and the Admin UI. The rest take their official spelling.
The galaxy descriptions all repeated the product name the Name column
already carries, one named a role id in prose and one used a hyphen as a
dash. svc-virt-kata also claimed to install Kata, which it does not: it
installs gVisor and registers an already-present Kata shim.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consumers of the model plane speak one OpenAI-compatible endpoint and
carry one virtual key; which backend answers is the gateway's business.
svc-ai-litellm renders its model_list from the round's actual topology,
so an alias only exists when its backend role is in group_names, and it
emits an explicit empty list when no backend is present at all because
LiteLLM rejects a bare model_list: key.
Keys reach the gateway through the per-consumer credential rather than
being minted there: 01_keys.yml probes /key/info and only POSTs
/key/generate on 404, so reruns stay idempotent. The consumer list is
built in 00_core.yml, not inside the included file, because 01_keys.yml
delegates to the hosting node and delegate_to is templated before when.
The gateway declares ollama and lmstudio as consumer flags so the
network renderer attaches it to its own backends; without them it cannot
reach them.
svc-ai-lmstudio runs the headless llmster daemon behind expose only, so
it is reachable from the gateway but never bound on the host, matching
svc-ai-ollama.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
