Replaces the note that explained why the keys were discarded. Records where
the key lives and why there rather than the inventory, what it does and does
not protect, and the two costs: in swarm DIR_SECRETS is the NFS export, and
the key is root-equivalent.
Also corrects the MODE_RESET warning a second time. A reset is no longer a
lockout, because the key survives it and rebuilds the AppRole.
Six assertions: the key is persisted on the deploy that creates it, it is
loaded before the logins that may need it, recovery is gated on both logins
having failed and a key existing, the rebuilt AppRole is the inventory one
rather than the recorded one, the recovery path ends by re-pinning, and a
rejected key fails loudly.
Recovering to the recorded pair instead of the inventory pair would strand
the next deploy the same way, which is why that one is asserted directly.
bao operator init issued a recovery key on the first deploy and the play read
only root_token from the JSON, dropping it. That left the AppRole as the sole
administrative path, so losing both halves of it ended the deploy for good on
an instance whose data was still readable.
The key is now persisted at init and loaded before the logins that may need
it. When neither the inventory AppRole nor the recorded one is accepted, the
run rebuilds the AppRole from it instead of failing. The hard fail remains for
the case where there is no key either.
The three first-deploy tasks move into a block: a third sibling sharing the
same when is what test_no_repeated_when_runs rejects.
DIR_SECRETS is the durable home a role actually has. The inventory would be
the natural one, but no role writes host_vars -- the CLI writes it before the
play. DIR_SECRETS outlives MODE_RESET and a replaced swarm manager, both of
which take .applied.json with them, and svc-bkp-secrets-2-local backs it up.
Mints a root token with bao operator generate-root and hands over to
02_auth.yml, which re-pins both AppRole halves to the inventory values.
The four protocol steps run as separate tasks because the token comes back
XOR-wrapped with a one-time password and the image carries neither python nor
jq, so the CLI performs each step and Ansible parses the JSON -- the same
split the rest of the role already uses for bao status.
Records why the key is held out of the rotation, how to rotate it
deliberately, and what pinning costs. Also corrects the MODE_RESET warning:
with the seal key pinned the store stays readable, and what a reset destroys
is the AppRole record, leaving no administrative path back in.
The rationale rested on the key rotating every deploy, which it no longer
does. Deriving the id from the key is what keeps it in step with the key,
however rarely that happens.
The header claimed every generated credential rotates per deploy. The seal
key no longer does; it is recorded for the deploy that follows an operator
changing it.
Nothing stopped the seal key drifting back into the per-deploy rotation, and
the regression is invisible until a deploy is skipped and the node cannot
unseal. Assert the pin on the seal key and its absence on both AppRole
credentials.
The static seal carries one generation. Rotating the key on every deploy
spends that slack continuously, so a run that skips a generation, or dies
between the two AppRole pins, leaves the raft store readable only from a
volume backup plus the key that wrapped it -- and the recovery keys that
would authorise a rekey are discarded at init.
Frequent rotation buys little here in return. The key lives in the
vault-encrypted inventory either way, and whoever holds that inventory holds
the AppRole too, so they reach the data without needing the seal key. An
operator now rotates it deliberately by changing the inventory value; the
previous_key path stays in place to carry that.
The AppRole keeps rotating per deploy. It is the live authentication path,
and the fallback login exists to absorb exactly that.
The note told readers to set an explicit password until a commit landed that
pins one at inventory creation. That commit is an ancestor of this branch, so
the workaround it prescribed is no longer needed, and the branch URL it linked
answers 404 now that the branch is merged and deleted.
Brings in Kevin's AI-agent work, the svc-net-firewall role and the update
tooling. One conflict, in the MCP proxy audit: both sides appended a role to
privileged_control_plane, ours web-app-openbao and upstream svc-net-firewall.
Taken as a union at 25, which is what test_mcp_audit_completeness requires -
neither role declares meta/mcp.yml, so the audit is the only place they are
classified.
The feature branch it linked to has been merged and deleted, so the URL now
answers 404 and fails the reachability suite. The commit the sentence already
named survives the branch.
The role installs its own tooling only when wg-quick is absent, and these
images already carry wireguard-tools, so the package list the role declares
is never applied here. The sysctl binary has to be baked in instead.
Enabling forwarding on the hub goes through ansible.posix.sysctl, which
shells out to the sysctl binary. Minimal RedHat and Arch images ship no
procps, so the hub failed with 'Failed to find required executable sysctl'
after every interface had already rendered.
The swarm drill against web-app-openbao establishes both meshes, rotates
every key mid-run and completes the update pass without an operator touch,
and every spoke crosses the tunnel with a full-size payload.
CI job 105881436871 on 0b7a8a95d:
A container path declared in meta/volumes.yml is written out again.
roles/svc-ai-lmstudio/tasks/utils/preload/node.yml:7: /root/.lmstudio/models
(declared as 'lmstudio_models')
The disk guard typed the mount target instead of deriving it. That matters more
since the split: the role now carries two nested volumes, and which of them holds
models/ is decided solely by meta/volumes.yml. A second copy of the path drifts
the moment the mount moves, which is the drift 29015311b6 built this lint for.
LMSTUDIO_MODELS_TARGET resolves it through lookup('volume', …).target, next to
LMSTUDIO_SHARED_MODELS_PATH, which already reads .name from the same entry.
The test script held the same path a second time. The lint only reads yaml and
templates, so it would never have reported that copy, and it would have gone
stale just as quietly. It now receives the value through test.env.
Not caught locally, and the reason is worth recording: the preceding `make
quality` was red on three formalities of the new test assets. I fixed those,
re-ran only those three lints, and deployed. This finding sat a few lines below
them in the same output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the mirror case pruning has to survive and its inverse, that an owned mesh
is left untouched.
The existing assertions searched the file for 'swarm:' and 'data:', which the
credential keys mesh_private_key_swarm and mesh_private_key_data also match, so
they passed against a file whose mesh block was empty. They now read the mesh
names out of the parsed document.
Runs first so a mirrored entry cannot be mistaken for this host's own, and
reports what it dropped, because a silently removed mesh is exactly the kind of
change that should be visible in a deploy log.
Writing a member's own entry never removes another's. After the swarm reset
mirrors one node's host_vars onto every other, the NFS server keeps the hub's
swarm entry and brings up a second interface claiming 10.100.0.1. Its own data
mesh stays healthy, so nothing looks wrong on it -- but it now answers for the
hub's address and swallows the return path, and every worker's route to the
export dies there.
Observed as a worker with a live tunnel, a correct route and 100% packet loss
to the NFS server, while the hub reached the same address fine.
c6e8990e25 kept the daemon's auth key off the shared volume and made the red
job green, but it left multi-replica operation as broken as it found it: three
daemons still shared one /root/.lmstudio, where LM Studio rewrites
extensions/plugins with rm -rf at every start, writes settings.json in place and
resets it to defaults on a torn read, and re-copies a 102 MB bin/lms while peers
execute it.
Splitting the tree by access pattern fixes that. /root/.lmstudio is node-local,
because it is per-daemon state the daemon rewrites in place. models/ stays
shared, because a blob is written once and read forever.
That leaves one question, and it is temporal rather than mechanical: the daemon
discovers models by an unconditional scan at startup and by an inotify watcher,
and inotify never sees writes made by another NFS client. So a model seeded
after a daemon started is invisible to it, whichever tool wrote the file. The
previous design answered that with `container service update --force`, which
restarts the whole inference backend svc-ai-litellm points at, on every deploy.
The seed now runs before the stack does. 00_core.yml places the blob on the NFS
export ahead of the compose include that deploys the stack, so every daemon
finds a complete tree at its own startup scan and nothing has to be restarted.
Measured on a three-node swarm with NFS:
Fetch the models into the shared store 136.14s one 531 MB download
Pre-pull on mgr-01 0.69s
Pre-pull on wrk-01 0.50s
Pre-pull on wrk-02 0.49s
Half a second is not a download. All three daemons recognised the pre-placed
model, so the cluster now stores and transfers it once instead of three times.
The nested mount across two devices raises no EXDEV, and the per-node disk guard
correctly skipped itself four times because the pre-seed covers that check.
Vendor coupling is gone from the role. `repo` and `url` are declared fields
rather than pieces cut out of a HuggingFace URL with a regex; a model on any
host needs only those two. `lms get` still knows only HuggingFace, so `source`
stays for it, and a non-HF model simply leaves it unset: the blob is seeded and
the startup scan picks it up. The external check follows, comparing against the
HuggingFace API only for entries whose url actually points there.
The guard arithmetic was wrong and is now derived rather than guessed. Peak
usage is the missing bytes plus the largest single model, not twice the declared
total, so the read-back moved ahead of the check and subtracts what the store
already holds. Before, the requirement grew after every successful seed. Staging
moved beside the tree instead of inside it, so an aborted transfer cannot leave
orphans where the indexer scans.
A deployment test ships with it and states its findings rather than its exit
code: it fails on `authPacket` by naming the shared key, on a missing or
mismatching blob by naming path and digest, and on a model that is on disk but
absent from /v1/models by saying the startup scan did not happen.
Gone with the restart: import.sh and its container-cp suppression, seed.yml,
rescan.yml, and the disk-copy factor of three that only existed because the
import staged through the container.
Residual: compose is untested here. There the pre-seed is off, the node-local
guard is the one that runs, and `lms get` is the only fetch, so a non-HF model
is not loadable in that mode. Seeding node-local volumes is the follow-up that
closes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The failure this covers is silent in the inventory and total on the cluster:
every node adopts the hub's key and address, every interface claims 10.100.0.1,
and nothing handshakes. Asserts the mirrored hosts read as unkeyed and that a
rewrite gives the three swarm members three distinct addresses again.
The gate mirrors the manager's host_vars onto every node, so the per-host mesh
each member needs is gone by the time the update pass deploys. Running the
writer again restores it, which is also what 'the tool runs before every
deploy' has to mean once anything rewrites the inventory mid-round.
The swarm rotation gate mirrors one node's host_vars over every other, which is
correct for credentials that are identical per host and hands every node the
hub's identity here. Observed on a live cluster: a worker's swarm.conf was
byte-identical to the manager's, same private key, Address 10.100.0.1, a
ListenPort it should not have, and both spokes as peers. Nothing handshaked and
every NFS mount over the mesh failed.
Re-running the writer alone does not fix it, because a mirrored file presents
the hub's public key and credential and would be read as already keyed. The
entry now names its owner, so one that names a different host counts as unkeyed
and the pair is reminted.
The preload treats "the model is already there" as success in three places, and
all three were dead. They match on the literal `already downloaded`, which the
shipped CLI never emits: `strings` over /app/.bundle/lms in the deployed image
returns zero hits. What it prints is
You already have this model. You can load it with:
One of the three was doubly dead. The `until:` reads `stdout`, but the CLI
builds its logger as
const console = new console$1.Console({
stdout: process.stderr,
stderr: process.stderr,
so every level including info goes to stderr and that clause could never fire
whatever string it carried. The other two are fine on the stream: `failed_when`
concatenates stdout and stderr, and the reap path goes through
async_failures._said(), which joins msg, stdout and stderr.
Effect until now: an already-present model was tolerated only when `lms get`
happened to exit 0. On a non-zero exit the task failed with the model sitting
on disk, and none of the three guards could catch it.
This matters more from here on, not less. The role is moving towards seeding
the model before the daemons start, which makes "already present" the normal
case rather than the exception.
Measured, not inferred: both strings and the logger construction were read out
of image 48abf3b7e061, the one the failing CI job ran.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The export ACL was built from each swarm node's default_ipv4 fact, which is its
underlay address. A client mounting over WireGuard arrives from its mesh
address instead, so the server refused every mount from the cluster: 'mount
system call failed', not a timeout, because the packets reached it and were
rejected.
The pool is allowed as a range rather than the members enumerated, because a
worker is not a data-plane member and reaches the export through the hub, so
which address it presents depends on that hop. The pool is the VPN itself, and
only its members can source from it.
Job 105845259891 (svc-ai-lmstudio#0@swarm, debian/btrfs) died in the model
pre-pull, two seconds in, not on a timeout:
Error: Failed to authenticate: Provided authPacket lms-cli is not allowed.
The daemon was up and rejecting deliberately. Its own log carries the server
half of the same line, through SystemExternalAPIProvider.
Reproduced locally in compose, without swarm and without NFS, which is what
pins the cause. Two daemons sharing one /root/.lmstudio:
daemon 1 writes .internal/lms-key-2 md5 c82691a9b6cb54c0eb233d2842f9f358
daemon 2 rewrites it md5 e407c5e17d7392b88080e8ed216f641d
lms against daemon 1 Provided authPacket lms-cli is not allowed
The key is the runtime half of the CLI's passkey and it lives in .internal,
not in credentials/, which is empty. /app/daemon-run.sh starts `lms server
start` in the entrypoint, so the CLI authenticates at container start and the
loser of the race is whichever daemon started first.
Swarm only supplied the second daemon: the role pins no replicas, so
compose_replicas defaults to one task per host, and /root/.lmstudio carries no
`nfs: false`, so the swarm rewrite puts all three on one NFS export. The
diagnostics show three task slots and five .ready-<timestamp> markers in one
directory.
.internal now has its own node-local volume. The models stay shared, which is
the point: the blobs are what the nodes may share, the daemon's identity is
not. Deobfuscating the daemon bundle confirms the split lines up with the
vendor's own layout: modelIndexCache, ggufMetadataCache and modelData resolve
inside .internal, defaultModelsFolder and hubModelsFolder resolve outside it,
and the folder watcher explicitly skips .internal when looking for models. A
missing index is a supported cold start: the accessor falls back to
emptyModelIndex with neverError, and startup re-runs the full directory walk.
`name:` is pinned because backup_volume.py emits `entry.get("name") or
semantic_name`, while the volume materialises as <stack>_<key>. Without the pin
the exclusion would name lmstudio_internal, the volume would be
lmstudio_lmstudio_internal as the run's artifact shows, and `backup: false`
would silently miss.
Verified against a real three-node swarm with NFS, the same shape that was red:
the pre-pull ran `ok: [mgr-01 -> wrk-02] => (item=qwen2.5:0.5b)`, the service
reports 3/3, `lms ls` inside a container lists the model with no auth error, and
the job ends 🏁 Job succeeded with no fatal and no failed host.
Shape follows web-app-matomo, whose `tmp` is an `nfs: false` child volume nested
in an NFS-backed parent with the same nocheck; 32 such nested pairs exist across
9 roles.
Residual, and it is not small: multi-replica operation stays unsafe for other
reasons this does not touch. extensions/plugins is rm -rf'd and re-extracted at
every daemon start, settings.json is written in place and resets itself to
defaults on a torn read, bin/lms is a 102 MB copy rewritten while peers execute
it, and model downloads land on their final shared path with no temp file. All
of that was equally true before this commit; a shared .internal merely added the
auth collision on top, and the index is opened noWatch so peers would have
overwritten each other's catalogue anyway. Making that safe means either one
replica or a fully node-local lmstudio_data with a per-node preload.
Second residual, affecting an earlier commit rather than this one: on a node
whose index does not know a model whose file is already in the shared models
folder, `lms get` sits at 0.00% until it is killed. That is the signature
df2fc5084a attributed to a wedged daemon and eef3c3be71 bounded with a dynamic
timeout. The seed imports the file and the pre-pull then asks for it again, so
the wedge may be this collision rather than the daemon. Measured once, on one
node; not yet a diagnosis.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It carries the same shebang as every sibling __main__.py and none of their
executable bit, which ruff rejects as EXE001.
Not reproducible from here: the sandbox masks the mode, so ruff reports the
file as executable whatever it actually is, and a local run passes with the bit
cleared and a cold cache. CI is the only place this rule can fail.
The rotation gate wipes every generated credential so the next deploy has to
carry it. That is right for a per-host secret and wrong for this one: the mesh
keys are correlated across hosts, and the writer that mints them runs once per
round, before the first deploy. Wiped and not rewritten, the update pass would
render an interface with an empty PrivateKey.
rotatable: false says who owns them. The mesh rotates when the writer is asked
to rotate it, as one atomic pass over every member, which is the only way a
mesh can rotate without partitioning.
Declaring meta/meshes.yml made the validator accept the generated block, and
handed every host both meshes whether or not it is a member. The role then
rendered an interface for a mesh with no address and died in the template:
'object of type list has no attribute 1', from splitting an empty subnet.
The failure pattern named the cause exactly -- the NFS server failed on the
swarm mesh, the workers on the data mesh, and the manager, which belongs to
both, passed.
An entry the writer filled in carries an address; a declared-but-unassigned one
does not. That is what now separates a real membership from the shape the
contract declares.
run.sh drives every suite with -n auto and no scope, so xdist hands out single
test METHODS. Each worker is its own process, so setUpClass runs once per worker
the class happens to be scattered over, and an expensive fixture is paid as many
times as the scheduler split its methods.
The integration report shows the shape without interpretation:
TestMcpEnvRender's nine methods came back as six at 277-287s and three at
0.003s. The fast three landed on workers that had already paid. That is 1776 of
the suite's 2282 CPU-seconds spent computing one identical result six times,
twenty Ansible templar renders of the MCP providers each time.
--dist loadscope groups by class, so a class occupies one worker and its
fixture runs once.
1a908a6e5b introduced -n auto and hoisted fixtures in the same commit, so the
hoisting was undercut from the start: work moved from per-method to per-class,
and the scheduler it arrived with promptly charged it per worker instead. This
finishes that commit rather than correcting it.
Measured against the same tree, four suites, before and after:
integration 296.83s -> 219.32s
lint 105.64s -> 84.27s
unit 65.74s -> 53.02s
external 75.69s -> 71.99s
Phase 1's critical path drops from 4m57s to 3m40s. That unit and lint move too
says the pattern was not unique to the MCP class.
Coverage is unchanged, which is the point of the change rather than a side note.
All four suites report identical counts: unit 5857 passed / 9 skipped / 2301
subtests, external 19 / 2, lint 521 / 1 / 374, integration 328 / 2 / 1562.
A second effect is worth naming because it cuts the right way. Several sweeps
keep their non-empty guard in a separate method, e.g. test_fragments.py:139
proving the corpus is not empty before the sweep asserts over it. Under
--dist load those two ran in different processes with separate caches, so the
guard proved nothing about the process the sweep ran in. loadscope puts them
back together.
Residual: loadscope trades balance for locality, and here one class sets the
floor. TestMcpEnvRender holds a worker for ~250s on its own, so the suite cannot
go below that until the ~12s per render is addressed. That cost is the render
guard in utils/cache/applications.py:415 returning the merged map without
caching it, which is production config resolution and deliberately left alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every key a host_vars file carries under applications.<role> must exist in that
role's meta tree, and the mesh is generated rather than authored, so the
inventory validator rejected it: 'Missing default for svc-net-wireguard:
meshes.swarm.address' and one line per key on every member.
It surfaced only at the rotation gate, which revalidates, so the first pass of
a round looked clean and the update pass aborted before any task ran.
meta/<topic>.yml maps to applications.<app>.<topic>, so declaring the shape in
meta/meshes.yml is enough. Values stay empty because the writer supplies them,
and peers is a list, which the validator does not walk.
_all_project_files carried @lru_cache(maxsize=1) and read PROJECT_ROOT from the
module global inside its body, so the root was not part of the cache key. The
first call decided the answer for the rest of the process: point PROJECT_ROOT at
a fixture afterwards and the walk still reports the real repository, or, in the
other order, a sweep after the fixture is torn down enumerates a directory that
no longer exists.
Both failure modes are silent. iter_project_files, iter_project_files_with_content
and iter_non_ignored_files all sit on this one walk, 202 files in the tree use
them, and a lint whose corpus collapses to zero paths reports zero offenders and
passes. Nothing raises.
The root is now a parameter and therefore the key, with room for four of them.
_reset() still clears it through cache_clear(), and the only other caller is
iter_project_files, which passes str(PROJECT_ROOT) explicitly.
Live impact today is none, and that is worth stating: exactly one test patches
PROJECT_ROOT in this module, and it calls _reset() in both setUp and
addCleanup, so the defect was fully masked. Every other patch site in the repo
targets PROJECT_ROOT in a different module. This is about the next test that
forgets, not about a failure anyone has seen.
A regression test covers it rather than a comment. Negative control: with the
key removed the new test fails on the second root with
Items in the first set but not the second: '/tmp/.../first.txt'
and passes once the key is back; the file is byte-identical to its pre-probe
state afterwards.
The lint docstring that quoted the old cache shape is deleted rather than
updated, because a cross-module implementation reference goes stale the moment
that module changes, which is exactly what happened here.
Same class as c4556fe72d, which keyed the applications SPOT by the groups its
payload reads. That one was a live defect in utils/cache/base.py; this is the
latent twin one module over.
Residual: maxsize=4 is a choice, not a measurement. More than four roots alive
at once evicts the oldest, which re-walks rather than answers wrongly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lint fan-out listed lint-php twice, once before and once after
lint-playwright. Two commits added it independently: 82218087c1 for the PHP and
Ruby syntax checks, 6fee7ca136 for the per-language unit suites, and the second
did not see the first because lint-playwright had since been inserted between
them.
The wasted slot was the harmless half. parallel.sh keys target_start,
target_dur, target_rc and target_log by target NAME, so a repeat overwrites the
first run's log path and exit code, and the final pass reads each name once.
Two consequences: drain_one prints the second run's log twice and the first
run's output never appears, and a failing first invocation is reported green as
soon as the second one passes. A duplicate therefore turns a target into a blind
spot rather than into a slow one.
The gate showed the symptom plainly, two lint-php rows with an identical
0m 03s, which reads as a rendering quirk rather than as one target run twice.
Both halves are fixed: the list loses the repeat, and parallel.sh refuses a
duplicate before it loads the environment or starts a single make, naming the
offender and why a repeat cannot be tolerated. The two callers are Makefile:440
and Makefile:822; the test list has no duplicates, so the guard is inert there.
Negative control: `parallel.sh lint-php lint-python lint-php` exits 2 with
"duplicate target(s): lint-php" and launches nothing. The full quality-high run
afterwards dispatches 15 lint targets with lint-php once and stays green.
Residual: the guard covers scripts/make/parallel.sh only. scripts/tests/dns/
parallel.sh is a separate script with its own argument handling and was not
examined.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mirror loop added in eef3c3be71 iterates with
`LMSTUDIO_PRELOAD_MODELS | subelements('mirrors', skip_missing=True)`. That
filter ships with ansible-core rather than with this repository's
`filter_plugins/`, and the allowlist that separates "provided by the runtime"
from "must be defined here" did not carry it, so test-integration went red on a
filter that is perfectly valid:
These filters are used in templates/YAML but have no local definition
(and are not in BUILTIN_FILTERS):
- subelements
Measured against the interpreter the suite runs on, `FilterModule().filters()`
from `ansible.plugins.filter.core` contains `subelements`, so the entry states
a fact rather than a suppression. The set has exactly one reader, the
subtraction in the same file, and the entry removes that one name from the
candidates and nothing else.
This is the same one-line extension as 9118b47654 (regex_findall) and
fa59b29b2a (urlsplit).
Residual: the check can no longer report this name, so a future local filter
called `subelements` would shadow the builtin unnoticed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
infinito-nexus-updater[bot]
evangelostsak
kevinveenbirkenbach
