A backend pulls its preload model at deploy time but only loads it into
memory on the first inference. That cold load outlives a reverse proxy's
read timeout, so the first consumer request comes back as a 504 while every
later one succeeds.
Send one max_tokens=1 completion from inside the litellm container right
after the per-consumer keys exist, where the container id, master key and
port are already resolved. Five retries cover a gateway that is not
listening yet.
This pairs with the ollama keep_alive setting: the warm-up pays the load
once, keep_alive stops it decaying after five idle minutes.
Note this hardens the deploy: a gateway that cannot serve its configured
model now fails the role instead of surfacing later as a consumer error.
That makes the openrouter/auto fallback with its empty API key loud rather
than silent.
The script ships with a mirrored unit test rather than a nocheck waiver; it
patches urlopen and asserts the URL, method, authorization header, model,
max_tokens and that the timeout exceeds a proxy read timeout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merged branch moved every role-local script out of files/ into a folder
named for its language (files/python/, files/php/, files/shell/), but the
moves lost their staging and never landed. HEAD still carried the old paths
while the worktree had the new ones.
Move the 24 remaining scripts and update the six places that resolve them:
the mcp-adapter instance/isolation/probe tasks, the litellm key task, the
adapter README and the endpoint-path lint. Every changed line in those
files is a path.
Also drop roles/sys-svc-compose/files/swarm/stack_ready.sh, a stale
duplicate of the copy already tracked under files/shell/swarm/.
The added comments are the branch's own `nocheck: mirrored-unit-test`
directives, which state why a script whose measurement only exists inside
its container cannot have a mirrored unit test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ollama unloads an idle model after five minutes by default. Preloading pulls
the model at deploy time but only the first inference loads it into memory,
so a consumer prompting a while after the deploy pays a cold load that eats
into its own provider timeout.
Declare keep_alive next to preload_async_time in the services SPOT, read it
in vars like its neighbours, and pass it to the container as
OLLAMA_KEEP_ALIVE. `-1` keeps the model loaded indefinitely; `ollama ps`
then reports UNTIL: Forever instead of Stopping...
This removes real latency but it is not what fixed the zammad timeout that
prompted it: that was a model which never emitted a stop token, addressed
separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The matrix variant preloaded smollm2:135m, which is too small to emit a
stop token reliably. Ollama's own log for the prompt "hi" shows it running
away rather than running slowly:
n_gen = 22011, tg = 86.14 t/s
slot context shift, n_keep = 4, n_left = 4091, n_discard = 2045
aborting completion request due to client closing the connection
86 tokens per second is fast; 22000 tokens with repeated context shifts is
a model that never finishes. Every completion without a hard token cap ran
until the consumer's own timeout, which is why zammad reported
AI::Provider::ResponseError after exactly 300s while short warm-ups and
provider pings returned 200 immediately.
Preload qwen2.5:0.5b instead: a comparable footprint (397MB vs 270MB) that
is instruction-tuned and terminates.
web-app-openwebui asserted the served model with a hardcoded
id.startsWith("smollm2"), so it would have failed on any model change.
Render the deployed preload model into its Playwright env and compare
against that instead of a literal family name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LITELLM_BACKEND_ROLES decides which backend branch LITELLM_CHAT_MODEL takes:
the first non-embedding ollama preload model, lmstudio/default, or the
openrouter/auto vendor fallback. It read lookup('deployment').running, which
plugins/lookup/deployment.py resolves as `whitelist or groups`.
On any targeted apps= deploy that whitelist holds only the roles named on
the command line, while backends like svc-ai-ollama arrive as dependencies
of the round. Both branches therefore missed and every consumer was
configured with openrouter/auto, a vendor model the in-cluster gateway does
not serve. The gateway answered each prompt with
400 Invalid model name passed in model=openrouter/auto
while /v1/models listed smollm2:135m and lmstudio/default, so the surface
looked correctly wired while no consumer could complete a prompt.
Read .groups instead, the round's deployed closure, which is what the
`'<role>' in group_names` form used across meta/services.yml already means.
This reverses the selector introduced in 46a16127fc, whose rationale in
requirement 036 assumed the two selectors agree on a single-host round;
they do not once apps= is set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
TOR_EGRESS_VIRTUAL_NET_IPV4 was Tor's upstream default 10.192.0.0/10, which
spans 10.192.0.0 to 10.255.255.255 and therefore contains the whole docker
pool that 08_networks.yml allocates as 10.208.0.0/12.
The value drives an iptables rule in svc-net-tor/tasks/router.yml that
REDIRECTs TCP for that range into Tor's TransPort. With the ranges
overlapping, every connection from the controller to a container IP or to
the docker bridge gateway was captured and handed to Tor, which has no
mapping for a private address and resets it. The failure surfaced as
`errno 104 Connection reset by peer` against containers that had been up
for tens of minutes without a restart, and it reached anything addressing a
container through DOCKER_REACH_HOST, not just the LDAP schema load where it
was first seen.
Move the virtual range to 10.240.0.0/12, which sits above the docker pool.
Nothing else in group_vars or roles allocates the upper 10/8 space.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The container healthcheck logged in as root with MARIADB_ROOT_PASSWORD, so
it asserted two things at once: that the server is up, and that the root
credential matches. MariaDB only applies MARIADB_ROOT_PASSWORD while the
data directory is empty, so any later credential regeneration left the
container permanently unhealthy while every application user kept working.
That mixing produced a deadlock. sys-svc-compose waits for database health
before the play reaches svc-db-mariadb, but the repair for a drifted root
password lives in tasks/02_reset_root_password.yml, which runs after that
gate. The repair could therefore never run when it was needed: a failed
round showed the health wait retrying 65 times with the reset reached 0
times, while the round with this change reached the reset and root logins
worked again afterwards.
Use the credential-free check the image ships instead. It authenticates as
the mysql system user over the unix socket, so health answers only "is the
server up?". The credential question stays covered by the role itself:
tasks/00_core.yml probes the desired root login, includes the reset on
mismatch, and then waits until MariaDB accepts root credentials.
Verified against a container Docker reported as unhealthy: the new command
exits 0 there, and after the deploy the container came up healthy with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without a zip extension or an unzip/7z binary Composer silently drops to
--prefer-source and mirror-clones every dependency, which exceeds its
300 s process timeout on phpunit.git. That took the Arch workspace job
from a 2m16s test-unit to a 16m08s failure. Debian never saw it because
apt pulls php-zip and unzip in as Composer dependencies; pacman does not.
Arch ships no zip extension and no php-zip package, so the extractor has
to be the unzip binary rather than the extension. Its mapping already
covers all five package managers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
requirements.git.yml carried 3 of the 7 collections the Galaxy file requests, and none of the transitive ones. When galaxy.ansible.com started returning 502, scripts/install/ansible.sh fell back to the git file, cloned its three collections, then resolved community.general's dependency on community.library_inventory_filtering_v1 against Galaxy again and died — five times over, taking make install and the whole CI workspace matrix with it.
All seven collections plus that dependency now resolve from git. Verified with an install run while galaxy.ansible.com was still 502: exit 0, seven collections installed, zero Galaxy contact.
A lint pins the parity so the fallback cannot rot back out of sync unnoticed; opt out per entry with the galaxy-git-parity nocheck marker.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A cancel request does not stop a running step. The runner asks it to end, and a step that ignores the signal keeps going, bounded only by its own timeout-minutes -- 350 for a deploy. Meanwhile the concurrency group holds the newer run on pending until the occupant is terminal, not until it is cancel-requested. Measured on run 32859801382: the cancel was accepted at 15:27:45 and thirteen minutes later ten deploy jobs were still running and the newer run still had not started.
The sweep now waits FORCE_CANCEL_AFTER_SECONDS and force-cancels whatever is still not completed. Cleanup steps are skipped for those, which is the price: the artefacts of an already superseded run help nobody. Only entry-cancel-superseded.yml sets the variable -- a closed pull request and a deleted branch have nothing waiting on the group, so they keep cancelling and walking away.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
force_cancel_in_progress existed so a manual sweep could supersede a running orchestrator on any ref. It was a flag every caller had to remember to pass, and only one ever did.
A called workflow inherits github.event_name from its caller, so the orchestrator can see for itself that a human dispatched the run. The input is gone; the group reads the event instead. Automatic triggers keep following the ref alone -- everything but main -- and a manual dispatch supersedes everywhere, main included, because someone typed it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds entry-cancel-superseded.yml to the workflow catalog and widens the cancel emoji's legend, which named only PR close and branch delete.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A long orchestrator sweep sometimes survives its concurrency group: the occupant sits on queued while its matrix waits for runners, cancel-in-progress does not reap it, and the newer run parks on pending until someone cancels the sweep by hand. PR #380 sat that way for nine hours behind a run from the previous commit.
entry-cancel-superseded.yml cancels those leftovers through the API. The group stays primary and untouched: it reaps within seconds and needs no runner, so the fallback normally arrives to find nothing left to do. It carries no concurrency group of its own, because the run holding a group cannot be the run that frees it.
Selection mirrors what the group would have done: cancel every matched run except the newest in the group instance, keyed the way the group is keyed. Not by head SHA, which goes stale while the job waits for a runner and would cancel the newest commit's CI while sparing the stale one. INCLUDE_PATHS keeps the sweep inside the group's own members, so a push to main cannot take the cron workflows with it.
deleted_branch_runs.sh becomes branch_runs.sh, since it now also serves live branches, and the shared parts move to lib.sh. A rejected cancel is logged and the remaining runs are still attempted; only a denied token fails the job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI_CANCEL_IN_PROGRESS gated whether a new run cancels the one before it, and it had to be remembered per repository: unset upstream meant cancel, set to false meant do not, and a fork inherited neither. Cancellation now follows the ref. Every branch but main supersedes its own runs; main never does, because a half-finished pipeline on the default branch is worse than a queued one.
call-orchestrator.yml compares github.ref rather than github.ref_name: a called workflow inherits the caller's ref, which is refs/pull/<n>/merge for a pull_request caller. The manual entry workflow keeps overriding the rule through force_cancel_in_progress.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role tracked `main`, so every rebuild took whatever upstream had merged
since. On 2026-08-24 that was e7c6f80, "bump typing-validation from 1.2.12 to
2.2.1" - a release requiring Python >= 3.14, which turned three CI drills red
in one run. The image now names the commit it was tested against.
Upstream publishes no tags at all, so a semver pin is impossible and the
non-semver pin carries the marker the freshness lint asks for, rather than
opting out silently.
`git clone --branch` accepts a branch or a tag and refuses a commit, so the
clone becomes a shallow fetch of the single ref. GitHub serves an arbitrary
reachable commit that way, and the form takes a branch or tag unchanged if the
pin ever moves back to one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A defaulted build argument answers a question the caller failed to answer. The
build then succeeds against a base image, version or path nobody chose, and the
mistake surfaces as wrong behaviour at runtime instead of a failed build.
web-app-bridgy-fed shows the cost: its metadata declared python 3.14 while the
Dockerfile default silently built against 3.12. Both look plausible, so nothing
pointed at the drift. It and web-app-postmarks now take the base image from the
same services declaration the rest of the role reads.
The lint walks every tracked Dockerfile, template variants and paths outside
roles/ included, which the existing ARG-scope check does not. A second
assertion fails when the scan finds no Dockerfile at all, so the rule cannot
pass vacuously if the walk ever breaks.
Touching the two Dockerfiles brought their pre-existing narration under the
comment lint; what carried substance moved into the file header.
