With single sign-on disabled, the platform administrator had no indexer
account, so the dashboard login answered "Invalid username or password".
internal_users.yml now carries that account with backend role "admin"
while sso is off. The bcrypt salt is fixed per role, so a repeat deploy
renders the same hash.
The dashboard queries the Wazuh API with run_as. The default security
rules only match the users "elastic" and "admin", so every other signed-in
user got "You have no permissions" on the Wazuh pages. A new task runs
api_administrators.py inside wazuh.manager with the manager's own API
account. It creates the rule "infinito_administrators", corrects it when
its content drifts, and links it to the "administrator" role. The rule
matches the same three backend roles that roles_mapping.yml maps to
all_access, so it grants nothing to a principal that did not already hold
indexer all_access. The task prints CHANGED or OK and is idempotent.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
reason() calls itself the single source of truth for both ends of the
pipeline, and the write path honoured that: client.py:228 passes the
directory code into unmask(). Only damaged() read catalog.locale, so
for 50 catalogs the two ends judged against different languages.
44 catalogs carry no locale header, 8 of them non-Latin and therefore
silently past the script check. 6 more name a code their directory
disagrees with, so tl was judged as fil, bh as bho, tw as ak.
The language becomes a required parameter; both production callers
already held the code. Measured over every catalog, header language
against path language, one moves: core/av from 0 to 29. Those 29 hold
pseudo-Norwegian and a degeneration loop in a Cyrillic catalog, and
make i18n-prune has since cleared them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cfe20f0726 and 4915a401d4 wrapped three statements more cautiously than
the formatter wants. Without this the lint phase rewrites the same diff
into the tree on every run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
failed_selections reproduced mode, onion state, distro and architecture
but not the mesh axis. These tokens feed the priority line, where an
unpinned axis offers both states, so a failed direct swarm row came back
as itself and as a meshed sibling: two jobs to reproduce one.
Only a mode the axis covers takes the pin. pinned_vpn_states returns an
empty list for a compose row pinned to the mesh, and an empty list drops
the row from the matrix -- the class ff75343aba already had to fix once
for held runs.
The fixture had drifted from what the matrix emits: row_label built
titles with no mesh glyph while axes.assign writes one for every swarm
and compose row, so parse_label fell back to its False default and the
new &direct assertion would have been right by accident. row_label and
deploy_job_name now place the glyph under the same condition as
axes.assign, and a meshed row has a case of its own, which nothing
covered before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The main merge left two step watchers: run.run_watched from main and the
newer watchdog.run_watched from here. Their disk and memory floors are
byte-identical (6144M, 6% of RAM, abort 75), so run.py now keeps only
run_step and takes the watcher from watchdog. mesh.py reaches it through
run_step and matrix.py already imported it directly.
That alone would have been a regression. The watchdog aborts on io60
alone, which its own docstring says cannot separate a wedged mount from
a busy disk, and the six mesh steps had no I/O kill before: a healthy
sleep on a host with 126G free and no task in D state died at rc=75
after 10s where the old watcher let it finish. The first sample also
lands 10s in, so 50 of the 60 seconds it averages belong to whatever ran
before -- an image pull could abort its successor before that step read
a byte.
The ceiling now takes three things together: a task in D state on the
same sample, WEDGE_SAMPLES breaches in a row, and a step old enough to
own a whole STALL_WINDOW_S of the average. Both refuted scenarios are
regression tests. A real wedge is caught about a minute later than
before; the disk floor that covers ENOSPC is an instantaneous read and
is untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Our side added the architecture axis and main the vpn axis; combining
both put axes.py at 502 lines against the 500 the structure lint allows.
check_pins and row_architectures answer one question: may this row take
what the selection token demands of it. Assigning the axes of a row that
carries no pin is the other job, so the two separate cleanly and axes.py
drops to 397 lines.
axes.__all__ still exports both names, so cli/meta/ci/validate.py keeps
reaching them through axes. The test patches get_role_architectures,
which moved with them, so its patch target moves too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
e17d182d0a gave web-app-docs a switch that turns the language assertion
off on a row where docs only rides along, and read it from the
guide_host environment variable. The deploy workflow sets that variable
on the runner, but a compose deploy runs Ansible inside the infinito
container, where only _FORWARDED_ENV_KEYS arrives. guide_host was not in
it, so the lookup resolved empty, the comparison against the row's own
id was true everywhere, and the switch never fired.
Run 39137606155 shows both halves: the passenger rows 117711219652 and
117711220306 carry guide_host: web-app-docs and each spent about 7000 s
of a 20400 s budget in that one assertion, while the docs row itself,
117711219660, carries an empty guide_host and therefore keeps running
it. Forwarding the variable preserves that split rather than silencing
the assertion everywhere.
APP_ID is in the tuple for the same reason and feeds the guide_role
lookup from the same workflow step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The design and docs commits landed with --no-verify, so the hook that
runs make test never saw them and four repository lints went red on
code that was already pushed.
test_init_files wants an __init__.py in every python tree under tests/;
the sys-front-inj-design directory arrived without one.
ruff S701 rejects Environment(autoescape=False). Both call sites render
from a string, where select_autoescape(default_for_string=False)
returns the same False, so what the test exercises is unchanged and the
explicit escape filter keeps doing its own work.
test_no_env_defaults_in_javascript rejects a default for an environment
value in JavaScript, because it is a second source that hides an unfed
variable. Normalising to the empty string stays allowed and keeps the
expression's result identical for every input.
The env-file schema caps a comment at 140 characters.
The two lint/filesystem files carry nothing but the autoformat reflow.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The instruction replay provisions a fresh inventory inside a throwaway
target, so its compose pull fetched every application image from the
upstream registry while every other CI deploy pulls through the GHCR
mirror. On the moodle row of run 39082924588, 512 s of the 1646 s
ansible run went into that pull.
mirror.sh rewrites the extracted block and not the README template the
site publishes: it inserts the mirrors.yml resolver ahead of the
provision call and passes --mirror. A reader keeps the upstream
references, because a reader has no GHCR mirror to pull from, and the
replay exists to prove the instructions a reader copies. Outside CI the
two GitHub variables are empty and the block stays byte-identical.
The prefix is read from default.env in the checkout, the way machine.sh
already reads INFINITO_PARENT_IMAGE, so no Ansible template has to look
up an INFINITO_ key that belongs in an inventory.
A block that carries no provision call fails the harness instead of
quietly falling back to the upstream registry, and test_mirror.py
renders both published shapes to keep the two sed patterns matched to
what the template emits.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The inline design script carries a CSP sha256 computed at render time. On an
onion request body_filter.lua rewrites the clearnet CDN base inside the whole
head payload, the script's bytes stop matching their hash, and every role
whose script held a CDN URL has its branding refused wherever script-src-elem
has no 'unsafe-inline'. web-svc-logout and web-svc-html are on nearly every
deployment, so this was not a corner case.
The config now rides in the script tag's data-design attribute, which CSP
does not hash and onionize may rewrite freely, and the first statement of the
script reads it back into INFINITO_DESIGN. The executable body is identical on
both vhosts, so one hash authorizes both.
All 35 design scripts now read the config instead of interpolating
design.urls. The 13 that still needed converting keep their render-time
guards, so a role with branding disabled still emits no script at all, exactly
as before. slots left the config object: no script has ever read it.
A new check refuses a branding URL in a script body, so the class cannot
return, and a unit test drives the config through to_json, Jinja's escape and
an HTML parser with a hostile title to prove the attribute round trip.
Not yet proven: that the hash matches on a real onion vhost. This stack runs
without tor and every role deployed here has 'unsafe-inline', so the check
needs a tor-enabled deploy of one of the seven roles that disable logout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A meta/volumes.yml entry that declares backup: false without pinning a
name: had its semantic key emitted into --volumes-no-backup-required.
Docker names such a volume <project>_<key>, and baudolo compares the string
exactly, so the exclusion matched nothing and the volume was copied anyway.
web-app-docs hit it in CI: docs_sites was emitted, the live volume is
docs_docs_sites, and the 17 GB regenerable sphinx tree was backed up into
the 30 GB container root. That filled it and took four jobs of run
39137606155 down with three different symptoms: no space left during the
copy, rsync reading the tree while the builder still wrote to it, and
hlth-disc-space then refusing to start on a full filesystem.
deployed_volume_name() now derives the name the way compose does: the
pinned name: when there is one, otherwise the role's entity name, which is
the compose project, joined to the key. The docstring claimed the old
behaviour was correct and is fixed too.
Three roles were silently still backed up and are now excluded as declared:
web-app-docs, web-app-jellyfin and web-svc-api. The nine entries that pin a
name are unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sys-front-inj-design now renders templates/css/app.css.j2 ahead of the
role's stylesheet and templates/js/design.js.j2 ahead of its script. The
branding URLs, the slot dimensions and the logo guard live there instead of
being repeated per role: the guard that stood in 41 sheets is one condition
in the composing template, and a sheet reads branding through
--design-url-* / --design-slot-* custom properties.
29 stylesheets and 43 design scripts lose their Jinja and move to static
files. Roles whose sheet carries app-specific URLs declare them as
design_css_urls and read them as --design-url-app-*.
The script prelude is prepended only when the role script references it,
and it carries urls only while a logo is configured, so every role that
needs neither is emitted byte for byte as before.
Five checks keep the mechanism honest: a composed sheet may only read
design custom properties that app.css.j2 declares for its role, a template
may not carry Jinja expression text outside its delimiters, a design script
may not quote its config object, and the injected-script parse check now
covers the design channel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two bots bumped the one baudolo version independently. The pip pin
updater rewrites the literal package_name pin in
roles/svc-bkp-volume-2-local/tasks/01_install.yml and opens a PR scoped
to roles/<entry>/, and dependabot bumps the same package in
pyproject.toml. Each PR alone splits the pins that
tests/lint/repository/dependencies/test_baudolo_pin.py holds equal; #995
merged such a split and turned main's test-lint red until #996, and the
next upstream release (7.1.2 is tagged) would repeat it.
The version lookup now takes an optional distribution name and returns
its exact == pin from [project.dependencies]. The install task asks it
for backup-docker-to-local, so pyproject.toml is the only place the
version lives, dependabot is the only bot that moves it, and the pip
updater no longer finds a literal to bump. The lint now forbids a
literal pin in the task and requires the lookup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
compose/cache.shared.override.yml attaches every worktree stack's
infinito service to the primary's network (infinito-nexus-core_default,
172.30.0.0/24) without an ipv4_address. The default network declared
no ip_range, so Docker IPAM handed out the lowest free address and the
worktree stacks took 172.30.0.2, .3, .8, .9 and, whenever the primary
container was down, .10, the primary's INFINITO_IP4. The primary's
`make compose-up` then failed with "Address already in use", and with
it `make quality-high` on main.
The default network now declares ip_range: ${INFINITO_IP_RANGE}, the
upper half of INFINITO_SUBNET. Every static address of the stack (.1,
.4 to .6, .10, .53) sits in the lower half, so a container without a
static address can no longer take one.
INFINITO_IP_RANGE is derived from the final INFINITO_SUBNET after
custom.env is applied, because a worktree pins its subnet there and
`make dotenv-force` regenerates without it in the environment. Existing
worktrees therefore get their own slot's range without a new pin.
A lint checks that every static IP in default.env and every
172.30.${slot}.N pin in scripts/system/worktree/up.sh lies below the
pool.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
make quality-high reformatted three expressions this branch added today: the chat-target chain in the translate role's vars, an assertEqual in the gateway engine tests, and the wanted comprehension in the lane probe. No logic and no rendered value changes; the YAML break folds to a space inside a double-quoted scalar.
Kept separate from the changes that caused it, because a reflow mixed into a behaviour commit hides the behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every catalog got a lane derived from the CPU count, so twenty requests met one model instance. That is not throughput, it is self-inflicted saturation: a single catalog recorded 425 translated, 9 discarded and 40 requests refused with HTTP 503, and a discarded entry is simply left pending.
The work is now split by the backend that will answer it. The languages the engine serves keep the wide pool, because Argos answers in about 0.15s; the rest run after them with two lanes, one to occupy the model and one to refill it while the first writes back. Each pass builds its own client, so the chat pass does not inherit the worker count the engine pass was sized with.
Order is the larger half of this. The table is alphabetical, which is why the last run spent its first 45 minutes on af, a language with no Argos model, while fr and de sat untouched. Cost order delivers the engine languages first. For domain=docs that is 47 catalogs of 3963 entries each in roughly an hour; for domain=core it changes nothing today, because those catalogs are already translated.
The test picks aa against fr on purpose: both orders agree on every pair the fixtures held before, so a test built from them would have passed against the old single pass as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In swarm, docker stack deploy ignores depends_on, so the precheck
service and every init replica ran search_allocation_precheck.sh at the
same time. They shared one probe index: one invocation's PUT could fail
the shards_acknowledged check because another had created the index
first, and either one's cleanup could delete the other's probe. The
init service carries no replica override, so compose_replicas gives it
one task per host in the application group and the collision also
happens between init replicas.
A failing precheck task also never reached has_fatal_task in
stack_ready.sh: with restart_policy none it records a single error,
while has_fatal_task needs three. stack_ready.sh then kept answering
"not converged" until the handler ran out of retries. And a Running
init task counts as converged through running_current_spec, so the
swarm deploy never depended on init failing; the role's "wait for
Zammad init" task is what stops it.
The probe index now carries the container's hostname, which docker
sets to the container id, so concurrent invocations no longer share an
index. compose.yml.j2 renders the precheck service, and init's
service_completed_successfully dependency on it, only outside swarm.
Compose keeps its fast stop on a refusal; in swarm, init runs the same
check in front of zammad-init, a refusal leaves Zammad unseeded, the
wait for Zammad init fails the deploy, and the init service logs carry
the allocation explanation.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On a data disk with less free space than the high disk watermark asks
for, Elasticsearch creates indices but assigns none of their primary
shards. `zammad-init` then reports every search index update as failed
and never exits, so `compose up` waits on
`service_completed_successfully` until its 1800 s task timeout and the
deploy fails without naming the cause (#850). Elasticsearch's default
high watermark is an absolute 150 GB headroom on a large disk.
A new one-shot `precheck` service runs `search_allocation_precheck.sh`
with `restart: "no"`, and `init` depends on it with
`service_completed_successfully`:
- it waits a bounded time for `_cluster/health`;
- it creates a one-shard, zero-replica probe index with
`wait_for_active_shards=1`;
- once Elasticsearch allocates it, it deletes the probe and exits 0;
- otherwise it prints the creation answer and the answer of
`_cluster/allocation/explain` and exits 1.
The check sits in its own service because `init` keeps
`restart: on-failure`: wrapped into `init`, the failing check was
restarted seven times within eight minutes on a local deploy while
`compose up` kept waiting. Inside the stack, a `restart: "no"`
dependency that exits 1 made `compose up` fail after one second with
`service "pre" didn't complete successfully: exit 1`.
Every curl call carries --connect-timeout and --max-time, so a hanging
request cannot outlast the bounded wait. The image's entrypoint passes
any non-`zammad-*` command through `exec "$@"` (bin/docker-entrypoint of
7.1.2), and the image ships `curl`. The watermark values themselves come
with feature/store-i18n (b82591ba9d); this change covers inventories
without a variant cap.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Both new tests read a file their own run has just written, which the uncached-read lint flags because utils.cache.files.read_text is an lru_cache with no mtime invalidation. The two cases want different answers. The rendered inventory is YAML, so load_yaml_any reads and parses it the way the lint asks and needs no exception at all. The healthcheck probe writes its request through bash inside the test, so a cached read would compare against the previous run's bytes and the assertion would pass without proving anything; that one keeps a cache-read directive naming why.
make test-lint went from 2 failed to green, and the first attempt is the reason the inventory test carries no directive: suppressing the read there only moved the complaint to yaml.safe_load, which the right reader answers in one call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The oracle asks a chat model whether a translation states its source. llama3.2:1b cannot answer that: on the suite's own fixture it scored an unrelated text at the middle level instead of the bottom one, and asked as a yes-or-no question it called two identical texts different. With it the suite reached no verdict at all, reporting that the judge read none of the 47 languages the catalogs hold machine translations for. Qwen2.5-3B answers both correctly, and the same run then judged 248 samples.
The pin lives in the lane inventory rather than in AI_LOCAL_MODELS, because a judge that can separate a faithful rendering from an unrelated text is a demand no deployment makes, and the default should not grow a gigabyte to satisfy a check. The litellm config builds its routes from the merged application config, so a lane override reaches the model list and its window without touching the repository default.
LANE_MODEL_ALIAS repeats the one name in utils/inventory/tools.py, because ensure() asks whether the lane already serves before the deploy that writes the rendered inventory, so the probe cannot read it from there. A lane serving one model while the probe expects another never reports ready and redeploys on every call, so a test renders the template and holds the two names equal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lane ran litellm but never republished its port, so a host-side probe hit the compose proxy on the same number and read 502. The lane could therefore never report ready, and the oracle suite skipped itself to a green exit instead of asking a judge. A handler reads the port from the role the way translate.py already does, the runner forwards it, and TOOLS_LANE_KEYS takes it up: compose_file_args layers the override in without checking its variables, so an .env written before the key existed would abort every compose command rather than merely drop the lane.
The gateway's master key is pinned in the lane inventory for the reason the System One key already is: the provisioner would otherwise generate it into a vault the checks cannot read back, and the oracle judge is not a deployed consumer with a virtual key of its own.
serving() stops treating reachability as readiness. A lane brought up before AI_LOCAL_MODELS named a different model keeps serving the old one, and a gateway holding a key these checks cannot use looks healthy from outside while refusing every call they make. It now asks /models with the bearer token and requires the configured alias, and requires the gateway to offer every language the catalogs name, because that advertisement is what the translation run waits on. Without this the lane waved itself through and no change to its roles ever shipped.
Two lane-scoped overrides come with it. The ollama role caps itself at four CPUs, which left the host at 361 percent while twenty cores idled, so the lane gives it the host count the way the engine entry above it already does. And the lane pins the router to its declared order: the decider may send a pair the engine serves to the chat model, which a batch over 183 catalogs cannot pay for. Deployments keep the decider.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generated tcp and http_status probes built their request with echo -e, and echo appends a newline, so the bytes on the wire ended with a stray bare LF after the header terminator of a Connection: close request. uvicorn's h11 reader treats that as trailing data and answers 400 before the handler's 200 reaches the socket, so the probe's grep for a 2xx or 3xx status line never matched and the container stayed unhealthy for good.
In compose the name resolves straight to the container, so an unhealthy but running service still served and only the backup drill noticed, failing with litellm is running/unhealthy after 1200s. In swarm a task joins the service VIP only once it is healthy, so the litellm service sat at 0/1 with no backend and every consumer of http://litellm:4000 got ECONNREFUSED, which is where the openwebui model publishing died.
The tcp flavor had been masking the same malformed request all along, because its grep for HTTP/1 matches 400 Bad Request as happily as 200.
printf is a bash builtin like echo and the probe already requires bash for /dev/tcp, so nothing new is demanded of the image. The new test runs the generated request through bash and compares the bytes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make i18n-translate addressed the engine directly, so a run could never reach more than the 47 languages Argos ships models for, while meta/languages.yml names 183. service_url now resolves web-svc-translate, which routes a pair the engine does not serve to a chat model, and the deploy brings both roles up.
The gateway advertised only what its backends declared, and the chat backend declared nothing, so /languages reported 51 targets of 183 and the readiness wait spun to its hour-long timeout without translating a line. The chat backend now declares every language the table names.
Routing offered libretranslate for every pair, because an undeclared catalogue read as any pair, so a language it holds no model for was answered with HTTP 400 instead of falling through to the next backend. serves() now reads the backend's own /languages, which catalogue() already exposed for the advertisement. Declaring the pairs in the role was the first attempt and carried a second copy that was already stale: the engine reports zh-Hans where the role's list says zh.
A small model copies the prompt markers back, echoes the source line and labels its answer with the target language. All three reached the catalog, and an entry carrying its own source scores below the oracle's middle level. clean() strips them, and only from an answer that carried scaffolding, because on its own a short word and a colon is a translation, not a label.
server() had lost its redeploy parameter while cli/build/i18n kept passing it, so every run raised TypeError before translating a line and the only symptom was an empty log.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sibling container is reached at http://<entity name>:<internal port>,
and that string was written out per call site: a literal protocol, an
entity_name interpolation for the host, and a lookup('config', ...,
'services.<key>.ports.internal.http') for the port. Nine of them across
seven roles, each needing its own `# nocheck: literal-protocol-lookup`,
because the sibling lint rightly objects to a protocol glued in front of
a lookup - the protocol is supposed to come from wherever the TLS
posture is decided.
A container network has no TLS posture. It speaks plaintext by
construction, which is a property of the network rather than of any
caller, so stating it nine times stated it eight times too often.
lookup('container_url', application_id, service_key) holds all three
facts, and because no literal protocol is left at the call site the
suppression markers go with them.
The port is deliberately the internal one. A sibling that goes out to
the published port leaves the container network and comes back through
the ingress, which the dns-search suffix turns into the public FQDN.
The lookup refuses a service that declares no internal http port rather
than falling back to anything.
test_container_url_lookup.py keeps the shape from coming back: a
protocol, an interpolation mentioning entity_name, a colon, a second
interpolation. Nothing else matches that, and `# nocheck:
container-url-lookup` covers an origin this lookup genuinely cannot
express - a port that is not ports.internal.http, or a host that is not
the application's entity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/languages.yml carried `libretranslate: true|false` per entry, which
answered two different questions with one field: which languages the
repository translates, and which engine renders them. The first is the
table's business; the second is not. The gateway offers every backend
that serves a pair, falls through when one refuses or mangles the
answer, drops an engine whose failures pile up, and settles a request
class on the engine that wins its comparisons - and `signature()` is
"{source}:{target}:{length bucket}:{markup|plain}", so that learning is
already per target language. A static column beside it is a second
opinion that goes stale the moment the router learns something.
So the field is gone, 184 lines, and the table now holds name, endonym
and direction. Every language it names carries a catalog in every
domain.
What one engine can do is declared with that engine instead:
services.libretranslate-engine.serves, 47 codes. Not a detour - both
readers run before any engine answers. The lane prefetches one model per
code, and the readiness probe is itself the thing that would have asked.
`translatable()` split along that seam. Catalog-scoped callers take
`domain_languages()`, engine-scoped ones take the new
`served_languages()`: the CI matrix, extraction and translation the
first, the model prefetch and the tuning sweep the second. Collapsing
the two is what this change is guarding against - `baked_languages()`
renders identically before and after, verified by diffing both outputs.
`translate()` also stops dying on a language whose catalog has not been
extracted yet. It reads a missing catalog as nothing pending rather than
raising, because 136 docs catalogs do not exist until the next
extraction and refusing those languages would quietly shrink the set the
repository promises to translate.
The oracle suite follows: tests/oracle/test_translation_score.py asks
utils/i18n/judge.py about every language instead of routing a laya
checkpoint per language. It cannot run yet - the tools lane carries no
LiteLLM - and its skip message says exactly that.
test_catalogs_complete_on_gpu now measures against the table rather than
one engine's coverage. It is live on a GPU host and will report the 136
missing catalogs as the gap they are, which is the point: narrowing it
to the engine would hide exactly the languages worth watching.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The oracle suite needs a verdict in every language meta/languages.yml
names, and laya cannot give one. Measured against the suite's fixture,
its english checkpoint separates a faithful rendering from an unrelated
text in 17 of 47 languages and its multilingual checkpoint in none,
answering around 1.5 either way. One judge for all of them beats a route
per language that is noise in most of them.
The question keeps the shape the checkpoint answered: the levels are
written out and the model names an index over them, never a bare
probability. A probability carries this deployment's calibration, and a
fixed threshold on it would encode that calibration into the suite;
"below the middle level" instead means what the middle level says.
Three refusals are deliberate. temperature 0, so the same comparison
does not come out differently on a rerun. An answer naming no index
raises instead of scoring zero, which would read a refusal as "states
something else". An index outside the scale raises instead of being
clamped, because a model answering 7 has not placed the pair on this
scale at all.
The request goes to the model gateway rather than to a model server, so
it carries a consumer identity and resolves through the gateway's model
list like every other model call here.
Nothing imports this yet: the oracle suite moves to it once the tools
lane carries LiteLLM and a loaded model, which is what lets the judge be
exercised against a real one for the first time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.
The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.
call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two jobs of run 39040406916 died with the runner's own annotation, 'The
hosted runner lost communication with the server'. Both left every
diagnostic step pending, so no artifact exists for either. For the swarm
one the in-band sampler ruled starvation out — 9324M of 15947M available,
stall60 0.0%, 25 seconds before the cut — and nothing in the repository
can tell the remaining causes apart.
A stuck NFS mount wedges tasks in uninterruptible D state, which shows as
I/O pressure while memory pressure stays flat, and this drill runs its own
NFS node. /proc/pressure/io and /proc/pressure/cpu were read nowhere in
the repository. They are now sampled beside memory, with the D-state count,
every ten seconds instead of thirty, and an I/O stall at or above half the
minute aborts the step while the runner is still alive to upload the
banner.
cpu reads 'some', not 'full': the full line is pinned at 0.00 system-wide,
measured 13.50 against 0.00 on this host, so reading it like memory would
have shipped a metric that can never move.
Dump runner state was switched off for swarm, so the one post-mortem that
survives a step failure never ran there. It runs now, and the swarm drill
gets the step timeout its compose sibling has had all along — 350, the
number test_distro_budget_fits_timeout already derives the distro budget
from.
The sampler moved to its own module: the additions took matrix.py past the
file-size cap, and the lint asks for a split rather than a marker.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An unmounted VOLUME does not fail: docker creates an anonymous volume for
it, the container works, and the gap shows up later as a volume nobody can
attribute. sys-ctl-hlth-volumes fails the deploy over exactly that, and its
whitelist keys on volume ids, which are new on every deploy, so the gap
cannot be suppressed there either. e0b973f9bb fixed one such case in
web-svc-xmpp by hand; this is the check that would have found it.
Attribution is the whole difficulty. meta/services.yml declares an image
under a key, and the compose file often runs it under another name:
web-app-jitsi declares the image under 'jitsi' and renders the service as
'web', web-app-erpnext runs one image as seven services, and the two RDBMS
consumer keys name no service of their own because the container comes from
sys-svc-rdbms. The sweep therefore unions the entry key with its
proxy.service_key and delegates the RDBMS keys to their provider role. A
first version keyed on the entry alone and reported six roles that mount
every path correctly, web-svc-xmpp among them.
Reading a config blob section is now its own module beside the registry
client, because the two readers were the same walk twice and registry.py
sat three lines under the file-size cap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three faults, one mechanism each, that together let the oracle suite report
a verdict it never reached.
The address. lane_url handed out the runner's compose service name, but the
stack's containers resolve through the deployment's CoreDNS, which never
learns one: getent hosts tools answers nothing inside them, so every check
that asked the lane from there skipped itself. The bring-up now writes the
runner's address, picked as the one inside INFINITO_SUBNET because the
runner sits on every network it was ever brought up on, and the checks read
it from there. It is rewritten on both paths, because a restart gives the
runner a new one.
The readiness. serving judged the engine alone, so a lane whose engine make
i18n had already brought up counted as up while System One was down. It now
probes every role the caller names, which is why the runner republishes
System One's port: a container IP is unreachable from the host that decides
whether to deploy.
The verdict. A silent lane made the suite skip, and a skip exits green.
Only a deploy that is already running leaves the stack alone and exits 0
now; a crashed deploy and a deployed-but-silent role both exit 1.
The override's third :? guard is added to the gate that decides whether the
file is layered in at all, next to the one that was already missing. Without
it an .env written before these keys aborts every compose command, down to
compose-down, instead of dropping the lane. A test now keeps that list and
the override's guards in step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every cold render of the applications SPOT walks the templates of all
269 roles, and those templates call lookup('config'), lookup('tls'),
lookup('domain') and others. Each of these re-enters
get_merged_applications while the render guard is set, and each one
rebuilt the unrendered view from scratch: a deepcopy of all role
defaults plus the override merge, about 8 ms, never cached. A local
cold render of the development inventory made 351 such calls, 2.82 s
of a 6.86 s render.
The outer render now keeps a copy of its unrendered view and its cache
key on the render guard, taken before the inventory volumes are
popped, so it is the same view a nested call used to build. A nested
call with the same key returns it; any other key still builds its own.
The guard drops it when the render ends.
Measured on the same inventory: the nested calls took 0.07 s instead
of 2.82 s and the cold render 3.60 s instead of 6.86 s. The rendered
SPOT is unchanged: the old and new dumps differ in 69 user passwords,
which the development inventory regenerates on every read (two runs of
the old code differ in 68), and in one absolute checkout path, the only
other difference.
In CI each cold render took 40 to 77 s and the heaviest swarm cell of
#950 missed its budget by a few minutes (#984).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
4aba897ca2 gave the pull and build subprocess a 3600s ceiling so a hang
could not stall a deploy. A pull of a large image set legitimately outlives
any fixed ceiling while it is still making progress, and the ceiling then
kills a healthy run.
run_streamed measures the gap between two output lines instead. Two reader
threads keep stdout and stderr separated the way run_cmd does, because
compose writes its warnings to stderr and merging them poisons every parser
downstream. Only the two calls that legitimately run long use it; the short
calls whose output is parsed keep run_cmd.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils/yaml_bootstrap.py promises that load_block(text) either equals
yaml.safe_load(text) or raises BootstrapYamlError. A fuzzer comparing
both found two inputs that broke the promise:
- str.splitlines() also breaks lines at \x0b, \x0c and \x1c to \x1e,
which PyYAML's reader rejects as non-printable. "\x1e/: R" came back
as {"/": "R"}, and a non-printable character inside a comment line was
skipped with the comment.
- An indented " ---" on line 1 was taken as the document marker, while
PyYAML reads it as a plain scalar and rejects the mapping that follows.
load_block now refuses any character outside PyYAML's own printable
class before it splits lines, and takes "---" as the document marker
only at column 0. The line breaks splitlines() still honours (\r, \r\n,
NEL, LS, PS) are line breaks for PyYAML too, so splitting stays as is.
The eight inputs join STRUCTURES in the divergence test; against the
previous reader, four of them diverge and fail it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The full gate failed test_a_distro_needs_the_slowest_run_plus_headroom_to_start on a quiet machine, so the load is the gate own parallel workers rather than anything beside it. The branch under test needs d < B < 2.15*d for a measured first duration d, the rule 040955ebf2 wrote down when it widened the same fixture from sleep 2 / 3s to sleep 4 / 7s. At 4 / 7 the window admits a first iteration of 4 to 6 seconds, three seconds of slack, sized then against the single second CI had overshot by. Under the gate it overshoots further, takes the budget exhausted branch and never prints the line the test greps for. At 10 / 19 the window admits 10 to 18 seconds, nine seconds of slack. The assertion and the branch are unchanged and the suite pays six more seconds.
The second hunk shortens the Odoo reload task name to 60 characters; at 83 it broke the 80 character convention the name lint enforces.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge left the role with two ways to the same end. The wired shell path called setup.py update-user, so it could neither create a missing account nor clear a lockout, and it decided its own idempotency from a sha256 marker in the persistent volume, which lies after a restore: marker matching, database not, reported unchanged and did nothing. The python converger from 0709de6fa1 queries the application instead, creates what is missing, verifies the password, unlocks, resets the failed-login counter and ensures the Administrator role, and it carried the only tests either path has; it was wired to nothing after the merge resolved towards manager_ops.yml. That engine now serves the surviving wiring: renamed into the family spelling the sibling roles use, reading the env names the task already passes and emitting the markers the task already parses. tasks/administrator.yml had no caller anywhere in the tree and goes with it.
Trade-off worth knowing: the deleted path used the supported pgAdmin CLI while this one uses internal APIs, so a pgAdmin upgrade can move it. Neither path has ever executed in CI, the pgadmin job died at the health wait before reaching the realign, so the choice rests on capability and tests rather than on field evidence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
