A deploy job name repeats the category path on every row of the matrix. One Egyptian hieroglyph costs four UTF-8 bytes where the two Chinese characters it replaces cost six. Unicode names each sign by its Gardiner code, so the opaque value stays identifiable without a comment per entry.
A label has to stay one codepoint: a sign written with a variation selector costs seven bytes and is worse than what it replaced. tests/integration/meta/categories/test_glyph.py enforces that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
27d59f4bd4 caught a non-Latin target answered in Latin letters. Its
mirror was missing, so the model answering a language it cannot write
with one it can passed every criterion:
ch 'AI Services' -> '人工智能服务' Chamorro, answered in Han
tr 'Sales' -> 'إدارة'
ae (76 entries) -> Arabic, Persian, Hebrew, Chinese
Chamorro writes in the Latin script and lives on Guam; the code 'ch'
reads as Chinese to a model that does not know the language, which is
how the whole catalog came back in Mandarin.
Measured over the 121 Latin-script catalogs before the rule was
written: 79 entries condemned, 76 of them in ae and 3 in tr. All three
Turkish ones were read individually and none is a false positive; two
are an Arabic fragment with a markdown rule and one is English plus
Bengali repeated four times.
Its own criterion rather than an arm of untranslated(): the catalogs
record the reason on the entry, and 'answered in another script' is a
different diagnosis from 'came back untranslated'. The two together
cover the script dimension for all 184 languages.
What it still cannot see: Spanish served as Corsican. Telling two
Latin-script languages apart needs a detector this repository does not
have, so that class stays open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway had two engine kinds and neither reaches the catalogue.
LibreTranslate covers roughly 48 languages. The chat engine is backed
by the single entry in AI_LOCAL_MODELS, llama3.2:1b, which officially
speaks eight, while web-svc-translate/vars/main.yml declares every
language in meta/languages.yml as a chat target. The gateway therefore
asserted 183-language capability for a model that invents the rest:
Belarusian came back as Polish in 86 of 87 new entries, Abkhaz had no
Cyrillic in any of 83, Amharic 10 of 236 in Ethiopic, and Avaric,
Assamese and Bashkir returned the source verbatim.
NLLB-200 is built for translation rather than generation. A third
engine kind serves it beside the two libretranslate instances, since
it is seq2seq and does not belong as an ollama alias.
The honest ceiling is 136 of 184, not 184. flores.py maps 136 ISO
codes and names the 48 the model has no code for, and the test asserts
mapped plus unserved equals exactly the keys of meta/languages.yml so
neither side can drift. Belarusian, Amharic, Assamese and Bashkir are
covered; Abkhaz and Avaric are not, so two of the worst catalogues
still have no local engine.
The script half of each mapping is derived rather than typed: the test
re-derives every endonym's script from unicodedata and asserts it
matches the suffix of the FLORES code, ja and zh excepted because Han
cannot distinguish Jpan from Hans from Hant. The subtag half cannot be
derived, because where ISO 639-1 names a macrolanguage NLLB ships one
variety (ar->arb_Arab, fa->pes_Arab, ms->zsm_Latn and ten more), so it
is stated once in one table.
Counting languages surfaced a real bug: meta/languages.yml has 184
entries, not 183. 'no' is quoted because YAML reads bare no as false.
Inert until deployed. The engine entry is multiplied by zero while
TRANSLATE_NLLB_ENABLED is false, so TRANSLATE_ENGINES is unchanged;
the two new vars are evaluated unconditionally and every key they read
exists in the new role. No variant round was added, so CI does not
exercise this engine yet, and a first deploy needs an inventory
regeneration for the new service key, volume and subnet.
Assumed facebook/nllb-200-distilled-600M, about 3 GB resident, which
fits the envelope svc-ai-s1 already occupies. 1.3B buys roughly
+2 chrF++ concentrated in low-resource languages for twice the memory;
the README tables the sizes so that choice is visible.
Owed before this is offered beyond private use: the weights are
CC BY-NC 4.0, and the role does not yet say so or credit the NLLB
team. Tracked with the per-role licence index.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A swarm job lost every published port of the mail stack without anything
reporting it. dockerd aborted the whole ingress program:
Failed to add ingress: failed to program ingress ports:
failed to bind host port 0.0.0.0:995/tcp: address already in use
/proc/net/tcp named the squatter: 10.100.1.1:995 -> 10.100.1.11:2049,
the node's own NFS client. The mount carries no noresvport, so the
kernel drew a source port from the 665-1023 reserved window and landed
on pop3s. Afterwards none of 25/465/993/995/4190/8095 was bound while a
sibling service's port was, and stalwart still reported 1/1 healthy
because its healthcheck runs inside the container. The e2e CLI probe
found it two hours later as ECONNREFUSED on SMTP 25.
993 and 995 are the only published ports repository-wide inside that
window, so this is luck of the draw on any NFS-backed swarm host that
runs the mail role.
insecure on the kernel export is not separate hygiene, it is the other
half: kernel nfsd defaults to secure and answers a mount from an
unprivileged source port with EACCES, so noresvport without it would
break every kernel-flavor mount. It reaches the kernel flavor only,
since templates/exports.j2 is reached from tasks/flavor/kernel/main.yml
alone. The ganesha flavor needs nothing: ganesha.conf.j2 sets no
Privileged_Port and the default already accepts unprivileged clients.
What it costs: the next deploy of every existing NFS client sees a
pending opts change and remounts the shared state. The role already
anticipates that, releasing DIR_BACKUPS first when the check-mode probe
reports a change. Nothing bind-mounts that path into a container, so
the exposure is a host-side job holding it at that instant, which fails
loudly rather than silently.
Not yet proven in effect: a source port is chosen when the RPC
connection is established, so a plain remount can leave a running mount
on its old reserved port until the next real mount. Reading
/proc/net/tcp after the first deploy on an existing client is what
confirms it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
coreutils timeout without --foreground calls setpgid, so the command it
wraps leaves the caller's process group. A wrapped command that then
reads the controlling terminal takes SIGTTIN and stops forever instead
of being killed at the deadline, which inverts the purpose of the
wrapper.
That is the cause behind d31b49e30d: container exec wraps docker in
this prefix, an ansible.builtin.script task gives the remote shell a
pty, and docker exec -i stops on its first read. Three jobs of run
39186571513 idled 1h37m to 2h22m before the outer budget killed them.
The scripts there close their own stdin; this closes the class.
utils/diagnostics/container.py gets the flag too. docker events reads
no stdin and was never at risk, but leaving one non-compliant
construction would have forced an exemption on the new rule's first
day.
The rule itself cannot be a text search. The prefix comes from a
function and the -i from the caller's argv, so timeout and the command
that reads stdin never meet in one line of source: a grep over the
tree scores zero hits both before and after the defect. The lint reads
the parsed tree instead and requires --foreground in any list literal
that starts with timeout. Checked against the pre-fix line, it reports
it.
Cost, deliberately accepted: --foreground signals the direct child
only, not a process tree. For docker exec the direct child is the CLI,
which is what has to die to release the task, but a wrapped command
that spawns children now leaves them where the group kill would have
taken them. covers the case where that
trade goes the other way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baudolo 7.1.x writes volume generations with `rsync -aHAX`, so the
producing host keeps `user.*` extended attributes (OpenCloud keeps its
storage metadata in `user.oc.*`). Every later copy dropped them again,
because none of the other rsync calls passed `-X` (nor `-A`/`-H`):
- NFS export snapshots (svc-bkp-nfs-2-local `script.sh`),
- the backup host's pull (svc-bkp-remote-2-local `pull_specific_host.py`),
- the device sync (svc-bkp-local-2-device `script.py`),
- every recover into a live target (`utils/recovery/base.py`, used by the
volume, NFS and device recover scripts),
- the `recover full` and `recover remote` tree pulls.
A disaster recovery through the documented chain therefore restored
OpenCloud without `user.oc.*`, silently: the copies succeeded and the
dirval stamp does not cover attributes.
All of these now run `rsync -aHAX` (`-aHAXP` for the device sync). A
target filesystem without xattr support makes rsync exit non-zero as soon
as a source file carries attributes, so the hop fails loudly instead of
dropping them; there is no per-device opt-out. The single-image pull in
`recover remote` keeps `-a`. The provider's ssh-wrapper needs no change:
its glob accepts any server flags and its character guard allows them.
Two tests write a `user.oc.id` attribute and assert it survives the NFS
snapshot and the recover; with the old `-a` both fail with
`OSError: [Errno 61] No data available`.
Closes#1021
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Moodle#2 of run 39186571513 failed the health check 26 times with
academy.main.infinito.test: Unable to reach https://academy.main.infinito.test/
(Navigation timeout of 20000 ms exceeded)
while every other vhost on the stack printed 'No CSP or network blocks
detected' and nginx answered 200 or 303 to every request. Nothing was
blocked; the page is slow. Moodle #0, #1 and #3 pass, and only the arm64
swarm row fails.
The role already claimed to own that budget. HEALTH_CSP_TIMEOUT_MS sat in
defaults with a comment calling it the clearnet navigation budget, but
script.py ran the clearnet batch as (clearnet_domains, '', 0), so clearnet
silently inherited the checker's own 20000 and the knob did nothing.
Only the onion batch ever received a real --timeout.
--clearnet-timeout closes that, defaulting to 0 so a call without the flag
behaves exactly as before. The value lives in its own variable rather than
raising the shared base, which would have multiplied through
HEALTH_CSP_ONION_TIMEOUT_MS and taken the public onion budget from 100s to
225s, toward the 3600s TimeoutStartSec cap. Onion budgets are unchanged.
45000 is derived, not picked: the slowest measured navigation on that
target reached domcontentloaded after 26s, with single requests stalling
up to 5.8s, so 26 + 3*5.8 = 43.4s, rounded. That is 1.7x the worst
navigation observed there.
TimeoutStartSec is deliberately NOT derived from the new budget. The
per-domain rate sizes the whole unit and nearly every domain answers in
well under a second; using the worst case as the scheduling rate would
have doubled the clearnet term and pushed large stacks toward the cap for
no measured reason. Scheduling is unchanged.
Known cost: above 20s the check no longer notices that an app got slow.
That coverage was incidental, not declared, and a page-load regression
needs its own assertion rather than a CSP probe.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
untranslated() is the one criterion that knows a catalog's script, and it
never fired. It returned False below LATIN_FLOOR = 6 words, while the
catalogs are mostly one- to four-word UI labels, so the guard sat above
its own corpus.
Measured across the 62 non-Latin catalogs: the floor condemns 0 entries.
Without it, 536, concentrated in the languages the model cannot write:
av 99 as 92 ba 87 be 84 ab 84 am 56 bh 13
Every entry in the low-count tail was read before the floor was removed,
because that tail is where a false positive would live. There is none:
ko 'Applications' -> a refusal sentence from the model
zh 'Updates & Package Mgmt' -> the string plus a version changelog
fa 'Cloud' -> 'Cloud' and a markdown rule
ab 'Communication' -> 'Comunicacion'
A word count was the wrong instrument. reason() consults this only after
an exact echo is ruled out, so for a non-Latin target, Latin text that
DIFFERS from the source cannot be a term the language keeps in English.
That distinction is what an earlier attempt at floor=2 lacked, which is
why it condemned 4053 entries including legitimate ones.
Scope, stated because the numbers invite a larger reading: this guards
the 62 non-Latin catalogs only. For the ~122 Latin-script languages a
wrong-language answer still passes, and catching those needs a detector
this repository does not have.
LATIN_FLOOR and its re-export are deleted rather than left at 1; nothing
else consumed them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When the IPAM config of a compose project's network changes (#999 added
`ip_range`), compose recreates the network on the next `up`. With a
foreign endpoint attached, as every worktree stack is on the primary's
network, compose first stops the project's own containers, then fails
with `error while removing network: ... has active endpoints`, and a
repeated `up` or `up --force-recreate` does not recover. On 2026-10-19
this took the shared registry/package cache and coredns down for every
local session.
`make compose-network-recreate` (alias `i8conere`) runs the sequence that
recovered it: detach every running container of another project from the
network, `docker compose down --remove-orphans` without `-v` so the
caches keep their volumes, the normal `up`, then attach the detached
containers again, which draw addresses from the dynamic pool. If `up`
fails, the still-detached containers are listed instead of reattached.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`security-apparmor-teardown` ran `aa-teardown`, which unloads every
profile including dockerd's in-memory `docker-default`. dockerd only loads
that profile when it starts, so every container start afterwards that
requests it fails until dockerd restarts. Since #1067 runs the teardown on
the GitHub runners, BuildKit failed each RUN of the swarm node image three
times (run 39332343490, 39 of 40 swarm jobs):
runc run failed: unable to start container process: error during
container init: unable to apply apparmor profile: apparmor failed to
apply profile: write .../attr/apparmor/exec: no such file or directory
The teardown now removes each loaded profile by name through securityfs
and skips `docker-default` and child profiles, which leave with their
parent. Dev hosts gain the same: no dockerd restart is needed after
`make environment-bootstrap`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workflows pin `runs-on: ubuntu-NN.NN`, so no job moves to a new Ubuntu
release behind our back, but nothing raised the pin either. A sixth job of
`🔄 Update versions` closes that: `utils/update/runner.py` reads the image
table of actions/runner-images, takes the newest x64 Ubuntu label whose
image carries no preview or beta badge, and raises every older bare
`runs-on: ubuntu-NN.NN` to it; arch-suffixed labels stay untouched. The job
opens the change as one PR over `.github/workflows`, so its CI run tests
the new image before it reaches main.
This commit carries the updater's first result: every job now runs on
ubuntu-26.04 (49 `runs-on`).
actionlint 1.7.12, the newest release, does not know `ubuntu-26.04` yet
(rhysd/actionlint#682, fix proposed in rhysd/actionlint#743), so
`.github/actionlint.yaml` declares the label until a release knows it.
act maps each `ubuntu-NN.NN` label the workflows use to the local platform
image, derived from `.github/workflows`, instead of a fixed list that did
not know ubuntu-26.04.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
reason() calls itself the single source of truth for both ends of the
pipeline, and the write path honoured that: client.py:228 passes the
directory code into unmask(). Only damaged() read catalog.locale, so
for 50 catalogs the two ends judged against different languages.
44 catalogs carry no locale header, 8 of them non-Latin and therefore
silently past the script check. 6 more name a code their directory
disagrees with, so tl was judged as fil, bh as bho, tw as ak.
The language becomes a required parameter; both production callers
already held the code. Measured over every catalog, header language
against path language, one moves: core/av from 0 to 29. Those 29 hold
pseudo-Norwegian and a degeneration loop in a Cyrillic catalog, and
make i18n-prune has since cleared them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cfe20f0726 and 4915a401d4 wrapped three statements more cautiously than
the formatter wants. Without this the lint phase rewrites the same diff
into the tree on every run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
failed_selections reproduced mode, onion state, distro and architecture
but not the mesh axis. These tokens feed the priority line, where an
unpinned axis offers both states, so a failed direct swarm row came back
as itself and as a meshed sibling: two jobs to reproduce one.
Only a mode the axis covers takes the pin. pinned_vpn_states returns an
empty list for a compose row pinned to the mesh, and an empty list drops
the row from the matrix -- the class ff75343aba already had to fix once
for held runs.
The fixture had drifted from what the matrix emits: row_label built
titles with no mesh glyph while axes.assign writes one for every swarm
and compose row, so parse_label fell back to its False default and the
new &direct assertion would have been right by accident. row_label and
deploy_job_name now place the glyph under the same condition as
axes.assign, and a meshed row has a case of its own, which nothing
covered before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The main merge left two step watchers: run.run_watched from main and the
newer watchdog.run_watched from here. Their disk and memory floors are
byte-identical (6144M, 6% of RAM, abort 75), so run.py now keeps only
run_step and takes the watcher from watchdog. mesh.py reaches it through
run_step and matrix.py already imported it directly.
That alone would have been a regression. The watchdog aborts on io60
alone, which its own docstring says cannot separate a wedged mount from
a busy disk, and the six mesh steps had no I/O kill before: a healthy
sleep on a host with 126G free and no task in D state died at rc=75
after 10s where the old watcher let it finish. The first sample also
lands 10s in, so 50 of the 60 seconds it averages belong to whatever ran
before -- an image pull could abort its successor before that step read
a byte.
The ceiling now takes three things together: a task in D state on the
same sample, WEDGE_SAMPLES breaches in a row, and a step old enough to
own a whole STALL_WINDOW_S of the average. Both refuted scenarios are
regression tests. A real wedge is caught about a minute later than
before; the disk floor that covers ENOSPC is an instantaneous read and
is untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Our side added the architecture axis and main the vpn axis; combining
both put axes.py at 502 lines against the 500 the structure lint allows.
check_pins and row_architectures answer one question: may this row take
what the selection token demands of it. Assigning the axes of a row that
carries no pin is the other job, so the two separate cleanly and axes.py
drops to 397 lines.
axes.__all__ still exports both names, so cli/meta/ci/validate.py keeps
reaching them through axes. The test patches get_role_architectures,
which moved with them, so its patch target moves too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
e17d182d0a gave web-app-docs a switch that turns the language assertion
off on a row where docs only rides along, and read it from the
guide_host environment variable. The deploy workflow sets that variable
on the runner, but a compose deploy runs Ansible inside the infinito
container, where only _FORWARDED_ENV_KEYS arrives. guide_host was not in
it, so the lookup resolved empty, the comparison against the row's own
id was true everywhere, and the switch never fired.
Run 39137606155 shows both halves: the passenger rows 117711219652 and
117711220306 carry guide_host: web-app-docs and each spent about 7000 s
of a 20400 s budget in that one assertion, while the docs row itself,
117711219660, carries an empty guide_host and therefore keeps running
it. Forwarding the variable preserves that split rather than silencing
the assertion everywhere.
APP_ID is in the tuple for the same reason and feeds the guide_role
lookup from the same workflow step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The design and docs commits landed with --no-verify, so the hook that
runs make test never saw them and four repository lints went red on
code that was already pushed.
test_init_files wants an __init__.py in every python tree under tests/;
the sys-front-inj-design directory arrived without one.
ruff S701 rejects Environment(autoescape=False). Both call sites render
from a string, where select_autoescape(default_for_string=False)
returns the same False, so what the test exercises is unchanged and the
explicit escape filter keeps doing its own work.
test_no_env_defaults_in_javascript rejects a default for an environment
value in JavaScript, because it is a second source that hides an unfed
variable. Normalising to the empty string stays allowed and keeps the
expression's result identical for every input.
The env-file schema caps a comment at 140 characters.
The two lint/filesystem files carry nothing but the autoformat reflow.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The instruction replay provisions a fresh inventory inside a throwaway
target, so its compose pull fetched every application image from the
upstream registry while every other CI deploy pulls through the GHCR
mirror. On the moodle row of run 39082924588, 512 s of the 1646 s
ansible run went into that pull.
mirror.sh rewrites the extracted block and not the README template the
site publishes: it inserts the mirrors.yml resolver ahead of the
provision call and passes --mirror. A reader keeps the upstream
references, because a reader has no GHCR mirror to pull from, and the
replay exists to prove the instructions a reader copies. Outside CI the
two GitHub variables are empty and the block stays byte-identical.
The prefix is read from default.env in the checkout, the way machine.sh
already reads INFINITO_PARENT_IMAGE, so no Ansible template has to look
up an INFINITO_ key that belongs in an inventory.
A block that carries no provision call fails the harness instead of
quietly falling back to the upstream registry, and test_mirror.py
renders both published shapes to keep the two sed patterns matched to
what the template emits.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The inline design script carries a CSP sha256 computed at render time. On an
onion request body_filter.lua rewrites the clearnet CDN base inside the whole
head payload, the script's bytes stop matching their hash, and every role
whose script held a CDN URL has its branding refused wherever script-src-elem
has no 'unsafe-inline'. web-svc-logout and web-svc-html are on nearly every
deployment, so this was not a corner case.
The config now rides in the script tag's data-design attribute, which CSP
does not hash and onionize may rewrite freely, and the first statement of the
script reads it back into INFINITO_DESIGN. The executable body is identical on
both vhosts, so one hash authorizes both.
All 35 design scripts now read the config instead of interpolating
design.urls. The 13 that still needed converting keep their render-time
guards, so a role with branding disabled still emits no script at all, exactly
as before. slots left the config object: no script has ever read it.
A new check refuses a branding URL in a script body, so the class cannot
return, and a unit test drives the config through to_json, Jinja's escape and
an HTML parser with a hostile title to prove the attribute round trip.
Not yet proven: that the hash matches on a real onion vhost. This stack runs
without tor and every role deployed here has 'unsafe-inline', so the check
needs a tor-enabled deploy of one of the seven roles that disable logout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A meta/volumes.yml entry that declares backup: false without pinning a
name: had its semantic key emitted into --volumes-no-backup-required.
Docker names such a volume <project>_<key>, and baudolo compares the string
exactly, so the exclusion matched nothing and the volume was copied anyway.
web-app-docs hit it in CI: docs_sites was emitted, the live volume is
docs_docs_sites, and the 17 GB regenerable sphinx tree was backed up into
the 30 GB container root. That filled it and took four jobs of run
39137606155 down with three different symptoms: no space left during the
copy, rsync reading the tree while the builder still wrote to it, and
hlth-disc-space then refusing to start on a full filesystem.
deployed_volume_name() now derives the name the way compose does: the
pinned name: when there is one, otherwise the role's entity name, which is
the compose project, joined to the key. The docstring claimed the old
behaviour was correct and is fixed too.
Three roles were silently still backed up and are now excluded as declared:
web-app-docs, web-app-jellyfin and web-svc-api. The nine entries that pin a
name are unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sys-front-inj-design now renders templates/css/app.css.j2 ahead of the
role's stylesheet and templates/js/design.js.j2 ahead of its script. The
branding URLs, the slot dimensions and the logo guard live there instead of
being repeated per role: the guard that stood in 41 sheets is one condition
in the composing template, and a sheet reads branding through
--design-url-* / --design-slot-* custom properties.
29 stylesheets and 43 design scripts lose their Jinja and move to static
files. Roles whose sheet carries app-specific URLs declare them as
design_css_urls and read them as --design-url-app-*.
The script prelude is prepended only when the role script references it,
and it carries urls only while a logo is configured, so every role that
needs neither is emitted byte for byte as before.
Five checks keep the mechanism honest: a composed sheet may only read
design custom properties that app.css.j2 declares for its role, a template
may not carry Jinja expression text outside its delimiters, a design script
may not quote its config object, and the injected-script parse check now
covers the design channel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two bots bumped the one baudolo version independently. The pip pin
updater rewrites the literal package_name pin in
roles/svc-bkp-volume-2-local/tasks/01_install.yml and opens a PR scoped
to roles/<entry>/, and dependabot bumps the same package in
pyproject.toml. Each PR alone splits the pins that
tests/lint/repository/dependencies/test_baudolo_pin.py holds equal; #995
merged such a split and turned main's test-lint red until #996, and the
next upstream release (7.1.2 is tagged) would repeat it.
The version lookup now takes an optional distribution name and returns
its exact == pin from [project.dependencies]. The install task asks it
for backup-docker-to-local, so pyproject.toml is the only place the
version lives, dependabot is the only bot that moves it, and the pip
updater no longer finds a literal to bump. The lint now forbids a
literal pin in the task and requires the lookup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
compose/cache.shared.override.yml attaches every worktree stack's
infinito service to the primary's network (infinito-nexus-core_default,
172.30.0.0/24) without an ipv4_address. The default network declared
no ip_range, so Docker IPAM handed out the lowest free address and the
worktree stacks took 172.30.0.2, .3, .8, .9 and, whenever the primary
container was down, .10, the primary's INFINITO_IP4. The primary's
`make compose-up` then failed with "Address already in use", and with
it `make quality-high` on main.
The default network now declares ip_range: ${INFINITO_IP_RANGE}, the
upper half of INFINITO_SUBNET. Every static address of the stack (.1,
.4 to .6, .10, .53) sits in the lower half, so a container without a
static address can no longer take one.
INFINITO_IP_RANGE is derived from the final INFINITO_SUBNET after
custom.env is applied, because a worktree pins its subnet there and
`make dotenv-force` regenerates without it in the environment. Existing
worktrees therefore get their own slot's range without a new pin.
A lint checks that every static IP in default.env and every
172.30.${slot}.N pin in scripts/system/worktree/up.sh lies below the
pool.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
make quality-high reformatted three expressions this branch added today: the chat-target chain in the translate role's vars, an assertEqual in the gateway engine tests, and the wanted comprehension in the lane probe. No logic and no rendered value changes; the YAML break folds to a space inside a double-quoted scalar.
Kept separate from the changes that caused it, because a reflow mixed into a behaviour commit hides the behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every catalog got a lane derived from the CPU count, so twenty requests met one model instance. That is not throughput, it is self-inflicted saturation: a single catalog recorded 425 translated, 9 discarded and 40 requests refused with HTTP 503, and a discarded entry is simply left pending.
The work is now split by the backend that will answer it. The languages the engine serves keep the wide pool, because Argos answers in about 0.15s; the rest run after them with two lanes, one to occupy the model and one to refill it while the first writes back. Each pass builds its own client, so the chat pass does not inherit the worker count the engine pass was sized with.
Order is the larger half of this. The table is alphabetical, which is why the last run spent its first 45 minutes on af, a language with no Argos model, while fr and de sat untouched. Cost order delivers the engine languages first. For domain=docs that is 47 catalogs of 3963 entries each in roughly an hour; for domain=core it changes nothing today, because those catalogs are already translated.
The test picks aa against fr on purpose: both orders agree on every pair the fixtures held before, so a test built from them would have passed against the old single pass as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In swarm, docker stack deploy ignores depends_on, so the precheck
service and every init replica ran search_allocation_precheck.sh at the
same time. They shared one probe index: one invocation's PUT could fail
the shards_acknowledged check because another had created the index
first, and either one's cleanup could delete the other's probe. The
init service carries no replica override, so compose_replicas gives it
one task per host in the application group and the collision also
happens between init replicas.
A failing precheck task also never reached has_fatal_task in
stack_ready.sh: with restart_policy none it records a single error,
while has_fatal_task needs three. stack_ready.sh then kept answering
"not converged" until the handler ran out of retries. And a Running
init task counts as converged through running_current_spec, so the
swarm deploy never depended on init failing; the role's "wait for
Zammad init" task is what stops it.
The probe index now carries the container's hostname, which docker
sets to the container id, so concurrent invocations no longer share an
index. compose.yml.j2 renders the precheck service, and init's
service_completed_successfully dependency on it, only outside swarm.
Compose keeps its fast stop on a refusal; in swarm, init runs the same
check in front of zammad-init, a refusal leaves Zammad unseeded, the
wait for Zammad init fails the deploy, and the init service logs carry
the allocation explanation.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On a data disk with less free space than the high disk watermark asks
for, Elasticsearch creates indices but assigns none of their primary
shards. `zammad-init` then reports every search index update as failed
and never exits, so `compose up` waits on
`service_completed_successfully` until its 1800 s task timeout and the
deploy fails without naming the cause (#850). Elasticsearch's default
high watermark is an absolute 150 GB headroom on a large disk.
A new one-shot `precheck` service runs `search_allocation_precheck.sh`
with `restart: "no"`, and `init` depends on it with
`service_completed_successfully`:
- it waits a bounded time for `_cluster/health`;
- it creates a one-shard, zero-replica probe index with
`wait_for_active_shards=1`;
- once Elasticsearch allocates it, it deletes the probe and exits 0;
- otherwise it prints the creation answer and the answer of
`_cluster/allocation/explain` and exits 1.
The check sits in its own service because `init` keeps
`restart: on-failure`: wrapped into `init`, the failing check was
restarted seven times within eight minutes on a local deploy while
`compose up` kept waiting. Inside the stack, a `restart: "no"`
dependency that exits 1 made `compose up` fail after one second with
`service "pre" didn't complete successfully: exit 1`.
Every curl call carries --connect-timeout and --max-time, so a hanging
request cannot outlast the bounded wait. The image's entrypoint passes
any non-`zammad-*` command through `exec "$@"` (bin/docker-entrypoint of
7.1.2), and the image ships `curl`. The watermark values themselves come
with feature/store-i18n (b82591ba9d); this change covers inventories
without a variant cap.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Both new tests read a file their own run has just written, which the uncached-read lint flags because utils.cache.files.read_text is an lru_cache with no mtime invalidation. The two cases want different answers. The rendered inventory is YAML, so load_yaml_any reads and parses it the way the lint asks and needs no exception at all. The healthcheck probe writes its request through bash inside the test, so a cached read would compare against the previous run's bytes and the assertion would pass without proving anything; that one keeps a cache-read directive naming why.
make test-lint went from 2 failed to green, and the first attempt is the reason the inventory test carries no directive: suppressing the read there only moved the complaint to yaml.safe_load, which the right reader answers in one call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The oracle asks a chat model whether a translation states its source. llama3.2:1b cannot answer that: on the suite's own fixture it scored an unrelated text at the middle level instead of the bottom one, and asked as a yes-or-no question it called two identical texts different. With it the suite reached no verdict at all, reporting that the judge read none of the 47 languages the catalogs hold machine translations for. Qwen2.5-3B answers both correctly, and the same run then judged 248 samples.
The pin lives in the lane inventory rather than in AI_LOCAL_MODELS, because a judge that can separate a faithful rendering from an unrelated text is a demand no deployment makes, and the default should not grow a gigabyte to satisfy a check. The litellm config builds its routes from the merged application config, so a lane override reaches the model list and its window without touching the repository default.
LANE_MODEL_ALIAS repeats the one name in utils/inventory/tools.py, because ensure() asks whether the lane already serves before the deploy that writes the rendered inventory, so the probe cannot read it from there. A lane serving one model while the probe expects another never reports ready and redeploys on every call, so a test renders the template and holds the two names equal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lane ran litellm but never republished its port, so a host-side probe hit the compose proxy on the same number and read 502. The lane could therefore never report ready, and the oracle suite skipped itself to a green exit instead of asking a judge. A handler reads the port from the role the way translate.py already does, the runner forwards it, and TOOLS_LANE_KEYS takes it up: compose_file_args layers the override in without checking its variables, so an .env written before the key existed would abort every compose command rather than merely drop the lane.
The gateway's master key is pinned in the lane inventory for the reason the System One key already is: the provisioner would otherwise generate it into a vault the checks cannot read back, and the oracle judge is not a deployed consumer with a virtual key of its own.
serving() stops treating reachability as readiness. A lane brought up before AI_LOCAL_MODELS named a different model keeps serving the old one, and a gateway holding a key these checks cannot use looks healthy from outside while refusing every call they make. It now asks /models with the bearer token and requires the configured alias, and requires the gateway to offer every language the catalogs name, because that advertisement is what the translation run waits on. Without this the lane waved itself through and no change to its roles ever shipped.
Two lane-scoped overrides come with it. The ollama role caps itself at four CPUs, which left the host at 361 percent while twenty cores idled, so the lane gives it the host count the way the engine entry above it already does. And the lane pins the router to its declared order: the decider may send a pair the engine serves to the chat model, which a batch over 183 catalogs cannot pay for. Deployments keep the decider.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generated tcp and http_status probes built their request with echo -e, and echo appends a newline, so the bytes on the wire ended with a stray bare LF after the header terminator of a Connection: close request. uvicorn's h11 reader treats that as trailing data and answers 400 before the handler's 200 reaches the socket, so the probe's grep for a 2xx or 3xx status line never matched and the container stayed unhealthy for good.
In compose the name resolves straight to the container, so an unhealthy but running service still served and only the backup drill noticed, failing with litellm is running/unhealthy after 1200s. In swarm a task joins the service VIP only once it is healthy, so the litellm service sat at 0/1 with no backend and every consumer of http://litellm:4000 got ECONNREFUSED, which is where the openwebui model publishing died.
The tcp flavor had been masking the same malformed request all along, because its grep for HTTP/1 matches 400 Bad Request as happily as 200.
printf is a bash builtin like echo and the probe already requires bash for /dev/tcp, so nothing new is demanded of the image. The new test runs the generated request through bash and compares the bytes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make i18n-translate addressed the engine directly, so a run could never reach more than the 47 languages Argos ships models for, while meta/languages.yml names 183. service_url now resolves web-svc-translate, which routes a pair the engine does not serve to a chat model, and the deploy brings both roles up.
The gateway advertised only what its backends declared, and the chat backend declared nothing, so /languages reported 51 targets of 183 and the readiness wait spun to its hour-long timeout without translating a line. The chat backend now declares every language the table names.
Routing offered libretranslate for every pair, because an undeclared catalogue read as any pair, so a language it holds no model for was answered with HTTP 400 instead of falling through to the next backend. serves() now reads the backend's own /languages, which catalogue() already exposed for the advertisement. Declaring the pairs in the role was the first attempt and carried a second copy that was already stale: the engine reports zh-Hans where the role's list says zh.
A small model copies the prompt markers back, echoes the source line and labels its answer with the target language. All three reached the catalog, and an entry carrying its own source scores below the oracle's middle level. clean() strips them, and only from an answer that carried scaffolding, because on its own a short word and a colon is a translation, not a label.
server() had lost its redeploy parameter while cli/build/i18n kept passing it, so every run raised TypeError before translating a line and the only symptom was an empty log.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sibling container is reached at http://<entity name>:<internal port>,
and that string was written out per call site: a literal protocol, an
entity_name interpolation for the host, and a lookup('config', ...,
'services.<key>.ports.internal.http') for the port. Nine of them across
seven roles, each needing its own `# nocheck: literal-protocol-lookup`,
because the sibling lint rightly objects to a protocol glued in front of
a lookup - the protocol is supposed to come from wherever the TLS
posture is decided.
A container network has no TLS posture. It speaks plaintext by
construction, which is a property of the network rather than of any
caller, so stating it nine times stated it eight times too often.
lookup('container_url', application_id, service_key) holds all three
facts, and because no literal protocol is left at the call site the
suppression markers go with them.
The port is deliberately the internal one. A sibling that goes out to
the published port leaves the container network and comes back through
the ingress, which the dns-search suffix turns into the public FQDN.
The lookup refuses a service that declares no internal http port rather
than falling back to anything.
test_container_url_lookup.py keeps the shape from coming back: a
protocol, an interpolation mentioning entity_name, a colon, a second
interpolation. Nothing else matches that, and `# nocheck:
container-url-lookup` covers an origin this lookup genuinely cannot
express - a port that is not ports.internal.http, or a host that is not
the application's entity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/languages.yml carried `libretranslate: true|false` per entry, which
answered two different questions with one field: which languages the
repository translates, and which engine renders them. The first is the
table's business; the second is not. The gateway offers every backend
that serves a pair, falls through when one refuses or mangles the
answer, drops an engine whose failures pile up, and settles a request
class on the engine that wins its comparisons - and `signature()` is
"{source}:{target}:{length bucket}:{markup|plain}", so that learning is
already per target language. A static column beside it is a second
opinion that goes stale the moment the router learns something.
So the field is gone, 184 lines, and the table now holds name, endonym
and direction. Every language it names carries a catalog in every
domain.
What one engine can do is declared with that engine instead:
services.libretranslate-engine.serves, 47 codes. Not a detour - both
readers run before any engine answers. The lane prefetches one model per
code, and the readiness probe is itself the thing that would have asked.
`translatable()` split along that seam. Catalog-scoped callers take
`domain_languages()`, engine-scoped ones take the new
`served_languages()`: the CI matrix, extraction and translation the
first, the model prefetch and the tuning sweep the second. Collapsing
the two is what this change is guarding against - `baked_languages()`
renders identically before and after, verified by diffing both outputs.
`translate()` also stops dying on a language whose catalog has not been
extracted yet. It reads a missing catalog as nothing pending rather than
raising, because 136 docs catalogs do not exist until the next
extraction and refusing those languages would quietly shrink the set the
repository promises to translate.
The oracle suite follows: tests/oracle/test_translation_score.py asks
utils/i18n/judge.py about every language instead of routing a laya
checkpoint per language. It cannot run yet - the tools lane carries no
LiteLLM - and its skip message says exactly that.
test_catalogs_complete_on_gpu now measures against the table rather than
one engine's coverage. It is live on a GPU host and will report the 136
missing catalogs as the gap they are, which is the point: narrowing it
to the engine would hide exactly the languages worth watching.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The oracle suite needs a verdict in every language meta/languages.yml
names, and laya cannot give one. Measured against the suite's fixture,
its english checkpoint separates a faithful rendering from an unrelated
text in 17 of 47 languages and its multilingual checkpoint in none,
answering around 1.5 either way. One judge for all of them beats a route
per language that is noise in most of them.
The question keeps the shape the checkpoint answered: the levels are
written out and the model names an index over them, never a bare
probability. A probability carries this deployment's calibration, and a
fixed threshold on it would encode that calibration into the suite;
"below the middle level" instead means what the middle level says.
Three refusals are deliberate. temperature 0, so the same comparison
does not come out differently on a rerun. An answer naming no index
raises instead of scoring zero, which would read a refusal as "states
something else". An index outside the scale raises instead of being
clamped, because a model answering 7 has not placed the pair on this
scale at all.
The request goes to the model gateway rather than to a model server, so
it carries a consumer identity and resolves through the gateway's model
list like every other model call here.
Nothing imports this yet: the oracle suite moves to it once the tools
lane carries LiteLLM and a loaded model, which is what lets the judge be
exercised against a real one for the first time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.
The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.
call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
