Both new tests read a file their own run has just written, which the uncached-read lint flags because utils.cache.files.read_text is an lru_cache with no mtime invalidation. The two cases want different answers. The rendered inventory is YAML, so load_yaml_any reads and parses it the way the lint asks and needs no exception at all. The healthcheck probe writes its request through bash inside the test, so a cached read would compare against the previous run's bytes and the assertion would pass without proving anything; that one keeps a cache-read directive naming why.
make test-lint went from 2 failed to green, and the first attempt is the reason the inventory test carries no directive: suppressing the read there only moved the complaint to yaml.safe_load, which the right reader answers in one call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The oracle asks a chat model whether a translation states its source. llama3.2:1b cannot answer that: on the suite's own fixture it scored an unrelated text at the middle level instead of the bottom one, and asked as a yes-or-no question it called two identical texts different. With it the suite reached no verdict at all, reporting that the judge read none of the 47 languages the catalogs hold machine translations for. Qwen2.5-3B answers both correctly, and the same run then judged 248 samples.
The pin lives in the lane inventory rather than in AI_LOCAL_MODELS, because a judge that can separate a faithful rendering from an unrelated text is a demand no deployment makes, and the default should not grow a gigabyte to satisfy a check. The litellm config builds its routes from the merged application config, so a lane override reaches the model list and its window without touching the repository default.
LANE_MODEL_ALIAS repeats the one name in utils/inventory/tools.py, because ensure() asks whether the lane already serves before the deploy that writes the rendered inventory, so the probe cannot read it from there. A lane serving one model while the probe expects another never reports ready and redeploys on every call, so a test renders the template and holds the two names equal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lane ran litellm but never republished its port, so a host-side probe hit the compose proxy on the same number and read 502. The lane could therefore never report ready, and the oracle suite skipped itself to a green exit instead of asking a judge. A handler reads the port from the role the way translate.py already does, the runner forwards it, and TOOLS_LANE_KEYS takes it up: compose_file_args layers the override in without checking its variables, so an .env written before the key existed would abort every compose command rather than merely drop the lane.
The gateway's master key is pinned in the lane inventory for the reason the System One key already is: the provisioner would otherwise generate it into a vault the checks cannot read back, and the oracle judge is not a deployed consumer with a virtual key of its own.
serving() stops treating reachability as readiness. A lane brought up before AI_LOCAL_MODELS named a different model keeps serving the old one, and a gateway holding a key these checks cannot use looks healthy from outside while refusing every call they make. It now asks /models with the bearer token and requires the configured alias, and requires the gateway to offer every language the catalogs name, because that advertisement is what the translation run waits on. Without this the lane waved itself through and no change to its roles ever shipped.
Two lane-scoped overrides come with it. The ollama role caps itself at four CPUs, which left the host at 361 percent while twenty cores idled, so the lane gives it the host count the way the engine entry above it already does. And the lane pins the router to its declared order: the decider may send a pair the engine serves to the chat model, which a batch over 183 catalogs cannot pay for. Deployments keep the decider.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generated tcp and http_status probes built their request with echo -e, and echo appends a newline, so the bytes on the wire ended with a stray bare LF after the header terminator of a Connection: close request. uvicorn's h11 reader treats that as trailing data and answers 400 before the handler's 200 reaches the socket, so the probe's grep for a 2xx or 3xx status line never matched and the container stayed unhealthy for good.
In compose the name resolves straight to the container, so an unhealthy but running service still served and only the backup drill noticed, failing with litellm is running/unhealthy after 1200s. In swarm a task joins the service VIP only once it is healthy, so the litellm service sat at 0/1 with no backend and every consumer of http://litellm:4000 got ECONNREFUSED, which is where the openwebui model publishing died.
The tcp flavor had been masking the same malformed request all along, because its grep for HTTP/1 matches 400 Bad Request as happily as 200.
printf is a bash builtin like echo and the probe already requires bash for /dev/tcp, so nothing new is demanded of the image. The new test runs the generated request through bash and compares the bytes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make i18n-translate addressed the engine directly, so a run could never reach more than the 47 languages Argos ships models for, while meta/languages.yml names 183. service_url now resolves web-svc-translate, which routes a pair the engine does not serve to a chat model, and the deploy brings both roles up.
The gateway advertised only what its backends declared, and the chat backend declared nothing, so /languages reported 51 targets of 183 and the readiness wait spun to its hour-long timeout without translating a line. The chat backend now declares every language the table names.
Routing offered libretranslate for every pair, because an undeclared catalogue read as any pair, so a language it holds no model for was answered with HTTP 400 instead of falling through to the next backend. serves() now reads the backend's own /languages, which catalogue() already exposed for the advertisement. Declaring the pairs in the role was the first attempt and carried a second copy that was already stale: the engine reports zh-Hans where the role's list says zh.
A small model copies the prompt markers back, echoes the source line and labels its answer with the target language. All three reached the catalog, and an entry carrying its own source scores below the oracle's middle level. clean() strips them, and only from an answer that carried scaffolding, because on its own a short word and a colon is a translation, not a label.
server() had lost its redeploy parameter while cli/build/i18n kept passing it, so every run raised TypeError before translating a line and the only symptom was an empty log.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sibling container is reached at http://<entity name>:<internal port>,
and that string was written out per call site: a literal protocol, an
entity_name interpolation for the host, and a lookup('config', ...,
'services.<key>.ports.internal.http') for the port. Nine of them across
seven roles, each needing its own `# nocheck: literal-protocol-lookup`,
because the sibling lint rightly objects to a protocol glued in front of
a lookup - the protocol is supposed to come from wherever the TLS
posture is decided.
A container network has no TLS posture. It speaks plaintext by
construction, which is a property of the network rather than of any
caller, so stating it nine times stated it eight times too often.
lookup('container_url', application_id, service_key) holds all three
facts, and because no literal protocol is left at the call site the
suppression markers go with them.
The port is deliberately the internal one. A sibling that goes out to
the published port leaves the container network and comes back through
the ingress, which the dns-search suffix turns into the public FQDN.
The lookup refuses a service that declares no internal http port rather
than falling back to anything.
test_container_url_lookup.py keeps the shape from coming back: a
protocol, an interpolation mentioning entity_name, a colon, a second
interpolation. Nothing else matches that, and `# nocheck:
container-url-lookup` covers an origin this lookup genuinely cannot
express - a port that is not ports.internal.http, or a host that is not
the application's entity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/languages.yml carried `libretranslate: true|false` per entry, which
answered two different questions with one field: which languages the
repository translates, and which engine renders them. The first is the
table's business; the second is not. The gateway offers every backend
that serves a pair, falls through when one refuses or mangles the
answer, drops an engine whose failures pile up, and settles a request
class on the engine that wins its comparisons - and `signature()` is
"{source}:{target}:{length bucket}:{markup|plain}", so that learning is
already per target language. A static column beside it is a second
opinion that goes stale the moment the router learns something.
So the field is gone, 184 lines, and the table now holds name, endonym
and direction. Every language it names carries a catalog in every
domain.
What one engine can do is declared with that engine instead:
services.libretranslate-engine.serves, 47 codes. Not a detour - both
readers run before any engine answers. The lane prefetches one model per
code, and the readiness probe is itself the thing that would have asked.
`translatable()` split along that seam. Catalog-scoped callers take
`domain_languages()`, engine-scoped ones take the new
`served_languages()`: the CI matrix, extraction and translation the
first, the model prefetch and the tuning sweep the second. Collapsing
the two is what this change is guarding against - `baked_languages()`
renders identically before and after, verified by diffing both outputs.
`translate()` also stops dying on a language whose catalog has not been
extracted yet. It reads a missing catalog as nothing pending rather than
raising, because 136 docs catalogs do not exist until the next
extraction and refusing those languages would quietly shrink the set the
repository promises to translate.
The oracle suite follows: tests/oracle/test_translation_score.py asks
utils/i18n/judge.py about every language instead of routing a laya
checkpoint per language. It cannot run yet - the tools lane carries no
LiteLLM - and its skip message says exactly that.
test_catalogs_complete_on_gpu now measures against the table rather than
one engine's coverage. It is live on a GPU host and will report the 136
missing catalogs as the gap they are, which is the point: narrowing it
to the engine would hide exactly the languages worth watching.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The oracle suite needs a verdict in every language meta/languages.yml
names, and laya cannot give one. Measured against the suite's fixture,
its english checkpoint separates a faithful rendering from an unrelated
text in 17 of 47 languages and its multilingual checkpoint in none,
answering around 1.5 either way. One judge for all of them beats a route
per language that is noise in most of them.
The question keeps the shape the checkpoint answered: the levels are
written out and the model names an index over them, never a bare
probability. A probability carries this deployment's calibration, and a
fixed threshold on it would encode that calibration into the suite;
"below the middle level" instead means what the middle level says.
Three refusals are deliberate. temperature 0, so the same comparison
does not come out differently on a rerun. An answer naming no index
raises instead of scoring zero, which would read a refusal as "states
something else". An index outside the scale raises instead of being
clamped, because a model answering 7 has not placed the pair on this
scale at all.
The request goes to the model gateway rather than to a model server, so
it carries a consumer identity and resolves through the gateway's model
list like every other model call here.
Nothing imports this yet: the oracle suite moves to it once the tools
lane carries LiteLLM and a loaded model, which is what lets the judge be
exercised against a real one for the first time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two jobs of run 39040406916 died with the runner's own annotation, 'The
hosted runner lost communication with the server'. Both left every
diagnostic step pending, so no artifact exists for either. For the swarm
one the in-band sampler ruled starvation out — 9324M of 15947M available,
stall60 0.0%, 25 seconds before the cut — and nothing in the repository
can tell the remaining causes apart.
A stuck NFS mount wedges tasks in uninterruptible D state, which shows as
I/O pressure while memory pressure stays flat, and this drill runs its own
NFS node. /proc/pressure/io and /proc/pressure/cpu were read nowhere in
the repository. They are now sampled beside memory, with the D-state count,
every ten seconds instead of thirty, and an I/O stall at or above half the
minute aborts the step while the runner is still alive to upload the
banner.
cpu reads 'some', not 'full': the full line is pinned at 0.00 system-wide,
measured 13.50 against 0.00 on this host, so reading it like memory would
have shipped a metric that can never move.
Dump runner state was switched off for swarm, so the one post-mortem that
survives a step failure never ran there. It runs now, and the swarm drill
gets the step timeout its compose sibling has had all along — 350, the
number test_distro_budget_fits_timeout already derives the distro budget
from.
The sampler moved to its own module: the additions took matrix.py past the
file-size cap, and the lint asks for a split rather than a marker.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An unmounted VOLUME does not fail: docker creates an anonymous volume for
it, the container works, and the gap shows up later as a volume nobody can
attribute. sys-ctl-hlth-volumes fails the deploy over exactly that, and its
whitelist keys on volume ids, which are new on every deploy, so the gap
cannot be suppressed there either. e0b973f9bb fixed one such case in
web-svc-xmpp by hand; this is the check that would have found it.
Attribution is the whole difficulty. meta/services.yml declares an image
under a key, and the compose file often runs it under another name:
web-app-jitsi declares the image under 'jitsi' and renders the service as
'web', web-app-erpnext runs one image as seven services, and the two RDBMS
consumer keys name no service of their own because the container comes from
sys-svc-rdbms. The sweep therefore unions the entry key with its
proxy.service_key and delegates the RDBMS keys to their provider role. A
first version keyed on the entry alone and reported six roles that mount
every path correctly, web-svc-xmpp among them.
Reading a config blob section is now its own module beside the registry
client, because the two readers were the same walk twice and registry.py
sat three lines under the file-size cap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three faults, one mechanism each, that together let the oracle suite report
a verdict it never reached.
The address. lane_url handed out the runner's compose service name, but the
stack's containers resolve through the deployment's CoreDNS, which never
learns one: getent hosts tools answers nothing inside them, so every check
that asked the lane from there skipped itself. The bring-up now writes the
runner's address, picked as the one inside INFINITO_SUBNET because the
runner sits on every network it was ever brought up on, and the checks read
it from there. It is rewritten on both paths, because a restart gives the
runner a new one.
The readiness. serving judged the engine alone, so a lane whose engine make
i18n had already brought up counted as up while System One was down. It now
probes every role the caller names, which is why the runner republishes
System One's port: a container IP is unreachable from the host that decides
whether to deploy.
The verdict. A silent lane made the suite skip, and a skip exits green.
Only a deploy that is already running leaves the stack alone and exits 0
now; a crashed deploy and a deployed-but-silent role both exit 1.
The override's third :? guard is added to the gate that decides whether the
file is layered in at all, next to the one that was already missing. Without
it an .env written before these keys aborts every compose command, down to
compose-down, instead of dropping the lane. A test now keeps that list and
the override's guards in step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every cold render of the applications SPOT walks the templates of all
269 roles, and those templates call lookup('config'), lookup('tls'),
lookup('domain') and others. Each of these re-enters
get_merged_applications while the render guard is set, and each one
rebuilt the unrendered view from scratch: a deepcopy of all role
defaults plus the override merge, about 8 ms, never cached. A local
cold render of the development inventory made 351 such calls, 2.82 s
of a 6.86 s render.
The outer render now keeps a copy of its unrendered view and its cache
key on the render guard, taken before the inventory volumes are
popped, so it is the same view a nested call used to build. A nested
call with the same key returns it; any other key still builds its own.
The guard drops it when the render ends.
Measured on the same inventory: the nested calls took 0.07 s instead
of 2.82 s and the cold render 3.60 s instead of 6.86 s. The rendered
SPOT is unchanged: the old and new dumps differ in 69 user passwords,
which the development inventory regenerates on every read (two runs of
the old code differ in 68), and in one absolute checkout path, the only
other difference.
In CI each cold render took 40 to 77 s and the heaviest swarm cell of
#950 missed its budget by a few minutes (#984).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
4aba897ca2 gave the pull and build subprocess a 3600s ceiling so a hang
could not stall a deploy. A pull of a large image set legitimately outlives
any fixed ceiling while it is still making progress, and the ceiling then
kills a healthy run.
run_streamed measures the gap between two output lines instead. Two reader
threads keep stdout and stderr separated the way run_cmd does, because
compose writes its warnings to stderr and merging them poisons every parser
downstream. Only the two calls that legitimately run long use it; the short
calls whose output is parsed keep run_cmd.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The full gate failed test_a_distro_needs_the_slowest_run_plus_headroom_to_start on a quiet machine, so the load is the gate own parallel workers rather than anything beside it. The branch under test needs d < B < 2.15*d for a measured first duration d, the rule 040955ebf2 wrote down when it widened the same fixture from sleep 2 / 3s to sleep 4 / 7s. At 4 / 7 the window admits a first iteration of 4 to 6 seconds, three seconds of slack, sized then against the single second CI had overshot by. Under the gate it overshoots further, takes the budget exhausted branch and never prints the line the test greps for. At 10 / 19 the window admits 10 to 18 seconds, nine seconds of slack. The assertion and the branch are unchanged and the suite pays six more seconds.
The second hunk shortens the Odoo reload task name to 60 characters; at 83 it broke the 80 character convention the name lint enforces.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge left the role with two ways to the same end. The wired shell path called setup.py update-user, so it could neither create a missing account nor clear a lockout, and it decided its own idempotency from a sha256 marker in the persistent volume, which lies after a restore: marker matching, database not, reported unchanged and did nothing. The python converger from 0709de6fa1 queries the application instead, creates what is missing, verifies the password, unlocks, resets the failed-login counter and ensures the Administrator role, and it carried the only tests either path has; it was wired to nothing after the merge resolved towards manager_ops.yml. That engine now serves the surviving wiring: renamed into the family spelling the sibling roles use, reading the env names the task already passes and emitting the markers the task already parses. tasks/administrator.yml had no caller anywhere in the tree and goes with it.
Trade-off worth knowing: the deleted path used the supported pgAdmin CLI while this one uses internal APIs, so a pgAdmin upgrade can move it. Neither path has ever executed in CI, the pgadmin job died at the health wait before reaching the realign, so the choice rests on capability and tests rather than on field evidence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A recover mirrored a snapshot with rsync -a, which transfers no extended attributes, so a restored volume came back byte-identical while every application that keeps identity in xattrs read the tree as unknown. OpenCloud posixfs is the case that surfaced it: after the DR drill its storage log reports node.Xattrs on the platform administrator space as no data available, the showcase files are assimilated as new objects, and the graph call that lists drives returns no personal drive, so the design spec dereferences undefined. Measured here, source tagged with user.oc.name, on tmpfs and on ext4 alike: rsync -a returns 0 and drops the attribute, rsync -aHX and rsync -aHAX return 0 and carry it. The shared restore flow and the NFS half of this repo now use -aHAX, the form ensure_subject_subvolume.sh already uses, continuing what 69321d4135 started for numeric ownership.
The backup half still has the same gap: backup-docker-to-local 7.0.2 builds its command as rsync -aP --no-D --delete --delete-excluded with no flag passthrough, so until it carries -X and the pin moves, a snapshot holds no attributes for this restore to put back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard patched the silence budget down to 0.3 s, but the watchdog starts its clock at Popen while python -c still has to boot before it can print, so under the parallel load of a full gate run the startup alone exhausted the window and the process was killed before its first line. The same file passed in four seconds on its own. The budget is now 1.0 s against a process that prints every 0.2 s for three seconds, which keeps all three conditions true at once: the total run outlives the budget so the old total-runtime timer still kills it, the gap between lines stays under it, and so does interpreter startup. Verified against the old shape in isolation: it dies with signal 9 after five of fifteen lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The watchdog armed a single timer at process start, so it killed the German build with SIGKILL while that build was still emitting progress, and Library.failed() then answered 500 for the rest of the run. The container was not out of memory: OOMKilled is false, the restart count is zero and neither the kernel ring buffer nor the journal records a kill, while the sphinx worker BrokenPipeError that follows is the signature of killing the parent alone. The watchdog now stamps the time of every line read and kills only after a full silence window, which the existing test name already described. A regression guard runs a process that keeps printing past the deadline and requires its last line in the log.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`deadline` is whole seconds from `date +%s`. A run that starts just
before a second boundary reaches the deadline after the first probe and
one `sleep 1`, so the gate correctly gives up after one probe while the
test demanded two. With a stubbed clock (1000, 1000, 1002) the gate
probes once and prints `deadline of 2s reached`. The deadline message
already proves the pending migration held the entrypoint back; require
at least one probe.
Reported by CodeRabbit on #979.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Swarm Nextcloud#3 on main 748189e890 (job 117182318941) wedged the
OpenProject schema: the convergence check gave up after 74 attempts
while the seeder kept exiting 1, first with
`relation "ar_internal_metadata" already exists` from its structure
load, then with `relation "announcements" already exists` on every
`db:migrate`.
The seeder (`docker/prod/seeder`) loads `db/structure.sql` through
psql, statement by statement, when `\dt` finds no relation. In swarm,
`depends_on` is ignored, so web and worker start alongside it and poll
`await_migrations.sh`. That gate called
`rails db:abort_if_pending_migrations`, which runs
`migration_context.open`, i.e. `Migrator.new`, whose constructor
executes `schema_migration.create_table` and
`internal_metadata.create_table` (Rails 8.0.4,
activerecord/lib/active_record/migration.rb:1430-1431). The seeder
saw an empty database at 05:33:20; by 05:33:38 the gate had created
`ar_internal_metadata`, the structure load aborted at line 255 with
everything alphabetically before it already committed, and every
later attempt took the migrate path into tables that already existed.
Probe with `rails db:migrate:status` instead. It aborts while
`schema_migrations` is missing and otherwise only reads the recorded
versions (database_tasks.rb:303, migration.rb:1318); the gate treats
any `down` row as pending.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two conflicts, both resolved by combining rather than by picking a side.
`roles/web-app-socialhome/tasks/02_design.yml`: designer restructured the pass,
moving the token request and the theme write into the new
`02_design/converge.yml` and adding a stat of the name-state file. HEAD still
carried the converge block inline. Kept designer's stat, dropped HEAD's inline
block: the merged body below the conflict already includes `converge.yml` and
reads `socialhome_name_marker`, which only designer defines, so keeping HEAD's
side would have duplicated the token request and left that variable undefined.
HEAD's `retries: 0` on the token request is deliberately not carried over; the
default of 30 retries covers a service that is not yet accepting connections,
which matters more here than failing fast on bad credentials.
`roles/web-app-fider/files/playwright/test-design.js`: the import line takes
both sides, since the merged body uses designer's `apiGetOnion` and HEAD's
`requireDotenvValue`. The env block keeps HEAD's `requireDotenvValue` form,
because designer's `|| ""` defaults are what
`tests/lint/ansible/.../test_unguarded_env_default.py` exists to reject, and
drops HEAD's module-scope `title` and `logoEnabled`: the merged body defines
`title` locally and reads `DESIGN_LOGO_ENABLED` directly, so both were dead and
would fail `no-unused-vars`.
Checked after resolution: no conflict markers (`git diff --check`), no `nocheck`
line dropped by the merge, and no duplicated YAML key in the seven merged files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without `--all`, `journalctl --output=json` encodes a field over 4096
bytes as `null`, so a long host error matched nothing and the check
passed. A repeated `MESSAGE` field arrives as a JSON array of its
values, and `bytes()` on that array raised `TypeError`, which aborted
the whole check.
Pass `--all`, and tell a byte array (all integers) from an array of
field values, decoding each value on its own.
Reproduced against the host journal: a 5000-byte error line read back
as `"MESSAGE":null` and produced no match; an entry with two `MESSAGE`
fields, sent over the native journal socket, read back as
`["error first","error second"]` and crashed `host_errors` with
`TypeError`. With the fix both are reported.
Also clear the two test-lint findings of this PR:
`test_comments_valid` rejects the narrative comment above the
`journalctl` Popen, so it goes; the stderr-to-file choice stays, because
an unread stderr pipe that fills up would block `journalctl` while
stdout is still being read. `test_no_uncached_file_reads` flags the
read of the stub's argument file, which lives in the test's tempdir and
is written per run, so the path-keyed `utils.cache.files.read_text`
could serve a stale body; the read opts out with
`# nocheck: cache-read tempdir fixture`, as the other stub-driven
script tests do.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`lookup('tls', ...)` aligns a cross-app domain to the consumer named by
`application_id` (utils/tls_common.py `align_domain_to_consumer`): a
clearnet-primary consumer gets the clearnet sibling of an onion-primary
target. The carrier key does not cover `application_id`, so a park that
renders inside a role hands that role's alignment to every later role
whose key matches.
The constructor and `tasks/utils/setup/role.yml` park at play scope,
where `application_id` is undefined. The re-park in
`roles/sys-token-store/tasks/write.yml` (67f6c8d7e8) and the nested-load
re-park render inside the caller: `svc-prx-openresty` at the preload
loader, `web-app-stalwart` or a provider role on the NFS host, the web
app itself under `sys-stk-front-base`. In a tor deploy that turned onion
targets into clearnet domains for onion-primary roles served from such a
park, and baked the caller's group into Nextcloud's sociallogin
`groupMapping`.
Drop `application_id` from the render's variables and from the
templar's for the duration of the render, so every park carries the
play-scope result the constructor parks have always carried. Rename the
nested re-park task so its log lines do not read as a role boundary.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SH_ADMIN_PASSWORD seeds the administrator on first boot only: upstream provision_admin early-returns once platform_users holds a row. The rotation gate regenerates users.platform_administrator.password, the container keeps the old hash, and the household theme token request fails. The 31 retries then drained the 5-per-15-min budget upstream enforces on /api/auth/token, so every attempt after the fifth answered 429 and hid the underlying 401.
realign_admin_password.py runs inside the container, resolves the database through the config helper the application itself exposes, and rewrites platform_users.password_hash with the scrypt helper the application itself exposes when the declared password no longer verifies. It writes that one column, because set_password would also write is_admin and demote the administrator.
The token request drops its retries. 31 attempts into a five-per-window budget also poisons the bucket the Playwright design spec shares, and 0 is the only value multiply-timeouts.sh cannot inflate.
Proven by a two-pass compose deploy of variant 0, the row that failed in CI and the only one with design enabled: pass 1 [UNCHANGED], pass 2 [CHANGED] after the rotation, and the token task that failed in CI with 31 attempts and status 429 answering on attempt 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Problem:
Every compose service logs to journald through
sys-svc-container/templates/base.yml.j2. The daily journal health
check grepped the whole journal for "error", so routine application
errors from containers triggered the host alert (email/Telegram) and
buried real host problems.
Change:
- Read the last day of the journal as JSON (MESSAGE, CONTAINER_ID,
SYSLOG_IDENTIFIER, _SYSTEMD_UNIT) and skip every entry that carries
a CONTAINER_ID field. Docker daemon and system errors are still
reported.
- Report each match as one readable "source: message" line instead
of raw journal output.
- Stream the journalctl output line by line instead of buffering a
whole day of logs in memory.
- Exit 2 with journalctl's stderr if journalctl itself fails, instead
of silently reporting "no errors".
- Rewrite the check in Python (files/python/script.py, deployed by
sys-service like the other sys-ctl-hlth roles) and decode non-UTF-8
messages.
- Fix the success message, which claimed "All docker containers are
healthy".
Impact:
Container log errors no longer raise this host alert on any
installation. Container health stays covered by sys-ctl-hlth-container;
application log analysis belongs in a log platform such as Wazuh.
system_service_suppress_flush is unchanged until a deploy confirms
that no host errors remain.
Tests:
Unit tests run the script against a fake journalctl (8 cases):
container error ignored, daemon error reported, identifier fallback,
non-UTF-8 message, journalctl failure, many container lines streamed
past, clean journal passes. The core cases fail against the previous
script.
The 🐳 Workspace (ubuntu:latest) job on main (run 38879556215, 1b36ef6120)
failed test-unit-python in the hook's second make test:
AssertionError: Regex didn't match: '^ localhost=ok/\d+ms$'
not found in ' localhost=ok/-1612931482679767779ms'
ubuntu:latest is Ubuntu 26.04, whose date is uutils coreutils. uutils
0.7.0 ignores the field width in %3N and prints all nine nanosecond
digits, so `date +%s%3N` yields a 19-digit value instead of epoch
milliseconds. Against main's sampler, uutils 0.7.0 date reports
localhost=ok/7081859ms, and an unpadded nanosecond value turns the
difference negative.
answer_once now takes both timestamps from /proc/uptime, which the
kernel prints with two decimals whatever date the image ships. The
sampler already reads /proc/net/udp, so it was Linux-only before. The
resolution is 10 ms; the same probe reports 10 to 20 ms with uutils
date on PATH.
The new test puts a date on PATH that does not understand %3N; main's
sampler dies on it, the fixed one does not call date for the latency.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The WordPress MCP endpoint answered 401 to every authenticated call, and
web-app-hermes and web-app-openclaw failed their CLI tests as cascades of the
same refusal. The WordPress container's own Apache log settles it: 121 POST,
100 GET and 21 HEAD on the MCP path, each with %u = mcp-web-app-wordpress and
status 401. A populated %u rules out a stripped Authorization header, and the
body is infinito_mcp_unauthorized, which only the mu-plugin's rest_pre_dispatch
guard emits - a wrong password for an existing user would surface WP core's
incorrect_password instead. So WordPress never resolved a user, because the
deploy sent the users-dict key while the login the deploy creates is
mcp-wordpress.
7916c0177e introduced the split when it capped usernames, deliberately keeping
the key because meta/mcp.yml's owner and the token store wire to it.
web-app-wordpress is the only role with auth: basic_auth - every other MCP
provider uses bearer, where the owner never travels - which is why only this
one broke.
resolve_credential now also returns the login, mcp_servers emits it on every
discovered entry, and the filter encodes login:token. It raises when no login
resolves rather than authenticating as the empty user, so the failure is loud
instead of another 401. The new mcp_client_login lookup shares
resolve_provider_credential with mcp_client_bearer so a second resolver cannot
drift from the first, and a test pins the two to the same function object.
test.env.j2 is the one place that hand-builds the server dict instead of
reading MCP_DISCOVERED_SERVERS, so it passes login as well.
The lint closes the class: auth: basic_auth requires credential.owner to be a
key some role's meta/users.yml declares. Without it a future basic-auth
provider sourced from credentials would hit the same fallback silently. It
imports BASIC_AUTHS from the filter so the rule and the renderer cannot drift
onto different spellings.
Tool names were verified against the adapter version the addon actually
installs, v0.7.0, not the v0.5.0 recorded in meta/mcp.yml: the sanitizer turns
the three registered abilities into infinito-get-post,
infinito-list-categories and infinito-search-posts, matching the rendered
allowlist exactly. Adding login to a discovered entry changes nothing for the
five consumers - three of them do not list basic_auth in supported_auths and
never see this server at all, and the other two read named keys only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Under SSO the platform administrator signed in and landed on #profile as a
Customer, so the design spec never reached #manage/users. Three defects
stacked.
The group-to-role map was written with Jinja in its mapping keys. Ansible
renders a mapping's values and leaves its keys alone, so the rendered Ruby
keys were literal {{ ... }} and could never match a group path. The map is now
one expression that computes its keys through rbac_group_path, the shape
meta/addons already uses elsewhere.
The initializer patched User.create_from_hash and User.update_from_hash behind
`next unless User.respond_to?`. Zammad 7.1.2 defines only create_from_hash!,
so both guards fell through in silence and nothing was installed: the
initializer's name does not appear once in the failing run's Rails log. Patch
Authorization.find_from_hash and Authorization.create_from_hash instead, which
sessions_controller calls before current_user_set, so first and subsequent
logins are both mirrored.
administrator mapped to Admin alone. Zammad's Admin carries no ticket
permission, and the Dashboard navbar entry is gated on ticket.agent, so an
Admin-only account has neither Dashboard nor Overviews. Grant Admin and Agent,
matching what the password path converges.
The write is defended rather than assumed. Zammad's has_roles concern runs
validate_roles, validate_agent_limit_by_role and last_admin_check_by_role on
collection assignment and they raise, which inside the omniauth callback would
answer the sign-in with 500 - a lockout, worse than the demotion being fixed.
The assignment is wrapped, resets the association and logs on refusal, and the
sign-in completes on the roles the account already has. Role lookup filters on
active and reports a missing role instead of silently granting the subset. A
map that rendered as a String is refused at boot, because map[path] would
otherwise be String#[] returning a substring and the feature would be a silent
no-op. The patch targets are checked by signature, not by respond_to?, since
respond_to? was exactly what hid defect two.
An absent groups claim leaves the roles untouched and logs, rather than
demoting every account to Customer on a misconfiguration. A present but empty
claim still means "in no group" and maps to Customer.
biber is unaffected: inventories/development/default.yml declares roles: [] for
it, so it is in no RBAC group, maps to Customer as before, and keeps
#customer_ticket_new.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
assertReadable takes a selector list and measures contrast on whatever it
finds. An entry marked optional is skipped when it matches nothing, so a list
whose entries are all optional can measure nothing at all - and then the call
asserts nothing while reporting green. Reject such a list before the browser
is touched. A plain string normalises to a required entry, so only a
deliberately all-optional list is refused, and an empty list with it.
web-app-matomo found the defect the hard way in CI run 38818071194. Its KPI
card check listed four optional selectors, all of them trend modifiers that
the dashboard only renders at a non-zero change; every card read 0%, so
KPICard emitted kpiTrendNeutral and SitesTableSite left its evolution cell
unclassed, nothing matched, and the test failed on the minimum-measured guard
rather than on contrast. The sibling evolution-badge check has the same
all-optional shape and was green only by luck.
Give every all-optional call site an anchor that is already proven visible in
its own spec: .kpiCardEvolutionTrend and .evolutionBadge for matomo,
header.header for web-app-dashboard (waitForDashboardReady asserts it
unconditionally) and #login .panel-body for web-app-espocrm. The espocrm case
needs care: #btn-login and #sign-in each exist in only one branch of the sso
conditional, so neither is variant-safe, while assertLightAndDark measures
.panel-body on the line above.
The matomo theme rule is part of the fix, not a cosmetic extra. Without it the
new anchor keeps Matomo's hardcoded #76838f, which measures 3.60 against
--design-surface-1 and 3.82 against --design-surface-2 in light mode and would
fail the 4.5 threshold the anchor now enforces. The role already replaces
Matomo's hardcoded trend colours; this closes the same gap for the neutral
caption. Equal-specificity rules resolve by order and the role's block is
concatenated about 204 KB after Matomo's in the single served bundle, measured
by byte offset in the failing run's own trace.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
00_core.yml pins xwiki_blobstore_s3_enabled_switch false for the bootstrap
render on every deploy. On a wiki whose attachment content already lives in
the SeaweedFS bucket that strips the whole store.* block - the blob store type
and the object-store credentials with it - and restarts the container. In CI
run 38818071194 the credentials were rotated between the two passes at 04:49,
the SeaweedFS identities were re-rendered at 05:04, and the bootstrap render at
05:39 then removed the only configuration that would have carried the new keys
in. 06_design/logo.yml:7 is the first task that reads an attachment, so that is
where the play died, after 31 retries, with "could not be found in the
filesystem attachment store".
Move the bucket render behind 04_extensions in 02_manager_ops.yml, which
installs and asserts xwiki-commons-store-blob-s3. The precondition 26d588fa41
names - nothing answers to 's3' before the extension is installed - is then
established by execution instead of inferred, and 00_core.yml stays byte
identical to what it was.
An earlier attempt probed the deployed xwiki.properties for store.blobStoreType
and selected s3 when it was already there. That is not safe: the purge
primitives in scripts/container/purge/apps.sh:86-94 each run under `|| true`,
so a failed `rm -rf` leaves the config dir behind with the volume gone, and a
plain `compose down -v` in a debugging session does the same while the central
Postgres keeps the installed-extension rows. The probe would then have written
s3 into a wiki that cannot start. Removing that exposure beat pricing it.
Nothing before the new task can behave differently from today, so the only new
behaviour is one render into oidc off, ldap off, s3 on - two configurations
that each booted in this run, in a flat properties file where the blocks do not
interact. In the bootstrap window the new shape is strictly better: today's
window is a superset that extends through 06_design.
The test asserts both halves of the invariant, that no render in 00_core.yml
selects the bucket and that the bucket render precedes the 06_design include.
Against the previous tree it reports renders [8] design [7] ordered_ok False.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
weblate, beat and worker mount the same weblate_data and weblate_cache.
Docker fills a fresh named volume from the image once per
POST /containers/create, and that copy does an Lstat plus os.Mkdir per
directory without tolerating EEXIST. Compose created all three at once, two
populations of the image's /app/data raced, and the deploy died with
"failed to mkdir /var/lib/docker/volumes/weblate_data/_data/python: file
exists" in CI run 38818071194. The daemon log names the route as
containers/create, so this is the population path, not a mount of a
pre-existing file.
Make weblate and worker depend on beat. The lever is not startup order but
the Create phase, which compose runs in dependency order: beat populates both
volumes alone, and by the time the other two are created no copy runs. The
failing run proves that phase is ordered - COMPOSE_PARALLEL_LIMIT was 4, yet
only the two existing depends_on targets reached Created before the other
three began.
This is compose-only. In swarm, container_depends_on drops to a bare list
(plugins/lookup/container_depends_on.py:103) and docker stack deploy ignores
it, so swarm behaviour is unchanged and the same race stays reachable there. A
mode-independent fix needs nocopy on the non-owning mounts, which
plugins/lookup/container_volumes.py cannot express today.
The test reads meta/volumes.yml, collects every service sharing a volume and
requires each non-beat sharer to name beat. Against the previous template it
yields weblate False, beat True, worker False, so it fails without this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Since 778c18fb23 the carrier key holds the tokens file's mtime and size.
sys-token-store changed that stat on every inclusion (00_core.yml
touched the file) and on every persisted token, and nothing re-parked a
carrier afterwards. Every later task's fresh worker then rendered the
applications SPOT cold, 30 to 80 s each. On main fea6f31d9f (run
38812964308) seven heavy jobs logged 194 to 287 such tasks and ran out
of the 20400 s budget; the same jobs before 778c18fb23 logged 65 to 83,
all of them real waits.
- 00_core.yml touches the store with modification_time and access_time
preserved, so it only creates a missing file. In swarm the controller
sees the store over NFS with attribute caching, and the touch alone
made the second pass go cold without any write.
- write.yml re-parks the carrier when the persist or the controller
mirror changed the store. Both run once, so every host that ran the
include gets the same verdict.
- tasks/utils/setup/role.yml re-parks after every role on every host,
so hosts that did not run the writing role recover too. Where the
key still matches this serves the carried payload; measured 0.1 s per
role in a local compose deploy.
- The tokens file enters the key as the SHA-256 of its content,
memoised per inode, mtime and size, so a touch no longer changes the
key. A stat that still matches serves the memo (1.67 us against
1.41 us for the old stat signature), and a stale stat keeps key and
YAML cache on the same old content instead of pairing new content
with an old key.
- applications_carrier opens the tokens file before it computes the
key, so on the NFS-mounted controller the parked key reflects the
revalidated attributes rather than a cached stat that changes a
minute later.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The web-app-litellm stylesheet swapped the navbar image with content: url(), so LiteLLM's own logo setting stayed empty and its UI theme page showed nothing. svc-ai-litellm now renders litellm_settings.ui_theme_config.logo_url from LITELLM_UI_LOGO_URL, the design lockup of web-app-litellm, only when that role is deployed with design and a logo; the stylesheet override is removed. The design spec asserts the served setting, the navbar src, the absence of a CSS swap, the loaded wide lockup and the value on the UI theme page; the config render test covers both the logo and the no-logo case.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`make quality-high` on main (fea6f31d9f) rewrote two files in its
autoformat step:
- roles/sys-service/handlers/refresh_systemctl_service.yml: the
handler's `when` was wrapped by hand; the formatter wraps it at line
width. Both forms parse to the same YAML.
- tests/unit/python/roles/web-app-nextcloud/test_onlyoffice_check_deadline.py:
two over-long assertIn calls that ruff format splits.
A second autoformat run leaves the tree unchanged, so main is stable
under the formatter again.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
identity.py built the services.yml path by hand, which test_file_spot.py names: utils/roles/mapping.py:175 holds ROLE_FILE_META_SERVICES, and bed4b3c84f already uses it in this role. The module meant to make services.yml the single source had duplicated a path of its own.
test_mirrored_unit_tests.py also required the matching test. It proves three things against a synthetic tree rather than the real values, so it cannot pass tautologically: the configured values arrive under the environment names conf.py reads, an undefined group variable is refused instead of rendering empty, and the repository's own configuration resolves with no {{ left in any value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
230d9a1e5e moved scripts/docker/cache/package-frontend/ca.sh to roles/svc-cache-package/files/shell/ca.sh as a pure rename. On origin/main the script stayed where it was, and 2756fd56f1 changed it there and added this test. The merge took the test without the move, so every case ran bash against a path that no longer exists and failed with exit 127.
Only the path changes. The script it exercises is the same file the rename produced, and every other PROJECT_ROOT path under tests/ was checked against git ls-files: registry-ca.sh is still at its old location and its test is correct, cert-gen.sh moved along but has no test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
services.docs.sphinx is the single place the project, copyright and author are configured. conf.py takes them from the environment, which env.j2 fills inside the container, but two callers run outside it: the host side of make i18n-extract and the unit tests. Both had a copy of the values, which is the duplication this removes.
infinito_docs/identity.py renders services.docs.sphinx with the group variables it references, through the same Jinja environment utils/templating/jinja.py:62 uses: StrictUndefined, so an undefined variable raises instead of rendering empty, and select_autoescape for html and xml, which leaves a from_string template unescaped and satisfies ruff S701.
utils/i18n/extract.py loads that module from the tooling path it already computes for the subprocess PYTHONPATH, without touching sys.path. The two tests import it through the _TOOLING entry they already insert. conf.py is unchanged and still requires the environment: reading the configuration there would turn it back into a default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The third post-merge quality-high pass wrapped the two context managers of 00393bb1b1 into the parenthesised form. Both revisions parse to an identical AST.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
On the Fedora, CentOS Stream and Manjaro workspace jobs install-lint
ran `puppeteer browsers install chrome-headless-shell --install-deps`
as root, and puppeteer rejected it with "apt-get does not seem to be
available". The installer only warned ("optional lint tool"), so
install-lint passed while mmdc had no browser.
--install-deps is now passed only where apt-get exists. Everywhere else
the libraries `ldd` reports missing are installed by soname: dnf and
yum through the RPM provide `<soname>()(64bit)`, pacman through the
first package `pacman -F usr/lib/<soname>` names (libgbm.so.1 is owned
by both mesa and mesa-amber, which conflict). browser_installed() also
requires every library to resolve, so the install-lint stamp stays
stale until the browser can actually load.
A failed npm install, a missing mmdc, a failed or timed-out browser
download and unresolved libraries now raise instead of warning.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Manjaro workspace job failed test-unit-ruby with "cannot load such
file -- minitest". Debian vendors minitest in libruby, but Arch ships it
as ruby-minitest and Fedora and CentOS Stream as rubygem-minitest, none
of which ruby pulls in. ensure_ruby_present now probes
`require "minitest"` and installs the distro package when the probe
fails, then fails loudly if minitest is still missing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Fedora and CentOS Stream workspace jobs failed TLS to pypi.org,
registry.npmjs.org and repo.packagist.org (CERTIFICATE_VERIFY_FAILED,
curl error 7). package-frontend/ca.sh put the frontend CA into the Arch
anchor dir /etc/ca-certificates/trust-source/anchors, which p11-kit on
the RedHat family never reads.
Probing for the anchor dir does not fix it: with-ca-trust.sh creates
both /etc/pki/ca-trust/source/anchors and
/etc/ca-certificates/trust-source/anchors on every update-ca-trust host,
so the probe in registry-ca.sh already picks the RedHat dir on a reused
Arch runner. Both scripts now install into both dirs, rebuild only when
one of them changed, and take CA_ROOT so unit tests run them against a
fake root.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
