make quality-high reformatted three expressions this branch added today: the chat-target chain in the translate role's vars, an assertEqual in the gateway engine tests, and the wanted comprehension in the lane probe. No logic and no rendered value changes; the YAML break folds to a space inside a double-quoted scalar.
Kept separate from the changes that caused it, because a reflow mixed into a behaviour commit hides the behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every catalog got a lane derived from the CPU count, so twenty requests met one model instance. That is not throughput, it is self-inflicted saturation: a single catalog recorded 425 translated, 9 discarded and 40 requests refused with HTTP 503, and a discarded entry is simply left pending.
The work is now split by the backend that will answer it. The languages the engine serves keep the wide pool, because Argos answers in about 0.15s; the rest run after them with two lanes, one to occupy the model and one to refill it while the first writes back. Each pass builds its own client, so the chat pass does not inherit the worker count the engine pass was sized with.
Order is the larger half of this. The table is alphabetical, which is why the last run spent its first 45 minutes on af, a language with no Argos model, while fr and de sat untouched. Cost order delivers the engine languages first. For domain=docs that is 47 catalogs of 3963 entries each in roughly an hour; for domain=core it changes nothing today, because those catalogs are already translated.
The test picks aa against fr on purpose: both orders agree on every pair the fixtures held before, so a test built from them would have passed against the old single pass as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In swarm, docker stack deploy ignores depends_on, so the precheck
service and every init replica ran search_allocation_precheck.sh at the
same time. They shared one probe index: one invocation's PUT could fail
the shards_acknowledged check because another had created the index
first, and either one's cleanup could delete the other's probe. The
init service carries no replica override, so compose_replicas gives it
one task per host in the application group and the collision also
happens between init replicas.
A failing precheck task also never reached has_fatal_task in
stack_ready.sh: with restart_policy none it records a single error,
while has_fatal_task needs three. stack_ready.sh then kept answering
"not converged" until the handler ran out of retries. And a Running
init task counts as converged through running_current_spec, so the
swarm deploy never depended on init failing; the role's "wait for
Zammad init" task is what stops it.
The probe index now carries the container's hostname, which docker
sets to the container id, so concurrent invocations no longer share an
index. compose.yml.j2 renders the precheck service, and init's
service_completed_successfully dependency on it, only outside swarm.
Compose keeps its fast stop on a refusal; in swarm, init runs the same
check in front of zammad-init, a refusal leaves Zammad unseeded, the
wait for Zammad init fails the deploy, and the init service logs carry
the allocation explanation.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On a data disk with less free space than the high disk watermark asks
for, Elasticsearch creates indices but assigns none of their primary
shards. `zammad-init` then reports every search index update as failed
and never exits, so `compose up` waits on
`service_completed_successfully` until its 1800 s task timeout and the
deploy fails without naming the cause (#850). Elasticsearch's default
high watermark is an absolute 150 GB headroom on a large disk.
A new one-shot `precheck` service runs `search_allocation_precheck.sh`
with `restart: "no"`, and `init` depends on it with
`service_completed_successfully`:
- it waits a bounded time for `_cluster/health`;
- it creates a one-shard, zero-replica probe index with
`wait_for_active_shards=1`;
- once Elasticsearch allocates it, it deletes the probe and exits 0;
- otherwise it prints the creation answer and the answer of
`_cluster/allocation/explain` and exits 1.
The check sits in its own service because `init` keeps
`restart: on-failure`: wrapped into `init`, the failing check was
restarted seven times within eight minutes on a local deploy while
`compose up` kept waiting. Inside the stack, a `restart: "no"`
dependency that exits 1 made `compose up` fail after one second with
`service "pre" didn't complete successfully: exit 1`.
Every curl call carries --connect-timeout and --max-time, so a hanging
request cannot outlast the bounded wait. The image's entrypoint passes
any non-`zammad-*` command through `exec "$@"` (bin/docker-entrypoint of
7.1.2), and the image ships `curl`. The watermark values themselves come
with feature/store-i18n (b82591ba9d); this change covers inventories
without a variant cap.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Both new tests read a file their own run has just written, which the uncached-read lint flags because utils.cache.files.read_text is an lru_cache with no mtime invalidation. The two cases want different answers. The rendered inventory is YAML, so load_yaml_any reads and parses it the way the lint asks and needs no exception at all. The healthcheck probe writes its request through bash inside the test, so a cached read would compare against the previous run's bytes and the assertion would pass without proving anything; that one keeps a cache-read directive naming why.
make test-lint went from 2 failed to green, and the first attempt is the reason the inventory test carries no directive: suppressing the read there only moved the complaint to yaml.safe_load, which the right reader answers in one call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The oracle asks a chat model whether a translation states its source. llama3.2:1b cannot answer that: on the suite's own fixture it scored an unrelated text at the middle level instead of the bottom one, and asked as a yes-or-no question it called two identical texts different. With it the suite reached no verdict at all, reporting that the judge read none of the 47 languages the catalogs hold machine translations for. Qwen2.5-3B answers both correctly, and the same run then judged 248 samples.
The pin lives in the lane inventory rather than in AI_LOCAL_MODELS, because a judge that can separate a faithful rendering from an unrelated text is a demand no deployment makes, and the default should not grow a gigabyte to satisfy a check. The litellm config builds its routes from the merged application config, so a lane override reaches the model list and its window without touching the repository default.
LANE_MODEL_ALIAS repeats the one name in utils/inventory/tools.py, because ensure() asks whether the lane already serves before the deploy that writes the rendered inventory, so the probe cannot read it from there. A lane serving one model while the probe expects another never reports ready and redeploys on every call, so a test renders the template and holds the two names equal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lane ran litellm but never republished its port, so a host-side probe hit the compose proxy on the same number and read 502. The lane could therefore never report ready, and the oracle suite skipped itself to a green exit instead of asking a judge. A handler reads the port from the role the way translate.py already does, the runner forwards it, and TOOLS_LANE_KEYS takes it up: compose_file_args layers the override in without checking its variables, so an .env written before the key existed would abort every compose command rather than merely drop the lane.
The gateway's master key is pinned in the lane inventory for the reason the System One key already is: the provisioner would otherwise generate it into a vault the checks cannot read back, and the oracle judge is not a deployed consumer with a virtual key of its own.
serving() stops treating reachability as readiness. A lane brought up before AI_LOCAL_MODELS named a different model keeps serving the old one, and a gateway holding a key these checks cannot use looks healthy from outside while refusing every call they make. It now asks /models with the bearer token and requires the configured alias, and requires the gateway to offer every language the catalogs name, because that advertisement is what the translation run waits on. Without this the lane waved itself through and no change to its roles ever shipped.
Two lane-scoped overrides come with it. The ollama role caps itself at four CPUs, which left the host at 361 percent while twenty cores idled, so the lane gives it the host count the way the engine entry above it already does. And the lane pins the router to its declared order: the decider may send a pair the engine serves to the chat model, which a batch over 183 catalogs cannot pay for. Deployments keep the decider.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generated tcp and http_status probes built their request with echo -e, and echo appends a newline, so the bytes on the wire ended with a stray bare LF after the header terminator of a Connection: close request. uvicorn's h11 reader treats that as trailing data and answers 400 before the handler's 200 reaches the socket, so the probe's grep for a 2xx or 3xx status line never matched and the container stayed unhealthy for good.
In compose the name resolves straight to the container, so an unhealthy but running service still served and only the backup drill noticed, failing with litellm is running/unhealthy after 1200s. In swarm a task joins the service VIP only once it is healthy, so the litellm service sat at 0/1 with no backend and every consumer of http://litellm:4000 got ECONNREFUSED, which is where the openwebui model publishing died.
The tcp flavor had been masking the same malformed request all along, because its grep for HTTP/1 matches 400 Bad Request as happily as 200.
printf is a bash builtin like echo and the probe already requires bash for /dev/tcp, so nothing new is demanded of the image. The new test runs the generated request through bash and compares the bytes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make i18n-translate addressed the engine directly, so a run could never reach more than the 47 languages Argos ships models for, while meta/languages.yml names 183. service_url now resolves web-svc-translate, which routes a pair the engine does not serve to a chat model, and the deploy brings both roles up.
The gateway advertised only what its backends declared, and the chat backend declared nothing, so /languages reported 51 targets of 183 and the readiness wait spun to its hour-long timeout without translating a line. The chat backend now declares every language the table names.
Routing offered libretranslate for every pair, because an undeclared catalogue read as any pair, so a language it holds no model for was answered with HTTP 400 instead of falling through to the next backend. serves() now reads the backend's own /languages, which catalogue() already exposed for the advertisement. Declaring the pairs in the role was the first attempt and carried a second copy that was already stale: the engine reports zh-Hans where the role's list says zh.
A small model copies the prompt markers back, echoes the source line and labels its answer with the target language. All three reached the catalog, and an entry carrying its own source scores below the oracle's middle level. clean() strips them, and only from an answer that carried scaffolding, because on its own a short word and a colon is a translation, not a label.
server() had lost its redeploy parameter while cli/build/i18n kept passing it, so every run raised TypeError before translating a line and the only symptom was an empty log.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sibling container is reached at http://<entity name>:<internal port>,
and that string was written out per call site: a literal protocol, an
entity_name interpolation for the host, and a lookup('config', ...,
'services.<key>.ports.internal.http') for the port. Nine of them across
seven roles, each needing its own `# nocheck: literal-protocol-lookup`,
because the sibling lint rightly objects to a protocol glued in front of
a lookup - the protocol is supposed to come from wherever the TLS
posture is decided.
A container network has no TLS posture. It speaks plaintext by
construction, which is a property of the network rather than of any
caller, so stating it nine times stated it eight times too often.
lookup('container_url', application_id, service_key) holds all three
facts, and because no literal protocol is left at the call site the
suppression markers go with them.
The port is deliberately the internal one. A sibling that goes out to
the published port leaves the container network and comes back through
the ingress, which the dns-search suffix turns into the public FQDN.
The lookup refuses a service that declares no internal http port rather
than falling back to anything.
test_container_url_lookup.py keeps the shape from coming back: a
protocol, an interpolation mentioning entity_name, a colon, a second
interpolation. Nothing else matches that, and `# nocheck:
container-url-lookup` covers an origin this lookup genuinely cannot
express - a port that is not ports.internal.http, or a host that is not
the application's entity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/languages.yml carried `libretranslate: true|false` per entry, which
answered two different questions with one field: which languages the
repository translates, and which engine renders them. The first is the
table's business; the second is not. The gateway offers every backend
that serves a pair, falls through when one refuses or mangles the
answer, drops an engine whose failures pile up, and settles a request
class on the engine that wins its comparisons - and `signature()` is
"{source}:{target}:{length bucket}:{markup|plain}", so that learning is
already per target language. A static column beside it is a second
opinion that goes stale the moment the router learns something.
So the field is gone, 184 lines, and the table now holds name, endonym
and direction. Every language it names carries a catalog in every
domain.
What one engine can do is declared with that engine instead:
services.libretranslate-engine.serves, 47 codes. Not a detour - both
readers run before any engine answers. The lane prefetches one model per
code, and the readiness probe is itself the thing that would have asked.
`translatable()` split along that seam. Catalog-scoped callers take
`domain_languages()`, engine-scoped ones take the new
`served_languages()`: the CI matrix, extraction and translation the
first, the model prefetch and the tuning sweep the second. Collapsing
the two is what this change is guarding against - `baked_languages()`
renders identically before and after, verified by diffing both outputs.
`translate()` also stops dying on a language whose catalog has not been
extracted yet. It reads a missing catalog as nothing pending rather than
raising, because 136 docs catalogs do not exist until the next
extraction and refusing those languages would quietly shrink the set the
repository promises to translate.
The oracle suite follows: tests/oracle/test_translation_score.py asks
utils/i18n/judge.py about every language instead of routing a laya
checkpoint per language. It cannot run yet - the tools lane carries no
LiteLLM - and its skip message says exactly that.
test_catalogs_complete_on_gpu now measures against the table rather than
one engine's coverage. It is live on a GPU host and will report the 136
missing catalogs as the gap they are, which is the point: narrowing it
to the engine would hide exactly the languages worth watching.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The oracle suite needs a verdict in every language meta/languages.yml
names, and laya cannot give one. Measured against the suite's fixture,
its english checkpoint separates a faithful rendering from an unrelated
text in 17 of 47 languages and its multilingual checkpoint in none,
answering around 1.5 either way. One judge for all of them beats a route
per language that is noise in most of them.
The question keeps the shape the checkpoint answered: the levels are
written out and the model names an index over them, never a bare
probability. A probability carries this deployment's calibration, and a
fixed threshold on it would encode that calibration into the suite;
"below the middle level" instead means what the middle level says.
Three refusals are deliberate. temperature 0, so the same comparison
does not come out differently on a rerun. An answer naming no index
raises instead of scoring zero, which would read a refusal as "states
something else". An index outside the scale raises instead of being
clamped, because a model answering 7 has not placed the pair on this
scale at all.
The request goes to the model gateway rather than to a model server, so
it carries a consumer identity and resolves through the gateway's model
list like every other model call here.
Nothing imports this yet: the oracle suite moves to it once the tools
lane carries LiteLLM and a loaded model, which is what lets the judge be
exercised against a real one for the first time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.
The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.
call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two jobs of run 39040406916 died with the runner's own annotation, 'The
hosted runner lost communication with the server'. Both left every
diagnostic step pending, so no artifact exists for either. For the swarm
one the in-band sampler ruled starvation out — 9324M of 15947M available,
stall60 0.0%, 25 seconds before the cut — and nothing in the repository
can tell the remaining causes apart.
A stuck NFS mount wedges tasks in uninterruptible D state, which shows as
I/O pressure while memory pressure stays flat, and this drill runs its own
NFS node. /proc/pressure/io and /proc/pressure/cpu were read nowhere in
the repository. They are now sampled beside memory, with the D-state count,
every ten seconds instead of thirty, and an I/O stall at or above half the
minute aborts the step while the runner is still alive to upload the
banner.
cpu reads 'some', not 'full': the full line is pinned at 0.00 system-wide,
measured 13.50 against 0.00 on this host, so reading it like memory would
have shipped a metric that can never move.
Dump runner state was switched off for swarm, so the one post-mortem that
survives a step failure never ran there. It runs now, and the swarm drill
gets the step timeout its compose sibling has had all along — 350, the
number test_distro_budget_fits_timeout already derives the distro budget
from.
The sampler moved to its own module: the additions took matrix.py past the
file-size cap, and the lint asks for a split rather than a marker.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An unmounted VOLUME does not fail: docker creates an anonymous volume for
it, the container works, and the gap shows up later as a volume nobody can
attribute. sys-ctl-hlth-volumes fails the deploy over exactly that, and its
whitelist keys on volume ids, which are new on every deploy, so the gap
cannot be suppressed there either. e0b973f9bb fixed one such case in
web-svc-xmpp by hand; this is the check that would have found it.
Attribution is the whole difficulty. meta/services.yml declares an image
under a key, and the compose file often runs it under another name:
web-app-jitsi declares the image under 'jitsi' and renders the service as
'web', web-app-erpnext runs one image as seven services, and the two RDBMS
consumer keys name no service of their own because the container comes from
sys-svc-rdbms. The sweep therefore unions the entry key with its
proxy.service_key and delegates the RDBMS keys to their provider role. A
first version keyed on the entry alone and reported six roles that mount
every path correctly, web-svc-xmpp among them.
Reading a config blob section is now its own module beside the registry
client, because the two readers were the same walk twice and registry.py
sat three lines under the file-size cap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three faults, one mechanism each, that together let the oracle suite report
a verdict it never reached.
The address. lane_url handed out the runner's compose service name, but the
stack's containers resolve through the deployment's CoreDNS, which never
learns one: getent hosts tools answers nothing inside them, so every check
that asked the lane from there skipped itself. The bring-up now writes the
runner's address, picked as the one inside INFINITO_SUBNET because the
runner sits on every network it was ever brought up on, and the checks read
it from there. It is rewritten on both paths, because a restart gives the
runner a new one.
The readiness. serving judged the engine alone, so a lane whose engine make
i18n had already brought up counted as up while System One was down. It now
probes every role the caller names, which is why the runner republishes
System One's port: a container IP is unreachable from the host that decides
whether to deploy.
The verdict. A silent lane made the suite skip, and a skip exits green.
Only a deploy that is already running leaves the stack alone and exits 0
now; a crashed deploy and a deployed-but-silent role both exit 1.
The override's third :? guard is added to the gate that decides whether the
file is layered in at all, next to the one that was already missing. Without
it an .env written before these keys aborts every compose command, down to
compose-down, instead of dropping the lane. A test now keeps that list and
the override's guards in step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every cold render of the applications SPOT walks the templates of all
269 roles, and those templates call lookup('config'), lookup('tls'),
lookup('domain') and others. Each of these re-enters
get_merged_applications while the render guard is set, and each one
rebuilt the unrendered view from scratch: a deepcopy of all role
defaults plus the override merge, about 8 ms, never cached. A local
cold render of the development inventory made 351 such calls, 2.82 s
of a 6.86 s render.
The outer render now keeps a copy of its unrendered view and its cache
key on the render guard, taken before the inventory volumes are
popped, so it is the same view a nested call used to build. A nested
call with the same key returns it; any other key still builds its own.
The guard drops it when the render ends.
Measured on the same inventory: the nested calls took 0.07 s instead
of 2.82 s and the cold render 3.60 s instead of 6.86 s. The rendered
SPOT is unchanged: the old and new dumps differ in 69 user passwords,
which the development inventory regenerates on every read (two runs of
the old code differ in 68), and in one absolute checkout path, the only
other difference.
In CI each cold render took 40 to 77 s and the heaviest swarm cell of
#950 missed its budget by a few minutes (#984).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
4aba897ca2 gave the pull and build subprocess a 3600s ceiling so a hang
could not stall a deploy. A pull of a large image set legitimately outlives
any fixed ceiling while it is still making progress, and the ceiling then
kills a healthy run.
run_streamed measures the gap between two output lines instead. Two reader
threads keep stdout and stderr separated the way run_cmd does, because
compose writes its warnings to stderr and merging them poisons every parser
downstream. Only the two calls that legitimately run long use it; the short
calls whose output is parsed keep run_cmd.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils/yaml_bootstrap.py promises that load_block(text) either equals
yaml.safe_load(text) or raises BootstrapYamlError. A fuzzer comparing
both found two inputs that broke the promise:
- str.splitlines() also breaks lines at \x0b, \x0c and \x1c to \x1e,
which PyYAML's reader rejects as non-printable. "\x1e/: R" came back
as {"/": "R"}, and a non-printable character inside a comment line was
skipped with the comment.
- An indented " ---" on line 1 was taken as the document marker, while
PyYAML reads it as a plain scalar and rejects the mapping that follows.
load_block now refuses any character outside PyYAML's own printable
class before it splits lines, and takes "---" as the document marker
only at column 0. The line breaks splitlines() still honours (\r, \r\n,
NEL, LS, PS) are line breaks for PyYAML too, so splitting stays as is.
The eight inputs join STRUCTURES in the divergence test; against the
previous reader, four of them diverge and fail it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The full gate failed test_a_distro_needs_the_slowest_run_plus_headroom_to_start on a quiet machine, so the load is the gate own parallel workers rather than anything beside it. The branch under test needs d < B < 2.15*d for a measured first duration d, the rule 040955ebf2 wrote down when it widened the same fixture from sleep 2 / 3s to sleep 4 / 7s. At 4 / 7 the window admits a first iteration of 4 to 6 seconds, three seconds of slack, sized then against the single second CI had overshot by. Under the gate it overshoots further, takes the budget exhausted branch and never prints the line the test greps for. At 10 / 19 the window admits 10 to 18 seconds, nine seconds of slack. The assertion and the branch are unchanged and the suite pays six more seconds.
The second hunk shortens the Odoo reload task name to 60 characters; at 83 it broke the 80 character convention the name lint enforces.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge left the role with two ways to the same end. The wired shell path called setup.py update-user, so it could neither create a missing account nor clear a lockout, and it decided its own idempotency from a sha256 marker in the persistent volume, which lies after a restore: marker matching, database not, reported unchanged and did nothing. The python converger from 0709de6fa1 queries the application instead, creates what is missing, verifies the password, unlocks, resets the failed-login counter and ensures the Administrator role, and it carried the only tests either path has; it was wired to nothing after the merge resolved towards manager_ops.yml. That engine now serves the surviving wiring: renamed into the family spelling the sibling roles use, reading the env names the task already passes and emitting the markers the task already parses. tasks/administrator.yml had no caller anywhere in the tree and goes with it.
Trade-off worth knowing: the deleted path used the supported pgAdmin CLI while this one uses internal APIs, so a pgAdmin upgrade can move it. Neither path has ever executed in CI, the pgadmin job died at the health wait before reaching the realign, so the choice rests on capability and tests rather than on field evidence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A recover mirrored a snapshot with rsync -a, which transfers no extended attributes, so a restored volume came back byte-identical while every application that keeps identity in xattrs read the tree as unknown. OpenCloud posixfs is the case that surfaced it: after the DR drill its storage log reports node.Xattrs on the platform administrator space as no data available, the showcase files are assimilated as new objects, and the graph call that lists drives returns no personal drive, so the design spec dereferences undefined. Measured here, source tagged with user.oc.name, on tmpfs and on ext4 alike: rsync -a returns 0 and drops the attribute, rsync -aHX and rsync -aHAX return 0 and carry it. The shared restore flow and the NFS half of this repo now use -aHAX, the form ensure_subject_subvolume.sh already uses, continuing what 69321d4135 started for numeric ownership.
The backup half still has the same gap: backup-docker-to-local 7.0.2 builds its command as rsync -aP --no-D --delete --delete-excluded with no flag passthrough, so until it carries -X and the pin moves, a snapshot holds no attributes for this restore to put back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard patched the silence budget down to 0.3 s, but the watchdog starts its clock at Popen while python -c still has to boot before it can print, so under the parallel load of a full gate run the startup alone exhausted the window and the process was killed before its first line. The same file passed in four seconds on its own. The budget is now 1.0 s against a process that prints every 0.2 s for three seconds, which keeps all three conditions true at once: the total run outlives the budget so the old total-runtime timer still kills it, the gap between lines stays under it, and so does interpreter startup. Verified against the old shape in isolation: it dies with signal 9 after five of fifteen lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The watchdog armed a single timer at process start, so it killed the German build with SIGKILL while that build was still emitting progress, and Library.failed() then answered 500 for the rest of the run. The container was not out of memory: OOMKilled is false, the restart count is zero and neither the kernel ring buffer nor the journal records a kill, while the sphinx worker BrokenPipeError that follows is the signature of killing the parent alone. The watchdog now stamps the time of every line read and kills only after a full silence window, which the existing test name already described. A regression guard runs a process that keeps printing past the deadline and requires its last line in the log.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`deadline` is whole seconds from `date +%s`. A run that starts just
before a second boundary reaches the deadline after the first probe and
one `sleep 1`, so the gate correctly gives up after one probe while the
test demanded two. With a stubbed clock (1000, 1000, 1002) the gate
probes once and prints `deadline of 2s reached`. The deadline message
already proves the pending migration held the entrypoint back; require
at least one probe.
Reported by CodeRabbit on #979.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Swarm Nextcloud#3 on main 748189e890 (job 117182318941) wedged the
OpenProject schema: the convergence check gave up after 74 attempts
while the seeder kept exiting 1, first with
`relation "ar_internal_metadata" already exists` from its structure
load, then with `relation "announcements" already exists` on every
`db:migrate`.
The seeder (`docker/prod/seeder`) loads `db/structure.sql` through
psql, statement by statement, when `\dt` finds no relation. In swarm,
`depends_on` is ignored, so web and worker start alongside it and poll
`await_migrations.sh`. That gate called
`rails db:abort_if_pending_migrations`, which runs
`migration_context.open`, i.e. `Migrator.new`, whose constructor
executes `schema_migration.create_table` and
`internal_metadata.create_table` (Rails 8.0.4,
activerecord/lib/active_record/migration.rb:1430-1431). The seeder
saw an empty database at 05:33:20; by 05:33:38 the gate had created
`ar_internal_metadata`, the structure load aborted at line 255 with
everything alphabetically before it already committed, and every
later attempt took the migrate path into tables that already existed.
Probe with `rails db:migrate:status` instead. It aborts while
`schema_migrations` is missing and otherwise only reads the recorded
versions (database_tasks.rb:303, migration.rb:1318); the gate treats
any `down` row as pending.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two conflicts, both resolved by combining rather than by picking a side.
`roles/web-app-socialhome/tasks/02_design.yml`: designer restructured the pass,
moving the token request and the theme write into the new
`02_design/converge.yml` and adding a stat of the name-state file. HEAD still
carried the converge block inline. Kept designer's stat, dropped HEAD's inline
block: the merged body below the conflict already includes `converge.yml` and
reads `socialhome_name_marker`, which only designer defines, so keeping HEAD's
side would have duplicated the token request and left that variable undefined.
HEAD's `retries: 0` on the token request is deliberately not carried over; the
default of 30 retries covers a service that is not yet accepting connections,
which matters more here than failing fast on bad credentials.
`roles/web-app-fider/files/playwright/test-design.js`: the import line takes
both sides, since the merged body uses designer's `apiGetOnion` and HEAD's
`requireDotenvValue`. The env block keeps HEAD's `requireDotenvValue` form,
because designer's `|| ""` defaults are what
`tests/lint/ansible/.../test_unguarded_env_default.py` exists to reject, and
drops HEAD's module-scope `title` and `logoEnabled`: the merged body defines
`title` locally and reads `DESIGN_LOGO_ENABLED` directly, so both were dead and
would fail `no-unused-vars`.
Checked after resolution: no conflict markers (`git diff --check`), no `nocheck`
line dropped by the merge, and no duplicated YAML key in the seven merged files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without `--all`, `journalctl --output=json` encodes a field over 4096
bytes as `null`, so a long host error matched nothing and the check
passed. A repeated `MESSAGE` field arrives as a JSON array of its
values, and `bytes()` on that array raised `TypeError`, which aborted
the whole check.
Pass `--all`, and tell a byte array (all integers) from an array of
field values, decoding each value on its own.
Reproduced against the host journal: a 5000-byte error line read back
as `"MESSAGE":null` and produced no match; an entry with two `MESSAGE`
fields, sent over the native journal socket, read back as
`["error first","error second"]` and crashed `host_errors` with
`TypeError`. With the fix both are reported.
Also clear the two test-lint findings of this PR:
`test_comments_valid` rejects the narrative comment above the
`journalctl` Popen, so it goes; the stderr-to-file choice stays, because
an unread stderr pipe that fills up would block `journalctl` while
stdout is still being read. `test_no_uncached_file_reads` flags the
read of the stub's argument file, which lives in the test's tempdir and
is written per run, so the path-keyed `utils.cache.files.read_text`
could serve a stale body; the read opts out with
`# nocheck: cache-read tempdir fixture`, as the other stub-driven
script tests do.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`lookup('tls', ...)` aligns a cross-app domain to the consumer named by
`application_id` (utils/tls_common.py `align_domain_to_consumer`): a
clearnet-primary consumer gets the clearnet sibling of an onion-primary
target. The carrier key does not cover `application_id`, so a park that
renders inside a role hands that role's alignment to every later role
whose key matches.
The constructor and `tasks/utils/setup/role.yml` park at play scope,
where `application_id` is undefined. The re-park in
`roles/sys-token-store/tasks/write.yml` (67f6c8d7e8) and the nested-load
re-park render inside the caller: `svc-prx-openresty` at the preload
loader, `web-app-stalwart` or a provider role on the NFS host, the web
app itself under `sys-stk-front-base`. In a tor deploy that turned onion
targets into clearnet domains for onion-primary roles served from such a
park, and baked the caller's group into Nextcloud's sociallogin
`groupMapping`.
Drop `application_id` from the render's variables and from the
templar's for the duration of the render, so every park carries the
play-scope result the constructor parks have always carried. Rename the
nested re-park task so its log lines do not read as a role boundary.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
