The watchdog armed a single timer at process start, so it killed the German build with SIGKILL while that build was still emitting progress, and Library.failed() then answered 500 for the rest of the run. The container was not out of memory: OOMKilled is false, the restart count is zero and neither the kernel ring buffer nor the journal records a kill, while the sphinx worker BrokenPipeError that follows is the signature of killing the parent alone. The watchdog now stamps the time of every line read and kills only after a full silence window, which the existing test name already described. A regression guard runs a process that keeps printing past the deadline and requires its last line in the log.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
After a version build the builder queued every translated language, not the
configured ones. The list now reaches the library: server.py reads
DOCS_I18N_PRELOAD_LANGUAGES, Library freezes it into a set that drops the empty
string ''.split(',') yields when nothing is configured, and builder.py queues a
language only when it is in that set. Every other translation is built when a
visitor opens it.
The warm-up task accepted 202, which means queued, so the deploy walked on
while the sites were still building. A second task now waits for 200. Both read
DOCS_I18N_PRELOAD_PATHS, hoisted out of the loop it used to be written inline
in, because two tasks need the same language-code-to-URL-path mapping.
DOCS_BUILD_PARALLEL divides by overhead + 2 * process_mb instead of
overhead + process_mb: with a translation building beside the version build a
parallel slot holds two process-sized allocations. The new divisor is strictly
larger, so the value can only fall, never rise; with the configured 3072 and
2048 MB it differs from the old one only between 10 and 16 GB of service
memory, where it goes from 2 to 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve jobs of run 37865281419 failed on the docs language spec:
tests/test-languages.js:25:7 the deployed working tree is published in de
with a language switcher (54.6m)
Error: Expected the de site of the deployed working tree to be built
The spec polls /deployed/de/ until the suite budget runs out, so a build that
hangs is reported as a spec timeout with nothing to read. None of the five
subprocess calls the build makes carried a timeout, so any of them could hold
the queue open forever:
library.py _git() rev-parse, tag --list and remote update --prune
library.py fetch() git clone --mirror
i18n.py each generator of generate_commands
i18n.py sphinx -b gettext
builder.py _run() the sphinx build of one version
The two git paths reach the network, which is where an unbounded call hangs
without failing.
`_run` streams its child's output into the build state, so it cannot take
`communicate(timeout=)` without losing the progress that `_append_log` and
`_save_state` record, and Popen has no timeout of its own. A threading.Timer
kills the child instead. The kill lands in the existing returncode branch, so
the build is reported through `_failed`, which already writes the log tail into
the state file.
RUN_TIMEOUT_SECONDS is set to catch a hang, not to beat the spec's clock: no
measurement of a successful de build exists, so a tighter value risks turning a
slow build into a failing one. With one, the value belongs below the poll
window the spec scales to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jobs and parallel were derived from the host CPU count with no memory term,
so a 4g container could be asked to run two builds of two sphinx workers
each. One sphinx process reached 2.39 GB anon-rss while still growing, and
the kernel killed two of them ("Memory cgroup out of memory: Killed process
... (sphinx-build)", cgroup usage 4194304kB of limit 4194304kB). Every build
died there, so every page kept answering 202 until the spec gave up.
mem_limit now follows the host, and both knobs follow mem_limit: parallel
takes the builds the limit can hold, then jobs takes the workers left once
every concurrent build has reserved its overhead. Deriving jobs first let it
claim all the memory and pinned parallel at one; reserving the overhead only
once overcommitted an 8-core host by 683 MB.
The two inputs are declared beside the limit they are measured against.
Neither is a ceiling: no docs build has ever completed in CI, so
sphinx_process_mb is a lower bound that the first green build can correct.
mem_limit calls lookup("resource", ...) directly rather than through a
group_var, the way svc-opt-swapfile does: a meta file renders with a flat
variable dict, which cannot expand a group_var that itself holds a lookup
and would have left the limit at 0.
The generator env also lacked the PYTHONUNBUFFERED that _sphinx_env sets,
so a generator killed under that memory pressure lost whatever its stdout
buffer still held. That is why the integrations failure logged its argv and
nothing else.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 37553816857 failed 59 jobs, and web-app-docs is in five of every six
that named a role. The spec waits 88 minutes for a German site that never
arrives:
tests/test-languages.js:25:7 › the deployed working tree is published in de
Expected the de site of the deployed working tree to be built
Expected: 200 Received: 202
The container log of any job that deployed the role carries the whole chain:
"GET /deployed/\1/ HTTP/1.1" 202
build latest failed: Command '[... 'infinito_docs.generators.integrations' ...]'
returned non-zero exit status 1.
"GET /deployed/de/ HTTP/1.1" 202 (every 31s, for 88 minutes)
Four defects, in the order they fire.
The preload asked for a path named `\1/`. The replacement reached re.sub with
its backslash still escaped, so the backreference stayed literal and no
language was ever warmed; the first build then started when a visitor polled.
Reproduced through Templar rather than argued: the old expression renders
['', '\1/'] and the new one ['de/', '']. Plain Jinja2 with Ansible's filters
renders the old one correctly, which is why the first measurement of this
cleared code that the access log had already convicted. The replacement
appends with '$' and carries no backreference to escape.
`latest` then failed to build at all. builder.py puts the documented checkout
first on PYTHONPATH, so the image's generator meets the utils of the version
it is documenting, and a05071fe78 dropped the get_ prefix from entity_name.
A builder that documents every version has to tolerate every version's API,
so the import falls back.
A failed build kept answering 202 and re-queueing itself, so every client
waited its whole timeout and a doomed build restarted on each poll. This is
what turned a one-minute failure into 88-minute waits and drove one job into
the six-hour ceiling, where it was killed with 137. The server now answers
500, through a queue accessor that reuses the comparison `request` already
makes: a failure at an older ref still retries. For a translation the parent
version counts too, because here /deployed/de/ polled while `latest` was the
build that had failed.
The failure line now carries the captured log tail. The detail was never
lost -- _run merges stderr into the build state -- but it was absent from the
container log, which is what a rescue bundle holds, so the traceback had to be
reproduced locally to be read at all.
Nothing here is deployed. The 500 is a live change for three consumers that
accept 202 -- the preload task, test-front-page and test-on-demand-build --
but it is reachable only when a version is unservable AND its build failed at
the ref it would build now, so a healthy deployment sees what it saw before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make autoformat reflowed a resource_filter call, a generator expression and five test files it touched along the way. The new architecture test also carried the two findings the first gate run stopped on: three RUF012, where a TestCase class attribute needs ClassVar, and two PERF401, where an append loop becomes list.extend over a generator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A translated site repeated the whole checkout and generation of its version, and the builder ran one marker at a time. The prepared tree is now cached per (ref, tooling digest), hardlinked into each build's scratch and sealed read-only, and a bounded thread pool claims the markers a free slot can build now - a language only once its version is current, so a stale one is no longer claimed and bounced in a spin.
services.docs.build.jobs and services.docs.build.parallel replace build_jobs and default to half the host's CPUs, parallel capped at 2 so the two knobs cannot multiply past mem_limit. DOCS_I18N_PRELOAD_LANGUAGES replaces DOCS_I18N_LANGUAGE, defaults to the host language and warms [en, de] in both matrix variants, where docs_sites is backed up under swarm so the DR drill restores the built sites instead of rebuilding them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The scratch directory of a translated build was named
"{version}{QUEUE_SEPARATOR}{code}", and QUEUE_SEPARATOR is ":". _generate then
exports PYTHONPATH as os.pathsep.join([src, package_dir.parent]), and
os.pathsep on POSIX is also ":", so a path like /scratch/deployed:de/src split
into /scratch/deployed and de/src and the generators could not import the
checkout.
The untranslated build was never affected: build() uses `self.scratch /
version` with no separator in it. Only the language path carried the colon,
which is why the German site was the one that failed.
QUEUE_SEPARATOR stays as it is. It still separates version from language in
the queue marker, which build_queue partitions on; the directory name was the
only place where that character had to be something else.
The language spec also stops retrying and stops recording: the test budgets
90 minutes for two Sphinx builds, so a retry costs three hours of runner for
no new information and a trace over that span is an artefact nobody opens. It
now runs before the on-demand build test rather than after, so the long build
is not queued behind the test that drives the same machinery.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
build() runs eleven generator subprocesses before Sphinx; build_language() went straight to sphinx-build. The translated site therefore rendered without generated/ - no role pages, no overview, no regenerated READMEs - while the .pot extractor does call the generators (i18n.py:47). The catalog described a superset of what the translation could ever show. The loop moves into _generate() and both builds call it. Extracting it also surfaced a division by zero on an empty generator list, now an early return.
build_language() collapsed two different reasons into one bail and treated them alike: drop the marker, request the base. Dropping is right when the code is not translated. It is wrong when the base is merely stale, because the base build short-circuits and never reaches the tail that re-queues the languages, so the translation was lost for good and /deployed/de/ would answer 202 forever. The two reasons are now separate and the stale case re-queues the translation alongside its base.
fetch() re-requests every 900 s and _current() can never be true for an unservable site, so a failed build rebuilt itself forever and starved the background lane. _failed now records the ref it failed on, and request() refuses a BACKGROUND entry for a marker that failed at the ref it would build now. A waiting visitor still gets a retry; only work nobody waits on is throttled. The guard needs a non-empty ref so an empty snapshot ref cannot lock everything out, and _ready replaces the whole state dict, so a later success clears it.
Verified: 158 passed over the roles entire docs unit suite in the container. The extraction is behaviour-identical on the build() path - same PYTHONPATH, phase, progress formula and run call - so the measured delta there is zero. Not verified: build_language has never executed the generator phase anywhere yet, which is the one real risk this carries.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`/snapshot` is a tar of `DOCS_SNAPSHOT_PATHS` and the list carried 32 entries,
neither of them `Makefile` or `aliases`. Both are tracked at the repository
root and `commands.py:80-90` hands them to two generators,
`make_targets --makefile {src}/Makefile` and `aliases --aliases-file
{src}/aliases`. `generators/make_targets.py:45` reads with no existence guard,
so the build dies with FileNotFoundError before sphinx is reached.
It dies without saying so anywhere. `_failed` wrote the exception into
`states/deployed.json` and printed nothing, and `library.status` iterates
`[LATEST, *tags]`, so `deployed` never appears in `/api/versions`. No log, no
API and no test carried the failure. `/deployed/de/` therefore answered 202 for
as long as the container lived and the Playwright suite polled a dead build for
its entire budget - eight jobs of run 36651124458, three hours each.
The failing test's trace settles which build it was: the 202 body carries
`<title>Building deployed</title>`, not `deployed/de`. The English `deployed`
version is the one that never lands, so the German build is never even queued.
That also gives the discriminator: `latest` and the release tags build from a
`git archive` of the mirror, where both files are present, and they reach
`ready` in the same run.
Three parts. The list gains the two entries, in the file's ASCII order - that
is the fix. `_failed` also prints to stderr, because without it the next such
gap is just as invisible; the moodle job's docs container log was 484 access
log lines and nothing else. And a test derives the requirement instead of
restating it: it calls `generate_commands()` and requires every first path
segment under `src` to be in the snapshot, `generated` excepted, since that one
the build creates. Adding two entries by hand would have broken the same way on
the next generator.
Main is unaffected: the role does not exist there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The replay of a role's "### Production" block moves out of the CI guide
scripts and into the documentation role itself. The block is read off the
rendered page over HTTP, so what runs is what a reader copies, and the role
that publishes the page is also the role that proves it.
scripts/github/guide/{one,compose_deploy,host_deploy}.sh and
cli/build/docs/readme/production.py are gone with the workflow step that
called them. The instructions marker now only adds web-app-docs to the row's
apps; test-e2e-cli discovers the role through its new templates/test.env.j2
and runs files/test/test.sh.
The block runs in a throwaway machine, never on the stack host. compose.yml
starts one service built from files/test/Dockerfile, whose base is
INFINITO_PARENT_IMAGE out of the mounted checkout's .env, so the machine is
the stack host's own distribution and the replay exercises the installation
instead of a prebuilt image. files/test/init.sh adds an init when the base
carries none and keeps the full -Syu that ef0f03ea98 established for arch.
scripts/tests/workspace/base/01_install.sh then prepares the machine before
the block runs.
Nothing is skipped any more. The previous host-install branch exited 0 and
passed silently over 65 of the 178 invokable roles; a block without a
"docker run" is now replayed against systemd in that same machine, with the
git clone dropped in favour of the mounted tree under test.
The target role comes from the new guide_role lookup, which reads
list_invokable_app_ids() rather than matching role-name prefixes, and falls
back to the asking role when the round deploys nothing else. deployment.py
grows running_apps() so both lookups share one definition of what the round
deploys. required_by_coverage.role_is_invokable drops its own walk over
categories.yml and delegates to the same SPOT; measured equal across all 280
roles.
Carried in the same index, because the refactor pass ran over this change
set: library.py splits into sites.py and build_queue.py mixins, builder.py
hands translated_languages to catalogs.py and folds three repeated log-tail
and state writes, and placeholders.py becomes a facade over spans.py,
repairs.py and damage.py. No test changed for those splits, and the two
ref-resolution dicts builder.py carried are replaced by _wanted_ref, which
also stops hashing the deployed tree on every tag build.
Verified: 7132 unit and lint tests pass over the whole tree, the guide_role
lookup renders through a real Templar, the base image resolves from .env,
and compose config validates the service. The replay itself has never been
executed; that needs a deploy of web-app-docs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cosmos, Quick Setup and Credits follow from meta/services.yml, categories.yml
and galaxy_info.author, yet 279 role READMEs carried a copy that went stale
whenever the metadata moved and nobody re-ran make docs. The documentation
build now runs the generators itself, against its own scratch checkout, and
the committed READMEs keep only the prose a person writes.
utils.cache.PROJECT_ROOT derives from the module's location, so putting the
checkout first on PYTHONPATH makes the generator operate on the scratch tree.
i18n.py and builder.py add it; the one remaining coupling, a relative_to over
PROJECT_ROOT in a log line, is gone.
CI extracted the "### Production" block from the committed README with awk.
Stripping the section made guide_deployable() return "" for every role, which
selects no deploy row and turns nothing red: a fail-open regression. It now
asks _is_role_invokable, which matches the old condition exactly (178 roles
either way, both difference sets empty), and cli/build/docs/readme/production.py
renders the block for the two guide scripts. All 178 renders are byte-identical
to what HEAD served, and both scripts' extraction pipelines produce identical
output for a web app, a service and a desktop role.
test_readme.py inverts: the three sections are now a hard failure in a
committed README. readme_md.md documents what each is derived from.
Removed: the docs, cosmos, readme-generate, readme-index and readme-check
targets, the five aliases pointing at them, and docs from make quality.
Verified by running generate_commands() in series against a copy of the
checkout: every command exits 0, no README is left without Cosmos or Credits,
and exactly the 178 invokable roles get a Quick Setup. The ansible_roles step
needs pandoc, which only the docs image carries, so that one link of the chain
is unexercised here; it is unchanged by this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
request(..., background=True) skipped a language as soon as translation_servable() found an index.html, which is an existence check with no ref in it. A language was therefore built exactly once: every later commit advanced the English site while every translated site stayed frozen on the commit it was first built from. The eager loop this replaced rebuilt them all, so the freeze arrived with f5eff500c1.
_publish_site already stamps the ref beside each translated site and nothing ever read it. translation_ref() reads it and _translation_current() compares it against _wanted_ref(), which also replaces the eager {LATEST: head, DEPLOYED: snapshot_ref()} dict so an unreadable snapshot no longer kills the builder while building an unrelated version.
Two more from the same review: build_language wrote into the version's state file, so a failing language reported the whole version as failed and leaked its sphinx log into /api/versions; it now uses its own version:code key. And the language index was written after the site was published, so a failure in between left a current site with a stale index, which _current() rejects forever and fetch() would rebuild on every tick; the index is written first.
library.py had grown past the 500-line cap, so the build machinery moves into builder.py as a Builder mixin. An AST diff proves every function, method and constant is byte-identical apart from _write_json -> write_json, and the six build-focused test cases move to test_builder.py with the count unchanged at 16.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5