After a version build the builder queued every translated language, not the
configured ones. The list now reaches the library: server.py reads
DOCS_I18N_PRELOAD_LANGUAGES, Library freezes it into a set that drops the empty
string ''.split(',') yields when nothing is configured, and builder.py queues a
language only when it is in that set. Every other translation is built when a
visitor opens it.
The warm-up task accepted 202, which means queued, so the deploy walked on
while the sites were still building. A second task now waits for 200. Both read
DOCS_I18N_PRELOAD_PATHS, hoisted out of the loop it used to be written inline
in, because two tasks need the same language-code-to-URL-path mapping.
DOCS_BUILD_PARALLEL divides by overhead + 2 * process_mb instead of
overhead + process_mb: with a translation building beside the version build a
parallel slot holds two process-sized allocations. The new divisor is strictly
larger, so the value can only fall, never rise; with the configured 3072 and
2048 MB it differs from the old one only between 10 and 16 GB of service
memory, where it goes from 2 to 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve jobs of run 37865281419 failed on the docs language spec:
tests/test-languages.js:25:7 the deployed working tree is published in de
with a language switcher (54.6m)
Error: Expected the de site of the deployed working tree to be built
The spec polls /deployed/de/ until the suite budget runs out, so a build that
hangs is reported as a spec timeout with nothing to read. None of the five
subprocess calls the build makes carried a timeout, so any of them could hold
the queue open forever:
library.py _git() rev-parse, tag --list and remote update --prune
library.py fetch() git clone --mirror
i18n.py each generator of generate_commands
i18n.py sphinx -b gettext
builder.py _run() the sphinx build of one version
The two git paths reach the network, which is where an unbounded call hangs
without failing.
`_run` streams its child's output into the build state, so it cannot take
`communicate(timeout=)` without losing the progress that `_append_log` and
`_save_state` record, and Popen has no timeout of its own. A threading.Timer
kills the child instead. The kill lands in the existing returncode branch, so
the build is reported through `_failed`, which already writes the log tail into
the state file.
RUN_TIMEOUT_SECONDS is set to catch a hang, not to beat the spec's clock: no
measurement of a successful de build exists, so a tighter value risks turning a
slow build into a failing one. With one, the value belongs below the poll
window the spec scales to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A translated site repeated the whole checkout and generation of its version, and the builder ran one marker at a time. The prepared tree is now cached per (ref, tooling digest), hardlinked into each build's scratch and sealed read-only, and a bounded thread pool claims the markers a free slot can build now - a language only once its version is current, so a stale one is no longer claimed and bounced in a spin.
services.docs.build.jobs and services.docs.build.parallel replace build_jobs and default to half the host's CPUs, parallel capped at 2 so the two knobs cannot multiply past mem_limit. DOCS_I18N_PRELOAD_LANGUAGES replaces DOCS_I18N_LANGUAGE, defaults to the host language and warms [en, de] in both matrix variants, where docs_sites is backed up under swarm so the DR drill restores the built sites instead of rebuilding them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
On a cold container the docs Playwright suite triggers four serial Sphinx builds and waits for the last one. test-languages.js polls /deployed/de/ for 200 and got 202 for 55.6 minutes in job 110003713825 of run 36745666606, after which the suite budget expired and five further specs never ran. The suite has never passed on a first attempt; it used to go green only on the harness retry, which 7127c6b96b removed for budget timeouts, so a structural overrun became a red job.
Measured cost per build, from run 36558500327 where the whole queue drained on a kept stack: container up 12:15:52, the spec green at 13:21:41 answering on its first poll, next container start only at 13:30:45. Four builds in 65 minutes 49 seconds, so at most 16.5 minutes each - not the sixty to ninety minutes 40688e4ded estimated.
Three of the four builds were none of the test's business. fetch() queued latest in the foreground although nothing waits on it at startup, and _refresh_stale_sites did the same for every servable version; both now use the background lane, which is what its own docstring describes. test-front-page.js reached GET /latest/, and reaching an unbuilt version queues its build - it now asserts the redirect from the Location header without following it, and checks the same page shape against /deployed/, which is built anyway. The release tag that test-on-demand-build.js clicks stays, and now runs behind the awaited work instead of in front of it.
The translation could only be queued after its base finished, because the 202 branch returns before _translation is reachable, so it always sorted behind whatever the sibling specs had queued meanwhile. The 202 branch now carries the language segment along, admitted by translatable(), which answers from the snapshot catalog on disk before any build; build_language still re-checks the written index.
Moving a version into the background lane exposed that status() read only the foreground one, so a waiting version would have been reported as missing rather than queued to the versions overview and to /api/versions. is_queued() now covers both lanes, mirroring _dequeue, which always removed from both. Two of the five unit failures this change first produced disappeared on that fix alone, which is what shows they pinned real behaviour; the three that pinned the lane itself were rewritten to assert the behaviour instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The site answers 202 while it builds a version, and 200 once a build
exists. Whether that build matches what the version currently names was
never asked at the serving gate: servable() only checks that an index.html
is on disk. So the first request after a new snapshot got the previous
build with 200, and the rebuild it triggered finished into a directory
nobody was waiting for.
For a release tag that cannot happen, and for latest it is a reasonable
trade, since a visitor would rather read the previous page than a spinner.
For deployed it is wrong by definition: the name means the working tree
this run staged, so an earlier snapshot is not a stale answer but a
different question's answer. The post-deploy test that reads the published
page therefore validated the round before it, and reported a role broken
whose page had already been fixed.
The comparison already existed. _current() weighs built_ref against
_wanted_ref, which resolves deployed to the snapshot digest, and the
builder consults it to decide whether to rebuild. Only the server did not.
outdated() exposes it for deployed alone and the serving branch now treats
a mismatch as not yet servable, so the poll waits for the build it asked
for.
Measured on the stack while the previous round was live: the served page
was written 16:44:41 from the older snapshot, its own source carried the
newer content, and the correct build landed at 17:10:56, twenty-six
minutes after the test had already read the stale one and failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The replay of a role's "### Production" block moves out of the CI guide
scripts and into the documentation role itself. The block is read off the
rendered page over HTTP, so what runs is what a reader copies, and the role
that publishes the page is also the role that proves it.
scripts/github/guide/{one,compose_deploy,host_deploy}.sh and
cli/build/docs/readme/production.py are gone with the workflow step that
called them. The instructions marker now only adds web-app-docs to the row's
apps; test-e2e-cli discovers the role through its new templates/test.env.j2
and runs files/test/test.sh.
The block runs in a throwaway machine, never on the stack host. compose.yml
starts one service built from files/test/Dockerfile, whose base is
INFINITO_PARENT_IMAGE out of the mounted checkout's .env, so the machine is
the stack host's own distribution and the replay exercises the installation
instead of a prebuilt image. files/test/init.sh adds an init when the base
carries none and keeps the full -Syu that ef0f03ea98 established for arch.
scripts/tests/workspace/base/01_install.sh then prepares the machine before
the block runs.
Nothing is skipped any more. The previous host-install branch exited 0 and
passed silently over 65 of the 178 invokable roles; a block without a
"docker run" is now replayed against systemd in that same machine, with the
git clone dropped in favour of the mounted tree under test.
The target role comes from the new guide_role lookup, which reads
list_invokable_app_ids() rather than matching role-name prefixes, and falls
back to the asking role when the round deploys nothing else. deployment.py
grows running_apps() so both lookups share one definition of what the round
deploys. required_by_coverage.role_is_invokable drops its own walk over
categories.yml and delegates to the same SPOT; measured equal across all 280
roles.
Carried in the same index, because the refactor pass ran over this change
set: library.py splits into sites.py and build_queue.py mixins, builder.py
hands translated_languages to catalogs.py and folds three repeated log-tail
and state writes, and placeholders.py becomes a facade over spans.py,
repairs.py and damage.py. No test changed for those splits, and the two
ref-resolution dicts builder.py carried are replaced by _wanted_ref, which
also stops hashing the deployed tree on every tag build.
Verified: 7132 unit and lint tests pass over the whole tree, the guide_role
lookup renders through a real Templar, the base image resolves from .env,
and compose config validates the service. The replay itself has never been
executed; that needs a deploy of web-app-docs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fetch() requested latest and run_builder() requested deployed once at startup, so a release tag was built on demand and then never revisited. A tag carrying a languages.json from before the translated-site split stays servable, nothing ever requests it, translates() stays false for every code and /v1.0.0/de/ answers a hard 404 forever.
_refresh_stale_sites() asks for every version that is already servable; request() checks _current() itself, so only a genuinely stale one is queued and an unbuilt tag stays on demand instead of triggering a mass rebuild.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
request(..., background=True) skipped a language as soon as translation_servable() found an index.html, which is an existence check with no ref in it. A language was therefore built exactly once: every later commit advanced the English site while every translated site stayed frozen on the commit it was first built from. The eager loop this replaced rebuilt them all, so the freeze arrived with f5eff500c1.
_publish_site already stamps the ref beside each translated site and nothing ever read it. translation_ref() reads it and _translation_current() compares it against _wanted_ref(), which also replaces the eager {LATEST: head, DEPLOYED: snapshot_ref()} dict so an unreadable snapshot no longer kills the builder while building an unrelated version.
Two more from the same review: build_language wrote into the version's state file, so a failing language reported the whole version as failed and leaked its sphinx log into /api/versions; it now uses its own version:code key. And the language index was written after the site was published, so a failure in between left a current site with a stale index, which _current() rejects forever and fetch() would rebuild on every tick; the index is written first.
library.py had grown past the 500-line cap, so the build machinery moves into builder.py as a Builder mixin. An AST diff proves every function, method and constant is byte-identical apart from _write_json -> write_json, and the six build-focused test cases move to test_builder.py with the count unchanged at 16.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
build() ran sphinx once per translated language before returning, so a deploy blocked for roughly 30 minutes per language. With 29 translated catalogs German sat eighth and its site appeared after about three and a half hours, which is why the language spec polled /deployed/de/ into a 404 for its whole run. The server already answered an unbuilt version with 202 and a build request; an unbuilt language got a bare 404.
Give languages the same treatment. request() takes a code and writes a version:code marker, build_language() builds exactly one from its own checkout via the extracted _checkout(), and the server answers a translated but unbuilt language with 202 while an untranslated one keeps its 404. No language is dropped: build() now queues every translated one into queue/background/, which next_queued() drains only after the lane a visitor is waiting on, so the requested language overtakes the backlog.
languages.json grows a translated list beside known, and _language_index() reads a pre-split file as known-only so an existing volume keeps naming its languages instead of rendering an empty switcher.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every version whose tree holds a locale/<code>/LC_MESSAGES/docs.po with at
least one translation now gets one site per such language next to the
English one, built with `-D language=<code> -D html_copy_source=0` after
the English site is published. Sphinx reads one docs.po per language
(locale_dirs, gettext_compact = "docs"); YAML pages stay untranslated
through an extension that runs before the Locale transform.
The deployed working tree is built as the unlisted version `deployed` from a
snapshot baked into the image, because the git mirror only holds what is
pushed to main. The server answers /<version>/<code>/ and
/api/languages/<version>, a language without a site is a 404 rather than the
English page, and a switcher on every page lists the built languages and
keeps the page path. min_storage grows to 13GB for the translated sites.
The 47 docs.po catalogs hold the 24869 documentation messages; German is
partially machine-translated so far. The raw-docker and primary-domain lints
skip .po files, which carry the Markdown's own commands and hosts as
translation sources.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Sphinx configuration, extensions and generators lived in the separate
infinito-sphinx repository, whose image baked a snapshot of core at build time
from a base image name core no longer publishes. Only core consumed it, and its
generators read core's role layout, so the two repositories drifted without any
CI covering the pair.
The tooling now ships in roles/web-app-docs/files/, and web-app-sphinx is
renamed to web-app-docs. The container is a small service instead of a static
image:
- it keeps a git mirror of the repository in the docs_sites volume;
- /latest/ is the last commit of the default branch, rebuilt whenever a fetch
finds a new one while the previous build stays online;
- every release tag is selectable in the sidebar switcher and on /versions/,
and is built the first time someone opens it, behind a progress bar;
- replicas share the volume: the one holding builder.lock fetches and builds,
the queue and the build states are files beside the sites, so another
replica resumes the queue when the builder dies.
The theme rewrites every page serially after the build, which took about 75
minutes for the 4544 pages; that pass now runs on the -j workers with the
theme's own function. sphinx-apidoc skips tests/, because
tests/utils/bare_bootstrap_probe.py installs an import blocker when autodoc
imports it and killed the Sphinx workers. The roles overview is generated by a
separate process, so autodoc resolves utils from the version being built, not
from main. The role pages now show galaxy_info.description instead of the
top-level key that never existed.
The Sphinx packages join the dev extra, because the mirrored unit tests of the
tooling import them. min_storage now budgets the built image, the latest site
and a running build's scratch instead of the old image's size. The Playwright
suite is split into front page, versions and on-demand build files; the last
one selects an unbuilt tag in the overview and asserts the build gets queued.
Operators must rename the web-app-sphinx inventory group to web-app-docs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5