After a version build the builder queued every translated language, not the
configured ones. The list now reaches the library: server.py reads
DOCS_I18N_PRELOAD_LANGUAGES, Library freezes it into a set that drops the empty
string ''.split(',') yields when nothing is configured, and builder.py queues a
language only when it is in that set. Every other translation is built when a
visitor opens it.
The warm-up task accepted 202, which means queued, so the deploy walked on
while the sites were still building. A second task now waits for 200. Both read
DOCS_I18N_PRELOAD_PATHS, hoisted out of the loop it used to be written inline
in, because two tasks need the same language-code-to-URL-path mapping.
DOCS_BUILD_PARALLEL divides by overhead + 2 * process_mb instead of
overhead + process_mb: with a translation building beside the version build a
parallel slot holds two process-sized allocations. The new divisor is strictly
larger, so the value can only fall, never rise; with the configured 3072 and
2048 MB it differs from the old one only between 10 and 16 GB of service
memory, where it goes from 2 to 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 37553816857 failed 59 jobs, and web-app-docs is in five of every six
that named a role. The spec waits 88 minutes for a German site that never
arrives:
tests/test-languages.js:25:7 › the deployed working tree is published in de
Expected the de site of the deployed working tree to be built
Expected: 200 Received: 202
The container log of any job that deployed the role carries the whole chain:
"GET /deployed/\1/ HTTP/1.1" 202
build latest failed: Command '[... 'infinito_docs.generators.integrations' ...]'
returned non-zero exit status 1.
"GET /deployed/de/ HTTP/1.1" 202 (every 31s, for 88 minutes)
Four defects, in the order they fire.
The preload asked for a path named `\1/`. The replacement reached re.sub with
its backslash still escaped, so the backreference stayed literal and no
language was ever warmed; the first build then started when a visitor polled.
Reproduced through Templar rather than argued: the old expression renders
['', '\1/'] and the new one ['de/', '']. Plain Jinja2 with Ansible's filters
renders the old one correctly, which is why the first measurement of this
cleared code that the access log had already convicted. The replacement
appends with '$' and carries no backreference to escape.
`latest` then failed to build at all. builder.py puts the documented checkout
first on PYTHONPATH, so the image's generator meets the utils of the version
it is documenting, and a05071fe78 dropped the get_ prefix from entity_name.
A builder that documents every version has to tolerate every version's API,
so the import falls back.
A failed build kept answering 202 and re-queueing itself, so every client
waited its whole timeout and a doomed build restarted on each poll. This is
what turned a one-minute failure into 88-minute waits and drove one job into
the six-hour ceiling, where it was killed with 137. The server now answers
500, through a queue accessor that reuses the comparison `request` already
makes: a failure at an older ref still retries. For a translation the parent
version counts too, because here /deployed/de/ polled while `latest` was the
build that had failed.
The failure line now carries the captured log tail. The detail was never
lost -- _run merges stderr into the build state -- but it was absent from the
container log, which is what a rescue bundle holds, so the traceback had to be
reproduced locally to be read at all.
Nothing here is deployed. The 500 is a live change for three consumers that
accept 202 -- the preload task, test-front-page and test-on-demand-build --
but it is reachable only when a version is unservable AND its build failed at
the ref it would build now, so a healthy deployment sees what it saw before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A translated site repeated the whole checkout and generation of its version, and the builder ran one marker at a time. The prepared tree is now cached per (ref, tooling digest), hardlinked into each build's scratch and sealed read-only, and a bounded thread pool claims the markers a free slot can build now - a language only once its version is current, so a stale one is no longer claimed and bounced in a spin.
services.docs.build.jobs and services.docs.build.parallel replace build_jobs and default to half the host's CPUs, parallel capped at 2 so the two knobs cannot multiply past mem_limit. DOCS_I18N_PRELOAD_LANGUAGES replaces DOCS_I18N_LANGUAGE, defaults to the host language and warms [en, de] in both matrix variants, where docs_sites is backed up under swarm so the DR drill restores the built sites instead of rebuilding them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
On a cold container the docs Playwright suite triggers four serial Sphinx builds and waits for the last one. test-languages.js polls /deployed/de/ for 200 and got 202 for 55.6 minutes in job 110003713825 of run 36745666606, after which the suite budget expired and five further specs never ran. The suite has never passed on a first attempt; it used to go green only on the harness retry, which 7127c6b96b removed for budget timeouts, so a structural overrun became a red job.
Measured cost per build, from run 36558500327 where the whole queue drained on a kept stack: container up 12:15:52, the spec green at 13:21:41 answering on its first poll, next container start only at 13:30:45. Four builds in 65 minutes 49 seconds, so at most 16.5 minutes each - not the sixty to ninety minutes 40688e4ded estimated.
Three of the four builds were none of the test's business. fetch() queued latest in the foreground although nothing waits on it at startup, and _refresh_stale_sites did the same for every servable version; both now use the background lane, which is what its own docstring describes. test-front-page.js reached GET /latest/, and reaching an unbuilt version queues its build - it now asserts the redirect from the Location header without following it, and checks the same page shape against /deployed/, which is built anyway. The release tag that test-on-demand-build.js clicks stays, and now runs behind the awaited work instead of in front of it.
The translation could only be queued after its base finished, because the 202 branch returns before _translation is reachable, so it always sorted behind whatever the sibling specs had queued meanwhile. The 202 branch now carries the language segment along, admitted by translatable(), which answers from the snapshot catalog on disk before any build; build_language still re-checks the written index.
Moving a version into the background lane exposed that status() read only the foreground one, so a waiting version would have been reported as missing rather than queued to the versions overview and to /api/versions. is_queued() now covers both lanes, mirroring _dequeue, which always removed from both. Two of the five unit failures this change first produced disappeared on that fix alone, which is what shows they pinned real behaviour; the three that pinned the lane itself were rewritten to assert the behaviour instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The site answers 202 while it builds a version, and 200 once a build
exists. Whether that build matches what the version currently names was
never asked at the serving gate: servable() only checks that an index.html
is on disk. So the first request after a new snapshot got the previous
build with 200, and the rebuild it triggered finished into a directory
nobody was waiting for.
For a release tag that cannot happen, and for latest it is a reasonable
trade, since a visitor would rather read the previous page than a spinner.
For deployed it is wrong by definition: the name means the working tree
this run staged, so an earlier snapshot is not a stale answer but a
different question's answer. The post-deploy test that reads the published
page therefore validated the round before it, and reported a role broken
whose page had already been fixed.
The comparison already existed. _current() weighs built_ref against
_wanted_ref, which resolves deployed to the snapshot digest, and the
builder consults it to decide whether to rebuild. Only the server did not.
outdated() exposes it for deployed alone and the serving branch now treats
a mismatch as not yet servable, so the poll waits for the build it asked
for.
Measured on the stack while the previous round was live: the served page
was written 16:44:41 from the older snapshot, its own source carried the
newer content, and the correct build landed at 17:10:56, twenty-six
minutes after the test had already read the stale one and failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
build() ran sphinx once per translated language before returning, so a deploy blocked for roughly 30 minutes per language. With 29 translated catalogs German sat eighth and its site appeared after about three and a half hours, which is why the language spec polled /deployed/de/ into a 404 for its whole run. The server already answered an unbuilt version with 202 and a build request; an unbuilt language got a bare 404.
Give languages the same treatment. request() takes a code and writes a version:code marker, build_language() builds exactly one from its own checkout via the extracted _checkout(), and the server answers a translated but unbuilt language with 202 while an untranslated one keeps its 404. No language is dropped: build() now queues every translated one into queue/background/, which next_queued() drains only after the lane a visitor is waiting on, so the requested language overtakes the backlog.
languages.json grows a translated list beside known, and _language_index() reads a pre-split file as known-only so an existing volume keeps naming its languages instead of rendering an empty switcher.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every version whose tree holds a locale/<code>/LC_MESSAGES/docs.po with at
least one translation now gets one site per such language next to the
English one, built with `-D language=<code> -D html_copy_source=0` after
the English site is published. Sphinx reads one docs.po per language
(locale_dirs, gettext_compact = "docs"); YAML pages stay untranslated
through an extension that runs before the Locale transform.
The deployed working tree is built as the unlisted version `deployed` from a
snapshot baked into the image, because the git mirror only holds what is
pushed to main. The server answers /<version>/<code>/ and
/api/languages/<version>, a language without a site is a 404 rather than the
English page, and a switcher on every page lists the built languages and
keeps the page path. min_storage grows to 13GB for the translated sites.
The 47 docs.po catalogs hold the 24869 documentation messages; German is
partially machine-translated so far. The raw-docker and primary-domain lints
skip .po files, which carry the Markdown's own commands and hosts as
translation sources.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Sphinx configuration, extensions and generators lived in the separate
infinito-sphinx repository, whose image baked a snapshot of core at build time
from a base image name core no longer publishes. Only core consumed it, and its
generators read core's role layout, so the two repositories drifted without any
CI covering the pair.
The tooling now ships in roles/web-app-docs/files/, and web-app-sphinx is
renamed to web-app-docs. The container is a small service instead of a static
image:
- it keeps a git mirror of the repository in the docs_sites volume;
- /latest/ is the last commit of the default branch, rebuilt whenever a fetch
finds a new one while the previous build stays online;
- every release tag is selectable in the sidebar switcher and on /versions/,
and is built the first time someone opens it, behind a progress bar;
- replicas share the volume: the one holding builder.lock fetches and builds,
the queue and the build states are files beside the sites, so another
replica resumes the queue when the builder dies.
The theme rewrites every page serially after the build, which took about 75
minutes for the 4544 pages; that pass now runs on the -j workers with the
theme's own function. sphinx-apidoc skips tests/, because
tests/utils/bare_bootstrap_probe.py installs an import blocker when autodoc
imports it and killed the Sphinx workers. The roles overview is generated by a
separate process, so autodoc resolves utils from the version being built, not
from main. The role pages now show galaxy_info.description instead of the
top-level key that never existed.
The Sphinx packages join the dev extra, because the mirrored unit tests of the
tooling import them. min_storage now budgets the built image, the latest site
and a running build's scratch instead of the old image's size. The Playwright
suite is split into front page, versions and on-demand build files; the last
one selects an unbuilt tag in the overview and asserts the build gets queued.
Operators must rename the web-app-sphinx inventory group to web-app-docs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5