Run 37553816857 failed 59 jobs, and web-app-docs is in five of every six
that named a role. The spec waits 88 minutes for a German site that never
arrives:
tests/test-languages.js:25:7 › the deployed working tree is published in de
Expected the de site of the deployed working tree to be built
Expected: 200 Received: 202
The container log of any job that deployed the role carries the whole chain:
"GET /deployed/\1/ HTTP/1.1" 202
build latest failed: Command '[... 'infinito_docs.generators.integrations' ...]'
returned non-zero exit status 1.
"GET /deployed/de/ HTTP/1.1" 202 (every 31s, for 88 minutes)
Four defects, in the order they fire.
The preload asked for a path named `\1/`. The replacement reached re.sub with
its backslash still escaped, so the backreference stayed literal and no
language was ever warmed; the first build then started when a visitor polled.
Reproduced through Templar rather than argued: the old expression renders
['', '\1/'] and the new one ['de/', '']. Plain Jinja2 with Ansible's filters
renders the old one correctly, which is why the first measurement of this
cleared code that the access log had already convicted. The replacement
appends with '$' and carries no backreference to escape.
`latest` then failed to build at all. builder.py puts the documented checkout
first on PYTHONPATH, so the image's generator meets the utils of the version
it is documenting, and a05071fe78 dropped the get_ prefix from entity_name.
A builder that documents every version has to tolerate every version's API,
so the import falls back.
A failed build kept answering 202 and re-queueing itself, so every client
waited its whole timeout and a doomed build restarted on each poll. This is
what turned a one-minute failure into 88-minute waits and drove one job into
the six-hour ceiling, where it was killed with 137. The server now answers
500, through a queue accessor that reuses the comparison `request` already
makes: a failure at an older ref still retries. For a translation the parent
version counts too, because here /deployed/de/ polled while `latest` was the
build that had failed.
The failure line now carries the captured log tail. The detail was never
lost -- _run merges stderr into the build state -- but it was absent from the
container log, which is what a rescue bundle holds, so the traceback had to be
reproduced locally to be read at all.
Nothing here is deployed. The 500 is a live change for three consumers that
accept 202 -- the preload task, test-front-page and test-on-demand-build --
but it is reachable only when a version is unservable AND its build failed at
the ref it would build now, so a healthy deployment sees what it saw before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make autoformat reflowed a resource_filter call, a generator expression and five test files it touched along the way. The new architecture test also carried the two findings the first gate run stopped on: three RUF012, where a TestCase class attribute needs ClassVar, and two PERF401, where an append loop becomes list.extend over a generator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A translated site repeated the whole checkout and generation of its version, and the builder ran one marker at a time. The prepared tree is now cached per (ref, tooling digest), hardlinked into each build's scratch and sealed read-only, and a bounded thread pool claims the markers a free slot can build now - a language only once its version is current, so a stale one is no longer claimed and bounced in a spin.
services.docs.build.jobs and services.docs.build.parallel replace build_jobs and default to half the host's CPUs, parallel capped at 2 so the two knobs cannot multiply past mem_limit. DOCS_I18N_PRELOAD_LANGUAGES replaces DOCS_I18N_LANGUAGE, defaults to the host language and warms [en, de] in both matrix variants, where docs_sites is backed up under swarm so the DR drill restores the built sites instead of rebuilding them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
build() runs eleven generator subprocesses before Sphinx; build_language() went straight to sphinx-build. The translated site therefore rendered without generated/ - no role pages, no overview, no regenerated READMEs - while the .pot extractor does call the generators (i18n.py:47). The catalog described a superset of what the translation could ever show. The loop moves into _generate() and both builds call it. Extracting it also surfaced a division by zero on an empty generator list, now an early return.
build_language() collapsed two different reasons into one bail and treated them alike: drop the marker, request the base. Dropping is right when the code is not translated. It is wrong when the base is merely stale, because the base build short-circuits and never reaches the tail that re-queues the languages, so the translation was lost for good and /deployed/de/ would answer 202 forever. The two reasons are now separate and the stale case re-queues the translation alongside its base.
fetch() re-requests every 900 s and _current() can never be true for an unservable site, so a failed build rebuilt itself forever and starved the background lane. _failed now records the ref it failed on, and request() refuses a BACKGROUND entry for a marker that failed at the ref it would build now. A waiting visitor still gets a retry; only work nobody waits on is throttled. The guard needs a non-empty ref so an empty snapshot ref cannot lock everything out, and _ready replaces the whole state dict, so a later success clears it.
Verified: 158 passed over the roles entire docs unit suite in the container. The extraction is behaviour-identical on the build() path - same PYTHONPATH, phase, progress formula and run call - so the measured delta there is zero. Not verified: build_language has never executed the generator phase anywhere yet, which is the one real risk this carries.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
On a cold container the docs Playwright suite triggers four serial Sphinx builds and waits for the last one. test-languages.js polls /deployed/de/ for 200 and got 202 for 55.6 minutes in job 110003713825 of run 36745666606, after which the suite budget expired and five further specs never ran. The suite has never passed on a first attempt; it used to go green only on the harness retry, which 7127c6b96b removed for budget timeouts, so a structural overrun became a red job.
Measured cost per build, from run 36558500327 where the whole queue drained on a kept stack: container up 12:15:52, the spec green at 13:21:41 answering on its first poll, next container start only at 13:30:45. Four builds in 65 minutes 49 seconds, so at most 16.5 minutes each - not the sixty to ninety minutes 40688e4ded estimated.
Three of the four builds were none of the test's business. fetch() queued latest in the foreground although nothing waits on it at startup, and _refresh_stale_sites did the same for every servable version; both now use the background lane, which is what its own docstring describes. test-front-page.js reached GET /latest/, and reaching an unbuilt version queues its build - it now asserts the redirect from the Location header without following it, and checks the same page shape against /deployed/, which is built anyway. The release tag that test-on-demand-build.js clicks stays, and now runs behind the awaited work instead of in front of it.
The translation could only be queued after its base finished, because the 202 branch returns before _translation is reachable, so it always sorted behind whatever the sibling specs had queued meanwhile. The 202 branch now carries the language segment along, admitted by translatable(), which answers from the snapshot catalog on disk before any build; build_language still re-checks the written index.
Moving a version into the background lane exposed that status() read only the foreground one, so a waiting version would have been reported as missing rather than queued to the versions overview and to /api/versions. is_queued() now covers both lanes, mirroring _dequeue, which always removed from both. Two of the five unit failures this change first produced disappeared on that fix alone, which is what shows they pinned real behaviour; the three that pinned the lane itself were rewritten to assert the behaviour instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`request()` ended in `touch(exist_ok=True)`, and on an existing file that
refreshes the mtime. `next_queued()` sorts by mtime and serves the oldest
marker, so re-requesting a build that is already queued moved it behind
everything else. The server calls `request()` on every visit, and a visitor
waiting for a page polls every thirty seconds, so waiting for a build was
what prevented it:
queued de, then a tag : ['deployed:de', 'v14.2.0'] -> builds deployed:de
after one poll on de : ['v14.2.0', 'deployed:de'] -> builds v14.2.0
One poll is enough. In the docs suite the spec before it deliberately clicks
an unbuilt release tag, so another hour-long build is always sitting there to
be overtaken by.
Observed in run 36596159856: `/deployed/de/` answered 202 and never 200 for
two hours and four minutes in one deployment and two hours fifty-seven in
another, from those jobs' own rescue artifacts, while the suite budget is
fifty-nine minutes on clearnet and eighty-nine on tor.
This removes the starvation, not the arithmetic: a full Sphinx build costs
sixty to ninety minutes here and the suite needs two of them in sequence,
English then German. Whether one round now fits is for a run to say.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The replay of a role's "### Production" block moves out of the CI guide
scripts and into the documentation role itself. The block is read off the
rendered page over HTTP, so what runs is what a reader copies, and the role
that publishes the page is also the role that proves it.
scripts/github/guide/{one,compose_deploy,host_deploy}.sh and
cli/build/docs/readme/production.py are gone with the workflow step that
called them. The instructions marker now only adds web-app-docs to the row's
apps; test-e2e-cli discovers the role through its new templates/test.env.j2
and runs files/test/test.sh.
The block runs in a throwaway machine, never on the stack host. compose.yml
starts one service built from files/test/Dockerfile, whose base is
INFINITO_PARENT_IMAGE out of the mounted checkout's .env, so the machine is
the stack host's own distribution and the replay exercises the installation
instead of a prebuilt image. files/test/init.sh adds an init when the base
carries none and keeps the full -Syu that ef0f03ea98 established for arch.
scripts/tests/workspace/base/01_install.sh then prepares the machine before
the block runs.
Nothing is skipped any more. The previous host-install branch exited 0 and
passed silently over 65 of the 178 invokable roles; a block without a
"docker run" is now replayed against systemd in that same machine, with the
git clone dropped in favour of the mounted tree under test.
The target role comes from the new guide_role lookup, which reads
list_invokable_app_ids() rather than matching role-name prefixes, and falls
back to the asking role when the round deploys nothing else. deployment.py
grows running_apps() so both lookups share one definition of what the round
deploys. required_by_coverage.role_is_invokable drops its own walk over
categories.yml and delegates to the same SPOT; measured equal across all 280
roles.
Carried in the same index, because the refactor pass ran over this change
set: library.py splits into sites.py and build_queue.py mixins, builder.py
hands translated_languages to catalogs.py and folds three repeated log-tail
and state writes, and placeholders.py becomes a facade over spans.py,
repairs.py and damage.py. No test changed for those splits, and the two
ref-resolution dicts builder.py carried are replaced by _wanted_ref, which
also stops hashing the deployed tree on every tag build.
Verified: 7132 unit and lint tests pass over the whole tree, the guide_role
lookup renders through a real Templar, the base image resolves from .env,
and compose config validates the service. The replay itself has never been
executed; that needs a deploy of web-app-docs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5