No conflicts. Two files were touched by both sides and both kept each
side: the Makefile, where upstream retires the mcp-audit target and this
branch pins the throwaway cluster's vpn axis, and requirement 035, where
the role table keeps the single svc-net-wireguard row.
Incoming carries the fixes for two applications this branch's last run
failed on -- penpot probing its frontend without curl, and mobilizon's
postgres 18 migrations -- plus dependency bumps and a lint repair.
Gates on the merged tree: unit 6202, lint 523, integration 330,
ansible-lint clean.
The wait job picks the newest privileged run on the head commit of the
PR. Two fork PRs that share a head repository, branch and commit but
target different bases each start such a run, and the commit alone cannot
tell them apart. A failed run of the other PR would then end the wait of
this one while its images are still being built.
The run name of the workflow carries the PR number, so the match now
requires it next to the head commit. A run without that name matches
nothing, and the wait does what it does whenever no run is found: it
keeps polling for the images until they exist or the wait times out.
Raised by the CodeRabbit review of this PR.
Refs #784
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The duplicate check accepted any open PR whose head branch started with
the dedupe prefix and ended in the fingerprint of the fresh proposal. The
listing also returns PRs from forks, and the fingerprint can be derived
from public data, so a fork PR with such a branch name would have made the
bot skip its own update PR without carrying the change.
The shortcut now only applies to PRs whose head branch lives in this
repository, which only accounts with write access can create. A fork PR
is still compared by the content of its head, as before.
Raised by the CodeRabbit review of PR #781.
Refs #780
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Docker updater watches image plus version, the repository updater
repository plus ref. A dozen pins fell outside both - an app_version, a
release, a tag in a registry neither speaks - so nothing ever told anyone
they had gone stale.
Such a pin now names its upstream next to itself in an `update:` block:
git_tags, registry_tags, npm, http_regex or a role-local script, with
`match` selecting a tag flavour and `strip` removing a prefix the pin does
not carry. utils/update/source.py resolves them beside the two existing
updaters, `python -m cli.contributing.update.source` bumps them, the
update-version-sources job opens one PR per role, and
tests/external/update/source warns while a bump is pending.
Two lints hold the rule: every semver pin is watched by one of the three
updaters or opted out with a reason, and a semver only ever sits behind
`version`, `<what>_version` or `ref` - which renames moodle's release to
app_version and simpleicons' local_version to local_tag, the latter being
the tag of a locally built image rather than an upstream version. A
declaration naming a key the entity does not carry, or a type the resolver
does not know, fails the lint instead of resolving nothing in silence.
Picked from feature/store-i18n (94d607fd0a) onto current main. Two things
differ from the original: svc-ai-lmstudio no longer carries an app_version
pin, so its declaration is dropped, and web-app-stalwart's webui_version,
added since, declares the git tags of the webui repository.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The wait for fork CI images never found the privileged workflow run it
watches. It selected a run only when the run's pull_requests array held
the PR number, and for a run whose head repository is a fork the API
returns that array empty. The job only runs for fork PRs, so the lookup
matched in no case it exists for.
The early exit for a failed privileged run was therefore unreachable.
When that run failed, the job kept polling until its timeout of 1980
attempts and held a runner for about five and a half hours.
The lookup now asks the API for the runs of the PR head commit and
selects by head_sha. That also replaces a paginated listing of every run
of the workflow with one small request per attempt. Without a known head
commit the match by PR number stays.
Refs #784
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The job that waits for the CI images of a fork PR ended on the first
failed GitHub API call. The lookup of the privileged workflow run is
assigned with run_json="$(find_matching_run)" under set -e, so a failing
gh ended the script, although the loop is built for 1980 attempts. PR #782
went red that way: GitHub answered one request with HTTP 504 in attempt
31, while the privileged run was still queued.
A failed lookup now counts as "no information in this attempt" and the
loop goes on waiting for the images, which are its success criterion.
Thirty failed lookups in a row end the job with a message that names the
cause, so a broken token does not hold a runner until the timeout.
The test runs the script against a gh that fails every lookup. The
previous script fails both cases at the first lookup.
Closes#784
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The update bot recognised an open PR as a duplicate by hashing the blobs
of the files its commit changed, read at the PR head. A follow-up commit
that edits one of those files changes the hash, so the next run opened a
second PR with the identical bump: PR #778 next to PR #775 for
web-svc-collabora, after a fix had edited the bumped services.yml.
The branch name of a bot PR ends in the first seven characters of the
fingerprint of the change the bot proposed. An open PR whose branch ends
in the fingerprint of the fresh proposal now counts as a duplicate before
its head is compared. The head comparison stays as the second criterion.
The test runs pr.sh in a throwaway repository against a stubbed gh. It
fails on the previous script with exit 1, because that one goes on to
push.
Closes#780
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
No conflicts, and no file touched by both sides. Incoming is a run of
Docker image version bumps plus one CI change, which moves the diff
baseline from a force-pushed mirror of main to a fetch in each consumer --
entry-push-latest.yml and the resolve scripts, none of which this branch
touches.
Gates on the merged tree: unit 6199, lint 525, integration 330, external
19, ansible-lint clean, markdownlint clean, ruff clean.
A round that drills has to deploy twice, so the caller needs the answer
before it deploys anything. Answering here keeps the rule deciding which
applications have something to restore in the script that already owns it,
rather than restating it in the caller.
The verdict is printed rather than signalled by exit status: a probe that
dies on the way to the decision must not read as 'no drill' and leave the
caller skipping the deploy the drill depends on.
The sync job force-pushed the source repository's main onto this one so that
`origin/main` would mean "upstream main" for the three consumers that diff
against it. Carrying .github/workflows/** in that push is what GitHub refuses
for an App token, and `workflows` is not a GITHUB_TOKEN permission scope, so the
job needed an app installation that may write workflow files. Run 37169357364
showed the fork has no such secret at all.
Nothing reads the pushed branch. push_ci_policy.sh, affected_roles.sh and
new_roles.sh each fetch `origin main` into their own runner's clone before
reading it, and no caller queries the branch through the API. Pointing those
fetches at the configured source repository gives the same baseline read-only:
no push, no credential, no app permission to widen.
baseline_remote.sh resolves where that baseline comes from and keeps the
normalization and the disabled-value list in one place: `origin` for the source
repository itself, for a disabled sync and for a local clone without
GITHUB_REPOSITORY, and the upstream fetch URL otherwise. The fetches now carry
an explicit `main:refs/remotes/origin/main` refspec, which a URL needs and a
remote name tolerates.
The release guard keeps fetching `origin`: it asks whether a tag commit sits on
this repository's own main, which is a different question from the diff
baseline.
Removed: the sync-main job, its `needs` edge, main_from_repository.sh and the
directory that held it.
test_diff_affected_roles.py copies only the script under test into its fixture,
so it needed the new helper too. It also inherited the full environment, and a
GITHUB_REPOSITORY present in CI would have sent the fixture's baseline fetch to
the real upstream instead of its own bare remote: three of its five cases fail
that way, and the run takes 24s instead of 0.3s because it goes to the network.
The variable is now dropped alongside BASH_ENV.
Verified by running the helper in all four contexts and affected_roles.sh from
an empty refs/remotes/origin/main in a scratch clone as the fork, as the source
repository and locally; each populated the ref and completed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One conflict, in the generated role table of README.md. Both sides changed
the same rows: upstream swapped WordPress's Mailu dependency for Stalwart,
while this branch replaced the three svc-net-wireguard-{core,plain,
firewalled} rows with the single svc-net-wireguard it deploys.
Resolved by regenerating rather than by hand, since make readme-index owns
that table and gates it with --check. The regenerated table carries both
sides and the check reports it up to date.
README.md was the only file both sides touched, and nothing in the tree
still names the three replaced roles.
A push to main whose head commit starts with "Release version " (the subject pkgmgr writes for every release) now cancels the main run in progress, so the release starts at once instead of queueing behind it. Every other push to main still leaves the running pipeline alone.
When the pushed main commit carries a vX.Y.Z tag, the orchestrator runs with chunk_gate false: every chunk deploys and reports, and a failed chunk still turns the run red, so release-guard and release-version stay skipped.
entry-cancel-superseded.yml mirrors the new rule behind its push guard, parenthesised so && does not bind the release clause; the mirror test composes the expected rule the same way.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One conflict, in the MCP exclusion list of requirement 035. Both sides
edited the same line: main added svc-net-tor-smtp, this branch replaced the
three svc-net-wireguard-{core,plain,firewalled} entries with the single
svc-net-wireguard it deploys. Resolved to the union of both — 24 entries —
which is what the merged tree actually holds.
Everything else merged clean and was checked for overlap rather than
assumed: no reference anywhere still names the three replaced roles;
matrix.py kept main's PRIMARY_APPS alongside the reordered round, and the
bootstrap pass does not need it, since only one template reads it and
group_vars defaults it empty; and main's ansible.cfg addition sits in
[defaults], so the ssh_args the mesh tool derives its own connection
settings from are untouched.
SWARM_REQUIRED_SERVICES no longer force-keeps wireguard: with the axis off the
role is absent from the inventory, so there is no disabled-service conflict to
work around.
The drill strips the services it cannot run without from the incoming disable
list, and wireguard was not among them. For an app whose own graph does not
reach it, the provision step disabled the service while extend_inventory had
already placed svc-net-wireguard on every swarm host, and the services
disabler rejected the pair:
service 'wireguard' is disabled, but provider role 'svc-net-wireguard' is
still active in .../devices.yml
svc-swarm-node declares the service required for exactly this reason, so the
drill has no more business disabling it than it has disabling node.
An image bump in a provider redeployed every variant of every consumer,
including the variants that switch that provider off and therefore never
deploy the bumped image at all. The diff-derived whitelist now narrows each
affected role to the variants whose own round closure still reaches a seed,
emitted as `role#0,2` selection tokens. That closure is walked on the same
variant-merged services maps the matrix planner builds its rounds from, so the
narrowing follows a parent's variant choice across every generation of the
chain. It covers every auto-derived whitelist: the weekly update PRs,
Dependabot, and any other diff-driven run.
The narrowing never shrinks the matrix on partial information. A seed role
keeps every variant, because its own files changed. A role reached only
through a run_after edge keeps every variant too, because the round closure
does not follow that edge and a role whose hits are empty is therefore
unnarrowed rather than dropped.
A pinned token naming a role the discovery query does not return now warns
instead of aborting the run. The diff-derived whitelist legitimately names
roles the lifecycle envelope filters out, and once it pins variants such a
token would abort every chunk; role names are still checked against roles/
before the run starts, so a typo is caught one layer earlier.
The round's variant merge moves out of the deploy inventory package into
utils/roles/applications/variants.py. The CI selection has to build the same
maps, and importing the deploy package for them pulls in a module that reads
INFINITO_SRC_DIR at import time, which the runner host does not set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
run.sh drives every suite with -n auto and no scope, so xdist hands out single
test METHODS. Each worker is its own process, so setUpClass runs once per worker
the class happens to be scattered over, and an expensive fixture is paid as many
times as the scheduler split its methods.
The integration report shows the shape without interpretation:
TestMcpEnvRender's nine methods came back as six at 277-287s and three at
0.003s. The fast three landed on workers that had already paid. That is 1776 of
the suite's 2282 CPU-seconds spent computing one identical result six times,
twenty Ansible templar renders of the MCP providers each time.
--dist loadscope groups by class, so a class occupies one worker and its
fixture runs once.
1a908a6e5b introduced -n auto and hoisted fixtures in the same commit, so the
hoisting was undercut from the start: work moved from per-method to per-class,
and the scheduler it arrived with promptly charged it per worker instead. This
finishes that commit rather than correcting it.
Measured against the same tree, four suites, before and after:
integration 296.83s -> 219.32s
lint 105.64s -> 84.27s
unit 65.74s -> 53.02s
external 75.69s -> 71.99s
Phase 1's critical path drops from 4m57s to 3m40s. That unit and lint move too
says the pattern was not unique to the MCP class.
Coverage is unchanged, which is the point of the change rather than a side note.
All four suites report identical counts: unit 5857 passed / 9 skipped / 2301
subtests, external 19 / 2, lint 521 / 1 / 374, integration 328 / 2 / 1562.
A second effect is worth naming because it cuts the right way. Several sweeps
keep their non-empty guard in a separate method, e.g. test_fragments.py:139
proving the corpus is not empty before the sweep asserts over it. Under
--dist load those two ran in different processes with separate caches, so the
guard proved nothing about the process the sweep ran in. loadscope puts them
back together.
Residual: loadscope trades balance for locality, and here one class sets the
floor. TestMcpEnvRender holds a worker for ~250s on its own, so the suite cannot
go below that until the ~12s per render is addressed. That cost is the render
guard in utils/cache/applications.py:415 returning the merged map without
caching it, which is production config resolution and deliberately left alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lint fan-out listed lint-php twice, once before and once after
lint-playwright. Two commits added it independently: 82218087c1 for the PHP and
Ruby syntax checks, 6fee7ca136 for the per-language unit suites, and the second
did not see the first because lint-playwright had since been inserted between
them.
The wasted slot was the harmless half. parallel.sh keys target_start,
target_dur, target_rc and target_log by target NAME, so a repeat overwrites the
first run's log path and exit code, and the final pass reads each name once.
Two consequences: drain_one prints the second run's log twice and the first
run's output never appears, and a failing first invocation is reported green as
soon as the second one passes. A duplicate therefore turns a target into a blind
spot rather than into a slow one.
The gate showed the symptom plainly, two lint-php rows with an identical
0m 03s, which reads as a rendering quirk rather than as one target run twice.
Both halves are fixed: the list loses the repeat, and parallel.sh refuses a
duplicate before it loads the environment or starts a single make, naming the
offender and why a repeat cannot be tolerated. The two callers are Makefile:440
and Makefile:822; the test list has no duplicates, so the guard is inert there.
Negative control: `parallel.sh lint-php lint-python lint-php` exits 2 with
"duplicate target(s): lint-php" and launches nothing. The full quality-high run
afterwards dispatches 15 lint targets with lint-php once and stays green.
Residual: the guard covers scripts/make/parallel.sh only. scripts/tests/dns/
parallel.sh is a separate script with its own argument handling and was not
examined.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
71888d86af moved twelve workflow files behind trigger prefixes and updated the
references GitHub validates. Three mentions in prose kept the old names: an echo
line, a file header and a lint's failure message. The same commit already fixed
the REST dispatch URL two lines below the echo it missed.
roles/svc-runner/files/test/external.sh:36 test-runner-smoke.yml
scripts/github/runner/cleanup_docker.sh:4 images-build-ci.yml
tests/lint/repository/test_distros_spot.py:116 test-workspace.yml
None of them executes; the workflow-reference lint deliberately does not reach
them, because outside .github/workflows a bare *.yml collides with services.yml,
compose.yml and variants.yml. A repo-wide grep over all eleven legacy names now
returns nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image runs this module from a layer that holds three files of utils:
__init__.py, install/__init__.py and install/collections.py, the slice
ef45d69dc3 cut so the source tree cannot invalidate the collection install.
67f5398985 established that the module therefore stays on the standard
library, and d4ae5cf9f0 broke it 15 minutes later by routing the reads
through utils.cache.files to satisfy the cache-read lint. The import raises
ModuleNotFoundError: No module named 'utils.cache' in that layer, which no
import inside a full checkout can show, because this repository installs
utils into site-packages.
Nothing turned red. The caller reads only the exit code, and the module's
contract is exit 1 with the missing collections on stdout, so a crash - exit
1, nothing on stdout - is indistinguishable from work to do. Every image
build since has reinstalled every collection with --force-with-deps, which
is the cost the check exists to avoid. The deploy job the traceback was
found in finished green.
The reads go back to Path.read_text with a cache-read marker each. The cache
was never earning its place here: read_text is an lru_cache keyed on the
path, and one run reads the requirements file once and one MANIFEST.json per
collection, eight distinct paths, none of them twice. Copying utils/cache
into the slice instead would work - the closure is three stdlib-only files -
but it binds a package with 51 commits in the last twelve months to the
layer that installs ansible, in front of a cache with no possible hit.
The caller now exits 3 when a non-zero status arrives with an empty stdout,
a state the contract cannot produce, so the next crash stops the install
instead of quietly reinstalling everything. all.sh runs under set -e and
touches the stamp only after ansible.sh returns, so the failure also leaves
the stamp stale rather than recording an install that did not happen.
The probe that would have caught this builds the slice in a tempdir from the
Dockerfile's own COPY lines and runs the module inside it. It has to pass
python -S: without it the interpreter resolves every submodule from the
installed checkout however the slice is built, and the test passes on a
module that cannot import in the image. Verified by injecting the import
back into the tempdir copy, which reproduces the traceback verbatim.
Delta measured by loading the committed module and this one side by side:
identical on both requirements files, against the real collections directory
and an empty one, on a missing file, on non-UTF-8 input and on a corrupt
MANIFEST.json. The one behavioural difference is that a file changed within
a process is now read fresh rather than from the memo.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A codename carries no ordering, so 'bookworm' and 'trixie' are two opaque
strings and no updater can tell that one supersedes the other. A pin that
names its Debian release that way is frozen on that release for as long as
it stands, and the freeze is invisible because the tag keeps resolving.
Numeric tags are comparable and stay inside the freshness pipeline, so the
rule is worth holding for the whole repository rather than fixing case by
case.
The check reads two shapes, because a codename reaches the tree in two
spellings. A 'name:tag' reference is recognised by the absence of a space
after the colon, which is what separates 'debian:bookworm' from the YAML
'image: python' and covers Dockerfile FROM lines and heredocs without a
second parser. A 'version:' value is read as a key, because '3.11-bookworm'
carries no colon at all.
False positives are kept out by two independent gates: a word-boundary
prefilter, so 'residual' and 'inside' are not read as 'sid', and a split of
the tag on '-', '.' and '_' that requires a whole component to be a
codename. That is what lets 'cache buster', '.stretched-link',
'fa-bullseye' and a tags list containing 'stretch' through. Every codename
occurrence in the repository was checked against the matcher by hand, and
the check was negative-controlled in both shapes, including an untracked
file.
Four references had to move for the rule to hold. meta/distros.yml takes
'debian:13' rather than '13.6': at depth 1 the updater can raise it across
a major, while depth 2 pins it inside the 13.x line forever. The whole
'make install' path was run in a debian:13 container to back that: all
twenty-four Depends and Recommends resolve on trixie, the docker repository
bootstrap derives the right suite and installs docker-ce-cli, and the
package builds and installs.
The workspace DiD probe moves to 'debian:13-slim'. An earlier version of
this change carried a suppression claiming the probe had to follow the
suite the apt proxy mirrors. That was wrong twice over: it cited a function
that has nothing to do with the probe, and the coupling does not exist,
because the cache frontend rewrites by path prefix and the suite never
enters a routing decision. Replaying the probe verbatim against
debian:13-slim fetches all three trixie suites through the
bookworm-configured proxies, so the suppression is gone rather than
reworded.
The rule ships with an escape hatch and no uses of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Penpot job 105068912535 and MediaWiki job 105068912565 both died on one lab node whose dnsmasq kept its :53 socket bound and rejected every query with 'ignoring query from non-local network', once for a container address and once for 127.0.0.1. Rejecting loopback as non-local only makes sense if --local-service is active and dnsmasq's own interface list went stale, but the bundle records neither the flag nor where it would come from.
Capture the process command line, /etc/default/dnsmasq, the unit including its drop-ins, and the config that actually landed on disk. Addresses and listening sockets were already collected and are not duplicated.
Read-only. Every call falls back to a text line when the file or the unit is absent, so a node without dnsmasq reports that rather than failing the collector.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
For a fork PR the build step is skipped -- the images come from the
privileged run -- so the job went green in 29 seconds while that run was
still pushing. Everything gated on it inherited the lie: the DNS test
pulled arch 3m30s too early on one run and 53s too early on the next,
both times 'manifest unknown', and the four deploy chunks were spared
only by starting later.
The job now polls until every distro image for its tag is in the
registry. It runs unconditionally: where the job built the images itself
the first poll passes, and a push that silently failed is caught too.
The poll drives scripts/meta/resolve/missing/ci_images.sh, which already
owned that check for the release workflow, so the ref shape is not
written a fourth time. A resolver that cannot run aborts with 2 rather
than reading as 'not yet pushed' and hiding behind the full budget.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The job could sit silent for its full 30-minute budget and only then
print five groups, so a hanging distro was indistinguishable from a slow
one. Each container's output is stamped with [distro][HH:MM:SS] as it
arrives, and the per-distro verdict is printed when that distro finishes
rather than at the end.
A pipeline hands $! the pid of tee rather than of the test, which would
have turned every DNS failure into a green job, so each run goes through
a subshell under pipefail. The ::error:: annotation, the per-distro logs
and the container cleanup are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Computing the distro list no longer justifies a runner of its own: it is
two steps, and the job that consumes them already checks the repository
out. The workflow now occupies one runner instead of seven.
The step also went silent for minutes at a time, printing five pids and
then nothing until every build had finished. Each build's output is piped
through a [distro][HH:MM:SS] stamp as it arrives, so the log shows what is
happening while it happens; tee still files the per-distro logs for the
artifact, and the replay block at the end is gone as duplication.
A pipeline hands $! the pid of tee rather than of the build, which would
have swallowed every failure, so each build runs in a subshell under
pipefail and wait reads that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gate flagged six of my own changes. INFINITO_ANSIBLE_COLLECTIONS_SOURCE
reached default.env without a handler, so the generator dropped it
silently; ansible.sh then carried a shell-side default for the same key,
a second source that drifts. Both are now one declaration, read as
required, so a missing key fails loudly instead of falling back.
File reads route through the session cache except where the same run
rewrites the file, which is the documented opt-out and now says so per
call. The updater reads yaml through utils.cache.yaml rather than calling
safe_load itself.
restore_collection_history.sh was committed without its executable bit,
and the new workflow's name was 31 characters where the dispatch menu
truncates at 25; the explanatory wording lives in run-name instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The parent directory was created and chowned on every run, before the
script looked at whether the virtualenv exists. Where the parent is not
writable that reached for sudo, which a sandboxed or no-new-privileges
container refuses -- so an install that had nothing to do failed with
'The "no new privileges" flag is set'.
Nothing needs preparing once the interpreter is there, so the check moves
ahead of the preparation and returns.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The matrix spent five runners on work that is almost entirely network
wait: the collection install measured 0% CPU for its whole duration, and
the parent-image pull, the package managers and the push are no
different. One runner carries all five concurrently and frees the other
four.
Losing the matrix loses fail-fast: false with it, so push_all.sh awaits
every build and reports each one by name before exiting non-zero. A
broken distro no longer hides the other four, which is what made today's
triage find three causes instead of one.
Starts are spread by fibonacci seconds because each build opens with a
parent-image pull of several hundred MB. Per-distro logs are grouped in
the job output and uploaded as an artifact, and the pressure report now
carries free -h: memory is the one resource this consolidation contends
for, and an OOM kill would otherwise read as a build error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ignore-error traded a loud fault for a silent one. The cache writes to
the same GHCR repository as the image and under the same packages:write
token, so every shared failure mode -- permissions, outage, rate limit --
already fails --push. What remains is buildkit's own cache manifest type,
which a registry can reject while accepting an ordinary image push.
That case is exactly the one worth hearing about. Swallowed, it leaves CI
green while the cache never fills and every build silently pays the full
855-899s again; raised, the first run answers it once and for good.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Which of the two sources is faster is not a constant. galaxy.ansible.com
degraded from 207s to 687s over one day and then served 502s outright,
while the pinned git fallback carried the same install in 31s; a week
later the order may be the other way round. The daily job times both and
swaps the default only when the preferred one's median reaches three
times the other's over twelve samples.
A failed install is recorded as None and ranks worse than any duration,
so an outage counts instead of dropping out of the statistic. Below a
full window nothing changes, which keeps a single bad day from flipping
the default.
History lives in an artifact rather than the Actions cache: a schedule
run resolves to the default branch and has read-only cache access there,
so it could not carry the window forward.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pinning the collections froze them, and nothing thaws a pin: dependabot
reads pyproject.toml, the docker updater reads image tags, and neither
sees a version inside requirements/. The new job resolves upstream
releases with git ls-remote --tags and opens a PR the same way the pip,
image and repository-ref updaters do.
Both requirement files are bumped together, because the git fallback has
to install what the Galaxy file declares; moving only one would break
that. A collection whose two files disagree, or whose version is not a
semver, is left alone -- that covers the branch ref and the commit pin,
which state an intent a bumper must not guess.
The job calls pr.sh rather than role_prs.sh: the latter splits per
sub-directory and would open no PR at all for flat paths like
requirements/*.yml.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
COPY . preceded every expensive RUN, so each commit rebuilt the whole
chain and ansible-galaxy ran on every build of every distro. The
collection install now sits in its own layer fed only by default.env,
scripts/install, the dev-python shell scripts, requirements/ and the
three files the version check imports. Everything that needs the full
tree, the project's own distro package included, stays behind COPY ..
Measured on a role-only change: the collection layer stays CACHED and
the build drops to 3m10s against a CI baseline of 855-899s. An
unchanged rebuild is 15 cached layers in 1.5s.
The gha cache imported and exported nothing in a plain run step, so the
export moves to the registry the job already authenticates against.
ignore-error keeps a cache write failure from taking the build with it,
which it otherwise does: proven against an unreachable registry, where
the build fails hard without it and exits 0 with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The requirements named seven collections without versions, so every
install re-resolved them against galaxy.ansible.com and --force-with-deps
redownloaded all 37 MB even when nothing had changed. Exact pins make the
installed set checkable, and the check short-circuits the whole roundtrip:
make install-ansible drops from 8m24s to 0.077s, and a Galaxy outage stops
mattering once the pinned set is present.
The git fallback is pinned to the same versions so it installs what the
Galaxy file declares rather than whatever the branches currently hold.
kewlfft.aur pins a commit because its tags carry no galaxy.yml and master
has moved past the released version.
The lint installer kept a second, weaker truth about the same directory,
guessing from directory presence for three hardcoded collections; it now
shares the version-exact check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-net-tor aborts at roles/svc-net-tor/tasks/router.yml:2 when networks.internet.dns is empty, and rightly so: dnsmasq runs with no-resolv, so an empty upstream leaves the node answering .onion and nothing else. That value is set in exactly two places, both under inventories/development/, and the documented production path reaches neither - the provisioner writes no networks key at all. A reader following the role README verbatim therefore cannot deploy a Tor node.
CI has been proving this since 2026-08-31 without anyone seeing it: the README replay rides on the role's smallest variant and is skipped when the deploy is red, so the only row that ever reached it is one where svc-net-tor itself went green. That happened first on Ubuntu, where the local-service defect abb6f37297 fixes never fired.
The template now emits the DNS assignment, the -e passthrough and the networks entry in --vars for a role whose tasks name networks.internet.dns, which today is svc-net-tor alone; the generator derives that rather than taking a declaration, so a second role asking for the value documents it by doing so. scripts/github/guide/compose_deploy.sh substitutes the placeholder from INFINITO_DNS_IP, the same SPOT inventories/development/default.yml reads, next to the existing domain substitution.
This follows 972596a758, which added the administrator key to the same block for the same reason. No fallback inside the role: defaulting to NETWORK_PUBLIC_DNS_RESOLVERS would route a Tor node's clearnet queries to Cloudflare without the operator choosing it, which is what the assert exists to prevent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34893363518 lost the fedora workspace job at certs.sh with exit 125. docker run resolved alpine:latest itself, the token request to auth.docker.io came back as 'read: connection reset by peer', and that single transport fault ended the job. Five of the six workspace jobs pulled the same image successfully in the same run, so the cause is transport, not a Docker Hub policy response, whose signature would be toomanyrequests after a successful token exchange.
Every other network operation on this path already retries: scripts/github/runner/pull_with_retry.sh serves two workflows, scripts/image/build.sh has its own ladder for the build frontend, and cli/administration/deploy/development/compose.py retries compose up six times. This call site had no attempts at all, so pre-pull through the existing helper rather than a new loop.
The attempt budget is 3x10s instead of the helper's 7x20s default because this runs as a prerequisite on the interactive local path, where two minutes of sleeping without a network would be felt. The call stays non-fatal so a locally cached image still carries an unreachable registry, the same semantics 0df75d83a6 established for the buildx bootstrap; set -euo pipefail is why it needs the if-guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two places in the DR drill ruled on a systemd unit without looking at what the
unit was doing.
[8b/9] took the exit code of `systemctl try-restart nfs-ganesha` as the verdict.
The unit carries Restart=on-failure from the role's own drop-in, and that
drop-in also hangs a 20 second readiness probe on port 2049 as ExecStartPost.
Right after the backing FS is wiped and restored, that probe runs out while the
container still waits on its own stale NFS mount, so systemd kills the daemon it
just started and reports a failed start - then restarts it three seconds later
and succeeds. Measured in run 34699757369, jobs 103572236910 (Matrix#0) and
103638497583 (Gitea#0): 'Control process exited, code=exited, status=124',
'Failed to start', then 'Scheduled restart job, restart counter is at 1'. The
restart attempt may now fail; the assertion is a bounded wait on is-active, and
it aborts loudly with systemctl status when the unit never arrives. The
coherence probe that follows is unchanged and still has to pass.
[0/9] waited for in-flight backups with `systemctl is-active --quiet
'svc-bkp-*.service'` and did not wait. In job 103588089327 (Moodle#0) the volume
unit ran from 01:00:56 to 01:01:12 while that loop returned after 0.36s for four
nodes. The drill then attached to the running job instead of starting its own,
and [3b/9] refused the generation it found because it predated the drill by ten
seconds.
That second change is a hardening, not a proven fix: the reason the loop
returned early is still unknown. The obvious explanation does not hold - the
systemd manual says is-active "Returns an exit code 0 if at least one is
active", so a pattern matching one running unit among four idle ones should have
blocked. list-units --state=active expresses the same intent without relying on
that semantics. Closing the gap needs the call run on a node with exactly that
mix, which no local container offers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The config held literals for values vars/main.yml already owned, and the two
drifted without a sound. actionTimeout carried `|| 30_000` for its whole life
while nothing ever set PLAYWRIGHT_ACTION_TIMEOUT, so every run used the
unscaled 30 s. expect had no entry at all, leaving Playwright's built-in 5 s to
govern every assertion over Tor while its neighbours were scaled by the onion
multiplier. That is what killed the second Nextcloud login attempt: an
expect(usernameField).toBeVisible() that gave up after 5 s inside a 150 s
budget.
vars/main.yml now owns the values in TEST_E2E_PLAYWRIGHT_CONFIG_TIMEOUTS, the
harness renders that map into the staged .env both the deploy and
scripts/tests/e2e/rerun-spec.sh already pass, and the config demands each name
through requiredTimeout() instead of inventing a fallback. The two `-e` flags
that fed the old pair are gone with it.
scripts/lint/playwright.sh stages the same config and runs `playwright test
--list`, which loads it, but built its stub env only from each role's
playwright.env.j2 and no such template carries a timeout. Without this it would
have aborted all 87 specs on a missing variable. It now takes the names from
the same map; the value is a placeholder because --list consumes none of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ubuntu CI image build (job 103277644636) hung on Canonical's
archive: apt fetched 26 MB at 417 kB/s, then 61.6 kB in 29 s, then sat
on the package download until the job was cancelled after 23 minutes,
while download.docker.com delivered 42.8 MB in under a second on the
same runner. Locally archive.ubuntu.com and security.ubuntu.com took
20-26 s to the first byte against 2 s for a university mirror.
INFINITO_APT_UBUNTU_MIRRORS in default.env is now the one list of
Ubuntu mirrors, in failover order: de.archive.ubuntu.com, ftp.fau.de,
ftp.halifax.rwth-aachen.de, archive.ubuntu.com.
- Image builds: the Dockerfile sources the list alongside the install
scripts and runs scripts/install/apt-mirrors.sh first. In an image
with ubuntu.sources it points both stanzas at apt's mirror+file
method, priorities in list order, and sets Acquire::http::Timeout to
10 s, so apt moves on from a mirror that fails or goes silent; other
distros are untouched. In ubuntu:latest apt-get update took 8 s
against 110 s on the stock sources, a dead first mirror was left
behind within 25 s against 62 s on apt's defaults, and the swarm node
packages (80 debs) came in 17 s. The Dockerfile's own RUN lines ran
clean on ubuntu, debian, fedora and arch.
- Package cache: package.sh proxies the first entry as the primary
Ubuntu repos and the second as their fallback. Nexus revalidates
metadata on every request here (metadataMaxAge 0), which costs a
full download from a mirror without an ETag; both de.archive and
ftp.fau.de send one, and the 15 MB universe index now comes back
from the cache in 0.02 s.
- tests/external/repository/test_apt_ubuntu_mirrors.py keeps the list
honest: the first mirror must answer InRelease within 5 s (median of
three) or the test names a faster later one, no mirror's -updates may
lag the freshest by more than 24 h, and every entry must be the
primary archive, a Canonical country alias or an enabled official
mirror in Launchpad's registry. It caught the first draft of the
list: mirrors.edge.kernel.org is not an official Ubuntu mirror.
After pulling, an existing checkout needs `make dotenv-force`: plain
`make dotenv` re-reads the old .env through BASH_ENV and keeps its
values.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The version cron updates docker image tags and repository refs, and tests/external/update covers docker, addons and repository. Neither sees a version inside an Ansible task variable, and dependabot reads pyproject.toml rather than roles/. A pin set once therefore froze and nothing thawed it: backup-docker-to-local has not moved since it was written, and the automtu pin one commit back would have gone the same way.
utils/update/pip mirrors the docker module: collect the pins, ask PyPI, rewrite in place. Only exact == pins are collected, because a range or a VCS reference states an intent an updater must not overrule. The external check warns rather than fails, like its docker sibling, so a slow PyPI cannot redden a run and a release the repository has deliberately not taken is not a regression.
The cron job is the docker job with one script swapped, so a changed pin arrives as its own PR under update/pip-pins.
Executed: the collector finds both pins in the repository; a synthetic file rewrites from 1.0.0 to 1.1.1 with the indent, quoting and neighbouring line untouched; automtu==1.0.0 would be reported while a bare name and a git+ reference are passed over.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils/roles/guide.py picks "compose" if the role ships a stack and "host"
otherwise. The compose branch runs make compose-up, which starts the coredns
service from compose.yml, so the replay resolves the deployment's own names.
The host branch runs this script, a bare container on the default bridge with
no resolver anywhere.
That only started to matter with 4986e80dd4, which threaded DOMAIN into the
README replay so it deploys under INFINITO_DOMAIN instead of inheriting
infinito.localhost. Before it, sys-svc-nss resolved *.localhost through
nss-myhostname and no DNS server was needed; after it, a stackless role asks
for infinito.test and gets nothing.
web-svc-html shows the whole chain in run 33923190220. Its guide replay logs
"Verify 'html.infinito.test' is served after the reload" with [Errno -2] Name
or service not known, sys-ctl-hlth-webserver probes every NGINX domain and
exits 1 through script.py's `return 1 if errors else 0`, systemd refuses to
start hlth-webserver, and sys-service-terminator fails the play after 25
attempts. The same run's Fider replay, a compose-mode role, answers the
identical task with real HTTP headers.
Rather than rebuild the resolver by hand, the script now brings up the very
service the compose branch uses and reads the network back off the container,
so subnet, gateway, MTU, image, Corefile and command all stay in compose.yml.
The Corefile is rendered through CoreDNSCorefileRenderer so the script does
not depend on an earlier phase having produced it. coredns declares no
depends_on, so the infinito service stays down and INFINITO_IP4 is free for
the guide container; the run's own log shows the CI phase removing
infinito-nexus-core_default seventeen seconds before the replay starts.
-v /srv is the second failure in the same class. svc-storage-nfs-server dies
with "mount.nfs4 ... reason given by server: No such file or directory" on a
host reporting Storage Driver: overlay2, because name_to_handle_at answers
EOPNOTSUPP there and the server builds no export root. A real volume under
the export path takes it off the overlay.
From triage of run 33923190220. Each fix is independently complete.
penpot: ansible templates delegate_to before the task's merged when, and AnsibleUndefinedVariable escapes get_delegated_vars_and_hostname. PENPOT_BACKEND_HOST_NODE is set only by the resolve include, so the bootstrap task hard-failed on every host that include skipped instead of skipping with it. A block-level when cannot prevent this; move the tasks into a dynamic include.
matrix: the MDAD reconfigure looked for volume matrix_matrix_mdad_state (real name matrix_mdad_state) and container matrix-mdad (real name matrix). Both if-bodies were skipped while the task still reported changed, so a rotated S3 secret never reached synapse and every media write got SignatureDoesNotMatch. Move the shell into its own script and pass the real names through the environment. Also add no_log to the vars.yml render, whose diff printed 16 secrets in cleartext.
n8n: when the settings file and N8N_ENCRYPTION_KEY disagree, oclif's findCommand throws for every command, so 'n8n export:credentials' is unreachable in exactly the state that would need it. The re-key is one-way; pin it rotatable: false.
swarm-drill: distros.sh sees only rc=124 and cannot tell a wedged deploy from one the sweep deadline cut short. Say so where the governor caps it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d14354e892 added both probes reading only /etc/dnsmasq.conf, but svc-net-tor writes its onion zone to /etc/dnsmasq.d/tor-onion.conf and leaves the main file with nothing but a conf-dir pointer. Every Tor host of run 33698844864 therefore captured resolve-inzone.txt as '[no output, exit 2]' and all 471 sampler lines as inzone=unconfigured: during the resolver outage that ended Gitea#0, nobody ever asked dnsmasq for an onion name, and two investigations went without the one measurement that would have settled them.
Both probes now follow the conf-dir pointer, main file first so a lab node with its zone in the main config answers as before. Each is exercised by a test that runs it against real files rather than comparing its text, which the rescue constant had no test for at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0b1565c493 went in with --no-verify and without the autoformat pass, so the
trailing nocheck comment in run_within_deadline() sat sixteen columns right
of the comment column the next line uses. The quality gate's autoformat step
realigns it on every pass; this lands that realignment so the tree stops
dirtying itself after each gate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Production block never set one. inventory provision writes only
ansible_connection for a local host, so DOMAIN_PRIMARY stayed at the
group_vars default of infinito.localhost and the deployment served a zone its
own resolver knows nothing about. CoreDNS templates answers for
INFINITO_DOMAIN_RE and nothing else, so every lookup of the deployment's own
name came back NXDOMAIN.
Mastodon is where that surfaced. tootctl accounts create validates the address
through EmailMxValidator, which resolves MX and then, for the domain itself,
A and AAAA. All six queries returned NXDOMAIN in run 33574056480, the
validator recorded email :unreachable, and the admin account was never created.
Only the guide rows were affected: the matrix rows write
DOMAIN_PRIMARY: "{{ lookup('env', 'INFINITO_DOMAIN') }}" into their host_vars
and resolve fine.
The template now carries DOMAIN=<your-domain> in both the host and the
container form and threads it into --vars, with the container form passing it
through the docker run boundary as well. The two guide runners substitute it
the way they already substitute the SSH key, so the replay deploys under
INFINITO_DOMAIN like every other CI deploy. compose_deploy.sh moves its env
load ahead of the substitutions to have the value; host_deploy.sh had no env
load at all.
A production reader is asked for a domain now instead of silently inheriting
infinito.localhost, which no real deployment wants either. The shipped default
keeps its value: sys-svc-nss documents that *.localhost resolves through the
myhostname NSS module without any zone, and host-side tasks lean on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The EXIT trap handed out four fixed budgets, 1500 + 900 + 900 + 900, against
the 900s reserve the governor sets aside for it. Its first step alone was
allowed to run longer than the whole reserve, so the trap regularly outlived
the deadline and the outer `timeout -k 60` escalated to SIGKILL. Nextcloud#3
and Nextcloud#11 in run 33574056480 died that way, both at 20460s to the
second, with the diagnostics still running in their own process group and the
teardown never reached.
Each step now takes the smaller of its nominal budget and the time actually
left, recomputed before it runs, and leaves behind what every later step needs.
The worst case where each one exhausts its cap fits the reserve exactly: 480 +
120 + 30 + 120 + 30 + 120. Two steps that carried no timeout at all join the
accounting, because one unbounded step defeats the whole arrangement.
The cap only engages near the deadline. A deploy that fails on its own with
hours of budget left sees the nominal budgets unchanged, so the depth is
traded away only in the case where the exit code already names the cause.
What that case gains is a truthful ending. The job now reports the deploy's own
rc rather than 137, the cluster is torn down instead of leaked into the next
distro, and the reports are collected. A step cut short by its cap says so on
stdout and returns 124 to the caller, so a half-written rescue bundle can no
longer read as a complete one.
The drill is not made faster by any of this. A round too slow for
INFINITO_CI_DISTRO_BUDGET_SECONDS still fails, now as a clean 124.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
