Output of 'make autoformat': ruff collapses expressions that were wrapped across lines without need, shfmt and the yaml formatters drop surplus blank lines. No semantic change - the whole diff is 10 insertions against 39 deletions, and every hunk joins lines that already said the same thing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The regular rows began wherever (sweep * capacity) % count pointed. On run 210
that was offset 68: chunk 0 opened at candidate 69, the coverage-first head of
the list went undeployed, and no two runs could be compared because the window
moved with the run number. A run that looked wrong turned out to be merely
rotated, and finding that out meant recomputing the arithmetic by hand.
The rotation is gone. Chunk 0 reads the ranking from the top unless an offset
says otherwise, and the offset is an input now, threaded from the manual form
through the orchestrator and the deploy call down to the matrix builder. The
plan renderer takes the same flag, so the table shows the window the run will
take.
The offset is a row count or a selection token. A token is the honest form:
the ranking shifts whenever a variant appears, so 'offset 84' means something
different tomorrow while 'web-app-nextcloud#5' still means the same row. Axes
the token leaves open match anything, so a bare role name starts at its first
row and a pasted job label starts at exactly the job it names. A token that
matches no regular row aborts rather than silently starting at the head an
operator meant to skip.
The run title carries it as ⏩ followed by the value, and only when one was
asked for, so a rotated-looking run says so in its own name and the retrigger
reads it back from there when a job log has expired.
resume_from_chunk leaves the manual form with it. Skipping whole chunks was
the coarse way to say 'carry on further down'; the offset says it exactly and
keeps the rest of the chain intact. The orchestrator input stays for the other
entry points.
What this gives up is automatic tail coverage. Rows beyond the budget are
reached by moving the offset, widening the budget or narrowing the selection,
not by waiting for consecutive sweeps to walk the list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge assertion was a job nobody depended on. ci-orchestrator needed the
scope, the branch prefix and the affected roles -- not it -- so a PR whose base
was not merged in went on to spend five distros on a deploy sweep and reported
a single red job beside a chain that had already finished.
Validation is now a reusable workflow every entry calls, and what it checks
follows the event and the inputs rather than the caller: a pull request gets
its branch prefix and merge base asserted, a run carrying a selection gets that
selection checked, a fork PR waits here for the privileged run's images. The
wait and its gate move in with it, so the orchestrator no longer keeps a second
copy of that logic. Every downstream job depends on the gate.
The selection check is what the last two nights needed. cli.meta.ci.validate
resolves every token against the branch that would deploy it -- role, variant,
mode, onion state -- and reports all offenders at once. Until now the matrix
builder raised on the first bad token from inside a chunk job, so a stale
priority line cost one full CI round per entry and only ever showed the next
problem after the previous one was fixed. Measured on a real list: three
unusable tokens, one pass.
A bare role name that matches nothing is a warning, not an error: the
diff-derived whitelist legitimately names roles the lifecycle envelope filters
out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were left by earlier commits and are collapsed by markdownlint-cli2 and shfmt respectively, so every make quality run left the tree dirty and the change would eventually have ridden along in an unrelated commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The converge gate polls STACK_NAME_PRIMARY_SERVICE_KEY. Four roles never
create that name and each was given a static `workload: node-local` to make
the gate skip itself:
21fe9e984 web-svc-cdn ships only cdn_build, never cdn_cdn
2c6c8b763 web-app-matrix names its services matrix_synapse and friends
76508c946 svc-runner deploys node-local compose projects per node
cb52a72f2 bigbluebutton phantom service, and the DR drill must still run
Four hand-written claims for one observable fact. The claim is also static
where the truth is not: web-app-discourse renders `services: {}` only while
its sidecars are shared, and job 94965456921 of run 31864448027 polled
discourse_discourse ninety times for a stack that had none. Declaring the
role node-local would have blinded the gate in the variants where its
sidecars are local and a stack does exist.
has_swarm_service asks the deployed stack for the exact name instead. Every
caller runs behind `if rc == 0`, so an absent service is the role's shape and
not a broken deploy. Exact-line matching, not docker's name filter, which
matches substrings.
No coverage is lost: 03_wait_converge cannot pass without that service, and
05/06/07 plus the backup drill only run once it has. The four roles keep the
skip they were given; discourse and every future case get it without a
declaration. The teardown branch in the drill keeps its meaning - stack rm
where the service exists, compose stop per node where it does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A whitelist or priority entry was a role name and nothing else, so a run
could not ask for "this variant, on swarm, behind the onion". It either
covered everything the role offers or waited for the rotation to come
around to the combination that broke.
An entry is now a token that may carry the axes, either as
web-app-x#0,2@swarm+tor or as the job label pasted straight back out of
the run that failed. What a token leaves open still belongs to the line
it stands in: priority covers every remaining combination in one sweep,
whitelist keeps the rotation and its position in the matrix. A pin the
row cannot take aborts the matrix rather than dropping the row, because
a dropped row reports a green run for a combination that never ran.
The onion state spells out as +clearnet, not as a -tor suffix: svc-net-tor
would otherwise read as role svc-net on the clearnet.
Chunk membership still follows the discovery ranking, which is what
decides who makes the budget cut, but the rows inside a chunk are sorted
by display name, variant, mode and onion state, so a chunk's job list
reads like the plan table instead of like the sweep's rotation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dnsmasq and apt-resolution investigations both stalled on facts the
snapshot never recorded.
The snapshot is taken by docker exec inside the DiD container and the
recursion only descends, so the runner's own docker - where the resolver
runs as ${INFINITO_RUNNER_PREFIX}-coredns - was structurally invisible.
dedicated.sh now runs a second pass on the CI host, with RESCUE_SEEN
seeded with the DiD id so it does not collect the first snapshot twice.
For the rest: a process table, so the MainPID that service-state reports
can be checked against something; the dnsmasq drop-in directory, because
this project's config lives at /etc/dnsmasq.d/tor-onion.conf and never in
the main file; nsswitch.conf and hosts, without which a getent verdict is
ambiguous; one probe per name plus, where the distro ships dig or
nslookup, one per nameserver, so a refusing resolver is distinguishable
from an unreachable mirror; the full firewall table set, since a dropped
query is recorded in filter and -t nat was actively hiding it; routes and
neighbours; os-release, because the matrix is distro-shaped and a bundle
could not say which leg it came from; the runtime's networks, which hold
the forwarder behind every container's 127.0.0.11; free inodes, which a
df -h reads as free space; and the runtime's disk, volume and image
inventory, the last with full references, since an engine was once picked
from a registry host embedded in an image name.
ps runs without args: the bundle is uploaded as an artifact and argv
carries whatever credentials a process was invoked with - a real proxy
password showed up in the first live run of this capture.
Two exec budgets are corrected while the per-container cost grows to
three calls. The per-container journal was borrowing the 600s meant for a
host boot journal, so one wedged container could cost 840s; container
journals span a deploy, not a boot. The cheap questions get 30s, and the
one collection site that had no outer bound gets the timeout 1500 the
others already carry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
02_performance deployed matomo with no disable set at all, so svc-net-tor joined the groups. It is exclusive, matomo follows it through tor.enabled in its own services meta, and the clearnet vhost disappears. The assertion two lines down then aborts in TLS before HTTP: 'curl: (35) TLS connect error: tlsv1 unrecognized name', reported as '[FAIL] https://matomo.infinito.example returned HTTP 000, expected 200'.
The counter-example sits in the same job: 01_minimal deploys dashboard with the group-conditional services disabled, tor among them, and its clearnet assertion returns 200. This suite tests the inventory, not the transport -- that a dedicated deploy replaces the previous app set. So it takes the route the sibling script already proves, rather than asserting against an onion address the runner never touched.
03_reuse stays untouched: it runs mode=update against the inventory 02 just built.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
call-test-deploy-{compose,host,swarm,single-node}.yml differed only in
which mode they pinned and which artifact prefix they wrote; compose and
host already ran the identical drill. They become one call-test-deploy.yml
whose matrix rows carry the mode, so the swarm-specific steps are guarded
on it and everything else is shared.
The orchestrator loses its eight deploy jobs and the sequencing job for
three serial chunk blocks. GitHub Actions cannot generate a variable
number of jobs, so the blocks are written out and INFINITO_CI_MAX_CHUNKS
tells the budget how many exist; blocks whose slice is empty skip
themselves. chunk_gate stops the chain at the first red chunk,
resume_from_chunk re-enters after a fix instead of re-running the green
chunks, and modes collapses to a single mode choice.
apps.sh now emits matrix entries for one chunk rather than a token list
for one mode, and no longer sorts them: the discovery order is what the
chunker slices on. split_runner_apps.sh reads its runner quota from
INFINITO_CI_CONCURRENCY instead of carrying a second copy of the 20.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
30% was a guess. Measuring it: across 60 successor distro runs in 22 jobs of run
31756400129, each against the running max of the runs before it, the median
overshoot is 0.991 - a successor is typically slightly faster - and the worst
ever observed is 1.149 (fedora 5541s against a 4822s max_seen).
15% therefore covers every observed case, and nothing above it buys more. It
costs less, too: 30% refuses 23 of 74 start decisions, 15% refuses 18. Five
distro runs that would have fitted, for no additional protection.
The tests read the percentage back out of the log line and check the formula
against it, so they hold at any value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
508669809 booked a budget SIGKILL as skipped so the job went green. That is a
soft-skip: a kill is indistinguishable from the command hanging, so it hid a
real failure behind a green tick.
Fail on it again, and prevent it upstream instead. A distro now starts only
while the remaining budget covers the slowest run so far plus 30%. Fediwall run
31756400129 shows why the bare comparison was not enough: max_seen=6770s against
remaining=6878s let the distro start, and it took 6942s, 64s past the deadline.
max_seen is the maximum of the durations already observed in the same sweep -
a small sample over distros of genuinely different weight, drawn in randomised
order, so it underestimates systematically.
Removing the soft-skip also removed its diagnosis, so a killed distro now
reports both numbers without claiming a cause: the run only knows the exit code
and how much budget was left when it started.
The suite drops from 85s to 4.1s. The 68s case made a process ignore SIGTERM to
force the -k escalation, which exercised coreutils rather than this script; the
124 path reaches the same branch in a second.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ad9222630 wrapped each distro in timeout so a run that outlives the budget stops instead of drifting into the action timeout, and mapped rc 124 to skipped because stopping at the budget is the documented intent rather than a defect. It verified that timeout returns 124 for its own kill. It did not cover the other one: timeout -k escalates to SIGKILL when the command survives SIGTERM, and then reports 137, which is equally its own kill. An ansible run with live docker children survives SIGTERM routinely, so 137 is the common case rather than the exotic one.
The result was that the governor never fired as documented. Fediwall swarm and Nextcloud swarm 0 in run 31756400129 and 31756560361 both ended on [ERROR] Run failed for distro=arch (rc=137) with budget=20400s remaining=-60s. Fediwalls playbook had already printed PLAYBOOK RECAP, the stack had converged and every node port answered; it was booked as a failure 0.3 seconds later.
The two narrowing conditions carry the discrimination and are unchanged. dur >= remaining - 5 rejects a SIGKILL that arrives at an arbitrary time, such as an OOM kill mid-run, because a budget kill lands at the deadline by construction. passed > 0 still requires an earlier distro to have succeeded.
A unit case in tests/unit/scripts/tests/deploy/test_distros.py drives a distro that traps SIGTERM and sleeps. Without the change it reproduces the CI failure locally, exit 137 and Run failed for distro; with it the sweep exits 0 and the summary marks the distro skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
default.env declares INFINITO_DOMAIN and everything is meant to derive from it.
Nothing checked that, so copies accumulated. A copy is a second source nobody
updates, and it fails in the least useful way: the code keeps running and
quietly talks about the wrong host. integration_mastodon is the worked example
-- it asserted the provisioned partner equalled a clearnet literal, so the spec
passed on a clearnet node and failed on an onion one while the product was
right in both.
tests/lint/repository/spot/test_no_hardcoded_primary_domain.py now fails on the
literal anywhere in tracked source. It reads the value it hunts for FROM
default.env, so the rule follows the SPOT instead of repeating it and would
otherwise be its own first violation.
Scope, and why each exclusion is not a loophole: default.env itself is the
source; .md is documentation; tests/ and scripts/tests/ hold fixtures that must
assert against a named host; inventories/ is where writing the real values is
the entire point; gitignored output carries the value because it correctly
derived it. Everything else is in.
Two real hardcodes are repaired rather than suppressed.
utils/domains/default_primary.py mirrors utils/roles/lifecycle.py -- one
function reading one key through parse_static_env -- and replaces the module
constant in utils/cleanup/nginx_vhosts.py and the argparse default in
cli/meta/domains/__main__.py. Both resolve to the same value as before.
Seven documentation sites become <primary-domain> placeholders instead of
carrying a nocheck: a Python docstring cannot host one without polluting the
docstring, and a placeholder does not rot when the default changes. The single
nocheck is scripts/system/tls/trust/wsl2.sh:61, where the marker is matched
verbatim against what an earlier run wrote into the Windows hosts file --
deriving it would orphan that block instead of replacing it.
Collateral, not intent: touching roles/sys-ctl-hlth-csp/files/script.py brought
it under the comment lint, which found five pre-existing narration comments.
Three deleted, `fn[:-5] # strip ".conf"` replaced by fn.removesuffix(".conf"),
and the root-user comment rewritten in the sanctioned Exception form. Its 33
tests stay green.
tests/lint/repository/test_no_hardcoded_container_name.py (05f425ea5) is the
same family for INFINITO_CONTAINER and could move into spot/ later; this change
leaves it where it is.
make test-lint: 326 passed. The rule found 18 sites before the cleanup and none
after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The runner sent both streams to /dev/null and then classified by exit code, so
a failure produced one line -- "FAILED ansible-lint" -- with no code and no
message. A lint finding, a crash and a lock conflict were indistinguishable,
which is exactly the state a quality run left me in while ansible-lint was
clean on the host at the same commit.
Output now goes to a temp file, is discarded on success because ansible-lint is
verbose when it passes, and is written to stderr together with the exit code
when it fails. The accepted codes are untouched, so a passing run behaves
exactly as before. run_ruff already reports this way, which is why the TID252
error it hit was fixable on sight.
This changes no outcome. The failure that motivated it has not recurred, so the
new path is still unexercised against a real fault.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`cli build include` generated one tasks/groups/<group>-roles.yml per invokable category during setup, and the stage plays included those files. They were written by the in-container UID, which is why a dedicated clean-container-owned helper had to delete them from inside the container.
The stage plays now loop over lookup('group_roles', <group>) and include tasks/utils/setup/{group,role}.yml directly. ordered_roles() in utils/roles/order.py holds the run_after topological sort the generator used to perform at build time.
utils/roles/stage.py:_group_names() no longer globs the generated files; it reads get_invokable_paths(), the same categories.yml source those files were derived from.
Measured: handlers still flush after every role (ansible-core 2.20.2, three loop items, three separate handler runs), and resolving all 18 groups costs 0.60s cold against 1.4s for the generator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill starts the backup units itself, but the deploy arms their calendar
timers first, and a job that runs long enough crosses every backup window in
group_vars/all/07_schedule.yml. web-app-decidim ran 23:05-02:24 and hit exactly
that: the drill's own volume backup succeeded at 01:00:49, then the timer fired
a second, unattended run at 01:03:18 while step [8b/9] was lazily unmounting
/var/lib/infinito underneath it.
The failure is silent by construction. baudolo writes the dump in two
processes - 'pg_dump > x.tmp', then 'mv x.tmp x'. The redirect shell exits 0
because a shell never reports a failed writeback on a soft NFS mount, but the
unmount discarded that mount instance together with the unflushed file, so the
mv resolved through the fresh instance and got ENOENT. The unit died at
01:04:04, the same second the drill finished remounting, and 80 minutes later
the CLI test read 'failed' and aborted the job before running its own backup.
Disarming the timers makes the drill the only thing driving those units.
Stopping a timer does not stop its service, and both the drill and
roles/svc-bkp-volume-2-local/files/test/test.sh start their units explicitly,
so nothing under test loses coverage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The PR-close canceller had no concurrency group at all, so cancelling
depended entirely on scripts/github/cancel/pull_request_runs.sh, which
needs a runner before it can call the API. Runners were 20 minutes to 21
hours out, and run 31451265380 stayed alive that whole time.
The canceller now joins the group entry-pr-change-orchestrate.yml renders
for the same PR. GitHub clears the group when the run is created, before a
runner is assigned, so the orchestrator dies immediately and the script
becomes a backstop instead of the only path.
The group spells pull_request out literally rather than interpolating
github.event_name. This workflow is triggered by pull_request_target, so
the interpolated form would join the twin the orchestrator's own job
guards already skip on a same-repo PR and leave the working run untouched.
A fork PR runs both twins; the pull_request_target one is still left to
the script.
The branch-delete sweep filtered runs by head_branch alone and shot this
canceller mid-cancel. It now excludes the canceller's own path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The hourly sweep fought the rest of the CI control plane instead of
helping it: it preempted itself (17 of the last 29 runs concluded
cancelled 1-2s after their successor was created, for 8s of work behind
a 74-80min queue wait) and shipped MAX_ATTEMPTS=0, which the script
reads as no ceiling while its own README promised the opposite. The 24h
age cutoff could not bound it either, because every rerun refreshes the
run's updatedAt and so keeps it inside the window.
Removes the workflow, the two scripts it was the sole consumer of, its
unit test, and the CI_RERUN_* documentation that would otherwise point
at a deleted file. The workflow is disabled in both origin and fork so
no sweep fires before this lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The quiesce guard spelled the dev container as the literal
infinito_nexus_debian, so it only ever matched a checkout whose .env was
generated for debian without a project prefix. It missed the self-hosted
runner form infinito_<project>_nexus_<distro>, any custom.env pin, and any
.env generated for another distro — and it missed them silently, because a
non-matching grep just skips the block and leaves the competing onion
publisher running.
INFINITO_CONTAINER is the SPOT for that name: compose.yml uses it as
container_name, and scripts/meta/env/load.sh (already sourced at line 19)
exports it. Proven against the container running on the author's host: the
literal does not match it, the variable does.
The outer `docker ps | grep -qx` pre-check is gone with it — `docker exec`
on an absent container already exits non-zero with stderr suppressed, and
the `if` absorbs that even under `set -euo pipefail`.
The variable joins the line-21 assertion so a broken env fails by name
instead of dying inside a pipeline subshell, which would reproduce exactly
the silent-skip class this commit removes. compose.yml already requires the
same variable with `:?`, so this is not a new demand on the environment.
tests/lint/repository/test_no_hardcoded_container_name.py pins it: run
against the pre-fix file it flags all three lines, and it stays green on
the distro-agnostic spellings in stack_refresh.sh and worktree/up.sh.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The split left the swarm track without any teardown: a `base swarm` run
started the development stack in base/04_bootstrap.sh and never brought it
down, so it leaked AppArmor, dev DNS and IPv6 state onto the host and ran
the swarm cluster alongside a live dev stack the old suite had already
released.
The old single job ran `make compose-down` + `make environment-teardown`
as step 10, immediately before 12_swarm.sh and 13_roundtrip.sh. The swarm
track now reproduces exactly that: 01_teardown releases the stack before
02_zombie and 03_roundtrip (which are byte-identical to the old steps apart
from their source path), and 04_teardown leaves the host clean — something
the old suite never did, since it ended on the roundtrip's restarted stack.
Both closing steps and the new leading one share utils/teardown.sh so the
teardown cannot drift between tracks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scripts/tests/environment/ becomes scripts/tests/workspace/ with one
subdirectory per track. base/ prepares the workspace and is a prerequisite;
compose/ and swarm/ are independent of each other, so CI can run them as
separate jobs with a budget each. 00_orchestrator.sh runs the tracks named
by INFINITO_WORKSPACE_TRACKS and aborts on an unknown one.
The job count stays at five: cli.meta.ci.dev_runtime matrix assigns exactly
one track per distro and flips the split with the run number, so two
consecutive runs cover both tracks on every distro without doubling the
matrix. The workspace_track input (rotate|compose|swarm) pins it from manual
steer through the orchestrator for a targeted re-run.
Job names carry the track glyph from utils/symbol_glossary and the image
shortened to its last two segments, so quay.io/centos/centos:latest reads
as centos:latest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The hourly rerun bot looked only at the newest run's conclusion. When that run was cancelled while an older one was still pending, reviving it added a second waiter to the ref's global-ci concurrency group — and cancel-in-progress evicts only in_progress runs, so neither could start and every later dispatch queued behind them.
Query the branch once with status included and leave it alone while anything on it is still active. The selection itself is unchanged; it now reads the same response instead of issuing a second one.
Scope, measured rather than assumed: this does NOT explain today's deadlock. The blocking revival happened at 14:51 on an idle branch, which the guard permits, and the later revival at 15:51 never produced a second attempt (run 31405424413 is still attempt 1). The guard covers a narrower case than the incident that motivated it.
scripts/github/rerun had no tests at all. The harness added here runs the script against a stubbed gh, following the pattern of tests/unit/scripts/tests/deploy/utils/test_rescue_index.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The routine only carried an EXIT trap, so a run terminated by the enclosing
`timeout` in distros.sh:128 left no rescue diagnostics, no playwright reports
and no teardown — the exact case where the artifacts matter most.
Add a TERM trap and make the handler take the status as an argument, so EXIT
keeps using `$?` while TERM passes 143 explicitly. The `_teardown_done` latch
is required because the EXIT trap still fires after the TERM trap; without it
the whole collect-and-teardown block, including its 1500s and 900s timeouts,
would run twice.
This works because GNU timeout signals the whole process group: bash defers a
trap while it waits on a foreground child, so a bare kill to the script PID
alone would not surface it until that child exited. Reproduced under the real
shape (timeout -k): TEARDOWN rc=143, exactly once, after 2.01s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Host and container share this checkout through a bind mount but run as
different uids -- 1000 and 0, measured. Whichever writes .ruff_cache or
.import_linter_cache first owns it, and the other is then locked out:
ruff aborts with "Failed to create temporary file: Permission denied" and
test_import_contracts fails the same way, reporting a broken contract that
is in fact perfectly intact.
Point both at build/<tool>-cache-$(id -u). build/ is already 0777 and
gitignored, and this is the mechanism the lint stamp
(build/install-lint-<hash>.stamp) already uses to keep the two environments
apart. Passing --no-cache would have silenced the symptom at the cost of the
cache on every single run.
The pre-existing .ruff_cache and .import_linter_cache are nobody-owned and
cannot be removed without root; they are simply unused from now on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm DR stage aborted with "no backup unit installed on <mgr> or
<nfs-server>" on all five workspace distros. routine/backup/base.sh triggers
svc-bkp-volume-2-local / svc-bkp-secrets-2-local on the manager and
svc-bkp-nfs-2-local on the export host; all three trigger_units calls returned
rc=2 (no unit matches), so _triggered stayed 0.
The units were never installed because the roles were never in the closure.
utils/tests/swarm/README.md pins the contract: derive_includes pulls
svc-bkp-volume-2-local in via the `container_backup` consumer and
svc-bkp-nfs-2-local via `nfs_backup`. derive_includes._disabled_provider_roles
maps those same keys back to those roles and `found -= disabled_roles` prunes
them, so any caller that lists them in `disable` silently removes what the
drill then requires.
scripts/tests/environment/12_swarm.sh and 13_roundtrip.sh both do exactly that:
`variable_services <app> node nfs-server` keeps three keys and disables every
other group-conditional one, which sweeps up container_backup and nfs_backup as
collateral. Reproduced offline byte-identically to the CI line:
$ python3 -m cli.meta.roles.applications.dynamic_services \
--exclude mariadb node nfs-server
asset,...,container_backup,...,nfs_backup,...
Filtering in those two callers would leave `make swarm-zombie app=X disable=...`
and any future caller exposed, so the filter goes to the single entry point every
swarm drill passes through: the CI matrix reaches routine/00_one.sh via
distros.sh, and swarm-zombie reaches the same script through act. Both `disable`
consumers -- the provision-time services_disabler and derive_includes in the
matrix subprocess -- read the env var this script re-exports.
Measured before/after on the derive_includes closure:
APP_ID=svc-db-mariadb -> {} / {svc-bkp-secrets-2-local, svc-bkp-volume-2-local}
APP_ID=svc-storage-nfs-server-> {} / {svc-bkp-nfs-2-local}
node and nfs-server join the set for the same reason db87a1a1c added them to the
callers: they build the lab cluster itself.
Token matching is exact, so nfs_backup2 and mynode survive; `IFS=', ' read -a`
avoids the pathname expansion an unquoted split would have allowed. The CI swarm
workflow passes no `disable` at all, so the filter is a no-op there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
shellcheck SC2016 flagged the single-quoted 'bash -c' body. The fd form
of flock keeps the guard in the current shell, so ${MAKE:-make} expands
normally and the lock is released before the test run starts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Parallel `make test` targets each auto-ran `compose-up`, which rendered
into a fixed Corefile.tmp; the first replace() stole the file from the
others and surfaced as "envsubst did not produce an output file".
mkstemp per process plus a flock'd re-check in the test wrapper so only
one target brings the stack up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extend_inventory._host_topology writes svc-swarm-node and
svc-storage-nfs-server into devices.yml unconditionally — without them
there is no swarm and no NFS export, so the throwaway cluster cannot
exist. Deriving the disable set from dynamic_services put their service
keys node and nfs-server on that same list, and the deploy aborted on
the provider-role branch of the consistency guard.
That branch never reads shared, so the force-shared fix cannot reach it:
with clean host_vars the guard still returns these two conflicts, and it
raises on any non-empty list.
Keep both keys instead of relaxing the guard — the guard is reporting a
real contradiction between the disable set and the lab topology, and the
disable set is the wrong side. roundtrip drives the same swarm-zombie
and extend_inventory path, so it needs the same keep-keys.
The keys are copied from _host_topology by hand; deriving them from it
would remove the duplication, at the cost of a helper this does not need
yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were created without the executable bit, which every sibling under scripts/ carries and which tests/lint/filesystem/shell/test_executable.py enforces.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workflow directory mixed entry points, reusable stages and scheduled jobs into one flat list sorted by topic, so the first question about any file - what starts it - was the one the name did not answer. Every workflow now carries one of three prefixes, derived from its on: block: cron- for a schedule trigger, call- for one only reachable through workflow_call, entry- for everything a repository event or a human starts. Scheduled workflows additionally carry an alarm clock as the first character of their name.
Two lints keep it that way. One derives the expected prefix from the triggers and compares it against the file name; it flagged all 17 offenders before the rename and none after. The other resolves every .github/workflows/<file> string in the repository against the directory, which is what a rename silently breaks in Markdown links, Python constants and shell API URLs alike.
Renaming pulled 69 pre-existing comments into the comment lint's changed-file scope. The vendored CodeQL template got the nocheck marker its scorecard sibling already carried, two workflow headers moved above name: where they count as file headers, and the remaining banners were deleted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A run cancelled by a concurrency group loses the work it had already finished. An hourly sweep now walks every branch, and when its latest run ended cancelled it restarts only the failed jobs through gh run rerun --failed, keeping the successful ones.
CI_RERUN_INTERVAL_MINUTES, CI_RERUN_MAX_ATTEMPTS and CI_RERUN_MAX_AGE_HOURS tune the sweep. A cron expression cannot read a repository variable, so the workflow is triggered every full hour and a gate lets only the hours through that open the configured window; the gate reads the hour, not the minute, so a late start still counts for its hour. An unset attempt ceiling means no ceiling, bounded only by the 50 reruns GitHub allows per run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub shipped the concurrency queue property on 2026-05-07. actionlint 1.7.12 and act v0.2.89 both predate it: actionlint reports an unexpected key, act refuses to parse the whole workflow. Ignore the finding in the lint call, and hand act a copy with those lines stripped so local iteration survives once the workflows use the key.
Refs: rhysd/actionlint#654, nektos/act#6095, nektos/act#6152
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every conflicted hunk resolves onto the swarm-nfs side, because on all six
files it is the strict superset: it carries the display-name codec
(utils/roles/display.py) the CI role selections now encode through, and the
sequencing glyph move from 910748ec4. svc-net-tor holds no unique content on
any of them -- runs.py even had non-conflicted hunks already calling `codec`,
so keeping its side would have left a NameError.
The lint fallout is settled in the same commit, as 4afc9b770 did for the
previous merge: the shell-or-true lint lives on svc-net-tor (c505ce746) and
swarm-nfs brings two unmarked `|| true` sites with it. Both get a nocheck
reason rather than a rewrite, because neither swallows anything -- the
slaptest call is the measured exception of c4bbbba93 and has_cn_config decides
right below it, and the dockerd restart is followed by an is-active poll that
fatals. In both the exit code was never the authority.
make quality green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI deploy jobs were titled with the raw role id, so a run page read as a
column of dash-joined prefixes. They now carry a display name: the category
path as hanzi labels, then the role's README heading or the part of the id
the categories do not already say, whichever is shorter.
utils/roles/display.py owns both directions. Encoding happens once, at
discovery time, as a `label` key on every matrix entry; the workflows render
`<glyph> ${{ matrix.label }}` and keep the glyph as a literal because the
retrigger reads the deploy mode off it.
Decoding is what makes the retrigger keep working. It accepts a display name
with or without its variant shard and a bare role id unchanged, so job names
from runs that predate this still parse and a hand-typed dispatch never has
to spell hanzi. The parser therefore stopped matching an id-shaped tail and
asks the codec instead; a job the codec does not recognise is skipped rather
than counted as a role.
The role selections go out as display names too, so the run title reads like
the jobs it spawns. Four consumers decode them back, each the single place
the value is actually used: the whitelist resolver before it checks the role
directory, the discovery query on its env, and the sequencing and plan CLIs
on their arguments.
Every fragment is joined by `·`, including before the title, and the heading
loses its whitespace and any emoji variation selector. A display name holding
a space would make the space-separated whitelist and priority inputs
unparseable with more than one role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-openwebui 0,1,2 was killed by the 350-minute action timeout with no
failed task anywhere in its 1.33M log lines. The sweep admitted fedora at
remaining=10372s after debian took 10028s, then never looked again: the
budget was only checked between distros. Its internal deadline passed 594
seconds before GitHub killed the job - the run knew it was over time for ten
minutes and had no way to act. Worst-case runtime was deadline plus the full
duration of the last admitted distro, unbounded.
The trigger was a single stalled task (803s for a no-op that takes 1.3s
warm, and a fully cold transaction 8 minutes later cost 24s), but that only
decided which job hit the wall. The wall itself is structural.
remaining is already computed at the top of the loop, so wrapping the call
in timeout adds no new state. rc 124 becomes 'skipped' rather than 'failed',
because a job that stops at its budget is the documented intent, not a
defect - test_distro_budget_fits_timeout.py already asserts this invariant,
which until now was simply untrue.
Two conditions narrow it, and both matter:
dur >= remaining - 5 keeps an inner timeout that fires early as a failure.
Verified: timeout returns 124 only for its own kill and passes inner exit
codes through unchanged (0 for success, 3 for an inner exit 3).
passed > 0 is the one that protects the test result. Without it, a job whose
very first distro overruns would report 'skipped' and exit 0 - green while
having verified nothing. That would have been a silent false pass across all
243 jobs of a run. The guard was executed over the full truth table,
including passed=0.
-k 60 is deliberate: one.sh execs into dedicated.sh and its ansible and
docker children can outlive SIGTERM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The REST API answers inputs: null for a finished workflow_dispatch, so
the run name is the only record of what a manual run was dispatched
with. It carried one value, distros; everything else silently fell back
to the workflow defaults, and a failure that only reproduces on swarm
with btrfs came back green from a run that deployed all three modes on
whatever filesystem it drew.
Render every input as its own <emoji><value> segment instead, omitted
when it holds its default, so the title carries exactly what deviates
and an absent input means 'leave it on the default'. run_name derives
those segments from the workflow rather than knowing them, so the format
cannot drift away from the parser without a failing test.
distros no longer defaults to debian -- empty draws one at random, the
way untagged pushes already do. That leaves nothing in the title, so the
effective list is recovered from the discover job names, matched by
content against the meta/distros.yml SPOT rather than by job-name shape.
Priority roles that fell behind the budget cut in every mode hold no job
at all, so failed_roles cannot see them: they are neither green nor red,
they never ran. Carry them along or they never deploy.
--run now resolves before the selection, so it reproduces a run whatever
picks the roles, not only with --failed.
tests/utils/ci_*.py move into tests/utils/ci/ -- two files sharing a
delimiter-bounded prefix are a folder in disguise, and the structure
lint says so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
07_compose_minimal.sh claimed to disable 'all variable (group-conditional)
services' and then named seven by hand. tor was not among them, so a
deploy meant to be minimal pulled svc-net-tor, which hard-depends on
web-svc-mirror, whose group-conditional prometheus flag the closure
resolver treats as true — dragging the whole web stack in. Steps 12 and
13 passed no disable at all, which is why a mariadb swarm test deployed
keycloak, mailu, matomo, prometheus and dashboard.
The set is now derived from the contract the repo already enforces
(tests/integration/roles/meta/services/test_dynamic_flags.py): a key is
group-conditional when enabled OR shared is a Jinja expression. Checking
only enabled would miss mariadb, postgres, redis, memcached, typesense.
Exclusions are load-bearing, not cosmetic: dashboard because disabling
the primary app aborts init; javascript because it aliases cdn, which
web-app-dashboard declares a hard dependency with no runtime fallback;
mariadb and postgres because steps 12 and 13 test them.
variable_services assigns to a local first — disable="$(cmd)" does not
set the exit status of a simple command, so a failing CLI would have
yielded an empty disable set and steps 12/13 would have run the full
deploy and reported green.
roundtrip could not take disable at all; it now reaches both the compose
and the swarm leg.
Not verified by a deploy: 33 of the keys are new to step 07, and the
variant axis of svc-db-mariadb/svc-db-postgres collapses because tor is
its only dimension.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-openwebui 0,1,2 was killed by the 350-minute action timeout with no
failed task anywhere in its 1.33M log lines. The sweep admitted fedora at
remaining=10372s after debian took 10028s, then never looked again: the
budget was only checked between distros. Its internal deadline passed 594
seconds before GitHub killed the job - the run knew it was over time for ten
minutes and had no way to act. Worst-case runtime was deadline plus the full
duration of the last admitted distro, unbounded.
The trigger was a single stalled task (803s for a no-op that takes 1.3s
warm, and a fully cold transaction 8 minutes later cost 24s), but that only
decided which job hit the wall. The wall itself is structural.
remaining is already computed at the top of the loop, so wrapping the call
in timeout adds no new state. rc 124 becomes 'skipped' rather than 'failed',
because a job that stops at its budget is the documented intent, not a
defect - test_distro_budget_fits_timeout.py already asserts this invariant,
which until now was simply untrue.
Two conditions narrow it, and both matter:
dur >= remaining - 5 keeps an inner timeout that fires early as a failure.
Verified: timeout returns 124 only for its own kill and passes inner exit
codes through unchanged (0 for success, 3 for an inner exit 3).
passed > 0 is the one that protects the test result. Without it, a job whose
very first distro overruns would report 'skipped' and exit 0 - green while
having verified nothing. That would have been a silent false pass across all
243 jobs of a run. The guard was executed over the full truth table,
including passed=0.
-k 60 is deliberate: one.sh execs into dedicated.sh and its ansible and
docker children can outlive SIGTERM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The REST API answers inputs: null for a finished workflow_dispatch, so
the run name is the only record of what a manual run was dispatched
with. It carried one value, distros; everything else silently fell back
to the workflow defaults, and a failure that only reproduces on swarm
with btrfs came back green from a run that deployed all three modes on
whatever filesystem it drew.
Render every input as its own <emoji><value> segment instead, omitted
when it holds its default, so the title carries exactly what deviates
and an absent input means 'leave it on the default'. run_name derives
those segments from the workflow rather than knowing them, so the format
cannot drift away from the parser without a failing test.
distros no longer defaults to debian -- empty draws one at random, the
way untagged pushes already do. That leaves nothing in the title, so the
effective list is recovered from the discover job names, matched by
content against the meta/distros.yml SPOT rather than by job-name shape.
Priority roles that fell behind the budget cut in every mode hold no job
at all, so failed_roles cannot see them: they are neither green nor red,
they never ran. Carry them along or they never deploy.
--run now resolves before the selection, so it reproduces a run whatever
picks the roles, not only with --failed.
tests/utils/ci_*.py move into tests/utils/ci/ -- two files sharing a
delimiter-bounded prefix are a folder in disguise, and the structure
lint says so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A debian:bookworm workspace job died on `apt-get update`: the mounted
sources list carried the Ubuntu entry too, Nexus had auto-blocked
archive.ubuntu.com, and apt fails the whole update on one bad source.
Three holes, three plugs:
- One sources list per INFINITO_DISTRO under compose/package-cache/apt/,
selected by the compose overrides. A Debian runner no longer fetches
an Ubuntu suite it has no use for. The non-apt distros get an empty
file so the bind mount has a source to point at.
- autoBlock off. While a remote is auto-blocked Nexus answers `404
Remote Auto Blocked` to every client instead of falling back to the
copy it already has cached.
- Mirror failover in the frontend. Every apt suite gets a second proxy
repo on an independent upstream; a 5xx from the primary re-runs the
request against it. Nexus has no group repository type for apt, so
the retry lives in nginx.
Measured on a live Nexus 3.91.1: dead remote without a cached copy
answers 502, with a cached copy 200 regardless of metadataMaxAge. The
frontend serves 149 kB of real bookworm `Release` from the mirror repo
when the primary is dead, and the new regex locations produce upstream
request lines byte-identical to the `rewrite … break` form they replace,
including `%3a` epochs in pool paths.
INFINITO_CACHE_PACKAGE_APT_LIST is gone; the mount interpolates
INFINITO_DISTRO directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In job 91983553540 all five lab nodes derived the same pool name,
infinito_ci_unknown, and the run died on `cannot destroy
'infinito_ci_unknown': pool is busy`. The name came from `hostname`, which
returns non-zero inside the node containers, so the `|| echo unknown` fallback
answered for every one of them. One node won the race and created the pool; the
others found it, took it for a leftover, and tried to tear down a pool a
sibling was actively using.
234c1158c already fixed this collision class three days ago by deriving the
name from `hostname` instead of a constant, and its message named the zfs
exclusion in resolve.sh as the only thing keeping the collision out of sight.
That fix was never exercised, because the exclusion stayed up until ed1121053
removed it - and it does not work, because the call it rests on fails in the
containers it was written for. Guessing the identity is the defect; the
playbook has known it all along as inventory_hostname, and ci/one.sh knows its
own deploy mode. Both now state it, as a required fourth argument with no
default, the way SIZE is already stated. verdict() stops guessing too: every
row it wrote into the job summary said `unknown`, which is why the summary was
never worth reading.
The second half is independent of the naming. reclaim_stale_pool ran
`zpool destroy -f` as a bare command under `set -euo pipefail`, the only
failure path in this script that bypassed decline(). A pool that refuses to be
destroyed is by definition not a leftover - something holds it - so that case
now declines through the usual contract: a drawn pick falls back and reports,
a stated one fails loudly as required-but-unavailable. A reclaim can no longer
be more destructive than the state it exists to clean up.
Verified over the real invocation path rather than a model of it: an
ansible-playbook run using the same `script:` task and docker connection as
compose/swarm/playbook.yml, against three containers in one play, has each node
address its own pool - infinito_ci_race-mgr-01, -wrk-01, -nfs-01 - with
failed=0 throughout. A stubbed-pool harness covers the paths that need a kernel
module this machine does not have: a sibling holding the pool declines with rc
0 instead of ending the run, the same case with a stated pick exits 1 through
the contract, a genuine leftover is still destroyed, and a missing identity is
a hard error. The earlier replacement harness (ext4 recognised, btrfs
replacing it, no loop leaked) still passes against the stricter signature.
Pool names now carry dashes, which zfs_namecheck.c accepts; at 42 characters
they sit far inside the 240 limit and match no reserved prefix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role README claimed the backup machinery "operates transparently against the mount
paths" and demoted the source-of-truth rule to advice in a prose box - both written
before 2efc73d3b made that rule the code's behaviour. It now states what is enforced:
this role skips export-backed volumes, svc-bkp-nfs-2-local is their single capture, and
manager-pinned roles, roles forcing another compose_mode_force and nfs: false opt-outs
stay covered here. It also names the limit the code actually has - the exclusion checks
that some host is in the svc-bkp-nfs-2-local inventory group, not that the role ever ran
successfully - because a README promising more than the code delivers is worse than one
promising less.
Its Schema section carried the role's timer flow; it now carries the mechanism the
change set is about: meta/volumes.yml -> swarm_nfs_backed() as the single predicate ->
its two consumers (the compose rewrite and the backup exclusion) -> the inventory-group
gate -> the export host's single capture -> the restore path with its mountpoint probe
-> the drill's [3b] guard against over-exclusion.
The two swarm READMEs drew the drill with eight steps and pre-773a25ab6 semantics. Both
now draw the nine that exist, including what makes the proof a proof: the pulled tree is
wiped before the device recover, so the device holds the only copy left; the live export
is wiped with the marker proven gone before it is restored; and [9] deletes the marker
server-side before restoring the volume through its bound backing store.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
