A cancel request does not stop a running step. The runner asks it to end, and a step that ignores the signal keeps going, bounded only by its own timeout-minutes -- 350 for a deploy. Meanwhile the concurrency group holds the newer run on pending until the occupant is terminal, not until it is cancel-requested. Measured on run 32859801382: the cancel was accepted at 15:27:45 and thirteen minutes later ten deploy jobs were still running and the newer run still had not started.
The sweep now waits FORCE_CANCEL_AFTER_SECONDS and force-cancels whatever is still not completed. Cleanup steps are skipped for those, which is the price: the artefacts of an already superseded run help nobody. Only entry-cancel-superseded.yml sets the variable -- a closed pull request and a deleted branch have nothing waiting on the group, so they keep cancelling and walking away.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
force_cancel_in_progress existed so a manual sweep could supersede a running orchestrator on any ref. It was a flag every caller had to remember to pass, and only one ever did.
A called workflow inherits github.event_name from its caller, so the orchestrator can see for itself that a human dispatched the run. The input is gone; the group reads the event instead. Automatic triggers keep following the ref alone -- everything but main -- and a manual dispatch supersedes everywhere, main included, because someone typed it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A long orchestrator sweep sometimes survives its concurrency group: the occupant sits on queued while its matrix waits for runners, cancel-in-progress does not reap it, and the newer run parks on pending until someone cancels the sweep by hand. PR #380 sat that way for nine hours behind a run from the previous commit.
entry-cancel-superseded.yml cancels those leftovers through the API. The group stays primary and untouched: it reaps within seconds and needs no runner, so the fallback normally arrives to find nothing left to do. It carries no concurrency group of its own, because the run holding a group cannot be the run that frees it.
Selection mirrors what the group would have done: cancel every matched run except the newest in the group instance, keyed the way the group is keyed. Not by head SHA, which goes stale while the job waits for a runner and would cancel the newest commit's CI while sparing the stale one. INCLUDE_PATHS keeps the sweep inside the group's own members, so a push to main cannot take the cron workflows with it.
deleted_branch_runs.sh becomes branch_runs.sh, since it now also serves live branches, and the shared parts move to lib.sh. A rejected cancel is logged and the remaining runs are still attempted; only a denied token fails the job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI_CANCEL_IN_PROGRESS gated whether a new run cancels the one before it, and it had to be remembered per repository: unset upstream meant cancel, set to false meant do not, and a fork inherited neither. Cancellation now follows the ref. Every branch but main supersedes its own runs; main never does, because a half-finished pipeline on the default branch is worse than a queued one.
call-orchestrator.yml compares github.ref rather than github.ref_name: a called workflow inherits the caller's ref, which is refs/pull/<n>/merge for a pull_request caller. The manual entry workflow keeps overriding the rule through force_cancel_in_progress.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32118850138 lost the :53 listener, so the sampler counts listeners.
Run 32538854067 lost DNS with the listener still bound: sockets-udp.txt
showed dnsmasq holding 0.0.0.0:53 while every forwarded query timed out,
and the sampler reported that hour as healthy.
Each tick now also records what a glibc caller gets back — for the zone
this resolver owns, derived from its own address= line, and for one name
it must forward — with the latency that caller paid. The rescue collector
probes the in-zone name too: probing external names alone cannot separate
"forwarding is gone" from "dnsmasq is gone".
The sampler now also runs on the swarm lab DNS nodes; it only ever ran in
the compose path, which is why the Magento artifact carried no sampler log
at all. Its location comes from group_vars FILE_DNS53_SAMPLER_LOG and is
required at every call site, so a run cannot quietly collect nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every scheduled workflow opened with an alarm clock, so the Actions list
read as a column of identical icons that said nothing about the run. The
clock now lives in run-name behind github.event_name == 'schedule', where
it marks the one thing it can mark: a run nobody triggered. The name says
what the run does, in two or three words.
The dispatch menu shows names in one narrow column and truncates them, so
a new lint caps a workflow_dispatch name at 25 characters and points the
longer, explanatory wording at run-name, which carries no such limit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Scorecard and the version updater had no concurrency group, so every merge
to main queued another full run alongside the ones already in flight.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A run-wide draw proved one distribution and said nothing about the other four, while reporting green. Both are matrix-row axes now, assigned by the same deterministic rotation mode and tor already use, so one sweep exercises every distribution and every filesystem.
The two read the row's position like an odometer, the distro as the low digit. Turning both on the position directly covers only n of the n x m pairs whenever the pools are the same length, and no sweep unlocks it.
Both are part of a row's identity: glyphs in the job label, columns in the plan table, shards in the artifact name. Both are selectable through %distro and /filesystem, and a retrigger replays the distro off a failed title. It leaves the filesystem out, because a title states the kind the matrix assigned and a deploy may have fallen back from it.
A row's filesystem is a preference unless a human names it, on the run or in a token. Reading the matrix's own narrowing as a demand would fail rows for every transient condition the applying step is built to tolerate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two scripts sharing a delimiter-bounded name prefix are a folder in disguise, and tests/lint/repository/structure/test_prefix_grouped_files.py refuses them. A second release_* resolver is coming, so give the group its directory now and drop the prefix from the name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
86389f856 added the input with a 113-character description; test_name_emoji_and_length caps a dispatch input at 80. 53e5e9ba2 set the convention this missed: the description says what the input is, the workflow README says what the values do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workspace swarm track passes disable=<39 provider keys> to make swarm-zombie, which forwards it to act as a runner env var. Both deploy steps then declare 'env: disable: ${{ matrix.disable }}', and a step env beats the runner env - so for variant 0, whose matrix cell is empty, the whole set was dropped.
Effect in job 95881470584: the MariaDB-only swarm deploy also brought up web-app-keycloak (629s), web-app-mailu (295s), svc-db-openldap, web-app-matomo, web-app-dashboard, svc-bkp-volume-2-local and sys-svc-cdn, plus test-e2e-playwright (997s) across all of them - while svc-db-mariadb itself, the role under test, took 169s. sso, email, ldap, matomo, dashboard, container_backup and css all sit in the disable set that never arrived. The job died at the 345-minute workspace limit.
The input defaults to an empty string and matrix.disable still wins where it is set, so the main CI passes nothing and behaves exactly as before. Verified: act accepts --input disable= with an empty value (exit 0); the || expression already runs in call-release-version.yml and call-test-runner-smoke.yml.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The id leads, the variant follows the name it belongs to, and covered_by,
weight and the new clone column close the row. ✂️ said only that a row was cut
while two different reasons hide behind it: 🔰 points at the earlier row that
already embeds this one, 🐑 says the row shares its service set with an earlier
role that stands in for it. 27 rows of a full plan carry 🐑 and were until now
indistinguishable from the rest.
_COLUMNS is the order and the headers again. tor and triggered used to sit
outside it as special cases, so a reordering had to be made in two places and
the tests indexed by hand; they now ask _COLUMNS for a column's position.
The chunk column loses its ⏭️ marker, which named the block the calling job was
running. Without it cells() has no use for `current` and --chunk no effect, so
both go, along with the workflow's argument.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-entry group serialized independent role deploys and kept jobs
queued instead of cancellable. Commented out with a TODO so the group key
can be reworked before it comes back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
clone and covered_by both name a stand-in row: another row carries this one's coverage, so the sweep may drop it. Both were decided over every row and then applied to a filtered set, so a run that selected a subset deferred to a stand-in the filter had already removed - the row was cut in favour of a deploy that never happens. --whitelist svc-db-mariadb yielded no row at all, and the swarm-zombie job passed green while drilling nothing.
The sort's last-resort tie-breaker was drawn per process, so discovery and the plan renderer could name different assignments for the same run. INFINITO_DISCOVERY_SEED derives it from the row's own identity instead; the workflow feeds github.run_id, so the rotation still happens, once per run rather than once per process.
Measured against the previous order: on the mode-only filter both cut sets are identical (swarm 220 rows, 44 cut; compose 243 rows, 54 cut), so the full sweep is unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two calls of call-validation.yml activate disjoint halves of it -- the
entry passes pr_scope and gets the branch-prefix and merge checks, the
orchestrator passes the selection and gets the selection check plus the fork
wait -- but both rendered as a bare '✅ Validation', reading like the same gate
run twice.
Name the calling jobs after the half they activate, and take the full history
only on the path that needs it: main_merged.sh resolves a merge base, the
selection check shells out to no git at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The regular rows began wherever (sweep * capacity) % count pointed. On run 210
that was offset 68: chunk 0 opened at candidate 69, the coverage-first head of
the list went undeployed, and no two runs could be compared because the window
moved with the run number. A run that looked wrong turned out to be merely
rotated, and finding that out meant recomputing the arithmetic by hand.
The rotation is gone. Chunk 0 reads the ranking from the top unless an offset
says otherwise, and the offset is an input now, threaded from the manual form
through the orchestrator and the deploy call down to the matrix builder. The
plan renderer takes the same flag, so the table shows the window the run will
take.
The offset is a row count or a selection token. A token is the honest form:
the ranking shifts whenever a variant appears, so 'offset 84' means something
different tomorrow while 'web-app-nextcloud#5' still means the same row. Axes
the token leaves open match anything, so a bare role name starts at its first
row and a pasted job label starts at exactly the job it names. A token that
matches no regular row aborts rather than silently starting at the head an
operator meant to skip.
The run title carries it as ⏩ followed by the value, and only when one was
asked for, so a rotated-looking run says so in its own name and the retrigger
reads it back from there when a job log has expired.
resume_from_chunk leaves the manual form with it. Skipping whole chunks was
the coarse way to say 'carry on further down'; the offset says it exactly and
keeps the rest of the chain intact. The orchestrator input stays for the other
entry points.
What this gives up is automatic tail coverage. Rows beyond the budget are
reached by moving the offset, widening the budget or narrowing the selection,
not by waiting for consecutive sweeps to walk the list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge assertion was a job nobody depended on. ci-orchestrator needed the
scope, the branch prefix and the affected roles -- not it -- so a PR whose base
was not merged in went on to spend five distros on a deploy sweep and reported
a single red job beside a chain that had already finished.
Validation is now a reusable workflow every entry calls, and what it checks
follows the event and the inputs rather than the caller: a pull request gets
its branch prefix and merge base asserted, a run carrying a selection gets that
selection checked, a fork PR waits here for the privileged run's images. The
wait and its gate move in with it, so the orchestrator no longer keeps a second
copy of that logic. Every downstream job depends on the gate.
The selection check is what the last two nights needed. cli.meta.ci.validate
resolves every token against the branch that would deploy it -- role, variant,
mode, onion state -- and reports all offenders at once. Until now the matrix
builder raised on the first bad token from inside a chunk job, so a stale
priority line cost one full CI round per entry and only ever showed the next
problem after the previous one was fixed. Measured on a real list: three
unusable tokens, one pass.
A bare role name that matches nothing is a warning, not an error: the
diff-derived whitelist legitimately names roles the lifecycle envelope filters
out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were left by earlier commits and are collapsed by markdownlint-cli2 and shfmt respectively, so every make quality run left the tree dirty and the change would eventually have ridden along in an unrelated commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An input description is what an operator reads in the dispatch form, one line
per input, exactly like a job or step name -- and it was drifting the other
way: eighteen of them opened with a bare word where every name in this
repository opens with an emoji, and sixteen ran past the line, the longest at
382 characters explaining four tor values inside a form field.
The descriptions now say what the input is; the workflow README says what the
values do. It gains the selection-token syntax and a table of the four tor
values, which is where a paragraph belongs when it outgrows its form field.
workflow_call inputs are left alone. They are wiring between workflows and are
rendered for nobody.
Two explanations did not survive the cut and are worth knowing: the mirror
cleanup no longer warns that the default GITHUB_TOKEN may lack the
delete:packages scope, and the workspace track no longer spells out that
'rotate' flips with the run number.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A whitelist or priority entry was a role name and nothing else, so a run
could not ask for "this variant, on swarm, behind the onion". It either
covered everything the role offers or waited for the rotation to come
around to the combination that broke.
An entry is now a token that may carry the axes, either as
web-app-x#0,2@swarm+tor or as the job label pasted straight back out of
the run that failed. What a token leaves open still belongs to the line
it stands in: priority covers every remaining combination in one sweep,
whitelist keeps the rotation and its position in the matrix. A pin the
row cannot take aborts the matrix rather than dropping the row, because
a dropped row reports a green run for a combination that never ran.
The onion state spells out as +clearnet, not as a -tor suffix: svc-net-tor
would otherwise read as role svc-net on the clearnet.
Chunk membership still follows the discovery ranking, which is what
decides who makes the budget cut, but the rows inside a chunk are sorted
by display name, variant, mode and onion state, so a chunk's job list
reads like the plan table instead of like the sweep's rotation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A workflow_dispatch input declared type: number reaches the calling
expression as a string. Handing that to a workflow_call input declared
number is a type mismatch, and GitHub rejects the job before it is
created -- no failed job, no log, just a run that stops after
pick-distro.
Run 31863925900 died exactly there. The push entry on the same sha
(31863720155) built all 33 jobs including the orchestrator, so the
callee chain was never the problem: only the manual entry passes
resume_from_chunk, and it was the sole type-risky input among the four
it adds.
Both sides become string with default "0". The comparisons are
unaffected -- GitHub casts a string operand to a number, so
'inputs.resume_from_chunk <= 1' in the chunk gates and '!= 0' in the
run-name keep their meaning. No type: number input remains in any
workflow.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
call-test-deploy-{compose,host,swarm,single-node}.yml differed only in
which mode they pinned and which artifact prefix they wrote; compose and
host already ran the identical drill. They become one call-test-deploy.yml
whose matrix rows carry the mode, so the swarm-specific steps are guarded
on it and everything else is shared.
The orchestrator loses its eight deploy jobs and the sequencing job for
three serial chunk blocks. GitHub Actions cannot generate a variable
number of jobs, so the blocks are written out and INFINITO_CI_MAX_CHUNKS
tells the budget how many exist; blocks whose slice is empty skip
themselves. chunk_gate stops the chain at the first red chunk,
resume_from_chunk re-enters after a fix instead of re-running the green
chunks, and modes collapses to a single mode choice.
apps.sh now emits matrix entries for one chunk rather than a token list
for one mode, and no longer sorts them: the discovery order is what the
chunker slices on. split_runner_apps.sh reads its runner quota from
INFINITO_CI_CONCURRENCY instead of carrying a second copy of the 20.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test-dns depended on build-ci-images, which skips its push step for a fork PR
and reports success anyway, so the dependency was satisfied by a job that built
nothing. The images it pulls come from the sibling pull_request_target run, and
nothing made it wait for them: arch finished at 00:10:40 while the pull ran at
00:09:28. fedora was no safer -- it completed four seconds after the job even
started, and only survived because its pull fell later in the probe order. The
race was thin for four of five distros, so no retry count would have settled it.
The gate for exactly this already existed and only done consumed it.
wait-fork-prereq-run polls docker manifest inspect through
scripts/meta/wait/images.sh until every ci-<sha> distro ref is present, and
fails early once the privileged run ends unsuccessfully; fork-prereqs-ready
enforces its result. The three jobs that pull those images directly now depend
on it.
test-deploy-swarm-priority and test-instructions were exposed the same way and
are wired too -- gating only the job that happened to fail would have left both
to lose the next close race. The remaining test-deploy-* jobs inherit the gate
transitively, which was computed over the needs graph rather than assumed.
Same-repo PRs are unaffected: wait-fork-prereq-run carries no job-level if,
reports "implicitly ready" outside fork PRs, and the gate script accepts both
success and skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
For a fork PR both clauses of the gate were true, so assert-main-merged ran
once per event. checkout@v7.0.0 now refuses the fork-head checkout under
pull_request_target outright, and the trusted copy fails on every push while
the untrusted one passes. Dropping the second clause removes a checkout of
fork code held against the base repo token rather than authorising one with
allow-unsafe-pr-checkout.
Nothing is lost. The assertion reads git history and needs no token, the
pull_request run already performs it, and no job declares needs on it. Same-repo
PRs never matched the second clause anyway.
The three jobs that genuinely need the trusted context keep it untouched:
images-mirror-missing-fork pins source_repository to github.repository, so it
never sees fork code despite holding packages: write; build-ci-images-fork
switches to the fork head only behind the "Trusted" label; prepare-fork-merge-ref
checks out the base with no ref at all. Those were wired through
prepare-fork-merge-ref in 158d57342. assert-main-merged never was, which is why
the guard caught it alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The onion glyph alone left a job name ambiguous: no glyph meant either
'clearnet' or 'this mode has no tor axis'. Swarm and compose entries now open
with the onion or the globe, so the name states which of the two networks the
job tested. Host keeps neither — it shares the matrix builder but has no onion,
and a glyph there would claim something the job never did.
'enabled' becomes 'enforced' so the manual selector says what it does: tor on
every role that can take one, not tor on everything.
'exclusive' is the new mode. keep_tor_variants runs before assign_tor and cuts
each entry down to its tor-capable variants, dropping entries left with none, so
a run tests the onion and nothing else. A bundle can come out with a hole in it
— variants 0 and 2 capable, 1 not, yields '0,2' — which filter_plan_to_variants
already handles by set membership; both shapes are now pinned by tests.
Note the word is overloaded: services.tor.exclusive drops a deployed app's
clearnet vhost, while this mode only narrows what CI runs. The docstring says so
where a maintainer will look.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The PR-close canceller had no concurrency group at all, so cancelling
depended entirely on scripts/github/cancel/pull_request_runs.sh, which
needs a runner before it can call the API. Runners were 20 minutes to 21
hours out, and run 31451265380 stayed alive that whole time.
The canceller now joins the group entry-pr-change-orchestrate.yml renders
for the same PR. GitHub clears the group when the run is created, before a
runner is assigned, so the orchestrator dies immediately and the script
becomes a backstop instead of the only path.
The group spells pull_request out literally rather than interpolating
github.event_name. This workflow is triggered by pull_request_target, so
the interpolated form would join the twin the orchestrator's own job
guards already skip on a same-repo PR and leave the working run untouched.
A fork PR runs both twins; the pull_request_target one is still left to
the script.
The branch-delete sweep filtered runs by head_branch alone and shot this
canceller mid-cancel. It now excludes the canceller's own path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tor coverage was decided per role variant and nowhere else: 1d9d355bc pins
services.tor.enabled true in variant 0 and false in every other variant, so an
operator had no way to steer it and a job name never said whether it ran behind
an onion. INFINITO_TOR now controls the axis from manual steer down to the
drill, and the entries that get an onion wear one in their label.
assign_tor decides per entry. 'enabled' gives tor to every capable entry,
'disabled' to none, 'auto' to every second one with the run number's parity
flipping which one starts — measured over the full matrix: 397 entries, 257
capable, 129 on even runs, 128 on odd, union 257, overlap 0. Two consecutive
runs therefore cover every capable entry exactly once.
tor_capable reads the entry's own variants, not just meta/services.yml.
Reading only the base config claimed an onion for 131 of 388 entries whose
variant had pinned the gate off — a third of the glyphs would have lied. A
bundle counts as capable when any variant it covers can use tor; a reactive
Jinja flag counts as capable, since that is the role saying 'onion when the
node is dark'.
The 'disable' token is the working half: it prunes svc-net-tor from the include
closure (utils/tests/swarm/derive_includes.py) and clears the service flags in
host_vars (cli/administration/inventory/provision/services_disabler.py). Both
halves are required — with only the first, extend_inventory walks the provider
back in. It runs after the variant overlay, so it also overrides a variant that
pinned the gate true.
Only swarm and compose join the axis. Host discovery shares the same matrix
builder, so without the allowlist its jobs would carry onion glyphs for a mode
that has no onion at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The hourly sweep fought the rest of the CI control plane instead of
helping it: it preempted itself (17 of the last 29 runs concluded
cancelled 1-2s after their successor was created, for 8s of work behind
a 74-80min queue wait) and shipped MAX_ATTEMPTS=0, which the script
reads as no ceiling while its own README promised the opposite. The 24h
age cutoff could not bound it either, because every rerun refreshes the
run's updatedAt and so keeps it inside the window.
Removes the workflow, the two scripts it was the sole consumer of, its
unit test, and the CI_RERUN_* documentation that would otherwise point
at a deleted file. The workflow is disabled in both origin and fork so
no sweep fires before this lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scripts/tests/environment/ becomes scripts/tests/workspace/ with one
subdirectory per track. base/ prepares the workspace and is a prerequisite;
compose/ and swarm/ are independent of each other, so CI can run them as
separate jobs with a budget each. 00_orchestrator.sh runs the tracks named
by INFINITO_WORKSPACE_TRACKS and aborts on an unknown one.
The job count stays at five: cli.meta.ci.dev_runtime matrix assigns exactly
one track per distro and flips the split with the run number, so two
consecutive runs cover both tracks on every distro without doubling the
matrix. The workspace_track input (rotate|compose|swarm) pins it from manual
steer through the orchestrator for a targeted re-run.
Job names carry the track glyph from utils/symbol_glossary and the image
shortened to its last two segments, so quay.io/centos/centos:latest reads
as centos:latest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`timeout-minutes` reports the job as cancelled, not as failed, so a step gated
on `if: failure()` never runs. The upload beside it is gated on `always()`, runs,
and finds an empty directory — the workspace centos job logged exactly that:
No files were found with the provided path: /tmp/rescue-diagnostics/.
That the upload did run also proves the runner still executes steps after the
cancellation, so widening the gate is enough. Every one of these four steps sits
on a job with a 345-355 minute budget, and three workspace distros burned that
budget in full on 2026-08-10 and produced no diagnostics at all.
call-test-deploy-swarm.yml needs no change: its routine carries an EXIT/TERM
trap (954b9f1ae) that covers the same case. There is no .github/actions/, and no
other `if: failure()` exists outside .github/workflows/.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every write scope sat at workflow level, so it reached each job in the file: entry-push-latest handed contents, packages, actions, security-events and issues write to all eight of its jobs, and call-orchestrator to all twenty-four, including every deploy and test job of the 134-job runs. The scopes now sit on the jobs that push, release or upload SARIF.
call-test-workspace, -install-make and -install-pkgmgr declared actions: write with no consumer at all, while passing the raw GITHUB_TOKEN into containers that start from mutable public tags and mount the docker socket. cron-cleanup-stale gave packages: write over the whole namespace to two jobs running third-party JavaScript that never touch a registry.
Eight single-job workflows are deliberately left alone: their one job needs the scope, so moving it produces an identical effective token while risking a job block that silently replaces contents: read. A check across every uses: edge confirms no callee now declares more than its caller grants.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
assert-main-merged checked out the PR head and then executed scripts/git/assert/main_merged.sh from that same tree. Under pull_request_target the job holds the base repository's token, so a fork PR could run anything it liked with write access. This is the run's only critical scanner finding.
The workflow file itself comes from the base branch and is trusted; only the checked-out files are not. Two checkouts now separate them: the script comes from .base-tooling, the tree it inspects is pr-head, and neither checkout persists credentials. Both repositories are public, so the script's git fetch still resolves anonymously, and git rev-parse --show-toplevel follows the working directory rather than the script's location - both verified by execution.
The same pass drops the top-level write scopes to contents: read, moves packages: write onto the two fork image jobs, and downgrades the orchestrator call site to actions: read now that call-orchestrator no longer declares actions: write.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both ran as soon as the mirror was ready, in parallel with lint and test, so a lint error still cost a full dev-runtime matrix at a 345 minute timeout. With lint and test in their needs, GitHub skips them when either fails - no if: is required because neither job carries always().
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workflow directory mixed entry points, reusable stages and scheduled jobs into one flat list sorted by topic, so the first question about any file - what starts it - was the one the name did not answer. Every workflow now carries one of three prefixes, derived from its on: block: cron- for a schedule trigger, call- for one only reachable through workflow_call, entry- for everything a repository event or a human starts. Scheduled workflows additionally carry an alarm clock as the first character of their name.
Two lints keep it that way. One derives the expected prefix from the triggers and compares it against the file name; it flagged all 17 offenders before the rename and none after. The other resolves every .github/workflows/<file> string in the repository against the directory, which is what a rename silently breaks in Markdown links, Python constants and shell API URLs alike.
Renaming pulled 69 pre-existing comments into the comment lint's changed-file scope. The vendored CodeQL template got the nocheck marker its scorecard sibling already carried, two workflow headers moved above name: where they count as file headers, and the remaining banners were deleted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A run cancelled by a concurrency group loses the work it had already finished. An hourly sweep now walks every branch, and when its latest run ended cancelled it restarts only the failed jobs through gh run rerun --failed, keeping the successful ones.
CI_RERUN_INTERVAL_MINUTES, CI_RERUN_MAX_ATTEMPTS and CI_RERUN_MAX_AGE_HOURS tune the sweep. A cron expression cannot read a repository variable, so the workflow is triggered every full hour and a gate lets only the hours through that open the configured window; the gate reads the hour, not the minute, so a late start still counts for its hour. An unset attempt ceiling means no ceiling, bounded only by the 50 reruns GitHub allows per run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Workspace, instructions and the compose, host and swarm deploy matrices now carry a job-level concurrency group, so one entry at a time runs repository-wide instead of per branch. Job level is required: only there does the matrix context exist. The deploy key joins apps and variant_slug with '#' because both fields contain hyphens themselves and a hyphen would make the pair ambiguous; apps alone is not unique, swarm expands one entry per variant and compose splits large roles into bundles. queue: max lifts the pending slot from one to a hundred, so a third concurrent run queues instead of cancelling the waiting one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every conflicted hunk resolves onto the swarm-nfs side, because on all six
files it is the strict superset: it carries the display-name codec
(utils/roles/display.py) the CI role selections now encode through, and the
sequencing glyph move from 910748ec4. svc-net-tor holds no unique content on
any of them -- runs.py even had non-conflicted hunks already calling `codec`,
so keeping its side would have left a NameError.
The lint fallout is settled in the same commit, as 4afc9b770 did for the
previous merge: the shell-or-true lint lives on svc-net-tor (c505ce746) and
swarm-nfs brings two unmarked `|| true` sites with it. Both get a nocheck
reason rather than a rewrite, because neither swallows anything -- the
slaptest call is the measured exception of c4bbbba93 and has_cn_config decides
right below it, and the dockerd restart is followed by an is-active poll that
fatals. In both the exit code was never the authority.
make quality green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The traffic light said "gate" everywhere else in the pipeline, on the PR
scope check and on the dependency gate, so reusing it for the deploy-mode
order made an ordering knob look like another pass/fail barrier. A crawler
reads as what sequencing is: segments following each other in a line.
Both failure-report entries in the orchestrator move to the ladybug in the
same commit, because they already held the caterpillar and one workflow
cannot spend the same glyph on ordering and on failures. The ladybug is what
the bug-report issue template already uses.
Only the test fixture spells the glyph out; the parser derives its segment
heads from the workflow's own run-name, so it follows the change on its own.
The traffic light stays where it means a gate, and as the release-candidate
symbol in the glossary.
A run title created before this commit still carries the traffic light, so
recovering its sequencing input from the title yields nothing and a retrigger
leaves that one input on the workflow default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The category labels now run together into a single block instead of being
separated from each other, the glyph sits flush against that block, and the
only `·` left is the one between the categories and the role's own fragment:
`🐳网络应用·Nextcloud` where it used to read `🐳 网络·应用·Nextcloud`.
The deploy-job parser drops the whitespace requirement behind the glyph in
the same move. Without it no job name from a fresh run matches any more, and
the retrigger would find zero roles without reporting an error.
The name still never holds a space, so the heading keeps collapsing its own
whitespace to `·` and the space-separated whitelist and priority inputs stay
parseable with more than one role. Empty parts are dropped when joining, so a
role with no category at all (pkgmgr, test-e2e-*) does not gain a leading
separator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI deploy jobs were titled with the raw role id, so a run page read as a
column of dash-joined prefixes. They now carry a display name: the category
path as hanzi labels, then the role's README heading or the part of the id
the categories do not already say, whichever is shorter.
utils/roles/display.py owns both directions. Encoding happens once, at
discovery time, as a `label` key on every matrix entry; the workflows render
`<glyph> ${{ matrix.label }}` and keep the glyph as a literal because the
retrigger reads the deploy mode off it.
Decoding is what makes the retrigger keep working. It accepts a display name
with or without its variant shard and a bare role id unchanged, so job names
from runs that predate this still parse and a hand-typed dispatch never has
to spell hanzi. The parser therefore stopped matching an id-shaped tail and
asks the codec instead; a job the codec does not recognise is skipped rather
than counted as a role.
The role selections go out as display names too, so the run title reads like
the jobs it spawns. Four consumers decode them back, each the single place
the value is actually used: the whitelist resolver before it checks the role
directory, the discovery query on its env, and the sequencing and plan CLIs
on their arguments.
Every fragment is joined by `·`, including before the title, and the heading
loses its whitespace and any emoji variation selector. A display name holding
a space would make the space-separated whitelist and priority inputs
unparseable with more than one role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The REST API answers inputs: null for a finished workflow_dispatch, so
the run name is the only record of what a manual run was dispatched
with. It carried one value, distros; everything else silently fell back
to the workflow defaults, and a failure that only reproduces on swarm
with btrfs came back green from a run that deployed all three modes on
whatever filesystem it drew.
Render every input as its own <emoji><value> segment instead, omitted
when it holds its default, so the title carries exactly what deviates
and an absent input means 'leave it on the default'. run_name derives
those segments from the workflow rather than knowing them, so the format
cannot drift away from the parser without a failing test.
distros no longer defaults to debian -- empty draws one at random, the
way untagged pushes already do. That leaves nothing in the title, so the
effective list is recovered from the discover job names, matched by
content against the meta/distros.yml SPOT rather than by job-name shape.
Priority roles that fell behind the budget cut in every mode hold no job
at all, so failed_roles cannot see them: they are neither green nor red,
they never ran. Carry them along or they never deploy.
--run now resolves before the selection, so it reproduces a run whatever
picks the roles, not only with --failed.
tests/utils/ci_*.py move into tests/utils/ci/ -- two files sharing a
delimiter-bounded prefix are a folder in disguise, and the structure
lint says so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub cancels a job queued for 24h. A line starting swarm, compose and
host at once queues its whole matrix in one go, so with more roles than
runner slots the tail waits past that cut and dies unrun -- 54 of 241
jobs in run 30937913632.
Deploy the modes one after the other instead, heaviest first, which
restarts the queue clock per mode. cli.meta.ci.sequencing counts a
line's matrix jobs on the row basis the discover steps use and
serialises above INFINITO_CI_SEQUENTIAL_THRESHOLD; below it the modes
still start together, so a small diff pays no wall-clock for this.
A needs: edge is static, so both layouts exist as jobs and the decision
skips one. Their conditions are structurally asserted, not evaluated:
ten near-identical if expressions invite a copied sentinel or a missing
chain edge, and both failures are silent at runtime.
mode_fail_fast stops a serial line at its first failed mode, on by
default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The REST API answers inputs: null for a finished workflow_dispatch, so
the run name is the only record of what a manual run was dispatched
with. It carried one value, distros; everything else silently fell back
to the workflow defaults, and a failure that only reproduces on swarm
with btrfs came back green from a run that deployed all three modes on
whatever filesystem it drew.
Render every input as its own <emoji><value> segment instead, omitted
when it holds its default, so the title carries exactly what deviates
and an absent input means 'leave it on the default'. run_name derives
those segments from the workflow rather than knowing them, so the format
cannot drift away from the parser without a failing test.
distros no longer defaults to debian -- empty draws one at random, the
way untagged pushes already do. That leaves nothing in the title, so the
effective list is recovered from the discover job names, matched by
content against the meta/distros.yml SPOT rather than by job-name shape.
Priority roles that fell behind the budget cut in every mode hold no job
at all, so failed_roles cannot see them: they are neither green nor red,
they never ran. Carry them along or they never deploy.
--run now resolves before the selection, so it reproduces a run whatever
picks the roles, not only with --failed.
tests/utils/ci_*.py move into tests/utils/ci/ -- two files sharing a
delimiter-bounded prefix are a folder in disguise, and the structure
lint says so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub cancels a job queued for 24h. A line starting swarm, compose and
host at once queues its whole matrix in one go, so with more roles than
runner slots the tail waits past that cut and dies unrun -- 54 of 241
jobs in run 30937913632.
Deploy the modes one after the other instead, heaviest first, which
restarts the queue clock per mode. cli.meta.ci.sequencing counts a
line's matrix jobs on the row basis the discover steps use and
serialises above INFINITO_CI_SEQUENTIAL_THRESHOLD; below it the modes
still start together, so a small diff pays no wall-clock for this.
A needs: edge is static, so both layouts exist as jobs and the decision
skips one. Their conditions are structurally asserted, not evaluated:
ten near-identical if expressions invite a copied sentinel or a missing
chain edge, and both failures are silent at runtime.
mode_fail_fast stops a serial line at its first failed mode, on by
default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pick was made once per matrix entry, in a workflow step ahead of the drill,
and every distro of that entry then ran on it. Coverage followed: one entry
exercised one filesystem, and the swarm never reached zfs at all, because
resolve.sh's userland_for() kept it out of any entry containing arch, centos or
fedora - which is every automatic sweep. The draw now happens inside the loop in
distros.sh, once per distro, so a five-distro entry covers zfs with 87% and all
three kinds with 62% probability instead of one kind with certainty.
The workflow input changes meaning with it. It was a single kind, offered as a
choice in the manual entry; it is now a space-separated allow-list. Empty draws
from all three, and naming kinds narrows the draw to them and marks the pick
required, so a host that cannot deliver one of them fails rather than quietly
running on something else. A name that is not a filesystem this project puts a
data root on is rejected outright instead of reaching the applying step.
For the draw to mean anything the images have to carry the userlands, so
scripts/install/zfs.sh becomes scripts/install/filesystem.sh and installs ext4
and btrfs alongside zfs. That is not symmetric work: btrfs-progs does not exist
in the EL family at all, because RHEL dropped btrfs, and a plain
`dnf install btrfs-progs` there fails the whole transaction and takes e2fsprogs
with it. EPEL carries it, so the EL family goes through EPEL from the start -
split by distro rather than by a failed first attempt, since Fedora has no
epel-release to fall back to and routing a transient failure there would turn it
into a fatal one. verify() now demands all four binaries, so a packaging
regression fails the build instead of leaving a hole the drill discovers later.
Measured on all five bases: debian trixie, ubuntu 26.04, arch, fedora 44 and
centos 10 each end with mkfs.ext4, mkfs.btrfs, zfs and zpool present and ldd
fully resolved.
Two defects only became reachable once a single job could ask for different
filesystems in turn:
* docker_dataroot.sh answered "already prepared" and returned for any mount it
found, so from the second distro on, every draw would have been silently
ignored on a host that survives the sweep - the runner in compose and host
mode. It now compares what is mounted against what was asked for and
dismantles the old root when they differ. Everything docker cached there
goes with it; that is what covering more than one filesystem per entry
costs.
* reclaim_stale_pool ran only when the new pick was zfs, so a zfs to btrfs
transition left the previous distro's pool imported on the shared kernel,
pinning a loop device over an image its dead container had already unlinked.
It now runs on every pick, which needed the /dev/zfs handling lifted out of
the zfs branch into zfs_device_ready().
The restore_docker trap is armed before anything can stop docker, rather than
after, so a failure while dismantling a data root cannot leave a host without a
daemon. resolve.sh splits its allow-list into an array before testing it, so a
whitespace-only input is the empty list it looks like rather than a narrowed one
that yields an empty pick marked required. pick.sh clears both keys when it
declines to draw: they are declared in default.env, so a generated .env carries
whatever was exported when it was made, and a caller gating its apply on the
value would otherwise repoint the developer's own docker in the very case the
skip exists to prevent.
Verified by execution rather than by reading: a privileged container ran
docker_dataroot.sh with real loop devices and mkfs through ext4 -> ext4 -> btrfs
-> ext4, confirming the recognition, the replacement in both directions and that
no loop leaks across it; a second harness with a recording zpool stub confirmed
that a btrfs pick does reclaim a stale pool, that no pool means no destroy, and
that a host without /dev/zfs still completes an ext4 pick without touching zpool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
