The Validation step "Assert the base branch is fully merged into the PR
head" ran main_merged.sh, which fetches the live tip of main when the job
starts. PR runs wait in the queue for hours while image-update PRs land
on main several times an hour, so nearly every queued run failed
Validation on start and had to be re-pushed after another main merge
(PR #1265: behind_by=0 at push, behind_by=9 while its run still queued).
main_merged.sh takes a new INFINITO_MAIN_COMMIT override: when set, it
fetches that full commit SHA from the remote and requires HEAD to
contain it instead of the live tip; a set but malformed or empty value
exits 2. The Validation step passes github.event.pull_request.base.sha,
the main tip at event time, which the step's pull_request-only guard
always provides. The run tests the merge against that base, so the gate
still guarantees the tested tree contains it.
The local pre-push hook (make test-main-merged) leaves the override
unset and keeps checking the live tip: a push happens right now, so
lagging the current main is the real defect there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The workflows pin `runs-on: ubuntu-NN.NN`, so no job moves to a new Ubuntu
release behind our back, but nothing raised the pin either. A sixth job of
`🔄 Update versions` closes that: `utils/update/runner.py` reads the image
table of actions/runner-images, takes the newest x64 Ubuntu label whose
image carries no preview or beta badge, and raises every older bare
`runs-on: ubuntu-NN.NN` to it; arch-suffixed labels stay untouched. The job
opens the change as one PR over `.github/workflows`, so its CI run tests
the new image before it reaches main.
This commit carries the updater's first result: every job now runs on
ubuntu-26.04 (49 `runs-on`).
actionlint 1.7.12, the newest release, does not know `ubuntu-26.04` yet
(rhysd/actionlint#682, fix proposed in rhysd/actionlint#743), so
`.github/actionlint.yaml` declares the label until a release knows it.
act maps each `ubuntu-NN.NN` label the workflows use to the local platform
image, derived from `.github/workflows`, instead of a fixed list that did
not know ubuntu-26.04.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`ubuntu-latest` moved from ubuntu-24.04 to ubuntu-26.04 on its own
schedule, in the middle of run 39166850273, and every job that landed on
the new image met a different OS: sudo-rs instead of sudo, no ip_tables
loaded, and now the NFS probe-write of svc-storage-nfs-client failing on
26.04 swarm nodes. The same branch went red without a code change.
Every `runs-on` now names `ubuntu-24.04`. GitHub still rebuilds that image
weekly, so patches and tool updates keep arriving; only the release jump
becomes a deliberate change of the label. The CodeQL job drops its dead
swift/macos-latest branch, since no swift language is in its matrix. act
already maps ubuntu-24.04 to the local platform image.
A lint rejects any job whose `runs-on` contains a `*-latest` label.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Closing a PR left "Cancel: PR Runs on Close" (run 36385268627) on
pending for hours. The canceller joins the PR orchestrator's
concurrency group, so GitHub cancelled the orchestrator run within a
second. But a cancel re-evaluates the job-level if: of every live job
and spares each one whose condition still holds. The deploy chunks
started with always(), which always holds: test-workspace died at
06:11:59, while Chunk 0 kept deploying and its ~60 queued matrix jobs
kept waiting for runners. The group stays occupied until the occupant
is terminal, so the canceller parked behind the very run it had
already cancelled.
The chunks, test-runner-smoke, the fork-prereqs gate and the release
job now guard with !cancelled(). Outside a cancel it matches always():
they still run after a failed or skipped need. On a cancel it turns
false and the job is cancelled with the run.
done and report-main-failures keep always(). They exist to report a
cancelled run, and a done skipped on cancel would count as a passing
required check.
The new lint walks every workflow whose concurrency cancels in
progress, plus everything it calls transitively, and rejects a
job-level always() outside that allowlist. It found the release job,
which a line-based grep had missed. It also fails when an allowlisted
job stops needing the exemption, and when its scope no longer reaches
the deploy workflows, so it cannot go silently empty. Against the
pre-fix tree it reports all six offenders.
Behaviour change: manually cancelling a release run now stops the
retag too. cron-release-highest re-dispatches the release.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The validation job checks out the pull request head and runs make install from it, then setup-python saves the pip cache in its post-step, after that code has run with write access to the cache directory. The workflow is reachable through workflow_dispatch, where the cache key resolves in the default-branch scope, so a poisoned entry written there is restored by every later trusted run. CodeQL flags each run step after the checkout as poisonable.
Removing the cache removes the sink. The checkout stays as it is: read-only token, no persisted credentials, excluded under pull_request_target. Nine other setup-python steps keep their cache; this is the one job that executes untrusted code, and the comment says so where the next tidy-up would put the cache back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A run-wide draw proved one distribution and said nothing about the other four, while reporting green. Both are matrix-row axes now, assigned by the same deterministic rotation mode and tor already use, so one sweep exercises every distribution and every filesystem.
The two read the row's position like an odometer, the distro as the low digit. Turning both on the position directly covers only n of the n x m pairs whenever the pools are the same length, and no sweep unlocks it.
Both are part of a row's identity: glyphs in the job label, columns in the plan table, shards in the artifact name. Both are selectable through %distro and /filesystem, and a retrigger replays the distro off a failed title. It leaves the filesystem out, because a title states the kind the matrix assigned and a deploy may have fallen back from it.
A row's filesystem is a preference unless a human names it, on the run or in a token. Reading the matrix's own narrowing as a demand would fail rows for every transient condition the applying step is built to tolerate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two calls of call-validation.yml activate disjoint halves of it -- the
entry passes pr_scope and gets the branch-prefix and merge checks, the
orchestrator passes the selection and gets the selection check plus the fork
wait -- but both rendered as a bare '✅ Validation', reading like the same gate
run twice.
Name the calling jobs after the half they activate, and take the full history
only on the path that needs it: main_merged.sh resolves a merge base, the
selection check shells out to no git at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge assertion was a job nobody depended on. ci-orchestrator needed the
scope, the branch prefix and the affected roles -- not it -- so a PR whose base
was not merged in went on to spend five distros on a deploy sweep and reported
a single red job beside a chain that had already finished.
Validation is now a reusable workflow every entry calls, and what it checks
follows the event and the inputs rather than the caller: a pull request gets
its branch prefix and merge base asserted, a run carrying a selection gets that
selection checked, a fork PR waits here for the privileged run's images. The
wait and its gate move in with it, so the orchestrator no longer keeps a second
copy of that logic. Every downstream job depends on the gate.
The selection check is what the last two nights needed. cli.meta.ci.validate
resolves every token against the branch that would deploy it -- role, variant,
mode, onion state -- and reports all offenders at once. Until now the matrix
builder raised on the first bad token from inside a chunk job, so a stale
priority line cost one full CI round per entry and only ever showed the next
problem after the previous one was fixed. Measured on a real list: three
unusable tokens, one pass.
A bare role name that matches nothing is a warning, not an error: the
diff-derived whitelist legitimately names roles the lifecycle envelope filters
out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5.5