Files
computer-playbook/docs/agents/action/ci-trigger-priority.md
T
kevinveenbirkenbachandClaude Opus 5 d9f8af9b16 fix(ci): refuse a mesh pin the row cannot take
A selection token may pin the mesh state as &vpn or &direct, but check_pins
had no arm for it. pinned_vpn_states intersects the pin with the states the
mode allows and only swarm meshes, so a token like @compose+clearnet&vpn left
an empty state list: the row produced no job at all and the run reported green
for a combination that never deployed. That is the outcome _reject exists to
prevent, and every sibling axis already refuses an impossible pin.

The new arm asks whether any candidate mode keeps a state rather than whether
the pin is reachable, so a held vpn axis still overrides the pin instead of
being refused. That behaviour has its own test and is unchanged.

An earlier check was removed with the compose-mesh revert of 2026-10-16
because it rested on meshable() and smallest_variant(), which that revert
deleted. This one reuses pinned_vpn_states, applies to the swarm-only axis the
revert restored, and covers both mesh states instead of only &vpn.

pin_vpn and vpn_mode now reach check_pins from all three callers. The run's
vpn axis reaches the validation job through call-orchestrator and
call-validation, which passed every other axis but this one. The priority
procedure and its skill state all eight axes a token carries, not six:
architecture and mesh were pinnable and, for architecture, already validated,
yet neither was documented.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-21 18:13:54 +02:00

8.1 KiB

Lead a CI run with a priority line 🔎

Turn the questions a branch still has open into the priority line of one manual CI run, so it answers as many of them as possible in the cheapest combination that still counts as an answer.

The run this builds differs from a retrigger: a retrigger replays what a run already judged, this one asks about rows nobody has judged yet or whose verdict is still disputed. It therefore states its axes instead of carrying them over.

Scope

  • At most 20 selections on the priority line. The bound is the point: a priority line long enough to fill the whole queue window stops being a priority.
  • workspace=false and chunk_gate=false.
  • Every selection names every axis: role, variant, mode, onion state, distro, filesystem, architecture and mesh state. Nothing is left to the rotation.

1. Collect the candidates

Two sources, run verdicts first:

  1. The selections of the last finished deploy run on the branch that are not green. cli.administration.deploy.ci.selections.failed_selections reads them straight out of the job titles, so each candidate arrives with the variant, mode and onion state it failed in.
  2. The root causes that are still open without a run to point at: a fix landed whose effect nobody has seen, or a diagnosis that was never confirmed against the deployed code.

Take source 1 in full, then fill up to 20 from source 2. A row that source 1 already covers is not added twice.

Stop and say so when the two sources together yield fewer than three candidates. A run that verifies two rows is cheaper to ask for by name.

2. Measure the axes before choosing them

python -m cli.administration.deploy.ci.timings --runs 6

It reports the median wall-clock per axis value off the finished runs of the branch, with the sample count behind each one. Measured on this repository, the spread is not marginal: swarm costs roughly twice what compose costs, and an onion row roughly a third more than a clearnet one.

Every token pins every axis. role#variant@mode+tor|clearnet%distro/filesystem:architecture&vpn|direct carries all eight, and none is left to the rotation. A run whose rows the rotation is still free to move is not a verification run: the combination that comes back is not the one the question was asked about, and a green row then proves nothing about the red one it was supposed to answer.

The measurement decides which value each axis takes, never whether it takes one:

Axis Pin the cheap value when Pin the expensive value when
mode the failure is not mode-specific the row failed only in one mode, or the question is about swarm placement, overlay networking or replicas
tor the onion is not involved the failure mentions onion addresses, circuits or the tor proxy
distro the failure is not package-manager specific the question names a distro, or the row failed on exactly one
filesystem the kind is deliverable, see the next step the question is about the storage driver
architecture the failure is not instruction-set specific the row failed on exactly one, or the question is about a native build
vpn the row is not swarm, where direct is the only state the failure is about overlay routing across the mesh

When timings.fastest returns None, the measurement has too few samples to choose. That is not permission to leave the axis open; it moves the choice one rung down:

  1. the value the candidate's own failing row carried, read off the source run's job title. It reproduces the failure, which is the point of the line.
  2. failing that, the only value the measurement saw at all, even below the sample minimum.
  3. failing that, the run's pool default: compose, clearnet, debian, btrfs.

Say in the report which rung each pin came from. A pin from rung 3 is a guess wearing a token, and the reader has to know that.

3. The filesystem is the one axis with a host condition

scripts/tests/deploy/utils/filesystem/resolve.sh serves a kind only when the host kernel carries it: zfs needs /dev/zfs or a modinfo zfs hit, the others need an entry in /proc/filesystems or a module. The distro images carry only the userland (scripts/install/filesystem.sh).

A pool or token of exactly one kind is read as a human naming it, so the applying step fails the row instead of substituting a kind the host can serve. Pinning a kind the runner cannot deliver therefore produces a red job that says nothing about the question.

The axis is still pinned; the condition decides which kind. Pick the kind that satisfies both:

  1. the measurement has samples of it, and
  2. no fallback was reported for it. One deploy job log that was assigned the kind settles it: a delivered kind logs docker-dataroot-filesystem: status=applied requested=<kind> effective=<kind>, a substituted one logs a fallback out of note instead. Use the ci-artifacts skill to pull that one log rather than every log of the run.

No kind satisfies both, because the branch has no samples yet: pin btrfs and say in the report that the pin rests on the pool default rather than on evidence. Never drop the axis to avoid the question.

4. Validate, then dispatch

The trigger validates on its own before it dispatches, and refuses the whole run when a token cannot deploy on this branch. That refusal is not about deploy minutes: call-orchestrator.yml runs the same check as its first job and every other job needs it, so a bad token never reaches a chunk. It reaches the branch's concurrency group, which entry-manual-steer.yml declares with cancel-in-progress, so the doomed dispatch cancels the run currently in flight before failing its own validation.

Check it first anyway when building the line by hand, because the same check outside the trigger costs a second and names every problem at once:

python -m cli.meta.ci.validate --priority "<tokens>" --modes auto --tor auto

--modes auto --tor auto at run level, because the tokens pin those axes per row: a run-level pool of one would refuse every token that names the other value, and a line that verifies a swarm question and a compose question at once needs both open at run level and closed per token.

Then dispatch:

python -m cli.administration.deploy.ci.trigger \
  --priority "<tokens>" \
  --workspace false \
  --chunk-gate false

No --mode, --tor, --distros or --filesystem. Those inputs pin an axis for the whole run, which the tokens already do per row, and a run-level pool that disagrees with a token refuses it. The only reason to reach for them is a line where every row genuinely shares one value, and even then the token form is what the report has to show.

Add --mode, --tor, --distros or --filesystem only for an axis that step 2 settled for the whole run rather than per token.

A refusal names the token and why. Fix the token, never the check: a variant that no longer exists means the list was carried over from an older run, and dropping the pin only moves the wrong row into the queue.

A warning does not refuse, and it carries two very different meanings. Tell them apart before ignoring it:

  • the role id is wrong. Candidates assembled from memory or from a conversation get prefixes wrong: web-svc-openresty warns, svc-prx-openresty resolves. Check roles/ for the id before dropping the candidate.
  • the role is outside the envelope. web-app-minio warns whatever is pinned, because its lifecycle is eol. That candidate does not belong on a verification line at all; no run will judge it.

5. Report what the run can and cannot answer

Every axis is pinned, so the report is not about what was left open. It is about what each pin rests on: the measurement, the row's own failing combination, or the pool default. A pin from the pool default is a guess, and a green row under it answers a question nobody asked.

Name every question the run will not answer because a candidate was pinned to one side of it. A line that verifies a fix in compose and says nothing about swarm is a legitimate choice; a line that does that silently is not, because its green result reads like a full answer.