mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-09 12:17:13 +00:00
kevinveenbirkenbachandClaude Opus 5
d9f8af9b16
A selection token may pin the mesh state as &vpn or &direct, but check_pins had no arm for it. pinned_vpn_states intersects the pin with the states the mode allows and only swarm meshes, so a token like @compose+clearnet&vpn left an empty state list: the row produced no job at all and the run reported green for a combination that never deployed. That is the outcome _reject exists to prevent, and every sibling axis already refuses an impossible pin. The new arm asks whether any candidate mode keeps a state rather than whether the pin is reachable, so a held vpn axis still overrides the pin instead of being refused. That behaviour has its own test and is unchanged. An earlier check was removed with the compose-mesh revert of 2026-10-16 because it rested on meshable() and smallest_variant(), which that revert deleted. This one reuses pinned_vpn_states, applies to the swarm-only axis the revert restored, and covers both mesh states instead of only &vpn. pin_vpn and vpn_mode now reach check_pins from all three callers. The run's vpn axis reaches the validation job through call-orchestrator and call-validation, which passed every other axis but this one. The priority procedure and its skill state all eight axes a token carries, not six: architecture and mesh were pinnable and, for architecture, already validated, yet neither was documented. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
173 lines
8.1 KiB
Markdown
173 lines
8.1 KiB
Markdown
# Lead a CI run with a priority line 🔎
|
|
|
|
Turn the questions a branch still has open into the priority line of one manual
|
|
CI run, so it answers as many of them as possible in the cheapest combination
|
|
that still counts as an answer.
|
|
|
|
The run this builds differs from a retrigger: a retrigger replays what a run
|
|
already judged, this one asks about rows nobody has judged yet or whose verdict
|
|
is still disputed. It therefore states its axes instead of carrying them over.
|
|
|
|
## Scope
|
|
|
|
- At most 20 selections on the priority line. The bound is the point: a
|
|
priority line long enough to fill the whole queue window stops being a
|
|
priority.
|
|
- `workspace=false` and `chunk_gate=false`.
|
|
- Every selection names every axis: role, variant, mode, onion state, distro,
|
|
filesystem, architecture and mesh state. Nothing is left to the rotation.
|
|
|
|
## 1. Collect the candidates
|
|
|
|
Two sources, run verdicts first:
|
|
|
|
1. The selections of the last finished deploy run on the branch that are not
|
|
green. `cli.administration.deploy.ci.selections.failed_selections` reads
|
|
them straight out of the job titles, so each candidate arrives with the
|
|
variant, mode and onion state it failed in.
|
|
2. The root causes that are still open without a run to point at: a fix landed
|
|
whose effect nobody has seen, or a diagnosis that was never confirmed
|
|
against the deployed code.
|
|
|
|
Take source 1 in full, then fill up to 20 from source 2. A row that source 1
|
|
already covers is not added twice.
|
|
|
|
Stop and say so when the two sources together yield fewer than three
|
|
candidates. A run that verifies two rows is cheaper to ask for by name.
|
|
|
|
## 2. Measure the axes before choosing them
|
|
|
|
```bash
|
|
python -m cli.administration.deploy.ci.timings --runs 6
|
|
```
|
|
|
|
It reports the median wall-clock per axis value off the finished runs of the
|
|
branch, with the sample count behind each one. Measured on this repository, the
|
|
spread is not marginal: `swarm` costs roughly twice what `compose` costs, and
|
|
an onion row roughly a third more than a clearnet one.
|
|
|
|
**Every token pins every axis.**
|
|
`role#variant@mode+tor|clearnet%distro/filesystem:architecture&vpn|direct`
|
|
carries all eight, and none is left to the rotation. A run whose rows the
|
|
rotation is still free to move is not a verification run: the combination that
|
|
comes back is not the one the question was asked about, and a green row then
|
|
proves nothing about the red one it was supposed to answer.
|
|
|
|
The measurement decides **which** value each axis takes, never **whether** it
|
|
takes one:
|
|
|
|
| Axis | Pin the cheap value when | Pin the expensive value when |
|
|
|---|---|---|
|
|
| `mode` | the failure is not mode-specific | the row failed only in one mode, or the question is about swarm placement, overlay networking or replicas |
|
|
| `tor` | the onion is not involved | the failure mentions onion addresses, circuits or the tor proxy |
|
|
| `distro` | the failure is not package-manager specific | the question names a distro, or the row failed on exactly one |
|
|
| `filesystem` | the kind is deliverable, see the next step | the question is about the storage driver |
|
|
| `architecture` | the failure is not instruction-set specific | the row failed on exactly one, or the question is about a native build |
|
|
| `vpn` | the row is not swarm, where `direct` is the only state | the failure is about overlay routing across the mesh |
|
|
|
|
When `timings.fastest` returns `None`, the measurement has too few samples to
|
|
choose. That is not permission to leave the axis open; it moves the choice one
|
|
rung down:
|
|
|
|
1. the value the candidate's own failing row carried, read off the source run's
|
|
job title. It reproduces the failure, which is the point of the line.
|
|
2. failing that, the only value the measurement saw at all, even below the
|
|
sample minimum.
|
|
3. failing that, the run's pool default: `compose`, `clearnet`, `debian`,
|
|
`btrfs`.
|
|
|
|
Say in the report which rung each pin came from. A pin from rung 3 is a guess
|
|
wearing a token, and the reader has to know that.
|
|
|
|
## 3. The filesystem is the one axis with a host condition
|
|
|
|
`scripts/tests/deploy/utils/filesystem/resolve.sh` serves a kind only when the
|
|
**host kernel** carries it: `zfs` needs `/dev/zfs` or a `modinfo zfs` hit, the
|
|
others need an entry in `/proc/filesystems` or a module. The distro images carry
|
|
only the userland (`scripts/install/filesystem.sh`).
|
|
|
|
A pool or token of exactly one kind is read as a human naming it, so the
|
|
applying step **fails the row instead of substituting** a kind the host can
|
|
serve. Pinning a kind the runner cannot deliver therefore produces a red job
|
|
that says nothing about the question.
|
|
|
|
The axis is still pinned; the condition decides **which kind**. Pick the kind
|
|
that satisfies both:
|
|
|
|
1. the measurement has samples of it, and
|
|
2. no fallback was reported for it. One deploy job log that was assigned the
|
|
kind settles it: a delivered kind logs
|
|
`docker-dataroot-filesystem: status=applied requested=<kind> effective=<kind>`,
|
|
a substituted one logs a `fallback out of` note instead. Use the
|
|
`ci-artifacts` skill to pull that one log rather than every log of the run.
|
|
|
|
No kind satisfies both, because the branch has no samples yet: pin `btrfs` and
|
|
say in the report that the pin rests on the pool default rather than on
|
|
evidence. Never drop the axis to avoid the question.
|
|
|
|
## 4. Validate, then dispatch
|
|
|
|
The trigger validates on its own before it dispatches, and refuses the whole
|
|
run when a token cannot deploy on this branch. That refusal is not about deploy
|
|
minutes: `call-orchestrator.yml` runs the same check as its first job and every
|
|
other job needs it, so a bad token never reaches a chunk. It reaches the
|
|
branch's concurrency group, which `entry-manual-steer.yml` declares with
|
|
`cancel-in-progress`, so the doomed dispatch cancels the run currently in flight
|
|
before failing its own validation.
|
|
|
|
Check it first anyway when building the line by hand, because the same check
|
|
outside the trigger costs a second and names every problem at once:
|
|
|
|
```bash
|
|
python -m cli.meta.ci.validate --priority "<tokens>" --modes auto --tor auto
|
|
```
|
|
|
|
`--modes auto --tor auto` at run level, because the tokens pin those axes per
|
|
row: a run-level pool of one would refuse every token that names the other
|
|
value, and a line that verifies a swarm question and a compose question at once
|
|
needs both open at run level and closed per token.
|
|
|
|
Then dispatch:
|
|
|
|
```bash
|
|
python -m cli.administration.deploy.ci.trigger \
|
|
--priority "<tokens>" \
|
|
--workspace false \
|
|
--chunk-gate false
|
|
```
|
|
|
|
No `--mode`, `--tor`, `--distros` or `--filesystem`. Those inputs pin an axis
|
|
for the whole run, which the tokens already do per row, and a run-level pool
|
|
that disagrees with a token refuses it. The only reason to reach for them is a
|
|
line where every row genuinely shares one value, and even then the token form
|
|
is what the report has to show.
|
|
|
|
Add `--mode`, `--tor`, `--distros` or `--filesystem` only for an axis that step
|
|
2 settled for the whole run rather than per token.
|
|
|
|
A refusal names the token and why. Fix the token, never the check: a variant
|
|
that no longer exists means the list was carried over from an older run, and
|
|
dropping the pin only moves the wrong row into the queue.
|
|
|
|
A warning does not refuse, and it carries two very different meanings. Tell
|
|
them apart before ignoring it:
|
|
|
|
- **the role id is wrong.** Candidates assembled from memory or from a
|
|
conversation get prefixes wrong: `web-svc-openresty` warns, `svc-prx-openresty`
|
|
resolves. Check `roles/` for the id before dropping the candidate.
|
|
- **the role is outside the envelope.** `web-app-minio` warns whatever is
|
|
pinned, because its `lifecycle` is `eol`. That candidate does not belong on a
|
|
verification line at all; no run will judge it.
|
|
|
|
## 5. Report what the run can and cannot answer
|
|
|
|
Every axis is pinned, so the report is not about what was left open. It is
|
|
about what each pin rests on: the measurement, the row's own failing
|
|
combination, or the pool default. A pin from the pool default is a guess, and a
|
|
green row under it answers a question nobody asked.
|
|
|
|
Name every question the run will **not** answer because a candidate was pinned
|
|
to one side of it. A line that verifies a fix in compose and says nothing about
|
|
swarm is a legitimate choice; a line that does that silently is not, because its
|
|
green result reads like a full answer.
|