--roles-only collapsed the entire failed set, so a job that never reached a verdict came back as a bare role name and the rotation re-picked axes the run had already proved green, dropping the one combination it never got to. Nothing distinguished a verdict from an interruption.
collapse_failures() takes the failed set and the hard-failure subset: a role that failed outright collapses to its name, everything else keeps the exact role#variant@mode+tor it was given. A role with a hard failure absorbs its own unfinished rows, since the collapsed name already covers every variant of it.
With --strict the two arguments are the same list, so i8ciall, i8ciallon and i8cifo are byte-identical to before; a test asserts that agreement rather than leaving it to inspection. The new combination is exposed as i8civar, which is i8ciallon without --strict.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The trigger could only derive a selection: --failed from a source run's verdicts, --apps as a whitelist, and every axis carried over from that run. A verification run has no source run whose axes it wants to inherit; it wants the combination that answers an open question. --priority states the tokens, and five overrides state the axes.
--priority composes with --apps but collides with --failed, which builds its own line, and it skips the resume-offset computation: that only means something when a source run says where the regular line stopped.
Every dispatch is now validated first. call-orchestrator.yml already runs the same check as its first job and every other job needs it, so a bad token never reaches a chunk. What it reaches is the branch's concurrency group, which entry-manual-steer.yml declares with cancel-in-progress: a doomed dispatch cancels the run currently in flight and then fails its own validation. That cancellation is what this refuses, not the deploy minutes.
The whitelist is passed as empty when it is the all-sentinel, which is not a role name and would be reported as an unknown one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cheapest combination that still answers a question was picked by assertion, because nothing recorded the cost. Deploy job titles already carry the whole row and the API gives every job a start and an end, so the cost per axis value is read back off the runs that already happened.
Measured over six runs of this branch: compose 47m against swarm 83m, clearnet 52m against tor 79m. Choosing the axis is therefore worth minutes per row, not seconds.
Medians, not means, so one queue wait cannot outvote a hundred normal jobs, and a value below the sample minimum is dropped rather than ranked. fastest() answers None there, which means the caller leaves that axis to the rotation instead of guessing it.
The job labels are parsed with the same primitives the retrigger uses, so a title format change breaks both together rather than leaving this one silently measuring nothing. A filesystem row states the kind the matrix assigned, not the kind the deploy ran on, because a fallback is not recorded in the title.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both architectures were claimed to work and only one was ever exercised. The
deploy matrix now carries an architecture the way it carries mode, tor, distro
and filesystem, so every run proves both.
A job cannot ask for an architecture the way it asks for a distro, because the
architecture is the machine rather than something the job installs on it. The
matrix therefore carries a runner label and `runs-on` reads it, and an
architecture without a label in RUNNERS is a build error rather than a row that
lands on the wrong hardware and still reports green. assert_architecture.sh
checks the runner it actually got, because a label that stops resolving to
arm64 would otherwise go unnoticed forever.
Assignment is rotation, not chance: two architectures against five distros and
fifteen distro/filesystem positions are coprime enough that every combination
comes round. A role narrows the pool through `architectures` in its
meta/services.yml; without one it takes whatever the rotation hands it. An
empty intersection aborts instead of silently removing the role from the run.
Images are built per architecture and pushed under an architecture-suffixed
tag, then merged into the plain tag by manifest_all.sh; pushing the plain tag
from both builds would let the last one win and send half the rows to the wrong
platform. cleanup.sh now protects the `ci-<sha>-<architecture>` children of a
protected tag, because the plain tag is an index over them and deleting a child
leaves it unpullable on that platform.
label.py is split out of axes.py, which the 500-line limit no longer fits.
Writing a job title and reading one back belong together: a retrigger that
recovers an axis wrongly replays a different deploy than the one that failed,
and a drifting artifact name makes the failure reporter download nothing. The
leading `.*` in LABEL_RE is greedy on purpose so it anchors on the last mode
glyph; a reusable-workflow caller path carries a glyph of its own, and
test_selection.py guards the case.
Verified: 313 unit tests and 48 subtests across utils/github/variant,
utils/roles and cli/meta/ci. Locally, a deploy pinned to arm64 reaches ok=1444
and its image builds log aarch64-linux-gnu paths.
Not verified here: actionlint over the seven changed workflows, and a full
`make test`. The matrix doubles the row count, so runtime and runner
availability change for every CI run from here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The retrigger turned a suite off only when the source run had been
dispatched with that suite explicitly on. b1ca39e9dd introduced that
guard together with the feature itself, and it contradicts the same
commit's message and docstring, both of which state that the job record
is the only evidence.
The guard made the feature a no-op in its most common case. An input
left on its default renders no title segment and resolves to empty, so
config.get('workspace') was never 'true' for a run that reached the
suite on the 'auto' default. call-orchestrator.yml reads 'auto' as
global scope off the whitelist alone, and a --failed retrigger puts its
selection on the priority line while leaving the whitelist empty, so
every retrigger in the chain redeployed a suite that had already gone
green. Measured on run 33785198875: six workspace jobs green, and the
dispatch still went out carrying workspace=auto.
Drop the guard. suite_passed already refuses to retire anything that
failed, was cancelled, timed out, was skipped or never ran at all, so
the job record decides on its own.
Three tests cover the paths 8600f1d9aa left open, all of which pinned
the explicit 'true': green on 'auto', green with no recorded input, and
red on 'auto' staying on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule lived only in suite_passed's body, and it inverts easily on a
casual read: ask for the suite again when it failed. Under that reading
the retrigger aliases would drop the suite in their most common case,
because they force-cancel the source run before reading it, so a shard
that was still running comes back cancelled rather than red.
Only an all-green suite is retired. Failed, cancelled, skipped, timed
out, still running and never dispatched all keep it on the retrigger,
and each of those now has a test. The two docstrings enumerate them, and
so does the i8ciall description, which is where an operator reads it.
The regular line restarted at the head on every retrigger. resume_offset
compared the source run's deploy tokens against a freshly computed
ranking, but every axis rotates with the sweep number and a retrigger is
a new run with a new one: measured against the current 277-row ranking,
2 of 277 tokens matched. The walk stopped on its first row every time,
and the 66 rows past the job budget never deployed at all.
Membership now compares role#variant, the identity the rotation cannot
move, and a row counts as proven only when every job it had was green.
A priority row deploys its whole cross-product, so one role#variant can
carry several verdicts, and a green compose job must not carry a red
swarm one past the line. The offset also inherits the source run's own
offset as its floor rather than falling back to the head.
The priority line drops --include-unrun with it: only the hard failures
and the priority entries the run never deployed belong there. The rows
it never reached are the regular line's turn now that the offset walks.
A retrigger copies every dispatch input from its source run, chunk_gate
included, so a sweep that stopped at its first red chunk came back stopping
there again, and nothing short of a manual dispatch could ask the chain to
deploy every chunk and report all of it at once.
`infinito administration deploy ci trigger --chunk-gate {true,false}` now
overrides the carried value after the source run's configuration is read; it
is the one deliberate departure from "reproduce the source run exactly", and
the module docstring says so. Omitted, the carried value or the workflow
default applies as before.
`i8ciallon` is `i8ciall` with `--chunk-gate false`: force-cancel the run, put
everything without a verdict back on the priority line, and keep deploying
past a failed chunk. The alias lint caps a name at two adjacent consonants,
which is why it is not `i8ciallnonstop`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A retrigger by role hands the priority line a bare role name so the rotation
can pick fresh axes, and the matrix then drops that role from the regular
query entirely. The resume offset was computed against a ranking discovered
without the priority line, so it could name one of exactly those rows, and
the run it was handed to had already guaranteed the row would not be there.
Chunk 0 of sweep 77 died on it before deploying anything:
offset 'web-app-nextcloud#11' names no row of the regular line
and with it every other chunk of the run. The priority line of that run
promoted 27 roles whole, nextcloud among them; its regular line held no
nextcloud row at all.
Give resume_offset the priority pins and let it skip what they claim, by the
same two rules the matrix applies: a variant-less pin takes its whole role, a
pinned one takes those variants. The walk continues past a claimed row rather
than stopping, so a promoted role in the middle of the green window does not
truncate the resume point, and a window that is claimed end to end yields no
offset and starts at the head.
The ranking itself still discovers without the priority line, so an
unresolvable token there still cannot abort the helper.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The instructions test held a job of its own that picked one random role per
run, so a README's Production block was proven roughly once per 265 roles.
The replay now rides on a deploy row instead: no extra runner, and every role
the sweep touches gets its documented one-command deploy exercised.
The row is the role's smallest variant -- fewest services enabled -- because
that is the shape the README describes, and it is chosen among the variants
the sweep actually deploys, so a row cut as redundant coverage cannot silently
drop the role from the test. A role qualifies only when its README carries a
Production block and the guide's own mode is one the role runs and tests.
The marked entry carries the guide mode in `instructions` and a 📖 in its job
title, between the variant number and a priority ⭐. The guide mode is
independent of the mode the rotation gave the row: a swarm row still replays
in compose. The step runs last in the deploy job, after the swarm teardown,
because the replay deploys the role a second time and needs the runner's
ports.
call-test-instructions.yml, cli.meta.ci.guide_select and the `instructions`
dispatch input go with it. `redundant()` moves next to the marker so the
chunker, the plan table and the marker share one definition.
The plan table trades its 📊 Weight column for 📖 Instructions; nothing read
the weight any more, so it leaves the matrix entries too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The offset a retrigger carries skipped every row the source run merely
started. A row that failed or was cancelled therefore retired its place in
the ranking: the regular line resumed past it, and the row came back only on
the priority line, pinned to the one combination it was recorded in. Every
other combination of that row was silently dropped from the sweep.
resume_offset now stops at the first row that is not green, so only proven
coverage is skipped. passed_selections supplies it; the three token
comprehensions that only differed in their state filter collapse into one
helper. deployed_selections and settled_selections keep feeding
--include-unrun unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two switches and an alias for the case f4c5a7cc36 was not written for: a run
where most of the ranking is red and much of it never ran.
--roles-only collapses the failed selections to their role ids. f4c5a7cc36
replaced exactly that form with pinned tokens, and its reason holds: the
rotation may not draw the combination that broke. It holds for a handful of red
rows. At thirty roles the question stops being which combination broke and
becomes whether the role comes up at all, and a token per combination spends
the priority budget proving the same role red twice. The default is unchanged;
this is the trade, named, behind a flag.
--include-unrun puts the rows the source run holds no job for on the priority
line, with their axes. resume_offset covers a run that walked a contiguous
prefix; it stops at the first gap, and a run whose priority line consumed the
budget before the head of the ranking leaves a hole at row one. Then the offset
is empty, the retrigger starts over, and the rows nobody judged queue behind
the ones already settled.
With --strict the two agree on what settles a row. Only a success or a failure
does. A cancelled row, or one still running when the run was read, was never
given the chance to be either, so it comes back as itself rather than as a role
name that would let the rotation draw some other combination instead. On the
run this was written against that moves 24 rows: 123 started, 99 judged.
Membership is decided on role#variant. Mode, onion state and distro are
assigned by the sweep rotation, so comparing full tokens between a source run
and a freshly computed ranking reports almost every row as unrun -- measured,
266 of 269, against 177 once the identity is right. Each ranking row is a
distinct role#variant, which is what makes it the identity to compare on.
i8ciall carries the combination, next to i8cifail which reproduces the exact
red rows and i8cifo which takes only the hard failures.
resume_offset and the new code now build their tokens through one helper
rather than two copies of the same five lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
