report_failures asked `gh api --paginate` for the run's jobs with the jq
filter `[.jobs[] | {name, conclusion}]`. gh applies that filter to each page
and prints one array per page, so a run with more jobs than one page holds
produced two JSON documents, and json.loads stopped with "Extra data: line 2
column 1". Main run 38218402254 (113 jobs) failed its "Report main role
failures" job that way, and no issue was opened for its failed roles.
The listing now emits one job object per line and parses the lines one by
one, the same way logs/download/github.py lists jobs.
Verified: the old parse of the real output of run 38218402254 raises the
same "Extra data: line 2 column 1 (char 8491)" as CI; list_jobs on that run
returns all 113 jobs, 16 of them failed, grouped into 12 failed roles.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The discovery step builds its table with plan.py, not matrix.py, so passing
--vpn aborted the chunk before a runner was spent on it. The plan now takes
the flag and shows the state as a column, the way it already shows the onion.
An image bump in a provider redeployed every variant of every consumer,
including the variants that switch that provider off and therefore never
deploy the bumped image at all. The diff-derived whitelist now narrows each
affected role to the variants whose own round closure still reaches a seed,
emitted as `role#0,2` selection tokens. That closure is walked on the same
variant-merged services maps the matrix planner builds its rounds from, so the
narrowing follows a parent's variant choice across every generation of the
chain. It covers every auto-derived whitelist: the weekly update PRs,
Dependabot, and any other diff-driven run.
The narrowing never shrinks the matrix on partial information. A seed role
keeps every variant, because its own files changed. A role reached only
through a run_after edge keeps every variant too, because the round closure
does not follow that edge and a role whose hits are empty is therefore
unnarrowed rather than dropped.
A pinned token naming a role the discovery query does not return now warns
instead of aborting the run. The diff-derived whitelist legitimately names
roles the lifecycle envelope filters out, and once it pins variants such a
token would abort every chunk; role names are still checked against roles/
before the run starts, so a typo is caught one layer earlier.
The round's variant merge moves out of the deploy inventory package into
utils/roles/applications/variants.py. The CI selection has to build the same
maps, and importing the deploy package for them pulls in a module that reads
INFINITO_SRC_DIR at import time, which the runner host does not set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
chunk_size() multiplied the runner slots a chunk may assume by the waves the
queue window affords, and nothing could say otherwise. The derivation is the
right default -- too few rows leaves runners idle, too many push a chunk's tail
past the queue cancel -- but a run that wants smaller chunks had to edit
default.env to get them.
INFINITO_CI_CHUNK_SIZE now takes precedence when it is greater than zero, and
zero keeps the derivation. The input travels the same path offset already
travels: entry-manual-steer.yml offers the field and maps empty to 0,
call-orchestrator.yml hands it to all four chunk blocks, call-test-deploy.yml
exports it as the env var. It is a string rather than a number for the reason
resume_from_chunk already documents: a workflow_dispatch number input reaches
the caller as a string, and a number-typed workflow_call input then makes
GitHub refuse the job before it exists.
Two places in call-test-deploy.yml set it, not one. The plan summary resolves
the chunk size itself, so exporting it only for the app resolution would print
a plan that disagrees with the matrix the run then deploys.
Measured against the live helper: 0 gives 80 rows over 3 chunks, 25 gives 25
over 4, and returning to 0 restores 80 and 3.
The gate found three faults in the first attempt and all three are in this
commit: the key sorted after INFINITO_CI_CONCURRENCY rather than before it, the
dispatch description ran to 81 characters against a limit of 80, and the
retrigger fixture did not carry the new input. That last one matters beyond the
test: it is the check that would otherwise let a retrigger drop the chunk size
silently. carried_inputs() itself needed nothing, since it derives from the
declared inputs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A codename carries no ordering, so 'bookworm' and 'trixie' are two opaque
strings and no updater can tell that one supersedes the other. A pin that
names its Debian release that way is frozen on that release for as long as
it stands, and the freeze is invisible because the tag keeps resolving.
Numeric tags are comparable and stay inside the freshness pipeline, so the
rule is worth holding for the whole repository rather than fixing case by
case.
The check reads two shapes, because a codename reaches the tree in two
spellings. A 'name:tag' reference is recognised by the absence of a space
after the colon, which is what separates 'debian:bookworm' from the YAML
'image: python' and covers Dockerfile FROM lines and heredocs without a
second parser. A 'version:' value is read as a key, because '3.11-bookworm'
carries no colon at all.
False positives are kept out by two independent gates: a word-boundary
prefilter, so 'residual' and 'inside' are not read as 'sid', and a split of
the tag on '-', '.' and '_' that requires a whole component to be a
codename. That is what lets 'cache buster', '.stretched-link',
'fa-bullseye' and a tags list containing 'stretch' through. Every codename
occurrence in the repository was checked against the matcher by hand, and
the check was negative-controlled in both shapes, including an untracked
file.
Four references had to move for the rule to hold. meta/distros.yml takes
'debian:13' rather than '13.6': at depth 1 the updater can raise it across
a major, while depth 2 pins it inside the 13.x line forever. The whole
'make install' path was run in a debian:13 container to back that: all
twenty-four Depends and Recommends resolve on trixie, the docker repository
bootstrap derives the right suite and installs docker-ce-cli, and the
package builds and installs.
The workspace DiD probe moves to 'debian:13-slim'. An earlier version of
this change carried a suppression claiming the probe had to follow the
suite the apt proxy mirrors. That was wrong twice over: it cited a function
that has nothing to do with the probe, and the coupling does not exist,
because the cache frontend rewrites by path prefix and the suite never
enters a routing decision. Replaying the probe verbatim against
debian:13-slim fetches all three trixie suites through the
bookworm-configured proxies, so the suppression is gone rather than
reworded.
The rule ships with an escape hatch and no uses of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sweep may deploy 210 rows (available) in chunks of 80. Three declared
blocks hold exactly that when there is no priority line, but priority
rows get a chunk of their own that is never topped up with regular rows.
Nine priority rows therefore occupied a whole block, left the regular
rows 2 x 80 = 160, and 42 rows of budget went unused.
Adding a block to the YAML alone would not have changed that:
matrix.chunks_of handed the split slots.chunk_count(), which is
min(ceil(available / chunk size), INFINITO_CI_MAX_CHUNKS) and ignores
the priority seam, so the plan still stopped at three blocks.
call-orchestrator.yml now declares test-deploy-chunk-3, gated on chunks
0 to 2 like every chunk gates on all before it, and report-main-failures
and done need it. INFINITO_CI_MAX_CHUNKS is 4, one above what the budget
fills, and slots.chunk_blocks() returns it. chunks_of hands the split
every declared block; available still caps the rows, and a chunk whose
slice comes back empty skips its deploy. chunk_count() stays the "chunks
filled" figure of the budget report.
Tests: the split spends 9 + 80 + 80 + 41 = 210 rows with a spare block;
chunks_of plans three chunks from chunk_blocks=3 while chunk_count=2
(the previous wiring yields [1, 10]); an integration guard requires the
declared chunk jobs to equal INFINITO_CI_MAX_CHUNKS and to exceed
ceil(available / chunk size) by one, and fails on the stale value 3.
A shell that inherited INFINITO_CI_MAX_CHUNKS=3 keeps failing that guard
until it is restarted; make dotenv-force rewrites .env.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rows are indexed by `(name, variant)` and a variant is an int. A token that
names only a role carries no variant, so the lookup ran with `None` and could
never hit, and every such token was reported as matching no discovered row.
That is the form `collapse_to_roles` produces, so it is the form a retriggered
priority line arrives in. Run 34348821577 shows the result: all twenty-one
priority entries carried
priority: '<role>' matches no discovered row
(no row in this run's mode and lifecycle envelope)
while baserow, checkmk, decidim and the rest went on to deploy fine. The
warning was not wrong about one entry and right about twenty; it was blind. A
role that genuinely has no row in the envelope looked exactly like a role with a
full set, which is how priority work goes missing without anyone being able to
see which entry did.
A token that narrows nothing beyond the role name is now answered by existence:
it asks for the role in whatever axes the rotation picks, so any discovered row
satisfies it. `Pin.pinned` already draws that line - "whether the token narrows
anything at all beyond the role name" - and pinned tokens keep the old path,
`check_pins` included.
Three tests come with it. There was already one for a bare role that matches
nothing, and none for a bare role that matches something, which is why the
lookup could stay broken. One of the new ones uses variant 4 so a future
implementation cannot pass by quietly guessing variant 0. Checked against HEAD:
the same input returns the spurious warning there, so the tests fail without
this change.
This restores the signal, not the lost jobs. The other silent path, `priority`
being truncated to the job budget in cli/meta/ci/chunks.py, is untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The instructions test held a job of its own that picked one random role per
run, so a README's Production block was proven roughly once per 265 roles.
The replay now rides on a deploy row instead: no extra runner, and every role
the sweep touches gets its documented one-command deploy exercised.
The row is the role's smallest variant -- fewest services enabled -- because
that is the shape the README describes, and it is chosen among the variants
the sweep actually deploys, so a row cut as redundant coverage cannot silently
drop the role from the test. A role qualifies only when its README carries a
Production block and the guide's own mode is one the role runs and tests.
The marked entry carries the guide mode in `instructions` and a 📖 in its job
title, between the variant number and a priority ⭐. The guide mode is
independent of the mode the rotation gave the row: a swarm row still replays
in compose. The step runs last in the deploy job, after the swarm teardown,
because the replay deploys the role a second time and needs the runner's
ports.
call-test-instructions.yml, cli.meta.ci.guide_select and the `instructions`
dispatch input go with it. `redundant()` moves next to the marker so the
chunker, the plan table and the marker share one definition.
The plan table trades its 📊 Weight column for 📖 Instructions; nothing read
the weight any more, so it leaves the matrix entries too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A run-wide draw proved one distribution and said nothing about the other four, while reporting green. Both are matrix-row axes now, assigned by the same deterministic rotation mode and tor already use, so one sweep exercises every distribution and every filesystem.
The two read the row's position like an odometer, the distro as the low digit. Turning both on the position directly covers only n of the n x m pairs whenever the pools are the same length, and no sweep unlocks it.
Both are part of a row's identity: glyphs in the job label, columns in the plan table, shards in the artifact name. Both are selectable through %distro and /filesystem, and a retrigger replays the distro off a failed title. It leaves the filesystem out, because a title states the kind the matrix assigned and a deploy may have fallen back from it.
A row's filesystem is a preference unless a human names it, on the run or in a token. Reading the matrix's own narrowing as a demand would fail rows for every transient condition the applying step is built to tolerate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The id column printed the discovery id, so every mode/tor combination of a
role repeated the same number. Count the rows instead and remap 'covered by'
onto the row number of the covering row's first occurrence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The id leads, the variant follows the name it belongs to, and covered_by,
weight and the new clone column close the row. ✂️ said only that a row was cut
while two different reasons hide behind it: 🔰 points at the earlier row that
already embeds this one, 🐑 says the row shares its service set with an earlier
role that stands in for it. 27 rows of a full plan carry 🐑 and were until now
indistinguishable from the rest.
_COLUMNS is the order and the headers again. tor and triggered used to sit
outside it as special cases, so a reordering had to be made in two places and
the tests indexed by hand; they now ask _COLUMNS for a column's position.
The chunk column loses its ⏭️ marker, which named the block the calling job was
running. Without it cells() has no use for `current` and --chunk no effect, so
both go, along with the workflow's argument.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
i8cifail died twice on the same run, each failure hiding the next.
The retrigger recovers a source run's dispatch inputs from a job log, because the title truncates at 512 bytes and REST answers inputs: null for a finished workflow_dispatch. That reader picked the first depth-1 job outright. A skipped job never uploads a log blob, so its log endpoint answers BlobNotFound as a bare HTTP 404 -- and a force-cancelled run can leave every depth-1 job skipped, which is why deeper jobs are now read too rather than assumed inputs-free. Candidates are taken shallowest first, so the orchestrator's own inputs still win over what a caller forwarded. gh 2.97 also refuses to emit a log holding terminal escape sequences and exits 1; the log downloader read that as no log and silently wrote none. A missed fetch is now tolerated instead of aborting, so the guard in dispatched_priority reports an expired log in words rather than the command dying on gh: HTTP 404.
Behind that sat the resolver: it decoded whole priority entries, but an entry may pin axes (web-app-zammad#1@compose+tor), which the display-name codec does not parse. Every pinned entry aborted as corrupt. split_axes -- already the seam decode_list uses -- separates name from suffix, so only the name is resolved and the truncation guard stays sharp. The pin rides along into the retrigger: the entry never ran, so the run holds no evidence its axes were wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A clone and a row an earlier one already embeds prove nothing the sweep does
not already prove, but both still took a slot. Drop them when the regular line
is chunked, so the budget buys unique coverage only, and mark them in the plan
table with a status of their own instead of letting them read as 'beyond the
budget'.
A priority row is exempt: an operator who pins a clone asked for that deploy,
and dropping it would be the silent withdrawal 6605c7ef9 just removed.
The cut sits in chunks_of, not in entries_of, so the plan still lists every
candidate while no chunk can see one. 55 of 254 rows are cut on this branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
INFINITO_DISCOVERY_SORT is the single source of truth for the discovery
order, but 'asc clone' lived in sort_spec() and was concatenated in front of
it, so the declared value never showed the order a run actually takes.
Declare the key where the rest of the order is declared and let sort_spec
return the value verbatim. test_clones_sort_last_ahead_of_the_declared_spec
now guards the declaration instead of the concatenation, and covered rows
sort directly behind the clone key so they land at the tail of the ranking.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The regular line was queried with selection.names(leading) as its blacklist,
and names() yields role ids only -- so pinning one variant in the priority
line withdrew every variant of that role from the sweep. Run 31927586060
deployed nextcloud only as #10 and #5; the other twelve variants existed
nowhere.
Blacklist only the tokens that name no variant, and drop the pinned
(role, variant) pairs from the regular rows afterwards. The variant-level cut
cannot go into the query filter: its tokenizer has no '#', so an axis token
aborts the parse.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
env_setting returns a non-empty process value, else the default.env value,
else KeyError -- an empty process override falls back through the strip check
in parser.py. INFINITO_DISCOVERY_SORT is declared non-empty, so the spec the
ternary guarded against could never be falsy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The regular rows began wherever (sweep * capacity) % count pointed. On run 210
that was offset 68: chunk 0 opened at candidate 69, the coverage-first head of
the list went undeployed, and no two runs could be compared because the window
moved with the run number. A run that looked wrong turned out to be merely
rotated, and finding that out meant recomputing the arithmetic by hand.
The rotation is gone. Chunk 0 reads the ranking from the top unless an offset
says otherwise, and the offset is an input now, threaded from the manual form
through the orchestrator and the deploy call down to the matrix builder. The
plan renderer takes the same flag, so the table shows the window the run will
take.
The offset is a row count or a selection token. A token is the honest form:
the ranking shifts whenever a variant appears, so 'offset 84' means something
different tomorrow while 'web-app-nextcloud#5' still means the same row. Axes
the token leaves open match anything, so a bare role name starts at its first
row and a pasted job label starts at exactly the job it names. A token that
matches no regular row aborts rather than silently starting at the head an
operator meant to skip.
The run title carries it as ⏩ followed by the value, and only when one was
asked for, so a rotated-looking run says so in its own name and the retrigger
reads it back from there when a job log has expired.
resume_from_chunk leaves the manual form with it. Skipping whole chunks was
the coarse way to say 'carry on further down'; the offset says it exactly and
keeps the rest of the chain intact. The orchestrator input stays for the other
entry points.
What this gives up is automatic tail coverage. Rows beyond the budget are
reached by moving the offset, widening the budget or narrowing the selection,
not by waiting for consecutive sweeps to walk the list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge assertion was a job nobody depended on. ci-orchestrator needed the
scope, the branch prefix and the affected roles -- not it -- so a PR whose base
was not merged in went on to spend five distros on a deploy sweep and reported
a single red job beside a chain that had already finished.
Validation is now a reusable workflow every entry calls, and what it checks
follows the event and the inputs rather than the caller: a pull request gets
its branch prefix and merge base asserted, a run carrying a selection gets that
selection checked, a fork PR waits here for the privileged run's images. The
wait and its gate move in with it, so the orchestrator no longer keeps a second
copy of that logic. Every downstream job depends on the gate.
The selection check is what the last two nights needed. cli.meta.ci.validate
resolves every token against the branch that would deploy it -- role, variant,
mode, onion state -- and reports all offenders at once. Until now the matrix
builder raised on the first bad token from inside a chunk job, so a stale
priority line cost one full CI round per entry and only ever showed the next
problem after the previous one was fixed. Measured on a real list: three
unusable tokens, one pass.
A bare role name that matches nothing is a warning, not an error: the
diff-derived whitelist legitimately names roles the lifecycle envelope filters
out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The plan told you a row is being deployed, never that another row in the same
sweep already proves whatever it would prove. The complexity matrix has always
known -- covered_by holds the id of the earlier row that embeds this one -- but
the number stopped at the matrix, and 46 of 256 rows carry one.
The matrix entry now carries the row's discovery id and that covering id
through to the plan, which renders them as two columns and drops them again
before the matrix reaches the deploy jobs.
The id column had to change with it. It was a running count over the rendered
lines, so a covered_by of 1 pointed at a row that numbering does not have. It
is now the discovery id, the same number the complexity matrix prints, which is
what makes the reference resolvable inside the table it appears in.
Two tests indexed their columns by position and broke on the new one; they ask
_COLUMNS for the index now, so the next column does not break them again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two shell aliases rebuilt the discovery order by hand, and both got it wrong
in the same two ways: they passed INFINITO_DISCOVERY_SORT straight through,
dropping the clones-last prefix that keeps one representative per dna cluster
ahead of the budget cut, and they filtered on `compose` -- what a role can do
-- where discovery filters on `test_compose`, what CI actually tests. Measured
against the real query that is 13 rows of difference and a divergence from row
two onward.
The query module already owned both pieces. It now also renders the whole-role
view (--matrix --roles), which is the only thing the hand-rolled version could
do that this one could not, so nothing outside this module has a reason to
assemble a --sort or a --filter again.
A unit test pins the contract: the role view may drop the --variant flag and
nothing else, and the filter reads the tested column.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A whitelist or priority entry was a role name and nothing else, so a run
could not ask for "this variant, on swarm, behind the onion". It either
covered everything the role offers or waited for the rotation to come
around to the combination that broke.
An entry is now a token that may carry the axes, either as
web-app-x#0,2@swarm+tor or as the job label pasted straight back out of
the run that failed. What a token leaves open still belongs to the line
it stands in: priority covers every remaining combination in one sweep,
whitelist keeps the rotation and its position in the matrix. A pin the
row cannot take aborts the matrix rather than dropping the row, because
a dropped row reports a green run for a combination that never ran.
The onion state spells out as +clearnet, not as a -tor suffix: svc-net-tor
would otherwise read as role svc-net on the clearnet.
Chunk membership still follows the discovery ranking, which is what
decides who makes the budget cut, but the rows inside a chunk are sorted
by display name, variant, mode and onion state, so a chunk's job list
reads like the plan table instead of like the sweep's rotation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
report_failures matched job names against a raw role id
((?:web-app|web-svc|svc|sys)-[a-z0-9-]+), but job titles have carried
display names for a while, so the regex matched nothing and every failed
deploy on main went unreported. runs.py had the same shape and stopped
at the first mode glyph, which swallowed a caller path like
'z / Host / sys-front-proxy' into the role name.
Both now go through axes.parse_label, kept next to the function that
builds the label so the two cannot drift, and resolve the middle through
the display-name codec. The parser anchors on the LAST mode glyph so a
reusable-workflow caller path carrying one of its own cannot capture the
role.
The onion state is part of what identifies a job, because a priority row
runs the same mode and variant twice; artifact_name takes it so the
reporter looks for the artifact the run actually uploaded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One discovery query per run replaces one per mode, so the row basis, the
ranking and the budget cut all speak the same role#variant unit. The
result is deployed in serial chunk blocks rather than three parallel
mode matrices.
Chunk size is derived, not pinned: GitHub cancels a job queued for
INFINITO_CI_QUEUE_HOURS, so with INFINITO_CI_CONCURRENCY runners free a
chunk holds concurrency * floor(queue_hours / the deploy job's
timeout-minutes) rows. Assuming every job burns its full timeout is
deliberately pessimistic -- no job can outlast it -- so the drain
estimate can only overshoot.
The run job cap bounds the sweep as a whole; rows beyond it roll into
the next sweep, whose regular line starts at an offset so consecutive
sweeps walk the whole list instead of re-testing the same head. Priority
rows lead every sweep, never move with that offset, and force a chunk
boundary at the seam so a chunk is either all priority or all regular.
sequencing.py goes with the mode lines it ordered: chunking restarts the
queue clock at a controlled size, which is what serialising the modes
was approximating with three buckets of arbitrary size.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scripts/tests/environment/ becomes scripts/tests/workspace/ with one
subdirectory per track. base/ prepares the workspace and is a prerequisite;
compose/ and swarm/ are independent of each other, so CI can run them as
separate jobs with a budget each. 00_orchestrator.sh runs the tracks named
by INFINITO_WORKSPACE_TRACKS and aborts on an unknown one.
The job count stays at five: cli.meta.ci.dev_runtime matrix assigns exactly
one track per distro and flips the split with the run number, so two
consecutive runs cover both tracks on every distro without doubling the
matrix. The workspace_track input (rotate|compose|swarm) pins it from manual
steer through the orchestrator for a targeted re-run.
Job names carry the track glyph from utils/symbol_glossary and the image
shortened to its last two segments, so quay.io/centos/centos:latest reads
as centos:latest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guide selector asked only meta/tests.yml whether a role opts out of its
deploy mode, so a role that switches the mode off in meta/services.yml stayed
in the random candidate pool and could be deployed by an orchestrator run that
passes no role, priority or whitelist.
Consulting get_role_mode_enabled covers both modes the selector can pick, host
included, which get_role_skip does not reach.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workflow directory mixed entry points, reusable stages and scheduled jobs into one flat list sorted by topic, so the first question about any file - what starts it - was the one the name did not answer. Every workflow now carries one of three prefixes, derived from its on: block: cron- for a schedule trigger, call- for one only reachable through workflow_call, entry- for everything a repository event or a human starts. Scheduled workflows additionally carry an alarm clock as the first character of their name.
Two lints keep it that way. One derives the expected prefix from the triggers and compares it against the file name; it flagged all 17 offenders before the rename and none after. The other resolves every .github/workflows/<file> string in the repository against the directory, which is what a rename silently breaks in Markdown links, Python constants and shell API URLs alike.
Renaming pulled 69 pre-existing comments into the comment lint's changed-file scope. The vendored CodeQL template got the nocheck marker its scorecard sibling already carried, two workflow headers moved above name: where they count as file headers, and the remaining banners were deleted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every conflicted hunk resolves onto the swarm-nfs side, because on all six
files it is the strict superset: it carries the display-name codec
(utils/roles/display.py) the CI role selections now encode through, and the
sequencing glyph move from 910748ec4. svc-net-tor holds no unique content on
any of them -- runs.py even had non-conflicted hunks already calling `codec`,
so keeping its side would have left a NameError.
The lint fallout is settled in the same commit, as 4afc9b770 did for the
previous merge: the shell-or-true lint lives on svc-net-tor (c505ce746) and
swarm-nfs brings two unmarked `|| true` sites with it. Both get a nocheck
reason rather than a rewrite, because neither swallows anything -- the
slaptest call is the measured exception of c4bbbba93 and has_cn_config decides
right below it, and the dockerd restart is followed by an is-active poll that
fatals. In both the exit code was never the authority.
make quality green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI deploy jobs were titled with the raw role id, so a run page read as a
column of dash-joined prefixes. They now carry a display name: the category
path as hanzi labels, then the role's README heading or the part of the id
the categories do not already say, whichever is shorter.
utils/roles/display.py owns both directions. Encoding happens once, at
discovery time, as a `label` key on every matrix entry; the workflows render
`<glyph> ${{ matrix.label }}` and keep the glyph as a literal because the
retrigger reads the deploy mode off it.
Decoding is what makes the retrigger keep working. It accepts a display name
with or without its variant shard and a bare role id unchanged, so job names
from runs that predate this still parse and a hand-typed dispatch never has
to spell hanzi. The parser therefore stopped matching an id-shaped tail and
asks the codec instead; a job the codec does not recognise is skipped rather
than counted as a role.
The role selections go out as display names too, so the run title reads like
the jobs it spawns. Four consumers decode them back, each the single place
the value is actually used: the whitelist resolver before it checks the role
directory, the discovery query on its env, and the sequencing and plan CLIs
on their arguments.
Every fragment is joined by `·`, including before the title, and the heading
loses its whitespace and any emoji variation selector. A display name holding
a space would make the space-separated whitelist and priority inputs
unparseable with more than one role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub cancels a job queued for 24h. A line starting swarm, compose and
host at once queues its whole matrix in one go, so with more roles than
runner slots the tail waits past that cut and dies unrun -- 54 of 241
jobs in run 30937913632.
Deploy the modes one after the other instead, heaviest first, which
restarts the queue clock per mode. cli.meta.ci.sequencing counts a
line's matrix jobs on the row basis the discover steps use and
serialises above INFINITO_CI_SEQUENTIAL_THRESHOLD; below it the modes
still start together, so a small diff pays no wall-clock for this.
A needs: edge is static, so both layouts exist as jobs and the decision
skips one. Their conditions are structurally asserted, not evaluated:
ten near-identical if expressions invite a copied sentinel or a missing
chain edge, and both failures are silent at runtime.
mode_fail_fast stops a serial line at its first failed mode, on by
default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub cancels a job queued for 24h. A line starting swarm, compose and
host at once queues its whole matrix in one go, so with more roles than
runner slots the tail waits past that cut and dies unrun -- 54 of 241
jobs in run 30937913632.
Deploy the modes one after the other instead, heaviest first, which
restarts the queue clock per mode. cli.meta.ci.sequencing counts a
line's matrix jobs on the row basis the discover steps use and
serialises above INFINITO_CI_SEQUENTIAL_THRESHOLD; below it the modes
still start together, so a small diff pays no wall-clock for this.
A needs: edge is static, so both layouts exist as jobs and the decision
skips one. Their conditions are structurally asserted, not evaluated:
ten near-identical if expressions invite a copied sentinel or a missing
chain edge, and both failures are silent at runtime.
mode_fail_fast stops a serial line at its first failed mode, on by
default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
variant_bundle_limits.py next to variant_bundles.py formed the prefix group
'variant', which the structure lint reads as a folder in disguise. It stayed
green locally because that lint asks git ls-files: the new file was untracked
through every quality run and only became visible once committed. CI saw it
immediately - the sole failure of run 30675284663.
Both move to utils/github/variant/ as bundles.py and limits.py, and the test
mirrors along.
The BLE001 per-file ignore is removed rather than repointed. parse_size raises
humanfriendly.InvalidSize for a malformed string and TypeError only for a
non-string, and the value reaching it is always str - so the catch-all was
never needed. Measured, not assumed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The compose and host deploy jobs tee the whole deploy stdout into the file
ANSIBLE_LOG_PATH names, and both downstream steps parse that stream. Only
stdout carries the host-side "matrix-deploy: round N/M ... PASS n" markers,
so it is the sole source the per-variant runtime tables can come from.
dedicated.sh's cleanup then copied the container-side ansible log onto that
same path. docker cp unlinks and recreates the destination, so tee kept
writing into an orphaned inode and the visible file became ansible's own
log_path output — whose "<ts> p= u= n= LEVEL| " line prefix defeats the
anchored recap matchers in logparse.py while the unanchored has_sections
substring test still fires. in_roles_recap never flips, every role-time line
is skipped, and the step dies with "no profile_roles entries found" although
the deploy succeeded. Run 30578244959: 14 of 14 compose+host jobs.
Why it looked like a flake: docker cp cannot read a /tmp that is a tmpfs
mounted inside the container, and the Debian image enables systemd's
tmp.mount while CentOS Stream 10 does not. The copy silently failed on
debian (host file survived, jobs green) and landed on centos (host file
destroyed). feature/swarm-nfs never had the block; it returned here through
merge drift from main, where the copy is still load-bearing because those
workflows do not tee.
The copy is dropped: stdout is a strict superset of what it provided, and
the in-container verifier reads the log inside the container. Alongside it:
the swarm workflow's file-exists guard is adopted so an aborted deploy does
not paint the summary step red on top; the log_path prefix moves to
utils/ansible_log.py and is applied in logparse.py, so an ansible run log
parses instead of yielding an empty summary; a lint pins that nothing writes
onto ANSIBLE_LOG_PATH again; and the parser regression test now covers the
recap headers, not just the role rows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workspace workflow asked the distro SPOT two questions through embedded python one-liners: the matrix image list, and the distro behind a matrix entry. Give both a name -- distro_of_dev_runtime_image() as the counterpart to dev_runtime_image(), and a cli.meta.ci.dev_runtime module shaped like its image_ref neighbour. The empty-SPOT guard that protected the matrix from silently skipping every job moves along with it. No caller yet; the workflow switches over in the next commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workspace workflow asked the distro SPOT two questions through embedded python one-liners: the matrix image list, and the distro behind a matrix entry. Give both a name -- distro_of_dev_runtime_image() as the counterpart to dev_runtime_image(), and a cli.meta.ci.dev_runtime module shaped like its image_ref neighbour. The empty-SPOT guard that protected the matrix from silently skipping every job moves along with it. No caller yet; the workflow switches over in the next commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test-environment.yml and test-guide.yml become test-workspace.yml and
test-instructions.yml. Orchestrator job ids, the distros-SPOT matrix
lint, the act example, the diagnostics slug and the rescue artefact
names follow.
Replace the roles_only input with per-job workspace/instructions
tri-state inputs (auto|true|false). 'auto' derives the former
roles_only skip straight from whitelist: both jobs run on a global
scope (whitelist empty or __ALL__) and are skipped on a role-scoped
run, which keeps the PR behaviour identical because the diff resolver
never emits an empty whitelist. entry-manual exposes both as choice
dropdowns defaulting to false, so a manual run opts in explicitly.
The now redundant roles_only output is dropped from
pr_affected_roles.sh and subset_roles.py.
Branch-protection required-status-check names change along with the
job renames.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
guide_select picked the role, the mode and the distro in one draw, so a run proved the README instructions worked on exactly one randomly chosen distribution. The distro list was also a literal in the workflow, which meant the manual-CI distros input never reached the guide at all.
The distro draw is removed. guide_select still picks the role and derives the mode, while the shared loop from 1bbe7a0eb drives the distro dimension, and ci-orchestrator forwards its distros input.
The four mode-specific steps collapse into guide/one.sh, which the loop runs once per distro: it renders the pkgmgr base image for the current distro, then either execs host_deploy.sh or brings the compose server up and replays the README block against it.
one.sh releases the compose server in an EXIT trap. Without it the second distro would start against the stack the first one left behind. Rescue diagnostics gain a distro segment for the same reason the swarm collectors did.
The job timeout goes from 90 to 345 minutes because up to five rounds now have to fit; the loop's own budget still decides how many actually run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
categories.yml moves from roles/ to the repository-root meta/, joining packages.yml. utils/roles/categories.py owns its location plus the tree-walking helpers that were stranded in entity/name.py, so no caller spells the path or duplicates the lookup.
meta/distros.yml becomes the SPOT for every distribution: os_family, Galaxy platform name, dev-environment base image, and the pkgmgr/pkgmgr-virgin/environment reference templates. It replaces seven scattered copies (DISTRO_FAMILY, VALID_DISTROS, _DISTROS, _CANONICAL_PLATFORMS, two shell arrays, a Makefile literal). INFINITO_DISTROS is no longer a static default but generated by utils/env/handlers/infinito/distros.py; the shell consumers read it through the env layer.
Drops dead CLI surface found along the way: --categories-file and --roles-dir on the provision CLI plus three unused parameters of generate_dynamic_inventory, and the positional roles_file of the invokable categories command.
tests/lint/repository/test_distros_spot.py pins the copies that cannot call the SPOT (workflow literals, docs table) and forbids composing a pkgmgr reference outside it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
guide_select built its random pool from invokable + lifecycle only, so
the Guide job could pick roles whose meta/tests.yml declares every
deploy mode untestable (svc-storage-nfs-client, which hard-asserts a
reachable NFS server). Filter the pool by the role's declared skip for
its guide mode; a malformed tests.yml now fails selection loudly
instead of being silently ignored.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
report_failures.decisive_excerpt and new_in_branch._declares_application_id
now read via utils.cache.files.read_text instead of Path.read_text, and the
new_in_branch test constructs its vars path from ROLE_FILE_VARS_MAIN. Clears
the uncached-read and role-file-spot lint gates; behaviour is unchanged
(same UTF-8 decode, same exceptions, same resolved path).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a main run, report-main-failures parses the failed deploy jobs, downloads
each role's rescue-diagnostics artifact, and opens a 'CI failure: <role>' issue
(label ci-failure) carrying the run link, the artifact names, and the decisive
rescue excerpt inline. An existing open issue for the role gets a comment
instead. Mode emojis are resolved through the symbol glossary, not hardcoded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
