89f53a942 landed the canonical mount surface without running autoformat,
so ruff reflowed six files the next time the quality gate ran. Pure
formatter output: expressions that fit on one line joined back up, one
import pair sorted, no executable change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3183814b9 put the marker at the end of the mock-IP line, which pushed
that line to 118 chars. `make quality` autoformats before it lints, the
formatter split the call across lines and left the comment sitting on
the closing paren -- and is_suppressed_at only honours a marker on the
construct's own line or the one above, never below. The suppression
silently detached and test-lint failed on a tree that looked annotated.
Bind the literal to a name so marker and IP share one short line that no
reflow can separate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 32118850138 left a dnsmasq that was alive with zero sockets,
nothing logged, and a 6m39s death window between two host DNS
consumers. Capture each dnsmasq's fd table and rlimits in the rescue
snapshot (closed listeners vs exhausted fd table vs lost netlink
watch), and sample the :53 listener set every 10s inside the DiD
container so the next occurrence is bounded to one bucket.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bridge resolver comes from ansible_facts docker0, a fact that does not exist on a first deploy: the same role installs docker, and the daemon only starts on this very template's notify. Run 1 therefore renders without the bridge and run 2 with it. Where bridge and clearnet are the same address that is one entry against two - different bytes, so the template reports changed, the handler fires, and dockerd restarts underneath a live stack. In CI run 32119014832 that restart killed postgres on the manager at 12:42:26 and the deploy unravelled from there.
Deduplicating makes both renders collide on identical bytes. Order is preserved: the bridge must stay first or dnsmasq never sees the onion names.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
engine_by_key merges the generation manifest's volume-to-engine entries and
its docstring promises they win, but replay() called it without the
generation it was already holding, so only repository-derived names resolved.
A dump whose names the application model does not claim aborted the drill:
matrix variant 2 seeds five mautrix bridge databases into the central
postgres, their dumps sit in postgres_data under names that are no entity,
and both Matrix#2 CI jobs died on
no database service in the repository claims volume 'postgres_data'
or database 'mautrix_meta_bridge', so its engine is unknown
tearing the stack down mid-cycle and cascading into four more CLI tests.
The drill's marker seeder has passed the generation since dd07275e8, so one
run was resolving engines from two different maps; replay now reads the same
one. A generation without a manifest contributes nothing, exactly as before,
and a corrupt manifest already fails earlier in dumps_of.
The new test pins the shape: a dump only the manifest can place is replayed.
Before the change it raised with the CI message verbatim.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test feeds daemon_json_dns() a literal {"dns": "1.1.1.1"} to prove a
string value is treated as empty; the IP is test data, not configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mount pipeline was split across two layers: compose_volumes and
container_volumes lived as filter modules that ansible's filter loader
rejected on every start ("has no attribute 'FilterModule'", two warnings
per run), while the same-named lookups wrapped them thinly. The filter
bodies now live inside their lookups (the tor_socks precedent), the
filter files are gone, and the render tests moved next to the lookup
tests.
As lookups they see the templating context, so the application_id term
every call site passed back to its own role is gone too: both lookups
read application_id from context and fail loudly when it is missing.
107 templates and nfs_prep.yml drop the term; the calls collapse to
lookup('compose_volumes') and lookup('container_volumes', service_name).
The container_volumes call is now canonical - four leading spaces,
service_name in the service position, | indent(4) - enforced by a new
lint (test_container_volumes_call_shape). Normalizing the 85 literal
call sites surfaced that erpnext keyed its meta/volumes.yml mounts by
container name instead of compose service key, the only role to do so;
its meta now uses the service keys. wordpress's column-0 call, which
rendered its first line at top level, is indented like every other.
Template logic that only restated meta/volumes.yml is deleted: mailu fed
compose_volumes an extra_volumes dict whose eight names were looked up
from the very meta file compose_volumes reads itself; zammad's init and
railsserver extra mounts were the volumes.yml entries minus the two
services, which the meta now lists (the when-gated OIDC binds included).
Conditionality that meta cannot express stays in code, but in role-local
lookup plugins instead of template loops: matrix_bridge_mounts renders
the per-bridge registration mounts, nextcloud_extra_volumes the
feature-gated whiteboard/recording volumes (type: volume entries are
declared unconditionally by compose_volumes, so gating must happen
before it).
Parity is proven, not assumed: every mechanically rewritten template is
byte-identical to HEAD after the transformation regexes (95/95), and the
semantic roles render identically through the real functions against
HEAD's meta - zammad across six sso/tls polarities, all seven erpnext
services, mailu's top level. make test-lint passes complete.
Also: suitecrm's compose template pins container_extra_hosts with
host_alias (the SOCKS proxy URL host.docker.internal never resolved in
compose+tor), and the five disabled mautrix bridge addons use the
empty-default config lookup so templating the applications dict stops
emitting 125 errors per batch in every CI job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A Tor node rewrites resolv.conf to the local dnsmasq alone, so the
rescue bundle of run 32118850138 could not prove whether the pinned
upstream (172.30.0.53) was alive while that dnsmasq sat silent on a
dead :53. Probe the daemon.json dns servers too, deduped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lookup read services.<type>.name only for the central provider and derived
the dedicated one from the entity, which holds as long as sys-svc-rdbms renders
the engine. bigbluebutton brings its own and declares bigbluebutton-postgres-1,
so the lookup answered bigbluebutton-database - a container that does not exist.
Nothing reads that field for bigbluebutton today, which is why the mismatch
survived: it was wrong and unused at the same time. The central branch is
untouched, where `container` is the compose service key rather than a container
name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Docker's built-in pools yield about thirty networks. The stack holds
seventy-two, so every further network creation failed with 'all predefined
address pools have been fully subnetted' — which is why web-app-nextcloud
never came up: it brings its own networks and was the first role to hit the
ceiling.
Declare the pools beside the nested daemon's in group_vars, so both docker
daemons read their allocation from one place, and render them from there.
The base moves to 10.208.0.0/12 because 172.16/12 is exhausted: a probe
subnet in 172.28 was refused as overlapping while one in 10.210 was
created.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sys-svc-mcp-reconcile sets application_id from a loop variable, so the
lookup received the literal {{ mcp_client.id }}. An unrendered id matches
no allowed_consumers, which rejected every provider as
consumer_not_allowed and emptied each client's registry without raising.
Pull the resolution into resolve_consumer_id() so the branch is reachable
from a test, and cover the three cases: a deferred id is rendered, a plain
id is kept, a missing id aborts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The id leads, the variant follows the name it belongs to, and covered_by,
weight and the new clone column close the row. ✂️ said only that a row was cut
while two different reasons hide behind it: 🔰 points at the earlier row that
already embeds this one, 🐑 says the row shares its service set with an earlier
role that stands in for it. 27 rows of a full plan carry 🐑 and were until now
indistinguishable from the rest.
_COLUMNS is the order and the headers again. tor and triggered used to sit
outside it as special cases, so a reordering had to be made in two places and
the tests indexed by hand; they now ask _COLUMNS for a column's position.
The chunk column loses its ⏭️ marker, which named the block the calling job was
running. Without it cells() has no use for `current` and --chunk no effect, so
both go, along with the workflow's argument.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
a2789f7bb was proven by a deploy run and nothing else. The pair here states
both halves of the rule: the engine holding the dump's volume does not block
its own replay, and a real consumer beside it still aborts - naming that
consumer and not the engine.
The second case is what keeps the first honest. On its own the first would
also pass if the guard were widened until it stopped refusing anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d130f5559 moved the swarm converge wait onto compose_mode_force-or-
DEPLOYMENT_MODE and left the compose health wait beside it reading
DEPLOYMENT_MODE raw. A role pinned with compose_mode_force: compose on a
swarm cluster then satisfied neither condition - the compose branch saw
'swarm' and the swarm branch saw 'compose' - so it got no post-deploy
readiness wait at all. web-app-matrix in its ansible flavour is the live
case.
The database lookup read DEPLOYMENT_MODE raw for the same reason, handing a
pinned role a swarm service name and a resolve-container-id address for a
stack that docker compose had created.
The override is scoped to the dedicated database only. A dedicated engine
runs inside the consumer's own stack, so the consumer's pin governs it; a
central engine is its own stack under the cluster mode, and letting a
consumer's pin reach it would rename a service swarm really did create.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
i8cifail died twice on the same run, each failure hiding the next.
The retrigger recovers a source run's dispatch inputs from a job log, because the title truncates at 512 bytes and REST answers inputs: null for a finished workflow_dispatch. That reader picked the first depth-1 job outright. A skipped job never uploads a log blob, so its log endpoint answers BlobNotFound as a bare HTTP 404 -- and a force-cancelled run can leave every depth-1 job skipped, which is why deeper jobs are now read too rather than assumed inputs-free. Candidates are taken shallowest first, so the orchestrator's own inputs still win over what a caller forwarded. gh 2.97 also refuses to emit a log holding terminal escape sequences and exits 1; the log downloader read that as no log and silently wrote none. A missed fetch is now tolerated instead of aborting, so the guard in dispatched_priority reports an expired log in words rather than the command dying on gh: HTTP 404.
Behind that sat the resolver: it decoded whole priority entries, but an entry may pin axes (web-app-zammad#1@compose+tor), which the display-name codec does not parse. Every pinned entry aborted as corrupt. split_axes -- already the seam decode_list uses -- separates name from suffix, so only the name is resolved and the truncation guard stays sharp. The pin rides along into the retrigger: the entry never ran, so the run holds no evidence its axes were wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test climbed to the repository root itself. Moving the file one directory would have left that index pointing somewhere else, and the test would then fail with FileNotFoundError instead of saying anything about compose.py. Its own package already defines PROJECT_ROOT and every sibling test module imports it from there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Output of 'make autoformat': ruff collapses expressions that were wrapped across lines without need, shfmt and the yaml formatters drop surplus blank lines. No semantic change - the whole diff is 10 insertions against 39 deletions, and every hunk joins lines that already said the same thing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A recent refactor imported baudolo into utils.recovery for three layout strings, and cli.administration.recover.__main__ imports full unconditionally - so the whole recovery CLI, --help included, died at import on every host without baudolo. That is most of them: recover device|nfs|secrets runs on swarm nodes, the NFS server and bare rescue hosts, while only the backup node carries svc-bkp-volume-2-local. utils/recovery had been deliberately stdlib-only for exactly that reason.
baudolo 5.0.0 writes a manifest into every generation stating the layout it used, so the names are now asked of the generation being read rather than of the installed package. A tree written by any version is walked with the names that version used, and utils.recovery.layout answers only for generations older than the manifest, which cannot describe themselves. That also retires two of core's guesses: the '<entity>_database' naming convention now yields to the engine the run actually recorded.
A repo whose manifest cannot be read is skipped rather than guessed at, but no longer takes the other repos with it - recovering two of three beats recovering none because one manifest is corrupt. The exit code reports the gap.
The pin lint compared two declared version strings and was therefore green while a different distribution was installed. It now checks the installed distribution and the layout literals against baudolo's own BackupPaths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
seed/token.py sat on sys.path beside the scripts that ran it, so any import chain reaching tokenize picked up the drill's seeder instead of the stdlib's token module - dataclasses -> inspect -> tokenize -> token is enough to trigger it. marker.py names what the file does and collides with nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4.0.0 renamed --dump-only-sql and rejects the old spelling with exit 2, so the timer's assembled exec string would have failed on every run against the version the previous commit pins. The launcher fixtures and the role README carry the same rename; nothing else in the repo still says it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils/recovery restated three things baudolo owns: the generation layout (suffixes, sql/ and files/ directories), the \connect parser for cluster dumps, and the databases.csv reader. Each was a second implementation of a contract whose author is one pip install away - the connect parser had even received the same quoted-identifier fix twice, once per repo.
The layout constants are now derived from BackupPaths, the inventory comes from dump_inventory, and the CSV goes through read_rows. dump_inventory is stricter than the parser it replaces: it also reads CREATE DATABASE lines, so a database a stream creates but never reconnects to is found - a new test pins that. recover/full.py and the drill's token.py take their globs from the same constants instead of spelling them out.
The dependency this creates is declared: backup-docker-to-local==4.0.0 in the base dependencies, because the deploy CLI must import from the base set alone. The role installs the same version, and a lint holds the two pins together - they drifted apart is exactly the failure this whole commit removes, so it must not come back one level up.
The generation-layout tests now write their fixtures through BackupPaths itself; before, the fixture hand-built the paths with the same assumption the test claimed to verify.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
5bb803054 tied the dnsmasq listener to group membership instead of the deploy mode, but left these assertions behind, so the branch carries two red tests. A Tor node runs the same listener in compose and in swarm, so a swarm node now expects the bridge resolver first and the clearnet upstream behind it, and the same node without an upstream expects the bridge alone rather than nothing.
The empty list did not lose its case, only its reason: it belongs to a node that is neither on Tor nor given a clearnet resolver, which now has a test of its own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four files outside the healthcheck work, all flagged by make quality-high.
The shopware marker sat on the first line of a value the YAML formatter wraps, where the lint never looked; it now sits on the last line, beside the trailing comment the formatter keeps there. The two ci meta tests get ClassVar for a mutable class attribute and unpacking instead of list concatenation.
The keycloak realm-import test asked jinja2 for an Environment without naming autoescape. It stays False - the template renders JSON, and HTML escaping corrupts the values - and the rule is silenced the way 9a9c2f239 already silences it for the dotenv filter test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A container without a healthcheck counts as healthy the moment its process starts. That is how a service which came back empty passes a restore drill: the drill asks the host's health service, and the host says yes.
The image side is read from the registry rather than by pulling: the manifest names the config blob, the blob carries the HEALTHCHECK. The answer is three-valued on purpose. A mapping means the image ships a probe, an empty mapping means it definitely ships none, and None means the question could not be answered. Only the empty mapping is evidence; reading None as absent turns one network hiccup into a hundred false findings.
Probes address the GHCR mirror first and fall back upstream only for images the mirror does not carry, and a second test requires mirror plus cache to answer more often than upstream, so the fallback cannot quietly become the rule and spend the anonymous rate limit again.
Detection spans the four places a probe can come from: the services.yml declaration, a literal block in a template, a partial pulled in by include or lookup('template'), and a HEALTHCHECK in a Dockerfile the role builds its own image from. Without the last three the sweep reported nextcloud, every redis sidecar and openldap as gaps they are not.
The opposite direction gets its own lint: a declared healthcheck that no template renders is dead configuration. Fixing it surfaced that the shared resolver silently dropped every service_name call site, because it accepted only quoted literals and a role var resolves to a bare key. And the literal-protocol lint could never be suppressed on a wrapped value, since autoformat moves the marker to the value's last line while the offence is reported on its first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A databases.csv row with database = '*' makes the backup dump a whole instance with pg_dumpall. The recover chain had no path for that file: it sorted cluster dumps out and printed SKIP, so a host whose engine is dumped that way could not be restored at all, and the drill refused to certify it.
Cluster dumps now go back through 'baudolo-restore cluster' with the instance's superuser. Three details decide the shape. A cluster row states '*' where a single-database row names a database, so credentials_of would file every instance under one key and keep the first, hence a lookup keyed by instance. Which applications must be down is stated by the dump's own connect lines and nowhere else, and reading them needs a real identifier parser: a quoted name may hold spaces and psql options precede the target. And the consumers of a cluster sit in the compose project of its engine, not in projects named after each database, so both are checked before anything is dropped.
The version gate moves out. baudolo compares the dump's header against the running server before its own pre-clean, so the copy here was a second implementation of the same rule, free to drift from the one that actually gates the replay. The test asserts the delegation negatively: the replay must not query the engine version itself. The pin follows, and baudolo-restore joins the command assertion.
The docker probing splits into utils/recovery/docker.py, which keeps the replay module under the size limit and gives the two halves one job each. Calls go through the module rather than through imported names, or a test patching the new module would not reach them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill removed each volume and re-created it. Docker refuses to remove a volume any container still references, and a container outside compose is only stopped here, not removed, so discourse's launcher container aborted the run at that point: after the host was already down and part of the volumes were gone. Re-creating a removed volume also drops its driver options, handing an NFS- or bind-backed volume back as a plain local one.
Volumes are emptied in place instead, from inside a throwaway container, and the emptiness is proven in the same call. Nothing has to be re-created, so a launcher container starts again exactly as it was built.
The engine that receives a replay is started by the service name on its own compose label. The previous fixed name existed for no engine on the host: the central Postgres serves project postgres from volume postgres_data, bigbluebutton serves project bigbluebutton from service postgres. Neither is called database, so the drill died at 'up -d database' after the teardown and before any restore.
A cluster dump yields one marker per database it recreates, or a replay that brought back only part of an instance would still pass. The drill runs once, in the sync pass; repeating it in the async pass would only tear the host down twice for the same answer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backup generation stores databases as sql dumps, not as a file tree
(--dump-only-sql), and the recover chain only ever restored trees:
`git grep sql HEAD -- cli/administration/recover utils/recovery` was
empty. An operator running `recover full` after a real loss got their
files back, their databases untouched, and an exit code of 0.
Adds the missing leg as a fifth recoverer, planned right after the
volume steps so the dumps land once the volumes are back. Three
properties make it a restore rather than an attempt:
- It refuses while a consumer of that database runs. `--empty`
pre-cleans the schema in one psql session and replays in the next; a
booting consumer recreates it in between and the dump's own CREATE
TABLE fails. That is what turned CI red with `relation
"ar_internal_metadata" already exists`.
- The engine is read, not guessed: the service key in the consuming
role's meta/services.yml, the same value that fed the backup seed.
The two previous call sites each sniffed it from the container image
and could disagree.
- The dump's own header states which server it came out of, and that is
compared against the running engine before the pre-clean drops
anything. Forward across a major version is the upgrade path and
stays allowed; backward is refused. The version deliberately does not
travel in databases.csv: that file is written at seed time and would
state the version deployed back then, and a generation copied to
another host carries its header along while a csv row stays behind.
Mind the header formats, they are not symmetric - postgres puts
`Dumped from database version` on line 7, while mariadb's line 2 opens
with mariadb-dump's own version and only the tab-separated `Server
version` line names the server. Both are pinned as test fixtures,
captured from postgres:17-alpine and mariadb:11 themselves.
baudolo is untouched: nothing beyond the credentials it already carries
is read from databases.csv, and no upstream format changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The realm carried exactly one client, the shared OIDC one, and no SAML client
existed anywhere in the repo - `flavor: saml` was in the resolver's enum with
zero consumers. Apps that cannot speak OIDC therefore had no way to
authenticate against Keycloak at all; they could only be fronted by the
oauth2-proxy, which authenticates at the edge and leaves the app itself without
a session.
KEYCLOAK_SAML_APPS collects every deployed app whose services.sso.flavor is
saml. It filters stepwise - value.services, then .sso, then .flavor, each
guarded by `defined` - because Ansible's undefined is a Marker(StrictUndefined,
Tripwire): resolving that chain in one selectattr raises for every app without
an sso block instead of yielding undefined, which would have broken the realm
render for every deployment in the repo.
The include is unconditional and the loop emits a leading comma per client, so
the realm needs no branch on the list being empty. That matters: an Ansible
`>-` value that failed to convert would be a non-empty string and a truthiness
guard would then splice a broken clients array.
The client's shape is not guesswork - every attribute comes from a failure
reproduced against a live Keycloak 26.7.1 until a SuiteCRM SAML login completed:
defaultClientScopes is emptied because the built-in role_list scope emits one
Role attribute per realm role and the SP rejects the assertion with "Found an
Attribute element with duplicated Name"; saml.client.signature is false because
the SP does not sign its AuthnRequests; the username mapper's attribute name
comes from OIDC.ATTRIBUTES.USERNAME rather than a hardcoded claim.
The new test renders the complete realm - every include, the role's own
ldap_groups_filter, StrictUndefined - and parses it as JSON for zero, one and
several SAML apps. The file had no test before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A clone and a row an earlier one already embeds prove nothing the sweep does
not already prove, but both still took a slot. Drop them when the regular line
is chunked, so the budget buys unique coverage only, and mark them in the plan
table with a status of their own instead of letting them read as 'beyond the
budget'.
A priority row is exempt: an operator who pins a clone asked for that deploy,
and dropping it would be the silent withdrawal 6605c7ef9 just removed.
The cut sits in chunks_of, not in entries_of, so the plan still lists every
candidate while no chunk can see one. 55 of 254 rows are cut on this branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The regular line was queried with selection.names(leading) as its blacklist,
and names() yields role ids only -- so pinning one variant in the priority
line withdrew every variant of that role from the sweep. Run 31927586060
deployed nextcloud only as #10 and #5; the other twelve variants existed
nowhere.
Blacklist only the tokens that name no variant, and drop the pinned
(role, variant) pairs from the regular rows afterwards. The variant-level cut
cannot go into the query filter: its tokenizer has no '#', so an axis token
aborts the parse.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard sat at line 61, between TestChunksOf and TestOffsetIndex, because 0f450722b appended the offset tests behind an already-present guard. unittest.main() does not return, so running the file directly executed only the classes defined above it and TestOffsetIndex never ran. Under pytest the placement made no difference, which is why the suite stayed green and the drift stayed invisible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four findings the quality gate raised against the offset and resume work, all
of them mine.
INFINITO_CI_OFFSET sat next to INFINITO_CI_CHUNK, where it reads well and
sorts wrong, and no handler claimed it -- so `make dotenv` would never have
materialised the default and every run outside CI would have resolved the
offset from an unset variable. It is in ASC order now and passes through the
static handler.
runs.py crossed 500 lines. The three helpers that translate job titles back
into selection tokens move to selections.py, which is a seam rather than a
quota: reading a finished run and dispatching a new one is one subject,
turning a job title into `role#variant@mode+tor` is another, and only the
latter is what lets a retrigger replay exactly what failed.
The sshd handler still told operators to look for a handler named
'sshd restart' after the topic became sshd-restart, and three lines from the
offset commit were past the format width.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
render_replicated_templates.yml built its expanded spec list in a 20-line Jinja set_fact, and both it and _render_one_template.yml carried their own copy of the same five defaults. The two drifted: a46348b6c changed the notify default to the compose-up listen topic in _render_one_template.yml and missed the identical line in its sibling, which kept notifying a handler named with a space. Ansible resolves handler names only when they fire, so nothing complained until a role took the src_glob path -- web-app-nextcloud variant 3 then died 49 minutes into a CI run with "The requested handler compose up was not found", after the rescue bundle had filled up with unrelated network diagnostics.
The expansion moves to plugins/lookup/role_templates_expanded.py, where DEFAULTS holds owner, group, mode, condition and notify once and applies them to both the glob-expanded and the passed-through branch. render_replicated_templates.yml stays as the wrapper every consumer includes by absolute path, so no call site changes; it shrinks to the include_tasks loop and no longer leaks _role_templates_expanded into the play namespace. The twelve now-dead defaults in _render_one_template.yml are removed, so a missing key fails loudly instead of falling through to a second definition.
The plugin lives in plugins/lookup rather than in the role, because role-local plugin directories are only on the search path while that role runs and the wrapper is included from magento, n8n, matrix, nextcloud and others; sys-stk-full does not include sys-svc-compose as a role and no consumer declares it as a dependency.
Verified through real Ansible, not only unit tests: the lookup resolves by name, returns a list of dicts that loop can iterate, builds dest from an actual templates directory, and keeps an explicit empty notify empty -- the shape magento relies on to suppress the handler. Twelve unit tests pin the contract, including both falsy cases and parity with fileglob, which drops directories and sorts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A retrigger started the regular line at the head again, redeploying the exact
window the source run had already judged. The red rows come back on the
priority line anyway, so that window was runner time spent on questions with
an answer.
The offset is computed now: the last row of the leading run of rows the source
run reached, as a selection token. Green or red does not matter -- what matters
is that the row ran. The scan stops at the first gap, because a hole inside the
window means that row was filtered out, not that the run got further, and
resuming past it would skip everything behind it.
The ranking is recomputed under the configuration the retrigger carries, since
that is the list the offset resolves against. The priority line is left out of
that computation on purpose: it only blacklists rows from the regular query,
and a token in it that no longer resolves would abort the very helper whose job
is to save time.
offset therefore joins priority as an input the retrigger decides rather than
carries. The integration test forced that: it asserts every carried input
arrives unchanged, and offset no longer did. The category was wrong, not the
test.
Costs a full local discovery per retrigger. There is no cheaper answer -- jobs
inside a chunk are sorted by name, so how far a run got cannot be read off the
job list without the ranking.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The regular rows began wherever (sweep * capacity) % count pointed. On run 210
that was offset 68: chunk 0 opened at candidate 69, the coverage-first head of
the list went undeployed, and no two runs could be compared because the window
moved with the run number. A run that looked wrong turned out to be merely
rotated, and finding that out meant recomputing the arithmetic by hand.
The rotation is gone. Chunk 0 reads the ranking from the top unless an offset
says otherwise, and the offset is an input now, threaded from the manual form
through the orchestrator and the deploy call down to the matrix builder. The
plan renderer takes the same flag, so the table shows the window the run will
take.
The offset is a row count or a selection token. A token is the honest form:
the ranking shifts whenever a variant appears, so 'offset 84' means something
different tomorrow while 'web-app-nextcloud#5' still means the same row. Axes
the token leaves open match anything, so a bare role name starts at its first
row and a pasted job label starts at exactly the job it names. A token that
matches no regular row aborts rather than silently starting at the head an
operator meant to skip.
The run title carries it as ⏩ followed by the value, and only when one was
asked for, so a rotated-looking run says so in its own name and the retrigger
reads it back from there when a job log has expired.
resume_from_chunk leaves the manual form with it. Skipping whole chunks was
the coarse way to say 'carry on further down'; the offset says it exactly and
keeps the rest of the chain intact. The orchestrator input stays for the other
entry points.
What this gives up is automatic tail coverage. Rows beyond the budget are
reached by moving the offset, widening the budget or narrowing the selection,
not by waiting for consecutive sweeps to walk the list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge assertion was a job nobody depended on. ci-orchestrator needed the
scope, the branch prefix and the affected roles -- not it -- so a PR whose base
was not merged in went on to spend five distros on a deploy sweep and reported
a single red job beside a chain that had already finished.
Validation is now a reusable workflow every entry calls, and what it checks
follows the event and the inputs rather than the caller: a pull request gets
its branch prefix and merge base asserted, a run carrying a selection gets that
selection checked, a fork PR waits here for the privileged run's images. The
wait and its gate move in with it, so the orchestrator no longer keeps a second
copy of that logic. Every downstream job depends on the gate.
The selection check is what the last two nights needed. cli.meta.ci.validate
resolves every token against the branch that would deploy it -- role, variant,
mode, onion state -- and reports all offenders at once. Until now the matrix
builder raised on the first bad token from inside a chunk job, so a stale
priority line cost one full CI round per entry and only ever showed the next
problem after the previous one was fixed. Measured on a real list: three
unusable tokens, one pass.
A bare role name that matches nothing is a warning, not an error: the
diff-derived whitelist legitimately names roles the lifecycle envelope filters
out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two tokens naming one row is the case the axes exist for: a variant that
failed in compose and in swarm comes back as web-app-x#1@compose+tor and
web-app-x#1@swarm+tor, and both have to deploy. The applier could not express
it. It walked the rows, took the first pin that matched and broke out, so the
second token matched nothing, counted as a typo, and aborted the chunk's
discovery with 'matches no discovered row'.
Two CI runs died on it before the cause was clear, both reporting a token that
was perfectly valid.
Pins and rows are now paired both ways round: every pin that matches a row
yields its own entry. Two tokens that narrow a row identically collapse into
one, so a duplicate in the input cannot become two jobs racing for a single
artifact name, and a bare role name loses to any token that narrows the same
row -- asking for role#1@swarm should not also deploy a rotation-picked
variant of that role.
The old unit test paired two pins on two different variants, which is why it
stayed green; the collision on one variant now has its own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
runs.py stood at 568 lines against the 500-line cap from 4b37e123d. The git and gh helpers were the clearest seam in it: eight functions that know nothing about deploy runs and only drive subprocess and the gh CLI. They move to cli/administration/deploy/ci/gh.py, leaving 482 lines behind. The module is deliberately not named runs_gh.py, which would have formed a prefix group with runs.py and tripped the structure lint.
Re-exporting the moved names from runs.py does not survive the formatter: autoformat strips them as unused imports, and seven call sites in status/__main__.py and trigger/__main__.py would then raise AttributeError at runtime. The call sites therefore reach into gh directly, and the mock targets move with them -- patching a name in the module that no longer resolves it makes a test green without testing anything.
fetch_jobs stays reachable through runs, because find_last_deploy_run calls it internally and test_status.py covers both the URL and the branch path with a single patch. Splitting that name would have left one of the two paths unpatched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The job titles in these fixtures still separated the variant with a space. Since 4dc89cf17 gave the variant a separator, utils/roles/display.py sets VARIANT_SEPARATOR to a hash and axes.LABEL_RE expects it, so parse_label matched but folded the variant into the role name -- Label(name="web-app-xwiki 0", variant="") -- and display_names().decode() then returned None. Every variant-carrying job dropped out of failed_roles, which is the one job class the function exists for.
That commit updated test_axes.py and test_display.py and missed this file. Real job titles from a run created before it still show the space form, which is why the drift stayed invisible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A retrigger answered a red run with role names. Three things were lost on the
way, and each of them let the rerun test something other than what broke.
The mode was a scope: --failed swarm dropped whatever fell in compose, and two
aliases passed that scope permanently. Every mode is read now, and a leftover
scope argument is accepted and ignored rather than breaking the callers.
The selection was a role id, so the rerun deployed whichever variant, mode and
onion state the sweep rotation picked next -- not necessarily the combination
that failed. Each red job now contributes its own token
(web-app-x#0@swarm+tor); a role that failed in two modes comes back twice.
The carried configuration was a hand-written list of eight inputs against the
thirteen the form declares, so resume_from_chunk, sweep, workspace_track and
whitelist silently fell back to defaults. The list is read off
entry-manual-steer.yml now, and everything but the priority line travels
verbatim, values taken from the job log that records them all.
That rule is deliberate and total: it supersedes the suite-skip behaviour on
fork/feature/ai-agent-employees (b1ca39e9d), which turned workspace and
instructions off when the source run had passed them. A retrigger reproduces
its source; deciding which suites are worth re-running is not its job.
The integration test walks the workflow's own input declaration, so an input
added to the form without being carried fails here rather than in a run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The plan told you a row is being deployed, never that another row in the same
sweep already proves whatever it would prove. The complexity matrix has always
known -- covered_by holds the id of the earlier row that embeds this one -- but
the number stopped at the matrix, and 46 of 256 rows carry one.
The matrix entry now carries the row's discovery id and that covering id
through to the plan, which renders them as two columns and drops them again
before the matrix reaches the deploy jobs.
The id column had to change with it. It was a running count over the rendered
lines, so a covered_by of 1 pointed at a row that numbering does not have. It
is now the discovery id, the same number the complexity matrix prints, which is
what makes the reference resolvable inside the table it appears in.
Two tests indexed their columns by position and broke on the new one; they ask
_COLUMNS for the index now, so the next column does not break them again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two shell aliases rebuilt the discovery order by hand, and both got it wrong
in the same two ways: they passed INFINITO_DISCOVERY_SORT straight through,
dropping the clones-last prefix that keeps one representative per dna cluster
ahead of the budget cut, and they filtered on `compose` -- what a role can do
-- where discovery filters on `test_compose`, what CI actually tests. Measured
against the real query that is 13 rows of difference and a divergence from row
two onward.
The query module already owned both pieces. It now also renders the whole-role
view (--matrix --roles), which is the only thing the hand-rolled version could
do that this one could not, so nothing outside this module has a reason to
assemble a --sort or a --filter again.
A unit test pins the contract: the role view may drop the --variant flag and
nothing else, and the filter reads the tested column.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
roles/sys-svc-mail-msmtp/tasks/ca_bundle.yml stat'ed candidate bundle paths on
the Ansible target and wrote the winner into msmtprc. Moodle and WordSpress then
copied that file into a container, where the named path is a different file - or
none. Only update-ca-certificates populates a container's system bundle, and
that runs solely from the with-ca-trust.sh entrypoint, which compose_ca.py sets
only when 'docker image inspect' succeeded. On a fresh host the image is built
minutes later, so it never ran: job 94965457602 of run 31864448027 shows
Config.Entrypoint still moodle-entrypoint.sh while the CA sits mounted beside it,
and msmtp reporting 'The certificate issuer is unknown'.
An explicit tls_trust_file also outranks SSL_CERT_FILE, so the env injection the
override always performs could not compensate.
Name CA_TRUST.inject_cert_container when the CA is actually injected, and emit
no line at all otherwise - msmtp's own 'system' default is the right answer for
a letsencrypt relay, and it is what the container's stock bundle verifies. This
also removes a latent defect on letsencrypt: a RedHat host would have written
/etc/pki/... into a Debian container.
The removal of the probing apparatus is taken from 9a72d01eb on
fork/feature/ai-agent-employees, which diagnosed the wrong-machine measurement
correctly. Its own conclusion - drop the line entirely and rely on the trust
store - does not hold where the wrapper never runs, and in swarm it never does:
handlers/swarm.yml passes wrapper=False by design.
The three-term predicate behind all of this was hand-copied into four places.
lookup('ca_injected', application_id) is now its single source; the swarm and
compose gates and sys-svc-compose-ca's own var read it too.
compose_ca.py finally says when it degrades to mount-plus-env, naming the service
and the image it could not inspect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A whitelist or priority entry was a role name and nothing else, so a run
could not ask for "this variant, on swarm, behind the onion". It either
covered everything the role offers or waited for the rotation to come
around to the combination that broke.
An entry is now a token that may carry the axes, either as
web-app-x#0,2@swarm+tor or as the job label pasted straight back out of
the run that failed. What a token leaves open still belongs to the line
it stands in: priority covers every remaining combination in one sweep,
whitelist keeps the rotation and its position in the matrix. A pin the
row cannot take aborts the matrix rather than dropping the row, because
a dropped row reports a green run for a combination that never ran.
The onion state spells out as +clearnet, not as a -tor suffix: svc-net-tor
would otherwise read as role svc-net on the clearnet.
Chunk membership still follows the discovery ranking, which is what
decides who makes the budget cut, but the rows inside a chunk are sorted
by display name, variant, mode and onion state, so a chunk's job list
reads like the plan table instead of like the sweep's rotation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dnsmasq and apt-resolution investigations both stalled on facts the
snapshot never recorded.
The snapshot is taken by docker exec inside the DiD container and the
recursion only descends, so the runner's own docker - where the resolver
runs as ${INFINITO_RUNNER_PREFIX}-coredns - was structurally invisible.
dedicated.sh now runs a second pass on the CI host, with RESCUE_SEEN
seeded with the DiD id so it does not collect the first snapshot twice.
For the rest: a process table, so the MainPID that service-state reports
can be checked against something; the dnsmasq drop-in directory, because
this project's config lives at /etc/dnsmasq.d/tor-onion.conf and never in
the main file; nsswitch.conf and hosts, without which a getent verdict is
ambiguous; one probe per name plus, where the distro ships dig or
nslookup, one per nameserver, so a refusing resolver is distinguishable
from an unreachable mirror; the full firewall table set, since a dropped
query is recorded in filter and -t nat was actively hiding it; routes and
neighbours; os-release, because the matrix is distro-shaped and a bundle
could not say which leg it came from; the runtime's networks, which hold
the forwarder behind every container's 127.0.0.11; free inodes, which a
df -h reads as free space; and the runtime's disk, volume and image
inventory, the last with full references, since an engine was once picked
from a registry host embedded in an image name.
ps runs without args: the bundle is uploaded as an artifact and argv
carries whatever credentials a process was invoked with - a real proxy
password showed up in the first live run of this capture.
Two exec budgets are corrected while the per-container cost grows to
three calls. The per-container journal was borrowing the 600s meant for a
host boot journal, so one wedged container could cost 840s; container
journals span a deploy, not a boot. The cheap questions get 30s, and the
one collection site that had no outer bound gets the timeout 1500 the
others already carry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three captures read two sources into one file - devices.txt held
/proc/devices and /proc/misc, net-snmp.txt held snmp and netstat,
sockets-proc.txt held four socket tables - so a grep hit no longer named
a source. Derive the filename from the path instead: the twelve cat
captures become a list of paths, and source_name() turns each into its
own file. Nine names are unchanged; conntrack gains the two segments its
old name dropped, and it was pointing at /proc/net/conntrack while
reading /proc/net/stat/nf_conntrack.
The daemon and kill-marker journals sat under containers/ between the
per-container dumps with an underscore prefix to sort them to the top.
They are host journals; name them as such and put them beside the others.
Give every journalctl call its own budget. The boot journal is now the
largest file in the bundle by two orders of magnitude - 224 MB of 220 MB
total on a one-day host - and timing out writes the exception text into
the file instead of the log.
The nocheck marker on the probe read repairs the lint gate for the
capture test added in 58c06fc8d.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RESCUE_MAX_DEPTH did two jobs without saying so: it bounded real nesting,
and it terminated cycles. Real nesting terminates on its own - every level
is a container - so the cap only ever cost snapshots. Cycles do not
terminate: svc-runner (templates/compose.yml.j2:14) and svc-prx-openresty
(meta/volumes.yml:86) bind-mount the host's docker socket, so those
containers list the host's containers including themselves and would
descend into themselves without end.
Carry the ids already entered instead. Container ids are shared across a
socket and distinct across daemons, so this cuts exactly the cycle and
lets true nesting run to whatever depth it has. A container whose id will
not resolve is skipped rather than entered: without an identity the cycle
cannot be seen, and a missing nested snapshot costs less than a recursion
that never returns.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
getent exits 2 with neither stdout nor stderr when it cannot resolve a
name, so resolve-probe.txt landed in the bundle as zero bytes - the same
shape a capture takes when the collector never reached it. The one file
that answers 'could this host resolve anything' could not distinguish a
failure from its own absence. Record the exit status whenever a command
produced nothing.
write() swallowed OSError, which hides exactly the incident that produces
it: once the filesystem holding the bundle is full, every later file goes
missing without a reason. The job log is the only channel left at that
point, so announce it there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
