builder.py imported utils.env.handlers.infinito.ip_range directly, while
ip_range imports EnvBuilder from builder for its annotations. CodeQL
reported that as py/unsafe-cyclic-import on PR #999. The import only
runs under TYPE_CHECKING, so nothing breaks at runtime, but builder now
takes ip_range from utils.env.handlers like ORDERED_HANDLERS, the path
every other handler module already uses.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
compose/cache.shared.override.yml attaches every worktree stack's
infinito service to the primary's network (infinito-nexus-core_default,
172.30.0.0/24) without an ipv4_address. The default network declared
no ip_range, so Docker IPAM handed out the lowest free address and the
worktree stacks took 172.30.0.2, .3, .8, .9 and, whenever the primary
container was down, .10, the primary's INFINITO_IP4. The primary's
`make compose-up` then failed with "Address already in use", and with
it `make quality-high` on main.
The default network now declares ip_range: ${INFINITO_IP_RANGE}, the
upper half of INFINITO_SUBNET. Every static address of the stack (.1,
.4 to .6, .10, .53) sits in the lower half, so a container without a
static address can no longer take one.
INFINITO_IP_RANGE is derived from the final INFINITO_SUBNET after
custom.env is applied, because a worktree pins its subnet there and
`make dotenv-force` regenerates without it in the environment. Existing
worktrees therefore get their own slot's range without a new pin.
A lint checks that every static IP in default.env and every
172.30.${slot}.N pin in scripts/system/worktree/up.sh lies below the
pool.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.
The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.
call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The three-arm comparison on web-app-gitea#0 put the meshed arm at 414ms
per task over the direct arm, and the tunnel at 4% of that. What it could not
split is the rest: the arms differ both in connection plugin (docker against
ssh) and in escalation (root with no prompt against administrator with a
password sudo on every task).
A fixed set of module calls, timed either side of the transport switch with
become off and on, gives four numbers that separate the two. It costs about a
minute where a fourth deploy arm would have cost three hours, and it needs no
root login and no sudoers change.
INFINITO_SWARM_TRANSPORT_BENCH guards it, so a normal round times nothing.
The arm existed to answer one question: how much of the meshed arm's
cost is the connection plugin and how much is the tunnel. Runs 37592061960
(docker), 37753506661 (SSH on the underlay) and 37621556682 (SSH over
WireGuard) answered it on web-app-gitea#0, debian/ext4:
pass 1 5964s / 8511s / 8832s
pass 2 5319s / 7472s / 7705s
Leaving ansible_connection: docker costs +4700s, the tunnel +554s - 89.5%
against 10.5%. The answer is recorded; the scaffolding does not need to ship.
Reverts d792519725, 1b28f87b37, 89b8a8c8d1, 0470491e1a, 529c5ace43,
dd3d10a9a2, c20dd0e5f5, 458d8956e1, 63ade1e03b, d281555945 and 4adfb6e3d2.
95cfc8b7b7 is already superseded by bbfd82f89, which sets the profiling limit
upstream.
One conflict, in the MCP exclusion list of requirement 035. Both sides
edited the same line: main added svc-net-tor-smtp, this branch replaced the
three svc-net-wireguard-{core,plain,firewalled} entries with the single
svc-net-wireguard it deploys. Resolved to the union of both — 24 entries —
which is what the merged tree actually holds.
Everything else merged clean and was checked for overlap rather than
assumed: no reference anywhere still names the three replaced roles;
matrix.py kept main's PRIMARY_APPS alongside the reordered round, and the
bootstrap pass does not need it, since only one template reads it and
group_vars defaults it empty; and main's ansible.cfg addition sits in
[defaults], so the ssh_args the mesh tool derives its own connection
settings from are untouched.
chunk_size() multiplied the runner slots a chunk may assume by the waves the
queue window affords, and nothing could say otherwise. The derivation is the
right default -- too few rows leaves runners idle, too many push a chunk's tail
past the queue cancel -- but a run that wants smaller chunks had to edit
default.env to get them.
INFINITO_CI_CHUNK_SIZE now takes precedence when it is greater than zero, and
zero keeps the derivation. The input travels the same path offset already
travels: entry-manual-steer.yml offers the field and maps empty to 0,
call-orchestrator.yml hands it to all four chunk blocks, call-test-deploy.yml
exports it as the env var. It is a string rather than a number for the reason
resume_from_chunk already documents: a workflow_dispatch number input reaches
the caller as a string, and a number-typed workflow_call input then makes
GitHub refuse the job before it exists.
Two places in call-test-deploy.yml set it, not one. The plan summary resolves
the chunk size itself, so exporting it only for the app resolution would print
a plan that disagrees with the matrix the run then deploys.
Measured against the live helper: 0 gives 80 rows over 3 chunks, 25 gives 25
over 4, and returning to 0 restores 80 and 3.
The gate found three faults in the first attempt and all three are in this
commit: the key sorted after INFINITO_CI_CONCURRENCY rather than before it, the
dispatch description ran to 81 characters against a limit of 80, and the
retrigger fixture did not carry the new input. That last one matters beyond the
test: it is the check that would otherwise let a retrigger drop the chunk size
silently. carried_inputs() itself needed nothing, since it derives from the
declared inputs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gate flagged six of my own changes. INFINITO_ANSIBLE_COLLECTIONS_SOURCE
reached default.env without a handler, so the generator dropped it
silently; ansible.sh then carried a shell-side default for the same key,
a second source that drifts. Both are now one declaration, read as
required, so a missing key fails loudly instead of falling back.
File reads route through the session cache except where the same run
rewrites the file, which is the documented opt-out and now says so per
call. The updater reads yaml through utils.cache.yaml rather than calling
safe_load itself.
restore_collection_history.sh was committed without its executable bit,
and the new workflow's name was 31 characters where the dispatch menu
truncates at 25; the explanatory wording lives in run-name instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway could reach exactly one remote provider, through a bare
OPENROUTER_API_KEY group_var with an empty-string default and an `{% if %}`
around a single hardcoded model entry. OpenAI and Anthropic had no path at all.
All three are now declared in roles/svc-ai-litellm/meta/secrets.yml, which
mapping.py already calls the application config schema, so a key lands at
applications.svc-ai-litellm.secrets.credentials.<provider>_api_key like every
other credential. Each carries the `type` and `regex` its value must satisfy and
defaults to the central API.<provider>.api_key block, which keeps one
operator-facing place to fill a provider in while the role owns the shape.
type and regex had to become real to be worth declaring. 192 `validation:`
blocks sit in the role schemas today and not one line of Python enforces them,
so a `regex` would have joined a pile of decoration. validate_supplied_value
now runs on every path a value arrives from outside — the --set override, a
value already in the inventory, and the value standing next to a `default:`.
A generated value stays exempt because the algorithm that produced it is named
by the same schema and generate_value takes no schema argument. min_length is
deliberately left unenforced: the generators ignore it, so holding them to it
would fail where nothing is wrong.
The pattern is written for the paste, not for the vendor. OpenAI moved issuance
from sk- to sk-proj- without notice and Anthropic issues sk-ant-api03-; a schema
pinning a prefix would reject a valid key and abort inventory creation. `\S{20,}`
accepts every form those three issue today and one nobody has announced, and
rejects a trailing newline, an embedded space, a truncated copy and a
<placeholder>. A key pasted into the wrong provider's slot is caught at deploy
instead, where the route it publishes answers nothing.
The gateway iterates one LITELLM_REMOTE_BACKENDS list instead of three `{% if %}`
blocks; config.yaml.j2 and env.j2 read the same list and selectattr('key')
decides. LITELLM_CHAT_MODEL falls back to the first configured remote alias
rather than to a hardcoded openrouter/auto.
For CI the keys travel as INFINITO_API_{OPENAI,ANTHROPIC,OPENROUTER}_KEY through
default.env, the passthrough allowlist, the infinito service in compose.yml, the
workflow env block and the development inventory. The compose leg is not
optional: the env lint rejected the change without it, because a key the
inventory reads but compose does not forward resolves to empty inside the
container with no error. The workflow reads `secrets.X || vars.X` so a variable
works as asked while a secret wins, which matters because a variable is not
masked and the development inventory sets MASK_CREDENTIALS_IN_LOGS: false.
An unset key is not an error anywhere: the provider's routes stay unpublished
and the CLI test asserts their absence rather than skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
INFINITO_DOMAIN_RE carried an already escaped regex through the env
chain: EnvBuilder.setdefault prefers a value already in the process env,
quote_for_dotenv escapes every backslash, and load.sh unescapes it again.
One inherited value with a doubled backslash was enough to write
"infinito\\\\.test" into .env, the rendered Corefile then matched a
literal backslash, and getent hosts test.infinito.test failed with rc 2.
The renderer now substitutes DOMAIN_RE = re.escape(INFINITO_DOMAIN) at
render time, so no backslash travels through the env or .env at all. The
template placeholder is renamed because it is a derived render value, not
an env key, and INFINITO_DOMAIN_RE is removed from default.env and the
passthrough keys, leaving INFINITO_DOMAIN as the single source.
A renderer test renders the real template and requires a single-escaped
match with no doubled backslash and no leftover placeholder. Verified
live: the Corefile renders ^infinito\.test\.$, test.infinito.test
resolves to 172.30.0.10 and test.infinitoXtest does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ubuntu CI image build (job 103277644636) hung on Canonical's
archive: apt fetched 26 MB at 417 kB/s, then 61.6 kB in 29 s, then sat
on the package download until the job was cancelled after 23 minutes,
while download.docker.com delivered 42.8 MB in under a second on the
same runner. Locally archive.ubuntu.com and security.ubuntu.com took
20-26 s to the first byte against 2 s for a university mirror.
INFINITO_APT_UBUNTU_MIRRORS in default.env is now the one list of
Ubuntu mirrors, in failover order: de.archive.ubuntu.com, ftp.fau.de,
ftp.halifax.rwth-aachen.de, archive.ubuntu.com.
- Image builds: the Dockerfile sources the list alongside the install
scripts and runs scripts/install/apt-mirrors.sh first. In an image
with ubuntu.sources it points both stanzas at apt's mirror+file
method, priorities in list order, and sets Acquire::http::Timeout to
10 s, so apt moves on from a mirror that fails or goes silent; other
distros are untouched. In ubuntu:latest apt-get update took 8 s
against 110 s on the stock sources, a dead first mirror was left
behind within 25 s against 62 s on apt's defaults, and the swarm node
packages (80 debs) came in 17 s. The Dockerfile's own RUN lines ran
clean on ubuntu, debian, fedora and arch.
- Package cache: package.sh proxies the first entry as the primary
Ubuntu repos and the second as their fallback. Nexus revalidates
metadata on every request here (metadataMaxAge 0), which costs a
full download from a mirror without an ETag; both de.archive and
ftp.fau.de send one, and the 15 MB universe index now comes back
from the cache in 0.02 s.
- tests/external/repository/test_apt_ubuntu_mirrors.py keeps the list
honest: the first mirror must answer InRelease within 5 s (median of
three) or the test names a faster later one, no mirror's -updates may
lag the freshest by more than 24 h, and every entry must be the
primary archive, a Canonical country alias or an enabled official
mirror in Launchpad's registry. It caught the first draft of the
list: mirrors.edge.kernel.org is not an official Ubuntu mirror.
After pulling, an existing checkout needs `make dotenv-force`: plain
`make dotenv` re-reads the old .env through BASH_ENV and keeps its
values.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b8a6205396 put INFINITO_SWARM_TEARDOWN_RESERVE_SECONDS into default.env and
made 00_one.sh require it with a :? guard, but wired it into no handler.
default.env is only the source of the value; a key reaches the generated .env
solely through a handler's setdefault, so make dotenv dropped it and the
swarm routine aborted on its own guard.
The two nocheck markers cover the neighbouring key on those lines:
INFINITO_CI_DISTRO_DEADLINE_EPOCH is computed by distros.sh from the sweep
budget at run time, so a static default would be a stale timestamp. They have
to sit on the same line; above the statement the marker does not apply.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
48845db97 made three capability decisions declarable and named them INFINITO_*, which put them under setdefault: the live environment wins and the value is written back. Makefile exports BASH_ENV, so .env is re-exported into every make recipe, and scripts/tests/code/run.sh reaches dotenv through 'make setup' on every test run. A declaration made for one run therefore became a machine fact on first regeneration and stayed one - that is what made the workspace failure intermittent rather than reproducible, and why fedora died on 11 assertions where debian passed the same code.
A dedicated handler writing through eb.set recomputes the value every time and discards whatever the environment held, so the key stays registered in default.env and visible to the lints while remaining structurally unpinnable. running_on/github.py already worked this way; the distinction now has a name in utils/env/README.md. The exporting shell still steers its own run, because load.sh preserves non-empty caller values.
The other two declarations are deleted rather than moved. image_mirror_enabled and docker_root_ephemeral both derive from runs_on_github(), and the orchestrator does not unset INFINITO_RUNNING_ON_GITHUB, so inside the workspace they were already true and the declarations restated a value that held anyway. Measured across all four runtime contexts: only the workspace row changes, and only for the cache.
Verified by reversing the probe: 'INFINITO_CACHE_STACK=false make dotenv' writes .env=true, where the registry path wrote the exported value back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_used_keys_declared flags every INFINITO_* key a source file reads without a registry entry; the three capabilities 48845db97 introduced had none, so the lint suite failed on 7 use sites.
Their defaults stay empty on purpose. _declared() distinguishes an explicit 'true'/'false' from an absent value, and only the absent case lets the capability derive from the runtime identity — a concrete default would pin all three and make that derivation dead. INFINITO_CACHE_NETWORK next to them carries the same empty-means-unset shape. Same registration path as c768bbf01.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The unit suites had to install their toolchain on every run. The image
now carries the interpreters, so the make prerequisite finds them present.
The Composer vendor tree deliberately stays out of the image build: the
package cache is reached through the compose extra_hosts mapping, which
only exists for a running container, so a build-time composer install
would bypass the cache and talk to packagist directly. Fetching it from
the test target keeps it on the cached path.
The two ca_* handlers sat flat in utils/env/handlers/infinito, which the
prefix-grouping lint rejects, and the bundle-candidates comment exceeded
the .env comment length limit.
The package-cache frontend terminates TLS for package traffic and signs
with its own CA: probed from inside the test container, pypi.org arrives
as subject CN=pypi.org issued by CN=Infinito Package-Cache Frontend CA.
package-frontend-ca.sh installs that CA into the OS store, which is why
apt, pip, curl and urllib work — but requests reads certifi, which has
never heard of it, so test-external failed its reachability precondition
with CERTIFICATE_VERIFY_FAILED. Upgrading certifi does not help; the
issuer is not a public root.
load.sh is where the runner puts the venv on PATH, so it is also where the
interpreter carrying the pip certifi is chosen; pointing REQUESTS_CA_BUNDLE
and SSL_CERT_FILE at the store the stack deliberately fills keeps one trust
decision instead of two. A caller-set value still wins.
The bundle location was about to become a fourth copy: dev-nix and the
network diagnose CLI each carried their own candidate list, and the two had
already drifted apart in their fourth entry. All of them now read
group_vars CA_BUNDLE_CANDIDATES — Ansible directly, the shell through a
generated env key, python through read_group_paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32118850138 lost the :53 listener, so the sampler counts listeners.
Run 32538854067 lost DNS with the listener still bound: sockets-udp.txt
showed dnsmasq holding 0.0.0.0:53 while every forwarded query timed out,
and the sampler reported that hour as healthy.
Each tick now also records what a glibc caller gets back — for the zone
this resolver owns, derived from its own address= line, and for one name
it must forward — with the latency that caller paid. The rescue collector
probes the in-zone name too: probing external names alone cannot separate
"forwarding is gone" from "dnsmasq is gone".
The sampler now also runs on the swarm lab DNS nodes; it only ever ran in
the compose path, which is why the Magento artifact carried no sampler log
at all. Its location comes from group_vars FILE_DNS53_SAMPLER_LOG and is
required at every call site, so a run cannot quietly collect nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A run-wide draw proved one distribution and said nothing about the other four, while reporting green. Both are matrix-row axes now, assigned by the same deterministic rotation mode and tor already use, so one sweep exercises every distribution and every filesystem.
The two read the row's position like an odometer, the distro as the low digit. Turning both on the position directly covers only n of the n x m pairs whenever the pools are the same length, and no sweep unlocks it.
Both are part of a row's identity: glyphs in the job label, columns in the plan table, shards in the artifact name. Both are selectable through %distro and /filesystem, and a retrigger replays the distro off a failed title. It leaves the filesystem out, because a title states the kind the matrix assigned and a deploy may have fallen back from it.
A row's filesystem is a preference unless a human names it, on the run or in a token. Reading the matrix's own narrowing as a demand would fail rows for every transient condition the applying step is built to tolerate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
clone and covered_by both name a stand-in row: another row carries this one's coverage, so the sweep may drop it. Both were decided over every row and then applied to a filtered set, so a run that selected a subset deferred to a stand-in the filter had already removed - the row was cut in favour of a deploy that never happens. --whitelist svc-db-mariadb yielded no row at all, and the swarm-zombie job passed green while drilling nothing.
The sort's last-resort tie-breaker was drawn per process, so discovery and the plan renderer could name different assignments for the same run. INFINITO_DISCOVERY_SEED derives it from the row's own identity instead; the workflow feeds github.run_id, so the rotation still happens, once per run rather than once per process.
Measured against the previous order: on the mode-only filter both cut sets are identical (swarm 220 rows, 44 cut; compose 243 rows, 54 cut), so the full sweep is unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four findings the quality gate raised against the offset and resume work, all
of them mine.
INFINITO_CI_OFFSET sat next to INFINITO_CI_CHUNK, where it reads well and
sorts wrong, and no handler claimed it -- so `make dotenv` would never have
materialised the default and every run outside CI would have resolved the
offset from an unset variable. It is in ASC order now and passes through the
static handler.
runs.py crossed 500 lines. The three helpers that translate job titles back
into selection tokens move to selections.py, which is a seam rather than a
quota: reading a finished run and dispatching a new one is one subject,
turning a job title into `role#variant@mode+tor` is another, and only the
latter is what lets a retrigger replay exactly what failed.
The sshd handler still told operators to look for a handler named
'sshd restart' after the topic became sshd-restart, and three lines from the
offset commit were past the format width.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One discovery query per run replaces one per mode, so the row basis, the
ranking and the budget cut all speak the same role#variant unit. The
result is deployed in serial chunk blocks rather than three parallel
mode matrices.
Chunk size is derived, not pinned: GitHub cancels a job queued for
INFINITO_CI_QUEUE_HOURS, so with INFINITO_CI_CONCURRENCY runners free a
chunk holds concurrency * floor(queue_hours / the deploy job's
timeout-minutes) rows. Assuming every job burns its full timeout is
deliberately pessimistic -- no job can outlast it -- so the drain
estimate can only overshoot.
The run job cap bounds the sweep as a whole; rows beyond it roll into
the next sweep, whose regular line starts at an offset so consecutive
sweeps walk the whole list instead of re-testing the same head. Priority
rows lead every sweep, never move with that offset, and force a chunk
boundary at the seam so a chunk is either all priority or all regular.
sequencing.py goes with the mode lines it ordered: chunking restarts the
queue clock at a controlled size, which is what serialising the modes
was approximating with three buckets of arbitrary size.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Variant bundling packed several variants onto one runner and made the
job count depend on storage sums. Every variant now gets its own matrix
row instead, and the two things that used to be fixed per workflow --
which deploy mode a row runs in, and whether it sits behind the node
onion -- become axes assigned per row.
Both axes are a deterministic rotation over the row's position and the
sweep number, never random, so a red job reproduces by re-running the
same sweep. Tor turns on sweep//2 so it does not flip in lockstep with
the mode: a row walks all four mode/tor combinations over four sweeps
instead of only two.
A role offers at most two modes in practice, since swarm requires its
own stack and host requires the absence of one.
Drops with the bundler: limits.py, the variant_bundle env handlers,
INFINITO_VARIANT_BUNDLE_SIZE/_MAX_STORAGE, the per-role
meta/tests.yml variant_bundle_size key and its only user, and the
ComplexityRow bundles/jobs fields with the --deploy-mode and --max-jobs
flags that consumed them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tor coverage was decided per role variant and nowhere else: 1d9d355bc pins
services.tor.enabled true in variant 0 and false in every other variant, so an
operator had no way to steer it and a job name never said whether it ran behind
an onion. INFINITO_TOR now controls the axis from manual steer down to the
drill, and the entries that get an onion wear one in their label.
assign_tor decides per entry. 'enabled' gives tor to every capable entry,
'disabled' to none, 'auto' to every second one with the run number's parity
flipping which one starts — measured over the full matrix: 397 entries, 257
capable, 129 on even runs, 128 on odd, union 257, overlap 0. Two consecutive
runs therefore cover every capable entry exactly once.
tor_capable reads the entry's own variants, not just meta/services.yml.
Reading only the base config claimed an onion for 131 of 388 entries whose
variant had pinned the gate off — a third of the glyphs would have lied. A
bundle counts as capable when any variant it covers can use tor; a reactive
Jinja flag counts as capable, since that is the role saying 'onion when the
node is dark'.
The 'disable' token is the working half: it prunes svc-net-tor from the include
closure (utils/tests/swarm/derive_includes.py) and clears the service flags in
host_vars (cli/administration/inventory/provision/services_disabler.py). Both
halves are required — with only the first, extend_inventory walks the provider
back in. It runs after the variant overlay, so it also overrides a variant that
pinned the gate true.
Only swarm and compose join the axis. Host discovery shares the same matrix
builder, so without the allowlist its jobs would carry onion glyphs for a mode
that has no onion at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scripts/tests/environment/ becomes scripts/tests/workspace/ with one
subdirectory per track. base/ prepares the workspace and is a prerequisite;
compose/ and swarm/ are independent of each other, so CI can run them as
separate jobs with a budget each. 00_orchestrator.sh runs the tracks named
by INFINITO_WORKSPACE_TRACKS and aborts on an unknown one.
The job count stays at five: cli.meta.ci.dev_runtime matrix assigns exactly
one track per distro and flips the split with the run number, so two
consecutive runs cover both tracks on every distro without doubling the
matrix. The workspace_track input (rotate|compose|swarm) pins it from manual
steer through the orchestrator for a targeted re-run.
Job names carry the track glyph from utils/symbol_glossary and the image
shortened to its last two segments, so quay.io/centos/centos:latest reads
as centos:latest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The frontend proxies to the package-cache by name, and nginx resolves a
literal proxy_pass hostname once while parsing the config and keeps it for the
worker's lifetime; the resolver only re-resolves targets that contain a
variable. The package-cache was the one cache service without a pinned
address, so recreating it left the frontend dialling a dead one, answering 502
to everything and reporting unhealthy.
That surfaces far from its cause: pip.conf deliberately keeps the real index
URL and extra_hosts points pypi.org at the frontend, so a pip install reports
no versions at all and looks like a network fault.
Pins it the way the registry-cache and the frontend are already pinned, and
carries the address into the runner slots.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
EnvBuilder.setdefault already reads os.environ.get(key).strip() and prefers it over the computed value. Three handlers repeated that check in front of eb.set: four dead lines each plus an import, restating a contract that lives one module over. If setdefault ever changes, the three copies do not follow.
Behaviour is identical and was measured on both paths: each handler yields the same derived value as before, and with os.environ[KEY] set the override still wins. The only change in flow is that an overridden value is now computed and discarded, costing one df or /proc read per make dotenv.
The host-path guard in registry_cache_max_size stays: it prevents a df on an empty path, which is a different concern from the override.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The five DinD lab nodes run privileged with cgroupns=host, a writable host cgroupfs and seccomp/apparmor off, and carried no resource limit of any kind. A runaway node therefore competes with the Actions runner agent itself, and the platform reports that only as 'lost communication', never as an OOM.
Only pids_limit for now. A memory cap needs a measured per-node peak, and every measurement available is a failure-time snapshot, i.e. a lower bound: 4.4 GiB on the busiest node of a variant-0 drill. Sizing the cap off that risks OOM-killing a legitimate container instead of the runaway node. The pid ceiling has no such problem: 8192 against a measured 981 is 8.3x.
The value is a static default.env key on purpose. A handler-derived key is absent from .env on a fresh runner, and the topology export step runs before the install step that generates it, so compose interpolation would have failed every swarm job before anything deployed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When the host runs out of memory the Actions runner loses its heartbeat. GitHub then reports only 'lost communication': no step completes, so every if-always upload step is skipped and the rescue diagnostics, the inventory artifact and the Playwright results are all lost. Job 92710765494 died exactly that way.
The 30s poll in _run already did this for disk. It now also samples MemAvailable, MemTotal and the memory PSI stall, prints one line per poll, and aborts below 6 percent of RAM. The abort returns 75, which propagates to the EXIT trap in 00_one.sh, so the existing post-mortem apparatus runs normally instead of dying with the runner.
The sample line is the main point: artifacts provably do not survive a lost runner, streamed log lines largely do, and the numbers discriminate memory thrash from a kernel wedge on the next occurrence. The floor is relative to MemTotal so a dev box does not inherit a runner-sized threshold; measured margin across four real drills is 9-15x. PSI is sampled but not used as a trigger, since no drill has ever measured it.
mem_total_mb and mem_stall_pct join the existing host probes in utils/env/runtime.py and share the /proc/meminfo parser. The probes read only /proc, never the host-side NFS mount, which would block in D-state under the very condition they exist to catch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub cancels a job queued for 24h. A line starting swarm, compose and
host at once queues its whole matrix in one go, so with more roles than
runner slots the tail waits past that cut and dies unrun -- 54 of 241
jobs in run 30937913632.
Deploy the modes one after the other instead, heaviest first, which
restarts the queue clock per mode. cli.meta.ci.sequencing counts a
line's matrix jobs on the row basis the discover steps use and
serialises above INFINITO_CI_SEQUENTIAL_THRESHOLD; below it the modes
still start together, so a small diff pays no wall-clock for this.
A needs: edge is static, so both layouts exist as jobs and the decision
skips one. Their conditions are structurally asserted, not evaluated:
ten near-identical if expressions invite a copied sentinel or a missing
chain edge, and both failures are silent at runtime.
mode_fail_fast stops a serial line at its first failed mode, on by
default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub cancels a job queued for 24h. A line starting swarm, compose and
host at once queues its whole matrix in one go, so with more roles than
runner slots the tail waits past that cut and dies unrun -- 54 of 241
jobs in run 30937913632.
Deploy the modes one after the other instead, heaviest first, which
restarts the queue clock per mode. cli.meta.ci.sequencing counts a
line's matrix jobs on the row basis the discover steps use and
serialises above INFINITO_CI_SEQUENTIAL_THRESHOLD; below it the modes
still start together, so a small diff pays no wall-clock for this.
A needs: edge is static, so both layouts exist as jobs and the decision
skips one. Their conditions are structurally asserted, not evaluated:
ten near-identical if expressions invite a copied sentinel or a missing
chain edge, and both failures are silent at runtime.
mode_fail_fast stops a serial line at its first failed mode, on by
default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
