A cancel request does not stop a running step. The runner asks it to end, and a step that ignores the signal keeps going, bounded only by its own timeout-minutes -- 350 for a deploy. Meanwhile the concurrency group holds the newer run on pending until the occupant is terminal, not until it is cancel-requested. Measured on run 32859801382: the cancel was accepted at 15:27:45 and thirteen minutes later ten deploy jobs were still running and the newer run still had not started.
The sweep now waits FORCE_CANCEL_AFTER_SECONDS and force-cancels whatever is still not completed. Cleanup steps are skipped for those, which is the price: the artefacts of an already superseded run help nobody. Only entry-cancel-superseded.yml sets the variable -- a closed pull request and a deleted branch have nothing waiting on the group, so they keep cancelling and walking away.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A long orchestrator sweep sometimes survives its concurrency group: the occupant sits on queued while its matrix waits for runners, cancel-in-progress does not reap it, and the newer run parks on pending until someone cancels the sweep by hand. PR #380 sat that way for nine hours behind a run from the previous commit.
entry-cancel-superseded.yml cancels those leftovers through the API. The group stays primary and untouched: it reaps within seconds and needs no runner, so the fallback normally arrives to find nothing left to do. It carries no concurrency group of its own, because the run holding a group cannot be the run that frees it.
Selection mirrors what the group would have done: cancel every matched run except the newest in the group instance, keyed the way the group is keyed. Not by head SHA, which goes stale while the job waits for a runner and would cancel the newest commit's CI while sparing the stale one. INCLUDE_PATHS keeps the sweep inside the group's own members, so a push to main cannot take the cron workflows with it.
deleted_branch_runs.sh becomes branch_runs.sh, since it now also serves live branches, and the shared parts move to lib.sh. A rejected cancel is logged and the remaining runs are still attempted; only a denied token fails the job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The coherence step wrapped a blocking 'systemctl try-restart' in a 120s
budget, one second short of the wedge-recovery cycle the unit performs
when the liburcu-bp startup wedge (f10a4a181) fires - observed in run
32657210518, where the Akaunting drill died on rc=124 while ganesha came
back healthy 4s later. 240s covers even the uncapped 123s cycle; with the
abort grace capped at 10s (64b39e227) one cycle is ~43s.
The '|| systemctl try-restart nfs-server' fallback also masked exactly
this failure: nfs-server.service is 'inactive dead' on a ganesha node,
try-restart on an inactive unit is a documented no-op exiting 0, so a
larger budget alone would have turned a SIGKILLed ganesha into a green
drill. Dispatching on the installed unit file keeps the kernel-NFS branch
and lets a real restart failure surface: a wedge leg now fails loud after
one ~43s cycle instead of passing on self-healing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
48845db97 made three capability decisions declarable and named them INFINITO_*, which put them under setdefault: the live environment wins and the value is written back. Makefile exports BASH_ENV, so .env is re-exported into every make recipe, and scripts/tests/code/run.sh reaches dotenv through 'make setup' on every test run. A declaration made for one run therefore became a machine fact on first regeneration and stayed one - that is what made the workspace failure intermittent rather than reproducible, and why fedora died on 11 assertions where debian passed the same code.
A dedicated handler writing through eb.set recomputes the value every time and discards whatever the environment held, so the key stays registered in default.env and visible to the lints while remaining structurally unpinnable. running_on/github.py already worked this way; the distinction now has a name in utils/env/README.md. The exporting shell still steers its own run, because load.sh preserves non-empty caller values.
The other two declarations are deleted rather than moved. image_mirror_enabled and docker_root_ephemeral both derive from runs_on_github(), and the orchestrator does not unset INFINITO_RUNNING_ON_GITHUB, so inside the workspace they were already true and the declarations restated a value that held anyway. Measured across all four runtime contexts: only the workspace row changes, and only for the cache.
Verified by reversing the probe: 'INFINITO_CACHE_STACK=false make dotenv' writes .env=true, where the registry path wrote the exported value back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One boolean answered five unrelated questions: whether the pull-through cache stack runs, whether image pulls take the GHCR mirror, whether the docker data root may be wiped, plus runtime naming and inventory resolution. The workspace is the environment where those answers diverge — it simulates a developer machine on a real runner — so getting one of them meant misreporting where the suite runs and silently paying for the other four.
is_ci() and runs_on_github() stay facts about the machine. cache_stack_enabled, image_mirror_enabled and docker_root_ephemeral become decisions that default off those facts and can be declared explicitly, so an environment states what it wants. Every default reproduces the previous condition exactly, including the distinction the old mirror and wipe gates drew between a real runner and act.
73ae40585 got the cache stack by flipping the identity flag and lost the docker-root wipe with it; the workspace now declares all three capabilities instead. Pairs with 4c068d1cf, which makes the resulting stray bind sources harmless rather than fatal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
03_wait_converge built the DB service name from the engine type:
"${DB_DEP}_${DB_DEP}" -> mariadb_mariadb. That name is only correct when
the database is central. force_shared_db deliberately leaves a
manager-pinned role carrying a literal `shared: false` alone, so
web-app-magento keeps its embedded engine, and sys-svc-rdbms deploys it as
magento_database inside magento's own stack.
The name came from the engine, which is global, and was looked up with
`docker service ls` across the whole swarm instead of the role's stack, so
the `shared` flag never entered the check at any step. Both polarities were
wrong:
variant #1 - no consumer pulls the central provider in, so
mariadb_mariadb does not exist. Run 32630684501 job 97175522113 polled it
ninety times and failed the drill, while the rescue artifact shows
magento_database at 1/1 Running and the play at failed=0.
variant #0 - matomo and mailu declare a Jinja `shared`, get force-shared
and drag svc-db-mariadb in. Job 95662862942 then reported
`db(mariadb): 1/1 Running 2 hours ago` beside `magento: Running 21
minutes ago` - green on a database magento never touches.
resolve_db_service asks the deployed swarm which service carries this
role's database: <entity>_database first, else <dep>_<dep>. The two cannot
collide, because sys-svc-rdbms renders the _database sidecar only under
`lookup('database', application_id, 'local')`. Exact-line matching, not
docker's substring name filter, per 246558a13.
Only web-app-magento changes behaviour: it is the one role that is both
pinned by force_shared_db and seen by the DB_DEP grep. matrix and
bigbluebutton never reach the DB half - their entity key carries no image,
so skip_if_no_swarm_service exits first - and discourse resolves
postgres_postgres exactly as before, in every variant.
A role that declares a DB dep no service serves now says so and gates the
app half alone, instead of polling a name nobody deployed.
diagnostics.sh carried the same guess in its container filter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workspace suite represents a developer machine, but Profile.is_ci() reads three signals and the block neutralised only two. INFINITO_RUNNING_ON_GITHUB is baked into .env by the runner-side make dotenv before the container starts, and load.sh regenerates .env only when it is missing, so the value survived into a container that had already unset GITHUB_ACTIONS. registry_cache_active() stayed false, make compose-up omitted compose/cache.override.yml, and the compose track then asserted pull-through on a cache stack that was never created.
40375decb wrote this block before that third signal existed; 94936f198 added the Profile gate a month later.
Accepted side effect: the workspace no longer resolves as GitHub, so it skips the GHCR mirrors and the docker-root wipe in down.py, which key off the same flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make test execs four targets into the same container and each one runs dev-extras.sh. The stamp guarding it was written after the install, so all four passed the check and ran concurrent pip install -e into a single venv, deleting each other's files mid-install. 1a908a6e5 introduced the fan-out; the retry loop in python.sh has masked the fallout ever since, at the cost of a silent fallback to serial pytest when it runs out of attempts.
Taking the lock before the stamp is read makes check, install and stamp one atomic section.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The step summary renders every junit report under build/test-reports,
so the three suites were absent from it for want of a report file. They
now write one under the same name the python runner derives from
INFINITO_TEST_TYPE.
minitest ships no junit reporter, and a gem for one file would be a
dependency the otherwise stdlib-only suite does not need, so it gets a
small plugin instead. The runners also provision their own toolchain
now that the host-side make prerequisites are gone.
Only test-unit-python went through the container dispatcher; the
javascript, php and ruby targets called their runner directly and so
executed against the host, where the toolchain may be absent and the
sandbox blocks the install.
The dispatcher was never python-specific, it just hardcoded run.sh.
It now takes the runner as an argument, and the host-side install
prerequisites drop out with it.
node expands a `**` pattern only from version 21 on, and install-node
resolves to node 18 on debian, so the suite reported "Could not find" and
exited 1 without running a single test. find now collects the files and
node receives them as arguments.
An empty suite exits non-zero for the same reason php and ruby already
do: reporting success for a suite that never ran hides its absence.
The unit suites had to install their toolchain on every run. The image
now carries the interpreters, so the make prerequisite finds them present.
The Composer vendor tree deliberately stays out of the image build: the
package cache is reached through the compose extra_hosts mapping, which
only exists for a running container, so a build-time composer install
would bypass the cache and talk to packagist directly. Fetching it from
the test target keeps it on the cached path.
test-unit-php and test-unit-ruby exited 0 whenever the interpreter or the
vendor tree was missing, so both suites were silently absent from every
gate. They now provision their toolchain through install-php/install-ruby
and fail loudly when that is impossible.
The three ensure_*_present helpers collapse into ensure_command_present
in system_pkg, and the ruby runner rejects an empty suite because
minitest reports success when nothing was required.
The package-cache frontend terminates TLS for package traffic and signs
with its own CA: probed from inside the test container, pypi.org arrives
as subject CN=pypi.org issued by CN=Infinito Package-Cache Frontend CA.
package-frontend-ca.sh installs that CA into the OS store, which is why
apt, pip, curl and urllib work — but requests reads certifi, which has
never heard of it, so test-external failed its reachability precondition
with CERTIFICATE_VERIFY_FAILED. Upgrading certifi does not help; the
issuer is not a public root.
load.sh is where the runner puts the venv on PATH, so it is also where the
interpreter carrying the pip certifi is chosen; pointing REQUESTS_CA_BUNDLE
and SSL_CERT_FILE at the store the stack deliberately fills keeps one trust
decision instead of two. A caller-set value still wins.
The bundle location was about to become a fourth copy: dev-nix and the
network diagnose CLI each carried their own candidate list, and the two had
already drifted apart in their fourth entry. All of them now read
group_vars CA_BUNDLE_CANDIDATES — Ansible directly, the shell through a
generated env key, python through read_group_paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32118850138 lost the :53 listener, so the sampler counts listeners.
Run 32538854067 lost DNS with the listener still bound: sockets-udp.txt
showed dnsmasq holding 0.0.0.0:53 while every forwarded query timed out,
and the sampler reported that hour as healthy.
Each tick now also records what a glibc caller gets back — for the zone
this resolver owns, derived from its own address= line, and for one name
it must forward — with the latency that caller paid. The rescue collector
probes the in-zone name too: probing external names alone cannot separate
"forwarding is gone" from "dnsmasq is gone".
The sampler now also runs on the swarm lab DNS nodes; it only ever ran in
the compose path, which is why the Magento artifact carried no sampler log
at all. Its location comes from group_vars FILE_DNS53_SAMPLER_LOG and is
required at every call site, so a run cannot quietly collect nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test-unit-javascript ran `node --test` without declaring where node
comes from, while its python sibling declares `install`. On a bare
distro image the target died with `exec: node: not found` (127), which
took down the pre-commit hook and all five test-workspace jobs.
The installer stack already knows how to pull a command from the system
package manager; node was simply missing from the table. Wiring the
target to install-node provisions it the same way php, ruby and npm are
provisioned, and fails loudly when the runtime cannot be installed
rather than skipping the suite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every scheduled workflow opened with an alarm clock, so the Actions list
read as a column of identical icons that said nothing about the run. The
clock now lives in run-name behind github.event_name == 'schedule', where
it marks the one thing it can mark: a run nobody triggered. The name says
what the run does, in two or three words.
The dispatch menu shows names in one narrow column and truncates them, so
a new lint caps a workflow_dispatch name at 25 characters and points the
longer, explanatory wording at run-name, which carries no such limit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four parse checks join the `lint-*` family, each using the language's own
parser rather than a reimplementation: `php -l`, `ruby -c`, `sqlfluff parse`,
and stylelint with an empty rule set (it raises CssSyntaxError whatever the
rules say, so the check stays syntax-only and grows no style opinion). Three
unit runners join `test-unit`: node:test, PHPUnit, minitest -- the first and
third ship with their runtime, so only PHPUnit adds a dependency.
Provisioning is best effort and the lint script decides whether it can run. A
machine without PHP must not fail the gate, but it must not look like success
either: the skip names how many files went unchecked. The first version of
these scripts read `git ls-files`, which inside the compose container returns
nothing (git refuses the bind-mounted repo as "dubious ownership") and reported
"No SQL files found." over 52 files -- green, and a lie. They use `find` now,
as the surrounding lint scripts already did.
sqlfluff needed two corrections before it parsed anything real. `%(name)s` is a
psycopg parameter marker, not SQL, so the placeholder templater is configured
rather than the files exempted; and the dialect follows each role's declared
engine (`meta/services.yml`), which is postgres for six roles and mysql for
svc-db-mariadb and pixelfed.
Recipes stay in scripts, not in the Makefile: the targets delegate, matching
every other target in the file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0d8ed0ba8 bounded these call sites with one ceiling for both operations.
The two are not equally safe to cut. A SIGTERM landing mid-postinst
leaves dpkg interrupted, and in docker_dataroot.sh that fails every later
filesystem candidate and sends the run through decline() - one truncated
package install becomes a dead job. An interrupted update has no such
cascade; it just repeats.
Measured on ubuntu:latest with the exact commands: update 10s, install
zfsutils-linux 20s. 1200s is a 60x margin on the operation that carries
the cascade, and update keeps the 600s it cannot plausibly need more than.
This breaks the single-number invariant with scripts/install/apt.sh:19,
which is already broken in fact - that call site carries a different
Acquire option set than these three do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A run-wide draw proved one distribution and said nothing about the other four, while reporting green. Both are matrix-row axes now, assigned by the same deterministic rotation mode and tor already use, so one sweep exercises every distribution and every filesystem.
The two read the row's position like an odometer, the distro as the low digit. Turning both on the position directly covers only n of the n x m pairs whenever the pools are the same length, and no sweep unlocks it.
Both are part of a row's identity: glyphs in the job label, columns in the plan table, shards in the artifact name. Both are selectable through %distro and /filesystem, and a retrigger replays the distro off a failed title. It leaves the filesystem out, because a title states the kind the matrix assigned and a deploy may have fallen back from it.
A row's filesystem is a preference unless a human names it, on the run or in a token. Reading the matrix's own narrowing as a demand would fail rows for every transient condition the applying step is built to tolerate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two scripts sharing a delimiter-bounded name prefix are a folder in disguise, and tests/lint/repository/structure/test_prefix_grouped_files.py refuses them. A second release_* resolver is coming, so give the group its directory now and drop the prefix from the name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ba8a72309 bounded scripts/install/apt.sh and d50ba46b0 bounded the swarm
bootstrap; these three were the remainder, and one of them was still
costing whole jobs. docker_dataroot.sh hung Nextcloud#7 (job 96179137223,
run 32284644282) for 5h40m: "installing zfsutils-linux for zpool" at
19:50:39 is the last line before "rc=124 after 20400s" at 01:30:39, with
nothing in between. distros.sh's global budget is the only ceiling below
these calls, so one stalled fetch also takes the untried distros with it.
Both update and install are wrapped, not just update - that hang sat in
the install, and apt-get install fetches from the same mirror. Constants
come verbatim from scripts/install/apt.sh:19-20 rather than being
re-derived.
host_deploy.sh runs inside a container via `docker exec ... sh -c`, where
POSIX sh has no arrays, so its options are a word-split string instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
01_bootstrap.sh ran apt-get update with no timeout, so a stalled mirror
connection hung until the drill's global budget killed it - twice in run
32284541284 (jobs 96178507396 funkwhale, 96178507443 friendica), each
after hours of silence. The journal pins it exactly: sudo[462047] opened
19:51:36 and closed 00:07:30, 4h15m54s in a single apt-get update, and
the session only ended because timeout fired. Healthy runs of the same
line in the same jobs took 3s, 21s and 41s.
The cost is not one red job. distros.sh:130 is the only ceiling and it
wraps the whole per-distro sequence in the remaining global budget, so
one stalled fetch also takes the untried distros with it: funkwhale ran
2 of 5, friendica 3 of 5, both ending at budget remaining=-19s/-20s.
-qq hid the stall and || true cannot catch a hang.
Same failure class and same remedy as ba8a72309, which bounded
scripts/install/apt.sh; this closes the sibling call site it left open.
Constants are taken verbatim from there rather than re-derived. timeout
runs under sudo so SIGTERM reaches apt-get instead of the sudo wrapper -
the 4h session above is what the other order looks like. -E stays so the
package-cache proxy variables survive.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Azure Ubuntu mirror stalled during runs 32265854076 and 32265543004, apt-get update burned its 10m timeout and exited 124, and 40 deploy jobs died in the install step. Every package the wrapper is ever asked for ships with the ubuntu-24.04 runner image, so the network round trip bought nothing and cost 60s per job.
dpkg-query decides per package now; apt only runs for what is genuinely missing, which keeps the act image (no envsubst, no skopeo, no gh) working.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scripts/install/apt.sh ran apt-get without any timeout, so a stalled mirror connection hung until the 355-minute job limit killed the job - three times across both runs (95883075054, 95883075011, 95883448555), each after roughly 5h50m of silence following the last apt line, "Get:5 noble-security InRelease". The Install-jq step carries no timeout-minutes of its own, so nothing bounded it.
Acquire retries/timeouts mirror the SPOT that 07d31d900 established for deployed hosts in roles/update/tasks/debian.yml. The timeout wrapper is the backstop for what Acquire cannot cover (dpkg lock, hanging post-install), and runs under sudo so the signal reaches apt-get itself rather than the sudo wrapper; -k 30 follows up with SIGKILL. 10m is 5.5x the slowest successful run measured in these same runs (6.7s / 41.1s / 81.3s / 109.8s).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32118850138 left a dnsmasq that was alive with zero sockets,
nothing logged, and a 6m39s death window between two host DNS
consumers. Capture each dnsmasq's fd table and rlimits in the rescue
snapshot (closed listeners vs exhausted fd table vs lost netlink
watch), and sample the :53 listener set every 10s inside the DiD
container so the next occurrence is bounded to one bucket.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The restore cycle stops every subject of the host, replays into the database
services alone, and starts the rest again at the very end. Any exit in
between skips that last step, so the host keeps its subjects down - and
nothing recorded it.
The next deploy then failed on a symptom of the teardown rather than on
whatever ended the drill. It cost two runs: once 'rake db:migrate' against a
database the cycle had emptied but not yet replayed, once
Redis::CannotConnectError because the cycle never restarted redis. Both times
the message pointed away from the cause, and fixing what it named would have
fixed nothing.
A trap armed before the teardown now records the teardown and the exit code,
and mode=update refuses to run while that record stands. purge=true and
mode=reinstall pass, because both rebuild the state the drill destroyed; a
completed cycle clears the record itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Output of 'make autoformat': ruff collapses expressions that were wrapped across lines without need, shfmt and the yaml formatters drop surplus blank lines. No semantic change - the whole diff is 10 insertions against 39 deletions, and every hunk joins lines that already said the same thing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The regular rows began wherever (sweep * capacity) % count pointed. On run 210
that was offset 68: chunk 0 opened at candidate 69, the coverage-first head of
the list went undeployed, and no two runs could be compared because the window
moved with the run number. A run that looked wrong turned out to be merely
rotated, and finding that out meant recomputing the arithmetic by hand.
The rotation is gone. Chunk 0 reads the ranking from the top unless an offset
says otherwise, and the offset is an input now, threaded from the manual form
through the orchestrator and the deploy call down to the matrix builder. The
plan renderer takes the same flag, so the table shows the window the run will
take.
The offset is a row count or a selection token. A token is the honest form:
the ranking shifts whenever a variant appears, so 'offset 84' means something
different tomorrow while 'web-app-nextcloud#5' still means the same row. Axes
the token leaves open match anything, so a bare role name starts at its first
row and a pasted job label starts at exactly the job it names. A token that
matches no regular row aborts rather than silently starting at the head an
operator meant to skip.
The run title carries it as ⏩ followed by the value, and only when one was
asked for, so a rotated-looking run says so in its own name and the retrigger
reads it back from there when a job log has expired.
resume_from_chunk leaves the manual form with it. Skipping whole chunks was
the coarse way to say 'carry on further down'; the offset says it exactly and
keeps the rest of the chain intact. The orchestrator input stays for the other
entry points.
What this gives up is automatic tail coverage. Rows beyond the budget are
reached by moving the offset, widening the budget or narrowing the selection,
not by waiting for consecutive sweeps to walk the list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge assertion was a job nobody depended on. ci-orchestrator needed the
scope, the branch prefix and the affected roles -- not it -- so a PR whose base
was not merged in went on to spend five distros on a deploy sweep and reported
a single red job beside a chain that had already finished.
Validation is now a reusable workflow every entry calls, and what it checks
follows the event and the inputs rather than the caller: a pull request gets
its branch prefix and merge base asserted, a run carrying a selection gets that
selection checked, a fork PR waits here for the privileged run's images. The
wait and its gate move in with it, so the orchestrator no longer keeps a second
copy of that logic. Every downstream job depends on the gate.
The selection check is what the last two nights needed. cli.meta.ci.validate
resolves every token against the branch that would deploy it -- role, variant,
mode, onion state -- and reports all offenders at once. Until now the matrix
builder raised on the first bad token from inside a chunk job, so a stale
priority line cost one full CI round per entry and only ever showed the next
problem after the previous one was fixed. Measured on a real list: three
unusable tokens, one pass.
A bare role name that matches nothing is a warning, not an error: the
diff-derived whitelist legitimately names roles the lifecycle envelope filters
out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were left by earlier commits and are collapsed by markdownlint-cli2 and shfmt respectively, so every make quality run left the tree dirty and the change would eventually have ridden along in an unrelated commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The converge gate polls STACK_NAME_PRIMARY_SERVICE_KEY. Four roles never
create that name and each was given a static `workload: node-local` to make
the gate skip itself:
21fe9e984 web-svc-cdn ships only cdn_build, never cdn_cdn
2c6c8b763 web-app-matrix names its services matrix_synapse and friends
76508c946 svc-runner deploys node-local compose projects per node
cb52a72f2 bigbluebutton phantom service, and the DR drill must still run
Four hand-written claims for one observable fact. The claim is also static
where the truth is not: web-app-discourse renders `services: {}` only while
its sidecars are shared, and job 94965456921 of run 31864448027 polled
discourse_discourse ninety times for a stack that had none. Declaring the
role node-local would have blinded the gate in the variants where its
sidecars are local and a stack does exist.
has_swarm_service asks the deployed stack for the exact name instead. Every
caller runs behind `if rc == 0`, so an absent service is the role's shape and
not a broken deploy. Exact-line matching, not docker's name filter, which
matches substrings.
No coverage is lost: 03_wait_converge cannot pass without that service, and
05/06/07 plus the backup drill only run once it has. The four roles keep the
skip they were given; discourse and every future case get it without a
declaration. The teardown branch in the drill keeps its meaning - stack rm
where the service exists, compose stop per node where it does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A whitelist or priority entry was a role name and nothing else, so a run
could not ask for "this variant, on swarm, behind the onion". It either
covered everything the role offers or waited for the rotation to come
around to the combination that broke.
An entry is now a token that may carry the axes, either as
web-app-x#0,2@swarm+tor or as the job label pasted straight back out of
the run that failed. What a token leaves open still belongs to the line
it stands in: priority covers every remaining combination in one sweep,
whitelist keeps the rotation and its position in the matrix. A pin the
row cannot take aborts the matrix rather than dropping the row, because
a dropped row reports a green run for a combination that never ran.
The onion state spells out as +clearnet, not as a -tor suffix: svc-net-tor
would otherwise read as role svc-net on the clearnet.
Chunk membership still follows the discovery ranking, which is what
decides who makes the budget cut, but the rows inside a chunk are sorted
by display name, variant, mode and onion state, so a chunk's job list
reads like the plan table instead of like the sweep's rotation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dnsmasq and apt-resolution investigations both stalled on facts the
snapshot never recorded.
The snapshot is taken by docker exec inside the DiD container and the
recursion only descends, so the runner's own docker - where the resolver
runs as ${INFINITO_RUNNER_PREFIX}-coredns - was structurally invisible.
dedicated.sh now runs a second pass on the CI host, with RESCUE_SEEN
seeded with the DiD id so it does not collect the first snapshot twice.
For the rest: a process table, so the MainPID that service-state reports
can be checked against something; the dnsmasq drop-in directory, because
this project's config lives at /etc/dnsmasq.d/tor-onion.conf and never in
the main file; nsswitch.conf and hosts, without which a getent verdict is
ambiguous; one probe per name plus, where the distro ships dig or
nslookup, one per nameserver, so a refusing resolver is distinguishable
from an unreachable mirror; the full firewall table set, since a dropped
query is recorded in filter and -t nat was actively hiding it; routes and
neighbours; os-release, because the matrix is distro-shaped and a bundle
could not say which leg it came from; the runtime's networks, which hold
the forwarder behind every container's 127.0.0.11; free inodes, which a
df -h reads as free space; and the runtime's disk, volume and image
inventory, the last with full references, since an engine was once picked
from a registry host embedded in an image name.
ps runs without args: the bundle is uploaded as an artifact and argv
carries whatever credentials a process was invoked with - a real proxy
password showed up in the first live run of this capture.
Two exec budgets are corrected while the per-container cost grows to
three calls. The per-container journal was borrowing the 600s meant for a
host boot journal, so one wedged container could cost 840s; container
journals span a deploy, not a boot. The cheap questions get 30s, and the
one collection site that had no outer bound gets the timeout 1500 the
others already carry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
02_performance deployed matomo with no disable set at all, so svc-net-tor joined the groups. It is exclusive, matomo follows it through tor.enabled in its own services meta, and the clearnet vhost disappears. The assertion two lines down then aborts in TLS before HTTP: 'curl: (35) TLS connect error: tlsv1 unrecognized name', reported as '[FAIL] https://matomo.infinito.example returned HTTP 000, expected 200'.
The counter-example sits in the same job: 01_minimal deploys dashboard with the group-conditional services disabled, tor among them, and its clearnet assertion returns 200. This suite tests the inventory, not the transport -- that a dedicated deploy replaces the previous app set. So it takes the route the sibling script already proves, rather than asserting against an onion address the runner never touched.
03_reuse stays untouched: it runs mode=update against the inventory 02 just built.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
call-test-deploy-{compose,host,swarm,single-node}.yml differed only in
which mode they pinned and which artifact prefix they wrote; compose and
host already ran the identical drill. They become one call-test-deploy.yml
whose matrix rows carry the mode, so the swarm-specific steps are guarded
on it and everything else is shared.
The orchestrator loses its eight deploy jobs and the sequencing job for
three serial chunk blocks. GitHub Actions cannot generate a variable
number of jobs, so the blocks are written out and INFINITO_CI_MAX_CHUNKS
tells the budget how many exist; blocks whose slice is empty skip
themselves. chunk_gate stops the chain at the first red chunk,
resume_from_chunk re-enters after a fix instead of re-running the green
chunks, and modes collapses to a single mode choice.
apps.sh now emits matrix entries for one chunk rather than a token list
for one mode, and no longer sorts them: the discovery order is what the
chunker slices on. split_runner_apps.sh reads its runner quota from
INFINITO_CI_CONCURRENCY instead of carrying a second copy of the 20.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
30% was a guess. Measuring it: across 60 successor distro runs in 22 jobs of run
31756400129, each against the running max of the runs before it, the median
overshoot is 0.991 - a successor is typically slightly faster - and the worst
ever observed is 1.149 (fedora 5541s against a 4822s max_seen).
15% therefore covers every observed case, and nothing above it buys more. It
costs less, too: 30% refuses 23 of 74 start decisions, 15% refuses 18. Five
distro runs that would have fitted, for no additional protection.
The tests read the percentage back out of the log line and check the formula
against it, so they hold at any value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
508669809 booked a budget SIGKILL as skipped so the job went green. That is a
soft-skip: a kill is indistinguishable from the command hanging, so it hid a
real failure behind a green tick.
Fail on it again, and prevent it upstream instead. A distro now starts only
while the remaining budget covers the slowest run so far plus 30%. Fediwall run
31756400129 shows why the bare comparison was not enough: max_seen=6770s against
remaining=6878s let the distro start, and it took 6942s, 64s past the deadline.
max_seen is the maximum of the durations already observed in the same sweep -
a small sample over distros of genuinely different weight, drawn in randomised
order, so it underestimates systematically.
Removing the soft-skip also removed its diagnosis, so a killed distro now
reports both numbers without claiming a cause: the run only knows the exit code
and how much budget was left when it started.
The suite drops from 85s to 4.1s. The 68s case made a process ignore SIGTERM to
force the -k escalation, which exercised coreutils rather than this script; the
124 path reaches the same branch in a second.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ad9222630 wrapped each distro in timeout so a run that outlives the budget stops instead of drifting into the action timeout, and mapped rc 124 to skipped because stopping at the budget is the documented intent rather than a defect. It verified that timeout returns 124 for its own kill. It did not cover the other one: timeout -k escalates to SIGKILL when the command survives SIGTERM, and then reports 137, which is equally its own kill. An ansible run with live docker children survives SIGTERM routinely, so 137 is the common case rather than the exotic one.
The result was that the governor never fired as documented. Fediwall swarm and Nextcloud swarm 0 in run 31756400129 and 31756560361 both ended on [ERROR] Run failed for distro=arch (rc=137) with budget=20400s remaining=-60s. Fediwalls playbook had already printed PLAYBOOK RECAP, the stack had converged and every node port answered; it was booked as a failure 0.3 seconds later.
The two narrowing conditions carry the discrimination and are unchanged. dur >= remaining - 5 rejects a SIGKILL that arrives at an arbitrary time, such as an OOM kill mid-run, because a budget kill lands at the deadline by construction. passed > 0 still requires an earlier distro to have succeeded.
A unit case in tests/unit/scripts/tests/deploy/test_distros.py drives a distro that traps SIGTERM and sleeps. Without the change it reproduces the CI failure locally, exit 137 and Run failed for distro; with it the sweep exits 0 and the summary marks the distro skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
default.env declares INFINITO_DOMAIN and everything is meant to derive from it.
Nothing checked that, so copies accumulated. A copy is a second source nobody
updates, and it fails in the least useful way: the code keeps running and
quietly talks about the wrong host. integration_mastodon is the worked example
-- it asserted the provisioned partner equalled a clearnet literal, so the spec
passed on a clearnet node and failed on an onion one while the product was
right in both.
tests/lint/repository/spot/test_no_hardcoded_primary_domain.py now fails on the
literal anywhere in tracked source. It reads the value it hunts for FROM
default.env, so the rule follows the SPOT instead of repeating it and would
otherwise be its own first violation.
Scope, and why each exclusion is not a loophole: default.env itself is the
source; .md is documentation; tests/ and scripts/tests/ hold fixtures that must
assert against a named host; inventories/ is where writing the real values is
the entire point; gitignored output carries the value because it correctly
derived it. Everything else is in.
Two real hardcodes are repaired rather than suppressed.
utils/domains/default_primary.py mirrors utils/roles/lifecycle.py -- one
function reading one key through parse_static_env -- and replaces the module
constant in utils/cleanup/nginx_vhosts.py and the argparse default in
cli/meta/domains/__main__.py. Both resolve to the same value as before.
Seven documentation sites become <primary-domain> placeholders instead of
carrying a nocheck: a Python docstring cannot host one without polluting the
docstring, and a placeholder does not rot when the default changes. The single
nocheck is scripts/system/tls/trust/wsl2.sh:61, where the marker is matched
verbatim against what an earlier run wrote into the Windows hosts file --
deriving it would orphan that block instead of replacing it.
Collateral, not intent: touching roles/sys-ctl-hlth-csp/files/script.py brought
it under the comment lint, which found five pre-existing narration comments.
Three deleted, `fn[:-5] # strip ".conf"` replaced by fn.removesuffix(".conf"),
and the root-user comment rewritten in the sanctioned Exception form. Its 33
tests stay green.
tests/lint/repository/test_no_hardcoded_container_name.py (05f425ea5) is the
same family for INFINITO_CONTAINER and could move into spot/ later; this change
leaves it where it is.
make test-lint: 326 passed. The rule found 18 sites before the cleanup and none
after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The runner sent both streams to /dev/null and then classified by exit code, so
a failure produced one line -- "FAILED ansible-lint" -- with no code and no
message. A lint finding, a crash and a lock conflict were indistinguishable,
which is exactly the state a quality run left me in while ansible-lint was
clean on the host at the same commit.
Output now goes to a temp file, is discarded on success because ansible-lint is
verbose when it passes, and is written to stderr together with the exit code
when it fails. The accepted codes are untouched, so a passing run behaves
exactly as before. run_ruff already reports this way, which is why the TID252
error it hit was fixable on sight.
This changes no outcome. The failure that motivated it has not recurred, so the
new path is still unexercised against a real fault.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`cli build include` generated one tasks/groups/<group>-roles.yml per invokable category during setup, and the stage plays included those files. They were written by the in-container UID, which is why a dedicated clean-container-owned helper had to delete them from inside the container.
The stage plays now loop over lookup('group_roles', <group>) and include tasks/utils/setup/{group,role}.yml directly. ordered_roles() in utils/roles/order.py holds the run_after topological sort the generator used to perform at build time.
utils/roles/stage.py:_group_names() no longer globs the generated files; it reads get_invokable_paths(), the same categories.yml source those files were derived from.
Measured: handlers still flush after every role (ansible-core 2.20.2, three loop items, three separate handler runs), and resolving all 18 groups costs 0.60s cold against 1.4s for the generator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill starts the backup units itself, but the deploy arms their calendar
timers first, and a job that runs long enough crosses every backup window in
group_vars/all/07_schedule.yml. web-app-decidim ran 23:05-02:24 and hit exactly
that: the drill's own volume backup succeeded at 01:00:49, then the timer fired
a second, unattended run at 01:03:18 while step [8b/9] was lazily unmounting
/var/lib/infinito underneath it.
The failure is silent by construction. baudolo writes the dump in two
processes - 'pg_dump > x.tmp', then 'mv x.tmp x'. The redirect shell exits 0
because a shell never reports a failed writeback on a soft NFS mount, but the
unmount discarded that mount instance together with the unflushed file, so the
mv resolved through the fresh instance and got ENOENT. The unit died at
01:04:04, the same second the drill finished remounting, and 80 minutes later
the CLI test read 'failed' and aborted the job before running its own backup.
Disarming the timers makes the drill the only thing driving those units.
Stopping a timer does not stop its service, and both the drill and
roles/svc-bkp-volume-2-local/files/test/test.sh start their units explicitly,
so nothing under test loses coverage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The PR-close canceller had no concurrency group at all, so cancelling
depended entirely on scripts/github/cancel/pull_request_runs.sh, which
needs a runner before it can call the API. Runners were 20 minutes to 21
hours out, and run 31451265380 stayed alive that whole time.
The canceller now joins the group entry-pr-change-orchestrate.yml renders
for the same PR. GitHub clears the group when the run is created, before a
runner is assigned, so the orchestrator dies immediately and the script
becomes a backstop instead of the only path.
The group spells pull_request out literally rather than interpolating
github.event_name. This workflow is triggered by pull_request_target, so
the interpolated form would join the twin the orchestrator's own job
guards already skip on a same-repo PR and leave the working run untouched.
A fork PR runs both twins; the pull_request_target one is still left to
the script.
The branch-delete sweep filtered runs by head_branch alone and shot this
canceller mid-cancel. It now excludes the canceller's own path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The hourly sweep fought the rest of the CI control plane instead of
helping it: it preempted itself (17 of the last 29 runs concluded
cancelled 1-2s after their successor was created, for 8s of work behind
a 74-80min queue wait) and shipped MAX_ATTEMPTS=0, which the script
reads as no ceiling while its own README promised the opposite. The 24h
age cutoff could not bound it either, because every rerun refreshes the
run's updatedAt and so keeps it inside the window.
Removes the workflow, the two scripts it was the sole consumer of, its
unit test, and the CI_RERUN_* documentation that would otherwise point
at a deleted file. The workflow is disabled in both origin and fork so
no sweep fires before this lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The quiesce guard spelled the dev container as the literal
infinito_nexus_debian, so it only ever matched a checkout whose .env was
generated for debian without a project prefix. It missed the self-hosted
runner form infinito_<project>_nexus_<distro>, any custom.env pin, and any
.env generated for another distro — and it missed them silently, because a
non-matching grep just skips the block and leaves the competing onion
publisher running.
INFINITO_CONTAINER is the SPOT for that name: compose.yml uses it as
container_name, and scripts/meta/env/load.sh (already sourced at line 19)
exports it. Proven against the container running on the author's host: the
literal does not match it, the variable does.
The outer `docker ps | grep -qx` pre-check is gone with it — `docker exec`
on an absent container already exits non-zero with stderr suppressed, and
the `if` absorbs that even under `set -euo pipefail`.
The variable joins the line-21 assertion so a broken env fails by name
instead of dying inside a pipeline subshell, which would reproduce exactly
the silent-skip class this commit removes. compose.yml already requires the
same variable with `:?`, so this is not a new demand on the environment.
tests/lint/repository/test_no_hardcoded_container_name.py pins it: run
against the pre-fix file it flags all three lines, and it stays green on
the distro-agnostic spellings in stack_refresh.sh and worktree/up.sh.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
