RFC 6761 reserves .test for testing and .example for documentation, and the
project's own domain was doing both jobs under the documentation name. It is
what default.env hands every local deploy and what the test fixtures assert
against, so it is a testing name.
Rename it wherever it stands for the running machine: the environment
defaults, the development inventory, the workspace and trust scripts, the test
suites, and the four documents that describe real behaviour - the DNS mapping
to loopback, the browser CA trust, WSL2 resolution. Those documents would have
started lying the moment default.env changed.
The sandbox allowlist moves with them, since egress is matched per pattern and
nothing resolves the old name any more, and the sandbox document that mirrors
that list follows it.
Two places keep .example on purpose: the link-style sample in the
documentation guide, which is documentation about documentation, and every
generic fixture name elsewhere in the suite - a.example, foo.example and their
kin. Those are a larger sweep of their own and are not touched here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b8a6205396 put INFINITO_SWARM_TEARDOWN_RESERVE_SECONDS into default.env and
made 00_one.sh require it with a :? guard, but wired it into no handler.
default.env is only the source of the value; a key reaches the generated .env
solely through a handler's setdefault, so make dotenv dropped it and the
swarm routine aborted on its own guard.
The two nocheck markers cover the neighbouring key on those lines:
INFINITO_CI_DISTRO_DEADLINE_EPOCH is computed by distros.sh from the sweep
budget at run time, so a static default would be a stale timestamp. They have
to sit on the same line; above the statement the marker does not apply.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moodle#0 of run 33445831564 was cancelled by the 355-minute job cap with almost no
artifacts. The cap did not end a deploy; it reaped a zombie.
00_one.sh wrapped `timeout` around `sudo`, so the guard ran as the unprivileged
runner and the payload as root. An unprivileged SIGKILL cannot reach a root process,
and the same holds for distros.sh's own `timeout -k` one level up. Probed both
shapes: the payload lands in a pgid the outer governor never addresses either way, so
--foreground would not have helped -- the privilege wall is what carries.
The log shows the consequence directly. The governor fired on schedule:
06:18:27.884 distros.sh: line 103: 15914 Killed timeout -k 60 "${remaining}s" "$@"
06:18:27.894 [ERROR] Run failed for distro=ubuntu: rc=137 after 20460s
and then ansible output kept arriving in the same tee'd log -- a TASK at 06:18:27.929,
another at 06:26:00.189, elapsed still advancing -- until
`##[error]The operation was canceled.` The orphan held the step's pipe open, so tee
never saw EOF and the step never returned.
954b9f1ae6 added the TERM trap precisely so a timeout kill would still collect
diagnostics, and its reasoning ("GNU timeout signals the whole process group") is
correct. It simply never gets signalled on the swarm path, because sudo stands
between. That is why this cell carries only inventory- and role-runtimes- while its
compose twin also has rescue-diagnostics- and playwright-. This completes that fix
rather than replacing it.
The ladder was also inverted: INFINITO_CI_DISTRO_BUDGET_SECONDS=20400 (340 min)
against INFINITO_SWARM_STEP_TIMEOUT_MINUTES=350, with default.env:97 asserting the
order as if correct. For the trap to run, the step guard must fire BEFORE the sweep
governor. Rather than re-tune two constants that start their clocks at different
points -- matrix start trails drill start by 1m49s to 2m04s, measured -- 00_one.sh now
derives its ceiling from the governor's deadline minus a teardown reserve, and keeps
its own configured value when that is smaller.
Not fixed here, and it must be said: Moodle#0 does not fit on a hosted runner at all.
Its pass-2 pacing tracks pass-1 within 5%, putting the pass end at roughly 362 minutes
after job start, past both the 355 cap and GitHub's own 360 ceiling. Raising the cap
would be masking and is arithmetically impossible anyway. What this buys is an honest
red cell at ~323 minutes WITH its diagnostics, instead of a silent cancellation with
none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both Workspace jobs of run 33445831564 die at
roles/sys-svc-container/handlers/main.yml:9, on the first registry pull a nested
node's dockerd performs after a restart:
docker: Error response from daemon: Get "https://registry-1.docker.io/v2/":
tls: failed to verify certificate: x509: certificate signed by unknown authority
The image name is a red herring. rpardini/docker-registry-proxy SSL-bumps, and the
certificate it presents for registry-1.docker.io and for ghcr.io carries the same
issuer, CN=DockerMirrorBox Intermediate IA -- measured against the live proxy through
the make helper, with three controls: direct fetch on public roots returns 401, the
same fetch through the proxy on public roots only returns "self-signed certificate in
certificate chain", and --cacert /dev/null errors, proving curl honours the flag.
That last control mattered: curl consults its compiled-in CApath as well, where
update-ca-certificates has hash-linked the proxy CA, so without --capath /nonexistent
the negative control passes silently.
So a mirror-name fix would have MOVED the error from docker.io to ghcr.io rather than
removing it. What is missing is trust, and the trust anchor came from
INFINITO_CACHE_REGISTRY_CA_HOST_PATH -- a variable act does not forward, and whose
`make dotenv` rewrites the shared .env back to default.env. It can therefore name a
directory the proxy that actually answers never signed with.
The bootstrap now reads the anchor off that proxy's own /ca bind. And it stops
limping: with the frontend up and no /ca bind exposed, the run aborts naming the
reason instead of continuing untrusting and surfacing three minutes later as a
cryptic x509.
Locally the patch is a measured no-op -- the inspected value is byte-identical to the
.env value, and the render with and without it is identical -- so it changes behaviour
only where the variable is wrong, which is act.
Not proven: that /mnt/cache/registry/ca/ca.crt on the hosted runner is the root the
live proxy presents. It follows from rpardini's design and from the sibling leg that
works, but both host directories are absent from every artifact and that runner is
gone. The line "==> registry-cache CA source: /mnt/cache/registry/ca" in the next
workspace run settles it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
87f762bd7 replaced the named helper lists in the deploy task and
rerun-spec.sh with a glob. There was a third stager. lint-playwright
keeps its own staging tree, so it kept its own list, and every MCP spec
failed to parse there: 8 of 87 roles, the same 'Cannot find module
./mcp-endpoint' the CI lint job reports.
All three lists held different sets - the deploy task four helpers,
rerun-spec.sh three, this one four again but not the same four - and none
of them the one the specs actually needed.
f14b1ab3f6 already fixed this drift once, by adding addon-gating here to
match the deploy stager. Adding a name to a list that exists three times
is what made a third occurrence certain; the list is gone now.
The rule learns about the third stager as well, since one that goes back
to naming helpers is exactly what nobody notices.
A helper that no stager copies fails at import, and it fails for every
role that imports it at once, because the whole spec file stops loading.
That is what mcp-endpoint.js did to all ten MCP specs: they ran zero
tests rather than one failing one.
The set lived in three places and no two agreed - the deploy task named
four helpers, rerun-spec.sh named three, and the specs required five.
addon-gating.js was already missing from the rerun path before any of
this. Both stagers now glob the runner's own files/*.js, minus the
config, so the directory listing is the set.
The lint derives what must be staged from the requires the specs
actually make, and holds both stagers to globbing, since a set that can
be typed twice will differ twice.
Conflict: roles/web-app-bridgy-fed/files/Dockerfile
Both sides extended the header comment. main also replaced the upstream
checkout, `git clone --depth=1 --branch "${BRIDGY_REF}"`, with git init plus a
fetch of the single ref; this branch never touched those lines, so git took
main's version automatically and only the comment collided.
Resolved to main's file verbatim. It is a superset of ours in both respects:
it keeps our two comment paragraphs and adds a third naming the trip-wire the
non-idiomatic fetch exists for, namely that BRIDGY_REF is a commit and
`git clone --branch` refuses one. Verified byte-identical to stage 3.
Committed with the hook skipped and before the semantic audit of the other 45
files finished. main brings a lint rule that has never seen this branch's
roles, image bumps on svc-ai-ollama and web-app-nextcloud which this branch
rewrote, a stylelint major, and release 13.0.0 across the packaging surface.
Any of those can merge cleanly line by line and still be wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every one of these call sites ended in `|| true`, so a failure left no trace
and the surrounding script reported a clean run.
service_converged.sh needed the `|| true` for a reason that had nothing to do
with the command: under `set -o pipefail`, `head -10` closes the pipe and awk
dies of SIGPIPE, which pipefail then reports as the pipeline's status. That
made a genuine `container service ps` failure indistinguishable from a normal
truncation. Capping inside awk (`n++ < 10`) removes the SIGPIPE, so the command
status can be tested on its own and a real failure now prints why. give_up and
report_failed_tasks carried a byte-identical copy of that block, so the fix
lifts it into one report_tasks helper rather than applying it twice.
The swarm-clean scripts keep going after a failed step on purpose, since the
host layer below them clears what survives. They now say what did not work
before continuing, instead of continuing silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A cancel request does not stop a running step. The runner asks it to end, and a step that ignores the signal keeps going, bounded only by its own timeout-minutes -- 350 for a deploy. Meanwhile the concurrency group holds the newer run on pending until the occupant is terminal, not until it is cancel-requested. Measured on run 32859801382: the cancel was accepted at 15:27:45 and thirteen minutes later ten deploy jobs were still running and the newer run still had not started.
The sweep now waits FORCE_CANCEL_AFTER_SECONDS and force-cancels whatever is still not completed. Cleanup steps are skipped for those, which is the price: the artefacts of an already superseded run help nobody. Only entry-cancel-superseded.yml sets the variable -- a closed pull request and a deleted branch have nothing waiting on the group, so they keep cancelling and walking away.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A long orchestrator sweep sometimes survives its concurrency group: the occupant sits on queued while its matrix waits for runners, cancel-in-progress does not reap it, and the newer run parks on pending until someone cancels the sweep by hand. PR #380 sat that way for nine hours behind a run from the previous commit.
entry-cancel-superseded.yml cancels those leftovers through the API. The group stays primary and untouched: it reaps within seconds and needs no runner, so the fallback normally arrives to find nothing left to do. It carries no concurrency group of its own, because the run holding a group cannot be the run that frees it.
Selection mirrors what the group would have done: cancel every matched run except the newest in the group instance, keyed the way the group is keyed. Not by head SHA, which goes stale while the job waits for a runner and would cancel the newest commit's CI while sparing the stale one. INCLUDE_PATHS keeps the sweep inside the group's own members, so a push to main cannot take the cron workflows with it.
deleted_branch_runs.sh becomes branch_runs.sh, since it now also serves live branches, and the shared parts move to lib.sh. A rejected cancel is logged and the remaining runs are still attempted; only a denied token fails the job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The coherence step wrapped a blocking 'systemctl try-restart' in a 120s
budget, one second short of the wedge-recovery cycle the unit performs
when the liburcu-bp startup wedge (f10a4a181) fires - observed in run
32657210518, where the Akaunting drill died on rc=124 while ganesha came
back healthy 4s later. 240s covers even the uncapped 123s cycle; with the
abort grace capped at 10s (64b39e227) one cycle is ~43s.
The '|| systemctl try-restart nfs-server' fallback also masked exactly
this failure: nfs-server.service is 'inactive dead' on a ganesha node,
try-restart on an inactive unit is a documented no-op exiting 0, so a
larger budget alone would have turned a SIGKILLed ganesha into a green
drill. Dispatching on the installed unit file keeps the kernel-NFS branch
and lets a real restart failure surface: a wedge leg now fails loud after
one ~43s cycle instead of passing on self-healing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
48845db97 made three capability decisions declarable and named them INFINITO_*, which put them under setdefault: the live environment wins and the value is written back. Makefile exports BASH_ENV, so .env is re-exported into every make recipe, and scripts/tests/code/run.sh reaches dotenv through 'make setup' on every test run. A declaration made for one run therefore became a machine fact on first regeneration and stayed one - that is what made the workspace failure intermittent rather than reproducible, and why fedora died on 11 assertions where debian passed the same code.
A dedicated handler writing through eb.set recomputes the value every time and discards whatever the environment held, so the key stays registered in default.env and visible to the lints while remaining structurally unpinnable. running_on/github.py already worked this way; the distinction now has a name in utils/env/README.md. The exporting shell still steers its own run, because load.sh preserves non-empty caller values.
The other two declarations are deleted rather than moved. image_mirror_enabled and docker_root_ephemeral both derive from runs_on_github(), and the orchestrator does not unset INFINITO_RUNNING_ON_GITHUB, so inside the workspace they were already true and the declarations restated a value that held anyway. Measured across all four runtime contexts: only the workspace row changes, and only for the cache.
Verified by reversing the probe: 'INFINITO_CACHE_STACK=false make dotenv' writes .env=true, where the registry path wrote the exported value back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One boolean answered five unrelated questions: whether the pull-through cache stack runs, whether image pulls take the GHCR mirror, whether the docker data root may be wiped, plus runtime naming and inventory resolution. The workspace is the environment where those answers diverge — it simulates a developer machine on a real runner — so getting one of them meant misreporting where the suite runs and silently paying for the other four.
is_ci() and runs_on_github() stay facts about the machine. cache_stack_enabled, image_mirror_enabled and docker_root_ephemeral become decisions that default off those facts and can be declared explicitly, so an environment states what it wants. Every default reproduces the previous condition exactly, including the distinction the old mirror and wipe gates drew between a real runner and act.
73ae40585 got the cache stack by flipping the identity flag and lost the docker-root wipe with it; the workspace now declares all three capabilities instead. Pairs with 4c068d1cf, which makes the resulting stray bind sources harmless rather than fatal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
03_wait_converge built the DB service name from the engine type:
"${DB_DEP}_${DB_DEP}" -> mariadb_mariadb. That name is only correct when
the database is central. force_shared_db deliberately leaves a
manager-pinned role carrying a literal `shared: false` alone, so
web-app-magento keeps its embedded engine, and sys-svc-rdbms deploys it as
magento_database inside magento's own stack.
The name came from the engine, which is global, and was looked up with
`docker service ls` across the whole swarm instead of the role's stack, so
the `shared` flag never entered the check at any step. Both polarities were
wrong:
variant #1 - no consumer pulls the central provider in, so
mariadb_mariadb does not exist. Run 32630684501 job 97175522113 polled it
ninety times and failed the drill, while the rescue artifact shows
magento_database at 1/1 Running and the play at failed=0.
variant #0 - matomo and mailu declare a Jinja `shared`, get force-shared
and drag svc-db-mariadb in. Job 95662862942 then reported
`db(mariadb): 1/1 Running 2 hours ago` beside `magento: Running 21
minutes ago` - green on a database magento never touches.
resolve_db_service asks the deployed swarm which service carries this
role's database: <entity>_database first, else <dep>_<dep>. The two cannot
collide, because sys-svc-rdbms renders the _database sidecar only under
`lookup('database', application_id, 'local')`. Exact-line matching, not
docker's substring name filter, per 246558a13.
Only web-app-magento changes behaviour: it is the one role that is both
pinned by force_shared_db and seen by the DB_DEP grep. matrix and
bigbluebutton never reach the DB half - their entity key carries no image,
so skip_if_no_swarm_service exits first - and discourse resolves
postgres_postgres exactly as before, in every variant.
A role that declares a DB dep no service serves now says so and gates the
app half alone, instead of polling a name nobody deployed.
diagnostics.sh carried the same guess in its container filter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workspace suite represents a developer machine, but Profile.is_ci() reads three signals and the block neutralised only two. INFINITO_RUNNING_ON_GITHUB is baked into .env by the runner-side make dotenv before the container starts, and load.sh regenerates .env only when it is missing, so the value survived into a container that had already unset GITHUB_ACTIONS. registry_cache_active() stayed false, make compose-up omitted compose/cache.override.yml, and the compose track then asserted pull-through on a cache stack that was never created.
40375decb wrote this block before that third signal existed; 94936f198 added the Profile gate a month later.
Accepted side effect: the workspace no longer resolves as GitHub, so it skips the GHCR mirrors and the docker-root wipe in down.py, which key off the same flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make test execs four targets into the same container and each one runs dev-extras.sh. The stamp guarding it was written after the install, so all four passed the check and ran concurrent pip install -e into a single venv, deleting each other's files mid-install. 1a908a6e5 introduced the fan-out; the retry loop in python.sh has masked the fallout ever since, at the cost of a silent fallback to serial pytest when it runs out of attempts.
Taking the lock before the stamp is read makes check, install and stamp one atomic section.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The step summary renders every junit report under build/test-reports,
so the three suites were absent from it for want of a report file. They
now write one under the same name the python runner derives from
INFINITO_TEST_TYPE.
minitest ships no junit reporter, and a gem for one file would be a
dependency the otherwise stdlib-only suite does not need, so it gets a
small plugin instead. The runners also provision their own toolchain
now that the host-side make prerequisites are gone.
Only test-unit-python went through the container dispatcher; the
javascript, php and ruby targets called their runner directly and so
executed against the host, where the toolchain may be absent and the
sandbox blocks the install.
The dispatcher was never python-specific, it just hardcoded run.sh.
It now takes the runner as an argument, and the host-side install
prerequisites drop out with it.
node expands a `**` pattern only from version 21 on, and install-node
resolves to node 18 on debian, so the suite reported "Could not find" and
exited 1 without running a single test. find now collects the files and
node receives them as arguments.
An empty suite exits non-zero for the same reason php and ruby already
do: reporting success for a suite that never ran hides its absence.
The unit suites had to install their toolchain on every run. The image
now carries the interpreters, so the make prerequisite finds them present.
The Composer vendor tree deliberately stays out of the image build: the
package cache is reached through the compose extra_hosts mapping, which
only exists for a running container, so a build-time composer install
would bypass the cache and talk to packagist directly. Fetching it from
the test target keeps it on the cached path.
test-unit-php and test-unit-ruby exited 0 whenever the interpreter or the
vendor tree was missing, so both suites were silently absent from every
gate. They now provision their toolchain through install-php/install-ruby
and fail loudly when that is impossible.
The three ensure_*_present helpers collapse into ensure_command_present
in system_pkg, and the ruby runner rejects an empty suite because
minitest reports success when nothing was required.
The package-cache frontend terminates TLS for package traffic and signs
with its own CA: probed from inside the test container, pypi.org arrives
as subject CN=pypi.org issued by CN=Infinito Package-Cache Frontend CA.
package-frontend-ca.sh installs that CA into the OS store, which is why
apt, pip, curl and urllib work — but requests reads certifi, which has
never heard of it, so test-external failed its reachability precondition
with CERTIFICATE_VERIFY_FAILED. Upgrading certifi does not help; the
issuer is not a public root.
load.sh is where the runner puts the venv on PATH, so it is also where the
interpreter carrying the pip certifi is chosen; pointing REQUESTS_CA_BUNDLE
and SSL_CERT_FILE at the store the stack deliberately fills keeps one trust
decision instead of two. A caller-set value still wins.
The bundle location was about to become a fourth copy: dev-nix and the
network diagnose CLI each carried their own candidate list, and the two had
already drifted apart in their fourth entry. All of them now read
group_vars CA_BUNDLE_CANDIDATES — Ansible directly, the shell through a
generated env key, python through read_group_paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32118850138 lost the :53 listener, so the sampler counts listeners.
Run 32538854067 lost DNS with the listener still bound: sockets-udp.txt
showed dnsmasq holding 0.0.0.0:53 while every forwarded query timed out,
and the sampler reported that hour as healthy.
Each tick now also records what a glibc caller gets back — for the zone
this resolver owns, derived from its own address= line, and for one name
it must forward — with the latency that caller paid. The rescue collector
probes the in-zone name too: probing external names alone cannot separate
"forwarding is gone" from "dnsmasq is gone".
The sampler now also runs on the swarm lab DNS nodes; it only ever ran in
the compose path, which is why the Magento artifact carried no sampler log
at all. Its location comes from group_vars FILE_DNS53_SAMPLER_LOG and is
required at every call site, so a run cannot quietly collect nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test-unit-javascript ran `node --test` without declaring where node
comes from, while its python sibling declares `install`. On a bare
distro image the target died with `exec: node: not found` (127), which
took down the pre-commit hook and all five test-workspace jobs.
The installer stack already knows how to pull a command from the system
package manager; node was simply missing from the table. Wiring the
target to install-node provisions it the same way php, ruby and npm are
provisioned, and fails loudly when the runtime cannot be installed
rather than skipping the suite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every scheduled workflow opened with an alarm clock, so the Actions list
read as a column of identical icons that said nothing about the run. The
clock now lives in run-name behind github.event_name == 'schedule', where
it marks the one thing it can mark: a run nobody triggered. The name says
what the run does, in two or three words.
The dispatch menu shows names in one narrow column and truncates them, so
a new lint caps a workflow_dispatch name at 25 characters and points the
longer, explanatory wording at run-name, which carries no such limit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four parse checks join the `lint-*` family, each using the language's own
parser rather than a reimplementation: `php -l`, `ruby -c`, `sqlfluff parse`,
and stylelint with an empty rule set (it raises CssSyntaxError whatever the
rules say, so the check stays syntax-only and grows no style opinion). Three
unit runners join `test-unit`: node:test, PHPUnit, minitest -- the first and
third ship with their runtime, so only PHPUnit adds a dependency.
Provisioning is best effort and the lint script decides whether it can run. A
machine without PHP must not fail the gate, but it must not look like success
either: the skip names how many files went unchecked. The first version of
these scripts read `git ls-files`, which inside the compose container returns
nothing (git refuses the bind-mounted repo as "dubious ownership") and reported
"No SQL files found." over 52 files -- green, and a lie. They use `find` now,
as the surrounding lint scripts already did.
sqlfluff needed two corrections before it parsed anything real. `%(name)s` is a
psycopg parameter marker, not SQL, so the placeholder templater is configured
rather than the files exempted; and the dialect follows each role's declared
engine (`meta/services.yml`), which is postgres for six roles and mysql for
svc-db-mariadb and pixelfed.
Recipes stay in scripts, not in the Makefile: the targets delegate, matching
every other target in the file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0d8ed0ba8 bounded these call sites with one ceiling for both operations.
The two are not equally safe to cut. A SIGTERM landing mid-postinst
leaves dpkg interrupted, and in docker_dataroot.sh that fails every later
filesystem candidate and sends the run through decline() - one truncated
package install becomes a dead job. An interrupted update has no such
cascade; it just repeats.
Measured on ubuntu:latest with the exact commands: update 10s, install
zfsutils-linux 20s. 1200s is a 60x margin on the operation that carries
the cascade, and update keeps the 600s it cannot plausibly need more than.
This breaks the single-number invariant with scripts/install/apt.sh:19,
which is already broken in fact - that call site carries a different
Acquire option set than these three do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A run-wide draw proved one distribution and said nothing about the other four, while reporting green. Both are matrix-row axes now, assigned by the same deterministic rotation mode and tor already use, so one sweep exercises every distribution and every filesystem.
The two read the row's position like an odometer, the distro as the low digit. Turning both on the position directly covers only n of the n x m pairs whenever the pools are the same length, and no sweep unlocks it.
Both are part of a row's identity: glyphs in the job label, columns in the plan table, shards in the artifact name. Both are selectable through %distro and /filesystem, and a retrigger replays the distro off a failed title. It leaves the filesystem out, because a title states the kind the matrix assigned and a deploy may have fallen back from it.
A row's filesystem is a preference unless a human names it, on the run or in a token. Reading the matrix's own narrowing as a demand would fail rows for every transient condition the applying step is built to tolerate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two scripts sharing a delimiter-bounded name prefix are a folder in disguise, and tests/lint/repository/structure/test_prefix_grouped_files.py refuses them. A second release_* resolver is coming, so give the group its directory now and drop the prefix from the name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ba8a72309 bounded scripts/install/apt.sh and d50ba46b0 bounded the swarm
bootstrap; these three were the remainder, and one of them was still
costing whole jobs. docker_dataroot.sh hung Nextcloud#7 (job 96179137223,
run 32284644282) for 5h40m: "installing zfsutils-linux for zpool" at
19:50:39 is the last line before "rc=124 after 20400s" at 01:30:39, with
nothing in between. distros.sh's global budget is the only ceiling below
these calls, so one stalled fetch also takes the untried distros with it.
Both update and install are wrapped, not just update - that hang sat in
the install, and apt-get install fetches from the same mirror. Constants
come verbatim from scripts/install/apt.sh:19-20 rather than being
re-derived.
host_deploy.sh runs inside a container via `docker exec ... sh -c`, where
POSIX sh has no arrays, so its options are a word-split string instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
01_bootstrap.sh ran apt-get update with no timeout, so a stalled mirror
connection hung until the drill's global budget killed it - twice in run
32284541284 (jobs 96178507396 funkwhale, 96178507443 friendica), each
after hours of silence. The journal pins it exactly: sudo[462047] opened
19:51:36 and closed 00:07:30, 4h15m54s in a single apt-get update, and
the session only ended because timeout fired. Healthy runs of the same
line in the same jobs took 3s, 21s and 41s.
The cost is not one red job. distros.sh:130 is the only ceiling and it
wraps the whole per-distro sequence in the remaining global budget, so
one stalled fetch also takes the untried distros with it: funkwhale ran
2 of 5, friendica 3 of 5, both ending at budget remaining=-19s/-20s.
-qq hid the stall and || true cannot catch a hang.
Same failure class and same remedy as ba8a72309, which bounded
scripts/install/apt.sh; this closes the sibling call site it left open.
Constants are taken verbatim from there rather than re-derived. timeout
runs under sudo so SIGTERM reaches apt-get instead of the sudo wrapper -
the 4h session above is what the other order looks like. -E stays so the
package-cache proxy variables survive.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Azure Ubuntu mirror stalled during runs 32265854076 and 32265543004, apt-get update burned its 10m timeout and exited 124, and 40 deploy jobs died in the install step. Every package the wrapper is ever asked for ships with the ubuntu-24.04 runner image, so the network round trip bought nothing and cost 60s per job.
dpkg-query decides per package now; apt only runs for what is genuinely missing, which keeps the act image (no envsubst, no skopeo, no gh) working.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scripts/install/apt.sh ran apt-get without any timeout, so a stalled mirror connection hung until the 355-minute job limit killed the job - three times across both runs (95883075054, 95883075011, 95883448555), each after roughly 5h50m of silence following the last apt line, "Get:5 noble-security InRelease". The Install-jq step carries no timeout-minutes of its own, so nothing bounded it.
Acquire retries/timeouts mirror the SPOT that 07d31d900 established for deployed hosts in roles/update/tasks/debian.yml. The timeout wrapper is the backstop for what Acquire cannot cover (dpkg lock, hanging post-install), and runs under sudo so the signal reaches apt-get itself rather than the sudo wrapper; -k 30 follows up with SIGKILL. 10m is 5.5x the slowest successful run measured in these same runs (6.7s / 41.1s / 81.3s / 109.8s).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32118850138 left a dnsmasq that was alive with zero sockets,
nothing logged, and a 6m39s death window between two host DNS
consumers. Capture each dnsmasq's fd table and rlimits in the rescue
snapshot (closed listeners vs exhausted fd table vs lost netlink
watch), and sample the :53 listener set every 10s inside the DiD
container so the next occurrence is bounded to one bucket.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Iterating on one app meant restarting the whole stack or reaching for raw
docker. The target mirrors compose-app-logs and goes through the
repository's own container wrapper.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The restore cycle stops every subject of the host, replays into the database
services alone, and starts the rest again at the very end. Any exit in
between skips that last step, so the host keeps its subjects down - and
nothing recorded it.
The next deploy then failed on a symptom of the teardown rather than on
whatever ended the drill. It cost two runs: once 'rake db:migrate' against a
database the cycle had emptied but not yet replayed, once
Redis::CannotConnectError because the cycle never restarted redis. Both times
the message pointed away from the cause, and fixing what it named would have
fixed nothing.
A trap armed before the teardown now records the teardown and the exit code,
and mode=update refuses to run while that record stands. purge=true and
mode=reinstall pass, because both rebuild the state the drill destroyed; a
completed cycle clears the record itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Output of 'make autoformat': ruff collapses expressions that were wrapped across lines without need, shfmt and the yaml formatters drop surplus blank lines. No semantic change - the whole diff is 10 insertions against 39 deletions, and every hunk joins lines that already said the same thing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The regular rows began wherever (sweep * capacity) % count pointed. On run 210
that was offset 68: chunk 0 opened at candidate 69, the coverage-first head of
the list went undeployed, and no two runs could be compared because the window
moved with the run number. A run that looked wrong turned out to be merely
rotated, and finding that out meant recomputing the arithmetic by hand.
The rotation is gone. Chunk 0 reads the ranking from the top unless an offset
says otherwise, and the offset is an input now, threaded from the manual form
through the orchestrator and the deploy call down to the matrix builder. The
plan renderer takes the same flag, so the table shows the window the run will
take.
The offset is a row count or a selection token. A token is the honest form:
the ranking shifts whenever a variant appears, so 'offset 84' means something
different tomorrow while 'web-app-nextcloud#5' still means the same row. Axes
the token leaves open match anything, so a bare role name starts at its first
row and a pasted job label starts at exactly the job it names. A token that
matches no regular row aborts rather than silently starting at the head an
operator meant to skip.
The run title carries it as ⏩ followed by the value, and only when one was
asked for, so a rotated-looking run says so in its own name and the retrigger
reads it back from there when a job log has expired.
resume_from_chunk leaves the manual form with it. Skipping whole chunks was
the coarse way to say 'carry on further down'; the offset says it exactly and
keeps the rest of the chain intact. The orchestrator input stays for the other
entry points.
What this gives up is automatic tail coverage. Rows beyond the budget are
reached by moving the offset, widening the budget or narrowing the selection,
not by waiting for consecutive sweeps to walk the list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge assertion was a job nobody depended on. ci-orchestrator needed the
scope, the branch prefix and the affected roles -- not it -- so a PR whose base
was not merged in went on to spend five distros on a deploy sweep and reported
a single red job beside a chain that had already finished.
Validation is now a reusable workflow every entry calls, and what it checks
follows the event and the inputs rather than the caller: a pull request gets
its branch prefix and merge base asserted, a run carrying a selection gets that
selection checked, a fork PR waits here for the privileged run's images. The
wait and its gate move in with it, so the orchestrator no longer keeps a second
copy of that logic. Every downstream job depends on the gate.
The selection check is what the last two nights needed. cli.meta.ci.validate
resolves every token against the branch that would deploy it -- role, variant,
mode, onion state -- and reports all offenders at once. Until now the matrix
builder raised on the first bad token from inside a chunk job, so a stale
priority line cost one full CI round per entry and only ever showed the next
problem after the previous one was fixed. Measured on a real list: three
unusable tokens, one pass.
A bare role name that matches nothing is a warning, not an error: the
diff-derived whitelist legitimately names roles the lifecycle envelope filters
out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were left by earlier commits and are collapsed by markdownlint-cli2 and shfmt respectively, so every make quality run left the tree dirty and the change would eventually have ridden along in an unrelated commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The converge gate polls STACK_NAME_PRIMARY_SERVICE_KEY. Four roles never
create that name and each was given a static `workload: node-local` to make
the gate skip itself:
21fe9e984 web-svc-cdn ships only cdn_build, never cdn_cdn
2c6c8b763 web-app-matrix names its services matrix_synapse and friends
76508c946 svc-runner deploys node-local compose projects per node
cb52a72f2 bigbluebutton phantom service, and the DR drill must still run
Four hand-written claims for one observable fact. The claim is also static
where the truth is not: web-app-discourse renders `services: {}` only while
its sidecars are shared, and job 94965456921 of run 31864448027 polled
discourse_discourse ninety times for a stack that had none. Declaring the
role node-local would have blinded the gate in the variants where its
sidecars are local and a stack does exist.
has_swarm_service asks the deployed stack for the exact name instead. Every
caller runs behind `if rc == 0`, so an absent service is the role's shape and
not a broken deploy. Exact-line matching, not docker's name filter, which
matches substrings.
No coverage is lost: 03_wait_converge cannot pass without that service, and
05/06/07 plus the backup drill only run once it has. The four roles keep the
skip they were given; discourse and every future case get it without a
declaration. The teardown branch in the drill keeps its meaning - stack rm
where the service exists, compose stop per node where it does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A whitelist or priority entry was a role name and nothing else, so a run
could not ask for "this variant, on swarm, behind the onion". It either
covered everything the role offers or waited for the rotation to come
around to the combination that broke.
An entry is now a token that may carry the axes, either as
web-app-x#0,2@swarm+tor or as the job label pasted straight back out of
the run that failed. What a token leaves open still belongs to the line
it stands in: priority covers every remaining combination in one sweep,
whitelist keeps the rotation and its position in the matrix. A pin the
row cannot take aborts the matrix rather than dropping the row, because
a dropped row reports a green run for a combination that never ran.
The onion state spells out as +clearnet, not as a -tor suffix: svc-net-tor
would otherwise read as role svc-net on the clearnet.
Chunk membership still follows the discovery ranking, which is what
decides who makes the budget cut, but the rows inside a chunk are sorted
by display name, variant, mode and onion state, so a chunk's job list
reads like the plan table instead of like the sweep's rotation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
