The Must-Use plugins screen lists a mu-plugin under the Plugin Name its header
declares, and infinito-http-ca-trust.php has carried one since 117d246a3. The
assertion I wrote in 2f7cd543a looked for the filename, so it could never hold:
the plugin was loaded the whole time, and the gate discarded an hour-long run
over it.
The expected string is the one the failing run printed, not a second guess, and
the message now names where it comes from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The data-root task runs on all five swarm nodes at once, and they are
containers on one host kernel sharing its loop devices. losetup -f only names
the next free one; by the time the following losetup attached it, a sibling had
taken it - 'failed to set up loop device: Device or resource busy'. Claim with
--find --show, which picks and attaches in one call, and retry with a short
pause when a sibling wins anyway.
The step runs before anything is pulled, so on some nodes docker is not
installed yet and the closing 'systemctl start docker' died with rc=5, taking
the whole task with it under set -e. The two systemctl stop calls above it had
tolerated that all along. Start only when the unit exists, and report
applied-pending-docker instead of applied so the job summary stops claiming a
data root that no daemon has picked up yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
variant_bundle_limits.py next to variant_bundles.py formed the prefix group
'variant', which the structure lint reads as a folder in disguise. It stayed
green locally because that lint asks git ls-files: the new file was untracked
through every quality run and only became visible once committed. CI saw it
immediately - the sole failure of run 30675284663.
Both move to utils/github/variant/ as bundles.py and limits.py, and the test
mirrors along.
The BLE001 per-file ignore is removed rather than repointed. parse_size raises
humanfriendly.InvalidSize for a malformed string and TypeError only for a
non-string, and the value reaching it is always str - so the catch-all was
never needed. Measured, not assumed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mbake wraps the WORDPRESS_ADDONS_REQUIRED jinja chain, ruff collapses a path
that now fits one line and normalises the quotes of a stub literal. No value,
expression or string content changes; each line was read rather than trusted.
These files were committed during this session without autoformat having run
over them, so the next quality gate would have produced exactly this diff.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight inputs stack up in the form, and the two that decide which roles run sat
apart. Move priority next to whitelist, lead every description with an emoji so
the purpose lands before the first word, and cut the two longest down to what
the operator has to decide - the priority text spent 178 characters describing
the workflow's execution order rather than the input.
Defaults, types and option lists are untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three violations in test files written earlier today, all surfaced by the first
full quality run rather than by the targeted checks I had been running.
test_addons_vendored_files built its finding list with append in a loop.
test_env_handlers_registered walked the tree with rglob, bypassing the shared
project cache the lint exists to protect on a 250+ role tree.
test_resolve read two keys that resolve.sh writes per matrix entry into
GITHUB_ENV, which can never be .env entries; the second key now derives from the
first so a single line carries the marker - and stays short enough that ruff
format leaves it on the line the lint inspects.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The retrigger sent only whitelist and priority, so entry-manual.yml applied its
own distros default of 'debian'. A run that swept five distros came back as one,
and a failure that only reproduces on centos returned green with nothing fixed.
The dispatch inputs are not recoverable from the API - it answers inputs: null
for a finished workflow_dispatch - so they are read back from the run title.
The title format is not copied here: run_name.py parses the workflow's own
run-name declaration and derives the literals framing an input, and the tests
build their titles through the same helper instead of spelling one out.
Known gap: a run from another entry point carries an unrelated title and yields
no distros, leaving the retrigger on the workflow default. Job step names carry
the list for every entry point and arrive in the same gh call, so that is where
to look when it matters.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Registering variant_bundle/max_storage in ORDERED_HANDLERS was right - the key
had never reached a generated .env - but the handler imported its default from
utils.github.variant_bundles, and that module pulls humanfriendly and, through
utils.cache.applications, yaml. The .env generator imports every registered
handler and has to run on the bare bootstrap python, so one constant import
took it down.
Move both defaults into a stdlib-only module. This also ends a duplication the
handler layer had already been carrying: size.py spelled out '3' while
variant_bundles held the same 3 - the very thing max_storage tried to avoid by
importing, at the cost of the layer boundary.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rescue path was built from two workflow env interpolations, and neither
name exists in the job env block - both are written to $GITHUB_ENV at runtime.
An undefined expression expands to the empty string, so the path became
"" + "/" + "" = literally /. The steps that use it run under if: failure() and
if: always(), i.e. exactly when the writing step did not finish.
The grep cannot fail: default.env is tracked and carries the value. What made
it unreachable was sitting behind a python3 call in the same step, which aborts
under set -e before the redirect group runs. Split it out, the way the four
sibling workflows have always done it, and let the diagnostics step build its
own path with :? so a missing value is loud rather than a walk of /.
Verified under act with every variable forced empty: the old shape yields
DIR=[/] and a passing [ -d / ]; the new one keeps the paths across the failing
distro step and leaves INFINITO_DISTRO unset.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
find and du write what they already collected and only then return nonzero, so
the guards added in ce1c5b5fe were throwing away the evidence the index exists
to report. One vanished entry mid-walk blanked the whole listing: '0 file(s)',
no paths, on a tree that was mostly readable. The rescue tree is indexed while
a failed run is still tearing down, so that is the normal case, not the corner
one.
Stop resetting LISTING, fall back on SIZE only when du produced nothing, and
turn the early exit into a warning so the partial listing still prints.
The existing test could not see this: its stub find emitted nothing, so it only
covered a total failure. The new case stubs a find that prints rows and then
fails, and it fails against ce1c5b5fe.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Bash(git commit*) sits in ask and Bash(git push*) in deny, but both match on
the argv prefix: 'git --git-dir=<path> commit' and 'git --work-tree=<path>
push' start with neither, so they slipped straight through. The confirmation
dialog never appeared for a commit issued that way.
Deny both flags outright. Repo-local work has no need for them - the worktree
tooling covers the legitimate cases - and leaving them open makes every other
git rule advisory.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DEFAULT_OUTPUT is built from tempfile.gettempdir(), which honours TMPDIR, so
asserting it equals /tmp/infinito-runner-deploy.log contradicts the source it
tests. The assertion held only where TMPDIR happens to be unset and failed on
any runner, container or sandbox that sets it.
The line above it stays: that the parser default wires to DEFAULT_OUTPUT is
the behaviour worth pinning, and it holds wherever the temp dir points.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make lint-action was red repo-wide: a type:choice input may not carry an
unnamed option, and entry-manual.yml used the empty string to mean 'pick one
at random'. The lint gated nothing while it failed on every run.
Call the case 'auto', the spelling the workspace and instructions inputs in
the same file already use, and resolve it in resolve.sh - the one place that
decides random-versus-stated. The pass-through stays verbatim, so the six
downstream declarations keep seeing '' when a caller says nothing and 'auto'
when a human deliberately chose random. Both resolve the same way; the origin
stays distinguishable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rescue artifact is a tarball of whatever the collectors managed to grab, so
reading a failed run meant downloading it to find out which files exist at all.
Run rescue_index.sh right after each collection instead, so the job log names
the tree the artifact contains.
All four sites call it without a fallback, which is what the preceding commit's
exit-status contract is for. host_deploy.sh and one.sh hoist the rescue path
into a variable first - it is now needed twice, once for the collector's
INFINITO_RESCUE_DIAGNOSTICS_DIR and once for the index.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every caller runs this in a cleanup path: three shell teardown handlers and a
GitHub step under bash -e, none of them with a fallback. A nonzero rc therefore
aborts the caller's remaining teardown or adds a second red step on top of the
failure being rescued. Until now that could not happen only because each
internal failure path happened to be caught - a property that dies with the
next edit. A terminal 'trap exit 0' states the guarantee once and makes it
independent of the enumeration.
The failure is not hypothetical: a find that rejects -printf (BusyBox) makes
the old script exit 1, reproduced with a stub.
Collapses three walks of the same tree into one, and counts files with awk
rather than grep -c, which answers 1 on zero matches and takes the assignment
down with it under pipefail. A walk that fails now says so - it used to print
'0 file(s)', which reads exactly like an empty tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every unit test for a shell script sat flat in tests/unit/scripts/ with the
path encoded into the filename - test_worktree_lib.py, test_filesystem_resolve.py,
test_swarm_diagnostics_routing.py. Finding the test for a script meant guessing
which words of its path had made it into the name.
Mirror the layout instead, the way tests/unit/utils/ already does:
scripts/system/worktree/lib.sh is now tested by
tests/unit/scripts/system/worktree/test_lib.py. The name prefixes that stood in
for directories disappear with them.
Pure relocation - 0 insertions, 0 deletions. All eight moves are recorded as
renames, so git log --follow still reaches the history.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
build_env() iterates an explicit registry, so a handler module that nobody
imports never runs and its key is silently absent from the generated .env -
the consumer reads an empty value and behaves as if the feature were off.
Nothing errors, which is what makes the omission survive review.
The check parses the registry with ast rather than importing it, because one
handler pulls a third-party dependency at import time and a registry check
must not depend on the environment it describes.
It immediately found variant_bundle/max_storage.py unregistered, so
INFINITO_VARIANT_BUNDLE_MAX_STORAGE never reached any .env although
utils/github/variant_bundles.py reads it; registering it lifts the generated
file from 129 to 130 keys.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
container.py hardcoded /tmp/infinito-rescue-diagnostics, a second copy of
DIR_RESCUE_DIAGNOSTICS from group_vars/all/05_paths.yml. Move the value to
a generated env key so the paths SPOT stays single: a new handler seeds
INFINITO_RESCUE_LOCAL_DUMPS_DIR from read_group_path(), recurse() forwards
it into nested runtimes, and an unset key now prints why the in-play role
dumps are missing rather than silently collecting nothing.
Registers the handler in ORDERED_HANDLERS - the registry is explicit, not
discovered - and moves rescue_diagnostics_dir.py into the new rescue/
package so the two siblings satisfy the prefix-group lint.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The structure lint requires files sharing a name prefix to live in a
directory of that name. tasks/utils/install_plugin.yml and
tasks/utils/install_mu_plugin.yml violated that from the moment they
landed in 4d9bf6da3; move both into tasks/utils/install/ and repoint the
three includes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-matrix selection never picked anything but ext4. candidates()
intersected what every distribution in the entry supports, and with the list the
orchestrator passes - arch debian ubuntu fedora centos - that leaves ext4 alone:
centos stream has no btrfs, zfs is debian/ubuntu only. btrfs was reachable only
by stating it in the manual input.
The intersection is right for swarm and wrong for the other two. distros.sh
runs 00_one.sh per distribution and each round builds its own lab, so a swarm
pick must hold in every round's nodes. compose and host carve the data root on
the runner itself, whose OS is fixed; there DISTROS names the application images
and has nothing to do with the filesystem underneath docker.
A fourth argument says which, rather than parsing it out of the label, and
guards with `:?` so a forgotten call site fails loudly. The runner pool omits
zfs: the runner-image build installs neither the zfs userland nor
linux-modules-extra, where the Ubuntu azure kernel keeps zfs.ko, so a drawn zfs
would decline on every run. A stated one still overrides the pool and is marked
required, so `filesystem: zfs` fails loudly instead of quietly falling back.
The test executes the resolver rather than reading it, and pins the relation
that breaks silently: the keys it writes to GITHUB_ENV are the keys the three
workflows read. A typo on either side hands docker_dataroot.sh an empty string,
which reports "no filesystem stated" and exits 0 - the feature switched off
under a green run. Pool contents are policy and are not asserted; the per-distro
expectation is derived from the resolver's own answers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The healthy replica threw `StorageUnavailableException: LDAP connection has been
closed` while the other two were down for an unrelated reason. slapd sets no
idletimeout, so it never closes a connection itself; what cuts them is the swarm
path, where an idle flow through a service VIP loses its conntrack entry without
either side seeing a FIN. The pool then hands the dead connection to the next
caller.
Capping the wait was the obvious remedy and is not available: reading the class
files of the pinned keycloak-ldap-federation jar, the provider knows
connectionPooling, connectionTimeout and readTimeout, and passes only
com.sun.jndi.ldap.connect.pool and .pool.protocol to JNDI. There is no
connectionPoolingTimeout - setting one would have parked a dead key in the
component config. Disabling the pool removes the failure class instead of
shortening the wait on a connection that is already gone; with importEnabled the
directory is not on a hot path, so the extra bind per operation costs little.
The flag lives in meta/services.yml and is read without a fallback, so an
inventory can put the pool back and a missing key fails the deploy rather than
resolving to a guess.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four checks and one assertion asserted that a literal appears in a file:
`PATTERN="${1:?`, `UNIT_DUMPS="${…:?`, three strings inside two other files,
and a message the collector prints. None of them describes a relation - they
describe the spelling of code that is itself the source of truth, and each one
would have failed a legitimate rewording while proving nothing about behaviour.
What stays checks something the code cannot state alone: that every call site
of the trigger passes the directory (two counts of the same file compared
against each other, which is what found two call sites a grep for the filename
could not see), and that neither file carries its own copy of the rescue path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The collector carried a shell-side default for INFINITO_RESCUE_DIAGNOSTICS_DIR,
which the SPOT lint forbids because a second source drifts unnoticed - and this
one already had: the fallback spelled /tmp/rescue-diagnostics, the value of
INFINITO_RESCUE_DIAGNOSTICS_BASE, while the variable it stood in for resolves to
/tmp/infinito-rescue-diagnostics. Had it ever been used, the collector would have
written where no artifact collects.
A `:?` guard replaces it, matching the six the DR drill already carries. The
only caller sets the variable explicitly for both collectors, and _context.sh
sources the env layer regardless, so the guard should never fire.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nextcloud drill watched /dev/root go from 40G used to 124G used with 21G
free on a 145G disk shared by five nested nodes, the NFS export, both backup
repos and a 14.2GB pull. Four staging trees survived the whole run - grep for
`rm -rf` found nothing after the secrets leg - so the drill kept every copy it
made until the containers went away.
DR_RESTORE_STAGE, DR_VOL_STAGE, DR_SEC_STAGE and RESTORE_ROOT are released
after their last reader, each checked individually against the line that reads
it. All four sit on the success path: `set -e` aborts before them on any
failure, so a drill that fails still hands its evidence to the rescue
collector.
The device step now measures free space before it starts and refuses with both
numbers rather than running into ENOSPC inside a nested systemd unit. The
threshold is USB_SIZE_MB, which the script already computes, so no new constant
enters - and it states the real ledger: the drill holds the pulled tree twice,
once for the device sync and again while the restore root exists alongside the
image. It sits after 02_luks_device.sh because that is where the only stale-loop
reaper runs; a guard placed earlier would abort forever on a leaked loop device
and the reap that would clear it would never happen.
Reading free space is split from evaluating it. Under pipefail a df that
rejects --output fails the command substitution and set -e kills the drill
before the check can speak - which is how the guard managed to produce exactly
the silence it exists to prevent. Driving the body against six df behaviours
(roomy, tight, no header, option unsupported, empty, non-numeric) is what
surfaced that.
This does not address why the observed run failed. It died in step [4/9] with
21G still free, so this guard would not have fired on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The deploy died in the drill's second pass on `wp plugin install activitypub
--activate`: the first pass had installed it, the DR restore put the tree back,
and wp-cli refuses to unpack onto an occupied directory. `--force` would have
made that green by overwriting whatever the restore produced - on a branch about
NFS-backed volumes, the worst possible place to paper over a restore. The probe
keeps it red instead: `wp plugin is-installed` answers from `get_plugins()`
alone (verified in the pinned wp-cli phar: `Fetchers\Plugin::get()` iterates
`get_plugins()` and nothing else), so a tree WordPress cannot parse still
reaches the install and still fails on folder_exists.
Activation moves into its own step. Chained to `install --activate`, wp-cli sets
chained_command and skips the result check, so an activation failure returns 0
today.
The same install path now serves both call sites; the addon hook carried its own
copy, which is why one defect needed fixing twice.
mu-plugins follow the declaration the way discourse already does. The loop reads
`mechanism: mu_plugin` from meta/addons/ instead of spelling two filenames, which
is how `infinito-http-ca-trust` shipped undeclared while `infinito-oidc-rbac-mapper`
carried a `required: true` that nothing evaluated. Both install paths honour that
field now - a required addon aborts the play, an optional one warns and is
skipped - which is the first place in the repo to read it.
Three guards, because the loop replaced two guaranteed literals: an assert that
every required addon is claimed by some install path, a lint pairing each
vendored addon with its shipped file in both directions, and the spec the addon
contract demands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three of the six failures triaged today were explainable only after downloading
a rescue artifact, and two not even then. The job log carried "not converged:
docker_docker", a bare "rc=1", and "failed after 3 attempt(s)".
The convergence gate now prints the task rows of a service that will not
converge - but only those whose error field is filled. A stack coming up
normally stays silent, while the one that killed svc-db-postgres would have
shown `error=Head "https://ghcr.io/...": request canceled` in the job log
instead of hiding it in an artifact. awk rather than grep: grep exits 1 when
nothing matches, which under set -e would take the script with it.
The remote pull decodes its exit status. os.system returns rc << 8, so the
message said 5888 where rsync said 23, and every attempt now names its code.
The sleep after the final attempt is gone - five wasted minutes per run.
The matomo poller hands its last 40 lines to stderr on deadline, redacted at
the source, and the failing task interpolates them the way the compose flavor
already does. No no_log: MASK_CREDENTIALS_IN_LOGS is force-set anyway and would
have censored exactly the payload this exists to deliver.
The `-p warning` filter added to trigger_units.sh a few commits ago is removed.
systemd journals a unit's stdout and stderr at info, so that filter would have
swallowed the per-provider attribution the same commit introduced.
stack_ready.sh gets its first tests: replica comparison, the completed-oneshot
branch, and both halves of the new reporting - the error reaches the caller, the
error-free rows do not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three of the failures triaged today were a single unretried external fetch:
deb.debian.org went dark for 76 seconds on one node while three other distro
rounds of the same job built the same image; packagist answered 502 for two
metadata files while the identical build passed thirteen minutes later; and
ghcr.io timed out on a manifest HEAD for six minutes.
Both build tasks were the only network-bound tasks in their file without a
retry, and their in-shell fallback made a transient fault worse: it re-ran
immediately, with no wait, and threw away the cache it had just filled. The
ladder now waits 30 seconds between three attempts, and the cache-discarding
rebuild moved out into its own task that runs only once those are exhausted -
at which point the transient explanation is spent and a corrupt cache is the
next candidate. Three attempts rather than ten because ten would run 36270s
against a 21000s step cap.
svc-registry-docker is excluded from the registry sync - it cannot mirror
itself into its own registry - which left it the one role fetching from a public
registry through the swarm agent, single-shot, where every other role gets ten
tries. It now pre-pulls through a retried task, gated on the sync task having
been skipped so the exclusion keeps one source.
The accompanying check computes the ladder's worst-case wall clock from the
handlers and measures it against default.env and the workflow step cap rather
than restating the numbers, and it fails if an ignore_errors ever loses its
paired fallback - which would turn a failed build into a silent pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The summary named the filesystem that was picked, never the one the run got.
A host that declines - no root, missing tooling, no /dev/zfs - still deploys,
and nothing recorded on what, so a red run could not be reproduced.
docker_dataroot.sh now ends every path with a verdict that measures the data
root with `stat -f -c %T` and reports applied, declined, already-prepared,
unchanged or required-but-unavailable. Compose and host write it straight into
the step summary; swarm runs the script inside each node, where that variable
does not exist, so the playbook lifts the verdict line into the deploy log and
report.sh aggregates it afterwards.
`hostname` is resolved with a fallback: it is not installed by default on the
RedHat family, and under `set -euo pipefail` a bare command substitution would
have turned the decline path - which exits 0 - into a hard failure.
test-deploy-host.yml gains the `filesystem` input. single-node has been passing
it since the per-matrix selection landed, and a reusable workflow that receives
an undeclared input fails the whole run with a startup_failure whose only trace
is a banner on the run page. The single-node line was skipped in the run that
would have shown it. A lint now pairs every local `uses:` with the callee's
declared inputs, and also demands that required inputs without a default are
supplied.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill's remote pull exits 1 after 65 minutes and nothing says why. The
unit loops over two providers and only counts errors, so which one failed is
unrecoverable; the pull runs `rsync -abP`, whose --progress writes one journal
line per file - 77137 of them on a nextcloud pull - and the drill captured
`journalctl -n 60`. The entire diagnostic was therefore rsync filenames.
The unit now names the host it could not pull from. rsync keeps --partial and
trades --progress for --info=stats2, turning the flood into a closing balance.
trigger_units.sh writes the complete journal of a failed unit into the rescue
dump directory the artifact ships as local-dumps, and prints only the lines
from warning upwards.
That directory arrives as an argument instead of a second copy of the paths
SPOT: the drill reads INFINITO_RESCUE_DIAGNOSTICS_DIR once, which the env
handler derives from group_vars/all/05_paths.yml, and hands it to every call
site. A test pins that contract, and it earned its place immediately - it found
two call sites that a grep for the file name cannot see, and both were the
units this change exists for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A build that dies on "Temporary failure resolving deb.debian.org" leaves no
trace of why. The rescue snapshot carried journals, disks, containers and swarm
tasks, but nothing about name resolution, so the one question that decided the
case - whether the node's own dnsmasq answered at that second - could not be
settled from any artifact.
collect_host now also takes resolv.conf, daemon.json, the listening UDP and TCP
sockets, the NAT rules, and a resolve probe against the three hosts that killed
builds today. Each entry goes through the same best-effort capture as its
neighbours, so a node without ss or iptables-save records that fact rather than
disturbing the collection.
Because the collector copies itself into every runtime it finds, this lands per
level: runner, node, and the containers inside them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Of six CI failures triaged today, three were explainable only from downloaded
artifacts and two not even there: the job log carried "not converged:
docker_docker" or a bare "rc=1", while this collector drowned that same log in
compose trees, env listings and ganesha thread stacks.
diagnostics.sh switches its stdout per section with exec, so every topic lands
in its own file under the rescue directory and the log keeps one line. An
unwritable output directory degrades to stdout rather than killing the
collector - a failed mkdir would otherwise abort the script at the first
redirection, which the caller swallows with `|| true`, leaving no diagnostics
at all and no sign of it. For the same reason the collector says so when it
ends up having captured nothing.
00_one.sh hands the same directory to both collectors. The variable was scoped
to the python one, so the shell collector wrote where no artifact picks it up.
The capture grows by exactly what today's investigations went without: per-node
resolv.conf, daemon.json, listening sockets, NAT rules and a resolve probe
against the three hosts that killed builds, plus df and docker system df.
rescue_index.sh runs last and lists what was collected together with the note
that it is downloadable. The index is the one thing worth printing.
A test pins the routing against a stub docker: one file per section, no bleed
between them, a single log line, and the degrade path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test annotation summary rendered 595 warning rows inline, which pushed
everything below it off the job summary page. Each level now sits inside a
details block with its heading as the summary; errors keep `details open`, so a
failure is never one collapsed section away from being missed.
The blank line after the summary tag is load-bearing rather than cosmetic:
GitHub parses a Markdown table inside an HTML block only when a blank line
separates the two, and without it the page shows raw pipes.
utils/github/pytest_summary.py already renders its slowest-first table that way,
which is where the shape comes from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backup snapshot mode only engages on btrfs or zfs, so on CI - where the
data root is whatever the runner or the node image happens to provide - it
was never exercised at all. Nothing proved the snapshot path beyond unit
tests and a synthetic probe.
`entry-manual` gains a `filesystem` choice that reaches both deploy modes.
Empty means each matrix entry draws its own, so coverage spreads over runs
instead of pinning one path forever; the pick and its origin go to the step
summary, because reproducing a red run means re-stating it verbatim. The
draw is restricted to what every distribution of that entry can deliver -
centos stream has no btrfs at all, and zfs is a third-party repository
outside debian and ubuntu - so a run never dies on a packaging gap. A
stated filesystem skips that restriction: it is an instruction, and a host
that cannot deliver it fails loudly rather than deploying on something
else.
docker_dataroot.sh formats a sparse loop image, mounts it at /mnt/docker-fs
and carves the data root inside it, then points the daemon there through
daemon.json - merged, not overwritten, so the registry-cache entry
survives. It runs on the runner for compose and inside each node for swarm,
before anything is pulled, and installs the filesystem tooling it needs.
Two container facts it has to work around: a privileged container's /dev is
a fresh tmpfs, so neither /dev/loopN nor /dev/zfs exists there and both are
mknod'ed from /proc/devices before use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The launcher declined on every btrfs host for a reason no operator could
guess: `/var/lib/docker` is created by mkdir, so it is an ordinary
directory inside the enclosing subvolume, and `btrfs subvolume snapshot`
needs a subvolume root as its source. Documenting that as a manual
prerequisite would have left the feature switched on and never firing.
ensure_subject_subvolume.sh converts the data root on the deploy that
introduces the mode. The copy uses `cp --reflink=always`: both trees live
on the same btrfs, so extents are shared instead of duplicated and the
conversion costs seconds and no extra space no matter how large the data
root is - where an rsync copy would have meant hours of downtime and a
free-space requirement of 110%. The original tree is kept until the daemon
has answered on the converted root, so a failed copy, a mismatched
verification pass or a daemon that will not start all put it back and
start docker on it again.
The mode itself moves from group_vars into meta/services.yml as
`snapshot_mode`, read through `lookup('config', ...)`, so a single host can
override it from the inventory. Falsey spellings normalise to `never` and
anything else fails a deploy-time assert, because YAML reads a bare
off/no/false as a boolean and `--mode False` would kill the unit nightly.
btrfs-progs is installed wherever a btrfs mount exists, with centos opted
out: centos stream ships neither the kernel module nor the package, and
demanding it there would fail the package-availability test for the whole
RedHat family.
The pin moves to 3.2.2, which is not cosmetic: that release carves the
snapshot inside its subject, and the reaper here looks for leftovers in
exactly that place. Against 3.2.1 it would search beside the data root and
never find one.
Two drills run before anything else in test.sh: subvolume_probe.sh proves
the conversion carries every path, payload and symlink across and that the
result can actually be snapshotted; snapshot_probe.sh pins the launcher's
decision, including the refusals that keep a host on the live copy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ruff`'s B905 wants every `zip()` to say whether it tolerates operands of
different length. Here it never sees one: both operands are the `packed`
form of an address from the same network, so both are four bytes for IPv4
and sixteen for IPv6. `strict=True` states that, and turns a silently
truncated - and therefore too permissive - regex into an error should the
invariant ever break.
The three other files carry `ruff format` reflows the same run produced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The file list came from `find .`, so every path carried a leading './'.
shellcheck writes those paths verbatim into its `-f diff` output, and
`git apply` rejects a diff whose paths start with './' as "invalid path",
which aborts the whole autoformat run.
The autofix branch was therefore dead from the start and nobody noticed:
it only runs when shellcheck actually has something to correct, and until
now it never did. The first real finding turned a silent no-op into a red
`make quality`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In a linked worktree `.git` is a file naming that worktree's gitdir, not a
directory, so reading `PROJECT_ROOT/.git/config` raised NotADirectoryError.
`_git_config` catches OSError and returns an empty mapping, which left
`_owner_and_repository` without a remote and `mirror_image` returning None
for every image.
The consequence was silent: every probe addressed the upstream registry
directly, so all image work done from a worktree bypassed the GHCR mirror
and hit Docker Hub instead. The rate limits and handshake timeouts that
showed up as flaky `test-external` runs came from there.
`_config_path` follows the indirection: a `.git` directory keeps its own
config, a `.git` file is read for its gitdir and that dir's `commondir`
points at the shared config. Reads go through utils.cache.files, which the
uncached-read lint requires under utils/.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The launcher decides per run whether baudolo may capture the volumes from
a filesystem snapshot. That decision reads mountinfo, inode numbers and
`btrfs subvolume list`, so a unit test can only mock it - and the one
defect that shipped in this area (baudolo 3.2.0 dropping the trailing
separator) survived precisely because the production path was never
driven end to end.
snapshot_probe.sh builds a throwaway btrfs on a loop device under
mktemp -d, puts a stub on PATH in place of the docker daemon, and asserts
both halves of the contract: a snapshot is taken where it would be
faithful, and where it would not be, `auto` degrades to the live copy
while a stated kind fails the run. It covers the plain case, a volume
declaring its own backing store, a nested subvolume, a `.baudolo-*`
leftover of a killed run, and a plain directory as the data root. The
host's real data root is never touched and a trap releases the mount, the
loop device and the directory.
It runs first in test.sh, before the backup unit is even waited on,
because it depends on nothing a backup produces. Where root or the btrfs
tooling is missing it reports SKIP after checking that baudolo-snapshot
resolves on PATH and that `--mode never` hands argv through unchanged -
on such a host the launcher declines anyway, so there is nothing else to
assert.
The README documents all four drills with a diagram each, and how to run
the snapshot probe from a host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baudolo 3.2.1 can capture every volume from one atomic filesystem
snapshot, which removes both the container stops and the torn copy a
live rsync of a database produces. It states the snapshot kind rather
than probing it, so naming a kind the host cannot deliver costs a whole
backup run.
Deciding that at deploy time is not possible: the set of docker volumes
changes between deploys, and a volume created afterwards would sit in the
snapshot as an empty directory, so the copy would succeed while holding
nothing. The decision therefore moves into the run itself. ExecStart now
points at baudolo-snapshot, which probes the host, appends
--snapshot/--snapshot-subject when a snapshot will be faithful, and
becomes the baudolo process either way.
STORAGE_BACKUP_SNAPSHOT_MODE ships as `never`, so no host changes
behaviour until an operator opts in - baudolo's snapshot path has never
been driven end to end on a real deployment. `auto` probes and degrades
to today's live two-pass copy with the reason in the journal; a stated
`btrfs`/`zfs` skips only the fstype lookup and fails the unit when the
host cannot deliver, which is the point of stating it. The token is
`never` rather than `off` because YAML 1.1 reads a bare off/no/false as a
boolean, which would render `--mode False` and kill the unit on every
timer fire; falsey spellings are normalised and a deploy-time assert
rejects anything the launcher's choices would not accept.
The probe refuses unless the data root is a btrfs subvolume root whose
parent is writable and on the same filesystem with no nested subvolume
below it, and unless every docker volume uses the local driver, declares
no Options of its own, sits under the data root and has nothing mounted
inside. The volume rules read the declaration rather than the mount
table, because docker mounts an NFS-backed volume lazily and unmounts it
when the last container stops - such a volume is a separate filesystem
that a snapshot of the data root captures as an empty directory. `auto`
accepts btrfs only; zfs needs to be stated, since none of its premises
could be exercised here.
Before taking a snapshot the launcher removes the `.baudolo-*` leftovers
a killed run leaves beside the data root: baudolo removes its snapshot in
a finally, which a SIGKILL or a reboot skips, and a read-only snapshot of
the data root pins every block the live root later frees.
--hard-restart-projects is stripped when the snapshot flags are appended:
it exists for stacks whose database cannot be backed up hot, which is
what a snapshot removes.
The pin moves 3.1.3 -> 3.2.1. That version is not on PyPI yet; publish it
before deploying, or the pip task fails on every host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The OIDC --add-host derived the gateway by string surgery on the subnet,
replacing a trailing .0/<prefix> with .1. That is correct for every subnet this
repo declares, so this changes no behaviour: checked old against new across all
98 role subnets, zero differences.
What it buys is that the derivation is now reachable from tests, that an
unusable subnet fails at render time with the role and the value named instead
of reaching Docker as a malformed address, and that a subnet not ending in .0
gets its real gateway rather than being passed through unchanged - a /25 now
yields .129 instead of an invalid IP.
Not exercised: the --add-host path runs only in a matrix deploy with OIDC
enabled, which this change was not run against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm branch turns on clustered Infinispan but never constrains the JGroups
transport, so Keycloak falls back to SITE_LOCAL. Every keycloak task is attached
to four 192.168.x overlays plus the node-local docker_gwbridge, all of them
site-local, so SITE_LOCAL cannot disambiguate and each task publishes whichever
interface enumerates first.
In run 30586213932 two of the three replicas published a 172.18.x gwbridge
address into the jdbc-ping table. gwbridge is per-node and unreachable from a
peer, so those JOINs timed out: ISPN000196 "failed to recover cluster state after
the current node became the coordinator", 25 Infinispan TimeoutExceptions, 17 of
them through StateTransferLockImpl. The cluster split, one replica froze on an
old topology and answered HTTP 500 on the auth endpoint, and the Playwright
sign-out assertion saw "We are sorry... An internal server error has occurred"
instead of the login form. Eight tests failed across three roles, because
web-app-dashboard and web-app-prometheus authenticate against the same Keycloak.
The matcher comes from the subnet_address_regex lookup, so the role's declared
192.168.24.0/24 yields match-address:192\.168\.24\.\d+ - its own overlay address
matches, gwbridge and the sibling overlays do not. The line sits inside the
existing swarm guard, so compose mode is untouched.
This complements 72e275d41 rather than replacing it: that one makes a broken
replica detectable and evicts it from the VIP, this one keeps it from breaking.
Not verified by me: that Keycloak 26.7.0 spells the option
KC_CACHE_EMBEDDED_NETWORK_BIND_ADDRESS and evaluates match-address as a Java
regex. That is read from the upstream documentation, not executed, and it is the
one assumption this change rests on. If it is wrong the container fails to start,
loudly, on the first swarm run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Subnet arithmetic lived in Jinja expressions - octet slicing for a JGroups bind
matcher, a regex_replace for a gateway address. Neither is reachable from a
test, and neither says anything useful when it meets a shape it cannot handle.
utils/networks/address.py carries the computation without an Ansible dependency,
so tests/unit reaches it directly.
subnet_address_regex works for any prefix length, aligned or not. A prefix mask
leaves a contiguous run of low bits free, so per octet the addresses span exactly
network_address to broadcast_address, and the cross product of those four ranges
is the network. That makes the octet decomposition exact without a case analysis:
/24 yields 192\.168\.24\.\d+ and /23 yields 192\.168\.(24|25)\.\d+ from the same
three lines.
subnet_gateway returns the address Docker assigns as the gateway, and raises for
a network too small to hold one.
Both lookups follow compose_networks: no positional terms, application_id from
context, networks.local.subnet resolved internally, ValueError turned into an
AnsibleError naming the role and the reason. The lookup form rather than a filter
is deliberate - the caller states its intent and the plugin owns where the value
comes from, instead of repeating the config key at every call site.
Verified exhaustively rather than by sampling: over all 65536 addresses of a /16,
for eight subnets, regex match equals network membership - about 524000
comparisons. Boundaries checked for every prefix length /8 to /32 including the
addresses immediately outside, gateways checked against ipaddress for /8 to /31,
and a cross check that each network's gateway is matched by its own regex.
The regex uses \d+ for a fully free octet, so it is exact over valid IPv4
addresses rather than over arbitrary strings; JGroups only ever matches addresses
of local interfaces. The test module docstring records that boundary.
No consumer at this commit; the call sites follow.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tests/lint/filesystem/python/test_no_uncached_file_reads.py requires every read
under tests/, utils/, cli/, plugins/ and library/ to go through read_text from
utils.cache.files, because the same files are read many times across the suite.
3a77165d6 landed with two raw Path.read_text() calls and left that lint red; the
commit only ran the comment lint before landing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every pull_request run of this workflow has ended in startup_failure - ten of
ten across feature/swarm-nfs, feature/svc-net-tor and feature/ai-agent-employees,
including runs that predate the current work. No jobs, no logs, no check runs,
because run-graph validation rejects the workflow before any job is created:
entry-pull-request-change.yml (Line: 130, Col: 3): Error calling workflow
'ci-orchestrator.yml'. The workflow is requesting 'issues: write', but is only
allowed 'issues: none'.
ci-orchestrator.yml requests issues:write at workflow level (:69) and on its
report-main-failures job (:283). The caller declares an explicit permissions
block listing only actions/contents/packages/security-events, and an explicit
block pins every unlisted scope to none. A called workflow may request only a
subset of its caller's declared set, so the graph is invalid. Introduced by
8b58b67121, a three-caller fan-out where only two callers were patched.
The grant goes on the job rather than at workflow level. Job permissions REPLACE
the workflow block instead of merging with it, so the four existing scopes are
restated verbatim alongside the new one.
Do not move this block to workflow level. That block also governs the
pull_request_target half of this file, whose build-ci-images-fork and
images-mirror-missing-fork jobs check out fork-controlled refs with
allow_unsafe_checkout, one of them with secrets: inherit. Widening that token is
the only way this change becomes a security problem.
The scope itself carries no new capability on this path: report-main-failures is
gated on github.ref == refs/heads/main and is unreachable from a PR ref, forks
get a read-only token for pull_request regardless of the declared ceiling, and
same-repo PR authors already hold issue rights. The block already granted
actions:write and packages:write, both broader than issues:write.
Not provable locally - act does not evaluate nested permission escalation, so
this needs a push to confirm.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tcp healthcheck ends its probe with `cat <&3 | grep -q 'HTTP/1'`, which
matches the status LINE. A backend answering HTTP/1.1 500 therefore passes the
probe exactly like a 200. Swarm's only mechanism for taking a task out of the
VIP is the healthcheck, so a Keycloak replica serving 500s stays in rotation
indefinitely, and with replicas: 3 it catches roughly every third request.
That is what failed run 30586213932's keycloak job. The Playwright artifact's
error context shows the sign-out assertion waiting on the login form while the
page rendered "We are sorry... An internal server error has occurred" on
/realms/master/protocol/openid-connect/auth, and the rescue artifact's container
log shows why the replica was broken: ISPN000196, failed to recover cluster
state after becoming coordinator, followed by 25 Infinispan TimeoutExceptions,
17 of them through StateTransferLockImpl. Eight tests failed across three roles
because web-app-dashboard and web-app-prometheus authenticate against the same
Keycloak.
The status grading lands in a new template rather than in tcp.yml.j2, because
that one has fifteen other consumers whose probe paths were not audited. They
keep the behaviour they have; only Keycloak moves. The new probe reads the
status line with head -n1 and accepts 2xx/3xx only.
Keycloak also moves from health/live to health/ready: liveness answers "the JVM
is up", which a broken replica satisfies, while readiness answers "requests can
be served", which is the distinction the VIP needs.
Readiness returns 503 through schema migration, so the strict probe needs a
start period that outlives it. Measured across all five replica logs of that
run, Keycloak reaches "started" in 13.0-13.2s including migration, so 180s is
roughly thirteen times the observed boot. The deploy gate that waits for
health=healthy allows 600 retries at 2s, i.e. 1200s, so the longer start period
consumes at most an eighth of its budget.
Tool availability verified in quay.io/keycloak/keycloak:26.7.0 (digest
0f198be292568439d700cdbfb893e69a6009bb43a94a06a945b1d3d506c76b13): bash, cat,
grep and head are all present, so the probe's head -n1 resolves.
This fixes detection, not the clustering fault behind it. Why the Infinispan
cluster fails to form is still unknown; a replica that never clusters is now
pulled from the VIP instead of serving errors, but if no replica clusters there
is nothing healthy to route to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm gate asked whether the drill ran by grepping the job log for
"Matrix-deploy <app>: provision/deploy/e2e/verify per round" — a copy of a
GitHub Actions step display name. d9a4498b3 renamed that step to
"🚀 Drill <apps> per distro: <distros>" and did not touch roundtrip.sh, so at
19d56f07e the literal existed in exactly one place in the tree: the grep itself.
It could no longer match anything, and four of the five workspace legs of run
30586213932 false-failed on deploys that had passed — svc-db-postgres converged,
answered on 5432, completed its DR drill and reported "Job succeeded" three lines
above the failure.
This is the second time the same coupling drifted. 0ca2ed56c is titled "align
roundtrip swarm gate to the live workflow step name" and records that the
previous stale marker made every swarm leg false-fail on a passing deploy.
Nothing cross-checked the literal, so it rots silently after each rename.
The marker now comes from the drill itself, printed after 07_assert_state.sh.
Every step before it runs under set -euo pipefail and the EXIT trap tears down
without printing, so the line is unreachable unless the drill completed for that
app. APP_ID and INFINITO_DISTRO are :?-guarded at the top of the script, so the
marker cannot degrade into an empty string that matches everything. The gate
matches it with grep -qF and names what it checked instead of asserting "empty
matrix" — a diagnosis that was never true and sent this investigation into the
wrong layer.
The gate itself stays. 214672a94 added it because act exits 0 on a zero-leg
matrix, which silently false-passed every local swarm roundtrip.
tests/unit/scripts/test_roundtrip_marker.py pins emitter and gate to the same
literal, pins the marker behind the state assertion, and fails if the old step
name reappears — the cross-check whose absence allowed both drifts.
Verified against the failing run's log: 0 occurrences of the old literal, while
DR drill PASSED, ran=1 skipped=0 and "Job succeeded" prove the drill reached the
point where the new marker is emitted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baudolo 3.2.1 can capture every volume from one atomic filesystem
snapshot, which removes both the container stops and the torn copy a
live rsync of a database produces. It states the snapshot kind rather
than probing it, so naming a kind the host cannot deliver costs a whole
backup run.
Deciding that at deploy time is not possible: the set of docker volumes
changes between deploys, and a volume created afterwards would sit in the
snapshot as an empty directory, so the copy would succeed while holding
nothing. The decision therefore moves into the run itself. ExecStart now
points at baudolo-snapshot, which probes the host, appends
--snapshot/--snapshot-subject when a snapshot will be faithful, and
becomes the baudolo process either way.
STORAGE_BACKUP_SNAPSHOT_MODE ships as `never`, so no host changes
behaviour until an operator opts in - baudolo's snapshot path has never
been driven end to end on a real deployment. `auto` probes and degrades
to today's live two-pass copy with the reason in the journal; a stated
`btrfs`/`zfs` skips only the fstype lookup and fails the unit when the
host cannot deliver, which is the point of stating it. The token is
`never` rather than `off` because YAML 1.1 reads a bare off/no/false as a
boolean, which would render `--mode False` and kill the unit on every
timer fire; falsey spellings are normalised and a deploy-time assert
rejects anything the launcher's choices would not accept.
The probe refuses unless the data root is a btrfs subvolume root whose
parent is writable and on the same filesystem with no nested subvolume
below it, and unless every docker volume uses the local driver, declares
no Options of its own, sits under the data root and has nothing mounted
inside. The volume rules read the declaration rather than the mount
table, because docker mounts an NFS-backed volume lazily and unmounts it
when the last container stops - such a volume is a separate filesystem
that a snapshot of the data root captures as an empty directory. `auto`
accepts btrfs only; zfs needs to be stated, since none of its premises
could be exercised here.
Before taking a snapshot the launcher removes the `.baudolo-*` leftovers
a killed run leaves beside the data root: baudolo removes its snapshot in
a finally, which a SIGKILL or a reboot skips, and a read-only snapshot of
the data root pins every block the live root later frees.
--hard-restart-projects is stripped when the snapshot flags are appended:
it exists for stacks whose database cannot be backed up hot, which is
what a snapshot removes.
The pin moves 3.1.3 -> 3.2.1. That version is not on PyPI yet; publish it
before deploying, or the pip task fails on every host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role tracked a rolling main tag, so no deploy could state which
build it ran and the native MCP client could not be relied on. 0.11.0 is
the current stable release and is past the v0.6.31 that first ships the
client.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
