Seven include templates under sys-svc-container decided how a container was
probed, and they read their inputs from template-local `{% set %}` scope.
That scope leaks: a value set for one service silently reached every service
below it, which is how taiga's front and gateway ended up probing a port
declared for a different service, and how bluesky's frontend probed a PDS
endpoint.
Probes now live in utils/docker/healthcheck.py, one class per flavor behind a
PROBES registry, and each service declares its own:
healthcheck:
flavor: curl
path: health/ready
start_period: 20m
`port_key` picks another entry under ports.internal for the services that
expose their probe elsewhere (keycloak on management, nextcloud talk on
signaling). Without a flavor the entry must carry an explicit test argv, so a
missing declaration fails loudly instead of rendering a probe against nothing.
Rendering HEAD's templates against the new output confirms 49 of 56 call
sites are byte-identical; the remaining seven are covered by an Ansible run.
Three probes change on purpose: erpnext's backend gains a check it never had,
bluesky's frontend drops the leaked PDS path, and msmtp_curl moves from a
hardcoded localhost:80 to the service's own port on 127.0.0.1 like every
other flavor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repo's async convention is fire and forget: ASYNC_POLL is 0 whenever
ASYNC_ENABLED is true, and the swarm test lane forces it true through
extra vars. A task dispatched that way returns a stub, so a changed_when
or failed_when reading its own register evaluates against nothing and can
never report changed and never fail. Several tasks carried a
"not ASYNC_ENABLED and" prefix in front of those conditions, added to
avoid touching undefined rc and stdout, which turned the guard off
entirely rather than adapting it.
That is how the nextcloud plugin configuration stayed broken unseen: the
batch reported ok after 0.8s while its delegated commands kept running,
the next task read the half written configuration, and no failure in any
of them could surface.
Every async in the repo was decided individually rather than swept. Where
nothing consumes the result, the async stays and a one line rationale now
records what does not read it. Where a later task, a handler or a browser
does consume it, either the async goes or the result is collected.
Collecting is what the new reaper does. It waits for a whole batch, then
reports every broken job at once instead of aborting on the first, so one
run shows whether a single item or the whole set is broken. Deciding what
counts as broken lives in a filter plugin, because the distinction is
subtle enough to deserve tests: a module reports failure in msg, a command
in rc and stdout, an idempotent API answers "already exists" and means
success, and a job that cannot be read at all comes back marked finished
with no outcome, which would otherwise read as green.
Timing cost is roughly one to two minutes on a deploy measured in hours.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, each resolved by combining both sides rather than picking one.
compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.
daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.
The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add a blocker vocabulary so a role whose upstream MCP path this
deployment cannot use says why instead of reporting an absence. Without
it OpenProject reads as "upstream has no MCP", which is false: the server
ships, it is Enterprise-only. A blocker and a served surface are mutually
exclusive, so a role cannot claim both.
Add endpoint.key_credential and endpoint.suffix for servers that address
a session through the URL path. Baserow streams from
{base}/mcp/{key}/sse, and endpoint fields must stay literal, so the
composition belongs in the discovery layer. A server that declares a path
key without a resolvable value is dropped rather than handed to clients
half-built.
Carry the upstream provenance in the role block, so the audit derives it
instead of keeping a second copy.
Require an explicit mutating_tools_enabled: false for a privileged
auth_subject; the rule previously only fired when the key was present and
true, so omitting it slipped an administrator-scoped surface through.
Move the MCP filters into plugins/filter/mcp/ so they stop sharing a name
prefix with their siblings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The consumer count came from roles_with_service, which builds dashboard card
targets and therefore requires a single canonical domain. web-app-matrix
declares domains.canonical as a mapping (synapse, element) and was dropped;
in a deployment where it was the only consumer the count reached zero, the
server started with -volume.max=7, the unnamed default collection took all
seven slots on first write, and every S3 PUT for the matrix collection failed
with "No writable volumes and no free volumes left". Synapse answered 500 on
media upload and no avatar ever reached the bucket. web-app-openwebui carries
the same mapping shape and the same latent failure.
Consumers are now enumerated by the binding itself: a role is a consumer when
its merged config binds the engine and marks it shared, whatever the role is
called and however many domains it serves. The S3 identity file and the volume
budget read that one list, so the identities granted write access and the
slots budgeted for their collections cannot drift apart.
Selection reads the merged applications config rather than meta/services.yml,
because an operator may bind a store through the applications inventory
variable alone; that path is covered by a test so a later optimisation cannot
quietly reintroduce a naming or declaration assumption.
f956de889 made the slot ceiling generous, which removed the accidental brake the old autosizing had provided: while the ceiling was derived from free disk space, growth stopped long before the disk filled. With a fixed ceiling the only thing between growth and ENOSPC is the free-space guard, and its stock 1% is 300 MB on a 30 GB data root - less than the 1024 MB unit SeaweedFS allocates in, so a single growing volume can outrun it.
The floor is now expressed as whole allocation units rather than a percentage, which is meaningless without knowing the disk size. Two units: one for the growth in flight, one for concurrent writers. The size limit is stated alongside it so both come from one constant instead of the floor deriving from a value the image injects - verified that our flag wins, since -master.volumeSizeLimitMB=512 makes the master report 512 MB.
-volume.minFreeSpace replaces the deprecated -volume.minFreeSpacePercent, per the image's own help text. The rendered command was run against chrislusf/seaweedfs:4.39: master, volume server and filer all start.
Left open deliberately: web-app-seaweedfs declares min_storage: 304MB, which cannot hold with a 2 GiB floor. That value feeds the CI storage filter in scripts/meta/resolve/apps.sh, where 60GB is the budget, so nothing is dropped today - but the declaration understates what the role now needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
a3fcd3e7f stopped the entrypoint from autosizing the ceiling out of free disk space, but replaced it with a flat 64 that I adopted without deriving. Each bucket is a SeaweedFS collection and the master grows one in batches of seven - 'create 7 volume' in the artifact, copy_1 = 7 in the image's own master scaffold - so 64 covers nine collections. Twenty-five roles declare the service, and the tenth consumer would have hit 'No writable volumes and no free volumes left' again, just later and without the disk-space link.
The ceiling now comes from volume_slots(): one grow batch per bucket plus one for the default collection, with the consumer count read from roles_with_service. Note what that count is: the consumer gate gets whether seaweedfs is in group_names, not whether the consumer itself is installed, so the value is derived from the repository rather than from the host. It grows with the codebase, which is what the ceiling has to keep up with.
The collection count is a required parameter with no default. A silent fallback would be worse here than a loud failure: without the flag the stock entrypoint injects -volume.max=0 and the autosizing trap is back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role passed no -volume.max, so the stock entrypoint injects -volume.max=0, which means 'derive the ceiling from free disk space'. That derivation charges every non-full volume its full volumeSizeLimit: the master's default grow batch creates seven 1 GB volumes on the first write, reserving ~7.0 GB, and the manager's docker data root had 6.8 GB free. unclaimedSpaces goes non-positive, maxVolumeCount collapses to volumeCount, and the node reports zero free slots.
Those seven volumes belong to the default collection, so they can never serve collection 'nextcloud'. The master says it plainly - 'topo failed to pick 1 from 0 node candidates', 'create 7 volume, created 0' - and every S3 PUT returns 500 while seven empty volumes sit on disk. The volume server never left the topology: RestartCount 0, one unbroken log stream across the window.
That 500 is what starts the web-app-nextcloud 0 chain: occ maintenance:install fails eleven times, leaves a half-installed instance, and slot 1 then holds nextcloud-init-sync.lock for four minutes while every other replica dies on a blocking flock. Reproduced independently in variant 5 with collection 'peertube'; both compose jobs stayed green.
-volume.max makes isArgPassed true, so the entrypoint stops injecting the autosizing value. Slots become a fixed budget rather than a function of disk space; volumes stay sparse and unpreallocated, and -volume.minFreeSpacePercent still guards a genuinely full disk.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nfs(5) lists both 'nolock' and 'local_lock=' under "Options for NFS versions 2 and 3 only". NFSv4 carries locking in the protocol and has no NLM sideband to switch off, so the kernel accepts the token and reports local_lock=none. mount_opts had the condition inverted: the branch that can work sat on v3, the inert one on v4 - and svc-storage-nfs-server pins nfs_version: 4, so the effective locking configuration was always the dead branch.
The artifact proves it landed nowhere: the mount module ran with local_lock=flock in opts on every node, every other token from that same string is present in /proc/mounts (soft, timeo=50, retrans=3, _netdev), and only local_lock reads 'none'.
The false premise had spread. docker-entrypoint-suitecrm.sh justified its mkdir mutex with 'the swarm NFS mount forces local_lock=flock, so flock(2) never crosses nodes'; that comment is removed. The mkdir mutex stays correct for a different reason: against this ganesha export a non-blocking flock works while a blocking one returns EIO.
Nothing about the effective mount changes - the option was already being discarded. On an existing host the first run rewrites fstab and remounts /var/lib/infinito once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_any_bridge_partner_deployed matched a bridge against deployed role ids by equality or '-<bridge>' suffix. That reaches a partner whose entity carries the name (matrix -> web-app-matrix) but never one registered under a service key: 'sso' is provided by web-app-keycloak and 'ldap' by svc-db-openldap, neither of which ends in the bridge name. Those five declarations therefore evaluated to false in every run, so their addon flags were false everywhere and skipUnlessAddonEnabled skipped the specs in every variant - silently, because a gate that never opens looks exactly like a variant that does not apply.
The lookup now resolves a bridge through build_service_registry_from_applications first and keeps the entity-suffix rule as the fallback, so both spellings work. Existing bridge tests are unaffected: their applications stub yields an empty registry and takes the old path.
The new lint asserts that every bridges: value names a service some meta/services.yml declares. It fails on a typo or an invented name while staying agnostic about which run deploys the service, which is the lookup's question at runtime. Verified both ways: no findings against the real service set, thirty against an empty one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Exclusion was a property of the image: --images-no-backup-required resolves through volume_is_fully_ignored, which skips a volume only when every container using it is ignored. A container holding a derived tree beside state that must be kept could not express the exclusion at all.
matrix_mdad_docker is that case - the data root of a nested docker daemon, sharing one container with matrix_mdad_matrix and matrix_mdad_state, which carry the actual Matrix data. Copying it is not merely wasteful: overlay2 encodes a deletion both as a character-device whiteout and as a trusted.overlay.* xattr, and no rsync in the chain carries -X, so the copy cannot be restored faithfully. The whiteouts also cannot be written onto an overlayfs destination at all, which is what kept the web-app-matrix DR drill red for 58 minutes a round.
meta/volumes.yml gains a 'backup:' key, validated like 'nfs:'. The backup_volume lookup resolves every 'backup: false' entry to its pinned docker name and feeds baudolo's --volumes-no-backup-required. A lint requires the decision to be written down for any volume mounted at /var/lib/docker or /var/lib/containers, so the next docker-in-docker role cannot repeat this silently.
Requires backup-docker-to-local 3.3.0, which carries the flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add basic_auth to the MCP auth schema so Jenkins can declare its
HTTP-Basic endpoint through the contract instead of around it.
Introduce the client-side renderers: mcp_authorization builds the
Authorization header per scheme, mcp_tool_server_connections builds the
Open WebUI tool-server entries and drops servers whose scheme it cannot
present, and mcp_renderable_servers filters a server list, since Jinja
select() needs a test and the repo ships no test plugins.
Replace the inline MCP_DISCOVERED_SERVERS Jinja with a mcp_servers
lookup, and carry auth and username per entry so basic_auth consumers can
build their credential.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The runner now ships the execution modes as one JSON extra-var, so the old assertIn on MODE_CLEANUP=true no longer matches. Locate the last -e instead, parse its argument and compare the whole mapping: that catches a missing or extra mode, which the substring check could not, and it no longer passes when the text happens to appear inside an unrelated argument.
Drop the narration comments the file carried; the assertions state the same thing. Carry the autoformat wrap on the AUR retry test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A module that dies on an unhandled exception carries no msg: the AnsiballZ wrapper returns only failed and exception, whose event holds the sole account of what went wrong. The aggregation defaulted to the literal string package install failed and discarded it, so an AUR fetch that threw in run 30713666732 reported nothing at all and took a full investigation to place. Walk msg, exception.event.msg, exception, module_stderr, stderr instead, and tag every result with the module that produced it. This applies to apt, dnf, pacman and every other module the plugin drives, not only to the AUR.
The aur module fetches the RPC record and the snapshot tarball through bare open_url calls with no error handling, so one blip kills a deploy; distribution repositories need none of this because apt, dnf and pacman retry inside their own transport. Give the calls that reach an endpoint of their own a single shared policy. Only the AUR path is declared today; copr, ppa and build carry it against the day they are.
Split the result along its responsibilities so no file passes 250 lines: calls.py holds the vocabulary, sources.py one builder per acquisition path, plan.py only the dispatcher, outcome.py the reporting and facts.py the fact shapes. The action plugin is left with executing modules. Tests follow the code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An untyped inspect searches every docker object kind, and a role's overlay network carries the same entity name as its main container, so the call can resolve the network and report a missing State key instead of naming the container. That is how the openresty assert died on every swarm job of run 30709070050.
Extend container-exec-resolver to cover inspect, stripping raw markers and quoted spans first so the flag walker survives Go template format strings. Add container-inspect-type, scoped to roles/ because the collision follows the entity naming convention and the outer act namespace carries no role networks. Sweep 40 call sites; suppress the delegate_to-resolved CIDs and the compose co-location probes with a stated reason.
Move the five container rules into tests/lint/ansible/shell/container/ and drop the container_ prefix from their filenames. Register both markers in the suppression docs, where the resolver marker was missing entirely.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RUNNER_POOL and NODE_POOL were assertions. With a distro list of just debian
and ubuntu the node pool offered zfs, because the table records that those
distros ship the zfs userland - but the module comes from the runner's kernel,
which carries none. One run in three would then have drawn zfs and either
declined silently, testing ext4 while the summary claimed zfs, or hard-failed on
all five nodes with nothing actually broken.
Ask the kernel instead: /proc/filesystems for what is loaded, modinfo for what
could be. Both answer unprivileged, which resolve.sh needs since it runs without
sudo. The modinfo half carries weight rather than being belt-and-braces - xfs is
absent from /proc/filesystems on this host while modinfo finds the module, so
without it the pool would shrink to whatever happens to be loaded.
The distro table stays above it for the other question: which userland a
container can install. Two layers because they are two things, and only the
kernel one cannot be fixed by a package.
Every exclusion now names its reason on stderr and distinguishes a missing tool
from a missing capability; a filesystem used to drop out of the pool in silence.
For the full five-distro sweep the pool is unchanged: ext4 btrfs, measured
before and after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An opencloud snapshot carried 174 entries, and every line repeated the whole
path - the deepest ran about 150 characters, most of it the same prefix as the
line above. Print the basename at its depth instead, with a glyph per kind:
folder, json, journal or log, plain text.
The separator is swapped for \001 across the sort. Without it a directory and
its children come apart: '/' outranks '.', so a sibling file sorts between the
two and reads as though it lived inside. The swap also makes the order
independent of the locale, which was deciding it before.
The existing cut-short test looked for 'logs/container.log', which no longer
appears, so it now checks the basename; a fourth case pins the structure rather
than the wording - a child is indented deeper than its folder and siblings sit
level.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
variant_bundle_limits.py next to variant_bundles.py formed the prefix group
'variant', which the structure lint reads as a folder in disguise. It stayed
green locally because that lint asks git ls-files: the new file was untracked
through every quality run and only became visible once committed. CI saw it
immediately - the sole failure of run 30675284663.
Both move to utils/github/variant/ as bundles.py and limits.py, and the test
mirrors along.
The BLE001 per-file ignore is removed rather than repointed. parse_size raises
humanfriendly.InvalidSize for a malformed string and TypeError only for a
non-string, and the value reaching it is always str - so the catch-all was
never needed. Measured, not assumed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mbake wraps the WORDPRESS_ADDONS_REQUIRED jinja chain, ruff collapses a path
that now fits one line and normalises the quotes of a stub literal. No value,
expression or string content changes; each line was read rather than trusted.
These files were committed during this session without autoformat having run
over them, so the next quality gate would have produced exactly this diff.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three violations in test files written earlier today, all surfaced by the first
full quality run rather than by the targeted checks I had been running.
test_addons_vendored_files built its finding list with append in a loop.
test_env_handlers_registered walked the tree with rglob, bypassing the shared
project cache the lint exists to protect on a 250+ role tree.
test_resolve read two keys that resolve.sh writes per matrix entry into
GITHUB_ENV, which can never be .env entries; the second key now derives from the
first so a single line carries the marker - and stays short enough that ruff
format leaves it on the line the lint inspects.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The retrigger sent only whitelist and priority, so entry-manual.yml applied its
own distros default of 'debian'. A run that swept five distros came back as one,
and a failure that only reproduces on centos returned green with nothing fixed.
The dispatch inputs are not recoverable from the API - it answers inputs: null
for a finished workflow_dispatch - so they are read back from the run title.
The title format is not copied here: run_name.py parses the workflow's own
run-name declaration and derives the literals framing an input, and the tests
build their titles through the same helper instead of spelling one out.
Known gap: a run from another entry point carries an unrelated title and yields
no distros, leaving the retrigger on the workflow default. Job step names carry
the list for every entry point and arrive in the same gh call, so that is where
to look when it matters.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
find and du write what they already collected and only then return nonzero, so
the guards added in ce1c5b5fe were throwing away the evidence the index exists
to report. One vanished entry mid-walk blanked the whole listing: '0 file(s)',
no paths, on a tree that was mostly readable. The rescue tree is indexed while
a failed run is still tearing down, so that is the normal case, not the corner
one.
Stop resetting LISTING, fall back on SIZE only when du produced nothing, and
turn the early exit into a warning so the partial listing still prints.
The existing test could not see this: its stub find emitted nothing, so it only
covered a total failure. The new case stubs a find that prints rows and then
fails, and it fails against ce1c5b5fe.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DEFAULT_OUTPUT is built from tempfile.gettempdir(), which honours TMPDIR, so
asserting it equals /tmp/infinito-runner-deploy.log contradicts the source it
tests. The assertion held only where TMPDIR happens to be unset and failed on
any runner, container or sandbox that sets it.
The line above it stays: that the parser default wires to DEFAULT_OUTPUT is
the behaviour worth pinning, and it holds wherever the temp dir points.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every caller runs this in a cleanup path: three shell teardown handlers and a
GitHub step under bash -e, none of them with a fallback. A nonzero rc therefore
aborts the caller's remaining teardown or adds a second red step on top of the
failure being rescued. Until now that could not happen only because each
internal failure path happened to be caught - a property that dies with the
next edit. A terminal 'trap exit 0' states the guarantee once and makes it
independent of the enumeration.
The failure is not hypothetical: a find that rejects -printf (BusyBox) makes
the old script exit 1, reproduced with a stub.
Collapses three walks of the same tree into one, and counts files with awk
rather than grep -c, which answers 1 on zero matches and takes the assignment
down with it under pipefail. A walk that fails now says so - it used to print
'0 file(s)', which reads exactly like an empty tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every unit test for a shell script sat flat in tests/unit/scripts/ with the
path encoded into the filename - test_worktree_lib.py, test_filesystem_resolve.py,
test_swarm_diagnostics_routing.py. Finding the test for a script meant guessing
which words of its path had made it into the name.
Mirror the layout instead, the way tests/unit/utils/ already does:
scripts/system/worktree/lib.sh is now tested by
tests/unit/scripts/system/worktree/test_lib.py. The name prefixes that stood in
for directories disappear with them.
Pure relocation - 0 insertions, 0 deletions. All eight moves are recorded as
renames, so git log --follow still reaches the history.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
container.py hardcoded /tmp/infinito-rescue-diagnostics, a second copy of
DIR_RESCUE_DIAGNOSTICS from group_vars/all/05_paths.yml. Move the value to
a generated env key so the paths SPOT stays single: a new handler seeds
INFINITO_RESCUE_LOCAL_DUMPS_DIR from read_group_path(), recurse() forwards
it into nested runtimes, and an unset key now prints why the in-play role
dumps are missing rather than silently collecting nothing.
Registers the handler in ORDERED_HANDLERS - the registry is explicit, not
discovered - and moves rescue_diagnostics_dir.py into the new rescue/
package so the two siblings satisfy the prefix-group lint.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-matrix selection never picked anything but ext4. candidates()
intersected what every distribution in the entry supports, and with the list the
orchestrator passes - arch debian ubuntu fedora centos - that leaves ext4 alone:
centos stream has no btrfs, zfs is debian/ubuntu only. btrfs was reachable only
by stating it in the manual input.
The intersection is right for swarm and wrong for the other two. distros.sh
runs 00_one.sh per distribution and each round builds its own lab, so a swarm
pick must hold in every round's nodes. compose and host carve the data root on
the runner itself, whose OS is fixed; there DISTROS names the application images
and has nothing to do with the filesystem underneath docker.
A fourth argument says which, rather than parsing it out of the label, and
guards with `:?` so a forgotten call site fails loudly. The runner pool omits
zfs: the runner-image build installs neither the zfs userland nor
linux-modules-extra, where the Ubuntu azure kernel keeps zfs.ko, so a drawn zfs
would decline on every run. A stated one still overrides the pool and is marked
required, so `filesystem: zfs` fails loudly instead of quietly falling back.
The test executes the resolver rather than reading it, and pins the relation
that breaks silently: the keys it writes to GITHUB_ENV are the keys the three
workflows read. A typo on either side hands docker_dataroot.sh an empty string,
which reports "no filesystem stated" and exits 0 - the feature switched off
under a green run. Pool contents are policy and are not asserted; the per-distro
expectation is derived from the resolver's own answers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four checks and one assertion asserted that a literal appears in a file:
`PATTERN="${1:?`, `UNIT_DUMPS="${…:?`, three strings inside two other files,
and a message the collector prints. None of them describes a relation - they
describe the spelling of code that is itself the source of truth, and each one
would have failed a legitimate rewording while proving nothing about behaviour.
What stays checks something the code cannot state alone: that every call site
of the trigger passes the directory (two counts of the same file compared
against each other, which is what found two call sites a grep for the filename
could not see), and that neither file carries its own copy of the rescue path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three of the six failures triaged today were explainable only after downloading
a rescue artifact, and two not even then. The job log carried "not converged:
docker_docker", a bare "rc=1", and "failed after 3 attempt(s)".
The convergence gate now prints the task rows of a service that will not
converge - but only those whose error field is filled. A stack coming up
normally stays silent, while the one that killed svc-db-postgres would have
shown `error=Head "https://ghcr.io/...": request canceled` in the job log
instead of hiding it in an artifact. awk rather than grep: grep exits 1 when
nothing matches, which under set -e would take the script with it.
The remote pull decodes its exit status. os.system returns rc << 8, so the
message said 5888 where rsync said 23, and every attempt now names its code.
The sleep after the final attempt is gone - five wasted minutes per run.
The matomo poller hands its last 40 lines to stderr on deadline, redacted at
the source, and the failing task interpolates them the way the compose flavor
already does. No no_log: MASK_CREDENTIALS_IN_LOGS is force-set anyway and would
have censored exactly the payload this exists to deliver.
The `-p warning` filter added to trigger_units.sh a few commits ago is removed.
systemd journals a unit's stdout and stderr at info, so that filter would have
swallowed the per-provider attribution the same commit introduced.
stack_ready.sh gets its first tests: replica comparison, the completed-oneshot
branch, and both halves of the new reporting - the error reaches the caller, the
error-free rows do not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill's remote pull exits 1 after 65 minutes and nothing says why. The
unit loops over two providers and only counts errors, so which one failed is
unrecoverable; the pull runs `rsync -abP`, whose --progress writes one journal
line per file - 77137 of them on a nextcloud pull - and the drill captured
`journalctl -n 60`. The entire diagnostic was therefore rsync filenames.
The unit now names the host it could not pull from. rsync keeps --partial and
trades --progress for --info=stats2, turning the flood into a closing balance.
trigger_units.sh writes the complete journal of a failed unit into the rescue
dump directory the artifact ships as local-dumps, and prints only the lines
from warning upwards.
That directory arrives as an argument instead of a second copy of the paths
SPOT: the drill reads INFINITO_RESCUE_DIAGNOSTICS_DIR once, which the env
handler derives from group_vars/all/05_paths.yml, and hands it to every call
site. A test pins that contract, and it earned its place immediately - it found
two call sites that a grep for the file name cannot see, and both were the
units this change exists for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Of six CI failures triaged today, three were explainable only from downloaded
artifacts and two not even there: the job log carried "not converged:
docker_docker" or a bare "rc=1", while this collector drowned that same log in
compose trees, env listings and ganesha thread stacks.
diagnostics.sh switches its stdout per section with exec, so every topic lands
in its own file under the rescue directory and the log keeps one line. An
unwritable output directory degrades to stdout rather than killing the
collector - a failed mkdir would otherwise abort the script at the first
redirection, which the caller swallows with `|| true`, leaving no diagnostics
at all and no sign of it. For the same reason the collector says so when it
ends up having captured nothing.
00_one.sh hands the same directory to both collectors. The variable was scoped
to the python one, so the shell collector wrote where no artifact picks it up.
The capture grows by exactly what today's investigations went without: per-node
resolv.conf, daemon.json, listening sockets, NAT rules and a resolve probe
against the three hosts that killed builds, plus df and docker system df.
rescue_index.sh runs last and lists what was collected together with the note
that it is downloadable. The index is the one thing worth printing.
A test pins the routing against a stub docker: one file per section, no bleed
between them, a single log line, and the degrade path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test annotation summary rendered 595 warning rows inline, which pushed
everything below it off the job summary page. Each level now sits inside a
details block with its heading as the summary; errors keep `details open`, so a
failure is never one collapsed section away from being missed.
The blank line after the summary tag is load-bearing rather than cosmetic:
GitHub parses a Markdown table inside an HTML block only when a blank line
separates the two, and without it the page shows raw pipes.
utils/github/pytest_summary.py already renders its slowest-first table that way,
which is where the shape comes from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The launcher declined on every btrfs host for a reason no operator could
guess: `/var/lib/docker` is created by mkdir, so it is an ordinary
directory inside the enclosing subvolume, and `btrfs subvolume snapshot`
needs a subvolume root as its source. Documenting that as a manual
prerequisite would have left the feature switched on and never firing.
ensure_subject_subvolume.sh converts the data root on the deploy that
introduces the mode. The copy uses `cp --reflink=always`: both trees live
on the same btrfs, so extents are shared instead of duplicated and the
conversion costs seconds and no extra space no matter how large the data
root is - where an rsync copy would have meant hours of downtime and a
free-space requirement of 110%. The original tree is kept until the daemon
has answered on the converted root, so a failed copy, a mismatched
verification pass or a daemon that will not start all put it back and
start docker on it again.
The mode itself moves from group_vars into meta/services.yml as
`snapshot_mode`, read through `lookup('config', ...)`, so a single host can
override it from the inventory. Falsey spellings normalise to `never` and
anything else fails a deploy-time assert, because YAML reads a bare
off/no/false as a boolean and `--mode False` would kill the unit nightly.
btrfs-progs is installed wherever a btrfs mount exists, with centos opted
out: centos stream ships neither the kernel module nor the package, and
demanding it there would fail the package-availability test for the whole
RedHat family.
The pin moves to 3.2.2, which is not cosmetic: that release carves the
snapshot inside its subject, and the reaper here looks for leftovers in
exactly that place. Against 3.2.1 it would search beside the data root and
never find one.
Two drills run before anything else in test.sh: subvolume_probe.sh proves
the conversion carries every path, payload and symlink across and that the
result can actually be snapshotted; snapshot_probe.sh pins the launcher's
decision, including the refusals that keep a host on the live copy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ruff`'s B905 wants every `zip()` to say whether it tolerates operands of
different length. Here it never sees one: both operands are the `packed`
form of an address from the same network, so both are four bytes for IPv4
and sixteen for IPv6. `strict=True` states that, and turns a silently
truncated - and therefore too permissive - regex into an error should the
invariant ever break.
The three other files carry `ruff format` reflows the same run produced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baudolo 3.2.1 can capture every volume from one atomic filesystem
snapshot, which removes both the container stops and the torn copy a
live rsync of a database produces. It states the snapshot kind rather
than probing it, so naming a kind the host cannot deliver costs a whole
backup run.
Deciding that at deploy time is not possible: the set of docker volumes
changes between deploys, and a volume created afterwards would sit in the
snapshot as an empty directory, so the copy would succeed while holding
nothing. The decision therefore moves into the run itself. ExecStart now
points at baudolo-snapshot, which probes the host, appends
--snapshot/--snapshot-subject when a snapshot will be faithful, and
becomes the baudolo process either way.
STORAGE_BACKUP_SNAPSHOT_MODE ships as `never`, so no host changes
behaviour until an operator opts in - baudolo's snapshot path has never
been driven end to end on a real deployment. `auto` probes and degrades
to today's live two-pass copy with the reason in the journal; a stated
`btrfs`/`zfs` skips only the fstype lookup and fails the unit when the
host cannot deliver, which is the point of stating it. The token is
`never` rather than `off` because YAML 1.1 reads a bare off/no/false as a
boolean, which would render `--mode False` and kill the unit on every
timer fire; falsey spellings are normalised and a deploy-time assert
rejects anything the launcher's choices would not accept.
The probe refuses unless the data root is a btrfs subvolume root whose
parent is writable and on the same filesystem with no nested subvolume
below it, and unless every docker volume uses the local driver, declares
no Options of its own, sits under the data root and has nothing mounted
inside. The volume rules read the declaration rather than the mount
table, because docker mounts an NFS-backed volume lazily and unmounts it
when the last container stops - such a volume is a separate filesystem
that a snapshot of the data root captures as an empty directory. `auto`
accepts btrfs only; zfs needs to be stated, since none of its premises
could be exercised here.
Before taking a snapshot the launcher removes the `.baudolo-*` leftovers
a killed run leaves beside the data root: baudolo removes its snapshot in
a finally, which a SIGKILL or a reboot skips, and a read-only snapshot of
the data root pins every block the live root later frees.
--hard-restart-projects is stripped when the snapshot flags are appended:
it exists for stacks whose database cannot be backed up hot, which is
what a snapshot removes.
The pin moves 3.1.3 -> 3.2.1. That version is not on PyPI yet; publish it
before deploying, or the pip task fails on every host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Subnet arithmetic lived in Jinja expressions - octet slicing for a JGroups bind
matcher, a regex_replace for a gateway address. Neither is reachable from a
test, and neither says anything useful when it meets a shape it cannot handle.
utils/networks/address.py carries the computation without an Ansible dependency,
so tests/unit reaches it directly.
subnet_address_regex works for any prefix length, aligned or not. A prefix mask
leaves a contiguous run of low bits free, so per octet the addresses span exactly
network_address to broadcast_address, and the cross product of those four ranges
is the network. That makes the octet decomposition exact without a case analysis:
/24 yields 192\.168\.24\.\d+ and /23 yields 192\.168\.(24|25)\.\d+ from the same
three lines.
subnet_gateway returns the address Docker assigns as the gateway, and raises for
a network too small to hold one.
Both lookups follow compose_networks: no positional terms, application_id from
context, networks.local.subnet resolved internally, ValueError turned into an
AnsibleError naming the role and the reason. The lookup form rather than a filter
is deliberate - the caller states its intent and the plugin owns where the value
comes from, instead of repeating the config key at every call site.
Verified exhaustively rather than by sampling: over all 65536 addresses of a /16,
for eight subnets, regex match equals network membership - about 524000
comparisons. Boundaries checked for every prefix length /8 to /32 including the
addresses immediately outside, gateways checked against ipaddress for /8 to /31,
and a cross check that each network's gateway is matched by its own regex.
The regex uses \d+ for a fully free octet, so it is exact over valid IPv4
addresses rather than over arbitrary strings; JGroups only ever matches addresses
of local interfaces. The test module docstring records that boundary.
No consumer at this commit; the call sites follow.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tests/lint/filesystem/python/test_no_uncached_file_reads.py requires every read
under tests/, utils/, cli/, plugins/ and library/ to go through read_text from
utils.cache.files, because the same files are read many times across the suite.
3a77165d6 landed with two raw Path.read_text() calls and left that lint red; the
commit only ran the comment lint before landing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm gate asked whether the drill ran by grepping the job log for
"Matrix-deploy <app>: provision/deploy/e2e/verify per round" — a copy of a
GitHub Actions step display name. d9a4498b3 renamed that step to
"🚀 Drill <apps> per distro: <distros>" and did not touch roundtrip.sh, so at
19d56f07e the literal existed in exactly one place in the tree: the grep itself.
It could no longer match anything, and four of the five workspace legs of run
30586213932 false-failed on deploys that had passed — svc-db-postgres converged,
answered on 5432, completed its DR drill and reported "Job succeeded" three lines
above the failure.
This is the second time the same coupling drifted. 0ca2ed56c is titled "align
roundtrip swarm gate to the live workflow step name" and records that the
previous stale marker made every swarm leg false-fail on a passing deploy.
Nothing cross-checked the literal, so it rots silently after each rename.
The marker now comes from the drill itself, printed after 07_assert_state.sh.
Every step before it runs under set -euo pipefail and the EXIT trap tears down
without printing, so the line is unreachable unless the drill completed for that
app. APP_ID and INFINITO_DISTRO are :?-guarded at the top of the script, so the
marker cannot degrade into an empty string that matches everything. The gate
matches it with grep -qF and names what it checked instead of asserting "empty
matrix" — a diagnosis that was never true and sent this investigation into the
wrong layer.
The gate itself stays. 214672a94 added it because act exits 0 on a zero-leg
matrix, which silently false-passed every local swarm roundtrip.
tests/unit/scripts/test_roundtrip_marker.py pins emitter and gate to the same
literal, pins the marker behind the state assertion, and fails if the old step
name reappears — the cross-check whose absence allowed both drifts.
Verified against the failing run's log: 0 occurrences of the old literal, while
DR drill PASSED, ran=1 skipped=0 and "Job succeeded" prove the drill reached the
point where the new marker is emitted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baudolo 3.2.1 can capture every volume from one atomic filesystem
snapshot, which removes both the container stops and the torn copy a
live rsync of a database produces. It states the snapshot kind rather
than probing it, so naming a kind the host cannot deliver costs a whole
backup run.
Deciding that at deploy time is not possible: the set of docker volumes
changes between deploys, and a volume created afterwards would sit in the
snapshot as an empty directory, so the copy would succeed while holding
nothing. The decision therefore moves into the run itself. ExecStart now
points at baudolo-snapshot, which probes the host, appends
--snapshot/--snapshot-subject when a snapshot will be faithful, and
becomes the baudolo process either way.
STORAGE_BACKUP_SNAPSHOT_MODE ships as `never`, so no host changes
behaviour until an operator opts in - baudolo's snapshot path has never
been driven end to end on a real deployment. `auto` probes and degrades
to today's live two-pass copy with the reason in the journal; a stated
`btrfs`/`zfs` skips only the fstype lookup and fails the unit when the
host cannot deliver, which is the point of stating it. The token is
`never` rather than `off` because YAML 1.1 reads a bare off/no/false as a
boolean, which would render `--mode False` and kill the unit on every
timer fire; falsey spellings are normalised and a deploy-time assert
rejects anything the launcher's choices would not accept.
The probe refuses unless the data root is a btrfs subvolume root whose
parent is writable and on the same filesystem with no nested subvolume
below it, and unless every docker volume uses the local driver, declares
no Options of its own, sits under the data root and has nothing mounted
inside. The volume rules read the declaration rather than the mount
table, because docker mounts an NFS-backed volume lazily and unmounts it
when the last container stops - such a volume is a separate filesystem
that a snapshot of the data root captures as an empty directory. `auto`
accepts btrfs only; zfs needs to be stated, since none of its premises
could be exercised here.
Before taking a snapshot the launcher removes the `.baudolo-*` leftovers
a killed run leaves beside the data root: baudolo removes its snapshot in
a finally, which a SIGKILL or a reboot skips, and a read-only snapshot of
the data root pins every block the live root later frees.
--hard-restart-projects is stripped when the snapshot flags are appended:
it exists for stacks whose database cannot be backed up hot, which is
what a snapshot removes.
The pin moves 3.1.3 -> 3.2.1. That version is not on PyPI yet; publish it
before deploying, or the pip task fails on every host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The compose and host deploy jobs tee the whole deploy stdout into the file
ANSIBLE_LOG_PATH names, and both downstream steps parse that stream. Only
stdout carries the host-side "matrix-deploy: round N/M ... PASS n" markers,
so it is the sole source the per-variant runtime tables can come from.
dedicated.sh's cleanup then copied the container-side ansible log onto that
same path. docker cp unlinks and recreates the destination, so tee kept
writing into an orphaned inode and the visible file became ansible's own
log_path output — whose "<ts> p= u= n= LEVEL| " line prefix defeats the
anchored recap matchers in logparse.py while the unanchored has_sections
substring test still fires. in_roles_recap never flips, every role-time line
is skipped, and the step dies with "no profile_roles entries found" although
the deploy succeeded. Run 30578244959: 14 of 14 compose+host jobs.
Why it looked like a flake: docker cp cannot read a /tmp that is a tmpfs
mounted inside the container, and the Debian image enables systemd's
tmp.mount while CentOS Stream 10 does not. The copy silently failed on
debian (host file survived, jobs green) and landed on centos (host file
destroyed). feature/swarm-nfs never had the block; it returned here through
merge drift from main, where the copy is still load-bearing because those
workflows do not tee.
The copy is dropped: stdout is a strict superset of what it provided, and
the in-container verifier reads the log inside the container. Alongside it:
the swarm workflow's file-exists guard is adopted so an aborted deploy does
not paint the summary step red on top; the log_path prefix moves to
utils/ansible_log.py and is applied in logparse.py, so an ansible run log
parses instead of yielding an empty summary; a lint pins that nothing writes
onto ANSIBLE_LOG_PATH again; and the parser regression test now covers the
recap headers, not just the role rows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker info reports DockerRootDir as the DAEMON sees it, and
host_storage_constrained fed that string straight into shutil.disk_usage,
which statvfs()es it in the CALLING process's mount namespace. The two
coincide only when the caller is the daemon's host. In the nested act
container of the workspace jobs they do not, so os.statvfs('/var/lib/docker')
raised FileNotFoundError, matrix.py died, and because 00_one.sh invokes it
bare under set -euo pipefail the whole swarm drill went down with it. All
five workspace jobs of run 30560683608 failed this way, byte-identically,
across five base images and five nested distros.
docker_root_free_bytes() now takes the vantage point as a required keyword:
it measures the daemon's data root when that path resolves here, and the
caller's stated local path when it does not. matrix.py passes "/", which is
the same vantage it already trusts for its ENOSPC abort 270 lines earlier;
overlayfs passes statfs through to the upper filesystem, so a container of a
daemon measures that daemon's data-root filesystem through its own rootfs.
Nothing is caught and nothing is guessed. An unreachable daemon still raises
RuntimeError out of docker_data_root(), which is the guard this module exists
for, and local_vantage carries no default because only the caller knows where
it is running.
The compose-side twin loses its || true and its hardcoded /var/lib/docker
substitution, so an unresolvable root no longer measures a guessed filesystem.
Its compose.exec locality was already correct and is unchanged.
Verified against the pre-fix module under the CI condition: with the data root
stubbed to a path absent from this namespace, the old code raises
FileNotFoundError and the new code measures the local vantage; with the daemon
stubbed unreachable, the new code still raises RuntimeError.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two problems, same two files, neither ever gated: the earlier run died in
autoformat before test-unit started.
ruff wanted Path.stat/Path.chmod over their os counterparts and an explicit
check on subprocess.run — worth stating here, since down() is called precisely
to observe a non-zero exit while its sibling git() sets check=True.
The write-bit pinning the four remaining tests rely on does not restrain root,
and the suite runs as root in the container: the script deleted the directory
anyway, the assertion inverted, and the finally block then chmod'd a path that
was already gone. Those four now skip when euid is 0, because the scenario
cannot exist there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
git worktree remove deletes the checkout first and the metadata dir
second. Under the agent sandbox, which holds config.worktree and
commondir as read-only bind mounts, it therefore dies with EBUSY after
the checkout is already gone, aborting make worktree-down with rc 255
even though the branch is in fact free. down.sh now falls back to the
unregistration prune.sh already used (drop gitdir + HEAD, best-effort
rm -rf) and reports any dir the sandbox still pins. That routine moves
into lib.sh so both callers cannot drift apart, and gains a third
outcome for the case where gitdir/HEAD themselves survive: only then is
the branch still claimed, and only then must the teardown fail.
The discriminator for the fallback is whether the checkout survived,
not a match on "Device or resource busy" — the identical half-removed
state arises from EACCES, EPERM and EROFS.
A failing git status probe no longer reads as clean: on a broken .git
link it exits 128 with empty output, which used to satisfy the guard,
tear down the compose stack, and only then die at git worktree remove
with an unexplained 128.
Measured against the unfixed scripts with real fixtures: pinned
metadata dir 255 -> 0, broken .git link 128 -> 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two problems, same two files, neither ever gated: the earlier run died in
autoformat before test-unit started.
ruff wanted Path.stat/Path.chmod over their os counterparts and an explicit
check on subprocess.run — worth stating here, since down() is called precisely
to observe a non-zero exit while its sibling git() sets check=True.
The write-bit pinning the four remaining tests rely on does not restrain root,
and the suite runs as root in the container: the script deleted the directory
anyway, the assertion inverted, and the finally block then chmod'd a path that
was already gone. Those four now skip when euid is 0, because the scenario
cannot exist there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
git worktree remove deletes the checkout first and the metadata dir
second. Under the agent sandbox, which holds config.worktree and
commondir as read-only bind mounts, it therefore dies with EBUSY after
the checkout is already gone, aborting make worktree-down with rc 255
even though the branch is in fact free. down.sh now falls back to the
unregistration prune.sh already used (drop gitdir + HEAD, best-effort
rm -rf) and reports any dir the sandbox still pins. That routine moves
into lib.sh so both callers cannot drift apart, and gains a third
outcome for the case where gitdir/HEAD themselves survive: only then is
the branch still claimed, and only then must the teardown fail.
The discriminator for the fallback is whether the checkout survived,
not a match on "Device or resource busy" — the identical half-removed
state arises from EACCES, EPERM and EROFS.
A failing git status probe no longer reads as clean: on a broken .git
link it exits 128 with empty output, which used to satisfy the guard,
tear down the compose stack, and only then die at git worktree remove
with an unexplained 128.
Measured against the unfixed scripts with real fixtures: pinned
metadata dir 255 -> 0, broken .git link 128 -> 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker info reports the daemon's data root. Under act only the socket is
mounted, so that path is not in the runner's mount namespace and every
swarm run died at the STORAGE_CONSTRAINED lookup with FileNotFoundError.
The container's own writable layer lives under that data root, so its
root filesystem carries the same free space.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
