Six of thirteen failures in run 30937913632 landed on the one package
acquisition path that carries no retry: package_call() built every ordinary
distro-repo install with retry=None, and the enable_existing branch did the
same. EXTERNAL_FETCH was attached to AUR, COPR, PPA and source builds only,
and its docstring said so outright.
The dichotomy that justified it - external endpoints are flaky, distro repos
are not - is one the repo already rejects elsewhere: the image build wraps
every dnf call in RETRIES=3, and sys-svc-ssh hand-rolls retries: 5 for apt
with the comment 'nested CI egress is flaky'. Line 14 of that same file then
hands the RedHat branch to the unarmoured path, which is where lam 2 and
prometheus 0 died.
The measurement that settles it, same task across the run:
2.98s / 5.34s / 5.63s / 6.78s / 18.09s / 20.46s / 68.41s / 367.71s (FAIL) /
769.29s (FAIL) / 815.18s (SUCCESS). The slowest run is a success - failure
and success sit on one continuous tail, so the split is not mirror-up versus
mirror-down. xwiki 2 hit the identical epel metadata error, same checksum,
404 from ten mirrors including dl.fedoraproject.org, and is green only
because the Docker build retried it five seconds later.
The three upstreams that failed - EPEL, updates-testing and www.centos.org -
are exactly the three the package cache does not proxy, so the cache cannot
substitute for this.
test_a_plain_repository_install_is_not_retried was the defect's own tripwire
and inverts accordingly.
Also here because it is the same class: roles/update pinned dnf to retries=5,
below dnf's own default of 10 - a hardening task that halved what it meant to
raise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The REST API answers inputs: null for a finished workflow_dispatch, so
the run name is the only record of what a manual run was dispatched
with. It carried one value, distros; everything else silently fell back
to the workflow defaults, and a failure that only reproduces on swarm
with btrfs came back green from a run that deployed all three modes on
whatever filesystem it drew.
Render every input as its own <emoji><value> segment instead, omitted
when it holds its default, so the title carries exactly what deviates
and an absent input means 'leave it on the default'. run_name derives
those segments from the workflow rather than knowing them, so the format
cannot drift away from the parser without a failing test.
distros no longer defaults to debian -- empty draws one at random, the
way untagged pushes already do. That leaves nothing in the title, so the
effective list is recovered from the discover job names, matched by
content against the meta/distros.yml SPOT rather than by job-name shape.
Priority roles that fell behind the budget cut in every mode hold no job
at all, so failed_roles cannot see them: they are neither green nor red,
they never ran. Carry them along or they never deploy.
--run now resolves before the selection, so it reproduces a run whatever
picks the roles, not only with --failed.
tests/utils/ci_*.py move into tests/utils/ci/ -- two files sharing a
delimiter-bounded prefix are a folder in disguise, and the structure
lint says so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub cancels a job queued for 24h. A line starting swarm, compose and
host at once queues its whole matrix in one go, so with more roles than
runner slots the tail waits past that cut and dies unrun -- 54 of 241
jobs in run 30937913632.
Deploy the modes one after the other instead, heaviest first, which
restarts the queue clock per mode. cli.meta.ci.sequencing counts a
line's matrix jobs on the row basis the discover steps use and
serialises above INFINITO_CI_SEQUENTIAL_THRESHOLD; below it the modes
still start together, so a small diff pays no wall-clock for this.
A needs: edge is static, so both layouts exist as jobs and the decision
skips one. Their conditions are structurally asserted, not evaluated:
ten near-identical if expressions invite a copied sentinel or a missing
chain edge, and both failures are silent at runtime.
mode_fail_fast stops a serial line at its first failed mode, on by
default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five independent blockers, none of which the role could get past.
1. Credentials. create-project pulled the metapackage from repo.magento.com,
which needs Adobe Marketplace keys; without them the bootstrap got HTTP 401
after 40 minutes of stack build. meta/schema.yml declared both keys with
validation ^.*$, so an empty value passed inventory validation silently.
mirror.mage-os.org serves the same packages unauthenticated and hosts the
dists itself - verified on the deploy's own network path, HTTP 200 for
project-community-edition-2.4.8.zip. Repository, package and version now
live in meta/services.yml, which also makes a later switch to the Mage-OS
distribution proper a data change rather than a code change.
2. Premature handler flush. The config renders notified 'compose up', so the
handler chain ran before sys-stk-full had rendered compose.yml and the CA
handler died on the missing file. Worse, the render had already created the
instance directory, so the repo's own guard for exactly this
(Fail: instance dir missing at handler flush) no longer tripped.
3. Anonymous volumes. Both markoshust images declare VOLUME /var/www while
the named volume mounted one level deeper at /var/www/html, so docker created
an anonymous volume per container and sys-ctl-hlth-volumes refused to start.
The mount moves to /var/www, where the image declares it; the image ships an
empty html/ that a fresh named volume initialises from.
4. Unescaped expansions. setup:install passed --db-host=\"$MYSQL_HOST\" and
friends, but only the quotes were escaped for the outer shell, not the
dollars. The DiD shell expanded them to empty, Magento fell back to localhost,
took that as a unix socket and reported SQLSTATE[HY000] [2002]. 18 expansions.
5. A non-resumable bootstrap. create-project needs an empty target and cannot
be retried, yet it carried the dependency download - and mirror.mage-os.org
runs nginx with the default http2_max_requests=1000, so composer's ~490
requests on one connection hit GOAWAY mid-transfer at the same package twice
in a row (curl error 18, not the flake it looked like). It is now
create-project --no-install plus a retryable composer install, which resumes
from the cache; measured succeeding in the live container after the abort.
The creates: guards follow: composer.json marks the skeleton, bin/magento the
finished install - it only exists once dependencies are in.
MAGENTO_COMPOSER_SPEC folds package and version because autoformat broke the
create-project line inside a Jinja expression; it rendered correctly by luck.
Not fixed here: the base URL renders with a double slash, and the lifecycle
stays pre-alpha - the role reaches setup:install now but has not yet served
a page.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role could not deliver anything. The markoshust nginx image ships a
default vhost built for a different topology: fastcgi points at
unix:/sock/phpfpm.sock, port 8000 only returns 301 to https, the real
server sits on 8443 with a self-signed cert, and it references an upstream
host 'phpfpm' that does not exist here - so nginx aborted at config load
with 'host not found in upstream' and the container never started.
php-fpm listened on that same unix socket, because its
/usr/local/etc/php-fpm.conf carries no include line and php-fpm.d/*.conf
is therefore dead config.
Both are now replaced through type: config mounts. The role never had any
- that absence is why every template it previously rendered was inert.
The vhost stays minimal and includes Magento's own nginx.conf, copied from
nginx.conf.sample by the bootstrap. Its location blocks carry the .php
whitelist, the /static/version rewrites and the /media to get.php fallback
that a hand-written vhost has to reimplement - the previous attempt did
not, and would have executed every .php under pub/.
PHP moves from 8.5 to 8.4: Magento 2.4.8 requires ~8.2.0||~8.3.0||~8.4.0
and the bootstrap passes no platform-requirement override, so
composer create-project failed before nginx ever mattered.
nginx waits for the php socket before starting, for the reason given in
the erpnext frontend commit: the wildcard search domain turns a DNS miss
into a routable wrong answer, and nginx pins upstreams at config load.
Also: the oauth2 origin port named 8080, where nothing listens, and the
healthcheck proved only that some listener existed - it stayed green on a
redirect loop and on 502.
Measured in a rebuild of the role topology against Magento's real
nginx.conf.sample: HTTP 200 on / and /index.php, healthy healthcheck, and
the counter-test - without the gate and with php down, nginx exits 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
01_runtime_conf.yml rendered a nginx.conf and a php-fpm pool override that
never reached a container: no task included the file, and meta/volumes.yml
declares no mount for either output. Verified against HEAD in a second
checkout - both templates were already dead there, so removing them changes
runtime behaviour by exactly nothing.
Their four path variables go with them, as do MAGENTO_PHP_HOST,
MAGENTO_PHP_PORT and MAGENTO_NGINX_PORT, whose only consumers were the
assert and the two templates.
Closing the numbering gap renames 02_setup.yml to 01_setup.yml, the name it
could not take while 01_runtime_conf.yml existed.
This does not make the role work - it does not work today either. The
markoshust nginx image cannot even start under this topology
(host not found in upstream "phpfpm"), and its php-fpm listens on a unix
socket rather than TCP 9000. Both are addressed separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The frappe nginx entrypoint envsubsts BACKEND into a static upstream
block, so nginx resolves the name once at config-parse time and pins the
address for the container's life. When backend is briefly absent from
docker's embedded DNS, libc does not return NXDOMAIN: it falls through to
the infinito.example search domain, whose wildcard answers with the
ingress host. nginx then starts successfully against an address nothing
listens on and 502s forever.
Observed in run 30937913632 from both directions: compose logged 93
identical connect-refused errors to 172.30.0.10:8000 over 31 minutes
while the backend was healthy at 192.168.214.7, and all three swarm
replicas logged the same against the manager IP with zero 200s. The
compose trigger was the svc-bkp-volume-2-local backup, which stops and
starts every container mounting a volume without regard to dependencies;
in swarm, stack deploy drops depends_on outright.
The gate probes TCP rather than DNS because the wildcard resolves the
name either way. Five of the six other long-running services in this
template already use the same bash -c gate idiom.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both creates: guards pointed at paths that could never match, so the
bootstrap and setup:install shells re-ran on every deploy.
The first guard went through docker_volume_path, which hardcodes
/var/lib/docker/volumes/<name>/_data and misses any relocated docker
data-root. The second was worse: path_join over the bare volume name
produced a relative path, resolved against the Ansible working
directory, which matched nowhere at all.
Both now derive from the mountpoint the daemon reports. The stack-host
gate moves to a single include, mirroring web-app-akaunting.
With its last caller converted, docker_volume_path is deleted. It could
not be repaired: a Jinja filter runs on the controller and cannot know a
remote host's data-root.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AKAUNTING_SETUP_MARKER was built from docker_volume_path, which hardcodes
/var/lib/docker/volumes/<name>/_data. The CI swarm lab relocates each
node's docker data-root to /mnt/docker-fs/docker, so the marker touch hit
ENOENT on a directory that was never there, while the volume itself sat
healthy at the relocated path.
Same root cause, silent: the marker stat reported exists=false on every
relocated host, so the one-shot installer re-ran on each deploy.
The mountpoint now comes from the daemon. The resolve sits behind
sys-stk-full because on a genuine first deploy the volume does not exist
yet and volume inspect would abort the very first install. The stack-host
gate moved to a single include, so nothing evaluates the marker on hosts
where the register is undefined.
docker_volume_path stays for now; web-app-magento is its last caller and
is converted in the follow-up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The inline repo definition pinned gpgkey to www.centos.org, a single
non-CDN origin. In run 30937913632 it was unreachable for at least
17m25s and killed 2 of 23 centos ganesha installs; dnf's own retries=5
was already active on the host and did not help.
centos-release-nfs-ganesha11 ships the same repo with metalink failover
and gpgkey=file://, planting the key through Requires:
centos-release-storage-common. Verified end to end against
quay.io/centos/centos:latest and ghcr.io/kevinveenbirkenbach/pkgmgr-centos:stable.
The BOOTSTRAP_BASEURL entry keeps the availability probe verifying both
names instead of degrading them to an inconclusive warning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3.4.1 drops -b from the volume rsync, so a generation no longer keeps a file
the source replaced or deleted between the hot and the cold pass as name~.
That twin is what made the restore drill's elasticsearch unrecoverable: Lucene
parses every file starting with segments as a radix-36 generation, so a restored
segments_3~ leaves the shard store unreadable and .security-7 unallocatable,
and the reserved elastic user then has no password hash to authenticate against.
Only new generations are clean. A restore from one written before this bump
still carries the twins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rsync -b turns a deletion into a rename, so a file the source dropped between
two writes of the same generation survives as name~ beside the real one and is
restored into live data. baudolo 3.4.1 fixes the copy that caused the
elasticsearch restore failure; this is the sweep over the sibling call sites
here.
svc-bkp-remote-2-local is the live one: the rsync command is assembled once and
re-executed up to twelve times against the same destination, which is created
with exist_ok before the loop starts. A retry after a partial transfer is
structurally the same second write as baudolo's hot/cold pair, only triggered by
--timeout instead of a container stop. That path pulls other hosts' backups, and
retries are its normal case rather than an exception.
svc-bkp-local-2-device cannot fire today - it writes one pass into a directory
named after the current second - but mkdir(exist_ok=True) makes a re-entry the
same two-write case, so the flag goes rather than the argument for keeping it.
The integration test mirrors the production command and follows.
The lint is what makes this complete: it scans every tracked .py/.sh/.j2/.yml
for a backup flag near an rsync token, and it found the integration test that
the manual sweep had missed. It excludes exactly one file, its own, because the
rule has to name the token it forbids; the sibling guard for || true avoids the
same problem only by scanning .sh alone.
Verified: make quality green across all four targets, the ssh-wrapper
integration test passes with real rsync, and test-lint goes 307 -> 308 with the
new guard collected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make quality failed on three ruff errors, all in the CI deploy-sequencing
work: RUF007 twice in test_ci_deploy_sequencing.py and TID252 in
tests/utils/ci/job_names.py. autoformat is chained ahead of test, so the whole
test stage never ran while they stood.
zip(CHAIN, CHAIN[1:], strict=False) becomes pairwise(CHAIN). Same two pairs
over the three-element chain, but strict=False was switching off a length
check that this call genuinely needs -- CHAIN[1:] is shorter by construction,
so strict=True would raise. pairwise has no second sequence and cannot have
the problem, so the suppression goes away with its reason. Both assertIn calls
fit on one line again as a result; that reflow is ruff format's, not a
hand-made edit.
job_names.py was the only file reaching PROJECT_ROOT through a relative parent
import; test_ci_deploy_sequencing.py:20 already spells it absolutely.
Verified rather than assumed: make quality now green end to end (ruff in the
ran list, all four targets green, 4030 + 299 passed). The sequencing file on
its own reports the same 24 subtests as before the change, so pairwise yields
the identical pair set and not merely a passing one. All four importers of
job_names run green (60 passed), which is what rules out the absolute import
resolving differently under another sys.path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
07_compose_minimal.sh claimed to disable 'all variable (group-conditional)
services' and then named seven by hand. tor was not among them, so a
deploy meant to be minimal pulled svc-net-tor, which hard-depends on
web-svc-mirror, whose group-conditional prometheus flag the closure
resolver treats as true — dragging the whole web stack in. Steps 12 and
13 passed no disable at all, which is why a mariadb swarm test deployed
keycloak, mailu, matomo, prometheus and dashboard.
The set is now derived from the contract the repo already enforces
(tests/integration/roles/meta/services/test_dynamic_flags.py): a key is
group-conditional when enabled OR shared is a Jinja expression. Checking
only enabled would miss mariadb, postgres, redis, memcached, typesense.
Exclusions are load-bearing, not cosmetic: dashboard because disabling
the primary app aborts init; javascript because it aliases cdn, which
web-app-dashboard declares a hard dependency with no runtime fallback;
mariadb and postgres because steps 12 and 13 test them.
variable_services assigns to a local first — disable="$(cmd)" does not
set the exit status of a simple command, so a failing CLI would have
yielded an empty disable set and steps 12/13 would have run the full
deploy and reported green.
roundtrip could not take disable at all; it now reaches both the compose
and the swarm leg.
Not verified by a deploy: 33 of the keys are new to step 07, and the
variant axis of svc-db-mariadb/svc-db-postgres collapses because tor is
its only dimension.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deleting the external onion probe left check_external, _CURL_REACHED_RCS,
the internal|external|both dispatch and the optional socks argument with
no caller, while the docstring still promised 'Exit 0 only when both
succeed' and that the external path 'must actually work end to end' —
the exact contract that deletion removed.
The script now takes one argument. Real-network reachability stays gated
where it blocks something: test-e2e-playwright's wait_onion_reachable.sh.
Wiring the CLI test into e2e-cli also armed its default retry budget:
20 attempts x curl --max-time 30 plus 19 x 15s sleep is 885s per failing
domain, serial. TEST_E2E_CLI_TIMEOUT is 3600s, so four failing domains
exhaust it and a tor deploy serves about twelve vhosts. test.env.j2 now
pins RETRIES=6 and SLEEP_SECONDS=10 — 230s per domain.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_fingerprint_mapping memoised digests under id(obj) while holding no
reference to obj, so CPython reused the address of a freed mapping and
served its digest for an unrelated live one. Measured 398/400 wrong
digests for short-lived mappings; in-place mutation was invisible and
the domain-index collision guard could be bypassed entirely.
Two unit tests pinned that behaviour as the contract —
test_id_cache_short_circuits_recompute asserted that mutating a dict in
place leaves its fingerprint unchanged — so the defect shipped green.
Both now assert the opposite.
build_domain_index rebuilt the whole reverse index on every lookup,
scanning every application's domains and aliases to answer one key. It
is now memoised on that fingerprint, which is only sound because of the
fix above. 4x on a 100-app payload.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-matrix 0,1,2 failed with no assertion at all - JUnit reports
errors=1, failures=0. Chromium crashed: dmesg holds exactly two traps in the
whole 04:34-05:55 window, second-exact to both attempts and at the identical
binary offset, so one fatal CHECK() twice. int3 is Chromium aborting itself,
not a kill from outside.
The same job proves the budget is what decided the outcome:
tests/test-seaweedfs.js hit the identical error class twice, ~5s in each
time, and passed on attempt 3 because it inherits retries: 2 from
playwright.config.js. The DM test opted down to 1 and died.
Ruled out with evidence rather than by elimination: not the application
(zero 502s in the whole openresty log, Synapse 200 throughout), not the auth
chain (both OIDC logins succeeded on attempt 1), not the test (no assertion
reached), not the timeout (effective budget 1200s, attempts took 170s and
164s), not OOM (dmesg-oom.txt empty, 9G available).
The comment goes with it. It attributed the crashes to processes being
'OOM-killed', which is exactly what the artifacts refute, and it would have
sent the next investigator after memory pressure that was never there.
Worst case is 8.5 min instead of 5.7 against a 59-minute suite budget.
Not addressed here: which CHECK fires at that offset is unknowable from
these artifacts because Playwright discards the browser's stderr.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-openwebui 0,1,2 was killed by the 350-minute action timeout with no
failed task anywhere in its 1.33M log lines. The sweep admitted fedora at
remaining=10372s after debian took 10028s, then never looked again: the
budget was only checked between distros. Its internal deadline passed 594
seconds before GitHub killed the job - the run knew it was over time for ten
minutes and had no way to act. Worst-case runtime was deadline plus the full
duration of the last admitted distro, unbounded.
The trigger was a single stalled task (803s for a no-op that takes 1.3s
warm, and a fully cold transaction 8 minutes later cost 24s), but that only
decided which job hit the wall. The wall itself is structural.
remaining is already computed at the top of the loop, so wrapping the call
in timeout adds no new state. rc 124 becomes 'skipped' rather than 'failed',
because a job that stops at its budget is the documented intent, not a
defect - test_distro_budget_fits_timeout.py already asserts this invariant,
which until now was simply untrue.
Two conditions narrow it, and both matter:
dur >= remaining - 5 keeps an inner timeout that fires early as a failure.
Verified: timeout returns 124 only for its own kill and passes inner exit
codes through unchanged (0 for success, 3 for an inner exit 3).
passed > 0 is the one that protects the test result. Without it, a job whose
very first distro overruns would report 'skipped' and exit 0 - green while
having verified nothing. That would have been a silent false pass across all
243 jobs of a run. The guard was executed over the full truth table,
including passed=0.
-k 60 is deliberate: one.sh execs into dedicated.sh and its ansible and
docker children can outlive SIGTERM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-penpot 2 failed with 'Port 2049 never opened'. ganesha had not
crashed: the unit was active, 5 threads, 18ms CPU, and the journal stopped
after the export CLIENT warnings without ever reaching NFS SERVER
INITIALIZED. All five threads sat on one futex word (FUTEX_WAIT_PRIVATE,
val 2) with no live thread holding it - an orphaned mutex in liburcu-bp,
entered from ReadExports -> fsal_cfg_commit -> urcu_bp_register. Bind_sockets()
is strictly downstream of ReadExports(), so nothing was ever bound.
This is not the Fedora 43 / ganesha 7.2 SEGV: that one reached listen state
and died on accept, 100% reproducible. This one never listens, alive and
idle, and hits roughly 1-3% of fedora legs - lam 2 started the same build
twice in the same run.
The recovery that already existed could not fire. Type=forking plus an
ExecStart parent that exited 0 means systemd considers the unit started, so
Restart=on-failure never saw a failure. ExecStartPost runs before the start
job completes, so a failing probe there is the event it has been waiting for.
Verified rather than assumed, with a throwaway transient unit carrying the
same properties: a Type=forking unit whose ExecStartPost fails ends in
'failed' with the same message this class produces in CI, Restart=on-failure
then schedules restarts 1, 2, 3, and StartLimitBurst stops it with 'Start
request repeated too quickly'. StartLimitIntervalSec was 0, i.e. unbounded -
without the burst cap this fix would trade a rare wedge for a silent loop.
The 20s budget is measured, not guessed: a healthy ganesha binds 2049 in
0.32-0.38s across eight successful gates in lam 2, so the probe sits ~57x
above normal. /usr/bin/timeout and /bin/bash were checked on all four node
images before hardcoding them.
TimeoutStopSec=10 is required, not cosmetic: the wedged process blocks
SIGTERM in main() and its sigwait consumer is created downstream of the
wedge, so every recovery cycle would otherwise burn ~45s to SIGABRT.
The Ansible gate goes 60s -> 120s because one recovery cycle is ~32s and 60
expires before the first one finishes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six of thirteen failures in run 30937913632 landed on the one package
acquisition path that carries no retry: package_call() built every ordinary
distro-repo install with retry=None, and the enable_existing branch did the
same. EXTERNAL_FETCH was attached to AUR, COPR, PPA and source builds only,
and its docstring said so outright.
The dichotomy that justified it - external endpoints are flaky, distro repos
are not - is one the repo already rejects elsewhere: the image build wraps
every dnf call in RETRIES=3, and sys-svc-ssh hand-rolls retries: 5 for apt
with the comment 'nested CI egress is flaky'. Line 14 of that same file then
hands the RedHat branch to the unarmoured path, which is where lam 2 and
prometheus 0 died.
The measurement that settles it, same task across the run:
2.98s / 5.34s / 5.63s / 6.78s / 18.09s / 20.46s / 68.41s / 367.71s (FAIL) /
769.29s (FAIL) / 815.18s (SUCCESS). The slowest run is a success - failure
and success sit on one continuous tail, so the split is not mirror-up versus
mirror-down. xwiki 2 hit the identical epel metadata error, same checksum,
404 from ten mirrors including dl.fedoraproject.org, and is green only
because the Docker build retried it five seconds later.
The three upstreams that failed - EPEL, updates-testing and www.centos.org -
are exactly the three the package cache does not proxy, so the cache cannot
substitute for this.
test_a_plain_repository_install_is_not_retried was the defect's own tripwire
and inverts accordingly.
Also here because it is the same class: roles/update pinned dnf to retries=5,
below dnf's own default of 10 - a hardening task that halved what it meant to
raise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The REST API answers inputs: null for a finished workflow_dispatch, so
the run name is the only record of what a manual run was dispatched
with. It carried one value, distros; everything else silently fell back
to the workflow defaults, and a failure that only reproduces on swarm
with btrfs came back green from a run that deployed all three modes on
whatever filesystem it drew.
Render every input as its own <emoji><value> segment instead, omitted
when it holds its default, so the title carries exactly what deviates
and an absent input means 'leave it on the default'. run_name derives
those segments from the workflow rather than knowing them, so the format
cannot drift away from the parser without a failing test.
distros no longer defaults to debian -- empty draws one at random, the
way untagged pushes already do. That leaves nothing in the title, so the
effective list is recovered from the discover job names, matched by
content against the meta/distros.yml SPOT rather than by job-name shape.
Priority roles that fell behind the budget cut in every mode hold no job
at all, so failed_roles cannot see them: they are neither green nor red,
they never ran. Carry them along or they never deploy.
--run now resolves before the selection, so it reproduces a run whatever
picks the roles, not only with --failed.
tests/utils/ci_*.py move into tests/utils/ci/ -- two files sharing a
delimiter-bounded prefix are a folder in disguise, and the structure
lint says so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub cancels a job queued for 24h. A line starting swarm, compose and
host at once queues its whole matrix in one go, so with more roles than
runner slots the tail waits past that cut and dies unrun -- 54 of 241
jobs in run 30937913632.
Deploy the modes one after the other instead, heaviest first, which
restarts the queue clock per mode. cli.meta.ci.sequencing counts a
line's matrix jobs on the row basis the discover steps use and
serialises above INFINITO_CI_SEQUENTIAL_THRESHOLD; below it the modes
still start together, so a small diff pays no wall-clock for this.
A needs: edge is static, so both layouts exist as jobs and the decision
skips one. Their conditions are structurally asserted, not evaluated:
ten near-identical if expressions invite a copied sentinel or a missing
chain edge, and both failures are silent at runtime.
mode_fail_fast stops a serial line at its first failed mode, on by
default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five independent blockers, none of which the role could get past.
1. Credentials. create-project pulled the metapackage from repo.magento.com,
which needs Adobe Marketplace keys; without them the bootstrap got HTTP 401
after 40 minutes of stack build. meta/schema.yml declared both keys with
validation ^.*$, so an empty value passed inventory validation silently.
mirror.mage-os.org serves the same packages unauthenticated and hosts the
dists itself - verified on the deploy's own network path, HTTP 200 for
project-community-edition-2.4.8.zip. Repository, package and version now
live in meta/services.yml, which also makes a later switch to the Mage-OS
distribution proper a data change rather than a code change.
2. Premature handler flush. The config renders notified 'compose up', so the
handler chain ran before sys-stk-full had rendered compose.yml and the CA
handler died on the missing file. Worse, the render had already created the
instance directory, so the repo's own guard for exactly this
(Fail: instance dir missing at handler flush) no longer tripped.
3. Anonymous volumes. Both markoshust images declare VOLUME /var/www while
the named volume mounted one level deeper at /var/www/html, so docker created
an anonymous volume per container and sys-ctl-hlth-volumes refused to start.
The mount moves to /var/www, where the image declares it; the image ships an
empty html/ that a fresh named volume initialises from.
4. Unescaped expansions. setup:install passed --db-host=\"$MYSQL_HOST\" and
friends, but only the quotes were escaped for the outer shell, not the
dollars. The DiD shell expanded them to empty, Magento fell back to localhost,
took that as a unix socket and reported SQLSTATE[HY000] [2002]. 18 expansions.
5. A non-resumable bootstrap. create-project needs an empty target and cannot
be retried, yet it carried the dependency download - and mirror.mage-os.org
runs nginx with the default http2_max_requests=1000, so composer's ~490
requests on one connection hit GOAWAY mid-transfer at the same package twice
in a row (curl error 18, not the flake it looked like). It is now
create-project --no-install plus a retryable composer install, which resumes
from the cache; measured succeeding in the live container after the abort.
The creates: guards follow: composer.json marks the skeleton, bin/magento the
finished install - it only exists once dependencies are in.
MAGENTO_COMPOSER_SPEC folds package and version because autoformat broke the
create-project line inside a Jinja expression; it rendered correctly by luck.
Not fixed here: the base URL renders with a double slash, and the lifecycle
stays pre-alpha - the role reaches setup:install now but has not yet served
a page.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role shipped a complete probe script at files/test.sh and no
templates/test.env.j2. discover_cli_roles globs the env template alone,
so the role was never discovered and the script had never run — and the
script hard-requires TOR_SOCKS and NGINX_SERVERS_DIR via ${VAR:?}, so it
could not have run without one either. Its own usage hint pointed at a
path that does not exist.
Moving the script to the discovered path and adding the env template
makes both halves line up. Verified against the live stack: the onion
answers through the Tor SOCKS proxy, 1 probed, 0 failed.
The new lint keeps the pair together and rejects a script left at
files/test.sh. The two halves fail differently and only one is loud: an
env template without a script aborts the play, a script without an env
template is never discovered and its silence reads like a pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
variants.yml held a single entry, so the CI coverage the previous commit
unlocked would have exercised chutney only — a private, offline Tor net
built for tests. The public flavor is the production path and stayed
uncovered.
Variant 1 overrides flavor and use_entry_guards and nothing else:
resources fall through to meta/services.yml, which is sized for the one
tor container, whereas variant 0 raises them for chutney's ~9 in-container
nodes. Guards stay off for the same reason variant 0 turns them off — a
degraded entry guard must not stall the node.
Swarm splits per variant and now runs two jobs for this role; compose
bundles both into one.
Unproven: whether a public-flavor container bootstraps into the real Tor
network from a GitHub runner. The internal loopback probe is fatal, so a
failed bootstrap fails the job hard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
modes.compose.enabled and modes.swarm.enabled were both false, and a
stack role is forced host=false, so all three test_<mode> columns were
false and the CI discovery filter (test_<mode> == true) dropped the role
from every matrix. It was resolved into the priority whitelist as a new
role and then silently filtered out again — svc-net-tor was only ever
exercised indirectly, as a dependency other roles pull in.
The flag is read solely by the discovery model and the README generator;
nothing in the deploy path consults it, so this has no runtime effect.
meta/categories.yml pins the whole net category to compose/swarm false,
which fits svc-net-wireguard-{core,plain,firewalled}: none of them ships
a compose template. svc-net-tor does, and is the only role in the
category that does, so it takes the category-modes exemption the lint
provides rather than bending the category default for three host roles.
Regenerated READMEs follow: svc-net-tor now carries its mode symbols in
every cosmos diagram that names it as a dependency.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The probe carried retries: 90, delay: 5 but failed_when: false, so it
could never gate a deploy — its only consumer was the debug warning
below it. Measured in CI run 30963648828: 4 blocks at 7:40/7:43/1:08/8:03,
283 retries, 3 of 4 ending non-fatal 'not reachable'.
Onion reachability is already gated where it matters, by
test-e2e-playwright's wait_onion_reachable.sh: it fails hard, budgets
2 x 40 x 15s, and force-recreates Tor for fresh circuits in between. In
the same job that gate ran 28 times for 6s total, while this probe was
reporting the onion unreachable.
The internal loopback probe stays and remains fatal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3.4.1 drops -b from the volume rsync, so a generation no longer keeps a file
the source replaced or deleted between the hot and the cold pass as name~.
That twin is what made the restore drill's elasticsearch unrecoverable: Lucene
parses every file starting with segments as a radix-36 generation, so a restored
segments_3~ leaves the shard store unreadable and .security-7 unallocatable,
and the reserved elastic user then has no password hash to authenticate against.
Only new generations are clean. A restore from one written before this bump
still carries the twins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rsync -b turns a deletion into a rename, so a file the source dropped between
two writes of the same generation survives as name~ beside the real one and is
restored into live data. baudolo 3.4.1 fixes the copy that caused the
elasticsearch restore failure; this is the sweep over the sibling call sites
here.
svc-bkp-remote-2-local is the live one: the rsync command is assembled once and
re-executed up to twelve times against the same destination, which is created
with exist_ok before the loop starts. A retry after a partial transfer is
structurally the same second write as baudolo's hot/cold pair, only triggered by
--timeout instead of a container stop. That path pulls other hosts' backups, and
retries are its normal case rather than an exception.
svc-bkp-local-2-device cannot fire today - it writes one pass into a directory
named after the current second - but mkdir(exist_ok=True) makes a re-entry the
same two-write case, so the flag goes rather than the argument for keeping it.
The integration test mirrors the production command and follows.
The lint is what makes this complete: it scans every tracked .py/.sh/.j2/.yml
for a backup flag near an rsync token, and it found the integration test that
the manual sweep had missed. It excludes exactly one file, its own, because the
rule has to name the token it forbids; the sibling guard for || true avoids the
same problem only by scanning .sh alone.
Verified: make quality green across all four targets, the ssh-wrapper
integration test passes with real rsync, and test-lint goes 307 -> 308 with the
new guard collected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role could not deliver anything. The markoshust nginx image ships a
default vhost built for a different topology: fastcgi points at
unix:/sock/phpfpm.sock, port 8000 only returns 301 to https, the real
server sits on 8443 with a self-signed cert, and it references an upstream
host 'phpfpm' that does not exist here - so nginx aborted at config load
with 'host not found in upstream' and the container never started.
php-fpm listened on that same unix socket, because its
/usr/local/etc/php-fpm.conf carries no include line and php-fpm.d/*.conf
is therefore dead config.
Both are now replaced through type: config mounts. The role never had any
- that absence is why every template it previously rendered was inert.
The vhost stays minimal and includes Magento's own nginx.conf, copied from
nginx.conf.sample by the bootstrap. Its location blocks carry the .php
whitelist, the /static/version rewrites and the /media to get.php fallback
that a hand-written vhost has to reimplement - the previous attempt did
not, and would have executed every .php under pub/.
PHP moves from 8.5 to 8.4: Magento 2.4.8 requires ~8.2.0||~8.3.0||~8.4.0
and the bootstrap passes no platform-requirement override, so
composer create-project failed before nginx ever mattered.
nginx waits for the php socket before starting, for the reason given in
the erpnext frontend commit: the wildcard search domain turns a DNS miss
into a routable wrong answer, and nginx pins upstreams at config load.
Also: the oauth2 origin port named 8080, where nothing listens, and the
healthcheck proved only that some listener existed - it stayed green on a
redirect loop and on 502.
Measured in a rebuild of the role topology against Magento's real
nginx.conf.sample: HTTP 200 on / and /index.php, healthy healthcheck, and
the counter-test - without the gate and with php down, nginx exits 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
01_runtime_conf.yml rendered a nginx.conf and a php-fpm pool override that
never reached a container: no task included the file, and meta/volumes.yml
declares no mount for either output. Verified against HEAD in a second
checkout - both templates were already dead there, so removing them changes
runtime behaviour by exactly nothing.
Their four path variables go with them, as do MAGENTO_PHP_HOST,
MAGENTO_PHP_PORT and MAGENTO_NGINX_PORT, whose only consumers were the
assert and the two templates.
Closing the numbering gap renames 02_setup.yml to 01_setup.yml, the name it
could not take while 01_runtime_conf.yml existed.
This does not make the role work - it does not work today either. The
markoshust nginx image cannot even start under this topology
(host not found in upstream "phpfpm"), and its php-fpm listens on a unix
socket rather than TCP 9000. Both are addressed separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The frappe nginx entrypoint envsubsts BACKEND into a static upstream
block, so nginx resolves the name once at config-parse time and pins the
address for the container's life. When backend is briefly absent from
docker's embedded DNS, libc does not return NXDOMAIN: it falls through to
the infinito.example search domain, whose wildcard answers with the
ingress host. nginx then starts successfully against an address nothing
listens on and 502s forever.
Observed in run 30937913632 from both directions: compose logged 93
identical connect-refused errors to 172.30.0.10:8000 over 31 minutes
while the backend was healthy at 192.168.214.7, and all three swarm
replicas logged the same against the manager IP with zero 200s. The
compose trigger was the svc-bkp-volume-2-local backup, which stops and
starts every container mounting a volume without regard to dependencies;
in swarm, stack deploy drops depends_on outright.
The gate probes TCP rather than DNS because the wildcard resolves the
name either way. Five of the six other long-running services in this
template already use the same bash -c gate idiom.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both creates: guards pointed at paths that could never match, so the
bootstrap and setup:install shells re-ran on every deploy.
The first guard went through docker_volume_path, which hardcodes
/var/lib/docker/volumes/<name>/_data and misses any relocated docker
data-root. The second was worse: path_join over the bare volume name
produced a relative path, resolved against the Ansible working
directory, which matched nowhere at all.
Both now derive from the mountpoint the daemon reports. The stack-host
gate moves to a single include, mirroring web-app-akaunting.
With its last caller converted, docker_volume_path is deleted. It could
not be repaired: a Jinja filter runs on the controller and cannot know a
remote host's data-root.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AKAUNTING_SETUP_MARKER was built from docker_volume_path, which hardcodes
/var/lib/docker/volumes/<name>/_data. The CI swarm lab relocates each
node's docker data-root to /mnt/docker-fs/docker, so the marker touch hit
ENOENT on a directory that was never there, while the volume itself sat
healthy at the relocated path.
Same root cause, silent: the marker stat reported exists=false on every
relocated host, so the one-shot installer re-ran on each deploy.
The mountpoint now comes from the daemon. The resolve sits behind
sys-stk-full because on a genuine first deploy the volume does not exist
yet and volume inspect would abort the very first install. The stack-host
gate moved to a single include, so nothing evaluates the marker on hosts
where the register is undefined.
docker_volume_path stays for now; web-app-magento is its last caller and
is converted in the follow-up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The inline repo definition pinned gpgkey to www.centos.org, a single
non-CDN origin. In run 30937913632 it was unreachable for at least
17m25s and killed 2 of 23 centos ganesha installs; dnf's own retries=5
was already active on the host and did not help.
centos-release-nfs-ganesha11 ships the same repo with metalink failover
and gpgkey=file://, planting the key through Requires:
centos-release-storage-common. Verified end to end against
quay.io/centos/centos:latest and ghcr.io/kevinveenbirkenbach/pkgmgr-centos:stable.
The BOOTSTRAP_BASEURL entry keeps the availability probe verifying both
names instead of degrading them to an inconclusive warning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3.4.1 drops -b from the volume rsync, so a generation no longer keeps a file
the source replaced or deleted between the hot and the cold pass as name~.
That twin is what made the restore drill's elasticsearch unrecoverable: Lucene
parses every file starting with segments as a radix-36 generation, so a restored
segments_3~ leaves the shard store unreadable and .security-7 unallocatable,
and the reserved elastic user then has no password hash to authenticate against.
Only new generations are clean. A restore from one written before this bump
still carries the twins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rsync -b turns a deletion into a rename, so a file the source dropped between
two writes of the same generation survives as name~ beside the real one and is
restored into live data. baudolo 3.4.1 fixes the copy that caused the
elasticsearch restore failure; this is the sweep over the sibling call sites
here.
svc-bkp-remote-2-local is the live one: the rsync command is assembled once and
re-executed up to twelve times against the same destination, which is created
with exist_ok before the loop starts. A retry after a partial transfer is
structurally the same second write as baudolo's hot/cold pair, only triggered by
--timeout instead of a container stop. That path pulls other hosts' backups, and
retries are its normal case rather than an exception.
svc-bkp-local-2-device cannot fire today - it writes one pass into a directory
named after the current second - but mkdir(exist_ok=True) makes a re-entry the
same two-write case, so the flag goes rather than the argument for keeping it.
The integration test mirrors the production command and follows.
The lint is what makes this complete: it scans every tracked .py/.sh/.j2/.yml
for a backup flag near an rsync token, and it found the integration test that
the manual sweep had missed. It excludes exactly one file, its own, because the
rule has to name the token it forbids; the sibling guard for || true avoids the
same problem only by scanning .sh alone.
Verified: make quality green across all four targets, the ssh-wrapper
integration test passes with real rsync, and test-lint goes 307 -> 308 with the
new guard collected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mirror failover turned the four apt locations into regex locations.
nginx only synthesises the `/debian` -> `/debian/` redirect for prefix
locations, so the slash-less form started answering 404. Inside a runner
`deb.debian.org` is DNS-hijacked to this frontend and the URL
reachability test probes exactly that form, which took the debian,
fedora and centos workspace jobs down.
The locations are prefix locations again, byte for byte as before. The
only thing that stays is the error path: the named location picks up the
already-rewritten `$uri` and rewrites it once more onto the mirror repo,
so the failover survives without a capture variable.
Measured against a simulated hijack: both bare paths 200 again, all four
suite indexes 200, and with `apt-debian` pointed at a dead remote
`dists/bookworm/InRelease` still arrives with 151 kB through
apt-debian-mirror.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A debian:bookworm workspace job died on `apt-get update`: the mounted
sources list carried the Ubuntu entry too, Nexus had auto-blocked
archive.ubuntu.com, and apt fails the whole update on one bad source.
Three holes, three plugs:
- One sources list per INFINITO_DISTRO under compose/package-cache/apt/,
selected by the compose overrides. A Debian runner no longer fetches
an Ubuntu suite it has no use for. The non-apt distros get an empty
file so the bind mount has a source to point at.
- autoBlock off. While a remote is auto-blocked Nexus answers `404
Remote Auto Blocked` to every client instead of falling back to the
copy it already has cached.
- Mirror failover in the frontend. Every apt suite gets a second proxy
repo on an independent upstream; a 5xx from the primary re-runs the
request against it. Nexus has no group repository type for apt, so
the retry lives in nginx.
Measured on a live Nexus 3.91.1: dead remote without a cached copy
answers 502, with a cached copy 200 regardless of metadataMaxAge. The
frontend serves 149 kB of real bookworm `Release` from the mirror repo
when the primary is dead, and the new regex locations produce upstream
request lines byte-identical to the `rewrite … break` form they replace,
including `%3a` epochs in pool paths.
INFINITO_CACHE_PACKAGE_APT_LIST is gone; the mount interpolates
INFINITO_DISTRO directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In job 91983553540 all five lab nodes derived the same pool name,
infinito_ci_unknown, and the run died on `cannot destroy
'infinito_ci_unknown': pool is busy`. The name came from `hostname`, which
returns non-zero inside the node containers, so the `|| echo unknown` fallback
answered for every one of them. One node won the race and created the pool; the
others found it, took it for a leftover, and tried to tear down a pool a
sibling was actively using.
234c1158c already fixed this collision class three days ago by deriving the
name from `hostname` instead of a constant, and its message named the zfs
exclusion in resolve.sh as the only thing keeping the collision out of sight.
That fix was never exercised, because the exclusion stayed up until ed1121053
removed it - and it does not work, because the call it rests on fails in the
containers it was written for. Guessing the identity is the defect; the
playbook has known it all along as inventory_hostname, and ci/one.sh knows its
own deploy mode. Both now state it, as a required fourth argument with no
default, the way SIZE is already stated. verdict() stops guessing too: every
row it wrote into the job summary said `unknown`, which is why the summary was
never worth reading.
The second half is independent of the naming. reclaim_stale_pool ran
`zpool destroy -f` as a bare command under `set -euo pipefail`, the only
failure path in this script that bypassed decline(). A pool that refuses to be
destroyed is by definition not a leftover - something holds it - so that case
now declines through the usual contract: a drawn pick falls back and reports,
a stated one fails loudly as required-but-unavailable. A reclaim can no longer
be more destructive than the state it exists to clean up.
Verified over the real invocation path rather than a model of it: an
ansible-playbook run using the same `script:` task and docker connection as
compose/swarm/playbook.yml, against three containers in one play, has each node
address its own pool - infinito_ci_race-mgr-01, -wrk-01, -nfs-01 - with
failed=0 throughout. A stubbed-pool harness covers the paths that need a kernel
module this machine does not have: a sibling holding the pool declines with rc
0 instead of ending the run, the same case with a stated pick exits 1 through
the contract, a genuine leftover is still destroyed, and a missing identity is
a hard error. The earlier replacement harness (ext4 recognised, btrfs
replacing it, no loop leaked) still passes against the stricter signature.
Pool names now carry dashes, which zfs_namecheck.c accepts; at 42 characters
they sit far inside the 240 limit and match no reserved prefix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role README claimed the backup machinery "operates transparently against the mount
paths" and demoted the source-of-truth rule to advice in a prose box - both written
before 2efc73d3b made that rule the code's behaviour. It now states what is enforced:
this role skips export-backed volumes, svc-bkp-nfs-2-local is their single capture, and
manager-pinned roles, roles forcing another compose_mode_force and nfs: false opt-outs
stay covered here. It also names the limit the code actually has - the exclusion checks
that some host is in the svc-bkp-nfs-2-local inventory group, not that the role ever ran
successfully - because a README promising more than the code delivers is worse than one
promising less.
Its Schema section carried the role's timer flow; it now carries the mechanism the
change set is about: meta/volumes.yml -> swarm_nfs_backed() as the single predicate ->
its two consumers (the compose rewrite and the backup exclusion) -> the inventory-group
gate -> the export host's single capture -> the restore path with its mountpoint probe
-> the drill's [3b] guard against over-exclusion.
The two swarm READMEs drew the drill with eight steps and pre-773a25ab6 semantics. Both
now draw the nine that exist, including what makes the proof a proof: the pulled tree is
wiped before the device recover, so the device holds the only copy left; the live export
is wiped with the marker proven gone before it is restored; and [9] deletes the marker
server-side before restoring the volume through its bound backing store.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three steps of the DR drill, one theme: every step now proves what it claims.
[9] restored into the volume's mountpoint while the stack was down - a bare directory,
because docker mounts a volume declaring its own backing store only while a container
holds it. The payload landed on the node disk and the real mount shadowed it on the next
deploy: success reported, nothing delivered, and roughly a gigabyte added to the very
device the drill later runs out of. It now deletes the marker server-side, binds the
backing store at the mountpoint, runs the recover CLI through it, and asserts the marker
on the NFS server's own disk. That also exercises the positive branch of the guard from
773a25ab6 - its refusal path is unit-tested, its pass-through was tested nowhere. When
the manager exposes no backing store the branch skips with a reason: docker volume
inspect is node-local, and with replicas: 1 the volume object usually lives elsewhere.
[3b] replaces a proof that used to happen by accident. The double capture 2efc73d3b
removed also showed, incidentally, that the volume unit really writes - the drill found
its marker in the volume repo too. The new step checks it directly: the newest
backup-docker-to-local generation must be at least as new as DRILL_START_TS (taken on
the manager, the same clock that names generations) and hold a non-empty file under a
volume subtree. Its failure message names the suspicion it exists for: an exclusion that
swallowed every volume. With no volume unit deployed it skips, tracked by
VOLUME_TRIGGERED alongside the existing SECRETS_TRIGGERED.
[5] prints du -sm per machine and repo when the space precondition fails, so the next red
run names what fills the pulled tree instead of leaving it to arithmetic.
Four adversarial review rounds found five defects in this block and confirmed each fix:
an NFS mount reconstructed from driver options the codebase never emits (it is a bind),
the recover CLI's second positional passed a volume name where it takes a host, a missing
volume object aborting the drill instead of skipping, and an umount ordered before the
return-code check, which under set -e would have killed the drill on a busy mount with no
diagnostic and left the bind behind.
None of this has run in a swarm yet. Every failure path is loud and names its cause.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Since the swarm-NFS migration two chains copied the same bytes: svc-bkp-nfs-2-local
reads the export from the NFS server's own filesystem, while baudolo on the manager
copied the same files again THROUGH the NFS mount. That was a migration leftover -
svc-bkp-nfs-2-local arrived in adb8bcf69 (2026-07-10) and nobody taught the older role
to step back - and the role README already forbade it as a rule nobody enforced. In the
DR drill the duplicate costs three times over: the pulled tree, the device copy and the
restore root.
swarm_nfs_backed() in utils/storage/nfs.py is now the single answer to "does the swarm
NFS rewrite back this volume". plugins/filter/compose_volumes.py (which performs the
rewrite) and this role's backup_volume lookup (which feeds baudolo's
--volumes-no-backup-required) both call it, so the two cannot drift. On the current tree
that excludes 83 of 158 type: volume entries.
Four conditions keep the exclusion from ever dropping a volume nobody else captures:
* a manager-pinned role's volumes are not on the export (web-app-seaweedfs holds
Nextcloud's object store, deliberately node-local per its meta/services.yml)
* a role forcing another mode via compose_mode_force is not on the export either.
That override is per role (roles/sys-svc-compose/defaults/main.yml:3), so it is
resolved per role inside the predicate from roles/<id>/vars/main.yml, not read once
from ambient scope. web-app-bigbluebutton pins compose statically;
web-app-matrix computes it, and a value the predicate cannot resolve counts as
forcing - unknown mode keeps the volume backed up
* an entry opting out with nfs: false stays node-local
* the exclusion applies at all only when the svc-bkp-nfs-2-local inventory group is
populated. That role enters a play by group membership alone
(tasks/groups/svc-bkp-roles.yml:19-22) and svc-storage-nfs-server ships a variant
with nfs_backup: {enabled: false}, so without the gate an export whose repository
nobody deployed would lose 83 volumes from both capture paths
Every condition fails toward the old double capture: wasted space, never lost data.
Compose mode is untouched - the exclusion cannot fire without swarm and an nfs backend.
DEPLOYMENT_MODE and storage are templated through the lookup's templar, because
group_vars/all/18_swarm.yml:2 defines the mode as a Jinja expression; comparing the raw
source string would silently disable the whole exclusion.
Twelve new tests pin the contract, including the two failure modes that would be
invisible in production: an untemplated mode, and an exclusion without an export
repository. Both were verified by mutation - removing either guard makes its test fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pick was made once per matrix entry, in a workflow step ahead of the drill,
and every distro of that entry then ran on it. Coverage followed: one entry
exercised one filesystem, and the swarm never reached zfs at all, because
resolve.sh's userland_for() kept it out of any entry containing arch, centos or
fedora - which is every automatic sweep. The draw now happens inside the loop in
distros.sh, once per distro, so a five-distro entry covers zfs with 87% and all
three kinds with 62% probability instead of one kind with certainty.
The workflow input changes meaning with it. It was a single kind, offered as a
choice in the manual entry; it is now a space-separated allow-list. Empty draws
from all three, and naming kinds narrows the draw to them and marks the pick
required, so a host that cannot deliver one of them fails rather than quietly
running on something else. A name that is not a filesystem this project puts a
data root on is rejected outright instead of reaching the applying step.
For the draw to mean anything the images have to carry the userlands, so
scripts/install/zfs.sh becomes scripts/install/filesystem.sh and installs ext4
and btrfs alongside zfs. That is not symmetric work: btrfs-progs does not exist
in the EL family at all, because RHEL dropped btrfs, and a plain
`dnf install btrfs-progs` there fails the whole transaction and takes e2fsprogs
with it. EPEL carries it, so the EL family goes through EPEL from the start -
split by distro rather than by a failed first attempt, since Fedora has no
epel-release to fall back to and routing a transient failure there would turn it
into a fatal one. verify() now demands all four binaries, so a packaging
regression fails the build instead of leaving a hole the drill discovers later.
Measured on all five bases: debian trixie, ubuntu 26.04, arch, fedora 44 and
centos 10 each end with mkfs.ext4, mkfs.btrfs, zfs and zpool present and ldd
fully resolved.
Two defects only became reachable once a single job could ask for different
filesystems in turn:
* docker_dataroot.sh answered "already prepared" and returned for any mount it
found, so from the second distro on, every draw would have been silently
ignored on a host that survives the sweep - the runner in compose and host
mode. It now compares what is mounted against what was asked for and
dismantles the old root when they differ. Everything docker cached there
goes with it; that is what covering more than one filesystem per entry
costs.
* reclaim_stale_pool ran only when the new pick was zfs, so a zfs to btrfs
transition left the previous distro's pool imported on the shared kernel,
pinning a loop device over an image its dead container had already unlinked.
It now runs on every pick, which needed the /dev/zfs handling lifted out of
the zfs branch into zfs_device_ready().
The restore_docker trap is armed before anything can stop docker, rather than
after, so a failure while dismantling a data root cannot leave a host without a
daemon. resolve.sh splits its allow-list into an array before testing it, so a
whitespace-only input is the empty list it looks like rather than a narrowed one
that yields an empty pick marked required. pick.sh clears both keys when it
declines to draw: they are declared in default.env, so a generated .env carries
whatever was exported when it was made, and a caller gating its apply on the
value would otherwise repoint the developer's own docker in the very case the
skip exists to prevent.
Verified by execution rather than by reading: a privileged container ran
docker_dataroot.sh with real loop devices and mkfs through ext4 -> ext4 -> btrfs
-> ext4, confirming the recognition, the replacement in both directions and that
no loop leaks across it; a second harness with a recording zpool stub confirmed
that a btrfs pick does reclaim a stale pool, that no pool means no destroy, and
that a host without /dev/zfs still completes an ext4 pick without touching zpool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dockerd's zfs graphdriver resolves `zfs` on PATH and opens /dev/zfs before it
accepts a data root (moby daemon/graphdriver/zfs/zfs.go:54,59), so the userland
has to sit in the image, not on the host. Only the apt images carried it, which
is why resolve.sh's userland_for() kept zfs out of every entry containing arch,
centos or fedora - that is every automatic swarm sweep, so the zfs data root and
the zfs branch of the backup launcher were never once exercised. Run
30781723102 shows it: "zfs left out: not every distro of this entry carries the
userland".
scripts/install/zfs.sh joins the three existing install scripts in the image
build and takes a different route per distro, each one run in a real container
of the matching base image before landing here:
debian trixie 2.3.2 zfsutils-linux, contrib enabled first
ubuntu 26.04 2.4.1 zfsutils-linux
arch 2.4.3 the AUR PKGBUILD, which configures --with-config=user
fedora 44 2.4.3 zfsonlinux repo plus a dummy zfs-kmod provider
centos 10 2.2.10 the same, from the epel path
The dummy is what keeps the RPM route usable: the `zfs` package carries an
ungated `Requires: %{name}-kmod`, so a plain `dnf install zfs` resolves through
zfs-dkms and installs 389 extra packages and 1.44 GB of kernel and toolchain to
build a module against a kernel the container never boots. Providing zfs-kmod
from an empty package keeps the same install at 60 packages with no kernel
package at all, and the transaction is checked afterwards so a regression fails
the build instead of bloating the image. The repository is removed again once
the userland is in, because the dummy pins one version and an upgrade would
otherwise re-resolve the kmod at deploy time.
Version drift against the host module is not a concern: zpool create enables a
feature only when the loaded module reports it through /sys/module/zfs/features
.pool (cmd/zpool/zpool_main.c, module/zcommon/zfeature_common.c), and libzfs
compares no versions at all (lib/libzfs/os/linux/libzfs_util_os.c:123-134), so a
newer userland degrades rather than breaking.
With every image carrying it, userland_for() and every_distro_carries() have
nothing left to decide and go. Two defects that only became reachable once zfs
could be drawn are fixed in the same move:
* /dev/zfs is a misc device, so the old lookup in /proc/devices could never
find its major and the node was never created. A node container's /dev is
populated at creation and udevd is masked, so nothing else fills the gap
either: the first bring-up of every job would have declined, blaming a
module that modprobe had just loaded. The minor now comes from /proc/misc.
* Pools live in the kernel the lab nodes share with the runner, while the
teardown only removes containers, and the node hostnames repeat across the
distro sweep. From the second distro on, zpool create hit the previous
iteration's pool. It is destroyed and its loop detached first.
The node image drops its own apt-only zfs branch and instead masks the six boot
units the userland enables by preset, which on a node can only fail: the pool is
created after boot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three CI entry workflows drop the "⚡ CI:" prefix for one emoji each -
🕹️ Manual, 🔀 Pull Request, 📤 Push - completing the trigger triad by symbol.
The manual run-name loses its prose prefix and separator dots; the segments now
frame themselves: 🕹️ for the dispatch, 🐧 before the distro sweep, ⭐ for the
priority line, 🎯 for a verbatim whitelist, 🔀 for diff derivation.
before: Manual CI triggered: debian arch; web-app-x ⭐; diff-derived (origin/main)
after: 🕹️🐧 debian arch ⭐ web-app-x 🔀 diff (origin/main)
Dropping the dots removes the fixed literal after inputs.distros that
utils/github/run_name.py cut the distro list at - the list is itself
space-separated, and the next segment starts with a branch-dependent emoji.
value_from_title therefore derives the terminator dynamically when the framing
literal is whitespace-only: _openings_after collects what any later template
segment can render first (the leading literal of each quoted string in later
expressions, format placeholders stripped, plus later inter-expression
literals) and cuts at the earliest occurrence. The SPOT property holds: the
openings are read from the workflow's own run-name, so a future emoji change
flows into the parser without touching it.
Titles from before this change no longer match the template and yield '' - the
documented unknown-title behaviour, so a retrigger of a pre-rename run falls
back to the workflow's distro default instead of guessing.
The dispatch CLI is unaffected (it targets entry-manual.yml by file name), job
names and the orchestrator are untouched, and no workflow_run trigger
references the old names. Required status checks configured against the old
workflow names in repository settings are the one surface this repo cannot
see - re-check them after the rename lands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
