The role shipped a complete probe script at files/test.sh and no
templates/test.env.j2. discover_cli_roles globs the env template alone,
so the role was never discovered and the script had never run — and the
script hard-requires TOR_SOCKS and NGINX_SERVERS_DIR via ${VAR:?}, so it
could not have run without one either. Its own usage hint pointed at a
path that does not exist.
Moving the script to the discovered path and adding the env template
makes both halves line up. Verified against the live stack: the onion
answers through the Tor SOCKS proxy, 1 probed, 0 failed.
The new lint keeps the pair together and rejects a script left at
files/test.sh. The two halves fail differently and only one is loud: an
env template without a script aborts the play, a script without an env
template is never discovered and its silence reads like a pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rsync -b turns a deletion into a rename, so a file the source dropped between
two writes of the same generation survives as name~ beside the real one and is
restored into live data. baudolo 3.4.1 fixes the copy that caused the
elasticsearch restore failure; this is the sweep over the sibling call sites
here.
svc-bkp-remote-2-local is the live one: the rsync command is assembled once and
re-executed up to twelve times against the same destination, which is created
with exist_ok before the loop starts. A retry after a partial transfer is
structurally the same second write as baudolo's hot/cold pair, only triggered by
--timeout instead of a container stop. That path pulls other hosts' backups, and
retries are its normal case rather than an exception.
svc-bkp-local-2-device cannot fire today - it writes one pass into a directory
named after the current second - but mkdir(exist_ok=True) makes a re-entry the
same two-write case, so the flag goes rather than the argument for keeping it.
The integration test mirrors the production command and follows.
The lint is what makes this complete: it scans every tracked .py/.sh/.j2/.yml
for a backup flag near an rsync token, and it found the integration test that
the manual sweep had missed. It excludes exactly one file, its own, because the
rule has to name the token it forbids; the sibling guard for || true avoids the
same problem only by scanning .sh alone.
Verified: make quality green across all four targets, the ssh-wrapper
integration test passes with real rsync, and test-lint goes 307 -> 308 with the
new guard collected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The inline repo definition pinned gpgkey to www.centos.org, a single
non-CDN origin. In run 30937913632 it was unreachable for at least
17m25s and killed 2 of 23 centos ganesha installs; dnf's own retries=5
was already active on the host and did not help.
centos-release-nfs-ganesha11 ships the same repo with metalink failover
and gpgkey=file://, planting the key through Requires:
centos-release-storage-common. Verified end to end against
quay.io/centos/centos:latest and ghcr.io/kevinveenbirkenbach/pkgmgr-centos:stable.
The BOOTSTRAP_BASEURL entry keeps the availability probe verifying both
names instead of degrading them to an inconclusive warning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rsync -b turns a deletion into a rename, so a file the source dropped between
two writes of the same generation survives as name~ beside the real one and is
restored into live data. baudolo 3.4.1 fixes the copy that caused the
elasticsearch restore failure; this is the sweep over the sibling call sites
here.
svc-bkp-remote-2-local is the live one: the rsync command is assembled once and
re-executed up to twelve times against the same destination, which is created
with exist_ok before the loop starts. A retry after a partial transfer is
structurally the same second write as baudolo's hot/cold pair, only triggered by
--timeout instead of a container stop. That path pulls other hosts' backups, and
retries are its normal case rather than an exception.
svc-bkp-local-2-device cannot fire today - it writes one pass into a directory
named after the current second - but mkdir(exist_ok=True) makes a re-entry the
same two-write case, so the flag goes rather than the argument for keeping it.
The integration test mirrors the production command and follows.
The lint is what makes this complete: it scans every tracked .py/.sh/.j2/.yml
for a backup flag near an rsync token, and it found the integration test that
the manual sweep had missed. It excludes exactly one file, its own, because the
rule has to name the token it forbids; the sibling guard for || true avoids the
same problem only by scanning .sh alone.
Verified: make quality green across all four targets, the ssh-wrapper
integration test passes with real rsync, and test-lint goes 307 -> 308 with the
new guard collected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Since the swarm-NFS migration two chains copied the same bytes: svc-bkp-nfs-2-local
reads the export from the NFS server's own filesystem, while baudolo on the manager
copied the same files again THROUGH the NFS mount. That was a migration leftover -
svc-bkp-nfs-2-local arrived in adb8bcf69 (2026-07-10) and nobody taught the older role
to step back - and the role README already forbade it as a rule nobody enforced. In the
DR drill the duplicate costs three times over: the pulled tree, the device copy and the
restore root.
swarm_nfs_backed() in utils/storage/nfs.py is now the single answer to "does the swarm
NFS rewrite back this volume". plugins/filter/compose_volumes.py (which performs the
rewrite) and this role's backup_volume lookup (which feeds baudolo's
--volumes-no-backup-required) both call it, so the two cannot drift. On the current tree
that excludes 83 of 158 type: volume entries.
Four conditions keep the exclusion from ever dropping a volume nobody else captures:
* a manager-pinned role's volumes are not on the export (web-app-seaweedfs holds
Nextcloud's object store, deliberately node-local per its meta/services.yml)
* a role forcing another mode via compose_mode_force is not on the export either.
That override is per role (roles/sys-svc-compose/defaults/main.yml:3), so it is
resolved per role inside the predicate from roles/<id>/vars/main.yml, not read once
from ambient scope. web-app-bigbluebutton pins compose statically;
web-app-matrix computes it, and a value the predicate cannot resolve counts as
forcing - unknown mode keeps the volume backed up
* an entry opting out with nfs: false stays node-local
* the exclusion applies at all only when the svc-bkp-nfs-2-local inventory group is
populated. That role enters a play by group membership alone
(tasks/groups/svc-bkp-roles.yml:19-22) and svc-storage-nfs-server ships a variant
with nfs_backup: {enabled: false}, so without the gate an export whose repository
nobody deployed would lose 83 volumes from both capture paths
Every condition fails toward the old double capture: wasted space, never lost data.
Compose mode is untouched - the exclusion cannot fire without swarm and an nfs backend.
DEPLOYMENT_MODE and storage are templated through the lookup's templar, because
group_vars/all/18_swarm.yml:2 defines the mode as a Jinja expression; comparing the raw
source string would silently disable the whole exclusion.
Twelve new tests pin the contract, including the two failure modes that would be
invisible in production: an untemplated mode, and an exclusion without an export
repository. Both were verified by mutation - removing either guard makes its test fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pick was made once per matrix entry, in a workflow step ahead of the drill,
and every distro of that entry then ran on it. Coverage followed: one entry
exercised one filesystem, and the swarm never reached zfs at all, because
resolve.sh's userland_for() kept it out of any entry containing arch, centos or
fedora - which is every automatic sweep. The draw now happens inside the loop in
distros.sh, once per distro, so a five-distro entry covers zfs with 87% and all
three kinds with 62% probability instead of one kind with certainty.
The workflow input changes meaning with it. It was a single kind, offered as a
choice in the manual entry; it is now a space-separated allow-list. Empty draws
from all three, and naming kinds narrows the draw to them and marks the pick
required, so a host that cannot deliver one of them fails rather than quietly
running on something else. A name that is not a filesystem this project puts a
data root on is rejected outright instead of reaching the applying step.
For the draw to mean anything the images have to carry the userlands, so
scripts/install/zfs.sh becomes scripts/install/filesystem.sh and installs ext4
and btrfs alongside zfs. That is not symmetric work: btrfs-progs does not exist
in the EL family at all, because RHEL dropped btrfs, and a plain
`dnf install btrfs-progs` there fails the whole transaction and takes e2fsprogs
with it. EPEL carries it, so the EL family goes through EPEL from the start -
split by distro rather than by a failed first attempt, since Fedora has no
epel-release to fall back to and routing a transient failure there would turn it
into a fatal one. verify() now demands all four binaries, so a packaging
regression fails the build instead of leaving a hole the drill discovers later.
Measured on all five bases: debian trixie, ubuntu 26.04, arch, fedora 44 and
centos 10 each end with mkfs.ext4, mkfs.btrfs, zfs and zpool present and ldd
fully resolved.
Two defects only became reachable once a single job could ask for different
filesystems in turn:
* docker_dataroot.sh answered "already prepared" and returned for any mount it
found, so from the second distro on, every draw would have been silently
ignored on a host that survives the sweep - the runner in compose and host
mode. It now compares what is mounted against what was asked for and
dismantles the old root when they differ. Everything docker cached there
goes with it; that is what covering more than one filesystem per entry
costs.
* reclaim_stale_pool ran only when the new pick was zfs, so a zfs to btrfs
transition left the previous distro's pool imported on the shared kernel,
pinning a loop device over an image its dead container had already unlinked.
It now runs on every pick, which needed the /dev/zfs handling lifted out of
the zfs branch into zfs_device_ready().
The restore_docker trap is armed before anything can stop docker, rather than
after, so a failure while dismantling a data root cannot leave a host without a
daemon. resolve.sh splits its allow-list into an array before testing it, so a
whitespace-only input is the empty list it looks like rather than a narrowed one
that yields an empty pick marked required. pick.sh clears both keys when it
declines to draw: they are declared in default.env, so a generated .env carries
whatever was exported when it was made, and a caller gating its apply on the
value would otherwise repoint the developer's own docker in the very case the
skip exists to prevent.
Verified by execution rather than by reading: a privileged container ran
docker_dataroot.sh with real loop devices and mkfs through ext4 -> ext4 -> btrfs
-> ext4, confirming the recognition, the replacement in both directions and that
no loop leaks across it; a second harness with a recording zpool stub confirmed
that a btrfs pick does reclaim a stale pool, that no pool means no destroy, and
that a host without /dev/zfs still completes an ext4 pick without touching zpool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dockerd's zfs graphdriver resolves `zfs` on PATH and opens /dev/zfs before it
accepts a data root (moby daemon/graphdriver/zfs/zfs.go:54,59), so the userland
has to sit in the image, not on the host. Only the apt images carried it, which
is why resolve.sh's userland_for() kept zfs out of every entry containing arch,
centos or fedora - that is every automatic swarm sweep, so the zfs data root and
the zfs branch of the backup launcher were never once exercised. Run
30781723102 shows it: "zfs left out: not every distro of this entry carries the
userland".
scripts/install/zfs.sh joins the three existing install scripts in the image
build and takes a different route per distro, each one run in a real container
of the matching base image before landing here:
debian trixie 2.3.2 zfsutils-linux, contrib enabled first
ubuntu 26.04 2.4.1 zfsutils-linux
arch 2.4.3 the AUR PKGBUILD, which configures --with-config=user
fedora 44 2.4.3 zfsonlinux repo plus a dummy zfs-kmod provider
centos 10 2.2.10 the same, from the epel path
The dummy is what keeps the RPM route usable: the `zfs` package carries an
ungated `Requires: %{name}-kmod`, so a plain `dnf install zfs` resolves through
zfs-dkms and installs 389 extra packages and 1.44 GB of kernel and toolchain to
build a module against a kernel the container never boots. Providing zfs-kmod
from an empty package keeps the same install at 60 packages with no kernel
package at all, and the transaction is checked afterwards so a regression fails
the build instead of bloating the image. The repository is removed again once
the userland is in, because the dummy pins one version and an upgrade would
otherwise re-resolve the kmod at deploy time.
Version drift against the host module is not a concern: zpool create enables a
feature only when the loaded module reports it through /sys/module/zfs/features
.pool (cmd/zpool/zpool_main.c, module/zcommon/zfeature_common.c), and libzfs
compares no versions at all (lib/libzfs/os/linux/libzfs_util_os.c:123-134), so a
newer userland degrades rather than breaking.
With every image carrying it, userland_for() and every_distro_carries() have
nothing left to decide and go. Two defects that only became reachable once zfs
could be drawn are fixed in the same move:
* /dev/zfs is a misc device, so the old lookup in /proc/devices could never
find its major and the node was never created. A node container's /dev is
populated at creation and udevd is masked, so nothing else fills the gap
either: the first bring-up of every job would have declined, blaming a
module that modprobe had just loaded. The minor now comes from /proc/misc.
* Pools live in the kernel the lab nodes share with the runner, while the
teardown only removes containers, and the node hostnames repeat across the
distro sweep. From the second distro on, zpool create hit the previous
iteration's pool. It is destroyed and its loop detached first.
The node image drops its own apt-only zfs branch and instead masks the six boot
units the userland enables by preset, which on a node can only fail: the pool is
created after boot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three CI entry workflows drop the "⚡ CI:" prefix for one emoji each -
🕹️ Manual, 🔀 Pull Request, 📤 Push - completing the trigger triad by symbol.
The manual run-name loses its prose prefix and separator dots; the segments now
frame themselves: 🕹️ for the dispatch, 🐧 before the distro sweep, ⭐ for the
priority line, 🎯 for a verbatim whitelist, 🔀 for diff derivation.
before: Manual CI triggered: debian arch; web-app-x ⭐; diff-derived (origin/main)
after: 🕹️🐧 debian arch ⭐ web-app-x 🔀 diff (origin/main)
Dropping the dots removes the fixed literal after inputs.distros that
utils/github/run_name.py cut the distro list at - the list is itself
space-separated, and the next segment starts with a branch-dependent emoji.
value_from_title therefore derives the terminator dynamically when the framing
literal is whitespace-only: _openings_after collects what any later template
segment can render first (the leading literal of each quoted string in later
expressions, format placeholders stripped, plus later inter-expression
literals) and cuts at the earliest occurrence. The SPOT property holds: the
openings are read from the workflow's own run-name, so a future emoji change
flows into the parser without touching it.
Titles from before this change no longer match the template and yield '' - the
documented unknown-title behaviour, so a retrigger of a pre-rename run falls
back to the workflow's distro default instead of guessing.
The dispatch CLI is unaffected (it targets entry-manual.yml by file name), job
names and the orchestrator are untouched, and no workflow_run trigger
references the old names. Required status checks configured against the old
workflow names in repository settings are the one surface this repo cannot
see - re-check them after the rename lands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Docker mounts a volume that declares its own backing store (Options set: nfs,
cifs, bind) only while a container holds it. recover.py resolved the volume's
Mountpoint and rsynced into it regardless, so with the consuming stack stopped -
the state its own docstring demands - the payload landed on the node's disk
under the bare mountpoint directory, and the next stack start mounted the real
backing store over it: a restore that reports success, delivers nothing, and
leaves an invisible, unreclaimable orphan behind.
That orphan had weight. In run 30859490335 the swarm drill's [9/9] volume leg
restored ~1 GiB of nextcloud_data onto the manager's 30 GiB loop device in
exactly the minute the device crossed SeaweedFS's 2 GiB minFreeSpace floor -
the last straw on a device already >=93% consumed (the sizing is ea14535f4) -
after which every authenticated Nextcloud request died on an unwritable object
store.
recover.py now probes the mountpoint with `mountpoint -q` (over ssh for
--docker-host) whenever the volume declares Options, and dies naming the
mechanism and the way out instead of writing a single byte. Plain local volumes
without Options keep the old path untouched - their _data directory on the node
disk is the legitimate target.
The drill skips its [9/9] volume leg for such volumes with the reason in the
log: its data restore already happens in [8/9] straight into the NFS export,
which is the restore path the role's README prescribes for NFS-backed volumes
on swarm. The leg never proved anything for them - it only filled the disk.
Three new unit tests pin the contract: unmounted refused, mounted accepted,
remote probed via ssh with the exact command line.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four roles declared auth_subject: user while handing the client an
account-bound credential. Nextcloud and Home Assistant mint against the
administrator account, Mattermost issues the administrator's personal access
token, and Baserow creates a dedicated mcp@<domain> account with its own
workspace. The declarations now name what the minting tasks do; Baserow's
README additionally claimed the administrator owns its endpoint, which it
never did.
The tools declarations follow one convention instead of two: the field records
the deployment's intent and the README names the missing enforcement, which is
what GitLab already did and what Baserow and Mattermost now do.
The discovery lookup carries auth_subject through so clients can see the
subject. No consumer branches on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight server roles declare an `mcp` RBAC role, so the platform mints
roles/<app>/mcp and the administrator joins it. Open WebUI now honours those
groups instead of flattening them to user/admin, and each MCP tool server is
restricted to the group of the role that serves it.
The grant cannot be rendered: Open WebUI matches it against a group id that
insert_new_group mints as a fresh UUID, so tasks/01_mcp.yml resolves the group
after the container is up and writes the grant through the admin API, the way
every instance-minted credential in this repository is provisioned.
Entries render disabled. An empty access_grants is not 'nobody', Open WebUI
reads it as every administrator, and ENABLE_PERSISTENT_CONFIG=false makes the
env authoritative again on every start. Enabling only together with the grant
keeps the fallback at no access rather than too much.
The provisioning account is the role-internal openwebui-api-bot, never a
platform identity: OAUTH_MERGE_ACCOUNTS_BY_EMAIL ships disabled and an OIDC
login whose email a local account already holds is refused, so seeding a
person's address would lock them out.
Known limits are documented rather than claimed away: a bare restart leaves MCP
off until the next deploy, a user's roles list carries no application context so
roles: [mcp] grants every deployed mcp group at once, and Open WebUI drops a
stale membership only while the groups claim is non-empty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Leftovers from extracting lookup_context and attachments: the mock.patch target
got short enough to fit on one line again, and the file split left a spare blank
line in each module. Whitespace only, produced by autoformat during make
quality, so the 298 tests of that run already covered this state.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two places decided independently what the pre-created <entity> network is called
inside the compose project. render.py emits the key; create.yml asserted it was
always `default`. They diverged in July and nobody noticed, because compose
5.3.1 resolved networks by name and never read the claim.
Compose 5.4.0, published 2026-08-03T14:39:12Z, resolves them by label. For
web-app-seaweedfs - the one deployed role that provides its own shared_net and
therefore renders an unnamed `default` - the mislabelled `seaweedfs` network is
adopted as the project default, `seaweedfs_default` is never created, and the
containers fail to start against it. That failed web-app-nextcloud 0 and 3 in
run 30830377785.
_shared_network_key() is now the single derivation. render_compose_networks
takes its branch from it, so the rendering decision IS the key decision;
shared_network_compose_key() exports the same value, the shared_network_key
lookup hands it to Ansible, and create.yml stamps it: `seaweedfs` for a
self-provider, `default` for everyone else. Both true, both from the source that
renders the file.
Verified on one host with only the compose binary swapped:
5.4.0 network=default -> FAIL, network <project>_default not found
5.4.0 network=<entity> -> OK
5.3.1 network=default -> OK
5.3.1 network=<entity> -> OK
so this is not a fix that trades one version for the other.
test_key_matches_the_key_the_renderer_emits renders the block and asserts the
reported key appears in it, for both role shapes and both modes. The drift that
sat unnoticed for a month now breaks a test.
The move of the attachment computation into utils/networks/attachments.py rides
along because render.py would otherwise be broken between two commits: it is a
pure relocation, re-exported so no importer changes, and it brings the file from
352 to 243 lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
compose_networks, container_networks and compose_external_networks each rebuilt
the same Ansible-side context before calling their renderer: resolve the
variables, settle the deployment mode, load applications, build the service
registry, and close over the config and database lookups. Around forty lines,
three times, each with its own copy of _resolve_var. A change to how the
registry or the closures are obtained could reach one lookup and miss another.
build_context() returns that context once. honour_mode_force carries the only
real difference between the callers: compose_external_networks never consulted
compose_mode_force, and still does not.
The lookups keep only what is theirs - term validation, their own arguments,
their renderer call - and drop from 94/90/87 lines to 59/54/53.
No assertion changed in the tests; only the mock targets moved with the seam,
from the plugin modules to utils.networks.lookup_context. Twelve legacy comments
in those test files lost their grandfathering when the files entered the lint's
diff scope: two became docstrings naming the call contract, the rest were
narration and are gone.
Behaviour-preserving: the same 13 lookup tests pass before and after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No role declared it, no client rendered it and no probe read it. The key only
put a "health_path": None into every discovery entry and suggested a capability
that does not exist.
Removing it from the vocabulary, the lookup and the lookup test has to happen
together: the schema lint would keep accepting it, the entry would keep carrying
the None, and the test compares the exact key set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The loop only left when average_version_directories dropped to 0. With exactly
one version per application that value is 1, yet every version is also its
application's last one and therefore undeletable. A pass freed nothing, disk
usage never moved, and the job spun until the time limit raised, exiting 1 and
failing the deploy.
Count the deletions and leave when a full pass reaches zero. A cleanup that did
everything it is allowed to do has succeeded; reporting a full disk belongs to
sys-ctl-hlth-disc, so the threshold stays untouched.
The regression test needed an advancing clock: with time.time pinned the limit
never fires, so a regression would hang the suite instead of failing it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
autoformat joined the two operands of the message concatenation during
make quality; 1408c6770 had them on separate lines. Formatting only, same
expression and same output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule checked whether both keys contain the token `omit`. That catches the
Ollama shape, where async carried no omit at all, but it does not enforce what
the failure message claims: two keys can both carry omit and still hang on
different switches, and the old check passed that.
_gate() now reads the switch each key states inline - the test of an
`if ... else omit`, or the left side of a `| ternary(..., omit)` - and the two
must match. A key that names no gate matches another that names none, which is
the shape sys-service and the canonical ASYNC_TIME/ASYNC_POLL pair use; both
then inherit whatever their variables do. An unrecognised omit-bearing form
falls into a single bucket, so it only matches another unrecognised form.
Checked against all five shapes in the tree: the pre-fix Ollama pair is an
offender, the fixed pair and the ternary and bare-variable pairs are not, and
two different conditions are now an offender where the token check passed them.
The rule still does not catch two ungated keys whose variables diverge. That is
deliberate: group_vars makes ASYNC_TIME evaluate to omit while ASYNC_POLL stays
numeric, and a poll without async is ignored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pre-pull task set `async` unconditionally while gating only `poll`. With
async disabled, `poll` became `omit` and ansible fell back to its built-in 15s
poll while `async` still routed the task through async_wrapper: ansible
collected the job itself and then ran `async_status mode=cleanup`, deleting the
job file that tasks/utils/async/reap.yml expects to read. The reap reported a
successful `ollama pull` as a failed async job, failing web-app-nextcloud 2 in
run 30781723102.
Both keys are now switched off together, matching all six peer call sites. On
the resulting synchronous path the reap's `async_reap_tolerate` list no longer
applies, so the tolerated "up to date" answer is carved out in `failed_when`
instead. It keys on `failed` rather than `rc`, so a module-level failure that
carries no rc is not swallowed.
async_failures._which() now honours loop_control.loop_var: the fired result
records the chosen name under `ansible_loop_var`, so a batch failure names
`smollm2:135m` instead of the bare job id.
A second lint case asserts that async and poll carry `omit` together - the rule
this call site was the only one of seven to break.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merged branch renumbered the Nextcloud tasks to 01/02, leaving the
MCP tasks stranded at 09 with a gap the numbering lint rejects; rename
them to 03 and follow every reference.
The Mattermost select statements only formed a prefix group once the
incoming branch's own select landed beside them, so they move into
sql/select/, and the two user-scoped ones again into sql/select/user/.
Pin the object kind on the kata isolation inspect: this repository names
a role's overlay network after the same entity as its main container, so
an untyped inspect can resolve the network and report a missing State key
instead of the missing container. Suppress the address-lookup rule there
because both callers gate the assertion on compose, where the container
name is the address.
Give the sandbox-runtime test the data-root variable the incoming
daemon.json template reads. Both sides were green alone: their template
gained the variable, this test renders it under StrictUndefined.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A service without a healthcheck.flavor or an explicit test only breaks
while its compose file renders, one role at a time during a deploy.
Replay the lookup's own validation against the static config so the
failure surfaces in CI, and reject declarations the lookup would refuse
anyway: an unknown flavor, or a flavor and a test fighting over the
same entry.
Both tests share _probe_defect and import PROBES from the same SPOT the
lookup uses, so the known flavors cannot drift apart. 49 of the 56 call
sites resolve statically; the rest set service_name in an included base
template and are skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
container_healthcheck resolves application_id from the play vars
instead of taking it as a positional term. Vars handed over by
include_role arrive unrendered, so under sys-stk-full the lookup got
the literal '{{ sys_stk_full_application_id }}', resolved no app and
refused to render the dashboard compose file.
Template such values before use. container_hostname had the same
defect: it is a Jinja string in every role's vars and was baked
verbatim into the curl Host header. Raise when a value survives
templating unrendered, since ansible-core returns untrusted templates
as-is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven include templates under sys-svc-container decided how a container was
probed, and they read their inputs from template-local `{% set %}` scope.
That scope leaks: a value set for one service silently reached every service
below it, which is how taiga's front and gateway ended up probing a port
declared for a different service, and how bluesky's frontend probed a PDS
endpoint.
Probes now live in utils/docker/healthcheck.py, one class per flavor behind a
PROBES registry, and each service declares its own:
healthcheck:
flavor: curl
path: health/ready
start_period: 20m
`port_key` picks another entry under ports.internal for the services that
expose their probe elsewhere (keycloak on management, nextcloud talk on
signaling). Without a flavor the entry must carry an explicit test argv, so a
missing declaration fails loudly instead of rendering a probe against nothing.
Rendering HEAD's templates against the new output confirms 49 of 56 call
sites are byte-identical; the remaining seven are covered by an Ansible run.
Three probes change on purpose: erpnext's backend gains a check it never had,
bluesky's frontend drops the leaked PDS path, and msmtp_curl moves from a
hardcoded localhost:80 to the service's own port on 127.0.0.1 like every
other flavor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The applications registry is built before any role-vars binding, so a
`{{ SOME_ROLE_VAR }}` in meta/*.yml stays an unrendered Jinja string and
silently resolves to nothing. The sibling rule already covers
`application_id`; this extends it to every key declared under
`roles/*/vars/`.
String literals, keyword arguments and filter or attribute names are
stripped before matching, so `lookup('domain', ...)` and
`lookup('rbac_group_path', application_id=...)` are not mistaken for
variable reads. Scope stays at meta/*.yml: files under meta/addons/ are
rendered inside the role's own play, where role vars are bound.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`shell` populates stdout on every path where changed_when is evaluated,
so the default filter guarded nothing: a non-zero exit and even a chdir
failure both carry stdout, a skipped task never evaluates the condition,
and on a module failure Ansible records the conditional error in
changed_when_result while still reporting the real cause.
The async lint walked the tree itself instead of going through the
shared cache. Both selections cover the same 1079 files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An async task is a deployment wide decision taken in one line. Whether it
holds depends on things the line does not show: in swarm a delegated
command crosses a node boundary and runs far longer than the same command
in compose, so a window that never opens on one node routinely opens
across three, and the playwright run starts the moment the play ends, so a
command still writing in the background surfaces as a flaky spec rather
than a failed task.
Require one sentence per async naming what does not consume the result.
"It is slow" is why async is tempting, never why it is safe, and the
docstring says so. There is no nocheck escape: a task that cannot be given
the sentence has not earned its async.
The comment lint learns `rationale:` as a marker so those lines are not
read as narration. It is deliberately narrower than the exception marker
next to it: a rationale is demanded by another lint that refuses the
construct without it, so it cannot be self granted, which is exactly the
failure mode of comments that explain themselves.
This lands after the change it describes, so every commit stays green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repo's async convention is fire and forget: ASYNC_POLL is 0 whenever
ASYNC_ENABLED is true, and the swarm test lane forces it true through
extra vars. A task dispatched that way returns a stub, so a changed_when
or failed_when reading its own register evaluates against nothing and can
never report changed and never fail. Several tasks carried a
"not ASYNC_ENABLED and" prefix in front of those conditions, added to
avoid touching undefined rc and stdout, which turned the guard off
entirely rather than adapting it.
That is how the nextcloud plugin configuration stayed broken unseen: the
batch reported ok after 0.8s while its delegated commands kept running,
the next task read the half written configuration, and no failure in any
of them could surface.
Every async in the repo was decided individually rather than swept. Where
nothing consumes the result, the async stays and a one line rationale now
records what does not read it. Where a later task, a handler or a browser
does consume it, either the async goes or the result is collected.
Collecting is what the new reaper does. It waits for a whole batch, then
reports every broken job at once instead of aborting on the first, so one
run shows whether a single item or the whole set is broken. Deciding what
counts as broken lives in a filter plugin, because the distinction is
subtle enough to deserve tests: a module reports failure in msg, a command
in rc and stdout, an idempotent API answers "already exists" and means
success, and a job that cannot be read at all comes back marked finished
with no outcome, which would otherwise read as green.
Timing cost is roughly one to two minutes on a deploy measured in hours.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, each resolved by combining both sides rather than picking one.
compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.
daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.
The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The wall-clock budgets in this suite asserted a property the test cannot
control. `make test` dispatches four suites concurrently through
scripts/make/parallel.sh, each running `pytest -n auto`, so on a 12-core box
up to 48 workers contend for 12 cores and a 14s single-threaded render
stretches to anywhere between 65s and 141s. Five consecutive local gate runs
failed here on four different assertions while CI, whose integration suite was
green throughout, saw nothing.
Measurement, not guesswork: the render costs ~14s of CPU and wall matches CPU
almost exactly when the suite runs alone, so CPU seconds are the contention-
invariant quantity. An A/B against the 2026-07-30 tree renders 12% slower than
HEAD, which rules out a regression as the cause of the breaches.
The assertions therefore measure `time.process_time()`, and both cold-render
ceilings collapse into one 30s CPU budget. That is stricter than the 60s and
100s wall-clock ceilings it replaces, since CPU time never exceeds wall time.
The cold-to-hot ratio assertion is untouched: numerator and denominator are
sampled in the same process under the same load, so it was already immune.
The suite moves to tests/performance/, which takes it out of the `test`
fan-out because run.sh derives the suite directory from the test type, and
gains a `test-performance` target plus the alias the lint requires. What
remains in the commit gate is deterministic.
Two redundant cold renders are gone with it: the applications and domains
budgets each had a second test that re-rendered the same state purely to
assert a ceiling the first test had already measured. All eight assertions
survive as subTests, with each cold-render check placed immediately after its
measurement so an exception in the hot loop cannot swallow its verdict. The
file drops from 88s to 49s.
Read platform_arch from the role config and append a node.platform.arch
constraint where it is set, so a role whose image exists for one
architecture only cannot be scheduled onto a node that could never run
it. LM Studio ships x86_64 alone and is the first consumer.
The constraint lives with the other placement rules rather than in the
consuming role, because architecture binding is a property any role with
a platform-specific image can carry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six roles have an upstream MCP path this deployment cannot use, so they
declare a blocker instead of a direction. Without it they would read as
"upstream has no MCP", which is false for all of them and worst for
OpenProject: the server ships, it is Enterprise-only, and upstream's own
request specification asserts a 404 on the Community edition this role
deploys. Jira, Confluence, WordPress, Odoo and Discourse are hosted-only
or unreviewed third-party paths.
The dynamic-flag suppressions are load-bearing: a blocker block is
mutually exclusive with a served surface, so a group_names expression
there could never evaluate true.
Add the test that keeps a served surface honest: every role declaring one
must gate it on the role's MCP-enabled flag, so switching MCP off removes
the endpoint instead of leaving it reachable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add a blocker vocabulary so a role whose upstream MCP path this
deployment cannot use says why instead of reporting an absence. Without
it OpenProject reads as "upstream has no MCP", which is false: the server
ships, it is Enterprise-only. A blocker and a served surface are mutually
exclusive, so a role cannot claim both.
Add endpoint.key_credential and endpoint.suffix for servers that address
a session through the URL path. Baserow streams from
{base}/mcp/{key}/sse, and endpoint fields must stay literal, so the
composition belongs in the discovery layer. A server that declares a path
key without a resolvable value is dropped rather than handed to clients
half-built.
Carry the upstream provenance in the role block, so the audit derives it
instead of keeping a second copy.
Require an explicit mutating_tools_enabled: false for a privileged
auth_subject; the rule previously only fired when the key was present and
true, so omitting it slipped an administrator-scoped surface through.
Move the MCP filters into plugins/filter/mcp/ so they stop sharing a name
prefix with their siblings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The suite rendered the merged applications tree twice, once per scenario.
Under xdist that competed with the runtime-performance budgets and pushed
the cold applications render from well under its 60s budget to 96s.
Both scenarios now resolve against a single render: the second lookup call
hits the applications cache, so the file costs one render instead of two
and its runtime halves.
Each scenario keeps exactly one consumer, which is what makes the volume
bound bite -- with one consumer the budget is two grow batches and dropping
that consumer leaves one, which is the budget that starved the matrix
collection. Merging the scenarios into a single set would have made the
bound tolerate a dropped consumer. The consumer sets are now asserted by
equality rather than membership, so a role that is silently added is caught
as well as one that is dropped.
The role path and the file read go through the ROLE_FILE_META_SERVICES
constant and the cached reader, as the repository's own lints require.
The lint walked each role's tasks directory itself, once per role. The
project's own test_no_raw_project_walk rejects that: utils.cache.files
shares a single filesystem walk and one read per file across the whole
pytest session, so 251 separate directory traversals are 250 too many.
Roles are now identified from the relative path of each yielded file
instead of by iterating the roles directory. The scanned set is unchanged
at 806 files, with empty differences in both directions, and the
unresolvable-target injection still produces the expected finding.
The consumer count came from roles_with_service, which builds dashboard card
targets and therefore requires a single canonical domain. web-app-matrix
declares domains.canonical as a mapping (synapse, element) and was dropped;
in a deployment where it was the only consumer the count reached zero, the
server started with -volume.max=7, the unnamed default collection took all
seven slots on first write, and every S3 PUT for the matrix collection failed
with "No writable volumes and no free volumes left". Synapse answered 500 on
media upload and no avatar ever reached the bucket. web-app-openwebui carries
the same mapping shape and the same latent failure.
Consumers are now enumerated by the binding itself: a role is a consumer when
its merged config binds the engine and marks it shared, whatever the role is
called and however many domains it serves. The S3 identity file and the volume
budget read that one list, so the identities granted write access and the
slots budgeted for their collections cannot drift apart.
Selection reads the merged applications config rather than meta/services.yml,
because an operator may bind a store through the applications inventory
variable alone; that path is covered by a test so a later optimisation cannot
quietly reintroduce a naming or declaration assumption.
main.yml now reads as the chronological sequence it describes: it includes
01_fullstack.yml, then the single IS_STACK_HOST-gated 02_manager_ops.yml,
whose substeps live in the matching 02_manager_ops/ directory numbered in
execution order. The plugin pipeline moves under its parent step so its
include resolves against its own directory instead of an inherited search
root.
The flush that deploys the stack moves out of the config step to the top of
02_manager_ops.yml. It previously ran after the swarm resolver, which then
waited 600 times for a service that did not exist yet and killed two CI jobs
with "no such service: nextcloud_nextcloud". The second flush and the
re-resolve behind it are removed: neither included file notifies a handler,
so they could never fire.
A new lint pins what made this refactor risky. A relative include is resolved
at runtime, so a typo or a moved file survives ansible-lint and
--syntax-check and only fails mid-deploy; worse, a bare name that matches
under two search roots silently shadows one file with the other.
Comments that only narrated code are dropped from the addon hooks and the
plugin install step.
apply is a parameter of the include_tasks action, not a task keyword. Written as a sibling of the module Ansible sees two action statements and aborts: 'conflicting action statements: include_tasks, apply' at 08_manager_ops.yml:9. All five delegated includes from a963280c1 carried it that way.
Nothing static caught it. ansible-lint passes the file, and ansible-playbook --syntax-check - which scripts/lint/ansible.sh runs and make quality-high covers - reports 0 failures against the broken tree, because a task file reached through a dynamic include_tasks is first parsed at runtime. The play therefore died 26 minutes into the deploy, after everything before it had run.
The new lint reads every task file on its own and rejects apply beside a module, which is never valid. Verified both ways: the sibling form is reported, the nested form is not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f956de889 made the slot ceiling generous, which removed the accidental brake the old autosizing had provided: while the ceiling was derived from free disk space, growth stopped long before the disk filled. With a fixed ceiling the only thing between growth and ENOSPC is the free-space guard, and its stock 1% is 300 MB on a 30 GB data root - less than the 1024 MB unit SeaweedFS allocates in, so a single growing volume can outrun it.
The floor is now expressed as whole allocation units rather than a percentage, which is meaningless without knowing the disk size. Two units: one for the growth in flight, one for concurrent writers. The size limit is stated alongside it so both come from one constant instead of the floor deriving from a value the image injects - verified that our flag wins, since -master.volumeSizeLimitMB=512 makes the master report 512 MB.
-volume.minFreeSpace replaces the deprecated -volume.minFreeSpacePercent, per the image's own help text. The rendered command was run against chrislusf/seaweedfs:4.39: master, volume server and filer all start.
Left open deliberately: web-app-seaweedfs declares min_storage: 304MB, which cannot hold with a 2 GiB floor. That value feeds the CI storage filter in scripts/meta/resolve/apps.sh, where 60GB is the budget, so nothing is dropped today - but the declaration understates what the role now needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
a3fcd3e7f stopped the entrypoint from autosizing the ceiling out of free disk space, but replaced it with a flat 64 that I adopted without deriving. Each bucket is a SeaweedFS collection and the master grows one in batches of seven - 'create 7 volume' in the artifact, copy_1 = 7 in the image's own master scaffold - so 64 covers nine collections. Twenty-five roles declare the service, and the tenth consumer would have hit 'No writable volumes and no free volumes left' again, just later and without the disk-space link.
The ceiling now comes from volume_slots(): one grow batch per bucket plus one for the default collection, with the consumer count read from roles_with_service. Note what that count is: the consumer gate gets whether seaweedfs is in group_names, not whether the consumer itself is installed, so the value is derived from the repository rather than from the host. It grows with the codebase, which is what the ceiling has to keep up with.
The collection count is a required parameter with no default. A silent fallback would be worse here than a loud failure: without the flag the stock entrypoint injects -volume.max=0 and the autosizing trap is back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role passed no -volume.max, so the stock entrypoint injects -volume.max=0, which means 'derive the ceiling from free disk space'. That derivation charges every non-full volume its full volumeSizeLimit: the master's default grow batch creates seven 1 GB volumes on the first write, reserving ~7.0 GB, and the manager's docker data root had 6.8 GB free. unclaimedSpaces goes non-positive, maxVolumeCount collapses to volumeCount, and the node reports zero free slots.
Those seven volumes belong to the default collection, so they can never serve collection 'nextcloud'. The master says it plainly - 'topo failed to pick 1 from 0 node candidates', 'create 7 volume, created 0' - and every S3 PUT returns 500 while seven empty volumes sit on disk. The volume server never left the topology: RestartCount 0, one unbroken log stream across the window.
That 500 is what starts the web-app-nextcloud 0 chain: occ maintenance:install fails eleven times, leaves a half-installed instance, and slot 1 then holds nextcloud-init-sync.lock for four minutes while every other replica dies on a blocking flock. Reproduced independently in variant 5 with collection 'peertube'; both compose jobs stayed green.
-volume.max makes isArgPassed true, so the entrypoint stops injecting the autosizing value. Slots become a fixed budget rather than a function of disk space; volumes stay sparse and unpreallocated, and -volume.minFreeSpacePercent still guards a genuinely full disk.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nfs(5) lists both 'nolock' and 'local_lock=' under "Options for NFS versions 2 and 3 only". NFSv4 carries locking in the protocol and has no NLM sideband to switch off, so the kernel accepts the token and reports local_lock=none. mount_opts had the condition inverted: the branch that can work sat on v3, the inert one on v4 - and svc-storage-nfs-server pins nfs_version: 4, so the effective locking configuration was always the dead branch.
The artifact proves it landed nowhere: the mount module ran with local_lock=flock in opts on every node, every other token from that same string is present in /proc/mounts (soft, timeo=50, retrans=3, _netdev), and only local_lock reads 'none'.
The false premise had spread. docker-entrypoint-suitecrm.sh justified its mkdir mutex with 'the swarm NFS mount forces local_lock=flock, so flock(2) never crosses nodes'; that comment is removed. The mkdir mutex stays correct for a different reason: against this ganesha export a non-blocking flock works while a blocking one returns EIO.
Nothing about the effective mount changes - the option was already being discarded. On an existing host the first run rewrites fstab and remounts /var/lib/infinito once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_any_bridge_partner_deployed matched a bridge against deployed role ids by equality or '-<bridge>' suffix. That reaches a partner whose entity carries the name (matrix -> web-app-matrix) but never one registered under a service key: 'sso' is provided by web-app-keycloak and 'ldap' by svc-db-openldap, neither of which ends in the bridge name. Those five declarations therefore evaluated to false in every run, so their addon flags were false everywhere and skipUnlessAddonEnabled skipped the specs in every variant - silently, because a gate that never opens looks exactly like a variant that does not apply.
The lookup now resolves a bridge through build_service_registry_from_applications first and keeps the entity-suffix rule as the fallback, so both spellings work. Existing bridge tests are unaffected: their applications stub yields an empty registry and takes the old path.
The new lint asserts that every bridges: value names a service some meta/services.yml declares. It fails on a typo or an invented name while staying agnostic about which run deploys the service, which is the lookup's question at runtime. Verified both ways: no findings against the real service set, thirty against an empty one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
