Mastodon moved its reserved-usernames list out of config/settings.yml into the
username_blocks table, seeded on every fresh install by
db/seeds/05_blocked_usernames.rb - 'administrator' included, along with admin,
root, owner, mod, staff and more. Shipped in v4.5.0 (2025-11-06); this repo is
exposed since 5a041982f pinned v4.6.2 on 2026-07-02, because the same-day
db:prepare migrate service seeds the fresh database.
The role's pre-create task still ran `sed -i '/- administrator/d'
config/settings.yml` - the old location, which at v4.6.2 holds no reserved list
at all. sed exits 0 on a non-matching pattern, so the task reported success
while doing nothing, and `tootctl accounts create administrator` died on the
UnreservedUsernameValidator with the output censored by no_log. That failed the
Instructions job in run 30884701334 (job 91914016159). Diagnosed against
upstream sources at both tags; run 29150751430 in the org repo is the control -
its image predated v4.5 (db:seed rc=0 one minute before a successful create,
impossible with the seed present), which is why the failure never showed there.
The sed becomes a rails-runner unblock at the new location, following the
stdin-fed pattern the no-inline-rails-runner lint prescribes (see
web-app-decidim/tasks/01_setup.yml): the Ruby lives in
files/ruby/unblock_admin_username.rb, reads the username via ENV.fetch, deletes
its username_blocks rows and prints the count; changed_when keys on that count.
The old `when: username == "administrator"` guard is gone - the seed also
blocks admin, root, owner and others, so any configured admin name can collide.
The create task's "taken" tolerance for re-runs is untouched.
Not covered: whatever waits behind the create - approve, plugins and the
playwright specs have not run with an existing admin since the pin.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Docker mounts a volume that declares its own backing store (Options set: nfs,
cifs, bind) only while a container holds it. recover.py resolved the volume's
Mountpoint and rsynced into it regardless, so with the consuming stack stopped -
the state its own docstring demands - the payload landed on the node's disk
under the bare mountpoint directory, and the next stack start mounted the real
backing store over it: a restore that reports success, delivers nothing, and
leaves an invisible, unreclaimable orphan behind.
That orphan had weight. In run 30859490335 the swarm drill's [9/9] volume leg
restored ~1 GiB of nextcloud_data onto the manager's 30 GiB loop device in
exactly the minute the device crossed SeaweedFS's 2 GiB minFreeSpace floor -
the last straw on a device already >=93% consumed (the sizing is ea14535f4) -
after which every authenticated Nextcloud request died on an unwritable object
store.
recover.py now probes the mountpoint with `mountpoint -q` (over ssh for
--docker-host) whenever the volume declares Options, and dies naming the
mechanism and the way out instead of writing a single byte. Plain local volumes
without Options keep the old path untouched - their _data directory on the node
disk is the legitimate target.
The drill skips its [9/9] volume leg for such volumes with the reason in the
log: its data restore already happens in [8/9] straight into the NFS export,
which is the restore path the role's README prescribes for NFS-backed volumes
on swarm. The leg never proved anything for them - it only filled the disk.
Three new unit tests pin the contract: unmounted refused, mounted accepted,
remote probed via ssh with the exact command line.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every lab node got an identical 30 GiB loop device because docker_dataroot.sh
defaulted SIZE to 30G and no caller ever stated it. The swarm manager hosts the
in-cluster registry, ~60 containers, and - by deliberate placement, since /data
is NFS-unsafe - the SeaweedFS object store, whose sizing filter claims 14 GiB
of slots plus a 2 GiB minFreeSpace floor. The idle nfs-server node got the same
30 GiB and used 1% of it.
In run 30859490335 the manager's device stood at 30G/30G (98%), SeaweedFS
crossed its floor at 02:04:45 and marked all 14 volumes unwritable, permanently.
Nextcloud in variant 0 stores objects in that SeaweedFS via OBJECTSTORE_S3_*,
so every authenticated page render performed an object write that failed as a
10-attempt assign ladder - 15 requests, 15 bursts of exactly 10, burst end
matching the nginx 500 to the second. That is the "🐝 web-app-nextcloud 0"
failure: 5 addon specs, 3 attempts each, all dead on the post-login page.
The default is gone: SIZE is now a required argument and a missing value dies
naming the call site. The swarm playbook states it per node - 60G for the
manager, 30G for the rest. 60G covers the observed demand (>=31 GiB: a full
device plus the denied writes) with headroom; the image is sparse, so the
raised ceiling costs only what is actually written, and the runner-side
watchdog at 6 GiB free remains the backstop. The compose and host workflows
state their unchanged 30G explicitly.
Not addressed here: what fills the first 28 GiB (registry + image layers +
volumes composition is unmeasured - the rescue artifact holds exactly one df
snapshot), and the drill-side shadow restore that supplied the last ~1 GiB;
both are separate changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nextcloud's log_type was never set, so the default 'file' applied and every
application exception went to data/nextcloud.log on the NFS volume. The rescue
diagnostics collect `docker logs` per service, which only ever saw the FPM
access log - so an exception is invisible in CI artifacts.
That cost a diagnosis. In run 30859490335, job 91840410448, the OIDC login
completed (302 /login -> 302 /apps/oidc_login/oidc -> 303 with the code) and
/apps/dashboard/ answered 500. The access log counts 15 x 303 against 15 x 500,
i.e. five addon specs times three attempts, every single authenticated request
failing - deterministic, not flaky. The trace holds only Nextcloud's generic
error page, "More details can be found in the server log", and that log is not
in the artifact.
'errorlog' sends the exception through PHP's error_log to FPM's stderr and from
there into `docker logs`, which the existing collection already captures. No new
collection code and no reads from the NFS volume. 'syslog' and 'systemd' would
both need a socket inside the container.
This does not explain the 500 in that run; it makes the next one explainable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
466f04503 replaced the container-run publish with a compose build service and
started copying build_assets.sh into <volumes>. ansible.builtin.copy does not
create parent directories, and nothing else creates <volumes> at that point:
sys-stk-full, which does, is included afterwards on purpose, because
meta/volumes.yml makes <volumes>/pkgsrc a bind source and
<volumes>/build_assets.sh a config source of the build service - both have to
exist before the stack is rendered.
<volumes> was therefore only ever created as a side effect of
utils/stage_manifest.yml:2-6, inside the loop that `manifest_role in groups`
filters. Run 30856149821, job "📖 Instructions": both candidates skipped, the
loop ran zero times, and the copy failed with
msg: Destination directory /opt/compose/cdn/volumes does not exist
The new task targets CDN_ASSET_STAGE rather than directories.volumes. That is
<volumes>/pkgsrc, so creating it with parents covers the build script's parent
AND the cdn_pkgsrc bind source. Aiming at directories.volumes would fix only the
task the log named: with no matched manifests, pkgsrc is missing too, and while
the compose daemon auto-creates a missing bind source, swarm does not.
The failure is not deterministic across reruns. cli/meta/ci/guide_select.py:70
picks the test role with random.choice, and of the three roles that require cdn
only web-app-dashboard ships a package-lock.json - a dashboard roll makes the
guard match and hides the bug. Pin inputs.role=web-svc-css to reproduce.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Takes the revert of the rendered system config. The MCP work had added the
proxy service name and its swarm DNS name as trusted domains to
system.config.php.j2, which the revert deletes; both entries move back to
vars/system.yml as trusted_domains 1 and 2, where they lived before aeefb1407.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three occ calls take an exclusive lock on config.php and gave up on the first
failure. In run 30830377785 the config:system:set batch hit
In Config.php line 306:
Could not acquire an exclusive lock on the config file
/var/www/html/config/config.php
after writing seven of nine values, and failed the swarm job. The same task ran
13 times across that run's swarm jobs and passed 12 times, including a second
fedora job on the same commit and topology, so the condition is transient rather
than a defect in the call.
maintenance:repair and app:update in 03_upgrade.yml already carry
retries/delay/until for this reason. This aligns the three that did not:
occ upgrade, maintenance:mode --off, and the config:system:set batch. The batch
is safe to repeat because config:system:set is idempotent and its `set -euo
pipefail` aborts at the failing call rather than continuing half-applied.
This does not mask the failure. Verified with a probe playbook of the same shape
(failed_when plus an until that never becomes true): the play ends failed=1 and
the downstream task does not run, so an exhausted retry still fails the deploy -
two attempts later than before.
The mechanism behind the lock error stays unproven: Config.php calls
flock(LOCK_EX) without LOCK_NB, which blocks rather than returning false, and no
errno appears in any of the 596 rescue-diagnostic files. A retry helps if the
condition is transient and does not if the NFSv4 lock layer returns a hard
error. That is why this is a mitigation of an ~8% flake, not a fix for a proven
cause.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This reverts commit 41063206a.
It broke every swarm nextcloud deploy. system.config.php.j2 opened with
<?php
{%- macro php_str(value) -%}
and the leading dash strips the newline after `<?php`, so under Ansible's
trim_blocks the file renders as `<?phpreturn [`. Run 30848540165:
ParseError: syntax error, unexpected token "=>", expecting "]"
in /var/www/infinito/system.config.php:2
#0 /var/www/html/lib/private/Config.php(229): include()
Config->readData() includes the file, so occ upgrade, every later occ call and
every php-fpm request died on an uncatchable compile error. The template had
been verified by rendering it with plain Jinja defaults, which do not match the
ones Ansible uses - the check confirmed the escaping and missed the line above.
A one-character fix (`{% macro`) is correct and was verified end to end, but the
commit is reverted rather than repaired, because its premise did not survive
review:
- The failure it targeted is not deterministic. The same task ran 13 times
across the swarm jobs of run 30830377785 and passed 12 times, including a
second fedora job on the same commit and topology. It is an ~8% flake.
- It does not remove the mechanism. 03_upgrade.yml:8 runs
`occ maintenance:repair --include-expensive` unconditionally, and upstream
Repair.php sets the maintenance system value twice with no guard - two
exclusive locks on the same NFS config.php per deploy, unretried.
- The benefit is largely illusory. Config.php:279 var_exports the whole merged
cache back into config.php, so the nine values land in the locked file anyway
on the first maintenance:repair, with a larger payload.
- It traded away occ's exit-code confirmation that Nextcloud accepted the
values, and moved them from the shared NFS volume to a node-local bind.
The remaining exposure is the unretried occ calls in 03_upgrade.yml; a bounded
retry there is the smaller and better-targeted change and is left for its own
commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds docs/requirements/035-mcp-proxy-expansion.md with the exhaustive
169-role MCP audit: architecture decision, target metadata contract,
identity/authorization/credential requirements, discovery and
reconciliation, Flowise 3.1.3 and n8n 1.95.3 integration paths, curated
adapter candidates, tool and adapter safety contract, acceptance
criteria, and implementation order.
Revalidates 025-mcp-role-integration.md against that audit: the tree
already carries 18 role-local mcp: blocks, so the empty-baseline claim
is dropped. Adds 'adapter' to the implementation precedence chain,
corrects the stale hosted_only classification on wordpress, odoo and
discourse, and unchecks the criteria that metadata presence alone does
not prove (real provisioned identity, application-scoped RBAC grants,
enforced proxy limits, runtime mutating-tool policy, openwebui and
flowise client provisioning, end-to-end real tool call).
Thirty-nine criteria are now ticked against thirteen new pieces of evidence,
strongest among them the five per-server tool counts hermes reported live and the
status-code check passing on every domain with MCP enabled.
Two criteria are reworded rather than ticked. 206 asked a client to restrict the
tool server to the role's group and closed on someone entitled to drive it not
needing to be an administrator; a user's roles list carries no application
context, so that is not what the platform can express today and the text now says
so. 208 allowed both readings of tools.mutating_tools_enabled, which is why two
roles carried opposite values for the same situation; it now states that the
field records intent and the README names the missing enforcement.
The design doc gains the public exposure section, including that the proxy ships
no rate limiting, and why stdio is not deployed.
Nine criteria stay open: two are PR artefacts, three are blocked upstream, and
four need deploys that have not run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The key left the vocabulary, the discovery lookup and its test earlier. Jenkins
was the only role still declaring it and was missed there, because that pass
searched for consumers rather than declarations. The schema lint no longer
accepts it under endpoint, so leaving it would have failed every run.
The Health rows in the jenkins and mattermost READMEs stay: those document real
upstream URLs, not this schema key.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Neither client exposes an administrator-visible server list the way Open WebUI
does, so their configured set is proven at deploy time instead: hermes runs
'hermes mcp test <server>' per server, openclaw probes the agent's mcp surface
in tasks/01_mcp_assert.yml. OpenClaw's assert already existed but was documented
nowhere, so reading the role suggested it had no verification at all.
Both sections also name what the Playwright spec covers, which is the
complementary case: the instance holding the MCP credentials refuses an
unauthenticated caller.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The plugin tracked :latest, so nothing recorded which version served the MCP
endpoint. The pin is the version read out of the running container's
MANIFEST.MF, and a redeploy read it back unchanged.
Only mcp-server is pinned. The other twenty plugins and the jenkins/jenkins:lts
image keep this role's existing convention; changing that is a separate concern.
The README also gains the Authorization subject section that belongs with the
earlier auth_subject commit. It was missed there and cannot be split off a file
without an interactive stage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The probe asserted an HTTP 200 on initialize and treated that as a working MCP
surface. It only proves the path exists and the token is accepted. This role
deploys the CE images while GitLab serves MCP through Duo, which is Premium and
Ultimate only, so an endpoint that answers but advertises nothing was
indistinguishable from a working one.
The probe now completes the handshake and calls tools/list, reading an SSE frame
when the response is one, and reports the count. A separate assertion fails the
deploy on zero tools and names the licence as the reason, which would put this
role behind 'blocker: licence' like web-app-openproject.
The assertion uses two substring checks rather than regex_search: backslash
escaping inside YAML scalars is a trap this repository has been bitten by.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The assertions were embedded in each role's playwright.spec.js. Extracting them
follows how the other specs are laid out and completes the set: all eight MCP
server roles now register a guest spec, where only five did before.
Both assertions matter. A non-2xx alone would pass for an endpoint that answers
401 and still names its tools in the body, so the inventory is checked too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four roles declared auth_subject: user while handing the client an
account-bound credential. Nextcloud and Home Assistant mint against the
administrator account, Mattermost issues the administrator's personal access
token, and Baserow creates a dedicated mcp@<domain> account with its own
workspace. The declarations now name what the minting tasks do; Baserow's
README additionally claimed the administrator owns its endpoint, which it
never did.
The tools declarations follow one convention instead of two: the field records
the deployment's intent and the README names the missing enforcement, which is
what GitLab already did and what Baserow and Mattermost now do.
The discovery lookup carries auth_subject through so clients can see the
subject. No consumer branches on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight server roles declare an `mcp` RBAC role, so the platform mints
roles/<app>/mcp and the administrator joins it. Open WebUI now honours those
groups instead of flattening them to user/admin, and each MCP tool server is
restricted to the group of the role that serves it.
The grant cannot be rendered: Open WebUI matches it against a group id that
insert_new_group mints as a fresh UUID, so tasks/01_mcp.yml resolves the group
after the container is up and writes the grant through the admin API, the way
every instance-minted credential in this repository is provisioned.
Entries render disabled. An empty access_grants is not 'nobody', Open WebUI
reads it as every administrator, and ENABLE_PERSISTENT_CONFIG=false makes the
env authoritative again on every start. Enabling only together with the grant
keeps the fallback at no access rather than too much.
The provisioning account is the role-internal openwebui-api-bot, never a
platform identity: OAUTH_MERGE_ACCOUNTS_BY_EMAIL ships disabled and an OIDC
login whose email a local account already holds is refused, so seeding a
person's address would lock them out.
Known limits are documented rather than claimed away: a bare restart leaves MCP
off until the next deploy, a user's roles list carries no application context so
roles: [mcp] grants every deployed mcp group at once, and Open WebUI drops a
stale membership only while the groups claim is non-empty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bdb3b29a9 passed the watchdog floor to the drill as INFINITO_DISK_FLOOR_MB and
broke test-lint:
AssertionError: INFINITO_* keys referenced in code without a registry entry
INFINITO_DISK_FLOOR_MB
The prefix is reserved for the deployment env registry, where every key needs a
default in default.env or a handler that sets it. This value is neither: it is a
handoff from matrix.py to the script matrix.py starts, and it only exists for
the duration of that call. DRILL_EXTRAS, one line above, already goes the same
way under an unprefixed name.
So the key becomes DISK_FLOOR_MB and its declaration moves up beside
DRILL_EXTRAS in the `: "${VAR:?...}"` form the file uses for matrix handoffs,
leaving the INFINITO_* block below to the values that really come from .env.
Registering it in default.env would have satisfied the lint by writing 6144 down
a second time, which is the duplication bdb3b29a9 set out to remove; a nocheck
would have hidden the naming mistake rather than fixed it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hostname(1) is not installed in every node image. In run 30830377785 the arch
drill printed
trigger_units.sh: line 22: hostname: command not found
starting svc-bkp-local-2-device.11.6.0.infinito.nexus.service on
so the node name came out empty on all three call sites. The cosmetic one is the
progress line; the damaging one is the journal dump path, which becomes
"${unit}..journal.txt" and is identical on every node. Four hosts writing the
same filename into the shared dumps directory means the artifact keeps whichever
one wrote last, and a unit failure gets investigated against another host's
journal.
uname is coreutils and present wherever the drill runs. Reading it once also
drops three subshells from the loop.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill's space check required FREE >= PULLED, but the run is killed from
outside by matrix.py's watchdog as soon as the runner drops below 6G free. The
check knew nothing about that floor, so it waved through a drill that could not
finish: 19008M free against a 13961M pulled tree passes FREE >= PULLED, and the
sync then leaves 5047M, which is under the floor.
That is what happened to the arch swarm drill in run 30830377785. The device
sync reported the marker at 19:16:38.34, the watchdog fired at 19:16:39.73 on
its next 30s tick, and the step died with rc 75 thirteen minutes into work that
was already doomed when the precondition ran.
The requirement is PULLED + FLOOR, not FREE >= PULLED and not 2 * PULLED. FREE
is measured after the pull, so the first copy is already deducted; exactly one
further copy is ever added on top of it - the device copy during the sync, and
later the restore root, by which time base.sh has deleted the pulled tree. The
two extra copies are sequential, never simultaneous.
matrix.py owns the floor now and exports it as INFINITO_DISK_FLOOR_MB. It was
previously a bare `6 * 2**30` the shell had no way to see, which is why the two
sides could disagree about what "enough disk" means.
This makes the failure honest and immediate; it does not create disk. The arch
runner is short by about 1.1G, and closing that needs a capacity decision -
bigger runner, smaller test dataset, or reclaiming build cache before the drill.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Leftovers from extracting lookup_context and attachments: the mock.patch target
got short enough to fit on one line again, and the file split left a spare blank
line in each module. Whitespace only, produced by autoformat during make
quality, so the 298 tests of that run already covered this state.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine sequential `occ config:system:set` calls each open config.php exclusively.
In swarm that file lives on the NFS share behind every replica, and the seventh
call could not take the lock:
In Config.php line 306:
Could not acquire an exclusive lock on the config file
/var/www/html/config/config.php
stdout shows six values written before the abort, so the run left a partial
configuration behind. That failed the swarm web-app-nextcloud 5 job in run
30830377785 while every PLAY RECAP on the other nodes reported failed=0.
The same nine values now come from templates/config/system.config.php.j2, which
rides the additional-config channel the role already uses for redis, ldap and
the rest: rendered to NEXTCLOUD_HOST_CONF_ADD_PATH, mounted read-only, merged by
zzz-infinito.config.php after config.php so its values win. That path never
takes an exclusive lock, and it runs before the stack starts, since all nine
values are statically derived and none needs a live container.
Two behaviours are deliberately not carried over. The --type flags become native
PHP types, which is stricter than the string coercion occ applied. And nothing
confirms the file was accepted any more, where occ's exit code did; a read-only
`config:system:get` would restore that assurance at the cost of an occ call, and
is left out until someone wants it.
php_str() escapes backslashes and single quotes, matching the idiom already used
for the redis password in memcache.config.php.j2. Without it a quote in any
value would be a PHP syntax error, and include_once would drop every additional
config file, not just this one.
vars/system.yml and 02_manager_ops/05_system_config.yml are gone; the later task
files shift down one number so test_task_file_numbering stays satisfied.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two places decided independently what the pre-created <entity> network is called
inside the compose project. render.py emits the key; create.yml asserted it was
always `default`. They diverged in July and nobody noticed, because compose
5.3.1 resolved networks by name and never read the claim.
Compose 5.4.0, published 2026-08-03T14:39:12Z, resolves them by label. For
web-app-seaweedfs - the one deployed role that provides its own shared_net and
therefore renders an unnamed `default` - the mislabelled `seaweedfs` network is
adopted as the project default, `seaweedfs_default` is never created, and the
containers fail to start against it. That failed web-app-nextcloud 0 and 3 in
run 30830377785.
_shared_network_key() is now the single derivation. render_compose_networks
takes its branch from it, so the rendering decision IS the key decision;
shared_network_compose_key() exports the same value, the shared_network_key
lookup hands it to Ansible, and create.yml stamps it: `seaweedfs` for a
self-provider, `default` for everyone else. Both true, both from the source that
renders the file.
Verified on one host with only the compose binary swapped:
5.4.0 network=default -> FAIL, network <project>_default not found
5.4.0 network=<entity> -> OK
5.3.1 network=default -> OK
5.3.1 network=<entity> -> OK
so this is not a fix that trades one version for the other.
test_key_matches_the_key_the_renderer_emits renders the block and asserts the
reported key appears in it, for both role shapes and both modes. The drift that
sat unnoticed for a month now breaks a test.
The move of the attachment computation into utils/networks/attachments.py rides
along because render.py would otherwise be broken between two commits: it is a
pure relocation, re-exported so no importer changes, and it brings the file from
352 to 243 lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
compose_networks, container_networks and compose_external_networks each rebuilt
the same Ansible-side context before calling their renderer: resolve the
variables, settle the deployment mode, load applications, build the service
registry, and close over the config and database lookups. Around forty lines,
three times, each with its own copy of _resolve_var. A change to how the
registry or the closures are obtained could reach one lookup and miss another.
build_context() returns that context once. honour_mode_force carries the only
real difference between the callers: compose_external_networks never consulted
compose_mode_force, and still does not.
The lookups keep only what is theirs - term validation, their own arguments,
their renderer call - and drop from 94/90/87 lines to 59/54/53.
No assertion changed in the tests; only the mock targets moved with the seam,
from the plugin modules to utils.networks.lookup_context. Twelve legacy comments
in those test files lost their grandfathering when the files entered the lint's
diff scope: two became docstrings naming the call contract, the rest were
narration and are gone.
Behaviour-preserving: the same 13 lookup tests pass before and after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No role declared it, no client rendered it and no probe read it. The key only
put a "health_path": None into every discovery entry and suggested a capability
that does not exist.
Removing it from the vocabulary, the lookup and the lookup test has to happen
together: the schema lint would keep accepting it, the entry would keep carrying
the None, and the test compares the exact key set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
YAML treats # inside a block scalar as text, so the directive became part of the
command. ansible.builtin.command starts no shell and splits on whitespace, which
turned '#', 'nocheck:' and 'container-exec-resolver' into container names the
daemon then reported as missing. The real inspect succeeded; the non-zero exit
came only from those three.
Put the directive behind the scalar indicator, where YAML reads it as a comment,
matching how the rest of the repository writes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The loop only left when average_version_directories dropped to 0. With exactly
one version per application that value is 1, yet every version is also its
application's last one and therefore undeletable. A pass freed nothing, disk
usage never moved, and the job spun until the time limit raised, exiting 1 and
failing the deploy.
Count the deletions and leave when a full pass reaches zero. A cleanup that did
everything it is allowed to do has succeeded; reporting a full disk belongs to
sys-ctl-hlth-disc, so the threshold stays untouched.
The regression test needed an advancing clock: with time.time pinned the limit
never fires, so a regression would hang the suite instead of failing it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sys-stk-full passes `application_id` on as a role var holding the Jinja
expression, and the lookup read it raw from the play vars. It then looked up an
application named `{{ sys_stk_full_application_id }}`, found no healthcheck and
blamed the role for not declaring one, which web-svc-logout does.
The container_ports lookup already resolves this, so take its idiom. The
re-trust has to follow the str() coercion: coercing strips the trust tag and
templar.template would silently no-op.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pre-pull hardcoded its `async:` value while `poll:` resolved to `omit`
whenever async is disabled. The job therefore stayed asynchronous but Ansible
polled it to completion itself and removed the job file, so the shared reaper
died with `could not find job`, naming a different job id on every run.
Gate `async:` the way the other seven reaper callers already do, and skip the
reaper itself when async is off, so no caller can reap jobs that were never
started asynchronously.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
autoformat joined the two operands of the message concatenation during
make quality; 1408c6770 had them on separate lines. Formatting only, same
expression and same output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The login probe concludes "login failed, so the password is wrong" and hands
over to 02_reset_root_password.yml, which stops the container, scales the swarm
service to 0, boots a recovery server with --skip-grant-tables, rewrites the
root password and scales back. That is a full teardown of the central database,
so what triggers it has to be right.
container exec also fails when the container is not on this node. The probe
delegates to STACK_HOST, and resolve_container_id.sh tries a local lookup first
but otherwise falls back to `container service ps` + `container inspect
--type=task`, which returns the container id of a task on ANY node and exits 0.
On a manager that does not host the task, container exec then answers "No such
container".
Two amplifiers turn that into a silent misdiagnosis: failed_when: false, and -
verified by execution, since the opposite was claimed - retry exhaustion under
until: does NOT abort a task carrying failed_when: false. The task stays ok,
`failed: True` only appears in the result, and the gate fires.
The clause excludes exactly that case. mariadb never prints "No such container"
for a bad password, so no legitimate trigger is lost, and the exec failure stays
as visible as it was.
Not reachable today: the swarm topology has one manager and svc-db-mariadb
declares placement: manager, so the local branch of the resolver always matches.
It becomes reachable with two or more managers, which the comment on that
placement line names as planned.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker_dataroot.sh installs zfsutils-linux at runtime, so the zfs pick depends
on the package cache being up. In run 30781723102 that decided the two swarm
drills differently: the cache stack was removed at 06:39:49 and rebuilt at
07:18:25, so the mariadb drill had no zfsutils candidate at all and declined,
while the postgres drill got one and reached zpool create. Installing it at
build time removes that dependency.
Debian keeps zfsutils-linux in contrib, which the base image does not enable, so
the branch adds a contrib source first - only for ID=debian, since Ubuntu ships
it in its default components. A failed update after adding the source removes
the source again rather than leaving a broken repo line behind.
The install is deliberately separate from the mandatory packages and
non-fatal. Node images for centos, fedora and arch have no zfs userland either
and resolve.sh's userland_for() already accounts for that, so a missing zfs
userland is a supported state - whereas folding it into the mandatory install
would let a packaging error break the node image for all five distros. A failure
now prints to stderr and the run declines the pick visibly.
Verified against the real bookworm contrib index (deb.debian.org, HTTP 200):
zfsutils-linux 2.1.11-1+deb12u1 exists there, all four libzfs libraries are
Depends rather than Recommends so --no-install-recommends keeps them, and the
only Recommends are zfs-modules|zfs-dkms and zfs-zed, which we do not want - the
module comes from the host via /lib/modules. Control flow exercised with a
stubbed apt-get across six combinations of distro, package availability and
update failure; every one leaves the build at rc 0.
Not covered here: whether a 2.1 userland can drive the runner's newer host
module. zfs-version.txt and modules.txt from 264556a31 answer that next run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A job whose nodes all declined the filesystem tested none of it, and the table
row saying so reads like an ordinary result. That is how the debian arm lost its
zfs coverage without anyone noticing: docker_dataroot.sh exists to exercise the
snapshot mode of svc-bkp-volume-2-local, which engages only on btrfs and zfs,
and on that arm it has been engaging on neither.
The summary now carries an explicit warning when no node applied the requested
filesystem and at least one declined it. Partial coverage is coverage and stays
unflagged; a job with no verdicts at all still exits early.
Deliberately a finding, not a failure. Stating a filesystem instead of `auto`
would set REQUIRED=true in resolve.sh and turn a decline into a hard error, but
that would paint the debian arm red for a cause that is understood and not yet
fixed - the node image has no zfsutils candidate without the package cache. The
goal was never to enforce the pick, only to stop losing it silently.
Exercised against three real job logs from run 30781723102: the debian arm
(5 nodes, all declined) produces the warning; the ubuntu arm (10 nodes, zfs
applied) and an ext4 job do not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The MODE_DEBUG block of the restart handler carries no host gate, unlike its two
siblings in the same file which are pinned to a deployment mode and, in swarm,
to IS_STACK_HOST. A handler runs on the host that notified it, and a worker
reaches this notify: sys-stk-front-proxy/tasks/main.yml:112-118 is an
always-changed `command: true` whose only guard is `not nginx_conf.changed`, and
the stack-host template action forces changed:False on workers, so the guard is
always satisfied there.
On a worker the container_address subshell then exits 64 - resolve-container-id
refuses to run off a manager - the substitution collapses to empty,
`container logs --tail=200` fails, and failed_when:false plus ignore_errors:true
swallow it. The debug task below prints an empty string, so the diagnostic
reports "no logs" on exactly the node someone enabled MODE_DEBUG to inspect.
container_address returns a string to embed, not a delegation; it decides what
is addressed, not from where. delegate_to supplies the second half.
STACK_HOST rather than the resolve_host_cid include, because svc-prx-openresty
declares placement: manager (meta/services.yml:9), so the task can only run on a
manager and the full host resolution buys nothing. In compose STACK_HOST is
inventory_hostname, so this is a no-op there. The sibling `container ps -a` stays
undelegated: a per-node container listing is legitimately node-local.
Diagnostics only - nothing gates on openresty_logs, and MODE_DEBUG defaults to
false. Not executed: no swarm worker run reproduces the empty dump.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both waits are needed and check different things: wait_converged inside
resolve_host_cid asks swarm whether the task reached running, the wait_for asks
whether the service answers. Running the endpoint wait first meant a partner
that is not deployed at all paid both budgets in sequence - up to 600s of HTTP
timeout followed by 120x5s of convergence - before the play gave up.
Ordering the cheap service-level check first halves that without giving up
either guarantee. The resolver needs no HTTP: it runs container service ps,
container ps and set_fact, and its two outputs are first consumed by the rails
call further down, i.e. after both waits.
Task order only; nothing added, removed or reparameterised. 8b2cbcb04 applies
the same ordering to the Zammad twin, where the wait was newly introduced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
resolve_host_cid's convergence wait is swarm-only (wait_converged.yml:9), so in
compose mode nothing made sure Zammad answers before the rails provisioning ran
- the only budget there was the provisioning task's own 12x10s against a rails
server whose cold start can exceed it. integration_openproject_provision.yml
already guards its partner this way; this restores the symmetry.
The wait sits after the resolver rather than before it. Both waits are needed
and check different things - wait_converged asks swarm whether the task reached
running, the wait_for asks whether the service answers - but ordering the cheap
service-level check first means a partner that is not deployed at all fails
after one budget instead of two. The resolver itself needs no HTTP: it only runs
container service ps / container ps and set_fact.
Not executed: that lookup('domain', 'web-app-zammad') resolves to a name
answering on port 80 is taken from the openproject twin, not measured.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two investigations of run 30781723102 stalled on files the rescue capture does
not take.
The zfs question - why the debian lab nodes cannot use a module their ubuntu
counterparts apply on ten nodes - needed /proc/modules to see whether the module
is loaded, /proc/misc to see whether it registered (a misc device never appears
in /proc/devices, which is what docker_dataroot.sh was searching), `ls -l
/dev/zfs` to tell a real node from the static one every privileged container
inherits, and `zfs version` for the libzfs version a whole refuted hypothesis
turned on.
The matrix-mdad question - whether ~31.8s per git clone is a connect stall or a
server-side queue - needs the TCP retransmission counters in /proc/net/snmp and
/proc/net/netstat and the conntrack drop counters. Neither can be distinguished
from per-clone timings alone.
Executed on a host with no zfs: a missing binary and a missing proc file each
land as text in the artifact rather than aborting the capture, which is the
best-effort contract run() already implements.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule checked whether both keys contain the token `omit`. That catches the
Ollama shape, where async carried no omit at all, but it does not enforce what
the failure message claims: two keys can both carry omit and still hang on
different switches, and the old check passed that.
_gate() now reads the switch each key states inline - the test of an
`if ... else omit`, or the left side of a `| ternary(..., omit)` - and the two
must match. A key that names no gate matches another that names none, which is
the shape sys-service and the canonical ASYNC_TIME/ASYNC_POLL pair use; both
then inherit whatever their variables do. An unrecognised omit-bearing form
falls into a single bucket, so it only matches another unrecognised form.
Checked against all five shapes in the tree: the pre-fix Ollama pair is an
offender, the fixed pair and the ternary and bare-variable pairs are not, and
two different conditions are now an offender where the token check passed them.
The rule still does not catch two ungated keys whose variables diverge. That is
deliberate: group_vars makes ASYNC_TIME evaluate to omit while ASYNC_POLL stays
numeric, and a poll without async is ignored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
18abce480 added kmod so the data-root setup can run its one explicit `modprobe
zfs`. It also handed a working loader to the four modprobe@ instances systemd
pulls at every node boot - configfs, drm, efi_pstore and fuse - which until now
failed on the missing binary, as every node's journal in
rescue-diagnostics-workspace-debian-bookworm shows. They would have started
loading host modules from a privileged container, on every node of every swarm
job in all five distros.
Masking the template keeps them exactly as dead as they are today and leaves the
loader available to the caller that needs it, so the behaviour change of
18abce480 is confined to the one site it was meant for. It sits with the
existing masks for systemd-resolved and systemd-udevd.
The image was not built; that the mask covers every instance of the template is
read from systemd's behaviour, not measured. The next run shows it: the units
report masked instead of failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard compared free space against USB_SIZE_MB = PULLED_MB*2+256, which is
the nominal size of an image 02_luks_device.sh creates with truncate - sparse,
so nothing ever allocates it. It also ran after that script, i.e. once the image
was already created, formatted and mounted, so the image's own footprint was
already subtracted from the df it read. It charged the pulled tree a second time
while that tree was counted as used, not free. In run 30781723102 it demanded
28108M against 16582M and refused a drill that fits: the peak is 14555M
additional, derived from rescue-diagnostics-swarm-web-app-nextcloud-0, and the
same run's debian job contains a full 9/9 completion of the same drill.
Rather than predicting the image's ext4 overhead - two earlier attempts at that
constant were wrong in both directions - both thresholds are now read:
df on DEV_MOUNT reports what the encrypted filesystem actually offers after
mkfs, and df on DIR_BACKUPS reports what the host still has for the copy the
sync writes into the sparse file and, later, the restore root. Both are compared
against PULLED_MB, a measured quantity. USB_SIZE_MB keeps sizing the image.
Every abort now tears the device down (umount, luksClose, rm -f); a refused
drill previously left the mapping and the image behind.
Exercised against the recorded numbers and six edge cases: 26624/16582/13926
passes where the old guard refused, an undersized device or host is refused, and
an empty or non-numeric df reading is reported as unreadable.
Both thresholds sit exactly at PULLED_MB, so second-order allocation inside the
image (extent trees, directory blocks) is uncovered - tens of MiB against a
13.9 GiB tree. The drill has not been run with this guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node image ships no module loader, so `modprobe zfs` in
scripts/tests/deploy/utils/filesystem/docker_dataroot.sh was a no-op whose error
was additionally discarded by `2>/dev/null || true`. Every node's own systemd
says so at boot, in rescue-diagnostics-workspace-debian-bookworm of run
30781723102:
(modprobe)[27]: modprobe@configfs.service: Unable to locate executable
'/sbin/modprobe': No such file or directory
The module itself is reachable: the nodes run privileged with /lib/modules
bind-mounted read-only, and their df confirms it. What was missing is the
loader, not the module. zfs-load-module.service is installed and enabled in the
image and sits inactive for the same reason.
This does not by itself make the debian lane apply zfs. `The ZFS modules are not
loaded.` in job 91588086713 came from libzfs at `zpool create`, not from the
suppressed modprobe - the kernel ring buffer shows five 30 GiB loop attachments
at 07:37:32, i.e. the script was past the /dev/zfs gate. And a Debian bookworm
node only has a zfsutils-linux candidate while the package cache stack is up:
the cache was removed at 06:39:49 and rebuilt at 07:18:25, which is why the
mariadb drill declined at the userland check while the postgres drill reached
zpool create. That cache dependency is untouched here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The marker claimed to suppress container-exec-resolver, but that rule does not
fire on this task. Its extractor reads the first argument after `container exec`
and finds `-i`, so it returns no target; the container id is four lines further
down. The position was wrong too: the rule accepts a marker on the same or the
immediately preceding non-empty line, and three lines separated the two.
That leaves a suppression marker carrying a rationale, which is the shape the
comment policy exists to prevent. The reason it stated - that container_address
would expand to a manager-only subshell - is already in 04b8cfead's message.
tests/lint/ansible/shell/container/ stays green without it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The script stops the daemon at :157-158 and starts it again only on the success
path at :220. decline() exits 0 and set -e exits non-zero, and both walk past
that start, so any other outcome leaves the host without docker. One of the
three callers is test-deploy-host.yml:171, where the script runs on the GitHub
runner itself rather than in a lab node container.
An EXIT trap rather than a restart inside decline(), because decline is not the
only affected exit: arm_autoclear's exit 1 and every set -e abort at mkdir,
truncate, zfs create, mkfs, mount, btrfs subvolume create and the daemon.json
writer sit in the same window. The trap is armed only after the stop, and
DOCKER_STOPPED=false after a successful start makes it a no-op on the green
path. A failed restart is reported rather than swallowed, so it cannot hide the
failure that led there.
This hazard predates 02dc003bb: the loop-device decline added in the commit
before it has the same property. 02dc003bb widened it by putting a second
decline behind the stop.
Verified with a stubbed-PATH harness driving the real script: after a failed
zpool create the call log contains `systemctl start docker`, and it comes after
`losetup -d`, so the teardown still precedes the restart. REQUIRED=true still
exits 1, the success path is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bootstrap.sh re-cloned all 31 unauthenticated GitHub repos on every bootstrap,
including warm re-runs where roles/galaxy was already fully populated. The
install is now keyed on the sha256 of requirements.yml: the stamp is removed
before the install and written only after it, `--force` stays inside the branch
so a changed requirements.yml still refreshes every role, and set -euo pipefail
still aborts on a galaxy failure. Nothing is masked and no retry is added.
The stamp lives in roles/galaxy/, not in the matrix_mdad_state volume, so it
shares one lifecycle with the roles it describes. The volume-vs-bind-mount split
is the same cross-domain state coupling that 5d6d1c7ca blames for this run's
matrix failure, where the bootstrap gate read its marker from a volume and the
synapse unit from the container rootfs.
This is not a fix for the HTTP 503 that failed web-app-nextcloud 3 in run
30781723102: that job ran a cold container, which always clones. It removes
re-clones on redeploys of a persistent host. The 503's mechanism is still open -
the observed ~31.8s per clone is flat and payload-independent, and neither
runner bandwidth nor host load explains it.
PYTHONUNBUFFERED=1 line-buffers ansible's stdout in the unit. Without it every
progress line carries one flush timestamp at process exit, which is why the 667s
of the failing install cannot be split into per-clone times.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three places where a data-root verdict was produced and then lost.
report.sh matched `requested=[a-z]+`, which never matches `ext4`. Against the
real deploy log of a run that recorded five ext4 verdicts, the old pattern
yields 0 matches and the summary table was silently empty of them; the fixed
pattern aggregates the same five into one row.
docker_dataroot.sh wrote the reason only to GITHUB_STEP_SUMMARY, which is unset
inside the lab node containers - so on exactly the machines that decline, the
reason went nowhere. It now rides on the machine-readable report line too. This
is what leaves the mariadb drill's decline reason unrecoverable from run
30781723102.
resolve.sh's exclusions went to stderr only, i.e. into the deploy log rather
than the job summary.
The reason is emitted as a list below the table, not as a column: reasons embed
per-node data (the pool name carries the hostname, plus the loop device), so
folding them into the `sort | uniq -c` key would split one five-node row into
five one-node rows exactly in the failure case the reason exists to explain.
`reason=[^']*` keeps ansible's closing quote out of the value, and `|` is
replaced so a reason cannot break the markdown table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
02_manager_ops.yml includes the plugin chain with `apply: delegate_to:
NEXTCLOUD_HOST_NODE`, so the addon hook runs wherever that node happens to be.
On a swarm worker the partner probe could not answer: it used
NEXTCLOUD_ZAMMAD_CONTAINER_ADDRESS, which expands to a
`resolve-container-id zammad railsserver` subshell, and that helper exits 64 off
a manager. The substitution came back empty, `container inspect ""` failed,
`failed_when: false` swallowed it, and the gate
`(nextcloud_zammad_partner_running.stdout | trim) == 'true'` was false.
The OAuth provisioning was therefore skipped, client_id and client_secret stayed
empty, and integration_zammad's personal view fell back to its access-token UI -
so the 'Connect to Zammad' control the spec waits for never existed. That is
what failed web-app-nextcloud 5 in run 30781723102.
The probe and its clause are gone; provisioning now resolves the railsserver's
hosting node and local container id through the standard
sys-svc-compose/utils/swarm/resolve_host_cid.yml and delegates the rails call
there. resolve_host_cid pins its own delegate_to on STACK_HOST and
<prefix>_HOST_NODE, so it is unaffected by the inherited one, and the trailing
occ tasks keep no delegate_to and still run on the Nextcloud node.
Losing the probe also loses its quiet skip. The addon is still gated on
services.zammad.enabled, i.e. 'web-app-zammad' in group_names
(meta/addons/integration_zammad.yml:2), so the exposure is a targeted deploy
where the partner is in the inventory but excluded from the run: wait_converged
then spends 120x5s before failing instead of skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ActiveState=failed` only says the oneshot exited non-zero, and the unit runs
two phases: `ansible-galaxy install -r requirements.yml --force` and then
`ansible-playbook setup.yml` (bootstrap.sh:39-44, under set -euo pipefail).
Claiming the playbook failed sends the reader looking for an inner PLAY RECAP.
In run 30781723102 the web-app-nextcloud 3 job died in the first phase, at role
21 of 31, on a git clone that got HTTP 503. The container journal logs
"starting ansible-galaxy install" and never reaches "galaxy install done" or
"starting ansible-playbook setup.yml", so there was no recap to find.
Wording only; the branch, its condition and the diagnostics are unchanged. The
cause of that 503 is still open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
