VaultHandler.encrypt_string started one ansible-vault process for every single credential. The swarm matrix' rotation gate between PASS 1 and PASS 2 drives 113 application subprocesses over 855 credential keys, so the step cost 1078s in run 33923190220 while the green baseline never ran it at all. Measured per value: 226ms via the CLI, 3.3ms via VaultLib, so 855 keys drop from 193s to 2.8s.
The snippet layout is a contract: encrypt_leaves splits the returned lines and strips the body indent, and existing inventories were written by the CLI. Verified byte-compatible (same 'name: !vault |' header, same 10-space body indent, same $ANSIBLE_VAULT;1.1;AES256 marker), and ansible-vault decrypt reads back what the new path writes.
Reading the password file delegates to ansible's own FileVaultSecret rather than approximating it: ansible strips, then decrypts a vaulted password file, then rejects an empty secret. A hand-rolled .strip() covered only the first of those. _vault stays lazy because callers construct VaultHandler with a password file that need not exist yet.
The tests replace two subprocess.run mocks that asserted against invented output. They now pin the layout encrypt_leaves depends on, and pin the guarantee from 052cea02e2 (values starting with a dash), which landed without a test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
checkmk crash-looped in swarm on run 33923190220, 82 convergence attempts,
with PermissionError [Errno 13] on '/omd/sites/cmk/tmp/nagvis' raised from
omdlib/tmpfs.py:294 while OMD populated the site's tmp skeleton.
The mount is not the problem and must stay. Upstream's docker-entrypoint.sh
runs `omd create --no-tmpfs` and the comment above it states the site tmp
directory "might have been pre-created by docker when the --tmpfs option is
used", so supplying that tmpfs externally is the intended pattern, not a
workaround.
The mode was the problem. container_volumes coerces a declared mode with
int(text, 8), so "01777" reaches the stack file as the decimal 1023. Docker
29.7.2 reads that as raw mode bits and mounts 1777; the CI hosts run 28.0.4
and produced a directory the owning uid could not write. 0o1023 leaves the
owner with no permission bits at all, which is exactly the failure.
Docker's own default is already 1777, sticky and world-writable, which is
what a site tmp needs, so the declaration could only ever match it or break
it. Dropping the key restores the default. Measured on 29.7.2 in both
`docker compose` and `docker stack deploy`: a mode-less tmpfs mounts 1777 and
uid 1000, the uid `omd create -u 1000` gives the site, creates nagvis in it.
The validator now rejects tmpfs 'mode' outright rather than checking it is
octal. An octal check cannot help when the encoding itself does not survive
to the container the same way on every engine version. merge/svc-net-tor
still carries the old declaration and will fail this check if it lands.
The front proxy hardcoded [200, 301, 302] and escalated to a full OpenResty redeploy on anything else. Apps that answer 403, 404 or 405 by design tripped it on every run: 19 escalations in one Nextcloud job at a measured median of 40.5s each.
roles/<app>/meta/server.yml already declares status_codes per canonical key, and sys-ctl-hlth-webserver already reads it. Move the resolution into utils/roles/applications/status_codes.py so both consumers share one definition of DEFAULT_OK, and expose it to templates as the accepted_status_codes filter.
The proxy gate takes the union of the default and the declaration rather than replacing it. Most declarations are narrower than the triple, so replacing would escalate vhosts that answer perfectly well. Contract checking stays with sys-ctl-hlth-webserver, which keeps replacement semantics.
_normalize_codes and _valid_http_code became dead in the health filter and are removed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The step banner announced a start and nothing else, so a phase duration had to be read off the next banner, and a phase with no output of its own was invisible. In the budget-killed Nextcloud#0 of run 33828309745 the credential rotation spent 17 minutes with nothing but the 30-second sampler between its banner and the next one, and reconstructing the split between provisioning, both passes, the DR drill and the rotation meant subtracting timestamps by hand.
The watch loop moves unchanged into _run_watched. _run now brackets it and prints the elapsed time and the exit code, so this holds for every phase, not only the rotation.
The test stubs _run_watched and pins the three things that matter: the exit code passes through, the opening banner is printed, and the closing line carries duration and rc. It is the first test over this module, whose absence was named earlier as the reason a wrong change to the pass sequence would turn nothing red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
supported_version and minimum_version sat in 24 roles and were read by no template, no task and no lookup. Their only consumer was the audit report version column, which turned false the moment an automated image bump ran: qdrant reported a verified version nobody had verified, two releases long. Both leave MCP_KEYS, so a role that declares them again fails the schema lint instead of carrying a field with no effect.
Requirement 025 mandated minimum_version as a MUST and 035 carried it in an example; both are corrected here, otherwise the repository would contradict its own code.
The gate then surfaced five dead docs.mattermost.com links. Mattermost moved to /administration-guide/ and /deployment-guide/, and every replacement was probed for 200 rather than guessed. The MCP page is gone with no successor, so mattermost now points at mcpserver/README.md in the Agents repository, the source rather than a rendered mirror.
Both concerns share three files - mattermost's meta/mcp.yml, requirement 025, and the generated audit.md, which derives from a source of each - so splitting them would commit an artifact that does not match its sources.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four module-level constants were defined and never read: _OPERATORS in the complexity filter, superseded by the op group of _TOKEN_RE two lines below; _VALID_SWARM_CONDITIONS, which swarm_restart_condition never checks because it maps to one of three literals by construction; _MISSING in services/config.py, where resolution already goes through get(strict=False, default={}); and _DEFAULT_FLAG_VALUES_THAT_ARE_NOT_NAMES in the exec-resolver lint, referenced by no rule in it. The nfs_fstype lookup rebound its variables parameter and returned without reading it.
Every name was private to its module and a repo-wide search over every file type finds no reader. CodeQL flagged them as unused; ruff cannot, because pyflakes has no rule for module globals and treats a rebound parameter as used.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CodeQL reports cmd and sort_spec as possibly uninitialised after the except branch that calls parser.error, because it does not model the argparse error method as NoReturn. It is: usage is printed and the process exits with status 2. An explicit return 2 on that branch states the exit status the call already produces, so the analyser and a reader see the same path.
The retry loop in the registry client counted two attempts in a variable the body never read; the underscore prefix was the convention for that, but CodeQL only honours the bare name.
No path changes behaviour; the three alerts stop pointing at correct code so the scanner stays readable for the ones that are not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d14354e892 added both probes reading only /etc/dnsmasq.conf, but svc-net-tor writes its onion zone to /etc/dnsmasq.d/tor-onion.conf and leaves the main file with nothing but a conf-dir pointer. Every Tor host of run 33698844864 therefore captured resolve-inzone.txt as '[no output, exit 2]' and all 471 sampler lines as inzone=unconfigured: during the resolver outage that ended Gitea#0, nobody ever asked dnsmasq for an onion name, and two investigations went without the one measurement that would have settled them.
Both probes now follow the conf-dir pointer, main file first so a lab node with its zone in the main config answers as before. Each is exercised by a test that runs it against real files rather than comparing its text, which the rescue constant had no test for at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils/roles/applications/config.py turns a missing key with a None default into False, so the provider-side probe in attachments.py read an omitted mcp_consumer flag as an explicit refusal. No MCP client ever joined a provider's shared_net: open-webui rendered no gitlab network, http://gitlabmcp:8080/mcp did not resolve from its namespace, and Open WebUI's verify call answered 400 in both GitLab#1 jobs of run 33698844864, clearnet and tor alike. Measured on that run's own inventory: without the change open-webui does not join the gitlab overlay, with it it does, and a sentinel probe shows the provider carries no mcp_consumer key at all.
utils/roles/applications/mcp.py:295 already states the rule this restores, that an omitted flag inherits the client's own declaration and only an explicit false excludes.
The stub in the render tests answered an unset path with the caller's default, which is why the regression passed them; it now coerces the way the accessor does, and the new case fails without the fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
26 roles resolve a domain in their metadata, so lookup('tls') and lookup('domains') run inside the applications render. Under the re-entry guard those calls receive the unrendered tree, whose service flags are still Jinja strings, and _as_bool reads "{{ 'svc-net-tor' in group_names }}" as false. The resulting map skipped onion injection for every app that does not enable tor literally, and it was cached, so it answered every later read in that process.
Before the applications carrier each worker rendered and poisoned its own cache the same way, which is why the shared Keycloak client registered 5 onion origins then and 65 now. The render state joins the cache key, so the render's own call sites still share one map while a later read builds the finished one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The SSO gate handed X-Forwarded-Access-Token to every oauth2-fronted upstream, so each of the 28 gated roles received a full-scope bearer for the signed-in user while exactly one, n8n, decodes it. The token also grows with the realm's registered origins, and LAM's Apache rejects a header field above 8190 bytes (Moodle#0, run 33689701895).
Consumers now declare services.sso.oauth2.pass_access_token. Without it oauth2-proxy already answers the auth request without the token, and the gate blanks the upstream header rather than dropping the directive, because nginx inherits proxy_set_header as an array and an unset header would let a client-supplied one through. The new lint keeps declaration and use in step in both directions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible forks a fresh worker process for every task, so the in-process
cache behind lookup('applications') was cold in every task. Each task that
touched the merged applications tree (config, domains, nginx, tls, cdn and
their callers) paid a full render: about 16 s on main and 21 to 23 s on this
branch, where ten more application roles carry MCP and AI Jinja. Measured on
the workspace jobs of run 33574056480 against the main run: 331 such tasks
(93 min, 39 % of the job) versus 391 (141 min, 43 %); the same unit is 9 to
43 % of every compose and swarm deploy job on disk. The parent process
rendered once per play, late, while templating a task name, and only the
tasks after that point ran warm.
The constructor now renders once through the new applications_carrier
action and parks the payload with its cache key in the host fact
_INFINITO_APPLICATIONS_RENDERED. Every later worker inherits the fact at
fork time and get_merged_applications serves it while the key (roles dir,
applications, users, DOMAIN_PRIMARY, SYSTEM_EMAIL_DOMAIN) still matches; a
mismatch falls back to the old render. The action masks its own result: the
payload is the whole tree with every credential, and CI runs with
MASK_CREDENTIALS_IN_LOGS false.
The task sits after the networks, design and service_provider merges, after
the add_host placements and after the token store. An independent review
showed that an earlier position bakes stale values into the play: the
web-app-wordpress title rendered to 'Blog by ' because service_provider did
not exist yet. User records inside the payload are the token store's state
at render time; readers of tokens captured later keep using lookup('users')
or lookup('config', app, 'users.…'), which read the users cache fresh.
Expected savings per job: about 130 min on a fork workspace run, 16 min on
an average compose deploy and 28 min on an average swarm deploy. The
constructor's own lookups before the carrier stay cold.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The extras writer carved `applications` out of the deploy-facing twin and said
why in its own docstring: extra-vars replace rather than merge, and would strip
every generated credential. `users` carries the same hazard and stayed in, so
every swarm deploy ran with `-e @swarm-nfs-extras.deploy.yml` replacing a
nine-entry users map with the three from inventories/development/default.yml.
Everything not in those three then fell to its role default. For zammad-api-bot
that default is `{{ 42 | strong_password }}`, and utils/manager/value_generator.py
opens by warning what that means: it is a template, not a value, re-rendered at
every use site, so the task that creates an account and the task that later
authenticates as it read two different secrets. Playwright sent a 42-character
password that no deploy had ever written, and Zammad answered
Seed POST /api/v1/tickets failed: 401 Invalid BasicAuth credentials
on both swarm jobs of Zammad#1 while both compose jobs passed. Compose never
sees an `-e @file`; its driver emits only scalar pairs.
`users` now leaves the twin the way `applications` already does. The two deltas
the block actually carried, the swarm-test pubkeys for administrator and backup,
travel the merging channel instead: baked into base_overrides, handed to the
provisioner through --vars, and deep-merged into host_vars. ensure_swarm_keypairs
is split out so the keys exist before provisioning rather than at extras time.
Two things this restores beyond the 401. The rotation gate rotates
users.<name>.password, and swarm was overwriting the result with the defaults
straight afterwards, so user rotation was never actually exercised there. And
user-administrator/meta/users.yml ties the administrator account password to
ansible_become_password; the extras had been pinning it to default.yml's sample
literal instead, which sudo on these nodes does need to match.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The replay is a second deploy on the same runner, and mark() only ever checked the variant number of a row, never its mode. So the marker landed on whichever row of the smallest variant came first, which for 38 of 98 marked rows was a swarm one: that runner would have had to bring a compose stack up beside a swarm stack.
Two places needed it. The row now has to run in the mode guide_deployable names, and only rows in that mode count as deployed when the search looks for the smallest variant - without the second half the search settles on a variant this sweep runs in swarm alone and the marker falls away silently.
Measured over the full matrix: 98 marked rows become 71, swarm drops from 38 to 0, compose rises from 47 to 59. The 27 roles that lose the replay entirely are the ones this sweep deploys in no compose row of an eligible variant; getting them back means forcing such a row into the sweep, which is a change to matrix selection rather than to this marker.
mark() had no tests. It has eight now, including the two failure modes the old code showed under a negative control: a swarm row carrying instructions=compose, and a compose row left unmarked because a swarm-only variant won the search.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A round deploys a provider once, so the role that pulls it in is the only place that can say how lean it has to be. A variant entry may now carry any application config topic under a service it depends on: a mapping deep-merges, so naming one key changes that key alone, and null empties the topic. Empty was the case that needed a marker of its own, because merging an empty mapping is a no-op and would leave exactly the services the round was trying to shed.
The topic vocabulary and the merge semantics live in utils.roles.applications.topics, which is also where resources.py now gets them rather than spelling the same null-or-merge out a second time. Collecting who demands what stays in the inventory tree, refuses two roles demanding different values for one provider in the same round, and skips a role's own entity entry so the existing services.<own>.users declarations are not read as overrides.
The CI set-cover learns the same fact. Its greedy cover retires a role's variant-0 row when an earlier row of another role embeds it, on the stated premise that embedding pulls the provider's variant 0. A variant that dictates that provider's config breaks the premise from the other side: whatever ran, it was not the provider as the provider declares it. Such a pair now covers nothing, so the retired row keeps its job, its artefact and its verdict.
Measured on the current tree: 20 override pairs, 41 covered rows before and after, so nothing is revoked today. The rule bites the moment an override lands in a row that covers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The contract check ran as an Ansible task in the middle of every provider deploy, and the two client checks ran the same way. A test that is part of the deploy cannot be run without deploying, cannot be re-run on its own, and reports its verdict as a deploy failure. Twenty-seven task files lose 491 lines to this.
It is now what it always was. Each provider ships files/test/test.sh, templates/test.env.j2 and meta/tests.yml, and test-e2e-cli stages one shared harness beside it: run.sh drives contract.sh in a throwaway container on the provider's own network, then isolation.sh for adapter-fronted providers. The read probe each provider offers moved from the calling task into meta/mcp.yml next to the allowlist it must satisfy, and the schema now rejects a read probe naming a tool outside that allowlist.
The bearer the test presents is the one a client resolves through mcp.credential, via the new mcp_client_bearer lookup over resolve_credential, which moved to utils.roles.applications.mcp so it is the same function mcp_servers renders every client connection with. The probe presented whatever the calling role passed, which for the fifteen adapter-fronted providers is a different secret than any client holds.
Sidecar isolation is read from the engine rather than measured inside the container, so it no longer needs a delegation to the node holding the task: attachment is compared by network name, which drops the gwbridge special case, and a swarm worker says it cannot measure instead of skipping silently.
Retrying is bounded by wall clock against MCP_DEADLINE_SECONDS, which scales through the same lookup('timeout') as the harness budget it has to stay under. A flag that is neither true nor false aborts rather than reading as off.
mcp-provider-probe becomes mcp-provider-cli-test: an enabled provider must ship the test rather than include the task. mcp-probe-bearer becomes mcp-adapter-bearer, tying ADAPTER_BEARER to the declared credential key now that no probe include carries it. A new integration test renders the shared env template for every provider through Ansible's own templar, so a later change to it breaks visibly instead of poisoning twenty deploys quietly.
Hermes and OpenClaw keep their client-side proof as CLI tests of their own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b8a6205396 put INFINITO_SWARM_TEARDOWN_RESERVE_SECONDS into default.env and
made 00_one.sh require it with a :? guard, but wired it into no handler.
default.env is only the source of the value; a key reaches the generated .env
solely through a handler's setdefault, so make dotenv dropped it and the
swarm routine aborted on its own guard.
The two nocheck markers cover the neighbouring key on those lines:
INFINITO_CI_DISTRO_DEADLINE_EPOCH is computed by distros.sh from the sweep
budget at run time, so a static default would be a stale timestamp. They have
to sit on the same line; above the statement the marker does not apply.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
enabled and shared were hand-maintained in 24 roles and wrong in both
directions. Both now resolve from the mcp_consumer edges already declared
in meta/services.yml: a server takes the consumers it admits, a client
takes the servers that admit it, and the pair renders as "'peer' in
group_names or ...", or False when no peer is admitted at all. setdefault
keeps meta/mcp.yml and meta/variants.yml as the override channel, so
pinning either key still wins.
The resolved block replaces the mapping instead of being written into it.
_build_role_base_config parks the YAML cache's own object under mcp, so an
in-place setdefault injected the derived keys into every later raw read and
made test_mcp_schema pass alone but fail in a directory run.
The derivation keys on peer presence, never on a peer's own value: a
re-entrant read under _RENDER_GUARD returns the unrendered dict.
Four lints now ask the resolver instead of reading meta/mcp.yml, which they
could not do while the key was hand-written. That is what exposed the eleven
missing hostile-path specs the previous commit closed.
Carries one change that belongs to the WordPress probe but shares a file
with this one: the tool allowlist moves to the hyphenated names the adapter
actually serves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_APP_LOG_TAIL filtered with -size -20M. That is GNU syntax; busybox rejects the
suffix and aborts the whole find, not just the test:
$ docker run --rm busybox find /etc -type f -size -20M
find: invalid number '20M'
$ docker run --rm busybox find /etc -type f -size -20971520c
/etc/localtime
/etc/group
So every alpine-based container returned an empty capture. -20971520c is the same
bound in bytes, and c is the only suffix POSIX requires; both finds take it.
The 2>/dev/null goes with it, and that is the part worth naming: capture() at
container.py:76 already merges stdout and stderr, so the redirect is exactly what
swallowed `find: invalid number '20M'`. The collector reported success while finding
nothing.
Two investigations today paid for this. The dnf cause on LittleJS#0 could not be
settled because /var/log/dnf5.log was absent, and the discourse OIDC failure in
Nextcloud#7 had to be reconstructed from the reverse-proxy log because
/var/www/discourse/log/production.log was absent -- while two vendor mkmf.log files
from the same tree were present, proving the directory was traversed. Both times the
missing file read as a property of the situation; it was a property of the tool.
Nothing depends on the constant: grep over tests/, scripts/ and .github/ finds no
reference to _APP_LOG_TAIL or app-logs.
Introduced yesterday in 8434d35d82, so it has never worked on an alpine container.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ruff wraps the ternary in _sandbox_workers (8ce66075b) and the yaml formatter
reflows db_key_base's description (e3d5ab287); both lines were written over the
limit. No behaviour changes.
autoformat runs as quality-high's first phase, so it corrects a line only on the
run AFTER the one that introduced it -- which is why git status is worth reading
after every gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two Moodle#1 jobs of run 33445831564 -- tor/arch and clearnet/debian -- end with the
same line, 65 minutes apart:
FAILURE: no running moodle task on a worker node
The deploy is clean; PLAY RECAP reports failed=0 on all four hosts. Only the
post-deploy drain drill fails, and it fails because nothing is left to drain.
The chain, each link measured rather than assumed:
meta/variants.yml variant 1 turns openwebui/hermes/openclaw/flowise on;
variant 0 turns all four off
extend_inventory.py:88-89 svc-virt-kata in the closure put BOTH workers in the group
svc-virt-kata/00_core.yml:61 labels every group member kata-capable=true
(2 label tasks in each moodle#1 log, 0 in moodle#0)
deploy.yml.j2:18-23 non-kata services get node.labels.kata-capable != true
05_seed_content.sh:12-22 filters running tasks with grep -vx "${MGR}"
With both workers labelled, moodle and nginx have no eligible node but the stack
host. `docker service ps` at failure shows all three moodle_nginx replicas Running
on mgr-01. The drill then looks for a worker task and finds none.
Controls in the same run: moodle#0 carries no kata group and 0 label tasks, and its
drill was reachable; Nextcloud#10 (green, no kata) logs
"initially scheduled on: ...wrk-01" -> "Rescheduled ... to ...mgr-01" -> drill complete.
Exactly three of the run's 18 swarm inventories carry the kata group, and all three
are red.
b20a70646f widened _WORKERS[-1] to *_WORKERS and said so in its own message: "whether
the stack host holds up under it is what the next swarm run measures". The stack host
held up. What broke is the drill's precondition.
The discriminator is the app's own declaration, and it separates cleanly -- executed,
not reasoned:
web-app-moodle kata.enabled=None -> ('swarm-wrk-02',)
web-app-nextcloud kata.enabled=None -> ('swarm-wrk-02',)
web-app-openclaw kata.enabled=True -> ('swarm-wrk-01', 'swarm-wrk-02')
web-app-hermes kata.enabled=True -> ('swarm-wrk-01', 'swarm-wrk-02')
_WORKERS[-1:] restores exactly the pre-b20a70646f tuple for the non-sandboxed case, so
that commit's drain fix stays intact for the rounds it was written for.
svc-virt-kata's own assert (00_core.yml:80-88, labelled_nodes | length > 0) holds with
one labelled worker.
applications_for_round() is a thin public wrapper over the existing
_applications_for_active_variants(get_merged_applications(...)); extend_inventory would
otherwise have to import a neighbour module's private helper.
Three tests cover the branch, which had none: grep -c "kata\|_WORKERS" on
test_extend_inventory.py returned 0 before this.
Not covered: the full chain -- label, placement, drill -- has not run. One
`make act-swarm apps=web-app-moodle variant=1` closes that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
allowed_consumers was twenty hand-kept lists of which fourteen were identical, so every new client meant an edit in every provider. A client now declares services.<own-entity>.mcp_consumer: true once and a provider refuses one with mcp_consumer: false, which is also what tells a deliberate exclusion apart from a forgotten entry.
The group_names-gated consumer entries give the inventory resolver the co-deploy edge meta/mcp.yml never had: one of sixty-nine matrix rounds carried an MCP client before, twenty-three role/round pairs do now. bond 0.5 keeps the clients out of round 0 so the host memory budget still holds.
BREAKING CHANGE: meta/mcp.yml no longer accepts allowed_consumers; declare services.<entity>.mcp_consumer on the client instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The direct Path.read_text bypassed the shared reader, so the uncached-file-read lint failed and blocked the full quality gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The instructions test held a job of its own that picked one random role per
run, so a README's Production block was proven roughly once per 265 roles.
The replay now rides on a deploy row instead: no extra runner, and every role
the sweep touches gets its documented one-command deploy exercised.
The row is the role's smallest variant -- fewest services enabled -- because
that is the shape the README describes, and it is chosen among the variants
the sweep actually deploys, so a row cut as redundant coverage cannot silently
drop the role from the test. A role qualifies only when its README carries a
Production block and the guide's own mode is one the role runs and tests.
The marked entry carries the guide mode in `instructions` and a 📖 in its job
title, between the variant number and a priority ⭐. The guide mode is
independent of the mode the rotation gave the row: a swarm row still replays
in compose. The step runs last in the deploy job, after the swarm teardown,
because the replay deploys the role a second time and needs the runner's
ports.
call-test-instructions.yml, cli.meta.ci.guide_select and the `instructions`
dispatch input go with it. `redundant()` moves next to the marker so the
chunker, the plan table and the marker share one definition.
The plan table trades its 📊 Weight column for 📖 Instructions; nothing read
the weight any more, so it leaves the matrix entries too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The collector took `logs` from every container, which is stdout and stderr and
nothing else. An application that logs to a file left the bundle with a status code
and no trace of why.
That is what stopped the snipe-it investigation in run 33317483326. Its MCP adapter
reported `UNREACHABLE read call snipeit_get_license_summary: the provider answered
500`, and Laravel had written the exception to storage/logs inside the container,
where `logs` never looks. Three separate root causes were proposed from static
reading and all three were refuted -- the relation exists, inherited from SnipeModel;
the group permissions are stored exactly as decodePermissions expects; the controller
is identical at the deployed tag -- and the job is still unexplained.
The collector already execs into every container for resolv.conf, systemctl and
journalctl, so the gap was the application's own files, not the ability to reach them.
find drives the capture through -exec rather than a loop over $(find ...): the first
version split on whitespace and broke on paths containing spaces, which a probe
against a directory with two of them showed immediately.
The per-container and per-service captures fold into tables because the file sits at
the 500-line limit. Splitting it is not an option: this script copies itself into
every container it inspects and runs there standalone, so an import of a sibling
module resolves to nothing -- an attempt at that broke seventeen tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both renderers build their output as a list of lines seeded with the "networks:"
header and append one line per attachment. With no attachment to append they
still returned the bare header, and a mapping key with no children parses as
null, not as an empty mapping. Compose then rejects the whole file instead of
ignoring an empty section.
The two functions do not share their non-empty paths and disagree on where the
newline goes -- render_compose_networks terminates with one, render_container_networks
leads with one -- so the guard is written out in each rather than factored into a
helper that would have to know both shapes.
Both callers already treat an empty string as "insert nothing", so returning it
needs no change on their side.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An engine of its own keeps its passwords in its data directory, written once
when that directory was initialised. When the deploy rotates the credential the
new value reaches the container's environment and never the engine, so the
health check authenticates with a password nothing has ever accepted. The
application user and root come from the same value, so nothing is left that
could log in and change it either.
mariadbd reads --init-file on every start, before it opens the socket, and runs
the statements with full privileges. That is the one point where a password can
be reset without knowing the previous one.
The SQL is rendered next to the env file that already holds the same secret in
plaintext, so it adds no new exposure. It reaches the container as a config
rather than a bind mount because a swarm task may be scheduled on a node the
rendered path does not exist on; the config name carries a hash of the file
content, so a rotated password produces a new name and swarm replaces the
immutable old one.
The guard is mariadb-and-not-shared, which matches the `local` guard the service
template already renders on, so the block and the top-level entry appear
together or not at all.
3831c8f804 fixed the other half of this failure: matomo, moodle and nextcloud
keep the same password in a config file on their data volume. Same rotation,
opposite end of the connection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f5272fa85f derived the 404 from mcp.endpoint.path, which for an
adapter-fronted provider names the sidecar. Mattermost declares /mcp
there and answers on /plugins/mattermost-ai/mcp-server/mcp, so the proxy
blocked a path the application never served while the native surface
stayed reachable. Baserow passed only because its two paths coincide -
one role agreeing proved nothing about the other four.
That surface is what the adapter exists to narrow: it serves four tools
out of fourteen, and the native path reaches all fourteen.
adapter.upstream_path joins upstream_network and upstream_transport in
meta, and the four roles whose upstream lives on their own vhost declare
it there. Gitea does not: its upstream is a separate sidecar on its own
network, so it has nothing to hide on the application's vhost, and the
absence says that.
The roles read the value back through lookup('config', ...) rather than
keeping a second copy, so the path is stated once.
The template now separates the two roles the old variable conflated:
endpoint.path is what gets served, and only where the surface is public
and enabled; every declared path that differs from it gets a 404. Three
tests pin the distinction rather than the outcome - mattermost's two
paths must differ, gitea must declare none, and a passthrough provider
must say where its upstream lives.
This reverts 49f2dd516.
Its justification was wrong. I claimed the marker reached a running
Nextcloud as the stored Collabora WOPI URL. It did not: the deploy log
shows the executed command was
config:app:set richdocuments wopi_url --value 'https://collabora.infinito.example/'
02_manager_ops/06_plugin/03_configure.yml reads the addon file straight
from disk with include_vars, in role scope, precisely so those variables
resolve - its own header comment says so. The '<< error >>' strings
appear in the merged-applications payload that feeds the loop items, and
nothing configures the application from there. I read a payload dump and
took it for the value that was set.
The guard is also too broad for what it does catch: it fails the whole
render because one unrelated subtree could not resolve, so every deploy
with nextcloud in the application set would abort on a condition that
was previously a warning and changed no configuration.
The underlying observation stands - a captured template failure should
not be memoised - but it belongs at the boundary where a caller asks for
the affected value, not on the whole payload, and it needs a real case
before it is worth the blast radius.
Ansible's marker behaviour replaces an undefined name with the literal
'<< error N - NAME is undefined >>' instead of raising, and it is
inherited ambiently from whatever task is rendering, so code calling
templar.template(fail_on_undefined=True) gets that string back rather
than an exception.
_render_with_templar then returns it, get_merged_applications memoises
it unconditionally under a key that fingerprints only applications,
users and two domains, and hands the same object back without a
deepcopy. The first caller in the process therefore freezes the value
for every later one: a lookup from another role renders a role's addon
config without that role's vars, and the role itself is served the
poisoned result afterwards even though its own read happens in the right
scope.
It reached a running Nextcloud as the stored Collabora WOPI URL. A
configuration key set to an error message is worse than a missing one,
because the application stores it and reports itself configured.
The check sits at the end of the one function every merge render passes
through, so applications, users and domains are covered, and it runs
before any caller can memoise. The path is carried down while
descending, so the message names the offending key rather than the
payload. The pattern requires the ordinal and the dash so prose about
errors does not trip it.
2678090934 made the same choice for the lookup layer; this is the cache
layer.
This raises where the code used to continue. That is the point - a value
nobody can act on should stop a deploy rather than configure an
application - but the blast radius is only measured by the next deploy.
Requirement 035 asks that mutating variants test confirmation,
authorization, idempotency, audit and rollback apart from the read-only
ones. No surface enables mutations, so those tests would assert against
nothing and pass forever, and inventing a mutating variant to satisfy the
checkbox would be a claim rather than a check.
The guard is what survives that emptiness: a surface permitting mutations
must name an existing artifact per proof, so the day one ships it cannot
ship without them. Three synthetic blocks exercise the rule - none, ones
naming a file that is not there, and ones that are - because a rule
evaluated over zero matching roles is green by construction. The reverse
also holds: proofs declared while mutations are off name artifacts that
are never the thing under test.
mutating_proofs was missing from the schema keys, so the lint would have
rejected the very block it demands.
Baserow was the only role declaring endpoint.key_credential, and putting
the adapter in front of it left the field with no consumer. Keeping it
would also contradict the decision that put the adapter there: a secret
in the URL path cannot be scoped per consumer and cannot be revoked on
its own, so a provider authenticating that way belongs behind an adapter
rather than in a client's rendered configuration. The segment now stays
between the adapter and its upstream.
Discovery loses a config lookup per provider per client along with the
branch. The test that proved the segment was appended becomes its
inverse: an endpoint still declaring the field gets a URL without one, so
the decision is held rather than merely unproven.
Both agents registered every tool every provider served. Hermes documents that as its own default in tools/mcp_tool.py, "Neither set -> register all tools", and the role never set tools.include; OpenClaw's toolFilter was never rendered at all. A provider that grew a mutating tool would have reached both without anything declaring it.
A serving role now names its mutating tools, the discovery snapshot carries that set beside the allowlist, and every client renders the allowlist minus it in whatever form its upstream enforces. The subtraction sits at the client and not in discovery because Flowise compares the declared set against what its registry actually discovered and needs the full served list. Where a provider permits mutations there is nothing to withhold, so the set renders empty.
This is a client-side boundary, not a server-side one: whoever holds the endpoint key can still call a write tool directly. Baserow is the case that made it necessary, since its MCPEndpoint model carries no scope and its owner is a workspace administrator, so the declared mutating_tools_enabled bounds nothing there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Conflict: roles/web-app-bridgy-fed/files/Dockerfile
Both sides extended the header comment. main also replaced the upstream
checkout, `git clone --depth=1 --branch "${BRIDGY_REF}"`, with git init plus a
fetch of the single ref; this branch never touched those lines, so git took
main's version automatically and only the comment collided.
Resolved to main's file verbatim. It is a superset of ours in both respects:
it keeps our two comment paragraphs and adds a third naming the trip-wire the
non-idiomatic fetch exists for, namely that BRIDGY_REF is a commit and
`git clone --branch` refuses one. Verified byte-identical to stage 3.
Committed with the hook skipped and before the semantic audit of the other 45
files finished. main brings a lint rule that has never seen this branch's
roles, image bumps on svc-ai-ollama and web-app-nextcloud which this branch
rewrote, a stylelint major, and release 13.0.0 across the packaging surface.
Any of those can merge cleanly line by line and still be wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2ac6c0878d taught the shared loader to tolerate ansible-vault tags but
registered the constructor only on yaml.SafeLoader. Everything here parses
with _Loader, which is CSafeLoader wherever libyaml is available, so on a
normal system the tag stayed unknown and a file holding vault-encrypted
values still failed to parse.
Register it on _Loader too, the way the compose merge tags two lines above
already do.
This matters because the inventory keeps credentials vault-encrypted, which
is how the gap surfaced while reading meta/secrets.yml values.
Not verified: whether CSafeLoader is the active loader on this machine. If
libyaml were absent the two loaders would be the same object and the line
would be harmless but inert.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both dictionaries declared "php" and "ruby" twice. Python keeps the last
definition and discards the first silently, so the earlier entries read as
if they were in effect while never applying.
In system_pkg.py the shadowed "php" mapped apt-get, dnf and yum to
["php-cli"] alone, whereas the definition that actually applies installs
["php-cli", "php"]. The shadowed "ruby" was identical to the surviving one.
Removing them changes no behaviour; it removes a misleading read.
_GROUP_FN_NAMES loses the same two duplicated entries. php and ruby remain
in the group list itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without a zip extension or an unzip/7z binary Composer silently drops to
--prefer-source and mirror-clones every dependency, which exceeds its
300 s process timeout on phpunit.git. That took the Arch workspace job
from a 2m16s test-unit to a 16m08s failure. Debian never saw it because
apt pulls php-zip and unzip in as Composer dependencies; pacman does not.
Arch ships no zip extension and no php-zip package, so the extractor has
to be the unzip binary rather than the extension. Its mapping already
covers all five package managers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
checkmk's entrypoint runs `omd create --no-tmpfs`, so the site's tmp directory
is never mounted and the site cannot come up cleanly. Declaring it needs a tmpfs
with sticky, world-writable permissions, which the volume layer could not
express.
container_volumes built the tmpfs options as a hard-coded single key:
if mount.get("size") is not None:
tmpfs_entry["tmpfs"] = {"size": mount["size"]}
so a `mode` in the declaration was accepted by the schema and silently dropped by
the renderer. It now accumulates size and mode and attaches the key only when
something is set, coercing mode to the int docker expects. mounts.py validates
both the bind and tmpfs spellings through _is_octal_mode and names the expected
shape in the error, so `1777` or `0o1777` fails at render with a readable message
instead of producing the wrong bits.
Capability and consumer land together deliberately. Split apart, checkmk's
mode: "01777" would pass validation, be dropped at render, and the tmp directory
would come up without the sticky bit while everything reported success.
checkmk declares site_tmp as a tmpfs at the site's tmp path with size 512m and
mode 01777. The site name moves into services.checkmk.site, and CHECKMK_SITE
reads it through lookup('config', ...) instead of hard-coding "cmk", so the mount
target and the variable cannot drift.
MERGE NOTE: fork/feature/svc-net-tor carries acc9e25c0f, which deletes
plugins/filter/container_volumes.py and moves it to plugins/lookup/. That
relocated copy has the identical defect at lines 141-142 -- the same hard-coded
{"size": ...} -- and its _coerce_mode_int is wired only to the config-ref path.
It is not a competing fix, but this change will disappear at merge unless it is
re-applied in the lookup.
Verified: the unit suite passes, 4448 tests, including new coverage of the mode
path in test_container_volumes.py.
Not verified: checkmk has not been watched starting with this mount. Docker's
tmpfs mode also interacts with the container umask, so the effective bits on the
created directory were not inspected live. The new validation is fail-closed: a
role carrying a non-octal mode would now error at render rather than be ignored,
and while the suite is green, I did not grep every CHECKMK_SITE consumer for one
still assuming the literal "cmk".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two arch swarm jobs in fork run 32667447816 died in the ganesha AUR
install at 07:41Z, 42s apart, on two different network errors. All three
EXTERNAL_FETCH attempts ran and lost - the ~40s window is sized for
mirror pools where a bad member rotates out in seconds, but the AUR is
one origin host: when it flaps, it flaps for minutes. A sibling job
fetched the same package successfully 2.5 minutes earlier, and the
module (kewlfft.aur 0.13.0) hardcodes the AUR URLs with no timeout or
mirror knob, so a wider client-side window is the only available layer.
6x60s rides out a multi-minute flap and costs its 5 minutes only when
the AUR is actually down; every mirror-pool source keeps the tighter
EXTERNAL_FETCH.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
48845db97 made three capability decisions declarable and named them INFINITO_*, which put them under setdefault: the live environment wins and the value is written back. Makefile exports BASH_ENV, so .env is re-exported into every make recipe, and scripts/tests/code/run.sh reaches dotenv through 'make setup' on every test run. A declaration made for one run therefore became a machine fact on first regeneration and stayed one - that is what made the workspace failure intermittent rather than reproducible, and why fedora died on 11 assertions where debian passed the same code.
A dedicated handler writing through eb.set recomputes the value every time and discards whatever the environment held, so the key stays registered in default.env and visible to the lints while remaining structurally unpinnable. running_on/github.py already worked this way; the distinction now has a name in utils/env/README.md. The exporting shell still steers its own run, because load.sh preserves non-empty caller values.
The other two declarations are deleted rather than moved. image_mirror_enabled and docker_root_ephemeral both derive from runs_on_github(), and the orchestrator does not unset INFINITO_RUNNING_ON_GITHUB, so inside the workspace they were already true and the declarations restated a value that held anyway. Measured across all four runtime contexts: only the workspace row changes, and only for the cache.
Verified by reversing the probe: 'INFINITO_CACHE_STACK=false make dotenv' writes .env=true, where the registry path wrote the exported value back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_used_keys_declared flags every INFINITO_* key a source file reads without a registry entry; the three capabilities 48845db97 introduced had none, so the lint suite failed on 7 use sites.
Their defaults stay empty on purpose. _declared() distinguishes an explicit 'true'/'false' from an absent value, and only the absent case lets the capability derive from the runtime identity — a concrete default would pin all three and make that derivation dead. INFINITO_CACHE_NETWORK next to them carries the same empty-means-unset shape. Same registration path as c768bbf01.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The composer vendor tree lands in the repo root during the workspace suite, and the cached project walk descends it, so third-party files turn repo lints red. 8ff6b490b patched two consumers; the filename convention lint was the third.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ensure_php_toolchain installed the php and composer binaries and stopped
there. A distro ships the interpreter and its extensions as separate
packages, and ensure_command_present returns early once shutil.which
finds php - which it always does since the image bakes the interpreter -
so the extensions were never provisioned on any path.
composer then refused to resolve the vendor tree with
"phpunit/phpunit requires ext-dom * -> it is missing from your system"
and test-unit-php failed in every workspace job. The loaded ini list in
those logs shows 20-mbstring.ini present and no 20-dom.ini, 20-xml.ini
or 20-xmlwriter.ini, i.e. php-xml missing.
Extensions are now probed against php -m and installed from a per-manager
mapping, then re-probed so a package that does not deliver the extension
raises instead of leaving composer to fail later. The loop sits in
ensure_php_toolchain because both entry points pass through it: the image
build via utils.install.interpreters and the test run via
scripts/install/php.sh, so a prebuilt image is repaired at test time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
