start_interval is the cadence while start_period runs, and only while it runs.
Once the window closes the probe falls back to interval, so a service that
becomes ready just after it pays a whole tick anyway and the cadence never
reached it. Six of the seven flavors were shaped that way, Curl worst: a thirty
second window against a one minute interval.
Measured across 151 converge waits in four job logs, 63 sit exactly on a tick
boundary, 36 in the thirty second step and 27 in the sixty second one.
Attributed by application, web-app-seaweedfs is the largest at 1501s over
sixteen waits with a median of 57.7s: it probes with curl and fell straight into
the gap. web-app-zammad probes with connect, whose window is fifteen minutes,
and its 154s median is real boot time with nothing to recover.
So the window moves to ten minutes and the steady state interval to two, and the
flavor overrides that sat below them go: Curl, Wget and Connect lose interval
1m, Http its twenty second window, Nc its ten. Connect keeps fifteen minutes.
Neither number costs anything while a container works. start_period delays
nothing, it only stops failures counting, and interval no longer touches
convergence at all since that hangs on the window alone; it is steady state
load, which drops from 120 probes an hour per service to 30.
Forty six role files follow. Twenty seven declarations go that repeat the new
default or undercut it, an undercut window being the defect itself. Eight
windows longer than ten minutes keep their value and now say why. The literal
blocks and the three Dockerfiles this repository builds spell every timing out,
because outside the lookup an omission takes docker's value and not the
platform's.
A fourth lint rule keeps the shape: start_period must outlast one interval,
checked against the effective values, so raising interval without the window
cannot quietly bring the cliff back.
Two faults in the lint surfaced while building it and are fixed here. It
compared timings as strings, so 600s and 10m read as different values, and it
matched the healthcheck key exactly, so a line carrying a trailing nocheck was
skipped whole, which is how the two swarm-init partials went unexamined.
One regression, bounded. A container that starts and never turns healthy is
neither rejected nor errored, so stack_ready.sh only calls it fatal after three
killed tasks, and each cycle now spans the longer window. That reaches twenty
minutes rather than six, capped there by swarm_stack_converge_retries at 600
with a two second delay, and it lands only on runs that were failing anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
42ebd9ea71 gave the healthcheck SPOT start_interval, the cadence docker uses
while start_period is still running. A block a template writes out itself
bypasses the SPOT and never gained it. Measured across the repository: 46 such
blocks, 40 of them already carrying start_period, not one carrying
start_interval.
Without it a probe fires only every interval, so a service ready after two
seconds is reported healthy at the next tick and a stack converges no faster
than its coarsest quantisation. That lands in sys-svc-compose's swarm converge
wait, 1551.7s per CI job over 38 stack deploys.
Twenty five of the blocks move into meta/services.yml and render through
container_healthcheck. Each keeps an explicit test rather than picking a flavor:
a flavor would replace the command instead of carrying it across.
The move is lossless by construction. probes.py block() writes
overrides.get(key, default) for interval, timeout, retries, start_period and
start_interval, so a declaration repeating the literal values renders the same
block plus the missing cadence. Checked per block against the old values: the
only difference is start_interval 5s, nothing changed, nothing lost.
Twenty one stay literal. Seventeen interpolate a template variable, whose value
resolves in a scope meta/services.yml does not share, so moving them would be a
rewrite rather than a transcription. Four interpolate nothing and still cannot
move: svc-db-memcached and svc-db-redis probe every second in their swarm
service partial, where a 5s start cadence would report healthy later than the
block already does, and the sys-svc-rdbms and keycloak sso_proxy partials render
for the consuming role's application_id, so their declaration has no single
meta/services.yml to live in. Each carries a nocheck naming its reason.
All twenty one gain start_interval. Three also gain start_period, without which
the key does nothing at all, since it only applies while the start period runs.
That widens their failure tolerance rather than narrowing it: bluesky from 90 to
120 seconds, magento from 50 to 80, the mariadb partial from 180 to 210.
pixelfed spelled its probe as a bare string and now spells it CMD-SHELL. Docker
reads both as a shell command; only the YAML shape changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hermes_hermes.1 ran on a worker while run_one.yml ran its script on the loop
host. resolve_container_id.sh finds nothing node-locally on the manager, falls
through to the cluster-wide branch and returns the task's ContainerID, which
only the owning node's daemon knows, so `container exec` answered
`No such container: 65801ca259a1` twenty times over 116 seconds. Run
33785198875, job 100756596935.
Resolve the owning node and run there. meta/tests.yml gains
container_service_key for the two roles that exec into their own container; a
role without the key keeps the loop host, and role_tests returns the default
for a missing path, so absence is a polarity rather than an error.
The reset at run_one.yml:27-30 is unconditional and precedes every
delegate_to. Without it the first loop iteration dies with
'CLI_HOST_NODE' is undefined, because _cli_apps starts with a role that has no
key; the same task closes the fact leak that include_tasks leaves open between
iterations. The resolve include is gated on an explicit length check rather
than truthiness, or a keyless role would hang in resolve_host_cid's converge
wait, and it caps that wait at 30 x 2s instead of inheriting 600 x 2s: the
deploy is long finished by the time the CLI stage runs, so failing fast beats
waiting twenty minutes.
Staging, render and run all delegate. Delegating only the run would turn
`No such container` into `No such file or directory`, since the assets would
stay on the loop host. Both templates now carry CLI_LOCAL_CID: with the exec
on the owning node the id is node-local, so container_address's deferred
`$( ... )` is not needed, and it never survived `set -a && . test.env` anyway,
which evaluates it once at source time.
The stage directory is removed on the delegate too. tasks/main.yml clears it
without delegation, i.e. only on the loop host, and the task's own name calls
its contents decodable DB secrets.
Measured locally rather than argued: self-delegation leaves file, copy with
content, copy with src and shell byte-identical, including mode; the loop
reset gives keyless roles an empty container and keeps CLI_HOST_NODE on the
loop host across five iterations; and container_service_key resolves to
hermes, openclaw and empty strings through the same walk cli.timeout already
uses in production.
Not fixed, and named so it is not forgotten: web-app-joomla, web-app-yourls
and web-app-penpot exec from an IS_STACK_HOST gate without resolve_host_cid,
so a task of theirs landing on a worker hits the same error from the same
source.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Hermes and OpenClaw wrapped the tool allowlist in a guard that dropped the key when the callable set was empty. Both upstreams read an absent filter as "register every tool the server serves", so a provider resolving to no tools got the widest surface instead of none. Nextcloud keeps its contract in tools.categories with no allowlist and admits both clients, so it was the live case.
The templates now render the filter unconditionally, the way the n8n client node already pins include to selected; the two tests that asserted the omission are inverted. mcp_servers additionally rejects such a provider with the fatal code tools_unenforceable: a client filters by tool name, so a runtime-composed surface reaches it as no surface at all, and a server the agent can never call is worse wired in than refused with a reason.
A deployment pairing Nextcloud with either agent now aborts naming the provider, where it previously handed the agent every Context Agent tool unfiltered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The contract check ran as an Ansible task in the middle of every provider deploy, and the two client checks ran the same way. A test that is part of the deploy cannot be run without deploying, cannot be re-run on its own, and reports its verdict as a deploy failure. Twenty-seven task files lose 491 lines to this.
It is now what it always was. Each provider ships files/test/test.sh, templates/test.env.j2 and meta/tests.yml, and test-e2e-cli stages one shared harness beside it: run.sh drives contract.sh in a throwaway container on the provider's own network, then isolation.sh for adapter-fronted providers. The read probe each provider offers moved from the calling task into meta/mcp.yml next to the allowlist it must satisfy, and the schema now rejects a read probe naming a tool outside that allowlist.
The bearer the test presents is the one a client resolves through mcp.credential, via the new mcp_client_bearer lookup over resolve_credential, which moved to utils.roles.applications.mcp so it is the same function mcp_servers renders every client connection with. The probe presented whatever the calling role passed, which for the fifteen adapter-fronted providers is a different secret than any client holds.
Sidecar isolation is read from the engine rather than measured inside the container, so it no longer needs a delegation to the node holding the task: attachment is compared by network name, which drops the gwbridge special case, and a swarm worker says it cannot measure instead of skipping silently.
Retrying is bounded by wall clock against MCP_DEADLINE_SECONDS, which scales through the same lookup('timeout') as the harness budget it has to stay under. A flag that is neither true nor false aborts rather than reading as off.
mcp-provider-probe becomes mcp-provider-cli-test: an enabled provider must ship the test rather than include the task. mcp-probe-bearer becomes mcp-adapter-bearer, tying ADAPTER_BEARER to the declared credential key now that no probe include carries it. A new integration test renders the shared env template for every provider through Ansible's own templar, so a later change to it breaks visibly instead of poisoning twenty deploys quietly.
Hermes and OpenClaw keep their client-side proof as CLI tests of their own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The provider block ec736bd4d7 added carries only an id per model. OpenClaw
validates the whole file before serving anything and rejected it:
Gateway failed to start: Invalid config at /home/node/.openclaw/openclaw.json:
models.providers.litellm.models.0.name: Invalid input
A rejected config is not a degraded start. The gateway retries, trips its own
breaker after four unclean boots, and the container ends at exit 128. The
deploy notices only at the very end, when the container health check gives up
after 25 attempts - twenty-five minutes and a role away from the cause.
The model itself was fine: the rendered file on the host held
[{"id": "qwen2.5:0.5b"}], so this is the missing key and not the
LITELLM_CHAT_MODEL fallback. name now sits beside id with the same value.
Which of the two reaches the API is the upstream's business; identical values
cannot send a different model than the one agents.defaults names.
It shipped because nothing rendered it. All three existing cases in
test_mcp_tool_filter.py pass OPENCLAW_LITELLM_ENABLED=False, so the block was
skipped and its shape was never compared to anything. The new test renders it
with LiteLLM on, and against the old template it fails on KeyError: 'name'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both agents registered every tool every provider served. Hermes documents that as its own default in tools/mcp_tool.py, "Neither set -> register all tools", and the role never set tools.include; OpenClaw's toolFilter was never rendered at all. A provider that grew a mutating tool would have reached both without anything declaring it.
A serving role now names its mutating tools, the discovery snapshot carries that set beside the allowlist, and every client renders the allowlist minus it in whatever form its upstream enforces. The subtraction sits at the client and not in discovery because Flowise compares the declared set against what its registry actually discovered and needs the full served list. Where a provider permits mutations there is nothing to withhold, so the set renders empty.
This is a client-side boundary, not a server-side one: whoever holds the endpoint key can still call a write tool directly. Baserow is the case that made it necessary, since its MCPEndpoint model carries no scope and its owner is a workspace administrator, so the declared mutating_tools_enabled bounds nothing there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two persona-block-rationale directives, in the form aef525f91 used for
hermes, homeassistant and litellm.
The Control UI is a WebSocket SPA whose session sits behind gateway device
pairing and exposes no DOM logout control for the shared inAppLogout helper,
so files/playwright/test-oidc-login.js covers the SSO round trip instead.
biber is refused at the oauth2 proxy, which admits only the administrator
and mcp RBAC groups.
Both checkable claims verified: allowed_groups lists exactly those two
groups, and the substitute spec exists. Coverage moves rather than
disappears.
The missing logout control itself is not verified; openclaw was not deployed
this session, and it is a product claim that a UI update could invalidate
without anyone touching the directive.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
env.j2 hardcoded four container paths that meta/volumes.yml already declares.
Read the volume targets instead, so a changed mount cannot leave the
environment pointing at a path that no longer exists.
Value-identical, checked against the declaration rather than assumed:
openclaw_data.target is /home/node/.openclaw and openclaw_config.target is
/home/node/.openclaw/openclaw.json, and path_join reproduces the workspace
path exactly.
Same pattern as a772115b8 for litellm, but verified by value here rather
than by the service having worked.
openclaw was not deployed this session, so this is a static check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
acc9e25c0f moved the volume lookups to context-wired signatures, taking
application_id from the task context instead of an argument:
lookup('container_volumes', application_id, service_name) -> (service_name)
lookup('compose_volumes', application_id) -> ()
Six templates were left on the old form: the three AI services and hermes,
homeassistant and openclaw. One of them also picks up the matching
container_healthcheck lookup.
This is the same failure mode as the php paths: a refactor that leaves
callers behind, and the break only appears when the template renders, not in
the gate. A deploy in this session died exactly that way, on a template still
passing application_id to a lookup that no longer takes terms.
No file in the repository carries the old signature after this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The block sat under services but never behaved like a service. A service has
exactly one providing role, declared with provides, so the registry can map the
key to that role and consumers bind their flag to its presence. No role declares
provides: mcp. There are seventeen providers and the pairing is done by
allowed_consumers inside the block itself.
Every rule that enforces the service model therefore needed an exemption for it:
a shared-provider suppression in fourteen roles, a playwright-service-flag
suppression, and two freshly added exemptions in the dynamic-flag and
variants-static-override checks. Four independent rules, four escape hatches,
one cause. They are all gone now, together with a bond key that only the
resource budgeter reads and only for entries under services, and a wordpress
addon bridge that pointed at a service block that no longer exists.
The block moves to meta/mcp.yml. The applications loader maps meta/<name>.yml to
applications.<app>.<name> generically, so the move needs no wiring; the file
takes the same shape as meta/rbac.yml. Every config path follows, the variant
overrides are lifted out of their services map, and the vocabulary module leaves
the services package because it no longer describes one. roles_with_service
gains an explicit topic parameter rather than an implicit fallback, so nothing
silently resolves against the wrong container.
Files that are otherwise unrelated to this move are included where the committed
tree would otherwise keep pointing at a path that no longer resolves: thirteen
vars files, eleven playwright env templates and one task file.
Render every discovered server into the agent's own config, filtered to
the auth schemes a client can actually present, so no entry carries a
credential the server would reject.
Render hermes' config after the stack exists and before the handlers
flush; the other order fired compose up while the config was still
missing.
Own the config file as the container user. Rendered as root it stays
unreadable to the agent, which then falls back to its built-in defaults
silently and reports only that the server is unknown, pointing the reader
at the wrong layer.
Assert what these clients expose: neither ships an admin surface listing
its configured servers, so the specs hold the API to refusing an
unauthenticated caller, which is what keeps the per-server bearers out of
reach.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MCP_DISCOVERED_SERVERS resolves every server-direction role to its internal URL, the bearer the server role minted, and its transport, dropping any server without a token so no client renders a connection it cannot authenticate.
OpenClaw moves from 2026.3.11 to 2026.7.1 because the pinned version has no configuration surface for outbound MCP servers at all: its validator rejects mcpServers, the shipped bundle carries no bundle-mcp, and the ACP path refuses per-session servers while pointing at a gateway config that does not exist there. 2026.7.1 reads mcp.servers, which the template now renders. The gateway command gains its run subcommand because the bare gateway no longer starts anything.
Reachability was the missing half: openclaw sat on its own networks only, so the hub name resolved through the DNS wildcard to the ingress instead of the container. The hub overlay now declares an mcp consumer key, which attaches every role carrying services.mcp.
A probe task runs openclaw mcp probe after the handlers flush and fails the deploy when a configured server does not answer. Without it the previous run reported failed=0 while the client could not reach anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sandbox tier carried no check of its own. Three agent replicas once ran on
the shared kernel while every test stayed green, because nothing between the
runtime selection and the running container ever asked what the container
actually got. Each guard below sits where the promise first breaks.
A node is declared a sandbox target by a label that routes every sandboxed
service to it. It now has to prove it can carry them first: the node reads its
own default runtime and refuses the label while that is still runc. Doing it
before the label rather than after is the point, since the label is what makes
the promise.
Compose has no node-wide default to inspect, so the agent roles read the
runtime the container actually received once the stack is up. The check is
skipped in embodied mode, where the runtime is dropped on purpose because a
microVM between the agent and the hardware it drives defeats the point, and it
keys on the same services.kata switch that already drives the placement
constraint and the compose pin.
A runtime is only as good as the template around it. The new lint refuses a
privileged container, a host network, pid, ipc or user namespace, and a host
docker socket bind in any role that declares a kata service. It fails when it
finds no sandboxed role at all, so renaming the service cannot silently retire
the guard.
It caught its own subject on the first run: both agent templates emitted
privileged: true next to the device allowlist, which grants every device and
made the allowlist decorative. The requirement is explicit that no device
outside the allowlist reaches the agent, so the flag is gone and the role
README no longer claims a limit that was not enforced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`svc-virt-kata` is pulled in as a shared service of the agent roles, so it also
executes on hosts that are in no inventory group of that name. Gating the
runtime registration on `'svc-virt-kata' in group_names` therefore left such a
host with runsc installed and its node labelled kata-capable, while
`daemon.json` registered nothing and the daemon kept running everything under
runc: the exact isolation breach the label promises is closed. The stat probes
in `daemon_config.yml` decide instead, and they cannot be fooled by an
inventory that says otherwise.
Nothing else in the tree installs or references runsc, so a shim on disk is
proof that this role put it there.
The agent compose templates pin their runtime on the same `services.kata`
switch that already drives the swarm placement constraint, instead of pinning
unconditionally against a daemon config written under a different condition.
Both roles declare that key themselves, so the lookup is the strict form: a
deleted flag now aborts the render rather than silently starting the agent
without a sandbox.
The new unit test covers what nothing covered before, which is how the runc
breach went unnoticed: an absent shim is never registered, a default runtime is
always a registered one, the stack host keeps the shared kernel, compose never
sets a daemon-wide default, and registration follows the probe rather than the
inventory group.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Requirement 034 inverts the isolation model of the other agent roles: the
boundary is the whole machine rather than a kernel boundary inside it.
Adding this role to a device switches the agent from isolated to
embodied.
The agent then runs privileged with exactly the devices from
ROBOT_DEVICE_ALLOWLIST bound into its container, and the kernel-isolating
runtime is dropped, because a microVM between the agent and the hardware
it is meant to drive defeats the purpose. The allowlist lives in
group_vars so the agent templates can read it; a role variable is not
visible across roles.
The admission checks run inside the agent role before it deploys rather
than in this role afterwards. The agent is declared shared, so the
service loader brings it up during the constructor stage while svc-ai
roles only run at the following step; a check placed here would sit
behind the deployment it exists to prevent.
The dedicated-device check now compares against something group_names
actually carries: any other web-app role on the host. The previous list
held Docker node labels, which are a different namespace from Ansible
inventory groups, so that intersection could never match and the failure
message always blamed a multi-node swarm.
Which agent embodies the device follows from the topology: both agent
flags take the dynamic in-group_names form, Hermes Agent wins when both
are present, and the matrix pins one per round so neither path rots.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The second agent of requirement 032, pinned to the same isolating runtime
as hermes. Unlike hermes it exposes a browser Control UI rather than a
bearer API, so it takes the oauth2 proxy flavour with its own sso port;
a proxy in front of a browser surface is correct where it would break
OpenAI-compatible clients.
Three deploy failures shaped this role. OpenClaw maps no environment
variables onto config keys, so it restart-looped on 'Missing config' and
then on 'non-loopback Control UI requires allowedOrigins'; both now come
from a rendered openclaw.json mounted as a config. That render must run
after sys-stk-full, because it notifies compose up and the handler would
otherwise fire before the compose file exists. And the process runs as
root: OpenClaw refuses a config file not owned by the running uid and
writes cron state into its home, neither of which a non-root user can do
against the platform's root-owned volume and config mount. gVisor is the
isolation boundary here, not the in-guest uid.
The administrator persona is blocked: the Control UI is a WebSocket SPA
behind gateway device pairing with no DOM logout control for the shared
helper. The SSO round-trip itself is covered by test-oidc-login.js, which
drives the full unauthenticated to Keycloak to authenticated path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
