The CLI test discarded the exec status with `|| true` and judged the run on
its output alone: non-empty, and not carrying a `failed to start server`
line. `Error response from daemon: No such container: <id>` satisfies both,
so the script echoed the daemon's error, printed ALL CHECKS PASSED and
exited 0. CI reported green for a container it never reached.
That path was live in run 33785198875 job 100756596935, where hermes_hermes.1
sat on a worker while roles/test-e2e-cli ran the exec on the manager. The
harness half of that failure is a separate change; this one is about the
predicate, which would have reported success for any unreachable container in
any mode.
Capture the status and require it. The output guard stays as it is: no
positive success marker is introduced, because nothing in the repository
records what `mcp probe` prints when it succeeds, and a guessed marker can
break a working test. web-app-hermes keys on `Tools discovered:` for the same
reason the other way round, but that string is its own.
The condition gains a conjunct, so no case that was checked before is
unchecked now. The residual runs the other way: should the probe ever exit
non-zero while still producing usable output, a run that passed will now
fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The contract check ran as an Ansible task in the middle of every provider deploy, and the two client checks ran the same way. A test that is part of the deploy cannot be run without deploying, cannot be re-run on its own, and reports its verdict as a deploy failure. Twenty-seven task files lose 491 lines to this.
It is now what it always was. Each provider ships files/test/test.sh, templates/test.env.j2 and meta/tests.yml, and test-e2e-cli stages one shared harness beside it: run.sh drives contract.sh in a throwaway container on the provider's own network, then isolation.sh for adapter-fronted providers. The read probe each provider offers moved from the calling task into meta/mcp.yml next to the allowlist it must satisfy, and the schema now rejects a read probe naming a tool outside that allowlist.
The bearer the test presents is the one a client resolves through mcp.credential, via the new mcp_client_bearer lookup over resolve_credential, which moved to utils.roles.applications.mcp so it is the same function mcp_servers renders every client connection with. The probe presented whatever the calling role passed, which for the fifteen adapter-fronted providers is a different secret than any client holds.
Sidecar isolation is read from the engine rather than measured inside the container, so it no longer needs a delegation to the node holding the task: attachment is compared by network name, which drops the gwbridge special case, and a swarm worker says it cannot measure instead of skipping silently.
Retrying is bounded by wall clock against MCP_DEADLINE_SECONDS, which scales through the same lookup('timeout') as the harness budget it has to stay under. A flag that is neither true nor false aborts rather than reading as off.
mcp-provider-probe becomes mcp-provider-cli-test: an enabled provider must ship the test rather than include the task. mcp-probe-bearer becomes mcp-adapter-bearer, tying ADAPTER_BEARER to the declared credential key now that no probe include carries it. A new integration test renders the shared env template for every provider through Ansible's own templar, so a later change to it breaks visibly instead of poisoning twenty deploys quietly.
Hermes and OpenClaw keep their client-side proof as CLI tests of their own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role-files-layout lint wants scripts under files/ sorted into
per-language folders, which dfe7a7fdd did for 24 of them. Playwright suites
are exempt: the runner copies the tree as it stands, discovery works by path
and the specs import their helpers relatively, so sorting by extension would
break both.
83 roles already carry the marker; hermes, homeassistant, litellm and
openclaw were missing it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twenty-five specs called the Playwright APIs with default or hardcoded
timeouts. Routed over Tor those budgets are too small, so a spec fails on
transport latency rather than on the behaviour it asserts.
Route the waits through resolveTimeout and reach pages via gotoOnion, the
helpers the tor branch introduced for exactly this. The remaining changes in
the diff are import wrapping around those calls.
The repository lint enforces this: a raw numeric timeout in a role spec is
rejected with a pointer to resolveTimeout, which is how a hardcoded 500ms
wait added during this session was caught.
Merge work rather than authored here; not deployed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Render every discovered server into the agent's own config, filtered to
the auth schemes a client can actually present, so no entry carries a
credential the server would reject.
Render hermes' config after the stack exists and before the handlers
flush; the other order fired compose up while the config was still
missing.
Own the config file as the container user. Rendered as root it stays
unreadable to the agent, which then falls back to its built-in defaults
silently and reports only that the server is unknown, pointing the reader
at the wrong layer.
Assert what these clients expose: neither ships an admin surface listing
its configured servers, so the specs hold the API to refusing an
unauthenticated caller, which is what keeps the per-server bearers out of
reach.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The second agent of requirement 032, pinned to the same isolating runtime
as hermes. Unlike hermes it exposes a browser Control UI rather than a
bearer API, so it takes the oauth2 proxy flavour with its own sso port;
a proxy in front of a browser surface is correct where it would break
OpenAI-compatible clients.
Three deploy failures shaped this role. OpenClaw maps no environment
variables onto config keys, so it restart-looped on 'Missing config' and
then on 'non-loopback Control UI requires allowedOrigins'; both now come
from a rendered openclaw.json mounted as a config. That render must run
after sys-stk-full, because it notifies compose up and the handler would
otherwise fire before the compose file exists. And the process runs as
root: OpenClaw refuses a config file not owned by the running uid and
writes cron state into its home, neither of which a non-root user can do
against the platform's root-owned volume and config mount. gVisor is the
isolation boundary here, not the in-guest uid.
The administrator persona is blocked: the Control UI is a WebSocket SPA
behind gateway device pairing with no DOM logout control for the shared
helper. The SSO round-trip itself is covered by test-oidc-login.js, which
drives the full unauthenticated to Keycloak to authenticated path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5