The CLI test discarded the exec status with `|| true` and judged the run on
its output alone: non-empty, and not carrying a `failed to start server`
line. `Error response from daemon: No such container: <id>` satisfies both,
so the script echoed the daemon's error, printed ALL CHECKS PASSED and
exited 0. CI reported green for a container it never reached.
That path was live in run 33785198875 job 100756596935, where hermes_hermes.1
sat on a worker while roles/test-e2e-cli ran the exec on the manager. The
harness half of that failure is a separate change; this one is about the
predicate, which would have reported success for any unreachable container in
any mode.
Capture the status and require it. The output guard stays as it is: no
positive success marker is introduced, because nothing in the repository
records what `mcp probe` prints when it succeeds, and a guessed marker can
break a working test. web-app-hermes keys on `Tools discovered:` for the same
reason the other way round, but that string is its own.
The condition gains a conjunct, so no case that was checked before is
unchecked now. The residual runs the other way: should the probe ever exit
non-zero while still producing usable output, a run that passed will now
fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The contract check ran as an Ansible task in the middle of every provider deploy, and the two client checks ran the same way. A test that is part of the deploy cannot be run without deploying, cannot be re-run on its own, and reports its verdict as a deploy failure. Twenty-seven task files lose 491 lines to this.
It is now what it always was. Each provider ships files/test/test.sh, templates/test.env.j2 and meta/tests.yml, and test-e2e-cli stages one shared harness beside it: run.sh drives contract.sh in a throwaway container on the provider's own network, then isolation.sh for adapter-fronted providers. The read probe each provider offers moved from the calling task into meta/mcp.yml next to the allowlist it must satisfy, and the schema now rejects a read probe naming a tool outside that allowlist.
The bearer the test presents is the one a client resolves through mcp.credential, via the new mcp_client_bearer lookup over resolve_credential, which moved to utils.roles.applications.mcp so it is the same function mcp_servers renders every client connection with. The probe presented whatever the calling role passed, which for the fifteen adapter-fronted providers is a different secret than any client holds.
Sidecar isolation is read from the engine rather than measured inside the container, so it no longer needs a delegation to the node holding the task: attachment is compared by network name, which drops the gwbridge special case, and a swarm worker says it cannot measure instead of skipping silently.
Retrying is bounded by wall clock against MCP_DEADLINE_SECONDS, which scales through the same lookup('timeout') as the harness budget it has to stay under. A flag that is neither true nor false aborts rather than reading as off.
mcp-provider-probe becomes mcp-provider-cli-test: an enabled provider must ship the test rather than include the task. mcp-probe-bearer becomes mcp-adapter-bearer, tying ADAPTER_BEARER to the declared credential key now that no probe include carries it. A new integration test renders the shared env template for every provider through Ansible's own templar, so a later change to it breaks visibly instead of poisoning twenty deploys quietly.
Hermes and OpenClaw keep their client-side proof as CLI tests of their own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5