Commit Graph
24 Commits
Author SHA1 Message Date
evangelostsak bec593413c feat(web-app-homeassistant): let a deployment put its hosts into meshes 2026-10-16 03:40:18 +02:00
kevinveenbirkenbachandClaude Opus 5 4ac07edaea refactor(meta): move the info-media exemption to the file it speaks about
The marker names the reason a role has no upstream homepage or demo video, and
it sat in meta/main.yml because a comment-only info.yml parses to an empty
document, which the schema lint rejected outright. So the exemption lived one
file away from the two fields it explains, and a reader looking for the video
found neither the field nor the reason it was missing.

The schema lint now accepts an empty document exactly when it carries the
marker, and that is what allows the move. The media lint reads the marker from
info.yml rather than main.yml, and its failure message names that same file, so
the fix it suggests lands where the absent field would have gone.

A hundred and two roles follow. Every occurrence was a standalone comment on
the second line of main.yml, directly under the document start, so the move is
a line delete and an insert at the same offset. Seventy-three roles already had
an info.yml and gain the line; twenty-nine get one whose whole content is the
document start and the marker. All hundred and two reasons carry over verbatim,
including the thirty-five bespoke ones.

Nothing downstream sees a change. Of the four readers of meta/info.yml, three
coerce a null parse through `or {}` behind an isinstance guard, and
utils/cache/applications.py skips a falsy parse before assignment, so
applications.<role>.info stays absent for the twenty-nine exactly as it was
while the file did not exist.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 22:26:38 +02:00
kevinveenbirkenbachandClaude Opus 5 346ad875c6 refactor(mcp): drop the two provenance fields nothing reads
supported_version and minimum_version sat in 24 roles and were read by no template, no task and no lookup. Their only consumer was the audit report version column, which turned false the moment an automated image bump ran: qdrant reported a verified version nobody had verified, two releases long. Both leave MCP_KEYS, so a role that declares them again fails the schema lint instead of carrying a field with no effect.

Requirement 025 mandated minimum_version as a MUST and 035 carried it in an example; both are corrected here, otherwise the repository would contradict its own code.

The gate then surfaced five dead docs.mattermost.com links. Mattermost moved to /administration-guide/ and /deployment-guide/, and every replacement was probed for 200 rather than guessed. The MCP page is gone with no successor, so mattermost now points at mcpserver/README.md in the Agents repository, the source rather than a rendered mirror.

Both concerns share three files - mattermost's meta/mcp.yml, requirement 025, and the generated audit.md, which derives from a source of each - so splitting them would commit an artifact that does not match its sources.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-03 19:31:08 +02:00
kevinveenbirkenbachandClaude Opus 5 ebcfbce773 fix(homeassistant): enforce the read-only tool flag
mutating_tools_enabled: false was a claim with nothing behind it. The agent include list is built by subtracting tools.mutating from tools.allowlist, and the role named no mutating tool, so the subtraction removed nothing and every client was offered HassTurnOn, HassTurnOff, HassBroadcast, HassCancelAllTimers and the three list writers while the declaration said mutations were off.

Naming the seven turns the flag into the control it claims to be: a client now sees GetDateTime, GetLiveContext and todo_get_items, and setting the flag to true restores the full surface. The adapter already refuses a contract whose tool carries no mutating verdict; this brings the one native provider that serves actuating tools under the same rule. Entity exposure still bounds what the read tools reach.

The README described entity exposure as the only control and said the write tools stay listed; it now describes both layers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-03 16:22:54 +02:00
kevinveenbirkenbachandClaude Opus 5 5bf4d4d8ac refactor(variants)!: let an MCP round decide for itself what it carries
The mcp-consumer-variant guard demanded three things of a provider's MCP
round: every admitted consumer enabled, each of them pinned `services: null`
so it dragged no dependencies along, and no other service left enabled. Two
of those were the rule legislating a round's contents rather than checking
that the pairing gets proven at all.

Keep the first and drop the rest. What else the round carries is the role's
own call, so the tor/litellm exemption, the datastore exemption and the
`allowed` set they fed have no reader left and go with them.

Strip the 63 `services: null` pins the rule forced across 20 variants, all
written bare as `services:`. The topic-emptying capability behind them
stays: apply_topic still normalises None to {} for the inventory payload,
and the unit tests still cover it. Only the uses and the mandate are gone.

That leaves overridden_providers scanning a repository with nothing left to
find, so its three assertions passed over an empty dict and would have
stayed green however the function broke. Rebuild them on a synthetic role
tree, which also covers the case the repository scan never held: a variant
naming a plain flag is not an override.

Removing the pins changes what deploys. Measured across all 20 variants: 18
identical, prometheus#1 grows 23 to 31 roles, moodle#2 grows 31 to 32 and
crosses the memory budget until the following commit shrinks it back.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 01:01:14 +02:00
kevinveenbirkenbachandClaude Opus 5 f1635e6a59 feat(variants): let a variant dictate the config of the providers it pulls in
A round deploys a provider once, so the role that pulls it in is the only place that can say how lean it has to be. A variant entry may now carry any application config topic under a service it depends on: a mapping deep-merges, so naming one key changes that key alone, and null empties the topic. Empty was the case that needed a marker of its own, because merging an empty mapping is a no-op and would leave exactly the services the round was trying to shed.

The topic vocabulary and the merge semantics live in utils.roles.applications.topics, which is also where resources.py now gets them rather than spelling the same null-or-merge out a second time. Collecting who demands what stays in the inventory tree, refuses two roles demanding different values for one provider in the same round, and skips a role's own entity entry so the existing services.<own>.users declarations are not read as overrides.

The CI set-cover learns the same fact. Its greedy cover retires a role's variant-0 row when an earlier row of another role embeds it, on the stated premise that embedding pulls the provider's variant 0. A variant that dictates that provider's config breaks the premise from the other side: whatever ran, it was not the provider as the provider declares it. Such a pair now covers nothing, so the retired row keeps its job, its artefact and its verdict.

Measured on the current tree: 20 override pairs, 41 covered rows before and after, so nothing is revoked today. The rule bites the moment an override lands in a row that covers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 21:08:08 +02:00
kevinveenbirkenbachandClaude Opus 5 791eee7c9f refactor(mcp)!: prove the endpoint contract with a CLI test, not a deploy probe
The contract check ran as an Ansible task in the middle of every provider deploy, and the two client checks ran the same way. A test that is part of the deploy cannot be run without deploying, cannot be re-run on its own, and reports its verdict as a deploy failure. Twenty-seven task files lose 491 lines to this.

It is now what it always was. Each provider ships files/test/test.sh, templates/test.env.j2 and meta/tests.yml, and test-e2e-cli stages one shared harness beside it: run.sh drives contract.sh in a throwaway container on the provider's own network, then isolation.sh for adapter-fronted providers. The read probe each provider offers moved from the calling task into meta/mcp.yml next to the allowlist it must satisfy, and the schema now rejects a read probe naming a tool outside that allowlist.

The bearer the test presents is the one a client resolves through mcp.credential, via the new mcp_client_bearer lookup over resolve_credential, which moved to utils.roles.applications.mcp so it is the same function mcp_servers renders every client connection with. The probe presented whatever the calling role passed, which for the fifteen adapter-fronted providers is a different secret than any client holds.

Sidecar isolation is read from the engine rather than measured inside the container, so it no longer needs a delegation to the node holding the task: attachment is compared by network name, which drops the gwbridge special case, and a swarm worker says it cannot measure instead of skipping silently.

Retrying is bounded by wall clock against MCP_DEADLINE_SECONDS, which scales through the same lookup('timeout') as the harness budget it has to stay under. A flag that is neither true nor false aborts rather than reading as off.

mcp-provider-probe becomes mcp-provider-cli-test: an enabled provider must ship the test rather than include the task. mcp-probe-bearer becomes mcp-adapter-bearer, tying ADAPTER_BEARER to the declared credential key now that no probe include carries it. A new integration test renders the shared env template for every provider through Ansible's own templar, so a later change to it breaks visibly instead of poisoning twenty deploys quietly.

Hermes and OpenClaw keep their client-side proof as CLI tests of their own.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 20:46:26 +02:00
kevinveenbirkenbachandClaude Opus 5 ffdef28da3 refactor(mcp)!: derive a surface's presence from its peers
enabled and shared were hand-maintained in 24 roles and wrong in both
directions. Both now resolve from the mcp_consumer edges already declared
in meta/services.yml: a server takes the consumers it admits, a client
takes the servers that admit it, and the pair renders as "'peer' in
group_names or ...", or False when no peer is admitted at all. setdefault
keeps meta/mcp.yml and meta/variants.yml as the override channel, so
pinning either key still wins.

The resolved block replaces the mapping instead of being written into it.
_build_role_base_config parks the YAML cache's own object under mcp, so an
in-place setdefault injected the derived keys into every later raw read and
made test_mcp_schema pass alone but fail in a directory run.

The derivation keys on peer presence, never on a peer's own value: a
re-entrant read under _RENDER_GUARD returns the unrendered dict.

Four lints now ask the resolver instead of reading meta/mcp.yml, which they
could not do while the key was hand-written. That is what exposed the eleven
missing hostile-path specs the previous commit closed.

Carries one change that belongs to the WordPress probe but shares a file
with this one: the tool allowlist moves to the hyphenated names the adapter
actually serves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 16:13:01 +02:00
kevinveenbirkenbachandClaude Opus 5 83c1c729de refactor(mcp)!: derive admission from the client
allowed_consumers was twenty hand-kept lists of which fourteen were identical, so every new client meant an edit in every provider. A client now declares services.<own-entity>.mcp_consumer: true once and a provider refuses one with mcp_consumer: false, which is also what tells a deliberate exclusion apart from a forgotten entry.

The group_names-gated consumer entries give the inventory resolver the co-deploy edge meta/mcp.yml never had: one of sixty-nine matrix rounds carried an MCP client before, twenty-three role/round pairs do now. bond 0.5 keeps the clients out of round 0 so the host memory budget still holds.

BREAKING CHANGE: meta/mcp.yml no longer accepts allowed_consumers; declare services.<entity>.mcp_consumer on the client instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-31 18:33:22 +02:00
kevinveenbirkenbachandClaude Opus 5 896b72428c docs(csp): record why hermes, homeassistant and litellm allow unsafe-inline
Three roles set unsafe-inline without the rationale the repository lint
requires; 70 roles already carry it.

Only one of the three rationales is a justification. homeassistant's
frontend boots its panel from an inline module script, so the dashboard
stays blank without the flag. hermes and litellm state instead that the flag
was set when the role first deployed with no console error recorded against
it, and carry a TODO to re-verify against the running application and drop it
if nothing breaks.

unsafe-inline weakens the policy against inline script injection. For two of
these roles this documents the exemption rather than justifying it; whether
they work without it has not been tested.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 12:58:22 +02:00
kevinveenbirkenbachandClaude Opus 5 cd1f07a91e feat(roles): onboard hermes, homeassistant, litellm and openclaw to tor
Four roles gained a canonical domain without a tor service block. Declare it
the way 84b22b37a8 did for n8n: bond 0.5, enabled on 'svc-net-tor' in
group_names, shared, with the variant pins alongside.

The nocheck directives carry the two exemptions: the service has no persona
surface for the playwright flag, and the sso-variant-tor combination requires
the onion in every variant, so the false polarity has no variant left to
live in.

hermes also moves its local http port from 8089 to 8092, because 8089
belongs to web-app-socialhome. That is a separate concern riding along in the
same file; splitting it would need interactive staging.

This is merge work rather than authored here: it follows the established
pattern and the port collision is real, but none of it has been deployed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 12:48:10 +02:00
kevinveenbirkenbachandClaude Opus 5 481666bdb0 refactor(roles): finish renaming meta/schema.yml to meta/secrets.yml
Eleven roles still declared their credentials in meta/schema.yml while the
rest of the repository had moved to meta/secrets.yml. The name is not
cosmetic: the inventory resolves credentials at that path, as
d86913225b and c541e7c22f record, so a role left on the old name drops out
of that resolution.

All eleven are pure renames with no content change.

Also correct the four remaining places that still named the old path in
prose: the credentials reset CLI help, two test docstrings and the espocrm
README. None of them resolved a file, but they misdescribe where credentials
live, which is exactly what sent an earlier reader to the wrong location.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 12:13:48 +02:00
kevinveenbirkenbachandClaude Opus 5 28475455db feat(mcp): give every provider its own service account from the registry
Nine roles reached their MCP surface as the deployment administrator or with
an account password kept beside the account in credentials. A Moodle site
admin bypasses every capability check, a Nextcloud app password is not
scopable, a Gitea token reads every private repository, and a GitLab token on
root reads every private project. Each role now declares its own principal in
meta/users.yml and provisions it, so the exposed surface is what the service
role permits. jellyfin, zammad and snipe-it kept their account but took its
password from a credential named token; they read the registry too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 15:44:12 +02:00
kevinveenbirkenbach 09d4e496cc fix(mcp): build the adapter against the base image digest it declares
The metadata named infinito/mcp-adapter:1.0.0 with an all-zero digest,
an image that does not exist, while the build actually used
python:3.13-slim. Declaration and artifact disagreed, so the declaration
proved nothing. Both now name the real base image and pin it by digest,
which a tag cannot guarantee across a rebuild.

The adapter base moves from the provider's service version to
mcp.adapter.digest, alongside the tool specification and its checksum.
2026-08-09 23:20:58 +02:00
kevinveenbirkenbach 06bc856c32 refactor(mcp): make MCP a role topic instead of a service entry
The block sat under services but never behaved like a service. A service has
exactly one providing role, declared with provides, so the registry can map the
key to that role and consumers bind their flag to its presence. No role declares
provides: mcp. There are seventeen providers and the pairing is done by
allowed_consumers inside the block itself.

Every rule that enforces the service model therefore needed an exemption for it:
a shared-provider suppression in fourteen roles, a playwright-service-flag
suppression, and two freshly added exemptions in the dynamic-flag and
variants-static-override checks. Four independent rules, four escape hatches,
one cause. They are all gone now, together with a bond key that only the
resource budgeter reads and only for entries under services, and a wordpress
addon bridge that pointed at a service block that no longer exists.

The block moves to meta/mcp.yml. The applications loader maps meta/<name>.yml to
applications.<app>.<name> generically, so the move needs no wiring; the file
takes the same shape as meta/rbac.yml. Every config path follows, the variant
overrides are lifted out of their services map, and the vocabulary module leaves
the services package because it no longer describes one. roles_with_service
gains an explicit topic parameter rather than an implicit fallback, so nothing
silently resolves against the wrong container.

Files that are otherwise unrelated to this move are included where the committed
tree would otherwise keep pointing at a path that no longer resolves: thirteen
vars files, eleven playwright env templates and one task file.
2026-08-06 20:36:00 +02:00
kevinveenbirkenbach 10de64ff78 fix(networks): give MCP overlay membership a direction
A provider overlay keyed on services.mcp.enabled and .shared admitted every
MCP-capable role, servers included, because those flags carry no direction. Each
provider's network therefore collected every other provider and the result was a
mesh in which any sidecar could reach any application. The new mcp_client
consumer kind additionally requires services.mcp.direction to be client or both,
and the four provider overlays switch to it.

A sidecar rendered its networks with the same lookup as its provider and
inherited that provider's consumer attachments, which it neither needs nor
should have. own_network_only keeps it on the provider network alone. A provider
whose own topology is default_net contributes no attachment line at all, and a
bare networks: key is not a list, so the flag returns nothing in that case and
lets the sidecar fall back to the project default, which is that provider's own
network.
2026-08-06 14:27:07 +02:00
kevinveenbirkenbachandClaude Opus 5 19eaec2b6d fix(mcp): declare the subject each server actually authenticates as
Four roles declared auth_subject: user while handing the client an
account-bound credential. Nextcloud and Home Assistant mint against the
administrator account, Mattermost issues the administrator's personal access
token, and Baserow creates a dedicated mcp@<domain> account with its own
workspace. The declarations now name what the minting tasks do; Baserow's
README additionally claimed the administrator owns its endpoint, which it
never did.

The tools declarations follow one convention instead of two: the field records
the deployment's intent and the README names the missing enforcement, which is
what GitLab already did and what Baserow and Mattermost now do.

The discovery lookup carries auth_subject through so clients can see the
subject. No consumer branches on it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 22:08:49 +02:00
kevinveenbirkenbachandClaude Opus 5 e355bbdb8b feat(mcp): gate Open WebUI tool servers on the serving role's mcp group
Eight server roles declare an `mcp` RBAC role, so the platform mints
roles/<app>/mcp and the administrator joins it. Open WebUI now honours those
groups instead of flattening them to user/admin, and each MCP tool server is
restricted to the group of the role that serves it.

The grant cannot be rendered: Open WebUI matches it against a group id that
insert_new_group mints as a fresh UUID, so tasks/01_mcp.yml resolves the group
after the container is up and writes the grant through the admin API, the way
every instance-minted credential in this repository is provisioned.

Entries render disabled. An empty access_grants is not 'nobody', Open WebUI
reads it as every administrator, and ENABLE_PERSISTENT_CONFIG=false makes the
env authoritative again on every start. Enabling only together with the grant
keeps the fallback at no access rather than too much.

The provisioning account is the role-internal openwebui-api-bot, never a
platform identity: OAUTH_MERGE_ACCOUNTS_BY_EMAIL ships disabled and an OIDC
login whose email a local account already holds is refused, so seeding a
person's address would lock them out.

Known limits are documented rather than claimed away: a bare restart leaves MCP
off until the next deploy, a user's roles list carries no application context so
roles: [mcp] grants every deployed mcp group at once, and Open WebUI drops a
stale membership only while the groups claim is non-empty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 22:06:47 +02:00
kevinveenbirkenbachandClaude Opus 5 d350e88c6b feat(homeassistant): carry the MCP provenance in the role
Move source, minimum version and upstream notes into the role's own mcp
block, so the facts sit next to the surface they describe.

Drop the mcp_token credential declaration. Home Assistant issues that
token itself and it is persisted through sys-token-store, so a vault
credential of that name was never read and pointed at a place the value
does not live.

Document the surface: the endpoint answers an unauthenticated request
with 404 rather than 401, and the enforced control is entity exposure,
not a read-only tool policy. tools/list offers write tools, but an entity
that is not exposed to Assist stays unreachable through them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 22:07:45 +02:00
kevinveenbirkenbach f39073c95c Merge remote-tracking branch 'fork/feature/swarm-nfs' into feature/ai-agent-employees
# Conflicts:
#	roles/web-app-flowise/meta/services.yml
2026-07-30 15:39:29 +02:00
kevinveenbirkenbachandClaude Opus 5 e3481b7745 feat(openclaw): reach the declared MCP servers and prove it
MCP_DISCOVERED_SERVERS resolves every server-direction role to its internal URL, the bearer the server role minted, and its transport, dropping any server without a token so no client renders a connection it cannot authenticate.

OpenClaw moves from 2026.3.11 to 2026.7.1 because the pinned version has no configuration surface for outbound MCP servers at all: its validator rejects mcpServers, the shipped bundle carries no bundle-mcp, and the ACP path refuses per-session servers while pointing at a gateway config that does not exist there. 2026.7.1 reads mcp.servers, which the template now renders. The gateway command gains its run subcommand because the bare gateway no longer starts anything.

Reachability was the missing half: openclaw sat on its own networks only, so the hub name resolved through the DNS wildcard to the ingress instead of the container. The hub overlay now declares an mcp consumer key, which attaches every role carrying services.mcp.

A probe task runs openclaw mcp probe after the handlers flush and fails the deploy when a configured server does not answer. Without it the previous run reported failed=0 while the client could not reach anything.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 15:40:19 +02:00
kevinveenbirkenbachandClaude Opus 5 9164dcda4c feat(homeassistant): serve MCP and persist the token it mints
The role declared a server-direction MCP service but never provisioned one, and the declared endpoint pointed at /mcp_server/sse, the legacy path. Home Assistant serves MCP under /api/mcp, and its mcp_server integration has no YAML schema, so the config entry only exists through the config-entries flow API.

The provisioning script drives that path over the container loopback: onboard the administrator when the hub is fresh (falling back to the login flow when it is not), mint a long-lived token over the websocket API, then create the mcp_server config entry with it. The token is persisted through sys-token-store, which is why meta/users.yml registers the key; a second role registering a token for the same user only works since the per-key user merge.

The whole path is skipped when a token already exists, so a redeploy costs one lookup. Handlers flush before it runs because the hub container has to exist before anything talks to its API.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 15:39:13 +02:00
kevinveenbirkenbachandClaude Opus 5 3309038b56 fix(web-app-homeassistant): make the hub survive a swarm deploy
Two defects, the second only visible once the first was gone.

The hub answered every proxied request with HTTP 400. In swarm it joins the
proxy's own overlay, so the forwarded request arrives from that network and not
from the hub's, while `trusted_proxies` listed only the hub's own subnet. The
container log named the address it rejected, and the two networks resolve to
192.168.107.0/24 for the proxy against 192.168.79.0/24 for the hub. Compose has
no such split, which is why this never surfaced there. Use the project-wide
trusted-proxy CIDRs instead, the same list three other roles already consume.

With the proxy trusted the failure moved to HTTP 502: the backend was
crash-looping. Home Assistant guards its config directory with a PID lock, so
the second and third replica on the shared volume abort with "Another Home
Assistant instance is already running" and never come up. The service sat at
2/3 with `task: non-zero exit (1)` cycling over all three nodes. Upstream
supports no multi-instance mode, so the hub is pinned to a single replica like
xwiki and jellyfin before it.

Verified by a full swarm run: deploy, e2e and the drain-and-reschedule step all
green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-28 09:16:57 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0c4d74cc7c feat(web-app-homeassistant): expose the automation hub as an MCP server
The first MCP server role: where the agent roles consume tools, this one
offers them. Its services.mcp block declares an internal streamable-HTTP
endpoint gated on an agent being deployed, so the surface only exists
when something can use it, and it stays read-only because a privileged
auth subject must not carry mutating tools.

Home Assistant rejects any request carrying X-Forwarded-For from an
undeclared proxy, answering 400 for every call behind the platform's
reverse proxy. The role therefore renders a configuration.yaml that
trusts the stack subnet, keeping default_config so mounting the file does
not drop the integrations Home Assistant generates on first start.

As a shared_net provider its compose file references its own network as
external, which nothing creates in compose mode, so the role pre-creates
it the way prometheus, minio and seaweedfs already do. The vhost declares
status_codes [200, 302, 405] because the health probe sends HEAD and the
hub answers 405.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:15:18 +02:00