Commit Graph
363 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5 4e46858202 feat(n8n): provision the MCP server and client, and close the auth gate
The oauth2 gate declared no allowed_groups at all while the role carried an mcp RBAC role nothing consulted, so it admitted anyone the proxy authenticated. It now admits the role's own administrator and mcp groups, and a lint derives the client set from meta/mcp.yml rather than a list, which is how the gap surfaced.

Two idempotence defects went with it. The run minted a public-API key, found it again next time and counted that as a change, so the task never converged; and it left that workflow-scoped key standing after every deploy although nothing read it. It is now minted per run and revoked before the run ends. The server workflow also connected nothing, and the trigger serves exactly the tools that reach it over an ai_tool connection, so it served none; it now carries one checked-in read tool with an explicit input schema.

The client side did not exist. It renders an agent workflow with one MCP Client Tool per discovered provider, each pinned to that provider's allowlist, because the node defaults to including every tool an upstream happens to serve. Credentials go through the internal API since the public one of this tag can only create and delete them, never converge an existing one. Every node type, typeVersion and connection literal was read from the pinned source.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-29 00:46:57 +02:00
kevinveenbirkenbachandClaude Opus 5 d9f497688d feat(litellm): authenticate the admin UI through Keycloak
The role declared sso.enabled false behind three suppressions, so the checks were switched off rather than satisfied. LiteLLM authenticates its own admin UI instead of sitting behind an oauth2 proxy, so the declaration is flavor oidc and the client id, the three Keycloak endpoints, the proxy base URL and the admin id all render together under one flag: a half-set group would let the gateway fall back to its own form with nothing complaining. The variants pin both states so the matrix runs with and without SSO.

The proof has to measure the departure, not the arrival: a gateway ignoring the client id answers /sso/key/generate with its own username and password form and never redirects, so the spec asserts the round trip to Keycloak and back. The guest spec became SSO-aware, since its canonical-domain and password-field assertions only hold where no SSO is configured.

Two more gateway defects: the key warm-up ran even where no model was served, falling back to openrouter/auto so the gateway then answered every prompt with 400, and it is now gated on a served-model expression mirroring the model selection itself. A lint requires every declared svc-ai backend to have a model_list branch, because a backend without a route is an alias nobody serves. probe_auth.py proves revocation rather than creation: it mints a throwaway key, revokes it, and fails unless a real consumer key survives.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-29 00:44:26 +02:00
kevinveenbirkenbachandClaude Opus 5 b18c507cf6 feat(svc-ai-mcp-adapter): speak the classic HTTP+SSE transport
Streamable HTTP answers a POST on the same request; classic SSE splits the channels, so a GET opens the response stream and that stream announces a second URL as the request channel. The adapter only ever spoke the former, which is what reverted the Baserow migration twice. A contract now selects the transport and defaults to streamable_http, so no existing provider changes, and an unknown value is refused at load rather than at the first tool call.

The stream is opened once and held, because reopening it per call would lose whatever answer is already in flight. A notification carries no id, so it posts without awaiting a response the protocol never sends. A dead stream is now reset alongside the session it belongs to, since rebuilding the handshake over a dead stream would keep using it.

Two things this cleaned up on the way: NoRedirect and OPENER had been copied rather than reused, so server.py now takes the one in sse.py, and decode_jsonrpc moved to passthrough.py where it belongs, since it unwraps MCP responses rather than HTTP, which also brought server.py back under the file-size limit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-29 00:41:55 +02:00
kevinveenbirkenbachandClaude Opus 5 ec736bd4d7 feat(mcp): offer no client a tool its provider calls mutating
Both agents registered every tool every provider served. Hermes documents that as its own default in tools/mcp_tool.py, "Neither set -> register all tools", and the role never set tools.include; OpenClaw's toolFilter was never rendered at all. A provider that grew a mutating tool would have reached both without anything declaring it.

A serving role now names its mutating tools, the discovery snapshot carries that set beside the allowlist, and every client renders the allowlist minus it in whatever form its upstream enforces. The subtraction sits at the client and not in discovery because Flowise compares the declared set against what its registry actually discovered and needs the full served list. Where a provider permits mutations there is nothing to withhold, so the set renders empty.

This is a client-side boundary, not a server-side one: whoever holds the endpoint key can still call a write tool directly. Baserow is the case that made it necessary, since its MCPEndpoint model carries no scope and its owner is a workspace administrator, so the declared mutating_tools_enabled bounds nothing there.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-29 00:29:12 +02:00
kevinveenbirkenbachandClaude Opus 5 63a37dac45 fix(svc-ai-mcp-adapter): enforce the contract at runtime, not only at load
Four guarantees the contract made were never applied. concurrent_requests and stream_seconds appeared nowhere but the required-key list: ThreadingHTTPServer starts a thread per connection and asks no contract, so the ceiling only holds if the handler refuses the surplus itself, which it now does with 503 and gives the slot back on the failure path as well. The upstream was opened with urlopen, which follows redirects, and urllib copies the request headers onto the redirect target, so an upstream answering 302 with an off-host location would have handed the adapter's upstream credential to whoever it named.

input_schema was published to clients and never checked, which made it a suggestion: on the openapi path an argument the contract does not name is appended to the upstream request as a query parameter, so a client could add parameters the reviewed operation never granted, and a tool declaring no schema accepted anything. A tool that declares no inputs now takes none, a required input must be present, and the check runs after the size ceiling so a large payload is still refused for being large. No shipped contract regresses: the read tool every deploy probe calls declares no required argument, measured across all fourteen contracts.

sse.py adds a client for the classic HTTP+SSE transport, which splits the response and request channels across two URLs and needs a reader thread because the answer to a POST regularly arrives before the POST returns. It has no caller yet. The deploy probe keeps its own SSE implementation deliberately: it guards every enabled surface today and fails by exit code rather than by exception.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-29 00:26:11 +02:00
kevinveenbirkenbachandClaude Opus 5 493fedc6c6 refactor(tests): move the role unit mirrors under tests/unit/python
The role scripts moved into per-language folders but their unit mirrors did
not follow, so the index carried paths under tests/unit/roles/ that no longer
existed on disk while their counterparts under tests/unit/python/roles/ were
untracked.

That stale index broke the gate while naming the wrong culprit: the
repository lint reports "rsync --backup in 19 place(s)" because it counts
files it cannot open, and every entry was an [Errno 2] No such file or
directory rather than an actual rsync call.

Fifteen of the moves are recognised as renames; the rest are __init__ files
that collapsed into the new tree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 12:33:48 +02:00
kevinveenbirkenbachandClaude Opus 5 54a6f3b533 feat(svc-ai-litellm): warm the chat model during the deploy
A backend pulls its preload model at deploy time but only loads it into
memory on the first inference. That cold load outlives a reverse proxy's
read timeout, so the first consumer request comes back as a 504 while every
later one succeeds.

Send one max_tokens=1 completion from inside the litellm container right
after the per-consumer keys exist, where the container id, master key and
port are already resolved. Five retries cover a gateway that is not
listening yet.

This pairs with the ollama keep_alive setting: the warm-up pays the load
once, keep_alive stops it decaying after five idle minutes.

Note this hardens the deploy: a gateway that cannot serve its configured
model now fails the role instead of surfacing later as a consumer error.
That makes the openrouter/auto fallback with its empty API key loud rather
than silent.

The script ships with a mirrored unit test rather than a nocheck waiver; it
patches urlopen and asserts the URL, method, authorization header, model,
max_tokens and that the timeout exceeds a proxy read timeout.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 12:02:17 +02:00
kevinveenbirkenbachandClaude Opus 5 7162b7bf4c Merge fork/feature/svc-net-tor into feature/ai-agent-employees
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.

The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.

Resolutions that were not unions:

Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.

plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.

compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.

tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.

Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.

01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.

Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.

roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.

config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.

READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.

Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-24 16:32:30 +02:00
kevinveenbirkenbachandClaude Opus 5 b387c80528 fix(container): reject a directory standing in for a CA bind source
must_exist accepted any existing path, so a directory that dockerd auto-created where a bind source was missing passed the guard. The wrapper was then mounted over the container path and set as --entrypoint, and every image that has an ENTRYPOINT died with rc 126 'is a directory: permission denied' — seven identical retries, because the directory persists. Requiring a regular file routes that state into the existing 'CA injection disabled' fallback, which the same suite already exercises green when the wrapper is simply absent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-23 19:52:02 +02:00
kevinveenbirkenbach d391b4cbb8 test(unit): cover the postmarks, mediawiki and baserow shims
Bookwyrm has tests for the same trusted-header SSO bridge and settings
patcher; postmarks and baserow ship theirs untested, and the composer
require-dev stripper is a pure transformation nobody exercised.

Each test was checked by breaking its source: the admin gate failing
open, the group matcher loosened to a prefix, require-dev left in place
and the SSO gate forced open all turn the new tests red.
2026-08-22 22:31:37 +02:00
kevinveenbirkenbachandClaude Opus 5 b192415a5b fix(decidim): take the ruby fragment path from its one source
patch.py located files/ruby/ relative to its own __file__. That holds in
the repo, where the script sits next to the fragments, but the image
copies the script to /tmp/patch.py and the fragments to /tmp/ruby/, so
parent.parent resolved to / and the build died with
FileNotFoundError: /ruby/omniauth_provider.rb on every retry.

DECIDIM_PATCH_RUBY_DOCKER already decides where the COPY puts the
fragments. Reading it in the script makes that variable the only place
the location is stated, so the layout can change without a second
place silently disagreeing. Missing variable raises instead of
guessing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 13:03:48 +02:00
kevinveenbirkenbachandClaude Opus 5 8f4d842375 test(lint): require a mirrored unit test for every shipped script
`roles/<role>/files/<lang>/<rel>` is tested by
`tests/unit/<lang>/roles/<role>/files/<rel>`, renamed to that language's
convention. The language appears once, in `tests/unit/`, because the suite
already says what the file is. Without a fixed mapping a shipped script is
tested wherever its author happened to look, and nobody can tell an untested
file from one whose test lives somewhere unexpected.

Most of what a role ships is a script against a live console -- it boots a
Laravel kernel, imports frappe, or writes through an ActiveRecord model -- so
the escape is a `nocheck: mirrored-unit-test` comment in the file, with a
rationale of its own. A file carrying both a test and an exemption fails: the
exemption outlived its reason and reads as "untested" to everyone after you.

Five scripts turned out to have real units behind the console wrapper, so they
get tests rather than markers (62 cases):

  * svc-net-tor/onion_self_reach.py -- pins that an HTTP error status, a closed
    connection and a refused connection all count as reached, because each
    proves the loopback fast path resolved; only a timeout or an unresolved
    name is a failure.
  * sys-lock/sys-lock.py -- `parse_time_to_seconds` unit precedence, and that a
    service still activating holds the lock (otherwise the gate races it).
  * svc-opt-keyboard-color/script.py -- colour interpolation and the
    zero-length transition window that would otherwise divide by zero.
  * sys-svc-compose/swarm/secret_rotate.py -- including the binary fallback a
    TLS key needs, and that a missing source leaves the name alone rather than
    pinning the hash of a file that is not there.
  * sys-front-inj-logout/logout.js -- pins that `href="/logout"` alone does not
    match and a non-interactive tag is never patched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 01:31:10 +02:00
kevinveenbirkenbachandClaude Opus 5 575a5a87ac refactor(layout): sort shipped scripts by language and split the unit suite
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).

This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.

Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:

  * 7 `src:` values built from a role variable holding a bare filename
    (svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
    The variable doubles as the destination name inside the container, so the
    language segment belongs at the call site, not in the variable.
  * `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
    through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
    18 roles ship exactly that and have no template fallback: the play would
    have died on each.
  * `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
    interface now reads `files/javascript/javascript.js`.
  * `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
    path through `path_absolute(... 'files', ...)`.
  * 4 `ansible.builtin.script:` invocations naming a bare command
    (pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
  * `dev-python/files/shell/install.sh` pointed at a sibling role's script.

Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 01:31:09 +02:00