Commit Graph
28 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5 4ce588fea1 test(lint): forbid a JavaScript default for an unset environment variable
`process.env.X || <value>` is a second source for something the deployment
already renders, and the two drift without a sound. playwright.config.js read
`process.env.PLAYWRIGHT_ACTION_TIMEOUT || 30_000` while nothing ever set that
variable, so every run used the unscaled 30 s and no failure said so.

Normalising an absent variable stays allowed, because it substitutes nothing:
`|| ""`, `|| '[]'`, `|| false`, `|| 0` and falling through to another
`process.env` entry only let the caller test absence itself.

The six sites the rule found:

  n8n and flowise built a Postgres port from `|| '5432'` while env.j2 renders it
  from `lookup('database', application_id, 'port')`. Node throws
  ERR_SOCKET_BAD_PORT on the resulting NaN, so an unfed port now names itself
  instead of connecting somewhere plausible.

  The bluesky broker did the same with BROKER_PORT, which compose.yml.j2 renders
  unconditionally. Its INSECURE_TLS default was `"false"`, which is what `""`
  already evaluates to.

  matomo defaulted MATOMO_TRACKING_SCOPE to "sub"; the only comparison is against
  "root", so `""` takes the identical branch.

  moodle and simpleicons defaulted a name their playwright.env.j2 always renders.
  simpleicons already asserts the value is truthy; moodle now fails on the scope
  it cannot find rather than on a name nobody configured.

Four values the code itself owns keep a default under `// nocheck: env-default`:
PDS_INTERNAL_URL (which falls through to requireEnv, not to a literal),
HANDOFF_COOKIE_NAME, HANDOFF_COOKIE_MAX_AGE and PASSWORD_SALT_HASH_ROUNDS. No
role renders any of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-12 12:07:57 +02:00
kevinveenbirkenbachandClaude Opus 5 f7ad2ff0fa fix(flowise): read a refused registration as the deny list working
The deploy points a throwaway registry entry at loopback and at the metadata
address and fails unless Flowise refuses to reach it, because
HTTP_SECURITY_CHECK=false replaces the built-in deny list rather than merging
into it, and a typo there would disable every protection while the registry
kept working.

The check assumed the entry always registers and that only authorize can
refuse. Flowise consults the list at registration too, so the create came back

    400 {"statusCode":400,"success":false,
         "message":"Server URL is not allowed by policy"}

and the probe reported its own strongest proof as a failure, taking the deploy
with it after forty minutes.

Accept that answer, and only that answer: a 400 naming the policy is the list
refusing the address. Every other non-2xx still aborts, and a list that is not
live still returns the entry, so the authorize call below remains the judge of
that case. The check cannot pass on a dead deny list.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 12:12:07 +02:00
kevinveenbirkenbachandClaude Opus 5 40f2bb933d fix(flowise): scope the shim path to its layer instead of a build arg
f58ff5522 gave the socket-timeout shim a single point of truth by way of a
defaulted ARG. test_arg_without_default.py forbids that tree-wide:

    roles/web-app-flowise/files/Dockerfile:7: ARG FLOWISE_SOCKET_TIMEOUT_SHIM carries a default

The rule's reasoning does not fit this value at all: "A defaulted ARG answers a
question the caller failed to answer. The build then succeeds against a base
image, version or path nobody chose." An ARG is a contract with the caller, and
nobody needs to answer where a temp file lives that is written on line 21 and
deleted on line 25 of the same layer.

f45df04bf1 shows the house pattern for the case where the value IS a caller
decision: it dropped PY_BASE's default from bridgy-fed and had compose.yml.j2
supply it. Doing that here would invent a build contract for a scratch file.

A shell variable keeps the single point of truth and narrows the scope on top of
it -- the name was visible across the whole build context as an ARG, and now
lives only inside this RUN. Same path, same three references, same build.

Suppression was available and not used: a nocheck marker would have kept the
wrong mechanism and hidden the reason it was wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 00:20:29 +02:00
kevinveenbirkenbachandClaude Opus 5 f58ff55226 fix(flowise): survive a stalled prebuilt-binary download
GitLab#1 of run 33414875521 died in the flowise image build. The obvious line is
not the cause:

    npm error command sh -c prebuild-install -r napi || node-gyp rebuild
    npm error prebuild-install warn install Request timed out
    ...
    npm error gyp ERR! stack Error: not found: make

`not found: make` is the third domino. Counting the whole job settles it:
`Request timed out` 38, sqlite3 failures 6, `not found: make` 6 -- and sharp
failures 16, of 22 build attempts total. sharp emits no gyp output at all; its
fallback dies at `can-compile` because the image ships no libvips. A toolchain
would have touched 6 of 22 and left the dominant mode alone.

What actually aborts: node >=19 puts a 5s socket-inactivity timeout on
https.globalAgent (v24.19.0 lib/internal/http.js:263-268), while sharp and
prebuild-install fetch their prebuilt binaries through simple-get, which sets no
timeout of its own and never retries. Any 5s gap in the byte stream of the ~8 MB
libvips tarball drops the whole RUN layer.

The trigger was environmental, and it is worth being explicit that it was: the
same run, the same commit and the same base-image digest built flowise green
twice in another job, overlapping these failures. MTU is 1500 in both jobs, there
is no proxy configured, and svc-net-tor redirects only 10.240.0.0/12, so
github.com never enters Tor. The apt layer of the failing build itself fetched at
6178 kB/s and npm resolved all 3196 packages in the same 157s as the green one.

The fragility is ours. Eleven lines above, the apt layer already carries
-o Acquire::Retries=3; the npm layer had nothing. That asymmetry is what this
closes.

NODE_OPTIONS is attached to the install command alone rather than exported for
the layer: npm init and npm pkg set reach no network, and npm install is what
spawns the lifecycle scripts that download. --require is kAllowedInEnvvar on this
node (src/node_options.cc:1124-1127). No ENV, so the runtime image and flowise's
own HTTP behaviour are untouched. 60s is 12x headroom over a real stall while a
genuinely dead socket still fails inside a minute.

Not proven: that 60s clears this particular stall. The stall duration is nowhere
in the log, and per 850b795a51 a local quality gate cannot answer a
network-dependent question at all. Only a CI build closes that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 00:08:45 +02:00
kevinveenbirkenbachandClaude Opus 5 a206afaff5 fix(flowise): give the fixture nodes the keys buildChatflow iterates
The MCP registry reconcile retried thirty times and failed with Flowise's own words:

    FAILED calling qdrant_list_collections on svc-db-qdrant: 500
    {"message":"Error: predictionsServices.buildChatflow - inputParams is not iterable"}

fixture_flow_data hand-builds an Agentflow v2 definition, and both of its nodes
carried inputs, outputAnchors and outputs but no inputParams. buildChatflow iterates
that key, finds undefined and throws, so the fixture flow could be created and was
dead on its first invocation.

Empty is the right value rather than a placation: the fixture puts its arguments in
inputs literally, so there is nothing for Flowise to resolve. inputAnchors comes
along because a Flowise node carries both and the next node written here would
otherwise trip over the same edge.

The new test asserts the three keys on every node the fixture emits, not on the two
that exist today, so a third node cannot reintroduce the bug.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-31 15:13:04 +02:00
kevinveenbirkenbachandClaude Opus 5 ce29195b64 fix(qdrant,flowise): authenticate the vector engine and prove what Flowise can reach
Qdrant ran with no authentication at all: its environment set the HTTP port and disabled telemetry and nothing else, so every container that reached its network held full read, write and delete access to every collection, and the MCP surface's read-only promise rested entirely on the adapter's contract naming only GET routes. Qdrant 1.19 forces authentication on every request once a read-only key is set, so the engine now declares both keys, the adapter presents the read-only one, and the full key goes only to consumers that own data here. A write route is now refused by the engine rather than only by the adapter.

Three defects in the Flowise registry path. Its tool-name check compared against a contract the role always passed as empty, so it could not fail; it now compares against the provider's declared allowlist. Its managed flow pointed a customMCP node at an input that exists on no node at this tag, while the node that does take a registry id is customMcpServerTool, and the one that takes an inline config would have embedded the bearer in the flow; the flow is now an Agentflow whose Tool node calls one named tool with fixed arguments and is executed on every deploy. Its module docstring claimed only SSE providers could be registered, while the pinned MCPToolkit connects Streamable HTTP first and falls back to SSE, so the docstring contradicted the role's own declaration.

Reaching a provider inside the container network needs HTTP_SECURITY_CHECK=false, which replaces the built-in deny list rather than merging into it, so a typo there disables every protection silently while the registry keeps working. The deploy now points a throwaway entry at loopback and at the metadata address and fails unless both are refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-29 00:49:58 +02:00
kevinveenbirkenbachandClaude Opus 5 84ff470f7f refactor(flowise): move the reconcile script into files/javascript
The last script of the per-language sort follows its siblings:
files/reconcile/identity_password.js -> files/javascript/reconcile/.

The order matters here. 9967c806b already pointed
flowise/tasks/utils/mcp.yml at javascript/reconcile/identity_password.js,
but the file only existed there in the worktree, so HEAD has been resolving
a missing path for flowise's MCP provisioning since that commit. This
repairs it.

Same class of mistake as twice before in this relocation, reference and file
landing in different commits, this time with the reference first.

Verified in both directions now: every lookup('file', ...) target reachable
from HEAD exists in the tree, all 44 of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 13:04:53 +02:00
kevinveenbirkenbachandClaude Opus 5 27dcd975e5 test(e2e): scale the spec timeouts for onion transport
Twenty-five specs called the Playwright APIs with default or hardcoded
timeouts. Routed over Tor those budgets are too small, so a spec fails on
transport latency rather than on the behaviour it asserts.

Route the waits through resolveTimeout and reach pages via gotoOnion, the
helpers the tor branch introduced for exactly this. The remaining changes in
the diff are import wrapping around those calls.

The repository lint enforces this: a raw numeric timeout in a role spec is
rejected with a pointer to resolveTimeout, which is how a hardcoded 500ms
wait added during this session was caught.

Merge work rather than authored here; not deployed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 12:52:19 +02:00
kevinveenbirkenbachandClaude Opus 5 dfe7a7fdd4 refactor(roles): sort role scripts into per-language folders
The merged branch moved every role-local script out of files/ into a folder
named for its language (files/python/, files/php/, files/shell/), but the
moves lost their staging and never landed. HEAD still carried the old paths
while the worktree had the new ones.

Move the 24 remaining scripts and update the six places that resolve them:
the mcp-adapter instance/isolation/probe tasks, the litellm key task, the
adapter README and the endpoint-path lint. Every changed line in those
files is a path.

Also drop roles/sys-svc-compose/files/swarm/stack_ready.sh, a stale
duplicate of the copy already tracked under files/shell/swarm/.

The added comments are the branch's own `nocheck: mirrored-unit-test`
directives, which state why a script whose measurement only exists inside
its container cannot have a mirrored unit test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 12:00:59 +02:00
kevinveenbirkenbachandClaude Opus 5 7162b7bf4c Merge fork/feature/svc-net-tor into feature/ai-agent-employees
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.

The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.

Resolutions that were not unions:

Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.

plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.

compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.

tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.

Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.

01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.

Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.

roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.

config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.

READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.

Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-24 16:32:30 +02:00
kevinveenbirkenbachandClaude Opus 5 575a5a87ac refactor(layout): sort shipped scripts by language and split the unit suite
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).

This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.

Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:

  * 7 `src:` values built from a role variable holding a bare filename
    (svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
    The variable doubles as the destination name inside the container, so the
    language segment belongs at the call site, not in the variable.
  * `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
    through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
    18 roles ship exactly that and have no template fallback: the play would
    have died on each.
  * `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
    interface now reads `files/javascript/javascript.js`.
  * `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
    path through `path_absolute(... 'files', ...)`.
  * 4 `ansible.builtin.script:` invocations naming a bare command
    (pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
  * `dev-python/files/shell/install.sh` pointed at a sibling role's script.

Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 01:31:09 +02:00
kevinveenbirkenbachandClaude Opus 5 642055a20a style(e2e): drop the timeout argument isVisible discards
Playwright declares the timeout option on locator.isVisible() and isHidden() as deprecated and ignored: both return immediately and never wait. Passing resolveTimeout(10_000) to one therefore changes nothing at runtime and only tells the reader a wait happened that never did. That reading is what let 7538b8a0b conclude two genuine call sites were fine, and what hid the false negative b97cf8888 had to fix.

Sixty call sites across fifty-eight role-local specs lose the argument and nothing else. Every added line in this commit is a bare isVisible(); a grep for added lines that are not one returns zero, so a behaviour change cannot have entered here. Files whose diff also alters waiting semantics are deliberately not part of this commit.

A repository lint added in b97cf8888 keeps the construct out. It matches isVisible and isHidden only, because isChecked, isDisabled, isEditable and isEnabled take a real timeout.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 13:49:18 +02:00
kevinveenbirkenbach 05daa94733 fix(flowise): take the MCP transport from the declaration, not a constant
The reconciler hard-coded "sse" while meta/mcp.yml declared the same
fact, so the declaration decided which providers may admit the role and
the constant decided which ones it accepts. They disagreed, and only a
runtime error settled it. The transport now arrives as an environment
value and its absence aborts, because without it every provider reads as
incompatible and the registry converges to empty without complaining.

Registration now carries its response into login, so an instance that
refused to create the account says so instead of failing later as a bare
"User Not Found". The mcp group joins the SSO gate's allowed groups.
2026-08-10 00:00:50 +02:00
kevinveenbirkenbach 23ce4355e6 feat(web-app-flowise): build our own image and wire it as an MCP client
The published image lies about its version: flowiseai/flowise 3.1.3, 3.1.4 and
latest all ship package 3.1.2, measured in the running container rather than
inferred from the tag. An unpinned global npm install layer is reused between
releases. We build from the declared version instead.

The build installs locally rather than globally because flowise declares node
^24 from 3.1.3 on while resolving better-sqlite3 to 9.6.0, whose binding.gyp
pins -std=c++17 against node 24 headers demanding C++20. That version cannot
build there at all, and an npm override is only honoured for a local install.

Three upstream behaviours were read at the pinned source before being worked
around: cookie authentication is routed only behind an x-request-from header,
the key controller rejects an empty permissions list, and key creation answers
with the whole key list instead of the new key.

The identity password is written straight into the user row because flowise
accepts a password at registration only, and registration works once; a
redeploy carrying a rotated credential otherwise meets an account whose hash
still encodes the previous one.

Server registration refuses anything that does not speak SSE. This flowise
authorizes with an SSE toolkit, so registering another transport would claim a
connection that cannot be made.
2026-08-06 20:59:58 +02:00
kevinveenbirkenbachandClaude Opus 5 ab444915be feat(openwebui,flowise): wire the web UIs to the discovered MCP servers
Register every discovered server as an Open WebUI tool server, skipping
the auth schemes its ToolServerConnection cannot present so none is
stored with an unusable credential. Serialise that value with compact
separators: a space makes dotenv_quote wrap it, swarm keeps the quotes
inside the value, and Open WebUI then parses a string instead of a list
and crash-loops on startup.

Prepare the Flowise instance instead of registering servers in it.
Flowise 3.1.3 exposes no instance-level MCP registry; MCP lives in a
per-flow Custom MCP node. The role pins the transport to URL-based
servers so no flow can spawn a local command, and relaxes the host deny
list far enough to reach a container origin while loopback and the
cloud-metadata addresses stay denied. Selecting a server inside a flow
stays an operator step, and the README says so.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 22:27:46 +02:00
kevinveenbirkenbachandClaude Opus 4.8 07c6ad8e7c test(e2e): scale every nav click + timeout literal for onion targets
Over Tor the redirect chain behind a login/logout/OIDC click routinely exceeds Playwright's fixed 30s action timeout, so the click throws before the flow's scaled waits run. Sweep all role Playwright suites and the shared persona helpers: give every navigation/auth-transition click an onion-scaled timeout (.click({ timeout: resolveTimeout(30_000) })) and route the remaining raw millisecond literals (timeout options, waitForTimeout, sleep-promise setTimeout, positional wait args) through resolveTimeout. Local UI clicks (menus, toggles, modal dismissals, DOM clicks inside evaluate) stay untouched.

Two new lints keep it that way: test_no_raw_timeout_literals forbids raw numeric timeout literals in Playwright specs (opt-out: nocheck: raw-timeout), test_persona_clicks_scaled requires every click in the shared persona auth flows to carry a resolveTimeout timeout (opt-out: nocheck: unscaled-click).

Verified: node --check on all 133 changed JS files, all 13 playwright lints green, make autoformat clean, make test green (2745+196+163+4 tests OK).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 11:35:53 +02:00
kevinveenbirkenbachandClaude Opus 4.8 4c91df5de7 test(e2e): forbid raw page.goto, migrate all specs to Tor-resilient gotoOnion
New lint test_no_raw_goto.py forbids raw `page.goto(...)` in role playwright specs + persona flows (the gotoOnion wrapper in personas/utils is exempt) so navigation retries transient Tor-transport errors over onion. Migrates every raw call across ~75 roles (343 sites) to `gotoOnion(page, ...)` and teaches gotoOnion to detect relative onion URLs (`isOnionCanonical`). gotoOnion is a behaviour-identical drop-in on clearnet (single attempt, same return). Verified: no-raw-goto lint green, eslint 0 errors, make test green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-08 20:59:38 +02:00
kevinveenbirkenbachandClaude Opus 4.8 35dee215c8 test(playwright): route all timeouts through resolveTimeout for onion-compat
Add the global TIMEOUT_FACTOR and a shared timeouts.js helper: resolveTimeout(base) scales by TIMEOUT_FACTOR and, over a .onion target, an extra multiplier, so Tor's per-request latency no longer trips hard-coded timeouts. Migrate every raw timeout in the specs (timeout:/timeout=/setTimeout/waitForTimeout) to resolveTimeout, stage the helper in all three staging paths, and add test_no_raw_timeouts to forbid raw numeric timeouts. Clearnet behaviour is unchanged (factor 1: resolveTimeout(N) === N).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-08 10:38:06 +02:00
kevinveenbirkenbachandClaude Opus 4.8 9ffc52b9a7 refactor(web-app): disable LDAP variant for OIDC-only roles
akaunting, flowise, bluesky, opentalk and seaweedfs do not natively consume LDAP — their auth is OIDC / Keycloak-federated, so the LDAP deploy variant tested nothing app-specific. Set services.ldap.enabled/shared to false, remove the dedicated LDAP matrix variant plus the ldap: entries from the remaining variants, and drop the now-redundant LDAP playwright tests (akaunting/flowise test-ldap-login.js, bluesky test-ldap-broker-handoff.js). Matches the joomla/mattermost (enabled:false) and baserow (removed) precedent. opencloud is intentionally left untouched (LDAP is its real directory backend).

akaunting additionally bundles an unrelated seaweedfs object-store consumer change (test-seaweedfs.js, S3 env block, SEAWEEDFS_* flags) from concurrent work in the same role dir.

Verified against the staged tree: full lint suite passes (141 tests, 235 subtests), all meta YAML valid, node --check clean on the 3 modified specs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 15:29:55 +02:00
kevinveenbirkenbachandClaude Opus 4.8 64e64a6a81 test(web-app): split monolithic playwright specs into test-*.js
Split the monolithic playwright.spec.js of akaunting, flowise, joomla, opentalk and opencloud into per-scenario test-*.js files aggregated by a require-only playwright.spec.js, matching the baserow/mattermost convention. opentalk and opencloud gain a _shared.js holding their multi-test SSO-login helpers (attachDiagnostics + ssoLoginAndAssert*).

Pure structural split: all 28 tests are preserved verbatim (titles, asserts, selectors, timeouts, messages unchanged), the file-scoped beforeEach is inlined into the tests that relied on it, and each file re-declares only the env consts it uses. Verified: node --check on every file, old/new test-title inventory matches per role, and _shared exports cover the split files' imports.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 15:29:55 +02:00
kevinveenbirkenbachandClaude Opus 4.7 24d4ab8e4a refactor(sso)\!: collapse oauth2 + oidc into unified services.sso block (req 21)
Implements docs/requirements/021-sso-flavor-migration.md atomically:

* Schema: every role's meta/services.yml replaces the parallel
  `oauth2:` / `oidc:` top-level blocks with a single `sso:` block
  carrying `enabled`, `shared`, and a `flavor` discriminator
  (`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
  sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
  allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
  entirety; its 5 sidecar templates move under
  roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
  rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
  web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
  centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
  `is_oidc_native`) and the oauth2 sub-values
  (`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
  Three lookup plugins consume the resolver: `sso` (single-app
  predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
  consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
  plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
  and plugins/filter/compose_volumes.py all gate on
  `lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
  `enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
  → `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
  `OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
  `oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
  `OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
  `SSO_SERVICE_ENABLED`; Playwright gating helpers
  (isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
  safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
  is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
  `flavor: oauth2` with their existing `oauth2.acl.blacklist`
  (/user/login) and `oauth2.acl.whitelist` (federation + discovery
  endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
  (tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
  test_acl_mutual_exclusive.py) are deleted per Decision #10;
  test_oauth2_contract.py is renamed and rewritten to
  test_sso_contract.py, asserting that `flavor: oauth2` roles carry
  `oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
  (web-app integration) now requires the unified `sso:` block with an
  explicit flavor or a `# nocheck: sso` opt-out; new
  tests/lint/repository/no_legacy_sso_paths/ guards the source tree
  against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
  009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
  docs/contributing/actions/testing/suppression.md are updated to the
  unified SSO naming.

Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).

BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 12:59:59 +02:00
kevinveenbirkenbachandClaude Opus 4.7 c2c532c8e6 refactor(playwright): centralize dashboard-tile mechanics in web-app-dashboard SPOT; move specs into files/playwright/ subdir
Architecture
============
- New SPOT helper `roles/web-app-dashboard/files/playwright/dashboard-card-flow.js` (`runDashboardCardScenario(page, context, target)`): per `roles_with_service('dashboard')` target, asserts tile visible → click loads in `#main iframe` (outer URL gets `?iframe=` + iframe src matches canonical_domain) → header "Tab" button pops the embed into a fresh browser tab.
- `web-app-dashboard/files/playwright/playwright.spec.js` parametrised loop calls the helper for every `DASHBOARD_TARGET_ROLES_JSON` entry — the single owner of dashboard-tile mechanics across the whole codebase.

Spec relocation
===============
- 71 `roles/<role>/files/playwright.spec.js` → `roles/<role>/files/playwright/playwright.spec.js` (`git mv`, history preserved). odoo + pixelfed show as D+A in `git status` because of substantial content rewrites; coverage retained.
- New SPOT constant `ROLE_FILE_PLAYWRIGHT_SPEC` in `utils/roles/mapping.py`, registered as optional on `ROLE_TYPE_APPLICATION` (disallowed elsewhere, mirrors `ROLE_FILE_META_VARIANTS` pattern).
- `roles/test-e2e-playwright/tasks/run_one.yml`: staging task replaced with `with_fileglob` over `files/playwright/*.js` — roles can ship companion helpers (like `dashboard-card-flow.js`) next to the spec without further wiring.

Spec migrations (no more dashboard-iframe path in app specs)
============================================================
13 specs rewritten so the persona/SSO journey starts at the app URL directly, never through the dashboard tile or iframe:
* fider, mailu, mattermost, nextcloud, odoo, taiga: iframe-wrap removed (no more `frameLocator("#main iframe")`)
* bigbluebutton, discourse, kix, matrix, openwebui, peertube: dashboard goto preamble removed
* pixelfed: ~270 LOC of iframe-aware helpers/login/logout collapsed to direct-URL flow

Title cleanup
=============
- 64 specs: `"<persona>: dashboard → app → universal logout"` / `"administrator: dashboard → prometheus → app → universal logout"` → `"<persona>: app → universal logout"`.
- 13 specs: `"<persona>: dashboard to <app> ..."` → `"<persona>: <app> ..."`.

Dead-code cleanup
=================
- `dashboardBaseUrl` consts + beforeEach assertions removed from 11 specs (no functional uses; personas helper handles env validation).
- `DASHBOARD_BASE_URL=` env-template lines removed from fider, mattermost, odoo, taiga, pixelfed, kix.
- `DASHBOARD_SERVICE_ENABLED=` env-template lines removed (pixelfed retained for the still-gated spec).
- Stale "loads inside the iframe" / "FrameLocator parameter" comments cleaned in fider, odoo, prometheus, nextcloud.

Tooling
=======
- 5 lint Python files + 1 integration test now use `from utils.roles.mapping import ROLE_FILE_PLAYWRIGHT_SPEC`.
- `eslint.config.js` glob pattern + `roles/test-e2e-playwright/meta/main.yml` description + `roles/test-e2e-playwright/README.md` updated to new path.
- 25 markdown/docstring references in `docs/`, `roles/**/README.md`, `roles/**/TODO.md`, persona-helper docstrings updated to new path.
- `docs/contributing/artefact/files/role/playwright.specs.js.md`: file-placement section now documents the `files/playwright/` convention + `ROLE_FILE_PLAYWRIGHT_SPEC` SPOT constant; dashboard SPOT delegation explicit at 3 places.

Net diff: 124 files, +1142/-1564 (−422 LOC).
Integration suite: 183/183 green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 23:24:55 +02:00
kevinveenbirkenbachandClaude Opus 4.7 0e651e4a22 chore(lint): tighten ESLint rules and clean up flagged offenders
Promote four cheap rules in `eslint.config.js`:
* `no-unused-vars`: warn → error (with `argsIgnorePattern: "^_"`)
* `no-var`: error
* `eqeqeq`: error (always)
* `playwright/no-wait-for-navigation`: off → error
* `linterOptions.reportUnusedDisableDirectives`: off → warn

Add a repo-root CommonJS block so `eslint.config.js` itself sees Node
globals (`require`, `module`).

Fix every flagged offender:
* 138 stale persona-helper imports across 70 spec files
  (`isServiceEnabled`, `skipUnlessServiceEnabled`, `runAdminFlow`,
  ... — leftovers from the SPOT consolidation).
* 4 unused catch bindings → modern `catch {}` syntax.
* 2 unused `dashboardBaseUrl` declarations (minio, moodle).
* 1 unused `label` arg → renamed to `_label` to opt out per the
  argsIgnorePattern.
* 1 intentionally-kept legacy helper in web-app-wordpress's spec
  receives a per-line `eslint-disable-next-line` with reason; the
  preceding docstring already explained why it stays.
* 1 `page.waitForNavigation` in web-app-fediwall's Mastodon OIDC flow
  receives a per-line disable with reason — the navigation target
  depends on the runtime OIDC issuer; `waitForURL` would need a
  runtime-built pattern.
* 10 stale `// eslint-disable-next-line no-console` directives now
  surfaced by `reportUnusedDisableDirectives`; removed.

ESLint after this pass: 0 errors, 154 warnings (148 pre-existing
`playwright/expect-expect` plus 6 `playwright/no-force-option` that
were already inherited from `playwright/flat/recommended`).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 16:25:23 +02:00
kevinveenbirkenbachandClaude Opus 4.7 c55ad69c41 refactor: drop requirement-file references from code, add doc rule + lint
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`,
`requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from
non-Markdown source files. Comments anchored to requirement numbers
rot when requirements are renumbered or rewritten and conflate
present-state behaviour (which the comment should describe) with the
audit trail of how the code got there (which belongs in commit
messages, PR descriptions, and the requirement file itself).

* Document the rule in `docs/contributing/documentation.md` under
  "Comments". Code MUST describe the *why* in present-state language;
  contract enforcement lives in tests, not folklore.
* Clean 254 source files across `roles/`, `tests/`, `plugins/`,
  `cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root
  `eslint.config.js`. Pure comment / docstring / log-message edits;
  no behavioural change.
* Add `tests/lint/repository/test_no_req_references_in_code.py` to
  enforce the rule going forward across `.js`, `.py`, `.yml`,
  `.yaml`, `.j2`, and `.sh` files. The legacy
  `test_no_stale_requirement_refs.py` is exempt as the only file
  whose purpose is to detect these very tokens.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 16:25:22 +02:00
kevinveenbirkenbachandClaude Opus 4.7 47196731d7 refactor(playwright): migrate spec-local performKeycloakLogin to shared form helper
Nine role specs (akaunting, baserow, bluesky, bookwyrm, flowise,
fusiondirectory, jenkins, joomla, kix) carried a 3-arg
`performKeycloakLogin(page, username, password)` shadowing the shared
4-arg `performKeycloakLogin(page, username, password, canonicalDomain)`
from `personas/utils/keycloak.js`. The local definitions were
functionally identical to `performKeycloakLoginForm` (added in the
previous commit); they were missed by the earlier `performOidcLogin`
sweep because they happened to live under the canonical name.

Changes
-------
- The 9 specs strip their inline definition, rename every call site
  and comment reference `performKeycloakLogin` -> `performKeycloakLoginForm`
  (function, callers, and inline prose), and pull
  `performKeycloakLoginForm` from `./personas`.

Drift handling
--------------
- Cluster 1c681583 (5 specs) and 4b3fd133 (3 specs) used the
  input-name selector strategy (`input[name='username']` family).
  The canonical helper covers those via the `.or()` fallback in
  every selector chain.
- Cluster ffa30940 (kix) used role-based selectors with an
  exact-string `getByRole("textbox", { name: "Password" })`. The
  canonical's case-insensitive regex `/^password$/i` matches "Password"
  and any reasonable Keycloak theme variant (strictly more
  permissive).
- Sign-in button: kix's `/sign in/i` becomes the canonical's
  `/sign in|login|log in/i` plus the `input#kc-login` family via
  `.or()` (strictly broader).

Verification
------------
- 419 passed / 1 skipped / 0 failed across `tests/lint/`,
  `tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors, 283 warnings (unchanged from
  baseline).
- 0 inline `performKeycloakLogin` definitions remain in any role
  spec; the only definition lives in `personas/utils/keycloak.js`
  alongside the 4-arg variant and the deny-expecting variant.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 14:55:05 +02:00
kevinveenbirkenbachandClaude Opus 4.7 122c82910b refactor(playwright): share decodeDotenvQuotedValue + normalizeBaseUrl
Both helpers were inlined verbatim in 71 of the role-local
`playwright.spec.js` files (52 with the compact form, 19 with the
verbose form, plus one parameter-name variant). All bodies were
semantically identical.

Changes
-------
- New `roles/test-e2e-playwright/files/personas/utils/dotenv.js` holds
  the canonical pair (`decodeDotenvQuotedValue` and `normalizeBaseUrl`),
  re-exported through `personas/utils/index.js` so specs that
  `require("./personas")` get them via the existing aggregator.
- Every role spec drops its inline copies and pulls the helpers from
  `./personas` instead. Specs that only reference one of the two
  carry only that one in the destructuring (no unused-import noise).

Doc
---
`docs/contributing/artefact/files/role/playwright.specs.js.md` no
longer carries the "once a third spec needs them, MUST be promoted"
directive: the helpers live at the shared path and inline copies are
forbidden.

Verification
------------
- 419 passed / 1 skipped / 0 failed across `tests/lint/`,
  `tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors, 283 warnings (unchanged from
  baseline; no new unused-imports introduced).
- 0 inline `function decodeDotenvQuotedValue|normalizeBaseUrl`
  definitions remain in any role spec.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 14:55:05 +02:00
kevinveenbirkenbachandClaude Opus 4.7 735ae5e29d feat(playwright): migrate third batch of roles to shared persona library
Wires runGuestFlow / runBiberFlow / runAdminFlow from
roles/test-e2e-playwright/files/personas/ into the remaining 5 roles
(flowise, minio, opentalk, openwebui, wordpress). Each role gets the
spec.js persona scenarios plus matching meta/services.yml gate entries
and playwright.env.j2 env wiring; admin-flow callers pass a
role-specific adminInteraction that drives a real management surface.

Reason: continuation of d3e5ebe44 / 44d51744a — closes the long-tail
role list against the central library so per-role drift cannot
reintroduce the stub-shaped bodies the new lint forbids.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 16:11:49 +02:00
kevinveenbirkenbachandClaude Opus 4.7 91c57e0471 feat(req-013): alpha-to-beta promotion for 13 in-scope roles
Promote akaunting, baserow, bluesky, bookwyrm, bridgy-fed, flowise,
fusiondirectory, jenkins, joomla, minio, postmarks, libretranslate, xmpp
from `lifecycle: alpha` to `lifecycle: beta` per requirement 013.
socialhome stays alpha (operator-excluded).

SSO/LDAP integration per role:
- minio + jenkins + joomla + fusiondirectory: native first-party paths
  (MINIO_IDENTITY_OPENID/_LDAP, oic-auth + ldap + role-strategy JCasC,
  plg_system_keycloak, LDAP-as-storage). All V0+V1 verified.
- akaunting, baserow, bookwyrm, flowise, postmarks, libretranslate:
  oauth2-proxy perimeter gate as universal glue path. Native in-app
  OIDC integrations deferred to follow-up issues #219 (Baserow),
  #220 (BookWyrm), #221 (Akaunting).
- bluesky: variant A+ login-broker (oauth2-proxy → broker → social-app)
  with AES-256-GCM-encrypted PDS app-passwords cached in-process
  (Keycloak user-attribute storage caused LDAP federation conflicts
  that broke other realm-clients' OIDC logins).
- xmpp: native ejabberd LDAP backend for both variants
  (mod_oauth2_client is contrib-only and deferred).
- bridgy-fed: documented SSO/LDAP/RBAC exception (federated source-
  platform auth, no local user table).

Project-wide bugfixes:
- web-app-oauth2-proxy: propagate X-Forwarded-User /
  -Preferred-Username / -Email / -Groups via auth_request_set bridge
  (set_xauthrequest + pass_user_headers in cfg + nginx headers in
  following_directives.conf). Benefits all 8 oauth2-proxy consumers.
- scripts/tests/deploy/local/deploy/fresh-kept-app.sh: bake
  `RUNTIME=dev` into the matrix-init host vars so the Playwright
  E2E gate fires (was silently skipped on kept-deploys).

From-scratch role implementations: jenkins (JCasC + plugin pre-bake +
JVM cacerts CA-trust entrypoint), fusiondirectory (LDAP-as-storage
compose), xmpp (ejabberd LDAP backend ejabberd.yml).

Verification:
- Per-role baseline: each in-scope role passed standalone V0/V1
  Playwright suites green.
- Multi-app fresh-purged-apps over all 13 in-scope roles
  (FULL_CYCLE=true): Pass 1 sync + Pass 2 async both `failed=0`,
  every role's Playwright suite green.

Doc 013 acceptance section reflects the divergences (oauth2-proxy
glue path, in-memory bluesky cache, deferred contrib modules) with
explicit follow-up issue references.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 12:47:01 +02:00