Commit Graph
34 Commits
Author SHA1 Message Date
Alejandro Roman 333079cc2a feat(UI): fix migration, errors happening in local deploy and swarm integraton + merged main 2026-08-30 21:39:52 +04:00
Alejandro Roman 7332bd66ae Merge upstream/main into feature/role-stalwart-mail-swarm 2026-08-18 01:42:23 +04:00
kevinveenbirkenbachandClaude Opus 5 e16eb5728d test(lint): drop the minimal variant's exemption from the onion
72fd9c870 required the onion in every variant but exempted variant 1, on the
argument that the minimal round is the one place the stack runs without it. That
argument does not hold: the onion is part of the deployment under test, not a
feature bolted onto the full shape. A round that skips it is not a smaller
deployment, it is a different one.

So the exemption is gone and 95 variant-1 entries pin the onion on. Two
exclusions remain, both read from data rather than a comment: a role with no tor
block has no onion to pin, and a role whose meta/services.yml pins tor.enabled to
a literal false has already declared it has none.

That put the rule in conflict with test_coverage, which requires both polarities
of every dynamic flag: with tor on everywhere, the false side has no variant left
to live in. 94 roles take the exemption that contract already provides for
exactly this - "flags whose group-derived value is a hard invariant that variants
must never override", the same one 4e6ef2542 used for the seaweedfs frontend.
web-app-nextcloud needs none: its variant 8 pairs the clearnet-only PeerTube and
keeps the false polarity.

Measured, because the declaration is not what CI runs. assign_tor gives the onion
to every second capable entry with the run number as the parity switch, so more
capable variants do not mean proportionally more onion jobs: across five roles
and 21 entries the count goes 8 -> 10 on an even run and 9 on an odd one. The
clearnet path also survives - checkmk reports v1=false on run 0 and v1=true on
run 1, so both states are still covered across two consecutive runs. What the
94 markers give up is a declarative guarantee in variants.yml, not the coverage
itself.

The real cost is weight, not job count. web-app-nextcloud variant 1 goes from
3.09GB reservation / 7.27GB limit to 18.26GB / 43.01GB once svc-net-tor and its
closure join, so the cheap round is no longer cheap. Every variant stays under
the 32/64GB budget, and the same effect applies to the other 94 unmeasured.

Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles/meta/
variants plus the role-meta lints 39 passed. The rule was checked in both
directions - it reported the variant-1 offenders before the flips and none after.

Not measured: what the heavier rounds do to wall-clock, against swarm jobs capped
at 355 minutes and one that has hit that ceiling before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-15 06:09:02 +02:00
kevinveenbirkenbachandClaude Opus 5 b2a0a317fa refactor(tor): loosen the onion bond to 0.5
Every variants.yml header states the rule the bond feeds: variant 0 carries each
tightly-coupled (bond>=1, same-host) service, while loosely-coupled (bond<1,
separate-host) partners are excluded from the closure and distributed across
later variants to keep every host under budget. tor was declared bond: 1 in all
101 consumers, so it had to sit in every variant-0 closure.

It does not belong there. The onion is a separate-host provider reached over the
node's hidden service, the same shape as the bridged partners that already carry
0.5, and f1dff50fb established both the reasoning and the precedent when it
dropped xwiki and bigbluebutton from 1 to 0.5: bridged partners with their own
role and their own spec, whose bond declaration was the outlier rather than
their placement.

This also unblocks the variant-level tor contract. Requiring a non-baseline
variant to pin the onion on while tor still counted as tightly coupled would
have pulled variant 0's closure requirements along with it.

Verified: tests/lint 336 passed / 307 subtests and tests/integration/roles 107
passed / 610 subtests, including test_resource_budget over every variant's
deduplicated footprint and the bond-aware test_order closure.

Not measured, and the honest cost: f1dff50fb's own caveat applies unchanged
here. Whether redistributing a service out of the baseline closure buys back
wall-clock is not established; role-runtimes has the per-role seconds and no run
has been made with this shape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-15 06:09:01 +02:00
kevinveenbirkenbach e04c448cdd Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
Three conflicts, none resolved by picking a side wholesale:

- Makefile: took the incoming ACT_INPUTS, which routes through SWARM_DISTROS
  instead of inlining the same expression a second time.
- test-e2e-playwright/02_run_one.yml: took the incoming rc handling. An absent
  rc now counts as failure; the previous default(0) let an aborted Playwright
  run pass silently, contradicting the debug message right above it.
- web-app-peertube spec: combined both. Kept gotoOnion and resolveTimeout,
  which this branch's lint enforces, and adopted the incoming retry-inside-poll
  for the OIDC button plus its networkidle wait.
2026-07-30 15:38:15 +02:00
kevinveenbirkenbachandClaude Opus 5 028f4b85f4 chore(storage): set every image-backed min_storage to its image size plus 200MB
The declared values were guesses that ran up to 60x the size of the image they belong to: seaweedfs asked for 30GB against a 104MB image, nextcloud for 20GB against 353MB. Because min_storage feeds the storage guard, the runner packer and the sufficient-storage filter, those numbers made hosts look too small and pruned caches that deploys did not need to lose.

Every service that declares min_storage together with image and version now carries the measured compressed transfer size of that image plus 200MB of headroom. Sizes were read from the registries through utils.docker.registry.manifest_transfer_size, the same path the headroom test uses, so the declarations and the gate agree by construction. Values are written in MB throughout, matching the existing 500MB / 100MB / 50MB style.

119 declarations across 89 roles. Range: 206MB (memcached) to 3473MB (ollama); four exceed 1GB. The 84 min_storage entries that name no image are untouched, since there is nothing to measure them against.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:08:51 +02:00
Alejandro Roman ac101d2efa Merge kevinveenbirkenbach/feature/swarm-nfs into stalwart integration branch 2026-07-21 10:00:17 +02:00
kevinveenbirkenbachandClaude Fable 5 d88f272f9f Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 15:31:08 +02:00
kevinveenbirkenbachandClaude Opus 4.8 7a6268fa96 refactor(deploy): replace skip with services modes
Per-role deploy-mode opt-out moves from meta/tests.yml 'skip:' to meta/services.yml <entity>.modes.{compose,swarm}.enabled on the primary entity. get_role_skip now derives from it; get_role_mode_enabled added. A new lint requires every role shipping templates/*compose*.yml.j2 to declare both mode flags. Migrates 107 services.yml and drops 11 skip-only tests.yml. Also removes the variant_status deployability gate so swarm runs one job per variant unconditionally (variant_status.py and deployable_variant_indices deleted, complexity model simplified). Adds the i8matix alias.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-16 03:44:17 +02:00
kevinveenbirkenbachandClaude Opus 4.8 f76bfcf531 refactor(svc-bkp-volume-2-local): rename role from svc-bkp-container-2-local
Rename the role, its test tree, the primary entity key
(volume-2-local), the derived vars (BKP_VOLUME_2_LOCAL_*,
SYS_SERVICE_BACKUP_VOLUME_2_LOCAL), the run-once fact, and every
group_names consumer flag, include path, doc, and lint reference. The
container_backup service key and the on-disk backup-docker-to-local
tree keep their names: both are contracts with existing consumer
entries, snapshot trees, and the baudolo pull tooling.

Operators must rename the svc-bkp-container-2-local inventory group and
disable the previous versioned units on already-provisioned hosts; the
pre-deploy snapshot glob derives its base from the new role name and
will not find them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 18:52:02 +02:00
kevinveenbirkenbachandClaude Opus 4.8 4473edafb2 feat(roles): declare the container_backup consumer flag on stack roles with volumes
Every invokable stack role that declares named volumes in meta/volumes.yml now carries the standard container_backup consumer entry: enabled resolves via 'svc-bkp-container-2-local' in group_names, shared stays literally true since there is no per-consumer backup instance. Playwright/dynamic-flag/bond-baseline suppressions ride on the block per the established idioms; stale explanatory comments in the touched files are dropped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-09 23:04:40 +02:00
kevinveenbirkenbachandClaude Opus 4.8 dff306dd9a refactor(kix): drop the embedded postgres and scale via the house database pattern
The manager pin existed for the embedded kix-db container (private registry image with node-local pgdata). Remove that container and wire KIXDB_* through the database lookup: central svc-db-postgres (pg_trgm pre-activated via services.postgres.extensions) or the auto-rendered sidecar when no central provider is in the inventory. The proxy service floats and scales to the topology default; backend stays a pinned singleton (it bundles the kix.Daemon scheduler) and frontend too (websocket notifications without cross-instance pub/sub). Variants pin postgres.shared once true/once false. Existing embedded-postgres data is not migrated (fresh-install semantics for the DB move).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 02:37:26 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e191b91de0 style: apply make autoformat (whitespace + quote/line normalization)
make autoformat output across role meta files, a task file and 19_tor.yml (single->double quotes on mode values, Jinja line reflow). No semantic changes. Kept separate from the onion CSP / oauth2-proxy fixes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 13:25:44 +02:00
kevinveenbirkenbachandClaude Opus 4.8 77be21f5fd feat(tor): mark tor as non-playwright-gateable in domain services.yml
Onion routing is transport-layer with no persona/UI surface, so tor cannot be gated from the per-persona Playwright specs. Mark the tor block with '# nocheck: playwright-service-flag' in the 60 domain roles that ship a playwright.env.j2, per test_env_services_match's documented exemption for services that legitimately cannot be Playwright-gated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 23:53:17 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0a7164af48 feat(tor): per-role onion service block in meta/services.yml
Domain-serving roles reference the svc-net-tor provider via a tor: block. Consumers declare enabled (dynamic group_names gate) and shared: true (static — onion is always the shared node model, inline nocheck: dynamic-flag). The provider svc-net-tor additionally declares exclusive/primary defaults that consumers inherit via resolve_service_config unless overridden.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 23:53:16 +02:00
kevinveenbirkenbachandClaude Fable 5 9c84ae2240 feat(engine): shared-aware container field + SPOT container names via services.<x>.name
Container-name vars derived their value inconsistently (hard-coded literals, entity_name composites) instead of the role-owned SPOT. Add a lint requiring every *_CONTAINER var to resolve through lookup('config', app, 'services.<x>.name'), the bare entity_name form, or the engine lookup; give each multi-container worker a name: in meta/services.yml (with realistic resource limits) and switch its var to the lookup. Extend the engine lookup with a shared-aware 'container' field (central provider container when shared, embedded <entity>-<engine> otherwise) and gate zammad's embedded memcached sidecar on not-shared so its container_name never collides with the central memcached.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 00:00:49 +02:00
kevinveenbirkenbachandClaude Fable 5 30afef6bbe fix(resources): entity fallback, short compose service names + SPOT vars
resource_filter now falls back service_name -> entity_name -> hard
default, so entity-level limits keep applying to per-service compose
entries. Compose service names drop the entity prefix (kix, jitsi,
zammad, bluesky); the docker container_name keeps it via compose_only.
A new lint forbids '<entity>-'-prefixed service names.

Service names and internal ports are declared once: vars/main.yml
holds <ROLE>_<SERVICE>_SERVICE constants (container names derived via
entity_name), meta/services.yml declares ports.internal consumed
through config lookups. sso-proxy upstreams stack-qualify the origin
host in swarm because short aliases are ambiguous on shared overlays.
Stale proxy.service_key entries fixed for erpnext/shopware/nextcloud.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 17:05:56 +02:00
kevinveenbirkenbachandClaude Opus 4.8 d0ffad19a8 fix(kix): use embedded passwordless redis, not shared svc-db-redis
The KIX frontend image reads only REDIS_CACHE_HOST/PORT and ignores
REDIS_CACHE_PASSWORD, so it cannot authenticate against the shared svc-db-redis
(always requirepass + ACL) and crash-loops on NOAUTH. Pin redis.shared=false so
the engine lookup wires the embedded passwordless sidecar (host redis, no
password) that KIX's on-premise stack expects. Drop the now-static flag's
variant coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-02 16:47:34 +02:00
Alejandro Roman 9962429e59 Added playwright testing 2026-06-30 20:37:50 +02:00
Alejandro Roman 2f693f00e6 feat(stalwart):Created the role and the connection like mailu had it 2026-06-30 16:17:48 +02:00
kevinveenbirkenbachandClaude Opus 4.8 36ebf34b64 Merge branch 'main' into feature/swarm-nfs
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 17:13:55 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e1176b9613 test(lint): require ports.internal for published compose ports
The swarm resolve_upstream (utils/networks/proxy.py) reads
services.<entity>.ports.internal.<kind> and hard-fails (ValueError) when a
proxied role exposes a compose port mapping (services.<entity>.ports.local.<kind>)
but never declares its container-side internal port. Compose ignores
ports.internal, so the gap is invisible until a swarm deploy.

Add tests/lint/roles/test_swarm_internal_ports.py to catch the whole class, and
fix the three roles it flagged: web-app-funkwhale (http: 80), web-app-kix
(http: 80), web-app-zammad (http: 8080). web-app-espocrm was fixed earlier.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 03:30:11 +02:00
kevinveenbirkenbachandClaude Opus 4.8 f4774ec145 refactor(apps): remap consumers to central engines
About 38 web-app/web-svc roles now consume the central svc-db-* and svc-dns engines via lookup('engine', ...) instead of embedded sidecars: meta/services flags, variant matrices, and env/conf hosts point at the shared services. Lets these apps run unpinned and NFS-shareable in swarm.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 19:29:39 +02:00
kevinveenbirkenbachandClaude Opus 4.8 3fa781004d feat(roles): add bond integration-importance factor to shared services
`bond` is a 0..1 factor for how strongly a service is tied to the overall
integration: bond: 1 = always required, bond: 0 = omittable. Groundwork for
tooling that later scores service importance and decides what can be dropped.
Named `bond` (not `bound`) to avoid clashing with the resource bounds/limits.

Add test_service_bond.py (lint): every enabled + shared service must declare a
valid `bond` (number in 0..1). Seed `bond: 1` as a placeholder on all 638
enabled+shared service entries across 88 roles, to be refined later.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-21 06:03:17 +02:00
kevinveenbirkenbachandClaude Opus 4.7 24d4ab8e4a refactor(sso)\!: collapse oauth2 + oidc into unified services.sso block (req 21)
Implements docs/requirements/021-sso-flavor-migration.md atomically:

* Schema: every role's meta/services.yml replaces the parallel
  `oauth2:` / `oidc:` top-level blocks with a single `sso:` block
  carrying `enabled`, `shared`, and a `flavor` discriminator
  (`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
  sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
  allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
  entirety; its 5 sidecar templates move under
  roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
  rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
  web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
  centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
  `is_oidc_native`) and the oauth2 sub-values
  (`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
  Three lookup plugins consume the resolver: `sso` (single-app
  predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
  consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
  plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
  and plugins/filter/compose_volumes.py all gate on
  `lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
  `enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
  → `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
  `OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
  `oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
  `OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
  `SSO_SERVICE_ENABLED`; Playwright gating helpers
  (isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
  safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
  is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
  `flavor: oauth2` with their existing `oauth2.acl.blacklist`
  (/user/login) and `oauth2.acl.whitelist` (federation + discovery
  endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
  (tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
  test_acl_mutual_exclusive.py) are deleted per Decision #10;
  test_oauth2_contract.py is renamed and rewritten to
  test_sso_contract.py, asserting that `flavor: oauth2` roles carry
  `oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
  (web-app integration) now requires the unified `sso:` block with an
  explicit flavor or a `# nocheck: sso` opt-out; new
  tests/lint/repository/no_legacy_sso_paths/ guards the source tree
  against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
  009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
  docs/contributing/actions/testing/suppression.md are updated to the
  unified SSO naming.

Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).

BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 12:59:59 +02:00
kevinveenbirkenbachandClaude Opus 4.7 872708a288 chore(web-app-kix): bump lifecycle pre-alpha -> beta
Promote kix to beta so it enters the CI-tested lifecycle set
({alpha, beta, rc, stable}) used by scripts/meta/resolve/apps.sh, and
no longer trips the "non-CI-tested lifecycle" warning emitted by
tests/lint/ansible/roles/meta/test_invokable_lifecycle.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 20:40:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 d03db7f73e test(web-svc-css): own consumer CSS-indexing assertion + close variants/playwright gates
Adds the web-svc-css provider Playwright spec that mirrors the
dashboard/prometheus SPOT-provider pattern: emits CSS_TARGET_ROLES_JSON
via roles_with_service('css') and asserts every consumer's canonical
surface indexes a <link rel="stylesheet"> pointing at the shared CSS
asset host (CSS_BASE_URL / CDN_BASE_URL). The baseline test exercises
CDN reachability first so per-consumer failures unambiguously point
at the consumer's sys-front-inj-css wiring, not at the provider.

Suppresses the per-role playwright env-services-match guard above each
consumer's `css:` key with the canonical `# nocheck:
playwright-service-flag — verified by web-svc-css provider spec`
marker. web-svc-css/meta/services.yml gets the self-provider nocheck
on `css:` and `provider role; no consumer surface to gate` nochecks on
`cdn:` / `matomo:`.

Adds css.{enabled,shared} true/false pinning to every variants.yml
that declared the dynamic flag without coverage so
test_every_dynamic_flag_has_true_and_false_variant passes. Unifies
the css service declaration across espocrm / mediawiki / openproject
(was static enabled: false, shared: true) so the matrix matches the
rest of the consumer fleet.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 01:09:02 +02:00
kevinveenbirkenbachandClaude Opus 4.7 96e6a3daa5 feat(web-svc-css): introduce canonical owner of the css service flag
Decouples the CSS service concept from its underlying transport
(currently `web-svc-cdn`) so the flag becomes opt-in via inventory
group membership instead of staying literal-`true` everywhere.

- `roles/web-svc-css/`: new role. Marker / wrapper role that owns
  the `css` entry in the service registry, declares `web-svc-cdn` as
  the upstream that serves the actual bytes, and 301-redirects its
  own canonical domain to the cdn domain so health probes return a
  valid response.
- `roles/web-svc-cdn/meta/services.yml`: drop the `css` block — the
  canonical owner is now web-svc-css; cdn keeps `javascript`.
- 51 consumer roles (web-app-* and web-svc-{coturn,file,html,
  libretranslate,logout,onlyoffice}) migrated their `css:` block from
  literal `{ enabled: true, shared: true }` to dynamic
  `{ enabled: "{{ 'web-svc-css' in group_names }}", shared: ... }`,
  with matching matrix coverage added to each role's
  `meta/variants.yml` (true-pinning + false-pinning entries) so the
  dynamic-flag variant lint stays green.
- 41 `# nocheck: playwright-service-flag` rationale strings updated
  from the now-incorrect "canonical:cdn alias of static-asset
  provider" to "css consumer, no Playwright surface (provider:
  web-svc-css)".

Bundled refactor: `roles/web-svc-asset/tasks/01_core.yml` now
includes `web-opt-rdr-domains` directly with a single-entry mapping
to redirect its canonical domain to web-svc-file, instead of having
that wiring live in the global `tasks/stages/01_constructor.yml`
constructor. The `add_redirect_if_group` filter
(`plugins/filter/redirect_filters.py`) and its unit test are dropped
as dead code — both call sites are now role-local.

Dashboard's consumption of the new pattern is intentionally not part
of this commit and follows separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 02:20:37 +02:00
kevinveenbirkenbachandClaude Opus 4.7 ddbbda9b4a fix(roles+tests): migrate LDAP consumers to dynamic enabled + add variants-static-override guard
The nextcloud LDAP failure in CI 25939428268 traced to a variant-merge
propagation gap: when meta/services.yml declares a flag as a literal
and meta/variants.yml overrides that same flag, the merged value does
not reliably reach downstream `lookup('config', ...)` consumers.
Plugin tasks then run against a provider that the round's inventory
does not contain (e.g. `occ ldap:search` -> "Lost connection to LDAP
server"). The workaround is to anchor the base value to a
`'<role>' in group_names` expression so the inventory itself decides
whether the consumer activates.

Same anti-pattern applied to nine other shared-LDAP consumers:
funkwhale, keycloak, kix, openproject, opentalk, snipe-it, suitecrm,
taiga, web-svc-xmpp. Each had `services.ldap.enabled: true # nocheck:
dynamic-flag` paired with a dynamic `shared`. Replaced with
`"{{ 'svc-db-openldap' in group_names }}"` for both flags.

Add a regression guard at
tests/integration/roles/meta/services/test_variants_no_static_override.py
that fails whenever meta/variants.yml overrides a first-level
`services.<key>.enabled` or `services.<key>.shared` flag while the
base in meta/services.yml is a static literal. Rule
`variants-static-override` is catalogued in the suppression doc.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 15:37:11 +02:00
kevinveenbirkenbachandClaude Opus 4.7 44d51744a7 feat(playwright): migrate second batch of roles to shared persona library
Wires runGuestFlow / runBiberFlow / runAdminFlow from
roles/test-e2e-playwright/files/personas/ into another ~28 roles
(akaunting, baserow, bluesky, bookwyrm, bridgy-fed, decidim, discourse,
fider, friendica, fusiondirectory, gitea, hugo, jenkins, joomla, kix,
matrix, mattermost, mig, moodle, odoo, opencloud, peertube, pixelfed,
postmarks, sphinx, taiga, yourls, web-svc-libretranslate, web-svc-xmpp).
Each role gets the spec.js persona scenarios plus matching
meta/services.yml gate entries and playwright.env.j2 env wiring;
admin-flow callers pass a role-specific adminInteraction that drives
an actual management surface so the body is not a stub.

Reason: continuation of d3e5ebe44 — req 019 Rule 3 demands every
role exercise the canonical guest/biber/administrator flows; this
batch closes the long-tail role list against the central library so
per-role drift cannot reintroduce the stub-shaped bodies the new lint
forbids.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 16:09:46 +02:00
kevinveenbirkenbachandClaude Opus 4.7 5eb040eb9d chore: working-tree drift sync (meta/services.yml + meta/info.yml)
Picked up by ansible-lint --fix during a make-autoformat run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 02:07:04 +02:00
kevinveenbirkenbachandClaude Opus 4.7 e5de4e085a refactor(meta): wire services.<key>.{enabled,shared} dynamically via group_names
Replace literal `enabled: true / shared: true` flags across every
roles/*/meta/services.yml with a Jinja `"{{ '<role>' in group_names }}"`
form so the per-role services map becomes the single source of truth
for shared-service co-deployment, on a per-host basis. Inventory build
already runs through the templar before reaching the static `is True`
check, so runtime behaviour is unchanged; the new form only opens up
static-analysis and CLI tooling to the group-aware semantics.

Mechanism + lint:

- New integration test tests/integration/roles/meta/test_services_
  dynamic_flags.py enforces the dynamic shape per top-level service
  entry. Registered service keys must reference the registry-resolved
  provider role; covered keys (e.g. oauth2 -> web-app-keycloak via
  the new `covers:` block) reference the covering role.
- Two suppression placements documented in docs/contributing/actions/
  testing/suppression.md: per-flag inline `# nocheck: dynamic-flag`
  for databases (`enabled: true` stays literal, `shared` resolves
  dynamically) and block-level above the key for entries whose both
  flags legitimately stay literal (`css`, `javascript`, role-local
  `redis`, `recaptcha`, `hcaptcha`, role-local containers with
  `shared: false`).

Registry + helpers:

- utils/service_registry.py: new `is_explicit_truth(value)` helper
  accepts `True` literal OR a Jinja string carrying `in group_names`.
  Adopted by `resolve_service_dependency_roles_from_config`,
  `ServicesResolver._is_enabled_shared`, `test_web_app_email_
  integration`, `test_prometheus`, `test_universal_logout`, and the
  run_after-explicit guard so static analysis keeps resolving deps.
- New `covers:` field on a primary service entity declares
  transitive equivalents (web-app-keycloak.covers: [oauth2]).
  Exposed via `build_role_to_covered_keys` and
  `build_covered_key_to_role`, replacing the previously hard-coded
  `_TRANSITIVE_SERVICE_EQUIVALENTS` / `_SPECIAL_REFERENCE_ROLE` maps
  in the run_after and dynamic-flag tests.

Bulk migration:

- All 80 roles/*/meta/services.yml files migrated to the dynamic
  form. DB consumer keys (mariadb, postgres, ldap) keep
  `enabled: true  # nocheck: dynamic-flag` literal with a dynamic
  `shared`. Block-nocheck applied to css/javascript/recaptcha/
  hcaptcha and role-local containers (e.g. mailu's admin/imap/smtp/
  antispam/antivirus/oletools/webdav/fetchmail/webmail/resolver/
  front, nextcloud's talk).
- web-app-prometheus's primary entity now declares
  `provides: prometheus` + dynamic flags so the registry recognises
  it as a service provider. Consequence: tasks/01_core.yml split out
  with `- include_tasks: utils/once/flag.yml` as the first task and
  tasks/main.yml reduced to the standard run_once-guarded loader.
- Seven minimal no-login apps (web-app-hugo, -bridgy-fed, -chess,
  -littlejs, -navigator, -mini-qr, -mig) gained an explicit
  `logout: { enabled: false, shared: false }` opt-out (with
  `# nocheck: dynamic-flag` block-marker) to satisfy the universal-
  logout schema check.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 22:52:55 +02:00
kevinveenbirkenbach c9c8535034 Merge branch 'main' into feature/web-app-kix 2026-05-06 19:27:05 +02:00
kevinveenbirkenbachandClaude Opus 4.7 370b998635 feat(web-app-kix): KIX Service Management integration (req-015)
Deploys KIX Start as a first-class web-app-* role behind
sys-stk-front-proxy + web-app-oauth2-proxy + Keycloak realm-level
2FA at kix.<DOMAIN_PRIMARY>. The 4-container upstream stack
(proxy, backend, frontend, db) reuses the shared svc-db-redis for
OAuth2-proxy session storage and KIX cache.

Auth chain:
- OAuth2-proxy `allowed_groups` admits members of either
  `/roles/web-app-kix/administrator` or `/roles/web-app-kix/user`
  (the user role is declared via `meta/rbac.yml`).
- A custom kix-proxy nginx default.conf is bind-mounted (the
  upstream image ships an empty one) and forwards everything
  through the frontend Node server, plus copies the OAuth2-proxy
  X-Forwarded-User into a Remote-User upstream header.
- KIX `Auth::LDAP` + `Auth::Sync::LDAP` against svc-db-openldap
  with one backend per role group (admin GroupDN and user GroupDN).
  KIX UsageContext must be a single-string value; comma-list
  silently disables the backend.

Validated end-to-end via deploy-fresh-purged-apps APPS=web-app-kix
FULL_CYCLE=true (Pass 1 + Pass 2 both failed=0). Playwright 3/3
in both passes:
- TLS+HSTS at the canonical URL.
- administrator full flow: dashboard -> KIX card -> OAuth2-proxy
  -> Keycloak -> KIX-LDAP login form -> KIX UI -> universal logout
  -> re-anonymous redirect back to the OAuth2-proxy gate.
- biber full flow: same, with a Keycloak admin REST API fixture
  that puts biber into /roles/web-app-kix/user (mirrors what an
  operator does manually via the Keycloak UI).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 19:03:20 +02:00