Commit Graph
27 Commits
Author SHA1 Message Date
infinito-nexus-updater[bot] c471346ea1 update(images): update Docker image versions (web-svc-logout) 2026-10-31 09:14:06 +00:00
infinito-nexus-updater[bot] 5fdb311936 update(images): update Docker image versions (web-svc-logout) 2026-10-31 01:17:20 +00:00
infinito-nexus-updater[bot] e50dcf876c update(images): update Docker image versions (web-svc-logout) 2026-10-30 22:34:13 +00:00
infinito-nexus-updater[bot] 2ee04cdd1a update(images): update Docker image versions (web-svc-logout) 2026-10-14 19:55:27 +00:00
infinito-nexus-updater[bot] 34a76f6c88 update(images): update Docker image versions (web-svc-logout) 2026-09-22 14:35:40 +00:00
kevinveenbirkenbachandClaude Opus 5 3005fcf4fe update(images): pin web-svc-logout to universal-logout v1.3.1
The role tracked `latest`, which was not a choice: until today the upstream
repository had never published a versioned tag at all, so `latest` was the only
thing that existed. Its release pipeline is fixed now and v1.3.1 is in the
registry, so the pin can say what it means.

Two things follow. Deploys stop depending on when a host last pulled, and the
role joins the daily update-image-versions job, which only bumps entries whose
current value is semver-shaped — `latest` kept it out of the freshness pipeline
entirely.

The `v` prefix is the actual tag: the upstream build takes `github.ref_name`
verbatim, and the registry lists exactly `latest` and `v1.3.1`.

Verified against the live registry rather than assumed. `make test-external`
passes with the pin, and a control run with a deliberately absent tag fails and
names this very line — so the green run tested this entry rather than skipping
it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 01:31:10 +02:00
kevinveenbirkenbachandClaude Opus 5 e16eb5728d test(lint): drop the minimal variant's exemption from the onion
72fd9c870 required the onion in every variant but exempted variant 1, on the
argument that the minimal round is the one place the stack runs without it. That
argument does not hold: the onion is part of the deployment under test, not a
feature bolted onto the full shape. A round that skips it is not a smaller
deployment, it is a different one.

So the exemption is gone and 95 variant-1 entries pin the onion on. Two
exclusions remain, both read from data rather than a comment: a role with no tor
block has no onion to pin, and a role whose meta/services.yml pins tor.enabled to
a literal false has already declared it has none.

That put the rule in conflict with test_coverage, which requires both polarities
of every dynamic flag: with tor on everywhere, the false side has no variant left
to live in. 94 roles take the exemption that contract already provides for
exactly this - "flags whose group-derived value is a hard invariant that variants
must never override", the same one 4e6ef2542 used for the seaweedfs frontend.
web-app-nextcloud needs none: its variant 8 pairs the clearnet-only PeerTube and
keeps the false polarity.

Measured, because the declaration is not what CI runs. assign_tor gives the onion
to every second capable entry with the run number as the parity switch, so more
capable variants do not mean proportionally more onion jobs: across five roles
and 21 entries the count goes 8 -> 10 on an even run and 9 on an odd one. The
clearnet path also survives - checkmk reports v1=false on run 0 and v1=true on
run 1, so both states are still covered across two consecutive runs. What the
94 markers give up is a declarative guarantee in variants.yml, not the coverage
itself.

The real cost is weight, not job count. web-app-nextcloud variant 1 goes from
3.09GB reservation / 7.27GB limit to 18.26GB / 43.01GB once svc-net-tor and its
closure join, so the cheap round is no longer cheap. Every variant stays under
the 32/64GB budget, and the same effect applies to the other 94 unmeasured.

Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles/meta/
variants plus the role-meta lints 39 passed. The rule was checked in both
directions - it reported the variant-1 offenders before the flips and none after.

Not measured: what the heavier rounds do to wall-clock, against swarm jobs capped
at 355 minutes and one that has hit that ceiling before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-15 06:09:02 +02:00
kevinveenbirkenbachandClaude Opus 5 b2a0a317fa refactor(tor): loosen the onion bond to 0.5
Every variants.yml header states the rule the bond feeds: variant 0 carries each
tightly-coupled (bond>=1, same-host) service, while loosely-coupled (bond<1,
separate-host) partners are excluded from the closure and distributed across
later variants to keep every host under budget. tor was declared bond: 1 in all
101 consumers, so it had to sit in every variant-0 closure.

It does not belong there. The onion is a separate-host provider reached over the
node's hidden service, the same shape as the bridged partners that already carry
0.5, and f1dff50fb established both the reasoning and the precedent when it
dropped xwiki and bigbluebutton from 1 to 0.5: bridged partners with their own
role and their own spec, whose bond declaration was the outlier rather than
their placement.

This also unblocks the variant-level tor contract. Requiring a non-baseline
variant to pin the onion on while tor still counted as tightly coupled would
have pulled variant 0's closure requirements along with it.

Verified: tests/lint 336 passed / 307 subtests and tests/integration/roles 107
passed / 610 subtests, including test_resource_budget over every variant's
deduplicated footprint and the bond-aware test_order closure.

Not measured, and the honest cost: f1dff50fb's own caveat applies unchanged
here. Whether redistributing a service out of the baseline closure buys back
wall-clock is not established; role-runtimes has the per-role seconds and no run
has been made with this shape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-15 06:09:01 +02:00
kevinveenbirkenbach 342de43111 Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
# Conflicts:
#	roles/svc-ai-ollama/tasks/01_manager_ops.yml
#	roles/web-app-decidim/templates/compose.yml.j2
#	roles/web-app-joomla/templates/compose.yml.j2
#	roles/web-app-matrix/files/flavor/ansible/runner/bootstrap.service
#	roles/web-app-matrix/templates/flavor/compose/services.yml.j2
#	roles/web-app-nextcloud/files/playwright/_shared.js
#	roles/web-svc-cdn/tasks/01_assets.yml
#	scripts/tests/deploy/swarm/utils/trigger_units.sh
#	scripts/tests/deploy/utils/filesystem/docker_dataroot.sh
2026-08-05 02:32:58 +02:00
kevinveenbirkenbachandClaude Opus 5 e3e81e816f refactor(healthcheck): declare probes in services.yml, render them from one lookup
Seven include templates under sys-svc-container decided how a container was
probed, and they read their inputs from template-local `{% set %}` scope.
That scope leaks: a value set for one service silently reached every service
below it, which is how taiga's front and gateway ended up probing a port
declared for a different service, and how bluesky's frontend probed a PDS
endpoint.

Probes now live in utils/docker/healthcheck.py, one class per flavor behind a
PROBES registry, and each service declares its own:

    healthcheck:
      flavor: curl
      path: health/ready
      start_period: 20m

`port_key` picks another entry under ports.internal for the services that
expose their probe elsewhere (keycloak on management, nextcloud talk on
signaling). Without a flavor the entry must carry an explicit test argv, so a
missing declaration fails loudly instead of rendering a probe against nothing.

Rendering HEAD's templates against the new output confirms 49 of 56 call
sites are byte-identical; the remaining seven are covered by an Ansible run.
Three probes change on purpose: erpnext's backend gains a check it never had,
bluesky's frontend drops the leaked PDS path, and msmtp_curl moves from a
hardcoded localhost:80 to the service's own port on 127.0.0.1 like every
other flavor.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 04:05:38 +02:00
kevinveenbirkenbach e04c448cdd Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
Three conflicts, none resolved by picking a side wholesale:

- Makefile: took the incoming ACT_INPUTS, which routes through SWARM_DISTROS
  instead of inlining the same expression a second time.
- test-e2e-playwright/02_run_one.yml: took the incoming rc handling. An absent
  rc now counts as failure; the previous default(0) let an aborted Playwright
  run pass silently, contradicting the debug message right above it.
- web-app-peertube spec: combined both. Kept gotoOnion and resolveTimeout,
  which this branch's lint enforces, and adopted the incoming retry-inside-poll
  for the OIDC button plus its networkidle wait.
2026-07-30 15:38:15 +02:00
kevinveenbirkenbachandClaude Opus 5 028f4b85f4 chore(storage): set every image-backed min_storage to its image size plus 200MB
The declared values were guesses that ran up to 60x the size of the image they belong to: seaweedfs asked for 30GB against a 104MB image, nextcloud for 20GB against 353MB. Because min_storage feeds the storage guard, the runner packer and the sufficient-storage filter, those numbers made hosts look too small and pruned caches that deploys did not need to lose.

Every service that declares min_storage together with image and version now carries the measured compressed transfer size of that image plus 200MB of headroom. Sizes were read from the registries through utils.docker.registry.manifest_transfer_size, the same path the headroom test uses, so the declarations and the gate agree by construction. Values are written in MB throughout, matching the existing 500MB / 100MB / 50MB style.

119 declarations across 89 roles. Range: 206MB (memcached) to 3473MB (ollama); four exceed 1GB. The 84 min_storage entries that name no image are untouched, since there is nothing to measure them against.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:08:51 +02:00
kevinveenbirkenbachandClaude Fable 5 d88f272f9f Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 15:31:08 +02:00
kevinveenbirkenbachandClaude Opus 4.8 7a6268fa96 refactor(deploy): replace skip with services modes
Per-role deploy-mode opt-out moves from meta/tests.yml 'skip:' to meta/services.yml <entity>.modes.{compose,swarm}.enabled on the primary entity. get_role_skip now derives from it; get_role_mode_enabled added. A new lint requires every role shipping templates/*compose*.yml.j2 to declare both mode flags. Migrates 107 services.yml and drops 11 skip-only tests.yml. Also removes the variant_status deployability gate so swarm runs one job per variant unconditionally (variant_status.py and deployable_variant_indices deleted, complexity model simplified). Adds the i8matix alias.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-16 03:44:17 +02:00
kevinveenbirkenbachandClaude Opus 4.8 c2fa2da88e style: apply make autoformat (whitespace only)
Indentation/whitespace normalization emitted by make autoformat across role meta/services.yml, meta/variants.yml and a task file. No semantic changes (verified: git diff --ignore-all-space is empty for every file). Kept separate from the Tor egress-router feature.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 08:22:01 +02:00
kevinveenbirkenbachandClaude Opus 4.8 77be21f5fd feat(tor): mark tor as non-playwright-gateable in domain services.yml
Onion routing is transport-layer with no persona/UI surface, so tor cannot be gated from the per-persona Playwright specs. Mark the tor block with '# nocheck: playwright-service-flag' in the 60 domain roles that ship a playwright.env.j2, per test_env_services_match's documented exemption for services that legitimately cannot be Playwright-gated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 23:53:17 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0a7164af48 feat(tor): per-role onion service block in meta/services.yml
Domain-serving roles reference the svc-net-tor provider via a tor: block. Consumers declare enabled (dynamic group_names gate) and shared: true (static — onion is always the shared node model, inline nocheck: dynamic-flag). The provider svc-net-tor additionally declares exclusive/primary defaults that consumers inherit via resolve_service_config unless overridden.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 23:53:16 +02:00
kevinveenbirkenbachandClaude Opus 4.8 36ebf34b64 Merge branch 'main' into feature/swarm-nfs
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 17:13:55 +02:00
kevinveenbirkenbachandClaude Opus 4.8 3fa781004d feat(roles): add bond integration-importance factor to shared services
`bond` is a 0..1 factor for how strongly a service is tied to the overall
integration: bond: 1 = always required, bond: 0 = omittable. Groundwork for
tooling that later scores service importance and decides what can be dropped.
Named `bond` (not `bound`) to avoid clashing with the resource bounds/limits.

Add test_service_bond.py (lint): every enabled + shared service must declare a
valid `bond` (number in 0..1). Seed `bond: 1` as a placeholder on all 638
enabled+shared service entries across 88 roles, to be refined later.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-21 06:03:17 +02:00
kevinveenbirkenbachandClaude Opus 4.8 25f286d2bf feat(swarm): per-service internal ports and entity-named services
Give each reverse-proxied app a swarm-resolvable upstream: add services.<entity>.ports.internal.http to the role metadata and rename the main compose service from the generic 'application' to the entity name, so its short DNS alias on the shared overlay is unique. Thread the service name through compose, vars (*_SERVICE_NAME), volume mounts and the oauth2 origin host. Consolidate the split image/ports blocks in bridgy-fed and postmarks into the entity block.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 19:49:18 +02:00
kevinveenbirkenbachandClaude Opus 4.7 96e6a3daa5 feat(web-svc-css): introduce canonical owner of the css service flag
Decouples the CSS service concept from its underlying transport
(currently `web-svc-cdn`) so the flag becomes opt-in via inventory
group membership instead of staying literal-`true` everywhere.

- `roles/web-svc-css/`: new role. Marker / wrapper role that owns
  the `css` entry in the service registry, declares `web-svc-cdn` as
  the upstream that serves the actual bytes, and 301-redirects its
  own canonical domain to the cdn domain so health probes return a
  valid response.
- `roles/web-svc-cdn/meta/services.yml`: drop the `css` block — the
  canonical owner is now web-svc-css; cdn keeps `javascript`.
- 51 consumer roles (web-app-* and web-svc-{coturn,file,html,
  libretranslate,logout,onlyoffice}) migrated their `css:` block from
  literal `{ enabled: true, shared: true }` to dynamic
  `{ enabled: "{{ 'web-svc-css' in group_names }}", shared: ... }`,
  with matching matrix coverage added to each role's
  `meta/variants.yml` (true-pinning + false-pinning entries) so the
  dynamic-flag variant lint stays green.
- 41 `# nocheck: playwright-service-flag` rationale strings updated
  from the now-incorrect "canonical:cdn alias of static-asset
  provider" to "css consumer, no Playwright surface (provider:
  web-svc-css)".

Bundled refactor: `roles/web-svc-asset/tasks/01_core.yml` now
includes `web-opt-rdr-domains` directly with a single-entry mapping
to redirect its canonical domain to web-svc-file, instead of having
that wiring live in the global `tasks/stages/01_constructor.yml`
constructor. The `add_redirect_if_group` filter
(`plugins/filter/redirect_filters.py`) and its unit test are dropped
as dead code — both call sites are now role-local.

Dashboard's consumption of the new pattern is intentionally not part
of this commit and follows separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 02:20:37 +02:00
kevinveenbirkenbachandClaude Opus 4.7 6fa7bd2f57 feat(playwright): expose canonical_url in roles_with_service; add web-svc-logout injection spec; parameterize matomo tracker check
- plugins/lookup/roles_with_service.py: resolve url.base via the tls
  sublookup and return canonical_url alongside id/canonical_domain.
- web-app-matomo/files/playwright/playwright.spec.js: parameterised
  matomo-tracker injection assertion per consumer over
  MATOMO_TARGET_ROLES_JSON, using target.canonical_url.
- web-svc-logout: new files/playwright/playwright.spec.js + templates/
  playwright.env.j2; checks universal-logout injection on every consumer
  declared in LOGOUT_TARGET_ROLES_JSON. meta/services.yml gains nocheck
  markers explaining why this provider has no user-facing surface to
  assert on.
- tests/unit/plugins/lookup/test_roles_with_service.py: cover the new
  canonical_url field via a stubbed tls sublookup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 23:24:56 +02:00
kevinveenbirkenbachandClaude Opus 4.7 6d27cf5578 feat(roles/dashboard): forbid non-web-app dashboard integration + drop the six existing offenders
Add `tests/integration/roles/test_dashboard_integration_scope.py`,
which fails when any non-`web-app-*` role declares a `dashboard:`
block in `meta/services.yml` or in a `meta/variants.yml` override
whose `enabled` or `shared` flag is truthy according to the canonical
`is_explicit_truth` helper (literal `True` *or* the
`"{{ '<role>' in group_names }}"` Jinja form).

The dashboard tile grid is reserved for end-user-visible web-app
surfaces. Shared services (`web-svc-*`), system roles (`sys-*`),
desktop roles (`desk-*`), and driver roles (`drv-*`) integrating it
would spam the grid with non-clickable infrastructure entries.

Drop the now-forbidden `dashboard:` blocks from the six web-svc-*
roles that previously gated dashboard tiles via the dynamic-flag form
(both in `meta/services.yml` and in their `meta/variants.yml`
matrix-pinning entries):

* web-svc-collabora
* web-svc-file
* web-svc-html
* web-svc-libretranslate
* web-svc-logout
* web-svc-onlyoffice

Also drop a stray `q` keystroke at the tail of
`web-svc-file/meta/services.yml` that broke YAML parsing once the
trailing `dashboard:` block was removed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 18:18:22 +02:00
kevinveenbirkenbachandClaude Opus 4.7 951f6d36d2 chore(lint): unify autoformat/galaxy parallelism into one PARALLEL env var
* scripts/lint/autoformat.sh: rename AUTOFORMAT_PARALLEL to PARALLEL,
  declare it at file start with default `true`; still env-overridable.
  Reword the worker comment so the literal `# shellcheck` token no
  longer appears at start-of-line and trips SC1072/SC1073.
* scripts/lint/galaxy.sh: drop the GALAXY_FATAL / GALAXY_NAMESPACE /
  GALAXY_PARALLEL env vars (over-engineered for a single repo). Failure
  is always fatal. NAMESPACE is an internal constant. Add the same
  PARALLEL env var (default `true`, env-overridable); falsy demotes the
  xargs fan-out to one worker.
* docs/contributing/environment/variables.md: collapse the four
  AUTOFORMAT_*/GALAXY_* rows into one `PARALLEL` row covering both
  consumers.
* docs/contributing/tools/make.md: refer to `PARALLEL=0` everywhere;
  add the lint-galaxy parallelism note.

Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 00:59:24 +02:00
kevinveenbirkenbachandClaude Opus 4.7 0cc48be9ae test(integration): variants coverage + no-self-reference guards for services.yml
Two new integration lints under tests/integration/roles/meta/:

* test_variants_coverage.py — every services.yml flag declared via
  the dynamic ``"{{ '<role>' in group_names }}"`` form MUST be
  exercised by meta/variants.yml: at least one variant entry pins
  it to literal ``true`` AND at least one pins it to ``false``.
  Pairs may share variant entries — the test only checks the union
  of all variants covers both polarities for every dynamic flag.
* test_services_no_self_reference.py — a role's own primary entity
  (matching ``get_entity_name(role)``) MUST NOT use the dynamic form
  for ``enabled`` / ``shared``. Both MUST be literal ``true``: the
  role's own services.yml is loaded only when the role itself is in
  group_names, so a ``'<self>' in group_names`` Jinja is tautological.
  The dynamic-flag test now skips the role's own entity to avoid a
  contradiction with this new guard.

Self-reference fixes (Jinja → literal ``true: true``) on 19 roles:
svc-ai-ollama, svc-db-{mariadb, openldap, postgres}, web-app-{
dashboard, discourse, keycloak, mailu, matomo, prometheus},
web-svc-{asset, cdn, collabora, coturn, file, libretranslate, logout,
onlyoffice, simpleicons}.

Variants coverage rollout (62 new + 17 merged meta/variants.yml):

* Roles without a pre-existing variants.yml get a new file with two
  entries — one pinning every dynamic flag true, one pinning every
  dynamic flag false. No ``- {}`` baseline (matrix runs only the
  two coverage variants).
* Roles with a pre-existing variants.yml (akaunting/baserow/bluesky/
  bookwyrm/flowise/fusiondirectory/jenkins/jira/joomla/keycloak/lam/
  minio/phpmyadmin/postmarks/wordpress/web-svc-xmpp) keep their
  matrix variants (OIDC vs LDAP, Single-Site vs Multisite, dev creds
  + TOTP-disabled realm, autologin, etc.); the all-true coverage is
  deep-merged into variant 0 and the all-false coverage into
  variant 1, so each role still ships exactly two matrix rounds.
* svc-ai-ollama keeps its single CI image-shrink variant; after the
  self-reference fix it has no remaining dynamic flags so no extra
  coverage entry is required.
* svc-db-{mariadb, openldap, postgres} drop variants.yml entirely
  for the same reason.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 00:20:46 +02:00
kevinveenbirkenbachandClaude Opus 4.7 e5de4e085a refactor(meta): wire services.<key>.{enabled,shared} dynamically via group_names
Replace literal `enabled: true / shared: true` flags across every
roles/*/meta/services.yml with a Jinja `"{{ '<role>' in group_names }}"`
form so the per-role services map becomes the single source of truth
for shared-service co-deployment, on a per-host basis. Inventory build
already runs through the templar before reaching the static `is True`
check, so runtime behaviour is unchanged; the new form only opens up
static-analysis and CLI tooling to the group-aware semantics.

Mechanism + lint:

- New integration test tests/integration/roles/meta/test_services_
  dynamic_flags.py enforces the dynamic shape per top-level service
  entry. Registered service keys must reference the registry-resolved
  provider role; covered keys (e.g. oauth2 -> web-app-keycloak via
  the new `covers:` block) reference the covering role.
- Two suppression placements documented in docs/contributing/actions/
  testing/suppression.md: per-flag inline `# nocheck: dynamic-flag`
  for databases (`enabled: true` stays literal, `shared` resolves
  dynamically) and block-level above the key for entries whose both
  flags legitimately stay literal (`css`, `javascript`, role-local
  `redis`, `recaptcha`, `hcaptcha`, role-local containers with
  `shared: false`).

Registry + helpers:

- utils/service_registry.py: new `is_explicit_truth(value)` helper
  accepts `True` literal OR a Jinja string carrying `in group_names`.
  Adopted by `resolve_service_dependency_roles_from_config`,
  `ServicesResolver._is_enabled_shared`, `test_web_app_email_
  integration`, `test_prometheus`, `test_universal_logout`, and the
  run_after-explicit guard so static analysis keeps resolving deps.
- New `covers:` field on a primary service entity declares
  transitive equivalents (web-app-keycloak.covers: [oauth2]).
  Exposed via `build_role_to_covered_keys` and
  `build_covered_key_to_role`, replacing the previously hard-coded
  `_TRANSITIVE_SERVICE_EQUIVALENTS` / `_SPECIAL_REFERENCE_ROLE` maps
  in the run_after and dynamic-flag tests.

Bulk migration:

- All 80 roles/*/meta/services.yml files migrated to the dynamic
  form. DB consumer keys (mariadb, postgres, ldap) keep
  `enabled: true  # nocheck: dynamic-flag` literal with a dynamic
  `shared`. Block-nocheck applied to css/javascript/recaptcha/
  hcaptcha and role-local containers (e.g. mailu's admin/imap/smtp/
  antispam/antivirus/oletools/webdav/fetchmail/webmail/resolver/
  front, nextcloud's talk).
- web-app-prometheus's primary entity now declares
  `provides: prometheus` + dynamic flags so the registry recognises
  it as a service provider. Consequence: tasks/01_core.yml split out
  with `- include_tasks: utils/once/flag.yml` as the first task and
  tasks/main.yml reduced to the standard run_once-guarded loader.
- Seven minimal no-login apps (web-app-hugo, -bridgy-fed, -chess,
  -littlejs, -navigator, -mini-qr, -mig) gained an explicit
  `logout: { enabled: false, shared: false }` opt-out (with
  `# nocheck: dynamic-flag` block-marker) to satisfy the universal-
  logout schema check.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 22:52:55 +02:00
kevinveenbirkenbachandClaude Opus 4.7 ed04dad5d5 refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).

req-008 — role meta layout
  * Move every role's metadata under meta/<topic>.yml with file-root
    convention (no wrapping key matching the basename):
      - schema/main.yml + config/main.yml.credentials → meta/schema.yml
      - users/main.yml                                → meta/users.yml
      - config/main.yml.server                        → meta/server.yml
      - config/main.yml.rbac                          → meta/rbac.yml
      - config/main.yml.compose.services + every other top-level key
        of config/main.yml inlined under <entity>.<key>
                                                      → meta/services.yml
      - config/main.yml.compose.volumes               → meta/volumes.yml
  * Drop the legacy schema/, users/, config/ directories from every role.
  * meta/schema.yml: support nested credential keys, optional `default:`
    field (Jinja literal, not rendered/validated/regenerated), implicit
    `algorithm: plain`.
  * Materialised paths drop the `compose.` prefix:
      compose.services.<e>.<…> → services.<e>.<…>
      compose.volumes.<k>      → volumes.<k>
      <former-top-level>.<…>   → services.<entity_name>.<former-top-level>.<…>
  * Update every Python consumer (utils/cache, utils/manager, utils/applications,
    plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
    and task file to the new paths.
  * No legacy fallback. Repository-wide grep for `compose.services.` /
    `compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
    is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
  * Add tests/lint/repository/test_role_meta_layout.py guarding against the
    old paths returning, and an explicit lint guard that meta/variants.yml
    files do not carry a `compose:` wrapper.

req-009 — per-role networks/ports
  * Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
    dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
    dns_resolver}.
  * Delete group_vars/all/09_ports.yml entirely; ports now live on the
    service entity that exposes them at
    roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
  * Rename + int-cast every former top-level `services.<entity>.port`
    field to `services.<entity>.ports.inter` (single int).
  * Always-category-keyed maps for ports.local / ports.public; relay
    spans use {start, end} integers.
  * Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
    of truth for the per-<scope>.<category> band ranges, consumed by both
    the new CLI helpers and the lint test.
  * New CLI helpers (cli meta ports suggest, cli meta networks suggest)
    walk the live role tree gap-first then increment-fallback, with
    deterministic output and non-zero exit on capacity overflow.
  * cli create role consumes both helpers when scaffolding a new role.
  * Lint extensions: bare-int port shape rejection, host-bound port
    collision check (single ints + relay span integers, inter excluded),
    band-membership check with documented allow-list for the legacy
    web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.

req-010 — run_after / lifecycle migration
  * Move galaxy_info.run_after and galaxy_info.lifecycle out of
    meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
    lifecycle}; <primary_entity> is the value returned by
    get_entity_name(role_name).
  * For multi-entity roles whose primary entity has no compose service
    (web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
    gen roles), create a dedicated metadata-only top-level entry under
    <primary_entity>.
  * Introduce utils/roles/meta_lookup.py with get_role_run_after /
    get_role_lifecycle that resolve <primary_entity> once and read from
    meta/services.yml; degrade gracefully when meta/services.yml is
    absent OR the field is absent; raise MetaServicesShapeError when the
    file exists but is malformed.
  * Update every consumer (utils/service_registry, utils/invokable,
    cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
    lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
    cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
    init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
    use the helper.
  * Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
    values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
    run_after/lifecycle never on a non-primary entity; empty run_after
    lists rejected.

Also folded into this change set
  * services_disabler now writes `services.<svc>.{enabled,shared}` into
    host_vars instead of the old `compose.services.<svc>.{...}` path; the
    discover/find pair walks meta/services.yml and only treats entries
    carrying enabled/shared/provides as provider declarations (so
    metadata-only entries from req-010 do not create spurious provider
    rows).
  * service_registry skips `run_after` deps whose target was filtered
    out via SERVICES_DISABLED instead of failing the whole load.
  * cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
    consume the new applications.<role>.services.<svc>.image shape
    (mirror.md updated to match).
  * utils/domains/list.build_applications_from_roles now reads
    meta/server.yml directly (it had been silently returning {} since
    config/main.yml stopped existing).
  * utils/roles/resource_validation.filter_roles_by_min_storage reads
    meta/services.yml.<entity>.min_storage instead of the legacy
    compose.services path.
  * web-app-bluesky pre-clones the social-app build context BEFORE
    sys-svc-compose so the build context is on disk by the time
    sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
  * 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
    web-app-wordpress, web-app-keycloak) lose their `compose: services:`
    wrapper so deep-merge over meta/services.yml lands on the new
    file-root shape.

Validation
  * Unit tests:        1571 passed.
  * Lint tests:        50 passed.
  * Integration tests: 161 passed (1 skipped).
  * Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
    set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
    discourse,mattermost}.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 09:58:26 +02:00