Commit Graph
584 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 4.7 66b906a898 docs(requirements): archive 012, 013, 015-moodle, 015-opencloud
Ran `python -m cli.contributing.requirements.archive` after marking all
remaining acceptance criteria as `[x]` in:

- 012 - Package Cache (Nexus 3 OSS)
- 013 - Sub-Beta-to-Beta Lifecycle Promotion for `web-*` Roles
- 015 - OpenCloud and OpenTalk IAM integration
- 015 - Self-built Moodle image

Their headings are appended to the `## Archive` block of
`docs/requirements/README.md` and the per-requirement files are deleted.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 15:42:54 +02:00
kevinveenbirkenbachandClaude Opus 4.7 81d326ef6f docs(requirements): archive 9 completed requirements
Ran `python -m cli.contributing.requirements.archive` to move the
following fully-checked-off requirement files into the `## Archive`
block of `docs/requirements/README.md` and delete the per-requirement
files:

- 001 - Nextcloud Talk TURN relay
- 002 - Service Registry Refactoring
- 003 - Reduce `applications` and `users` to lookup-only access
- 004 - Generic RBAC role auto-provisioning via OpenLDAP
- 005 - WordPress Multisite support for OIDC-driven RBAC
- 006 - Service-gated Playwright tests
- 007 - WordPress to Discourse post round-trip
- 015 - KIX Service Management integration
- 016 - web-app-hugo

Cross-references from the still-open requirements (015-moodle,
015-opencloud, 017, 018, 019, 021) rewritten from the deleted file
paths to \`README.md#archive\`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 15:42:54 +02:00
kevinveenbirkenbachandClaude Opus 4.7 29377e1bd3 fix(cache/applications): stop variant-0 service leak via base-config defaults
`_build_application_defaults` returned variant 0 of every role with a
meta/variants.yml, which then deep-merged with the variant-N override
that build_dev_inventory bakes into the inventory. Any service flag
that variant N did not explicitly disable carried through from
variant 0 into the runtime applications map. The most visible casualty
was svc-bkp-container-2-local variant 1 (postgres + mariadb only),
where gitea + keycloak leaked in via variant 0, sys-service-loader
pulled in svc-db-openldap via keycloak.services.ldap, and the deploy
crashed in sys-svc-compose/04_files.yml on openldap's env.j2 with
"credentials is defined in schema at roles/svc-db-openldap/meta/schema.yml
but not set in application 'svc-db-openldap'".

Defaults now come from each role's variant-free base config
(meta/<topic>.yml plus schema-defaults, no variants overlay). Variants
stay accessible via get_variants for any caller that explicitly needs
a specific variant. Inventory-level applications.<app> overrides win
exactly as before.

Supporting changes that keep the variant matrix consistent under the
new defaults model:

- New lint guard tests/integration/roles/meta/variants/
  test_non_baseline_explicit_disables.py: for every dynamic
  service-key declared in a role's meta/services.yml that variant 0
  pins to literal true, every non-baseline variant MUST explicitly
  pin the same key to either literal true or to literal
  enabled: false + shared: false. Statically-enabled keys are exempt
  (same rule as test_auth_isolation.py).
- roles/web-app-{fusiondirectory,lam,nextcloud,phpldapadmin}/meta/
  variants.yml: explicit `false` pins for dashboard / email / logout /
  matomo / prometheus / talk in restrictive variants so deep-merge no
  longer hands them the deleted variant-0 inheritance.
- roles/web-app-moodle/templates/playwright.env.j2: gate the LAM
  block on `lookup('config', application_id, 'services.lam.enabled',
  false)`. LAM is a transitive helper, not a moodle-owned service, so
  the default-bearing lookup tolerates the missing path when the
  variant does not deploy LAM.
- tests/unit/utils/cache/test_variants.py: rename the legacy
  "always variant zero" coverage to TestMergedApplicationsUsesBaseConfig
  and add test_get_application_defaults_omits_variant_zero_only_keys
  as the regression guard for the leak this commit fixes.
- docs/contributing/artefact/files/role/variants.md +
  docs/contributing/design/variants.md: variant 0 now manually
  enumerates every dynamic services.yml key with
  enabled: true + shared: true; `{}` is only permitted when the role
  has no dynamic-enabled service-key.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 13:04:21 +02:00
kevinveenbirkenbachandClaude Opus 4.7 ee01f331a8 docs(variants): document variant-aware credentials generation
design/variants.md gains a "Credentials Generation" section that
explains both flows the credential machinery can take:

- CI/CD path: planner picks the round's variants -> build_dev_inventory
  emits --app-variants -> provision CLI forwards --variant -> credentials
  CLI hands the index to InventoryManager, which resolves the root
  role's variants.yml overlay so resolve_schema_includes_recursive
  sees the variant-merged services map and discovers every shared
  provider that the variant enables.
- Standalone path: any caller that omits --app-variants / --variant
  gets variant=None, InventoryManager skips the overlay, and shared-
  provider discovery walks the base meta config only.

Also adds four reference-table rows for the involved files and fixes
two unrelated wording issues in the surrounding text:

- "every dep round R needs MUST be redeployed" -> grammatical fix.
- "historical payload" / "rename the file back" -> drop historical
  framing per documentation.md's current-state-only rule.

artefact/files/role/variants.md gains a short "Credentials
Interaction" section that anchors back to the design doc so a role
author setting services.<key>.enabled+shared in a variant knows that
the credential generator will pick the provider up.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 13:04:20 +02:00
kevinveenbirkenbach 9ed04c22ab Revert "feat(test): add live registry-reachability probe for image pins"
This reverts commit 63c6912fae.
2026-05-23 07:00:51 +02:00
kevinveenbirkenbachandClaude Opus 4.7 bfe1d6fd79 refactor(sys-service-loader): rename + post-load queue mechanism
Renames the shared-services loader role from sys-utils-service-loader to
sys-service-loader (shorter, no orphaned "utils" prefix; entity name in
meta/services.yml realigns to service-loader per get_entity_name).

Introduces an explicit two-phase contract in the loader: after the
ordered preload pass, flush a `_service_loader_post_run` queue that
service roles register into during preload via the new helper
`list_or_shoot.yml` (list during preload, shoot directly outside it).

Replaces the `when: not _service_loader_preloading` guards in
sys-svc-compose and sys-svc-webserver-core with helper calls. Closes
the gap where non-web deploys would never load these post-load
dependencies; in normal web deploys behavior stays the same thanks to
each sub-role's `run_once_<role>` gate.

Docs, comments, test docstring, and constructor-stage invocation
follow the rename. Historical changelogs (CHANGELOG.md, packaging/*)
intentionally left at the original name.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 06:48:14 +02:00
kevinveenbirkenbachandClaude Opus 4.7 88bac5b8cf docs(make-exec-rename): update remaining make exec references
Trailing textual references missed in the prior cmd-migration commit:
three requirement docs and two lint-test docstrings still pointed at
`make exec`, but the target has always been `make compose-exec`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 05:31:16 +02:00
kevinveenbirkenbachandClaude Opus 4.7 63c6912fae feat(test): add live registry-reachability probe for image pins
Closes the gap left by test_image_versions, which silently skips any
non-semver pin (latest, stable, branch tags). The new test_image_reachable
walks every entry from iter_role_images and probes the Docker Registry v2
manifest endpoint via HTTPS with anonymous bearer-token negotiation
(docker.io, ghcr.io, quay.io, mcr.microsoft.com). HTTP 404 hard-fails as
a broken pin; auth/rate-limit/5xx/network errors warn. Bounded by per-
request, per-probe, and global deadlines and parallelized via
INFINITO_WORKER_FETCH.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 05:31:16 +02:00
kevinveenbirkenbachandClaude Opus 4.7 490f0919ab refactor(make): replace INFINITO_CMD with cmd parameter
INFINITO_CMD is a per-invocation argument, not a config value — drop it
from default.env, the passthrough STATIC_KEYS, and the dotenv-generator
test, and rename to lowercase `cmd` in Makefile targets, exec scripts,
internal callers, and user-facing docs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 05:31:15 +02:00
kevinveenbirkenbachandClaude Opus 4.7 f63118f34d feat(lint): enforce auth-isolation in variants.yml
Auth-isolated matrix variants (exactly one of LDAP / OIDC-OAuth2
enabled, and no non-auth `enabled: true` alongside) must explicitly
pin every non-auth service-key in the role's variant union to literal
`enabled: false, shared: false`. Without this, the variant's
auth-only test scope balloons into a full integration deploy via the
services.yml-default propagation, wasting CI minutes and obscuring
the regression signal.

Adds `tests/integration/roles/meta/variants/test_auth_isolation.py`
(rule key `variants-auth-isolation`) plus the matching catalog entry.
The trigger is narrow on purpose: a variant that enables auth
alongside other services is a deliberate integration test, not an
auth-isolated one, and the rule does not fire there.

Mass-fixes the 24 legacy auth-isolated variants the rule turned up:
each gets the missing `enabled: false, shared: false` declarations
appended to its `services:` block. Where the false-override clashes
with a static-literal base in meta/services.yml, the override line
also picks up a `# nocheck: variants-static-override` marker. The
svc-bkp-container-2-local variant 2 additionally carries
`# nocheck: variants-services-match` on the new keys because they
are resolver-only matrix hooks with no consumer contract in the base
meta.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 04:23:46 +02:00
kevinveenbirkenbachandClaude Opus 4.7 515b735522 feat(lint): add variants-services-match nocheck opt-out
The variants → services match guard (`test_services_match.py`) used to
have no exemption mechanism, forcing every variant service-key to also
appear as a top-level entry in `meta/services.yml`. That contract is
correct for application roles whose variants pin per-app service
flags, but it forces noise on invokable infrastructure roles whose
variants exist only to pin the round's companion topology — there the
service-key is a resolver hook, not a consumer-side contract, and
adding a stub in `meta/services.yml` just duplicates the variant
declaration.

Add a `# nocheck: variants-services-match` opt-out (same line or line
above the key) so legitimate resolver-only matrix hooks can stay in
variants.yml without a synthetic services.yml stub. Mirrors the
`variants-static-override` parser pattern (text walk + per-leaf line
indices + is_suppressed_at).

Adopts the new exemption on `svc-bkp-container-2-local`:
meta/services.yml drops the gitea / keycloak / postgres / mariadb /
ldap stubs, meta/variants.yml carries the nocheck marker on each
variant service-key entry. The resolver closure per round stays
identical (16 / 3 / 2 roles).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 04:23:46 +02:00
kevinveenbirkenbachandClaude Opus 4.7 aacca6025d docs(playwright-specs): require dedicated LDAP scenario per persona, drop contradictory variant-pinning sentence
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 04:23:44 +02:00
kevinveenbirkenbachandClaude Opus 4.7 5196283f6c refactor(test-e2e-playwright): bake @playwright/test into a shared node_modules
Per-role Playwright runs no longer call `npm install` against
registry.npmjs.org on every invocation. A new `01_setup.yml` runs once
per host/version, populates `_shared/node_modules/` inside the Playwright
container, and stamps the installed version in `.installed-version`.
Each `02_run_one.yml` mounts that tree read-only at `/e2e/node_modules`
and the container command becomes `npx --no-install playwright test`.
This eliminates the npmjs.org TLS-handshake flake that took out the
keycloak playwright in a recent variant-2 update, saves ~4 s per role,
and gives version bumps a single rebuild point.

Three retry layers cover transient registry failures: Ansible (3× with
5 s delay), npm fetch (5× with 10-60 s backoff), and image pull (3×).

Stage-base / shared-dir / stamp / node_modules paths live as SPOT
constants in `vars/main.yml` so call sites never reconstruct them.

Renames `tasks/run_one.yml` → `tasks/02_run_one.yml` (and adds
`01_setup.yml`) to make the include order visible by filename; updates
cross-references in `AGENTS.md`, the two `docs/contributing/...` pages,
`dump_disk_diagnostics.sh`, and the lint test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 11:23:24 +02:00
kevinveenbirkenbachandClaude Opus 4.7 b3c519b914 refactor(images): route role image refs through lookup('config', ...) so the GHCR mirror takes effect
`lookup('image', '<role>', '<svc>', 'ref')` only consulted
`images_overrides.*` plus role defaults, which the mirror pipeline
never populates — the pipeline writes into
`applications.<role>.services.<svc>.{image,version}`. As a result,
`sys-svc-container`'s `BUSYBOX_IMAGE` / `NODE_IMAGE`,
`sys-ctl-hlth-csp`'s `HEALTH_CSP_IMAGE`, and `test-e2e-playwright`'s
`TEST_E2E_PLAYWRIGHT_IMAGE` always resolved to the raw upstream ref
and hit Docker Hub directly. In CI this surfaced as
`toomanyrequests: ...unauthenticated pull rate limit` on
`busybox:1.37`.

Switches every productive `lookup('image', ...)` to
`lookup('config', '<role>', 'services.<svc>.image|version')` — the
same path templates like `web-app-oauth2-proxy` already use, which
resolves through the merged `applications.*` map and so transparently
picks up the GHCR mirror override written by the inventory creator.

Removes the now-unused `plugins/lookup/image.py` plugin and its unit
test (zero remaining consumers across roles, scripts, and tests).
Rewrites the image-origin / image-mirror / dockerfile docs to teach
the `lookup('config', ...)` pattern, dropping references to the
deleted plugin and the dead `images_overrides.*` variable. Doc edits
follow the `documentation.md` rules (no em dashes, one sentence per
line, no historical phrasing).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 02:53:05 +02:00
kevinveenbirkenbachandClaude Opus 4.7 c36929090d refactor(make): apply comment-block schema across Makefile targets
Splits mixed-content target descriptions into a single description line
plus `Note:` follow-ups, and dresses the targets that carry real
parameters (`compose-deploy`, `compose-exec`, `compose-inner-run`) with
explicit `Usage:` / `Example:` / `Param <name>:` lines. Adds a
`variant=<idx>` Make var alias to `compose-deploy` so multi-variant
pinning works without dropping to `INFINITO_VARIANT=` env prefixes.

Removes redundant inline `compose-deploy` / `compose-inner-run` param
enumerations from the docs and replaces them with
`make help target=<name>` pointers, so the help banner stays the SPOT.

Also fixes SC2016 in `scripts/make/help.sh` (literal backticks in the
help banner) with a scoped suppression.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 02:53:05 +02:00
kevinveenbirkenbachandClaude Opus 4.7 91fc422f19 feat(make): add compose-playwright target + schema-aware help
Adds `make compose-playwright role=<role> [pw="..."] [keep=true]` as the
canonical inner-loop entry point for rerunning a role-local Playwright
spec against the live stack — replaces the raw
`scripts/tests/e2e/rerun-spec.sh` invocation across docs.

Defines a small comment-block schema for Makefile targets (description
line on top, followed by `Usage:` / `Example:` / `Note:` / `Param <name>:`
markers). Lint enforces it; `make help` shows the description per target
in the short list and `make help target=<name>` prints the full block.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 02:53:05 +02:00
kevinveenbirkenbachandClaude Opus 4.7 605f29a2f8 refactor(make): rename deploy to compose-deploy
Move the deploy router into the compose- domain prefix to match the
rest of the local stack lifecycle (compose-up/down/exec/...). Reorder
its block alphabetically between compose-down and compose-entity-purge
to keep `make help` and the alphabetical-targets lint happy.

Update every active call site: contributor docs, agent-iteration docs,
role-level Administration/README/TODO files, the bundles README, the
test-environment scripts, the in-repo deploy helpers and main.sh, and
the server stage task. The two wildcard prose references (`make
deploy-*` in CLAUDE.md and testing/playwright.md) become single
`make compose-deploy` mentions since no other deploy-* target exists.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 21:38:08 +02:00
kevinveenbirkenbachandClaude Opus 4.7 2e42ade0d2 refactor(make): rename container-* purges/refresh into the compose-* family
Move the three container-* targets into the compose- domain prefix to
match their actual scope (compose stack maintenance) and to make `make
help` group them with the other compose lifecycle commands:

- container-purge-entity       -> compose-entity-purge
- container-purge-system       -> compose-system-purge
- container-refresh-inventory  -> compose-inventory-refresh

Update the prereq chain (compose-system-purge -> compose-entity-purge),
the local-deploy testing doc table, and reorder the block so the
compose- family stays alphabetically sorted between compose-down and
compose-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 21:38:08 +02:00
kevinveenbirkenbachandClaude Opus 4.7 c02229b4be refactor(make): align targets to domain prefixes (compose-, network-, etc.)
Apply a consistent naming convention to the Make target inventory so
every target is grouped by its domain:

- compose-*  (compose-up/down/restart/stop/exec/inner-run)
- network-*  (network-dns-setup/remove, network-ipv6-disable/restore,
              network-refresh, network-trust-ca)
- security-* (security-apparmor-restore/teardown)
- diagnose-* (diagnose-network, diagnose-disk-usage)
- fix-*      (fix-chmod, fix-dockerignore)
- clean-*    (clean-cache, clean-pycache-dirs)
- install-*  (install-agent)
- meta-*     (meta-list, meta-tree, meta-mig)

Drop the stale debug-network target and reintroduce its functionality
as diagnose-network. Cascade renames through:

- Makefile (.PHONY entries, prereqs, recursive $(MAKE) calls)
- contributor docs, agent-iteration docs, role-level docs
- bundle README, environment scripts, install/lint/test wrappers
- AGENTS.md, CLAUDE.md, compose/cache.override.yml, .github/workflows

Historical entries under docs/requirements/ and changelog/ stay
untouched as audit logs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 21:00:45 +02:00
kevinveenbirkenbachandClaude Opus 4.7 356e3366be feat(make): consolidate local deploy into single make deploy router
Replace the seven separate deploy-* make targets with a single `make
deploy` recipe that maps short Make variables to the canonical
INFINITO_* environment variables and dispatches via the new
scripts/tests/deploy/local/deploy/main.sh router. main.sh and all
sub-scripts continue to read the long-form names, so Ansible passthrough
and container forwarding stay unchanged.

Short Make CLI now exposes apps, mode, purge, type, bundles, disabled,
and full_cycle as aliases. The router picks the verb folder
(initialize, reinstall, update) and scope (all, selection) based on the
chosen mode plus the presence of INFINITO_APPS or INFINITO_BUNDLES.

Also folds in the global INFINITO_TEST_DEPLOY_TYPE -> INFINITO_DEPLOY_TYPE
rename across default.env, passthrough handler, dotenv test, CI deploy
workflows, discovery scripts, and the server stage task.

Active contributor and agent-iteration docs, role-level Administration
and README files, the bundles README, and the deploy reset helper are
all retargeted to the new short CLI. docs/requirements/ entries stay
untouched as historical audit logs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 20:22:28 +02:00
kevinveenbirkenbachandClaude Opus 4.7 9a0620f7bb fix(inventory): drop provision-CLI args that shadowed group_vars defaults
Root cause of the DinD DNS-check failure on test-deploy-server
(networks.internet.ip4 falling through to 127.0.0.1 despite
INFINITO_IP4=172.30.0.10 reaching every link of the env pipeline):

  cli/administration/inventory/provision/cli.py declared --ip4 with a
  hardcoded default of "127.0.0.1" and ensure_host_vars_file wrote that
  value unconditionally into host_vars/<host>.yml. group_vars'
  defaults_networks.internet.ip4 resolves "172.30.0.10" via
  \`lookup('env', 'INFINITO_IP4')\`, but tasks/stages/01_constructor.yml
  merges \`defaults_networks | combine(networks | default({}, true))\` so
  host_vars wins. INFINITO_IP4 looked broken end-to-end; the breakage
  actually lived in the provision-CLI host_vars overlay.

  INFINITO_DNS_IP and INFINITO_DOMAIN propagated correctly because
  the provision CLI had no --dns and the --primary-domain write was
  None-gated and never invoked.

While here, --ssl-disabled and --primary-domain are removed too:
neither had any effect in the dev flow. provision/cli.py applies
\`--vars-file inventories/development/default.yml\` AFTER
ensure_host_vars_file via apply_vars_overrides_from_file, whose
_deep_update_commented_map unconditionally overwrites
(\`target[key] = value\`). The dev vars-file always set
\`TLS_ENABLED: true\`, so \`--ssl-disabled\` (which forced
\`TLS_ENABLED: false\` in host_vars) was systematically reverted to
true. Dev/CI ran with TLS enabled the whole time. Inventory stays at
\`TLS_ENABLED: true\` to preserve that effective behaviour.

Removed from cli/administration/inventory/provision/cli.py:
- --ip4, --ip6: defaulted to 127.0.0.1 / ::1, shadowed group_vars
- --primary-domain: never set by any caller, None-gated write
- --ssl-disabled: always re-overwritten by --vars-file later in the
  same provision pass

Removed from cli/administration/inventory/provision/host_vars.py the
matching writes in ensure_host_vars_file; the function now only sets
\`ansible_connection: local\` for localhost-class hosts. Defaults for
DOMAIN_PRIMARY / TLS_ENABLED / networks.internet.{ip4,ip6} live in
group_vars (env-driven) and the inventory vars-file (deployment-wide
flags); host_vars stops duplicating them.

Callers cleaned up:
- cli/administration/deploy/development/inventory/builder.py
- cli/administration/deploy/container/command.py
- scripts/tests/deploy/local/utils/fresh-kept-all-init-and-deploy.sh
- scripts/tests/deploy/local/reset/inventory.sh

Tests + docs follow:
- tests/unit/cli/administration/inventory/provision/test_host_vars.py
  now asserts host_vars no longer carries baked DOMAIN_PRIMARY /
  TLS_ENABLED / networks defaults
- docs/administration/deploy.md drops --ssl-disabled from the
  provision example
- docs/contributing/design/inventory.md re-scopes the host_vars
  column: the DOMAIN_PRIMARY / TLS_ENABLED / networks.internet.{ip4,ip6}
  values arrive via --vars / --vars-file overlays, otherwise group_vars
  provides the env-driven default

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 17:05:41 +02:00
kevinveenbirkenbachandClaude Opus 4.7 13c93ad6d5 refactor(compose): drop the ci profile and its --profile ci arg everywhere
compose.yml no longer hides services behind \`profiles: ["ci"]\`. The
profile existed solely as a no-op safety net against a bare
\`docker compose up\` in the repo root, but the top-level compose.yml is
already explicitly scoped to local dev / CI testing (production goes
through the Ansible roles under roles/<role>/templates/), so the safety
net is redundant with the documented scope.

Every \`--profile ci\` callsite collapses:
- compose.yml: \`profiles: ["ci"]\` removed from coredns + infinito
- Profile.args() deleted; Profile keeps is_ci() and
  registry_cache_active() for the cache-override gating
- compose.py / down.py: profile.args() splat removed from the
  docker-compose command builder; down.py drops its now-unused
  Profile import
- 5 shell wrappers (install, lint, tests/code, meta/resolve/apps,
  meta/resolve/diff/affected_roles): plain \`docker compose\` instead
  of \`docker compose --profile ci\`
- tests/unit/.../test_profile.py: TestProfileArgs class removed
- tests/unit/.../test_compose.py: two cmd-shape asserts updated to
  reflect the missing --profile pair

Docs:
- docs/contributing/artefact/files/compose.yml.md: the
  \"every service MUST be behind a profiles: entry\" rule is gone
- docs/contributing/actions/debugging/ci.md: the
  \`docker compose ... --profile ci config\` debug snippet drops the
  flag; while touching the file the two em-dash clause separators in
  the reproduction-profiles shell block are rewritten to colons
  (documentation.md em-dash rule), and one list-item in
  compose.yml.md is collapsed to one sentence per line for
  consistency with its siblings

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 16:22:33 +02:00
kevinveenbirkenbachandClaude Opus 4.7 12a56e449d refactor(env): move env/default.env to repo-root default.env and drop env/
With env.development and env/ci.env both gone, env/default.env was the
last inhabitant of the directory. Hoisting it to the repo root removes
the empty-shell indirection: the SPOT is now [default.env](default.env),
sitting next to .env (its generated companion) and other repo-root
configuration artefacts like compose.yml, Dockerfile, and Makefile.

Path updates land in:
- generator: cli/meta/env/__main__.py (REPO_ROOT / "default.env")
- workflow: .github/workflows/test-install-make.yml (awk reads from
  default.env, error message updated in lockstep)
- lint tests: test_used_keys_declared.py and test_infinito_prefix.py
  switched from globbing env/*.env to targeting default.env directly;
  test_default_env_keys_handled.py points at the new path
- drift test: test_inventory_vars_file_spot.py whitelist + STATIC_ENV
- smoke test: tests/integration/meta/env/test_dotenv_generator.py
  STATIC_ENV + every docstring/error message

Doc-string and prose references in utils/env/*, scripts/*,
cli/administration/deploy/development/{common,coredns}.py,
Makefile, Dockerfile, and every doc page that mentioned env/default.env
are rewritten in the same pass so future readers see one path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 16:22:33 +02:00
kevinveenbirkenbachandClaude Opus 4.7 f20278b4d5 refactor(env): drop env.development marker and its compose layering
The env.development marker file existed for two purposes that no longer
justify the wiring cost: a per-machine compose --env-file override layer
appended after auto-loaded .env, and a fallback signal for
detect_runtime() to return "dev" when the marker exists. With every
INFINITO_* default now collapsed into env/default.env -> .env,
persistent per-machine overrides are best expressed as
\`export INFINITO_FOO=…\` before invoking make (load.sh honors
caller-set values via setdefault). The marker fallback in
detect_runtime was redundant with the explicit RUNTIME=dev that every
dev-flow callsite already passes (fresh-*.sh, reset/inventory.sh).

Removed:
- Makefile mark-development target + .PHONY entry
- env.development from .gitignore / .dockerignore
- env_local conditional in compose.up()
- env_development conditional in down._compose_run()
- env.development conditional in scripts/meta/resolve/apps.sh and
  scripts/meta/resolve/diff/affected_roles.sh
- env.development marker branch + the now-unused project_root parameter
  from cli/meta/runtime/detect_runtime; __main__.py caller follows
- "Optional Development Marker" section in docs/contributing/environment/
  setup.md and the marker reference in docs/administration/installation.md

Also drops two RUNTIME=='dev' guards in roles/web-app-bigbluebutton that
relied on the auto-detect path:
- the compose build etherpad task in tasks/02_compose.yml
- the freeswitch iptables-REJECT entrypoint override in
  templates/compose.override.yml.j2

With auto-detection of "dev" gone the guards would otherwise only fire
on explicit RUNTIME=dev runs, silently regressing the local-dev path
operators rely on; both behaviors now apply unconditionally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 16:22:33 +02:00
kevinveenbirkenbachandClaude Opus 4.7 07a9440ac7 refactor(env): fold ci.env into default.env so INFINITO_IP4/DNS/DOMAIN reach Ansible
The four ci.env keys (INFINITO_DNS_IP, INFINITO_DOMAIN, INFINITO_DOMAIN_RE,
INFINITO_IP4) were only honored by docker compose --env-file, so processes
that sourced .env via load.sh saw INFINITO_IP4 as unset. As a result
networks.internet.ip4 in the Ansible playbook fell through to the
127.0.0.1 default and the DinD DNS-check handler failed on the deploy
server matrix.

Moving the defaults into env/default.env makes them flow through the same
.env / load.sh / os.environ chain as every other INFINITO_* key, so
\${INFINITO_IP4} in compose.yml, lookup('env', 'INFINITO_IP4') in
group_vars, and envsubst over the CoreDNS Corefile template all see the
same value deterministically.

Side fix: utils/env/parser.py now undoes the writer's \\\\/\\" escaping for
quoted values, closing a round-trip gap that surfaced once
INFINITO_DOMAIN_RE (the only value containing a literal backslash) started
flowing through the generator.

All --env-file env/ci.env call sites (compose.py, down.py, install/lint/
test wrappers, resolve scripts, test-dns workflow + script) are removed.
coredns.py now reads .env via parse_static_env, docs and unit/integration
tests follow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 14:55:10 +02:00
kevinveenbirkenbachandClaude Opus 4.7 e4e861c261 refactor(cli): move ports/networks suggesters under cli/contributing/network and add diagnose subcommand
Relocates `cli/meta/{ports,networks/suggest}` to
`cli/contributing/network/{ports,address/suggest}` so port and subnet
allocators live alongside the rest of the contributor-side tooling
(mirror, update, changelog) instead of `meta`. Mirrors the tests
into `tests/unit/cli/contributing/network/` and renames the contract
docs to `docs/contributing/tools/network/{port,address}.md`. Prog
names update from `cli meta ports suggest` / `cli meta networks
suggest` to `cli contributing network ports suggest` / `cli
contributing network address suggest`; the corresponding lint
references and `utils.meta.scan` docstring follow.

Adds a third subcommand: `cli contributing network diagnose`
(stdlib-only DNS / TCP / TLS / PMTU probe against the standard
infrastructure hosts plus interface MTU, `/etc/resolv.conf`,
`/etc/hosts`, proxy env, CA-bundle summaries). Split across `cli.py`
(orchestration), `config.py` (SPOT for hosts / per-distro install
commands), `format.py` (output + subprocess helpers), `tools.py`
(distro-aware auto-install of `iputils-ping` + `iproute2`),
`report.py` (system-info dumps + IPv6-default-route detection), and
`probes.py` (per-host probes). Exposed as `make debug-network`.

Companion `docs/contributing/actions/debugging/network.md` absorbs
the diagnose-tool walkthrough (incl. how to read MTU-induced TLS
handshake timeouts) so the merged page covers DNS / MTU / IPv6 /
connectivity end-to-end.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 14:55:10 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a77a570409 refactor(env): consolidate INFINITO_* namespace and centralize path/image SPOTs
Closes a stack of env-pipeline regressions left over from the recent
INFINITO_* namespace refactor by:

* DOMAIN_PRIMARY now reads INFINITO_DOMAIN strictly (was: legacy DOMAIN
  env var → defaulted to infinito.localhost, which never resolved
  because CoreDNS serves the INFINITO_DOMAIN value, infinito.example).
* group_vars/all/08_networks.yml IPs (ip4/ip6/dns) now read
  INFINITO_IP4 / INFINITO_IP6 / INFINITO_DNS_IP.
* PLAYWRIGHT_KEEP_ALL renamed to INFINITO_PLAYWRIGHT_KEEP end-to-end
  (env-builder, role default, run_one.yml docker exec, playwright.config.js,
  lint allowlist, docs anchor, CI workflow vars).
* PKGMGR_IMAGE renamed to INFINITO_PARENT_IMAGE; Dockerfile ARG drops
  the inline default so env/default.env is the only SPOT.
* INFINITO_SRC_DIR centralized: Dockerfile ARG (no default) + ENV;
  compose forwards it as build-arg, volume mount path, and container env;
  every Python CLI + shell helper + CI workflow now reads it strictly
  (os.environ[...] / "${INFINITO_SRC_DIR:?...}") instead of the literal
  /opt/src/infinito.
* New entries in env/default.env: INFINITO_IP6=::1, INFINITO_PARENT_IMAGE,
  INFINITO_PLAYWRIGHT_KEEP=false. All three propagated through
  utils/env/handlers/passthrough.STATIC_KEYS and (where consumed in the
  container) through compose.yml's environment block.

Validated end-to-end with `make test` (2360 tests across lint, lint-*,
test-lint, test-external, test-unit, test-integration).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 02:30:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 80c396b839 refactor(env+install/spot): collapse python/venv resolution into shell SPOT
Pulls VENV/PYTHON/PIP/PYTHONPATH out of the dotenv handler pipeline
into scripts/meta/python/detect.sh. load.sh now sources detect.sh and
prepends ${VENV}/bin to PATH above the INFINITO_ENV_LOADED guard, so
every BASH_ENV re-entry re-resolves PYTHON the moment a venv
materialises mid-chain — the same fix as b79ed2869 but at the SPOT,
which retires the reload-trick from ansible.sh, python.sh and
pre-commit.sh.

Also renames INVENTORY_VARS_FILE → INFINITO_INVENTORY_VARS_FILE under
the new env-prefix lint, drops the 5 dead INFINITO_ENV_*_LOADED unsets
left over from the multi-loader era, and adds a lint guarding the
INFINITO_ prefix invariant on env/*.env keys.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 00:39:52 +02:00
kevinveenbirkenbachandClaude Opus 4.7 b278cd8ccd refactor(entry.sh): switch INFINITO_COMPILE{,_SILENCE} env-vars to flags
scripts/docker/entry.sh's two compile-control reads (INFINITO_COMPILE
and INFINITO_COMPILE_SILENCE) crashed during Dockerfile RUN because
that context has no env-pipeline: env/default.env exists in the repo
but no make/BASH_ENV/load.sh chain materialises it into the running
shell. The bare reads under `set -u` triggered unbound-variable exit
even though the keys were registered in default.env + passthrough.

Move build/silence control from env to script flags so entry.sh is
self-contained at bootstrap time. Each flag now triggers its action
directly inside the parse loop -- no COMPILE/SILENCE state vars:

    --compile         -> run_local_build (streaming)
    --compile-silent  -> run_local_build >/dev/null 2>&1
    --                -> end flag parsing, rest is the exec target

Drop INFINITO_COMPILE + INFINITO_COMPILE_SILENCE from the whole
registry surface: env/default.env, passthrough.STATIC_KEYS, the
infinito_compile handler (gone entirely), compose.yml's env
injection block, the dotenv-generator baseline test, the
build_env unit-test fixture, and the now-no-op `-e INFINITO_COMPILE=0`
in test-install-{make,pkgmgr} workflows.

Dockerfile RUN switches to `entry.sh --compile -- true`; runtime
ENTRYPOINT receives no flags so the default (no-compile) holds and
exec hands off to systemd via the CMD.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 23:25:39 +02:00
kevinveenbirkenbachandClaude Opus 4.7 eb4bcd23a8 refactor(env/spot): forbid .get() defaults + cache namespace rename
Three new lint tests close gaps in the INFINITO_* SPOT contract:
* tests/lint/filesystem/env/test_default_env_keys_handled.py
  Every default.env key must be wired through a handler.
* tests/lint/filesystem/env/test_used_keys_declared.py
  Every INFINITO_* in .sh/.py must live in env/{default,ci}.env or
  a handler; per-line `# nocheck:` suppresses runtime-only refs.
* tests/lint/filesystem/python/test_no_infinito_get_defaults.py
  Forbid ANY default in `.get("INFINITO_*", ...)` -- including the
  empty-string fallback (which silently masks an unset key). The
  nocheck check walks every physical line the call spans so an
  autoformat-split call still honours the marker.

Cache namespace flattens to a single prefix:
  INFINITO_PACKAGE_CACHE_*  -> INFINITO_CACHE_PACKAGE_*
  INFINITO_REGISTRY_CACHE_* -> INFINITO_CACHE_REGISTRY_*
The five previously driver-injected paths (APT_LIST, NPMRC,
PIP_CONF, REGISTRY_PROXY_CONF, FRONTEND_CA_FILE) move into
env/default.env and passthrough.STATIC_KEYS; the
`cache_env_overrides()` helper and its two call sites are gone
along with the test-fixture override for FRONTEND_CA_DIR.

SPOT-strict rewrites:
  - 19 sites: `something.get("INFINITO_X", "")` -> subscript
    `something["INFINITO_X"]` (loud KeyError when env loader did
    not run; all keys live in env/default.env).
  - 2 handler-internal bootstrap reads suppressed with
    `# nocheck: handler-bootstrap-read`.
  - 30 runtime-only / driver-injected / test-fixture references
    annotated with `# nocheck: <reason>` per-line.
  - Unit tests that previously removed SPOT-guaranteed keys via
    `{k: v ... if k \!= KEY}` or `clear=True` switch to setting
    the key to "" explicitly, matching the new contract.

Side fixes:
  - INFINITO_DNS_DOMAIN + INFINITO_DNS_HOSTS_FILE move from
    shell-side defaults in common.sh to env/default.env and
    passthrough.STATIC_KEYS.
  - dns_systemd_is_operational() drops the dead
    INFINITO_DNS_FORCE_HOSTS_FALLBACK override branch (the
    variable was never set anywhere).
  - INFINITO_WORKER_FETCH switches from `os.environ.get(KEY, "1")`
    to `os.environ[KEY]`.
  - scripts/tests/code/wrapper.sh exec args move into a bash
    array so `# nocheck:` on INFINITO_TEST_TYPE can live on its
    own element line (SC2215 fix).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 22:26:16 +02:00
kevinveenbirkenbachandClaude Opus 4.7 22cf6f38f1 refactor(env+make): rename static.env→default.env + sort + lint gates
Rename env/static.env → env/default.env across the entire repo
(string label, doc links, Python file-path constants) — the
`parse_static_env*` function names stay as a stable API surface.

Sort all `*.env` / `*.env.j2` entries alphabetically (52 in
default.env, 4 in ci.env, 46 of 49 j2 templates; 3 templates
with Jinja2 `{% %}` control flow are skipped because conditional
blocks would tear apart under a naive sort). Sort all 83
Makefile target blocks alphabetically; .PHONY group headers
stay in place.

Collapse the two remaining multi-line Makefile recipe comments
(dotenv-force, test-signed) to single-line WHY comments.

Add four lint tests under tests/lint/filesystem/ to keep the
invariants enforced going forward:
- env/test_alphabetical_sort.py
- env/test_single_line_comments.py        (# nocheck: markers exempt)
- makefile/test_alphabetical_targets.py
- makefile/test_single_line_comments.py   (# nocheck: markers exempt)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 22:26:16 +02:00
kevinveenbirkenbachandClaude Opus 4.7 4d4267e433 refactor(env): SPOT worker chain (INFINITO_WORKER_ENABLED/CPU/FETCH)
Rename INFINITO_PARALLEL -> INFINITO_WORKER_ENABLED and add two derived
keys so consumers no longer multiply by their own factor: the env
generator resolves the cascade once at `make dotenv` time and bakes
the integer counts into `.env`.

Chain:
- INFINITO_WORKER_ENABLED (operator toggle, default true)
- INFINITO_WORKER_CPU     = cpu_count() when ENABLED truthy, else 1
- INFINITO_WORKER_FETCH   = CPU * 20 for I/O-bound bulk fetches,
                            collapsing to 1 when CPU == 1

Handlers:
- utils/env/handlers/infinito_worker.py (the prior single key)
  removed
- utils/env/handlers/infinito_worker_cpu.py + infinito_worker_fetch.py
  added; registry orders them after `passthrough` so the cascade
  reads its dependency from the env-builder state.

Consumers (now 1-liners):
- tests/external/repository/test_urls_reachable.py reads
  INFINITO_WORKER_FETCH directly.
- utils/update/base.py::resolve_max_fetch_workers() reads it directly;
  utils/update/{docker,repository}.py keep using that helper plus a
  ThreadPoolExecutor for the actual bulk fetch (with `dict(pool.map(...))`
  satisfying PERF403).
- scripts/lint/autoformat.sh + docs/contributing/tools/make.md
  rename INFINITO_PARALLEL -> INFINITO_WORKER_ENABLED.

Tests:
- EXPECTED_BASELINE_KEYS in test_dotenv_generator updated to track the
  three new keys.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 20:40:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 770531395d chore(lint): drop galaxy-importer lint
galaxy-importer's legacy-role mode was dropped upstream in 0.5.0
(CLI no longer accepts --legacy-role / --namespace), and the project
does not publish roles to Ansible Galaxy. The check enforces a
publishing schema that never gets exercised, against an abandoned API
that can only be kept alive via pinning galaxy-importer==0.4.39 +
setuptools<81 in an isolated venv. The cost has stopped buying
anything.

Removed:
- scripts/lint/galaxy.sh
- utils/install/lint/galaxy_importer.py
- install_isolated_pip_pkg() in utils/install/pip.py (only galaxy_importer
  used it)
- Makefile lint-galaxy target + the entry in the `lint:` aggregate
- pre-commit make-lint-galaxy hook
- The galaxy_importer ensure() call in _install_ansible_tools
- Test assertion against cli.galaxy_importer in test_main.py
- Documentation rows mentioning lint-galaxy / galaxy-importer

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 20:40:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 92f11a6701 feat(install/lint): Python-ize lint install + host/docker dispatcher
Refactor scripts/install/lint.sh (~870 LOC bash) into a Python package
under utils/install/. scripts/install/lint.sh becomes a 7-line wrapper
that delegates to `python3 -m utils.install.lint`.

Layout:
- utils/install/{primitives,pip,npm,github_release,system_pkg}.py:
  channel-agnostic installers
- utils/install/lint/<tool>.py: one ensure() per tool (actionlint,
  shfmt, mbake, ruff, ansible_lint, galaxy_importer, markdownlint_cli2,
  eslint, shellcheck, ansible_commands, ansible_collections)
- utils/install/lint/__main__.py: CLI + group dispatch + stamp logic

INFINITO_LINT_RUNNER dispatch:
- New env var (host | docker, default docker), tracked in env/static.env
  + passthrough handler + drift test.
- scripts/install/wrapper.sh routes `make install-lint` to host or
  container based on the runner.
- scripts/lint/wrapper.sh already dispatched; container branch now uses
  bash (not --login) so /etc/profile does not reset PATH, and forwards
  BASH_ENV so the .env keys land in the lint script's env.

Stamp logic:
- Per-interpreter stamp at build/install-lint-<sha256(sys.executable)>.stamp
  so host venv and container venv track installs independently despite
  bind-mounted build/.
- `--force` drops the stamp; group args bypass the stamp.

Tests + docs:
- 43 unit tests under tests/unit/utils/install/{,lint/} cover primitives,
  pip/npm/github_release/system_pkg fallback chains, and the main CLI
  (stamp logic, group dispatch, error path).
- utils/install/README.md + utils/install/lint/README.md document the
  structure and the per-tool ensure() contract.
- docs/contributing/tools/make.md install-lint row updated to mention
  the runner dispatch + per-env stamp + install-lint-force.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 20:40:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 9a578952c3 feat(env): SPOT env-pipeline + INFINITO_* namespace
Replace 8 bash env modules with a single Python generator
(cli.meta.env) that materialises env/static.env + runtime context
into a gitignored .env. Each variable lives in its own handler
under utils/env/handlers/; defaults carry inline one-line `# ...`
comments.

Rename to the INFINITO_* schema:
- workflow inputs: APPS, SERVICES_DISABLED, CMD, DEBUG, WHITELIST,
  BUNDLES, FULL_CYCLE, VARIANT, RUN_FLAGS, LIMIT_HOST, PARALLEL
- inventory: INVENTORY_DIR / INVENTORY_FILE / HOST_VARS_FILE,
  PW_FILE -> INVENTORY_PASSWORD_FILE
- venv: VENV_BASE / VENV_FALLBACK / VENV_NAME
- new SPOT defaults: COMPILE_SILENCE, CONTAINER_NO_CA,
  WAIT_HEALTH_TIMEOUT_S, PACKAGE_CACHE_FRONTEND_INIT_IMAGE,
  promoted IMAGE_TAG from GHA-only to general passthrough

compose.yml hardening:
- inline \${VAR:-default} removed; strict \${VAR:?...} everywhere
- new integration lint blocks regressions

Tooling:
- scripts/install/all.sh owns the stamp-file install logic
  (Makefile target delegates)
- scripts/meta/env/load.sh is the single shell entry point;
  8 bash modules deleted
- new shell lint: no INFINITO_* defaults outside env/static.env

Tests:
- 93 new unit tests for utils/env/{builder,parser,runtime,writer}
- restructure tests/lint into repository/ (file-content rules) vs
  filesystem/{shell,python,env}/ (filesystem-shape rules)
- drift + smoke integration tests against the generator

Docs:
- variables.md is now a pointer; per-variable docs live next to
  the key in env/static.env / env/ci.env
- compose.yml.md drops the Default column

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 20:40:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 f9147e4486 refactor(scripts/github): split helpers into responsibility subfolders
Group the shell helpers under scripts/github/ into five subfolders
that match how the workflow files consume them:

  runner/   host-runner hygiene (disk, swap, Docker daemon, diagnostics)
  cancel/   run cancellation on PR-close and branch-delete events
  resolve/  derivation of workflow inputs and outputs from repo state
  update/   scheduled Update: Versions pipeline
  release/  release-time gates

Drop the now-redundant subfolder prefix from each filename, e.g.
dump_runner_state.sh becomes dump_state.sh and cancel_pull_request_runs.sh
becomes pull_request_runs.sh. Each subfolder carries a README that
describes its scope per documentation.md rules; the top-level README
points at the subfolders without inventorying files.

Propagate the new paths through ten workflow files, three docs
(workflow.md, pipeline.md, secrets.md), and two tests. Fix a real
runtime bug in enlarge_swap.sh, which still referenced the
pre-rename dump_runner_state.sh path. Extend the inline-block-size
lint hint to describe the new subfolder convention.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:50:26 +02:00
kevinveenbirkenbachandClaude Opus 4.7 0b5672d7d5 ci(update): mint GitHub App token so update PRs trigger full CI
Workflow-driven update PRs opened with GITHUB_TOKEN do not start
further workflow runs, so the CI orchestrator never executed against
them. Each PR-opening job in update.yml now mints a short-lived
GitHub App installation token via actions/create-github-app-token@v2
and forwards APP_SLUG to open_update_pr.sh, which derives the
verified-commit identity from the App user.

The job-level permissions block drops to read-only since write access
is provided exclusively by the App token. Behaviour with the secrets
unset is fail-loud: the token step fails, the PR step is skipped,
and no PR with an unintended identity is ever created.

Requires the BOT_APP_ID and BOT_APP_PRIVATE_KEY repository secrets
documented in docs/contributing/tools/github/actions/secrets.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:50:25 +02:00
kevinveenbirkenbach a38ab671ec Merge branch 'main' into feature/web-app-kix 2026-05-19 17:00:02 +02:00
kevinveenbirkenbachandClaude Opus 4.7 dd1481a9b3 feat(bundles): add deploy-bundles / redeploy-bundles flow
- new utils.inventory.bundle_apps resolver aggregates and dedupes role
  groups from inventories/bundles/{servers,workstations}/<name>/inventory.yml
- new make targets deploy-bundles (fresh-purged) and redeploy-bundles
  (reuse-kept), driven by BUNDLES=... env, wrapped by
  scripts/tests/deploy/local/deploy/{bundles,redeploy-bundles}.sh
- unit tests for the resolver under tests/unit/utils/inventory/
- document the new targets in docs/contributing/tools/make.md,
  docs/contributing/actions/testing.md, inventories/bundles/README.md
- collapse multi-line Makefile comments to one-liners for consistency
- disable nextcloud bbb plugin auto-enable (commented out)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-19 16:55:00 +02:00
kevinveenbirkenbachandClaude Opus 4.7 ab99f5dbc0 refactor(meta/services): rename ports.inter to ports.internal.<category>
The internal container port slot under <entity>.ports is now a
category-keyed map (matching the local/public shape) instead of a
single int. "internal" replaces "inter" because the latter is
ambiguous in English (could read as "between").

Affected:
- 8 role meta/services.yml (matrix, ollama, mig, shopware, oauth2-proxy,
  gitea, listmonk, sphinx) and every caller (vars, templates, group_vars).
- web-app-matrix: 8008 hardcoded across compose.yml.j2, homeserver.yaml.j2,
  tasks/01_docker.yml, and 7 mautrix bridges now reads
  MATRIX_SYNAPSE_CONTAINER_PORT from services.synapse.ports.internal.http.
- web-app-matrix: admin username/password lookups consolidated as
  MATRIX_ADMINISTRATOR_{USERNAME,PASSWORD} in vars/main.yml, sourced from
  lookup('users', 'administrator').
- Tooling: cli/meta/ports/suggest, utils/meta/scan, integration scan,
  migrate_meta_layout/role_builder updated to the new key name.
- Docs: design/role/services/layout.md, tools/ports-suggest.md,
  requirements 009/021, image/origin.md, PR template.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 11:35:43 +02:00
kevinveenbirkenbachandClaude Opus 4.7 3f12fbe83e docs(agents/iteration): mandate fresh-purged on first contact with a new VARIANT
Codify the rule that switching the live stack onto a previously-untouched
VARIANT must go through deploy-fresh-purged-apps, never a reuse target.
Reuse re-pins the stack onto stale volumes, DB rows and network aliases
from the previously-pinned variant and produces split-brain app state.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 09:44:24 +02:00
kevinveenbirkenbachandClaude Opus 4.7 e746368c20 fix(sys-svc-proxy): CSP is SPOT — strip deprecated security headers + add regression lint
Browser vendors deprecated X-Frame-Options/X-XSS-Protection/X-Download-Options/
X-Permitted-Cross-Domain-Policies; CSP (rendered from each role's
meta/server.yml via build_csp_header) is the modern replacement. To make the
SPOT actually win against upstream apps (e.g. Keycloak/Quarkus still emits
X-Frame-Options), the proxy now hides those headers on every location type.
Includes a fail-loud lint that catches any future add_header / Header set/add
re-introduction in .conf / .conf.j2 / .htaccess / .j2 / .jinja / .tmpl files.

- sys-svc-proxy: new upstream_strips.conf.j2 partial bundles HSTS + the 4
  deprecated header strips; included in html/media/ws/upload + web-svc-logout
  + web-app-oauth2-proxy locations (nginx array-replacement inheritance forces
  per-location placement)
- web-app-keycloak: meta/server.yml frame-ancestors allows the primary-domain
  wildcard with the dynamic TLS protocol so dashboard silent-check-sso iframe
  loads cleanly
- web-app-dashboard/playwright: drop isOidcSilentCheckNoise — root cause is
  now fixed, no noise filter needed
- web-app-dashboard/nginx + web-app-nextcloud/nginx: remove the deprecated
  add_header lines themselves
- tests + docs: new lint registers `deprecated-http-header` suppression rule

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 22:36:05 +02:00
kevinveenbirkenbachandClaude Opus 4.7 3f6e2dd68d refactor(sys-svc-compose): rename docker_git_repository_* → compose_*
Pure 1:1 rename (60 insertions / 60 deletions across 30 files) to
align the variable names with the role that owns them and to drop the
misleading ``docker_git_`` prefix — these knobs configure the
compose-stack's git clone step, not the docker CLI:

- docker_git_repository_address → compose_repository_url
- docker_git_repository_version → compose_repository_ref
- docker_git_repository_pull    → compose_pull_repository
- docker_repository_path        → compose_repository_path

``ref`` better captures git refs (branches/tags/SHAs are not
"versions"); ``url`` is the idiomatic name for the clone source.

Touches every consumer (web-app-akaunting/bigbluebutton/dashboard/
fediwall/hugo/littlejs/mini-qr/minio/openproject/openwebui/
roulette-wheel/taiga + svc-db-openldap/svc-prx-openresty/sys-svc-rdbms
+ web-svc-collabora/coturn/logout/onlyoffice + sys-front-inj-all), the
role's own defaults/tasks, the bigbluebutton compose_mods filter and
its unit test, the declared-in-services lint, and the hugo
requirements doc.

Breaking change for any external override that still sets the old
names — no alias is provided.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 13:43:43 +02:00
kevinveenbirkenbachandClaude Opus 4.7 904f713c99 docs(agents/role-loop): expand non-essential provider toggle to matomo,dashboard,prometheus,email,css
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 01:09:03 +02:00
kevinveenbirkenbachandClaude Opus 4.7 1fa2a50c1b docs(agents/cheatsheet): tighten Web Development Iteration prompt; drop trailing Use mode /caveman ultra from every prompt
Web Development Iteration prompt now spells out the in-container loop
explicitly: for every failing Playwright test, follow the inner-loop
procedure in role.md together with playwright.md; before every
redeploy run `make exec` and `make run` against the live stack and
fully fix and inspect every failure in-container; the next deploy
iteration only starts once every error is resolved and the fix is
empirically verified. Adds explicit prohibitions on commit/push and
on `ask`-mode approvals, and forbids premature termination — the
iteration is finished only when every role is green end-to-end.

Strips the `Use mode /caveman ultra` postscript from every other
prompt: it was an experimental mode marker that no longer applies
and only added noise. The prompts now stop at the standard
`act autonomously …` sentence.

Reformatted to comply with documentation.md: complete sentences
instead of semicolon chains, no em-dash thought breaks, RFC 2119
keywords used consistently.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 19:29:36 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a965461c53 docs(requirements): add 021 SSO flavor migration plan
Captures the unified `services.sso.{flavor,...}` target schema, the
per-role mapping for the 3 dual-block roles (bookwyrm → oauth2,
gitea/friendica → oidc), the 6-layer touchpoint inventory, the 7-step
atomic migration sequence, and the validation set. Plan only; no code
changes yet.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 19:07:01 +02:00
kevinveenbirkenbachandClaude Opus 4.7 16cab61397 fix(image-lookup): require explicit role_id to prevent wrong-role resolution
The `image` lookup inferred `role_id` from `role_name` when only
`service_name` was passed. Inference broke lazy re-evaluation: a default
like `HEALTH_CSP_IMAGE: "{{ lookup('image', 'csp-checker', 'ref') }}"`
defined in sys-ctl-hlth-csp got re-evaluated inside sys-service's
systemctl.service.j2 template (via `system_service_tpl_exec_start`),
where `role_name='sys-service'`. The lookup then searched
`roles/sys-service/meta/services.yml` for `csp-checker` and aborted with
`no image mapping found for role 'sys-service' service 'csp-checker'`.

CI run 25976825584 surfaced this on six apps (web-app-{bridgy-fed,chess,
mig,hugo,mini-qr,postmarks}) whose deploy reached sys-svc-webserver-core
outside the service-loader preload context, which is what actually
triggers the sys-ctl-hlth-csp -> sys-service include chain.

Make `role_id` mandatory in the plugin signature, remove `_infer_role_id`,
and update the three callsites that relied on inference
(sys-ctl-hlth-csp, sys-svc-container, test-e2e-playwright) to pass the
owning role id explicitly. A migration guard catches the removed
`(service_name, want)` form and emits a clear error instead of the
confusing "missing services.yml" file-not-found message.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 16:11:30 +02:00
kevinveenbirkenbachandClaude Opus 4.7 fbecc09d24 refactor(tests): regroup meta tests, add native-metrics contract
- New bidirectional contract test ensures `services.prometheus.native_metrics.enabled`
  and `roles/<role>/templates/prometheus.yml.j2` stay consistent across the roles tree,
  replacing per-role spot checks for gitea/mattermost/matrix. Document the `native-metrics`
  suppression rule and apply it to web-app-mattermost (prepared-but-default-off fragment).
- Move every test in `tests/integration/meta/config/` into the per-domain layout under
  `tests/integration/roles/meta/{services,server}`. Group prometheus tests under
  `services/prometheus/`, repositories tests under `services/repositories/`, the
  `server.csp` shape test under `server/`. Move `test_non_empty.py` one level up
  (scans all meta/*.yml, not just services.yml).
- Delete `tests/integration/meta/spots/` — both files were only `assertIn` over file
  contents (brittle, no behavioural coverage, knowledge belongs in code comments or
  proper integration tests).
- Harden `test_project_root_import.py` to catch chained `Path(__file__)…parent.parent.…`
  walks (the attribute-access twin of `parents[N]` that previously slipped through).
  Refactor 6 surfaced sites to import canonical `PROJECT_ROOT` / `ROLES_DIR`, plus
  delete the now-redundant `_find_roles_dir` / `_find_project_root` helpers in
  `native_metrics_apps` lookup and `invokable_paths` filter plugin.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 12:50:35 +02:00
kevinveenbirkenbach eb4d437989 docs(req-019): strike web-app-kix row in iteration matrix 2026-05-16 23:24:56 +02:00
kevinveenbirkenbachandClaude Opus 4.7 c2c532c8e6 refactor(playwright): centralize dashboard-tile mechanics in web-app-dashboard SPOT; move specs into files/playwright/ subdir
Architecture
============
- New SPOT helper `roles/web-app-dashboard/files/playwright/dashboard-card-flow.js` (`runDashboardCardScenario(page, context, target)`): per `roles_with_service('dashboard')` target, asserts tile visible → click loads in `#main iframe` (outer URL gets `?iframe=` + iframe src matches canonical_domain) → header "Tab" button pops the embed into a fresh browser tab.
- `web-app-dashboard/files/playwright/playwright.spec.js` parametrised loop calls the helper for every `DASHBOARD_TARGET_ROLES_JSON` entry — the single owner of dashboard-tile mechanics across the whole codebase.

Spec relocation
===============
- 71 `roles/<role>/files/playwright.spec.js` → `roles/<role>/files/playwright/playwright.spec.js` (`git mv`, history preserved). odoo + pixelfed show as D+A in `git status` because of substantial content rewrites; coverage retained.
- New SPOT constant `ROLE_FILE_PLAYWRIGHT_SPEC` in `utils/roles/mapping.py`, registered as optional on `ROLE_TYPE_APPLICATION` (disallowed elsewhere, mirrors `ROLE_FILE_META_VARIANTS` pattern).
- `roles/test-e2e-playwright/tasks/run_one.yml`: staging task replaced with `with_fileglob` over `files/playwright/*.js` — roles can ship companion helpers (like `dashboard-card-flow.js`) next to the spec without further wiring.

Spec migrations (no more dashboard-iframe path in app specs)
============================================================
13 specs rewritten so the persona/SSO journey starts at the app URL directly, never through the dashboard tile or iframe:
* fider, mailu, mattermost, nextcloud, odoo, taiga: iframe-wrap removed (no more `frameLocator("#main iframe")`)
* bigbluebutton, discourse, kix, matrix, openwebui, peertube: dashboard goto preamble removed
* pixelfed: ~270 LOC of iframe-aware helpers/login/logout collapsed to direct-URL flow

Title cleanup
=============
- 64 specs: `"<persona>: dashboard → app → universal logout"` / `"administrator: dashboard → prometheus → app → universal logout"` → `"<persona>: app → universal logout"`.
- 13 specs: `"<persona>: dashboard to <app> ..."` → `"<persona>: <app> ..."`.

Dead-code cleanup
=================
- `dashboardBaseUrl` consts + beforeEach assertions removed from 11 specs (no functional uses; personas helper handles env validation).
- `DASHBOARD_BASE_URL=` env-template lines removed from fider, mattermost, odoo, taiga, pixelfed, kix.
- `DASHBOARD_SERVICE_ENABLED=` env-template lines removed (pixelfed retained for the still-gated spec).
- Stale "loads inside the iframe" / "FrameLocator parameter" comments cleaned in fider, odoo, prometheus, nextcloud.

Tooling
=======
- 5 lint Python files + 1 integration test now use `from utils.roles.mapping import ROLE_FILE_PLAYWRIGHT_SPEC`.
- `eslint.config.js` glob pattern + `roles/test-e2e-playwright/meta/main.yml` description + `roles/test-e2e-playwright/README.md` updated to new path.
- 25 markdown/docstring references in `docs/`, `roles/**/README.md`, `roles/**/TODO.md`, persona-helper docstrings updated to new path.
- `docs/contributing/artefact/files/role/playwright.specs.js.md`: file-placement section now documents the `files/playwright/` convention + `ROLE_FILE_PLAYWRIGHT_SPEC` SPOT constant; dashboard SPOT delegation explicit at 3 places.

Net diff: 124 files, +1142/-1564 (−422 LOC).
Integration suite: 183/183 green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 23:24:55 +02:00