Commit Graph
859 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 4.7 552b8c6f1b fix(dependabot): disable maven ecosystem (no pom.xml in repo)
The maven ecosystem entry was added for a Bluesky Keycloak
event-listener SPI scaffold that has since been removed. Without a
pom.xml anywhere in the tree, every scheduled run fails with
manifest_not_found, producing a daily red Dependabot job.

Comment the block out in the same style as the other "no X found"
entries, with a note about why so a future re-enabler knows what to
restore.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:50:26 +02:00
kevinveenbirkenbachandClaude Opus 4.7 f9147e4486 refactor(scripts/github): split helpers into responsibility subfolders
Group the shell helpers under scripts/github/ into five subfolders
that match how the workflow files consume them:

  runner/   host-runner hygiene (disk, swap, Docker daemon, diagnostics)
  cancel/   run cancellation on PR-close and branch-delete events
  resolve/  derivation of workflow inputs and outputs from repo state
  update/   scheduled Update: Versions pipeline
  release/  release-time gates

Drop the now-redundant subfolder prefix from each filename, e.g.
dump_runner_state.sh becomes dump_state.sh and cancel_pull_request_runs.sh
becomes pull_request_runs.sh. Each subfolder carries a README that
describes its scope per documentation.md rules; the top-level README
points at the subfolders without inventorying files.

Propagate the new paths through ten workflow files, three docs
(workflow.md, pipeline.md, secrets.md), and two tests. Fix a real
runtime bug in enlarge_swap.sh, which still referenced the
pre-rename dump_runner_state.sh path. Extend the inline-block-size
lint hint to describe the new subfolder convention.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:50:26 +02:00
kevinveenbirkenbachandClaude Opus 4.7 0b5672d7d5 ci(update): mint GitHub App token so update PRs trigger full CI
Workflow-driven update PRs opened with GITHUB_TOKEN do not start
further workflow runs, so the CI orchestrator never executed against
them. Each PR-opening job in update.yml now mints a short-lived
GitHub App installation token via actions/create-github-app-token@v2
and forwards APP_SLUG to open_update_pr.sh, which derives the
verified-commit identity from the App user.

The job-level permissions block drops to read-only since write access
is provided exclusively by the App token. Behaviour with the secrets
unset is fail-loud: the token step fails, the PR step is skipped,
and no PR with an unintended identity is ever created.

Requires the BOT_APP_ID and BOT_APP_PRIVATE_KEY repository secrets
documented in docs/contributing/tools/github/actions/secrets.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 00:50:25 +02:00
kevinveenbirkenbachandClaude Opus 4.7 ab99f5dbc0 refactor(meta/services): rename ports.inter to ports.internal.<category>
The internal container port slot under <entity>.ports is now a
category-keyed map (matching the local/public shape) instead of a
single int. "internal" replaces "inter" because the latter is
ambiguous in English (could read as "between").

Affected:
- 8 role meta/services.yml (matrix, ollama, mig, shopware, oauth2-proxy,
  gitea, listmonk, sphinx) and every caller (vars, templates, group_vars).
- web-app-matrix: 8008 hardcoded across compose.yml.j2, homeserver.yaml.j2,
  tasks/01_docker.yml, and 7 mautrix bridges now reads
  MATRIX_SYNAPSE_CONTAINER_PORT from services.synapse.ports.internal.http.
- web-app-matrix: admin username/password lookups consolidated as
  MATRIX_ADMINISTRATOR_{USERNAME,PASSWORD} in vars/main.yml, sourced from
  lookup('users', 'administrator').
- Tooling: cli/meta/ports/suggest, utils/meta/scan, integration scan,
  migrate_meta_layout/role_builder updated to the new key name.
- Docs: design/role/services/layout.md, tools/ports-suggest.md,
  requirements 009/021, image/origin.md, PR template.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 11:35:43 +02:00
kevinveenbirkenbachandClaude Opus 4.7 181c46d742 feat(ci): render Playwright JUnit reports as workflow step summary
- New `utils/github/playwright_summary.py` parses every
  `playwright-junit.xml` under a given artefact directory and prints a
  markdown table to stdout with one row per test case: App, Test,
  Status (🟢 passed / 🔴 failed / 🔵 skipped), Duration (sub-second
  precision), Message. Sub-second formatting keeps useful per-test
  signal that integer rounding would erase. Edge cases (missing dir,
  no junit files, all-unparseable) emit a friendly placeholder note
  instead of crashing. Covered by 26 unit tests in
  `tests/unit/utils/github/test_playwright_summary.py`.
- `test-deploy-server.yml` runs the script with `if: always()` after
  the deploy step and pipes its output into `$GITHUB_STEP_SUMMARY`, so
  every matrix job's run page surfaces a per-app test table without
  needing to download artefacts.
- Playwright config adds the `github` reporter so failing tests also
  emit inline `::error file=...,line=...::`-annotations directly on the
  workflow run page and in PR diffs.
- New lint `tests/lint/filesystem/test_plugin_paths_files.py` enforces
  that files under the Ansible plugin source directories declared in
  `ansible.cfg` (`action_plugins`, `filter_plugins`, `lookup_plugins`,
  `library`, `module_utils`) are Python source files (README.md
  exempt). Paths are read live from `ansible.cfg` so a future config
  edit automatically extends the lint's coverage. Counterpart to the
  existing `test_scripts_folder_files.py` lint that enforces
  `scripts/` contains only `.sh`.

Known limitation (pre-existing pipeline behaviour, not introduced by
this change): when a role runs through multiple matrix variants, the
container path `/var/lib/infinito/logs/test-e2e-playwright/<app_id>/`
has no variant component, so each variant overwrites the previous
variant's reports. The summary therefore reflects only the LAST
variant per role. A future change to namespace the reports per
variant (or copy them after each variant pass) will make the table
cover every variant without touching the script or workflow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 15:30:05 +02:00
kevinveenbirkenbachandClaude Opus 4.7 c2c532c8e6 refactor(playwright): centralize dashboard-tile mechanics in web-app-dashboard SPOT; move specs into files/playwright/ subdir
Architecture
============
- New SPOT helper `roles/web-app-dashboard/files/playwright/dashboard-card-flow.js` (`runDashboardCardScenario(page, context, target)`): per `roles_with_service('dashboard')` target, asserts tile visible → click loads in `#main iframe` (outer URL gets `?iframe=` + iframe src matches canonical_domain) → header "Tab" button pops the embed into a fresh browser tab.
- `web-app-dashboard/files/playwright/playwright.spec.js` parametrised loop calls the helper for every `DASHBOARD_TARGET_ROLES_JSON` entry — the single owner of dashboard-tile mechanics across the whole codebase.

Spec relocation
===============
- 71 `roles/<role>/files/playwright.spec.js` → `roles/<role>/files/playwright/playwright.spec.js` (`git mv`, history preserved). odoo + pixelfed show as D+A in `git status` because of substantial content rewrites; coverage retained.
- New SPOT constant `ROLE_FILE_PLAYWRIGHT_SPEC` in `utils/roles/mapping.py`, registered as optional on `ROLE_TYPE_APPLICATION` (disallowed elsewhere, mirrors `ROLE_FILE_META_VARIANTS` pattern).
- `roles/test-e2e-playwright/tasks/run_one.yml`: staging task replaced with `with_fileglob` over `files/playwright/*.js` — roles can ship companion helpers (like `dashboard-card-flow.js`) next to the spec without further wiring.

Spec migrations (no more dashboard-iframe path in app specs)
============================================================
13 specs rewritten so the persona/SSO journey starts at the app URL directly, never through the dashboard tile or iframe:
* fider, mailu, mattermost, nextcloud, odoo, taiga: iframe-wrap removed (no more `frameLocator("#main iframe")`)
* bigbluebutton, discourse, kix, matrix, openwebui, peertube: dashboard goto preamble removed
* pixelfed: ~270 LOC of iframe-aware helpers/login/logout collapsed to direct-URL flow

Title cleanup
=============
- 64 specs: `"<persona>: dashboard → app → universal logout"` / `"administrator: dashboard → prometheus → app → universal logout"` → `"<persona>: app → universal logout"`.
- 13 specs: `"<persona>: dashboard to <app> ..."` → `"<persona>: <app> ..."`.

Dead-code cleanup
=================
- `dashboardBaseUrl` consts + beforeEach assertions removed from 11 specs (no functional uses; personas helper handles env validation).
- `DASHBOARD_BASE_URL=` env-template lines removed from fider, mattermost, odoo, taiga, pixelfed, kix.
- `DASHBOARD_SERVICE_ENABLED=` env-template lines removed (pixelfed retained for the still-gated spec).
- Stale "loads inside the iframe" / "FrameLocator parameter" comments cleaned in fider, odoo, prometheus, nextcloud.

Tooling
=======
- 5 lint Python files + 1 integration test now use `from utils.roles.mapping import ROLE_FILE_PLAYWRIGHT_SPEC`.
- `eslint.config.js` glob pattern + `roles/test-e2e-playwright/meta/main.yml` description + `roles/test-e2e-playwright/README.md` updated to new path.
- 25 markdown/docstring references in `docs/`, `roles/**/README.md`, `roles/**/TODO.md`, persona-helper docstrings updated to new path.
- `docs/contributing/artefact/files/role/playwright.specs.js.md`: file-placement section now documents the `files/playwright/` convention + `ROLE_FILE_PLAYWRIGHT_SPEC` SPOT constant; dashboard SPOT delegation explicit at 3 places.

Net diff: 124 files, +1142/-1564 (−422 LOC).
Integration suite: 183/183 green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 23:24:55 +02:00
kevinveenbirkenbachandClaude Opus 4.7 98457f98fd refactor(update): unify version bumpers under utils/update + cli/contributing/update + scripts/github/update
Folds the parallel Docker-image and git-ref version-update plumbing
into a single subsystem so the auto-update workflow, the external
freshness tests, and the unit tests all share one module layout.

utils/
  update/
    base.py        # semver primitives shared by both backends
    docker.py      # OCI registry lookups + version: line rewriter
    repository.py  # git ls-remote walker + ref: line rewriter

cli/contributing/update/
  docker/           # python -m cli.contributing.update.docker
  repository/       # python -m cli.contributing.update.repository

scripts/github/update/
  image_versions.sh
  repository_refs.sh

Tests mirror the same shape under tests/{unit,external}/update/{docker,repository}/.

Also widens the Docker bump scope to all roles (previously gated on
`web-*`) so non-web roles like sys-svc-container get bumped too;
pins Bookwyrm's python image at 3.11-bookworm via
`# nocheck: docker-version` with a TODO until upstream supports
Python >= 3.12.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 20:12:07 +02:00
kevinveenbirkenbachandClaude Opus 4.7 b1b73bc1df refactor(lint): tighten inline-script lint to 11 lines + extract overflows
Drops `MAX_LINES` from 12 to 11 in `test_inline_script_block_size.py`,
catching the five blocks that were sitting exactly on the previous
threshold. Pure-bash blocks move to dedicated `.sh` files invoked via
`ansible.builtin.script:` or `run: bash …`; the keycloak heredoc with a
templated JSON payload splits into a `set_fact` plus a folded
`printf | kcadm … -f -` one-liner.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 22:06:32 +02:00
kevinveenbirkenbachandClaude Opus 4.7 14700e0658 feat(test-e2e-playwright): add PLAYWRIGHT_KEEP_ALL artefact-retention toggle
Forwards `PLAYWRIGHT_KEEP_ALL` from host env / GitHub Actions repo var
through `TEST_E2E_PLAYWRIGHT_KEEP_ALL` (inventory-overridable) into the
container and flips trace / screenshot / video to `on` for every test
when truthy.

SPOT for the propagation chain lives in
`docs/contributing/actions/testing/playwright.md#artefact-retention-`;
CI Configuration and the agent iteration loops link to it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 22:06:31 +02:00
kevinveenbirkenbachandClaude Opus 4.7 2480cd1ce3 ci(deps): cover repo-root package.json + regenerate .dockerignore
Two follow-ups to 2986e8042 (introduces repo-root package.json with the
ESLint dev-deps and adds node_modules/ to .gitignore):

- `.github/dependabot.yml`: add `/` to the existing npm directories
  list. The new package.json was uncovered, so
  `tests/integration/workflows/test_dependabot_coverage.py` failed
  ("ecosystem: npm" required for /package.json).
- `.dockerignore`: regenerated by `make dockerignore` (the Makefile
  target that mirrors .gitignore into .dockerignore plus a trailing
  `.git`); now carries the new `node_modules/` line so build contexts
  don't ship the local install tree into images.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 14:55:04 +02:00
kevinveenbirkenbachandClaude Opus 4.7 0e877caafe build(lint/javascript): introduce ESLint + eslint-plugin-playwright
Wires JavaScript linting into the project for the first time, scoped to
the Playwright specs and persona helpers under `roles/**/files/**/*.js`.

Setup
-----
- `package.json` + `package-lock.json` pin eslint v9, eslint-plugin-
  playwright v2, @eslint/js, globals.
- `eslint.config.js` (flat config) loads `eslint:recommended` and
  `playwright/flat/recommended`. Browser globals are injected so
  `page.evaluate(() => window.foo)` callbacks lint cleanly. Project-
  idiomatic patterns are explicitly turned off with rationale: the
  service-gating contract (req 006/019) uses test.skip on disabled
  services; persona helpers branch on isVisible/isServiceEnabled before
  expect; some specs deliberately use waitForTimeout / networkidle for
  OIDC settle. `no-empty` allows the empty-catch idiom; the autofixer
  for `prefer-web-first-assertions` and `no-wait-for-navigation` is
  unsafe in this codebase (broken diff in dashboard.js), so both are
  off pending a hand-migration pass.
- `scripts/lint/javascript.sh` mirrors the markdown.sh shape (skip on
  missing, exit 127 with "make install-lint" hint).
- `scripts/lint/autoformat.sh` gets a `run_eslint` worker (treats rc≤1
  as OK, like markdownlint).
- `scripts/install/lint.sh` gets `ensure_eslint` (npm-local install in
  the repo root, matching the markdownlint pattern), a `javascript`
  group, and `install_javascript_tools` is wired into the `all` group.
- `Makefile` exposes `lint-javascript` and folds it into the aggregate
  `lint` target.
- `.github/workflows/lint-javascript.yml` follows the lint-makefile.yml
  shape (Node 20, npm cache, `npm ci`, `make lint-javascript`); the
  ci-orchestrator wires it into the lint pipeline + lint-gate `needs:`.
- `.gitignore` adds `node_modules/`.

Real lint findings fixed in this commit
---------------------------------------
- `roles/web-app-decidim/files/playwright.spec.js` was using
  `runBiberFlow` / `runAdminFlow` without ever requiring `./personas`
  (would have failed at runtime). Adds the require.
- Same file: two nutzlose `\\'csrf-token\\'` escapes in a double-quoted
  query selector — eslint flagged, fixed.
- `roles/web-app-moodle/files/playwright.spec.js`:
  `expect(await locator.count()).toBe(0)` →
  `await expect(locator).toHaveCount(0)` (Playwright auto-retrying
  assertion; safe `prefer-to-have-count` autofix).

Lint baseline: 0 errors, 285 warnings (mostly `no-unused-vars` and
`playwright/expect-expect` — addressable as follow-up cleanup, not
blocking).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 14:55:04 +02:00
kevinveenbirkenbachandClaude Opus 4.7 1f61653a91 chore(ci): drop tool suffix from lint-galaxy and lint-makefile job names
Tightens "🧹 Lint Galaxy (galaxy-importer)" → "🧹 Lint Galaxy" and
"🧹 Lint Makefile (mbake)" → "🧹 Lint Makefile" so the job titles
are uniform with the other lint-* jobs and don't lock the displayed
name to the underlying tool implementation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 14:55:03 +02:00
kevinveenbirkenbachandClaude Opus 4.7 e8f59e868d ci(lint-ansible): set PYTHONPATH=. when generating role include files
Follow-up to 971fda535: the new "Generate role include files" step
failed with `ModuleNotFoundError: No module named 'utils'` because
cli/build/include/__main__.py imports `utils.cache.yaml` and friends.
Locally scripts/meta/env/python.sh sets `PYTHONPATH:=.` via the
Makefile env layer, but the CI step invokes `bash scripts/setup.sh`
directly and bypasses that layer. Inline `PYTHONPATH=.` on the run
line; verified locally with
`PYTHONPATH=. python3 -c "from utils.cache.yaml import load_yaml"`
and `PYTHONPATH=. python3 -m cli.meta.categories.invokable -s "-"`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 02:46:08 +02:00
kevinveenbirkenbachandClaude Opus 4.7 971fda535a ci(lint-ansible): fix syntax-check and load-failure fatals
Two fixes for the lint-ansible CI job (failing run 25604499233):

1. Add `community.docker` to requirements/requirements.galaxy.yml so
   `ansible-galaxy collection install -r ...` pulls it in CI. The
   collection is referenced in 9+ files (web-app-discourse handlers,
   svc-db-mariadb/openldap, sys-svc-compose/container, web-app-keycloak/
   decidim) but was missing from the requirements list, so syntax-check
   could not resolve `community.docker.docker_container`.

2. Add a "Generate role include files" step to .github/workflows/
   lint-ansible.yml. `tasks/groups/*-roles.yml` are gitignored generated
   artefacts (per tasks/groups/.gitignore: `*-roles.yml`) referenced
   from tasks/stages/*.yml; without them ansible-lint reports a fatal
   load-failure on tasks/groups/desk-roles.yml. The new step runs
   `PYTHON=python3 bash scripts/setup.sh` (same path the local `make
   setup` target invokes); pyyaml added to the pip install line as
   that's the only non-stdlib dep the setup script needs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 02:31:16 +02:00
kevinveenbirkenbachandClaude Opus 4.7 256e027525 docs: drop stale requirement refs + reflow soft-wraps to one sentence per line
Removes 67 live references to docs/requirements/<NNN>-*.md files across
43 contributor docs, role READMEs, code comments, task files, and tests.
Requirement docs are initial-creation-only artefacts; live references
freeze a snapshot that rots once the per-feature SPOT (a contributor
doc, a code module, a lint test) takes over the live behaviour.

In the same pass, fixes documentation.md violations on the touched
files: drops em-dashes used as thought breaks, removes "SPOT for ..."
self-labels, strips historical phrasing ("the legacy", "previously",
"used to", "is no longer used", "matching the historical"), and
reflows soft-wrapped sentences so each sentence stays on one logical
line.

The `test_inventory_vars_file_spot.py` whitelist for the literal
inventory path drops the docs/requirements/003 entry and instead
skips docs/requirements/ entirely; future requirement docs that
mention the literal path are then ignored automatically.

Verified by tests/lint/repository/test_no_stale_requirement_refs.py
(green tree-wide).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 05:00:46 +02:00
kevinveenbirkenbachandClaude Opus 4.7 c52e4269bf chore(tooling): integrate mbake (Makefile linter/formatter) + working-tree drift sync
Makefile lint integration:
* scripts/install/lint.sh: replace the prior checkmake binary install with
  ensure_mbake (mbake is pip-installable, parses our Makefile correctly
  whereas checkmake bailed on .ONESHELL: / .SHELLFLAGS: / ifneq).
* scripts/lint/makefile.sh: new runner — `mbake format --check Makefile`
  reports formatting drift and missing .PHONY declarations.
* scripts/lint/autoformat.sh: add a two-pass `mbake format Makefile`
  (mbake's formatter is not strictly idempotent on the first pass).
* Makefile: split the multi-line `.PHONY:` block into 18 single-line
  `.PHONY:` declarations so mbake's parser can validate it; add the
  previously-missing `install-python-dev` and `build-dependency` PHONY
  targets; add `lint-makefile` aggregate-and-direct target.
* .pre-commit-config.yaml: new make-lint-makefile hook scoped to
  `^Makefile$`.
* .github/workflows/lint-makefile.yml: new CI workflow (pip-install mbake
  + make lint-makefile).
* .github/workflows/ci-orchestrator.yml: wire lint-makefile into
  lint-gate.needs.

Misc:
* .gitignore: add roles/importer_result.json (tooling artefact written by
  galaxy-importer when it runs in roles/).

Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 00:32:44 +02:00
kevinveenbirkenbachandClaude Opus 4.7 89b9a94525 ci(lint-ansible): install required Ansible collections before make lint-ansible
The lint-ansible workflow runs `ansible-playbook playbook.yml --syntax-check`
inside scripts/lint/ansible.sh, which needs every collection the playbook
references to resolve module names. Without them CI would fail with
"couldn't resolve module/action ...". Add an explicit
`ansible-galaxy collection install -r requirements/requirements.galaxy.yml`
step right after the pip install of ansible-core/ansible-lint so the
syntax-check can fully load community.general / hetzner.hcloud /
kewlfft.aur on a fresh runner.

lint-galaxy needs no parallel change — galaxy-importer in --legacy-role
mode treats unresolved modules as warnings, not fatal errors, so the
workflow can stay collection-less.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 23:50:23 +02:00
kevinveenbirkenbachandClaude Opus 4.7 79409ebf10 test(tooling): integrate markdownlint-cli2, ansible-lint, galaxy-importer + fix the resulting baseline
Tooling integration
* add .markdownlint-cli2.jsonc, .ansible-lint, .ansible-lint-ignore
  configs (permissive day-1: cosmetic rules disabled, semantic ones on)
* extend scripts/install/lint.sh with generic install_pip_pkg helper +
  ensure_ansible_lint / ensure_galaxy_importer / ensure_markdownlint_cli2
  ensure_* functions (pip / npm with sudo / --user / --break-system-packages
  fallbacks). install_ansible_tools now also pulls ansible-lint and
  galaxy-importer; new `markdown` install group.
* extend scripts/lint/ansible.sh to run ansible-lint after the existing
  syntax-check.
* add scripts/lint/markdown.sh and scripts/lint/galaxy.sh runners. The
  galaxy runner parallelizes via xargs (-P=$(nproc)), prints OK/FAIL per
  role, and treats `bare-name dependency` errors as success since our
  roles/ tree is not a Galaxy collection layout. All other Galaxy schema
  errors still fail.
* Makefile: new lint-markdown and lint-galaxy targets, both wired into
  `make lint`.
* CI: new workflows lint-markdown.yml and lint-galaxy.yml; lint-ansible.yml
  now installs ansible-core + ansible-lint; ci-orchestrator wires both
  new lint workflows into lint-gate.

Galaxy schema fixes
* shorten the canonical `galaxy_info.company` block from three lines to
  two ("Kevin Veen-Birkenbach\nhttps://www.veen.world", 45 chars) so it
  stays under Galaxy's hard 50-char limit. Updated everywhere: 229 role
  meta/main.yml plus _CANONICAL_COMPANY in
  tests/lint/repository/test_role_meta_main_galaxy_schema.py.
* shorten 19 over-long role descriptions to <=255 chars.
* add stub READMEs for sys-svc-compose-ca and sys-utils-service-loader.

Markdown baseline fixes
* markdownlint-cli2 --fix run repo-wide (formatting normalization
  across CHANGELOG, role READMEs, docs).
* manual MD041 fixes: prepend H1 in 15 files (CHANGELOG, LICENSE,
  TODO.md helpers, role README templates, etc.).
* MD025 / MD003 / MD001 / MD032 manual fixes for heading-style and
  list-spacing edge cases.
* role README content polish: "## Single Point of Truth (SPOT)" headings
  renamed to "## User data resolution" (D3 self-label rule).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 23:46:37 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a26bfc4e94 refactor(cli): reorg admin/contributing tree + per-folder help system
Module moves (with imports, paths, argv literals, scripts, docs, and
parents[N] formulas updated in lockstep):

- cli/deploy             -> cli/administration/deploy
- cli/validate/inventory -> cli/administration/inventory/validate
- cli/vault              -> cli/administration/secrets/vault
- cli/administration/inventory/encrypt -> cli/administration/secrets/seal
- cli/mirror             -> cli/contributing/mirror

The new cli/administration/secrets cluster collocates the whole-file
ansible-vault wrapper next to the inventory-aware per-field sealer.
The user-facing CLI subcommand paths follow the package layout, e.g.
"infinito administration deploy dedicated <inv>" and
"infinito administration inventory validate <inv>".

Help system redesign in cli/core/{discovery,help,app}.py:

- Discovery now distinguishes runnable commands (directories with
  __main__.py) from category folders (directories that only group
  further commands). Empty support packages are filtered out.
- Each category folder MAY carry a README.md; its H1 heading is used
  as the title and its first paragraph as the one-line description in
  the help output.
- "infinito" / "infinito --help" prints only the top-level entries
  (one level deep). "infinito <category>" navigates into a folder and
  shows its sub-folders and commands one level deep, with the README
  title as the page heading and the invocation as a dim subtitle.
- New "--tree [N]" flag walks the tree under the (optional) path
  prefix recursively. N caps the depth (1-based); omit N for an
  unbounded walk. The output uses Unicode tree connectors, an emoji
  per entry (folder for categories, gear for commands), a globally
  aligned description column, and multi-line wrapped descriptions
  with tree-continuity bars on continuation lines.
- Category and command listings use the full invocation
  ("infinito meta callorder") rather than the bare leaf name, so the
  output is copy-pasteable.

Convention doc + 17 CLI READMEs:

- New docs/contributing/artefact/files/cli/readme_md.md describes the
  H1+paragraph layout the help system reads, and how it differs from
  role/docs READMEs.
- README.md added for cli/{administration, build, contributing, meta}
  and their immediate sub-folders, plus cli/meta/roles/applications
  and its resolution sub-folder. Each follows the "title with trailing
  emoji + single descriptive paragraph" convention.

Mirror resolver/sync description strings (the help system reads them
from argparse --help output) added so they no longer surface as
"options:" in the global index. cli/mirror/cleanup already had a
description and is unchanged.

Fixes uncovered by the move:

- import unittest.mock added to four dedicated-deploy tests so
  unittest.mock.patch resolves under stdlib unittest discovery
  (Python 3.13 no longer surfaces unittest.mock through "import
  unittest" alone).
- parents[N] index in the new
  cli/administration/inventory/validate/__init__.py corrected from
  3 to 4 to match the deeper layout.
- test_paths.INVENTORY_VALIDATOR_PATH assertion and test_runner
  substring assertions updated to match the new validator path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 20:17:09 +02:00
kevinveenbirkenbach d239b2ab19 Merge branch 'main' into feature/web-app-fediwall 2026-05-06 18:33:32 +02:00
kevinveenbirkenbach c0891bd40a Merge branch 'main' into feature/web-app-fediwall 2026-05-06 00:02:15 +02:00
kevinveenbirkenbach 041510d193 Merge branch 'main' into feature/alpha-to-beta 2026-05-05 22:59:45 +02:00
kevinveenbirkenbachandClaude Opus 4.7 91fadbdcc5 feat(ci): diff-driven app whitelist for deploy tests
Narrow the test-deploy-{server,universal,workstation} matrix to roles
actually impacted by the branch's diff against origin/main, expanded
transitively over run_after + dependencies + shared services.

Why: full-matrix deploy CI on every PR is wasteful; explicit per-PR
whitelisting is error-prone. The resolver derives the affected set
automatically while remaining fail-safe.

Behaviour (in resolve_effective_whitelist.sh precedence order):
- whitelist input "__ALL__"  -> empty whitelist (force full deploy)
- whitelist input non-empty  -> verbatim
- whitelist input empty      -> diff vs origin/main
  - empty diff or any path outside roles/<role>/   -> __ALL__
  - all paths under roles/<role>/                  -> reverse closure

Fail-safe: any seed that is not modellable in CombinedResolver
(no application_id and not referenced by any role's run_after) and
any resolver runtime error MUST trigger a fall-back to __ALL__.
The closure never silently shrinks the deploy matrix.

test-deploy-local.yml is intentionally exempt (local/manual dispatch
keeps explicit whitelist semantics).

Documented in pipeline.md and workflows.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-05 22:59:45 +02:00
kevinveenbirkenbachandClaude Opus 4.7 9524c50928 feat(ci): diff-driven app whitelist for deploy tests
Narrow the test-deploy-{server,universal,workstation} matrix to roles
actually impacted by the branch's diff against origin/main, expanded
transitively over run_after + dependencies + shared services.

Why: full-matrix deploy CI on every PR is wasteful; explicit per-PR
whitelisting is error-prone. The resolver derives the affected set
automatically while remaining fail-safe.

Behaviour (in resolve_effective_whitelist.sh precedence order):
- whitelist input "__ALL__"  -> empty whitelist (force full deploy)
- whitelist input non-empty  -> verbatim
- whitelist input empty      -> diff vs origin/main
  - empty diff or any path outside roles/<role>/   -> __ALL__
  - all paths under roles/<role>/                  -> reverse closure

Fail-safe: any seed that is not modellable in CombinedResolver
(no application_id and not referenced by any role's run_after) and
any resolver runtime error MUST trigger a fall-back to __ALL__.
The closure never silently shrinks the deploy matrix.

test-deploy-local.yml is intentionally exempt (local/manual dispatch
keeps explicit whitelist semantics).

Documented in pipeline.md and workflows.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-05 22:49:41 +02:00
kevinveenbirkenbachandClaude Opus 4.7 f3845323f1 test(lint): cap inline literal script blocks at 12 logical lines + extract offenders
Add tests/lint/repository/test_inline_script_block_size.py: scans
roles/, tasks/, playbooks/ and .github/ YAML for literal block scalars
(`|` / `|-` / `|+`) following the keys `shell`, `command`, `script`,
`raw` (Ansible) and `run` (GitHub Actions). Counts logical lines —
trailing-`\` continuations collapse to one line, blank lines are
ignored — and fails when a block exceeds 12.

FOLDED scalars (`>` / `>-` / `>+`) are exempt: they are one shell
statement after YAML folding, even if they span many source lines for
readability.

The walk goes through `utils.cache.files.iter_project_files(...)` per
`docs/agents/action/testing.md` so the shared project-tree cache is
reused across the lint suite.

The fail message tells the author exactly how to fix: Ansible tasks
get the built-in `script:` module (collapses the task to one line);
GitHub Actions workflows extract into `scripts/github/<name>.sh` and
call it via `run: bash scripts/github/<name>.sh` (or
`_helpers/scripts/github/<name>.sh` when the workflow checks out the
repo into `_helpers/`).

Extract the pre-existing literal-block offenders so the new lint
ships green:

  Ansible (script: module, file under <role>/files/)
    web-app-friendica   sync_addon.sh
    web-app-fider       create_admin_user.sh
    web-app-fider       configure_oidc_provider.sh
    web-app-mediawiki   install_or_update.sh
    web-app-mediawiki   composer_install_extension.sh
    web-app-xwiki       debug_superadmin.sh

  GitHub Actions (scripts/github/)
    pick_distro.sh
    guard_release.sh
    resolve_release_inputs.sh
    disk_pressure_report.sh

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 20:42:44 +02:00
kevinveenbirkenbach f6aeb1480e Merge branch 'feature/package-cache-frontend' into feature/alpha-to-beta 2026-05-04 16:57:47 +02:00
kevinveenbirkenbachandClaude Opus 4.7 deaab3c736 chore(lint): sort .claude/settings.json + disable codeql java-kotlin
Two follow-up lint fixes after the bluesky bridge SPI deletion +
WebFetch domain auto-additions during the iteration session:

- Run `bash scripts/lint/sort_claude_settings.sh` (via `make exec`
  because the host sandbox denies writes to `.claude/settings.json`).
  Sorts `permissions.allow` (265 entries) — Claude Code had appended
  new WebFetch domain grants to the end as the iteration ran, which
  tripped `tests.lint.repository.test_claude_settings_sorted`.
- Comment out `language: java-kotlin` in
  `.github/workflows/security-codeql.yml`. The previous commit
  removed the only Java source in the repo (the unused
  `roles/web-app-bluesky/files/keycloak-bluesky-bridge/` SPI
  scaffold), and `tests.integration.workflows.test_security_coverage`
  fails when CodeQL has a language enabled with no matching files.
  Re-enable once a real Java component lands again.

`make test` now green (1833/1833). External URL test still green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 21:48:14 +02:00
kevinveenbirkenbach f9f3dfd10d test(environment): differentiate cache assertions, split helpers, verify DiD path
Replace the previous OR-based assertion in scripts/tests/environment
(which trivially passed because the Nexus log always grows from
health probes) with a richer set of helpers split across two files
under utils/ for SRP/SPOT.

scripts/tests/environment/utils/cache.sh (new):

* expected_system_proxy_for_runtime maps DEV_RUNTIME_IMAGE to the
  Nexus proxy that handles its system package manager (apt-debian,
  apt-ubuntu, yum-fedora, yum-rocky; empty for Manjaro/Arch where
  pacman is not cached).
* nexus_repo_requests counts only /repository/<name>/... entries in
  the Nexus request log, optionally scoped to one proxy. Filters out
  health probes, admin REST calls, and Web-UI traffic.
* registry_cache_flows counts every cache-flow status (HIT, MISS,
  REVALIDATED, UPDATING, STALE) so cold deploys also count.
* assert_caches_used fails with a per-cache breakdown when any of
  three independent conditions is violated: total Nexus repo
  traffic = 0, the runtime's expected system proxy has 0 cached
  requests, or the deploy itself produced 0 registry-cache flows.
* probe_caches actively fetches a known upstream resource through
  each cache (apt-debian/dists/bookworm/Release for the package
  cache, /ca.crt for the registry cache).
* probe_did_inner_build drives a throwaway docker build whose
  apt-get update runs against deb.debian.org. The build container
  is placed on the compose network and pinned via --add-host to the
  frontend IP, exactly like the runtime override that
  roles/sys-svc-compose/files/compose.py injects for app builds.
  After the build the test snapshots
  ``docker logs infinito-package-cache-frontend`` and asserts the
  frontend recorded new requests, proving DiD apt traffic flows
  through nginx -> Nexus apt-debian and not directly to upstream.
  DOCKER_BUILDKIT=0 because BuildKit refuses --network=<compose-net>.

scripts/tests/environment/utils/common.sh (renamed from lib.sh):
keeps the bootstrap, constants, inventory inspect, and
assert_http_status; cache helpers moved out so the file is
single-concern.

scripts/tests/environment/07_deploy_minimal.sh sources both files;
order is snapshot, deploy, probe_caches, probe_did_inner_build,
snapshot, assert_caches_used. Active probes contribute traffic that
the AND assertion verifies, so the assertion is robust on deploys
that do not naturally trigger inner-build apt traffic.

.github/workflows/test-environment.yml: propagate DEV_RUNTIME_IMAGE
into the inner ``docker run`` so per-distro probe target and proxy
expectation resolve correctly inside the orchestrator.

.gitignore + .dockerignore: also ignore .claude/settings.local.json,
which compose mounts as /dev/null and which the classic docker
builder would otherwise try (and fail) to read into the build
context.

Verified end-to-end with five iterations of
``make act-workflow ACT_WORKFLOW=.github/workflows/test-environment.yml
ACT_JOB=test-environment ACT_MATRIX='dev_runtime_image:debian:bookworm'``.
The final iteration reports +81 apt-debian, +235 registry-cache
flows, +13 frontend log lines from the DiD probe, and "[OK] both
cache stacks observed activity" plus "[OK] DiD inner-build flowed
through cache frontend".
2026-05-01 10:25:45 +02:00
kevinveenbirkenbachandClaude Opus 4.7 803ed8451d feat(req-013): variant A+ Bluesky OIDC integration via in-role login-broker
Promotes web-app-bluesky's OIDC integration to a working Keycloak SSO
flow that ends up in the official Bluesky social-app web client
WITHOUT exposing the synthesised PDS app-password to the browser.

* New `login-broker` Node sidecar under
  `roles/web-app-bluesky/files/login-broker/`. The broker sits behind
  `web-app-oauth2-proxy` and in front of the official
  `@bluesky-social/social-app` web client. On a user's first
  OIDC-authenticated visit it:
    1. Looks up the Keycloak user via the Admin REST API.
    2. Provisions a Bluesky PDS account via createAccount if no
       encrypted app-password is recorded yet, encrypts the
       synthesised app-password with AES-256-GCM, and stores the
       ciphertext as the `bluesky_app_password_enc` user attribute.
    3. Decrypts the password and exchanges it for a PDS session via
       createSession.
    4. Renders an HTML handoff page that drops the session JWTs
       into localStorage["BSKY_STORAGE"] (canonical social-app key)
       and redirects to /.
  For subsequent requests carrying the handoff cookie, the broker
  reverse-proxies to the social-app upstream.
* `social-app` is built from a pinned upstream tag (1.121.0) so the
  broker's BSKY_STORAGE schema stays in sync with the upstream
  persistence layer.
* `roles/web-app-bluesky/tasks/05_keycloak_admin_client.yml` adds a
  service-account Keycloak client for the broker (deterministic
  secret from the credentials vault, view-users + manage-users
  realm-management roles). kcadm output is filtered for the JVM
  cgroup-detection warnings on cgroup-v2 hosts.
* `roles/web-app-bluesky/tasks/01_core.yml` switches from
  sys-svc-compose to sys-stk-backend so the oauth2-proxy sidecar is
  auto-included when services.oauth2.enabled is true. A pre-task
  heals the oauth2-proxy bind-mount when a previous deploy left it
  as an empty directory.
* `meta/schema.yml` adds `bridge_encryption_key` (AES-256-GCM key)
  and `kc_admin_client_secret` (broker's KC service-account secret).
* `meta/services.yml` enables oauth2 with origin `bluesky-broker:8080`,
  whitelist for /xrpc/ and /.well-known/ so federation traffic to
  api.bluesky.<domain> stays open.
* `meta/variants.yml` adds the LDAP variant so the matrix-deploy
  exercises both auth-flow legs.
* `templates/extra_locations.conf.j2` uses the new
  NETWORK_PUBLIC_DNS_RESOLVERS SPOT and the on_off filter for the
  nginx resolver directive in front of the public ip.bsky.app /
  bsky.app upstreams.
* Documentation:
  - `roles/web-app-bluesky/README.md` rewritten per documentation.md
    and readme_md.md (software-focused description, current SSO
    flow, no em-dashes, canonical Credits format).
  - `roles/web-app-bluesky/files/login-broker/README.md` covers the
    handoff sequence, env contract, and failure modes.
  - `roles/web-app-bluesky/files/keycloak-bluesky-bridge/README.md`
    cleaned up per documentation.md.
  - `docs/requirements/013-alpha-to-beta-promotion.md` bluesky
    per-role-notes updated to reference the variant A+ broker; new
    "Future hardening" section tracks key rotation, an external
    secrets store, the Joomla `?fallback=local` IP allowlist, and
    the still-source-only Keycloak event-listener SPI.
* `.github/dependabot.yml` adds the npm ecosystem entry that covers
  the broker's package.json metadata stub.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 17:46:30 +02:00
kevinveenbirkenbachandClaude Opus 4.7 084e19202d feat(req-013): native OIDC for web-app-joomla via in-role plg_system_keycloak
Promotes web-app-joomla's OIDC integration from "documented future
work" to a working native plugin shipped inside the role.

* New `plg_system_keycloak` Joomla 6 plugin under
  `roles/web-app-joomla/files/joomla-oidc-plugin/`. The plugin's PHP
  dependency `jumbojett/openid-connect-php` is vendored at deploy time
  inside the Joomla container (Composer is baked into the role's
  custom image), then packaged as a ZIP and installed via the Joomla
  CLI. The repo working tree carries no `vendor/` tree.
* Modus 3 per requirement 013 (Force-Frontend, Local-Backup-Backend):
  every visit to `/` and `/administrator` redirects to Keycloak
  unless the request explicitly carries `?fallback=local` AND the
  env-var `JOOMLA_OIDC_FALLBACK_ENABLED` is `true`. The fallback hatch
  is env-toggleable so high-security deployments can flip to Modus 1
  (no local form).
* RBAC group mapping: the plugin maps the Keycloak `groups` claim
  onto Joomla's standard usergroup IDs:
    /roles/web-app-joomla/administrator -> Super Users (id 8)
    /roles/web-app-joomla/editor        -> Editor      (id 4)
    /roles/web-app-joomla               -> Registered  (id 2)
  Users whose Keycloak groups match none of these paths are refused.
* Matrix-deploy variants flip the auth-flow gate: variant 0 is the
  canonical OIDC variant (this commit's primary path), variant 1
  flips OIDC off and LDAP on so Joomla's core LDAP authentication
  plugin against svc-db-openldap is the exercised auth path.
* Role wiring:
  - `tasks/07_oidc_plugin.yml` builds + installs the plugin idempotently.
  - `templates/env.j2` propagates JOOMLA_OIDC_* env into the container.
  - `meta/services.yml` enables `oidc` with `fallback_enabled: true`.
  - `meta/rbac.yml` declares the `editor` role.
  - `templates/playwright.env.j2` keys `OIDC_SERVICE_ENABLED` off
    `services.oidc.enabled` for the Playwright test gate.
* Playwright e2e covers three scenarios:
  - native OIDC redirect from `/` to Keycloak and back to Joomla,
  - `/administrator?fallback=local` hatch bypassing Keycloak,
  - LDAP variant against the core Joomla LDAP plugin.
* Documentation:
  - `roles/web-app-joomla/README.md` rewritten per documentation.md
    and readme_md.md (software-focused description, no em-dashes,
    canonical Credits format).
  - `roles/web-app-joomla/files/joomla-oidc-plugin/README.md` covers
    plugin build, env contract, claimed routes, and watch points.
  - `docs/requirements/013-alpha-to-beta-promotion.md` joomla
    per-role-notes updated to reference the in-role plugin and the
    Modus 3 fallback contract.
* `.github/dependabot.yml` adds the composer ecosystem entry that
  covers the plugin's `composer.json`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 17:42:35 +02:00
kevinveenbirkenbachandClaude Opus 4.7 e1530c9912 test(docker): scope raw-docker guard to roles/ only
The wrapper convention applies inside Ansible roles. Bootstrap shell
scripts and CI workflows live outside `roles/` and either pre-date the
wrapper's installation or run on hosts that never see it; flagging them
adds noise without protecting the contract. Add a roles/ path-prefix
filter, rename the test method, drop the now-redundant `nocheck:
raw-docker` headers from the 6 GitHub workflows, and rewrite the two
suppression-doc examples on realistic in-scope scenarios (a per-line
marker on an Ansible task, a file-level marker on a role-bundled .sh).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 13:07:40 +02:00
kevinveenbirkenbachandClaude Opus 4.7 c54c40afa8 test(docker): migrate raw-docker guard onto unified suppression grammar
Drop the local `# no-raw-docker: skip` regex in favour of
utils.annotations.suppress.is_suppressed_at / is_suppressed_in_head
under rule key `raw-docker`, gaining per-line opt-outs (`# noqa:
raw-docker` / `# nocheck: raw-docker`) on top of the existing
file-level head suppression. Migrate the 6 GitHub-hosted-runner
workflow markers to the new grammar, lift the head scan to the catalog
default of 30 lines, and add the rule to the suppression catalog with
two examples.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 12:58:11 +02:00
kevinveenbirkenbachandClaude Opus 4.7 b99b2299ef test(docker): restore raw-docker guard with cache and skip marker
Reintroduce the no_raw_docker integration test that was lost in the
flat-to-clustered tests/integration reorg, scoped to .yml/.yaml/.j2/.sh
and powered by utils.cache.files.iter_non_ignored_files plus read_text
so file enumeration respects .gitignore and content reads are
lru-cached. Catch two argv-/inline-form blind spots that the original
single-line regex missed (argv: - docker / shell: "docker exec ...")
and add a `no-raw-docker: skip` header marker for files where the
container wrapper is genuinely unavailable.

Apply the marker to the 6 GitHub-hosted-runner workflows that cannot
reach the wrapper, and rewrite the 7 role-task argv invocations
(BigBlueButton, Discourse, Matomo, Joomla 03/04/06) onto
'container' so the engine stays swappable.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 12:50:36 +02:00
kevinveenbirkenbach 4e3ef819d4 Merge branch 'feature/package-cache-frontend' into feature/alpha-to-beta 2026-04-30 08:31:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 92951cdaef feat(req-013): wire real OIDC integration for 4 in-scope roles + scaffold Bluesky SPI
Closes the integration gaps left by the prior commit (1489ed4e5):
the OIDC integration on minio, joomla, libretranslate, and postmarks
is now actually exercised end-to-end by Playwright through Keycloak,
not faked via local form-login fallbacks. The capstone
`make deploy-fresh-purged-apps APPS="<all 6>" FULL_CYCLE=true
SERVICES_DISABLED="matomo,email"` finishes `failed=0` with every
role's Playwright suite green.

Per-role:

* MinIO (✅/✅/✅). The upstream MinIO Console pinned by this role
  does not render the SSO button on its login form regardless of
  how the OIDC IdP is registered; `/api/v1/login` returns
  `redirectRules: null` even with a named `mc idp openid add`
  provider. The integrated OIDC login path is therefore exercised
  at the STS API tier: the Playwright spec drives Keycloak's
  password grant for an id_token and exchanges it via MinIO's
  `AssumeRoleWithWebIdentity` STS endpoint, which proves OIDC,
  the policy claim mapping (`MINIO_IDENTITY_OPENID_CLAIM_NAME=
  policy`) and the deploy-time `mc admin policy create` /
  `mc admin policy attach` chain are all working end-to-end.

* Joomla (✅/✅/🛠️). Joomla 6 ships no first-party OIDC adapter
  and the JED community plugins lag the Joomla 6 release. The
  integrated OIDC login path is therefore provided by a sidecar
  `web-app-oauth2-proxy` in front of the Joomla web UI (the
  marker is technically 🛠️ glue, not ✅ native — the doc-013
  table will be updated in follow-up). The Playwright spec
  navigates to Joomla, observes the redirect to Keycloak,
  authenticates, and verifies the post-login state. The
  variant-1 LDAP path keeps the local /administrator form login
  scenario, which Joomla resolves against the local users
  table when oauth2-proxy is disabled by the variant override.

* LibreTranslate (🛠️/❌/❌). Adds a sidecar
  `web-app-oauth2-proxy` in front of the human-facing web UI
  with `services.oauth2.acl.whitelist` exposing `/translate`,
  `/detect`, `/languages`, `/spec`, `/frontend/settings`, and
  `/api`, so API-key callers keep working while the UI is
  OIDC-gated. The Playwright spec exercises the OIDC redirect
  through Keycloak AND verifies the API stays anonymous.

* Postmarks (🛠️/🛠️/❌). Adds a sidecar
  `web-app-oauth2-proxy` in front of the web UI. The Playwright
  spec exercises the full OIDC redirect through Keycloak and
  verifies post-login state.

* Bluesky (🛠️/🛠️/❌). The role now ships the source for the
  Keycloak event-listener SPI bridge that auto-provisions
  Bluesky PDS accounts on REGISTER / LOGIN events and stores
  the synthesised app-password back as a Keycloak user
  attribute (per the requirement-013 per-role notes):

    roles/web-app-bluesky/files/keycloak-bluesky-bridge/
      pom.xml
      README.md
      src/main/java/com/infinito/bluesky/
        BlueskyBridgeEventListener.java
        BlueskyBridgeProviderFactory.java
      src/main/resources/META-INF/services/
        org.keycloak.events.EventListenerProviderFactory

  The SPI implements handle sanitisation to the AT Protocol's
  allowed `[a-z0-9-]` charset, a 3-attempt retry budget for the
  PLC-bound provisioning step, and group-membership gating
  against `/roles/web-app-bluesky`. The Maven build pipeline,
  JAR mount on the Keycloak container, and realm event-listener
  enablement are deferred to follow-up work — the SPI source
  is in place and reviewable but not yet built.

* Bridgy-fed (❌/❌/❌). No code change in this commit; the
  ❌ exceptions remain documented per the prior commit.

Shared infrastructure / tooling:

* `.github/dependabot.yml`: enable Maven coverage for the new
  Bluesky SPI pom.xml.
* `.github/workflows/security-codeql.yml`: enable
  `java-kotlin` in the CodeQL matrix to scan the new Java source.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 08:14:09 +02:00
kevinveenbirkenbachandClaude Opus 4.7 c88d90677f ci(test-environment): cap caches and move bind paths to /mnt
The orchestrator at scripts/tests/environment/00_orchestrator.sh
unsets GITHUB_ACTIONS to simulate local dev, which flips
Profile.registry_cache_active() to True and activates the `cache`
compose profile (registry-cache + package-cache) even on hosted
runners. Without explicit caps, the half-of-free heuristic in
scripts/meta/env/cache/{registry,package}.sh would claim ~7g+~7g on
the runner's small / partition (~14 GB free after free-disk-space
step), competing with /home/runner/work and other ephemera.

Override INFINITO_REGISTRY_CACHE_*/INFINITO_PACKAGE_CACHE_* in the
test-environment.yml job env so:

- Bind paths land on /mnt/cache/... — shares the 74 GB ephemeral
  disk with /var/lib/docker (already at /mnt/docker) instead of
  stressing /.
- Caps sized for in-run post-prune re-pulls: rpardini=8g (fits one
  deploy's in-use image set), Nexus blobstore=4g (pip/npm/apt
  artefacts), heap=1g/direct=1g (OSS minimum is enough for a
  pull-through proxy). Leaves ~62 GB headroom on /mnt for Docker
  layer storage.

The cache.sh scripts gate every default with `if [[ -z … ]]` /
`: \"\${VAR:=…}\"`, so pre-set values bypass the heuristics; local
dev behaviour is unchanged.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-29 11:04:14 +02:00
kevinveenbirkenbach 07247b3552 Merge branch 'main' into feature/role-meta-refactor
# Conflicts:
#	roles/web-app-bridgy-fed/config/main.yml
#	roles/web-app-decidim/config/main.yml
#	roles/web-app-joomla/config/main.yml
#	roles/web-app-magento/config/main.yml
#	roles/web-app-shopware/config/main.yml
#	roles/web-app-suitecrm/config/main.yml
2026-04-27 10:16:52 +02:00
kevinveenbirkenbachandClaude Opus 4.7 ed04dad5d5 refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).

req-008 — role meta layout
  * Move every role's metadata under meta/<topic>.yml with file-root
    convention (no wrapping key matching the basename):
      - schema/main.yml + config/main.yml.credentials → meta/schema.yml
      - users/main.yml                                → meta/users.yml
      - config/main.yml.server                        → meta/server.yml
      - config/main.yml.rbac                          → meta/rbac.yml
      - config/main.yml.compose.services + every other top-level key
        of config/main.yml inlined under <entity>.<key>
                                                      → meta/services.yml
      - config/main.yml.compose.volumes               → meta/volumes.yml
  * Drop the legacy schema/, users/, config/ directories from every role.
  * meta/schema.yml: support nested credential keys, optional `default:`
    field (Jinja literal, not rendered/validated/regenerated), implicit
    `algorithm: plain`.
  * Materialised paths drop the `compose.` prefix:
      compose.services.<e>.<…> → services.<e>.<…>
      compose.volumes.<k>      → volumes.<k>
      <former-top-level>.<…>   → services.<entity_name>.<former-top-level>.<…>
  * Update every Python consumer (utils/cache, utils/manager, utils/applications,
    plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
    and task file to the new paths.
  * No legacy fallback. Repository-wide grep for `compose.services.` /
    `compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
    is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
  * Add tests/lint/repository/test_role_meta_layout.py guarding against the
    old paths returning, and an explicit lint guard that meta/variants.yml
    files do not carry a `compose:` wrapper.

req-009 — per-role networks/ports
  * Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
    dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
    dns_resolver}.
  * Delete group_vars/all/09_ports.yml entirely; ports now live on the
    service entity that exposes them at
    roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
  * Rename + int-cast every former top-level `services.<entity>.port`
    field to `services.<entity>.ports.inter` (single int).
  * Always-category-keyed maps for ports.local / ports.public; relay
    spans use {start, end} integers.
  * Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
    of truth for the per-<scope>.<category> band ranges, consumed by both
    the new CLI helpers and the lint test.
  * New CLI helpers (cli meta ports suggest, cli meta networks suggest)
    walk the live role tree gap-first then increment-fallback, with
    deterministic output and non-zero exit on capacity overflow.
  * cli create role consumes both helpers when scaffolding a new role.
  * Lint extensions: bare-int port shape rejection, host-bound port
    collision check (single ints + relay span integers, inter excluded),
    band-membership check with documented allow-list for the legacy
    web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.

req-010 — run_after / lifecycle migration
  * Move galaxy_info.run_after and galaxy_info.lifecycle out of
    meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
    lifecycle}; <primary_entity> is the value returned by
    get_entity_name(role_name).
  * For multi-entity roles whose primary entity has no compose service
    (web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
    gen roles), create a dedicated metadata-only top-level entry under
    <primary_entity>.
  * Introduce utils/roles/meta_lookup.py with get_role_run_after /
    get_role_lifecycle that resolve <primary_entity> once and read from
    meta/services.yml; degrade gracefully when meta/services.yml is
    absent OR the field is absent; raise MetaServicesShapeError when the
    file exists but is malformed.
  * Update every consumer (utils/service_registry, utils/invokable,
    cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
    lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
    cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
    init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
    use the helper.
  * Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
    values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
    run_after/lifecycle never on a non-primary entity; empty run_after
    lists rejected.

Also folded into this change set
  * services_disabler now writes `services.<svc>.{enabled,shared}` into
    host_vars instead of the old `compose.services.<svc>.{...}` path; the
    discover/find pair walks meta/services.yml and only treats entries
    carrying enabled/shared/provides as provider declarations (so
    metadata-only entries from req-010 do not create spurious provider
    rows).
  * service_registry skips `run_after` deps whose target was filtered
    out via SERVICES_DISABLED instead of failing the whole load.
  * cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
    consume the new applications.<role>.services.<svc>.image shape
    (mirror.md updated to match).
  * utils/domains/list.build_applications_from_roles now reads
    meta/server.yml directly (it had been silently returning {} since
    config/main.yml stopped existing).
  * utils/roles/resource_validation.filter_roles_by_min_storage reads
    meta/services.yml.<entity>.min_storage instead of the legacy
    compose.services path.
  * web-app-bluesky pre-clones the social-app build context BEFORE
    sys-svc-compose so the build context is on disk by the time
    sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
  * 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
    web-app-wordpress, web-app-keycloak) lose their `compose: services:`
    wrapper so deep-merge over meta/services.yml lands on the new
    file-root shape.

Validation
  * Unit tests:        1571 passed.
  * Lint tests:        50 passed.
  * Integration tests: 161 passed (1 skipped).
  * Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
    set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
    discourse,mattermost}.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 09:58:26 +02:00
kevinveenbirkenbachandClaude Opus 4.7 3e30694ee3 ci(updates): gate auto-update PRs via repo var
Add CI_ENABLE_AUTO_UPDATES (default unset/false). Workflow update
jobs and Dependabot PRs are skipped/auto-closed unless the variable
is set to true, so update noise is opt-in.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-26 16:16:58 +02:00
kevinveenbirkenbach 41b48f2385 ci(push): add update branch trigger 2026-04-26 15:42:19 +02:00
kevinveenbirkenbachandClaude Opus 4.7 3f2021b07d refactor: collapse INFINITO_DISTRO/INFINITO_CONTAINER to a single SPOT and align all consumers
Follow-up to d002796a5. The formula `infinito_nexus_${INFINITO_DISTRO}` is
now defined in exactly one place — `scripts/meta/env/defaults.sh:18`,
exported outside the load-once guard so it tracks INFINITO_DISTRO across
matrix iterations (caller re-sources defaults.sh after distro changes).
Every other site reads INFINITO_CONTAINER strictly from env; nothing
re-derives the formula, no lazy/colon-dash fallbacks, no per-call defaults.

Production code aligned:

- cli/deploy/development/common.py: add resolve_container() (strict env
  reader, raises SystemExit on unset). make_compose() calls it as an
  early-fail guard so a missing defaults.sh source surfaces here instead
  of as a confusing compose.yml interpolation error later.
- cli/deploy/development/deploy.py: stop re-deriving from compose.distro;
  use resolve_container().
- compose.yml: container_name reads ${INFINITO_CONTAINER:?...} strictly,
  so callers that bypass defaults.sh fail fast with a clear hint.
- scripts/meta/resolve/image/local.sh: drop ${INFINITO_DISTRO:-debian}
  fallback; require INFINITO_DISTRO via :? error.
- scripts/tests/deploy/ci/dedicated.sh: drop local _container fallback.
- scripts/tests/deploy/local/deploy/fresh-kept-app.sh: drop redundant
  ${INFINITO_DISTRO:-debian} reassignment after sourcing env/all.sh
  (defaults.sh already defaults the value).
- scripts/tests/deploy/local/deploy/fresh-purged-app.sh: drop
  target_container fallback; INFINITO_CONTAINER is already in env.
- scripts/tests/deploy/local/exec/container.sh: drop the lazy
  ${INFINITO_CONTAINER:=infinito_nexus_...} default; defaults.sh source
  in env/all.sh already guarantees the value.
- scripts/tests/dns.sh: source defaults.sh so callers only need to set
  INFINITO_DISTRO (the implicit dependency was previously a footgun).
- scripts/system/tls/trust/{wsl2,linux}.sh: same — source defaults.sh,
  drop the ${INFINITO_CONTAINER:-infinito_nexus_debian} fallback in
  linux.sh.
- .github/workflows/test-dns.yml: drop redundant
  INFINITO_CONTAINER: infinito_nexus_${{ matrix.distro }} from per-step
  env blocks (3 sites). The env now flows from INFINITO_DISTRO via the
  defaults.sh sourced inside the scripts.
- tests/unit/cli/deploy/development/test_matrix_deploy.py: replace the
  module-level os.environ.setdefault('INFINITO_CONTAINER', 'infinito_nexus_arch')
  shim — which itself encoded the formula — with a module-level patch of
  resolve_container that returns an opaque test-fixture string. Drop the
  now-unused distro="arch" field from the _args() Namespace fixture
  (deploy.handler stopped consuming it after the --distro arg removal).

Documentation aligned:

- docs/contributing/environment/variables.md: rewrite the INFINITO_CONTAINER
  row to spell out the SPOT contract — set automatically by defaults.sh,
  never re-derived elsewhere, callers re-source defaults.sh after
  changing INFINITO_DISTRO mid-process.
- docs/contributing/artefact/files/compose.yml.md: replace the
  INFINITO_DISTRO row (which described the now-removed
  `infinito_nexus_${INFINITO_DISTRO:-debian}` interpolation) with an
  INFINITO_CONTAINER row noting the strict ${INFINITO_CONTAINER:?...} read.

Verified:
  - Formula derivation: 1 production hit (the SPOT in defaults.sh:18).
  - INFINITO_CONTAINER fallback patterns (:-/:=): 0.
  - --distro CLI arg refs: 0 (one comment explaining the removal).
  - make test green: 1553 unit + 168 integration tests, no skipped suites.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-26 12:21:35 +02:00
kevinveenbirkenbachandClaude Opus 4.7 3466be99df fix(ci): use github.event.ref in delete-branch concurrency group
On `delete` events the workflow runs from the default-branch context, so
`github.ref_name` evaluated to "main" and the concurrency group collided
with entry-push-latest.yml's main group — cancelling the merge-commit
CI: Push run on main. `github.event.ref` carries the actually deleted
branch name, so the group matches the intended push-CI group.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-25 09:38:27 +02:00
kevinveenbirkenbachandClaude Opus 4.7 3f0b5481cc fix(ci): use github.event.ref in delete-branch concurrency group
On `delete` events the workflow runs from the default-branch context, so
`github.ref_name` evaluated to "main" and the concurrency group collided
with entry-push-latest.yml's main group — cancelling the merge-commit
CI: Push run on main. `github.event.ref` carries the actually deleted
branch name, so the group matches the intended push-CI group.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-24 23:18:21 +02:00
kevinveenbirkenbachandClaude Opus 4.7 8ce9179706 fix(lookups,web-app-peertube): finish users/applications migration and harden OIDC plugin install
- Resolve merge conflict in web-app-peertube/vars/main.yml; migrate
  remaining `applications |` and `users.*` references to
  `lookup('applications')` / `lookup('users', ...)` in web-app-odoo and
  web-app-decidim (compose, env, playwright, setup tasks).
- web-app-peertube: split OIDC tasks into oidc/{enable,disable}.yml,
  add install_mem_limit filter + PEERTUBE_MEM_LIMIT /
  PEERTUBE_OIDC_PLUGIN_INSTALL_MEM_LIMIT vars to raise the cgroup ceiling
  during the one-off `npm run plugin:install` exec (prevents OOM-SIGKILL
  that was surfacing as "Container is restarting"); add
  upsert-oidc-plugin.sql.j2 and diagnostics.sh.
- CI: test-deploy-{local,server,universal,workstation}.yml tweaks plus
  scripts/github/{dump_runner_state,enlarge_swap}.sh for constrained
  runners.
- cli/meta/applications/ressources: new CLI entry with unit tests.
- Misc config/docs updates across dashboard, mailu, mattermost, asset,
  cdn, file roles and contributing docs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-21 10:14:21 +02:00
kevinveenbirkenbachandClaude Opus 4.7 6fac8996ff Add runner-state diagnostic helper for CI deploy debugging
Extracts disk/memory/docker/swap dump into a reusable helper,
calls it pre/post swap resize, and runs it on deploy failure
across all four deploy workflows. Also adds SKIP_SWAP=1
short-circuit for A/B probing swap-related regressions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-20 12:37:54 +02:00
kevinveenbirkenbachandClaude Opus 4.7 8cdfcad9ef Enlarge CI runner swap dynamically across deploy workflows
Introduce scripts/github/enlarge_swap.sh that consumes all remaining
space on / minus a 25 GB buffer (Docker layers, checkout, runtime
state), falling back to /mnt when / is too small. Replaces the
default ~4 GB runner swap so transient memory spikes (e.g. peertube
plugin install, issue #162) don't trip the host OOM-killer on the
16 GB GitHub-hosted runner.

Wire the script into all four deploy test workflows (local, universal,
server, workstation) and flip free-disk-space to the aggressive config
(tool-cache, large-packages, docker-images, swap-storage all true):
the deploy actually runs inside the infinito container, so host-side
toolchains, language runtimes, and the default Docker layer cache are
dead weight. Update the workflow SPOT doc accordingly.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-20 03:18:45 +02:00
kevinveenbirkenbach b5c4b8a3d9 Merge branch 'main' into fix/users-applications.yml 2026-04-19 22:21:16 +02:00
kevinveenbirkenbach da140094a1 docs(agents): split iteration loops and refactor Playwright SPOT
Split docs/agents/action/iteration.md into role.md, playwright.md,
workflow.md, and a scope-focused README. Move Certificate Authority
and Inspect rules into role.md. Rewrite playwright.md as a machine-
oriented state machine with explicit preconditions, procedure, exit
gates, and escape path.

Extract Playwright spec content MUSTs into the new contrib SPOT
docs/contributing/artefact/files/role/playwright.specs.js.md. Reduce
the agent-side spec and env.j2 pages to procedure only. Align the
framework SPOT and the test-e2e-playwright role README with the
central templates/package.json.j2 + files/playwright.config.js model
and images.playwright.version as single source of truth.

Sweep em dashes and sentence-case role headings across all touched
docs per documentation.md and readme_md.md rules.
2026-04-19 22:04:24 +02:00
kevinveenbirkenbach 50e4cb8422 chore(group_vars/all): renumber files to close 04/05 gap
- Rename 06..19 → 04..17 so the sequence is dense and gap-free.
- Update hard-coded references in cli/, tests/, and PR template.
- Add README.md describing scope, load order, and naming rules
  per docs/contributing/documentation.md standards.
2026-04-19 15:55:52 +02:00
kevinveenbirkenbachandClaude Opus 4.7 85c008a90b chore(ci): add weekly CodeQL cron to backstop idle periods
Restores the weekly scan coverage previously provided by Default Setup,
so CodeQL analysis still runs when no CI triggers land on main.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-19 13:05:23 +02:00