Commit Graph
2780 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Fable 5 cc7662fe3a refactor(users): replace reserved flag with accounts + forward
Model change: per-user 'reserved: true|false' becomes 'accounts', a list of
host/mailbox/identity account types. Default when absent derives from roles
(bot/administrator -> [mailbox, identity], else [identity]) both for role-file
users and — via _apply_account_defaults after the inventory merge — for
inventory-only users, so users like biber keep their LDAP/Keycloak accounts.
Auto-generated role-suffix reservations get accounts: [].

New 'forward' field: mailu creates <username>@domain -> forward user's email
via the new 04b_manage_alias.yml (create-only; no_log on the loop so user
dicts never hit the log). Mailbox creation and password rotation now gate on
'mailbox' in accounts instead of role names; the mailu rbac role is renamed
mail-bot -> bot (token gate included).

svc-db-openldap drops the provision.reserved flag; OPENLDAP_USERS is always
the identity subset. reserved_usernames/non_reserved_users now classify by
'identity' in accounts, keeping the Keycloak deny-list semantics. root/backup
pin accounts: [host], administrator [host, mailbox, identity], helpdesk
[mailbox].

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-02 12:51:31 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0c57b055e4 feat(test/swarm): deploy bigbluebutton node-local (compose) in swarm clusters
BBB runs the external bigbluebutton/docker project (host networking,
node-local stateful stack) and cannot be a docker stack deploy service.
Add a per-role compose_mode override so it deploys via node-local
compose up on STACK_HOST even inside a swarm cluster:

- sys-svc-compose: compose_mode_force (per-role override) + computed
  compose_mode (SPOT) in defaults; up.yml + handlers route on
  compose_mode instead of DEPLOYMENT_MODE.
- container_address: honour compose_mode_force so a node-local role's
  exec targets its compose container (bare name), not a swarm service.
- web-app-bigbluebutton: compose_mode_force=compose, ungate compose
  generation for swarm, flush handlers in-scope, and add
  services.bigbluebutton.ports.internal.http for the swarm proxy upstream.
- web-app-matrix: add services.element.ports.internal.http (swarm proxy).
- test_variant_bundles: pin compose mode in the main test so it is not
  polluted by another test leaking INFINITO_DEPLOY_MODE=swarm.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-02 03:45:59 +02:00
kevinveenbirkenbachandClaude Opus 4.8 f4b293912f fix(ansible): drop {{ }} from folded conditionals + close two lint gaps
Three folded 'when:/failed_when: >- {{ ... }}' conditionals (mediawiki, joomla, sys-front-tls-selfsigned) triggered the ansible-core 'Conditionals should not be surrounded by templating delimiters' deprecation, which fails the deploy deprecation gate. Drop the delimiters (a folded conditional is already a bare Jinja expression).

Extend two lints so this cannot regress and does not misfire: test_no_templated_conditionals now catches folded/block scalars (was inline-only); test_vars_usage_in_yaml now scans failed_when/until/changed_when for var usage (was when/loop/with_* only), so bare-expression conditionals no longer make their vars look unused.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-02 03:45:59 +02:00
kevinveenbirkenbachandClaude Opus 4.8 72d472c35b fix(web-app-mastodon): generate a real linked VAPID keypair
Mastodon on the current mastodon/mastodon:latest (Ruby 4.0 / Rails 8.1.3) aborts at Zeitwerk eager_load_all (task: non-zero exit 1) without a usable VAPID key. The two keys were declared 'plain', so under CI allow-empty-plain they render empty and the web+sidekiq services never converge. Omitting the env vars (0fc5408db) was not enough: empty and absent both crash identically, so a valid key must be present. Add 'vapid_private'/'vapid_public' credential algorithms that emit one linked P-256 keypair (private = unpadded base64url 32-byte scalar, public = unpadded base64url uncompressed point 0x04||X||Y), wire the mastodon schema to them, and render both keys unconditionally.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-02 03:45:59 +02:00
kevinveenbirkenbach ffbd33d51f Ruff autoformat 2026-07-01 14:32:27 +02:00
kevinveenbirkenbachandClaude Opus 4.8 ccca758051 test(lint): ban templated conditionals in when/failed_when/changed_when/until
Catch the ansible-core 'conditionals surrounded by templating delimiters' deprecation at lint time. Excludes meta/{volumes,networks,services}.yml filter-when fields and templates; suppress with # nocheck: templated-conditional.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 14:28:42 +02:00
Alejandro Roman 9104b34795 add AV, Expose CalDAV/CardDAV/WebDAV and final testing refactoring 2026-07-01 13:04:15 +02:00
Alejandro Roman 233f15f3e0 fix OIDC and add more scenarios to tests 2026-07-01 11:19:49 +02:00
kevinveenbirkenbach 9b2f870777 Merge branch 'main' into feature/swarm-nfs 2026-07-01 05:37:36 +02:00
kevinveenbirkenbachandClaude Opus 4.8 9f833f3aae feat(swarm-test): force shared DB provider services in swarm inventory
Embedded per-app DBs (shared:false) resolve to mariadb-on-NFS with no placement pin in swarm, which InnoDB does not support (locking, split-brain); embedded DBs are a compose-only pattern. After provisioning each swarm-test round's inventory, force shared:true on every svc-db-* provider service so consumers route to the central provider.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 03:30:06 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e704e9a74d chore: autoformat + lint fixes; drop Invariant comment marker
autoformat reformats (settings sort, complexity/model.py, test_database_query.py, keycloak vars wrap); shorten akaunting swarm task name to <=80 chars; delete narration comments from keycloak vars; remove 'invariant' from the comment linter's valid marker set. make autoformat + make test verified green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 02:54:20 +02:00
kevinveenbirkenbachandClaude Opus 4.8 5e14c7e2d0 fix(database_query): read query_file on the controller via action plugin
database_query is a module: it opens query_file on the host it runs on. In swarm the task is delegated to the stack host, which has no repo checkout at role_path, so query_file is not found. A new action plugin (same name auto-wraps the module) reads query_file on the controller and passes the SQL as 'query', so every role's query_file: works in both compose and swarm with no call-site changes. Reverts the per-call-site fider workaround in favour of this root fix, which also covers mattermost, odoo, openproject and pixelfed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 01:27:46 +02:00
kevinveenbirkenbachandClaude Opus 4.8 337a0b2add feat(compose_replicas): respect explicit replicas override below 1
Floor-to-1 now applies only to the topology default (no override term). An explicit service_replicas=0 emits 'replicas: 0' (service created but unscheduled, to be scaled up by a later task) instead of being bumped to 1. Empty/missing inventory group on the default path still floors to 1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 01:08:43 +02:00
kevinveenbirkenbachandClaude Opus 4.8 f02da33128 fix(ci): clear compose deploy-gate deprecation warnings + tighten comment markers
The test-deploy-compose "Fail on deprecation warnings" gate failed ~10 apps
while the deploys themselves succeeded. Sources cleared:

- svc-db-postgres: community.postgresql `db` -> `login_db` (the `db` param
  alias is deprecated, removed in collection 5.0.0)
- assert_no_deprecation_warnings.sh: ignore ansible-core's legacy filter/test
  PluginLoader-aliases deprecation (needs a collection migration with
  meta/runtime.yml, not fixable in role YAML)
- joomla/wordpress: drop the `{{ }}` delimiters from task conditionals
  (when/until/changed_when), per ansible-core 2.23

Also drop `upstream`/`compat`/`security`/`bug` from the comment-policy markers
and delete those comments from the joomla install tasks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 22:48:26 +02:00
Alejandro Roman 9962429e59 Added playwright testing 2026-06-30 20:37:50 +02:00
kevinveenbirkenbachandClaude Opus 4.8 87c78275b8 feat(meta/complexity): multi-key sort, filter DSL, CI columns
Extend the 'meta roles applications complexity' report:

- --sort takes comma-separated multi-key clauses with per-key direction (e.g. 'desc embeds, asc weight'); replaces --order.
- --variant lists one row per meta/variants.yml variant; --format adds yaml.
- New columns: weight (was total), random, id, row, variants, bundles, jobs, lifecycle, covered_by, compose, swarm.
- covered_by marks the first green (1-based id) different-role predecessor that embeds the row; variant>0 never covered.
- bundles is the compose bundle/job count (whole-role) or 1 (per variant); jobs is the running sum.
- compose/swarm reflect the CI test-deploy skip logic (invokable + tested lifecycle + per-mode skip + swarm deployable variant).
- --filter is a boolean expression DSL (%% == \!= < > <= >=, and/or/xor/not, parens, set literals) over row fields.

Add shared SPOTs: variant_bundles.compose_bundle_counts and variant_status.deployable_variant_indices, reused by the swarm/compose matrix. Update roundtrip.sh/md to --sort "desc weight".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 20:33:01 +02:00
Alejandro Roman 2f693f00e6 feat(stalwart):Created the role and the connection like mailu had it 2026-06-30 16:17:48 +02:00
kevinveenbirkenbachandClaude Opus 4.8 3ee35c3c91 fix(ci): treat an absent mode job as N/A, not a failure
failed_roles for the swarm/docker scope only flags a role whose job in that mode actually ran and was not green; total_state is green when every mode that ran is green. A role with no job in a mode (a host driver that never deploys in swarm) is no longer counted as failed for that mode, so 'ci trigger --failed swarm' / 'ci status --failed swarm' stop listing drv-* and other roles that simply do not run in swarm.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 13:11:12 +02:00
kevinveenbirkenbachandClaude Opus 4.8 4187e1ebab feat(ci): one variant per swarm runner, skip all-disabled variants
In swarm mode (INFINITO_DEPLOY_MODE=swarm) variant_bundles emits one matrix entry per variant (bundle size 1) and drops variants whose meta/variants.yml override disables every service; the all-enabled variant already exercises the role so a bare all-off deploy adds no coverage. Absolute variant indices are preserved so the emitted index still maps to the deploy planner's round. Compose mode is unchanged. Adds the variant_disables_all_services predicate (utils.roles.applications.services.variant_status) plus unit coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 12:24:19 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6e4a62f754 fix(web-app-dashboard): make Simple Icons slugs ASCII-safe
slugify only stripped whitespace, so a card title carrying a non-ASCII
character (e.g. U+2011 non-breaking hyphen) produced a non-ASCII slug
that crashed the Simple Icons uri probe with 'ascii' codec can't encode
character. NFKD-normalize then drop non-ASCII; identity for ASCII titles
so existing slugs are unchanged. Adds a regression test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 12:24:18 +02:00
kevinveenbirkenbachandClaude Opus 4.8 8390cc4ce4 test(comments): lint .j2 templates and add file-level nocheck opt-out
Drop trap/warning from the exception markers; lint .j2 by stripping the suffix to the underlying language plus Jinja {# #} comments; add a head-level '# nocheck: comments-valid' opt-out via is_suppressed_in_head, documented in suppression.md; strip narration comments from friendica/nextcloud templates and opt the vendored discourse app.yml out with the new marker.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 12:24:18 +02:00
kevinveenbirkenbachandClaude Opus 4.8 926271478b fix(security): resolve all open code-scanning alerts and guard against recurrence
Resolve the 80 open CodeQL/code-scanning alerts on this branch.

Security-sensitive fixes:
- cli/administration/deploy/development: break the unsafe import cycle by
  extracting the env/arg helpers into a leaf module `env.py`, giving the
  load-time DAG compose -> common -> deps -> env.
- web-app-bluesky login-broker server.js: prototype-pollution guard on
  cookie parsing, CR/LF + control-char log sanitisation, HTML-escaped
  exception text, and a ReDoS-bounded handle regex.
- web-app-baserow SSO: harden the open-redirect `next` handling.
- network diagnose probes: require TLS >= 1.2.
- actionlint installer: validate tar members against path traversal.
- decidim/apt-purge lint regexes: rewrite to avoid catastrophic backtracking.
- 6x Playwright URL regexes anchored; crs-k/stale-branches pinned to a commit SHA.

Mechanical cleanups: unused imports/globals/locals, empty-except comments,
import-and-import-from, implicit-string-concat, multiple-definition,
unreachable/useless-assignment.

False positives are intentionally left in place (Django settings patches,
Odoo __manifest__, side-effect imports, cross-module symbols).

Recurrence guards:
- import-linter contract (pyproject + tests/lint) locks the dev-deploy
  package acyclic; a new cycle now fails CI instead of only CodeQL.
- new lint test requires every bandit (S###) `noqa` to carry a justification.
- BLE001 (blind-except) added to ruff as a ratchet (tests/roles exempt,
  current production offenders grandfathered).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 01:38:40 +02:00
kevinveenbirkenbachandClaude Opus 4.8 b6e79878fd fix(ci): make 🧩 Subset resolver comply with repo lint rules
The merged subset feature failed test-lint: a .py under scripts/, a local parents[N] PROJECT_ROOT, a direct yaml.safe_load, an uncached read, and narration comments.

- move scripts/meta/resolve/pr/subset_roles.py -> cli.meta.ci.subset_roles (scripts/ is .sh-only); dispatcher invokes it via python -m
- parse via utils.cache.yaml.load_yaml_str; import utils.PROJECT_ROOT
- test invokes the module via -m against real roles; nocheck the tempfile read
- strip narration comments; docs document HOW not WHY

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 00:59:49 +02:00
kevinveenbirkenbach 19b344d1dc Merge branch 'main' into feature/swarm-nfs 2026-06-30 00:59:49 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0d1e718abf feat(ci): add 🧩 Subset label for explicit role selection on PRs
The maintainer-only manual CI dispatch (workflow_dispatch) cannot be triggered by fork contributors. The 🧩 Subset label closes that gap: when present on a PR, the detect-affected-roles job resolves the deploy whitelist from an explicit roles: YAML block in the PR body instead of the diff, giving fork PRs the same narrowed matrix a maintainer would dispatch manually. Without the label, behavior is unchanged (the dispatcher falls through to the diff resolver, output byte-identical).

- subset_roles.py: parse and strictly validate the PR-body roles block (invalid YAML / empty / unknown id / path-traversal all hard-fail)
- pr_affected_roles.sh: single dispatcher selecting subset vs diff resolver
- entry-pull-request-change.yml: wire the label and PR body into the job
- PR templates, pipeline/workflows docs and READMEs, integration tests

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 00:59:48 +02:00
Alejandro Roman d832e88cec Merge remote-tracking branch 'upstream/main' into feature/self-host-runners 2026-06-30 00:14:59 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0bcd63b658 fix(ci): make 🧩 Subset resolver comply with repo lint rules
The merged subset feature failed test-lint: a .py under scripts/, a local parents[N] PROJECT_ROOT, a direct yaml.safe_load, an uncached read, and narration comments.

- move scripts/meta/resolve/pr/subset_roles.py -> cli.meta.ci.subset_roles (scripts/ is .sh-only); dispatcher invokes it via python -m
- parse via utils.cache.yaml.load_yaml_str; import utils.PROJECT_ROOT
- test invokes the module via -m against real roles; nocheck the tempfile read
- strip narration comments; docs document HOW not WHY

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-30 00:08:31 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6ad17c07de feat(ci): add 🧩 Subset label for explicit role selection on PRs
The maintainer-only manual CI dispatch (workflow_dispatch) cannot be triggered by fork contributors. The 🧩 Subset label closes that gap: when present on a PR, the detect-affected-roles job resolves the deploy whitelist from an explicit roles: YAML block in the PR body instead of the diff, giving fork PRs the same narrowed matrix a maintainer would dispatch manually. Without the label, behavior is unchanged (the dispatcher falls through to the diff resolver, output byte-identical).

- subset_roles.py: parse and strictly validate the PR-body roles block (invalid YAML / empty / unknown id / path-traversal all hard-fail)
- pr_affected_roles.sh: single dispatcher selecting subset vs diff resolver
- entry-pull-request-change.yml: wire the label and PR body into the job
- PR templates, pipeline/workflows docs and READMEs, integration tests

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 23:10:50 +02:00
kevinveenbirkenbachandClaude Opus 4.8 c1c5230302 Merge branch 'main' into feature/swarm-nfs
Conflicts resolved:
- .claude/.gitignore: union of ignore patterns
- plugins/lookup/rbac_group_path.py: drop both unused conflicting imports
  (body fetches apps via lookup_loader)
- roles/sys-svc-compose/handlers/main.yml: keep HEAD's refactor (compose-up
  handlers live in compose.yml, included per DEPLOYMENT_MODE)
- utils/cache/base.py: keep HEAD's merge_with_defaults/FILE_TOKENS imports

Post-merge fixes for main's newly-enabled lints:
- pyproject.toml: grandfather 5 files under the BLE001 ratchet
- justify 6 pre-existing security # noqa markers (S108/S311/S701)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 22:41:39 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e11bf6f31b refactor(lint): drop caveat/workaround comment markers
Remove caveat and workaround from the comment-lint marker vocabulary and delete
the comments that used them across the tree. Move test_comments_valid's own
git-safe.directory note from a marked comment into the function docstring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 21:54:10 +02:00
kevinveenbirkenbachandClaude Opus 4.8 7f23940436 test(lint): forbid whitespace around '=' in env.j2 templates
Docker env-files take everything after '=' literally, so cosmetic column
alignment (KEY=     value) leaks the spaces into the value (the cause of the
MariaDB "Access denied for user '     friendica'" failure). Add a lint scanning
every *env.j2 for whitespace before/after '=' or trailing whitespace, and strip
the existing offenders in espocrm, friendica, nextcloud and phpldapadmin.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 21:49:41 +02:00
Alejandro Roman b85729297b Merge remote-tracking branch 'upstream/main' into feature/self-host-runners
# Conflicts:
#	cli/administration/deploy/development/down.py
2026-06-29 21:16:40 +02:00
kevinveenbirkenbachandClaude Opus 4.8 5cce601eb2 style(tests): autoformat deprecation test files
ruff format line-wrapping only (the files were committed under --no-verify, so
the format hook had not run). No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 21:06:43 +02:00
kevinveenbirkenbachandClaude Opus 4.8 d95f8d8d22 fix(ansible): clear deploy deprecation warnings + gate against regressions
Remove the [DEPRECATION WARNING] entries the compose/swarm deploys emitted:

- apt_repository (removed in ansible-core 2.25): write docker.list via copy +
  an explicit apt update. deb822_repository is unusable here because
  apt_repo.sh's sanitize_sources deletes .sources every run.
- INJECT_FACTS_AS_VARS (default deprecated in 2.24): use ansible_facts['<name>']
  instead of the auto-injected top-level ansible_os_family / virtualization_* /
  default_ipv4 vars.
- community.mysql.* (removed in community.mysql 6.0.0): switch to ansible.mysql
  and pin the collection in requirements.galaxy.yml.

Add a compose deploy gate (assert_no_deprecation_warnings.sh + workflow step +
test) that fails on [DEPRECATION WARNING] in the deploy log, and a lint test
(test_no_deprecated_ansible_patterns) that forbids the three patterns above so
they cannot return.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 20:36:57 +02:00
kevinveenbirkenbachandClaude Opus 4.8 fbbbbc1234 feat(ci): exclude drv-* roles from swarm test-deploys
Driver roles configure the host (kernel modules, lid switch, vendor drivers)
and are meaningless as swarm services. Add `skip: [swarm]` to each drv-* role's
meta/services.yml so the swarm discovery (--skip-mode swarm) drops them; compose
deploys still run them.

Add tests/lint/ansible/roles/test_drv_roles_skip_swarm.py to enforce the policy
for the whole class: every drv-* role must list swarm in its skip list, so
future driver roles cannot silently enter the swarm matrix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 20:36:57 +02:00
kevinveenbirkenbachandClaude Opus 4.8 23d1447566 refactor(ansible): add path_absolute lookup, replace playbook_dir joins
Introduce plugins/lookup/path_absolute.py as the single source for the
repo-root anchor: lookup('path_absolute', 'roles/...') returns an absolute
path under playbook_dir, joining every term. Replace all hand-built
playbook_dir path constructions across .yml/.yaml and .j2 templates with it:

  [playbook_dir, '<rel>'] | path_join   -> lookup('path_absolute', '<rel>')
  playbook_dir ~ '/<rel>'               -> lookup('path_absolute', '<rel>')
  {{ playbook_dir }}/<rel>              -> lookup('path_absolute', '<rel>')

Unrelated repo_path symbols in cli/meta/roles/applications/resolution and the
single bare discover_playwright_roles(playbook_dir) call are left untouched.

Add tests/lint/ansible/test_playbook_dir_via_lookup.py: fails the build if any
of the three construction forms reappears in .yml/.yaml/.j2, pointing callers
at the lookup. Unit test covers the plugin.

Touching these files pulled their pre-existing narration comments into
test_comments_valid scope; clean them (delete restatement/banners, keep real
trip-wires as Workaround/Upstream/Invariant markers). Drop caution, important
and pitfall from the comment-lint marker vocabulary and remove the comments
that used them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 20:36:57 +02:00
kevinveenbirkenbachandClaude Opus 4.8 9413519e85 fix(ci): detect deploy failures by job glyph, not absent Compose/Swarm words
The --failed swarm/compose/total filters keyed the deploy-job regex on the
literal words "Compose"/"Swarm", which no real CI job carries. Job names are
"<glyph> <app> <variant>" (e.g. "... / test-deploy-swarm / 🐝 web-app-x 0,1"),
so the regex matched nothing and the CLI reported "Nothing failed" for runs
full of failures. app_for_job in the role-runtime path had the identical bug
(returned the shard token instead of the app id).

Match on the 🐳/🐝 glyph instead, tolerate the orchestrator caller prefix and
the " <variant>" shard suffix, and route both consumers through one parser
(runs.app_of_job). Centralize the naming as SPOTs: MODE_GLYPHS in runs.py and
ORCHESTRATOR_PREFIX plus a workflow-sourced deploy_job_name() helper in tests,
so fixtures render real names from the workflow YAML and cannot drift back to
job names GitHub never emits.

Add an integration test that renders names from test-deploy-{compose,swarm}.yml
and asserts the parser extracts app+mode for both modes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-29 20:36:56 +02:00
kevinveenbirkenbachandClaude Opus 4.8 18cf9b9ada fix(matomo): inject tracker into web-svc consumers on swarm re-injection
The matomo bootstrap re-injects early-rendered web-svc consumers (such as
web-svc-logout) after obtaining the admin token, but in swarm this never ran:
roles_with_service filters by the manager's group_names, which omits
worker-placed web-svc services, so the re-injection loop was empty and the
logout page shipped without the matomo tracker.

- roles_with_service: add a scope kwarg; scope='all' bypasses the per-host
  group_names filter (the merged applications config is already
  deployment-scoped, so iterating all of it equals the deployment's consumers).
- 05_bootstrap: enumerate consumers with scope='all', stat each compose project
  dir, and re-inject only deployed ones. Avoids a fatal chdir into an undeployed
  consumer's missing /opt/compose/<entity>/ that scope='all' would otherwise hit.
- sys-front-inj-matomo: resolve matomo_effective_token (override else
  lookup('users')); the disable-gate and the SitesManager getSitesId/addSite
  calls use it, so the re-injection passes the fresh bootstrap token and bypasses
  the play-start-stale controller-side users cache that left matomo disabled.

scope defaults to 'host' and the override defaults to empty, so every existing
caller and the per-consumer injection path are unchanged. Validated green in
both compose and swarm round 1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 16:37:56 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6eb1ce58e7 fix(lint): make comment-policy linter enforce in CI
test_comments_valid.py scopes to the working-tree diff via 'git diff --name-only HEAD', but the make-test container mounts the repo as a different uid, so git aborted with 'dubious ownership' and the linter silently scanned zero files. Pass '-c safe.directory=*' so it runs. Also drop 'gotcha' from the accepted exception markers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 16:37:55 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e55fb78f15 test(lint): comment-policy linter scoped to changed files
Add tests/lint/repository/test_comments_valid.py: across .py/.sh/.yml/.rb/
.php/.css, a comment is valid only as a file header, a doc comment directly
above a class/function/rule, a tool directive, or a mid-code EXCEPTION that
flags a real trip-wire (text starts with Warning/Exception/Caution/Gotcha/
Workaround/Security/Important/...). Plain narration, banners, commented-out
code and neutral notes are rejected.

Scope is the working-tree diff (staged + unstaged + untracked vs HEAD): the
legacy tree is grandfathered in and never breaks the build, while every file a
change touches must comply going forward. Python is parsed with tokenize
(string-aware); a multi-line trip-wire's marker on its first line validates the
whole block.

Also fold in autoformat normalization of 11 meta/{csp,domains,server}.yml files.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 22:45:06 +02:00
kevinveenbirkenbachandClaude Opus 4.8 cddb0dc59b feat(lint): require emoji-prefixed task/step/job names + convert existing
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.

- emit emoji names at the source: cli/build/include generator now writes
  "🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
  tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
  task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
  🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 22:45:06 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6ce6780575 refactor(meta): split server.yml into csp/domains/networks.yml
Extract the csp, domains, and networks blocks out of roles/*/meta/server.yml
into dedicated meta files, each mapping to its own top-level applications key
via the recursive meta-walk:

  csp:      -> meta/csp.yml      -> applications.<app>.csp
  domains:  -> meta/domains.yml  -> applications.<app>.domains
  networks: -> meta/networks.yml -> applications.<app>.networks

Residual nginx/proxy keys (status_codes, client_max_body_size, locations,
ip6_enabled, config_upstream_url) stay in server.yml; an emptied server.yml is
deleted. 98 server.yml -> 249 new files, 74 deleted.

- Consumers repointed applications.<app>.server.{csp,domains,networks} ->
  applications.<app>.{csp,domains,networks} across templates, vars, tasks,
  lookups and filters.
- Infra readers migrated: canonical_domains_map, application_domain_index,
  build_ldap_role_entries, service-registry overlay discovery,
  roles_with_service, and the file-based readers utils/domains/list.py +
  utils/meta/scan.py (read the now-deleted server.yml directly).
- Detection + mapping: _META_TOPICS gains csp/domains/networks;
  ROLE_FILE_META_{CSP,DOMAINS,NETWORKS} constants + ROLE_FILES entries.
- New self-expiring lint guard (test_server_meta_no_split_fields) fails if a
  server.yml still carries a split field; auto-fails from 2026-12-28 as a
  delete reminder.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 22:45:05 +02:00
kevinveenbirkenbachandClaude Opus 4.8 b9d86e176f test(lint): fail on invalid git branch names
Add tests/lint/repository/naming/test_branch_name.py validating the checked-out branch against the naming policy in branch.md. Rejects unknown prefixes (e.g. feat/ instead of feature/), non-kebab-case descriptions, and fix/ branches missing a ticket id; exempts main, master, and detached HEAD. Document the local enforcement in branch.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 21:13:46 +02:00
kevinveenbirkenbachandClaude Opus 4.8 238bc285c7 refactor(lookups): reach merged config via the lookup-loader SPOT
Consumer lookup plugins now reach the merged config through the Ansible
lookup loader instead of importing get_merged_applications /
get_merged_domains from utils.cache. The merge stays one cached
implementation surfaced via the applications/config/domains/users lookups;
consumers call lookup_loader.get(...).run() (the compose_networks idiom).

- get_merged_applications walks the whole meta/ tree recursively
  (meta/**/*.yml to applications.<app>.<path>, addons and any depth) and
  folds the raw meta/volumes.yml subtree onto applications.<app>.volumes
  after the templar render so the Jinja source: strings stay unrendered.
- 31 consumer lookups rerouted; providers (applications/config/
  applications_current_play/users/domains) and the raw-volume accessors
  (volume/nginx) keep the cache import behind # nocheck: lookup-cache-import.
- New lint test_no_cache_imports_in_lookups enforces the contract.
- nfs_flavor is now the flavor SPOT (svc-storage-nfs-server + nfs_client_src
  resolve via lookup('nfs_flavor')); removed the unused STORAGE_NFS_FLAVOR
  group_var. Extracted extract_default_credentials to
  utils/cache/schema_credentials.py to keep applications.py under the cap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 18:35:06 +02:00
kevinveenbirkenbachandClaude Opus 4.8 5e90c9cfd0 fix(nfs lookups): render via variables + read the applications var
self._templar.template('{{ var }}') no-ops on untrusted strings in ansible 2.19+, so the nfs_* lookups emitted literal Jinja into /etc/exports and exportfs failed (the fsid=0 export never came up). The base lookups now read resolved inputs from 'variables'. nfs_flavor/nfs_client_src read the flavor from the applications variable (the variant-baked config provisioning bakes under applications:), not a re-merge of the role files via get_merged_applications. Adds a real-Templar regression test that fails on the old code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 18:35:06 +02:00
kevinveenbirkenbachandClaude Opus 4.8 8cdf9d2f6d fix(web-app-dashboard): swarm simpleicons probe + tile iframe capability
Simpleicons: the reachability probe ran inside the controller-side Jinja filter, which has no route to the node's published port in swarm, so every card silently fell back to Font Awesome. Move the probe to a STACK_HOST-delegated uri_retry step and make the filter pure assembly. Compose output is byte-identical.

Tile reachability: add a per-card services.dashboard.iframe capability (default = enabled) so apps whose SPA top-redirects out of an embed (Keycloak admin console, iframe:false) open in a new tab; roles_with_service emits the flag into the target manifest and the spec branches on it. For iframe:false targets the spec also asserts the HTTP layer permits framing (no blocking X-Frame-Options, CSP frame-ancestors allows the dashboard) so the flag can never mask a real header block. Fix the Tab-button locator and reveal it from fullscreen before asserting.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 18:35:06 +02:00
kevinveenbirkenbachandClaude Opus 4.8 fc77d9d265 fix(security): resolve all open code-scanning alerts and guard against recurrence
Resolve the 80 open CodeQL/code-scanning alerts on this branch.

Security-sensitive fixes:
- cli/administration/deploy/development: break the unsafe import cycle by
  extracting the env/arg helpers into a leaf module `env.py`, giving the
  load-time DAG compose -> common -> deps -> env.
- web-app-bluesky login-broker server.js: prototype-pollution guard on
  cookie parsing, CR/LF + control-char log sanitisation, HTML-escaped
  exception text, and a ReDoS-bounded handle regex.
- web-app-baserow SSO: harden the open-redirect `next` handling.
- network diagnose probes: require TLS >= 1.2.
- actionlint installer: validate tar members against path traversal.
- decidim/apt-purge lint regexes: rewrite to avoid catastrophic backtracking.
- 6x Playwright URL regexes anchored; crs-k/stale-branches pinned to a commit SHA.

Mechanical cleanups: unused imports/globals/locals, empty-except comments,
import-and-import-from, implicit-string-concat, multiple-definition,
unreachable/useless-assignment.

False positives are intentionally left in place (Django settings patches,
Odoo __manifest__, side-effect imports, cross-module symbols).

Recurrence guards:
- import-linter contract (pyproject + tests/lint) locks the dev-deploy
  package acyclic; a new cycle now fails CI instead of only CodeQL.
- new lint test requires every bandit (S###) `noqa` to carry a justification.
- BLE001 (blind-except) added to ruff as a ratchet (tests/roles exempt,
  current production offenders grandfathered).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 15:42:03 +02:00
kevinveenbirkenbachandClaude Opus 4.8 9942dbdbcf fix(svc-storage-nfs-server): fsid=0 v4 root + nfs_* lookup SPOT
Root cause of the kernel-flavor swarm ENOENT: no export carried fsid=0, so the NFSv4 mount of the nested /srv/nfs/infinito-state sub-export had no pseudo-root anchor and the server returned NFS4ERR_NOENT on the PUTROOTFH+LOOKUP walk. crossmnt could not fix it (it only crosses a boundary mid-walk, never sets the starting filehandle). Kernel now exports the state dir itself at fsid=0 and clients mount the bare v4 root ':/' (zero LOOKUPs). ganesha keeps Pseudo=Path and v3 mounts the full path. The self-bind is kept so fsid=0 rides a real mount boundary (stable st_dev) under overlayfs DinD.

The NFS derivations move to per-value lookup plugins (nfs_flavor, nfs_client_src, nfs_fstype, nfs_mount_opts, nfs_state_path) backed by pure helpers in utils/storage/nfs.py with unit tests; group_vars STORAGE_NFS_* become thin lookup wrappers. The dev/act->ganesha flavor rule stays the single SPOT in the role config; nfs_flavor reads the merged value so client and server agree.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 15:27:53 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6559ca0b84 fix(web-app-prometheus): resolve swarm scrape targets + healthz via a SPOT lookup
In swarm the sibling exporters (alertmanager, cadvisor, node-exporter, blackbox) and the healthz backend are unreachable on the compose container_name / host.docker.internal / 127.0.0.1 addresses, so prometheus marked every target down and /healthz/ready returned 502. Add a scrape_target lookup emitting <host>:<port> per mode (swarm tasks.<entity>_<service>:<internal_port>, compose container_name/compose_host), reading the internal port from services.<svc>.ports.internal; declare those ports on the four exporters; route both healthz probes through the proxy_pass lookup via one set variable. Compose output stays byte-identical.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 15:27:53 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e9299d0871 fix(sys-svc-compose): skip 'compose up' when the project declares no services
A launcher-managed app (discourse) renders an empty services map, so 'docker compose up' fails with 'no service selected'. The wrapper now returns no command for an 'up' against a zero-service project (main() exits 0); a compose that fails to parse is not treated as serviceless, so real errors still surface.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 07:39:24 +02:00