The existing `rails r SiteSetting.force_https = true` exec runs in the
`run:` block, AFTER Discourse precompiles its assets. As a result the
generated stylesheets contain `@font-face { src: url(http://forum.../...) }`
references, which the CSP `font-src 'self' https://forum.../` blocks at
runtime, failing the Playwright `discourse enforces Content-Security-Policy`
test (CI runs 25387069690 / 25402722355).
Setting `DISCOURSE_FORCE_HTTPS` in the env section lets Discourse pick up
the value before asset precompilation, so font and asset URLs are baked
as https from the start.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Manual `browser.newContext()` + `try/finally context.close()` caused the
test to hang on close while the OpenCloud SPA kept the page alive with
periodic OIDC token-refresh and MutationObserver activity, exhausting
the 5-minute test timeout (CI run 25387069690). Switch to the default
`page` fixture, matching the administrator test; Playwright already
provides per-test context isolation and handles teardown.
Verified with full make deploy-fresh-purged-apps APPS=web-app-opencloud
FULL_CYCLE=true SERVICES_DISABLED=matomo,email — both passes green:
admin ~2.5s, biber ~1.5s.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both Ubuntu's universe package docker-compose-v2 and Docker upstream's
docker-compose-plugin ship /usr/libexec/docker/cli-plugins/docker-compose
without a Conflicts: marker between them. Any apt transaction on Ubuntu
26.04 (resolute) that selects both — for example bootstrap_docker_repo
followed by the local infinito-nexus deb pulling ansible — aborts at the
dpkg unpack phase.
The Ubuntu/Debian branch of install-cli.sh now drops a priority -1
preference for docker-compose-v2 (Ubuntu only) and purges any
pre-installed copy before the docker-compose-plugin install. apt then
picks the upstream package unambiguously and the conflict cannot
re-emerge through transitive deps later.
Verified via `make act-workflow` against the ubuntu:latest matrix slot
of test-environment.yml (1h28m, exit 0).
Tracked upstream: https://bugs.launchpad.net/ubuntu/+source/docker-compose-v2/+bug/2151249
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Tracks the realistic scope for a `web-app-kix` role: OAuth2-proxy in
front (Keycloak SSO with realm-level 2FA), header-trust auth into
KIX, LDAP as the user directory, SMTP for outbound, dashboard card,
universal-logout and a Playwright E2E for the gated login/logout
flow. Out-of-scope items (OpenProject/GitLab cross-link, XWiki
content embed, Nextcloud-backed attachments, ActivityPub) are listed
explicitly so future requirements can pick them up individually.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_no_raw_project_walk.py: AST-walks every
.py file under tests/ and flags call expressions matching the forbidden
shapes — `<x>.rglob(...)`, `os.walk(...)`, `glob.glob(...)` — including
their aliased imports. The lint blocks any NEW raw walk from sneaking
in; pre-existing offenders are migrated below.
Background
==========
Pytest runs every test in one process. A raw walk over the repo tree
re-iterates the same paths and re-stats the same inodes for every test
that needs them, so the cost scales linearly with the number of lint /
integration tests doing project scans. Routing every walk through
`utils.cache.files.iter_project_files(...)` (or
`iter_project_files_with_content(...)`, `iter_non_ignored_files(...)`,
`read_text(...)`) shares one filesystem walk + one read per file
across the whole pytest session. See docs/agents/action/testing.md.
Migration
=========
Refactored to the cached helpers (broad project-tree walks):
Lint
tests/lint/ansible/test_no_literal_no_log.py
tests/lint/ansible/test_no_sh_lc_pipefail.py
tests/lint/ansible/test_run_once_tags.py
tests/lint/ansible/test_service_shared_consistency.py
tests/lint/filesystem/test_python_shebangs.py
tests/lint/repository/test_folder_item_limit.py
tests/lint/repository/test_markdown_links.py
tests/lint/repository/test_role_meta_layout.py
tests/lint/repository/test_todo_transparency.py
Integration / external / unit
tests/integration/compose/test_no_raw_docker.py
tests/integration/config/test_non_empty.py
tests/integration/roles/includes/test_existence.py
tests/integration/vars/test_facts_not_overridden.py
tests/integration/workflows/test_dependabot_coverage.py
tests/external/repository/test_urls_reachable.py
tests/unit/cli/deploy/development/test_inventory_vars_file_spot.py
Tagged `# noqa: project-walk` (legacy; future commits migrate them
incrementally — the lint pinpoints each offender and the fail message
points at the cached helpers and explains when noqa is appropriate):
~30 further test files (small per-role globs, scripts/utils subtree
scans, dynamic Jinja-driven globs that need richer refactors).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Narrow the test-deploy-{server,universal,workstation} matrix to roles
actually impacted by the branch's diff against origin/main, expanded
transitively over run_after + dependencies + shared services.
Why: full-matrix deploy CI on every PR is wasteful; explicit per-PR
whitelisting is error-prone. The resolver derives the affected set
automatically while remaining fail-safe.
Behaviour (in resolve_effective_whitelist.sh precedence order):
- whitelist input "__ALL__" -> empty whitelist (force full deploy)
- whitelist input non-empty -> verbatim
- whitelist input empty -> diff vs origin/main
- empty diff or any path outside roles/<role>/ -> __ALL__
- all paths under roles/<role>/ -> reverse closure
Fail-safe: any seed that is not modellable in CombinedResolver
(no application_id and not referenced by any role's run_after) and
any resolver runtime error MUST trigger a fall-back to __ALL__.
The closure never silently shrinks the deploy matrix.
test-deploy-local.yml is intentionally exempt (local/manual dispatch
keeps explicit whitelist semantics).
Documented in pipeline.md and workflows.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Narrow the test-deploy-{server,universal,workstation} matrix to roles
actually impacted by the branch's diff against origin/main, expanded
transitively over run_after + dependencies + shared services.
Why: full-matrix deploy CI on every PR is wasteful; explicit per-PR
whitelisting is error-prone. The resolver derives the affected set
automatically while remaining fail-safe.
Behaviour (in resolve_effective_whitelist.sh precedence order):
- whitelist input "__ALL__" -> empty whitelist (force full deploy)
- whitelist input non-empty -> verbatim
- whitelist input empty -> diff vs origin/main
- empty diff or any path outside roles/<role>/ -> __ALL__
- all paths under roles/<role>/ -> reverse closure
Fail-safe: any seed that is not modellable in CombinedResolver
(no application_id and not referenced by any role's run_after) and
any resolver runtime error MUST trigger a fall-back to __ALL__.
The closure never silently shrinks the deploy matrix.
test-deploy-local.yml is intentionally exempt (local/manual dispatch
keeps explicit whitelist semantics).
Documented in pipeline.md and workflows.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Narrow the test-deploy-{server,universal,workstation} matrix to roles
actually impacted by the branch's diff against origin/main, expanded
transitively over run_after + dependencies + shared services.
Why: full-matrix deploy CI on every PR is wasteful; explicit per-PR
whitelisting is error-prone. The resolver derives the affected set
automatically while remaining fail-safe.
Behaviour (in resolve_effective_whitelist.sh precedence order):
- whitelist input "__ALL__" -> empty whitelist (force full deploy)
- whitelist input non-empty -> verbatim
- whitelist input empty -> diff vs origin/main
- empty diff or any path outside roles/<role>/ -> __ALL__
- all paths under roles/<role>/ -> reverse closure
Fail-safe: any seed that is not modellable in CombinedResolver
(no application_id and not referenced by any role's run_after) and
any resolver runtime error MUST trigger a fall-back to __ALL__.
The closure never silently shrinks the deploy matrix.
test-deploy-local.yml is intentionally exempt (local/manual dispatch
keeps explicit whitelist semantics).
Documented in pipeline.md and workflows.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Narrow the test-deploy-{server,universal,workstation} matrix to roles
actually impacted by the branch's diff against origin/main, expanded
transitively over run_after + dependencies + shared services.
Why: full-matrix deploy CI on every PR is wasteful; explicit per-PR
whitelisting is error-prone. The resolver derives the affected set
automatically while remaining fail-safe.
Behaviour (in resolve_effective_whitelist.sh precedence order):
- whitelist input "__ALL__" -> empty whitelist (force full deploy)
- whitelist input non-empty -> verbatim
- whitelist input empty -> diff vs origin/main
- empty diff or any path outside roles/<role>/ -> __ALL__
- all paths under roles/<role>/ -> reverse closure
Fail-safe: any seed that is not modellable in CombinedResolver
(no application_id and not referenced by any role's run_after) and
any resolver runtime error MUST trigger a fall-back to __ALL__.
The closure never silently shrinks the deploy matrix.
test-deploy-local.yml is intentionally exempt (local/manual dispatch
keeps explicit whitelist semantics).
Documented in pipeline.md and workflows.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The placeholder repo URL pointed at github.com/your-org/your-hugo-site
which 404s and fails the external URL reachability test. Switch to
git.example.com so the URL probe skips it as a reserved-host example.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The URL extractor truncates `…/stable<MAJOR><MINOR>/…` at the `<` and
treats the bare `…/direct/stable` as a live URL (HTTP 404). The
`api.github.com/repos/.../tags` reference uses a literal `...`
placeholder and 404s for the same reason. Both are illustrative, not
fetchable. Mark them with `<\!-- nocheck: url -->` per the suppression
catalog.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add tests/lint/repository/test_inline_script_block_size.py: scans
roles/, tasks/, playbooks/ and .github/ YAML for literal block scalars
(`|` / `|-` / `|+`) following the keys `shell`, `command`, `script`,
`raw` (Ansible) and `run` (GitHub Actions). Counts logical lines —
trailing-`\` continuations collapse to one line, blank lines are
ignored — and fails when a block exceeds 12.
FOLDED scalars (`>` / `>-` / `>+`) are exempt: they are one shell
statement after YAML folding, even if they span many source lines for
readability.
The walk goes through `utils.cache.files.iter_project_files(...)` per
`docs/agents/action/testing.md` so the shared project-tree cache is
reused across the lint suite.
The fail message tells the author exactly how to fix: Ansible tasks
get the built-in `script:` module (collapses the task to one line);
GitHub Actions workflows extract into `scripts/github/<name>.sh` and
call it via `run: bash scripts/github/<name>.sh` (or
`_helpers/scripts/github/<name>.sh` when the workflow checks out the
repo into `_helpers/`).
Extract the pre-existing literal-block offenders so the new lint
ships green:
Ansible (script: module, file under <role>/files/)
web-app-friendica sync_addon.sh
web-app-fider create_admin_user.sh
web-app-fider configure_oidc_provider.sh
web-app-mediawiki install_or_update.sh
web-app-mediawiki composer_install_extension.sh
web-app-xwiki debug_superadmin.sh
GitHub Actions (scripts/github/)
pick_distro.sh
guard_release.sh
resolve_release_inputs.sh
disk_pressure_report.sh
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New Ansible role rendering one Fediwall instance with one or more
configurable walls under https://fediwall.<DOMAIN_PRIMARY>/<slug>/. Each
wall's `wall-config.json` is forwarded VERBATIM from
meta/services.yml.fediwall.walls[*].config — the YAML there is the
single source of truth, the role's template applies no defaults. Two
matrix-deploy variants ship: variant 0 inherits the services.yml
default (one wall, polls all active Mastodon-API-compatible siblings via
the new `lookup('fediwall_active', ...)` plugin); variant 1 overrides
with two explicitly configured walls (default + mastodon-only) so the
per-wall server scoping is observable end-to-end.
End-to-end Playwright spec drives biber's first OIDC login on Mastodon
and ldapauth-mediated form login on Friendica through the same web UI a
human would use — biber's local accounts come into existence lazily on
first SSO login, exactly as in production. The cross-Fediverse test is
data-driven against each wall's `wall-config.json.servers`: a sibling's
post is visible iff its host is listed there.
Cross-cutting fixes that surfaced during the iteration ship alongside:
- utils/cache + utils/templating: type-preserving render path for
single-Jinja-expression scalars in `lookup('config', ...)`.
`_templar_render_best_effort` always coerced its output to `str`, so a
list-returning lookup like `{{ lookup('fediwall_active', 'domains') }}`
collapsed to its Python repr. `_render_scalar` now detects the
`{{ ... }}`-only shape and routes it through the new
`_templar_render_preserve_type` helper, which returns templar's
native value (list/dict/etc.) instead of stringifying.
- web-app-friendica/tasks/04_addons.yml: force a `disable` + `enable`
cycle for enabled addons. Friendica's `addon enable` only fires the
`_install` hooks on the disabled→enabled transition, so an
already-enabled-but-broken state (the addon shows up in `list
enabled` but its `_install`-registered hooks are missing from the
`hook` DB table) silently rejected every ldapauth login at runtime.
Cycling through disable+enable reliably re-registers the hooks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds tests/lint/repository/test_literal_protocol_with_lookup.py which
fails when a file hand-writes the protocol next to a lookup-derived
domain. The protocol must come from the same source as the domain so
TLS-on / TLS-off / self-signed / public-CA stays consistent across the
stack; the `tls` lookup plugin emits both at once via the
`url.base` / `protocols.web` / `protocols.websocket` keys.
Mechanics
- Three regex patterns cover the realistic shapes:
1. `https?://{{ ... lookup( ... }}` (Jinja interpolation)
2. `"https?://" ~ lookup(...)` (Jinja concat with lookup)
3. `"https?://" ~ ... canonical.<key>` (concat with canonical access)
- Internal docker-network upstreams that are genuinely plaintext
(`http://<container>:<port>`) are not detected because the regex
requires a `lookup(` or `canonical.` token; they would only need
the noqa marker if the upstream resolution itself uses one of those.
- Per-line opt-out: `# noqa: literal-protocol-lookup` (or `nocheck`).
- File reads route through `utils.cache.files.read_text` and the
per-file scan result is cached process-wide via `lru_cache`.
Repo cleanup
- web-app-discourse and web-app-littlejs CSP entries now use
`lookup('tls', '<role>', 'url.base')` instead of pinning `http://` /
`https://` next to `lookup('domain', ...)`.
- web-app-matrix and web-svc-cdn Playwright env templates compose the
base URL through the `tls` lookup so the test runner uses the same
protocol the live stack serves.
- web-app-oauth2-proxy carries a single `# noqa: literal-protocol-lookup`
on the upstream line; that one is intentionally plaintext because
oauth2-proxy talks to its sibling upstream over the internal docker
network with no TLS.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds tests/lint/repository/test_domain_primary_spot.py which fails when a
file constructs a hostname via a literal `<sub>.{{ DOMAIN_PRIMARY }}` outside
the role's `meta/server.yml` (or `meta/variants.yml`) `domains.canonical` /
`domains.aliases` blocks. Every other reference MUST flow through the
project's lookup plugins so the canonical/aliases entries stay the SPOT.
Mechanics
- Regex matches `<word>.{{ DOMAIN_PRIMARY }}` patterns; wildcard fragments
(`.{{ DOMAIN_PRIMARY }}`, `*.{{ DOMAIN_PRIMARY }}`) and bare
`{{ DOMAIN_PRIMARY }}` references are intentionally NOT flagged
- meta/server.yml and meta/variants.yml: a state-machine line scanner
exempts list items and dict entries inside `canonical:` / `aliases:`
blocks (handles compact, expanded, list, and dict YAML styles)
- README.md is exempt; docs/ and files/ subtrees inside roles are skipped
- Per-line opt-out via `# noqa: domain-spot` (or `# nocheck: domain-spot`)
- File reads route through utils.cache.files.read_text and the helpers
themselves are wrapped in lru_cache so repeated path lookups during
the traversal pay the parse cost once
Repo cleanup
- 11 role meta/server.yml files now reference Keycloak, Collabora,
OnlyOffice, Nextcloud, and their own canonical hostnames through
`lookup('domain', '<role>')` (and `lookup('config', '<role>',
'server.domains.canonical.<key>')` for the dict-canonical roles
Matrix and Bluesky) instead of duplicating the SPOT
- `roles/web-app-mattermost/tasks/main.yml` keeps a single
`# noqa: domain-spot` on the legacy domain cleanup task; the
hardcoded historical hostname must persist there to remove stale
on-disk nginx configs from the rename
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Iterates the role-local Playwright specs from a structural skeleton into
fully green end-to-end scenarios that exercise the SSO handoff for both
the administrator and the biber LDAP persona, against the actual
deployed stack, and adds optional meeting recording for OpenTalk.
OpenCloud (web-app-opencloud)
- Add Keycloak's URL to connect-src and frame-src in the role CSP via
lookup('tls', 'web-app-keycloak', 'url.base') so the SPA can hit the
OIDC discovery and authorize endpoints from the browser
- Switch to read-only LDAP semantics: PROXY_AUTOPROVISION_ACCOUNTS=false,
GRAPH_USERNAME_MATCH removed, GRAPH_ASSIGN_DEFAULT_USER_ROLE=true
- Set OC_LDAP_DISABLE_USER_MECHANISM=none so the user-lookup filter
stops referencing the OpenCloud-only `openclouduserenabled` attribute
that the central LDAP schema does not define
- Pin the image to opencloudeu/opencloud:4.0.6
- Spec primes localStorage forceAllowOldBrowser, bypasses the "browser
not supported" splash, submits via Enter, asserts the Files-view banner
OpenTalk (web-app-opentalk)
- Provision a dedicated confidential Keycloak client `opentalk-controller`
via kcadm so the controller can introspect access tokens; the shared
realm client stays public for the browser PKCE flow. The service
account is granted the realm-management view-users / query-users roles
for the Keycloak admin Web API user-search backend
- Inject an nginx vhost extension via webserver_extra_configuration that
routes /v1/, /signaling, and /livekit on open.talk.<DOMAIN> to the
controller and LiveKit upstreams; the default route stays on the
frontend
- Fix the frontend env vars to upstream names (OIDC_ISSUER, BASE_URL=""),
pin the frontend image to v2.8.1
- Spec races between auto-redirect and explicit "Sign in" CTA, submits
via Enter, asserts the post-login Home-link in the dashboard
- Add OpenTalk recorder (registry.opencode.de/opentalk/recorder v0.16.2)
plus RabbitMQ broker (rabbitmq:4.2.4); wire the controller to publish
recording jobs on the `opentalk_recorder` queue and the recorder to
consume them; provision a dedicated confidential `opentalk-recorder`
Keycloak client for the recorder service account
- Reference Keycloak and the cross-app domain through lookup('tls', ...)
and lookup('domain', ...) in meta/server.yml instead of literal
`auth.<DOMAIN>` / `open.cloud.<DOMAIN>` strings
Cross-cutting
- Rename canonical hostnames to the <vendor>.<category> pattern:
- web-app-opentalk: talk.<DOMAIN> -> open.talk.<DOMAIN>
- web-app-bigbluebutton: meet.<DOMAIN> -> bbb.talk.<DOMAIN>
- Restructure both READMEs to the per-role README contract; drop emojis
from headings; strip em dashes / history phrases / self-SPOT
declarations from IAM.md / LDAP.md / TODO.md
- Replace technical galaxy_info descriptions with user-facing copy so
the dashboard cards read clearly
Validated end-to-end via the Role Loop (deploy-fresh-kept-apps then
deploy-reuse-kept-apps with SERVICES_DISABLED=matomo,email): all
seven OpenTalk containers + the OpenCloud container come up healthy,
scripts/tests/e2e/rerun-spec.sh web-app-opencloud and
scripts/tests/e2e/rerun-spec.sh web-app-opentalk both finish with
`2 passed`, and `make test` is green (1604 + 162 tests).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reworks tests/lint/ansible/roles/test_compose_resource_limits.py from
warn-only-on-every-role to a strict failure scoped to actual
compose-service declarations:
1. Only roles that ship a `templates/compose.yml.j2` are scanned.
Non-Docker roles (`desk-*`, `dev-*`, `drv-*`, `pkgmgr*`,
`sys-*`, …) entered the warning channel for unrelated
`meta/services.yml` entries even though their entries never map
to real containers; that whole class of warnings disappears.
2. Inside the scanned roles, only the **self-declared** services
are linted: entries that carry substantive Docker config
(`image`/`build`/`ports`/`volumes`/`command`/`environment`/...)
OR a primary entity backed by a `files/Dockerfile` /
`templates/Dockerfile.j2`. Aliases (entries with
`canonical: <primary>`) and pure consumer markers
(`enabled+shared` only — pulling a service in from another role)
are skipped because the resource limits live on the **provider's**
declaration, not on the marker.
3. Failures are hard `self.fail` again so regressions block CI.
Concretely: `web-app-mailu/meta/services.yml.{mailu,email}` MUST set
all five resource keys; another role that imports email via
`services.email.enabled+shared: true` is a marker and is exempt.
Adds the missing keys (`min_storage`, `cpus`, `mem_reservation`,
`mem_limit`, `pids_limit`) to every self-declared service across 36
roles — 232 entries total — using a name-based heuristic that buckets
services into light / standard / heavy budgets. Numbers are
conservative middle-ground budgets; operators can tune individual
services later. Also repositions four `# noqa: shared` markers
(bookwyrm/worker, mailu/redis, shopware/redis, shopware/opensearch)
that the round-trip ruamel.yaml dump shifted into the wrong service
block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Marks each role's primary entity (`mastodon`, `friendica`, `pixelfed`)
as `shared: true; enabled: true` in meta/services.yml so the variant-
aware service registry resolves them. Consumer roles can now pull any
of the three in via `services.<entity>.enabled: true; shared: true`
(e.g. from a variant) without listing the role under
`meta/main.yml.dependencies` and without committing every deploy of
the consumer role to a heavyweight Fediverse pull-in.
To pass the service-role schema lint each tasks/main.yml now collapses
to a thin wrapper that includes a new tasks/01_core.yml under a
`run_once_<role_slug>` guard, and 01_core.yml ends with the role's
existing setup pipeline. The pre-existing step files are renumbered
to leave 01 free for the canonical entry point:
web-app-mastodon
00_overrides.yml -> 02_overrides.yml
01_setup.yml -> 03_setup.yml
02_wait.yml -> 04_wait.yml
03_cleanup.yml -> 05_cleanup.yml
04_administrator.yml -> 06_administrator.yml
web-app-friendica
01_ldap.yml -> 02_ldap.yml
02_patch_config.yml -> 03_patch_config.yml
03_addons.yml -> 04_addons.yml
web-app-pixelfed
00_bootstrap.yml -> 02_bootstrap.yml
01_administrator.yml -> 03_administrator.yml
02_oidc_mapping.yml -> 04_oidc_mapping.yml
Each role's 01_core.yml wires the renumbered files via include/import
exactly where the previous main.yml did. No behaviour change for
direct deploys (APPS=web-app-<role>) — load_app.yml already gates on
`run_once_<role>`, so the wrapper is invariant under the existing
direct-deploy path. Variant-driven service-registry pull-ins now work
end-to-end for these three roles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops the meta/main.yml meta-dependency from web-svc-legal on
web-svc-html and replaces it with the variant-aware service-registry
mechanism: web-svc-html marks its primary entity `shared: true;
enabled: false` so the registry resolves the `html` key, and
web-svc-legal declares `services.html.enabled: true; shared: true` in
its own meta/services.yml so the deploy planner pulls web-svc-html in
on demand instead of unconditionally on every web-svc-legal deploy.
To pass the service-role schema check this also splits
web-svc-html/tasks/main.yml into a thin wrapper that loads
tasks/01_core.yml under a `run_once_web_svc_html`-guard.
Adds tests/lint/ansible/roles/test_web_role_no_web_dependency.py: a
broad guard that fails if any `web-*` role declares another `web-*`
role in `meta/main.yml.dependencies`. The fix message points the
contributor at the service-registry pattern (`shared: true` + the
`run_once` schema follow-ups, plus the consumer-side
`services.<X>.enabled: true; shared: true`). The web-svc-legal /
web-svc-html cleanup above is exactly the migration this lint walks
new contributors through.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds tests/integration/roles/run_once/test_shared_primary_entity_run_once.py.
Walks every role that participates in the variant-aware service registry
(self-declared `shared: true` on the primary entity, declared via
`provides:`/aliases, OR consumer-activated through another role's
`services.<X>.enabled+shared`) and asserts that tasks/main.yml carries
a `run_once_<role_slug>` token.
Without that guard, sys-utils-service-loader can pull the role in twice
when both APPS= and a consumer's variant request it, triggering
duplicate execution / infinite recursion. The check is permissive about
the exact YAML shape (`include_tasks: 01_core.yml … when: run_once_…`,
a `block: … when: run_once_…`, or any other arrangement carrying the
literal token) so it complements the stricter
tests/lint/ansible/roles/test_service_core_first_task_run_once.py
schema lint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stops every direct `yaml.safe_load`/`safe_load_all`/`safe_dump`/`dump`
call across the test corpus from re-parsing files the production cache
already holds. Each migrated test loads role meta, group_vars, inventory
fixtures, etc. through the shared cache, so a single `make test` run
parses every YAML file at most once across all suites.
Cache module changes (utils/cache/yaml.py):
- Unify single- and multi-doc cache: one `_CACHE: Dict[sig, list]`
holding the `safe_load_all` payload. Single-doc helpers
(load_yaml / load_yaml_any) unwrap docs[0] and raise on >1
document, preserving the old strict semantics. A file consumed
both as single-doc and multi-doc is now parsed exactly once.
- Add load_yaml_all (path-cached) and load_yaml_all_str (string,
no cache) so multi-doc consumers no longer need a direct
`yaml.safe_load_all` plus a separate file read.
- dump_yaml now accepts any safe-dumpable structure (dict, list,
scalar) so test fixtures writing list-of-tasks YAML can use it.
Lint guard (tests/lint/repository/test_no_direct_yaml_calls.py):
- Extend the scanned tree to include `tests/` so future test code
is forced through the cache, with the existing
`# noqa: direct-yaml` per-call opt-out for genuine raw-yaml
fixture needs.
Test corpus:
- 89 test files migrated to load_yaml / load_yaml_any /
load_yaml_str / load_yaml_all / load_yaml_all_str / dump_yaml /
dump_yaml_str. No `# noqa` shortcuts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements requirement 015. Both new roles join the central Keycloak
realm and consume the central OpenLDAP through the same patterns as
web-app-nextcloud, so users can move between Files (OpenCloud) and
Video (OpenTalk) without separate accounts.
OpenCloud (web-app-opencloud)
- Single-binary deploy of opencloudeu/opencloud:4.0.6 at open.cloud.<DOMAIN>
- External Keycloak as IdP (OC_EXCLUDE_RUN_SERVICES=idp,
PROXY_AUTOPROVISION_ACCOUNTS=true, PROXY_ROLE_ASSIGNMENT_DRIVER=oidc)
- Direct OpenLDAP user/group source via OC_LDAP_*
- Co-deployable with web-app-nextcloud (next.cloud.<DOMAIN> stays put)
- WEB_OPTION_OPENTALK_URL surfaces the OpenTalk meeting launcher into
the Files UI when web-app-opentalk is in scope
OpenTalk (web-app-opentalk)
- Controller + LiveKit + web-frontend + dedicated Redis + dedicated MinIO
(controller refuses to start without an S3 backend) at talk.<DOMAIN>
- Uses shared svc-db-postgres for relational data and shared
web-svc-coturn for STUN/TURN (no dedicated TURN provisioning)
- Native Keycloak integration via controller.toml [oidc.*] +
[user_search] backend = "keycloak_webapi"; LDAP is consumed
transitively through Keycloak's federation
- Frontend env exposes SHARED_FOLDER_OPENCLOUD_URL when web-app-opencloud
is in scope, wiring the in-meeting OpenCloud file picker
- Bucket bootstrap is idempotent: an Ansible task waits for MinIO and
creates the configured bucket via mc
Cross-cutting
- Extends PORT_BANDS public.relay to 65535 so LiveKit's media UDP range
(60000-60999) can coexist with web-svc-coturn (20000-39999),
web-app-bigbluebutton (40000-49999), and web-app-nextcloud's onboard
TURN (50000-59999)
- All credentials (OIDC client secrets, LDAP bind, TURN auth_secret,
MinIO root, LiveKit API, OpenCloud JWT/transfer) are sourced via
lookup('config', ..., 'credentials.*'); none are committed in plain text
- Each role ships IAM/LDAP docs mirroring web-app-nextcloud and
role-local Playwright specs that cover the administrator AND biber
personas through OIDC
Validated end-to-end via the Role Loop (deploy-fresh-purged-apps then
deploy-reuse-kept-apps with SERVICES_DISABLED=matomo,email): all five
OpenTalk containers + the OpenCloud container come up cleanly, the
controller successfully completes OIDC discovery against Keycloak, and
both canonical hostnames return HTTP 200 through the central reverse
proxy. Browser-validated assertions (admin role mapping, full SSO logout
propagation, OpenCloud<->OpenTalk cross-launch from a real LDAP user)
remain as TODOs in each role's TODO.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements requirement 016: a `web-app-hugo` role that builds and
serves Hugo static sites via a multi-stage Dockerfile (extended Hugo
builder + nginx serve), modeled on `web-app-littlejs` (no DB, single
canonical domain, sys-stk-full + docker_git_repository_* for the
upstream content clone).
Default content source is the official Hugo documentation
(`gohugoio/hugoDocs` pinned to tag `v0.148.0`, commit
36e3d7b7b521a165bdbf8f63cd417720f37832c6) so a fresh deploy renders
a real Hugo site end-to-end. Content repo and ref are
inventory-overridable via `services.hugo.source_repository` /
`services.hugo.source_version`; build env (`hugo_environment`,
`hugo_minify`) are overridable too. `--baseURL` is derived from
`lookup('tls', application_id, 'url.base')`.
Validated end-to-end on the local stack:
- `make deploy-fresh-purged-apps APPS=web-app-hugo FULL_CYCLE=true`
succeeds (`failed=0`).
- Container `infinito-hugo` reports `Up (healthy)` on the wget
healthcheck.
- `https://hugo.infinito.example/` serves the rendered hugoDocs
homepage with HSTS + CSP headers.
- All three Playwright assertions (front-page reachability + HSTS,
CSP header present, non-empty title with Hugo content sentinel)
pass under the deploy-time `test-e2e-playwright` runner.
- Second `deploy-reuse-kept-apps` is idempotent: git pull reports
`changed: false`, no compose-build / compose-up handler fires.
V1 explicitly supports a single canonical domain — asserted in
`tasks/01_core.yml`. Multi-canonical-domain support is out of scope
and deferred to a follow-up. Theme is whatever the upstream content
repo bundles (themes/ or go.mod / Hugo Modules); a separate theme
override is also a follow-up.
The Dockerfile installs Node.js + npm in the builder stage and runs
`npm install` when the upstream repo ships a `package.json` —
hugoDocs uses Hugo's JS pipeline with `import 'alpinejs'`, which
requires `node_modules/`. Skipped silently when no `package.json` is
present, so a custom site without JS deps builds unchanged.
Allocations: local HTTP port 8008 (via
`cli meta ports suggest --scope local --category http`); local
subnet 192.168.102.240/28 (via `cli meta networks suggest --clients 8`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Pin OpenCloud to opencloudeu/opencloud image (Heinlein fork)
- Note OpenCloud uses embedded KV store, no external SQL backend
- Pick LiveKit as OpenTalk signaling backend (upstream default)
- Reuse svc-db-postgres for OpenTalk and web-svc-coturn for STUN/TURN
- Set canonical hostnames: open.cloud.* and talk.*; keep next.cloud.* for Nextcloud (parallel operation)
- Reference svc-db-openldap explicitly and use application_administrators LDAP group per req 004
- Require both roles enabled by default (not in SERVICES_DISABLED)
- Add TURN shared secret to no-plaintext list
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The project's `config` lookup (plugins/lookup/config.py) is strict by
default: a 2-term call `lookup('config', app, path)` raises AnsibleError
on a missing key. The Jinja `| default(...)` filter only applies to
Jinja `Undefined` values, which the lookup never produces — it either
returns a value or raises. Every `lookup('config', a, b) | default(X)`
in the tree is therefore dead code; the operator-visible behaviour is
"crash on missing key" no matter what `X` is.
Add a line-local lint that paren-walks each `lookup('config', ...)`
call to count its top-level args. When the call has exactly 2 commas
(i.e. no plugin-level default) and is followed by `| default(...)`
later on the same line, the test fails with the recommended fix:
pass the fallback as the 3rd lookup argument so the strict-mode
short-circuit is disabled and the default actually applies.
Fix the two existing occurrences:
- roles/web-app-moodle/vars/main.yml:85 (services.ldap.enabled)
- roles/web-app-matrix/templates/compose.yml.j2:86
(credentials.chatgpt_bridge_access_token)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The CI image-mirror job iterates `image`+`version` pairs in
roles/**/meta/services.yml and tries to mirror each to GHCR via skopeo.
The previous layout used `version: stable405` (Moodle Git branch,
naming stable<MAJOR>0<MINOR>) paired with `image: php`, which made the
mirror attempt `docker.io/library/php:stable405` — a non-existent tag.
skopeo aborted with `manifest unknown`, exit 1, breaking the run.
Swap so `image`+`version` carry the real upstream Docker pull ref the
Dockerfile builds FROM (php:8.3-fpm) and move the Moodle Git branch to
its own `branch` key. This both fixes the mirror failure and lets the
PHP base get cached on GHCR alongside other mirrored bases.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Specifies new web-app-opencloud and web-app-opentalk roles that
integrate with the central Keycloak (OIDC) and OpenLDAP services
analogously to web-app-nextcloud, including SSO, LDAP-driven
auto-provisioning, group-based admin mapping, OpenCloud<->OpenTalk
file/meeting interop, propagated logout, and Playwright coverage.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Introduces requirement 016 for a `web-app-hugo` role that builds and
serves Hugo static sites — one per canonical domain — with a build /
serve split (one-shot Hugo builder writing to a named volume, nginx
serving the result).
Default content source is `gohugoio/hugoDocs` pinned to tag `v0.148.0`
(commit 36e3d7b7b521a165bdbf8f63cd417720f37832c6) so an out-of-the-box
deploy validates the full pipeline against a real, non-trivial Hugo
project. `source.repo` / `source.ref` are inventory-overridable.
Numbered 016 because 013 (alpha-to-beta promotion), 014 (svc-runner
CI), and 015 (Moodle self-built image) are already reserved on other
branches.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`oidc_login_auto_redirect=true` is the happy path on /login: the
pulsejet/nextcloud-oidc-login boot hook bounces straight to Keycloak.
Under cold first-deploy load that hook can lose its `header('Location:...')`
when an earlier hook has already emitted output, and /login renders the
NC chrome with the password form hidden but the OIDC alt-login button
(`<a href="/apps/oidc_login/oidc" class="oidc-button">`) visible.
Treat that button as a valid entry point so the spec stays green
regardless of whether auto-redirect actually fires. The existing
`kind === "social-login"` branch in `loginToStandaloneNextcloud` already
clicks through to Keycloak and waits for the credential form, so adding
the candidates is the only required wiring.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous `production-fpm-alpine` floated to whatever Docker Hub tagged
production at deploy time, which produced non-reproducible Playwright runs in
CI: the same commit deployed against a slightly different NC build raced the
`pulsejet/nextcloud-oidc-login` boot-hook against earlier hooks emitting output,
losing the `header('Location:...')` and serving 200 on /login instead of the
expected 302 to Keycloak. Pin to the 33 major so cache, plugin compatibility,
and login flow are reproducible across host and CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two failures surfaced once the metadata-plugin fix stopped aborting
the Deploy web-app-nextcloud CI job before the Playwright suite ran:
✘ nextcloud talk admin settings (3 retries, 53s each)
✘ dashboard to nextcloud login (3 retries, 60s each)
Both failures funneled through `clickAllTalkTestServerButtonsAndVerify`
expecting the regex
`/OK:\s*Running version:\s*\S+[\s\S]*?OK:\s*Running version:\s*\S+/i`
to match `document.body.innerText` — i.e. two consecutive
`"OK: Running version: <ver>"` markers. The diagnostic from a fresh
deploy showed the actual rendered Talk-admin state:
Warning: Running version: 2.0.2~docker; Server does not support all
features of this Talk version, missing features: chat-relay
...
OK: Running version: 0.1 (the recording backend)
i.e. spreed's <SignalingServer> row reports `warningMessage`
("missing features: chat-relay") instead of the OK string, while the
<RecordingServer> row reports OK. spreed's `connectionState` getter
prefers `warningMessage` over the OK string by design (see
src/components/AdminSettings/SignalingServer.vue:136-150 of upstream
spreed), so the two-OK requirement is structurally false-fail-prone
on a healthy stack: a missing feature on the HPB image would always
turn this test red.
Fix the spec, not the deploy:
- Wait for `span.test-connection` to attach before counting buttons.
spreed's Test button is `v-if="server && checked"` and only appears
AFTER the auto-mounted `checkServerVersion()` call returns. Polling
too early found zero buttons, so `clicked.length > 0` could only
pass on a lucky race.
- Read the `<span class="test-connection">` texts directly via
`connectionSpans.allInnerTexts()` instead of pulling
`document.body.innerText`. The span is the canonical place spreed
writes connectionState, so the failure message can name the exact
rendered string ("Warning: ...", "Status: Checking connection",
"Error: ...") instead of dumping the whole admin chrome.
- Require ONE OK marker (recording row) instead of two. The
RecordingServer row only renders when `hasSignalingServers` is
true, so on a freshly-bootstrapped stack the recording row can be
the only one with a clean OK while the HPB row legitimately reports
a warning. Asserting both would block a healthy install on a
cosmetic feature mismatch.
- Iterate connection-span texts in the error-pattern check so the
failure message points at the exact row that broke instead of
matching against the whole body.
Verified locally via
`SERVICES_DISABLED='matomo,email' make deploy-reuse-kept-apps APPS=web-app-nextcloud`:
✓ nextcloud talk admin settings (23.8s)
✓ dashboard to nextcloud login (29.7s)
✓ biber logs into nextcloud via OIDC and logs out (4.7s)
PLAY RECAP: ok=1728, changed=131, failed=0, rescued=0, skipped=643.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`test_public_http_urls_are_reachable` flagged
`http://deb.debian.org/debian-security` as a hard fail (HTTP 404),
breaking the test-external CI gate.
The URL is correct — apt clients fetch concrete files underneath
`dists/bookworm-security/`, all of which resolve to 200. Only a bare
GET on the redirector path returns 404 because the redirector does
not serve a directory index. The same convention (`# nocheck: url`)
already covers raw-packagist on line 121 for the same reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
