Commit Graph
43 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5 c215aca058 fix(nextcloud): route sociallogin through the tor proxy, fail fast on a refused login callback, stop leaking plugin config
Run 34421944328 Nextcloud#1 timed out "waiting for a signed-in Nextcloud
shell after the login redirect". 4164f37981 turned sso on in variant 1
while ldap stays off there, so that round logs in through sociallogin
instead of oidc_login - the first variant, on this branch or on main,
that runs sociallogin against an onion issuer. oidc_login goes through
Nextcloud's HTTP client, which honours the proxy in proxy.config.php;
sociallogin builds its own client and reads only
social_login_http_client['proxy'], so its token exchange never reached
the onion Keycloak. The template now sets the same tor SOCKS proxy for
it, inside the existing "sso on and the OIDC host is an onion" branch.

The spec only treated 5xx documents as a page failure, so a refused
callback surfaced as a 60s (x5 over tor) wait for a shell that could not
come. A 4xx on /apps/sociallogin/ or /apps/oidc_login/ now counts as
the page itself failing, reported without the query so no code leaks.

The plugin loop printed every item, including the OIDC client secret,
into the include line of a public CI log; it now labels by plugin key and
honours MASK_CREDENTIALS_IN_LOGS.

_shared.js grew past 250 lines; its page and modal helpers move to
_page.js and _shared.js re-exports them, with an identical export
surface.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 13:21:39 +02:00
kevinveenbirkenbachandClaude Opus 5 5137465c6f style: delete every comment the comments-valid policy rejects
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.

Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 03:22:02 +02:00
kevinveenbirkenbachandClaude Opus 5 20ddda8c1c fix(nextcloud): reach the onion OIDC issuer over Tor SOCKS again
libcurl refuses a .onion host by name (RFC 7686, error 6) before it
consults /etc/hosts, extra_hosts or DNS, so the platform's DNS+iptables
onion egress cannot serve a server-side backchannel at all. Measured:

    curl --noproxy '*'                          -> exit 6
    curl --resolve auth.<onion>:80:127.0.0.1 …  -> exit 6

--resolve fills curl's own cache strictly earlier than getaddrinfo and
is still refused, which makes sys-svc-container's onion_oidc_socks
template -- extra_hosts, despite its name -- a no-op for every libcurl
client. Only a configured proxy makes curl skip the check.

Since 9d91fbea8 emptied proxy.config.php.j2 there is no such proxy, while
svc-net-tor's exclusive/primary defaults replace Keycloak's clearnet
domain and KC_HOSTNAME pins the issuer to the onion. Nextcloud's
oidc_login discovery call therefore dies inside PHP, /apps/oidc_login/oidc
returns 500, and oidc_login_auto_redirect sends /login straight there.

Restoring the old socks5h://host.docker.internal:9050 alone would make it
worse: tor runs as a single manager-pinned replica publishing SOCKS with
mode: host, so only the replica on that node could connect and a
deterministic failure would become a 2-in-3 flake. In swarm, though, tor
is NOT host-networked -- that branch is compose-only -- and it already
binds SocksPort 0.0.0.0, so the shared overlay is a usable path from
every node.

Give svc-net-tor a shared_net overlay with alias 'tor' and dial it by
service name. Attachment is gated by a new consumer kind, onion_sso: the
consumer's services.sso.enabled AND the provider's services.tor.enabled.
Both are group-membership derived, so the conjunction is exactly "the
issuer is an onion" and stays false in clearnet deployments -- which
matters because the service registry is not filtered to deployed roles.
The existing kinds could not express it: keying on sso alone would also
hold without tor, and keying on tor would miss Nextcloud's variant 2,
which disables its own onion yet still needs the client path.

tor_socks_proxy keeps the swarm/compose split in one place so the six
roles sharing this defect need not restate it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-10 00:50:48 +02:00
kevinveenbirkenbachandClaude Opus 5 9d91fbea83 feat(svc-net-tor): route onion names through tor via dns, not per-app proxies
Reaching an onion address required each role to point its own http client
at the tor socks proxy, which then carried clearnet traffic too — fatal
in the test network, which has no exit, and the reason nextcloud needed a
per-host exclusion list.

The egress router already redirects tor's virtual range to the TransPort
and dnsmasq already forwards .onion to tor's DNSPort; the piece missing
was reachability. dnsmasq listened on loopback only, which docker
discards, so containers never saw it. It now also answers on the docker
bridge, the daemon points there with the clearnet resolver kept as
fallback, and the role's own onion resolves to a container-reachable
backend instead of loopback.

Nextcloud drops its proxy block and the exclusion list; every other role
gets the same behaviour without configuring anything.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 01:13:33 +02:00
kevinveenbirkenbachandClaude Opus 5 d85603782f fix(web-app-nextcloud): keep appstore downloads off the tor proxy
The role routes all outgoing traffic through tor so it can reach
keycloak on its onion address, but the test tor network has no exit, so
every app download ran into a two-minute timeout and the deploy aborted
on the mandatory oidc_login plugin. All 20 required apps resolve to
githubusercontent hosts after redirects, so excluding those plus the
appstore API is enough; onion traffic still goes through the proxy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 23:51:17 +02:00
kevinveenbirkenbachandClaude Fable 5 d88f272f9f Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 15:31:08 +02:00
kevinveenbirkenbachandClaude Opus 4.8 d5aa44715b fix(web-app-nextcloud): route OIDC through Tor SOCKS on an onion node
On an onion node the OIDC issuer (Keycloak) is a .onion; libcurl (oidc_login's Guzzle client) refuses to resolve .onion per RFC 7686, so the server-side OIDC backchannel (discovery/token/jwks) fails. Set Nextcloud's system proxy to socks5h://host.docker.internal:<tor-socks> and expose the host Tor SOCKS to the app container via extra_hosts, both gated on SSO being enabled and the Keycloak domain being an onion. No-op on clearnet nodes.

Hoist the repeated services.sso.enabled and Keycloak-domain lookups into NEXTCLOUD_SSO_ENABLED and NEXTCLOUD_OIDC_HOST constants (mirroring MATRIX_OIDC_HOST) and use them across the role.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 10:50:09 +02:00
kevinveenbirkenbachandClaude Fable 5 b73706c022 fix(web-app-nextcloud): converge on swarm and run multi-node fpm safely
The fpm/cron occ-status healthcheck rendered a 30s start_period, far shorter
than the (image-serialized) first-boot init, so swarm killed every task with
exit 137 mid-init and the stack never converged. Give both healthchecks a 10m
start_period and raise swarm_stack_converge_retries to 600.

The nextcloud image already serializes first boot across replicas (its own
init lock), so unpin fpm from replicas: 1 and let it run on every node; keep
cron pinned as a genuine singleton. For correct multi-instance runtime, follow
the Nextcloud caching recommendation: memcache.local => APCu in both modes,
memcache.distributed and memcache.locking => Redis in swarm, backed by an
explicit redis connection array (which also fixes the RedisException at occ
upgrade, where OC\Memcache\Redis previously fell back to 127.0.0.1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 21:05:16 +02:00
kevinveenbirkenbachandClaude Opus 4.8 51c3d41ede Merge origin/main into feature/swarm-nfs
Reconcile main's object-store feature and pre-swarm deploy tasks with the branch's compose/networks/volumes SPOT and swarm lints.

objstore (seaweedfs/minio): standalone compose + embedded service templates migrated to the container_image/container_volumes/container_networks/compose_volumes/compose_networks/depends_on lookups; meta/volumes.yml on the canonical dict schema with s3.json and nginx.conf as type:config; embedded services made swarm-compatible via compose_only/compose_restart.

deploy tasks (erpnext, opentalk, seaweedfs, shopware, snipe-it, wordpress, yourls): container exec routed through the container_address resolver and gated on IS_STACK_HOST, shell module plus stack_host_template, no chdir or compose-verb forms.

Also removed orphaned vars, restored MAILU_INSTANCE_DIR, freed the seaweedfs subnet (192.168.206.0/24), stripped literal quotes from the nextcloud/bookwyrm env templates, and routed main's objstore compose_volumes tests through dir_var_lib.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 21:08:49 +02:00
kevinveenbirkenbachandClaude Opus 4.8 c86ab1e37b feat(web-app-nextcloud): provision on SeaweedFS S3 object store + consumer e2e test
Render the S3 objectstore config (templates/config/objectstore.config.php.j2) into NC's config dir before install, gated on objstore.enabled — NC cannot switch an already-installed instance to objectstore primary storage, so it must be present at install time.

Add the SeaweedFS consumer storage Playwright test (files/playwright/test-seaweedfs.js, playwright.env.j2 SEAWEEDFS_* vars, spec wiring; drop the playwright-service-flag nocheck for the seaweedfs service) plus the shared runSeaweedfsStorageCheck helper (admin logs into the Filer once, counts bucket objects via the Filer JSON listing, asserts growth) and the seaweedfs-consumer integration lint.

web-app-seaweedfs: bind weed to 0.0.0.0 (-ip.bind) so the multi-homed gateway is reachable by consumers that resolve the service name to its IP on the shared network.

nextcloud_install: classify 'was not found on the appstore' as unavailable (skippable, non-fatal) so a plugin without a release for the pinned NC version does not abort the whole deploy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 20:17:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 9cacfbf959 refactor(swarm): inventory-driven topology replaces per-role swarm flags
DEPLOYMENT_MODE is now derived per role from group membership:
swarm iff len(groups[application_id]) > 1, else compose. The cluster
infrastructure flag SWARM_CLUSTER_AVAILABLE is true whenever any host
sits in svc-docker-swarm, so DB and proxy services render attachable
overlay networks even when their own group lists only one host.

All service_is_stateful flags are removed across svc-db-mariadb,
svc-db-postgres, svc-db-openldap, svc-prx-openresty, svc-docker-registry
and svc-db-mariadb's compose template; the inventory shape now expresses
the same intent. Cluster-wide handlers (swarm build/push/validate/deploy,
external network creation, DB healthcheck) gain run_once + delegate_to
the manager so they execute exactly once regardless of how many hosts
the role's play targets. CA-injection and compose validate are gated to
compose-mode only; their compose-config strictness breaks when the
deploy block coexists with container_name under swarm.

Dead per-host swarm-resolve infrastructure is removed: swarm_resolve_
container.sh, resolve_container.yml, resolve_self.yml, the MariaDB-
specific swarm_mariadb_cid.sh, the cross-host template replication
step, and the _swarm_retries / _swarm_delay defaults. With run_once +
delegate_to: manager replacing the old "render every template on every
swarm peer" pattern, every consumer becomes a noop on workers and
collapses to the manager's one-shot path.

The web-app-nextcloud memcache override mapping flips from the legacy
"single" sentinel to "compose" (semantically equivalent under the new
model). Requirement 023 is updated to reflect the new model and drops
the now-irrelevant per-role opt-out / placement-constraint clauses.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-04 17:11:07 +02:00
kevinveenbirkenbachandClaude Opus 4.7 24d4ab8e4a refactor(sso)\!: collapse oauth2 + oidc into unified services.sso block (req 21)
Implements docs/requirements/021-sso-flavor-migration.md atomically:

* Schema: every role's meta/services.yml replaces the parallel
  `oauth2:` / `oidc:` top-level blocks with a single `sso:` block
  carrying `enabled`, `shared`, and a `flavor` discriminator
  (`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
  sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
  allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
  entirety; its 5 sidecar templates move under
  roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
  rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
  web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
  centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
  `is_oidc_native`) and the oauth2 sub-values
  (`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
  Three lookup plugins consume the resolver: `sso` (single-app
  predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
  consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
  plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
  and plugins/filter/compose_volumes.py all gate on
  `lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
  `enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
  → `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
  `OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
  `oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
  `OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
  `SSO_SERVICE_ENABLED`; Playwright gating helpers
  (isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
  safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
  is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
  `flavor: oauth2` with their existing `oauth2.acl.blacklist`
  (/user/login) and `oauth2.acl.whitelist` (federation + discovery
  endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
  (tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
  test_acl_mutual_exclusive.py) are deleted per Decision #10;
  test_oauth2_contract.py is renamed and rewritten to
  test_sso_contract.py, asserting that `flavor: oauth2` roles carry
  `oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
  (web-app integration) now requires the unified `sso:` block with an
  explicit flavor or a `# nocheck: sso` opt-out; new
  tests/lint/repository/no_legacy_sso_paths/ guards the source tree
  against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
  009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
  docs/contributing/actions/testing/suppression.md are updated to the
  unified SSO naming.

Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).

BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 12:59:59 +02:00
kevinveenbirkenbachandClaude Opus 4.7 79409ebf10 test(tooling): integrate markdownlint-cli2, ansible-lint, galaxy-importer + fix the resulting baseline
Tooling integration
* add .markdownlint-cli2.jsonc, .ansible-lint, .ansible-lint-ignore
  configs (permissive day-1: cosmetic rules disabled, semantic ones on)
* extend scripts/install/lint.sh with generic install_pip_pkg helper +
  ensure_ansible_lint / ensure_galaxy_importer / ensure_markdownlint_cli2
  ensure_* functions (pip / npm with sudo / --user / --break-system-packages
  fallbacks). install_ansible_tools now also pulls ansible-lint and
  galaxy-importer; new `markdown` install group.
* extend scripts/lint/ansible.sh to run ansible-lint after the existing
  syntax-check.
* add scripts/lint/markdown.sh and scripts/lint/galaxy.sh runners. The
  galaxy runner parallelizes via xargs (-P=$(nproc)), prints OK/FAIL per
  role, and treats `bare-name dependency` errors as success since our
  roles/ tree is not a Galaxy collection layout. All other Galaxy schema
  errors still fail.
* Makefile: new lint-markdown and lint-galaxy targets, both wired into
  `make lint`.
* CI: new workflows lint-markdown.yml and lint-galaxy.yml; lint-ansible.yml
  now installs ansible-core + ansible-lint; ci-orchestrator wires both
  new lint workflows into lint-gate.

Galaxy schema fixes
* shorten the canonical `galaxy_info.company` block from three lines to
  two ("Kevin Veen-Birkenbach\nhttps://www.veen.world", 45 chars) so it
  stays under Galaxy's hard 50-char limit. Updated everywhere: 229 role
  meta/main.yml plus _CANONICAL_COMPANY in
  tests/lint/repository/test_role_meta_main_galaxy_schema.py.
* shorten 19 over-long role descriptions to <=255 chars.
* add stub READMEs for sys-svc-compose-ca and sys-utils-service-loader.

Markdown baseline fixes
* markdownlint-cli2 --fix run repo-wide (formatting normalization
  across CHANGELOG, role READMEs, docs).
* manual MD041 fixes: prepend H1 in 15 files (CHANGELOG, LICENSE,
  TODO.md helpers, role README templates, etc.).
* MD025 / MD003 / MD001 / MD032 manual fixes for heading-style and
  list-spacing edge cases.
* role README content polish: "## Single Point of Truth (SPOT)" headings
  renamed to "## User data resolution" (D3 self-label rule).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 23:46:37 +02:00
kevinveenbirkenbachandClaude Opus 4.7 ed04dad5d5 refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).

req-008 — role meta layout
  * Move every role's metadata under meta/<topic>.yml with file-root
    convention (no wrapping key matching the basename):
      - schema/main.yml + config/main.yml.credentials → meta/schema.yml
      - users/main.yml                                → meta/users.yml
      - config/main.yml.server                        → meta/server.yml
      - config/main.yml.rbac                          → meta/rbac.yml
      - config/main.yml.compose.services + every other top-level key
        of config/main.yml inlined under <entity>.<key>
                                                      → meta/services.yml
      - config/main.yml.compose.volumes               → meta/volumes.yml
  * Drop the legacy schema/, users/, config/ directories from every role.
  * meta/schema.yml: support nested credential keys, optional `default:`
    field (Jinja literal, not rendered/validated/regenerated), implicit
    `algorithm: plain`.
  * Materialised paths drop the `compose.` prefix:
      compose.services.<e>.<…> → services.<e>.<…>
      compose.volumes.<k>      → volumes.<k>
      <former-top-level>.<…>   → services.<entity_name>.<former-top-level>.<…>
  * Update every Python consumer (utils/cache, utils/manager, utils/applications,
    plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
    and task file to the new paths.
  * No legacy fallback. Repository-wide grep for `compose.services.` /
    `compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
    is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
  * Add tests/lint/repository/test_role_meta_layout.py guarding against the
    old paths returning, and an explicit lint guard that meta/variants.yml
    files do not carry a `compose:` wrapper.

req-009 — per-role networks/ports
  * Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
    dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
    dns_resolver}.
  * Delete group_vars/all/09_ports.yml entirely; ports now live on the
    service entity that exposes them at
    roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
  * Rename + int-cast every former top-level `services.<entity>.port`
    field to `services.<entity>.ports.inter` (single int).
  * Always-category-keyed maps for ports.local / ports.public; relay
    spans use {start, end} integers.
  * Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
    of truth for the per-<scope>.<category> band ranges, consumed by both
    the new CLI helpers and the lint test.
  * New CLI helpers (cli meta ports suggest, cli meta networks suggest)
    walk the live role tree gap-first then increment-fallback, with
    deterministic output and non-zero exit on capacity overflow.
  * cli create role consumes both helpers when scaffolding a new role.
  * Lint extensions: bare-int port shape rejection, host-bound port
    collision check (single ints + relay span integers, inter excluded),
    band-membership check with documented allow-list for the legacy
    web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.

req-010 — run_after / lifecycle migration
  * Move galaxy_info.run_after and galaxy_info.lifecycle out of
    meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
    lifecycle}; <primary_entity> is the value returned by
    get_entity_name(role_name).
  * For multi-entity roles whose primary entity has no compose service
    (web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
    gen roles), create a dedicated metadata-only top-level entry under
    <primary_entity>.
  * Introduce utils/roles/meta_lookup.py with get_role_run_after /
    get_role_lifecycle that resolve <primary_entity> once and read from
    meta/services.yml; degrade gracefully when meta/services.yml is
    absent OR the field is absent; raise MetaServicesShapeError when the
    file exists but is malformed.
  * Update every consumer (utils/service_registry, utils/invokable,
    cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
    lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
    cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
    init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
    use the helper.
  * Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
    values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
    run_after/lifecycle never on a non-primary entity; empty run_after
    lists rejected.

Also folded into this change set
  * services_disabler now writes `services.<svc>.{enabled,shared}` into
    host_vars instead of the old `compose.services.<svc>.{...}` path; the
    discover/find pair walks meta/services.yml and only treats entries
    carrying enabled/shared/provides as provider declarations (so
    metadata-only entries from req-010 do not create spurious provider
    rows).
  * service_registry skips `run_after` deps whose target was filtered
    out via SERVICES_DISABLED instead of failing the whole load.
  * cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
    consume the new applications.<role>.services.<svc>.image shape
    (mirror.md updated to match).
  * utils/domains/list.build_applications_from_roles now reads
    meta/server.yml directly (it had been silently returning {} since
    config/main.yml stopped existing).
  * utils/roles/resource_validation.filter_roles_by_min_storage reads
    meta/services.yml.<entity>.min_storage instead of the legacy
    compose.services path.
  * web-app-bluesky pre-clones the social-app build context BEFORE
    sys-svc-compose so the build context is on disk by the time
    sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
  * 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
    web-app-wordpress, web-app-keycloak) lose their `compose: services:`
    wrapper so deep-merge over meta/services.yml lands on the new
    file-root shape.

Validation
  * Unit tests:        1571 passed.
  * Lint tests:        50 passed.
  * Integration tests: 161 passed (1 skipped).
  * Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
    set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
    discourse,mattermost}.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 09:58:26 +02:00
kevinveenbirkenbachandClaude Opus 4.7 51fc1d51ea feat(web-app-nextcloud): role-local oidc_flavor lookup with LDAP-aware default
Reintroduce the conditional OIDC-plugin selection that was lost when commit
77a0e16ea removed the _applications_nextcloud_oidc_flavor helper. Ship a
role-local lookup plugin at roles/web-app-nextcloud/lookup_plugins/oidc_flavor.py
with resolution order: explicit compose.services.oidc.flavor > oidc_login if
LDAP enabled > sociallogin. Wire all consumers (plugin toggles + oidc.config.php)
through lookup('oidc_flavor') and default compose.services.oidc.flavor to "" so
the LDAP-aware default takes effect (None would trip the non-empty-config
integration guard). Add 12 unit tests covering override, LDAP on/off,
whitespace, non-string, missing-app, and input-validation paths; evict
stubbed utils.applications.config leaked by the keycloak redirect_uris test
before loading the plugin so discovery-order no longer matters. Seed missing
__init__.py in tests/unit/roles/web-app-peertube and
tests/unit/cli/meta/applications/ressources so the repo-wide init-file guard
passes again.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-21 00:38:25 +02:00
Environment TestandClaude Opus 4.6 77a0e16ea6 refactor(web-app-nextcloud): consolidate oidc.flavor under compose.services.oidc
Single source of truth at compose.services.oidc.flavor. Removes stray
root-level keys, deletes the _applications_nextcloud_oidc_flavor helper,
and switches all consumers (plugin toggles + PHP template) to direct
lookups. Also fixes inherited precedence bug on plugin.enabled filters
and corrects OICD/floavor typos.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-15 11:15:27 +02:00
kevinveenbirkenbach b78ab29218 Renamed config entry from docker to compose
https://chatgpt.com/share/69870f54-d708-800f-9870-e9ae5a4bff44
2026-02-07 11:09:36 +01:00
kevinveenbirkenbach 497986e309 fix(mariadb): correct config lookup arguments for image setting
Fix invalid lookup('config') call for MARIADB_IMAGE that passed an extra
argument. Other replacements from get_app_conf to config lookup are
correct and unchanged.

https://chatgpt.com/share/69860678-cef8-800f-8402-7fe6231fdc2d
2026-02-06 16:26:25 +01:00
kevinveenbirkenbach 3fa7887dc8 Added whitespaces for better readability 2026-02-06 16:21:10 +01:00
kevinveenbirkenbach 5100c2c1ce refactor(config): replace get_app_conf filter with config lookup
Replace usages of the get_app_conf filter with the config lookup plugin
across roles and templates. Adjust quoting where required to keep YAML
and Jinja syntax valid. No functional changes intended.

https://chatgpt.com/share/69860678-cef8-800f-8402-7fe6231fdc2d
2026-02-06 16:19:14 +01:00
kevinveenbirkenbach 7cb881183b refactor: remove implicit True flag from get_app_conf calls
The get_app_conf() helper no longer requires an explicit third
parameter to enable application-specific lookups. All templates,
tasks, and vars have been updated to rely on the new default behavior
by removing the redundant `True` argument.

This simplifies usage, improves readability, and enforces a consistent
API across Jinja2 templates and Ansible logic.

https://chatgpt.com/share/6985f5f6-c084-800f-887a-4a70d62e6dcf
2026-02-06 15:08:47 +01:00
kevinveenbirkenbach 3bb8c0ce30 refactor(tls): rename tls_resolve lookup to tls
The TLS lookup has been renamed from `tls_resolve` to `tls` to better
reflect its responsibility as a strict TLS state resolver.

This refactor updates all usages across the codebase, including:
- lookup plugins and internal helpers
- roles, vars, tasks, and templates
- tests, stubs, and error messages
- documentation and changelog references

Behavior, lookup API, and semantics remain unchanged.

https://chatgpt.com/share/6985e635-5a94-800f-b794-937e566ee308
2026-02-06 14:01:32 +01:00
kevinveenbirkenbach 5e33e70892 refactor(database lookup usage): switch conditionals to enabled flag
Replace indirect type!='' checks with explicit database.enabled lookups
to improve semantics, readability, and alignment with strict lookup API.

https://chatgpt.com/share/6985d642-b40c-800f-9cdf-ed54fc484a92
2026-02-06 13:25:37 +01:00
kevinveenbirkenbach 8fe6b92e63 refactor(lookups): migrate nginx_paths, tls_resolve and cert_plan to strict positional want-path API
- Replace legacy want= keyword usage with mandatory positional want-path
- Align all nginx_paths, tls_resolve and cert_plan lookups to strict API
- Keep behavior unchanged while enforcing consistent lookup semantics

https://chatgpt.com/share/6985d642-b40c-800f-9cdf-ed54fc484a92
2026-02-06 13:06:09 +01:00
kevinveenbirkenbach f6f96a11ac Refactor: use database lookup helper everywhere
https://chatgpt.com/share/6983bcf6-85bc-800f-a0fc-5684b9078469
2026-02-04 22:52:08 +01:00
kevinveenbirkenbach 0b9f71463c refactor(tls): replace legacy tls filters with strict tls_resolve lookup
- Remove deprecated filter_plugins/tls.py and all dependent filters
- Introduce strict lookup plugin lookup_plugins/tls_resolve.py as single TLS authority
- Migrate all roles, templates, vars, and tasks to use tls_resolve (protocols, ports, URLs, cert paths)
- Enforce explicit TLS context (domains, applications, TLS_ENABLED, TLS_MODE)
- Support letsencrypt and self_signed with SAN handling and per-app overrides
- Update TLS provider roles to assert resolver state instead of setting facts
- Add comprehensive unit tests for tls_resolve lookup

https://chatgpt.com/share/6972b49b-b074-800f-a323-dd9fe69d02eb
2026-01-23 00:36:46 +01:00
kevinveenbirkenbach 9fdb0cddd0 Replace global WEB_PROTOCOL / WEB_PORT / WEBSOCKET_PROTOCOL logic
with explicit per-application TLS resolution.

Key changes:
- Introduce sys-front-tls-layer as the single TLS abstraction
- Centralize certificate paths via TLS_CERT_FILE / TLS_KEY_FILE / TLS_CA_FILE
- Replace sys-svc-letsencrypt SSL templates with generic TLS layer includes
- Add tls_* Jinja filters (enabled, mode, protocol, port, websocket)
- Resolve TLS mode per application (letsencrypt | self_signed | off)
- Remove implicit defaults and WEB_* globals to fail fast on misconfiguration
- Support self-signed certificates with SAN awareness
- Make TLS behavior explicit and consistent across all roles

This decouples application logic from certificate providers and
allows switching TLS flavor without touching app roles.

https://chatgpt.com/share/69727c61-8e68-800f-a267-473a7e627978
2026-01-22 20:36:49 +01:00
kevinveenbirkenbach 069030deea Made TLS for OIDC Nextcloud plugin variable 2026-01-20 02:06:56 +01:00
kevinveenbirkenbach c0fdfaa759 fix(nextcloud): stabilize config permissions, loader injection and bootstrap flow
- ensure host-mounted infinito config directory ownership and permissions (recursive)
- ensure includes.php ownership on host before container start
- inject /tmp/includes.php loader into config.php idempotently using printf (no heredoc)
- prepare config directory with correct permissions in Dockerfile to avoid root-owned bootstrap
- add read_only.config.php template (default disabled for installer phase)

This fixes broken config.php injection, prevents root-owned config dir on fresh volumes,
and stabilizes Nextcloud auto-installer and OCC plugin routines.

https://chatgpt.com/share/696ed102-9e74-800f-ad52-f09ad0a9da80
2026-01-20 01:48:46 +01:00
kevinveenbirkenbach 04c9490745 Solved unit and integration tests 2026-01-10 13:07:12 +01:00
kevinveenbirkenbach fc59c64273 Nextcloud Talk: fix virtual-background web check by
- adding explicit MIME types for .wasm and .tflite in internal Nginx
- relaxing CSP (script-src: allow 'unsafe-eval') for WebAssembly
- removing obsolete turnserver draft.
Details: https://chatgpt.com/share/68d7dd39-50b8-800f-ab59-cfb1d3cf07cb
2025-09-27 14:49:42 +02:00
kevinveenbirkenbach 4f5602c791 Nextcloud Talk: fix TURN/STUN config
- Removed duplicate Admin Manual link in README
- Fixed turnserver.config.php draft return syntax
- Unified onboard port handling in docker-compose and env
- Updated vars to define STUN/TURN configs with correct schemas
- Ensured spreed plugin config serializes clean JSON arrays

Ref: https://chatgpt.com/share/68d7cfa2-7378-800f-9ecf-09b6bb768f13
2025-09-27 13:51:17 +02:00
kevinveenbirkenbach 75d476267e Optimized Nextcloud variables 2025-09-27 12:14:57 +02:00
kevinveenbirkenbach 7ca8b7c71d feat(nextcloud): integrate Talk & Whiteboard; refactor to NEXTCLOUD_* vars; full-stack setup
config(ports): add Nextcloud websocket port (4003); canonical domains (nextcloud/talk/whiteboard)

refactor: unify get_app_conf usage & Jinja spacing; migrate paths/handlers to new NEXTCLOUD_* vars

feat(plugins): split plugin routines; configure Whiteboard via occ (URL + JWT)

fix(oidc): use NEXTCLOUD_URL for logout; correct LDAP attribute mappings; add OIDC flavor switch

feat: Whiteboard container & reverse-proxy location; Talk STUN/WS ports; Redis URL for Whiteboard

chore: drop obsolete TODO; minor cleanups in oauth2-proxy, matrix, peertube, pgadmin, phpldapadmin, pixelfed, phpmyadmin

security(schema): Bluesky jwt_secret now base64_prefixed_32; add Nextcloud whiteboard_jwt_secret

db: normalize postgres image tag templating; central DB host checks spacing fixes

ops: add full-stack bootstrap (certs, proxy, volumes); internal nginx config reload handler update

refs: https://chatgpt.com/share/68b5f5b7-8d64-800f-b001-1241f818dc0e
2025-09-01 21:37:02 +02:00
kevinveenbirkenbach cb66fb2978 Refactor LDAP variable schema to use top-level constant LDAP and nested ALL-CAPS keys.
- Converted group_vars/all/13_ldap.yml from lower-case to ALL-CAPS nested keys.
- Updated all roles, tasks, templates, and filter_plugins to reference LDAP.* instead of ldap.*.
- Fixed Keycloak JSON templates to properly quote Jinja variables.
- Adjusted svc-db-openldap filter plugins and unit tests to handle new LDAP structure.
- Updated integration test to only check uniqueness of TOP-LEVEL ALL-CAPS constants, ignoring nested keys.

See: https://chatgpt.com/share/68b01017-efe0-800f-a508-7d7e2f1c8c8d
2025-08-28 10:15:48 +02:00
kevinveenbirkenbach 9180182d5b Optimized variables 2025-08-21 16:27:10 +02:00
kevinveenbirkenbach 0de26fa6c7 Solved bug existed due to difference between mailu domain and hostname difference. also refactored during this to find the bug 2025-08-16 14:29:07 +02:00
kevinveenbirkenbach 022800425d THE HUGE REFACTORING CALENDER WEEK 33; Optimized Matrix and during this updated variables, and implemented better reset and cleanup mode handling, also solved some initial setup bugs 2025-08-15 15:15:48 +02:00
kevinveenbirkenbach db0e030900 Renamed general and mode constants and implemented a check to verify that constants are just defined ones over the whole repository 2025-08-13 19:11:14 +02:00
kevinveenbirkenbach d1c8036fa4 Implemented DB Credentials Update for Nextcloud and solved bug 2025-07-14 18:41:30 +02:00
kevinveenbirkenbach ad60f5fb37 Rmeoved is_feature_enabled function 2025-07-13 17:54:09 +02:00
kevinveenbirkenbach 756597668c Semi bsr for applications[] to prevent heavy to debug bugs in j2 - part 1 2025-07-13 15:11:38 +02:00
kevinveenbirkenbach 563d5fd528 Huge role refactoring/cleanup. Other commits will propably follow. Because some bugs will exist. Still important for longrun and also for auto docs/help/slideshow generation 2025-07-08 23:43:13 +02:00