Split the previous two-file plugin pipeline into three single-responsibility
stages and group them under tasks/plugin/, with per-plugin hooks now sitting
under tasks/plugin/hooks/. Each stage owns exactly one OCC call (app:install,
app:enable, config:app:set) and one continuation decision driven by the
nextcloud_install_status filter.
Layout:
tasks/
├── 06_plugin.yml (entry; includes plugin/01_install.yml)
└── plugin/
├── 01_install.yml (occ app:install + classify)
├── 02_enable.yml (occ app:enable + classify)
├── 03_configure.yml (occ config:app:set + run hook)
├── hooks/ (per-plugin extension tasks)
│ └── user_ldap.yml
└── README.md (pipeline contract + extension guide)
The new 02_enable.yml mirrors the version-compatibility tolerance that
01_install.yml already had: when occ rejects the enable with the
"is not compatible with this version of the server" marker, the
failed_when clause downgrades the task to a non-fatal classification,
emits a debug warning, and skips the entire 03_configure.yml include via
a single set_fact-driven flag instead of repeating the filter call on
every downstream task. This unblocks deploys against a Nextcloud release
whose appstore lags briefly while the plugin's compatibility metadata
catches up (e.g. metadata, drawio, files_bpm in the NC 33 window).
NEXTCLOUD_CNODE_PLUGIN_TASKS_PATH in vars/main.yml is rebased to the new
tasks/plugin/hooks/ directory so plugin hooks resolve at the new location
without touching call sites.
README.md follows the role + docs README contracts: one sentence per line,
no em dashes, RFC 2119 keywords, no historical narrative, cross-links to
the filter plugin and the role vars file.
Makefile lint integration:
* scripts/install/lint.sh: replace the prior checkmake binary install with
ensure_mbake (mbake is pip-installable, parses our Makefile correctly
whereas checkmake bailed on .ONESHELL: / .SHELLFLAGS: / ifneq).
* scripts/lint/makefile.sh: new runner — `mbake format --check Makefile`
reports formatting drift and missing .PHONY declarations.
* scripts/lint/autoformat.sh: add a two-pass `mbake format Makefile`
(mbake's formatter is not strictly idempotent on the first pass).
* Makefile: split the multi-line `.PHONY:` block into 18 single-line
`.PHONY:` declarations so mbake's parser can validate it; add the
previously-missing `install-python-dev` and `build-dependency` PHONY
targets; add `lint-makefile` aggregate-and-direct target.
* .pre-commit-config.yaml: new make-lint-makefile hook scoped to
`^Makefile$`.
* .github/workflows/lint-makefile.yml: new CI workflow (pip-install mbake
+ make lint-makefile).
* .github/workflows/ci-orchestrator.yml: wire lint-makefile into
lint-gate.needs.
Misc:
* .gitignore: add roles/importer_result.json (tooling artefact written by
galaxy-importer when it runs in roles/).
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five web-app roles read their own `credentials.database_password` via
`lookup('config', application_id, …)` from `vars/main.yml`. The
credential is wired in by the inventory's apply_schema step rather
than declared in each role's `meta/schema.yml`, so the static
lookup-config check cannot see it. Add the `lookup-config-path`
marker to each call so the role-local pass stays clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces inline `applications`/`users` merging with dedicated lookup
plugins and runtime hydration. Introduces `add_csp_hash` filter,
integrates sort into `find_dock_val_by_bkp_entr`, splits the Keycloak
one-shot realm import into `05a_realm_import_run.yml`, and fixes the
Mastodon administrator task YAML. Removes obsolete `csp_hashes` filter,
`sys-token-store` Jinja filter, and empty CLI setup modules.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add 07_admin_identities.yml: resolves LDAP-backed admin identities
for OCC calls that require the actual NC username, not the LDAP DN
- vars: add NEXTCLOUD_HPB_TURN/STUN_ACTIVE_CONFIG selecting standalone
or onboard TURN based on enabled flags; add TALK_ADMIN_URL
- config: talk container gated on spreed plugin flag
- env.j2 + playwright.env.j2: expose active TURN/STUN config and
Talk admin URL for container env and E2E tests
- playwright.spec.js: extend with Talk TURN/STUN E2E assertions
- schema: add spreed.enabled validation
- coturn compose: move relay port range before TLS block so ports are
always set regardless of TLS mode
- docs/requirements: document TURN relay requirement
- test_resolver: cover new resolution paths
- Rename lookup plugin from `docker` to `container`
- Switch all lookups from `lookup('docker', ...)` to `lookup('container', ...)`
- Normalize compose file keys:
- docker_compose → compose
- docker_compose_override → compose_override
- docker_compose_ca_override → compose_ca_override
- Centralize compose path resolution via docker_paths_utils
- Update compose_file_args to always derive paths from DIR_COMPOSITIONS
- Align all roles, templates, and vars with container/compose naming
- Update and rename unit tests accordingly
This prepares the codebase for engine-agnostic container backends
(e.g. Docker ↔ Podman) and removes legacy docker-specific assumptions.
https://chatgpt.com/share/6987a7cc-d374-800f-99ad-bb4c8f92c59e
Replace usages of the get_app_conf filter with the config lookup plugin
across roles and templates. Adjust quoting where required to keep YAML
and Jinja syntax valid. No functional changes intended.
https://chatgpt.com/share/69860678-cef8-800f-8402-7fe6231fdc2d
The TLS lookup has been renamed from `tls_resolve` to `tls` to better
reflect its responsibility as a strict TLS state resolver.
This refactor updates all usages across the codebase, including:
- lookup plugins and internal helpers
- roles, vars, tasks, and templates
- tests, stubs, and error messages
- documentation and changelog references
Behavior, lookup API, and semantics remain unchanged.
https://chatgpt.com/share/6985e635-5a94-800f-b794-937e566ee308
Migrate all lookup() usages from the deprecated want= keyword argument to the
new strict positional want-path API. Update call order where required
(e.g. nginx_paths) and ensure consistency across vars, tasks, and templates.
https://chatgpt.com/share/6985d642-b40c-800f-9cdf-ed54fc484a92
- Replace lookup('docker', ..., want='...') with lookup('docker', ..., 'path')
- Allow both top-level keys (directories/files) and dotted paths
- Simplify docker lookup API and reduce lazy dict propagation
- Update unit tests to reflect new lookup semantics
https://chatgpt.com/share/6984ccea-8290-800f-a6d7-ec0e94df95a0
Remove implicit docker_compose variable usage and resolve all docker paths
(files, directories, env) via lookup('docker', application_id).
This prevents scope leakage, centralizes path resolution, and makes docker
context deterministic across roles and handlers.
https://chatgpt.com/share/6984ccea-8290-800f-a6d7-ec0e94df95a0
- Mount includes.php read-only instead of docker cp
- Ensure include loader starts with proper PHP header
- Add overwriteprotocol, overwritehost, and trusted_proxies via system config
- Simplify env handling for protocol and trusted proxies
- Remove redundant permission and copy tasks
https://chatgpt.com/share/697f7aff-859c-800f-bfaa-0d4c2378fd85
Adds an integration test that fails when a service defines `build:` without an `image:` on the same indentation level. This enforces stable image names for tooling like CA-injection and makes compose behavior more predictable.
https://chatgpt.com/share/697bb4af-0d58-800f-8edd-aaef4c224319
- Remove deprecated filter_plugins/tls.py and all dependent filters
- Introduce strict lookup plugin lookup_plugins/tls_resolve.py as single TLS authority
- Migrate all roles, templates, vars, and tasks to use tls_resolve (protocols, ports, URLs, cert paths)
- Enforce explicit TLS context (domains, applications, TLS_ENABLED, TLS_MODE)
- Support letsencrypt and self_signed with SAN handling and per-app overrides
- Update TLS provider roles to assert resolver state instead of setting facts
- Add comprehensive unit tests for tls_resolve lookup
https://chatgpt.com/share/6972b49b-b074-800f-a323-dd9fe69d02eb
with explicit per-application TLS resolution.
Key changes:
- Introduce sys-front-tls-layer as the single TLS abstraction
- Centralize certificate paths via TLS_CERT_FILE / TLS_KEY_FILE / TLS_CA_FILE
- Replace sys-svc-letsencrypt SSL templates with generic TLS layer includes
- Add tls_* Jinja filters (enabled, mode, protocol, port, websocket)
- Resolve TLS mode per application (letsencrypt | self_signed | off)
- Remove implicit defaults and WEB_* globals to fail fast on misconfiguration
- Support self-signed certificates with SAN awareness
- Make TLS behavior explicit and consistent across all roles
This decouples application logic from certificate providers and
allows switching TLS flavor without touching app roles.
https://chatgpt.com/share/69727c61-8e68-800f-a267-473a7e627978
- pass application_id to sys-stk-full for proper app context
- remove recursive chown on /var/www/html/config (breaks with read-only infinito mount)
- set correct ownership and permissions for host config and include files
- normalize file modes for generated Nextcloud config snippets
https://chatgpt.com/share/696d5a7c-f248-800f-b228-b26afd8c24bf
This change removes --network=host usage, introduces DOCKER_IN_CONTAINER
for reliable environment detection, and replaces hardcoded localhost
bindings with an explicit DOCKER_BIND_HOST abstraction to avoid nftables
issues in CI/DiD environments.
https://chatgpt.com/share/69497616-7b00-800f-8fa3-ddc6eb9c5e98
Unify reverse proxy handling across apps via sys-stk-front-proxy and cleanly separate internal docker nginx configs from external vhosts.
Changes:
- docker-compose: use with_first_found for optional env and docker-compose.override files so roles can provide either a template or a static file without noisy 'Could not find or access' failures.
- sys-stk-front-proxy: remove front_proxy_domain_conf_src and switch nginx vhost rendering to with_first_found over app-specific proxy.conf.j2 and the generic roles/sys-svc-proxy vhost flavour; keep health-check and handler logic unchanged.
- web-app-nextcloud: migrate to sys-stk-full-stateful (front proxy + DB + docker), move internal nginx config to docker.conf.j2 under the volume path, and rename host.conf.j2 to proxy.conf.j2 for the external vhost.
- web-app-magento: rename nginx.conf.j2 to docker.conf.j2 and update the runtime template task accordingly to make the intent (internal nginx) explicit.
- web-app-matrix: rename nginx.conf.j2 to synapse.conf.j2 and adjust the webserver task to use the new template name for the synapse vhost.
- web-app-bridgy-fed & web-app-flowise: pass domain and http_port explicitly when including sys-stk-front-proxy so the front stack has all required context.
- web-svc-cdn/file/html: replace direct sys-stk-front-base + sys-util-csp-cert + nginx.conf.j2 handling with sys-stk-front-proxy and proxy.conf.j2, relying on the shared front-stack for TLS/CSP/vHost wiring.
- web-svc-collabora: drop the direct nginx.conf.j2 vhost generation and rename it to proxy.conf.j2 so it is picked up by sys-stk-front-proxy like other services.
- web-opt-rdr-domains: rename redirect.domain.nginx.conf.j2 to redirect-domain.conf.j2 and adjust the task for clearer and more consistent naming.
Context: see ChatGPT refactor discussion on 2025-11-30 (proxy unification, Collabora/Nextcloud/CDN stacks, CSP/header handling): https://chatgpt.com/share/692c64ea-a488-800f-ad42-7f7692a3742f
- New web-svc-onlyoffice role (Docker, env, CSP, proxy hooks)
- Nextcloud plugin integration with JWT and CSP updates
- Custom Nextcloud image with updated CA certificates
- Networking and port assignments
- Disabled Collabora by default
- Enabled ONLYOFFICE by default
- Link to conversation for reference: https://chatgpt.com/share/691f49ac-54d8-800f-83c3-60cf8c100183
- Add node_autosize filter (node_max_old_space_size) using get_app_conf
- Raise error when mem_limit < min_mb to prevent OOM-kill misconfigurations
- Wire Whiteboard NODE_OPTIONS and increase mem_limit to 1g; set cpus=1
- Refactor PeerTube to use the same filter; simplify vars
- Add unit tests; keep integration filters usage green
Context: https://chatgpt.com/share/690e0499-6a94-800f-b8ed-2c5124690103
- Added role tasks to deploy templated Dockerfile for Whiteboard service
- Configured build context and custom image name (nextcloud_whiteboard_custom)
- Increased PID limits and shm_size for stable recording
- Adjusted user ID variable naming consistency
- Integrated path_join for service directory variables
- Fixed build permissions (install as root, revert to nobody)
Reference: ChatGPT conversation https://chatgpt.com/share/68f771c6-0e98-800f-99ca-9e367f4cd0c2
- Added Chromium headless flags and writable font cache/tmp volumes
- Enabled WebSocket proxy forwarding for /whiteboard/
- Verified and adjusted CSP and frontend integration
- Added Whiteboard-related variables and volumes in main.yml
See ChatGPT conversation (20 Oct 2025):
https://chatgpt.com/share/68f655e1-fa3c-800f-b35f-4f875dfed4fd
- Removed ?transport=udp from Nextcloud Talk TURN server definitions
- Dropped --no-tcp-relay to allow TCP fallback
- Removed invalid UDP mapping on TLS port
- Introduced switch between REST secret auth and lt-cred-mech via COTURN_USER_AUTH_ENABLED
- Added user_auth_enabled flag in coturn config for flexibility
See: https://chatgpt.com/share/68d7d601-3558-800f-bc84-00d7e8fc3243
- Replace inline get_app_conf secrets in env.j2 with dedicated vars (TURN, signaling, internal)
- Correctly model signaling_servers as object {servers, secret} in spreed.yml
- Use UDP stun_turn port instead of TLS for transport=udp
- Add fallback logic for standalone Coturn role in main.yml
- Remove obsolete Greenlight section from BBB override
Ref: https://chatgpt.com/share/68d74e25-c068-800f-ae20-d0e34ac8ee12
- Switch to custom BBB Docker repository
- Externalize Coturn and Collabora by default
- Add dedicated 03_dependencies.yml for dependency handling
- Improve env templating with lowercased feature flags
- Add conditional healthcheck for Greenlight
- Refactor TURN/STUN/relay handling with role variable _BBB_COTURN_ROLE
- Extend Collabora/Greenlight dependency wiring in override file
- Nextcloud Talk: refine vars and enable/disable logic with separate plugin/service flags, add network_mode support and conditional nginx proxy block
Ref: https://chatgpt.com/share/68d741ff-a544-800f-9e81-a565e0bab0eb
- Split internal/external Coturn for BBB and Nextcloud
- Added dedicated relay port ranges per app
- Updated env and compose overrides for coturn
- Ensure coturn role is loaded conditionally
- Standardize credential/env passing for coturn
@See https://chatgpt.com/share/68d6f376-4878-800f-b4f7-62822caa49ea
