Commit Graph
44 Commits
Author SHA1 Message Date
kevinveenbirkenbach c8ff5e8605 feat(services): register every MCP provider as a shared service
Ten provider roles served an MCP surface that other roles consume, but
were never declared as shared services, so the registry did not know
them. Declaring them pulls in the recursion contract every registry
service owes: seven roles move their entry tasks into tasks/00_core.yml
behind the run_once flag, so the service loader cannot include them a
second time.

pretix already had a run_once block, but set the flag last; the flag now
comes first, before the body it protects.
2026-08-09 22:22:08 +02:00
kevinveenbirkenbach 91320b8deb refactor(nextcloud): make main.yml the orchestrator and fix the flush order
main.yml now reads as the chronological sequence it describes: it includes
01_fullstack.yml, then the single IS_STACK_HOST-gated 02_manager_ops.yml,
whose substeps live in the matching 02_manager_ops/ directory numbered in
execution order. The plugin pipeline moves under its parent step so its
include resolves against its own directory instead of an inherited search
root.

The flush that deploys the stack moves out of the config step to the top of
02_manager_ops.yml. It previously ran after the swarm resolver, which then
waited 600 times for a service that did not exist yet and killed two CI jobs
with "no such service: nextcloud_nextcloud". The second flush and the
re-resolve behind it are removed: neither included file notifies a handler,
so they could never fire.

A new lint pins what made this refactor risky. A relative include is resolved
at runtime, so a typo or a moved file survives ansible-lint and
--syntax-check and only fails mid-deploy; worse, a bare name that matches
under two search roots silently shadows one file with the other.

Comments that only narrated code are dropped from the addon hooks and the
plugin install step.
2026-08-02 19:52:22 +02:00
kevinveenbirkenbachandClaude Opus 4.8 cddb0dc59b feat(lint): require emoji-prefixed task/step/job names + convert existing
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.

- emit emoji names at the source: cli/build/include generator now writes
  "🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
  tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
  task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
  🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 22:45:06 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a4cb5b89e1 feat(swarm): swarm-aware container exec via container_address lookup
Adds a mode-aware `container_address` lookup plugin and a
`resolve-container-id` helper script that resolve swarm task IDs at
exec time (compose mode returns bare names unchanged). Migrates ~25
roles + ~28 task files from `command:` to `shell:` so the lookup's
`$(...)` subshell evaluates correctly, and extends `database` lookup
with an `address` key for cross-role DB exec.

Three new lint tests enforce the contract: every container exec
target goes through the resolver, lives behind an IS_STACK_HOST gate
(walks the include graph one hop deep), and uses `shell:` rather than
`command:` when the target is a Jinja reference. Existing nocheck
suppressions on bare-name DB exec sites (nextcloud redis, listmonk,
moodle OIDC, mailu) are replaced with the lookup-based form.

Fixes the systemd Environment= injection in sys-ctl-cln-disc-space
(it never expanded $(...)) by passing STACK / SERVICE_KEY parts and
resolving inside the script. Extracts shopware admin and mediawiki
extension install bodies into dedicated .sh files invoked via env.
Rewrites password-bearing shell tasks to either `| quote` directly or
the `-e VAR=... | quote` + `sh -c '... ${VAR} ...'` env-bridge pattern.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-10 03:46:28 +02:00
kevinveenbirkenbach eab16c0568 refactor(web-app-nextcloud): reshape plugin pipeline into install/enable/configure stages
Split the previous two-file plugin pipeline into three single-responsibility
stages and group them under tasks/plugin/, with per-plugin hooks now sitting
under tasks/plugin/hooks/. Each stage owns exactly one OCC call (app:install,
app:enable, config:app:set) and one continuation decision driven by the
nextcloud_install_status filter.

Layout:

  tasks/
  ├── 06_plugin.yml                 (entry; includes plugin/01_install.yml)
  └── plugin/
      ├── 01_install.yml            (occ app:install + classify)
      ├── 02_enable.yml             (occ app:enable + classify)
      ├── 03_configure.yml          (occ config:app:set + run hook)
      ├── hooks/                    (per-plugin extension tasks)
      │   └── user_ldap.yml
      └── README.md                 (pipeline contract + extension guide)

The new 02_enable.yml mirrors the version-compatibility tolerance that
01_install.yml already had: when occ rejects the enable with the
"is not compatible with this version of the server" marker, the
failed_when clause downgrades the task to a non-fatal classification,
emits a debug warning, and skips the entire 03_configure.yml include via
a single set_fact-driven flag instead of repeating the filter call on
every downstream task. This unblocks deploys against a Nextcloud release
whose appstore lags briefly while the plugin's compatibility metadata
catches up (e.g. metadata, drawio, files_bpm in the NC 33 window).

NEXTCLOUD_CNODE_PLUGIN_TASKS_PATH in vars/main.yml is rebased to the new
tasks/plugin/hooks/ directory so plugin hooks resolve at the new location
without touching call sites.

README.md follows the role + docs README contracts: one sentence per line,
no em dashes, RFC 2119 keywords, no historical narrative, cross-links to
the filter plugin and the role vars file.
2026-05-19 23:01:18 +02:00
kevinveenbirkenbachandClaude Opus 4.7 2b72d8b550 test(lint): forbid redundant default(...) on module IO in same-task conditionals
Adds tests/lint/ansible/test_no_redundant_default_on_module_io.py which
flags `<reg>.{stdout,stderr,rc} | default(...)` inside `changed_when:`,
`failed_when:`, or `until:` lines. Those conditionals only fire after
the module has actually executed, so the fallback can never apply --
keeping it just hides bugs and noise.

Cleans up the 21 existing offenders across 14 role files. The taiga
upsert-admin task additionally gains a `not ASYNC_ENABLED and ...`
short-circuit so its async-poll-0 path no longer leans on a silent
empty fallback for stdout/stderr; with the short-circuit, the
`default('')` is genuinely dead code.

Cross-task references (a later task reading a registered result of an
earlier task that may have been skipped via `when:`) stay out of scope
and remain a legitimate use of `| default(...)`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 22:06:32 +02:00
kevinveenbirkenbachandClaude Opus 4.7 2a023ca6a1 chore(lint): autoformat strict-failure semantics + claude.sh rename
* scripts/lint/autoformat.sh:
  - rename run_sort_claude_settings worker to run_claude (matches the
    renamed scripts/lint/claude.sh).
  - drop the EXIT trap that wiped status_dir; cleanup happens explicitly
    after `wait` returns, so a parallel worker that interrupts the
    parent never races a vanished status dir.
  - each worker now captures its tool's exit code and writes one of
    OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
    (markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
    real failures count as FAIL.
  - all workers run to completion regardless of individual tool failures
    (so the user always sees the full picture); after the summary, the
    script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
  (history preserved via git mv); update test_claude_settings_sorted.py
  fix-hint accordingly.

Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 01:30:49 +02:00
Environment Test 83cd881dee feat(nextcloud): wire standalone coturn TURN + admin identity tasks
- Add 07_admin_identities.yml: resolves LDAP-backed admin identities
  for OCC calls that require the actual NC username, not the LDAP DN
- vars: add NEXTCLOUD_HPB_TURN/STUN_ACTIVE_CONFIG selecting standalone
  or onboard TURN based on enabled flags; add TALK_ADMIN_URL
- config: talk container gated on spreed plugin flag
- env.j2 + playwright.env.j2: expose active TURN/STUN config and
  Talk admin URL for container env and E2E tests
- playwright.spec.js: extend with Talk TURN/STUN E2E assertions
- schema: add spreed.enabled validation
- coturn compose: move relay port range before TLS block so ports are
  always set regardless of TLS mode
- docs/requirements: document TURN relay requirement
- test_resolver: cover new resolution paths
2026-04-14 02:29:29 +02:00
kevinveenbirkenbach d44eee1048 Fix Ansible fact deprecations and loop variable collisions 2026-02-26 17:54:14 +01:00
kevinveenbirkenbach c1d50d78da refactor(container): replace BIN_CONTAINER usages with system-wide container wrapper
https://chatgpt.com/share/698660b7-4c78-800f-b25f-af5408cff034
2026-02-06 23:00:21 +01:00
kevinveenbirkenbach 46238d4c89 refactor: use BIN_CONTAINER wrapper for container exec commands
https://chatgpt.com/share/698657ee-e708-800f-ba5d-4c7c5198a5fe
2026-02-06 22:06:42 +01:00
kevinveenbirkenbach 4ca9bc61e0 Removed MODE_UPDATE conditions to make behaviour more deterministic 2026-02-05 11:39:09 +01:00
kevinveenbirkenbach d0a244ef3a Corrected variable name 2025-12-22 17:49:13 +01:00
kevinveenbirkenbach 2b5ff60a6f infra(docker): drop host networking and normalize port binding logic
This change removes --network=host usage, introduces DOCKER_IN_CONTAINER
for reliable environment detection, and replaces hardcoded localhost
bindings with an explicit DOCKER_BIND_HOST abstraction to avoid nftables
issues in CI/DiD environments.

https://chatgpt.com/share/69497616-7b00-800f-8fa3-ddc6eb9c5e98
2025-12-22 17:46:08 +01:00
kevinveenbirkenbach d0efc77b6c Optimized variable conditions for ansible update 2.18 -> 2.20 2025-12-04 12:24:01 +01:00
kevinveenbirkenbach 116c20d61d Refactor Nextcloud fullstack and config tasks:
- Load proxy in fullstack
- Simplify config merging logic
- Improve handler flushing and container config update

Context: https://chatgpt.com/share/692dc9d4-a884-800f-85e5-e92471c7b52b
2025-12-01 18:01:25 +01:00
kevinveenbirkenbach f8e5110730 Add Redis readiness check before Nextcloud upgrade and add retry logic for maintenance repair
This prevents OCC repair failures caused by Redis still loading its dataset after container restarts.
See context: https://chatgpt.com/share/690377ba-1520-800f-b8c1-bc93fbd9232f
2025-10-30 15:36:00 +01:00
kevinveenbirkenbach 75d476267e Optimized Nextcloud variables 2025-09-27 12:14:57 +02:00
kevinveenbirkenbach e6803e5614 refactor(ansible): normalize include_role syntax and unify host config paths via path_join
- Remove stray spaces after include_role: across many roles to ensure clean YAML and
  consistent linting/formatting.
- Listmonk:
  - Introduce LISTMONK_CONFIG_HOST = [ docker_compose.directories.config, 'config.toml' ] | path_join
  - Use that var in the template task (dest) and the docker-compose volume mount
- Matrix:
  - Build MATRIX_SYNAPSE_CONFIG_PATH_HOST, MATRIX_SYNAPSE_LOG_PATH_HOST, and
    MATRIX_ELEMENT_CONFIG_PATH_HOST via path_join
- Mobilizon:
  - Build mobilizon_host_conf_exs_file via path_join
  - Keep get_app_conf strictness unchanged (defaults to True in our filter), so behavior
    remains strict even though the explicit third arg was dropped
- Simpleicons:
  - Build server.js and package.json host paths via path_join
- Numerous web-app roles (Confluence, Discourse, EspoCRM, Friendica, Funkwhale, Gitea,
  GitLab, Jenkins, Joomla, Listmonk, Mailu, Mastodon, Matomo, Matrix, MediaWiki,
  Mobilizon, Moodle, Nextcloud, OpenProject, Peertube, Pixelfed, Pretix, Roulette Wheel,
  Snipe-IT, Syncope, Taiga, WordPress, XWiki, Yourls) and web-svc roles (coturn,
  libretranslate, simpleicons) updated for consistent include_role formatting

Why:
- path_join avoids double slashes and missing separators across different config roots
- Consistent include_role: formatting improves readability and prevents linter noise

Ref:
- Conversation: https://chatgpt.com/share/68d14711-727c-800f-b454-7dc4c3c1f4cb
2025-09-22 14:55:25 +02:00
kevinveenbirkenbach 7ca8b7c71d feat(nextcloud): integrate Talk & Whiteboard; refactor to NEXTCLOUD_* vars; full-stack setup
config(ports): add Nextcloud websocket port (4003); canonical domains (nextcloud/talk/whiteboard)

refactor: unify get_app_conf usage & Jinja spacing; migrate paths/handlers to new NEXTCLOUD_* vars

feat(plugins): split plugin routines; configure Whiteboard via occ (URL + JWT)

fix(oidc): use NEXTCLOUD_URL for logout; correct LDAP attribute mappings; add OIDC flavor switch

feat: Whiteboard container & reverse-proxy location; Talk STUN/WS ports; Redis URL for Whiteboard

chore: drop obsolete TODO; minor cleanups in oauth2-proxy, matrix, peertube, pgadmin, phpldapadmin, pixelfed, phpmyadmin

security(schema): Bluesky jwt_secret now base64_prefixed_32; add Nextcloud whiteboard_jwt_secret

db: normalize postgres image tag templating; central DB host checks spacing fixes

ops: add full-stack bootstrap (certs, proxy, volumes); internal nginx config reload handler update

refs: https://chatgpt.com/share/68b5f5b7-8d64-800f-b001-1241f818dc0e
2025-09-01 21:37:02 +02:00
kevinveenbirkenbach 110381e80c Refactored peertube role and implemented config volume 2025-09-01 18:19:50 +02:00
kevinveenbirkenbach b02d88adc0 Refactored server roles for better readability 2025-09-01 18:08:35 +02:00
kevinveenbirkenbach 6ea8301364 Refactor: migrate cmp/* and srv/* roles into sys-stk/* and sys-svc/* namespaces
- Removed obsolete 'cmp' category, introduced 'stk' category (fa-bars-staggered icon).
- Renamed roles:
  * cmp-db-docker → sys-stk-back-stateful
  * cmp-docker-oauth2 → sys-stk-back-stateless
  * srv-domain-provision → sys-stk-front
  * cmp-db-docker-proxy → sys-stk-full-stateful
  * cmp-docker-proxy → sys-stk-full-stateless
  * cmp-rdbms → sys-svc-rdbms
- Updated all include_role references, vars, templates and README.md files.
- Adjusted run_once comments and variable paths accordingly.
- Updated all web-app roles to use new sys-stk/* and sys-svc/* roles.

Conversation: https://chatgpt.com/share/68b0ba66-09f8-800f-86fc-76c47009d431
2025-08-28 22:23:09 +02:00
kevinveenbirkenbach a4f39ac732 Renamed webserver roles to more speakable names 2025-08-20 08:54:17 +02:00
kevinveenbirkenbach 9cfb8f3a60 Different optimations for collabora 2025-08-20 08:34:12 +02:00
kevinveenbirkenbach dc125e4843 Solved path bug 2025-08-20 06:18:52 +02:00
kevinveenbirkenbach 39a54294dd Moved update commands to nextcloud role 2025-08-20 06:07:33 +02:00
kevinveenbirkenbach b867a52471 Refactor and extend role dependency resolution:
- Introduced module_utils/role_dependency_resolver.py with full support for include_role, import_role, meta dependencies, and run_after.
- Refactored cli/build/tree.py to use RoleDependencyResolver (added toggles for include/import/dependencies/run_after).
- Extended filter_plugins/canonical_domains_map.py with optional 'recursive' mode (ignores run_after by design).
- Updated roles/web-app-nextcloud to properly include Collabora dependency.
- Added comprehensive unittests under tests/unit/module_utils for RoleDependencyResolver.

Ref: https://chatgpt.com/share/68a519c8-8e54-800f-83c0-be38546620d9
2025-08-20 02:42:07 +02:00
kevinveenbirkenbach 7019b307c5 Optimized collabora draft 2025-08-20 01:00:20 +02:00
kevinveenbirkenbach 022800425d THE HUGE REFACTORING CALENDER WEEK 33; Optimized Matrix and during this updated variables, and implemented better reset and cleanup mode handling, also solved some initial setup bugs 2025-08-15 15:15:48 +02:00
kevinveenbirkenbach 567b1365c0 Nextcloud: async overhaul & task refactor (conditional wait, faster polling)
• Add config.performance.async.wait_for and expose as nextcloud_wait_for_async_enabled to toggle waiting for async jobs.

• Split system/admin/index maintenance into separate tasks: 02_add_missing_indices.yml, 03_admin.yml, 04_system_config.yml.

• Refactor plugin flow: rename 02_plugin→05_plugin, 03_plugin_routines→06_plugin_routines, 04_plugin_enable_and_configure→07_plugin_enable_and_configure; remove old 03_plugin_routines and 05_system.

• Harden async handling: filter async_status loops by ansible_job_id; conditionally wait only when nextcloud_wait_for_async_enabled; reduce delay to 1s.

• Reorder main.yml to run system steps before plugin setup; keep handlers flush earlier.

• env.j2: simplify get_app_conf lookups (drop extra True flag).

• vars/main.yml: add nextcloud_host_nginx_path and nextcloud_wait_for_async_enabled.

https://chatgpt.com/share/689c9d4a-1748-800f-b490-06a5a48dd831
2025-08-13 16:13:00 +02:00
kevinveenbirkenbach 4fa1c6cfbd ansible: quote file modes; keycloak: robust LDAP bind update + config cleanup
Highlights
- Quote all file modes as strings ("0755"/"0770") across multiple roles to avoid YAML octal quirks and improve portability.
- Keycloak: introduce actions.{import_realm,update_ldap_bind} feature flags and wire them via vars/config.
- Implement idempotent LDAP bind updater (tasks/03_update-ldap-bind.yml):
  * kcadm login with no_log protection,
  * fetch LDAP UserStorage component by name,
  * compare current bindDn/bindCredential and update only when changed.
- Keycloak realm import template: keep providerId="ldap" and set name from keycloak_ldap_component_name.
- Centralize Keycloak readiness check in tasks/main.yml; remove duplicate waits from 02_update_client_redirects.yml and 04_ssh_public_key.yml.
- 01_import.yml: fix typo (keycloak), quote modes, tidy spacing, and replace Jinja-in-Jinja fileglob with concatenation.
- 02_update_client_redirects.yml: correct assert fail_msg filename; keep login-first flow.
- Minor template/vars tidy-ups (spacing, comments, consistent variable usage).

Files touched (excerpt)
- roles/*/*: replace 0755/0770 → "0755"/"0770"
- roles/web-app-keycloak/config/main.yml: add actions map
- roles/web-app-keycloak/vars/main.yml: unify Keycloak vars and feature flags
- roles/web-app-keycloak/tasks/{01_import,02_update_client_redirects,03_update-ldap-bind,04_ssh_public_key,main}.yml
- roles/web-app-keycloak/templates/{docker-compose.yml.j2,import/realm.json.j2}

https://chatgpt.com/share/689bda16-b138-800f-8258-e13f6d7d8239
2025-08-13 02:20:38 +02:00
kevinveenbirkenbach 6e04ac58d2 Moved blocks to include_tasks to raise performance. Deploy was really slow 2025-08-11 12:28:31 +02:00
kevinveenbirkenbach 27973c2773 Optimized injection layer on lua base, as replace for nginx replace. Also optimized cloudflare cache deletion(no everytime for cleanup). Still CDN is required for logout mechanism via JS and Nextcloud deploy is buggy after changing from nginx to openresty. Propably some variable overwritte topic. Should be solved tomorrow. 2025-07-24 19:13:13 +02:00
kevinveenbirkenbach f62355e490 Replaced nginx native with openresty for logout injection. Right now still buggy on nextcloud and espocrm 2025-07-24 03:19:16 +02:00
kevinveenbirkenbach ad449c3b6a Adapted roles to new architecture 2025-07-17 15:39:31 +02:00
kevinveenbirkenbach af3ea9039c Restructure and cleaned up in preparation of new backup logic 2025-07-15 23:51:51 +02:00
kevinveenbirkenbach d1c8036fa4 Implemented DB Credentials Update for Nextcloud and solved bug 2025-07-14 18:41:30 +02:00
kevinveenbirkenbach 756597668c Semi bsr for applications[] to prevent heavy to debug bugs in j2 - part 1 2025-07-13 15:11:38 +02:00
kevinveenbirkenbach 96268e7161 Renamed server roles by osi they work on 2025-07-10 12:33:46 +02:00
kevinveenbirkenbach c9c73cbdb2 Decoupeld database, docker and proxy 2025-07-09 14:21:30 +02:00
kevinveenbirkenbach 575df76ec3 Shortened service- to svc- 2025-07-09 05:00:41 +02:00
kevinveenbirkenbach 66198ca1ec Shortened webserver to srv-web- 2025-07-09 04:27:58 +02:00
kevinveenbirkenbach 563d5fd528 Huge role refactoring/cleanup. Other commits will propably follow. Because some bugs will exist. Still important for longrun and also for auto docs/help/slideshow generation 2025-07-08 23:43:13 +02:00