- Replace legacy utils/run_once.yml with the new helpers utils/once_flag.yml and utils/once_finalize.yml
- Introduce utils/compose_up.yml to ensure docker-compose stacks are up and to flush handlers safely without coupling to run-once flags
- Migrate all affected roles (desk-*, dev-*, sys-ctl-*, sys-svc-*, web-app-*, web-svc-*, util-*) to the new run-once helpers
- Rework sys-svc-msmtp to auto-load Mailu once per deploy, check reachability, and reuse the running stack instead of requiring multiple playbook passes
- Adjust web-app-mailu to integrate cert deployment, handler flushing, and run-once handling so Mailu is fully initialized in a single deploy
- Improve Matomo, CDN, logout and CSP/health-check related roles to cooperate with the new compose_up / once_* pattern
- Simplify alarm/backup/timer/service orchestration (sys-ctl-alm-*, sys-bkp-provider, sys-timer-cln-bkps, etc.) by moving run-once logic into dedicated 01_core.yml files
- Update integration tests so utils/once_flag.yml and utils/once_finalize.yml are recognised as valid run-once providers, keeping the global run_once_* guarantees consistent
- Align frontend injection and service dependencies so Mastodon- and Mailu-related services can be brought up coherently within a single deployment cycle rather than several iterations
- Moved database seed variables from vars/main.yml to task-level include in BigBlueButton
- Simplified core include logic in sys-ctl-bkp-docker-2-loc
- Ensured clean conditional for BKP_DOCKER_2_LOC_DB_ENABLED
See: https://chatgpt.com/share/68f216f7-62d8-800f-94e3-c82e4418e51b (deutsch)
- Added MODE_BACKUP to trigger backup before the rest of the deployment
- sys-ctl-bkp-docker-2-loc: force linear sync and force flush when MODE_BACKUP is true
- Unified name resolution via system_service_name across handlers and tasks
- Introduced system_service_force_linear_sync and system_service_force_flush (rename from system_force_flush)
- Drive async/poll via system_service_async/system_service_poll using omit when disabled
- Propagated per-role overrides (cleanup, repair, cert tasks) for clarity and safety
- Minor formatting and consistency cleanups
Why: Ensure the backup runs before the deployment routine to safeguard data integrity.
Refs: Conversation https://chatgpt.com/share/68de4c41-b6e4-800f-85cd-ce6949097b5e
Signed-off-by: Kevin Veen-Birkenbach <kevin@veen.world>
Integration tests added/updated:
- tests/integration/test_filters_usage.py: AST-based detection of filter definitions (FilterModule.filters), robust Jinja detection ({{ ... }}, {% ... %}, {% filter ... %}), plus Python call tracking; fails if a filter is used only under tests/.
- tests/integration/test_filters_are_defined.py: inverse check — every filter used in .yml/.yaml/.j2/.jinja2/.tmpl must be defined locally. Scans only inside Jinja blocks and ignores pipes inside strings (e.g., lookup('pipe', "... | grep ... | awk ...")) to avoid false positives like trusted_hosts, woff/woff2, etc.
Bug fixes & robustness:
- Build regexes without %-string formatting to avoid ValueError from literal '%' in Jinja tags.
- Strip quoted strings in usage analysis so sed/grep/awk pipes are not miscounted as filters.
- Prevent self-matches in the defining file.
Cleanup / removal of dead code:
- Removed unused filter plugins and related unit tests:
* filter_plugins/alias_domains_map.py
* filter_plugins/get_application_id.py
* filter_plugins/load_configuration.py
* filter_plugins/safe.py
* filter_plugins/safe_join.py
* roles/svc-db-openldap/filter_plugins/build_ldap_nested_group_entries.py
* roles/sys-ctl-bkp-docker-2-loc/filter_plugins/dict_to_cli_args.py
* corresponding tests under tests/unit/*
- roles/svc-db-postgres/filter_plugins/split_postgres_connections.py: dropped no-longer-needed list_postgres_roles API; adjusted tests.
Misc:
- sys-stk-front-proxy/defaults/main.yml: clarified valid vhost_flavour values (comma-separated).
Ref: https://chatgpt.com/share/68b56bac-c4f8-800f-aeef-6708dbb44199
- Added '| sort' to all service group lists and backup routine lists to ensure
deterministic ordering and stable checksums across Ansible runs.
- Adjusted systemctl templates to use a single service variable
('SYS_SERVICE_BACKUP_RMT_2_LOC') instead of rejecting dynamic list entries,
making the ignore logic simpler and more predictable.
- Fixed minor whitespace inconsistencies in Jinja templates to avoid
unnecessary changes.
This change was made to prevent spurious 'changed' states in Ansible caused by
non-deterministic list order and to reduce complexity in service definitions.
See discussion: https://chatgpt.com/share/68a74c20-6300-800f-a44e-da43ae2f3dea
- Swap role includes: sys-systemctl → sys-service in all roles
- Rename variables everywhere: systemctl_* → system_service_* (incl. systemctl_id → system_service_id)
- Templates: ExecStart now uses {{ system_service_script_exec }}; add optional RuntimeMaxSec via SYS_SERVICE_DEFAULT_RUNTIME
- Move SYS_SERVICE defaults into roles/sys-service/defaults (remove SYS_SERVICE_ALL_ENABLED & SYS_SERVICE_DEFAULT_STATE from group_vars/07_services.yml)
- Tidy group_vars/all/08_timer.yml formatting
- Introduce roles/sys-daemon:
- default manager timeouts (timeouts.conf)
- optional purge of /etc/systemd/system.conf.d
- validation via systemd-analyze verify
- handlers for daemon-reload & daemon-reexec
- Refactor sys-timer to system_service_* variables (docs and templates updated)
- Move filter_plugins/filetype.py under sys-service
- Update meta/README to point to official systemd docs
- Touch many roles (backup/cleanup/health/repair/certs/nginx/csp/wireguard/ssd-hdd/keyboard/update-docker/alarm compose/email/telegram/etc.) to new naming
BREAKING CHANGE:
- Role path/name change: use `sys-service` instead of `sys-systemctl`
- All `systemctl_*` vars are now `system_service_*` (e.g., on_calendar, state, timer_enabled, script_exec, id)
- If you have custom templates, adopt RuntimeMaxSec and new variable names
Chat context: https://chatgpt.com/share/68a47568-312c-800f-af3f-e98575446327
Replaced all hardcoded service name concatenations with the new get_service_name filter.
This ensures consistency, proper lowercase formatting, and correct handling of '@' suffixed units.
Added unittests for the filter (normal, custom suffix, '@'-units, and lowercase normalization).
Context: see ChatGPT discussion https://chatgpt.com/share/68a38beb-b9bc-800f-b7ed-cdd2b64b2604
- Unified service templates into generic systemctl templates
- Introduced reusable filter plugins for script path handling
- Updated path variables and service/timer definitions
- Migrated roles (backup, cleanup, repair, etc.) to use systemctl role
- Added sys-daemon role for core systemd cleanup
- Simplified timer handling via sys-timer role
Note: This is a large refactor and some errors may still exist. Further testing and adjustments will be needed.
This commit restructures systemctl service definitions and category mappings.
Motivation: Alarm-related bugs revealed inconsistencies in service and role handling.
Preparation step: lays the groundwork for fixing the alarm issues by aligning categories, roles, and service templates.
