Commit Graph

162 Commits

Author SHA1 Message Date
5e550ce3a3 sys-ctl-rpr-docker-soft: switch to STRICT label mode and adapt tests
- script.py now resolves docker-compose project and working_dir strictly from container labels
- removed container-name fallback logic
- adjusted sys-ctl-hlth-docker-container to include sys-ctl-rpr-docker-soft
- cleaned up sys-svc-docker dependencies
- updated unit tests to mock docker inspect and os.path.isfile for STRICT mode

Conversation: https://chatgpt.com/share/68b80927-b800-800f-a909-0fe8d110fd0e
2025-09-03 11:24:14 +02:00
7791bd8c04 Implement filter checks: ensure all defined filters are used and remove dead code
Integration tests added/updated:
- tests/integration/test_filters_usage.py: AST-based detection of filter definitions (FilterModule.filters), robust Jinja detection ({{ ... }}, {% ... %}, {% filter ... %}), plus Python call tracking; fails if a filter is used only under tests/.
- tests/integration/test_filters_are_defined.py: inverse check — every filter used in .yml/.yaml/.j2/.jinja2/.tmpl must be defined locally. Scans only inside Jinja blocks and ignores pipes inside strings (e.g., lookup('pipe', "... | grep ... | awk ...")) to avoid false positives like trusted_hosts, woff/woff2, etc.

Bug fixes & robustness:
- Build regexes without %-string formatting to avoid ValueError from literal '%' in Jinja tags.
- Strip quoted strings in usage analysis so sed/grep/awk pipes are not miscounted as filters.
- Prevent self-matches in the defining file.

Cleanup / removal of dead code:
- Removed unused filter plugins and related unit tests:
  * filter_plugins/alias_domains_map.py
  * filter_plugins/get_application_id.py
  * filter_plugins/load_configuration.py
  * filter_plugins/safe.py
  * filter_plugins/safe_join.py
  * roles/svc-db-openldap/filter_plugins/build_ldap_nested_group_entries.py
  * roles/sys-ctl-bkp-docker-2-loc/filter_plugins/dict_to_cli_args.py
  * corresponding tests under tests/unit/*
- roles/svc-db-postgres/filter_plugins/split_postgres_connections.py: dropped no-longer-needed list_postgres_roles API; adjusted tests.

Misc:
- sys-stk-front-proxy/defaults/main.yml: clarified valid vhost_flavour values (comma-separated).

Ref: https://chatgpt.com/share/68b56bac-c4f8-800f-aeef-6708dbb44199
2025-09-01 11:47:51 +02:00
94fe58b5da safe_join: raise ValueError on None parameters and update tests
Changed safe_join to raise ValueError if base or tail is None instead of returning 'None/path'.
Adjusted unit tests accordingly to expect exceptions for None inputs and kept empty-string handling valid.

Ref: https://chatgpt.com/share/68b55850-e854-800f-9702-09ea956b8dc4
2025-09-01 10:25:08 +02:00
231fd567b3 feat(frontend): rename inj roles to sys-front-*, add sys-svc-cdn, cache-busting lookup
Introduce sys-svc-cdn (cdn_paths/cdn_urls/cdn_dirs) and ensure CDN directories + latest symlink.

Rename sys-srv-web-inj-* → sys-front-inj-*; update includes/templates; serve shared/per-app CSS & JS via CDN.

Add lookup_plugins/local_mtime_qs.py for mtime-based cache busting; split CSS into default.css/bootstrap.css + optional per-app style.css.

CSP: use style-src-elem; drop unsafe-inline for styles. Services: fix SYS_SERVICE_ALL_ENABLED bool and controlled flush.

BREAKING CHANGE: role names changed; replace includes and references accordingly.

Conversation: https://chatgpt.com/share/68b55494-9ec4-800f-b559-44707029141d
2025-09-01 10:10:23 +02:00
3f8e7c1733 Refactor CSP filter:
- Move default 'unsafe-inline' for style-src and style-src-elem into get_csp_flags
- Ensure hashes are only added if 'unsafe-inline' not in final tokens
- Improve comments and structure
- Extend unit tests to cover default flags, overrides, and final-token logic
See: https://chatgpt.com/share/68b54520-5cfc-800f-9bac-45093740df78
2025-09-01 09:03:22 +02:00
3bfab9ef8e feat(filter_plugins/url_join): add query parameter support
- Support query elements starting with '?' or '&'
  * First query element normalized to '?', subsequent to '&'
  * Each query element must be exactly one 'key=value' pair
  * Query elements may only appear after path elements
  * Once query starts, no more path elements are allowed
- Extend test suite with success and failure cases for query handling

See: https://chatgpt.com/share/68b537ea-d198-800f-927a-940c4de832f2
2025-09-01 08:16:22 +02:00
f1870c07be refactor(filter_plugins/url_join): enforce mandatory scheme and raise specific AnsibleFilterError messages
Improved url_join filter:
- Requires first element to contain a valid '<scheme>://'
- Raises specific errors for None, empty list, wrong type, missing scheme,
  extra schemes in later parts, or string conversion failures
- Provides clearer error messages with index context in parts

See: https://chatgpt.com/share/68b537ea-d198-800f-927a-940c4de832f2
2025-09-01 08:06:48 +02:00
3a17b2979e Refactor CSP filters to use get_url for domain resolution and update tests to check CSP directives order-independently. See: https://chatgpt.com/share/68b49e5c-6774-800f-9d8e-a3f980799c08 2025-08-31 21:11:57 +02:00
aa2eb53776 fix(csp): always include internal CDN in script-src/connect-src and update tests accordingly
See ChatGPT conversation: https://chatgpt.com/share/68b492b8-847c-800f-82a9-fb890d4add7f
2025-08-31 20:22:05 +02:00
5f66c1a622 feat(postgres): add split_postgres_connections filter and average pool fact
Compute POSTGRES_ALLOWED_AVG_CONNECTIONS once and propagate to app roles (gitlab, mastodon, listmonk, matrix, pretix, mobilizon, openproject, discourse). Fix docker-compose postgres command (-c flags split). Add unit tests. Minor env/locale tweaks and includes.

Conversation: https://chatgpt.com/share/68b48e72-cc28-800f-9c21-270cbc17d82a
2025-08-31 20:04:14 +02:00
2aed0f97d2 Enhance timeout_start_sec_for_domains filter to accept dict, list, or str
- Updated filter to handle dict (domain map), list (flattened domains), or single str inputs.
- Prevents duplicate 'www.' prefixes by checking prefix before adding.
- Adjusted unit tests:
  * Replaced old non-dict test with invalid type tests (int, None).
  * Added explicit tests for list and string input types.

See conversation: https://chatgpt.com/share/68b1ae9a-1ac0-800f-b49d-2915386a1a23
2025-08-29 15:57:00 +02:00
f36c7831b1 Implement dynamic TimeoutStartSec filter for domains and update roles
- Added new filter plugin 'timeout_start_sec_for_domains' to calculate TimeoutStartSec based on number of domains.
- Updated sys-ctl-hlth-csp and sys-ctl-hlth-webserver tasks to use the filter.
- Removed obsolete systemctl.service.j2 in sys-ctl-hlth-csp.
- Adjusted variable naming (CURRENT_PLAY_DOMAINS_ALL etc.) in multiple roles.
- Updated srv-letsencrypt and sys-svc-certs to use uppercase vars.
- Switched pretix role to sys-stk-full-stateful and removed leftover javascript.js.
- Added unittests for the new filter under tests/unit/filter_plugins.

See conversation: https://chatgpt.com/share/68b1ae9a-1ac0-800f-b49d-2915386a1a23
2025-08-29 15:44:31 +02:00
cb66fb2978 Refactor LDAP variable schema to use top-level constant LDAP and nested ALL-CAPS keys.
- Converted group_vars/all/13_ldap.yml from lower-case to ALL-CAPS nested keys.
- Updated all roles, tasks, templates, and filter_plugins to reference LDAP.* instead of ldap.*.
- Fixed Keycloak JSON templates to properly quote Jinja variables.
- Adjusted svc-db-openldap filter plugins and unit tests to handle new LDAP structure.
- Updated integration test to only check uniqueness of TOP-LEVEL ALL-CAPS constants, ignoring nested keys.

See: https://chatgpt.com/share/68b01017-efe0-800f-a508-7d7e2f1c8c8d
2025-08-28 10:15:48 +02:00
b916173422 Renamed web-app-port-ui to web-app-desktop 2025-08-26 11:35:22 +02:00
9756a0f75f Extend repair scripts with env-file support and unit tests
- Added detect_env_file() to both sys-ctl-rpr-docker-soft and sys-ctl-rpr-docker-hard
  * prefer .env, fallback to .env/env
  * append --env-file parameter automatically
- Refactored soft script to use compose_cmd() for consistent command building
- Adjusted error recovery path in soft script to also respect env-file
- Extended unit tests for soft script to cover env-file priority and restart commands
- Added new unit tests for hard script verifying env-file priority, cwd handling,
  and --only filter logic

Ref: https://chatgpt.com/share/68ad7b30-7510-800f-8172-56f03a2f40f5
2025-08-26 11:15:59 +02:00
e417bc19bd Refactor sys-ctl-rpr-docker-soft role to use standalone Python script with argparse and unittests
- Replace Jinja2 template (script.py.j2) with raw Python script (files/script.py)
- Add argparse options: --manipulation, --manipulation-string, --timeout
- Implement timeout handling in wait_while_manipulation_running
- Update systemd ExecStart/ExecStartPre handling in tasks/01_core.yml
- Remove obsolete systemctl.service.j2 and script.py.j2 templates
- Add unittest suite under tests/unit/roles/sys-ctl-rpr-docker-soft/files/test_script.py
- Mock docker and systemctl calls in tests for safe execution

Reference: ChatGPT conversation (see https://chatgpt.com/share/68ad770b-ea84-800f-b378-559cb61fc43a)
2025-08-26 10:58:17 +02:00
b867a52471 Refactor and extend role dependency resolution:
- Introduced module_utils/role_dependency_resolver.py with full support for include_role, import_role, meta dependencies, and run_after.
- Refactored cli/build/tree.py to use RoleDependencyResolver (added toggles for include/import/dependencies/run_after).
- Extended filter_plugins/canonical_domains_map.py with optional 'recursive' mode (ignores run_after by design).
- Updated roles/web-app-nextcloud to properly include Collabora dependency.
- Added comprehensive unittests under tests/unit/module_utils for RoleDependencyResolver.

Ref: https://chatgpt.com/share/68a519c8-8e54-800f-83c0-be38546620d9
2025-08-20 02:42:07 +02:00
6e538eabc8 Enhance tree builder: detect include_role dependencies from tasks/*.yml
- Added logic to scan each role’s tasks/*.yml files for include_role usage
- Supports:
  * loop/with_items with literal strings → adds each role
  * patterns with variables inside literals (e.g. svc-db-{{database_type}}) → expanded to glob and matched
  * pure variable-only names ({{var}}) → ignored
  * pure literal names → added directly
- Merges discovered dependencies under graphs["dependencies"]["include_role"]
- Added dedicated unit test covering looped includes, glob patterns, pure literals, and ignoring pure variables

See ChatGPT conversation (https://chatgpt.com/share/68a4ace0-7268-800f-bd32-b475c5c9ba1d) for context.
2025-08-19 19:00:03 +02:00
7ef20474a0 Renamed sys-ctl-cln-backups to sys-ctl-cln-bkps 2025-08-19 12:15:33 +02:00
73b7d2728e Solved timer bug 2025-08-19 00:33:00 +02:00
b9461026a6 refactor: improve get_service_name suffix handling and handler usage
- Updated filter_plugins/get_service_name.py:
  * Default suffix handling: auto-select .service (no '@') or .timer (with '@')
  * Explicit False disables suffix entirely
  * Explicit string suffix still supported
- Updated sys-systemctl handler to use new filter instead of SYS_SERVICE_SUFFIX
- Extended unit tests to cover new suffix behavior

Ref: https://chat.openai.com/share/8c2de9e6-daa0-44dd-ae13-d7a7d8d8b6d9
2025-08-18 22:36:31 +02:00
bf63e01b98 refactor(systemd-services): migrate SYS_SERVICE_SUFFIX usage to get_service_name filter
Replaced all hardcoded service name concatenations with the new get_service_name filter.
This ensures consistency, proper lowercase formatting, and correct handling of '@' suffixed units.

Added unittests for the filter (normal, custom suffix, '@'-units, and lowercase normalization).

Context: see ChatGPT discussion https://chatgpt.com/share/68a38beb-b9bc-800f-b7ed-cdd2b64b2604
2025-08-18 22:24:33 +02:00
2569abc0be Refactor systemctl services and timers
- Unified service templates into generic systemctl templates
- Introduced reusable filter plugins for script path handling
- Updated path variables and service/timer definitions
- Migrated roles (backup, cleanup, repair, etc.) to use systemctl role
- Added sys-daemon role for core systemd cleanup
- Simplified timer handling via sys-timer role

Note: This is a large refactor and some errors may still exist. Further testing and adjustments will be needed.
2025-08-18 21:22:16 +02:00
3a839cfe37 Refactor systemctl services and categories due to alarm bugs
This commit restructures systemctl service definitions and category mappings.

Motivation: Alarm-related bugs revealed inconsistencies in service and role handling.

Preparation step: lays the groundwork for fixing the alarm issues by aligning categories, roles, and service templates.
2025-08-18 13:35:43 +02:00
29f50da226 Add custom Ansible filter plugin get_category_entries
This commit introduces a new Ansible filter plugin named
'get_category_entries', which returns all role names under the
roles/ directory that start with a given prefix.

Additionally, unit tests (unittest framework) have been added under
tests/unit/filterplugins/ to ensure correct behavior, including:

- Returns empty list when roles/ directory is missing
- Correctly filters and sorts by prefix
- Ignores non-directory entries
- Supports custom roles_path argument
- Returns all roles when prefix is empty

Reference: https://chatgpt.com/share/68a2f1ab-1fe8-800f-b22a-28c1c95802c2
2025-08-18 11:27:26 +02:00
0de26fa6c7 Solved bug existed due to difference between mailu domain and hostname difference. also refactored during this to find the bug 2025-08-16 14:29:07 +02:00
f0b323afee Added auto snippet for webserver injection 2025-08-16 01:31:49 +02:00
cc2c1dc730 Renamed injection services 2025-08-16 00:01:46 +02:00
3b4821f7e7 Solved missing logout injection bug and refactored srv-web-7-7-inj-compose 2025-08-15 23:55:19 +02:00
4a65a254ae replaced port-ui-desktop with desktop to make it more speakable 2025-08-14 11:45:08 +02:00
5e00deea19 Implemented desktop csp policies 2025-08-14 11:40:09 +02:00
b5d8ac5462 Reactivated keycloak docker and webserver tasks and implemented correct logic for element and synapse redirect handling 2025-08-11 02:21:02 +02:00
6e8ae793e3 Added auto setting for redirect urls for keycloak clients. Element and Synapse still need to be mapped 2025-08-11 00:17:18 +02:00
9f865dd215 Removed catetory domain präfix from redirect domains 2025-08-08 09:47:31 +02:00
5a0684fa2d Adapted CSP to new server dict structure 2025-08-07 13:00:52 +02:00
9228d51e86 Restructured server config 2025-08-07 11:31:06 +02:00
27973c2773 Optimized injection layer on lua base, as replace for nginx replace. Also optimized cloudflare cache deletion(no everytime for cleanup). Still CDN is required for logout mechanism via JS and Nextcloud deploy is buggy after changing from nginx to openresty. Propably some variable overwritte topic. Should be solved tomorrow. 2025-07-24 19:13:13 +02:00
f62355e490 Replaced nginx native with openresty for logout injection. Right now still buggy on nextcloud and espocrm 2025-07-24 03:19:16 +02:00
4b9e7dd3b7 Implemented universal logout 2025-07-22 13:14:06 +02:00
b1bf7aaba5 Fixed BBB stuff 2025-07-21 15:10:05 +02:00
a1643870db Optimized auto playbook creation 2025-07-21 15:09:38 +02:00
257d0c4673 Set entity name as default domain instead of application_id 2025-07-20 15:44:28 +02:00
c572d535e2 Optimized test for tree creation 2025-07-20 11:41:16 +02:00
8ccfb1dfbe Added icon to mig 2025-07-19 16:26:10 +02:00
85195e01f9 Activated loading of env depending on if it exist 2025-07-18 19:40:34 +02:00
45624037b1 Added shadow option to tree for mig 2025-07-18 19:35:44 +02:00
562603a8cd Restructured libraries 2025-07-17 16:38:20 +02:00
ad449c3b6a Adapted roles to new architecture 2025-07-17 15:39:31 +02:00
725fea1169 Solved database credentials bug 2025-07-17 06:32:53 +02:00
70f7953027 Added backup value key mapper 2025-07-16 20:03:00 +02:00