Commit Graph

30 Commits

Author SHA1 Message Date
46174125bc Refine deploy CLI, test-deploy workflow and Ansible output
Changes:
- Update GitHub Actions test-deploy workflow to run three staged deploys (normal+debug, reset+debug, async) using inventory-generated vault password files.
- Switch Ansible stdout_callback to ansible.builtin.default and enable YAML-style result_format via callback_default.
- Refactor cli/deploy.py: typed run_ansible_playbook(), structured MODE_* handling, better error reporting, and preserved vault/interactive behaviour.
- Add unit tests for deploy CLI (bool parsing, MODE_* loading, dynamic args, validation, and ansible-playbook command construction) under tests/unit/cli/test_deploy.py.

Context: see ChatGPT conversation on 2025-12-02: https://chatgpt.com/share/692f1035-6bc4-800f-91a9-342db54e1a75
2025-12-02 20:25:26 +01:00
5b18f39ccd Extend inventory create CLI with include/ignore filters and SSL/network defaults - https://chatgpt.com/share/692edecd-54c4-800f-b01b-35cf395a60f0 2025-12-02 13:44:16 +01:00
c0e26275f8 Refactor defaults generation, credential creation, and inventory management
### Overview
This commit introduces a broad set of improvements across the defaults
generator, credential creation subsystem, inventory creation workflow,
and InventoryManager core logic.

### Major Changes
- Support empty or  config/main.yml in defaults generator and ensure that
  applications with empty configs are still included in defaults_applications.
- Add '--snippet' and '--allow-empty-plain' modes to create/credentials.py
  with non-destructive merging and correct plain-secret handling.
- Ensure empty strings for 'plain' credentials are never encrypted.
- Update InventoryManager to fully support allow_empty_plain and prevent
  accidental overwriting or encrypting existing VaultScalar or dict values.
- Add full-size implementation of cli/create/inventory.py including
  dynamic inventory building, role filtering, host_vars management, and
  parallelised credential snippet generation.
- Fix schemas (Magento, Nextcloud, OAuth2-Proxy, keyboard-color, etc.) to
  align with the new credential model and avoid test failures.
- Improve get_app_conf consistency by ensuring credentials.* paths are
  always resolvable for applications even when config/main.yml is empty.

### Added Test Coverage
- Unit tests for defaults generator handling empty configs.
- Full test suite for create/inventory.py including merge logic and
  vault-safe host_vars loading.
- Extensive tests for InventoryManager: plain-secret behavior,
  vault handling, and recursion logic.
- Update or remove outdated tests referencing old schema behaviour.

### Context
This commit is associated with a refactoring and debugging session documented here:
https://chatgpt.com/share/692ec0e1-5018-800f-b568-d09a53e9d0ee
2025-12-02 11:54:55 +01:00
26dfab147d Implement reserved username handling for users, LDAP and Keycloak
Add end-to-end support for reserved usernames and tighten CAPTCHA / Keycloak logic.

Changes:

- Makefile: rename EXTRA_USERS → RESERVED_USERNAMES and pass it as --reserved-usernames to the users defaults generator.

- cli/build/defaults/users.py: propagate  flag into generated users, add --reserved-usernames CLI option and mark listed accounts as reserved.

- Add reserved_users filter plugin with  and  helpers for Ansible templates and tasks.

- Add unit tests for reserved_users filters and the new reserved-usernames behaviour in the users defaults generator.

- group_vars/all/00_general.yml: harden RECAPTCHA_ENABLED / HCAPTCHA_ENABLED checks with default('') and explicit > 0 length checks.

- svc-db-openldap: introduce OPENLDAP_PROVISION_* flags, add OPENLDAP_PROVISION_RESERVED and OPERNLDAP_USERS to optionally exclude reserved users from provisioning.

- svc-db-openldap templates/tasks: switch role/group LDIF and user import loops to use OPERNLDAP_USERS instead of the full users dict.

- networks: assign dedicated subnet for web-app-roulette-wheel.

- web-app-keycloak vars: compute KEYCLOAK_RESERVED_USERNAMES_LIST and KEYCLOAK_RESERVED_USERNAMES_REGEX from users | reserved_usernames.

- web-app-keycloak user profile template: inject reserved-username regex into username validation pattern and improve error message, fix SSH public key attribute usage and add component name field.

- web-app-keycloak update/_update.yml: strip subComponents from component payloads before update and disable async/poll for easier debugging.

- web-app-keycloak tasks/main.yml: guard cleanup include with MODE_CLEANUP and keep reCAPTCHA update behind KEYCLOAK_RECAPTCHA_ENABLED.

- user/users defaults: mark system/service accounts (root, daemon, mail, admin, webmaster, etc.) as reserved so they cannot be chosen as login names.

- svc-prx-openresty vars: simplify OPENRESTY_CONTAINER lookup by dropping unused default parameter.

- sys-ctl-rpr-btrfs-balancer: simplify main.yml by removing the extra block wrapper.

- sys-daemon handlers: quote handler name for consistency.

Context: change set discussed and refined in ChatGPT on 2025-11-29 (Infinito.Nexus reserved usernames & Keycloak user profile flow). See conversation: https://chatgpt.com/share/692b21f5-5d98-800f-8e15-1ded49deddc9
2025-11-29 17:40:45 +01:00
9c65bd4839 Refine role dependency graph/tree builders and tests
- Refactor cli/build/graph.py to use cached metadata and dependency indices
  for faster graph generation and cleaner separation of concerns
- Refactor cli/build/tree.py to delegate per-role processing to process_role()
  and support parallel execution via ProcessPoolExecutor
- Add unit tests for graph helper functions and build_mappings()
  under tests/unit/cli/build/test_graph.py
- Add unit tests for find_roles() and process_role() behaviour
  under tests/unit/cli/build/test_tree.py
- Remove the old include_role dependency integration test which relied on the
  previous tree.json dependencies bucket

For details see ChatGPT conversation: https://chatgpt.com/share/6926b805-28a0-800f-a075-e5250aab5c4a
2025-11-26 09:20:45 +01:00
e6f4f3a6a4 feat(cli/build/defaults): ensure deterministic alphabetical sorting for applications and users
- Added sorting by application key and user key before YAML output.
- Ensures stable and reproducible file generation across runs.
- Added comprehensive unit tests verifying key order and output stability.

See: https://chatgpt.com/share/68ef4778-a848-800f-a50b-a46a3b878797
2025-10-15 09:04:39 +02:00
32bc17e0c3 Optimized whitespacing 2025-09-04 12:41:11 +02:00
751615b1a4 Changed 09_ports.yml to 10_ports.yml 2025-09-03 17:41:14 +02:00
6e538eabc8 Enhance tree builder: detect include_role dependencies from tasks/*.yml
- Added logic to scan each role’s tasks/*.yml files for include_role usage
- Supports:
  * loop/with_items with literal strings → adds each role
  * patterns with variables inside literals (e.g. svc-db-{{database_type}}) → expanded to glob and matched
  * pure variable-only names ({{var}}) → ignored
  * pure literal names → added directly
- Merges discovered dependencies under graphs["dependencies"]["include_role"]
- Added dedicated unit test covering looped includes, glob patterns, pure literals, and ignoring pure variables

See ChatGPT conversation (https://chatgpt.com/share/68a4ace0-7268-800f-bd32-b475c5c9ba1d) for context.
2025-08-19 19:00:03 +02:00
3a839cfe37 Refactor systemctl services and categories due to alarm bugs
This commit restructures systemctl service definitions and category mappings.

Motivation: Alarm-related bugs revealed inconsistencies in service and role handling.

Preparation step: lays the groundwork for fixing the alarm issues by aligning categories, roles, and service templates.
2025-08-18 13:35:43 +02:00
a1643870db Optimized auto playbook creation 2025-07-21 15:09:38 +02:00
c572d535e2 Optimized test for tree creation 2025-07-20 11:41:16 +02:00
45624037b1 Added shadow option to tree for mig 2025-07-18 19:35:44 +02:00
562603a8cd Restructured libraries 2025-07-17 16:38:20 +02:00
dea2669de2 Solved unclosed file <_io.TextIOWrapper warnings 2025-07-16 14:33:10 +02:00
c8054ffbc3 Optimized get paths and applications generation 2025-07-15 22:48:59 +02:00
b6eb73dee4 Implemented that schemas are recognized 2025-07-15 22:22:22 +02:00
adec2aed84 Renamed generate commands to build 2025-07-12 17:11:41 +02:00
25cee9a4c7 Finished Graph and Tree implementation 2025-07-11 14:33:27 +02:00
3c63936970 Different optimations 2025-07-11 05:29:06 +02:00
d43fdc63ea Optimized inventory validator for wrong groups 2025-07-11 01:34:44 +02:00
1f4dee49bc Solved last open tests 2025-07-10 23:19:44 +02:00
c160c58a5c Restructured CLI logic 2025-07-10 21:26:44 +02:00
3c3739c234 Fixed desk roles application ids 2025-07-10 14:31:09 +02:00
73329506a9 Solved bug which existed due to user structure refactoring 2025-07-09 12:31:37 +02:00
ed0cd9b8c0 Restructured users 2025-07-09 02:26:50 +02:00
22b4342300 Implemented schema/main.yml und config/main.yml file 2025-07-09 02:03:32 +02:00
563d5fd528 Huge role refactoring/cleanup. Other commits will propably follow. Because some bugs will exist. Still important for longrun and also for auto docs/help/slideshow generation 2025-07-08 23:43:13 +02:00
b9e5c3a337 Added tests for create docker function 2025-07-07 04:58:03 +02:00
a9f55579a2 Reorgenized test structure and added validation of inventory before deployment 2025-07-04 01:14:00 +02:00