Commit Graph
72 Commits
Author SHA1 Message Date
kevinveenbirkenbach d2e88b6a8c feat(cli): eliminate runpy warnings for inventory + container entrypoints
* Remove legacy **main** re-export wrappers to avoid `runpy` "already in sys.modules" warnings
* Introduce `cli.deploy.container.command` as the stable implementation module
* Reduce `cli.deploy.container.__main__` to a minimal entrypoint delegating to `command.main`
* Re-export container helpers from package `__init__` without importing `__main__`
* Update unit tests to import/patch the new command module and keep CLI dispatch stable

https://chatgpt.com/share/69484a22-60dc-800f-a23d-068d7c45fe37
2025-12-21 20:27:26 +01:00
kevinveenbirkenbach 1fe0a420fa fix(cli): avoid runpy warnings by moving logic into command modules
Stop importing __main__ from package __init__.py to prevent runpy RuntimeWarning when using `python -m`.

- Move CLI logic into command.py modules
- Keep __main__.py as thin wrappers
- Update unit tests to import command modules directly

https://chatgpt.com/share/6947fb8e-550c-800f-8323-e36f92cb5bba
2025-12-21 14:52:09 +01:00
kevinveenbirkenbach 71c541ee05 test(cli): use unittest.mock.patch consistently
- Remove direct imports of unittest.mock alias
- Use unittest.mock.patch via unittest for all test decorators
- Keep behavior identical while simplifying imports
2025-12-21 11:10:03 +01:00
kevinveenbirkenbach e23187bccb refactor(deploy): remove skip-tests option from dedicated deploy
- Drop --skip-tests CLI flag and runner parameter
- Remove test phase execution from deploy workflow
- Update unit tests to match simplified deploy flow

https://chatgpt.com/share/69473607-22e4-800f-845b-a628677fb8b9
2025-12-21 00:49:21 +01:00
kevinveenbirkenbach 103659a534 refactor(deploy): modularize dedicated deploy CLI and fix make cwd issue
- Split dedicated deploy logic into focused modules (apps, modes, paths, proc, runner)
- Ensure all make targets run from the repository root via explicit cwd handling
- Fix "make: No rule to make target" errors caused by incorrect working directory
- Slim down __main__ to a thin CLI entrypoint
- Rework unit tests into one file per module and expand coverage

https://chatgpt.com/share/69472632-9740-800f-962c-bcb4474963d6
2025-12-20 23:41:41 +01:00
kevinveenbirkenbach a354992f27 refactor(create-inventory): modularize CLI, stabilize path handling, add unit tests
- Split create-inventory logic into focused modules (cli, project, filters, host_vars, credentials, etc.)
- Replace fragile path construction with python -m module execution
- Add robust project root detection
- Centralize password generation
- Improve test coverage with per-module unit tests
- Preserve existing inventory and host_vars behavior

https://chatgpt.com/share/69471157-79c0-800f-8dec-25c0f8e38f03
2025-12-20 22:12:44 +01:00
kevinveenbirkenbach 3da6e3a23e refactor(cli): modularize dispatcher and improve command discovery
https://chatgpt.com/share/6947052d-76fc-800f-9db3-98de6894ceb0
2025-12-20 21:20:47 +01:00
kevinveenbirkenbach 2e6d973839 fix(build): align CLI module invocations and stabilize include generation
- Switch Makefile targets to direct module execution (cli.build.*)
- Remove obsolete flags and normalize tree invocation
- Minor formatting cleanups for readability and test stability
- Keep inventory and include paths consistent after package layout migration

https://chatgpt.com/share/6946eac4-eb48-800f-becb-f08420ab03a6
2025-12-20 19:28:14 +01:00
kevinveenbirkenbach bb8c05a653 Replaced tree.py by tree/__main__.py 2025-12-20 18:38:52 +01:00
kevinveenbirkenbach 102735dba4 Renamed inventory.py to inventory/__main__.py 2025-12-20 18:30:20 +01:00
kevinveenbirkenbach 993ed8addd Renamed Makefile *.py includes 2025-12-20 18:28:37 +01:00
kevinveenbirkenbach 0ea828ab6e cleanup: unify git cleanup, simplify deploy flow, fix dockerignore
- Remove clean-keep-logs target and enforce git-repo guard in Makefile
- Always use `make clean` during deploy cleanup with live output
- Add shared subprocess runner for consistent stdout/stderr handling
- Align unit tests with simplified cleanup semantics
- Fix .dockerignore to ignore build artifacts and Ruff cache correctly

https://chatgpt.com/share/6945784a-daac-800f-bc02-d7e8d3087376
2025-12-19 17:07:33 +01:00
kevinveenbirkenbach f775ae9dfa ci(deploy): run deploy tests with --missing and docker-based invokable excludes 2025-12-19 16:07:59 +01:00
kevinveenbirkenbach a38f36837f style: reformat code for readability and lint compliance 2025-12-18 14:08:01 +01:00
kevinveenbirkenbachandCopilot Autofix powered by AI 1188bd51f3 Potential fix for code scanning alert no. 60: Module is imported with 'import' and 'import from'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-18 13:59:09 +01:00
kevinveenbirkenbach b512d6c4b6 Solved mock path bugs 2025-12-18 13:03:12 +01:00
kevinveenbirkenbach c2bb81a126 Merge pull request #13 from kevinveenbirkenbach/alert-autofix-61
Potential fix for code scanning alert no. 61: Module is imported with 'import' and 'import from'
2025-12-18 12:48:59 +01:00
kevinveenbirkenbach ffe966324c Merge pull request #14 from kevinveenbirkenbach/alert-autofix-59
Potential fix for code scanning alert no. 59: Module is imported with 'import' and 'import from'
2025-12-18 12:48:04 +01:00
kevinveenbirkenbachandCopilot Autofix powered by AI 1d381f9fe4 Potential fix for code scanning alert no. 80: File is not always closed
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-18 12:45:12 +01:00
kevinveenbirkenbach c1bf8035f7 Solved missing import bug 2025-12-18 12:38:36 +01:00
kevinveenbirkenbach 908d391359 ruff format . 2025-12-18 12:26:34 +01:00
kevinveenbirkenbachandCopilot Autofix powered by AI f7b2739b8e Potential fix for code scanning alert no. 63: Module is imported with 'import' and 'import from'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-18 12:00:56 +01:00
kevinveenbirkenbachandCopilot Autofix powered by AI 4bc29cd52c Potential fix for code scanning alert no. 59: Module is imported with 'import' and 'import from'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-18 10:34:11 +01:00
kevinveenbirkenbachandCopilot Autofix powered by AI a6a56a7cc3 Potential fix for code scanning alert no. 61: Module is imported with 'import' and 'import from'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-18 10:33:49 +01:00
kevinveenbirkenbachandCopilot Autofix powered by AI 9a6387a0c2 Potential fix for code scanning alert no. 81: File is not always closed
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-18 10:28:04 +01:00
kevinveenbirkenbach dc846b0672 refactor(cli): move main entrypoint to cli package and standardize execution
- Move main.py to cli/__main__.py and expose via console script
- Switch Makefile execution to `python -m cli`
- Remove fragile Docker symlink to main.py
- Ensure infinito console script is available via venv PATH
- Update tests and documentation to reflect new CLI structure
- Improve logging location and permissions (XDG_STATE_HOME)
- Handle Ctrl+C gracefully with POSIX-compliant exit code (130)

https://chatgpt.com/share/6942f140-fb8c-800f-a13a-641ebf1c644e
2025-12-17 19:06:49 +01:00
kevinveenbirkenbach 7123841fb4 style: apply ruff autoformat 2025-12-17 16:37:21 +01:00
kevinveenbirkenbach c549febe8a Autofix ruff F401 2025-12-17 14:41:42 +01:00
kevinveenbirkenbach 176c8799ce lint(ruff): fix E402 and related import issues
- Fix E402 by removing sys.path hacks and ensuring imports are at module top level
- Replace ad-hoc imports with proper package imports (filter_plugins, lookup_plugins, module_utils, cli)
- Make lookup plugins importable via package paths (e.g. lookup_plugins.application_gid)
- Clean up test imports to rely on installed package layout instead of sys.path manipulation
- Partially fix F821 caused by missing/invalid imports

This establishes a consistent, ruff-compliant import architecture across CLI, filters, plugins, and tests.
2025-12-17 14:36:29 +01:00
kevinveenbirkenbach 9db66f6f98 Ruff autofix 2025-12-17 12:56:47 +01:00
kevinveenbirkenbach 3123ac4a08 Fixed Unit Tests https://chatgpt.com/share/6941cd96-81b4-800f-9017-2e4bcdc4bd52 2025-12-16 22:22:47 +01:00
kevinveenbirkenbach 14548cbc52 Replaced false make messy-build by make setup 2025-12-16 20:51:37 +01:00
kevinveenbirkenbach d0882433c8 Refactor setup workflow and make install robust via virtualenv
- Introduce a dedicated Python virtualenv (deps target) and run all setup scripts through it
- Fix missing PyYAML errors in clean, CI, and Nix environments
- Refactor build defaults into cli/setup for clearer semantics
- Make setup deterministic and independent from system Python
- Replace early Makefile shell expansion with runtime evaluation
- Rename messy-test to test-messy and update deploy logic and tests accordingly
- Keep setup and test targets consistent across Makefile, CLI, and unit tests

https://chatgpt.com/share/693de226-00ac-800f-8cbd-06552b2f283c
2025-12-13 23:00:13 +01:00
kevinveenbirkenbach 0d18d86243 Add --vars support to inventory creation, implement deep JSON overrides for host_vars, and update CI workflow to pass MASK_CREDENTIALS_IN_LOGS=false.
Includes:
- New apply_vars_overrides() with deep merge logic
- New --vars CLI argument in cli/create/inventory.py
- Added unit tests for vars handling in test_inventory.py
- Updated test-deploy workflow to pass --vars in all deploy phases

Ref: ChatGPT conversation https://chatgpt.com/share/69320f49-6c00-800f-8875-49d36935ae3a
2025-12-04 23:48:43 +01:00
kevinveenbirkenbach bf0134b9c5 Extend inventory creation with --authorized-keys support, unify PATH_ADMINISTRATOR_HOME resolution via group_vars, update CI workflow to pass dummy SSH key, and add full unit test coverage for new features.
See details: https://chatgpt.com/share/6931f775-12d4-800f-9ccb-e6ce52097f9c
2025-12-04 22:06:10 +01:00
kevinveenbirkenbach d0aac64c67 Refactor CI container deploy wrapper: add modes, rebuild/no-cache flags, and transparent inventory/deploy arg forwarding; update tests accordingly (see ChatGPT conversation: https://chatgpt.com/share/6931e58c-2ddc-800f-88d2-f7887ec13e25) 2025-12-04 20:49:04 +01:00
kevinveenbirkenbach f2ace362bc Refactor container deploy pipeline:
- Replace inline shell scripts with Python-driven orchestration
- Introduce inner dockerd started via detached docker run
- Add docker exec–based inventory creation and dedicated deploy execution
- Add automatic vault password generation for CI
- Update GitHub Actions workflow to use new container deploy wrapper
- Add complete unit test suite for container deploy behavior
- Fix CLI argument forwarding and ensure single docker run call

Conversation reference:
https://chatgpt.com/share/6931c45d-4e40-800f-852f-6c9b1f7dc281
2025-12-04 18:27:15 +01:00
kevinveenbirkenbach 8c64f91a6d Refactor deploy CLI into cli/deploy/dedicated.py, add package inits, and update unit tests (see ChatGPT conversation: https://chatgpt.com/share/6931af9e-dad0-800f-9a8f-6d01c373de87) 2025-12-04 16:59:13 +01:00
kevinveenbirkenbach b914fb9789 Refactor CLI filters: rename --ignore to --exclude and update all related logic and tests
- Updated CLI argument parsing to use --exclude instead of --ignore.
- Adjusted help texts, comments, and error messages accordingly.
- Updated role filtering logic and references (include → exclude).
- Added new unit tests for parse_roles_list(), filter_inventory_by_include(), and filter_inventory_by_ignore().
- Improved wording and consistency in docstrings.

This change is part of the refactoring required for the Ansible 2.18 → 2.20 upgrade, ensuring naming clarity and avoiding confusion with Python's 'ignore' semantics.

Conversation reference: https://chatgpt.com/share/69307ef2-1fb4-800f-a2ec-d56020019269
2025-12-03 19:18:39 +01:00
kevinveenbirkenbach d12f7a10a6 Add optional --become-password support with automatic Vault encryption
- Implement ensure_become_password() to handle explicit, generated, and existing become passwords
- Integrate VaultHandler for encrypted ansible_become_password storage
- Add CLI parameter --become-password to inventory creation workflow
- Ensure backwards compatibility: existing passwords remain untouched unless explicitly overridden
- Add unit test verifying non-overwrite behaviour when no password is provided
- Part of migration and refactoring for Ansible 2.20 upgrade

Reference: https://chatgpt.com/share/69301a6d-e920-800f-b19c-e5ca7c3bdd24
2025-12-03 12:09:47 +01:00
kevinveenbirkenbach e22893bdcb Optimized test_deploy.sh for the new stream 2025-12-03 11:12:32 +01:00
kevinveenbirkenbach f06460db16 Fix: Automatically set ansible_connection=local for localhost in inventory generator
Adds automatic local Ansible connection settings and new unit test for localhost.
Details: https://chatgpt.com/share/692f7649-d35c-800f-948f-19d45cb93512
2025-12-03 00:29:29 +01:00
kevinveenbirkenbach 46174125bc Refine deploy CLI, test-deploy workflow and Ansible output
Changes:
- Update GitHub Actions test-deploy workflow to run three staged deploys (normal+debug, reset+debug, async) using inventory-generated vault password files.
- Switch Ansible stdout_callback to ansible.builtin.default and enable YAML-style result_format via callback_default.
- Refactor cli/deploy.py: typed run_ansible_playbook(), structured MODE_* handling, better error reporting, and preserved vault/interactive behaviour.
- Add unit tests for deploy CLI (bool parsing, MODE_* loading, dynamic args, validation, and ansible-playbook command construction) under tests/unit/cli/test_deploy.py.

Context: see ChatGPT conversation on 2025-12-02: https://chatgpt.com/share/692f1035-6bc4-800f-91a9-342db54e1a75
2025-12-02 20:25:26 +01:00
kevinveenbirkenbach 5b18f39ccd Extend inventory create CLI with include/ignore filters and SSL/network defaults - https://chatgpt.com/share/692edecd-54c4-800f-b01b-35cf395a60f0 2025-12-02 13:44:16 +01:00
kevinveenbirkenbach c0e26275f8 Refactor defaults generation, credential creation, and inventory management
### Overview
This commit introduces a broad set of improvements across the defaults
generator, credential creation subsystem, inventory creation workflow,
and InventoryManager core logic.

### Major Changes
- Support empty or  config/main.yml in defaults generator and ensure that
  applications with empty configs are still included in defaults_applications.
- Add '--snippet' and '--allow-empty-plain' modes to create/credentials.py
  with non-destructive merging and correct plain-secret handling.
- Ensure empty strings for 'plain' credentials are never encrypted.
- Update InventoryManager to fully support allow_empty_plain and prevent
  accidental overwriting or encrypting existing VaultScalar or dict values.
- Add full-size implementation of cli/create/inventory.py including
  dynamic inventory building, role filtering, host_vars management, and
  parallelised credential snippet generation.
- Fix schemas (Magento, Nextcloud, OAuth2-Proxy, keyboard-color, etc.) to
  align with the new credential model and avoid test failures.
- Improve get_app_conf consistency by ensuring credentials.* paths are
  always resolvable for applications even when config/main.yml is empty.

### Added Test Coverage
- Unit tests for defaults generator handling empty configs.
- Full test suite for create/inventory.py including merge logic and
  vault-safe host_vars loading.
- Extensive tests for InventoryManager: plain-secret behavior,
  vault handling, and recursion logic.
- Update or remove outdated tests referencing old schema behaviour.

### Context
This commit is associated with a refactoring and debugging session documented here:
https://chatgpt.com/share/692ec0e1-5018-800f-b568-d09a53e9d0ee
2025-12-02 11:54:55 +01:00
kevinveenbirkenbach 26dfab147d Implement reserved username handling for users, LDAP and Keycloak
Add end-to-end support for reserved usernames and tighten CAPTCHA / Keycloak logic.

Changes:

- Makefile: rename EXTRA_USERS → RESERVED_USERNAMES and pass it as --reserved-usernames to the users defaults generator.

- cli/build/defaults/users.py: propagate  flag into generated users, add --reserved-usernames CLI option and mark listed accounts as reserved.

- Add reserved_users filter plugin with  and  helpers for Ansible templates and tasks.

- Add unit tests for reserved_users filters and the new reserved-usernames behaviour in the users defaults generator.

- group_vars/all/00_general.yml: harden RECAPTCHA_ENABLED / HCAPTCHA_ENABLED checks with default('') and explicit > 0 length checks.

- svc-db-openldap: introduce OPENLDAP_PROVISION_* flags, add OPENLDAP_PROVISION_RESERVED and OPERNLDAP_USERS to optionally exclude reserved users from provisioning.

- svc-db-openldap templates/tasks: switch role/group LDIF and user import loops to use OPERNLDAP_USERS instead of the full users dict.

- networks: assign dedicated subnet for web-app-roulette-wheel.

- web-app-keycloak vars: compute KEYCLOAK_RESERVED_USERNAMES_LIST and KEYCLOAK_RESERVED_USERNAMES_REGEX from users | reserved_usernames.

- web-app-keycloak user profile template: inject reserved-username regex into username validation pattern and improve error message, fix SSH public key attribute usage and add component name field.

- web-app-keycloak update/_update.yml: strip subComponents from component payloads before update and disable async/poll for easier debugging.

- web-app-keycloak tasks/main.yml: guard cleanup include with MODE_CLEANUP and keep reCAPTCHA update behind KEYCLOAK_RECAPTCHA_ENABLED.

- user/users defaults: mark system/service accounts (root, daemon, mail, admin, webmaster, etc.) as reserved so they cannot be chosen as login names.

- svc-prx-openresty vars: simplify OPENRESTY_CONTAINER lookup by dropping unused default parameter.

- sys-ctl-rpr-btrfs-balancer: simplify main.yml by removing the extra block wrapper.

- sys-daemon handlers: quote handler name for consistency.

Context: change set discussed and refined in ChatGPT on 2025-11-29 (Infinito.Nexus reserved usernames & Keycloak user profile flow). See conversation: https://chatgpt.com/share/692b21f5-5d98-800f-8e15-1ded49deddc9
2025-11-29 17:40:45 +01:00
kevinveenbirkenbach 9c65bd4839 Refine role dependency graph/tree builders and tests
- Refactor cli/build/graph.py to use cached metadata and dependency indices
  for faster graph generation and cleaner separation of concerns
- Refactor cli/build/tree.py to delegate per-role processing to process_role()
  and support parallel execution via ProcessPoolExecutor
- Add unit tests for graph helper functions and build_mappings()
  under tests/unit/cli/build/test_graph.py
- Add unit tests for find_roles() and process_role() behaviour
  under tests/unit/cli/build/test_tree.py
- Remove the old include_role dependency integration test which relied on the
  previous tree.json dependencies bucket

For details see ChatGPT conversation: https://chatgpt.com/share/6926b805-28a0-800f-a075-e5250aab5c4a
2025-11-26 09:20:45 +01:00
kevinveenbirkenbach e6f4f3a6a4 feat(cli/build/defaults): ensure deterministic alphabetical sorting for applications and users
- Added sorting by application key and user key before YAML output.
- Ensures stable and reproducible file generation across runs.
- Added comprehensive unit tests verifying key order and output stability.

See: https://chatgpt.com/share/68ef4778-a848-800f-a50b-a46a3b878797
2025-10-15 09:04:39 +02:00
kevinveenbirkenbach 32bc17e0c3 Optimized whitespacing 2025-09-04 12:41:11 +02:00
kevinveenbirkenbach 751615b1a4 Changed 09_ports.yml to 10_ports.yml 2025-09-03 17:41:14 +02:00