Commit Graph
918 Commits
Author SHA1 Message Date
kevinveenbirkenbach 20d03575e7 fix(deploy): mount project root instead of cli/deploy in CI container
https://chatgpt.com/share/6946f9a9-bf94-800f-adab-f1baf49e1e12
2025-12-20 20:37:10 +01:00
kevinveenbirkenbach 2e6d973839 fix(build): align CLI module invocations and stabilize include generation
- Switch Makefile targets to direct module execution (cli.build.*)
- Remove obsolete flags and normalize tree invocation
- Minor formatting cleanups for readability and test stability
- Keep inventory and include paths consistent after package layout migration

https://chatgpt.com/share/6946eac4-eb48-800f-becb-f08420ab03a6
2025-12-20 19:28:14 +01:00
kevinveenbirkenbach a8a4948e39 **fix(cli): restore test patchability after package migration**
* Reintroduced proxy wrappers in cli subcommands to keep unit/integration tests patchable
* Synced test monkeypatches with **main** implementations (tree, deploy, fix, users)
* Fixed ROLES_DIR resolution to repo root after package refactor
* Re-exported private helpers required by tests
* Made cli/setup/users script-safe when executed directly

https://chatgpt.com/share/6946e285-2658-800f-b8f3-b68d66b91be9
2025-12-20 18:52:56 +01:00
kevinveenbirkenbach bb8c05a653 Replaced tree.py by tree/__main__.py 2025-12-20 18:38:52 +01:00
kevinveenbirkenbach 102735dba4 Renamed inventory.py to inventory/__main__.py 2025-12-20 18:30:20 +01:00
kevinveenbirkenbach 993ed8addd Renamed Makefile *.py includes 2025-12-20 18:28:37 +01:00
kevinveenbirkenbach 5c6afd2cf6 p2pkg executed 2025-12-20 18:25:37 +01:00
kevinveenbirkenbach 0ea828ab6e cleanup: unify git cleanup, simplify deploy flow, fix dockerignore
- Remove clean-keep-logs target and enforce git-repo guard in Makefile
- Always use `make clean` during deploy cleanup with live output
- Add shared subprocess runner for consistent stdout/stderr handling
- Align unit tests with simplified cleanup semantics
- Fix .dockerignore to ignore build artifacts and Ruff cache correctly

https://chatgpt.com/share/6945784a-daac-800f-bc02-d7e8d3087376
2025-12-19 17:07:33 +01:00
kevinveenbirkenbach f775ae9dfa ci(deploy): run deploy tests with --missing and docker-based invokable excludes 2025-12-19 16:07:59 +01:00
kevinveenbirkenbach aea82e8080 ci(deploy): use host Docker socket and local infinito image for deploy tests
- Install Docker CLI in the Docker image (distro-aware)
- Switch deploy container runner from Docker-in-Docker to host /var/run/docker.sock
  with clearer diagnostics when docker/permissions are missing
- Rewrite scripts/tests/deploy.sh to build a local infinito-<distro> image via Make
  and run deployment strictly against that local tag (no pkgmgr/GHCR pulls)

https://chatgpt.com/share/694513c7-daec-800f-a9b5-c8fb25274958
2025-12-19 09:58:43 +01:00
kevinveenbirkenbach 7c0c1321d5 Fix Hadolint code scanning identity and apply formatting cleanups
https://chatgpt.com/share/694420ac-5d70-800f-94b8-71549893936e
2025-12-18 16:41:25 +01:00
kevinveenbirkenbachandCopilot Autofix powered by AI aa586e0e2c Potential fix for code scanning alert no. 43: Unnecessary lambda
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-18 16:28:02 +01:00
kevinveenbirkenbachandCopilot Autofix powered by AI 0d9ad719f7 Potential fix for code scanning alert no. 66: Empty except
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-18 16:27:37 +01:00
kevinveenbirkenbach a38f36837f style: reformat code for readability and lint compliance 2025-12-18 14:08:01 +01:00
kevinveenbirkenbach 9047285e7a Fix CodeQL warning by avoiding implicit string concatenation in sh -c command
https://chatgpt.com/share/6943f9b0-7dd4-800f-9259-da3c9c4bf156
2025-12-18 13:54:55 +01:00
kevinveenbirkenbachandCopilot Autofix powered by AI 69b1df72f8 Potential fix for code scanning alert no. 53: Explicit returns mixed with implicit (fall through) returns
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-18 13:10:19 +01:00
kevinveenbirkenbach cbd11530f8 Merge pull request #19 from kevinveenbirkenbach/alert-autofix-40
Potential fix for code scanning alert no. 40: Jinja2 templating with autoescape=False
2025-12-18 12:46:53 +01:00
kevinveenbirkenbachandCopilot Autofix powered by AI 1da7a25474 Potential fix for code scanning alert no. 79: File is not always closed
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-18 12:45:53 +01:00
kevinveenbirkenbachandCopilot Autofix powered by AI 18ae1b2838 Potential fix for code scanning alert no. 40: Jinja2 templating with autoescape=False
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-18 12:42:33 +01:00
kevinveenbirkenbach 39b52f4449 fix(security): create vault password file with restrictive permissions
https://chatgpt.com/share/6943d45b-fc7c-800f-8742-413bde1b1516
2025-12-18 11:15:38 +01:00
kevinveenbirkenbach dc846b0672 refactor(cli): move main entrypoint to cli package and standardize execution
- Move main.py to cli/__main__.py and expose via console script
- Switch Makefile execution to `python -m cli`
- Remove fragile Docker symlink to main.py
- Ensure infinito console script is available via venv PATH
- Update tests and documentation to reflect new CLI structure
- Improve logging location and permissions (XDG_STATE_HOME)
- Handle Ctrl+C gracefully with POSIX-compliant exit code (130)

https://chatgpt.com/share/6942f140-fb8c-800f-a13a-641ebf1c644e
2025-12-17 19:06:49 +01:00
kevinveenbirkenbach 7123841fb4 style: apply ruff autoformat 2025-12-17 16:37:21 +01:00
kevinveenbirkenbach 324f9bebf4 fix(lint): resolve Ruff E402 import-order violations
Move all module-level imports to the top of affected files and remove
late imports after sys.path manipulation or runtime code.

Also normalize test imports to use package paths instead of ad-hoc
sys.path hacks, aligning tests with the new layered architecture.

No functional changes intended.

https://chatgpt.com/share/6942c6af-bd10-800f-886a-0ed8731675ee
2025-12-17 16:05:15 +01:00
kevinveenbirkenbach 9e6256dc6e lint(test,core): remove misplaced test code and fix undefined names
- Import subprocess where used to fix F821 in cli/setup/applications
- Remove accidental test methods from RoleDependencyResolver implementation
- Remove invalid and non-discoverable test functions from test_role_dependency_resolver
- Eliminate test code that was never executed due to wrong scope/indentation
- Restore clear separation between production code and unit tests

https://chatgpt.com/share/6942c3ec-8a10-800f-9a15-c319f5e93089
2025-12-17 15:53:25 +01:00
kevinveenbirkenbach 176c8799ce lint(ruff): fix E402 and related import issues
- Fix E402 by removing sys.path hacks and ensuring imports are at module top level
- Replace ad-hoc imports with proper package imports (filter_plugins, lookup_plugins, module_utils, cli)
- Make lookup plugins importable via package paths (e.g. lookup_plugins.application_gid)
- Clean up test imports to rely on installed package layout instead of sys.path manipulation
- Partially fix F821 caused by missing/invalid imports

This establishes a consistent, ruff-compliant import architecture across CLI, filters, plugins, and tests.
2025-12-17 14:36:29 +01:00
kevinveenbirkenbach 73f7c00acd Solved ruff E741 2025-12-17 13:54:51 +01:00
kevinveenbirkenbach 14a0268ec3 Solved ruff E741 2025-12-17 13:51:03 +01:00
kevinveenbirkenbach 9db66f6f98 Ruff autofix 2025-12-17 12:56:47 +01:00
kevinveenbirkenbach 6c3a6d9b20 Implemented correct path 2025-12-17 12:14:11 +01:00
kevinveenbirkenbach 14548cbc52 Replaced false make messy-build by make setup 2025-12-16 20:51:37 +01:00
kevinveenbirkenbach a7140f0097 Adapted python import path 2025-12-16 20:41:52 +01:00
kevinveenbirkenbach d0882433c8 Refactor setup workflow and make install robust via virtualenv
- Introduce a dedicated Python virtualenv (deps target) and run all setup scripts through it
- Fix missing PyYAML errors in clean, CI, and Nix environments
- Refactor build defaults into cli/setup for clearer semantics
- Make setup deterministic and independent from system Python
- Replace early Makefile shell expansion with runtime evaluation
- Rename messy-test to test-messy and update deploy logic and tests accordingly
- Keep setup and test targets consistent across Makefile, CLI, and unit tests

https://chatgpt.com/share/693de226-00ac-800f-8cbd-06552b2f283c
2025-12-13 23:00:13 +01:00
kevinveenbirkenbach 0d18d86243 Add --vars support to inventory creation, implement deep JSON overrides for host_vars, and update CI workflow to pass MASK_CREDENTIALS_IN_LOGS=false.
Includes:
- New apply_vars_overrides() with deep merge logic
- New --vars CLI argument in cli/create/inventory.py
- Added unit tests for vars handling in test_inventory.py
- Updated test-deploy workflow to pass --vars in all deploy phases

Ref: ChatGPT conversation https://chatgpt.com/share/69320f49-6c00-800f-8875-49d36935ae3a
2025-12-04 23:48:43 +01:00
kevinveenbirkenbach bf0134b9c5 Extend inventory creation with --authorized-keys support, unify PATH_ADMINISTRATOR_HOME resolution via group_vars, update CI workflow to pass dummy SSH key, and add full unit test coverage for new features.
See details: https://chatgpt.com/share/6931f775-12d4-800f-9ccb-e6ce52097f9c
2025-12-04 22:06:10 +01:00
kevinveenbirkenbach d0aac64c67 Refactor CI container deploy wrapper: add modes, rebuild/no-cache flags, and transparent inventory/deploy arg forwarding; update tests accordingly (see ChatGPT conversation: https://chatgpt.com/share/6931e58c-2ddc-800f-88d2-f7887ec13e25) 2025-12-04 20:49:04 +01:00
kevinveenbirkenbach f2ace362bc Refactor container deploy pipeline:
- Replace inline shell scripts with Python-driven orchestration
- Introduce inner dockerd started via detached docker run
- Add docker exec–based inventory creation and dedicated deploy execution
- Add automatic vault password generation for CI
- Update GitHub Actions workflow to use new container deploy wrapper
- Add complete unit test suite for container deploy behavior
- Fix CLI argument forwarding and ensure single docker run call

Conversation reference:
https://chatgpt.com/share/6931c45d-4e40-800f-852f-6c9b1f7dc281
2025-12-04 18:27:15 +01:00
kevinveenbirkenbach 8c64f91a6d Refactor deploy CLI into cli/deploy/dedicated.py, add package inits, and update unit tests (see ChatGPT conversation: https://chatgpt.com/share/6931af9e-dad0-800f-9a8f-6d01c373de87) 2025-12-04 16:59:13 +01:00
kevinveenbirkenbach b914fb9789 Refactor CLI filters: rename --ignore to --exclude and update all related logic and tests
- Updated CLI argument parsing to use --exclude instead of --ignore.
- Adjusted help texts, comments, and error messages accordingly.
- Updated role filtering logic and references (include → exclude).
- Added new unit tests for parse_roles_list(), filter_inventory_by_include(), and filter_inventory_by_ignore().
- Improved wording and consistency in docstrings.

This change is part of the refactoring required for the Ansible 2.18 → 2.20 upgrade, ensuring naming clarity and avoiding confusion with Python's 'ignore' semantics.

Conversation reference: https://chatgpt.com/share/69307ef2-1fb4-800f-a2ec-d56020019269
2025-12-03 19:18:39 +01:00
kevinveenbirkenbach d12f7a10a6 Add optional --become-password support with automatic Vault encryption
- Implement ensure_become_password() to handle explicit, generated, and existing become passwords
- Integrate VaultHandler for encrypted ansible_become_password storage
- Add CLI parameter --become-password to inventory creation workflow
- Ensure backwards compatibility: existing passwords remain untouched unless explicitly overridden
- Add unit test verifying non-overwrite behaviour when no password is provided
- Part of migration and refactoring for Ansible 2.20 upgrade

Reference: https://chatgpt.com/share/69301a6d-e920-800f-b19c-e5ca7c3bdd24
2025-12-03 12:09:47 +01:00
kevinveenbirkenbach a6ed047765 CLI: Fix playbook output freeze by removing capture_output
The deploy wrapper previously used subprocess.run(..., capture_output=True),
which buffered all Ansible output until the playbook finished.
This made the CLI appear stuck at 'Launching Ansible Playbook…'.

Switching to subprocess.run(cmd) restores live streaming of Ansible output.

Details: https://chatgpt.com/share/693008b4-b7b0-800f-bd35-5a307a76fc59
2025-12-03 10:54:31 +01:00
kevinveenbirkenbach e9b0760d08 Added localhost variable 2025-12-03 00:31:56 +01:00
kevinveenbirkenbach 46174125bc Refine deploy CLI, test-deploy workflow and Ansible output
Changes:
- Update GitHub Actions test-deploy workflow to run three staged deploys (normal+debug, reset+debug, async) using inventory-generated vault password files.
- Switch Ansible stdout_callback to ansible.builtin.default and enable YAML-style result_format via callback_default.
- Refactor cli/deploy.py: typed run_ansible_playbook(), structured MODE_* handling, better error reporting, and preserved vault/interactive behaviour.
- Add unit tests for deploy CLI (bool parsing, MODE_* loading, dynamic args, validation, and ansible-playbook command construction) under tests/unit/cli/test_deploy.py.

Context: see ChatGPT conversation on 2025-12-02: https://chatgpt.com/share/692f1035-6bc4-800f-91a9-342db54e1a75
2025-12-02 20:25:26 +01:00
kevinveenbirkenbach 5b18f39ccd Extend inventory create CLI with include/ignore filters and SSL/network defaults - https://chatgpt.com/share/692edecd-54c4-800f-b01b-35cf395a60f0 2025-12-02 13:44:16 +01:00
kevinveenbirkenbach c0e26275f8 Refactor defaults generation, credential creation, and inventory management
### Overview
This commit introduces a broad set of improvements across the defaults
generator, credential creation subsystem, inventory creation workflow,
and InventoryManager core logic.

### Major Changes
- Support empty or  config/main.yml in defaults generator and ensure that
  applications with empty configs are still included in defaults_applications.
- Add '--snippet' and '--allow-empty-plain' modes to create/credentials.py
  with non-destructive merging and correct plain-secret handling.
- Ensure empty strings for 'plain' credentials are never encrypted.
- Update InventoryManager to fully support allow_empty_plain and prevent
  accidental overwriting or encrypting existing VaultScalar or dict values.
- Add full-size implementation of cli/create/inventory.py including
  dynamic inventory building, role filtering, host_vars management, and
  parallelised credential snippet generation.
- Fix schemas (Magento, Nextcloud, OAuth2-Proxy, keyboard-color, etc.) to
  align with the new credential model and avoid test failures.
- Improve get_app_conf consistency by ensuring credentials.* paths are
  always resolvable for applications even when config/main.yml is empty.

### Added Test Coverage
- Unit tests for defaults generator handling empty configs.
- Full test suite for create/inventory.py including merge logic and
  vault-safe host_vars loading.
- Extensive tests for InventoryManager: plain-secret behavior,
  vault handling, and recursion logic.
- Update or remove outdated tests referencing old schema behaviour.

### Context
This commit is associated with a refactoring and debugging session documented here:
https://chatgpt.com/share/692ec0e1-5018-800f-b568-d09a53e9d0ee
2025-12-02 11:54:55 +01:00
kevinveenbirkenbach 26dfab147d Implement reserved username handling for users, LDAP and Keycloak
Add end-to-end support for reserved usernames and tighten CAPTCHA / Keycloak logic.

Changes:

- Makefile: rename EXTRA_USERS → RESERVED_USERNAMES and pass it as --reserved-usernames to the users defaults generator.

- cli/build/defaults/users.py: propagate  flag into generated users, add --reserved-usernames CLI option and mark listed accounts as reserved.

- Add reserved_users filter plugin with  and  helpers for Ansible templates and tasks.

- Add unit tests for reserved_users filters and the new reserved-usernames behaviour in the users defaults generator.

- group_vars/all/00_general.yml: harden RECAPTCHA_ENABLED / HCAPTCHA_ENABLED checks with default('') and explicit > 0 length checks.

- svc-db-openldap: introduce OPENLDAP_PROVISION_* flags, add OPENLDAP_PROVISION_RESERVED and OPERNLDAP_USERS to optionally exclude reserved users from provisioning.

- svc-db-openldap templates/tasks: switch role/group LDIF and user import loops to use OPERNLDAP_USERS instead of the full users dict.

- networks: assign dedicated subnet for web-app-roulette-wheel.

- web-app-keycloak vars: compute KEYCLOAK_RESERVED_USERNAMES_LIST and KEYCLOAK_RESERVED_USERNAMES_REGEX from users | reserved_usernames.

- web-app-keycloak user profile template: inject reserved-username regex into username validation pattern and improve error message, fix SSH public key attribute usage and add component name field.

- web-app-keycloak update/_update.yml: strip subComponents from component payloads before update and disable async/poll for easier debugging.

- web-app-keycloak tasks/main.yml: guard cleanup include with MODE_CLEANUP and keep reCAPTCHA update behind KEYCLOAK_RECAPTCHA_ENABLED.

- user/users defaults: mark system/service accounts (root, daemon, mail, admin, webmaster, etc.) as reserved so they cannot be chosen as login names.

- svc-prx-openresty vars: simplify OPENRESTY_CONTAINER lookup by dropping unused default parameter.

- sys-ctl-rpr-btrfs-balancer: simplify main.yml by removing the extra block wrapper.

- sys-daemon handlers: quote handler name for consistency.

Context: change set discussed and refined in ChatGPT on 2025-11-29 (Infinito.Nexus reserved usernames & Keycloak user profile flow). See conversation: https://chatgpt.com/share/692b21f5-5d98-800f-8e15-1ded49deddc9
2025-11-29 17:40:45 +01:00
kevinveenbirkenbach 1129b943fc Added --diff option for deploy to check file differences 2025-11-29 13:31:48 +01:00
kevinveenbirkenbach 9c65bd4839 Refine role dependency graph/tree builders and tests
- Refactor cli/build/graph.py to use cached metadata and dependency indices
  for faster graph generation and cleaner separation of concerns
- Refactor cli/build/tree.py to delegate per-role processing to process_role()
  and support parallel execution via ProcessPoolExecutor
- Add unit tests for graph helper functions and build_mappings()
  under tests/unit/cli/build/test_graph.py
- Add unit tests for find_roles() and process_role() behaviour
  under tests/unit/cli/build/test_tree.py
- Remove the old include_role dependency integration test which relied on the
  previous tree.json dependencies bucket

For details see ChatGPT conversation: https://chatgpt.com/share/6926b805-28a0-800f-a075-e5250aab5c4a
2025-11-26 09:20:45 +01:00
kevinveenbirkenbach e6f4f3a6a4 feat(cli/build/defaults): ensure deterministic alphabetical sorting for applications and users
- Added sorting by application key and user key before YAML output.
- Ensures stable and reproducible file generation across runs.
- Added comprehensive unit tests verifying key order and output stability.

See: https://chatgpt.com/share/68ef4778-a848-800f-a50b-a46a3b878797
2025-10-15 09:04:39 +02:00
kevinveenbirkenbach 61772d5916 Solved testing mode bug 2025-09-19 11:18:29 +02:00
kevinveenbirkenbach fcc9dc71ef Removed solved Todos 2025-09-12 04:05:02 +02:00