Includes:
- New apply_vars_overrides() with deep merge logic
- New --vars CLI argument in cli/create/inventory.py
- Added unit tests for vars handling in test_inventory.py
- Updated test-deploy workflow to pass --vars in all deploy phases
Ref: ChatGPT conversation https://chatgpt.com/share/69320f49-6c00-800f-8875-49d36935ae3a
- Replace inline shell scripts with Python-driven orchestration
- Introduce inner dockerd started via detached docker run
- Add docker exec–based inventory creation and dedicated deploy execution
- Add automatic vault password generation for CI
- Update GitHub Actions workflow to use new container deploy wrapper
- Add complete unit test suite for container deploy behavior
- Fix CLI argument forwarding and ensure single docker run call
Conversation reference:
https://chatgpt.com/share/6931c45d-4e40-800f-852f-6c9b1f7dc281
- Updated CLI argument parsing to use --exclude instead of --ignore.
- Adjusted help texts, comments, and error messages accordingly.
- Updated role filtering logic and references (include → exclude).
- Added new unit tests for parse_roles_list(), filter_inventory_by_include(), and filter_inventory_by_ignore().
- Improved wording and consistency in docstrings.
This change is part of the refactoring required for the Ansible 2.18 → 2.20 upgrade, ensuring naming clarity and avoiding confusion with Python's 'ignore' semantics.
Conversation reference: https://chatgpt.com/share/69307ef2-1fb4-800f-a2ec-d56020019269
- Implement ensure_become_password() to handle explicit, generated, and existing become passwords
- Integrate VaultHandler for encrypted ansible_become_password storage
- Add CLI parameter --become-password to inventory creation workflow
- Ensure backwards compatibility: existing passwords remain untouched unless explicitly overridden
- Add unit test verifying non-overwrite behaviour when no password is provided
- Part of migration and refactoring for Ansible 2.20 upgrade
Reference: https://chatgpt.com/share/69301a6d-e920-800f-b19c-e5ca7c3bdd24
### Overview
This commit introduces a broad set of improvements across the defaults
generator, credential creation subsystem, inventory creation workflow,
and InventoryManager core logic.
### Major Changes
- Support empty or config/main.yml in defaults generator and ensure that
applications with empty configs are still included in defaults_applications.
- Add '--snippet' and '--allow-empty-plain' modes to create/credentials.py
with non-destructive merging and correct plain-secret handling.
- Ensure empty strings for 'plain' credentials are never encrypted.
- Update InventoryManager to fully support allow_empty_plain and prevent
accidental overwriting or encrypting existing VaultScalar or dict values.
- Add full-size implementation of cli/create/inventory.py including
dynamic inventory building, role filtering, host_vars management, and
parallelised credential snippet generation.
- Fix schemas (Magento, Nextcloud, OAuth2-Proxy, keyboard-color, etc.) to
align with the new credential model and avoid test failures.
- Improve get_app_conf consistency by ensuring credentials.* paths are
always resolvable for applications even when config/main.yml is empty.
### Added Test Coverage
- Unit tests for defaults generator handling empty configs.
- Full test suite for create/inventory.py including merge logic and
vault-safe host_vars loading.
- Extensive tests for InventoryManager: plain-secret behavior,
vault handling, and recursion logic.
- Update or remove outdated tests referencing old schema behaviour.
### Context
This commit is associated with a refactoring and debugging session documented here:
https://chatgpt.com/share/692ec0e1-5018-800f-b568-d09a53e9d0ee
- Add ANSI color fallback when colorama is missing
- Refactor global help into print_global_help() and reuse for --help and --help-all
- Enhance show_full_help_for_all() with colorful Subcommand/File headers
- Extend unit tests for global help, child sound handling, and failure loop
Reference: https://chatgpt.com/share/692e88de-39f4-800f-ab7f-5ac178698831/share/this-conversation
Add end-to-end support for reserved usernames and tighten CAPTCHA / Keycloak logic.
Changes:
- Makefile: rename EXTRA_USERS → RESERVED_USERNAMES and pass it as --reserved-usernames to the users defaults generator.
- cli/build/defaults/users.py: propagate flag into generated users, add --reserved-usernames CLI option and mark listed accounts as reserved.
- Add reserved_users filter plugin with and helpers for Ansible templates and tasks.
- Add unit tests for reserved_users filters and the new reserved-usernames behaviour in the users defaults generator.
- group_vars/all/00_general.yml: harden RECAPTCHA_ENABLED / HCAPTCHA_ENABLED checks with default('') and explicit > 0 length checks.
- svc-db-openldap: introduce OPENLDAP_PROVISION_* flags, add OPENLDAP_PROVISION_RESERVED and OPERNLDAP_USERS to optionally exclude reserved users from provisioning.
- svc-db-openldap templates/tasks: switch role/group LDIF and user import loops to use OPERNLDAP_USERS instead of the full users dict.
- networks: assign dedicated subnet for web-app-roulette-wheel.
- web-app-keycloak vars: compute KEYCLOAK_RESERVED_USERNAMES_LIST and KEYCLOAK_RESERVED_USERNAMES_REGEX from users | reserved_usernames.
- web-app-keycloak user profile template: inject reserved-username regex into username validation pattern and improve error message, fix SSH public key attribute usage and add component name field.
- web-app-keycloak update/_update.yml: strip subComponents from component payloads before update and disable async/poll for easier debugging.
- web-app-keycloak tasks/main.yml: guard cleanup include with MODE_CLEANUP and keep reCAPTCHA update behind KEYCLOAK_RECAPTCHA_ENABLED.
- user/users defaults: mark system/service accounts (root, daemon, mail, admin, webmaster, etc.) as reserved so they cannot be chosen as login names.
- svc-prx-openresty vars: simplify OPENRESTY_CONTAINER lookup by dropping unused default parameter.
- sys-ctl-rpr-btrfs-balancer: simplify main.yml by removing the extra block wrapper.
- sys-daemon handlers: quote handler name for consistency.
Context: change set discussed and refined in ChatGPT on 2025-11-29 (Infinito.Nexus reserved usernames & Keycloak user profile flow). See conversation: https://chatgpt.com/share/692b21f5-5d98-800f-8e15-1ded49deddc9
This update sorts all expectation keys alphabetically to guarantee idempotent
Ansible deployments and stable systemd unit generation. Added two unit tests to
validate proper key ordering for canonical domains, aliases, redirects, and WWW
mappings.
Reference: https://chatgpt.com/share/692ae99b-dd88-800f-8fad-2ff62666e295
Implements automatic addition of 'unsafe-inline' when logout feature is enabled, ensuring inline handlers work for iframe logout bridge. Preserves strict base script-src when explicitly disabled. Includes full test suite for logout behaviour, propagation rules, and base override interactions.
Context: https://chatgpt.com/share/6927075c-4de0-800f-bcee-b1f5193e4a99
- Refactor cli/build/graph.py to use cached metadata and dependency indices
for faster graph generation and cleaner separation of concerns
- Refactor cli/build/tree.py to delegate per-role processing to process_role()
and support parallel execution via ProcessPoolExecutor
- Add unit tests for graph helper functions and build_mappings()
under tests/unit/cli/build/test_graph.py
- Add unit tests for find_roles() and process_role() behaviour
under tests/unit/cli/build/test_tree.py
- Remove the old include_role dependency integration test which relied on the
previous tree.json dependencies bucket
For details see ChatGPT conversation: https://chatgpt.com/share/6926b805-28a0-800f-a075-e5250aab5c4a
- Replace jvm_filters with unified memory_filters (JVM + Redis helpers)
- Add redis_maxmemory_mb filter and unit tests
- Introduce sys-ctl-cln-docker role (systemd-based Docker prune + anon volumes)
- Refactor disk space health check to Python script and wire SIZE_PERCENT_CLEANUP_DISC_SPACE
- Adjust schedules and services for Docker cleanup and disk space health
See discussion: https://chatgpt.com/share/6925c1c5-ee38-800f-84b6-da29ccfa7537
- Add node_autosize filter (node_max_old_space_size) using get_app_conf
- Raise error when mem_limit < min_mb to prevent OOM-kill misconfigurations
- Wire Whiteboard NODE_OPTIONS and increase mem_limit to 1g; set cpus=1
- Refactor PeerTube to use the same filter; simplify vars
- Add unit tests; keep integration filters usage green
Context: https://chatgpt.com/share/690e0499-6a94-800f-b8ed-2c5124690103
- Add CSP3 support for style/script: include -elem and -attr directives
- Base (style-src, script-src) now unions elem/attr (CSP2/Safari fallback)
- Respect explicit base disables (e.g. style-src.unsafe-inline: false)
- Hashes only when 'unsafe-inline' absent in the final base tokens
- Nginx: set CSP only for HTML/worker via header_filter_by_lua_block; drop for subresources
- Remove per-location header_filter; keep body_filter only
- Update app role flags to *-attr where appropriate; extend desktop CSS sources
- Add comprehensive unit tests for union/explicit-disable/no-mirror-back
Ref: https://chatgpt.com/share/68f87a0a-cebc-800f-bb3e-8c8ab4dee8ee
- Added run_openssl_dates() to extract notBefore/notAfter timestamps.
- Modified mapping logic to store multiple cert entries per SAN with metadata.
- find_cert_for_domain() now selects the newest certificate based on notBefore and mtime.
- Exact SAN matches take precedence over wildcard matches.
- Added new unit tests (test_cert_utils_newest.py) verifying freshness logic, fallback handling, and wildcard behavior.
Reference: https://chatgpt.com/share/68ef4b4c-41d4-800f-9e50-5da4b6be1105
- Added sorting by application key and user key before YAML output.
- Ensures stable and reproducible file generation across runs.
- Added comprehensive unit tests verifying key order and output stability.
See: https://chatgpt.com/share/68ef4778-a848-800f-a50b-a46a3b878797
- Added new optional parameter 'skip_missing_app' to get_app_conf() in module_utils/config_utils.py to safely return defaults when applications are missing.
- Updated group_vars/all/00_general.yml and roles/web-app-nextcloud/config/main.yml to include skip_missing_app=True in all Nextcloud-related calls.
- Added comprehensive unit tests under tests/unit/module_utils/test_config_utils.py covering missing app handling, schema enforcement, nested lists, and index edge cases.
Ref: https://chatgpt.com/share/68ee6b5c-6db0-800f-bc20-d51470d7b39f
feat(bbb/schema): auto-generate etherpad_api_key; set fsesl_password to alphanumeric_32
test(unit): add InventoryManager tests (Option B) expecting feature-generated creds as plain strings
docs: full autocreation of credentials for BigBlueButton now enabled
See: https://chatgpt.com/share/68d69ee8-3fd4-800f-9209-60026b338934
- Added _normalize_codes to support lists of valid HTTP status codes
- Updated web_health_expectations to handle multiple codes, deduplication, and fallback logic
- Extended unit tests with coverage for list/default combinations, invalid values, and alias behavior
- Fixed Flowise CSP flags and whitelist entries
- Adjusted Flowise, MinIO, and Pretix docker service resource limits
- Updated docker-compose templates with explicit service_name
- Corrected MinIO status_codes to 301 redirects
✅ All CSP errors fixed
See details: https://chatgpt.com/share/68d557ad-fc10-800f-b68b-0411d20ea6eb
Introduce filter_plugins/jvm_filters.py with jvm_max_mb/jvm_min_mb. Derive Xmx/Xms from docker mem_limit/mem_reservation using safe rules: Xmx=min(70% limit, limit-1024MB, 12288MB), floored at 1024MB; Xms=min(Xmx/2, reservation, Xmx), floored at 512MB. Parse human-readable sizes (k/m/g/t) with binary units.
Wire filters into roles: set JVM_MINIMUM_MEMORY/JVM_MAXIMUM_MEMORY via filters; stop relying on host RAM. Keep env templates simple and stable.
Add unit tests under tests/unit/filter_plugins/test_jvm_filters.py covering typical sizes, floors, caps, invalid inputs, and entity-name derivation.
Ref: https://chatgpt.com/share/68d3b9f6-8d18-800f-aa8d-8a743ddf164d
- Introduce new resource_filter plugin (mandatory hard_default, auto entity_name fallback)
- Replace get_app_conf calls with resource_filter in resource.yml.j2
- Add WEBSERVER_CPUS_EFFECTIVE, WEBSERVER_WORKER_PROCESSES, WEBSERVER_WORKER_CONNECTIONS to 05_webserver.yml
- Update Nginx templates (sys-svc-webserver, web-app-magento, web-app-nextcloud) to use new vars
- Extend svc-prx-openresty config with cpus/mem limits
- Add unit tests for resource_filter
Details: https://chatgpt.com/share/68d3a493-9a5c-800f-8cd2-bd2e7a3e3fda
Compute per-container CPU/RAM shares based on active services (web-/svc-*, enabled=true or undefined). Cast host facts to numbers, add safe min=1, and output compose-ready values. Include robust unit test.
Also: include resource.yml.j2 in base template and minor formatting tidy-up.
https://chatgpt.com/share/68d2d96c-9bf4-800f-bbec-d4f2c0051c06
- Extend filter signature with auto_build_alias flag to control automatic
default→canonical alias creation
- group_vars/all: introduce AUTO_BUILD_ALIASES variable for global toggle
- Update unit tests: adjust calls to new signature and add dedicated
test cases for auto_build_aliases=False
Ref: conversation https://chatgpt.com/share/68cd512c-c878-800f-bdf2-81737adf7e0e
Rename filter parent_build_records→wildcard_records; create only wildcard (*.parent) A/AAAA records (no base/apex); switch to CURRENT_PLAY_DOMAINS_ALL; update vars to SYN_DNS_WILDCARD_RECORDS; adjust role/task names, defaults, and docs; add unittest expecting *.a.b from www.a.b.example.com. See: https://chatgpt.com/share/68c35dc1-7170-800f-8fbe-772e61780597
Introduce sys-svc-dns to bootstrap Cloudflare DNS prerequisites. Validates CLOUDFLARE_API_TOKEN, (optionally) manages CAA for base SLDs, and delegates parent record creation to sys-dns-parent-hosts. Wired into sys-stk-front-pure.
sys-dns-parent-hosts: new parent_dns filter builds A/AAAA for each parent host and wildcard children (*.parent). Supports dict/list inputs for CURRENT_PLAY_DOMAINS, optional IPv6, proxied flag, and optional *.apex. Exposes a single parent_build_records entry point.
Let’s Encrypt role cleanup: remove DNS/C AA management from sys-svc-letsencrypt; it now focuses on webroot challenge config and renew timer. Fixed path joins and run_once guards.
Tests: update unit tests to allow wildcard outputs and dict-based CURRENT_PLAY_DOMAINS. Add generate_base_sld_domains filter. Documentation updates for both roles.
Conversation: https://chatgpt.com/share/68c342f7-d20c-800f-b61f-cefeebcf1cd8
- Move all domain→expected-status mapping to filter `web_health_expectations`.
- Require explicit app selection via non-empty `group_names`; only those apps are included.
- Add `www_enabled` flag (wired via `WWW_REDIRECT_ENABLED`) to generate/force www.* → 301.
- Support `redirect_maps` to include manual redirects (sources forced to 301), independent of app selection.
- Aliases always 301; canonicals use per-key override or `server.status_codes.default`, else [200,302,301].
- Remove legacy fallbacks (`server.status_codes.home` / `landingpage`).
- Wire filter output into systemd ExecStart script as JSON expectations.
- Normalize various templates to use `to_json` and minor spacing fixes.
- Update app configs (e.g., YOURLS default=301; Confluence default=302; Bluesky web=405; MediaWiki/Confluence canonical/aliases).
- Constructor now uses `WWW_REDIRECT_ENABLED` for domain generation.
Tests:
- Add comprehensive unit tests for filter: selection by group, keyed/default codes, aliases, www handling, redirect_maps, input sanitization.
- Add unit tests for the standalone checker script (JSON parsing, OK/mismatch counting, sanitization).
See conversation: https://chatgpt.com/share/68c2b93e-de58-800f-8c16-ea05755ba776
- script.py now resolves docker-compose project and working_dir strictly from container labels
- removed container-name fallback logic
- adjusted sys-ctl-hlth-docker-container to include sys-ctl-rpr-docker-soft
- cleaned up sys-svc-docker dependencies
- updated unit tests to mock docker inspect and os.path.isfile for STRICT mode
Conversation: https://chatgpt.com/share/68b80927-b800-800f-a909-0fe8d110fd0e
Integration tests added/updated:
- tests/integration/test_filters_usage.py: AST-based detection of filter definitions (FilterModule.filters), robust Jinja detection ({{ ... }}, {% ... %}, {% filter ... %}), plus Python call tracking; fails if a filter is used only under tests/.
- tests/integration/test_filters_are_defined.py: inverse check — every filter used in .yml/.yaml/.j2/.jinja2/.tmpl must be defined locally. Scans only inside Jinja blocks and ignores pipes inside strings (e.g., lookup('pipe', "... | grep ... | awk ...")) to avoid false positives like trusted_hosts, woff/woff2, etc.
Bug fixes & robustness:
- Build regexes without %-string formatting to avoid ValueError from literal '%' in Jinja tags.
- Strip quoted strings in usage analysis so sed/grep/awk pipes are not miscounted as filters.
- Prevent self-matches in the defining file.
Cleanup / removal of dead code:
- Removed unused filter plugins and related unit tests:
* filter_plugins/alias_domains_map.py
* filter_plugins/get_application_id.py
* filter_plugins/load_configuration.py
* filter_plugins/safe.py
* filter_plugins/safe_join.py
* roles/svc-db-openldap/filter_plugins/build_ldap_nested_group_entries.py
* roles/sys-ctl-bkp-docker-2-loc/filter_plugins/dict_to_cli_args.py
* corresponding tests under tests/unit/*
- roles/svc-db-postgres/filter_plugins/split_postgres_connections.py: dropped no-longer-needed list_postgres_roles API; adjusted tests.
Misc:
- sys-stk-front-proxy/defaults/main.yml: clarified valid vhost_flavour values (comma-separated).
Ref: https://chatgpt.com/share/68b56bac-c4f8-800f-aeef-6708dbb44199
Changed safe_join to raise ValueError if base or tail is None instead of returning 'None/path'.
Adjusted unit tests accordingly to expect exceptions for None inputs and kept empty-string handling valid.
Ref: https://chatgpt.com/share/68b55850-e854-800f-9702-09ea956b8dc4
- Move default 'unsafe-inline' for style-src and style-src-elem into get_csp_flags
- Ensure hashes are only added if 'unsafe-inline' not in final tokens
- Improve comments and structure
- Extend unit tests to cover default flags, overrides, and final-token logic
See: https://chatgpt.com/share/68b54520-5cfc-800f-9bac-45093740df78
- Support query elements starting with '?' or '&'
* First query element normalized to '?', subsequent to '&'
* Each query element must be exactly one 'key=value' pair
* Query elements may only appear after path elements
* Once query starts, no more path elements are allowed
- Extend test suite with success and failure cases for query handling
See: https://chatgpt.com/share/68b537ea-d198-800f-927a-940c4de832f2
Improved url_join filter:
- Requires first element to contain a valid '<scheme>://'
- Raises specific errors for None, empty list, wrong type, missing scheme,
extra schemes in later parts, or string conversion failures
- Provides clearer error messages with index context in parts
See: https://chatgpt.com/share/68b537ea-d198-800f-927a-940c4de832f2
