### Overview
This commit introduces a broad set of improvements across the defaults
generator, credential creation subsystem, inventory creation workflow,
and InventoryManager core logic.
### Major Changes
- Support empty or config/main.yml in defaults generator and ensure that
applications with empty configs are still included in defaults_applications.
- Add '--snippet' and '--allow-empty-plain' modes to create/credentials.py
with non-destructive merging and correct plain-secret handling.
- Ensure empty strings for 'plain' credentials are never encrypted.
- Update InventoryManager to fully support allow_empty_plain and prevent
accidental overwriting or encrypting existing VaultScalar or dict values.
- Add full-size implementation of cli/create/inventory.py including
dynamic inventory building, role filtering, host_vars management, and
parallelised credential snippet generation.
- Fix schemas (Magento, Nextcloud, OAuth2-Proxy, keyboard-color, etc.) to
align with the new credential model and avoid test failures.
- Improve get_app_conf consistency by ensuring credentials.* paths are
always resolvable for applications even when config/main.yml is empty.
### Added Test Coverage
- Unit tests for defaults generator handling empty configs.
- Full test suite for create/inventory.py including merge logic and
vault-safe host_vars loading.
- Extensive tests for InventoryManager: plain-secret behavior,
vault handling, and recursion logic.
- Update or remove outdated tests referencing old schema behaviour.
### Context
This commit is associated with a refactoring and debugging session documented here:
https://chatgpt.com/share/692ec0e1-5018-800f-b568-d09a53e9d0ee
- Add ANSI color fallback when colorama is missing
- Refactor global help into print_global_help() and reuse for --help and --help-all
- Enhance show_full_help_for_all() with colorful Subcommand/File headers
- Extend unit tests for global help, child sound handling, and failure loop
Reference: https://chatgpt.com/share/692e88de-39f4-800f-ab7f-5ac178698831/share/this-conversation
- Replace legacy utils/run_once.yml with the new helpers utils/once_flag.yml and utils/once_finalize.yml
- Introduce utils/compose_up.yml to ensure docker-compose stacks are up and to flush handlers safely without coupling to run-once flags
- Migrate all affected roles (desk-*, dev-*, sys-ctl-*, sys-svc-*, web-app-*, web-svc-*, util-*) to the new run-once helpers
- Rework sys-svc-msmtp to auto-load Mailu once per deploy, check reachability, and reuse the running stack instead of requiring multiple playbook passes
- Adjust web-app-mailu to integrate cert deployment, handler flushing, and run-once handling so Mailu is fully initialized in a single deploy
- Improve Matomo, CDN, logout and CSP/health-check related roles to cooperate with the new compose_up / once_* pattern
- Simplify alarm/backup/timer/service orchestration (sys-ctl-alm-*, sys-bkp-provider, sys-timer-cln-bkps, etc.) by moving run-once logic into dedicated 01_core.yml files
- Update integration tests so utils/once_flag.yml and utils/once_finalize.yml are recognised as valid run-once providers, keeping the global run_once_* guarantees consistent
- Align frontend injection and service dependencies so Mastodon- and Mailu-related services can be brought up coherently within a single deployment cycle rather than several iterations
Add end-to-end support for reserved usernames and tighten CAPTCHA / Keycloak logic.
Changes:
- Makefile: rename EXTRA_USERS → RESERVED_USERNAMES and pass it as --reserved-usernames to the users defaults generator.
- cli/build/defaults/users.py: propagate flag into generated users, add --reserved-usernames CLI option and mark listed accounts as reserved.
- Add reserved_users filter plugin with and helpers for Ansible templates and tasks.
- Add unit tests for reserved_users filters and the new reserved-usernames behaviour in the users defaults generator.
- group_vars/all/00_general.yml: harden RECAPTCHA_ENABLED / HCAPTCHA_ENABLED checks with default('') and explicit > 0 length checks.
- svc-db-openldap: introduce OPENLDAP_PROVISION_* flags, add OPENLDAP_PROVISION_RESERVED and OPERNLDAP_USERS to optionally exclude reserved users from provisioning.
- svc-db-openldap templates/tasks: switch role/group LDIF and user import loops to use OPERNLDAP_USERS instead of the full users dict.
- networks: assign dedicated subnet for web-app-roulette-wheel.
- web-app-keycloak vars: compute KEYCLOAK_RESERVED_USERNAMES_LIST and KEYCLOAK_RESERVED_USERNAMES_REGEX from users | reserved_usernames.
- web-app-keycloak user profile template: inject reserved-username regex into username validation pattern and improve error message, fix SSH public key attribute usage and add component name field.
- web-app-keycloak update/_update.yml: strip subComponents from component payloads before update and disable async/poll for easier debugging.
- web-app-keycloak tasks/main.yml: guard cleanup include with MODE_CLEANUP and keep reCAPTCHA update behind KEYCLOAK_RECAPTCHA_ENABLED.
- user/users defaults: mark system/service accounts (root, daemon, mail, admin, webmaster, etc.) as reserved so they cannot be chosen as login names.
- svc-prx-openresty vars: simplify OPENRESTY_CONTAINER lookup by dropping unused default parameter.
- sys-ctl-rpr-btrfs-balancer: simplify main.yml by removing the extra block wrapper.
- sys-daemon handlers: quote handler name for consistency.
Context: change set discussed and refined in ChatGPT on 2025-11-29 (Infinito.Nexus reserved usernames & Keycloak user profile flow). See conversation: https://chatgpt.com/share/692b21f5-5d98-800f-8e15-1ded49deddc9
This update sorts all expectation keys alphabetically to guarantee idempotent
Ansible deployments and stable systemd unit generation. Added two unit tests to
validate proper key ordering for canonical domains, aliases, redirects, and WWW
mappings.
Reference: https://chatgpt.com/share/692ae99b-dd88-800f-8fad-2ff62666e295
Implements automatic addition of 'unsafe-inline' when logout feature is enabled, ensuring inline handlers work for iframe logout bridge. Preserves strict base script-src when explicitly disabled. Includes full test suite for logout behaviour, propagation rules, and base override interactions.
Context: https://chatgpt.com/share/6927075c-4de0-800f-bcee-b1f5193e4a99
- Refactor cli/build/graph.py to use cached metadata and dependency indices
for faster graph generation and cleaner separation of concerns
- Refactor cli/build/tree.py to delegate per-role processing to process_role()
and support parallel execution via ProcessPoolExecutor
- Add unit tests for graph helper functions and build_mappings()
under tests/unit/cli/build/test_graph.py
- Add unit tests for find_roles() and process_role() behaviour
under tests/unit/cli/build/test_tree.py
- Remove the old include_role dependency integration test which relied on the
previous tree.json dependencies bucket
For details see ChatGPT conversation: https://chatgpt.com/share/6926b805-28a0-800f-a075-e5250aab5c4a
- Replace jvm_filters with unified memory_filters (JVM + Redis helpers)
- Add redis_maxmemory_mb filter and unit tests
- Introduce sys-ctl-cln-docker role (systemd-based Docker prune + anon volumes)
- Refactor disk space health check to Python script and wire SIZE_PERCENT_CLEANUP_DISC_SPACE
- Adjust schedules and services for Docker cleanup and disk space health
See discussion: https://chatgpt.com/share/6925c1c5-ee38-800f-84b6-da29ccfa7537
- Add node_autosize filter (node_max_old_space_size) using get_app_conf
- Raise error when mem_limit < min_mb to prevent OOM-kill misconfigurations
- Wire Whiteboard NODE_OPTIONS and increase mem_limit to 1g; set cpus=1
- Refactor PeerTube to use the same filter; simplify vars
- Add unit tests; keep integration filters usage green
Context: https://chatgpt.com/share/690e0499-6a94-800f-b8ed-2c5124690103
- Dynamically calculate PEERTUBE_MAX_OLD_SPACE_SIZE (~35% of container RAM, clamped between 768–3072 MB)
- Dynamically calculate PEERTUBE_TRANSCODING_CONCURRENCY (~½ vCPUs, min 1, max 8)
- Added default resource limits for Redis and Peertube containers
- Updated test suite to include human_to_bytes filter in built-in filter list
https://chatgpt.com/share/690914d2-6100-800f-a850-94e6d226e7c9
- Add CSP3 support for style/script: include -elem and -attr directives
- Base (style-src, script-src) now unions elem/attr (CSP2/Safari fallback)
- Respect explicit base disables (e.g. style-src.unsafe-inline: false)
- Hashes only when 'unsafe-inline' absent in the final base tokens
- Nginx: set CSP only for HTML/worker via header_filter_by_lua_block; drop for subresources
- Remove per-location header_filter; keep body_filter only
- Update app role flags to *-attr where appropriate; extend desktop CSS sources
- Add comprehensive unit tests for union/explicit-disable/no-mirror-back
Ref: https://chatgpt.com/share/68f87a0a-cebc-800f-bb3e-8c8ab4dee8ee
This update extends the regex to detect block-style variable definitions such as:
{% set var %} ... {% endset %}
Previously, only inline 'set var =' syntax was recognized, causing false positives
like '_snippet' being flagged as undefined in Jinja templates.
Reference: https://chatgpt.com/share/68f6799a-eb80-800f-ab5c-7c196d4c4661
- Added run_openssl_dates() to extract notBefore/notAfter timestamps.
- Modified mapping logic to store multiple cert entries per SAN with metadata.
- find_cert_for_domain() now selects the newest certificate based on notBefore and mtime.
- Exact SAN matches take precedence over wildcard matches.
- Added new unit tests (test_cert_utils_newest.py) verifying freshness logic, fallback handling, and wildcard behavior.
Reference: https://chatgpt.com/share/68ef4b4c-41d4-800f-9e50-5da4b6be1105
- Added sorting by application key and user key before YAML output.
- Ensures stable and reproducible file generation across runs.
- Added comprehensive unit tests verifying key order and output stability.
See: https://chatgpt.com/share/68ef4778-a848-800f-a50b-a46a3b878797
- Added new optional parameter 'skip_missing_app' to get_app_conf() in module_utils/config_utils.py to safely return defaults when applications are missing.
- Updated group_vars/all/00_general.yml and roles/web-app-nextcloud/config/main.yml to include skip_missing_app=True in all Nextcloud-related calls.
- Added comprehensive unit tests under tests/unit/module_utils/test_config_utils.py covering missing app handling, schema enforcement, nested lists, and index edge cases.
Ref: https://chatgpt.com/share/68ee6b5c-6db0-800f-bc20-d51470d7b39f
feat(bbb/schema): auto-generate etherpad_api_key; set fsesl_password to alphanumeric_32
test(unit): add InventoryManager tests (Option B) expecting feature-generated creds as plain strings
docs: full autocreation of credentials for BigBlueButton now enabled
See: https://chatgpt.com/share/68d69ee8-3fd4-800f-9209-60026b338934
- Added _normalize_codes to support lists of valid HTTP status codes
- Updated web_health_expectations to handle multiple codes, deduplication, and fallback logic
- Extended unit tests with coverage for list/default combinations, invalid values, and alias behavior
- Fixed Flowise CSP flags and whitelist entries
- Adjusted Flowise, MinIO, and Pretix docker service resource limits
- Updated docker-compose templates with explicit service_name
- Corrected MinIO status_codes to 301 redirects
✅ All CSP errors fixed
See details: https://chatgpt.com/share/68d557ad-fc10-800f-b68b-0411d20ea6eb
Introduce filter_plugins/jvm_filters.py with jvm_max_mb/jvm_min_mb. Derive Xmx/Xms from docker mem_limit/mem_reservation using safe rules: Xmx=min(70% limit, limit-1024MB, 12288MB), floored at 1024MB; Xms=min(Xmx/2, reservation, Xmx), floored at 512MB. Parse human-readable sizes (k/m/g/t) with binary units.
Wire filters into roles: set JVM_MINIMUM_MEMORY/JVM_MAXIMUM_MEMORY via filters; stop relying on host RAM. Keep env templates simple and stable.
Add unit tests under tests/unit/filter_plugins/test_jvm_filters.py covering typical sizes, floors, caps, invalid inputs, and entity-name derivation.
Ref: https://chatgpt.com/share/68d3b9f6-8d18-800f-aa8d-8a743ddf164d
- Introduce new resource_filter plugin (mandatory hard_default, auto entity_name fallback)
- Replace get_app_conf calls with resource_filter in resource.yml.j2
- Add WEBSERVER_CPUS_EFFECTIVE, WEBSERVER_WORKER_PROCESSES, WEBSERVER_WORKER_CONNECTIONS to 05_webserver.yml
- Update Nginx templates (sys-svc-webserver, web-app-magento, web-app-nextcloud) to use new vars
- Extend svc-prx-openresty config with cpus/mem limits
- Add unit tests for resource_filter
Details: https://chatgpt.com/share/68d3a493-9a5c-800f-8cd2-bd2e7a3e3fda
Compute per-container CPU/RAM shares based on active services (web-/svc-*, enabled=true or undefined). Cast host facts to numbers, add safe min=1, and output compose-ready values. Include robust unit test.
Also: include resource.yml.j2 in base template and minor formatting tidy-up.
https://chatgpt.com/share/68d2d96c-9bf4-800f-bbec-d4f2c0051c06
- Extend filter signature with auto_build_alias flag to control automatic
default→canonical alias creation
- group_vars/all: introduce AUTO_BUILD_ALIASES variable for global toggle
- Update unit tests: adjust calls to new signature and add dedicated
test cases for auto_build_aliases=False
Ref: conversation https://chatgpt.com/share/68cd512c-c878-800f-bdf2-81737adf7e0e
- Added regex masking to skip {{ var }} usages inside {% raw %}…{% endraw %} blocks.
- Simplified code by removing redundant comments.
- Cleaned up task file for XWiki role by removing outdated note.
Ref: https://chatgpt.com/share/68cd2558-e92c-800f-a80a-a79d3c81476e
Rename filter parent_build_records→wildcard_records; create only wildcard (*.parent) A/AAAA records (no base/apex); switch to CURRENT_PLAY_DOMAINS_ALL; update vars to SYN_DNS_WILDCARD_RECORDS; adjust role/task names, defaults, and docs; add unittest expecting *.a.b from www.a.b.example.com. See: https://chatgpt.com/share/68c35dc1-7170-800f-8fbe-772e61780597
Updated tests/integration/test_vars_usage_in_yaml.py:
- Variables immediately followed by '(' are now treated as function calls,
not as set variables. This prevents false errors.
- Fixed detection of redirect_domain_mappings so it is no longer flagged
as unused.
See: https://chatgpt.com/share/68c3542d-f44c-800f-a483-b3e43739f315
Introduce sys-svc-dns to bootstrap Cloudflare DNS prerequisites. Validates CLOUDFLARE_API_TOKEN, (optionally) manages CAA for base SLDs, and delegates parent record creation to sys-dns-parent-hosts. Wired into sys-stk-front-pure.
sys-dns-parent-hosts: new parent_dns filter builds A/AAAA for each parent host and wildcard children (*.parent). Supports dict/list inputs for CURRENT_PLAY_DOMAINS, optional IPv6, proxied flag, and optional *.apex. Exposes a single parent_build_records entry point.
Let’s Encrypt role cleanup: remove DNS/C AA management from sys-svc-letsencrypt; it now focuses on webroot challenge config and renew timer. Fixed path joins and run_once guards.
Tests: update unit tests to allow wildcard outputs and dict-based CURRENT_PLAY_DOMAINS. Add generate_base_sld_domains filter. Documentation updates for both roles.
Conversation: https://chatgpt.com/share/68c342f7-d20c-800f-b61f-cefeebcf1cd8
- Move all domain→expected-status mapping to filter `web_health_expectations`.
- Require explicit app selection via non-empty `group_names`; only those apps are included.
- Add `www_enabled` flag (wired via `WWW_REDIRECT_ENABLED`) to generate/force www.* → 301.
- Support `redirect_maps` to include manual redirects (sources forced to 301), independent of app selection.
- Aliases always 301; canonicals use per-key override or `server.status_codes.default`, else [200,302,301].
- Remove legacy fallbacks (`server.status_codes.home` / `landingpage`).
- Wire filter output into systemd ExecStart script as JSON expectations.
- Normalize various templates to use `to_json` and minor spacing fixes.
- Update app configs (e.g., YOURLS default=301; Confluence default=302; Bluesky web=405; MediaWiki/Confluence canonical/aliases).
- Constructor now uses `WWW_REDIRECT_ENABLED` for domain generation.
Tests:
- Add comprehensive unit tests for filter: selection by group, keyed/default codes, aliases, www handling, redirect_maps, input sanitization.
- Add unit tests for the standalone checker script (JSON parsing, OK/mismatch counting, sanitization).
See conversation: https://chatgpt.com/share/68c2b93e-de58-800f-8c16-ea05755ba776
- script.py now resolves docker-compose project and working_dir strictly from container labels
- removed container-name fallback logic
- adjusted sys-ctl-hlth-docker-container to include sys-ctl-rpr-docker-soft
- cleaned up sys-svc-docker dependencies
- updated unit tests to mock docker inspect and os.path.isfile for STRICT mode
Conversation: https://chatgpt.com/share/68b80927-b800-800f-a909-0fe8d110fd0e
