Stops every direct `yaml.safe_load`/`safe_load_all`/`safe_dump`/`dump`
call across the test corpus from re-parsing files the production cache
already holds. Each migrated test loads role meta, group_vars, inventory
fixtures, etc. through the shared cache, so a single `make test` run
parses every YAML file at most once across all suites.
Cache module changes (utils/cache/yaml.py):
- Unify single- and multi-doc cache: one `_CACHE: Dict[sig, list]`
holding the `safe_load_all` payload. Single-doc helpers
(load_yaml / load_yaml_any) unwrap docs[0] and raise on >1
document, preserving the old strict semantics. A file consumed
both as single-doc and multi-doc is now parsed exactly once.
- Add load_yaml_all (path-cached) and load_yaml_all_str (string,
no cache) so multi-doc consumers no longer need a direct
`yaml.safe_load_all` plus a separate file read.
- dump_yaml now accepts any safe-dumpable structure (dict, list,
scalar) so test fixtures writing list-of-tasks YAML can use it.
Lint guard (tests/lint/repository/test_no_direct_yaml_calls.py):
- Extend the scanned tree to include `tests/` so future test code
is forced through the cache, with the existing
`# noqa: direct-yaml` per-call opt-out for genuine raw-yaml
fixture needs.
Test corpus:
- 89 test files migrated to load_yaml / load_yaml_any /
load_yaml_str / load_yaml_all / load_yaml_all_str / dump_yaml /
dump_yaml_str. No `# noqa` shortcuts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
find_provider_roles now uses build_service_registry_from_roles_dir, the
same resolver the SERVICES_DISABLED pre-check uses. Previously it matched
only roles with a literal compose.services.<svc>.image entry, so services
exposed via compose.services.<entity>.provides (e.g. mailu → email) were
invisible to the init phase. The pre-check then flagged the unresolved
provider role as a conflict and aborted every fresh-purged deploy that
disabled "email".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Rename 06..19 → 04..17 so the sequence is dense and gap-free.
- Update hard-coded references in cli/, tests/, and PR template.
- Add README.md describing scope, load order, and naming rules
per docs/contributing/documentation.md standards.
Adds inventory consistency guard that fails fast when SERVICES_DISABLED
conflicts with an already-deployed inventory, and ensures the env var is
forwarded through the dedicated runner, development exec command, and
local test scripts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolve conflicts:
- docs/agents/action/iteration.md: keep both new bullets (SERVICES_DISABLED
proposal + make test requirement)
- docs/contributing/design/services.md: deleted (renamed to
docs/contributing/code/services.md in this branch)
- group_vars/all/20_services.yml: keep deletion from this branch (req-001)
- tests/lint/ansible/roles/test_service_core_first_task_run_once.py:
use build_service_registry_from_roles_dir (req-001) with repo_root()
from main for robust path resolution
Fix broken relative links caused by rename and deletion of files:
- docs/contributing/design/architecture.md: services.md → ../code/services.md
- docs/contributing/design/exceptions.md: load_frontend_service.yml →
load_service.yml
- docs/requirements/001-service-registry-refactoring.md: fix paths for
unit.md, integration.md, principles.md, documentation.md, AGENTS.md
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Delete group_vars/all/20_services.yml; all service metadata now lives
in each role's config/main.yml under compose.services.<name>
- Rename generic 'database' service key to explicit 'mariadb'/'postgres'
across all app roles; remove legacy type: field
- Add utils/service_registry.py and plugins/lookup/service_registry.py
as the single source of truth for service discovery
- Add utils/database_service.py to centralize DB service resolution
- Delete service_load_enabled filter and service_should_load lookup;
replaced by service lookup plugin
- Update sys-utils-service-loader to run in constructor stage and use
generic load_service.yml; remove load_frontend_service.yml
- Add _service_loader_preloading guard to sys-svc-compose,
sys-svc-webserver-core and sys-ctl-alm-compose to prevent
health-checks/alarms from firing during preload phase
- Replace manual load_app.yml calls for shared DB/Mailu with assert-
based preload validation in sys-svc-rdbms and sys-svc-mail
- Update networks.yml.j2 templates to resolve LDAP/AI networks via
service_registry instead of hardcoded role names
- Migrate all tests to new service discovery API; delete obsolete tests
- Update docs/contributing/code/services.md to reflect role-local model
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Scans roles_dir to find every role whose config/main.yml defines the
disabled service, then creates or updates the compose.services entry in
host_vars for each matching role — including roles not yet present in
applications. roles_dir is now a required parameter (no Optional fallback).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add services_disabler.py: parse SERVICES_DISABLED, disable services in host_vars,
remove provider roles from inventory
- Wire apply_services_disabled_from_env into cli/create/inventory/cli.py after deploy
- Add unit tests for all services_disabler functions
- Update hardware.md and variables.md with SERVICES_DISABLED documentation
- Update environment.sh: deploy dashboard with SERVICES_DISABLED=matomo first,
assert dashboard reachable + matomo not reachable, then deploy matomo and assert
reachable; extract assert_http_status / assert_http_status_not helpers
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Allow credentials generator to run for roles without schema/main.yml so
transitive config-based generation still works.
- Make InventoryManager resilient to missing YAML by defaulting to {} and
only loading schema when schema/main.yml exists.
- Add recursive shared-provider include resolution based on docker.services
config and apply provider schemas/special rules transitively.
- Require database_password injection only when database.enabled=true and shared=true.
- Update unit/integration tests to create required files for .exists() checks and
validate transitive provider role credential generation.
https://chatgpt.com/share/69639d93-4c10-800f-bc6c-8e5b9fb75088
- Migrate Mailu credentials from legacy mailu_token to tokens['web-app-mailu']
- Update all affected application templates to use the new token structure
- Adjust inventory validation to no longer special-case mailu_token
- Minor typing and formatting cleanups in inventory creation and tests
https://chatgpt.com/share/69554915-87c8-800f-afd5-12118dbb1107
* Move SSH key handling to users.<name>.authorized_keys (list-only) and write keys via shared user role
* Enforce at least one SSH key for administrator user
* Wrap backup user keys with restricted command wrapper before writing authorized_keys
* Remove legacy inventory file copy and CLI flag/function for administrator authorized_keys
* Extend user setup generator to include authorized_keys field
* Rename administrator scripts path to root scripts path and update dependent variables
* Update inventory sample and docs to reflect new SSH key workflow
https://chatgpt.com/share/695510b8-23f4-800f-b5db-877b07a1469b
* Replace global `SSL_ENABLED` flag with `TLS_ENABLED`
* Update web protocol/ports (http/https, ws/wss) to use `TLS_ENABLED`
* Switch cert-related roles and nginx TLS header gating to `TLS_ENABLED`
* Update app env templates (BookWyrm, Confluence, Jira, Magento, OpenProject, WordPress, etc.) to use `TLS_ENABLED`
* Adjust inventory generator to write `TLS_ENABLED` and update CLI help text accordingly
* Update unit tests for inventory vars overrides to expect `TLS_ENABLED`
https://chatgpt.com/share/695306ef-a798-800f-b118-e07b525e1811
- add --vars-file CLI option to load host_vars overrides from YAML
- merge vars-file overrides before JSON --vars (JSON takes precedence)
- implement deep-merge logic for YAML vars files
- update CI deploy script to use vars file instead of inline JSON
- add unit test for apply_vars_overrides_from_file
- remove obsolete MASK_CREDENTIALS_IN_LOGS TODO entry
https://chatgpt.com/share/6952e51d-6c50-800f-8764-31dc8c609cdb
- Fix E402 by removing sys.path hacks and ensuring imports are at module top level
- Replace ad-hoc imports with proper package imports (filter_plugins, lookup_plugins, module_utils, cli)
- Make lookup plugins importable via package paths (e.g. lookup_plugins.application_gid)
- Clean up test imports to rely on installed package layout instead of sys.path manipulation
- Partially fix F821 caused by missing/invalid imports
This establishes a consistent, ruff-compliant import architecture across CLI, filters, plugins, and tests.
Includes:
- New apply_vars_overrides() with deep merge logic
- New --vars CLI argument in cli/create/inventory.py
- Added unit tests for vars handling in test_inventory.py
- Updated test-deploy workflow to pass --vars in all deploy phases
Ref: ChatGPT conversation https://chatgpt.com/share/69320f49-6c00-800f-8875-49d36935ae3a
- Updated CLI argument parsing to use --exclude instead of --ignore.
- Adjusted help texts, comments, and error messages accordingly.
- Updated role filtering logic and references (include → exclude).
- Added new unit tests for parse_roles_list(), filter_inventory_by_include(), and filter_inventory_by_ignore().
- Improved wording and consistency in docstrings.
This change is part of the refactoring required for the Ansible 2.18 → 2.20 upgrade, ensuring naming clarity and avoiding confusion with Python's 'ignore' semantics.
Conversation reference: https://chatgpt.com/share/69307ef2-1fb4-800f-a2ec-d56020019269
- Implement ensure_become_password() to handle explicit, generated, and existing become passwords
- Integrate VaultHandler for encrypted ansible_become_password storage
- Add CLI parameter --become-password to inventory creation workflow
- Ensure backwards compatibility: existing passwords remain untouched unless explicitly overridden
- Add unit test verifying non-overwrite behaviour when no password is provided
- Part of migration and refactoring for Ansible 2.20 upgrade
Reference: https://chatgpt.com/share/69301a6d-e920-800f-b19c-e5ca7c3bdd24
### Overview
This commit introduces a broad set of improvements across the defaults
generator, credential creation subsystem, inventory creation workflow,
and InventoryManager core logic.
### Major Changes
- Support empty or config/main.yml in defaults generator and ensure that
applications with empty configs are still included in defaults_applications.
- Add '--snippet' and '--allow-empty-plain' modes to create/credentials.py
with non-destructive merging and correct plain-secret handling.
- Ensure empty strings for 'plain' credentials are never encrypted.
- Update InventoryManager to fully support allow_empty_plain and prevent
accidental overwriting or encrypting existing VaultScalar or dict values.
- Add full-size implementation of cli/create/inventory.py including
dynamic inventory building, role filtering, host_vars management, and
parallelised credential snippet generation.
- Fix schemas (Magento, Nextcloud, OAuth2-Proxy, keyboard-color, etc.) to
align with the new credential model and avoid test failures.
- Improve get_app_conf consistency by ensuring credentials.* paths are
always resolvable for applications even when config/main.yml is empty.
### Added Test Coverage
- Unit tests for defaults generator handling empty configs.
- Full test suite for create/inventory.py including merge logic and
vault-safe host_vars loading.
- Extensive tests for InventoryManager: plain-secret behavior,
vault handling, and recursion logic.
- Update or remove outdated tests referencing old schema behaviour.
### Context
This commit is associated with a refactoring and debugging session documented here:
https://chatgpt.com/share/692ec0e1-5018-800f-b568-d09a53e9d0ee
This commit restructures systemctl service definitions and category mappings.
Motivation: Alarm-related bugs revealed inconsistencies in service and role handling.
Preparation step: lays the groundwork for fixing the alarm issues by aligning categories, roles, and service templates.
