3 Commits

Include renames

Author	SHA1	Message	Date
kevinveenbirkenbachandClaude Opus 4.7	623035618c	refactor(cache): route every yaml call in tests/ through utils.cache.yaml ... Stops every direct `yaml.safe_load`/`safe_load_all`/`safe_dump`/`dump` call across the test corpus from re-parsing files the production cache already holds. Each migrated test loads role meta, group_vars, inventory fixtures, etc. through the shared cache, so a single `make test` run parses every YAML file at most once across all suites. Cache module changes (utils/cache/yaml.py): - Unify single- and multi-doc cache: one `_CACHE: Dict[sig, list]` holding the `safe_load_all` payload. Single-doc helpers (load_yaml / load_yaml_any) unwrap docs[0] and raise on >1 document, preserving the old strict semantics. A file consumed both as single-doc and multi-doc is now parsed exactly once. - Add load_yaml_all (path-cached) and load_yaml_all_str (string, no cache) so multi-doc consumers no longer need a direct `yaml.safe_load_all` plus a separate file read. - dump_yaml now accepts any safe-dumpable structure (dict, list, scalar) so test fixtures writing list-of-tasks YAML can use it. Lint guard (tests/lint/repository/test_no_direct_yaml_calls.py): - Extend the scanned tree to include `tests/` so future test code is forced through the cache, with the existing `# noqa: direct-yaml` per-call opt-out for genuine raw-yaml fixture needs. Test corpus: - 89 test files migrated to load_yaml / load_yaml_any / load_yaml_str / load_yaml_all / load_yaml_all_str / dump_yaml / dump_yaml_str. No `# noqa` shortcuts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-05 12:16:20 +02:00
kevinveenbirkenbachandClaude Opus 4.7	ed04dad5d5	refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration ... Single atomic landing for three back-to-back role-meta refactor requirements (see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md, 010-role-meta-runafter-lifecycle-migration.md). req-008 — role meta layout * Move every role's metadata under meta/<topic>.yml with file-root convention (no wrapping key matching the basename): - schema/main.yml + config/main.yml.credentials → meta/schema.yml - users/main.yml → meta/users.yml - config/main.yml.server → meta/server.yml - config/main.yml.rbac → meta/rbac.yml - config/main.yml.compose.services + every other top-level key of config/main.yml inlined under <entity>.<key> → meta/services.yml - config/main.yml.compose.volumes → meta/volumes.yml * Drop the legacy schema/, users/, config/ directories from every role. * meta/schema.yml: support nested credential keys, optional `default:` field (Jinja literal, not rendered/validated/regenerated), implicit `algorithm: plain`. * Materialised paths drop the `compose.` prefix: compose.services.<e>.<…> → services.<e>.<…> compose.volumes.<k> → volumes.<k> <former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…> * Update every Python consumer (utils/cache, utils/manager, utils/applications, plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml and task file to the new paths. * No legacy fallback. Repository-wide grep for `compose.services.` / `compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml` is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py. * Add tests/lint/repository/test_role_meta_layout.py guarding against the old paths returning, and an explicit lint guard that meta/variants.yml files do not carry a `compose:` wrapper. req-009 — per-role networks/ports * Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet, dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet, dns_resolver}. * Delete group_vars/all/09_ports.yml entirely; ports now live on the service entity that exposes them at roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}. * Rename + int-cast every former top-level `services.<entity>.port` field to `services.<entity>.ports.inter` (single int). * Always-category-keyed maps for ports.local / ports.public; relay spans use {start, end} integers. * Add PORT_BANDS map to group_vars/all/08_networks.yml — single source of truth for the per-<scope>.<category> band ranges, consumed by both the new CLI helpers and the lint test. * New CLI helpers (cli meta ports suggest, cli meta networks suggest) walk the live role tree gap-first then increment-fallback, with deterministic output and non-zero exit on capacity overflow. * cli create role consumes both helpers when scaffolding a new role. * Lint extensions: bare-int port shape rejection, host-bound port collision check (single ints + relay span integers, inter excluded), band-membership check with documented allow-list for the legacy web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry. req-010 — run_after / lifecycle migration * Move galaxy_info.run_after and galaxy_info.lifecycle out of meta/main.yml into meta/services.yml.<primary_entity>.{run_after, lifecycle}; <primary_entity> is the value returned by get_entity_name(role_name). * For multi-entity roles whose primary entity has no compose service (web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/ gen roles), create a dedicated metadata-only top-level entry under <primary_entity>. * Introduce utils/roles/meta_lookup.py with get_role_run_after / get_role_lifecycle that resolve <primary_entity> once and read from meta/services.yml; degrade gracefully when meta/services.yml is absent OR the field is absent; raise MetaServicesShapeError when the file exists but is malformed. * Update every consumer (utils/service_registry, utils/invokable, cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/ lifecycle_filter/__main__, cli/meta/applications/resolution/combined, cli/build/{tree,graph,role_include}, cli/deploy/development/{common, init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to use the helper. * Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated}; run_after/lifecycle never on a non-primary entity; empty run_after lists rejected. Also folded into this change set * services_disabler now writes `services.<svc>.{enabled,shared}` into host_vars instead of the old `compose.services.<svc>.{...}` path; the discover/find pair walks meta/services.yml and only treats entries carrying enabled/shared/provides as provider declarations (so metadata-only entries from req-010 do not create spurious provider rows). * service_registry skips `run_after` deps whose target was filtered out via SERVICES_DISABLED instead of failing the whole load. * cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/ consume the new applications.<role>.services.<svc>.image shape (mirror.md updated to match). * utils/domains/list.build_applications_from_roles now reads meta/server.yml directly (it had been silently returning {} since config/main.yml stopped existing). * utils/roles/resource_validation.filter_roles_by_min_storage reads meta/services.yml.<entity>.min_storage instead of the legacy compose.services path. * web-app-bluesky pre-clones the social-app build context BEFORE sys-svc-compose so the build context is on disk by the time sys-stk-front-proxy/tasks/main.yml flushes the docker build handler. * 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin, web-app-wordpress, web-app-keycloak) lose their `compose: services:` wrapper so deep-merge over meta/services.yml lands on the new file-root shape. Validation * Unit tests: 1571 passed. * Lint tests: 50 passed. * Integration tests: 161 passed (1 skipped). * Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea, discourse,mattermost}. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 09:58:26 +02:00
kevinveenbirkenbachandClaude Opus 4.7	9c35909752	refactor(utils/cache): split data.py into base/applications/users/domains ... The 934-line `utils/cache/data.py` monolith mixed cross-cutting helpers (constants, yaml-loaders, deep-merge, signatures, render machinery) with three independent domain caches (applications, users, domains). Per the KISS/SPOT/DRY guidelines, split it along domain boundaries so each file has a single, well-scoped responsibility: utils/cache/__init__.py — package; owns _reset_cache_for_tests() orchestrator utils/cache/base.py — cross-cutting helpers: constants, yaml-loaders, _deep_merge, cache-key + content-fingerprint signatures, the cross-domain re-entry guard, and the templar-render machinery (the only ansible-coupled symbol, lazy-imports its dependency) utils/cache/applications.py — variants + defaults + get_merged_applications. Strictly ansible-free at import time so the GitHub Actions runner-host CLI path (cli.deploy.development.init → plan_dev_inventory_matrix → get_variants) keeps working without ansible installed utils/cache/users.py — user defs, token store hydration, alias materialization, get_user_defaults, get_merged_users utils/cache/domains.py — get_merged_domains, derived from the merged applications view utils/cache/data.py — DELETED. Façade is gone; the package exports nothing the domain modules don't already Backward-compat: the facade was deleted (per operator request) and all 36+ consumers — 18 plugins, 2 CLI entry points, 9 test files, 4 doc files, plus a role-internal lookup_plugin — were rewired to import directly from the owning domain module. `_reset_cache_for_tests` is now exposed at `from utils.cache import _reset_cache_for_tests`. Tests added (per-module, focused): tests/unit/utils/cache/test_base.py (31 cases) tests/unit/utils/cache/test_applications.py ( 9 cases) tests/unit/utils/cache/test_users_module.py (21 cases; suffix disambiguates from the existing tests/unit/plugins/ lookup/test_users.py) tests/unit/utils/cache/test_domains.py ( 6 cases) The ansible-block invariants from CI runs 24934007615 and 24935979190 are now pinned via subprocess isolation: each ansible-less import test spawns a fresh `python3 -c '…'` with a meta_path filter denying every `ansible*` import and asserts the import + invocation succeed in that clean process. The earlier in-process sys.modules-manipulation shims fragmented module identity and broke under Python 3.13 inside the infinito container; subprocess isolation has neither problem. Doc-style audit (per docs/contributing/documentation.md): - Removed em-dash from utils/cache/README.md (split into two sentences) - Updated stale `utils.runtime_data` reference in docs/contributing/design/matrix-deploy.md - Removed two SPOT self-declarations (docs/agents/files/group_vars/applications.md and docs/contributing/artefact/files/plugins/lookup/users.md) per the documentation rule that SPOT status is conferred by cross-links, not by the page asserting it about itself Empirical proof: - All four domain modules import without ansible (subprocess isolation tests) - get_variants(roles_dir=ROLES_DIR) returns 99 apps in the ansible-blocked subprocess - 142 cache tests + 1085 wider-suite tests pass; the only remaining failures (test_host_vars free-function fixture bug, test_runtime_performance timing flake) are pre-existing and unrelated to this refactor (verified via `git stash`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 00:27:57 +02:00