mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-09 12:17:13 +00:00
05f2d9d6cc8e4d87db43d3141506fdebf58b1663
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  kevinveenbirkenbachandClaude Opus 4.7
|
fe54854fe5 |
refactor(resolver): variant-aware combined resolver and per-round include
Single SPOT for matrix-deploy topology
=======================================
Before this change two paths derived the deploy plan from different
sources and could disagree:
* Group inclusion (`--include` / `group_names`): driven by
`cli.meta.applications.resolution.combined.CombinedResolver`,
which read each role's `meta/services.yml` raw and ignored
`meta/variants.yml` entirely. A variant that flipped
`services.<X>.enabled` to a literal `true` did NOT pull `<X>`'s
provider role into the deploy plan.
* Inventory bake (`host_vars/applications.<app>.*`): driven by the
variant-aware payload merge in `cli.deploy.development.inventory`,
so the role saw the variant-merged services map at deploy time.
The split surfaced concretely for `web-app-wordpress` variant 0
(`services.discourse.enabled: true` in `meta/variants.yml`): the
inventory baked `discourse.enabled = true`, the wp-discourse plugin
task gate fired, and the install loop tried `docker exec discourse`
against a container the variant-blind resolver had never pulled.
This commit collapses both consumers onto the same per-round,
variant-merged services map. The variant-aware resolver (Phase 1+2)
is the additive plumbing, but the deploy/init wiring (Phase 3-5) is
what actually changes behaviour at runtime.
Variant-aware resolver (Phase 1+2)
----------------------------------
`role_introspection.load_shared_service_roles_for_app` accepts a
`services_override` kwarg; `CombinedResolver.__init__` accepts a
`{role_name: services_dict}` map and routes it through the loader.
A `CombinedResolver` instance is round-specific when overrides are
supplied — its per-role edge cache is variant-resolved, do NOT reuse
across rounds. The combined-resolver CLI gains
`--services-overrides <path>`.
`cli.deploy.development.deps.resolve_run_after` threads the
in-container path of the overrides JSON into the resolver
invocation.
Per-round include planner (Phase 3)
-----------------------------------
`cli.deploy.development.inventory.plan_dev_inventory_matrix` now
takes `primary_apps` (just the user-named apps) instead of a
pre-resolved `include` list and returns a 4-tuple per round
`(round_index, inv_dir, round_variants, include_R)`. Per round it:
1. Computes `round_variants` clamped to each primary's variant
count.
2. Builds variant-merged services overrides for every role with a
`meta/variants.yml`, picking that role's variant for the round
(clamped) — so transitive deps with their own variants follow
the round.
3. Runs `CombinedResolver(services_overrides=...)` per primary and
unions the result with the primary apps in user order.
4. Extends `round_variants` with the discovered deps' clamped
variant indices so the deploy step's per-round purge logic stays
accurate.
`build_dev_inventory_matrix` keeps its name but signature changes:
it now takes `primary_apps` and an optional `include_filter` for
SERVICES_DISABLED-style consumer filters.
Init / deploy wiring (Phase 4-5)
--------------------------------
`init.py` and `deploy.py` consume the per-round `include_R`
directly, drop the old `resolve_deploy_ids_for_apps` pre-resolution
step, and apply the SERVICES_DISABLED provider filter on the
include set so the inventory `--include` flag and the final ansible
deploy ids stay in lockstep.
Behaviour invariance for non-variant roles
------------------------------------------
Apps with no `meta/variants.yml`, or without a `services.<X>.enabled`
override on any variant, behave identically to before:
* The override map for those roles is empty — resolver falls
through to the disk-only path.
* Per-round `include_R` collapses to the same set every round.
* Inventory bake outputs the same `applications.<app>.*` shape.
The behaviour change is scoped to roles that legitimately use
variants to flip topology. In the current repo only
`web-app-wordpress` is affected (variant 0 pulls
`web-app-discourse` in; variant 1 leaves it out, exercising the
multisite-without-discourse path).
Test surface
------------
* `tests/unit/cli/deploy/development/test_inventory.py` updated
for the new planner signature, mocking
`_resolve_round_include` / `_build_services_overrides_for_round`
so the planner's tuple shape is asserted independently of the
live resolver.
* `tests/unit/cli/deploy/development/test_matrix_deploy.py` adds a
`_entry()` helper that defaults `include` to the keys of
`round_variants`, so existing fixtures stay terse — round 0 of
the deploy loop deploys whatever is in `include_R`, and the
variant-aware planner naturally emits the same set.
* `tests/unit/utils/cache/test_data.py` pins the planner contract
(`primary_apps=`, 4-tuple plan entries with `include_R`).
* `tests/utils/__init__.py` added so the
`tests.lint.filesystem.test_init_files` lint test passes
end-to-end (the directory existed in the tree without an
__init__.py and tripped the lint).
Verification
------------
* `make test` green (162 tests, 1 skipped) on the merged tree.
* `make deploy-fresh-kept-apps APPS="web-app-wordpress"
FULL_CYCLE=true SERVICES_DISABLED="matomo,email"` end-to-end:
round 0 (variant 0, single-site + Discourse integration) and
round 1 (variant 1, multisite) each ran PASS 1 (sync) and PASS 2
(async), all four passes with `failed=0 rescued=0`. Round 0's
per-round include contained `web-app-discourse`; round 1's did
not — exactly the variant-driven topology the resolver is
supposed to produce.
The unrelated Joomla CLI install race (CI run 25079786137 `Deploy
web-app-joomla`) is parked separately as infinito-nexus/core#209;
its candidate fix is in that issue body, not in this commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
ed04dad5d5 |
refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
9c35909752 |
refactor(utils/cache): split data.py into base/applications/users/domains
The 934-line `utils/cache/data.py` monolith mixed cross-cutting helpers
(constants, yaml-loaders, deep-merge, signatures, render machinery)
with three independent domain caches (applications, users, domains).
Per the KISS/SPOT/DRY guidelines, split it along domain boundaries so
each file has a single, well-scoped responsibility:
utils/cache/__init__.py — package; owns _reset_cache_for_tests()
orchestrator
utils/cache/base.py — cross-cutting helpers: constants,
yaml-loaders, _deep_merge, cache-key +
content-fingerprint signatures, the
cross-domain re-entry guard, and the
templar-render machinery (the only
ansible-coupled symbol, lazy-imports
its dependency)
utils/cache/applications.py — variants + defaults +
get_merged_applications. Strictly
ansible-free at import time so the
GitHub Actions runner-host CLI path
(cli.deploy.development.init →
plan_dev_inventory_matrix →
get_variants) keeps working without
ansible installed
utils/cache/users.py — user defs, token store hydration,
alias materialization,
get_user_defaults, get_merged_users
utils/cache/domains.py — get_merged_domains, derived from the
merged applications view
utils/cache/data.py — DELETED. Façade is gone; the package
exports nothing the domain modules
don't already
Backward-compat: the facade was deleted (per operator request) and all
36+ consumers — 18 plugins, 2 CLI entry points, 9 test files, 4 doc
files, plus a role-internal lookup_plugin — were rewired to import
directly from the owning domain module. `_reset_cache_for_tests` is
now exposed at `from utils.cache import _reset_cache_for_tests`.
Tests added (per-module, focused):
tests/unit/utils/cache/test_base.py (31 cases)
tests/unit/utils/cache/test_applications.py ( 9 cases)
tests/unit/utils/cache/test_users_module.py (21 cases; suffix
disambiguates from
the existing
tests/unit/plugins/
lookup/test_users.py)
tests/unit/utils/cache/test_domains.py ( 6 cases)
The ansible-block invariants from CI runs 24934007615 and 24935979190
are now pinned via subprocess isolation: each ansible-less import test
spawns a fresh `python3 -c '…'` with a meta_path filter denying every
`ansible*` import and asserts the import + invocation succeed in that
clean process. The earlier in-process sys.modules-manipulation shims
fragmented module identity and broke under Python 3.13 inside the
infinito container; subprocess isolation has neither problem.
Doc-style audit (per docs/contributing/documentation.md):
- Removed em-dash from utils/cache/README.md (split into two
sentences)
- Updated stale `utils.runtime_data` reference in
docs/contributing/design/matrix-deploy.md
- Removed two SPOT self-declarations
(docs/agents/files/group_vars/applications.md and
docs/contributing/artefact/files/plugins/lookup/users.md) per
the documentation rule that SPOT status is conferred by
cross-links, not by the page asserting it about itself
Empirical proof:
- All four domain modules import without ansible (subprocess
isolation tests)
- get_variants(roles_dir=ROLES_DIR) returns 99 apps in the
ansible-blocked subprocess
- 142 cache tests + 1085 wider-suite tests pass; the only
remaining failures (test_host_vars free-function fixture bug,
test_runtime_performance timing flake) are pre-existing and
unrelated to this refactor (verified via `git stash`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbach
|
12dba01156 |
refactor(utils): consolidate cache modules into utils/cache/ package
Both in-process cache layers — the cached YAML loader and the cached runtime-data getters — now live next to each other under `utils/cache/`. The package gains a README that documents the "orthogonal layers, no third cache" rule so future contributors hit the right primitive on the first look. Moves: - `utils/yaml_cache.py` -> `utils/cache/yaml.py` - `utils/runtime_data.py` -> `utils/cache/data.py` - `tests/unit/utils/test_yaml_cache.py` -> `tests/unit/utils/cache/test_yaml.py` - `tests/unit/utils/test_runtime_data.py` -> `tests/unit/utils/cache/test_data.py` - `tests/unit/utils/test_variants.py` -> `tests/unit/utils/cache/test_variants.py` New: - `utils/cache/__init__.py` (empty per project convention) - `utils/cache/README.md` describing the two layers, when to reach for which, lifetime, test-only helpers, and design guidelines (RFC 2119 keywords, emoji-after-headline, no self-SPOT declaration, no slash-in-link-text). - `tests/unit/utils/cache/__init__.py` Mechanical updates: - All imports across cli/, plugins/lookup/, tests/, and roles/ rewritten from `utils.yaml_cache` -> `utils.cache.yaml` and `utils.runtime_data` -> `utils.cache.data` (40+ files). - `REPO_ROOT` parents-index in the moved tests bumped from `parents[3]` to `parents[4]` to reflect the deeper nesting. - Six broken markdown links in docs/ updated to the new module paths (test_markdown_links lint test stays green). - Two test files that used the `from utils import runtime_data` form now use `from utils.cache import data as runtime_data` so the local alias remains. Tests: 848 passed across unit + lint suites (1 pre-existing collection error in test_host_vars.py unrelated to this change). |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
d97ed7a9e8 |
feat(deploy): per-role matrix-deploy variants with folder-per-round model
Introduces per-role `meta/variants.yml` for declaring multi-variant
deploys (e.g. WordPress Single-Site vs. Multisite) and reshapes the
development deploy CLI to materialise one inventory folder per variant
and iterate them with co-located sync+async passes.
Why:
- The previous single-inventory + Pass-1-then-Pass-2 split deployed each
variant twice and ran the async pass against a host left in a
different variant's state, masking real bugs.
- Inventory and runtime concerns were tangled: variant selection lived
as a runtime extra-var and required cross-stage coordination.
How:
- `roles/<role>/meta/variants.yml` (file format docs at
docs/contributing/artefact/files/role/variants.md): YAML list of
deep-merge overrides on top of `config/main.yml`. Empty entry `{}` is
variant 0 (= legacy payload). 6 roles get their dev overrides moved
here from inventories/dev.yml; WordPress declares 2 variants.
- `utils/runtime_data.py` gains `_build_variants` / `get_variants` and
drops the retired `_active_variants` runtime selector. Loader always
returns variant 0 as the default; non-zero variants reach
`get_merged_applications` as inventory-level `applications.<app>`
overrides baked at init time.
- New `cli/deploy/development/inventory.py`: `DevInventorySpec`,
`plan_dev_inventory_matrix` (pure planner shared between init and
deploy), `build_dev_inventory_matrix` (materialises one folder per
round at `<dir>-<R>`, or `<dir>` when single-variant). Single-folder
paths preserve full backward compat.
- `cli/deploy/development/deploy.py`: matrix loop with per-round
cleanup (only apps whose variant changed AND that are about to
re-deploy), `--variant <idx>` (or `VARIANT=` env) to pin a single
round, and `--full-cycle` (or `FULL_CYCLE=true`) which interleaves
sync + async per round so the async re-deploy targets exactly the
host state the matching sync deploy just produced.
- `cli/deploy/development/exec.py` gains repeatable `--env KEY=VAL` so
bash callers can inject per-run context into in-container helpers
without inlining heredocs.
- `scripts/meta/env/inventory.sh` exports `INVENTORY_VARS_FILE`
(Python mirror in `cli.deploy.development.common.DEV_INVENTORY_VARS_FILE`;
test_inventory_vars_file_spot.py locks them against drift).
- `make deploy-fresh-purged-apps` and `scripts/tests/deploy/ci/dedicated.sh`
switch to a single init + single deploy `--full-cycle`. The
`make deploy-reuse-*` bash scripts honour `VARIANT=` by suffixing
`INVENTORY_DIR`/`INVENTORY_FILE`.
- The 5 `scripts/tests/deploy/local/deploy/*.sh` scripts now delegate
their in-container heredocs to dedicated helpers in
`scripts/tests/deploy/local/utils/`, removing the bash-in-bash
quoting hell.
Tests:
- `tests/unit/utils/test_variants.py` (renamed from
`test_application_variants.py`): loader edge cases, variant-zero
default, retired-`_active_variants` guard.
- `tests/unit/cli/deploy/development/test_inventory.py` (new): planner,
variant baking, builder, `filter_plan_to_variant`, `DevInventorySpec`.
- `tests/unit/cli/deploy/development/test_matrix_deploy.py` (new):
per-round deploy filter, cleanup decisions, `--variant` pinning,
`--full-cycle` per-round interleave.
- `tests/unit/cli/deploy/development/test_inventory_vars_file_spot.py`
(new): drift guard for the bash <-> Python literal pair.
- `tests/unit/cli/deploy/development/test_exec.py` extended for `--env`.
Docs:
- `docs/contributing/artefact/files/role/variants.md` (new): file format.
- `docs/contributing/design/matrix-deploy.md` (new): runtime mechanism
(folder layout, --variant, --full-cycle, cleanup logic).
- `docs/contributing/actions/deploy.md` (new): contributor dev-deploy
flow (make targets, --variant pinning).
- `docs/administration/deploy.md` slimmed to admin-only (was mixed
admin+dev), cross-links to the new contributor doc.
- `docs/agents/action/iteration/role.md` keeps only agent-iteration
recipes; mechanism explanation moved out and linked.
- `docs/contributing/documentation.md` adds the "no self-SPOT
declaration" rule; `applications.md` renamed its SoT header for
consistency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|