mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 19:01:54 +00:00
kevinveenbirkenbachandClaude Opus 5.5
0cb8bae88d
Every cold render of the applications SPOT walks the templates of all
269 roles, and those templates call lookup('config'), lookup('tls'),
lookup('domain') and others. Each of these re-enters
get_merged_applications while the render guard is set, and each one
rebuilt the unrendered view from scratch: a deepcopy of all role
defaults plus the override merge, about 8 ms, never cached. A local
cold render of the development inventory made 351 such calls, 2.82 s
of a 6.86 s render.
The outer render now keeps a copy of its unrendered view and its cache
key on the render guard, taken before the inventory volumes are
popped, so it is the same view a nested call used to build. A nested
call with the same key returns it; any other key still builds its own.
The guard drops it when the render ends.
Measured on the same inventory: the nested calls took 0.07 s instead
of 2.82 s and the cold render 3.60 s instead of 6.86 s. The rendered
SPOT is unchanged: the old and new dumps differ in 69 user passwords,
which the development inventory regenerates on every read (two runs of
the old code differ in 68), and in one absolute checkout path, the only
other difference.
In CI each cold render took 40 to 77 s and the heaviest swarm cell of
#950 missed its budget by a few minutes (#984).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
91 lines
7.8 KiB
Markdown
91 lines
7.8 KiB
Markdown
# Cache Helpers 💾
|
|
|
|
This package contains the in-process cache layers Infinito.Nexus uses to keep CLI tools, lookup plugins, and Ansible filters fast without re-reading or re-merging the same data multiple times per Python process.
|
|
|
|
## Modules 🧭
|
|
|
|
| Module | Caches |
|
|
|---|---|
|
|
| [yaml.py](yaml.py) | A single YAML file's parsed root, keyed by absolute path. Every other module routes its YAML reads through here so each file is parsed at most once per process. |
|
|
| [files.py](files.py) | The project tree's full path list (one walk per process) and per-file UTF-8 contents (`read_text`). Used by lint/integration tests and CLI tools that scan the repo. |
|
|
| [base.py](base.py) | Cross-cutting helpers: filesystem constants (`PROJECT_ROOT`, `ROLES_DIR`, `DEFAULT_TOKENS_FILE`), `_deep_merge`, cache-key + content-fingerprint signatures, the cross-domain re-entry guard, and the templar-render machinery. The only ansible-coupled symbol is `_render_with_templar`, which lazy-imports its dependency. |
|
|
| [applications.py](applications.py) | Per-app variants + defaults + `get_merged_applications`. **Strictly ansible-free at import time** so the GitHub Actions runner-host CLI path (`cli.administration.deploy.development.init` → `plan_dev_inventory_matrix` → `get_variants`) keeps working without ansible installed. |
|
|
| [carrier.py](carrier.py) | The play-scoped carrier of the rendered applications payload: the fact name `_INFINITO_APPLICATIONS_RENDERED`, `merged_applications_cache_key`, and the reader `get_merged_applications` consults before it renders. |
|
|
| [users/](users/) | User definitions, token store hydration, alias materialization, `get_user_defaults`, `get_merged_users` (package; `users/placeholders.py` substitutes `DOMAIN_PRIMARY` / `ORGANIZATION` / `SOFTWARE_NAME` ahead of the templar render pass). |
|
|
| [domains.py](domains.py) | Canonical-domains map derived from the merged applications view: `get_merged_domains`. A read issued while the applications render is in progress (the re-entry guard is set) sees the unrendered view, so it is cached under its own key and never answers a later read. |
|
|
| [`__init__.py`](__init__.py) | Owns the package-level `_reset_cache_for_tests()` orchestrator that clears every cache plus the shared fingerprint memo in one call. |
|
|
|
|
## When To Use Which 🎯
|
|
|
|
- Use `utils.cache.yaml.load_yaml(path)` (or `load_yaml_any(path)` when the YAML root is a list or scalar) whenever a CLI tool, filter, or library reads individual YAML files where the same path MAY be touched repeatedly within a single invocation.
|
|
- Import the cached getters directly from the domain module that owns them: `utils.cache.applications` for `get_application_defaults` / `get_variants` / `get_merged_applications`, `utils.cache.users` for `get_user_defaults` / `get_merged_users`, and `utils.cache.domains` for `get_merged_domains`. The package's `_reset_cache_for_tests` is exposed at `from utils.cache import _reset_cache_for_tests`.
|
|
|
|
The layers are orthogonal: `yaml.py` caches "this file's parsed root", the `applications`/`users`/`domains` modules cache "this aggregated, rendered payload". They do not invalidate each other and MAY both be active at the same time.
|
|
|
|
## Lifetime 🕒
|
|
|
|
Both layers are process-wide in-process caches. They intentionally do NOT track on-disk file mtimes; CLI tools are short-lived and the assumption "the on-disk file does not change while my process runs" holds. The single exception is `utils.cache.yaml.dump_yaml`, which writes through and evicts the cached entry for the path it just wrote so a tool that edits a file mid-run sees the new content on the next read.
|
|
|
|
Inside a play the process boundary is the task: Ansible forks a fresh worker per task, so `_MERGED_APPLICATIONS_CACHE` starts empty in every worker. `get_merged_applications` therefore also accepts the play-scoped carrier fact `_INFINITO_APPLICATIONS_RENDERED` (payload plus cache key, parked by the constructor stage and again after every role, every nested application load and every persisted token) and serves it when the key matches. See [applications.md](../../docs/contributing/artefact/files/plugins/lookup/applications.md) for the contract.
|
|
|
|
While one render runs, the re-entry guard also holds that render's unrendered view and its cache key. A nested `get_merged_applications` call from a lookup inside a role template (`config`, `tls`, `domain`, ...) with the same key returns that view instead of copying and merging the role defaults again. The view is shared by every nested call of the render, so callers MUST NOT mutate it. The guard drops it when the render ends.
|
|
|
|
## Test-Only Helpers 🧪
|
|
|
|
Each module owns a private `_reset()` that clears its own cache dicts. The package-level `utils.cache._reset_cache_for_tests()` orchestrates all six (`base`, `applications`, `users`, `domains`, `yaml`, `files`) plus the shared fingerprint memo. Unit tests that exercise the cached paths MUST call it in `setUp` to guarantee clean state across test cases.
|
|
|
|
## Design Guidelines 📐
|
|
|
|
- You MUST NOT introduce a third parallel cache for the same concern. If you need to cache a derived shape, add a getter to the appropriate domain module (`applications`, `users`, or `domains`) that builds on the existing primitives.
|
|
- You SHOULD prefer the explicit cached getter over re-implementing the same memoisation in a calling module.
|
|
- You MUST NOT mutate values returned by `utils.cache.yaml.load_yaml`. The same dict instance is returned to every caller; callers that need to mutate MUST `copy.deepcopy()` the result first. Each domain getter documents whether it returns a fresh deep copy or the cached instance; respect that contract.
|
|
- You MUST keep dependencies inside this package minimal. Cache primitives MUST remain importable from filter plugins, lookup plugins, and CLI tools without dragging in heavy dependencies. **Specifically: `applications.py` MUST stay ansible-free at import time.** See CI runs 24934007615 / 24935979190 for the regression that motivated the split. Tests in `tests/unit/python/utils/cache/test_data.py::TestImportableWithoutAnsible` pin both the import-time and call-time invariants.
|
|
|
|
## Example Imports ✍️
|
|
|
|
```python
|
|
from utils.cache.yaml import load_yaml, load_yaml_any
|
|
|
|
from utils.cache.applications import (
|
|
get_application_defaults,
|
|
get_merged_applications,
|
|
get_variants,
|
|
)
|
|
from utils.cache.users import get_merged_users, get_user_defaults
|
|
from utils.cache.domains import get_merged_domains
|
|
|
|
# Tests
|
|
from utils.cache import _reset_cache_for_tests
|
|
```
|
|
|
|
## Schema 🗺️
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
decl["meta/services.yml<br/>a flag reading group_names<br/>114 role files"]
|
|
ctor["01_constructor.yml:191<br/>applications_carrier"]
|
|
fact["_INFINITO_APPLICATIONS_RENDERED<br/>per-host fact"]
|
|
lookup["lookup('database', ...)<br/>and every applications consumer"]
|
|
key["base.py:103<br/>_stable_variables_signature"]
|
|
proc["_MERGED_APPLICATIONS_CACHE<br/>process-global dict"]
|
|
render["base.py:198<br/>_render_with_templar"]
|
|
consumer["database.py:121<br/>central_enabled = shared"]
|
|
name["postgres_postgres<br/>vs pretix_database"]
|
|
|
|
decl -->|"read by"| render
|
|
ctor -->|"parks"| fact
|
|
lookup -->|"computes"| key
|
|
key -->|"1. looked up in"| proc
|
|
proc -->|"2. miss: falls back to"| fact
|
|
fact -->|"3. miss: renders"| render
|
|
render -->|"substitutes group_names,<br/>then stores under key"| proc
|
|
render --> consumer
|
|
consumer -->|"selects"| name
|
|
|
|
proc -.->|"TRIP-WIRE: read BEFORE the<br/>host's own fact, so an inherited<br/>entry outranks it"| fact
|
|
render -.->|"TRIP-WIRE: main process renders<br/>task names for one host; forked<br/>workers inherit that entry"| proc
|
|
key -.->|"TRIP-WIRE: carrier.py:68 stringifies<br/>only one side, so a non-scalar key<br/>part silently never matches again"| fact
|
|
```
|
|
|
|
The key covers `group_names`. `domains.py` derives from the applications view and shares that dependency; `users` reads `group_names` nowhere.
|