computer-playbook/utils/cache

Cache Helpers 💾

This package contains the in-process cache layers Infinito.Nexus uses to keep CLI tools, lookup plugins, and Ansible filters fast without re-reading or re-merging the same data multiple times per Python process.

Modules 🧭

Module	Caches
yaml.py	A single YAML file's parsed root, keyed by absolute path. Every other module routes its YAML reads through here so each file is parsed at most once per process.
files.py	The project tree's full path list (one walk per process) and per-file UTF-8 contents (read_text). Used by lint/integration tests and CLI tools that scan the repo.
base.py	Cross-cutting helpers: filesystem constants (PROJECT_ROOT, ROLES_DIR, DEFAULT_TOKENS_FILE), _deep_merge, cache-key + content-fingerprint signatures, the cross-domain re-entry guard, and the templar-render machinery. The only ansible-coupled symbol is _render_with_templar, which lazy-imports its dependency.
applications.py	Per-app variants + defaults + get_merged_applications. Strictly ansible-free at import time so the GitHub Actions runner-host CLI path (cli.administration.deploy.development.init → plan_dev_inventory_matrix → get_variants) keeps working without ansible installed.
carrier.py	The play-scoped carriers of the rendered applications and users payloads: the fact names _INFINITO_APPLICATIONS_RENDERED and _INFINITO_USERS_RENDERED, merged_applications_cache_key, merged_users_cache_key, and the readers get_merged_applications and get_merged_users consult before they render.
users/	User definitions, token store hydration, alias materialization, get_user_defaults, get_merged_users (package; users/placeholders.py substitutes DOMAIN_PRIMARY / ORGANIZATION / SOFTWARE_NAME ahead of the templar render pass; users/store.py reads the sys_token_store_users fact and digests the tokens the users are hydrated from, the digest the users cache and users carrier keys share).
domains.py	Canonical-domains map derived from the merged applications view: get_merged_domains. A read issued while the applications render is in progress (the re-entry guard is set) sees the unrendered view, so it is cached under its own key and never answers a later read.
__init__.py	Owns the package-level _reset_cache_for_tests() orchestrator that clears every cache plus the shared fingerprint memo in one call.

When To Use Which 🎯

• Use utils.cache.yaml.load_yaml(path) (or load_yaml_any(path) when the YAML root is a list or scalar) whenever a CLI tool, filter, or library reads individual YAML files where the same path MAY be touched repeatedly within a single invocation.
• Import the cached getters directly from the domain module that owns them: utils.cache.applications for get_application_defaults / get_variants / get_merged_applications, utils.cache.users for get_user_defaults / get_merged_users, and utils.cache.domains for get_merged_domains. The package's _reset_cache_for_tests is exposed at from utils.cache import _reset_cache_for_tests.

The layers are orthogonal: yaml.py caches "this file's parsed root", the applications/users/domains modules cache "this aggregated, rendered payload". They do not invalidate each other and MAY both be active at the same time.

Lifetime 🕒

Both layers are process-wide in-process caches. They intentionally do NOT track on-disk file mtimes; CLI tools are short-lived and the assumption "the on-disk file does not change while my process runs" holds. The single exception is utils.cache.yaml.dump_yaml, which writes through and evicts the cached entry for the path it just wrote so a tool that edits a file mid-run sees the new content on the next read.

Inside a play the process boundary is the task: Ansible forks a fresh worker per task, so _MERGED_APPLICATIONS_CACHE and _MERGED_USERS_CACHE start empty in every worker. get_merged_applications therefore also accepts the play-scoped carrier fact _INFINITO_APPLICATIONS_RENDERED (payload plus cache key, parked by the constructor stage and again after every role, every nested application load and every persisted token) and serves it when the key matches. The same action parks _INFINITO_USERS_RENDERED, which get_merged_users serves to templated reads under the applications key plus the token store digest and inventory_hostname. See applications.md and users.md for the contract.

While one render runs, the re-entry guard also holds that render's unrendered view and its cache key. A nested get_merged_applications call from a lookup inside a role template (config, tls, domain, ...) with the same key returns that view instead of copying and merging the role defaults again. The view is shared by every nested call of the render, so callers MUST NOT mutate it. The guard drops it when the render ends.

Test-Only Helpers 🧪

Each module owns a private _reset() that clears its own cache dicts. The package-level utils.cache._reset_cache_for_tests() orchestrates all six (base, applications, users, domains, yaml, files) plus the shared fingerprint memo. Unit tests that exercise the cached paths MUST call it in setUp to guarantee clean state across test cases.

Design Guidelines 📐

• You MUST NOT introduce a third parallel cache for the same concern. If you need to cache a derived shape, add a getter to the appropriate domain module (applications, users, or domains) that builds on the existing primitives.
• You SHOULD prefer the explicit cached getter over re-implementing the same memoisation in a calling module.
• You MUST NOT mutate values returned by utils.cache.yaml.load_yaml. The same dict instance is returned to every caller; callers that need to mutate MUST copy.deepcopy() the result first. Each domain getter documents whether it returns a fresh deep copy or the cached instance; respect that contract.
• You MUST keep dependencies inside this package minimal. Cache primitives MUST remain importable from filter plugins, lookup plugins, and CLI tools without dragging in heavy dependencies. Specifically: applications.py MUST stay ansible-free at import time. See CI runs 24934007615 / 24935979190 for the regression that motivated the split. Tests in tests/unit/python/utils/cache/test_data.py::TestImportableWithoutAnsible pin both the import-time and call-time invariants.

Example Imports ✍️

from utils.cache.yaml import load_yaml, load_yaml_any

from utils.cache.applications import (
    get_application_defaults,
    get_merged_applications,
    get_variants,
)
from utils.cache.users import get_merged_users, get_user_defaults
from utils.cache.domains import get_merged_domains

# Tests
from utils.cache import _reset_cache_for_tests

Schema 🗺️

flowchart TD
    decl["meta/services.yml<br/>a flag reading group_names<br/>114 role files"]
    ctor["01_constructor.yml:191<br/>applications_carrier"]
    fact["_INFINITO_APPLICATIONS_RENDERED<br/>per-host fact"]
    lookup["lookup('database', ...)<br/>and every applications consumer"]
    key["base.py:103<br/>_stable_variables_signature"]
    proc["_MERGED_APPLICATIONS_CACHE<br/>process-global dict"]
    render["base.py:198<br/>_render_with_templar"]
    consumer["database.py:121<br/>central_enabled = shared"]
    name["postgres_postgres<br/>vs pretix_database"]

    decl -->|"read by"| render
    ctor -->|"parks"| fact
    lookup -->|"computes"| key
    key -->|"1. looked up in"| proc
    proc -->|"2. miss: falls back to"| fact
    fact -->|"3. miss: renders"| render
    render -->|"substitutes group_names,<br/>then stores under key"| proc
    render --> consumer
    consumer -->|"selects"| name

    proc -.->|"TRIP-WIRE: read BEFORE the<br/>host's own fact, so an inherited<br/>entry outranks it"| fact
    render -.->|"TRIP-WIRE: main process renders<br/>task names for one host; forked<br/>workers inherit that entry"| proc
    key -.->|"TRIP-WIRE: carrier.py:121 stringifies<br/>only one side, so a non-scalar key<br/>part silently never matches again"| fact

The key covers group_names. domains.py derives from the applications view and shares that dependency; users reads group_names nowhere.