perf(applications): render the applications SPOT once per play instead of once per task

Ansible forks a fresh worker process for every task, so the in-process
cache behind lookup('applications') was cold in every task. Each task that
touched the merged applications tree (config, domains, nginx, tls, cdn and
their callers) paid a full render: about 16 s on main and 21 to 23 s on this
branch, where ten more application roles carry MCP and AI Jinja. Measured on
the workspace jobs of run 33574056480 against the main run: 331 such tasks
(93 min, 39 % of the job) versus 391 (141 min, 43 %); the same unit is 9 to
43 % of every compose and swarm deploy job on disk. The parent process
rendered once per play, late, while templating a task name, and only the
tasks after that point ran warm.

The constructor now renders once through the new applications_carrier
action and parks the payload with its cache key in the host fact
_INFINITO_APPLICATIONS_RENDERED. Every later worker inherits the fact at
fork time and get_merged_applications serves it while the key (roles dir,
applications, users, DOMAIN_PRIMARY, SYSTEM_EMAIL_DOMAIN) still matches; a
mismatch falls back to the old render. The action masks its own result: the
payload is the whole tree with every credential, and CI runs with
MASK_CREDENTIALS_IN_LOGS false.

The task sits after the networks, design and service_provider merges, after
the add_host placements and after the token store. An independent review
showed that an earlier position bakes stale values into the play: the
web-app-wordpress title rendered to 'Blog by ' because service_provider did
not exist yet. User records inside the payload are the token store's state
at render time; readers of tokens captured later keep using lookup('users')
or lookup('config', app, 'users.…'), which read the users cache fresh.

Expected savings per job: about 130 min on a fork workspace run, 16 min on
an average compose deploy and 28 min on an average swarm deploy. The
constructor's own lookups before the carrier stay cold.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kevinveenbirkenbachandClaude Fable 5.1 committed 2026-09-02 23:55:32 +02:00
1 parent 03e42bae6d
commit 113113e553
11 files changed
+350 -10

No files matched your search

@@ -18,6 +18,7 @@ The `applications` lookup is the ONLY supported runtime entry point for merged a
| `lookup('applications')` | Full merged mapping keyed by `application_id`. |
| `lookup('applications', '<application_id>')` | Merged entry for that application. Raises when missing. |
| `lookup('applications', '<application_id>', default_value)` | Merged entry, or `default_value` when the entry is missing. |
| `lookup('applications', carrier=True)` | `{"key": [...], "applications": {...}}`: the full mapping plus the cache key it is valid under. Reserved for the constructor's carrier task (see below). |
## Data Sources 📚
@@ -26,7 +27,19 @@ The lookup merges exactly two sources:
1. **Defaults** discovered from each role's per-topic meta files: `roles/*/meta/server.yml`, `roles/*/meta/rbac.yml`, `roles/*/meta/services.yml`, `roles/*/meta/volumes.yml`, plus `roles/*/meta/secrets.yml` (post-`apply_schema()`). Variants from `roles/*/meta/variants.yml` deep-merge over the assembled per-role payload. See [layout.md](../../../../design/role/services/layout.md).
2. **Overrides** supplied through the normal Ansible variable `applications` in inventory, group vars, host vars, or role vars.
No intermediate merged `applications` fact exists. No other source is consulted.
No other source is consulted.
## Play-Scoped Carrier 🧊
Ansible forks a fresh worker process for every task. The in-process cache inside `get_merged_applications` therefore never survives from one task to the next, and every task that touches the lookup would render the full tree again (about 16 to 23 seconds per task on a CI runner, measured on the workspace jobs).
The constructor stage renders once through the [applications_carrier](../../../../../../plugins/action/applications_carrier.py) action and parks the result together with its cache key in the host fact `_INFINITO_APPLICATIONS_RENDERED` (the action calls `lookup('applications', carrier=True)`). Every later worker inherits that fact at fork time; `get_merged_applications` serves it whenever the key still matches and renders only when it does not (a changed `applications`, `users`, `DOMAIN_PRIMARY` or `SYSTEM_EMAIL_DOMAIN`).
- The task sits in `tasks/stages/01_constructor.yml` after the `networks`, `design` and `service_provider` merges, after the `add_host` placements and after the token store, so the payload renders with the same `group_names`, `groups` and user records a role would see. Moving it earlier bakes stale values into the play.
- The action masks its own result: the fact is the whole rendered tree with every application's credentials, so it never reaches a log regardless of `MASK_CREDENTIALS_IN_LOGS`.
- User records inside the payload are the token store's state at render time. Readers of tokens captured later in the play MUST go through `lookup('users', ...)` or `lookup('config', <app>, 'users.<key>...')`, both of which read the users cache fresh.
- The carrier is private to the lookup. Consumers MUST NOT read `_INFINITO_APPLICATIONS_RENDERED`; they keep going through `lookup('applications')`, which is what makes the carrier transparent.
- The carrier is the only materialised merged view a play MAY hold. The rule against a top-level `applications` fact above is unchanged.
## Adding an Application Entry ➕
@@ -69,6 +82,7 @@ applications:
|---|---|
| [applications.py](../../../../../../plugins/lookup/applications.py) | Runtime entry point for the `applications` lookup. |
| [applications.py (cache)](../../../../../../utils/cache/applications.py) | Shared aggregation helper that builds and caches defaults (`get_application_defaults`, `get_variants`, `get_merged_applications`). |
| [carrier.py](../../../../../../utils/cache/carrier.py) | The play-scoped carrier: fact name, `merged_applications_cache_key`, and the reader that serves the carried payload. |
| [test_applications.py](../../../../../../tests/unit/python/plugins/lookup/test_applications.py) | Unit tests covering the full-dict, single-entry, override, strict missing, and non-strict missing cases. |
For the related users pattern see [users.md](users.md).
+2
View File
@@ -47,6 +47,8 @@ Rendering happens at **play runtime**, never during the build pipeline above. Th
`get_merged_applications` is invoked transitively by `lookup('applications', ...)` and by `lookup('config', application_id, 'services.<x>.<y>')`. Tasks that read service flags through these lookups always see real Python values, never raw Jinja strings.
The render runs once per play. The constructor stage parks the payload and its cache key in the host fact `_INFINITO_APPLICATIONS_RENDERED`, and every task's worker process (Ansible forks one per task) serves that payload instead of rendering again. See [applications.md](../artefact/files/plugins/lookup/applications.md) for the carrier contract.
A role's task that reads its own service flag therefore picks up the runtime-resolved value, even when the inventory carried a Jinja string for that flag: the conditional evaluates against the play's actual `group_names`, which reflects which other application groups the host joined.
## Interaction between the three inclusion mechanisms 🧬
+55
View File
@@ -0,0 +1,55 @@
"""Render the applications SPOT once and park it as the play-scoped carrier fact.
Ansible forks a fresh worker per task, so the in-process render cache of
``lookup('applications')`` is cold in every task until something renders in
a place all later workers inherit. This action renders in the worker of one
constructor task and returns the payload as a host fact; every later worker
inherits that fact at fork time and ``get_merged_applications`` serves it
instead of rendering. Sits after the token store in
``tasks/stages/01_constructor.yml`` so the embedded user records already
carry the loaded tokens::
- name: 🧊 Render the applications SPOT once and carry it into every task's worker
applications_carrier: {}
The result is masked by the action itself: the fact is the whole rendered
tree, about 0.4 MB with every application's credentials, and printing it at
``-v`` is neither credential debugging nor readable.
Args:
none. Any argument is an error.
"""
from __future__ import annotations
from typing import Any
from ansible.errors import AnsibleActionFail
from ansible.plugins.action import ActionBase
from ansible.plugins.loader import lookup_loader
from utils.cache.carrier import APPLICATIONS_RENDERED_FACT
class ActionModule(ActionBase):
TRANSFERS_FILES = False
def run(
self, tmp: Any = None, task_vars: dict[str, Any] | None = None
) -> dict[str, Any]:
result = super().run(tmp, task_vars)
task_vars = task_vars or {}
if self._task.args:
raise AnsibleActionFail("applications_carrier takes no arguments")
lookup = lookup_loader.get(
"applications", loader=self._loader, templar=self._templar
)
carrier = lookup.run([], variables=task_vars, carrier=True)[0]
result.update(
changed=False,
ansible_facts={APPLICATIONS_RENDERED_FACT: carrier},
_ansible_no_log=True,
)
return result
+23 -4
View File
@@ -9,6 +9,7 @@ from utils.cache import _reset_cache_for_tests as _reset_runtime_lookup_cache
# nocheck: lookup-cache-import (this lookup IS the applications SPOT provider)
from utils.cache.applications import get_merged_applications
from utils.cache.carrier import merged_applications_cache_key
def _reset_cache_for_tests() -> None:
@@ -22,21 +23,39 @@ class LookupModule(LookupBase):
variables: dict[str, Any] | None = None,
**kwargs: Any,
) -> list[Any]:
"""
Terms:
0) full merged mapping
1) application_id
2) application_id, default value
Keyword arguments:
roles_dir: roles tree to build from (defaults to the repository's)
carrier: with no terms, return ``{"key": [...], "applications": {...}}``
for the constructor to park in `_INFINITO_APPLICATIONS_RENDERED`
"""
terms = terms or []
if len(terms) > 2:
raise AnsibleError(
"applications: expected 0, 1, or 2 terms: "
"lookup('applications'[, application_id[, default]])"
)
carrier = bool(kwargs.get("carrier", False))
if carrier and terms:
raise AnsibleError("applications: carrier=True takes no terms")
variables = variables or getattr(self._templar, "available_variables", {}) or {}
roles_dir = kwargs.get("roles_dir")
applications = get_merged_applications(
variables=variables
or getattr(self._templar, "available_variables", {})
or {},
roles_dir=kwargs.get("roles_dir"),
variables=variables,
roles_dir=roles_dir,
templar=getattr(self, "_templar", None),
)
if carrier:
key = merged_applications_cache_key(variables, roles_dir=roles_dir)
return [{"key": [key[0], list(key[1])], "applications": applications}]
if len(terms) == 0:
return [applications]
+3
View File
@@ -180,6 +180,9 @@
name: sys-token-store
when: run_once_sys_token_store is not defined
- name: 🧊 Render the applications SPOT once and carry it into every task's worker
applications_carrier: {}
- name: 🛠️ Preload shared services
ansible.builtin.include_role:
name: sys-service-loader
@@ -0,0 +1,73 @@
import unittest
from unittest.mock import patch
from ansible.errors import AnsibleActionFail
from plugins.action.applications_carrier import ActionModule
from utils.cache.carrier import APPLICATIONS_RENDERED_FACT
class _FakeTask:
def __init__(self, args=None):
self.args = {} if args is None else dict(args)
class _FakeLookup:
def __init__(self, carrier):
self.carrier = carrier
self.calls = []
def run(self, terms, variables=None, **kwargs):
self.calls.append((terms, variables, kwargs))
return [self.carrier]
def _make_action(task: _FakeTask) -> ActionModule:
action = object.__new__(ActionModule)
action._task = task
action._loader = None
action._templar = None
return action
class TestApplicationsCarrierActionPlugin(unittest.TestCase):
def test_parks_the_carrier_as_a_masked_fact(self):
carrier = {"key": ["roles", ["a", "b", "c", "d"]], "applications": {"x": {}}}
lookup = _FakeLookup(carrier)
action = _make_action(_FakeTask())
with (
patch(
"plugins.action.applications_carrier.ActionBase.run",
autospec=True,
return_value={},
),
patch(
"plugins.action.applications_carrier.lookup_loader.get",
return_value=lookup,
),
):
result = action.run(task_vars={"applications": {}})
self.assertIs(result["ansible_facts"][APPLICATIONS_RENDERED_FACT], carrier)
self.assertTrue(result["_ansible_no_log"])
self.assertFalse(result["changed"])
terms, variables, kwargs = lookup.calls[0]
self.assertEqual(terms, [])
self.assertEqual(variables, {"applications": {}})
self.assertTrue(kwargs["carrier"])
def test_rejects_arguments(self):
action = _make_action(_FakeTask(args={"foo": 1}))
with (
patch(
"plugins.action.applications_carrier.ActionBase.run",
autospec=True,
return_value={},
),
self.assertRaises(AnsibleActionFail),
):
action.run(task_vars={})
if __name__ == "__main__":
unittest.main()
@@ -9,6 +9,7 @@ from ansible.errors import AnsibleError
from plugins.lookup.applications import LookupModule, _reset_cache_for_tests
from utils.cache import base as cache_base
from utils.cache.carrier import merged_applications_cache_key
from utils.cache.yaml import dump_yaml_str
from utils.roles.mapping import ROLE_FILE_META_SERVICES
@@ -103,6 +104,32 @@ class TestApplicationsLookup(unittest.TestCase):
)[0]
self.assertEqual(result, default_value)
def test_carrier_form_returns_cache_key_and_full_mapping(self) -> None:
variables = {
"applications": {
"web-app-foo": {"services": {"smtp": {"host": "override.example.org"}}}
}
}
roles_dir = str(self._tmp / "roles")
result = self.lookup.run(
[], variables=variables, roles_dir=roles_dir, carrier=True
)[0]
key = merged_applications_cache_key(variables, roles_dir=roles_dir)
self.assertEqual(result["key"], [key[0], list(key[1])])
self.assertEqual(
result["applications"]["web-app-foo"]["services"]["smtp"]["host"],
"override.example.org",
)
def test_carrier_form_rejects_terms(self) -> None:
with self.assertRaises(AnsibleError):
self.lookup.run(
["web-app-foo"],
variables={},
roles_dir=str(self._tmp / "roles"),
carrier=True,
)
if __name__ == "__main__":
unittest.main()
+69
View File
@@ -19,6 +19,10 @@ from pathlib import Path
from utils.cache import _reset_cache_for_tests
from utils.cache import applications as cache_apps
from utils.cache.carrier import (
APPLICATIONS_RENDERED_FACT,
merged_applications_cache_key,
)
from utils.cache.yaml import load_yaml_any
from utils.roles.mapping import (
ROLE_DIR_META_ADDONS,
@@ -335,6 +339,71 @@ class TestGetMergedApplicationsRespectsOverrides(unittest.TestCase):
)
class TestGetMergedApplicationsCarrierFact(unittest.TestCase):
"""The constructor renders once and parks the payload with its cache key in
`_INFINITO_APPLICATIONS_RENDERED`. A worker process whose in-process cache is
empty serves that payload when the key matches and renders only when it
does not.
"""
def setUp(self) -> None:
_reset_cache_for_tests()
@staticmethod
def _variables() -> dict:
return {
"applications": {
"web-app-foo": {"services": {"foo": {"image": "override"}}}
}
}
def test_matching_key_serves_the_carried_payload_without_rendering(self):
with tempfile.TemporaryDirectory() as tmp:
roles = _seed_minimal_roles(Path(tmp))
variables = self._variables()
key = merged_applications_cache_key(variables, roles_dir=roles)
carried = {"web-app-foo": {"carried": True}}
variables[APPLICATIONS_RENDERED_FACT] = {
"key": [key[0], list(key[1])],
"applications": carried,
}
merged = cache_apps.get_merged_applications(
variables=variables, roles_dir=roles, templar=None
)
self.assertIs(merged, carried)
def test_stale_key_falls_back_to_a_fresh_render(self):
with tempfile.TemporaryDirectory() as tmp:
roles = _seed_minimal_roles(Path(tmp))
variables = self._variables()
key = merged_applications_cache_key(variables, roles_dir=roles)
variables[APPLICATIONS_RENDERED_FACT] = {
"key": [key[0], ["stale"] * len(key[1])],
"applications": {"web-app-foo": {"carried": True}},
}
merged = cache_apps.get_merged_applications(
variables=variables, roles_dir=roles, templar=None
)
self.assertNotIn("carried", merged["web-app-foo"])
self.assertEqual(
merged["web-app-foo"]["services"]["foo"]["image"], "override"
)
def test_malformed_carrier_is_ignored(self):
with tempfile.TemporaryDirectory() as tmp:
roles = _seed_minimal_roles(Path(tmp))
for carrier in ("not-a-mapping", {"key": "x", "applications": {}}, {}):
_reset_cache_for_tests()
variables = self._variables()
variables[APPLICATIONS_RENDERED_FACT] = carrier
merged = cache_apps.get_merged_applications(
variables=variables, roles_dir=roles, templar=None
)
self.assertEqual(
merged["web-app-foo"]["services"]["foo"]["image"], "override"
)
class TestRecursiveMetaWalk(unittest.TestCase):
"""`_build_role_base_config` walks the whole `meta/` tree: every
`meta/**/*.yml` lands at the matching nested path under
+3
View File
@@ -10,6 +10,7 @@ This package contains the in-process cache layers Infinito.Nexus uses to keep CL
| [files.py](files.py) | The project tree's full path list (one walk per process) and per-file UTF-8 contents (`read_text`). Used by lint/integration tests and CLI tools that scan the repo. |
| [base.py](base.py) | Cross-cutting helpers: filesystem constants (`PROJECT_ROOT`, `ROLES_DIR`, `DEFAULT_TOKENS_FILE`), `_deep_merge`, cache-key + content-fingerprint signatures, the cross-domain re-entry guard, and the templar-render machinery. The only ansible-coupled symbol is `_render_with_templar`, which lazy-imports its dependency. |
| [applications.py](applications.py) | Per-app variants + defaults + `get_merged_applications`. **Strictly ansible-free at import time** so the GitHub Actions runner-host CLI path (`cli.administration.deploy.development.init` → `plan_dev_inventory_matrix` → `get_variants`) keeps working without ansible installed. |
| [carrier.py](carrier.py) | The play-scoped carrier of the rendered applications payload: the fact name `_INFINITO_APPLICATIONS_RENDERED`, `merged_applications_cache_key`, and the reader `get_merged_applications` consults before it renders. |
| [users/](users/) | User definitions, token store hydration, alias materialization, `get_user_defaults`, `get_merged_users` (package; `users/placeholders.py` substitutes `DOMAIN_PRIMARY` / `ORGANIZATION` / `SOFTWARE_NAME` ahead of the templar render pass). |
| [domains.py](domains.py) | Canonical-domains map derived from the merged applications view: `get_merged_domains`. |
| [`__init__.py`](__init__.py) | Owns the package-level `_reset_cache_for_tests()` orchestrator that clears every cache plus the shared fingerprint memo in one call. |
@@ -25,6 +26,8 @@ The layers are orthogonal: `yaml.py` caches "this file's parsed root", the `appl
Both layers are process-wide in-process caches. They intentionally do NOT track on-disk file mtimes; CLI tools are short-lived and the assumption "the on-disk file does not change while my process runs" holds. The single exception is `utils.cache.yaml.dump_yaml`, which writes through and evicts the cached entry for the path it just wrote so a tool that edits a file mid-run sees the new content on the next read.
Inside a play the process boundary is the task: Ansible forks a fresh worker per task, so `_MERGED_APPLICATIONS_CACHE` starts empty in every worker. `get_merged_applications` therefore also accepts the play-scoped carrier fact `_INFINITO_APPLICATIONS_RENDERED` (payload plus cache key, parked once by the constructor stage) and serves it when the key matches. See [applications.md](../../docs/contributing/artefact/files/plugins/lookup/applications.md) for the contract.
## Test-Only Helpers 🧪
Each module owns a private `_reset()` that clears its own cache dicts. The package-level `utils.cache._reset_cache_for_tests()` orchestrates all six (`base`, `applications`, `users`, `domains`, `yaml`, `files`) plus the shared fingerprint memo. Unit tests that exercise the cached paths MUST call it in `setUp` to guarantee clean state across test cases.
+7 -5
View File
@@ -25,8 +25,8 @@ from .base import (
_render_with_templar,
_resolve_override_mapping,
_resolve_roles_dir,
_stable_variables_signature,
)
from .carrier import _carried_applications, merged_applications_cache_key
from .schema_credentials import extract_default_credentials
from .yaml import load_yaml as _load_yaml_cached
from .yaml import load_yaml_any as _load_yaml_any_cached
@@ -396,14 +396,16 @@ def get_merged_applications(
variables = variables or {}
resolved_roles_dir = _resolve_roles_dir(roles_dir=roles_dir)
cache_key = (
_cache_key(resolved_roles_dir),
_stable_variables_signature(variables),
)
cache_key = merged_applications_cache_key(variables, roles_dir=resolved_roles_dir)
cached = _MERGED_APPLICATIONS_CACHE.get(cache_key)
if cached is not None:
return cached
carried = _carried_applications(variables, cache_key)
if carried is not None:
_MERGED_APPLICATIONS_CACHE[cache_key] = carried
return carried
defaults = get_application_defaults(roles_dir=resolved_roles_dir)
overrides = _resolve_override_mapping(variables, "applications", templar=templar)
+73
View File
@@ -0,0 +1,73 @@
"""Play-scoped carrier of the rendered applications payload.
Ansible forks a fresh worker process per task, so the in-process
`_MERGED_APPLICATIONS_CACHE` starts empty in every worker. The constructor
stage renders once and parks the payload plus its cache key in the host fact
named by `APPLICATIONS_RENDERED_FACT`; every later worker inherits that fact
at fork time and `get_merged_applications` serves it instead of rendering.
Ansible-free at import time like the rest of the package.
"""
from __future__ import annotations
from collections.abc import Mapping
from typing import TYPE_CHECKING, Any
from .base import _cache_key, _resolve_roles_dir, _stable_variables_signature
if TYPE_CHECKING:
import os
APPLICATIONS_RENDERED_FACT = "_INFINITO_APPLICATIONS_RENDERED"
def merged_applications_cache_key(
variables: Mapping[str, Any] | None,
*,
roles_dir: str | os.PathLike[str] | None = None,
) -> tuple[str, tuple]:
"""Key under which one rendered applications payload stays valid.
Args:
variables: The play's variable scope. Only the subset that
`_stable_variables_signature` reads influences the key.
roles_dir: Roles tree the payload is built from; defaults to the
repository's `roles/`.
Returns:
``(resolved roles dir, stable variables signature)``.
"""
return (
_cache_key(_resolve_roles_dir(roles_dir=roles_dir)),
_stable_variables_signature(variables),
)
def _carried_applications(
variables: Mapping[str, Any], cache_key: tuple[str, tuple]
) -> dict[str, Any] | None:
"""Return the payload carried in `APPLICATIONS_RENDERED_FACT` when its
key equals *cache_key*, else ``None``.
Args:
variables: The play's variable scope, read for the carrier fact.
cache_key: The key the caller would render under.
"""
carrier = variables.get(APPLICATIONS_RENDERED_FACT)
if not isinstance(carrier, Mapping):
return None
key = carrier.get("key")
payload = carrier.get("applications")
if (
not isinstance(payload, Mapping)
or not isinstance(key, (list, tuple))
or len(key) != 2
or not isinstance(key[1], (list, tuple))
):
return None
if [str(key[0]), [str(part) for part in key[1]]] != [
cache_key[0],
list(cache_key[1]),
]:
return None
return payload