diff --git a/docs/contributing/artefact/files/plugins/lookup/applications.md b/docs/contributing/artefact/files/plugins/lookup/applications.md index ac13d7e6e1..93b468d4dc 100644 --- a/docs/contributing/artefact/files/plugins/lookup/applications.md +++ b/docs/contributing/artefact/files/plugins/lookup/applications.md @@ -40,7 +40,8 @@ The constructor stage renders once through the [applications_carrier](../../../. - The action renders with `application_id` removed from its variables, so the carried tree holds the play-scope result of every consumer-aligned `lookup('tls', ...)`, whichever role parks it. - The token store enters the key as the SHA-256 of exactly the tokens the users are hydrated from: the controller's tokens file until `sys-token-store` loads, the `sys_token_store_users` fact after. Both sources key alike when they hold the same tokens, so loading an unchanged store keeps the carrier, and a touch keeps the key. The carrier action opens the file before it computes the key, so an NFS client keys on revalidated attributes. - The action masks its own result: the fact is the whole rendered tree with every application's credentials, so it never reaches a log regardless of `MASK_CREDENTIALS_IN_LOGS`. -- User records inside the payload are the token store's state at render time. Readers of tokens captured later in the play MUST go through `lookup('users', ...)` or `lookup('config', , 'users....')`, both of which read the users cache fresh. +- User records inside the payload are the token store's state at render time. Readers of tokens captured later in the play MUST go through `lookup('users', ...)` or `lookup('config', , 'users....')`, both of which key the users on the current token store. +- The action parks the users carrier `_INFINITO_USERS_RENDERED` at the same points, after the applications render; `clear: true` drops both. See [users.md](users.md). - The carrier is private to the lookup. Consumers MUST NOT read `_INFINITO_APPLICATIONS_RENDERED`; they keep going through `lookup('applications')`, which is what makes the carrier transparent. - The carrier is the only materialised merged view a play MAY hold. The rule against a top-level `applications` fact above is unchanged. diff --git a/docs/contributing/artefact/files/plugins/lookup/users.md b/docs/contributing/artefact/files/plugins/lookup/users.md index c3ab7ffcb9..22fdab26b2 100644 --- a/docs/contributing/artefact/files/plugins/lookup/users.md +++ b/docs/contributing/artefact/files/plugins/lookup/users.md @@ -18,6 +18,15 @@ The `users` lookup is the ONLY supported runtime entry point for merged user con | `lookup('users')` | Full merged mapping keyed by the canonical user key. | | `lookup('users', '')` | Merged entry for that user. Raises when missing. | | `lookup('users', '', default_value)` | Merged entry, or `default_value` when the entry is missing. | +| `lookup('users', carrier=True)` | `{"key": [...], "users": {...}}`: the full mapping plus the cache key it is valid under. Reserved for the `applications_carrier` action (see below). | + +## Play-Scoped Carrier 🧊 + +The [applications_carrier](../../../../../../plugins/action/applications_carrier.py) action parks the rendered users next to the applications in the host fact `_INFINITO_USERS_RENDERED`, at every point listed in [applications.md](applications.md), and `clear: true` drops both facts. `get_merged_users` serves the fact to a templated read when its key matches and renders otherwise. + +- The key is the applications carrier key plus `inventory_hostname`: a changed `applications`, `users`, `DOMAIN_PRIMARY`, `SYSTEM_EMAIL_DOMAIN`, `group_names`, token store or host rejects the carrier. +- A read without a templar (the raw users inside an applications render) never takes the carrier. +- The carrier is private to the lookup. Consumers MUST NOT read `_INFINITO_USERS_RENDERED`; they keep going through `lookup('users')` or `lookup('config', , 'users....')`. ## Data Sources 📚 @@ -26,7 +35,7 @@ The lookup merges exactly two sources: 1. **Defaults** discovered from every `roles/*/meta/users.yml` in the repository (file root IS the users map; no `users:` wrapper). 2. **Overrides** supplied through the normal Ansible variable `users` in inventory, group vars, host vars, or role vars. -No intermediate merged `users` fact exists. No other source is consulted. +No intermediate merged `users` fact exists besides the carrier above. No other source is consulted. ## Canonical User Key 🔑 @@ -82,6 +91,7 @@ users: |---|---| | [users.py](../../../../../../plugins/lookup/users.py) | Runtime entry point for the `users` lookup. | | [users/ (cache)](../../../../../../utils/cache/users/) | Shared aggregation helper package that builds and caches defaults, handles reserved users, UID/GID allocation, and uniqueness validation. Placeholder substitution (DOMAIN_PRIMARY / ORGANIZATION / SOFTWARE_NAME) lives in [users/placeholders.py](../../../../../../utils/cache/users/placeholders.py). | +| [carrier.py](../../../../../../utils/cache/carrier.py) | The play-scoped carrier: fact name, `merged_users_cache_key`, and the reader that serves the carried payload. | | [test_users.py](../../../../../../tests/unit/python/plugins/lookup/test_users.py) | Unit tests covering the full-dict, single-entry, override, strict missing, and non-strict missing cases. | For the related applications pattern see [applications.md](applications.md). diff --git a/plugins/action/applications_carrier.py b/plugins/action/applications_carrier.py index d7da126d30..5ad3fbf84c 100644 --- a/plugins/action/applications_carrier.py +++ b/plugins/action/applications_carrier.py @@ -1,11 +1,13 @@ -"""Render the applications SPOT once and park it as the play-scoped carrier fact. +"""Render the applications and users SPOTs once and park them as play-scoped carrier facts. -Ansible forks a fresh worker per task, so the in-process render cache of -``lookup('applications')`` is cold in every task until something renders in -a place all later workers inherit. This action renders in the worker of one -constructor task and returns the payload as a host fact; every later worker -inherits that fact at fork time and ``get_merged_applications`` serves it -instead of rendering:: +Ansible forks a fresh worker per task, so the in-process render caches of +``lookup('applications')`` and ``lookup('users')`` are cold in every task +until something renders in a place all later workers inherit. This action +renders in the worker of one constructor task and returns both payloads as +host facts; every later worker inherits them at fork time and +``get_merged_applications`` / ``get_merged_users`` serve them instead of +rendering. The users carrier is keyed like the applications carrier plus +``inventory_hostname``:: - name: 🧊 Render the applications SPOT once and carry it into every task's worker applications_carrier: {} @@ -26,14 +28,16 @@ rejected and the reader renders afresh:: ``tasks/utils/setup/role.yml`` parks one after every role, ``tasks/utils/load/app.yml`` one after every nested application load and ``roles/sys-token-store/tasks/write_many.yml`` one after every store write; -where the key still matches they serve the carried payload without rendering. +where the key still matches they serve the carried payload without rendering, +and a users carrier that still matches is left in place instead of being +returned again. -The result is masked by the action itself: the fact is the whole rendered -tree, about 0.4 MB with every application's credentials, and printing it at -``-v`` is neither credential debugging nor readable. +The result is masked by the action itself: the facts are the whole rendered +trees, about 0.4 MB with every application's and user's credentials, and +printing them at ``-v`` is neither credential debugging nor readable. Args: - clear: drop the carrier fact instead of rendering. Defaults to false. + clear: drop both carrier facts instead of rendering. Defaults to false. Any other argument is an error. """ @@ -46,7 +50,12 @@ from ansible.module_utils.parsing.convert_bool import boolean from ansible.plugins.action import ActionBase from ansible.plugins.loader import lookup_loader -from utils.cache.carrier import APPLICATIONS_RENDERED_FACT, revalidate_tokens_file +from utils.cache.carrier import ( + APPLICATIONS_RENDERED_FACT, + USERS_RENDERED_FACT, + _carried_users, + revalidate_tokens_file, +) class ActionModule(ActionBase): @@ -67,7 +76,11 @@ class ActionModule(ActionBase): if boolean(args.get("clear", False)): result.update( - changed=False, ansible_facts={APPLICATIONS_RENDERED_FACT: None} + changed=False, + ansible_facts={ + APPLICATIONS_RENDERED_FACT: None, + USERS_RENDERED_FACT: None, + }, ) return result @@ -77,16 +90,25 @@ class ActionModule(ActionBase): previous = self._templar.available_variables self._templar.available_variables = render_vars try: - lookup = lookup_loader.get( - "applications", loader=self._loader, templar=self._templar - ) - carrier = lookup.run([], variables=render_vars, carrier=True)[0] + carriers = { + fact: lookup_loader.get( + name, loader=self._loader, templar=self._templar + ).run([], variables=render_vars, carrier=True)[0] + for fact, name in ( + (APPLICATIONS_RENDERED_FACT, "applications"), + (USERS_RENDERED_FACT, "users"), + ) + } finally: self._templar.available_variables = previous + users_key = carriers[USERS_RENDERED_FACT]["key"] + if _carried_users(task_vars, (users_key[0], tuple(users_key[1]))) is not None: + del carriers[USERS_RENDERED_FACT] + result.update( changed=False, - ansible_facts={APPLICATIONS_RENDERED_FACT: carrier}, + ansible_facts=carriers, _ansible_no_log=True, ) return result diff --git a/plugins/lookup/users.py b/plugins/lookup/users.py index 099110bcba..cedcf6516a 100644 --- a/plugins/lookup/users.py +++ b/plugins/lookup/users.py @@ -6,6 +6,7 @@ from ansible.errors import AnsibleError from ansible.plugins.lookup import LookupBase from utils.cache import _reset_cache_for_tests as _reset_runtime_lookup_cache +from utils.cache.carrier import merged_users_cache_key # nocheck: lookup-cache-import (this lookup IS the users SPOT provider) from utils.cache.users import get_merged_users @@ -22,21 +23,39 @@ class LookupModule(LookupBase): variables: dict[str, Any] | None = None, **kwargs: Any, ) -> list[Any]: + """ + Terms: + 0) full merged mapping + 1) canonical_user_key + 2) canonical_user_key, default value + + Keyword arguments: + roles_dir: roles tree to build from (defaults to the repository's) + carrier: with no terms, return ``{"key": [...], "users": {...}}`` + for the constructor to park in `_INFINITO_USERS_RENDERED` + """ terms = terms or [] if len(terms) > 2: raise AnsibleError( "users: expected 0, 1, or 2 terms: " "lookup('users'[, canonical_user_key[, default]])" ) + carrier = bool(kwargs.get("carrier", False)) + if carrier and terms: + raise AnsibleError("users: carrier=True takes no terms") + variables = variables or getattr(self._templar, "available_variables", {}) or {} + roles_dir = kwargs.get("roles_dir") users = get_merged_users( - variables=variables - or getattr(self._templar, "available_variables", {}) - or {}, - roles_dir=kwargs.get("roles_dir"), + variables=variables, + roles_dir=roles_dir, templar=getattr(self, "_templar", None), ) + if carrier: + key = merged_users_cache_key(variables, roles_dir=roles_dir) + return [{"key": [key[0], list(key[1])], "users": users}] + if len(terms) == 0: return [users] diff --git a/tests/unit/python/plugins/action/test_applications_carrier.py b/tests/unit/python/plugins/action/test_applications_carrier.py index 4e0c8d213e..95f8cc355b 100644 --- a/tests/unit/python/plugins/action/test_applications_carrier.py +++ b/tests/unit/python/plugins/action/test_applications_carrier.py @@ -4,7 +4,7 @@ from unittest.mock import patch from ansible.errors import AnsibleActionFail from plugins.action.applications_carrier import ActionModule -from utils.cache.carrier import APPLICATIONS_RENDERED_FACT +from utils.cache.carrier import APPLICATIONS_RENDERED_FACT, USERS_RENDERED_FACT class _FakeTask: @@ -31,6 +31,16 @@ class _FakeLookup: return [self.carrier] +class _FakeLoader: + def __init__(self, applications, users): + self.lookups = {"applications": applications, "users": users} + self.order = [] + + def get(self, name, loader=None, templar=None): + self.order.append(name) + return self.lookups[name] + + def _make_action(task: _FakeTask, templar=None) -> ActionModule: action = object.__new__(ActionModule) action._task = task @@ -39,10 +49,19 @@ def _make_action(task: _FakeTask, templar=None) -> ActionModule: return action +def _lookups(templar=None): + applications = _FakeLookup( + {"key": ["roles", ["a", "b", "c", "d"]], "applications": {"x": {}}}, templar + ) + users = _FakeLookup( + {"key": ["roles", ["a", "b", "c", "d", "h"]], "users": {"u": {}}}, templar + ) + return applications, users, _FakeLoader(applications, users) + + class TestApplicationsCarrierActionPlugin(unittest.TestCase): - def test_parks_the_carrier_as_a_masked_fact(self): - carrier = {"key": ["roles", ["a", "b", "c", "d"]], "applications": {"x": {}}} - lookup = _FakeLookup(carrier) + def test_parks_both_carriers_as_masked_facts(self): + applications, users, loader = _lookups() action = _make_action(_FakeTask()) with ( patch( @@ -52,23 +71,78 @@ class TestApplicationsCarrierActionPlugin(unittest.TestCase): ), patch( "plugins.action.applications_carrier.lookup_loader.get", - return_value=lookup, + side_effect=loader.get, ), ): result = action.run(task_vars={"applications": {}}) - self.assertIs(result["ansible_facts"][APPLICATIONS_RENDERED_FACT], carrier) + self.assertIs( + result["ansible_facts"][APPLICATIONS_RENDERED_FACT], applications.carrier + ) + self.assertIs(result["ansible_facts"][USERS_RENDERED_FACT], users.carrier) self.assertTrue(result["_ansible_no_log"]) self.assertFalse(result["changed"]) - terms, variables, kwargs = lookup.calls[0] - self.assertEqual(terms, []) - self.assertEqual(variables, {"applications": {}}) - self.assertTrue(kwargs["carrier"]) + for lookup in (applications, users): + terms, variables, kwargs = lookup.calls[0] + self.assertEqual(terms, []) + self.assertEqual(variables, {"applications": {}}) + self.assertTrue(kwargs["carrier"]) + + def _park(self, task_vars): + applications, users, loader = _lookups() + action = _make_action(_FakeTask()) + with ( + patch( + "plugins.action.applications_carrier.ActionBase.run", + autospec=True, + return_value={}, + ), + patch( + "plugins.action.applications_carrier.lookup_loader.get", + side_effect=loader.get, + ), + patch("plugins.action.applications_carrier.revalidate_tokens_file"), + ): + return action.run(task_vars=task_vars), applications, users + + def test_a_users_carrier_that_still_matches_stays_in_place(self): + parked = {"key": ["roles", ["a", "b", "c", "d", "h"]], "users": {"u": {}}} + result, applications, _ = self._park({USERS_RENDERED_FACT: parked}) + + self.assertNotIn(USERS_RENDERED_FACT, result["ansible_facts"]) + self.assertIs( + result["ansible_facts"][APPLICATIONS_RENDERED_FACT], applications.carrier + ) + + def test_a_stale_users_carrier_is_replaced(self): + parked = {"key": ["roles", ["a", "b", "c", "d", "h2"]], "users": {"u": {}}} + result, _, users = self._park({USERS_RENDERED_FACT: parked}) + + self.assertIs(result["ansible_facts"][USERS_RENDERED_FACT], users.carrier) + + def test_renders_the_applications_before_the_users(self): + _, _, loader = _lookups() + action = _make_action(_FakeTask()) + with ( + patch( + "plugins.action.applications_carrier.ActionBase.run", + autospec=True, + return_value={}, + ), + patch( + "plugins.action.applications_carrier.lookup_loader.get", + side_effect=loader.get, + ), + patch("plugins.action.applications_carrier.revalidate_tokens_file"), + ): + action.run(task_vars={}) + + self.assertEqual(loader.order, ["applications", "users"]) def test_renders_without_the_callers_application_id(self): task_vars = {"applications": {}, "application_id": "web-app-bluesky"} templar = _FakeTemplar(task_vars) - lookup = _FakeLookup({"key": ["roles", []], "applications": {}}, templar) + applications, users, loader = _lookups(templar) action = _make_action(_FakeTask(), templar) with ( patch( @@ -78,22 +152,26 @@ class TestApplicationsCarrierActionPlugin(unittest.TestCase): ), patch( "plugins.action.applications_carrier.lookup_loader.get", - return_value=lookup, + side_effect=loader.get, ), patch("plugins.action.applications_carrier.revalidate_tokens_file"), ): action.run(task_vars=task_vars) - _, variables, _ = lookup.calls[0] - self.assertEqual(variables, {"applications": {}}) - self.assertEqual(lookup.templar_variables, [{"applications": {}}]) + for lookup in (applications, users): + _, variables, _ = lookup.calls[0] + self.assertEqual(variables, {"applications": {}}) + self.assertEqual(lookup.templar_variables, [{"applications": {}}]) self.assertIs(templar.available_variables, task_vars) self.assertEqual(task_vars["application_id"], "web-app-bluesky") def test_revalidates_the_tokens_file_before_rendering(self): order = [] - lookup = _FakeLookup({"key": ["roles", []], "applications": {}}) - lookup.run = lambda *args, **kwargs: order.append("render") or [{}] + applications, users, loader = _lookups() + applications.run = lambda *args, **kwargs: ( + order.append("applications") or [applications.carrier] + ) + users.run = lambda *args, **kwargs: order.append("users") or [users.carrier] action = _make_action(_FakeTask()) with ( patch( @@ -103,7 +181,7 @@ class TestApplicationsCarrierActionPlugin(unittest.TestCase): ), patch( "plugins.action.applications_carrier.lookup_loader.get", - return_value=lookup, + side_effect=loader.get, ), patch( "plugins.action.applications_carrier.revalidate_tokens_file", @@ -113,7 +191,8 @@ class TestApplicationsCarrierActionPlugin(unittest.TestCase): action.run(task_vars={"FILE_TOKENS": "/x/tokens.yml"}) self.assertEqual( - order, [("revalidate", {"FILE_TOKENS": "/x/tokens.yml"}), "render"] + order, + [("revalidate", {"FILE_TOKENS": "/x/tokens.yml"}), "applications", "users"], ) def test_clear_does_not_revalidate(self): @@ -144,8 +223,8 @@ class TestApplicationsCarrierActionPlugin(unittest.TestCase): ): action.run(task_vars={}) - def test_clear_drops_the_fact_without_rendering(self): - lookup = _FakeLookup({"key": ["roles", []], "applications": {}}) + def test_clear_drops_both_facts_without_rendering(self): + applications, users, loader = _lookups() action = _make_action(_FakeTask(args={"clear": True})) with ( patch( @@ -155,18 +234,21 @@ class TestApplicationsCarrierActionPlugin(unittest.TestCase): ), patch( "plugins.action.applications_carrier.lookup_loader.get", - return_value=lookup, + side_effect=loader.get, ), ): result = action.run(task_vars={"applications": {}}) - self.assertIsNone(result["ansible_facts"][APPLICATIONS_RENDERED_FACT]) + self.assertEqual( + result["ansible_facts"], + {APPLICATIONS_RENDERED_FACT: None, USERS_RENDERED_FACT: None}, + ) self.assertFalse(result["changed"]) - self.assertEqual(lookup.calls, []) + self.assertEqual(applications.calls, []) + self.assertEqual(users.calls, []) def test_clear_false_still_renders(self): - carrier = {"key": ["roles", ["a", "b", "c", "d"]], "applications": {"x": {}}} - lookup = _FakeLookup(carrier) + applications, users, loader = _lookups() action = _make_action(_FakeTask(args={"clear": "no"})) with ( patch( @@ -176,13 +258,17 @@ class TestApplicationsCarrierActionPlugin(unittest.TestCase): ), patch( "plugins.action.applications_carrier.lookup_loader.get", - return_value=lookup, + side_effect=loader.get, ), ): result = action.run(task_vars={"applications": {}}) - self.assertIs(result["ansible_facts"][APPLICATIONS_RENDERED_FACT], carrier) - self.assertEqual(len(lookup.calls), 1) + self.assertIs( + result["ansible_facts"][APPLICATIONS_RENDERED_FACT], applications.carrier + ) + self.assertIs(result["ansible_facts"][USERS_RENDERED_FACT], users.carrier) + self.assertEqual(len(applications.calls), 1) + self.assertEqual(len(users.calls), 1) if __name__ == "__main__": diff --git a/tests/unit/python/plugins/lookup/test_users.py b/tests/unit/python/plugins/lookup/test_users.py index 19f83592c5..b4d5c293f8 100644 --- a/tests/unit/python/plugins/lookup/test_users.py +++ b/tests/unit/python/plugins/lookup/test_users.py @@ -10,6 +10,7 @@ from ansible.errors import AnsibleError from plugins.lookup.users import LookupModule, _reset_cache_for_tests from utils.cache import base as runtime_data_base +from utils.cache.carrier import USERS_RENDERED_FACT, merged_users_cache_key from utils.cache.yaml import dump_yaml_str from utils.paths import FILE_TOKENS from utils.roles.mapping import ROLE_FILE_META_USERS @@ -303,6 +304,46 @@ class TestUsersLookup(unittest.TestCase): self.assertEqual(result["tokens"]["web-app-mailu"], "token-from-default-path") + def test_carrier_returns_the_users_with_the_key_they_are_valid_under( + self, + ) -> None: + roles = str(self._tmp / "roles") + variables = {"inventory_hostname": "h1"} + result = self.lookup.run( + [], variables=variables, roles_dir=roles, carrier=True + )[0] + key = merged_users_cache_key(variables, roles_dir=roles) + self.assertEqual(result["key"], [key[0], list(key[1])]) + self.assertEqual( + result["users"], + self.lookup.run([], variables=variables, roles_dir=roles)[0], + ) + + def test_carrier_takes_no_terms(self) -> None: + with self.assertRaises(AnsibleError): + self.lookup.run( + ["alice"], + variables={}, + roles_dir=str(self._tmp / "roles"), + carrier=True, + ) + + def test_a_parked_carrier_answers_a_fresh_worker_without_rendering(self) -> None: + roles = str(self._tmp / "roles") + self.lookup._templar = _DummyTemplar({"SYSTEM_EMAIL_DOMAIN": "example.org"}) + variables = {"inventory_hostname": "h1", "SYSTEM_EMAIL_DOMAIN": "example.org"} + parked = self.lookup.run( + [], variables=variables, roles_dir=roles, carrier=True + )[0] + _reset_cache_for_tests() + variables[USERS_RENDERED_FACT] = parked + with patch( + "utils.cache.users._render_with_templar", + side_effect=AssertionError("rendered"), + ): + result = self.lookup.run(["alice"], variables=variables, roles_dir=roles)[0] + self.assertIs(result, parked["users"]["alice"]) + class _DummyTemplar: def __init__(self, variables: dict[str, str]) -> None: diff --git a/tests/unit/python/utils/cache/users/test_module.py b/tests/unit/python/utils/cache/users/test_module.py index 0dc951ed81..ee1ce655d2 100644 --- a/tests/unit/python/utils/cache/users/test_module.py +++ b/tests/unit/python/utils/cache/users/test_module.py @@ -14,12 +14,14 @@ from __future__ import annotations import tempfile import textwrap import unittest +import unittest.mock as mock from collections import OrderedDict from pathlib import Path from utils.cache import _reset_cache_for_tests from utils.cache import base as cache_base from utils.cache import users as cache_users +from utils.cache.carrier import USERS_RENDERED_FACT, merged_users_cache_key from utils.cache.users import store as cache_users_store from utils.cache.yaml import dump_yaml_str from utils.roles.mapping import ROLE_FILE_META_SERVICES, ROLE_FILE_META_USERS @@ -415,6 +417,151 @@ class TestGetMergedUsersTokenSource(unittest.TestCase): self.assertEqual(after["foo"]["tokens"]["web-app-foo"], "minted") +class _Templar: + def __init__(self) -> None: + self.available_variables: dict = {} + + def template(self, value, fail_on_undefined=False): + return value + + +class TestGetMergedUsersCarrierFact(unittest.TestCase): + def setUp(self) -> None: + _reset_cache_for_tests() + + @staticmethod + def _variables(host: str = "h1") -> dict: + return { + "inventory_hostname": host, + "users": {"foo": {"email": "foo@example.org"}}, + cache_users_store.STORE_FACT: { + "foo": {"tokens": {"web-app-foo": "minted"}} + }, + } + + @staticmethod + def _park(roles: Path, variables: dict, payload: dict) -> dict: + key = merged_users_cache_key(variables, roles_dir=roles) + variables[USERS_RENDERED_FACT] = { + "key": [key[0], list(key[1])], + "users": payload, + } + return payload + + @staticmethod + def _merged(roles: Path, variables: dict, templar=None) -> tuple: + with mock.patch.object( + cache_users, + "_render_with_templar", + side_effect=lambda value, **_: {"rendered": value}, + ) as render: + merged = cache_users.get_merged_users( + variables=variables, + roles_dir=roles, + templar=_Templar() if templar is None else templar, + ) + return merged, render.call_count + + def test_matching_key_serves_the_carried_payload_without_rendering(self): + with tempfile.TemporaryDirectory() as tmp: + roles = _seed_minimal_user_role(Path(tmp)) + variables = self._variables() + carried = self._park(roles, variables, {"foo": {"carried": True}}) + merged, renders = self._merged(roles, variables) + self.assertIs(merged, carried) + self.assertEqual(renders, 0) + + def test_the_carried_payload_answers_the_rest_of_the_process(self): + with tempfile.TemporaryDirectory() as tmp: + roles = _seed_minimal_user_role(Path(tmp)) + variables = self._variables() + carried = self._park(roles, variables, {"foo": {"carried": True}}) + self._merged(roles, variables) + del variables[USERS_RENDERED_FACT] + merged, renders = self._merged(roles, variables) + self.assertIs(merged, carried) + self.assertEqual(renders, 0) + + def test_stale_key_falls_back_to_a_fresh_render(self): + with tempfile.TemporaryDirectory() as tmp: + roles = _seed_minimal_user_role(Path(tmp)) + variables = self._variables() + key = merged_users_cache_key(variables, roles_dir=roles) + variables[USERS_RENDERED_FACT] = { + "key": [key[0], ["stale"] * len(key[1])], + "users": {"foo": {"carried": True}}, + } + merged, renders = self._merged(roles, variables) + self.assertEqual(renders, 1) + self.assertEqual(merged["rendered"]["foo"]["email"], "foo@example.org") + + def test_another_hosts_carrier_is_rejected(self): + with tempfile.TemporaryDirectory() as tmp: + roles = _seed_minimal_user_role(Path(tmp)) + variables = self._variables("h1") + self._park(roles, variables, {"foo": {"carried": True}}) + variables["inventory_hostname"] = "h2" + merged, renders = self._merged(roles, variables) + self.assertEqual(renders, 1) + self.assertNotIn("carried", merged["rendered"]["foo"]) + + def test_a_rotated_store_token_rejects_the_carrier(self): + with tempfile.TemporaryDirectory() as tmp: + roles = _seed_minimal_user_role(Path(tmp)) + variables = self._variables() + self._park(roles, variables, {"foo": {"carried": True}}) + variables[cache_users_store.STORE_FACT] = { + "foo": {"tokens": {"web-app-foo": "rotated"}} + } + merged, renders = self._merged(roles, variables) + self.assertEqual(renders, 1) + self.assertEqual( + merged["rendered"]["foo"]["tokens"]["web-app-foo"], "rotated" + ) + + def test_a_rotated_inventory_password_rejects_the_carrier(self): + with tempfile.TemporaryDirectory() as tmp: + roles = _seed_minimal_user_role(Path(tmp)) + variables = self._variables() + variables["users"]["foo"]["password"] = "before" + self._park(roles, variables, {"foo": {"carried": True}}) + variables["users"] = {"foo": {"password": "after"}} + merged, renders = self._merged(roles, variables) + self.assertEqual(renders, 1) + self.assertEqual(merged["rendered"]["foo"]["password"], "after") + + def test_without_a_templar_the_carrier_is_ignored(self): + with tempfile.TemporaryDirectory() as tmp: + roles = _seed_minimal_user_role(Path(tmp)) + variables = self._variables() + self._park(roles, variables, {"foo": {"carried": True}}) + merged = cache_users.get_merged_users( + variables=variables, roles_dir=roles, templar=None + ) + self.assertNotIn("carried", merged["foo"]) + self.assertEqual(merged["foo"]["email"], "foo@example.org") + + def test_malformed_carrier_is_ignored(self): + with tempfile.TemporaryDirectory() as tmp: + roles = _seed_minimal_user_role(Path(tmp)) + key = merged_users_cache_key(self._variables(), roles_dir=roles) + for carrier in ( + "not-a-mapping", + {"key": "x", "users": {}}, + {"key": [key[0], list(key[1])], "users": "not-a-mapping"}, + {}, + ): + with self.subTest(carrier=carrier): + _reset_cache_for_tests() + variables = self._variables() + variables[USERS_RENDERED_FACT] = carrier + merged, renders = self._merged(roles, variables) + self.assertEqual(renders, 1) + self.assertEqual( + merged["rendered"]["foo"]["email"], "foo@example.org" + ) + + class TestStoreSignature(unittest.TestCase): def setUp(self) -> None: _reset_cache_for_tests() diff --git a/utils/cache/README.md b/utils/cache/README.md index 3be16f4ee2..2ecbb1ccb5 100644 --- a/utils/cache/README.md +++ b/utils/cache/README.md @@ -10,7 +10,7 @@ This package contains the in-process cache layers Infinito.Nexus uses to keep CL | [files.py](files.py) | The project tree's full path list (one walk per process) and per-file UTF-8 contents (`read_text`). Used by lint/integration tests and CLI tools that scan the repo. | | [base.py](base.py) | Cross-cutting helpers: filesystem constants (`PROJECT_ROOT`, `ROLES_DIR`, `DEFAULT_TOKENS_FILE`), `_deep_merge`, cache-key + content-fingerprint signatures, the cross-domain re-entry guard, and the templar-render machinery. The only ansible-coupled symbol is `_render_with_templar`, which lazy-imports its dependency. | | [applications.py](applications.py) | Per-app variants + defaults + `get_merged_applications`. **Strictly ansible-free at import time** so the GitHub Actions runner-host CLI path (`cli.administration.deploy.development.init` → `plan_dev_inventory_matrix` → `get_variants`) keeps working without ansible installed. | -| [carrier.py](carrier.py) | The play-scoped carrier of the rendered applications payload: the fact name `_INFINITO_APPLICATIONS_RENDERED`, `merged_applications_cache_key`, and the reader `get_merged_applications` consults before it renders. | +| [carrier.py](carrier.py) | The play-scoped carriers of the rendered applications and users payloads: the fact names `_INFINITO_APPLICATIONS_RENDERED` and `_INFINITO_USERS_RENDERED`, `merged_applications_cache_key`, `merged_users_cache_key`, and the readers `get_merged_applications` and `get_merged_users` consult before they render. | | [users/](users/) | User definitions, token store hydration, alias materialization, `get_user_defaults`, `get_merged_users` (package; `users/placeholders.py` substitutes `DOMAIN_PRIMARY` / `ORGANIZATION` / `SOFTWARE_NAME` ahead of the templar render pass; `users/store.py` reads the `sys_token_store_users` fact and digests the tokens the users are hydrated from, the digest the users and carrier keys share). | | [domains.py](domains.py) | Canonical-domains map derived from the merged applications view: `get_merged_domains`. A read issued while the applications render is in progress (the re-entry guard is set) sees the unrendered view, so it is cached under its own key and never answers a later read. | | [`__init__.py`](__init__.py) | Owns the package-level `_reset_cache_for_tests()` orchestrator that clears every cache plus the shared fingerprint memo in one call. | @@ -26,7 +26,7 @@ The layers are orthogonal: `yaml.py` caches "this file's parsed root", the `appl Both layers are process-wide in-process caches. They intentionally do NOT track on-disk file mtimes; CLI tools are short-lived and the assumption "the on-disk file does not change while my process runs" holds. The single exception is `utils.cache.yaml.dump_yaml`, which writes through and evicts the cached entry for the path it just wrote so a tool that edits a file mid-run sees the new content on the next read. -Inside a play the process boundary is the task: Ansible forks a fresh worker per task, so `_MERGED_APPLICATIONS_CACHE` starts empty in every worker. `get_merged_applications` therefore also accepts the play-scoped carrier fact `_INFINITO_APPLICATIONS_RENDERED` (payload plus cache key, parked by the constructor stage and again after every role, every nested application load and every persisted token) and serves it when the key matches. See [applications.md](../../docs/contributing/artefact/files/plugins/lookup/applications.md) for the contract. +Inside a play the process boundary is the task: Ansible forks a fresh worker per task, so `_MERGED_APPLICATIONS_CACHE` and `_MERGED_USERS_CACHE` start empty in every worker. `get_merged_applications` therefore also accepts the play-scoped carrier fact `_INFINITO_APPLICATIONS_RENDERED` (payload plus cache key, parked by the constructor stage and again after every role, every nested application load and every persisted token) and serves it when the key matches. The same action parks `_INFINITO_USERS_RENDERED`, which `get_merged_users` serves to templated reads under the applications key plus `inventory_hostname`. See [applications.md](../../docs/contributing/artefact/files/plugins/lookup/applications.md) and [users.md](../../docs/contributing/artefact/files/plugins/lookup/users.md) for the contract. While one render runs, the re-entry guard also holds that render's unrendered view and its cache key. A nested `get_merged_applications` call from a lookup inside a role template (`config`, `tls`, `domain`, ...) with the same key returns that view instead of copying and merging the role defaults again. The view is shared by every nested call of the render, so callers MUST NOT mutate it. The guard drops it when the render ends. @@ -84,7 +84,7 @@ flowchart TD proc -.->|"TRIP-WIRE: read BEFORE the
host's own fact, so an inherited
entry outranks it"| fact render -.->|"TRIP-WIRE: main process renders
task names for one host; forked
workers inherit that entry"| proc - key -.->|"TRIP-WIRE: carrier.py:68 stringifies
only one side, so a non-scalar key
part silently never matches again"| fact + key -.->|"TRIP-WIRE: carrier.py:118 stringifies
only one side, so a non-scalar key
part silently never matches again"| fact ``` The key covers `group_names`. `domains.py` derives from the applications view and shares that dependency; `users` reads `group_names` nowhere. diff --git a/utils/cache/carrier.py b/utils/cache/carrier.py index 989a0c2fc2..23f7972e70 100644 --- a/utils/cache/carrier.py +++ b/utils/cache/carrier.py @@ -1,11 +1,12 @@ -"""Play-scoped carrier of the rendered applications payload. +"""Play-scoped carriers of the rendered applications and users payloads. Ansible forks a fresh worker process per task, so the in-process -`_MERGED_APPLICATIONS_CACHE` starts empty in every worker. The constructor -stage renders once and parks the payload plus its cache key in the host fact -named by `APPLICATIONS_RENDERED_FACT`; every later worker inherits that fact -at fork time and `get_merged_applications` serves it instead of rendering. -Ansible-free at import time like the rest of the package. +`_MERGED_APPLICATIONS_CACHE` and `_MERGED_USERS_CACHE` start empty in every +worker. The constructor stage renders once and parks each payload plus its +cache key in the host facts named by `APPLICATIONS_RENDERED_FACT` and +`USERS_RENDERED_FACT`; every later worker inherits those facts at fork time +and `get_merged_applications` / `get_merged_users` serve them instead of +rendering. Ansible-free at import time like the rest of the package. """ from __future__ import annotations @@ -24,6 +25,7 @@ if TYPE_CHECKING: import os APPLICATIONS_RENDERED_FACT = "_INFINITO_APPLICATIONS_RENDERED" +USERS_RENDERED_FACT = "_INFINITO_USERS_RENDERED" def merged_applications_cache_key( @@ -53,6 +55,28 @@ def merged_applications_cache_key( ) +def merged_users_cache_key( + variables: Mapping[str, Any] | None, + *, + roles_dir: str | os.PathLike[str] | None = None, +) -> tuple[str, tuple]: + """Key under which one rendered users payload stays valid. + + Args: + variables: The play's variable scope. The applications key's inputs + plus ``inventory_hostname`` influence the key. + roles_dir: Roles tree the payload is built from; defaults to the + repository's `roles/`. + + Returns: + ``(resolved roles dir, applications key signature extended by the + host the payload was rendered for)``. + """ + roles_key, signature = merged_applications_cache_key(variables, roles_dir=roles_dir) + host = (variables or {}).get("inventory_hostname") + return (roles_key, (*signature, str(host or ""))) + + def revalidate_tokens_file(variables: Mapping[str, Any] | None) -> None: """Refresh an NFS client's attributes of the tokens file before a carrier is keyed. @@ -64,21 +88,26 @@ def revalidate_tokens_file(variables: Mapping[str, Any] | None) -> None: _revalidate_tokens_file(_resolve_tokens_file(variables)) -def _carried_applications( - variables: Mapping[str, Any], cache_key: tuple[str, tuple] +def _carried( + variables: Mapping[str, Any], + fact: str, + field: str, + cache_key: tuple[str, tuple], ) -> dict[str, Any] | None: - """Return the payload carried in `APPLICATIONS_RENDERED_FACT` when its - key equals *cache_key*, else ``None``. + """Return the payload carried in *fact* when its key equals *cache_key*, + else ``None``. Args: variables: The play's variable scope, read for the carrier fact. + fact: The carrier fact's name. + field: The carrier's payload field. cache_key: The key the caller would render under. """ - carrier = variables.get(APPLICATIONS_RENDERED_FACT) + carrier = variables.get(fact) if not isinstance(carrier, Mapping): return None key = carrier.get("key") - payload = carrier.get("applications") + payload = carrier.get(field) if ( not isinstance(payload, Mapping) or not isinstance(key, (list, tuple)) @@ -92,3 +121,29 @@ def _carried_applications( ]: return None return payload + + +def _carried_applications( + variables: Mapping[str, Any], cache_key: tuple[str, tuple] +) -> dict[str, Any] | None: + """Return the payload carried in `APPLICATIONS_RENDERED_FACT` when its + key equals *cache_key*, else ``None``. + + Args: + variables: The play's variable scope, read for the carrier fact. + cache_key: The key the caller would render under. + """ + return _carried(variables, APPLICATIONS_RENDERED_FACT, "applications", cache_key) + + +def _carried_users( + variables: Mapping[str, Any], cache_key: tuple[str, tuple] +) -> dict[str, Any] | None: + """Return the payload carried in `USERS_RENDERED_FACT` when its key + equals *cache_key*, else ``None``. + + Args: + variables: The play's variable scope, read for the carrier fact. + cache_key: The key the caller would render under. + """ + return _carried(variables, USERS_RENDERED_FACT, "users", cache_key) diff --git a/utils/cache/users/__init__.py b/utils/cache/users/__init__.py index a7a1a59527..79be183bc3 100644 --- a/utils/cache/users/__init__.py +++ b/utils/cache/users/__init__.py @@ -27,6 +27,7 @@ from utils.cache.base import ( _resolve_roles_dir, _stable_variables_signature, ) +from utils.cache.carrier import _carried_users, merged_users_cache_key from utils.cache.users import store as _store from utils.cache.users.placeholders import ( substitute_primary_domain_placeholder as _substitute_primary_domain_placeholder, @@ -442,6 +443,15 @@ def get_merged_users( if cached is not None: return cached + if templar is not None and source_variables: + carried = _carried_users( + source_variables, + merged_users_cache_key(source_variables, roles_dir=resolved_roles_dir), + ) + if carried is not None: + _MERGED_USERS_CACHE[cache_key] = carried + return carried + defaults = get_user_defaults(roles_dir=roles_dir) overrides = _resolve_override_mapping(variables, "users", templar=templar)