fix(cache): keep the applications carrier valid across token store writes

Since 778c18fb23 the carrier key holds the tokens file's mtime and size.
sys-token-store changed that stat on every inclusion (00_core.yml
touched the file) and on every persisted token, and nothing re-parked a
carrier afterwards. Every later task's fresh worker then rendered the
applications SPOT cold, 30 to 80 s each. On main fea6f31d9f (run
38812964308) seven heavy jobs logged 194 to 287 such tasks and ran out
of the 20400 s budget; the same jobs before 778c18fb23 logged 65 to 83,
all of them real waits.

- 00_core.yml touches the store with modification_time and access_time
  preserved, so it only creates a missing file. In swarm the controller
  sees the store over NFS with attribute caching, and the touch alone
  made the second pass go cold without any write.
- write.yml re-parks the carrier when the persist or the controller
  mirror changed the store. Both run once, so every host that ran the
  include gets the same verdict.
- tasks/utils/setup/role.yml re-parks after every role on every host,
  so hosts that did not run the writing role recover too. Where the
  key still matches this serves the carried payload; measured 0.1 s per
  role in a local compose deploy.
- The tokens file enters the key as the SHA-256 of its content,
  memoised per inode, mtime and size, so a touch no longer changes the
  key. A stat that still matches serves the memo (1.67 us against
  1.41 us for the old stat signature), and a stale stat keeps key and
  YAML cache on the same old content instead of pairing new content
  with an old key.
- applications_carrier opens the tokens file before it computes the
  key, so on the NFS-mounted controller the parked key reflects the
  revalidated attributes rather than a cached stat that changes a
  minute later.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs/contributing/artefact/files/plugins/lookup/applications.md

@@ -18,7 +18,7 @@ The `applications` lookup is the ONLY supported runtime entry point for merged a
 | `lookup('applications')` | Full merged mapping keyed by `application_id`. |
 | `lookup('applications', '<application_id>')` | Merged entry for that application. Raises when missing. |
 | `lookup('applications', '<application_id>', default_value)` | Merged entry, or `default_value` when the entry is missing. |
-| `lookup('applications', carrier=True)` | `{"key": [...], "applications": {...}}`: the full mapping plus the cache key it is valid under. Reserved for the constructor's carrier task (see below). |
+| `lookup('applications', carrier=True)` | `{"key": [...], "applications": {...}}`: the full mapping plus the cache key it is valid under. Reserved for the `applications_carrier` action (see below). |
 
 ## Data Sources 📚
 
@@ -33,9 +33,11 @@ No other source is consulted.
 
 Ansible forks a fresh worker process for every task. The in-process cache inside `get_merged_applications` therefore never survives from one task to the next, and every task that touches the lookup would render the full tree again (about 16 to 23 seconds per task on a CI runner, measured on the workspace jobs).
 
-The constructor stage renders once through the [applications_carrier](../../../../../../plugins/action/applications_carrier.py) action and parks the result together with its cache key in the host fact `_INFINITO_APPLICATIONS_RENDERED` (the action calls `lookup('applications', carrier=True)`). Every later worker inherits that fact at fork time; `get_merged_applications` serves it whenever the key still matches and renders only when it does not (a changed `applications`, `users`, `DOMAIN_PRIMARY` or `SYSTEM_EMAIL_DOMAIN`).
+The constructor stage renders once through the [applications_carrier](../../../../../../plugins/action/applications_carrier.py) action and parks the result together with its cache key in the host fact `_INFINITO_APPLICATIONS_RENDERED` (the action calls `lookup('applications', carrier=True)`). Every later worker inherits that fact at fork time; `get_merged_applications` serves it whenever the key still matches and renders only when it does not (a changed `applications`, `users`, `DOMAIN_PRIMARY`, `SYSTEM_EMAIL_DOMAIN` or `group_names`, or a changed path or content of the tokens file).
 
-- The task sits in `tasks/stages/01_constructor.yml` after the `networks`, `design` and `service_provider` merges, after the `add_host` placements and after the token store, so the payload renders with the same `group_names`, `groups` and user records a role would see. Moving it earlier bakes stale values into the play.
+- The stage-wide task sits in `tasks/stages/01_constructor.yml` after the `networks`, `design` and `service_provider` merges, after the `add_host` placements and after the token store, so the payload renders with the same `group_names`, `groups` and user records a role would see. Moving it earlier bakes stale values into the play.
+- The carrier is parked again after every role in `tasks/utils/setup/role.yml` and after every persisted token in `roles/sys-token-store/tasks/write.yml`. A re-park whose key still matches serves the carried payload without rendering.
+- The tokens file enters the key as the SHA-256 of its content, memoised per inode, mtime and size, so a touch keeps the key. The carrier action opens the file before it computes the key, so an NFS client keys on revalidated attributes.
 - The action masks its own result: the fact is the whole rendered tree with every application's credentials, so it never reaches a log regardless of `MASK_CREDENTIALS_IN_LOGS`.
 - User records inside the payload are the token store's state at render time. Readers of tokens captured later in the play MUST go through `lookup('users', ...)` or `lookup('config', <app>, 'users.<key>...')`, both of which read the users cache fresh.
 - The carrier is private to the lookup. Consumers MUST NOT read `_INFINITO_APPLICATIONS_RENDERED`; they keep going through `lookup('applications')`, which is what makes the carrier transparent.

plugins/action/applications_carrier.py

@@ -14,13 +14,17 @@ instead of rendering::
 for the merge block, which it drops again at the block's end, one after the
 ``add_host`` tasks for the swarm bootstrap roles, parked only when
 ``DEPLOYMENT_MODE`` is ``swarm``, and the stage-wide one after the token store. ``merged_applications_cache_key`` reads ``group_names`` and the
-tokens file's path, mtime and size, so a carrier whose groups or tokens no
+tokens file's path and content digest, so a carrier whose groups or tokens no
 longer match is rejected and the reader renders afresh::
 
     - name: 🧊 Drop the merge-block carrier so later stages re-render
       applications_carrier:
         clear: true
 
+``tasks/utils/setup/role.yml`` parks one after every role and
+``roles/sys-token-store/tasks/write.yml`` one after every persisted token;
+where the key still matches they serve the carried payload without rendering.
+
 The result is masked by the action itself: the fact is the whole rendered
 tree, about 0.4 MB with every application's credentials, and printing it at
 ``-v`` is neither credential debugging nor readable.
@@ -39,7 +43,7 @@ from ansible.module_utils.parsing.convert_bool import boolean
 from ansible.plugins.action import ActionBase
 from ansible.plugins.loader import lookup_loader
 
-from utils.cache.carrier import APPLICATIONS_RENDERED_FACT
+from utils.cache.carrier import APPLICATIONS_RENDERED_FACT, revalidate_tokens_file
 
 
 class ActionModule(ActionBase):
@@ -64,6 +68,7 @@ class ActionModule(ActionBase):
             )
             return result
 
+        revalidate_tokens_file(task_vars)
         lookup = lookup_loader.get(
             "applications", loader=self._loader, templar=self._templar
         )

roles/sys-token-store/tasks/00_core.yml

@@ -14,6 +14,8 @@
   ansible.builtin.file:
     path: "{{ FILE_TOKENS }}"
     state: touch
+    modification_time: preserve
+    access_time: preserve
     owner: "{{ SYS_TOKEN_STORE_OWNER }}"
     group: "{{ SYS_TOKEN_STORE_GROUP }}"
     mode: "{{ SYS_TOKEN_STORE_FILE_MODE }}"

roles/sys-token-store/tasks/write.yml

@@ -51,6 +51,7 @@
   run_once: true
   no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}"
   become: true
+  register: sys_token_store_persisted
 
 - name: "🪞 Mirror the store onto the controller, where every users lookup reads"
   ansible.builtin.copy:
@@ -63,3 +64,8 @@
   delegate_to: localhost
   run_once: true
   no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}"
+  register: sys_token_store_mirrored
+
+- name: 🧊 Re-render the applications SPOT the new token invalidated
+  applications_carrier: {}
+  when: sys_token_store_persisted is changed or sys_token_store_mirrored is changed

tasks/utils/setup/role.yml

@@ -6,3 +6,6 @@
 
 - name: "🚿 flush handlers after {{ role_entry.app }}"
   ansible.builtin.meta: flush_handlers
+
+- name: "🧊 Carry the applications SPOT past {{ role_entry.app }}"
+  applications_carrier: {}

tests/unit/python/plugins/action/test_applications_carrier.py

@@ -56,6 +56,48 @@ class TestApplicationsCarrierActionPlugin(unittest.TestCase):
         self.assertEqual(variables, {"applications": {}})
         self.assertTrue(kwargs["carrier"])
 
+    def test_revalidates_the_tokens_file_before_rendering(self):
+        order = []
+        lookup = _FakeLookup({"key": ["roles", []], "applications": {}})
+        lookup.run = lambda *args, **kwargs: order.append("render") or [{}]
+        action = _make_action(_FakeTask())
+        with (
+            patch(
+                "plugins.action.applications_carrier.ActionBase.run",
+                autospec=True,
+                return_value={},
+            ),
+            patch(
+                "plugins.action.applications_carrier.lookup_loader.get",
+                return_value=lookup,
+            ),
+            patch(
+                "plugins.action.applications_carrier.revalidate_tokens_file",
+                side_effect=lambda variables: order.append(("revalidate", variables)),
+            ),
+        ):
+            action.run(task_vars={"FILE_TOKENS": "/x/tokens.yml"})
+
+        self.assertEqual(
+            order, [("revalidate", {"FILE_TOKENS": "/x/tokens.yml"}), "render"]
+        )
+
+    def test_clear_does_not_revalidate(self):
+        action = _make_action(_FakeTask(args={"clear": True}))
+        with (
+            patch(
+                "plugins.action.applications_carrier.ActionBase.run",
+                autospec=True,
+                return_value={},
+            ),
+            patch(
+                "plugins.action.applications_carrier.revalidate_tokens_file"
+            ) as revalidate,
+        ):
+            action.run(task_vars={})
+
+        revalidate.assert_not_called()
+
     def test_rejects_arguments(self):
         action = _make_action(_FakeTask(args={"foo": 1}))
         with (

tests/unit/python/utils/cache/test_base.py

@@ -10,9 +10,11 @@ files; this file pins the contract of base itself.
 
 from __future__ import annotations
 
+import os
 import tempfile
 import textwrap
 import unittest
+import unittest.mock as mock
 from pathlib import Path
 
 from utils.cache import _reset_cache_for_tests, base
@@ -153,11 +155,13 @@ class TestStableVariablesSignature(unittest.TestCase):
 
 
 class TestTokensFileSignature(unittest.TestCase):
-    def test_missing_file_returns_zeroed_signature(self):
+    def setUp(self):
+        base._reset()
+
+    def test_missing_file_returns_an_empty_digest(self):
         with tempfile.TemporaryDirectory() as tmp:
-            sig = base._tokens_file_signature(Path(tmp) / "absent.yml")
-            self.assertEqual(sig[1], 0)
-            self.assertEqual(sig[2], 0)
+            path = Path(tmp) / "absent.yml"
+            self.assertEqual(base._tokens_file_signature(path), (str(path), ""))
 
     def test_signature_changes_when_file_changes(self):
         with tempfile.TemporaryDirectory() as tmp:
@@ -168,6 +172,37 @@ class TestTokensFileSignature(unittest.TestCase):
             sig_after = base._tokens_file_signature(path)
             self.assertNotEqual(sig_before, sig_after)
 
+    def test_a_touch_keeps_the_signature(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            path = Path(tmp) / "tokens.yml"
+            path.write_text("a: 1\n", encoding="utf-8")
+            sig_before = base._tokens_file_signature(path)
+            later = path.stat().st_mtime_ns + 5_000_000_000
+            os.utime(path, ns=(later, later))
+            self.assertEqual(base._tokens_file_signature(path), sig_before)
+
+    def test_an_unchanged_stat_serves_the_digest_without_reading(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            path = Path(tmp) / "tokens.yml"
+            path.write_text("a: 1\n", encoding="utf-8")
+            sig_before = base._tokens_file_signature(path)
+            with mock.patch.object(Path, "open", side_effect=AssertionError):
+                self.assertEqual(base._tokens_file_signature(path), sig_before)
+
+
+class TestRevalidateTokensFile(unittest.TestCase):
+    def test_opens_the_file(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            path = Path(tmp) / "tokens.yml"
+            path.write_text("a: 1\n", encoding="utf-8")
+            with mock.patch.object(Path, "open", wraps=path.open) as opened:
+                base._revalidate_tokens_file(path)
+            opened.assert_called_once_with("rb")
+
+    def test_a_missing_file_is_ignored(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            base._revalidate_tokens_file(Path(tmp) / "absent.yml")
+
 
 class TestRenderWithTemplar(unittest.TestCase):
     """`_render_with_templar` is the single ansible-coupled symbol in

utils/cache/README.md

@@ -26,7 +26,7 @@ The layers are orthogonal: `yaml.py` caches "this file's parsed root", the `appl
 
 Both layers are process-wide in-process caches. They intentionally do NOT track on-disk file mtimes; CLI tools are short-lived and the assumption "the on-disk file does not change while my process runs" holds. The single exception is `utils.cache.yaml.dump_yaml`, which writes through and evicts the cached entry for the path it just wrote so a tool that edits a file mid-run sees the new content on the next read.
 
-Inside a play the process boundary is the task: Ansible forks a fresh worker per task, so `_MERGED_APPLICATIONS_CACHE` starts empty in every worker. `get_merged_applications` therefore also accepts the play-scoped carrier fact `_INFINITO_APPLICATIONS_RENDERED` (payload plus cache key, parked once by the constructor stage) and serves it when the key matches. See [applications.md](../../docs/contributing/artefact/files/plugins/lookup/applications.md) for the contract.
+Inside a play the process boundary is the task: Ansible forks a fresh worker per task, so `_MERGED_APPLICATIONS_CACHE` starts empty in every worker. `get_merged_applications` therefore also accepts the play-scoped carrier fact `_INFINITO_APPLICATIONS_RENDERED` (payload plus cache key, parked by the constructor stage and again after every role and every persisted token) and serves it when the key matches. See [applications.md](../../docs/contributing/artefact/files/plugins/lookup/applications.md) for the contract.
 
 ## Test-Only Helpers 🧪
 

utils/cache/base.py

@@ -16,10 +16,12 @@ only when actually invoked.
 from __future__ import annotations
 
 import copy
+import hashlib
+import os
 import threading
 from collections.abc import Mapping
 from pathlib import Path
-from typing import TYPE_CHECKING, Any
+from typing import Any
 
 from plugins.filter.merge.with_defaults import (
     merge_with_defaults,  # noqa: F401  re-exported
@@ -28,8 +30,7 @@ from utils.paths import FILE_TOKENS
 
 from . import PROJECT_ROOT, ROLES_DIR  # noqa: F401
 
-if TYPE_CHECKING:
-    import os
+_TOKENS_DIGESTS: dict[tuple, str] = {}
 
 try:
     from ansible.parsing.vault import EncryptedString as _AnsibleEncryptedString
@@ -125,18 +126,46 @@ def _stable_variables_signature(variables: Mapping[str, Any] | None) -> tuple:
     )
 
 
-def _tokens_file_signature(path: Path) -> tuple:
-    """Return a cheap stat-based signature for the tokens file.
+def _stat_identity(st: os.stat_result) -> tuple:
+    return (st.st_ino, st.st_mtime_ns, st.st_size)
 
-    The merged-users cache must invalidate whenever sys-token-store persists a
-    new token — otherwise downstream `lookup('users', ...)` returns stale tokens
-    within the same play. stat() is cheap and captures in-place writes.
+
+def _tokens_file_signature(path: Path) -> tuple:
+    """Return ``(path, sha256 of the content)`` for the tokens file.
+
+    Args:
+        path: the tokens file.
+
+    Returns:
+        ``(str(path), "")`` when the file cannot be read.
     """
     try:
-        st = path.stat()
-    except (FileNotFoundError, OSError):
-        return (str(path), 0, 0)
-    return (str(path), st.st_mtime_ns, st.st_size)
+        identity = (str(path), *_stat_identity(path.stat()))
+    except OSError:
+        return (str(path), "")
+    digest = _TOKENS_DIGESTS.get(identity)
+    if digest is None:
+        try:
+            with path.open("rb") as handle:
+                identity = (str(path), *_stat_identity(os.fstat(handle.fileno())))
+                digest = hashlib.sha256(handle.read()).hexdigest()
+        except OSError:
+            return (str(path), "")
+        _TOKENS_DIGESTS[identity] = digest
+    return (str(path), digest)
+
+
+def _revalidate_tokens_file(path: Path) -> None:
+    """Open and close the tokens file so an NFS client refreshes its attributes.
+
+    Args:
+        path: the tokens file.
+    """
+    try:
+        with path.open("rb"):
+            pass
+    except OSError:
+        return
 
 
 def _deep_merge(base: Any, override: Any) -> Any:
@@ -287,5 +316,5 @@ def _render_with_templar(
 
 
 def _reset() -> None:
-    """No-op kept so the `data._reset_cache_for_tests` facade can call every
-    domain module uniformly. This module owns no cache."""
+    """Clear the tokens-file digest memo."""
+    _TOKENS_DIGESTS.clear()

utils/cache/carrier.py

@@ -16,6 +16,7 @@ from typing import TYPE_CHECKING, Any
 from .base import (
     _cache_key,
     _resolve_roles_dir,
+    _revalidate_tokens_file,
     _stable_variables_signature,
     _tokens_file_signature,
 )
@@ -42,7 +43,7 @@ def merged_applications_cache_key(
 
     Returns:
         ``(resolved roles dir, stable variables signature extended by the
-        tokens file's path, mtime and size)``.
+        tokens file's path and content digest)``.
     """
     from .users import _resolve_tokens_file
 
@@ -53,6 +54,17 @@ def merged_applications_cache_key(
     )
 
 
+def revalidate_tokens_file(variables: Mapping[str, Any] | None) -> None:
+    """Refresh an NFS client's attributes of the tokens file before a carrier is keyed.
+
+    Args:
+        variables: The play's variable scope the tokens file resolves from.
+    """
+    from .users import _resolve_tokens_file
+
+    _revalidate_tokens_file(_resolve_tokens_file(variables))
+
+
 def _carried_applications(
     variables: Mapping[str, Any], cache_key: tuple[str, tuple]
 ) -> dict[str, Any] | None: