Files
computer-playbook/utils/install/collections.py
kevinveenbirkenbachandClaude Opus 5 86f7d8c4f5 fix(install): keep the collection check inside the slice that runs it
The image runs this module from a layer that holds three files of utils:
__init__.py, install/__init__.py and install/collections.py, the slice
ef45d69dc3 cut so the source tree cannot invalidate the collection install.
67f5398985 established that the module therefore stays on the standard
library, and d4ae5cf9f0 broke it 15 minutes later by routing the reads
through utils.cache.files to satisfy the cache-read lint. The import raises
ModuleNotFoundError: No module named 'utils.cache' in that layer, which no
import inside a full checkout can show, because this repository installs
utils into site-packages.

Nothing turned red. The caller reads only the exit code, and the module's
contract is exit 1 with the missing collections on stdout, so a crash - exit
1, nothing on stdout - is indistinguishable from work to do. Every image
build since has reinstalled every collection with --force-with-deps, which
is the cost the check exists to avoid. The deploy job the traceback was
found in finished green.

The reads go back to Path.read_text with a cache-read marker each. The cache
was never earning its place here: read_text is an lru_cache keyed on the
path, and one run reads the requirements file once and one MANIFEST.json per
collection, eight distinct paths, none of them twice. Copying utils/cache
into the slice instead would work - the closure is three stdlib-only files -
but it binds a package with 51 commits in the last twelve months to the
layer that installs ansible, in front of a cache with no possible hit.

The caller now exits 3 when a non-zero status arrives with an empty stdout,
a state the contract cannot produce, so the next crash stops the install
instead of quietly reinstalling everything. all.sh runs under set -e and
touches the stamp only after ansible.sh returns, so the failure also leaves
the stamp stale rather than recording an install that did not happen.

The probe that would have caught this builds the slice in a tempdir from the
Dockerfile's own COPY lines and runs the module inside it. It has to pass
python -S: without it the interpreter resolves every submodule from the
installed checkout however the slice is built, and the test passes on a
module that cannot import in the image. Verified by injecting the import
back into the tempdir copy, which reproduces the traceback verbatim.

Delta measured by loading the committed module and this one side by side:
identical on both requirements files, against the real collections directory
and an empty one, on a missing file, on non-UTF-8 input and on a corrupt
MANIFEST.json. The one behavioural difference is that a file changed within
a process is now read fresh rather than from the memo.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 03:03:37 +02:00

110 lines
3.8 KiB
Python

"""Compare the pinned Galaxy requirements against what is on disk.
Imported by the lint bootstrap, which runs on a bare interpreter before any
dependency is installed, so this module stays on the standard library. It also
stays inside its own package: the Dockerfile copies it into the image beside
``utils/__init__.py`` and ``utils/install/__init__.py`` and nothing else, so
importing any other ``utils`` submodule raises ``ModuleNotFoundError`` in the
layer that runs it. The
requirements files it reads are this repository's own and hold one shape:
a ``collections:`` list of ``- name:`` entries with an optional ``version:``.
A line that does not match that shape leaves the entry unpinned, which the
caller treats as unsatisfied and installs.
"""
from __future__ import annotations
import json
import re
import sys
from pathlib import Path
_NAME = re.compile(r"^\s*-\s+name:\s*(?P<name>[A-Za-z0-9_.]+)\s*(?:#.*)?$")
_VERSION = re.compile(
r"^\s*version:\s*(?P<quote>[\"']?)(?P<version>[^\s#\"']+)(?P=quote)\s*(?:#.*)?$"
)
def declared_pins(requirements_file: Path) -> list[tuple[str, str | None]]:
"""The ``(fqcn, version)`` pairs a requirements file declares, in order.
Args:
requirements_file: the requirements.galaxy.yml to read.
"""
pins: list[tuple[str, str | None]] = []
try:
raw = requirements_file.read_text(encoding="utf-8") # nocheck: cache-read
except OSError:
return pins
lines = raw.splitlines()
for line in lines:
name_match = _NAME.match(line)
if name_match:
pins.append((name_match.group("name"), None))
continue
version_match = _VERSION.match(line)
if version_match and pins:
fqcn, _ = pins[-1]
pins[-1] = (fqcn, version_match.group("version"))
return pins
def _installed_version(collections_dir: Path, namespace: str, name: str) -> str | None:
"""The version recorded in a collection's manifest, or None when absent.
Args:
collections_dir: the ``-p`` target ansible-galaxy installs into.
namespace: collection namespace, the part before the dot.
name: collection name, the part after the dot.
"""
manifest = (
collections_dir / "ansible_collections" / namespace / name / "MANIFEST.json"
)
try:
raw = manifest.read_text(encoding="utf-8") # nocheck: cache-read
payload = json.loads(raw)
except (OSError, ValueError):
return None
version = payload.get("collection_info", {}).get("version")
return str(version) if version else None
def unsatisfied(requirements_file: Path, collections_dir: Path) -> list[str]:
"""Requirements that COLLECTIONS_DIR does not already hold at the pinned version.
An entry without a pinned version can never be confirmed from disk and is
always reported, so the caller falls through to a real install.
Args:
requirements_file: the requirements.galaxy.yml to read.
collections_dir: the ``-p`` target ansible-galaxy installs into.
"""
missing: list[str] = []
for fqcn, pinned in declared_pins(requirements_file):
if "." not in fqcn or not pinned:
missing.append(fqcn)
continue
namespace, _, name = fqcn.partition(".")
if _installed_version(collections_dir, namespace, name) != pinned:
missing.append(f"{fqcn}:{pinned}")
return missing
def main() -> int:
"""Exit 0 when every pinned collection is present at its pinned version.
Args:
argv[1]: path to requirements.galaxy.yml.
argv[2]: path to the collections directory.
"""
missing = unsatisfied(Path(sys.argv[1]), Path(sys.argv[2]))
if missing:
print(" ".join(missing))
return 1
return 0
if __name__ == "__main__":
sys.exit(main())