mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 02:54:26 +00:00
fix/dev-python/python-upper-bound
22
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  kevinveenbirkenbachandClaude Opus 5.5
|
284b0873dd |
fix(sys-ctl-hlth-runtime): harden LAM CLI, catch apk options, drop vars comment
Addresses three CodeRabbit findings on PR #1265. 1. LAM hardening invisible to the probe (thread 4413764929): the runtime check probes PHP with `php -r`, the CLI SAPI. web-app-lam mounted the hardening ini only into /etc/php/8.4/apache2/conf.d, which the Debian CLI does not scan, so the probe read unhardened CLI settings and the check failed for LAM. The php_hardening entry now also mounts the ini into /etc/php/8.4/cli/conf.d. No other role mounts it into a Debian per-SAPI directory; the rest use the shared scan dir of the official image or of Alpine. The TODO entry now names the remaining gap: the probe reads only the CLI copy. 2. Explanatory comment above HEALTH_RUNTIME_FLOORS (thread 4413764949): removed. The README already states the end-of-life rule and now also defines a floor as the lowest patched release of each line. 3. `apk --no-cache add` missed (thread 4415018662): PACKAGE_INSTALL_PATTERN required `add` right after `apk`, so `RUN apk --no-cache add php7` passed as an image without PHP. The apk branch now accepts options before `add`, like the apt branch. New tests cover the option form and guard that `apk` in a path or a comment is no package install. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5.5
|
d116dde659 |
fix(sys-ctl-hlth-runtime): bound the engine calls and catch hidden PHP runtimes
Three CodeRabbit findings on PR #1265: - script.py listed the running containers with `container ps` and `container inspect` without a timeout, so a hung engine kept the daily unit active forever and blocked the next timer run. Both calls now use EXEC_TIMEOUT_SECONDS; main turns a timeout or engine error into a one-line failure and exit 1 instead of a traceback. - runtime.py's PHP_PACKAGE_PATTERN requires a digit after "php", so an image installing an unversioned interpreter package (`php-cli`) and setting no PHP_VERSION was reported as PHP-free. The interpreter packages php, php-cli, php-fpm, php-cgi, php-phpdbg and libapache2-mod-php are now recorded as PhpRuntime.packages and reported with an unknown patch level; extension and library packages (php-mysql, php-pear, php-common, phpmyadmin) and paths do not count. - php_finding returned early for a patched PHP_VERSION, so an end-of-life package line installed next to it (e.g. php7) was never reported. The version only returns a finding now; otherwise the lines are judged too. Every finding has unit tests that fail on the previous code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5.5
|
feac35a5ea |
fix(sys-ctl-hlth-runtime): deploy only on container hosts and drop the bare Docker Hub API literal
Two CI failures on PR #1265 (run 40834164415):
- Chunk 0 / SuiteCRM#2 (swarm): the post-load flush included
sys-ctl-hlth-runtime on every play host, the NFS server too, which never
gets sys-svc-container and therefore has no `container` CLI. The unit
died with FileNotFoundError: 'container'. The role entry now runs only
where a container runtime exists: in compose mode, or on swarm nodes
('svc-swarm-node' in group_names), the same role-entry host scoping that
sys-ctl-hlth-csp and sys-ctl-hlth-webserver use with IS_STACK_HOST.
Workers stay covered because PHP replicas can run there.
- Workspace (ubuntu): tests/external/repository/test_urls_reachable.py
probed the literal "https://hub.docker.com/v2" from
utils/docker/image/runtime.py:20 and got HTTP 404. The constant is gone;
the endpoint is written as one f-string at the call site, the way
utils/update/docker.py:104 and utils/update/source.py do, which the URL
test skips as a template. With the scheme now visible in the literal,
ruff reported the noqa: S310 on urllib.request.Request as unused.
Both test files that imported unittest twice (CodeQL) now use one
`from unittest import ...` form.
The run_once queue leak that spreads the stack host's post-load queue to
every host is fixed separately on the updater line (
|
||
|
kevinveenbirkenbachandClaude Opus 5.5
|
3c743736a8 |
feat(sys-ctl-hlth-runtime): alert on EOL or unpatched PHP and missing PHP hardening
Answering "which deployed containers run a vulnerable PHP?" after CVE-2026-109618/109619 took a manual inventory of registry metadata, and nothing noticed a hardening ini that an image never loads. - New role sys-ctl-hlth-runtime, routed through sys-svc-compose next to sys-ctl-hlth-container. A daily timer runs `php` in every running container and fails, alerting through sys-ctl-alm-compose, when PHP is below the floor of its line or on a line older than every floor (7.4, 8.1). Every service that mounts php_hardening in meta/volumes.yml fails while expose_php or session.upload_progress.enabled is on; other PHP containers of such a stack are listed as info. Compose and swarm containers are matched by their project/stack and service labels. The check is not run at deploy time, so a finding never fails a deploy. - The floors (8.2.35, 8.3.36, 8.4.27, 8.5.12) live once, in HEALTH_RUNTIME_FLOORS of the role's vars/main.yml. - `make security-runtimes` lists the PHP of every pinned role image without pulling: Docker Hub tags through the Hub API, which no pull rate limit meters, other registries through the OCI config blob (new registry.config_blob, which the healthcheck probe now reuses). PHP_VERSION, distro PHP packages and a PHP_BASE build argument count. It exits 1 on an image below its floor or one the registry did not answer. Today it flags nfrastack/fusiondirectory:1.5 (line 7.4) and shopware/docker-base:8.3 (8.3.25). The role TODO.md keeps the follow-ups: further runtimes, the split web SAPI of web-app-lam, a scheduled CI run of the report and the CVE scan. Refs #1127 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5.5
|
8a29ba0bfd |
fix(images): leave eol roles out of image discovery
The scheduled mirror job has failed every day since 2026-09-25 on the same two
images: quay.io/minio/minio and quay.io/minio/mc answer `unauthorized` to
anonymous pulls ("2/197 failed", run 38352569503). Both belong to
web-app-minio, an eol role outside the lifecycle envelope that default.env
tests, so no CI deploy pulls them.
iter_role_images now skips every role that is_maintained() reports as
unmaintained, the check the version updaters already apply to the same list.
On the tree the discovery yields 187 images instead of 192; the five it drops
belong to the four eol roles web-app-confluence, web-app-jira, web-app-minio
and web-app-phpldapadmin. The mirror sync, resolver and wait read this list,
and their descriptions and mirror.md now say so.
The roles directory is resolved first: is_maintained reads a relative path as
a role name and would keep every role.
|
||
|
kevinveenbirkenbachandClaude Opus 5.5
|
1ba796d7f1 |
fix(ci): pull the images the runner uses itself from the GHCR mirror
The GHCR mirror covered role images only. The images the runner pulls on its own went straight to Docker Hub: coredns from compose.yml, the cache stack in compose/cache.override.yml and alpine:latest, the init image of scripts/docker/cache/package-frontend/certs.sh. With the anonymous githubactions quota spent, the guide step of Postmarks#0 (main run 37954203718) and Shopware#2 (#872) died on "toomanyrequests" while pulling alpine:latest, after coredns had hit the same limit. iter_runner_images (utils/docker/image/discovery.py) finds those images, and the mirror sync and the fork-PR wait now cover them next to the role images. On a real GitHub runner compose_file_args layers a generated build/compose/runner-mirror.override.yml that points only the services of the layered compose files at their mirror, and the certificate generator gets the mirrored init image. Local runs and act are unchanged, and the coredns literal stays in compose.yml for Dependabot. test-dns now waits for images-mirror-missing like the deploy and workspace jobs, since it brings up the stack too. Verified with docker compose config on a simulated GitHub runner: the service set is unchanged and only the runner images point at ghcr.io/infinito-nexus/core/mirror/docker.io/..., and an existing role mirror under that scheme pulls anonymously. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
e14e73fba0 |
refactor(docker): let a moving tag declare itself with the marker it already has
A pin outside semver was given its own 'docker-ref' marker on the theory that every class needs one. That holds for a digest and not for a tag. A digest pin declares something about the registry, that bytes move under a standing tag, which no other choice says. A tag outside semver declares that upstream publishes nothing orderable, which is exactly what 'docker-version' already means and what minio and the act runner already carry on their version lines. Keeping both would have put '# nocheck: docker-version, docker-ref' on every such line: two keys, two checks, one sentence said twice. 'docker-ref' is dropped instead. It lived in this one mapping, was never entered in the suppression catalog, and no check ever read it. The semver entry goes with it, because a semver pin never needs a declaration. The test that asserted three distinct rules is replaced by two that pin the separation where it is real: a digest marker differs from a tag marker, and a semver pin has no marker at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
be6a25517c |
feat(docker): classify image pins and build pull references from the class
A digest pin was emitted as `name:sha256:...`, which no registry accepts:
image_source('ghcr.io/.../csp-checker', 'sha256:d559...') returned
'ghcr.io/.../csp-checker:3.0.0'. The separator depends on the pin class, so
the class has to be decided before the reference is built.
utils/docker/image/pin.py names the three classes in preference order
(semver, then digest, then a general ref that no registry can order) and
maps each to the nocheck marker that declares it. Both discovery entry
points now end in pull_reference(), so '@' versus ':' is decided in one
place instead of at every call site.
The semver pin lint points a digest at 'docker-digest' instead of
'docker-version', so the two fallbacks stay distinguishable in the
suppression catalog. It stays warn-only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
5137465c6f |
style: delete every comment the comments-valid policy rejects
3529 invalid comments in 528 files, grandfathered in while the linter only saw the working-tree diff. Removal ran through the linter's own collectors, so detection and deletion cannot drift: a comment whose opener line is blank before it and whose closer line is blank after it loses all its lines, anything else loses only the comment region, and a line left holding a bare delimiter goes with it. Five files keep their comments behind a nocheck marker instead. Four are upstream configs whose inline docs are upstream's own (mautrix, systemd journald, nextcloud-oidc-login, MediaWiki). In littlejs the commented-out entries ARE the deactivation mechanism for an example tile, so deleting them would delete configuration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
b533863559 |
fix(env): keep the .env generation stdlib-only for fresh-host bootstrap
The nfs SPOT loaders pulled PyYAML into the .env generation, which bootstraps fresh hosts BEFORE python dependencies exist: the import crashed cli.meta.env, load.sh produced an empty environment and make install-system-python died on the required-var guard (environment/manjaro and pkgmgr/centos CI jobs). The loaders now line-parse the services.yml SPOTs via the stdlib-backed cached reader and the mapping path constant. The client's protocol version field becomes nfs_version, since services.yml reserves version for image tags (schema lint), and the image discovery casts versions defensively. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Fable 5
|
e35acc09bc |
feat(updater): support registry.gitlab.com images
Add registry.gitlab.com to the mirrorable registries so discovery yields the GitLab CNG images to the version updater and mirror sync. Seed the generic v2 tag scan with the OCI last cursor for v-prefixed pins and raise the page size to 1000: CNG repos carry tens of thousands of commit-sha tags that sort before the release tags, so an unseeded scan exhausted the page cap without reaching any semver tag. Pin every gitlab service version in meta/services.yml and drop the GITLAB_VERSION template override so services.yml is the single source for per-service versions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
e9b75f4ddf |
fix(mirror): enroll registry.opencode.de in the GHCR mirror (web-app-opentalk)
OpenTalk's images (web-frontend, controller, recorder) live on registry.opencode.de, which was absent from both _MIRRORABLE_REGISTRIES and _ALL_REGISTRY_PREFIXES in utils/docker/image/discovery.py. So is_mirrorable_image returned False, iter_role_images skipped them, the GHCR mirror never copied them, and the deploy pulled direct from registry.opencode.de -> TLS handshake timeout (deploy failed). Add the registry to both sets so the single iter_role_images gate enrolls the 3 images: the mirror-sync copies them to GHCR and the resolver rewrites the deploy to pull from the GHCR mirror. The prefix entry strips the host so the mirror destination is well-formed (no doubled registry). Add a regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
83466868e9 |
feat(update): support mcr.microsoft.com in docker version checker
- Add MCR_REGISTRY constant, is_mcr / mcr_repo / fetch_mcr_tags helpers mirroring the Docker Hub and GHCR pattern. Tags are fetched via the Docker Registry V2 API with RFC 5988 Link-header pagination (anonymous access; MCR is a public read mirror). Wire MCR into both the external test_image_versions warning surface and the find_outdated_updates orchestrator in utils.update.docker. - Bump mcr.microsoft.com/playwright in test-e2e-playwright from v1.59.1-noble to v1.60.0-noble — the update was surfaced by the new MCR detection path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
576fad84ff |
refactor(images): move role images from defaults/main.yml to meta/services.yml
Consolidates image declarations under a single schema: every role's
`<svc>.{image,version}` lives in `meta/services.yml` alongside compose
service entries. The `image` lookup now reads from disk per role,
`iter_role_images` collapses to a single source, and the mirror chain
(resolver + apply_mirror_overrides) drops the dead `images:` branch.
`test-e2e-playwright` resolves `TEST_E2E_PLAYWRIGHT_IMAGE` directly via
`lookup('image', 'playwright', 'ref')` in vars/main.yml; the runtime
set_fact + override-empty-string dance is gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
c55ad69c41 |
refactor: drop requirement-file references from code, add doc rule + lint
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`, `requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from non-Markdown source files. Comments anchored to requirement numbers rot when requirements are renumbered or rewritten and conflate present-state behaviour (which the comment should describe) with the audit trail of how the code got there (which belongs in commit messages, PR descriptions, and the requirement file itself). * Document the rule in `docs/contributing/documentation.md` under "Comments". Code MUST describe the *why* in present-state language; contract enforcement lives in tests, not folklore. * Clean 254 source files across `roles/`, `tests/`, `plugins/`, `cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root `eslint.config.js`. Pure comment / docstring / log-message edits; no behavioural change. * Add `tests/lint/repository/test_no_req_references_in_code.py` to enforce the rule going forward across `.js`, `.py`, `.yml`, `.yaml`, `.j2`, and `.sh` files. The legacy `test_no_stale_requirement_refs.py` is exempt as the only file whose purpose is to detect these very tokens. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
7dc6d6e491 |
chore(ruff): enable TC + N rules
TC (flake8-type-checking): typing-only imports moved into
``if TYPE_CHECKING:`` blocks across 8 source files via auto-fix
(117 hits). Reduces runtime import surface and rules out circular
imports for type-only deps. Project-wide ``from __future__ import
annotations`` (FA, already enabled) keeps annotations as strings,
so the moved imports never need to load at runtime.
N (pep8-naming): 32 hits handled.
Renames (semantic + lint):
- ``isSmallerThenMaximumBackupSize`` -> ``is_larger_than_maximum_backup_size``
(snake_case + fixes both the typo "Then -> Than" and the
semantic mismatch: the function returns ``current > maximum``)
- ``isDirectoryDeletable`` / ``deleteVersion`` /
``getAmountOfIteration`` / ``deleteIteration`` -> snake_case
- 8 test functions with ALL-CAPS env-var fragments
(``test_..._DOMAIN_PRIMARY_...`` etc.) -> snake_case
- Mock alias ``VH`` -> ``vh`` in test_host_vars.py
Exception class renames (``N818`` Error suffix):
- ``TimeLimitExceededException`` -> ``TimeLimitExceededError``
- ``AttemptException`` -> ``AttemptError``
- ``PathNotFound`` -> ``PathNotFoundError`` (propagated across
4 callers under tests/integration/lookups/config/)
- ``_ExitJson`` / ``_FailJson`` -> ``*_Error``
Refactor:
- ``METADATA = {...}`` local constant in two functions of
``plugins/filter/invokable_paths.py`` lifted to module-level
``_METADATA_KEYS``.
- ``from typing import TYPE_CHECKING`` in
``tasks/utils/migrate_meta_layout/__main__.py`` moved above
the ``sys.path.insert`` hack to satisfy E402 (collateral from
the TC auto-fix that landed it after the path mutation).
Per-file ignores in pyproject.toml for the legitimate cases the
rule cannot accept:
- ``pull-specific-host.py`` -> ``N999`` (Ansible-invoked role-level
script, hyphenated filename pinned by the surrounding stack).
- 4 matrix-deploy files -> ``N806`` for the ``include_R`` /
``raw_include_R`` round-index notation; the uppercase ``R``
here mirrors the formula in the docstring rather than naming
a constant.
- ``plugins/filter/value_generator.py`` -> ``N806`` for the
lazy-loaded class binding ``ValueGenerator =
_value_generator_cls()``, which retains the imported class
identifier per PEP 8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
f6c8c85b1b |
test(lint): role-file-spot guard + 139-file mapping migration
Add tests/lint/repository/test_role_file_spot.py: AST-walks every .py file under the production trees and tests/, flags any reference to a role file (standalone literal, f-string segment, pathlib `/` chain, `+` concat chain, os.path.join / Path.joinpath / posixpath.join / ntpath.join call) whose value or joined trailing operands match a ROLE_FILE_* constant in utils.roles.mapping. The flag set is auto-discovered from the module so adding a new constant extends coverage without further edits. ROLE_FILE_README is exempt as too generic. Per-line `# nocheck: role-file-spot` (same-or-above) is the documented opt-out, catalog row added in docs/contributing/actions/testing/suppression.md. Migrate 139 .py files (377 occurrences) to use the constants: - 135 files via direct constant import (`from utils.roles.mapping import ROLE_FILE_*`), with the symbol used in pathlib chains, in os.path.join / Path.joinpath calls, in f-string interpolation, and in glob patterns. - 4 files via `# nocheck: role-file-spot` where the literal lives in argparse help text or in an allowlist tuple meant to declare the literal path. Adjust utils/roles/dependency_resolver.py to use lazy-formatting positional args in the logger.exception call (G003-clean) instead of string concatenation against the constant. Drop a redundant `if cond: return True; return False` guard in the new test for `return cond` (SIM103-clean). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
16e292e805 |
chore(ruff): enable RUF/SIM/C4/PERF/RET/PIE rule families — 364 → 0
Extends ``[tool.ruff.lint].select`` with six additional rule families:
``RUF`` (ruff-specific), ``SIM`` (flake8-simplify), ``C4``
(flake8-comprehensions), ``PERF`` (perflint), ``RET`` (flake8-return),
``PIE`` (flake8-pie). Adds ``RUF001/002/003`` to ``ignore`` because
those exist for homoglyph-attack detection, not English typography —
the project uses em/en-dashes and ``ℹ`` as a UI-hint glyph in CLI help
output deliberately.
Auto-fixed (~225, mechanical & behavior-preserving)
====================================================
- RUF005: ``a + [b]`` → ``[*a, b]``
- C408: ``dict(k=v)`` → ``{"k": v}``
- C420: ``dict.fromkeys(seq)`` shape simplifications
- SIM103/SIM110/SIM118: truthy-return / dict-key / dict-method
idiom simplifications
- RET504/RET505: drop unneeded final assigns / ``else`` after
``return``
- PIE810: combine ``str.startswith(a) or .startswith(b)``
into one call with a tuple
- RUF010: ``f"{str(x)}"`` → ``f"{x\!s}"``
- RUF100: removed unused ``# noqa`` directives that
became stale (mostly old ``# noqa: E402``
markers that ruff's smart E402 implementation
no longer flags after ``sys.path`` shims)
- RUF015 / RUF013 / RUF022 / RUF046 / RUF059: misc small wins
- I001: re-sorted import blocks where the new
imports above shifted them
Hand-fixed (~140, scripted where mechanical)
============================================
PERF401 (42): scripted AST rewrite of
``for x in iter: target.append(expr)`` (with optional ``if`` guard)
→ ``target.extend(expr for x in iter [if cond])``.
``list.extend(genexp)`` consumes the generator immediately, so the
rewrite is semantically equivalent to the original eager loop.
SIM117 (41): scripted AST combiner for nested ``with`` blocks where
the inner is the only statement of the outer (38 sites). The remaining
3 multi-line ``patch.object(...)`` chains were combined by hand using
the parenthesised ``with (\\n A,\\n B,\\n):`` form.
SIM102 (14): scripted AST combiner for nested ``if`` (10 sites). The
remaining 4 carry an inline ``# noqa: SIM102`` with rationale —
combining would either change semantics (the outer body has siblings
beyond the inner ``if``, e.g. ``cli/encrypt/inventory/__main__.py``
where ``encrypted_value = …`` runs regardless of ``ask_confirmation``)
or hurt readability (long literal whitelists).
RUF012 (9): annotated mutable class attributes with ``ClassVar[…]``
and added ``from typing import ClassVar``.
SIM105 (6): ``try/except: pass`` → ``with contextlib.suppress(…):``
with the necessary ``import contextlib`` added. Verified on
``utils/templating.py`` (3 occurrences in a ``finally`` block) —
cleanup behaviour is identical.
SIM115 (4): wrapped ``tempfile.NamedTemporaryFile(..., delete=False)``
call patterns in a ``with`` block; ``delete=False`` keeps the file
on disk after the block, so callers' ``return f.name`` still works.
PERF402, PERF403, B007, SIM108, SIM103: 5 misc fixes (loop→list-copy,
loop→dict-comprehension, unused-loop-var rename, ternary, return
expression).
Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files already formatted
- 1621 unit tests → all pass
- Integration suite: 44 baseline failures, 1 skipped (unchanged)
- Sample-audited PERF401 rewrites in ``resolver.py``, ``inventory.py``,
``cache/files.py``, ``ressources/__main__.py``; SIM117 in
``test_main.py``; SIM105 in ``utils/templating.py``; SIM102 noqa
rationales in ``cli/encrypt/inventory/__main__.py``.
182 files changed, +993 / −960. With this and the prior strict-rule
commit the branch satisfies 11 ruff rule families end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
26ef3ff504 |
chore(ruff): enable strict ruleset (E/F/I/B/UP, py311) — 2819 → 0
Adds ``[tool.ruff.lint]`` with ``select = ["E", "F", "I", "B", "UP"]``
and ``target-version = "py311"``. ``E501`` is ignored because
``ruff format`` already reflows what it can; the remaining long lines
are unsplittable string literals / URLs.
Auto-fixed (2753 issues, behavior-preserving):
- UP006/UP035: ``List[X]`` → ``list[X]``, ``Dict[X]`` → ``dict[X]``,
``from typing import …`` redirected where superseded by builtins.
- UP045: ``Optional[X]`` → ``X | None``.
- UP010/UP009/UP015: dropped Python-2-era pragmas (``from __future__
import absolute_import, division, print_function``,
``# -*- coding: utf-8 -*-``), redundant ``open(..., "r")`` mode.
- UP004: dropped implicit ``class X(object):``.
- UP022/UP008/UP032: stdlib idiom modernization (``capture_output``,
super-call simplification, etc.).
- B905: ``zip(...)`` → ``zip(..., strict=False)`` (preserves prior
behavior; switching to ``strict=True`` is a separate semantic
choice not made here).
- I001: import-block ordering.
Hand-fixed (66 issues):
- B904 × 54: appended ``from <bound>`` (when the enclosing ``except``
binds a name) or ``from None`` (when it doesn't) so chained
exceptions stay distinguishable from errors-during-exception-
handling. Applied via a small AST-aware script that walked back to
the enclosing ``except`` clause and matched the binding.
- B017 × 4: narrowed broad ``assertRaises(Exception)`` to the actual
``AnsibleError`` / ``AnsibleFilterError`` raised by the code under
test.
- B023 × 2: bound captured loop variables via default args
(``def _record(name, *, _path_real=path_real, ...)``) so each
closure gets its own snapshot.
- B007 × 2: unused loop control variables renamed to ``_`` /
rewritten to ``.values()``.
- UP007 × 2: ``Union[X, Y]`` → ``X | Y`` (plus dropping the now-unused
``from typing import Union``).
- B018 × 1: Odoo addon manifests are bare dict expressions at module
level by design — suppressed with an inline ``# noqa: B018`` and
rationale comment.
- UP031 × 1: printf-style format → f-string.
The 65 ``# noqa: …`` markers introduced by the auto-fixer all carry
real ruff codes (``B018``, ``E402``, ``F401``, etc.), so
``test_noqa_only_ruff_codes`` stays green.
Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files already formatted
- ``tests.lint.repository.{test_project_root_import,
test_no_direct_yaml_calls, test_noqa_only_ruff_codes}`` and
``tests.unit.utils.annotations.test_suppress`` → 20 tests OK
- 1621 unit tests → all pass
- Integration suite: 44 failures (pre-existing baseline, none from
this change), 1 skipped.
Going forward, the new rules block regressions on E (style), F
(pyflakes), I (import order), B (likely-bug patterns), and UP
(modernization to py311) for any new code.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
a8a117c86e |
refactor(yaml): route every YAML touchpoint through utils.cache.yaml + lint
New lint guard
==============
tests/lint/repository/test_no_direct_yaml_calls.py AST-walks the
production tree (utils/, cli/, plugins/, filter_plugins/,
lookup_plugins/, roles/, scripts/, tasks/) and flags any direct
`yaml.safe_load` / `yaml.safe_dump` / `yaml.load` / `yaml.dump` /
`yaml.safe_load_all` / `yaml.dump_all` call. Aliased imports are
tracked: `import yaml as Y` ⇒ `Y.safe_load(...)` is flagged;
`from yaml import safe_load` ⇒ bare `safe_load(...)` is flagged.
Per-line opt-out: `# noqa: direct-yaml` on the call's physical line
(or any line spanned by a multi-line call). No central exemption
list — every legitimate exception lives next to its call with a
human-readable rationale in the surrounding comment.
Cache module additions
======================
* `dump_yaml_str(data)` — symmetric to `safe_dump(data)` with no file
path. Used for stdout output and filter return values where the
cache cannot help but the API needs to stay consistent.
* `load_yaml_str(text)` — symmetric to `safe_load(text)`. Used when
YAML comes from an HTTP body, a mutated buffer, or a mock-patched
source whose content does not correspond to a stable on-disk file.
Migrated production call sites (33 → 0 direct-yaml violations)
==============================================================
* utils/{entity_name_utils, valid_deploy_id, invokable, service_registry,
domains/list, applications/{config, in_group_deps},
docker/image/discovery, meta/{port_bands, scan},
roles/{dependency_resolver, meta_lookup, resource_validation}}
* cli/{build/{graph, inventory/full/command}, create/inventory/{host_vars,
mirror_overrides}, fix/vars_main_files, meta/{applications/in_group_deps,
callorder}, mirror/resolver, validate/inventory}
* plugins/{filter/{compose_volumes, get_all_application_ids, get_role,
invokable_paths, role_path_by_app_id}, lookup/applications_current_play}
* roles/{svc-db-postgres/filter_plugins/split_postgres_connections,
web-app-bigbluebutton/filter_plugins/compose_mods,
web-app-dashboard/lookup_plugins/docker_cards}
Behaviour adjustments
=====================
* utils/roles/dependency_resolver — `safe_load_all` (multi-doc) →
single-doc `load_yaml_any` wrapped in a list. Ansible task files are
single-doc by convention; the downstream multi-doc loop stays intact
for forward compatibility.
* roles/sys-svc-compose-ca/files/compose_ca.py — KEEPS direct
yaml.safe_load / yaml.safe_dump (annotated with noqa). The script is
copied to the target host via `ansible.builtin.copy` and the target
has no `utils/` package on PYTHONPATH.
noqa-annotated direct-yaml call sites (with rationale)
======================================================
* utils/cache/yaml.py — IS the cache; would recurse.
* utils/handler/yaml.py — vault-aware Loader subclass that wraps
\!vault scalars in VaultScalar; cache-by-path collapses the wrapping.
* cli/encrypt/inventory/__main__.py — vault-aware Dumper subclass that
emits \!vault literal blocks; the cached dump_yaml writes plain YAML
which is the wrong output here.
* roles/sys-svc-compose-ca/files/compose_ca.py — runtime-deployed.
* tasks/utils/migrate_meta_layout.py — once-off migration script.
Validation
==========
* Unit tests: 1573 passed.
* Lint: 56 passed (incl. new test_no_direct_yaml_calls).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
ed04dad5d5 |
refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
 Environment TestandClaude Sonnet 4.6
|
eebb3b9788 |
refactor(utils): move utils/docker/image_*.py into utils/docker/image/ subpackage
- Move image_discovery.py → utils/docker/image/discovery.py - Move image_ref.py → utils/docker/image/ref.py - Add utils/docker/image/__init__.py - Update all imports and references across cli/, tests/, utils/, and docs/ Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |