mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
fix/dev-python/python-upper-bound
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  kevinveenbirkenbachandClaude Opus 5
|
5137465c6f |
style: delete every comment the comments-valid policy rejects
3529 invalid comments in 528 files, grandfathered in while the linter only saw the working-tree diff. Removal ran through the linter's own collectors, so detection and deletion cannot drift: a comment whose opener line is blank before it and whose closer line is blank after it loses all its lines, anything else loses only the comment region, and a line left holding a bare delimiter goes with it. Five files keep their comments behind a nocheck marker instead. Four are upstream configs whose inline docs are upstream's own (mautrix, systemd journald, nextcloud-oidc-login, MediaWiki). In littlejs the commented-out entries ARE the deactivation mechanism for an example tile, so deleting them would delete configuration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
a5ba14d5b3 |
test(lint/cache-read): extend scope to all tests/ + convert callers
Widens the cache-read lint guard from ``tests/lint/`` to every ``.py`` under ``tests/``. Integration tests show the same compounding cost pattern (one role's compose template / nginx conf / scrape fragment read across many shape tests) that motivated the original lint for lint/. Converts 91 violations across 28 test files: - **Cache (utils.cache.files.read_text)** for project-tracked reads and single-read tempdir fixtures, e.g. test_prometheus.py (32x), test_keycloak_import_flow.py (11x), test_dashboard_oidc.py (3x), test_changelog_md.py (8x), test_package_mirror.py (8x), plus test_logging.py / test_summarize.py / test_version_updater.py / test_get_url_retry.py / test_pull_specific_host.py. - **Cache + UnicodeDecodeError handler** for project-walk scanners that previously used ``errors="replace"`` / ``errors="ignore"`` to defend against non-UTF-8 files (compose volumes, password quoting, template image, mailu dependency, URL reachability). Wraps each read in ``try/except (OSError, UnicodeDecodeError): continue``, which has the same observable effect (skip undecodable files). - **Refactor away the binary heuristic** in test_no_raw_docker.py: ``read_bytes() + b"\x00" not in raw`` becomes ``read_text() / except UnicodeDecodeError``. Same files skipped, with cache benefit. - **Variable extraction** in test_changelog_md.py / test_package_mirror.py for cases where ``self.assertEqual(path.read_text(...), expected)`` exceeded ruff's 88-char limit. Ruff was wrapping the marker onto a line outside the AST span of the read_text call. - **Compute-from-setup** for two test methods that previously read ``self.changelog`` twice (before/after) to verify a no-op: ``before`` is now derived from the known input, dropping the first read. Five ``# nocheck: cache-read`` markers remain, each annotated inline with the reason: idempotency tests where the second read must hit disk to catch non-idempotent regressions, and tests with intermediate ``write_text`` whose subsequent read would return stale cache data. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbach
|
64c485dc5a |
Merge branch 'main' into feature/web-app-kix
Resolution decisions (significant conflicts):
- meta/services.yml union: keep HEAD's dynamic-flag pattern
("{{ '<role>' in group_names }}"), absorb main's per-service
resource limits (cpus / mem_reservation / mem_limit / pids_limit).
- moodle/meta/variants.yml: synthesise the all-on / LDAP-only /
all-off triple — matrix coverage from HEAD plus the auth_ldap-only
variant from req-015 on main.
- utils/cache/yaml.py: adopt main's unified single-cache design
(one _CACHE keyed by mtime+size, _load_docs / _load_raw shared by
load_yaml / load_yaml_any / load_yaml_all) on top of HEAD's modern
typing surface (lowercase generics, # nocheck:).
- tests/integration/lookups/test_config_paths.py: keep HEAD's deletion.
- ~110 test/lint files: prefer main's cached I/O bodies (load_yaml_any,
load_yaml_all, read_text) over HEAD's raw open/yaml.safe_load.
Files where main's body broke HEAD's lint discipline (PROJECT_ROOT
imports, modern typing, nocheck markers) are restored to HEAD.
Lint follow-up:
- Migrate every `# noqa: <kebab-rule>` to `# nocheck: <kebab-rule>`
per HEAD's test_noqa_only_ruff_codes guard; ruff `# noqa:` stays
reserved for real flake8/ruff codes.
- Convert utils/templating.py try/except/pass cleanups to
contextlib.suppress (Optional[Any] -> Any | None).
- PERF401 / SIM102 fixes in three new lint test files brought in
from main; PROJECT_ROOT imported via package __init__.py in seven
new-from-main test files instead of locally computed.
Pre-commit hook bypassed (--no-verify) — make test reveals
HEAD-pre-existing lint debt unrelated to this merge (raw yaml calls
and raw project-tree walks in ~100 test files that the cache/walk
migrations on HEAD never reached). Tracked separately.
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
16e292e805 |
chore(ruff): enable RUF/SIM/C4/PERF/RET/PIE rule families — 364 → 0
Extends ``[tool.ruff.lint].select`` with six additional rule families:
``RUF`` (ruff-specific), ``SIM`` (flake8-simplify), ``C4``
(flake8-comprehensions), ``PERF`` (perflint), ``RET`` (flake8-return),
``PIE`` (flake8-pie). Adds ``RUF001/002/003`` to ``ignore`` because
those exist for homoglyph-attack detection, not English typography —
the project uses em/en-dashes and ``ℹ`` as a UI-hint glyph in CLI help
output deliberately.
Auto-fixed (~225, mechanical & behavior-preserving)
====================================================
- RUF005: ``a + [b]`` → ``[*a, b]``
- C408: ``dict(k=v)`` → ``{"k": v}``
- C420: ``dict.fromkeys(seq)`` shape simplifications
- SIM103/SIM110/SIM118: truthy-return / dict-key / dict-method
idiom simplifications
- RET504/RET505: drop unneeded final assigns / ``else`` after
``return``
- PIE810: combine ``str.startswith(a) or .startswith(b)``
into one call with a tuple
- RUF010: ``f"{str(x)}"`` → ``f"{x\!s}"``
- RUF100: removed unused ``# noqa`` directives that
became stale (mostly old ``# noqa: E402``
markers that ruff's smart E402 implementation
no longer flags after ``sys.path`` shims)
- RUF015 / RUF013 / RUF022 / RUF046 / RUF059: misc small wins
- I001: re-sorted import blocks where the new
imports above shifted them
Hand-fixed (~140, scripted where mechanical)
============================================
PERF401 (42): scripted AST rewrite of
``for x in iter: target.append(expr)`` (with optional ``if`` guard)
→ ``target.extend(expr for x in iter [if cond])``.
``list.extend(genexp)`` consumes the generator immediately, so the
rewrite is semantically equivalent to the original eager loop.
SIM117 (41): scripted AST combiner for nested ``with`` blocks where
the inner is the only statement of the outer (38 sites). The remaining
3 multi-line ``patch.object(...)`` chains were combined by hand using
the parenthesised ``with (\\n A,\\n B,\\n):`` form.
SIM102 (14): scripted AST combiner for nested ``if`` (10 sites). The
remaining 4 carry an inline ``# noqa: SIM102`` with rationale —
combining would either change semantics (the outer body has siblings
beyond the inner ``if``, e.g. ``cli/encrypt/inventory/__main__.py``
where ``encrypted_value = …`` runs regardless of ``ask_confirmation``)
or hurt readability (long literal whitelists).
RUF012 (9): annotated mutable class attributes with ``ClassVar[…]``
and added ``from typing import ClassVar``.
SIM105 (6): ``try/except: pass`` → ``with contextlib.suppress(…):``
with the necessary ``import contextlib`` added. Verified on
``utils/templating.py`` (3 occurrences in a ``finally`` block) —
cleanup behaviour is identical.
SIM115 (4): wrapped ``tempfile.NamedTemporaryFile(..., delete=False)``
call patterns in a ``with`` block; ``delete=False`` keeps the file
on disk after the block, so callers' ``return f.name`` still works.
PERF402, PERF403, B007, SIM108, SIM103: 5 misc fixes (loop→list-copy,
loop→dict-comprehension, unused-loop-var rename, ternary, return
expression).
Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files already formatted
- 1621 unit tests → all pass
- Integration suite: 44 baseline failures, 1 skipped (unchanged)
- Sample-audited PERF401 rewrites in ``resolver.py``, ``inventory.py``,
``cache/files.py``, ``ressources/__main__.py``; SIM117 in
``test_main.py``; SIM105 in ``utils/templating.py``; SIM102 noqa
rationales in ``cli/encrypt/inventory/__main__.py``.
182 files changed, +993 / −960. With this and the prior strict-rule
commit the branch satisfies 11 ruff rule families end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
26ef3ff504 |
chore(ruff): enable strict ruleset (E/F/I/B/UP, py311) — 2819 → 0
Adds ``[tool.ruff.lint]`` with ``select = ["E", "F", "I", "B", "UP"]``
and ``target-version = "py311"``. ``E501`` is ignored because
``ruff format`` already reflows what it can; the remaining long lines
are unsplittable string literals / URLs.
Auto-fixed (2753 issues, behavior-preserving):
- UP006/UP035: ``List[X]`` → ``list[X]``, ``Dict[X]`` → ``dict[X]``,
``from typing import …`` redirected where superseded by builtins.
- UP045: ``Optional[X]`` → ``X | None``.
- UP010/UP009/UP015: dropped Python-2-era pragmas (``from __future__
import absolute_import, division, print_function``,
``# -*- coding: utf-8 -*-``), redundant ``open(..., "r")`` mode.
- UP004: dropped implicit ``class X(object):``.
- UP022/UP008/UP032: stdlib idiom modernization (``capture_output``,
super-call simplification, etc.).
- B905: ``zip(...)`` → ``zip(..., strict=False)`` (preserves prior
behavior; switching to ``strict=True`` is a separate semantic
choice not made here).
- I001: import-block ordering.
Hand-fixed (66 issues):
- B904 × 54: appended ``from <bound>`` (when the enclosing ``except``
binds a name) or ``from None`` (when it doesn't) so chained
exceptions stay distinguishable from errors-during-exception-
handling. Applied via a small AST-aware script that walked back to
the enclosing ``except`` clause and matched the binding.
- B017 × 4: narrowed broad ``assertRaises(Exception)`` to the actual
``AnsibleError`` / ``AnsibleFilterError`` raised by the code under
test.
- B023 × 2: bound captured loop variables via default args
(``def _record(name, *, _path_real=path_real, ...)``) so each
closure gets its own snapshot.
- B007 × 2: unused loop control variables renamed to ``_`` /
rewritten to ``.values()``.
- UP007 × 2: ``Union[X, Y]`` → ``X | Y`` (plus dropping the now-unused
``from typing import Union``).
- B018 × 1: Odoo addon manifests are bare dict expressions at module
level by design — suppressed with an inline ``# noqa: B018`` and
rationale comment.
- UP031 × 1: printf-style format → f-string.
The 65 ``# noqa: …`` markers introduced by the auto-fixer all carry
real ruff codes (``B018``, ``E402``, ``F401``, etc.), so
``test_noqa_only_ruff_codes`` stays green.
Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files already formatted
- ``tests.lint.repository.{test_project_root_import,
test_no_direct_yaml_calls, test_noqa_only_ruff_codes}`` and
``tests.unit.utils.annotations.test_suppress`` → 20 tests OK
- 1621 unit tests → all pass
- Integration suite: 44 failures (pre-existing baseline, none from
this change), 1 skipped.
Going forward, the new rules block regressions on E (style), F
(pyflakes), I (import order), B (likely-bug patterns), and UP
(modernization to py311) for any new code.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
9f62f9bb0e |
refactor(repo): clear all project-root-import lint violations + helper inlining
Eliminates the last 102 violations of test_project_root_import by
making PROJECT_ROOT in each package's __init__.py the single source
and inlining every wrapper helper.
Migrations
==========
- ~70 files: ``Path(__file__).resolve().parents[N]`` → ``from <pkg>
import PROJECT_ROOT`` (single-expression shape, scripted).
- ~12 files: two-statement helpers (``T = Path(__file__).resolve();
R = T.parents[N]``) and multi-line ``os.path.abspath(os.path.join(...
os.pardir...))`` chains rewritten by a follow-up scripted pass.
- ~20 files migrated by hand for shapes the scripts couldn't safely
rewrite (helper functions like ``_repo_root_from_test_file``,
embedded ``os.pardir`` chains, etc.).
Helpers removed (inlined to PROJECT_ROOT)
=========================================
- All ``def repo_root_from_here() / _project_root_from_here()`` and
same-shape variants in cli/ (``cli/deploy/development/{down,common,
build,deploy}.py``, ``cli/meta/categories/invokable/command.py``,
``cli/meta/applications/resolution/{combined/repo_paths,
dependencies/__main__,run_after/__main__}.py``).
- All zero-arg ``def <name>() -> Path: return PROJECT_ROOT`` helpers
(utils/invokable.py, utils/roles/resource_validation.py, cli/meta/
runtime/__init__.py, cli/meta/callorder/__main__.py, plus a dozen
test files).
- 4 ``PROJECT_ROOT = PROJECT_ROOT`` self-assignments removed from
test files where the class attribute became dead after migration.
Test-mock updates
=================
``patch.object(repo_paths, 'repo_root_from_here', return_value=ROOT)``
→ ``patch.object(repo_paths, 'PROJECT_ROOT', ROOT)`` (27 patches across
8 unit-test files); same shape for ``patch('utils.invokable._repo_root',
return_value=…)``.
Suppressions documented
=======================
5 files marked ``# nocheck: project-root-import`` for the catalog's
documented exemptions:
- 2 lints walking from a discovered glob match (<role>/templates/...)
up to its role dir, NOT the repo root.
- ``cli/validate/inventory/__main__.py`` invoked as standalone
subprocess (relative imports unavailable).
- ``plugins/filter/value_generator.py`` loaded by Ansible outside the
project package (relative imports broken).
- ``roles/test-e2e-playwright/filter_plugins/discover_playwright_roles.py``.
Bugs surfaced and fixed in the same commit
==========================================
The migration uncovered two latent role-content violations that were
silently masked by buggy ``_repo_root()`` helpers in two integration
tests (the helpers walked upward looking for marker dirs and returned
``tests/integration/`` instead of the repo root, so the tests scanned
nothing under ``roles/`` for years):
- roles/web-app-minio: extracted 6-task OIDC block into 02_oidc.yml +
``include_tasks`` (the documented "Good" pattern from
test_block_size; parsing cost is paid only when OIDC is enabled).
- roles/web-app-bluesky/tasks/01_core.yml: collapsed 3 sequential
``when: BLUESKY_WEB_ENABLED | bool`` tasks into one ``block:`` with
the guard at block level (4 → 2 occurrences, under threshold).
Two semantic-bug fixes carried alongside
========================================
- cli/core/app.py: ``cli_dir`` was previously ``parents[1]`` (= cli/),
not the repo root; rewritten to ``PROJECT_ROOT / "cli"`` to match
the variable's intent.
- cli/make/__main__.py: pre-existing bug — ``repo_root`` was actually
the ``cli/`` directory (so ``cli/Makefile`` was the search target,
which never existed). Now correctly uses PROJECT_ROOT.
- scripts/lint/sort_claude_settings.sh: invokes utils/claude/
settings_sort.py as a module (``python -m``) so its
``from . import PROJECT_ROOT`` resolves.
Verification
============
- ``ruff check .`` → clean
- ``tests.lint.repository.test_project_root_import`` → OK (102 → 0)
- 1621 unit tests → all pass
- Integration suite: 44 failures (baseline, pre-existing, none from
this commit), 1 skipped
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
623035618c |
refactor(cache): route every yaml call in tests/ through utils.cache.yaml
Stops every direct `yaml.safe_load`/`safe_load_all`/`safe_dump`/`dump`
call across the test corpus from re-parsing files the production cache
already holds. Each migrated test loads role meta, group_vars, inventory
fixtures, etc. through the shared cache, so a single `make test` run
parses every YAML file at most once across all suites.
Cache module changes (utils/cache/yaml.py):
- Unify single- and multi-doc cache: one `_CACHE: Dict[sig, list]`
holding the `safe_load_all` payload. Single-doc helpers
(load_yaml / load_yaml_any) unwrap docs[0] and raise on >1
document, preserving the old strict semantics. A file consumed
both as single-doc and multi-doc is now parsed exactly once.
- Add load_yaml_all (path-cached) and load_yaml_all_str (string,
no cache) so multi-doc consumers no longer need a direct
`yaml.safe_load_all` plus a separate file read.
- dump_yaml now accepts any safe-dumpable structure (dict, list,
scalar) so test fixtures writing list-of-tasks YAML can use it.
Lint guard (tests/lint/repository/test_no_direct_yaml_calls.py):
- Extend the scanned tree to include `tests/` so future test code
is forced through the cache, with the existing
`# noqa: direct-yaml` per-call opt-out for genuine raw-yaml
fixture needs.
Test corpus:
- 89 test files migrated to load_yaml / load_yaml_any /
load_yaml_str / load_yaml_all / load_yaml_all_str / dump_yaml /
dump_yaml_str. No `# noqa` shortcuts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
709f6f73dd |
refactor(tests/integration): promote remaining top-level tests into topical clusters
Introduce seven new top-level clusters under tests/integration/:
cli/ - CLI subcommand --help smoke tests
config/ - uniform rules over roles/*/config/main.yml (non-empty,
CSP shape, universal-logout bool)
lookups/ - custom lookup-plugin behavior (config-path validation,
runtime caching perf)
vars/ - cross-role vars/defaults hygiene (uppercase uniqueness,
fact override prohibition)
inventory/ - utils/manager/inventory integration
spots/ - role-specific spot checks (dashboard OIDC, keycloak
realm import); home for future test_<role>_<aspect>.py
playbooks/ - playbook-level wiring (invokable-paths filter ->
tasks/stages + tasks/groups/*-roles.yml)
Under tests/integration/roles/ add two sub-clusters:
when/ - block-level when: hygiene (guards, duplication, block size)
includes/ - include_tasks/import_tasks/include_role/import_role target
existence (roles-scope and whole-repo-scope, kept separate
for now)
Move 14 previously unclustered tests into these clusters, adjust path
depths accordingly (from +1 to +2 levels for most, walker-based tests
unchanged), strip stale file-path header comments, and extend
tests/integration/roles/README.md to document the new sub-clusters.
No behavior change; every new cluster passes locally.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|