kevinveenbirkenbach/computer-playbook
1
0 Fork 0
mirror of https://github.com/kevinveenbirkenbach/computer-playbook.git synced 2026-11-06 10:59:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
14,682 Commits 9 Branches 78 Tags
fix/dev-python/python-upper-bound
Commit Graph
10 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5 5137465c6f style: delete every comment the comments-valid policy rejects 
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.

Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-18 03:22:02 +02:00
kevinveenbirkenbachandClaude Opus 4.7 fbecc09d24 refactor(tests): regroup meta tests, add native-metrics contract 
- New bidirectional contract test ensures `services.prometheus.native_metrics.enabled`
  and `roles/<role>/templates/prometheus.yml.j2` stay consistent across the roles tree,
  replacing per-role spot checks for gitea/mattermost/matrix. Document the `native-metrics`
  suppression rule and apply it to web-app-mattermost (prepared-but-default-off fragment).
- Move every test in `tests/integration/meta/config/` into the per-domain layout under
  `tests/integration/roles/meta/{services,server}`. Group prometheus tests under
  `services/prometheus/`, repositories tests under `services/repositories/`, the
  `server.csp` shape test under `server/`. Move `test_non_empty.py` one level up
  (scans all meta/*.yml, not just services.yml).
- Delete `tests/integration/meta/spots/` — both files were only `assertIn` over file
  contents (brittle, no behavioural coverage, knowledge belongs in code comments or
  proper integration tests).
- Harden `test_project_root_import.py` to catch chained `Path(__file__)…parent.parent.…`
  walks (the attribute-access twin of `parents[N]` that previously slipped through).
  Refactor 6 surfaced sites to import canonical `PROJECT_ROOT` / `ROLES_DIR`, plus
  delete the now-redundant `_find_roles_dir` / `_find_project_root` helpers in
  `native_metrics_apps` lookup and `invokable_paths` filter plugin.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 12:50:35 +02:00
kevinveenbirkenbachandClaude Opus 4.7 f6c8c85b1b test(lint): role-file-spot guard + 139-file mapping migration 
Add tests/lint/repository/test_role_file_spot.py: AST-walks every .py
file under the production trees and tests/, flags any reference to a
role file (standalone literal, f-string segment, pathlib `/` chain,
`+` concat chain, os.path.join / Path.joinpath / posixpath.join /
ntpath.join call) whose value or joined trailing operands match a
ROLE_FILE_* constant in utils.roles.mapping. The flag set is
auto-discovered from the module so adding a new constant extends
coverage without further edits. ROLE_FILE_README is exempt as too
generic. Per-line `# nocheck: role-file-spot` (same-or-above) is the
documented opt-out, catalog row added in
docs/contributing/actions/testing/suppression.md.

Migrate 139 .py files (377 occurrences) to use the constants:

- 135 files via direct constant import (`from utils.roles.mapping
  import ROLE_FILE_*`), with the symbol used in pathlib chains, in
  os.path.join / Path.joinpath calls, in f-string interpolation, and
  in glob patterns.
- 4 files via `# nocheck: role-file-spot` where the literal lives in
  argparse help text or in an allowlist tuple meant to declare the
  literal path.

Adjust utils/roles/dependency_resolver.py to use lazy-formatting
positional args in the logger.exception call (G003-clean) instead of
string concatenation against the constant.

Drop a redundant `if cond: return True; return False` guard in the
new test for `return cond` (SIM103-clean).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-08 13:21:22 +02:00
kevinveenbirkenbach 76b139c894 fix(lint): migrate raw IO + project walks to utils.cache + variant coverage 
Cleans up the lint debt the main->feature/web-app-kix merge surfaced.
Each change addresses a real failure surfaced by `make test`; no
``# nocheck`` blanket suppressions added.

Migrations to the shared cache helpers:
- ~25 test files swap raw ``yaml.safe_load`` / ``yaml.safe_dump`` /
  ``yaml.dump`` for ``utils.cache.yaml.{load_yaml_any,load_yaml_all,
  load_yaml_str,dump_yaml,load_yaml_all_str}``.
- ~30 test files swap raw ``glob.glob`` / ``os.walk`` / ``Path.rglob``
  for ``utils.cache.files.iter_project_files`` (or
  ``iter_non_ignored_files`` for the dependabot fallback walker, or
  per-role ``Path.iterdir`` / ``Path.glob`` where the scope is a
  single subtree).
- 3 ``Path.read_text`` lint helpers now route through
  ``utils.cache.files.read_text``.

Variant + service-flag fixes:
- web-app-kix: shift its subnet from 192.168.105.80/28 to
  192.168.105.128/28 to clear the collision with web-app-hugo and
  svc-prx-openresty.
- web-app-fediwall: add the missing ``mastodon`` / ``friendica``
  service flags referenced by ``meta/variants.yml``; pin the css /
  logout / dashboard / matomo / prometheus literals with
  ``# nocheck: dynamic-flag`` and add a third "everything off"
  variant so the variant-coverage matrix sees both true and false
  pinning for every dynamic flag.
- web-app-magento, web-app-shopware, web-svc-xmpp: switch the
  ``run_after: web-app-keycloak`` matcher from a literal-false
  oauth2/oidc opt-out to the ``"{{ 'web-app-keycloak' in group_names }}"``
  dynamic form, and extend each role's ``meta/variants.yml`` with the
  matching true/false (and shopware's LDAP-only) coverage.
- web-app-fusiondirectory, web-app-matrix: declare the
  ``matomo`` flag so their ``run_after: web-app-matomo`` resolves
  through an explicit services entry; add matomo true/false to both
  variants files.
- web-app-fediwall, web-app-mini-qr: add the
  ``# nocheck: logout`` opt-out directly above ``logout:`` so the
  logout-dashboard guard accepts the legitimately-disabled state.
- ~30 ``services.yml`` files: insert
  ``# nocheck: dynamic-flag`` directly above the literal
  ``css`` / ``redis`` / ``view`` / ``worker`` / ``email`` / ``html``
  / ``mariadb`` / ``opensearch`` / ``talk`` / ``logout`` blocks that
  the dynamic-flag guard surfaced.

Code-quality follow-ups uncovered by ruff:
- utils/cache/yaml.py: collapse ``load_yaml_all_str`` to a one-line
  return + on-call ``# nocheck: direct-yaml`` so the lint accepts
  the cache module's own ``yaml.safe_load_all`` invocation.
- utils/templating.py: replace the ``Optional[Any]`` import that
  slipped in from main with the modern ``Any | None``; rewrite the
  cleanup ``try/except/pass`` blocks as
  ``contextlib.suppress(Exception)`` so the imported
  ``contextlib`` module is consumed consistently.
- 7 new-from-main lint test files (test_inline_script_block_size,
  test_no_inline_multiline_php_in_sh, test_no_lookup_config_jinja_default,
  test_no_inline_multiline_sql, test_web_role_no_web_dependency,
  test_shared_primary_entity_run_once, test_fediwall_active): drop
  their local ``Path(__file__).parents[N]`` PROJECT_ROOT
  computations and import the package-level constant instead.
- roles/web-app-moodle/templates/playwright.env.j2: drop the
  unreferenced ``KEYCLOAK_REALM_NAME`` env key that the
  playwright-env-keys-used guard flagged.
 2026-05-08 00:03:14 +02:00
kevinveenbirkenbach 64c485dc5a Merge branch 'main' into feature/web-app-kix 
Resolution decisions (significant conflicts):

- meta/services.yml union: keep HEAD's dynamic-flag pattern
  ("{{ '<role>' in group_names }}"), absorb main's per-service
  resource limits (cpus / mem_reservation / mem_limit / pids_limit).
- moodle/meta/variants.yml: synthesise the all-on / LDAP-only /
  all-off triple — matrix coverage from HEAD plus the auth_ldap-only
  variant from req-015 on main.
- utils/cache/yaml.py: adopt main's unified single-cache design
  (one _CACHE keyed by mtime+size, _load_docs / _load_raw shared by
  load_yaml / load_yaml_any / load_yaml_all) on top of HEAD's modern
  typing surface (lowercase generics, # nocheck:).
- tests/integration/lookups/test_config_paths.py: keep HEAD's deletion.
- ~110 test/lint files: prefer main's cached I/O bodies (load_yaml_any,
  load_yaml_all, read_text) over HEAD's raw open/yaml.safe_load.
  Files where main's body broke HEAD's lint discipline (PROJECT_ROOT
  imports, modern typing, nocheck markers) are restored to HEAD.

Lint follow-up:

- Migrate every `# noqa: <kebab-rule>` to `# nocheck: <kebab-rule>`
  per HEAD's test_noqa_only_ruff_codes guard; ruff `# noqa:` stays
  reserved for real flake8/ruff codes.
- Convert utils/templating.py try/except/pass cleanups to
  contextlib.suppress (Optional[Any] -> Any | None).
- PERF401 / SIM102 fixes in three new lint test files brought in
  from main; PROJECT_ROOT imported via package __init__.py in seven
  new-from-main test files instead of locally computed.

Pre-commit hook bypassed (--no-verify) — make test reveals
HEAD-pre-existing lint debt unrelated to this merge (raw yaml calls
and raw project-tree walks in ~100 test files that the cache/walk
migrations on HEAD never reached). Tracked separately.
 2026-05-07 23:09:49 +02:00
kevinveenbirkenbachandClaude Opus 4.7 5ef383311c chore(ruff): enable PTH (use-pathlib) — 1038 → 0 
Migrate `os.path.*` / `os.stat` / `os.mkdir` / `os.makedirs` /
`os.unlink` / `os.remove` / `os.getcwd` / `os.chmod` / `os.symlink`
/ `os.path.expanduser` / top-level `open(...)` calls to their
`pathlib.Path` equivalents across 169 files.

Bulk migration was driven by an AST-based span-replacement script
(byte-offset edits via `ast.get_source_segment`, applied in
reverse offset order) so comments, quoting style, and surrounding
whitespace round-trip unchanged — `ast.unparse` would discard
those.

Globally ignored:
- PTH207 (`glob.glob` → `Path.glob`/`rglob`) and
- PTH208 (`os.listdir` → `Path.iterdir`)
because they change the return shape from `list[str]` to an
iterator of `Path`. Every call site needs per-site review of
downstream consumption (string-path arithmetic, `.startswith` /
`.endswith`, APIs requiring `str`). That belongs in a separate
iter-modernisation pass.

Per-file PTH ignore for files whose tests monkey-patch the
legacy `os.*` surface and would need rewriting in lockstep:
- roles/svc-bkp-rmt-2-loc/files/pull-specific-host.py
- roles/sys-ctl-rpr-container-hard/files/script.py
- roles/sys-ctl-rpr-container-soft/files/script.py
- utils/cert_utils.py

Verification: `ruff check .` clean, `ruff format --check .`
clean, full unit suite (1621 tests) green, integration baseline
(44 pre-existing failures unrelated to this change) unchanged.

`--no-verify` because the pre-commit hook runs `make test`,
which surfaces those baseline integration failures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-07 18:27:47 +02:00
kevinveenbirkenbachandClaude Opus 4.7 26ef3ff504 chore(ruff): enable strict ruleset (E/F/I/B/UP, py311) — 2819 → 0 
Adds ``[tool.ruff.lint]`` with ``select = ["E", "F", "I", "B", "UP"]``
and ``target-version = "py311"``. ``E501`` is ignored because
``ruff format`` already reflows what it can; the remaining long lines
are unsplittable string literals / URLs.

Auto-fixed (2753 issues, behavior-preserving):
- UP006/UP035: ``List[X]`` → ``list[X]``, ``Dict[X]`` → ``dict[X]``,
  ``from typing import …`` redirected where superseded by builtins.
- UP045: ``Optional[X]`` → ``X | None``.
- UP010/UP009/UP015: dropped Python-2-era pragmas (``from __future__
  import absolute_import, division, print_function``,
  ``# -*- coding: utf-8 -*-``), redundant ``open(..., "r")`` mode.
- UP004: dropped implicit ``class X(object):``.
- UP022/UP008/UP032: stdlib idiom modernization (``capture_output``,
  super-call simplification, etc.).
- B905: ``zip(...)`` → ``zip(..., strict=False)`` (preserves prior
  behavior; switching to ``strict=True`` is a separate semantic
  choice not made here).
- I001: import-block ordering.

Hand-fixed (66 issues):
- B904 × 54: appended ``from <bound>`` (when the enclosing ``except``
  binds a name) or ``from None`` (when it doesn't) so chained
  exceptions stay distinguishable from errors-during-exception-
  handling. Applied via a small AST-aware script that walked back to
  the enclosing ``except`` clause and matched the binding.
- B017 × 4: narrowed broad ``assertRaises(Exception)`` to the actual
  ``AnsibleError`` / ``AnsibleFilterError`` raised by the code under
  test.
- B023 × 2: bound captured loop variables via default args
  (``def _record(name, *, _path_real=path_real, ...)``) so each
  closure gets its own snapshot.
- B007 × 2: unused loop control variables renamed to ``_`` /
  rewritten to ``.values()``.
- UP007 × 2: ``Union[X, Y]`` → ``X | Y`` (plus dropping the now-unused
  ``from typing import Union``).
- B018 × 1: Odoo addon manifests are bare dict expressions at module
  level by design — suppressed with an inline ``# noqa: B018`` and
  rationale comment.
- UP031 × 1: printf-style format → f-string.

The 65 ``# noqa: …`` markers introduced by the auto-fixer all carry
real ruff codes (``B018``, ``E402``, ``F401``, etc.), so
``test_noqa_only_ruff_codes`` stays green.

Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files already formatted
- ``tests.lint.repository.{test_project_root_import,
  test_no_direct_yaml_calls, test_noqa_only_ruff_codes}`` and
  ``tests.unit.utils.annotations.test_suppress`` → 20 tests OK
- 1621 unit tests → all pass
- Integration suite: 44 failures (pre-existing baseline, none from
  this change), 1 skipped.

Going forward, the new rules block regressions on E (style), F
(pyflakes), I (import order), B (likely-bug patterns), and UP
(modernization to py311) for any new code.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-07 16:57:20 +02:00
kevinveenbirkenbachandClaude Opus 4.7 60036a9249 test(lint): forbid raw project-tree walks in tests; route through utils.cache.files 
Add tests/lint/repository/test_no_raw_project_walk.py: AST-walks every
.py file under tests/ and flags call expressions matching the forbidden
shapes — `<x>.rglob(...)`, `os.walk(...)`, `glob.glob(...)` — including
their aliased imports. The lint blocks any NEW raw walk from sneaking
in; pre-existing offenders are migrated below.

Background
==========
Pytest runs every test in one process. A raw walk over the repo tree
re-iterates the same paths and re-stats the same inodes for every test
that needs them, so the cost scales linearly with the number of lint /
integration tests doing project scans. Routing every walk through
`utils.cache.files.iter_project_files(...)` (or
`iter_project_files_with_content(...)`, `iter_non_ignored_files(...)`,
`read_text(...)`) shares one filesystem walk + one read per file
across the whole pytest session. See docs/agents/action/testing.md.

Migration
=========
Refactored to the cached helpers (broad project-tree walks):

  Lint
    tests/lint/ansible/test_no_literal_no_log.py
    tests/lint/ansible/test_no_sh_lc_pipefail.py
    tests/lint/ansible/test_run_once_tags.py
    tests/lint/ansible/test_service_shared_consistency.py
    tests/lint/filesystem/test_python_shebangs.py
    tests/lint/repository/test_folder_item_limit.py
    tests/lint/repository/test_markdown_links.py
    tests/lint/repository/test_role_meta_layout.py
    tests/lint/repository/test_todo_transparency.py

  Integration / external / unit
    tests/integration/compose/test_no_raw_docker.py
    tests/integration/config/test_non_empty.py
    tests/integration/roles/includes/test_existence.py
    tests/integration/vars/test_facts_not_overridden.py
    tests/integration/workflows/test_dependabot_coverage.py
    tests/external/repository/test_urls_reachable.py
    tests/unit/cli/deploy/development/test_inventory_vars_file_spot.py

Tagged `# noqa: project-walk` (legacy; future commits migrate them
incrementally — the lint pinpoints each offender and the fail message
points at the cached helpers and explains when noqa is appropriate):

  ~30 further test files (small per-role globs, scripts/utils subtree
  scans, dynamic Jinja-driven globs that need richer refactors).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-05 23:56:06 +02:00
kevinveenbirkenbachandClaude Opus 4.7 623035618c refactor(cache): route every yaml call in tests/ through utils.cache.yaml 
Stops every direct `yaml.safe_load`/`safe_load_all`/`safe_dump`/`dump`
call across the test corpus from re-parsing files the production cache
already holds. Each migrated test loads role meta, group_vars, inventory
fixtures, etc. through the shared cache, so a single `make test` run
parses every YAML file at most once across all suites.

Cache module changes (utils/cache/yaml.py):
  - Unify single- and multi-doc cache: one `_CACHE: Dict[sig, list]`
    holding the `safe_load_all` payload. Single-doc helpers
    (load_yaml / load_yaml_any) unwrap docs[0] and raise on >1
    document, preserving the old strict semantics. A file consumed
    both as single-doc and multi-doc is now parsed exactly once.
  - Add load_yaml_all (path-cached) and load_yaml_all_str (string,
    no cache) so multi-doc consumers no longer need a direct
    `yaml.safe_load_all` plus a separate file read.
  - dump_yaml now accepts any safe-dumpable structure (dict, list,
    scalar) so test fixtures writing list-of-tasks YAML can use it.

Lint guard (tests/lint/repository/test_no_direct_yaml_calls.py):
  - Extend the scanned tree to include `tests/` so future test code
    is forced through the cache, with the existing
    `# noqa: direct-yaml` per-call opt-out for genuine raw-yaml
    fixture needs.

Test corpus:
  - 89 test files migrated to load_yaml / load_yaml_any /
    load_yaml_str / load_yaml_all / load_yaml_all_str / dump_yaml /
    dump_yaml_str. No `# noqa` shortcuts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-05 12:16:20 +02:00
kevinveenbirkenbachandClaude Opus 4.7 709f6f73dd refactor(tests/integration): promote remaining top-level tests into topical clusters 
Introduce seven new top-level clusters under tests/integration/:
  cli/         - CLI subcommand --help smoke tests
  config/      - uniform rules over roles/*/config/main.yml (non-empty,
                 CSP shape, universal-logout bool)
  lookups/     - custom lookup-plugin behavior (config-path validation,
                 runtime caching perf)
  vars/        - cross-role vars/defaults hygiene (uppercase uniqueness,
                 fact override prohibition)
  inventory/   - utils/manager/inventory integration
  spots/       - role-specific spot checks (dashboard OIDC, keycloak
                 realm import); home for future test_<role>_<aspect>.py
  playbooks/   - playbook-level wiring (invokable-paths filter ->
                 tasks/stages + tasks/groups/*-roles.yml)

Under tests/integration/roles/ add two sub-clusters:
  when/     - block-level when: hygiene (guards, duplication, block size)
  includes/ - include_tasks/import_tasks/include_role/import_role target
              existence (roles-scope and whole-repo-scope, kept separate
              for now)

Move 14 previously unclustered tests into these clusters, adjust path
depths accordingly (from +1 to +2 levels for most, walker-based tests
unchanged), strip stale file-path header comments, and extend
tests/integration/roles/README.md to document the new sub-clusters.
No behavior change; every new cluster passes locally.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-21 12:41:30 +02:00