0 Fork 0

mirror of https://github.com/kevinveenbirkenbach/computer-playbook.git synced 2026-11-08 11:46:35 +00:00

Files

T

kevinveenbirkenbach 837d0c5335 test(suppression): unify # noqa / # nocheck marker grammar

Introduce a single grammar for per-line, per-block, and per-file
suppression markers across the test suite, replacing seven ad-hoc
syntaxes that had drifted apart.

Helper
  utils/annotations/suppress.py exposes is_suppressed_at,
  is_suppressed_in_head, is_suppressed_anywhere, line_has_rule, and
  suppressed_line_numbers. Markers may use either `noqa` or `nocheck`
  (case-insensitive synonyms), live inside `#`, `//`, `{# ... #}`, or
  `<\!-- ... -->` comments, and combine multiple kebab-case rules
  comma-separated. 17 unit tests cover the grammar in isolation.

Refactored consumers
  - tests/external/repository/test_urls_reachable.py (rule: url)
  - tests/external/docker/test_image_versions.py via
    utils/docker/version_updater.py (rule: docker-version)
  - tests/lint/repository/test_no_direct_yaml_calls.py (rule:
    direct-yaml)
  - tests/lint/ansible/test_service_shared_consistency.py (rule:
    shared)
  - tests/lint/ansible/roles/test_web_app_email_integration.py (rule:
    email)
  - tests/lint/repository/test_python_file_size.py (rule: file-size)
  - tests/lint/ansible/test_run_once_tags.py +
    tests/integration/roles/run_once/test_schema.py (rules: run-once,
    run-once-suffix)

Marker migration
  12 role tasks/main.yml files: `# run_once_<key>: deactivated`
  -> `# nocheck: run-once`. roles/web-app-suitecrm/meta/services.yml
  drops a malformed inline `noqa: intentional legacy pin` from a TODO
  comment so the new parser does not pick it up as a stray rule.

Caching wins
  Test files that previously loaded YAML / read files via raw
  yaml.safe_load + Path.read_text now route through utils.cache.yaml
  (load_yaml_any / load_yaml_str) and utils.cache.files.read_text:
    - test_service_shared_consistency.py
    - test_web_app_email_integration.py
    - tests/integration/roles/run_once/test_schema.py
    - utils/docker/version_updater.py
  This both speeds up repeated reads and stays consistent with the
  direct-yaml lint policy.

Documentation
  New SPOT page at docs/contributing/actions/testing/suppression.md
  with grammar, position semantics, the catalog of rules, examples,
  and an "adding a new rule" checklist. lint.md and external.md drop
  their per-rule tables and link to the new page. All touched docs
  conform to docs/contributing/documentation.md (no em-dash thought
  breaks, no SPOT self-labels, RFC-2119 keywords used consistently).

Settings sort tooling
  scripts/lint/sort_claude_settings.sh wraps utils/claude/settings_sort.py
  and is wired into scripts/lint/autoformat.sh. The tooling fixes the
  ASCII-ascending requirement enforced by
  tests/lint/repository/test_claude_settings_sorted.py; the test's
  failure message now points operators at the script. .claude/settings.json
  is committed in sorted form.

2026-04-30 11:54:46 +02:00

3.0 KiB

Raw Blame History

External Tests 🌐

External tests depend on live third-party systems such as Docker Hub, GHCR, or other external services. Run them with make test-external.

Goal 🎯

You MUST use external tests only when the behavior depends on a live third-party system and a hermetic unit, lint, or integration test would not verify the real condition.
You MUST keep external tests outside the default make test flow so normal validation stays reproducible without third-party network access.
You SHOULD keep external tests focused, advisory, and easy to run on demand.

Location 📁

External tests MUST live under tests/external/.
You SHOULD group them by dependency family such as tests/external/docker/ or tests/external/github/ when more than one test exists.
Every external test directory MUST contain __init__.py so repository-wide test structure checks keep passing.

Requirements 📋

You MUST document in the test file why the check is external and what live dependency it contacts.
You MUST NOT move an external test into tests/lint/, tests/unit/, or tests/integration/ just for convenience when it still depends on live third-party state.
External tests MAY emit warnings instead of hard failures when the goal is advisory freshness reporting rather than merge-blocking correctness.
You SHOULD keep network traffic narrow and query each external dependency only as much as needed for a meaningful result.
test_urls_reachable.py scans git-tracked plus untracked-but-not-ignored text files for literal public http:// and https:// URLs.
The URL reachability check skips template placeholders and reserved local or example hosts such as localhost and *.example.
The URL reachability check treats HTTP 401, 403, and 405 as reachable (auth-gated or method-restricted servers, not dead links).
The URL reachability check emits warning annotations for HTTP 429, 500, and 503 (rate-limited or temporarily unavailable) via utils.annotations.message.
The URL reachability check treats read and connect timeouts as warnings.
All other 4xx/5xx codes and connection errors (DNS, SSL) fail the test.

Suppression Comments 🚫

Lint and external checks share a single per-item suppression syntax based on # noqa / # nocheck markers. The full grammar, position semantics, and the catalog of rule keys live at suppression.md.

Running 🏃

Use the dedicated make target:

make test-external

To scope the run to one file:

make test-external TEST_PATTERN=test_image_versions.py

CI Workflow 🤖

The dedicated workflow lives at .github/workflows/test-code-external.yml. It MAY be triggered manually and is intentionally separate from the default make test flow. It is also wired into the orchestrated code-testing gate in CI so the live check can run there without changing local default validation.

3.0 KiB Raw Blame History