0 Fork 0

mirror of https://github.com/kevinveenbirkenbach/computer-playbook.git synced 2026-11-12 05:36:40 +00:00

Files

T

Environment TestandClaude Sonnet 4.6 8d6c01fa40 test(external): fail on client/server errors instead of warning for dead links

Change the URL reachability test to fail on 4xx, non-500 5xx, and
request errors (DNS, connection). Only HTTP 500 now emits a warning
annotation, distinguishing temporary upstream breakage from dead links.
Add a unit test for the probe classification logic.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-04-11 20:34:38 +00:00

3.0 KiB

Raw Blame History

External Tests 🌐

This page is the SPOT for tests that depend on live third-party systems such as Docker Hub, GHCR, or other external services. Run external tests with make test-external.

Goal 🎯

You MUST use external tests only when the behavior depends on a live third-party system and a hermetic unit, lint, or integration test would not verify the real condition.
You MUST keep external tests outside the default make test flow so normal validation stays reproducible without third-party network access.
You SHOULD keep external tests focused, advisory, and easy to run on demand.

Location 📁

External tests MUST live under tests/external/.
You SHOULD group them by dependency family such as tests/external/docker/ or tests/external/github/ when more than one test exists.
Every external test directory MUST contain __init__.py so repository-wide test structure checks keep passing.

Requirements 📋

You MUST document in the test file why the check is external and what live dependency it contacts.
You MUST NOT move an external test into tests/lint/, tests/unit/, or tests/integration/ just for convenience when it still depends on live third-party state.
External tests MAY emit warnings instead of hard failures when the goal is advisory freshness reporting rather than merge-blocking correctness.
You SHOULD keep network traffic narrow and query each external dependency only as much as needed for a meaningful result.
test_urls_reachable.py scans git-tracked plus untracked-but-not-ignored text files for literal public http:// and https:// URLs.
The URL reachability check skips template placeholders and reserved local or example hosts such as localhost and *.example.
The URL reachability check fails on 4xx, non-500 5xx, and request failures such as DNS or connection errors.
The URL reachability check emits a warning annotation for HTTP 500 via utils.annotations.message.

Suppression Comments 🚫

Some external checks support local inline suppressions for intentional exceptions. You MUST use these only when the check genuinely does not apply.

Comment	Placement	Affected test	Effect
`# nocheck: docker-version`	Line directly above `version:` in `config/main.yml`	test_image_versions.py	Skips the live version-update warning for that image

Running 🏃

Use the dedicated make target:

make test-external

To scope the run to one file:

TEST_PATTERN=test_image_versions.py make test-external

CI Workflow 🤖

The dedicated workflow lives at .github/workflows/test-code-external.yml. It MAY be triggered manually and is intentionally separate from the default make test flow. It is also wired into the orchestrated code-testing gate in CI so the live check can run there without changing local default validation.

3.0 KiB Raw Blame History