Files
computer-playbook/docs/contributing/actions/testing/external.md
T
kevinveenbirkenbachandClaude Opus 4.7 9a578952c3 feat(env): SPOT env-pipeline + INFINITO_* namespace
Replace 8 bash env modules with a single Python generator
(cli.meta.env) that materialises env/static.env + runtime context
into a gitignored .env. Each variable lives in its own handler
under utils/env/handlers/; defaults carry inline one-line `# ...`
comments.

Rename to the INFINITO_* schema:
- workflow inputs: APPS, SERVICES_DISABLED, CMD, DEBUG, WHITELIST,
  BUNDLES, FULL_CYCLE, VARIANT, RUN_FLAGS, LIMIT_HOST, PARALLEL
- inventory: INVENTORY_DIR / INVENTORY_FILE / HOST_VARS_FILE,
  PW_FILE -> INVENTORY_PASSWORD_FILE
- venv: VENV_BASE / VENV_FALLBACK / VENV_NAME
- new SPOT defaults: COMPILE_SILENCE, CONTAINER_NO_CA,
  WAIT_HEALTH_TIMEOUT_S, PACKAGE_CACHE_FRONTEND_INIT_IMAGE,
  promoted IMAGE_TAG from GHA-only to general passthrough

compose.yml hardening:
- inline \${VAR:-default} removed; strict \${VAR:?...} everywhere
- new integration lint blocks regressions

Tooling:
- scripts/install/all.sh owns the stamp-file install logic
  (Makefile target delegates)
- scripts/meta/env/load.sh is the single shell entry point;
  8 bash modules deleted
- new shell lint: no INFINITO_* defaults outside env/static.env

Tests:
- 93 new unit tests for utils/env/{builder,parser,runtime,writer}
- restructure tests/lint into repository/ (file-content rules) vs
  filesystem/{shell,python,env}/ (filesystem-shape rules)
- drift + smoke integration tests against the generator

Docs:
- variables.md is now a pointer; per-variable docs live next to
  the key in env/static.env / env/ci.env
- compose.yml.md drops the Default column

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 20:40:34 +02:00

3.0 KiB

External Tests 🌐

External tests depend on live third-party systems such as Docker Hub, GHCR, or other external services. Run them with make test-external.

Goal 🎯

  • You MUST use external tests only when the behavior depends on a live third-party system and a hermetic unit, lint, or integration test would not verify the real condition.
  • You MUST keep external tests outside the default make test flow so normal validation stays reproducible without third-party network access.
  • You SHOULD keep external tests focused, advisory, and easy to run on demand.

Location 📁

  • External tests MUST live under tests/external/.
  • You SHOULD group them by dependency family such as tests/external/update/ or tests/external/github/ when more than one test exists.
  • Every external test directory MUST contain __init__.py so repository-wide test structure checks keep passing.

Requirements 📋

  • You MUST document in the test file why the check is external and what live dependency it contacts.
  • You MUST NOT move an external test into tests/lint/, tests/unit/, or tests/integration/ just for convenience when it still depends on live third-party state.
  • External tests MAY emit warnings instead of hard failures when the goal is advisory freshness reporting rather than merge-blocking correctness.
  • You SHOULD keep network traffic narrow and query each external dependency only as much as needed for a meaningful result.
  • test_urls_reachable.py scans git-tracked plus untracked-but-not-ignored text files for literal public http:// and https:// URLs.
  • The URL reachability check skips template placeholders and reserved local or example hosts such as localhost and *.example.
  • The URL reachability check treats HTTP 401, 403, and 405 as reachable (auth-gated or method-restricted servers, not dead links).
  • The URL reachability check emits warning annotations for HTTP 429, 500, and 503 (rate-limited or temporarily unavailable) via utils.annotations.message.
  • The URL reachability check treats read and connect timeouts as warnings.
  • All other 4xx/5xx codes and connection errors (DNS, SSL) fail the test.

Suppression Comments 🚫

Lint and external checks share a single per-item suppression syntax based on # noqa / # nocheck markers. The full grammar, position semantics, and the catalog of rule keys live at suppression.md.

Running 🏃

Use the dedicated make target:

make test-external

To scope the run to one file:

make test-external INFINITO_TEST_PATTERN=test_image_versions.py

CI Workflow 🤖

The dedicated workflow lives at .github/workflows/test-code-external.yml. It MAY be triggered manually and is intentionally separate from the default make test flow. It is also wired into the orchestrated code-testing gate in CI so the live check can run there without changing local default validation.