mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-09 04:06:42 +00:00
kevinveenbirkenbachandClaude Opus 5
b005151a4f
`tests/semantic/` mixed three kinds of check under one name. They are now placed by what returns the verdict: - `tests/oracle/` holds only the checks a model decides, and `make test-semantic` becomes `make test-oracle` - `tests/external/i18n/` holds the back-translation check, which needs a served engine - `tests/integration/i18n/` holds the catalog scan, which reads the `.po` corpus and needs nothing served The shared reachability helpers move to `tests/utils/services.py`. Two checks were reading `auto_comments` while `utils/i18n/translate.py` writes the marker to `user_comments`. Over the `core` corpus that is 0 matches against 19,225, so both selected an empty set and passed having judged nothing. The back-translation check also addressed the engine by its pre-rename service key. The model-pin check treated a 200 that carries no file digests as proof the pinned file was withdrawn. An intermediary that answers with the file list but drops the `blobs` expansion produces exactly that, so a good pin went red; the comparison has no input in that case and the pin is now reported unverified. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
26 lines
1.2 KiB
Python
26 lines
1.2 KiB
Python
"""Checks whose verdict comes from System One rather than from an assertion.
|
|
|
|
A test oracle is the mechanism that decides pass or fail. Everywhere else in
|
|
this repository that mechanism is deterministic: a value equals another, a
|
|
file exists, a regex matches. Here it is a model, which makes the verdict
|
|
probabilistic and its calibration part of the test rather than a constant.
|
|
|
|
Two rules follow from that and every check in this suite keeps them:
|
|
|
|
* Read the answer as an ordering, never as an absolute. The server does not
|
|
centre its probabilities on 0.5, so a fixed threshold would encode one
|
|
deployment's calibration. Carry controls whose correct answers are known
|
|
and opposite, and judge a candidate against them.
|
|
* Declare the run inconclusive when the controls fail to separate. A model
|
|
that is not reading its input must not be allowed to return green.
|
|
|
|
The suite is not part of ``make test``: ``scripts/tests/code/run.sh`` picks
|
|
its directory from ``INFINITO_TEST_TYPE``, and only ``make test-oracle`` sets
|
|
it. It needs the tools lane, which that target brings up first.
|
|
"""
|
|
|
|
from pathlib import Path
|
|
|
|
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
|
SUITE = "oracle"
|