mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-08 19:56:53 +00:00
kevinveenbirkenbachandClaude Opus 5
efb53ddf9c
The router already ranked routes on traits.speed, but nothing ever set
one. No catalogue carries a rate, because a rate is a property of this
machine under this load rather than of the model, and a hand-written
number in services.yml would be a guess that rots.
The deploy now measures it. After the gateway is warm, measure_speed.py
asks it which models it serves and sends router_speed_samples requests to
each model that has no rate yet, discarding the first as a cold load and
taking the median of the rest. The result is stored on the stack host and
rendered back into model_info.traits.speed on the same deploy, so the
gateway restarts once with the numbers it just produced. The stored file
is read back with slurp rather than lookup('file'), because the config
directory lives on the stack host and a lookup reads the controller.
A model that already carries a rate sends no request at all. Re-measuring
everything every deploy would produce results the tolerance discards anyway,
and that traffic is the entire live surface of this step: what can hang,
what can fail the deploy, and what the deploy waits for.
router_speed_remeasure discards the stored numbers once, for the case
where the hardware under them changed.
Two budgets bound a run. A measured request may take
router_speed_timeout; the discarded first request gets upstream_timeout
instead, because it pays the model load and holding it to the sample
budget would time out exactly the models worth measuring. Exceeding a
budget abandons that model, not the deploy, and the model keeps whatever
rate it had.
Every matrix variant overrides the sample count to 1, where the single
sample is kept rather than discarded: a CI round proves that the
measurement, the store and the re-render ran, not what the rate was on a
shared runner.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>