mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-10 04:36:40 +00:00
kevinveenbirkenbachandClaude Opus 5
54de79c0a4
Both docstrings asserted that huggingface.co throttled the prober. It did not: the prober never reached it. The test container pins huggingface.co in /etc/hosts to the local package cache, and the 404 came from our own Nexus rejecting an asset it could not type. _host_gate now names only the one measured case, archlinux.org refusing connections under 22 parallel package probes, and says plainly that a host belongs there only with a reproduction showing the host itself refusing. test_urls_reachable gains what is actually true of it: for a pinned host the suite reports the cache's verdict rather than the upstream's, in both directions, and test_apt_ubuntu_mirrors._pinned_hosts is the existing way to tell which hosts those are. The gate stays. The package probes need it; only half its rationale was invented. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
42 lines
1.6 KiB
Python
42 lines
1.6 KiB
Python
"""One concurrency gate per host, shared by the probe suites in this package.
|
|
|
|
Both suites fan a thread pool out over many URLs at once, and one host has
|
|
been measured refusing connections under that load while answering each
|
|
request fine on its own: archlinux.org, under 22 parallel package probes.
|
|
The gate bounds how many requests may be in flight against one hostname
|
|
and leaves total parallelism alone.
|
|
|
|
That measurement is the only one. An earlier version of this file also
|
|
named huggingface.co, which was wrong: those requests never left the
|
|
runner, because `huggingface.co` is pinned in the test container's
|
|
`/etc/hosts` to the local package cache. Do not add a host here without a
|
|
reproduction that shows the host itself refusing.
|
|
|
|
The limit belongs to the caller, not to this module: a package index
|
|
tolerates less than a general web host, and the URL prober opens a fresh
|
|
connection per request where a browser would reuse a kept-alive one. Gates
|
|
are keyed by host *and* limit, so one suite's budget cannot silently
|
|
become another's.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import threading
|
|
import urllib.parse
|
|
|
|
_gates: dict[tuple[str, int], threading.Semaphore] = {}
|
|
|
|
|
|
def host_gate(url: str, limit: int) -> threading.Semaphore:
|
|
"""Return the gate shared by every URL on *url*'s host.
|
|
|
|
Args:
|
|
url: absolute URL that is about to be requested.
|
|
limit: how many requests this suite may hold in flight per host.
|
|
|
|
Returns:
|
|
A semaphore to hold for the duration of the request.
|
|
"""
|
|
host = (urllib.parse.urlsplit(url).hostname or "").lower()
|
|
return _gates.setdefault((host, limit), threading.Semaphore(limit))
|