Files
computer-playbook/compose/package-cache-frontend/upstreams.conf
T
kevinveenbirkenbachandClaude Opus 5 8d8701d9b3 feat(apt): one Ubuntu mirror list for the package cache and image builds, with failover, a short timeout and a health test
The ubuntu CI image build (job 103277644636) hung on Canonical's
archive: apt fetched 26 MB at 417 kB/s, then 61.6 kB in 29 s, then sat
on the package download until the job was cancelled after 23 minutes,
while download.docker.com delivered 42.8 MB in under a second on the
same runner. Locally archive.ubuntu.com and security.ubuntu.com took
20-26 s to the first byte against 2 s for a university mirror.

INFINITO_APT_UBUNTU_MIRRORS in default.env is now the one list of
Ubuntu mirrors, in failover order: de.archive.ubuntu.com, ftp.fau.de,
ftp.halifax.rwth-aachen.de, archive.ubuntu.com.

- Image builds: the Dockerfile sources the list alongside the install
  scripts and runs scripts/install/apt-mirrors.sh first. In an image
  with ubuntu.sources it points both stanzas at apt's mirror+file
  method, priorities in list order, and sets Acquire::http::Timeout to
  10 s, so apt moves on from a mirror that fails or goes silent; other
  distros are untouched. In ubuntu:latest apt-get update took 8 s
  against 110 s on the stock sources, a dead first mirror was left
  behind within 25 s against 62 s on apt's defaults, and the swarm node
  packages (80 debs) came in 17 s. The Dockerfile's own RUN lines ran
  clean on ubuntu, debian, fedora and arch.
- Package cache: package.sh proxies the first entry as the primary
  Ubuntu repos and the second as their fallback. Nexus revalidates
  metadata on every request here (metadataMaxAge 0), which costs a
  full download from a mirror without an ETag; both de.archive and
  ftp.fau.de send one, and the 15 MB universe index now comes back
  from the cache in 0.02 s.
- tests/external/repository/test_apt_ubuntu_mirrors.py keeps the list
  honest: the first mirror must answer InRelease within 5 s (median of
  three) or the test names a faster later one, no mirror's -updates may
  lag the freshest by more than 24 h, and every entry must be the
  primary archive, a Canonical country alias or an enabled official
  mirror in Launchpad's registry. It caught the first draft of the
  list: mirrors.edge.kernel.org is not an official Ubuntu mirror.

After pulling, an existing checkout needs `make dotenv-force`: plain
`make dotenv` re-reads the old .env through BASH_ENV and keeps its
values.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 17:13:54 +02:00

198 lines
6.0 KiB
Plaintext

proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Accept-Encoding "";
proxy_buffering on;
proxy_request_buffering off;
proxy_read_timeout 300s;
proxy_connect_timeout 30s;
# Docker embedded DNS, pinned to bypass image-specific resolv.conf order.
resolver 127.0.0.11 ipv6=off valid=30s;
# Nexus answers 502 when a proxy's remote is unreachable (measured) and has no
# group repository type for apt, so the mirror failover sits here rather than in
# the repo layout. 5xx only: a 404 means the file genuinely is not in the suite,
# and apt probes for missing files often enough that retrying those would double
# the request rate.
#
# Nexus answers that 502 only after trying every address of the remote in turn,
# one connect timeout each; for a remote with several addresses that outlasts a
# client's own timeout before the mirror is ever asked.
#
# These stay prefix locations. A regex location forfeits nginx's implicit
# `/debian` -> `/debian/` redirect, and the URL-reachability test probes exactly
# that slash-less form through the runner's DNS hijack, so it turns into a 404.
server {
listen 80;
server_name deb.debian.org;
location /debian/ {
rewrite ^/debian/(.*)$ /repository/apt-debian/$1 break;
proxy_intercept_errors on;
proxy_read_timeout 8s;
error_page 502 503 504 = @debian_mirror;
proxy_pass http://package-cache:8081;
}
location @debian_mirror {
rewrite ^/repository/apt-debian/(.*)$ /repository/apt-debian-mirror/$1 break;
proxy_pass http://package-cache:8081;
}
location /debian-security/ {
rewrite ^/debian-security/(.*)$ /repository/apt-debian-security/$1 break;
proxy_intercept_errors on;
proxy_read_timeout 8s;
error_page 502 503 504 = @debian_security_mirror;
proxy_pass http://package-cache:8081;
}
location @debian_security_mirror {
rewrite ^/repository/apt-debian-security/(.*)$ /repository/apt-debian-security-mirror/$1 break;
proxy_pass http://package-cache:8081;
}
}
server {
listen 80;
server_name archive.ubuntu.com;
location /ubuntu/ {
rewrite ^/ubuntu/(.*)$ /repository/apt-ubuntu/$1 break;
proxy_intercept_errors on;
proxy_read_timeout 8s;
error_page 502 503 504 = @ubuntu_mirror;
proxy_pass http://package-cache:8081;
}
location @ubuntu_mirror {
rewrite ^/repository/apt-ubuntu/(.*)$ /repository/apt-ubuntu-mirror/$1 break;
proxy_pass http://package-cache:8081;
}
}
server {
listen 80;
server_name security.ubuntu.com;
location /ubuntu/ {
rewrite ^/ubuntu/(.*)$ /repository/apt-ubuntu-security/$1 break;
proxy_intercept_errors on;
proxy_read_timeout 8s;
error_page 502 503 504 = @ubuntu_security_mirror;
proxy_pass http://package-cache:8081;
}
location @ubuntu_security_mirror {
rewrite ^/repository/apt-ubuntu-security/(.*)$ /repository/apt-ubuntu-security-mirror/$1 break;
proxy_pass http://package-cache:8081;
}
}
server {
listen 80;
server_name dl-cdn.alpinelinux.org;
location /alpine/ {
rewrite ^/alpine/(.*)$ /repository/raw-alpine/$1 break;
proxy_pass http://package-cache:8081;
}
}
server {
listen 443 ssl;
server_name pypi.org;
ssl_certificate /etc/nginx/certs/pypi.org.crt;
ssl_certificate_key /etc/nginx/certs/pypi.org.key;
location / {
proxy_pass http://package-cache:8081/repository/pypi-proxy/;
}
}
server {
listen 443 ssl;
server_name files.pythonhosted.org;
ssl_certificate /etc/nginx/certs/files.pythonhosted.org.crt;
ssl_certificate_key /etc/nginx/certs/files.pythonhosted.org.key;
location / {
proxy_pass http://package-cache:8081/repository/pypi-proxy/;
}
}
server {
listen 443 ssl;
server_name registry.npmjs.org;
ssl_certificate /etc/nginx/certs/registry.npmjs.org.crt;
ssl_certificate_key /etc/nginx/certs/registry.npmjs.org.key;
# Tarball URLs Nexus emits already carry /repository/npm-proxy/; pass them
# through as-is (no path on proxy_pass) so they aren't double-prefixed → 404.
location /repository/npm-proxy/ {
proxy_pass http://package-cache:8081;
}
location / {
proxy_pass http://package-cache:8081/repository/npm-proxy/;
}
}
server {
listen 443 ssl;
server_name rubygems.org;
ssl_certificate /etc/nginx/certs/rubygems.org.crt;
ssl_certificate_key /etc/nginx/certs/rubygems.org.key;
location / {
proxy_pass http://package-cache:8081/repository/gem-proxy/;
}
}
server {
listen 443 ssl;
server_name index.rubygems.org;
ssl_certificate /etc/nginx/certs/index.rubygems.org.crt;
ssl_certificate_key /etc/nginx/certs/index.rubygems.org.key;
location / {
proxy_pass http://package-cache:8081/repository/gem-proxy/;
}
}
server {
listen 443 ssl;
server_name repo.packagist.org;
ssl_certificate /etc/nginx/certs/repo.packagist.org.crt;
ssl_certificate_key /etc/nginx/certs/repo.packagist.org.key;
location / {
proxy_pass http://package-cache:8081/repository/raw-packagist/;
}
}
server {
listen 443 ssl;
server_name proxy.golang.org;
ssl_certificate /etc/nginx/certs/proxy.golang.org.crt;
ssl_certificate_key /etc/nginx/certs/proxy.golang.org.key;
location / {
proxy_pass http://package-cache:8081/repository/go-proxy/;
}
}
server {
listen 443 ssl;
server_name dl-cdn.alpinelinux.org;
ssl_certificate /etc/nginx/certs/dl-cdn.alpinelinux.org.crt;
ssl_certificate_key /etc/nginx/certs/dl-cdn.alpinelinux.org.key;
location / {
proxy_pass http://package-cache:8081/repository/raw-alpine/;
}
}
server {
listen 443 ssl;
server_name raw.githubusercontent.com;
ssl_certificate /etc/nginx/certs/raw.githubusercontent.com.crt;
ssl_certificate_key /etc/nginx/certs/raw.githubusercontent.com.key;
location / {
proxy_pass http://package-cache:8081/repository/raw-githubusercontent/;
}
}
server {
listen 443 ssl;
server_name codeload.github.com;
ssl_certificate /etc/nginx/certs/codeload.github.com.crt;
ssl_certificate_key /etc/nginx/certs/codeload.github.com.key;
location / {
proxy_pass http://package-cache:8081/repository/raw-codeload-github/;
}
}