The 16.2.0 release commit (e2501cb1de) shipped a locally generated
`compose/package-cache-frontend/upstreams.conf`, the output of the cache
generator that only exists on the feature branches (230d9a1e5e). It adds
TLS server blocks for argos-net.com, huggingface.co, index.crates.io,
registry.ollama.ai and static.crates.io, but main's `cert-gen.sh` issues
no certificates for them. nginx then dies on start with
cannot load certificate "/etc/nginx/certs/argos-net.com.crt"
the frontend stays unhealthy, pypi.org on the runner refuses connections,
and `make install-lint` fails in every workspace job (run 39482585010).
Restore the file byte for byte to its state before the release (blob
b5c5ef08e7), the version every workspace job passed with in run
39443109355.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The ubuntu CI image build (job 103277644636) hung on Canonical's
archive: apt fetched 26 MB at 417 kB/s, then 61.6 kB in 29 s, then sat
on the package download until the job was cancelled after 23 minutes,
while download.docker.com delivered 42.8 MB in under a second on the
same runner. Locally archive.ubuntu.com and security.ubuntu.com took
20-26 s to the first byte against 2 s for a university mirror.
INFINITO_APT_UBUNTU_MIRRORS in default.env is now the one list of
Ubuntu mirrors, in failover order: de.archive.ubuntu.com, ftp.fau.de,
ftp.halifax.rwth-aachen.de, archive.ubuntu.com.
- Image builds: the Dockerfile sources the list alongside the install
scripts and runs scripts/install/apt-mirrors.sh first. In an image
with ubuntu.sources it points both stanzas at apt's mirror+file
method, priorities in list order, and sets Acquire::http::Timeout to
10 s, so apt moves on from a mirror that fails or goes silent; other
distros are untouched. In ubuntu:latest apt-get update took 8 s
against 110 s on the stock sources, a dead first mirror was left
behind within 25 s against 62 s on apt's defaults, and the swarm node
packages (80 debs) came in 17 s. The Dockerfile's own RUN lines ran
clean on ubuntu, debian, fedora and arch.
- Package cache: package.sh proxies the first entry as the primary
Ubuntu repos and the second as their fallback. Nexus revalidates
metadata on every request here (metadataMaxAge 0), which costs a
full download from a mirror without an ETag; both de.archive and
ftp.fau.de send one, and the 15 MB universe index now comes back
from the cache in 0.02 s.
- tests/external/repository/test_apt_ubuntu_mirrors.py keeps the list
honest: the first mirror must answer InRelease within 5 s (median of
three) or the test names a faster later one, no mirror's -updates may
lag the freshest by more than 24 h, and every entry must be the
primary archive, a Canonical country alias or an enabled official
mirror in Launchpad's registry. It caught the first draft of the
list: mirrors.edge.kernel.org is not an official Ubuntu mirror.
After pulling, an existing checkout needs `make dotenv-force`: plain
`make dotenv` re-reads the old .env through BASH_ENV and keeps its
values.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_packages_available failed locally in every quality pass with
"91 package(s) could not be checked at all ... archive.ubuntu.com:
timed out". The dev stack routes archive.ubuntu.com to the package-cache
frontend, which re-runs a request against the -mirror repo only once
the primary Nexus repo answers 502/503/504. From this host's docker
network Canonical's IPv4 addresses take seconds to connect and then
send nothing, and Nexus tries all nine addresses of archive.ubuntu.com
in turn before it gives up: apt-ubuntu answered 502 after 21-25 s, so
the frontend's first byte arrived after 27-46 s, past the probe's 20 s
timeout. apt-ubuntu-mirror (de.archive.ubuntu.com) served the same
files in 0.08 s.
The four primary locations now carry proxy_read_timeout 8s: nginx
gives up on Nexus itself, its own 504 hits the existing error_page, and
the mirror answers. A shorter Nexus upstream timeout was tried first and
dropped, because Nexus spends it once per address and still took
9-21 s. The mirror locations keep the global 300 s, and a healthy
primary answers well inside the bound, as before.
Measured after a compose restart: the ubuntu index arrives after
8-12.6 s with Origin: Ubuntu, debian still in 0.03-0.16 s, npm and PyPI
in 0.13 s; make test-external passed 13 of 13 and the full
make quality-high went green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mirror failover turned the four apt locations into regex locations.
nginx only synthesises the `/debian` -> `/debian/` redirect for prefix
locations, so the slash-less form started answering 404. Inside a runner
`deb.debian.org` is DNS-hijacked to this frontend and the URL
reachability test probes exactly that form, which took the debian,
fedora and centos workspace jobs down.
The locations are prefix locations again, byte for byte as before. The
only thing that stays is the error path: the named location picks up the
already-rewritten `$uri` and rewrites it once more onto the mirror repo,
so the failover survives without a capture variable.
Measured against a simulated hijack: both bare paths 200 again, all four
suite indexes 200, and with `apt-debian` pointed at a dead remote
`dists/bookworm/InRelease` still arrives with 151 kB through
apt-debian-mirror.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A debian:bookworm workspace job died on `apt-get update`: the mounted
sources list carried the Ubuntu entry too, Nexus had auto-blocked
archive.ubuntu.com, and apt fails the whole update on one bad source.
Three holes, three plugs:
- One sources list per INFINITO_DISTRO under compose/package-cache/apt/,
selected by the compose overrides. A Debian runner no longer fetches
an Ubuntu suite it has no use for. The non-apt distros get an empty
file so the bind mount has a source to point at.
- autoBlock off. While a remote is auto-blocked Nexus answers `404
Remote Auto Blocked` to every client instead of falling back to the
copy it already has cached.
- Mirror failover in the frontend. Every apt suite gets a second proxy
repo on an independent upstream; a 5xx from the primary re-runs the
request against it. Nexus has no group repository type for apt, so
the retry lives in nginx.
Measured on a live Nexus 3.91.1: dead remote without a cached copy
answers 502, with a cached copy 200 regardless of metadataMaxAge. The
frontend serves 149 kB of real bookworm `Release` from the mirror repo
when the primary is dead, and the new regex locations produce upstream
request lines byte-identical to the `rewrite … break` form they replace,
including `%3a` epochs in pool paths.
INFINITO_CACHE_PACKAGE_APT_LIST is gone; the mount interpolates
INFINITO_DISTRO directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OpenSSL 3.x rejects a trust-anchor CA without keyUsage, but cert-gen.sh only regenerated on missing/expiring certs, so a CA created before the keyUsage addext was added (10y validity) never refreshed and broke all MITM-proxied HTTPS verification in the dev cache. Add a keyUsage guard so a deficient CA is regenerated even while still time-valid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`openssl req -x509` without explicit `-addext` produced a CA cert
without an X.509v3 keyUsage extension. Python 3.13's stricter ssl
module refuses to load the system trust bundle if any anchor lacks
keyUsage, so every outbound HTTPS — including hits against the
cache frontend that proxies codeload.github.com — fails with
`CA cert does not include key usage extension`. mediawiki's
OpenIDConnect extension download failed on this in CI.
Add `keyUsage=critical,keyCertSign,cRLSign`,
`basicConstraints=critical,CA:TRUE`, and `subjectKeyIdentifier=hash`
to the CA self-sign so every freshly generated CA is acceptable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extend the local cache stack with three independent capabilities:
1. Inner-`dockerd` build coverage for HTTP-only package mirrors.
`roles/sys-svc-compose/files/compose.py` (the runner's `compose`
wrapper) auto-generates a transient `compose.cache.override.yml`
per app when `INFINITO_PACKAGE_CACHE_FRONTEND_IP` is exported. The
override pins `build.extra_hosts` for `deb.debian.org`,
`archive.ubuntu.com`, and `security.ubuntu.com` to the frontend.
`pull.py` now delegates to the wrapper so build/pull operations see
the same `-f` set the wrapper auto-detects. nginx in the frontend
gains a port-80 listener with path rewrites for those hostnames,
and `package.sh` registers two new Nexus proxies
(`apt-debian-security`, `apt-ubuntu-security`) for the security
suites whose upstream URL prefix differs from the main archive.
Alpine is intentionally excluded from the inner-build extra_hosts:
3.18+ defaults to HTTPS, and inner-`dockerd` build containers do
not import the frontend CA. Routing apk through the frontend would
trip TLS verification.
2. General cache freshness env-var. `scripts/meta/env/cache/package.sh`
exports `INFINITO_PACKAGE_CACHE_MAX_AGE_MIN` (default `129600`,
i.e. 90 days). The bootstrap helper consumes it for every Nexus
proxy's `contentMaxAge`, `metadataMaxAge`, and
`negativeCache.timeToLive`, replacing the previous 1440-minute
literal. Lower the value when chasing fast-moving upstreams.
3. SPOT cache documentation at `docs/contributing/environment/cache.md`.
`compose.yml.md`, `variables.md`, the per-service READMEs, and
`requirements/012-...` now link to it instead of restating the
stack's behaviour. Net documentation lines drop substantially while
the cache page becomes the single user-facing reference.
Also: code-comment cleanup per `docs/contributing/documentation.md`
guidelines across all touched .py / .sh / .yml files. Multi-paragraph
docstrings, banner separators, and restate-the-obvious comments are
removed; one-line WHY comments remain where the rationale is not
already visible in the code.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move the `docker compose -f ...` flag list and the cache-override
env-var setup into single helpers in `cli/deploy/development/common.py`:
* `compose_file_args()` — decides whether to layer
`compose/cache.override.yml` on top of `compose.yml`.
* `cache_env_overrides()` — produces the env-var dict the
override file requires strictly via `${VAR:?…}`.
Both `cli.deploy.development.compose` (up flow) and
`cli.deploy.development.down` (down flow) now call these helpers, so
up and down resolve to the same compose-file set and the same env
vars. Previously the cache env-var setup was duplicated inline in
compose.py and missing entirely from down.py, which made `make down`
fail under the `cache` profile with a confusing "required variable
missing" error before even reaching the orphan-cleanup step.
Single SPOT discipline: the host CA directory default lives only in
`scripts/meta/env/cache/package.sh` (sourced via BASH_ENV by every
Makefile recipe). `cache_env_overrides()` refuses to fall back to a
duplicate literal — a missing env var raises SystemExit pointing at
the env script. Tests that bypass the env layer set the var
explicitly so the SPOT-strict guard does not raise.
Containerize cert-gen: the original
`scripts/docker/cache/package-frontend-certs.sh` ran `mkdir`/`openssl`
host-side, which fails when `/var/cache/infinito` is not writable from
the invoking shell (sandboxed tooling, non-root operators). Move the
openssl logic into `compose/package-cache-frontend/cert-gen.sh` and
have the host wrapper docker-run it inside a throw-away alpine
container so the docker daemon (root) creates the bind-mount source
paths. Same pattern as how rpardini's registry-cache generates its CA
in-container.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a TLS-terminating reverse proxy in front of the Sonatype Nexus 3 OSS
pull-through cache so the `infinito` runner can hit upstream
package-manager URLs (pypi.org, registry.npmjs.org, rubygems.org,
repo.packagist.org, proxy.golang.org, dl-cdn.alpinelinux.org,
files.pythonhosted.org, raw.githubusercontent.com, codeload.github.com)
unchanged and still flow through Nexus.
Mechanism:
* scripts/docker/cache/package-frontend-certs.sh — host-side helper,
idempotently generates a dedicated CA + per-hostname leaf certs.
* compose/package-cache-frontend/upstreams.conf — nginx server-blocks
that match SNI per hostname and reverse-proxy onto the matching
Nexus repo path.
* compose/cache.override.yml `infinito.extra_hosts` — DNS-hijack so
the runner's clients resolve each upstream onto the frontend's
static IP.
* scripts/docker/cache/package-frontend-ca.sh — runner-side helper,
drops the CA into the system trust store post-startup so apt/pip/
curl/etc. accept TLS against the hijacked hostnames.
* cli/deploy/development/compose.py — orchestrates cert gen before
compose-up, CA install after health.
Scope today: runner-only. App-service Dockerfile builds executed by
the inner dockerd are NOT routed through the frontend; that requires
build-time CA injection and is deferred.
Also extends scripts/docker/cache/package.sh with five additional
Nexus proxies registered for transparent use: gem-proxy, go-proxy,
yum-rocky, yum-fedora, raw-packagist, raw-alpine, raw-codeload-github.
Refactor: move all cache services + runner-side cache mounts/depends_on/
extra_hosts from compose.yml into compose/cache.override.yml. Base
compose.yml now describes only the bare CI/runtime stack (coredns +
infinito). The override file is the single gate for the cache stack;
`profiles: ["cache"]` attributes and `--profile cache` flag emission
were redundant once file inclusion controls existence and removed
accordingly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
