The package cache was configured in six places at once: the nginx upstream
map, the consumer compose override, pip.conf, .npmrc, the apt sources and the
twenty ensure_proxy_repo calls that bootstrap Nexus each spelled the same
hostnames, repository names and suites by hand. A new upstream meant editing
five files and hoping none was missed, and the docs told contributors to do
exactly that.
A role now declares what it needs under cache: in its meta/networks.yml, and
everything else is derived from it. svc-cache-package holds the general
upstreams; web-svc-libretranslate holds its own. All nine derived files are
generated and gitignored, so a second spelling of a hostname cannot exist.
utils.cache.hosts reads the declarations without PyYAML, because the .env
generator reaches them before any dependency is installed; it raises rather
than truncate when a comment sits at column 0 inside the block, which would
otherwise drop every declaration below it in silence. utils.cache.records
owns the Nexus record format and the Ubuntu mirror resolution, so the dev
stack and the role provision from one rendering.
A build may only resolve a cached host it can verify: the plain-HTTP hosts
are hijacked for every image, and the TLS hosts only for an image whose
Dockerfile installs the frontend CA. Hijacking all of them would fail every
pip and npm build with CERTIFICATE_VERIFY_FAILED.
svc-cache-package deploys the cache and its TLS frontend by calling the same
shell scripts the dev stack calls, rather than reimplementing them as tasks.
The role is invokable through the new svc-cache category and inert until a
host carries it.
Verified: the twenty Nexus payloads, the fifteen nginx server blocks and all
seven package-manager configs are byte-identical to the hand-written files
they replace, and the new category adds one role to the deploy order while
preserving the relative order of the other 180.
The role has not been deployed yet.
A debian:bookworm workspace job died on `apt-get update`: the mounted
sources list carried the Ubuntu entry too, Nexus had auto-blocked
archive.ubuntu.com, and apt fails the whole update on one bad source.
Three holes, three plugs:
- One sources list per INFINITO_DISTRO under compose/package-cache/apt/,
selected by the compose overrides. A Debian runner no longer fetches
an Ubuntu suite it has no use for. The non-apt distros get an empty
file so the bind mount has a source to point at.
- autoBlock off. While a remote is auto-blocked Nexus answers `404
Remote Auto Blocked` to every client instead of falling back to the
copy it already has cached.
- Mirror failover in the frontend. Every apt suite gets a second proxy
repo on an independent upstream; a 5xx from the primary re-runs the
request against it. Nexus has no group repository type for apt, so
the retry lives in nginx.
Measured on a live Nexus 3.91.1: dead remote without a cached copy
answers 502, with a cached copy 200 regardless of metadataMaxAge. The
frontend serves 149 kB of real bookworm `Release` from the mirror repo
when the primary is dead, and the new regex locations produce upstream
request lines byte-identical to the `rewrite … break` form they replace,
including `%3a` epochs in pool paths.
INFINITO_CACHE_PACKAGE_APT_LIST is gone; the mount interpolates
INFINITO_DISTRO directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Extend the local cache stack with three independent capabilities:
1. Inner-`dockerd` build coverage for HTTP-only package mirrors.
`roles/sys-svc-compose/files/compose.py` (the runner's `compose`
wrapper) auto-generates a transient `compose.cache.override.yml`
per app when `INFINITO_PACKAGE_CACHE_FRONTEND_IP` is exported. The
override pins `build.extra_hosts` for `deb.debian.org`,
`archive.ubuntu.com`, and `security.ubuntu.com` to the frontend.
`pull.py` now delegates to the wrapper so build/pull operations see
the same `-f` set the wrapper auto-detects. nginx in the frontend
gains a port-80 listener with path rewrites for those hostnames,
and `package.sh` registers two new Nexus proxies
(`apt-debian-security`, `apt-ubuntu-security`) for the security
suites whose upstream URL prefix differs from the main archive.
Alpine is intentionally excluded from the inner-build extra_hosts:
3.18+ defaults to HTTPS, and inner-`dockerd` build containers do
not import the frontend CA. Routing apk through the frontend would
trip TLS verification.
2. General cache freshness env-var. `scripts/meta/env/cache/package.sh`
exports `INFINITO_PACKAGE_CACHE_MAX_AGE_MIN` (default `129600`,
i.e. 90 days). The bootstrap helper consumes it for every Nexus
proxy's `contentMaxAge`, `metadataMaxAge`, and
`negativeCache.timeToLive`, replacing the previous 1440-minute
literal. Lower the value when chasing fast-moving upstreams.
3. SPOT cache documentation at `docs/contributing/environment/cache.md`.
`compose.yml.md`, `variables.md`, the per-service READMEs, and
`requirements/012-...` now link to it instead of restating the
stack's behaviour. Net documentation lines drop substantially while
the cache page becomes the single user-facing reference.
Also: code-comment cleanup per `docs/contributing/documentation.md`
guidelines across all touched .py / .sh / .yml files. Multi-paragraph
docstrings, banner separators, and restate-the-obvious comments are
removed; one-line WHY comments remain where the rationale is not
already visible in the code.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Brings up the package-cache compose service alongside the existing
registry-cache, gated by the same `cache` profile. Bootstrap helper
(scripts/docker/cache/package.sh) auto-rotates the admin password from
the well-known default, accepts the Sonatype CE EULA (3.71+ blocks
artifact downloads with HTTP 403 until /v1/system/eula is POSTed —
disclaimer must be sent verbatim with its unicode quotes), enables
anonymous read, creates one default blobstore, and registers the MVP
proxy set: apt-debian, apt-ubuntu, pypi-proxy, npm-proxy, helm-bitnami,
raw-githubusercontent. The runner gets pip.conf / .npmrc /
sources.list snippets bind-mounted in (each defaulting to /dev/null
when the cache profile is inactive) so the deploy is byte-identical
with cache off.
Cache plumbing:
- scripts/meta/env/cache/{registry,package}.sh: per-cache env defaults,
hostname-derived admin password seed (uses `hostname` command, not
$HOSTNAME shell var, so the value is identical across interactive
shells, non-interactive subshells, and Python subprocesses).
- scripts/docker/cache/{registry-ca,package}.sh: container-side
bootstrap helpers, both moved under scripts/docker/cache/.
- scripts/system/cache/clean.sh + `make cache-clean`: wipes both
/var/cache/infinito/core/cache/{registry,package} dirs and stops
the cache containers.
- cli/deploy/development/compose.py: Profile gate exports the three
new INFINITO_PACKAGE_CACHE_* client-snippet paths and runs the
bootstrap helper after the package-cache reports healthy.
Compose service uses sonatype/nexus3:3.91.1-ubi pinned, runs as
root (UBI image refuses to chown a fresh bind-mount), and forces
NEXUS_SECURITY_RANDOMPASSWORD=false so the bootstrap can log in with
admin/admin123 before rotating; the random-password mode otherwise
auto-completes onboarding silently and leaves no readable secret on
disk for an automated bootstrap to pick up.
Tests: tests/unit/cli/deploy/development/test_compose.py grows two
new assertions (cache services listed in `up` args when local; build
tests pin CI=true to retain ci-only assertions).
Closes req-012.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kevinveenbirkenbach
Alejandro Roman