Extend the local cache stack with three independent capabilities:
1. Inner-`dockerd` build coverage for HTTP-only package mirrors.
`roles/sys-svc-compose/files/compose.py` (the runner's `compose`
wrapper) auto-generates a transient `compose.cache.override.yml`
per app when `INFINITO_PACKAGE_CACHE_FRONTEND_IP` is exported. The
override pins `build.extra_hosts` for `deb.debian.org`,
`archive.ubuntu.com`, and `security.ubuntu.com` to the frontend.
`pull.py` now delegates to the wrapper so build/pull operations see
the same `-f` set the wrapper auto-detects. nginx in the frontend
gains a port-80 listener with path rewrites for those hostnames,
and `package.sh` registers two new Nexus proxies
(`apt-debian-security`, `apt-ubuntu-security`) for the security
suites whose upstream URL prefix differs from the main archive.
Alpine is intentionally excluded from the inner-build extra_hosts:
3.18+ defaults to HTTPS, and inner-`dockerd` build containers do
not import the frontend CA. Routing apk through the frontend would
trip TLS verification.
2. General cache freshness env-var. `scripts/meta/env/cache/package.sh`
exports `INFINITO_PACKAGE_CACHE_MAX_AGE_MIN` (default `129600`,
i.e. 90 days). The bootstrap helper consumes it for every Nexus
proxy's `contentMaxAge`, `metadataMaxAge`, and
`negativeCache.timeToLive`, replacing the previous 1440-minute
literal. Lower the value when chasing fast-moving upstreams.
3. SPOT cache documentation at `docs/contributing/environment/cache.md`.
`compose.yml.md`, `variables.md`, the per-service READMEs, and
`requirements/012-...` now link to it instead of restating the
stack's behaviour. Net documentation lines drop substantially while
the cache page becomes the single user-facing reference.
Also: code-comment cleanup per `docs/contributing/documentation.md`
guidelines across all touched .py / .sh / .yml files. Multi-paragraph
docstrings, banner separators, and restate-the-obvious comments are
removed; one-line WHY comments remain where the rationale is not
already visible in the code.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a TLS-terminating reverse proxy in front of the Sonatype Nexus 3 OSS
pull-through cache so the `infinito` runner can hit upstream
package-manager URLs (pypi.org, registry.npmjs.org, rubygems.org,
repo.packagist.org, proxy.golang.org, dl-cdn.alpinelinux.org,
files.pythonhosted.org, raw.githubusercontent.com, codeload.github.com)
unchanged and still flow through Nexus.
Mechanism:
* scripts/docker/cache/package-frontend-certs.sh — host-side helper,
idempotently generates a dedicated CA + per-hostname leaf certs.
* compose/package-cache-frontend/upstreams.conf — nginx server-blocks
that match SNI per hostname and reverse-proxy onto the matching
Nexus repo path.
* compose/cache.override.yml `infinito.extra_hosts` — DNS-hijack so
the runner's clients resolve each upstream onto the frontend's
static IP.
* scripts/docker/cache/package-frontend-ca.sh — runner-side helper,
drops the CA into the system trust store post-startup so apt/pip/
curl/etc. accept TLS against the hijacked hostnames.
* cli/deploy/development/compose.py — orchestrates cert gen before
compose-up, CA install after health.
Scope today: runner-only. App-service Dockerfile builds executed by
the inner dockerd are NOT routed through the frontend; that requires
build-time CA injection and is deferred.
Also extends scripts/docker/cache/package.sh with five additional
Nexus proxies registered for transparent use: gem-proxy, go-proxy,
yum-rocky, yum-fedora, raw-packagist, raw-alpine, raw-codeload-github.
Refactor: move all cache services + runner-side cache mounts/depends_on/
extra_hosts from compose.yml into compose/cache.override.yml. Base
compose.yml now describes only the bare CI/runtime stack (coredns +
infinito). The override file is the single gate for the cache stack;
`profiles: ["cache"]` attributes and `--profile cache` flag emission
were redundant once file inclusion controls existence and removed
accordingly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move the registry-cache pull-through proxy out of the always-on `ci`
profile into its own `cache` profile, and select it based on the runtime
environment instead of statically.
Why
The proxy only earns its keep when image pulls amortize across many
runs on the same host. That holds on developer machines (cross-run
image dedup, persistent host disk) but not on GitHub-hosted runners
(fresh disk per job), where the proxy startup overhead is net loss
without the corresponding payoff.
How
* Compose: split `registry-cache` into its own `cache` profile;
`infinito.depends_on.registry-cache` gets `required: false` so Compose
does not auto-start the proxy via the dependency when the profile is
inactive; the systemd drop-in bind-mount source is gated by the new
`INFINITO_REGISTRY_CACHE_PROXY_CONF` env var with `/dev/null` default,
so dockerd loads an empty drop-in (no `HTTP_PROXY`, no `ExecStartPre`)
when the proxy is not running and pulls images directly.
* New `cli/deploy/development/profile.py` `Profile` class is the SPOT
for which compose profiles activate. KISS + SRP: `is_ci()` reads the
three CI signals (`GITHUB_ACTIONS`, `RUNNING_ON_GITHUB`, `CI`),
`registry_cache_active()` is its strict inverse, and `args()` returns
the matching `--profile ...` flag list.
* `Compose` and `down.py` consume the same `Profile` so up/down cannot
drift on which services are part of the stack.
* Documentation under `compose/registry-cache/README.md`,
`docs/contributing/artefact/files/compose.yml.md` and
`docs/contributing/environment/variables.md` describes the new toggle
and the `INFINITO_REGISTRY_CACHE_PROXY_CONF` env var. Pre-existing
documentation.md violations in those files (one SPOT self-label, one
em dash, two path-as-link-text instances) corrected in the same pass
per the fix-nearby rule.
* Tests: 14 new `Profile` unit tests in `test_profile.py` plus two
scenario tests in `test_compose.py` that lock down the
`--profile ci` (CI runner) vs `--profile ci --profile cache` (local)
argument shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 4.7