kevinveenbirkenbach/computer-playbook
1
0 Fork 0
mirror of https://github.com/kevinveenbirkenbach/computer-playbook.git synced 2026-11-09 04:06:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
05f2d9d6cc8e4d87db43d3141506fdebf58b1663
computer-playbook/docs/contributing/environment/cache.md
T
kevinveenbirkenbachandClaude Opus 4.7 eb45f0ffaf feat(cache): inner-build override + TTL env-var + SPOT docs 
Extend the local cache stack with three independent capabilities:

1. Inner-`dockerd` build coverage for HTTP-only package mirrors.
   `roles/sys-svc-compose/files/compose.py` (the runner's `compose`
   wrapper) auto-generates a transient `compose.cache.override.yml`
   per app when `INFINITO_PACKAGE_CACHE_FRONTEND_IP` is exported. The
   override pins `build.extra_hosts` for `deb.debian.org`,
   `archive.ubuntu.com`, and `security.ubuntu.com` to the frontend.
   `pull.py` now delegates to the wrapper so build/pull operations see
   the same `-f` set the wrapper auto-detects. nginx in the frontend
   gains a port-80 listener with path rewrites for those hostnames,
   and `package.sh` registers two new Nexus proxies
   (`apt-debian-security`, `apt-ubuntu-security`) for the security
   suites whose upstream URL prefix differs from the main archive.

   Alpine is intentionally excluded from the inner-build extra_hosts:
   3.18+ defaults to HTTPS, and inner-`dockerd` build containers do
   not import the frontend CA. Routing apk through the frontend would
   trip TLS verification.

2. General cache freshness env-var. `scripts/meta/env/cache/package.sh`
   exports `INFINITO_PACKAGE_CACHE_MAX_AGE_MIN` (default `129600`,
   i.e. 90 days). The bootstrap helper consumes it for every Nexus
   proxy's `contentMaxAge`, `metadataMaxAge`, and
   `negativeCache.timeToLive`, replacing the previous 1440-minute
   literal. Lower the value when chasing fast-moving upstreams.

3. SPOT cache documentation at `docs/contributing/environment/cache.md`.
   `compose.yml.md`, `variables.md`, the per-service READMEs, and
   `requirements/012-...` now link to it instead of restating the
   stack's behaviour. Net documentation lines drop substantially while
   the cache page becomes the single user-facing reference.

Also: code-comment cleanup per `docs/contributing/documentation.md`
guidelines across all touched .py / .sh / .yml files. Multi-paragraph
docstrings, banner separators, and restate-the-obvious comments are
removed; one-line WHY comments remain where the rationale is not
already visible in the code.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-30 01:08:41 +02:00

124 lines
9.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Local Caches 📦
Three local cache services accelerate re-deploys, reduce upstream traffic, and harden the development workflow against transient upstream outages. They share one compose profile (`cache`) and one override file ([compose/cache.override.yml](../../../compose/cache.override.yml)) and are gated together by [profile.py](../../../cli/deploy/development/profile.py): active on developer machines, inactive on CI runners.
## Services 🧩
### Registry Cache 🐳
[`registry-cache`](../../../compose/registry-cache/README.md) runs `rpardini/docker-registry-proxy` and transparently MITMs every Docker image pull from the inner `dockerd`, regardless of upstream registry (`docker.io`, `ghcr.io`, `mcr.microsoft.com`, …). The runner trusts the proxy CA via [registry-ca.sh](../../../scripts/docker/cache/registry-ca.sh).
Pulls only. Pushes are not intercepted.
### Package Cache 📚
[`package-cache`](../../../compose/package-cache/README.md) runs `sonatype/nexus3` and exposes pull-through proxies for package-manager downloads:
| Format | Repo names | Upstream |
|---|---|---|
| `apt` | `apt-debian`, `apt-debian-security`, `apt-ubuntu`, `apt-ubuntu-security` | `deb.debian.org`, `archive.ubuntu.com`, `security.ubuntu.com` |
| `pypi` | `pypi-proxy` | `pypi.org` (incl. `files.pythonhosted.org`) |
| `npm` | `npm-proxy` | `registry.npmjs.org` |
| `rubygems` | `gem-proxy` | `rubygems.org` |
| `go` | `go-proxy` | `proxy.golang.org` |
| `helm` | `helm-bitnami` | `charts.bitnami.com/bitnami` |
| `yum` | `yum-rocky`, `yum-fedora` | `download.rockylinux.org`, `dl.fedoraproject.org` |
| `raw` | `raw-githubusercontent`, `raw-codeload-github`, `raw-packagist`, `raw-alpine` | `raw.githubusercontent.com`, `codeload.github.com`, `repo.packagist.org`, `dl-cdn.alpinelinux.org` |
Bootstrap is idempotent and runs from [package.sh](../../../scripts/docker/cache/package.sh) once the stack is healthy.
### Package Cache Frontend 🔐
[`package-cache-frontend`](../../../compose/package-cache-frontend/README.md) runs `nginx:alpine` and reverse-proxies upstream package-manager hostnames onto the matching Nexus repo path. Combined with `extra_hosts` DNS-hijack on consumers, package managers can hit their real upstream URL and still flow through the cache.
Two listener layers:
- HTTPS (port 443): per-hostname server certs signed by a dedicated CA. Used by the `infinito` runner (Ansible-driven `pip install`, `gem install`, `composer install`, `curl https://…`). The runner trusts the CA via [package-frontend-ca.sh](../../../scripts/docker/cache/package-frontend-ca.sh).
- HTTP (port 80): plain mirrors for `deb.debian.org`, `archive.ubuntu.com`, `security.ubuntu.com`, `dl-cdn.alpinelinux.org`. Used by inner-`dockerd` Dockerfile builds via `build.extra_hosts` DNS-hijack. No CA-trust required in the build container.
Cert generation runs in a throw-away alpine container driven by [package-frontend-certs.sh](../../../scripts/docker/cache/package-frontend-certs.sh) before the frontend starts.
## Activation 🎚️
The `cache` decision is exposed via `Profile.registry_cache_active()` in [profile.py](../../../cli/deploy/development/profile.py). When active:
- [compose/cache.override.yml](../../../compose/cache.override.yml) is layered on top of the base [compose.yml](../../../compose.yml) by [compose.py](../../../cli/deploy/development/compose.py) and [down.py](../../../cli/deploy/development/down.py) via [common.py](../../../cli/deploy/development/common.py)`compose_file_args`.
- The cache services are added.
- The runner's `infinito` service receives:
- bind-mounts for the registry-cache CA, the package-cache client snippets (`pip.conf`, `npmrc`, `apt.list`), and the frontend CA file
- `extra_hosts` entries DNS-hijacking the HTTPS upstream hostnames to the frontend's static IP
- `INFINITO_PACKAGE_CACHE_FRONTEND_IP` env var for the inner compose wrapper
- Cert generation, Nexus repo bootstrap, and runner trust-store install run from [compose.py](../../../cli/deploy/development/compose.py).
When inactive, the override is omitted: cache services do not exist, the runner has no cache mounts or DNS-hijack, package managers go direct to upstream.
CI signals (`GITHUB_ACTIONS=true`, `RUNNING_ON_GITHUB=true`, `CI=true`) deactivate. Fresh runner disks per CI job give no cross-run amortization.
## Coverage Matrix 📋
| Traffic | Mechanism | Cached |
|---|---|---|
| Image pulls (any registry, inner `dockerd`) | `registry-cache` MITM via `HTTP_PROXY` env on `dockerd` | ✓ |
| `apt-get install` (Ansible task in runner) | `apt.list` URL-rewrite to `package-cache:8081` | ✓ |
| `pip install` (Ansible task in runner) | `pip.conf` URL-rewrite | ✓ |
| `npm install` (Ansible task in runner) | `.npmrc` URL-rewrite | ✓ |
| `gem`, `composer`, `go`, `curl https://…` (Ansible task in runner) | DNS-hijack + frontend HTTPS + runner CA-trust | ✓ |
| `RUN apt-get install` (inner Dockerfile build, Debian/Ubuntu) | `build.extra_hosts` DNS-hijack + frontend HTTP | ✓ |
| `RUN apk add` (inner Dockerfile build, Alpine ≤3.17 HTTP) | `build.extra_hosts` DNS-hijack + frontend HTTP | ✓ |
| `RUN pip/npm/gem/composer/go install` (inner Dockerfile build) | none today | ✗ |
| `RUN apk add` (Alpine ≥3.18 HTTPS) | none today | ✗ |
| App container runtime traffic (HTTPS to external API) | none today | ✗ |
The HTTPS-only inner-build gap requires per-image CA-trust bootstrap, which is out of scope for the current minimal-invasive design.
## Compose Wrapper Auto-Detection 🪄
The runner's `compose` wrapper at [roles/sys-svc-compose/files/compose.py](../../../roles/sys-svc-compose/files/compose.py) auto-detects compose files when invoked from per-app directories under `/opt/compose/<app>/`:
| File | When |
|---|---|
| `compose.yml` | always |
| `compose.override.yml` | when present (role provides it) |
| `compose.ca.override.yml` | when present (TLS self-signed CA-inject runs) |
| `compose.cache.override.yml` | generated on the fly when `INFINITO_PACKAGE_CACHE_FRONTEND_IP` is set; emits `build.extra_hosts` for every service that has a `build:` key |
[pull.py](../../../roles/sys-svc-compose/files/pull.py) delegates to the same wrapper so `pull` and `build --pull` operations see the identical `-f` set.
## Environment Variables 🌳
The host-side env-vars live in [scripts/meta/env/cache/registry.sh](../../../scripts/meta/env/cache/registry.sh) and [scripts/meta/env/cache/package.sh](../../../scripts/meta/env/cache/package.sh). They are sourced via [scripts/meta/env/all.sh](../../../scripts/meta/env/all.sh), which `BASH_ENV` makes available to every Makefile recipe.
Per-variable defaults and purposes are in [compose.yml.md](../artefact/files/compose.yml.md) under the cache section.
`INFINITO_PACKAGE_CACHE_MAX_AGE_MIN` (default `129600` = 90 days) controls how long every Nexus proxy repo holds an upstream response before revalidating. The bootstrap helper applies it as `contentMaxAge`, `metadataMaxAge`, and `negativeCache.timeToLive`. Lower it for fast-moving upstreams (`apt-debian-security`, branch-pinned tarballs) when staleness becomes visible.
## Operations 🛠️
| Action | Command |
|---|---|
| Start the stack with caches | `make up` |
| Stop the stack | `make down` |
| Wipe local cache state | `make cache-clean` |
| Manually re-bootstrap Nexus repos | `bash scripts/docker/cache/package.sh` (after sourcing `scripts/meta/env/cache/package.sh`) |
| Manually regenerate frontend certs | `bash scripts/docker/cache/package-frontend-certs.sh` |
| Reload nginx in the frontend | `docker exec infinito-package-cache-frontend nginx -s reload` |
| Inspect cache hits | `docker logs -f infinito-package-cache` and `docker logs -f infinito-package-cache-frontend` |
Cache state persists under `/var/cache/infinito/core/cache/`. Paths are configurable via the env scripts above.
## Adding a New Upstream ➕
When a new package manager or upstream needs caching:
1. Register a Nexus proxy repo in [package.sh](../../../scripts/docker/cache/package.sh).
2. If the upstream uses HTTPS, add it to `HOSTNAMES` in [package-frontend-certs.sh](../../../scripts/docker/cache/package-frontend-certs.sh) so a leaf cert is issued.
3. Add a server-block in [upstreams.conf](../../../compose/package-cache-frontend/upstreams.conf) that reverse-proxies onto the new Nexus repo path (rewrite if upstream URL prefix differs from the Nexus repo path).
4. Add an `extra_hosts` entry on the `infinito` service in [compose/cache.override.yml](../../../compose/cache.override.yml) for runner-side traffic.
5. If the upstream is HTTP-only and inner-`dockerd` builds need it, also add it to `_CACHE_HTTP_HOSTNAMES` in [compose.py](../../../roles/sys-svc-compose/files/compose.py) so the per-app `compose.cache.override.yml` includes it in `build.extra_hosts`.
## Background and Requirements 📚
- Compose-file env-var contract: [compose.yml.md](../artefact/files/compose.yml.md)
- Original requirement and acceptance criteria: [docs/requirements/012-package-cache-nexus3-oss.md](../../requirements/012-package-cache-nexus3-oss.md)
- Profile gating mechanics: [profile.py](../../../cli/deploy/development/profile.py)
Reference in new issue View Git Blame Copy Permalink