The frontend proxies to the package-cache by name, and nginx resolves a
literal proxy_pass hostname once while parsing the config and keeps it for the
worker's lifetime; the resolver only re-resolves targets that contain a
variable. The package-cache was the one cache service without a pinned
address, so recreating it left the frontend dialling a dead one, answering 502
to everything and reporting unhealthy.
That surfaces far from its cause: pip.conf deliberately keeps the real index
URL and extra_hosts points pypi.org at the frontend, so a pip install reports
no versions at all and looks like a network fault.
Pins it the way the registry-cache and the frontend are already pinned, and
carries the address into the runner slots.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A debian:bookworm workspace job died on `apt-get update`: the mounted
sources list carried the Ubuntu entry too, Nexus had auto-blocked
archive.ubuntu.com, and apt fails the whole update on one bad source.
Three holes, three plugs:
- One sources list per INFINITO_DISTRO under compose/package-cache/apt/,
selected by the compose overrides. A Debian runner no longer fetches
an Ubuntu suite it has no use for. The non-apt distros get an empty
file so the bind mount has a source to point at.
- autoBlock off. While a remote is auto-blocked Nexus answers `404
Remote Auto Blocked` to every client instead of falling back to the
copy it already has cached.
- Mirror failover in the frontend. Every apt suite gets a second proxy
repo on an independent upstream; a 5xx from the primary re-runs the
request against it. Nexus has no group repository type for apt, so
the retry lives in nginx.
Measured on a live Nexus 3.91.1: dead remote without a cached copy
answers 502, with a cached copy 200 regardless of metadataMaxAge. The
frontend serves 149 kB of real bookworm `Release` from the mirror repo
when the primary is dead, and the new regex locations produce upstream
request lines byte-identical to the `rewrite … break` form they replace,
including `%3a` epochs in pool paths.
INFINITO_CACHE_PACKAGE_APT_LIST is gone; the mount interpolates
INFINITO_DISTRO directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Path-style invocation in scripts/system/network/dns/common.sh failed
with `ImportError: attempted relative import with no known parent
package` because cli/meta/domains/__main__.py uses
`from . import PROJECT_ROOT`. The DNS host generator silently fell
back to a static 3-host list on every bootstrap run (visible in the
test-development CI logs).
Switch the caller to `PYTHONPATH=<repo> python3 -m cli.meta.domains`
so the package context resolves. Keeps the PROJECT_ROOT SPOT
(tests/lint/filesystem/python/test_project_root_import.py) intact —
no local redefinition.
Also: one-line YAML rewrap in compose/cache.override.yml (formatter).
Three new lint tests close gaps in the INFINITO_* SPOT contract:
* tests/lint/filesystem/env/test_default_env_keys_handled.py
Every default.env key must be wired through a handler.
* tests/lint/filesystem/env/test_used_keys_declared.py
Every INFINITO_* in .sh/.py must live in env/{default,ci}.env or
a handler; per-line `# nocheck:` suppresses runtime-only refs.
* tests/lint/filesystem/python/test_no_infinito_get_defaults.py
Forbid ANY default in `.get("INFINITO_*", ...)` -- including the
empty-string fallback (which silently masks an unset key). The
nocheck check walks every physical line the call spans so an
autoformat-split call still honours the marker.
Cache namespace flattens to a single prefix:
INFINITO_PACKAGE_CACHE_* -> INFINITO_CACHE_PACKAGE_*
INFINITO_REGISTRY_CACHE_* -> INFINITO_CACHE_REGISTRY_*
The five previously driver-injected paths (APT_LIST, NPMRC,
PIP_CONF, REGISTRY_PROXY_CONF, FRONTEND_CA_FILE) move into
env/default.env and passthrough.STATIC_KEYS; the
`cache_env_overrides()` helper and its two call sites are gone
along with the test-fixture override for FRONTEND_CA_DIR.
SPOT-strict rewrites:
- 19 sites: `something.get("INFINITO_X", "")` -> subscript
`something["INFINITO_X"]` (loud KeyError when env loader did
not run; all keys live in env/default.env).
- 2 handler-internal bootstrap reads suppressed with
`# nocheck: handler-bootstrap-read`.
- 30 runtime-only / driver-injected / test-fixture references
annotated with `# nocheck: <reason>` per-line.
- Unit tests that previously removed SPOT-guaranteed keys via
`{k: v ... if k \!= KEY}` or `clear=True` switch to setting
the key to "" explicitly, matching the new contract.
Side fixes:
- INFINITO_DNS_DOMAIN + INFINITO_DNS_HOSTS_FILE move from
shell-side defaults in common.sh to env/default.env and
passthrough.STATIC_KEYS.
- dns_systemd_is_operational() drops the dead
INFINITO_DNS_FORCE_HOSTS_FALLBACK override branch (the
variable was never set anywhere).
- INFINITO_WORKER_FETCH switches from `os.environ.get(KEY, "1")`
to `os.environ[KEY]`.
- scripts/tests/code/wrapper.sh exec args move into a bash
array so `# nocheck:` on INFINITO_TEST_TYPE can live on its
own element line (SC2215 fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The URL-reachability probe now bounds its wall-clock end-to-end:
allow_redirects=False (a redirect chain previously stacked
timeouts), a 5h30m global deadline that marks in-flight probes
as warnings on expiry, and live `[done/total]` progress every 50
probes. Submission order is shuffled so same-host URLs no longer
batch at the front and self-DoS one origin.
The Docker image-version test drops its bespoke sequential
fetch loop and calls find_outdated_updates() from utils.update.docker,
so it now shares the INFINITO_WORKER_FETCH-sized worker pool with
the repository-refs check.
`scripts/tests/code/wrapper.sh` propagates BASH_ENV into the
container exec — without it INFINITO_WORKER_FETCH was unset
inside the container and the URL probe collapsed to 1 worker.
`make test` gains test-external; `make dotenv-force` regenerates
.env with a stripped environment so stale BASH_ENV-preloaded
INFINITO_* values cannot pin old numbers via setdefault.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace 8 bash env modules with a single Python generator
(cli.meta.env) that materialises env/static.env + runtime context
into a gitignored .env. Each variable lives in its own handler
under utils/env/handlers/; defaults carry inline one-line `# ...`
comments.
Rename to the INFINITO_* schema:
- workflow inputs: APPS, SERVICES_DISABLED, CMD, DEBUG, WHITELIST,
BUNDLES, FULL_CYCLE, VARIANT, RUN_FLAGS, LIMIT_HOST, PARALLEL
- inventory: INVENTORY_DIR / INVENTORY_FILE / HOST_VARS_FILE,
PW_FILE -> INVENTORY_PASSWORD_FILE
- venv: VENV_BASE / VENV_FALLBACK / VENV_NAME
- new SPOT defaults: COMPILE_SILENCE, CONTAINER_NO_CA,
WAIT_HEALTH_TIMEOUT_S, PACKAGE_CACHE_FRONTEND_INIT_IMAGE,
promoted IMAGE_TAG from GHA-only to general passthrough
compose.yml hardening:
- inline \${VAR:-default} removed; strict \${VAR:?...} everywhere
- new integration lint blocks regressions
Tooling:
- scripts/install/all.sh owns the stamp-file install logic
(Makefile target delegates)
- scripts/meta/env/load.sh is the single shell entry point;
8 bash modules deleted
- new shell lint: no INFINITO_* defaults outside env/static.env
Tests:
- 93 new unit tests for utils/env/{builder,parser,runtime,writer}
- restructure tests/lint into repository/ (file-content rules) vs
filesystem/{shell,python,env}/ (filesystem-shape rules)
- drift + smoke integration tests against the generator
Docs:
- variables.md is now a pointer; per-variable docs live next to
the key in env/static.env / env/ci.env
- compose.yml.md drops the Default column
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Makefile lint integration:
* scripts/install/lint.sh: replace the prior checkmake binary install with
ensure_mbake (mbake is pip-installable, parses our Makefile correctly
whereas checkmake bailed on .ONESHELL: / .SHELLFLAGS: / ifneq).
* scripts/lint/makefile.sh: new runner — `mbake format --check Makefile`
reports formatting drift and missing .PHONY declarations.
* scripts/lint/autoformat.sh: add a two-pass `mbake format Makefile`
(mbake's formatter is not strictly idempotent on the first pass).
* Makefile: split the multi-line `.PHONY:` block into 18 single-line
`.PHONY:` declarations so mbake's parser can validate it; add the
previously-missing `install-python-dev` and `build-dependency` PHONY
targets; add `lint-makefile` aggregate-and-direct target.
* .pre-commit-config.yaml: new make-lint-makefile hook scoped to
`^Makefile$`.
* .github/workflows/lint-makefile.yml: new CI workflow (pip-install mbake
+ make lint-makefile).
* .github/workflows/ci-orchestrator.yml: wire lint-makefile into
lint-gate.needs.
Misc:
* .gitignore: add roles/importer_result.json (tooling artefact written by
galaxy-importer when it runs in roles/).
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extend the local cache stack with three independent capabilities:
1. Inner-`dockerd` build coverage for HTTP-only package mirrors.
`roles/sys-svc-compose/files/compose.py` (the runner's `compose`
wrapper) auto-generates a transient `compose.cache.override.yml`
per app when `INFINITO_PACKAGE_CACHE_FRONTEND_IP` is exported. The
override pins `build.extra_hosts` for `deb.debian.org`,
`archive.ubuntu.com`, and `security.ubuntu.com` to the frontend.
`pull.py` now delegates to the wrapper so build/pull operations see
the same `-f` set the wrapper auto-detects. nginx in the frontend
gains a port-80 listener with path rewrites for those hostnames,
and `package.sh` registers two new Nexus proxies
(`apt-debian-security`, `apt-ubuntu-security`) for the security
suites whose upstream URL prefix differs from the main archive.
Alpine is intentionally excluded from the inner-build extra_hosts:
3.18+ defaults to HTTPS, and inner-`dockerd` build containers do
not import the frontend CA. Routing apk through the frontend would
trip TLS verification.
2. General cache freshness env-var. `scripts/meta/env/cache/package.sh`
exports `INFINITO_PACKAGE_CACHE_MAX_AGE_MIN` (default `129600`,
i.e. 90 days). The bootstrap helper consumes it for every Nexus
proxy's `contentMaxAge`, `metadataMaxAge`, and
`negativeCache.timeToLive`, replacing the previous 1440-minute
literal. Lower the value when chasing fast-moving upstreams.
3. SPOT cache documentation at `docs/contributing/environment/cache.md`.
`compose.yml.md`, `variables.md`, the per-service READMEs, and
`requirements/012-...` now link to it instead of restating the
stack's behaviour. Net documentation lines drop substantially while
the cache page becomes the single user-facing reference.
Also: code-comment cleanup per `docs/contributing/documentation.md`
guidelines across all touched .py / .sh / .yml files. Multi-paragraph
docstrings, banner separators, and restate-the-obvious comments are
removed; one-line WHY comments remain where the rationale is not
already visible in the code.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a TLS-terminating reverse proxy in front of the Sonatype Nexus 3 OSS
pull-through cache so the `infinito` runner can hit upstream
package-manager URLs (pypi.org, registry.npmjs.org, rubygems.org,
repo.packagist.org, proxy.golang.org, dl-cdn.alpinelinux.org,
files.pythonhosted.org, raw.githubusercontent.com, codeload.github.com)
unchanged and still flow through Nexus.
Mechanism:
* scripts/docker/cache/package-frontend-certs.sh — host-side helper,
idempotently generates a dedicated CA + per-hostname leaf certs.
* compose/package-cache-frontend/upstreams.conf — nginx server-blocks
that match SNI per hostname and reverse-proxy onto the matching
Nexus repo path.
* compose/cache.override.yml `infinito.extra_hosts` — DNS-hijack so
the runner's clients resolve each upstream onto the frontend's
static IP.
* scripts/docker/cache/package-frontend-ca.sh — runner-side helper,
drops the CA into the system trust store post-startup so apt/pip/
curl/etc. accept TLS against the hijacked hostnames.
* cli/deploy/development/compose.py — orchestrates cert gen before
compose-up, CA install after health.
Scope today: runner-only. App-service Dockerfile builds executed by
the inner dockerd are NOT routed through the frontend; that requires
build-time CA injection and is deferred.
Also extends scripts/docker/cache/package.sh with five additional
Nexus proxies registered for transparent use: gem-proxy, go-proxy,
yum-rocky, yum-fedora, raw-packagist, raw-alpine, raw-codeload-github.
Refactor: move all cache services + runner-side cache mounts/depends_on/
extra_hosts from compose.yml into compose/cache.override.yml. Base
compose.yml now describes only the bare CI/runtime stack (coredns +
infinito). The override file is the single gate for the cache stack;
`profiles: ["cache"]` attributes and `--profile cache` flag emission
were redundant once file inclusion controls existence and removed
accordingly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5
Alejandro Roman