fix(package-cache): survive a flaky apt mirror

A debian:bookworm workspace job died on `apt-get update`: the mounted
sources list carried the Ubuntu entry too, Nexus had auto-blocked
archive.ubuntu.com, and apt fails the whole update on one bad source.

Three holes, three plugs:

- One sources list per INFINITO_DISTRO under compose/package-cache/apt/,
  selected by the compose overrides. A Debian runner no longer fetches
  an Ubuntu suite it has no use for. The non-apt distros get an empty
  file so the bind mount has a source to point at.
- autoBlock off. While a remote is auto-blocked Nexus answers `404
  Remote Auto Blocked` to every client instead of falling back to the
  copy it already has cached.
- Mirror failover in the frontend. Every apt suite gets a second proxy
  repo on an independent upstream; a 5xx from the primary re-runs the
  request against it. Nexus has no group repository type for apt, so
  the retry lives in nginx.

Measured on a live Nexus 3.91.1: dead remote without a cached copy
answers 502, with a cached copy 200 regardless of metadataMaxAge. The
frontend serves 149 kB of real bookworm `Release` from the mirror repo
when the primary is dead, and the new regex locations produce upstream
request lines byte-identical to the `rewrite … break` form they replace,
including `%3a` epochs in pool paths.

INFINITO_CACHE_PACKAGE_APT_LIST is gone; the mount interpolates
INFINITO_DISTRO directly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kevinveenbirkenbachandClaude Opus 5 committed 2026-08-04 19:33:04 +02:00
1 parent f9d6f4525b
commit f9dea42c68
15 files changed
+96 -45

No files matched your search

+2 -2
View File
@@ -140,8 +140,8 @@ services:
- ${INFINITO_CACHE_PACKAGE_PIP_CONF:?Set by 'cli deploy development up' / 'make compose-up' (driver-injected, not produced by 'make
dotenv')}:/etc/pip.conf:ro
- ${INFINITO_CACHE_PACKAGE_NPMRC:?Set by 'cli deploy development up' / 'make compose-up' (driver-injected, not produced by 'make dotenv')}:/root/.npmrc:ro
- ${INFINITO_CACHE_PACKAGE_APT_LIST:?Set by 'cli deploy development up' / 'make compose-up' (driver-injected, not produced by 'make
dotenv')}:/etc/apt/sources.list.d/package-cache.list:ro
- ./compose/package-cache/apt/${INFINITO_DISTRO:?Run 'make dotenv' to generate the .env single source of
truth}.list:/etc/apt/sources.list.d/package-cache.list:ro
- ${INFINITO_CACHE_PACKAGE_FRONTEND_CA_FILE:?Set by 'cli deploy development up' / 'make compose-up' (driver-injected, not produced by 'make
dotenv')}:/opt/package-frontend-ca.crt:ro
- ./scripts/docker/cache/package-frontend/ca.sh:/usr/local/bin/package-frontend-ca.sh:ro
+33 -12
View File
@@ -12,33 +12,54 @@ proxy_connect_timeout 30s;
# Docker embedded DNS, pinned to bypass image-specific resolv.conf order.
resolver 127.0.0.11 ipv6=off valid=30s;
# Nexus answers 502 when a proxy's remote is unreachable (measured) and has no
# group repository type for apt, so the mirror failover sits here rather than in
# the repo layout. 5xx only: a 404 means the file genuinely is not in the suite,
# and apt probes for missing files often enough that retrying those would double
# the request rate.
server {
listen 80;
server_name deb.debian.org;
location /debian/ {
rewrite ^/debian/(.*)$ /repository/apt-debian/$1 break;
proxy_pass http://package-cache:8081;
location ~ ^/debian/(?<debian_path>.*)$ {
proxy_intercept_errors on;
error_page 502 503 504 = @debian_mirror;
proxy_pass http://package-cache:8081/repository/apt-debian/$debian_path;
}
location /debian-security/ {
rewrite ^/debian-security/(.*)$ /repository/apt-debian-security/$1 break;
proxy_pass http://package-cache:8081;
location @debian_mirror {
proxy_pass http://package-cache:8081/repository/apt-debian-mirror/$debian_path;
}
location ~ ^/debian-security/(?<debian_security_path>.*)$ {
proxy_intercept_errors on;
error_page 502 503 504 = @debian_security_mirror;
proxy_pass http://package-cache:8081/repository/apt-debian-security/$debian_security_path;
}
location @debian_security_mirror {
proxy_pass http://package-cache:8081/repository/apt-debian-security-mirror/$debian_security_path;
}
}
server {
listen 80;
server_name archive.ubuntu.com;
location /ubuntu/ {
rewrite ^/ubuntu/(.*)$ /repository/apt-ubuntu/$1 break;
proxy_pass http://package-cache:8081;
location ~ ^/ubuntu/(?<ubuntu_path>.*)$ {
proxy_intercept_errors on;
error_page 502 503 504 = @ubuntu_mirror;
proxy_pass http://package-cache:8081/repository/apt-ubuntu/$ubuntu_path;
}
location @ubuntu_mirror {
proxy_pass http://package-cache:8081/repository/apt-ubuntu-mirror/$ubuntu_path;
}
}
server {
listen 80;
server_name security.ubuntu.com;
location /ubuntu/ {
rewrite ^/ubuntu/(.*)$ /repository/apt-ubuntu-security/$1 break;
proxy_pass http://package-cache:8081;
location ~ ^/ubuntu/(?<ubuntu_security_path>.*)$ {
proxy_intercept_errors on;
error_page 502 503 504 = @ubuntu_security_mirror;
proxy_pass http://package-cache:8081/repository/apt-ubuntu-security/$ubuntu_security_path;
}
location @ubuntu_security_mirror {
proxy_pass http://package-cache:8081/repository/apt-ubuntu-security-mirror/$ubuntu_security_path;
}
}
+1 -1
View File
@@ -8,4 +8,4 @@ For services, activation, coverage, and operations of the local cache stack, see
- `pip.conf`: pip configuration. Mounted at `/etc/pip.conf`. Routes pip through `pypi-proxy`.
- `npmrc`: npm configuration. Mounted at `/root/.npmrc`. Routes npm through `npm-proxy`.
- `apt.list`: apt sources. Mounted at `/etc/apt/sources.list.d/package-cache.list`. Routes apt through `apt-debian` (Debian runners) and `apt-ubuntu` (Ubuntu runners).
- `apt/<distro>.list`: apt sources, one file per `INFINITO_DISTRO`. The matching file is mounted at `/etc/apt/sources.list.d/package-cache.list`. `apt/debian.list` routes apt through `apt-debian`, `apt/ubuntu.list` through `apt-ubuntu`; the pacman/dnf distros get an empty file, because `apt-get update` fails as a whole on a single unreachable source and must not carry a foreign distro's entry.
-13
View File
@@ -1,13 +0,0 @@
# Routes apt-get traffic from the infinito runner through the
# package-cache (Sonatype Nexus 3 OSS) pull-through proxy. Two repo
# entries to cover both Debian (`apt-debian`) and Ubuntu (`apt-ubuntu`)
# upstreams; the runner reads only the entry that matches its distro
# at runtime via apt's standard suite-resolution. The bootstrap helper
# pins debian -> bookworm and ubuntu -> jammy; older runners still
# resolve fine because Nexus serves the entire upstream tree.
#
# Uses the real mirror hostnames (mapped to the cache frontend via
# extra_hosts in compose/cache.override.yml), not the `package-cache`
# service name which coredns — the container's resolver — cannot resolve.
deb [trusted=yes] http://deb.debian.org/debian bookworm main
deb [trusted=yes] http://archive.ubuntu.com/ubuntu jammy main
+3
View File
@@ -0,0 +1,3 @@
# Empty on purpose: Arch uses pacman, which the package-cache does not
# proxy. The bind mount is unconditional in the compose overrides, and
# docker would create a host directory here if the source were missing.
+4
View File
@@ -0,0 +1,4 @@
# Empty on purpose: CentOS uses dnf, which reaches the package-cache
# through `yum-rocky`, not through apt. The bind mount is unconditional
# in the compose overrides, and docker would create a host directory
# here if the source were missing.
+11
View File
@@ -0,0 +1,11 @@
# apt sources for the Debian runner: routes apt-get through the
# package-cache (Sonatype Nexus 3 OSS) pull-through proxy.
#
# Real mirror hostname, not the `package-cache` service name, which
# coredns — the container's resolver — cannot resolve; the compose
# overrides map the hostname to the cache frontend via extra_hosts.
#
# One file per INFINITO_DISTRO: apt fails the whole `apt-get update`
# when a single source is unreachable, so a Debian runner must never
# carry the Ubuntu entry.
deb [trusted=yes] http://deb.debian.org/debian bookworm main
+4
View File
@@ -0,0 +1,4 @@
# Empty on purpose: Fedora uses dnf, which reaches the package-cache
# through `yum-fedora`, not through apt. The bind mount is unconditional
# in the compose overrides, and docker would create a host directory
# here if the source were missing.
+11
View File
@@ -0,0 +1,11 @@
# apt sources for the Ubuntu runner: routes apt-get through the
# package-cache (Sonatype Nexus 3 OSS) pull-through proxy.
#
# Real mirror hostname, not the `package-cache` service name, which
# coredns — the container's resolver — cannot resolve; the compose
# overrides map the hostname to the cache frontend via extra_hosts.
#
# One file per INFINITO_DISTRO: apt fails the whole `apt-get update`
# when a single source is unreachable, so an Ubuntu runner must never
# carry the Debian entry.
deb [trusted=yes] http://archive.ubuntu.com/ubuntu jammy main
+1 -1
View File
@@ -8,7 +8,7 @@ x-cache: &cache
- "security.ubuntu.com:${INFINITO_CACHE_PACKAGE_FRONTEND_IP}"
- "infinito-registry-cache:${INFINITO_CACHE_REGISTRY_IP:?Run 'make dotenv' to generate the .env single source of truth}"
volumes:
- ../package-cache/apt.list:/etc/apt/sources.list.d/package-cache.list:ro
- ../package-cache/apt/${INFINITO_DISTRO:?Run 'make dotenv' to generate the .env single source of truth}.list:/etc/apt/sources.list.d/package-cache.list:ro
- ${INFINITO_CACHE_REGISTRY_CA_HOST_PATH:?Run 'make dotenv' to generate the .env single source of truth}:/opt/registry-cache-ca:ro
- ../../scripts/docker/cache/registry-ca.sh:/usr/local/bin/registry-cache-ca.sh:ro
-3
View File
@@ -37,9 +37,6 @@ INFINITO_BUILDKIT_ENABLED=1
# External docker network carrying the shared cache stack; empty on the primary instance, which owns the stack itself.
INFINITO_CACHE_NETWORK=
# Bind-mounted apt sources.list fragment for the package-cache frontend.
INFINITO_CACHE_PACKAGE_APT_LIST=./compose/package-cache/apt.list
# Host directory holding the package-cache frontend CA materials.
INFINITO_CACHE_PACKAGE_FRONTEND_CA_DIR=/var/cache/infinito/core/cache/package/frontend/ca
@@ -77,7 +77,6 @@ The env-var contracts each service expects strictly via `${VAR:?…}` (consumed
| `INFINITO_CACHE_PACKAGE_PORT` | Host-side port mapped to Nexus REST/UI. Bound to `${INFINITO_BIND_IP}` only. |
| `INFINITO_CACHE_PACKAGE_PIP_CONF` | Bind source for `/etc/pip.conf`. Driver-injected by `make compose-up` under the `cache` profile. |
| `INFINITO_CACHE_PACKAGE_NPMRC` | Bind source for `/root/.npmrc`. Driver-injected. |
| `INFINITO_CACHE_PACKAGE_APT_LIST` | Bind source for `/etc/apt/sources.list.d/package-cache.list`. Driver-injected. |
| `INFINITO_CACHE_PACKAGE_FRONTEND_CA_DIR` | Host directory for the frontend CA. |
| `INFINITO_CACHE_PACKAGE_FRONTEND_CERTS_DIR` | Host directory for per-hostname leaf certs. |
| `INFINITO_CACHE_PACKAGE_FRONTEND_IP` | Static IPv4 address for the frontend on the compose default network. |
+7 -2
View File
@@ -17,6 +17,7 @@ Pulls only. Pushes are not intercepted.
| Format | Repo names | Upstream |
|---|---|---|
| `apt` | `apt-debian`, `apt-debian-security`, `apt-ubuntu`, `apt-ubuntu-security` | `deb.debian.org`, `archive.ubuntu.com`, `security.ubuntu.com` |
| `apt` (mirrors) | `apt-debian-mirror`, `apt-debian-security-mirror`, `apt-ubuntu-mirror`, `apt-ubuntu-security-mirror` | `ftp.debian.org`, `security.debian.org`, `de.archive.ubuntu.com` |
| `pypi` | `pypi-proxy` | `pypi.org` (incl. `files.pythonhosted.org`) |
| `npm` | `npm-proxy` | `registry.npmjs.org` |
| `rubygems` | `gem-proxy` | `rubygems.org` |
@@ -25,6 +26,10 @@ Pulls only. Pushes are not intercepted.
| `yum` | `yum-rocky`, `yum-fedora` | `download.rockylinux.org`, `dl.fedoraproject.org` |
| `raw` | `raw-githubusercontent`, `raw-codeload-github`, `raw-packagist`, `raw-alpine` | `raw.githubusercontent.com`, `codeload.github.com`, `repo.packagist.org`, `dl-cdn.alpinelinux.org` |
A proxy repo that holds a cached copy serves it when its remote is unreachable, regardless of `metadataMaxAge` (measured). `autoBlock` is therefore off: while a remote is auto-blocked Nexus answers `404 Remote Auto Blocked` instead of falling back to that cached copy. The mirror repos below only come into play when nothing is cached either.
Every apt suite exists twice, once per upstream mirror. Nexus has no group repository type for `apt` (only maven, raw, docker, yum, npm, pypi, rubygems, go and friends), so the failover lives in the frontend: a request that the primary repo answers with `502`/`503`/`504` is re-run against the `-mirror` repo, which proxies a different host. `404` deliberately does not retry, because a missing file is missing on both mirrors and apt probes for missing files often.
Bootstrap is idempotent and runs from [package.sh](../../../scripts/docker/cache/package.sh) once the stack is healthy.
### Package Cache Frontend 🔐
@@ -45,7 +50,7 @@ The `cache` decision is exposed via `Profile.registry_cache_active()` in [profil
- [compose/cache.override.yml](../../../compose/cache.override.yml) is layered on top of the base [compose.yml](../../../compose.yml) by [compose.py](../../../cli/administration/deploy/development/compose.py) and [down.py](../../../cli/administration/deploy/development/down.py) via [common.py](../../../cli/administration/deploy/development/common.py)`compose_file_args`.
- The cache services are added.
- The runner's `infinito` service receives:
- bind-mounts for the registry-cache CA, the package-cache client snippets (`pip.conf`, `npmrc`, `apt.list`), and the frontend CA file
- bind-mounts for the registry-cache CA, the package-cache client snippets (`pip.conf`, `npmrc`, `apt/${INFINITO_DISTRO}.list`), and the frontend CA file
- `extra_hosts` entries DNS-hijacking the HTTPS upstream hostnames to the frontend's static IP
- `INFINITO_CACHE_PACKAGE_FRONTEND_IP` env var for the inner compose wrapper
- Cert generation, Nexus repo bootstrap, and runner trust-store install run from [compose.py](../../../cli/administration/deploy/development/compose.py).
@@ -59,7 +64,7 @@ CI signals (`GITHUB_ACTIONS=true`, `INFINITO_RUNNING_ON_GITHUB=true`, `CI=true`)
| Traffic | Mechanism | Cached |
|---|---|---|
| Image pulls (any registry, inner `dockerd`) | `registry-cache` MITM via `HTTP_PROXY` env on `dockerd` | ✓ |
| `apt-get install` (Ansible task in runner) | `apt.list` URL-rewrite to `package-cache:8081` | ✓ |
| `apt-get install` (Ansible task in runner) | `apt/${INFINITO_DISTRO}.list` URL-rewrite to `package-cache:8081` | ✓ |
| `pip install` (Ansible task in runner) | `pip.conf` URL-rewrite | ✓ |
| `npm install` (Ansible task in runner) | `.npmrc` URL-rewrite | ✓ |
| `gem`, `composer`, `go`, `curl https://…` (Ansible task in runner) | DNS-hijack + frontend HTTPS + runner CA-trust | ✓ |
+19 -9
View File
@@ -93,44 +93,54 @@ ensure_blobstore() {
esac
}
ensure_proxy_repo() {
local format="$1" name="$2" payload="$3"
ensure_repo() {
local format="$1" kind="$2" name="$3" payload="$4"
local code
code="$(nexus_curl -sS -o /dev/null -w '%{http_code}' \
-u "${ADMIN_USER}:${ADMIN_PASS}" \
-H "Content-Type: application/json" \
-X POST "${NEXUS_REST}/v1/repositories/${format}/proxy" \
-X POST "${NEXUS_REST}/v1/repositories/${format}/${kind}" \
--data "${payload}" || true)"
case "${code}" in
201 | 204) log "${format} proxy ${name} created (HTTP ${code})" ;;
201 | 204) log "${format} ${kind} ${name} created (HTTP ${code})" ;;
400 | 409)
code="$(nexus_curl -sS -o /dev/null -w '%{http_code}' \
-u "${ADMIN_USER}:${ADMIN_PASS}" \
-H "Content-Type: application/json" \
-X PUT "${NEXUS_REST}/v1/repositories/${format}/proxy/${name}" \
-X PUT "${NEXUS_REST}/v1/repositories/${format}/${kind}/${name}" \
--data "${payload}" || true)"
case "${code}" in
200 | 204) log "${format} proxy ${name} updated (HTTP ${code})" ;;
200 | 204) log "${format} ${kind} ${name} updated (HTTP ${code})" ;;
*)
log "unexpected HTTP ${code} updating ${format} proxy ${name}"
log "unexpected HTTP ${code} updating ${format} ${kind} ${name}"
return 1
;;
esac
;;
*)
log "unexpected HTTP ${code} creating ${format} proxy ${name}"
log "unexpected HTTP ${code} creating ${format} ${kind} ${name}"
return 1
;;
esac
}
ensure_proxy_repo() { ensure_repo "$1" proxy "$2" "$3"; }
# autoBlock stays off: once Nexus blocks a remote it answers every client
# with `404 Remote Auto Blocked` for the whole block window, so a single
# upstream hiccup fails every concurrent job instead of just the request
# that hit it.
ensure_all_proxies() {
local storage='"storage":{"blobStoreName":"default","strictContentTypeValidation":true},"proxy":{"contentMaxAge":'"${CACHE_CONTENT_MAX_AGE_MIN}"',"metadataMaxAge":'"${CACHE_METADATA_MAX_AGE_MIN}"'},"negativeCache":{"enabled":true,"timeToLive":'"${CACHE_METADATA_MAX_AGE_MIN}"'},"httpClient":{"blocked":false,"autoBlock":true}'
local storage='"storage":{"blobStoreName":"default","strictContentTypeValidation":true},"proxy":{"contentMaxAge":'"${CACHE_CONTENT_MAX_AGE_MIN}"',"metadataMaxAge":'"${CACHE_METADATA_MAX_AGE_MIN}"'},"negativeCache":{"enabled":true,"timeToLive":'"${CACHE_METADATA_MAX_AGE_MIN}"'},"httpClient":{"blocked":false,"autoBlock":false}'
ensure_proxy_repo apt apt-debian "$(printf '{"name":"apt-debian","online":true,%s,"proxy":{"remoteUrl":"http://deb.debian.org/debian","contentMaxAge":'"${CACHE_CONTENT_MAX_AGE_MIN}"',"metadataMaxAge":'"${CACHE_METADATA_MAX_AGE_MIN}"'},"apt":{"distribution":"bookworm","flat":false}}' "${storage}")"
ensure_proxy_repo apt apt-ubuntu "$(printf '{"name":"apt-ubuntu","online":true,%s,"proxy":{"remoteUrl":"http://archive.ubuntu.com/ubuntu","contentMaxAge":'"${CACHE_CONTENT_MAX_AGE_MIN}"',"metadataMaxAge":'"${CACHE_METADATA_MAX_AGE_MIN}"'},"apt":{"distribution":"jammy","flat":false}}' "${storage}")"
ensure_proxy_repo apt apt-debian-security "$(printf '{"name":"apt-debian-security","online":true,%s,"proxy":{"remoteUrl":"http://deb.debian.org/debian-security","contentMaxAge":'"${CACHE_CONTENT_MAX_AGE_MIN}"',"metadataMaxAge":'"${CACHE_METADATA_MAX_AGE_MIN}"'},"apt":{"distribution":"bookworm-security","flat":false}}' "${storage}")" # nocheck: url (bare GET on the deb.debian.org redirector returns 404; apt-client paths underneath dists/bookworm-security/ resolve correctly)
ensure_proxy_repo apt apt-ubuntu-security "$(printf '{"name":"apt-ubuntu-security","online":true,%s,"proxy":{"remoteUrl":"http://security.ubuntu.com/ubuntu","contentMaxAge":'"${CACHE_CONTENT_MAX_AGE_MIN}"',"metadataMaxAge":'"${CACHE_METADATA_MAX_AGE_MIN}"'},"apt":{"distribution":"jammy-security","flat":false}}' "${storage}")"
ensure_proxy_repo apt apt-debian-mirror "$(printf '{"name":"apt-debian-mirror","online":true,%s,"proxy":{"remoteUrl":"http://ftp.debian.org/debian","contentMaxAge":'"${CACHE_CONTENT_MAX_AGE_MIN}"',"metadataMaxAge":'"${CACHE_METADATA_MAX_AGE_MIN}"'},"apt":{"distribution":"bookworm","flat":false}}' "${storage}")"
ensure_proxy_repo apt apt-ubuntu-mirror "$(printf '{"name":"apt-ubuntu-mirror","online":true,%s,"proxy":{"remoteUrl":"http://de.archive.ubuntu.com/ubuntu","contentMaxAge":'"${CACHE_CONTENT_MAX_AGE_MIN}"',"metadataMaxAge":'"${CACHE_METADATA_MAX_AGE_MIN}"'},"apt":{"distribution":"jammy","flat":false}}' "${storage}")"
ensure_proxy_repo apt apt-debian-security-mirror "$(printf '{"name":"apt-debian-security-mirror","online":true,%s,"proxy":{"remoteUrl":"http://security.debian.org/debian-security","contentMaxAge":'"${CACHE_CONTENT_MAX_AGE_MIN}"',"metadataMaxAge":'"${CACHE_METADATA_MAX_AGE_MIN}"'},"apt":{"distribution":"bookworm-security","flat":false}}' "${storage}")"
ensure_proxy_repo apt apt-ubuntu-security-mirror "$(printf '{"name":"apt-ubuntu-security-mirror","online":true,%s,"proxy":{"remoteUrl":"http://de.archive.ubuntu.com/ubuntu","contentMaxAge":'"${CACHE_CONTENT_MAX_AGE_MIN}"',"metadataMaxAge":'"${CACHE_METADATA_MAX_AGE_MIN}"'},"apt":{"distribution":"jammy-security","flat":false}}' "${storage}")"
ensure_proxy_repo pypi pypi-proxy "$(printf '{"name":"pypi-proxy","online":true,%s,"proxy":{"remoteUrl":"https://pypi.org/","contentMaxAge":'"${CACHE_CONTENT_MAX_AGE_MIN}"',"metadataMaxAge":'"${CACHE_METADATA_MAX_AGE_MIN}"'}}' "${storage}")"
ensure_proxy_repo npm npm-proxy "$(printf '{"name":"npm-proxy","online":true,%s,"proxy":{"remoteUrl":"https://registry.npmjs.org/","contentMaxAge":'"${CACHE_CONTENT_MAX_AGE_MIN}"',"metadataMaxAge":'"${CACHE_METADATA_MAX_AGE_MIN}"'}}' "${storage}")"
ensure_proxy_repo helm helm-bitnami "$(printf '{"name":"helm-bitnami","online":true,%s,"proxy":{"remoteUrl":"https://charts.bitnami.com/bitnami","contentMaxAge":'"${CACHE_CONTENT_MAX_AGE_MIN}"',"metadataMaxAge":'"${CACHE_METADATA_MAX_AGE_MIN}"'}}' "${storage}")"
-1
View File
@@ -102,7 +102,6 @@ STATIC_KEYS = (
"INFINITO_CACHE_REGISTRY_IP",
"INFINITO_CACHE_PACKAGE_HOST_PATH",
"INFINITO_CACHE_PACKAGE_PORT",
"INFINITO_CACHE_PACKAGE_APT_LIST",
"INFINITO_CACHE_PACKAGE_FRONTEND_CA_DIR",
"INFINITO_CACHE_PACKAGE_FRONTEND_CA_FILE",
"INFINITO_CACHE_PACKAGE_FRONTEND_CERTS_DIR",