The daily mirror waited about 70 s before each of its 195 copies (50 images
per hour), roughly 3.8 h of pauses, and logged no single rate-limit error.
37 of the copies come from ghcr.io, registry.gitlab.com, quay.io,
registry.opencode.de and mcr.microsoft.com, which the Docker Hub pull limit
does not cover, and --only-missing also waited before images it then skipped.
The throttle now waits only before an actual copy from docker.io, and the
schedule allows 150 Docker Hub copies per hour (24 s apart).
Part of #1324
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scan threshold becomes a severity-to-verdict mapping in
roles/sys-ctl-hlth-runtime/meta/services.yml, CRITICAL: fail and HIGH: warn,
for production and CI alike. A finding mapped to fail fails the service and
the deploy, a warn prints a warning line and passes, a suppressed one prints
an info line. Engine errors, missing databases and PHP hardening violations
still fail. The old list form is rejected with a config error. The daily
mirror scan stays strict: every unsuppressed fail or warn finding is a row,
so it fails while an image is red.
Every run publishes one list of judged findings four ways:
- report.json of the latest run (summary + records) under
DIR_HEALTH_RUNTIME_REPORT, written atomically;
- findings.jsonl under DIR_HEALTH_RUNTIME_LOG, one record per finding,
appended per run and rotated weekly by a logrotate drop-in;
- one JSON line per target plus a summary line on stdout for the journal;
- infinito_hlth_runtime.prom in DIR_METRICS_TEXTFILE when web-app-prometheus
is deployed: findings per target, kind, image, severity, scanner and
verdict (no CVE label), last run time and success, scanner errors.
Each record carries run_id, timestamp, host, environment, target, kind,
image, scanner, id, severity, package, installed_version, fixed_version,
suppressed, suppressed_until and verdict. The paths are SPOT variables in
group_vars/all/05_paths.yml, the script takes them as arguments.
Requirement 070 and the role README describe the mapping, the outputs and
the per-image vulnerability issue of the previous commit.
Part of #1324
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dry-run mirror scan (run 41286420589) reported 10,179 CVE/image pairs
across 135 red digests; one issue per pair would flood the tracker. The
sync now keeps exactly one open issue per pinned image:
- The body lists every CVE row (id, severity, engine, role, service,
package, installed, fixed), CRITICAL first, and stops below 60,000
characters with the number of rows left out and a pointer to the
vulnerability-findings artifact. On the dry-run data: 135 issues, 8
capped, the largest body 59,872 characters.
- A further open issue with the same title closes as a duplicate of the
oldest one.
- When an image's CVE table changed, the issue gets a "Still unresolved"
comment with the table and an updated body; an unchanged table leaves the
issue alone, so a daily run does not add 135 identical comments.
Part of #1324
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodeQL (py/call-to-non-callable, alerts 1073 and 1074 on PR #1326) could not
see the __call__ of the HangingEdit class defined inside the test method and
flagged both gh() calls in sync(). FakeGh now takes the verbs to hang as a
parameter, so the test double stays one module-level class.
Part of #1324
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodeRabbit review of PR #1326:
- fetch_databases downloads each OSV archive to all.zip.part and replaces
all.zip only after the transfer completes. An OSError removes the partial
file and comes back as an `osv <ecosystem>: <error>` line instead of a
traceback.
- run_gh bounds every gh call with a 120 s timeout. A failed or hung create,
edit or close becomes a `failed <action>: <error>` line while the other
issues and the close pass still run, and the command exits 1.
- The suppression lint checks every list item of a suppressions block, so an
entry that starts with `until:` still needs its `# nocheck: cve <reason>`
marker.
Part of #1324
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under act the deploy job pinned an engine but skipped the cache restores, so
every local swarm-* run stopped on a missing Grype database. The swarm
matrix also ran inside the act job container, where the shared host
directory did not exist, and the workspace job would have kept a copy of
about 5 GB in every worktree.
- scripts/tests/deploy/act/workflow.sh requires
INFINITO_VULNERABILITY_DB_HOST_PATH, creates it on the host and binds it
into the job container at the same path, because that container drives
the host daemon, which resolves every stack and swarm node bind on the
host.
- call-test-deploy.yml pins engine and database path in one step that runs
only outside act; under act the deploy draws an engine and refreshes the
shared directory. call-test-workspace.yml takes the shared path under act.
- The refresh lock moves into the shared directory, so host deploys and act
runs serialize on one file, and a root refresh hands its files to the
directory owner. hand_over skips paths a mocked download never created,
which failed four TestRefresh cases when the gate ran as root.
- A missing INFINITO_VULNERABILITY_DB_HOST_PATH stops compose up with a
"run 'make dotenv'" hint instead of a KeyError.
- Rationale paragraphs in the database and compose docstrings are dropped.
Part of #1324
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The daily mirror workflow gains a scan job after the sync. It runs
make test-security against the GHCR mirror, writes the failing findings per
image and per CVE into the job summary and uploads the report as the
vulnerability-findings artifact.
- One issue per CVE and image, labelled vulnerability, is opened, updated
while the finding stays and closed once no engine reports it. Issues are
only closed when the scan reported no error, so an image that failed to
scan keeps its issues.
- workflow_dispatch and workflow_call take an issues switch; false is a dry
run that only produces the summary and the artifact. Scheduled runs always
sync the issues.
- Only the scan job holds issues: write.
Part of #1324
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each CI deploy row now draws one vulnerability engine (trivy, grype or osv)
as a fifth rotation axis, shown in the job name next to distro and
filesystem (🔬 axis, 🔺 trivy, 🍇 grype, 🐙 osv). The deploy-time scan of
sys-ctl-hlth-runtime runs that engine behind Syft and the Grype runtime pass.
- A vulnerability-db job keeps one Actions cache per database and day and
downloads only the missing ones. Every deploy job restores Grype plus its
drawn engine, fails on a cache miss and never downloads a database itself.
- Local stacks share one host directory, INFINITO_VULNERABILITY_DB_HOST_PATH,
mounted read-only at INFINITO_VULNERABILITY_DB_DIR in compose.yml and in
every swarm node. A local deploy without a pinned engine draws one and
refreshes only the databases it reads, at most once a day, under a lock.
- A pinned engine whose database is missing stops the deploy before it
starts.
Part of #1324
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runtime check judged PHP by its upstream version and failed every
container on an end-of-life line. Distribution builds backport fixes without
raising that number, so main run 41154895895 failed Checkmk#0 on
`PHP 8.1.2-1ubuntu2.28 runs the end-of-life line 8.1`, a build Ubuntu lists
as fixed for CVE-2026-109618 and CVE-2026-109619.
- The role now runs Trivy, Grype and OSV-Scanner, with a Syft SBOM per
target, against the host and the images of the running containers, in
bounded parallel lanes. Targets, engines and the failing severities are
lists in meta/services.yml; the default is every target and engine at
CRITICAL and HIGH with a fix available.
- Grype always scans the binary-classified packages of the SBOM, so a PHP
built from source in an official php image (8.4.26) still fails on the
CVEs fixed in 8.4.27, which Trivy and OSV do not see.
- OSV reads an SBOM copy whose distro purls carry the source package name.
- A CVE is suppressed only in the role owning the image, under
services.<service>.security.suppressions with id, until and a
`# nocheck: cve <reason>` marker; a lint rejects bare, malformed and
expired entries.
- The deploy-time run fails on its first finding: sys-service gains
system_service_run_once and sys-service-terminator skips the retries
only for those units. The timer runs daily at 00:00 UTC.
- The PHP version floors are removed; the php_hardening check stays.
- make test-security replaces make security-runtimes: Syft builds an SBOM
of every pinned image straight from the GHCR mirror, cached by digest,
and the three engines scan it with the role's threshold and suppressions.
Part of #1324
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three CodeRabbit findings on PR #1265:
- script.py listed the running containers with `container ps` and
`container inspect` without a timeout, so a hung engine kept the daily
unit active forever and blocked the next timer run. Both calls now use
EXEC_TIMEOUT_SECONDS; main turns a timeout or engine error into a
one-line failure and exit 1 instead of a traceback.
- runtime.py's PHP_PACKAGE_PATTERN requires a digit after "php", so an
image installing an unversioned interpreter package (`php-cli`) and
setting no PHP_VERSION was reported as PHP-free. The interpreter
packages php, php-cli, php-fpm, php-cgi, php-phpdbg and
libapache2-mod-php are now recorded as PhpRuntime.packages and reported
with an unknown patch level; extension and library packages
(php-mysql, php-pear, php-common, phpmyadmin) and paths do not count.
- php_finding returned early for a patched PHP_VERSION, so an end-of-life
package line installed next to it (e.g. php7) was never reported. The
version only returns a finding now; otherwise the lines are judged too.
Every finding has unit tests that fail on the previous code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answering "which deployed containers run a vulnerable PHP?" after
CVE-2026-109618/109619 took a manual inventory of registry metadata, and
nothing noticed a hardening ini that an image never loads.
- New role sys-ctl-hlth-runtime, routed through sys-svc-compose next to
sys-ctl-hlth-container. A daily timer runs `php` in every running
container and fails, alerting through sys-ctl-alm-compose, when PHP is
below the floor of its line or on a line older than every floor (7.4,
8.1). Every service that mounts php_hardening in meta/volumes.yml fails
while expose_php or session.upload_progress.enabled is on; other PHP
containers of such a stack are listed as info. Compose and swarm
containers are matched by their project/stack and service labels. The
check is not run at deploy time, so a finding never fails a deploy.
- The floors (8.2.35, 8.3.36, 8.4.27, 8.5.12) live once, in
HEALTH_RUNTIME_FLOORS of the role's vars/main.yml.
- `make security-runtimes` lists the PHP of every pinned role image
without pulling: Docker Hub tags through the Hub API, which no pull rate
limit meters, other registries through the OCI config blob (new
registry.config_blob, which the healthcheck probe now reuses).
PHP_VERSION, distro PHP packages and a PHP_BASE build argument count. It
exits 1 on an image below its floor or one the registry did not answer.
Today it flags nfrastack/fusiondirectory:1.5 (line 7.4) and
shopware/docker-base:8.3 (8.3.25).
The role TODO.md keeps the follow-ups: further runtimes, the split web
SAPI of web-app-lam, a scheduled CI run of the report and the CVE scan.
Refs #1127
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Declare `__all__` in every package `__init__.py` whose `PROJECT_ROOT`
is imported by its modules (163 packages), so the constant counts as
exported; list every public name where the package defines more.
- Empty the 15 package `__init__.py` files whose `PROJECT_ROOT` no module
imports.
- Declare the public API of `utils.github.variant.axes` in `__all__`,
re-exports included, instead of silencing them with `noqa: F401`; the
list matches the one `feature/store-i18n` already carries.
- Use `import unittest.mock as mock` in `test_applications.py`, as the
rest of the suite does.
- Spell `_NON_PRINTABLE` in `utils/yaml_bootstrap.py` as the equivalent
BMP-only positive class; it matches exactly the same 2111 code points
as the old negated class across U+0000..U+10FFFF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
baudolo 7.1.x writes volume generations with `rsync -aHAX`, so the
producing host keeps `user.*` extended attributes (OpenCloud keeps its
storage metadata in `user.oc.*`). Every later copy dropped them again,
because none of the other rsync calls passed `-X` (nor `-A`/`-H`):
- NFS export snapshots (svc-bkp-nfs-2-local `script.sh`),
- the backup host's pull (svc-bkp-remote-2-local `pull_specific_host.py`),
- the device sync (svc-bkp-local-2-device `script.py`),
- every recover into a live target (`utils/recovery/base.py`, used by the
volume, NFS and device recover scripts),
- the `recover full` and `recover remote` tree pulls.
A disaster recovery through the documented chain therefore restored
OpenCloud without `user.oc.*`, silently: the copies succeeded and the
dirval stamp does not cover attributes.
All of these now run `rsync -aHAX` (`-aHAXP` for the device sync). A
target filesystem without xattr support makes rsync exit non-zero as soon
as a source file carries attributes, so the hop fails loudly instead of
dropping them; there is no per-device opt-out. The single-image pull in
`recover remote` keeps `-a`. The provider's ssh-wrapper needs no change:
its glob accepts any server flags and its character guard allows them.
Two tests write a `user.oc.id` attribute and assert it survives the NFS
snapshot and the recover; with the old `-a` both fail with
`OSError: [Errno 61] No data available`.
Closes#1021
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When the IPAM config of a compose project's network changes (#999 added
`ip_range`), compose recreates the network on the next `up`. With a
foreign endpoint attached, as every worktree stack is on the primary's
network, compose first stops the project's own containers, then fails
with `error while removing network: ... has active endpoints`, and a
repeated `up` or `up --force-recreate` does not recover. On 2026-10-19
this took the shared registry/package cache and coredns down for every
local session.
`make compose-network-recreate` (alias `i8conere`) runs the sequence that
recovered it: detach every running container of another project from the
network, `docker compose down --remove-orphans` without `-v` so the
caches keep their volumes, the normal `up`, then attach the detached
containers again, which draw addresses from the dynamic pool. If `up`
fails, the still-detached containers are listed instead of reattached.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.
The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.
call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Covers the declarations read from every host, the refusal of differing ones,
every peer pinned to its /32, no hub anywhere, the host of one mesh holding no
peer of the other, and a second run leaving every file alone.
No conflicts across 121 incoming commits. The one file both sides changed is
tasks/stages/01_constructor.yml: upstream parks an applications carrier for
the swarm bootstrap roles after the add_host tasks, and keys the carrier on
the tokens file as well. This branch's mesh bootstrap opens and drops its own
block-scoped carrier before that point, so both keep working as written.
Upstream also brings ruff 0.17.0, distro-derived env keys, a scoped act outer
teardown, the stalwart relay rule, the onion apex redirect and image and
version updates; ruff 0.17 reformats nothing on this branch.
Covers per-role and per-host membership, a host two roles put into one mesh,
the hub never being a spoke, an undeclared mesh being refused, and the
example's members matching its table.
The any-number-of-meshes test zipped the specs against the planned meshes
without strict=, which ruff reports as B905 and the lint-python job failed on.
Both sequences come from the same declarations, so strict=True also turns a
plan that drops a mesh into a failure instead of a silently shorter loop.
A compose deploy job labelled centos or arch built its stack image from
pkgmgr-debian and ran it as infinito_nexus_debian (#794). distros.sh
exports INFINITO_DISTRO per iteration and sources load.sh again, but the
loader returned early because INFINITO_ENV_LOADED and the .env mtime were
unchanged. A full reload would not have helped either: the debian values
were already exported, so the restore step kept them as caller intent.
load.sh now records the distro it loaded for. Another INFINITO_DISTRO
defeats the early return and drops INFINITO_CONTAINER,
INFINITO_PARENT_IMAGE and INFINITO_IMAGE, the three keys the env handlers
derive from the distro. When the requested distro also differs from the
one .env was generated for, the loader sources a set that
`python -m cli.meta.env --output` generates for it in a private temp
directory and leaves .env untouched, so a local multi-distro run does not
switch the checkout's distro. A caller value set for the loaded distro
still wins.
Verified: tests/unit/python/scripts/meta/env/test_load.py and the new
distros.sh case pass. Against main's loader both fail with the symptom of
the issue: INFINITO_DISTRO=centos next to infinito_nexus_debian and
pkgmgr-debian:stable, and distros.sh handing debian, arch and centos the
same debian parent. shellcheck is clean.
Consequence: CI jobs labelled arch, ubuntu, fedora or centos now build
and deploy that distro, so a failure that shows up there was masked
before.
feature/store-i18n carries 3f973eb137, which resolves the parent image
for scripts/image/push.sh on its own, and 05e4dbafcb, which inserts the
INFINITO_INSTANCE slot check into the block this commit restructures. The
merge keeps the slot check ahead of the distro check.
Closes#794
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The guarantees were asserted against the declared swarm and data pair,
so they proved two meshes work rather than that the planner works. A
deployment declares its own, and the review asks for any number of them with
the nodes arranged however the client wants.
The same invariants now run over generated topologies: one to eight meshes,
three arrangements each, hosts dealt out so some belong to several meshes and
some to none. 2247 subtests, four seconds, and the planner needed no change to
pass them.
report_failures asked `gh api --paginate` for the run's jobs with the jq
filter `[.jobs[] | {name, conclusion}]`. gh applies that filter to each page
and prints one array per page, so a run with more jobs than one page holds
produced two JSON documents, and json.loads stopped with "Extra data: line 2
column 1". Main run 38218402254 (113 jobs) failed its "Report main role
failures" job that way, and no issue was opened for its failed roles.
The listing now emits one job object per line and parses the lines one by
one, the same way logs/download/github.py lists jobs.
Verified: the old parse of the real output of run 38218402254 raises the
same "Extra data: line 2 column 1 (char 8491)" as CI; list_jobs on that run
returns all 113 jobs, 16 of them failed, grouped into 12 failed roles.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The repository is bind-mounted into the infinito container, and every deploy
runs `make setup` there, which depends on `dotenv`. The generator therefore
rewrote the host's .env with the values the container sees: the host lost
INFINITO_RUNNING_ON_GITHUB=true that way, and the guide replay pulled
alpine:latest past the GHCR mirror.
compose.yml now marks the infinito service with INFINITO_DOTENV_HOST_OWNED,
and `python -m cli.meta.env` keeps an existing .env while that marker is
"true". A missing .env is still generated, and `make dotenv-force` (env -i)
still rewrites it. The key stays out of the env registry: only compose.yml
sets it, it is no configuration value, and a .env entry could freeze the
host .env.
test_distros now also strips INFINITO_ENV_LOADED. load.sh returns early
while that guard is set and the mtime of .env is unchanged, so the test only
reloaded the budget keys it strips because the container kept rewriting .env.
Verified: unit tests cover a kept .env, a missing one being generated, and a
rewrite without the marker. Local compose deploys print "Kept .env: the host
checkout owns it" where they printed "Wrote .env" before, the host .env
checksum stayed the same across a deploy, and the four keys holding host
paths resolve inside the container under the same paths.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The GHCR mirror covered role images only. The images the runner pulls on
its own went straight to Docker Hub: coredns from compose.yml, the cache
stack in compose/cache.override.yml and alpine:latest, the init image of
scripts/docker/cache/package-frontend/certs.sh. With the anonymous
githubactions quota spent, the guide step of Postmarks#0 (main run
37954203718) and Shopware#2 (#872) died on "toomanyrequests" while pulling
alpine:latest, after coredns had hit the same limit.
iter_runner_images (utils/docker/image/discovery.py) finds those images,
and the mirror sync and the fork-PR wait now cover them next to the role
images. On a real GitHub runner compose_file_args layers a generated
build/compose/runner-mirror.override.yml that points only the services of
the layered compose files at their mirror, and the certificate generator
gets the mirrored init image. Local runs and act are unchanged, and the
coredns literal stays in compose.yml for Dependabot. test-dns now waits for
images-mirror-missing like the deploy and workspace jobs, since it brings up
the stack too.
Verified with docker compose config on a simulated GitHub runner: the
service set is unchanged and only the runner images point at
ghcr.io/infinito-nexus/core/mirror/docker.io/..., and an existing role mirror
under that scheme pulls anonymously.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The arm existed to answer one question: how much of the meshed arm's
cost is the connection plugin and how much is the tunnel. Runs 37592061960
(docker), 37753506661 (SSH on the underlay) and 37621556682 (SSH over
WireGuard) answered it on web-app-gitea#0, debian/ext4:
pass 1 5964s / 8511s / 8832s
pass 2 5319s / 7472s / 7705s
Leaving ansible_connection: docker costs +4700s, the tunnel +554s - 89.5%
against 10.5%. The answer is recorded; the scaffolding does not need to ship.
Reverts d792519725, 1b28f87b37, 89b8a8c8d1, 0470491e1a, 529c5ace43,
dd3d10a9a2, c20dd0e5f5, 458d8956e1, 63ade1e03b, d281555945 and 4adfb6e3d2.
95cfc8b7b7 is already superseded by bbfd82f89, which sets the profiling limit
upstream.
One test asserts ansible_host lands on the given address, the other
that a missing member raises. Either would have caught the UNREACHABLE run
before it was dispatched.
The carriage returns a pty adds are invisible until a role parses its own
output, so the absence of the tty is asserted where the transport is
written rather than left to the next role that reads stdout.
One conflict, in the MCP exclusion list of requirement 035. Both sides
edited the same line: main added svc-net-tor-smtp, this branch replaced the
three svc-net-wireguard-{core,plain,firewalled} entries with the single
svc-net-wireguard it deploys. Resolved to the union of both — 24 entries —
which is what the merged tree actually holds.
Everything else merged clean and was checked for overlap rather than
assumed: no reference anywhere still names the three replaced roles;
matrix.py kept main's PRIMARY_APPS alongside the reordered round, and the
bootstrap pass does not need it, since only one template reads it and
group_vars defaults it empty; and main's ansible.cfg addition sits in
[defaults], so the ssh_args the mesh tool derives its own connection
settings from are untouched.
The pass that runs over the mesh reaches every host by SSH where the
pass before it used the container connection, and it pays that difference
once per task. Measured in a container: an exec through a warm multiplexer
costs 7ms, a fresh connection 85ms, and 165ms once sudo asks for the
password every task demands. The pass runs thousands of tasks.
Two settings threw the multiplexer away. A persist window of 20s closes the
master whenever a host waits longer than that for its next task, which in a
linear play is most of them; and the named ControlPath points at a directory
nothing creates, so a master that cannot open its socket reconnects per task
without reporting it.
Neither is retuned for the repository. The tool that moves a host onto the
mesh now writes that host's own ssh_args, derived from the repository's
value so every other option is kept and only the window and the socket
differ. A host the mesh never touches keeps exactly what it had.
Confirmed against the built command: a switched host gets ControlPersist=600s
and a ControlPath under the directory Ansible creates, while a host beside it
keeps the repository's own.
