followed_tags caught OSError, ValueError, KeyError and RuntimeError, but
http.client.IncompleteRead, BadStatusLine and LineTooLong derive from
http.client.HTTPException only. One truncated digest response therefore
escaped to collect_updates (docker.py:344, no try) and aborted the whole
Docker update run instead of printing the follow warning for that image.
Every other fetch site of the updater already catches HTTPException
(docker.py, pip.py).
Reported by CodeRabbit on #1370.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Update run 41722395224 proposed no sys-os bump although fedora:latest
points at 45 and a local dry run proposes 44 -> 45. followed_tags turned
every failed digest lookup into "no candidate" without a word, so the run
cannot tell a rate limit from a pin that is current.
tag_digest now retries an HTTP 429 twice with a short backoff, as the tag
listing does, and followed_tags reports on stderr what it compared and
which tags it kept, or the error that made it propose nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runtime image pins lived in meta/distros.yml, which neither the docker
updater, the image mirror nor the vulnerability scan reads, so a pin would
only ever move by hand. The new role sys-os carries them in its
meta/services.yml under the distro id; utils/distros.py renders the
workspace matrix from there with the stdlib block reader, and
meta/distros.yml keeps the distro attributes only.
A new release now arrives as an updater pull request. follow: latest holds
debian, ubuntu and fedora to the release their latest tag points at, by
Docker Hub digest: Ubuntu moves between LTS releases only (26.10 shares the
digest of rolling, not latest) and Fedora skips rawhide (46 shares the
digest of rawhide, latest is 45). Arch stays rolling; CentOS stream tags
carry no number the updater can order.
dockerhub_repo and is_dockerhub move to utils/docker/image/ref.py beside
split_registry_and_name, so utils/update/follow.py can use them without an
import cycle and utils/update/docker.py stays under the size cap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A truncated chunked OSV download raises http.client.IncompleteRead, an
HTTPException rather than an OSError, so it skipped the .part cleanup and
the error line; the handler now catches both. The runtime check's warn
summary no longer claims the mirror scan tracks every warning, since host
findings and images outside the mirror never reach it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A suppression entry under <service>.security.suppressions now names either
one CVE (id) or one image tag (image_tag), each with until. An image_tag
entry accepts every finding of the service's image while the owner runs that
tag; each owner of a shared digest is judged by its own tag (Config.Image at
runtime, the pinned version in the mirror scan), so a container on any other
tag stays unsuppressed. Host suppressions are the id entries of the engine
services; an image_tag entry there covers the engine's own image.
The cve lint requires image_tag to equal the service's version, and the
Docker image updater drops a service's image_tag entries with their markers
when it moves the service to another version, so a tag suppression never
outlives its tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scan threshold becomes a severity-to-verdict mapping in
roles/sys-ctl-hlth-runtime/meta/services.yml, CRITICAL: fail and HIGH: warn,
for production and CI alike. A finding mapped to fail fails the service and
the deploy, a warn prints a warning line and passes, a suppressed one prints
an info line. Engine errors, missing databases and PHP hardening violations
still fail. The old list form is rejected with a config error. The daily
mirror scan stays strict: every unsuppressed fail or warn finding is a row,
so it fails while an image is red.
Every run publishes one list of judged findings four ways:
- report.json of the latest run (summary + records) under
DIR_HEALTH_RUNTIME_REPORT, written atomically;
- findings.jsonl under DIR_HEALTH_RUNTIME_LOG, one record per finding,
appended per run and rotated weekly by a logrotate drop-in;
- one JSON line per target plus a summary line on stdout for the journal;
- infinito_hlth_runtime.prom in DIR_METRICS_TEXTFILE when web-app-prometheus
is deployed: findings per target, kind, image, severity, scanner and
verdict (no CVE label), last run time and success, scanner errors.
Each record carries run_id, timestamp, host, environment, target, kind,
image, scanner, id, severity, package, installed_version, fixed_version,
suppressed, suppressed_until and verdict. The paths are SPOT variables in
group_vars/all/05_paths.yml, the script takes them as arguments.
Requirement 070 and the role README describe the mapping, the outputs and
the per-image vulnerability issue of the previous commit.
Part of #1324
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodeRabbit review of PR #1326:
- fetch_databases downloads each OSV archive to all.zip.part and replaces
all.zip only after the transfer completes. An OSError removes the partial
file and comes back as an `osv <ecosystem>: <error>` line instead of a
traceback.
- run_gh bounds every gh call with a 120 s timeout. A failed or hung create,
edit or close becomes a `failed <action>: <error>` line while the other
issues and the close pass still run, and the command exits 1.
- The suppression lint checks every list item of a suppressions block, so an
entry that starts with `until:` still needs its `# nocheck: cve <reason>`
marker.
Part of #1324
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The small GitHub runner shape (AWS, 72G root) ran out of disk in swarm and compose rows (#1329: WordPress#2 of run 41261531965, Nextcloud#6 and #7 of run 41262286962) while an empty instance-store disk nvme1n1 sat unused. mount_spare_disk.sh formats and mounts it at /mnt, but only on a GitHub-hosted runner outside act, and only a whole unpartitioned disk without any signature, mount or swap use of at least 75G; every decline leaves a notice. relocate_docker_dataroot.sh then bind-mounts /var/lib/containerd from the spare disk and moves the Docker data-root to /mnt/docker-host (not /mnt/docker, which is the nested daemon's volume that dedicated.sh wipes). It refuses before stopping anything when images, volumes, build cache, foreign containerd namespaces or shims exist or daemon.json is unreadable, and rolls back the bind mount and daemon.json on any later failure or cancel. Swap goes on / once containerd lives on /mnt, dump_state.sh logs disk size, model and free space per storage root, and the swarm watchdog samples / and /var/lib/containerd every 10 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under act the deploy job pinned an engine but skipped the cache restores, so
every local swarm-* run stopped on a missing Grype database. The swarm
matrix also ran inside the act job container, where the shared host
directory did not exist, and the workspace job would have kept a copy of
about 5 GB in every worktree.
- scripts/tests/deploy/act/workflow.sh requires
INFINITO_VULNERABILITY_DB_HOST_PATH, creates it on the host and binds it
into the job container at the same path, because that container drives
the host daemon, which resolves every stack and swarm node bind on the
host.
- call-test-deploy.yml pins engine and database path in one step that runs
only outside act; under act the deploy draws an engine and refreshes the
shared directory. call-test-workspace.yml takes the shared path under act.
- The refresh lock moves into the shared directory, so host deploys and act
runs serialize on one file, and a root refresh hands its files to the
directory owner. hand_over skips paths a mocked download never created,
which failed four TestRefresh cases when the gate ran as root.
- A missing INFINITO_VULNERABILITY_DB_HOST_PATH stops compose up with a
"run 'make dotenv'" hint instead of a KeyError.
- Rationale paragraphs in the database and compose docstrings are dropped.
Part of #1324
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each CI deploy row now draws one vulnerability engine (trivy, grype or osv)
as a fifth rotation axis, shown in the job name next to distro and
filesystem (🔬 axis, 🔺 trivy, 🍇 grype, 🐙 osv). The deploy-time scan of
sys-ctl-hlth-runtime runs that engine behind Syft and the Grype runtime pass.
- A vulnerability-db job keeps one Actions cache per database and day and
downloads only the missing ones. Every deploy job restores Grype plus its
drawn engine, fails on a cache miss and never downloads a database itself.
- Local stacks share one host directory, INFINITO_VULNERABILITY_DB_HOST_PATH,
mounted read-only at INFINITO_VULNERABILITY_DB_DIR in compose.yml and in
every swarm node. A local deploy without a pinned engine draws one and
refreshes only the databases it reads, at most once a day, under a lock.
- A pinned engine whose database is missing stops the deploy before it
starts.
Part of #1324
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The runtime check judged PHP by its upstream version and failed every
container on an end-of-life line. Distribution builds backport fixes without
raising that number, so main run 41154895895 failed Checkmk#0 on
`PHP 8.1.2-1ubuntu2.28 runs the end-of-life line 8.1`, a build Ubuntu lists
as fixed for CVE-2026-109618 and CVE-2026-109619.
- The role now runs Trivy, Grype and OSV-Scanner, with a Syft SBOM per
target, against the host and the images of the running containers, in
bounded parallel lanes. Targets, engines and the failing severities are
lists in meta/services.yml; the default is every target and engine at
CRITICAL and HIGH with a fix available.
- Grype always scans the binary-classified packages of the SBOM, so a PHP
built from source in an official php image (8.4.26) still fails on the
CVEs fixed in 8.4.27, which Trivy and OSV do not see.
- OSV reads an SBOM copy whose distro purls carry the source package name.
- A CVE is suppressed only in the role owning the image, under
services.<service>.security.suppressions with id, until and a
`# nocheck: cve <reason>` marker; a lint rejects bare, malformed and
expired entries.
- The deploy-time run fails on its first finding: sys-service gains
system_service_run_once and sys-service-terminator skips the retries
only for those units. The timer runs daily at 00:00 UTC.
- The PHP version floors are removed; the php_hardening check stays.
- make test-security replaces make security-runtimes: Syft builds an SBOM
of every pinned image straight from the GHCR mirror, cached by digest,
and the three engines scan it with the role's threshold and suppressions.
Part of #1324
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main CI run 40976399687, socialhome#0 (swarm, tor, centos): the CLI tests
of svc-net-tor (onion:25 "socks CONNECT failed with reply 0x01") and
web-app-stalwart (probe_mail relay: ConnectionRefusedError on port 25)
failed. dockerd on the manager had aborted the whole ingress program of
the mail stack:
Failed to add ingress: failed to program ingress ports:
failed to bind host port 0.0.0.0:993/tcp: address already in use
/proc/net/tcp named the squatter: 192.168.244.10:993 -> :2049, the
node's own NFS client. The mount carries no noresvport, so the kernel
drew a source port from the 665-1023 reserved window and landed on
imaps. None of 25/465/993/995/4190/8095 got a DNAT rule or a
docker-proxy listener while every other published port did, and
stalwart still reported healthy because its healthcheck runs inside the
container. 993 and 995 are the only published ports inside that window,
so this is luck of the draw on any NFS-backed swarm host that runs the
mail role.
mount_opts now emits noresvport. insecure on the kernel-flavor export is
the other half: kernel nfsd defaults to secure and answers a mount from
an unprivileged source port with EACCES. It reaches the kernel flavor
only; ganesha already accepts unprivileged clients.
What it costs: the next deploy of every existing NFS client sees a
pending opts change and remounts the shared state; a host-side job
holding that path at that instant fails loudly. A plain remount may keep
a running mount on its old reserved port until the next real mount.
Port of f46c4d0bab from feature/store-i18n, applied identically so the
later store-i18n -> main merge stays conflict-free.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rsync 3.5.0 (Debian security update 3.5.0+ds1-0+deb13u1) refuses a
destination symlink owned by neither root nor the calling user.
svc-opt-ssd-hdd turns a volume's `_data` into a symlink to /mnt/hdd or
/mnt/ssd. The restore drill empties the volume and runs `compose create`;
docker repopulates it and lchowns the `_data` symlink to the image
directory's owner (e.g. www-data). The restore then failed with
"refusing to follow a symlink owned by an untrusted user" and
"change_dir#1 .../_data/ failed: Too many levels of symbolic links",
the drill left its subjects down, and the dependent checks timed out
(CI run 40979252521, ssd-hdd#0, on feature/store-i18n; recover.py is
identical on main).
VolumeRecovery now resolves the mountpoint before handing it to rsync:
os.path.realpath locally, `ssh <host> realpath -e -- <mountpoint>` for
--docker-host, so rsync writes into the real directory and never meets
the symlink. `--insecure-links` was not used because older rsync builds
on other distributions lack it. Two new unit tests (local symlinked
mountpoint, remote resolution); both fail against the previous code.
Root cause and fix by the i18n session (feature/store-i18n triage of
ssd-hdd#0), ported to main.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses three CodeRabbit findings on PR #1265.
1. LAM hardening invisible to the probe (thread 4413764929): the runtime
check probes PHP with `php -r`, the CLI SAPI. web-app-lam mounted the
hardening ini only into /etc/php/8.4/apache2/conf.d, which the Debian
CLI does not scan, so the probe read unhardened CLI settings and the
check failed for LAM. The php_hardening entry now also mounts the ini
into /etc/php/8.4/cli/conf.d. No other role mounts it into a Debian
per-SAPI directory; the rest use the shared scan dir of the official
image or of Alpine. The TODO entry now names the remaining gap: the
probe reads only the CLI copy.
2. Explanatory comment above HEALTH_RUNTIME_FLOORS (thread 4413764949):
removed. The README already states the end-of-life rule and now also
defines a floor as the lowest patched release of each line.
3. `apk --no-cache add` missed (thread 4415018662): PACKAGE_INSTALL_PATTERN
required `add` right after `apk`, so `RUN apk --no-cache add php7`
passed as an image without PHP. The apk branch now accepts options
before `add`, like the apt branch. New tests cover the option form and
guard that `apk` in a path or a comment is no package install.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three CodeRabbit findings on PR #1265:
- script.py listed the running containers with `container ps` and
`container inspect` without a timeout, so a hung engine kept the daily
unit active forever and blocked the next timer run. Both calls now use
EXEC_TIMEOUT_SECONDS; main turns a timeout or engine error into a
one-line failure and exit 1 instead of a traceback.
- runtime.py's PHP_PACKAGE_PATTERN requires a digit after "php", so an
image installing an unversioned interpreter package (`php-cli`) and
setting no PHP_VERSION was reported as PHP-free. The interpreter
packages php, php-cli, php-fpm, php-cgi, php-phpdbg and
libapache2-mod-php are now recorded as PhpRuntime.packages and reported
with an unknown patch level; extension and library packages
(php-mysql, php-pear, php-common, phpmyadmin) and paths do not count.
- php_finding returned early for a patched PHP_VERSION, so an end-of-life
package line installed next to it (e.g. php7) was never reported. The
version only returns a finding now; otherwise the lines are judged too.
Every finding has unit tests that fail on the previous code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two CI failures on PR #1265 (run 40834164415):
- Chunk 0 / SuiteCRM#2 (swarm): the post-load flush included
sys-ctl-hlth-runtime on every play host, the NFS server too, which never
gets sys-svc-container and therefore has no `container` CLI. The unit
died with FileNotFoundError: 'container'. The role entry now runs only
where a container runtime exists: in compose mode, or on swarm nodes
('svc-swarm-node' in group_names), the same role-entry host scoping that
sys-ctl-hlth-csp and sys-ctl-hlth-webserver use with IS_STACK_HOST.
Workers stay covered because PHP replicas can run there.
- Workspace (ubuntu): tests/external/repository/test_urls_reachable.py
probed the literal "https://hub.docker.com/v2" from
utils/docker/image/runtime.py:20 and got HTTP 404. The constant is gone;
the endpoint is written as one f-string at the call site, the way
utils/update/docker.py:104 and utils/update/source.py do, which the URL
test skips as a template. With the scheme now visible in the literal,
ruff reported the noqa: S310 on urllib.request.Request as unused.
Both test files that imported unittest twice (CodeQL) now use one
`from unittest import ...` form.
The run_once queue leak that spreads the stack host's post-load queue to
every host is fixed separately on the updater line (70ed8a2e98); when it
meets main, sys-ctl-hlth-runtime moves into sys-svc-container's queue.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answering "which deployed containers run a vulnerable PHP?" after
CVE-2026-109618/109619 took a manual inventory of registry metadata, and
nothing noticed a hardening ini that an image never loads.
- New role sys-ctl-hlth-runtime, routed through sys-svc-compose next to
sys-ctl-hlth-container. A daily timer runs `php` in every running
container and fails, alerting through sys-ctl-alm-compose, when PHP is
below the floor of its line or on a line older than every floor (7.4,
8.1). Every service that mounts php_hardening in meta/volumes.yml fails
while expose_php or session.upload_progress.enabled is on; other PHP
containers of such a stack are listed as info. Compose and swarm
containers are matched by their project/stack and service labels. The
check is not run at deploy time, so a finding never fails a deploy.
- The floors (8.2.35, 8.3.36, 8.4.27, 8.5.12) live once, in
HEALTH_RUNTIME_FLOORS of the role's vars/main.yml.
- `make security-runtimes` lists the PHP of every pinned role image
without pulling: Docker Hub tags through the Hub API, which no pull rate
limit meters, other registries through the OCI config blob (new
registry.config_blob, which the healthcheck probe now reuses).
PHP_VERSION, distro PHP packages and a PHP_BASE build argument count. It
exits 1 on an image below its floor or one the registry did not answer.
Today it flags nfrastack/fusiondirectory:1.5 (line 7.4) and
shopware/docker-base:8.3 (8.3.25).
The role TODO.md keeps the follow-ups: further runtimes, the split web
SAPI of web-app-lam, a scheduled CI run of the report and the CVE scan.
Refs #1127
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a fresh Fedora dev container lint-mermaid failed most files with
ConnectionClosedError: the browser aborted with
`FATAL:third_party/skia/src/ports/SkFontMgr_FontConfigInterface.cpp:163]
Not implemented.` in its font fallback, because the host had no font at
all. install-lint provisions the browser's libraries by soname on dnf,
yum and pacman, and a font is not a soname; apt-get gets fonts through
puppeteer's --install-deps.
It looked load-dependent because lint-packages installs rpmlint in the
same parallel phase, which pulls fontconfig and Noto Sans: renders
crashed until that transaction landed, and every later run was green.
With every font removed, lint-mermaid alone fails 274 of 289 files the
same way.
ensure() now installs the distribution's font provide when
/usr/share/fonts holds no file (`font(:lang=en)` on dnf and yum,
`ttf-font` on pacman), and browser_installed() counts the font, so the
install-lint stamp is not fresh on a font-less host.
Closes#1154
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Three docstrings from #1173 claimed that ensure() leaves only the pinned
release on PATH or replaces whatever version PATH finds. It does neither:
it installs the release into /usr/local/bin and fails unless PATH then
resolves to it, so a distribution copy such as /usr/bin/shfmt stays on
disk behind it.
The test module and ensure() lose their docstrings; the module docstring
of the installer returns to the form its siblings use ("Install hadolint
via GitHub-release prebuilt binary.").
Reported by CodeRabbit on #1173.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`shfmt.ensure()` returned as soon as any `shfmt` was on PATH and tried
the distribution package before the release pinned in
roles/sys-lint/meta/services.yml. The Debian dev container therefore
formats with its packaged 3.8.0 (/usr/bin/shfmt), Fedora with its packaged
3.7.0, the CentOS one with the pinned v3.14.1 (/usr/local/bin/shfmt), and
they disagree on
scripts/tests/deploy/swarm/utils/unmount/nfs_mounts.sh: every gate in a
CentOS stack moved the `then` of the heredoc `if` below the terminator.
ensure() now follows ruff's installer: it reads `shfmt --version`, keeps
the binary only when it reports the pinned release, and otherwise installs
the release binary into /usr/local/bin. It fails when PATH still resolves
another version afterwards instead of formatting with it. The system
package path is gone.
nfs_mounts.sh is committed in the pinned release's format (byte-identical
to the v3.14.1 output from the CentOS container), so the first gate on any
distro leaves the tree clean.
The install-lint stamp now also depends on roles/sys-lint/meta/services.yml,
the pin file of actionlint, hadolint and shfmt, so a bumped pin reaches
environments whose stamp predates it; until now only pyproject.toml and
scripts/install/lint.sh invalidated it.
Fixes#1155
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
7e82ba6d71 moved web-app-bigbluebutton's bbb-pads to v1.5.11, the tag
https://raw.githubusercontent.com/bigbluebutton/docker/develop/repos/tags
names for the bbb-pads source. alangecker/bbb-docker-pads has no such
image (Docker Hub: newest v1.5.8), so every BigBlueButton deploy pulls a
missing tag and test-external fails "These pinned Docker images do not
exist in their registry (HTTP 404): web-app-bigbluebutton/bbb-pads".
bbb-pads goes back to v1.5.8. The version-source updater now knows the
image a `version` pin tags (pulled directly or as the base of a custom
build) and withholds a bump when the registry answers 404 for the new
tag; an indeterminate answer does not block it, the same line
test-external draws. Run against main, the updater now reports no
update for bbb-pads instead of v1.5.11.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`tarball_name()` picked the archive from `platform.machine()` alone, so
an end-of-life Node.js on Intel macOS would have unpacked a Linux build
into /usr/local, Homebrew's prefix there. The upstream install now
supports Linux only and stops with a message to install a supported
Node.js through the package manager everywhere else.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Debian trixie ships nodejs 20.19, whose release line reached its end of
life on 2026-04-30. `@mermaid-js/mermaid-cli` 12 and its puppeteer
declare `node >=22.13.0`, so `make install-lint` failed in every fresh
Debian stack container with `All providers failed for
chrome-headless-shell` (#1088).
`ensure_node_present()` still installs the distro package first, then
reads the release line of `node` on PATH and looks it up in the
nodejs/Release schedule. A line past its `end` date is replaced by the
newest LTS from nodejs.org/dist/index.json: SHASUMS256.txt is verified
with gpgv against the active Node.js release keys from
nodejs/release-keys, the tarball against its SHA-256 entry, and the
release is unpacked into /usr/local, after which `node` on PATH must be
that release. Nothing pins a major: the next LTS line is picked up the
same way. Ubuntu 26.04's Node.js 22, supported until 2027-04-30, stays.
`ensure_npm_present()` calls it first, so a container that already holds
the end-of-life node is upgraded too rather than skipped because npm
exists. The call runs once per process.
Closes#1088
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Declare `__all__` in every package `__init__.py` whose `PROJECT_ROOT`
is imported by its modules (163 packages), so the constant counts as
exported; list every public name where the package defines more.
- Empty the 15 package `__init__.py` files whose `PROJECT_ROOT` no module
imports.
- Declare the public API of `utils.github.variant.axes` in `__all__`,
re-exports included, instead of silencing them with `noqa: F401`; the
list matches the one `feature/store-i18n` already carries.
- Use `import unittest.mock as mock` in `test_applications.py`, as the
rest of the suite does.
- Spell `_NON_PRINTABLE` in `utils/yaml_bootstrap.py` as the equivalent
BMP-only positive class; it matches exactly the same 2111 code points
as the old negated class across U+0000..U+10FFFF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
baudolo 7.1.x writes volume generations with `rsync -aHAX`, so the
producing host keeps `user.*` extended attributes (OpenCloud keeps its
storage metadata in `user.oc.*`). Every later copy dropped them again,
because none of the other rsync calls passed `-X` (nor `-A`/`-H`):
- NFS export snapshots (svc-bkp-nfs-2-local `script.sh`),
- the backup host's pull (svc-bkp-remote-2-local `pull_specific_host.py`),
- the device sync (svc-bkp-local-2-device `script.py`),
- every recover into a live target (`utils/recovery/base.py`, used by the
volume, NFS and device recover scripts),
- the `recover full` and `recover remote` tree pulls.
A disaster recovery through the documented chain therefore restored
OpenCloud without `user.oc.*`, silently: the copies succeeded and the
dirval stamp does not cover attributes.
All of these now run `rsync -aHAX` (`-aHAXP` for the device sync). A
target filesystem without xattr support makes rsync exit non-zero as soon
as a source file carries attributes, so the hop fails loudly instead of
dropping them; there is no per-device opt-out. The single-image pull in
`recover remote` keeps `-a`. The provider's ssh-wrapper needs no change:
its glob accepts any server flags and its character guard allows them.
Two tests write a `user.oc.id` attribute and assert it survives the NFS
snapshot and the recover; with the old `-a` both fail with
`OSError: [Errno 61] No data available`.
Closes#1021
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workflows pin `runs-on: ubuntu-NN.NN`, so no job moves to a new Ubuntu
release behind our back, but nothing raised the pin either. A sixth job of
`🔄 Update versions` closes that: `utils/update/runner.py` reads the image
table of actions/runner-images, takes the newest x64 Ubuntu label whose
image carries no preview or beta badge, and raises every older bare
`runs-on: ubuntu-NN.NN` to it; arch-suffixed labels stay untouched. The job
opens the change as one PR over `.github/workflows`, so its CI run tests
the new image before it reaches main.
This commit carries the updater's first result: every job now runs on
ubuntu-26.04 (49 `runs-on`).
actionlint 1.7.12, the newest release, does not know `ubuntu-26.04` yet
(rhysd/actionlint#682, fix proposed in rhysd/actionlint#743), so
`.github/actionlint.yaml` declares the label until a release knows it.
act maps each `ubuntu-NN.NN` label the workflows use to the local platform
image, derived from `.github/workflows`, instead of a fixed list that did
not know ubuntu-26.04.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
compose/cache.shared.override.yml attaches every worktree stack's
infinito service to the primary's network (infinito-nexus-core_default,
172.30.0.0/24) without an ipv4_address. The default network declared
no ip_range, so Docker IPAM handed out the lowest free address and the
worktree stacks took 172.30.0.2, .3, .8, .9 and, whenever the primary
container was down, .10, the primary's INFINITO_IP4. The primary's
`make compose-up` then failed with "Address already in use", and with
it `make quality-high` on main.
The default network now declares ip_range: ${INFINITO_IP_RANGE}, the
upper half of INFINITO_SUBNET. Every static address of the stack (.1,
.4 to .6, .10, .53) sits in the lower half, so a container without a
static address can no longer take one.
INFINITO_IP_RANGE is derived from the final INFINITO_SUBNET after
custom.env is applied, because a worktree pins its subnet there and
`make dotenv-force` regenerates without it in the environment. Existing
worktrees therefore get their own slot's range without a new pin.
A lint checks that every static IP in default.env and every
172.30.${slot}.N pin in scripts/system/worktree/up.sh lies below the
pool.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.
The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.
call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every cold render of the applications SPOT walks the templates of all
269 roles, and those templates call lookup('config'), lookup('tls'),
lookup('domain') and others. Each of these re-enters
get_merged_applications while the render guard is set, and each one
rebuilt the unrendered view from scratch: a deepcopy of all role
defaults plus the override merge, about 8 ms, never cached. A local
cold render of the development inventory made 351 such calls, 2.82 s
of a 6.86 s render.
The outer render now keeps a copy of its unrendered view and its cache
key on the render guard, taken before the inventory volumes are
popped, so it is the same view a nested call used to build. A nested
call with the same key returns it; any other key still builds its own.
The guard drops it when the render ends.
Measured on the same inventory: the nested calls took 0.07 s instead
of 2.82 s and the cold render 3.60 s instead of 6.86 s. The rendered
SPOT is unchanged: the old and new dumps differ in 69 user passwords,
which the development inventory regenerates on every read (two runs of
the old code differ in 68), and in one absolute checkout path, the only
other difference.
In CI each cold render took 40 to 77 s and the heaviest swarm cell of
#950 missed its budget by a few minutes (#984).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
utils/yaml_bootstrap.py promises that load_block(text) either equals
yaml.safe_load(text) or raises BootstrapYamlError. A fuzzer comparing
both found two inputs that broke the promise:
- str.splitlines() also breaks lines at \x0b, \x0c and \x1c to \x1e,
which PyYAML's reader rejects as non-printable. "\x1e/: R" came back
as {"/": "R"}, and a non-printable character inside a comment line was
skipped with the comment.
- An indented " ---" on line 1 was taken as the document marker, while
PyYAML reads it as a plain scalar and rejects the mapping that follows.
load_block now refuses any character outside PyYAML's own printable
class before it splits lines, and takes "---" as the document marker
only at column 0. The line breaks splitlines() still honours (\r, \r\n,
NEL, LS, PS) are line breaks for PyYAML too, so splitting stays as is.
The eight inputs join STRUCTURES in the divergence test; against the
previous reader, four of them diverge and fail it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Since 778c18fb23 the carrier key holds the tokens file's mtime and size.
sys-token-store changed that stat on every inclusion (00_core.yml
touched the file) and on every persisted token, and nothing re-parked a
carrier afterwards. Every later task's fresh worker then rendered the
applications SPOT cold, 30 to 80 s each. On main fea6f31d9f (run
38812964308) seven heavy jobs logged 194 to 287 such tasks and ran out
of the 20400 s budget; the same jobs before 778c18fb23 logged 65 to 83,
all of them real waits.
- 00_core.yml touches the store with modification_time and access_time
preserved, so it only creates a missing file. In swarm the controller
sees the store over NFS with attribute caching, and the touch alone
made the second pass go cold without any write.
- write.yml re-parks the carrier when the persist or the controller
mirror changed the store. Both run once, so every host that ran the
include gets the same verdict.
- tasks/utils/setup/role.yml re-parks after every role on every host,
so hosts that did not run the writing role recover too. Where the
key still matches this serves the carried payload; measured 0.1 s per
role in a local compose deploy.
- The tokens file enters the key as the SHA-256 of its content,
memoised per inode, mtime and size, so a touch no longer changes the
key. A stat that still matches serves the memo (1.67 us against
1.41 us for the old stat signature), and a stale stat keeps key and
YAML cache on the same old content instead of pairing new content
with an old key.
- applications_carrier opens the tokens file before it computes the
key, so on the NFS-mounted controller the parked key reflects the
revalidated attributes rather than a cached stat that changes a
minute later.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
