Commit Graph
829 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5.5 2d7bf740b0 fix(ci): put containerd, Docker and swap on the runner's empty spare disk
The small GitHub runner shape (AWS, 72G root) ran out of disk in swarm and compose rows (#1329: WordPress#2 of run 41261531965, Nextcloud#6 and #7 of run 41262286962) while an empty instance-store disk nvme1n1 sat unused. mount_spare_disk.sh formats and mounts it at /mnt, but only on a GitHub-hosted runner outside act, and only a whole unpartitioned disk without any signature, mount or swap use of at least 75G; every decline leaves a notice. relocate_docker_dataroot.sh then bind-mounts /var/lib/containerd from the spare disk and moves the Docker data-root to /mnt/docker-host (not /mnt/docker, which is the nested daemon's volume that dedicated.sh wipes). It refuses before stopping anything when images, volumes, build cache, foreign containerd namespaces or shims exist or daemon.json is unreadable, and rolls back the bind mount and daemon.json on any later failure or cancel. Swap goes on / once containerd lives on /mnt, dump_state.sh logs disk size, model and free space per storage root, and the swarm watchdog samples / and /var/lib/containerd every 10 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:31:02 +01:00
kevinveenbirkenbach b0bacd072f Merge remote-tracking branch 'origin/main' into fix/web-app-stalwart/swarm-smtp-listener 2026-10-30 23:02:44 +01:00
kevinveenbirkenbachandClaude Opus 5.5 eb7056dafd fix(storage): keep the NFS client off the ports the platform publishes
Main CI run 40976399687, socialhome#0 (swarm, tor, centos): the CLI tests
of svc-net-tor (onion:25 "socks CONNECT failed with reply 0x01") and
web-app-stalwart (probe_mail relay: ConnectionRefusedError on port 25)
failed. dockerd on the manager had aborted the whole ingress program of
the mail stack:

  Failed to add ingress: failed to program ingress ports:
  failed to bind host port 0.0.0.0:993/tcp: address already in use

/proc/net/tcp named the squatter: 192.168.244.10:993 -> :2049, the
node's own NFS client. The mount carries no noresvport, so the kernel
drew a source port from the 665-1023 reserved window and landed on
imaps. None of 25/465/993/995/4190/8095 got a DNAT rule or a
docker-proxy listener while every other published port did, and
stalwart still reported healthy because its healthcheck runs inside the
container. 993 and 995 are the only published ports inside that window,
so this is luck of the draw on any NFS-backed swarm host that runs the
mail role.

mount_opts now emits noresvport. insecure on the kernel-flavor export is
the other half: kernel nfsd defaults to secure and answers a mount from
an unprivileged source port with EACCES. It reaches the kernel flavor
only; ganesha already accepts unprivileged clients.

What it costs: the next deploy of every existing NFS client sees a
pending opts change and remounts the shared state; a host-side job
holding that path at that instant fails loudly. A plain remount may keep
a running mount on its old reserved port until the next real mount.

Port of f46c4d0bab from feature/store-i18n, applied identically so the
later store-i18n -> main merge stays conflict-free.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 23:02:41 +01:00
kevinveenbirkenbachandClaude Opus 5.5 284b0873dd fix(sys-ctl-hlth-runtime): harden LAM CLI, catch apk options, drop vars comment
Addresses three CodeRabbit findings on PR #1265.

1. LAM hardening invisible to the probe (thread 4413764929): the runtime
   check probes PHP with `php -r`, the CLI SAPI. web-app-lam mounted the
   hardening ini only into /etc/php/8.4/apache2/conf.d, which the Debian
   CLI does not scan, so the probe read unhardened CLI settings and the
   check failed for LAM. The php_hardening entry now also mounts the ini
   into /etc/php/8.4/cli/conf.d. No other role mounts it into a Debian
   per-SAPI directory; the rest use the shared scan dir of the official
   image or of Alpine. The TODO entry now names the remaining gap: the
   probe reads only the CLI copy.

2. Explanatory comment above HEALTH_RUNTIME_FLOORS (thread 4413764949):
   removed. The README already states the end-of-life rule and now also
   defines a floor as the lowest patched release of each line.

3. `apk --no-cache add` missed (thread 4415018662): PACKAGE_INSTALL_PATTERN
   required `add` right after `apk`, so `RUN apk --no-cache add php7`
   passed as an image without PHP. The apk branch now accepts options
   before `add`, like the apt branch. New tests cover the option form and
   guard that `apk` in a path or a comment is no package install.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 08:57:21 +01:00
kevinveenbirkenbach ae497795ee Merge remote-tracking branch 'origin/main' into feature/sys-ctl-hlth-runtime/vulnerable-runtimes/1127 2026-10-29 18:20:17 +01:00
kevinveenbirkenbachandClaude Opus 5.5 d116dde659 fix(sys-ctl-hlth-runtime): bound the engine calls and catch hidden PHP runtimes
Three CodeRabbit findings on PR #1265:

- script.py listed the running containers with `container ps` and
  `container inspect` without a timeout, so a hung engine kept the daily
  unit active forever and blocked the next timer run. Both calls now use
  EXEC_TIMEOUT_SECONDS; main turns a timeout or engine error into a
  one-line failure and exit 1 instead of a traceback.

- runtime.py's PHP_PACKAGE_PATTERN requires a digit after "php", so an
  image installing an unversioned interpreter package (`php-cli`) and
  setting no PHP_VERSION was reported as PHP-free. The interpreter
  packages php, php-cli, php-fpm, php-cgi, php-phpdbg and
  libapache2-mod-php are now recorded as PhpRuntime.packages and reported
  with an unknown patch level; extension and library packages
  (php-mysql, php-pear, php-common, phpmyadmin) and paths do not count.

- php_finding returned early for a patched PHP_VERSION, so an end-of-life
  package line installed next to it (e.g. php7) was never reported. The
  version only returns a finding now; otherwise the lines are judged too.

Every finding has unit tests that fail on the previous code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 15:32:57 +01:00
kevinveenbirkenbachandClaude Opus 5.5 feac35a5ea fix(sys-ctl-hlth-runtime): deploy only on container hosts and drop the bare Docker Hub API literal
Two CI failures on PR #1265 (run 40834164415):

- Chunk 0 / SuiteCRM#2 (swarm): the post-load flush included
  sys-ctl-hlth-runtime on every play host, the NFS server too, which never
  gets sys-svc-container and therefore has no `container` CLI. The unit
  died with FileNotFoundError: 'container'. The role entry now runs only
  where a container runtime exists: in compose mode, or on swarm nodes
  ('svc-swarm-node' in group_names), the same role-entry host scoping that
  sys-ctl-hlth-csp and sys-ctl-hlth-webserver use with IS_STACK_HOST.
  Workers stay covered because PHP replicas can run there.

- Workspace (ubuntu): tests/external/repository/test_urls_reachable.py
  probed the literal "https://hub.docker.com/v2" from
  utils/docker/image/runtime.py:20 and got HTTP 404. The constant is gone;
  the endpoint is written as one f-string at the call site, the way
  utils/update/docker.py:104 and utils/update/source.py do, which the URL
  test skips as a template. With the scheme now visible in the literal,
  ruff reported the noqa: S310 on urllib.request.Request as unused.

Both test files that imported unittest twice (CodeQL) now use one
`from unittest import ...` form.

The run_once queue leak that spreads the stack host's post-load queue to
every host is fixed separately on the updater line (70ed8a2e98); when it
meets main, sys-ctl-hlth-runtime moves into sys-svc-container's queue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 11:47:41 +01:00
kevinveenbirkenbachandClaude Opus 5.5 3c743736a8 feat(sys-ctl-hlth-runtime): alert on EOL or unpatched PHP and missing PHP hardening
Answering "which deployed containers run a vulnerable PHP?" after
CVE-2026-109618/109619 took a manual inventory of registry metadata, and
nothing noticed a hardening ini that an image never loads.

- New role sys-ctl-hlth-runtime, routed through sys-svc-compose next to
  sys-ctl-hlth-container. A daily timer runs `php` in every running
  container and fails, alerting through sys-ctl-alm-compose, when PHP is
  below the floor of its line or on a line older than every floor (7.4,
  8.1). Every service that mounts php_hardening in meta/volumes.yml fails
  while expose_php or session.upload_progress.enabled is on; other PHP
  containers of such a stack are listed as info. Compose and swarm
  containers are matched by their project/stack and service labels. The
  check is not run at deploy time, so a finding never fails a deploy.
- The floors (8.2.35, 8.3.36, 8.4.27, 8.5.12) live once, in
  HEALTH_RUNTIME_FLOORS of the role's vars/main.yml.
- `make security-runtimes` lists the PHP of every pinned role image
  without pulling: Docker Hub tags through the Hub API, which no pull rate
  limit meters, other registries through the OCI config blob (new
  registry.config_blob, which the healthcheck probe now reuses).
  PHP_VERSION, distro PHP packages and a PHP_BASE build argument count. It
  exits 1 on an image below its floor or one the registry did not answer.
  Today it flags nfrastack/fusiondirectory:1.5 (line 7.4) and
  shopware/docker-base:8.3 (8.3.25).

The role TODO.md keeps the follow-ups: further runtimes, the split web
SAPI of web-app-lam, a scheduled CI run of the report and the CVE scan.

Refs #1127

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 00:12:25 +01:00
kevinveenbirkenbachandClaude Opus 5.5 1f8fc336e6 feat(proxy): per-app nosniff, Referrer-Policy, Permissions-Policy and HSTS directives
#1115 raised HSTS to one year but set no X-Content-Type-Options,
Referrer-Policy, Permissions-Policy or HSTS includeSubDomains/preload,
because none of them is safe globally: nosniff lets ORB block no-cors
responses typed text/html, includeSubDomains on the apex covers hosts
the platform does not serve, and a server-level add_header is dropped
in every location that sets its own.

WEBSERVER_SECURITY_HEADERS (group_vars/all/04_webserver.yml) now holds
the platform values, all opt-in: nosniff false, no Referrer-Policy, no
Permissions-Policy, HSTS max-age one year without includeSubDomains or
preload. A role overrides any key under security_headers in its
meta/server.yml, an inventory under
applications.<role>.server.security_headers.

The new security_headers filter merges both, rejects unknown keys,
wrong types, unknown Referrer-Policy tokens, non-printable header values
and preload without includeSubDomains or below a one-year max-age, and
composes the HSTS value.

- nosniff, Referrer-Policy and Permissions-Policy are set in the Lua
  header filter of content_security_policy.conf.j2, next to CSP, so a
  location's own add_header cannot drop them. Referrer-Policy is only
  set when the upstream sends none; Permissions-Policy only on HTML
  documents.
- HSTS stays the single server-level add_header in nginx_ssl_header.j2,
  now with the per-app value. A second emitter in Lua would duplicate
  the header. It stays inside the TLS branch, which onion vhosts never
  enter, so an onion vhost gets no HSTS even when its app opts in.

No role opts in yet, so rendered vhosts are unchanged.

Closes #1123

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 21:38:38 +01:00
kevinveenbirkenbach 62164178a2 Merge pull request #1204 from infinito-nexus/fix/web-app-matrix/probe-and-federation-delegation
fix(web-app-matrix): probe only served domains and serve the federation delegation
2026-10-24 13:04:32 +02:00
kevinveenbirkenbachandClaude Opus 5.5 bfd6f7c9b8 fix(install-lint): give chrome-headless-shell a font on dnf, yum and pacman
On a fresh Fedora dev container lint-mermaid failed most files with
ConnectionClosedError: the browser aborted with
`FATAL:third_party/skia/src/ports/SkFontMgr_FontConfigInterface.cpp:163]
Not implemented.` in its font fallback, because the host had no font at
all. install-lint provisions the browser's libraries by soname on dnf,
yum and pacman, and a font is not a soname; apt-get gets fonts through
puppeteer's --install-deps.

It looked load-dependent because lint-packages installs rpmlint in the
same parallel phase, which pulls fontconfig and Noto Sans: renders
crashed until that transaction landed, and every later run was green.
With every font removed, lint-mermaid alone fails 274 of 289 files the
same way.

ensure() now installs the distribution's font provide when
/usr/share/fonts holds no file (`font(:lang=en)` on dnf and yum,
`ttf-font` on pacman), and browser_installed() counts the font, so the
install-lint stamp is not fresh on a font-less host.

Closes #1154

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 12:59:42 +02:00
kevinveenbirkenbachandClaude Opus 5.5 7a82a92e84 fix(web-app-prometheus): probe only the domains an app serves itself
#1180 gave web-app-matrix a canonical `jitsi` domain for its own Jitsi
and listed it under `services.jitsi.domains`. When Jitsi is shared or
disabled nothing serves that domain, and the blackbox healthz job of
web-app-prometheus probed every canonical domain, so the probe of
https://jitsi.matrix.<domain>/healthz/ready failed with "tls:
unrecognized name" and `prometheus scrape: every consumer role reports
up=1 and passes its probe` went red (main run 40017166046: Matrix#2,
Nextcloud#12).

web_health_expectations already skipped such keys. The rule now lives
in utils/domains/unserved.py: a key claimed by a service entity that is
disabled, or enabled but shared, is not served by that app. The health
filter uses it unchanged, and prometheus.yml.j2 filters its targets
through the new `served_domains` filter, which also drops the
`<key>_onion` mirror of a skipped key.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 11:44:14 +02:00
kevinveenbirkenbachandClaude Opus 5.5 091c58307d fix(install): drop the shfmt docstrings that misstate ensure()
Three docstrings from #1173 claimed that ensure() leaves only the pinned
release on PATH or replaces whatever version PATH finds. It does neither:
it installs the release into /usr/local/bin and fails unless PATH then
resolves to it, so a distribution copy such as /usr/bin/shfmt stays on
disk behind it.

The test module and ensure() lose their docstrings; the module docstring
of the installer returns to the form its siblings use ("Install hadolint
via GitHub-release prebuilt binary.").

Reported by CodeRabbit on #1173.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 20:37:01 +02:00
kevinveenbirkenbachandClaude Opus 5.5 a9a12da75c fix(install): install the pinned shfmt release on every distro
`shfmt.ensure()` returned as soon as any `shfmt` was on PATH and tried
the distribution package before the release pinned in
roles/sys-lint/meta/services.yml. The Debian dev container therefore
formats with its packaged 3.8.0 (/usr/bin/shfmt), Fedora with its packaged
3.7.0, the CentOS one with the pinned v3.14.1 (/usr/local/bin/shfmt), and
they disagree on
scripts/tests/deploy/swarm/utils/unmount/nfs_mounts.sh: every gate in a
CentOS stack moved the `then` of the heredoc `if` below the terminator.

ensure() now follows ruff's installer: it reads `shfmt --version`, keeps
the binary only when it reports the pinned release, and otherwise installs
the release binary into /usr/local/bin. It fails when PATH still resolves
another version afterwards instead of formatting with it. The system
package path is gone.

nfs_mounts.sh is committed in the pinned release's format (byte-identical
to the v3.14.1 output from the CentOS container), so the first gate on any
distro leaves the tree clean.

The install-lint stamp now also depends on roles/sys-lint/meta/services.yml,
the pin file of actionlint, hadolint and shfmt, so a bumped pin reaches
environments whose stamp predates it; until now only pyproject.toml and
scripts/install/lint.sh invalidated it.

Fixes #1155

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 19:16:22 +02:00
kevinveenbirkenbachandClaude Opus 5.5 c65c5ce48c fix(update): pin only image tags the registry carries
7e82ba6d71 moved web-app-bigbluebutton's bbb-pads to v1.5.11, the tag
https://raw.githubusercontent.com/bigbluebutton/docker/develop/repos/tags
names for the bbb-pads source. alangecker/bbb-docker-pads has no such
image (Docker Hub: newest v1.5.8), so every BigBlueButton deploy pulls a
missing tag and test-external fails "These pinned Docker images do not
exist in their registry (HTTP 404): web-app-bigbluebutton/bbb-pads".

bbb-pads goes back to v1.5.8. The version-source updater now knows the
image a `version` pin tags (pulled directly or as the base of a custom
build) and withholds a bump when the registry answers 404 for the new
tag; an indeterminate answer does not block it, the same line
test-external draws. Run against main, the updater now reports no
update for bbb-pads instead of v1.5.11.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 10:30:10 +02:00
kevinveenbirkenbachandClaude Opus 5.5 d4b3609488 fix(install): refuse the upstream Node.js install on non-Linux hosts
`tarball_name()` picked the archive from `platform.machine()` alone, so
an end-of-life Node.js on Intel macOS would have unpacked a Linux build
into /usr/local, Homebrew's prefix there. The upstream install now
supports Linux only and stops with a message to install a supported
Node.js through the package manager everywhere else.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-21 16:52:56 +02:00
kevinveenbirkenbachandClaude Opus 5.5 30268cf552 fix(install): replace an end-of-life distro Node.js with the current LTS
Debian trixie ships nodejs 20.19, whose release line reached its end of
life on 2026-04-30. `@mermaid-js/mermaid-cli` 12 and its puppeteer
declare `node >=22.13.0`, so `make install-lint` failed in every fresh
Debian stack container with `All providers failed for
chrome-headless-shell` (#1088).

`ensure_node_present()` still installs the distro package first, then
reads the release line of `node` on PATH and looks it up in the
nodejs/Release schedule. A line past its `end` date is replaced by the
newest LTS from nodejs.org/dist/index.json: SHASUMS256.txt is verified
with gpgv against the active Node.js release keys from
nodejs/release-keys, the tarball against its SHA-256 entry, and the
release is unpacked into /usr/local, after which `node` on PATH must be
that release. Nothing pins a major: the next LTS line is picked up the
same way. Ubuntu 26.04's Node.js 22, supported until 2027-04-30, stays.

`ensure_npm_present()` calls it first, so a container that already holds
the end-of-life node is upgraded too rather than skipped because npm
exists. The call runs once per process.

Closes #1088

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-21 16:02:52 +02:00
kevinveenbirkenbachandClaude Opus 5.5 6e776a5780 refactor(python): clear the open CodeQL code-quality alerts
- Declare `__all__` in every package `__init__.py` whose `PROJECT_ROOT`
  is imported by its modules (163 packages), so the constant counts as
  exported; list every public name where the package defines more.
- Empty the 15 package `__init__.py` files whose `PROJECT_ROOT` no module
  imports.
- Declare the public API of `utils.github.variant.axes` in `__all__`,
  re-exports included, instead of silencing them with `noqa: F401`; the
  list matches the one `feature/store-i18n` already carries.
- Use `import unittest.mock as mock` in `test_applications.py`, as the
  rest of the suite does.
- Spell `_NON_PRINTABLE` in `utils/yaml_bootstrap.py` as the equivalent
  BMP-only positive class; it matches exactly the same 2111 code points
  as the old negated class across U+0000..U+10FFFF.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 21:40:58 +02:00
kevinveenbirkenbachandClaude Opus 5.5 21eba1e88f fix(backup): carry hard links, ACLs and xattrs on every backup and recover hop
baudolo 7.1.x writes volume generations with `rsync -aHAX`, so the
producing host keeps `user.*` extended attributes (OpenCloud keeps its
storage metadata in `user.oc.*`). Every later copy dropped them again,
because none of the other rsync calls passed `-X` (nor `-A`/`-H`):

- NFS export snapshots (svc-bkp-nfs-2-local `script.sh`),
- the backup host's pull (svc-bkp-remote-2-local `pull_specific_host.py`),
- the device sync (svc-bkp-local-2-device `script.py`),
- every recover into a live target (`utils/recovery/base.py`, used by the
  volume, NFS and device recover scripts),
- the `recover full` and `recover remote` tree pulls.

A disaster recovery through the documented chain therefore restored
OpenCloud without `user.oc.*`, silently: the copies succeeded and the
dirval stamp does not cover attributes.

All of these now run `rsync -aHAX` (`-aHAXP` for the device sync). A
target filesystem without xattr support makes rsync exit non-zero as soon
as a source file carries attributes, so the hop fails loudly instead of
dropping them; there is no per-device opt-out. The single-image pull in
`recover remote` keeps `-a`. The provider's ssh-wrapper needs no change:
its glob accepts any server flags and its character guard allows them.

Two tests write a `user.oc.id` attribute and assert it survives the NFS
snapshot and the recover; with the old `-a` both fail with
`OSError: [Errno 61] No data available`.

Closes #1021

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 18:20:46 +02:00
kevinveenbirkenbachandClaude Opus 5.5 b055eece05 feat(ci): keep GitHub-hosted runner labels current through the updater
The workflows pin `runs-on: ubuntu-NN.NN`, so no job moves to a new Ubuntu
release behind our back, but nothing raised the pin either. A sixth job of
`🔄 Update versions` closes that: `utils/update/runner.py` reads the image
table of actions/runner-images, takes the newest x64 Ubuntu label whose
image carries no preview or beta badge, and raises every older bare
`runs-on: ubuntu-NN.NN` to it; arch-suffixed labels stay untouched. The job
opens the change as one PR over `.github/workflows`, so its CI run tests
the new image before it reaches main.

This commit carries the updater's first result: every job now runs on
ubuntu-26.04 (49 `runs-on`).

actionlint 1.7.12, the newest release, does not know `ubuntu-26.04` yet
(rhysd/actionlint#682, fix proposed in rhysd/actionlint#743), so
`.github/actionlint.yaml` declares the label until a release knows it.

act maps each `ubuntu-NN.NN` label the workflows use to the local platform
image, derived from `.github/workflows`, instead of a fixed list that did
not know ubuntu-26.04.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 16:56:52 +02:00
kevinveenbirkenbachandClaude Opus 5.5 caec064e6b fix(env): reach ip_range through the handler registry
builder.py imported utils.env.handlers.infinito.ip_range directly, while
ip_range imports EnvBuilder from builder for its annotations. CodeQL
reported that as py/unsafe-cyclic-import on PR #999. The import only
runs under TYPE_CHECKING, so nothing breaks at runtime, but builder now
takes ip_range from utils.env.handlers like ORDERED_HANDLERS, the path
every other handler module already uses.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 09:46:18 +02:00
kevinveenbirkenbachandClaude Opus 5.5 d2b33a3b2a fix(compose): keep dynamic addresses off the stack's static IPs
compose/cache.shared.override.yml attaches every worktree stack's
infinito service to the primary's network (infinito-nexus-core_default,
172.30.0.0/24) without an ipv4_address. The default network declared
no ip_range, so Docker IPAM handed out the lowest free address and the
worktree stacks took 172.30.0.2, .3, .8, .9 and, whenever the primary
container was down, .10, the primary's INFINITO_IP4. The primary's
`make compose-up` then failed with "Address already in use", and with
it `make quality-high` on main.

The default network now declares ip_range: ${INFINITO_IP_RANGE}, the
upper half of INFINITO_SUBNET. Every static address of the stack (.1,
.4 to .6, .10, .53) sits in the lower half, so a container without a
static address can no longer take one.

INFINITO_IP_RANGE is derived from the final INFINITO_SUBNET after
custom.env is applied, because a worktree pins its subnet there and
`make dotenv-force` regenerates without it in the environment. Existing
worktrees therefore get their own slot's range without a new pin.

A lint checks that every static IP in default.env and every
172.30.${slot}.N pin in scripts/system/worktree/up.sh lies below the
pool.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-18 21:44:16 +02:00
kevinveenbirkenbach da449f8894 Merge pull request #323 from evangelostsak/feature/svc-net-wireguard
feat(svc-net-wireguard): Dockerized WireGuard role + test-e2e-cli DinD e2e
2026-10-18 19:50:42 +02:00
kevinveenbirkenbach 5601592872 Merge pull request #989 from infinito-nexus/feature/fuzz/yaml-bootstrap-differential
feat(fuzz): fuzz the bootstrap YAML reader against PyYAML weekly
2026-10-18 13:29:43 +02:00
evangelostsak 4a112c1295 chore(env): stop passing INFINITO_SWARM_TRANSPORT_BENCH through 2026-10-18 01:01:10 +02:00
evangelostsak 1e080c1a2e refactor(swarm-test): drop the transport bench from the mesh prologue 2026-10-18 01:01:10 +02:00
evangelostsak 5f945c2991 refactor(swarm-test): run the mesh playbooks from playbooks/ and drop the transport bench 2026-10-18 01:01:10 +02:00
kevinveenbirkenbachandClaude Opus 5.5 088c8be14f fix(ci): pull workspace images through the GHCR mirror like the deploy jobs
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.

The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.

call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 22:40:13 +02:00
evangelostsak 0c9ad653f7 Merge remote-tracking branch 'upstream/main' into feature/svc-net-wireguard 2026-10-17 22:19:57 +02:00
kevinveenbirkenbach 41e3146197 Merge remote-tracking branch 'origin/main' into feat/fuzz/yaml-bootstrap-differential 2026-10-17 20:41:07 +02:00
kevinveenbirkenbachandClaude Opus 5.5 0cb8bae88d perf(cache): let nested lookups reuse the unrendered applications view
Every cold render of the applications SPOT walks the templates of all
269 roles, and those templates call lookup('config'), lookup('tls'),
lookup('domain') and others. Each of these re-enters
get_merged_applications while the render guard is set, and each one
rebuilt the unrendered view from scratch: a deepcopy of all role
defaults plus the override merge, about 8 ms, never cached. A local
cold render of the development inventory made 351 such calls, 2.82 s
of a 6.86 s render.

The outer render now keeps a copy of its unrendered view and its cache
key on the render guard, taken before the inventory volumes are
popped, so it is the same view a nested call used to build. A nested
call with the same key returns it; any other key still builds its own.
The guard drops it when the render ends.

Measured on the same inventory: the nested calls took 0.07 s instead
of 2.82 s and the cold render 3.60 s instead of 6.86 s. The rendered
SPOT is unchanged: the old and new dumps differ in 69 user passwords,
which the development inventory regenerates on every read (two runs of
the old code differ in 68), and in one absolute checkout path, the only
other difference.
In CI each cold render took 40 to 77 s and the heaviest swarm cell of
#950 missed its budget by a few minutes (#984).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 19:52:34 +02:00
kevinveenbirkenbachandClaude Opus 5.5 32740d860f fix(utils): stop the bootstrap YAML reader from accepting what PyYAML rejects
utils/yaml_bootstrap.py promises that load_block(text) either equals
yaml.safe_load(text) or raises BootstrapYamlError. A fuzzer comparing
both found two inputs that broke the promise:

- str.splitlines() also breaks lines at \x0b, \x0c and \x1c to \x1e,
  which PyYAML's reader rejects as non-printable. "\x1e/: R" came back
  as {"/": "R"}, and a non-printable character inside a comment line was
  skipped with the comment.
- An indented " ---" on line 1 was taken as the document marker, while
  PyYAML reads it as a plain scalar and rejects the mapping that follows.

load_block now refuses any character outside PyYAML's own printable
class before it splits lines, and takes "---" as the document marker
only at column 0. The line breaks splitlines() still honours (\r, \r\n,
NEL, LS, PS) are line breaks for PyYAML too, so splitting stays as is.
The eight inputs join STRUCTURES in the divergence test; against the
previous reader, four of them diverge and fail it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 18:46:16 +02:00
evangelostsak e62ed8c56f Merge remote-tracking branch 'upstream/main' into feature/svc-net-wireguard 2026-10-17 09:34:00 +02:00
evangelostsak 99fcca4d1b feat(ci): compose titles carry the direct arm glyph
A compose row never joins a WireGuard mesh, so its title shows 🛣️ like a
direct swarm row. The vpn axis, the planner and the deploy are unchanged.
2026-10-17 00:03:18 +02:00
evangelostsak ec33add867 revert(env): stop passing INFINITO_COMPOSE_VPN through 2026-10-16 21:26:17 +02:00
evangelostsak 5d7327d911 revert(ci): compose titles carry no mesh glyph and take no mesh pin check 2026-10-16 21:26:16 +02:00
evangelostsak dc84a4c9ac revert(ci): the vpn axis covers swarm rows only again 2026-10-16 21:26:16 +02:00
kevinveenbirkenbachandClaude Opus 5.5 a58abd470c fix(cache): re-park the applications carrier after every nested app load
Job 116975298820 (Nextcloud#5, swarm, arch, btrfs, main 463df06202) hit
its budget (rc=124 after 20136 s). The app phase loads every service as a
nested `tasks/utils/load/app.yml` include under sys-service-loader, so
`tasks/utils/setup/role.yml` parks no carrier between `svc-virt-kata`
(L6073) and `update` (L38213). The only re-park in that stretch is the one
in `roles/sys-token-store/tasks/write.yml`, and Stalwart's user management
reaches it on the manager alone. After its last token write (L23574) the
manager carried the new key while both workers kept the old one and
rendered the full tree on every task: 85 non-wait tasks of 30 s or more, 63 min
from Keycloak (L25641) to the carry past `update` (L38237).

Park the carrier after every nested load on every host of the include.
A re-park whose key still matches serves the carried payload without
rendering.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 20:53:12 +02:00
evangelostsak dc991f97c0 refactor: drop rationale and narration from utils/tests/swarm/write/extras.py 2026-10-16 19:10:01 +02:00
evangelostsak 9b825c7096 refactor: drop rationale and narration from utils/tests/swarm/run.py 2026-10-16 19:10:01 +02:00
evangelostsak 8323786733 refactor: drop rationale and narration from utils/tests/swarm/mesh.py 2026-10-16 19:10:01 +02:00
evangelostsak eea44f1210 refactor: drop rationale and narration from utils/tests/swarm/matrix.py 2026-10-16 19:10:01 +02:00
evangelostsak 50c500eb6d refactor: drop rationale and narration from utils/tests/swarm/extend_inventory.py 2026-10-16 19:10:01 +02:00
evangelostsak 7f5382f3c8 refactor: drop rationale and narration from utils/packages/calls.py 2026-10-16 19:10:01 +02:00
evangelostsak 327ec01bf5 refactor: drop rationale and narration from utils/networks/mesh.py 2026-10-16 19:10:01 +02:00
evangelostsak 0b1bc82d57 refactor: drop rationale and narration from utils/github/variant/vpn.py 2026-10-16 19:10:01 +02:00
evangelostsak f5b2979c45 refactor: drop rationale and narration from utils/github/variant/selection.py 2026-10-16 19:10:01 +02:00
evangelostsak 216f6ea170 refactor: drop rationale and narration from utils/github/variant/label.py 2026-10-16 19:10:01 +02:00
evangelostsak aaecbccf8c refactor: drop rationale and narration from utils/github/variant/axes.py 2026-10-16 19:10:01 +02:00
evangelostsak 1860f587b4 feat(ci): show the mesh state on every compose title and refuse a mesh pin it cannot take
A compose row has two behaviours, with or without the mesh, but its title
showed three: no glyph on a role's larger variants, 🛣️ or 🚇 on its smallest.
No glyph and 🛣️ deploy the same thing, so every compose title now reads 🛣️
or 🚇 like a swarm title, and host titles stay without one. Only the smallest
variant still rotates onto the mesh.

A &vpn pin on a row no mode of which can take the mesh now aborts the run
instead of quietly deploying 🛣️, and a regular row pinned to the mesh only
rotates onto a mode that can carry it.
2026-10-16 14:48:36 +02:00