Commit Graph
145 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5.5 73a16bef09 perf(sys-svc-compose): prepare the compose instance in one module call
Every app's compose pass ran three remote tasks before compose up: create the directories, compose ps for running containers, and stat the pending marker (53 apps x ~2.7 s in a production run). compose_instance_prepare does all three in one call from 02_stack_host_prep.yml, sharing the directory creation with directory_batch through utils/directory_batch.py; compose ps runs only in compose mode. up.yml evaluates the registered result, named compose_instance_before_files because it predates 04_files: the running check is skipped whenever the compose, env or override file changed in this run (those changes notify compose-up themselves).

Replay harness old vs new: rendered files and the compose-up, marker and swarm handlers identical in every scenario; the only difference is the skipped running check when only the override changed, where compose-up still runs. About 85 s per full production run.

Merge note: store-i18n's a4c7fda974 hoists the removed marker stat into a block; resolve up.yml with this side.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-25 21:22:32 +01:00
kevinveenbirkenbachandClaude Opus 5.5 afb82f57a1 feat(web-app-dashboard): let a role name the canonical domain its card opens
The dashboard card of every app opened its primary domain, so Matrix led to the Synapse status page and Stalwart to its admin UI. A role can now declare services.dashboard.card: <key of meta/domains.yml canonical>; docker_cards builds the card URL with lookup('canonical_url', app, key), the Apps menu reuses it, and roles_with_service('dashboard') hands the same domain to the tile e2e spec. Matrix sets card: element, Stalwart card: webmail. Without card nothing changes, and the prometheus/matomo target lists are untouched. A lint requires the key to resolve in the role's own domains.canonical.

The Matrix ansible flavor also sets Synapse's web_client_location to Element through matrix_synapse_configuration_extension_yaml, as the compose flavor already does, so matrix.<domain>/ redirects there. Merge note: feature/iam-profile-rbac defines the same extension variable inside the OIDC block; the next merge must combine both into one block (agreed with that session).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-25 21:01:06 +01:00
kevinveenbirkenbachandClaude Opus 5.5 7b2dfcfac7 fix(tls): serve a redirect to another host through the redirect role
resolve_term follows a redirect source to its target so an alias inherits the TLS settings of the application it points at. A redirect to a domain another host serves ends that chain on a domain this host does not know, and the tls and cert lookups failed: galaxyserver full-1 stopped in web-opt-rdr-domains on 'tls: domain project.infinito.nexus not found' for its project.veen.world -> project.infinito.nexus redirect.

When the chain ends outside the host's domains and web-opt-rdr-domains is deployed, the source now resolves to web-opt-rdr-domains with the source itself as primary domain, which is the role that serves its vhost and whose certificate bundle names it. Only the outermost call falls back, so the source asked for stays the primary domain across a longer chain; without the redirect role, or for a target this host serves, nothing changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-25 11:04:28 +01:00
kevinveenbirkenbach 9d08fda117 Merge remote-tracking branch 'origin/main' into feature/updater-lockdown-probe 2026-10-24 15:09:52 +02:00
kevinveenbirkenbachandClaude Opus 5.5 d3b9ae59a9 fix(sys-token-store): let the users lookup read the store of the stack host
Lookups render on the controller and read FILE_TOKENS there, while sys-token-store writes the store on STACK_HOST and mirrors it to the controller only in swarm. A compose deploy run from another machine therefore resolved every token empty: on echoserver Friendica rendered an empty SMTP_AUTH_PASS (msmtp: authentication method PLAIN needs a password) and Stalwart minted new bot app passwords on every run.

get_merged_users and the applications carrier key now take the sys_token_store_users fact, which the play reads from the stack host, over the controller file; write.yml refreshes the fact after each write, and 01_load.yml slurps from STACK_HOST so every host reads the store that is written. Without the fact, as in tests and ad-hoc calls, the controller file still applies, and where controller and stack host share the file the merged users are identical.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 15:01:05 +02:00
kevinveenbirkenbachandClaude Opus 5.5 bfd6f7c9b8 fix(install-lint): give chrome-headless-shell a font on dnf, yum and pacman
On a fresh Fedora dev container lint-mermaid failed most files with
ConnectionClosedError: the browser aborted with
`FATAL:third_party/skia/src/ports/SkFontMgr_FontConfigInterface.cpp:163]
Not implemented.` in its font fallback, because the host had no font at
all. install-lint provisions the browser's libraries by soname on dnf,
yum and pacman, and a font is not a soname; apt-get gets fonts through
puppeteer's --install-deps.

It looked load-dependent because lint-packages installs rpmlint in the
same parallel phase, which pulls fontconfig and Noto Sans: renders
crashed until that transaction landed, and every later run was green.
With every font removed, lint-mermaid alone fails 274 of 289 files the
same way.

ensure() now installs the distribution's font provide when
/usr/share/fonts holds no file (`font(:lang=en)` on dnf and yum,
`ttf-font` on pacman), and browser_installed() counts the font, so the
install-lint stamp is not fresh on a font-less host.

Closes #1154

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 12:59:42 +02:00
kevinveenbirkenbachandClaude Opus 5.5 57eb64e0af fix(cache): render a templated DOMAIN_PRIMARY against the play variables
get_merged_domains rendered DOMAIN_PRIMARY with no templar and an empty variable scope. A primary declared as a template, such as echoserver's "{{ SOFTWARE_NAME | lower }}", therefore came out as 'none' inside the applications SPOT, while direct lookups saw the domain Ansible had already resolved. Every SPOT value built through it named file.none, and the dashboard waited for https://file.none/assets/img/logo.png until full-2 failed. The render now gets the templar and the play variables, which also covers services.tor.node.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-24 00:40:32 +02:00
kevinveenbirkenbachandClaude Opus 5.5 9152c884d8 fix(tls): serve every redirect source the certificate that names it
The cert lookup handed an alias or redirect source the live directory of its app's canonical domain or of its target, so social.cymais.cloud got the infinito.nexus lineage and failed with 'Hostname mismatch'. With the san and dedicated flavors a domain term now resolves to its own live/<domain>, which cert_link_domains points at the lineage that covers it; application terms keep the canonical path. resolve_term follows redirect chains and fails on a cycle.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:14:21 +02:00
kevinveenbirkenbachandClaude Opus 5.5 fb18077f57 perf(utils): key the domain index by the applications' domains only
build_domain_index fingerprinted the whole applications tree on every call, 77 % of a tls lookup. The index reads only each application's domains, so the cache key now covers exactly those; a tls lookup drops from 8.2 to 3.1 ms.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 20:03:40 +02:00
kevinveenbirkenbachandClaude Opus 5.5 c728b4427b fix(tls): resolve a redirect source through its target
With AUTO_BUILD_ALIASES enabled, current_play_redirect_domains adds
<entity>.<DOMAIN_PRIMARY> as a redirect source for every application whose
canonical domain differs. web-opt-rdr-domains then asks the tls and cert
lookups about each source, but resolve_term only knew the domains mapping
and applications[*].domains, so the Keycloak stage on echoserver stopped
with "tls: domain 'funkwhale.infinito.nexus' not found".

redirect_targets() turns CURRENT_PLAY_REDIRECT_DOMAINS into a source to
target map, and resolve_term falls back to it, one hop, only where it
raised before, so every term that resolved keeps its answer. The tls and
cert lookups pass the map.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 16:08:18 +02:00
kevinveenbirkenbachandClaude Opus 5.5 96fe8bd7e6 feat(playbooks/update): pull the images of the deploy before the upgrade window
pull.yml pulls, on the target host, every pinned image of the applications the host deploys, the same closure the update check judges, and lists the images a role builds itself instead. It never touches a container, retries each pull a bounded number of times and fails loudly with the image and the registry error. The update_check filter moves next to the new update_pull filter into plugins/filter/update/, and both read the code pins from one helper.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 21:30:18 +02:00
kevinveenbirkenbach 4a86376aeb Merge remote-tracking branch 'origin/main' into feature/updater 2026-10-22 21:08:26 +02:00
kevinveenbirkenbachandClaude Opus 5.5 091c58307d fix(install): drop the shfmt docstrings that misstate ensure()
Three docstrings from #1173 claimed that ensure() leaves only the pinned
release on PATH or replaces whatever version PATH finds. It does neither:
it installs the release into /usr/local/bin and fails unless PATH then
resolves to it, so a distribution copy such as /usr/bin/shfmt stays on
disk behind it.

The test module and ensure() lose their docstrings; the module docstring
of the installer returns to the form its siblings use ("Install hadolint
via GitHub-release prebuilt binary.").

Reported by CodeRabbit on #1173.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 20:37:01 +02:00
kevinveenbirkenbachandClaude Opus 5.5 a9a12da75c fix(install): install the pinned shfmt release on every distro
`shfmt.ensure()` returned as soon as any `shfmt` was on PATH and tried
the distribution package before the release pinned in
roles/sys-lint/meta/services.yml. The Debian dev container therefore
formats with its packaged 3.8.0 (/usr/bin/shfmt), Fedora with its packaged
3.7.0, the CentOS one with the pinned v3.14.1 (/usr/local/bin/shfmt), and
they disagree on
scripts/tests/deploy/swarm/utils/unmount/nfs_mounts.sh: every gate in a
CentOS stack moved the `then` of the heredoc `if` below the terminator.

ensure() now follows ruff's installer: it reads `shfmt --version`, keeps
the binary only when it reports the pinned release, and otherwise installs
the release binary into /usr/local/bin. It fails when PATH still resolves
another version afterwards instead of formatting with it. The system
package path is gone.

nfs_mounts.sh is committed in the pinned release's format (byte-identical
to the v3.14.1 output from the CentOS container), so the first gate on any
distro leaves the tree clean.

The install-lint stamp now also depends on roles/sys-lint/meta/services.yml,
the pin file of actionlint, hadolint and shfmt, so a bumped pin reaches
environments whose stamp predates it; until now only pyproject.toml and
scripts/install/lint.sh invalidated it.

Fixes #1155

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 19:16:22 +02:00
kevinveenbirkenbach 2d7f842613 Merge branch 'main' of github.com:infinito-nexus/core into feature/updater 2026-10-22 11:27:47 +02:00
kevinveenbirkenbachandClaude Opus 5.5 05135a7cee feat(svc-net-wireguard): let a compose deployment name a hub with roaming spokes
Without a swarm manager the mesh planner only built full meshes, whose
handshake gate waits for every peer, so laptops behind NAT could not join.
An inventory may now name a hub in networks.hub: the planner builds every
declared mesh around it, the hub forwards between its spokes, a spoke marked
networks.roaming gets no endpoint in the hub's peer list and dials the hub
with PersistentKeepalive, and the hub's gate skips roaming spokes while a
roaming spoke gates only on the hub. The mesh CLI takes the hub's public
address per host (--endpoint) and a vault password per host
(--host-vault-password-file). Full meshes and the swarm manager hub plan
exactly as before; NETWORK_MESH_POOL keeps the operator's subnet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 10:56:50 +02:00
kevinveenbirkenbachandClaude Opus 5.5 c65c5ce48c fix(update): pin only image tags the registry carries
7e82ba6d71 moved web-app-bigbluebutton's bbb-pads to v1.5.11, the tag
https://raw.githubusercontent.com/bigbluebutton/docker/develop/repos/tags
names for the bbb-pads source. alangecker/bbb-docker-pads has no such
image (Docker Hub: newest v1.5.8), so every BigBlueButton deploy pulls a
missing tag and test-external fails "These pinned Docker images do not
exist in their registry (HTTP 404): web-app-bigbluebutton/bbb-pads".

bbb-pads goes back to v1.5.8. The version-source updater now knows the
image a `version` pin tags (pulled directly or as the base of a custom
build) and withholds a bump when the registry answers 404 for the new
tag; an indeterminate answer does not block it, the same line
test-external draws. Run against main, the updater now reports no
update for bbb-pads instead of v1.5.11.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 10:30:10 +02:00
kevinveenbirkenbachandClaude Opus 5.5 d4b3609488 fix(install): refuse the upstream Node.js install on non-Linux hosts
`tarball_name()` picked the archive from `platform.machine()` alone, so
an end-of-life Node.js on Intel macOS would have unpacked a Linux build
into /usr/local, Homebrew's prefix there. The upstream install now
supports Linux only and stops with a message to install a supported
Node.js through the package manager everywhere else.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-21 16:52:56 +02:00
kevinveenbirkenbachandClaude Opus 5.5 30268cf552 fix(install): replace an end-of-life distro Node.js with the current LTS
Debian trixie ships nodejs 20.19, whose release line reached its end of
life on 2026-04-30. `@mermaid-js/mermaid-cli` 12 and its puppeteer
declare `node >=22.13.0`, so `make install-lint` failed in every fresh
Debian stack container with `All providers failed for
chrome-headless-shell` (#1088).

`ensure_node_present()` still installs the distro package first, then
reads the release line of `node` on PATH and looks it up in the
nodejs/Release schedule. A line past its `end` date is replaced by the
newest LTS from nodejs.org/dist/index.json: SHASUMS256.txt is verified
with gpgv against the active Node.js release keys from
nodejs/release-keys, the tarball against its SHA-256 entry, and the
release is unpacked into /usr/local, after which `node` on PATH must be
that release. Nothing pins a major: the next LTS line is picked up the
same way. Ubuntu 26.04's Node.js 22, supported until 2027-04-30, stays.

`ensure_npm_present()` calls it first, so a container that already holds
the end-of-life node is upgraded too rather than skipped because npm
exists. The call runs once per process.

Closes #1088

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-21 16:02:52 +02:00
kevinveenbirkenbachandClaude Opus 5.5 6e776a5780 refactor(python): clear the open CodeQL code-quality alerts
- Declare `__all__` in every package `__init__.py` whose `PROJECT_ROOT`
  is imported by its modules (163 packages), so the constant counts as
  exported; list every public name where the package defines more.
- Empty the 15 package `__init__.py` files whose `PROJECT_ROOT` no module
  imports.
- Declare the public API of `utils.github.variant.axes` in `__all__`,
  re-exports included, instead of silencing them with `noqa: F401`; the
  list matches the one `feature/store-i18n` already carries.
- Use `import unittest.mock as mock` in `test_applications.py`, as the
  rest of the suite does.
- Spell `_NON_PRINTABLE` in `utils/yaml_bootstrap.py` as the equivalent
  BMP-only positive class; it matches exactly the same 2111 code points
  as the old negated class across U+0000..U+10FFFF.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 21:40:58 +02:00
kevinveenbirkenbachandClaude Opus 5.5 21eba1e88f fix(backup): carry hard links, ACLs and xattrs on every backup and recover hop
baudolo 7.1.x writes volume generations with `rsync -aHAX`, so the
producing host keeps `user.*` extended attributes (OpenCloud keeps its
storage metadata in `user.oc.*`). Every later copy dropped them again,
because none of the other rsync calls passed `-X` (nor `-A`/`-H`):

- NFS export snapshots (svc-bkp-nfs-2-local `script.sh`),
- the backup host's pull (svc-bkp-remote-2-local `pull_specific_host.py`),
- the device sync (svc-bkp-local-2-device `script.py`),
- every recover into a live target (`utils/recovery/base.py`, used by the
  volume, NFS and device recover scripts),
- the `recover full` and `recover remote` tree pulls.

A disaster recovery through the documented chain therefore restored
OpenCloud without `user.oc.*`, silently: the copies succeeded and the
dirval stamp does not cover attributes.

All of these now run `rsync -aHAX` (`-aHAXP` for the device sync). A
target filesystem without xattr support makes rsync exit non-zero as soon
as a source file carries attributes, so the hop fails loudly instead of
dropping them; there is no per-device opt-out. The single-image pull in
`recover remote` keeps `-a`. The provider's ssh-wrapper needs no change:
its glob accepts any server flags and its character guard allows them.

Two tests write a `user.oc.id` attribute and assert it survives the NFS
snapshot and the recover; with the old `-a` both fail with
`OSError: [Errno 61] No data available`.

Closes #1021

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 18:20:46 +02:00
kevinveenbirkenbachandClaude Opus 5.5 b055eece05 feat(ci): keep GitHub-hosted runner labels current through the updater
The workflows pin `runs-on: ubuntu-NN.NN`, so no job moves to a new Ubuntu
release behind our back, but nothing raised the pin either. A sixth job of
`🔄 Update versions` closes that: `utils/update/runner.py` reads the image
table of actions/runner-images, takes the newest x64 Ubuntu label whose
image carries no preview or beta badge, and raises every older bare
`runs-on: ubuntu-NN.NN` to it; arch-suffixed labels stay untouched. The job
opens the change as one PR over `.github/workflows`, so its CI run tests
the new image before it reaches main.

This commit carries the updater's first result: every job now runs on
ubuntu-26.04 (49 `runs-on`).

actionlint 1.7.12, the newest release, does not know `ubuntu-26.04` yet
(rhysd/actionlint#682, fix proposed in rhysd/actionlint#743), so
`.github/actionlint.yaml` declares the label until a release knows it.

act maps each `ubuntu-NN.NN` label the workflows use to the local platform
image, derived from `.github/workflows`, instead of a fixed list that did
not know ubuntu-26.04.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 16:56:52 +02:00
kevinveenbirkenbachandClaude Opus 5.5 d2b33a3b2a fix(compose): keep dynamic addresses off the stack's static IPs
compose/cache.shared.override.yml attaches every worktree stack's
infinito service to the primary's network (infinito-nexus-core_default,
172.30.0.0/24) without an ipv4_address. The default network declared
no ip_range, so Docker IPAM handed out the lowest free address and the
worktree stacks took 172.30.0.2, .3, .8, .9 and, whenever the primary
container was down, .10, the primary's INFINITO_IP4. The primary's
`make compose-up` then failed with "Address already in use", and with
it `make quality-high` on main.

The default network now declares ip_range: ${INFINITO_IP_RANGE}, the
upper half of INFINITO_SUBNET. Every static address of the stack (.1,
.4 to .6, .10, .53) sits in the lower half, so a container without a
static address can no longer take one.

INFINITO_IP_RANGE is derived from the final INFINITO_SUBNET after
custom.env is applied, because a worktree pins its subnet there and
`make dotenv-force` regenerates without it in the environment. Existing
worktrees therefore get their own slot's range without a new pin.

A lint checks that every static IP in default.env and every
172.30.${slot}.N pin in scripts/system/worktree/up.sh lies below the
pool.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-18 21:44:16 +02:00
kevinveenbirkenbach da449f8894 Merge pull request #323 from evangelostsak/feature/svc-net-wireguard
feat(svc-net-wireguard): Dockerized WireGuard role + test-e2e-cli DinD e2e
2026-10-18 19:50:42 +02:00
kevinveenbirkenbach 5601592872 Merge pull request #989 from infinito-nexus/feature/fuzz/yaml-bootstrap-differential
feat(fuzz): fuzz the bootstrap YAML reader against PyYAML weekly
2026-10-18 13:29:43 +02:00
kevinveenbirkenbachandClaude Opus 5.5 088c8be14f fix(ci): pull workspace images through the GHCR mirror like the deploy jobs
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.

The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.

call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 22:40:13 +02:00
evangelostsak 0c9ad653f7 Merge remote-tracking branch 'upstream/main' into feature/svc-net-wireguard 2026-10-17 22:19:57 +02:00
kevinveenbirkenbach 41e3146197 Merge remote-tracking branch 'origin/main' into feat/fuzz/yaml-bootstrap-differential 2026-10-17 20:41:07 +02:00
kevinveenbirkenbachandClaude Opus 5.5 0cb8bae88d perf(cache): let nested lookups reuse the unrendered applications view
Every cold render of the applications SPOT walks the templates of all
269 roles, and those templates call lookup('config'), lookup('tls'),
lookup('domain') and others. Each of these re-enters
get_merged_applications while the render guard is set, and each one
rebuilt the unrendered view from scratch: a deepcopy of all role
defaults plus the override merge, about 8 ms, never cached. A local
cold render of the development inventory made 351 such calls, 2.82 s
of a 6.86 s render.

The outer render now keeps a copy of its unrendered view and its cache
key on the render guard, taken before the inventory volumes are
popped, so it is the same view a nested call used to build. A nested
call with the same key returns it; any other key still builds its own.
The guard drops it when the render ends.

Measured on the same inventory: the nested calls took 0.07 s instead
of 2.82 s and the cold render 3.60 s instead of 6.86 s. The rendered
SPOT is unchanged: the old and new dumps differ in 69 user passwords,
which the development inventory regenerates on every read (two runs of
the old code differ in 68), and in one absolute checkout path, the only
other difference.
In CI each cold render took 40 to 77 s and the heaviest swarm cell of
#950 missed its budget by a few minutes (#984).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 19:52:34 +02:00
kevinveenbirkenbachandClaude Opus 5.5 32740d860f fix(utils): stop the bootstrap YAML reader from accepting what PyYAML rejects
utils/yaml_bootstrap.py promises that load_block(text) either equals
yaml.safe_load(text) or raises BootstrapYamlError. A fuzzer comparing
both found two inputs that broke the promise:

- str.splitlines() also breaks lines at \x0b, \x0c and \x1c to \x1e,
  which PyYAML's reader rejects as non-printable. "\x1e/: R" came back
  as {"/": "R"}, and a non-printable character inside a comment line was
  skipped with the comment.
- An indented " ---" on line 1 was taken as the document marker, while
  PyYAML reads it as a plain scalar and rejects the mapping that follows.

load_block now refuses any character outside PyYAML's own printable
class before it splits lines, and takes "---" as the document marker
only at column 0. The line breaks splitlines() still honours (\r, \r\n,
NEL, LS, PS) are line breaks for PyYAML too, so splitting stays as is.
The eight inputs join STRUCTURES in the divergence test; against the
previous reader, four of them diverge and fail it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 18:46:16 +02:00
evangelostsak e62ed8c56f Merge remote-tracking branch 'upstream/main' into feature/svc-net-wireguard 2026-10-17 09:34:00 +02:00
evangelostsak b3a9e140af test(ci): the compose title carries the direct arm glyph 2026-10-17 00:03:18 +02:00
evangelostsak 9c93e32a73 revert(ci): the axis tests cover the swarm-only vpn axis again 2026-10-16 21:26:16 +02:00
evangelostsak 779e995bf3 refactor: drop rationale and narration from tests/unit/python/utils/tests/swarm/test_write_extras.py 2026-10-16 19:10:01 +02:00
evangelostsak 2a60fd1f32 refactor: drop rationale and narration from tests/unit/python/utils/tests/swarm/test_mesh_steps.py 2026-10-16 19:10:01 +02:00
evangelostsak 2698bb7996 refactor: drop rationale and narration from tests/unit/python/utils/github/variant/test_axes.py 2026-10-16 19:10:01 +02:00
evangelostsak d4179b747a test(ci): compose titles state their mesh and impossible mesh pins abort 2026-10-16 14:48:36 +02:00
kevinveenbirkenbachandClaude Opus 5.5 67f6c8d7e8 fix(cache): keep the applications carrier valid across token store writes
Since 778c18fb23 the carrier key holds the tokens file's mtime and size.
sys-token-store changed that stat on every inclusion (00_core.yml
touched the file) and on every persisted token, and nothing re-parked a
carrier afterwards. Every later task's fresh worker then rendered the
applications SPOT cold, 30 to 80 s each. On main fea6f31d9f (run
38812964308) seven heavy jobs logged 194 to 287 such tasks and ran out
of the 20400 s budget; the same jobs before 778c18fb23 logged 65 to 83,
all of them real waits.

- 00_core.yml touches the store with modification_time and access_time
  preserved, so it only creates a missing file. In swarm the controller
  sees the store over NFS with attribute caching, and the touch alone
  made the second pass go cold without any write.
- write.yml re-parks the carrier when the persist or the controller
  mirror changed the store. Both run once, so every host that ran the
  include gets the same verdict.
- tasks/utils/setup/role.yml re-parks after every role on every host,
  so hosts that did not run the writing role recover too. Where the
  key still matches this serves the carried payload; measured 0.1 s per
  role in a local compose deploy.
- The tokens file enters the key as the SHA-256 of its content,
  memoised per inode, mtime and size, so a touch no longer changes the
  key. A stat that still matches serves the memo (1.67 us against
  1.41 us for the old stat signature), and a stale stat keeps key and
  YAML cache on the same old content instead of pairing new content
  with an old key.
- applications_carrier opens the tokens file before it computes the
  key, so on the NFS-mounted controller the parked key reflects the
  revalidated attributes rather than a cached stat that changes a
  minute later.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 11:11:47 +02:00
evangelostsak e00008070d Merge remote-tracking branch 'upstream/main' into feature/svc-net-wireguard 2026-10-16 08:58:56 +02:00
evangelostsak a473d4e03d test(ci): the vpn axis rotates on smallest compose variants only 2026-10-16 03:40:16 +02:00
evangelostsak 759bb237b4 test(mesh): hub, spoke and full-mesh members listen and dial as planned 2026-10-16 03:40:15 +02:00
kevinveenbirkenbachandClaude Opus 5.5 fad0c60e52 fix(lint): provision mmdc's browser libraries off apt and fail hard
On the Fedora, CentOS Stream and Manjaro workspace jobs install-lint
ran `puppeteer browsers install chrome-headless-shell --install-deps`
as root, and puppeteer rejected it with "apt-get does not seem to be
available". The installer only warned ("optional lint tool"), so
install-lint passed while mmdc had no browser.

--install-deps is now passed only where apt-get exists. Everywhere else
the libraries `ldd` reports missing are installed by soname: dnf and
yum through the RPM provide `<soname>()(64bit)`, pacman through the
first package `pacman -F usr/lib/<soname>` names (libgbm.so.1 is owned
by both mesa and mesa-amber, which conflict). browser_installed() also
requires every library to resolve, so the install-lint stamp stays
stale until the browser can actually load.

A failed npm install, a missing mmdc, a failed or timed-out browser
download and unresolved libraries now raise instead of warning.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 23:45:54 +02:00
kevinveenbirkenbachandClaude Opus 5.5 e6f552ca0a fix(install): provision minitest next to ruby on Arch and the RedHat family
The Manjaro workspace job failed test-unit-ruby with "cannot load such
file -- minitest". Debian vendors minitest in libruby, but Arch ships it
as ruby-minitest and Fedora and CentOS Stream as rubygem-minitest, none
of which ruby pulls in. ensure_ruby_present now probes
`require "minitest"` and installs the distro package when the probe
fails, then fails loudly if minitest is still missing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 23:45:54 +02:00
evangelostsak 1c5ba164f2 Merge remote-tracking branch 'upstream/main' into feature/svc-net-wireguard
No conflicts across 121 incoming commits. The one file both sides changed is
tasks/stages/01_constructor.yml: upstream parks an applications carrier for
the swarm bootstrap roles after the add_host tasks, and keys the carrier on
the tokens file as well. This branch's mesh bootstrap opens and drops its own
block-scoped carrier before that point, so both keep working as written.

Upstream also brings ruff 0.17.0, distro-derived env keys, a scoped act outer
teardown, the stalwart relay rule, the onion apex redirect and image and
version updates; ruff 0.17 reformats nothing on this branch.
2026-10-15 21:27:11 +02:00
kevinveenbirkenbach c38f95457b Merge pull request #955 from infinito-nexus/fix/constructor/bootstrap-carrier/925
perf(constructor): carry the applications SPOT into the bootstrap roles
2026-10-15 17:54:46 +02:00
kevinveenbirkenbachandClaude Opus 5.5 33bb7e99e9 fix(lint): resolve puppeteer-core like node and keep other browser revisions
- browser_revision() reads revisions.js from the puppeteer-core that
  node resolves for mmdc's puppeteer: puppeteer/node_modules first, then
  the node_modules directories above it. When mermaid-cli pins another
  puppeteer-core, npm nests puppeteer's own copy, and the sibling held a
  different revision.
- A failed browser install removes only the directories of the pinned
  revision before the retry. Removing the whole chrome-headless-shell
  cache also deleted browsers that other puppeteer installs share.

Addresses the CodeRabbit review on #958.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 11:23:22 +02:00
kevinveenbirkenbach 706745162b Merge remote-tracking branch 'origin/main' into fix/lint/mermaid-browser-revision 2026-10-15 10:36:09 +02:00
kevinveenbirkenbachandClaude Opus 5.5 c74a31e83e fix(lint): install the browser revision mmdc's own puppeteer pins
install-lint provisioned chrome-headless-shell with `npx --yes
puppeteer browsers install`, which runs the latest puppeteer from the
registry. mmdc runs with the puppeteer npm installed next to it as a
peer dependency when mmdc itself was installed, and looks for exactly
the revision that puppeteer-core pins. Once puppeteer 25.14.0 was out,
npx installed 155.0.8059.79 while mmdc with puppeteer-core 25.13.0
wanted 155.0.8059.39, and lint-mermaid failed on every file. Because
browser_installed() accepted any chrome-headless-shell, a stack with
the wrong revision never repaired itself.

The browser is now installed through the puppeteer CLI next to mmdc,
which installs its puppeteer-core's revision, and only that revision,
read from puppeteer-core's revisions.js, counts as installed.

Closes #957

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 03:21:30 +02:00
kevinveenbirkenbachandClaude Opus 5.5 778c18fb23 perf(constructor): carry the applications SPOT into the bootstrap roles
The constructor parked no applications carrier between the merge block
and the token store, so every lookup in the svc-swarm-node, svc-storage,
svc-registry and svc-virt bootstrap rendered the SPOT cold, 25 to 30 s
each on a loaded host.

A carrier there was unsafe: in swarm, svc-storage-nfs-server mounts the
NFS export over the controller's DIR_VAR_LIB, so the tokens file the
render reads can appear in the middle of that section, and the cache key
did not notice. merged_applications_cache_key now includes the tokens
file's path, mtime and size, as the users cache key already did, so a
carrier rejects itself once the tokens change. A carrier parked after
the add_host tasks then serves the bootstrap roles. It is parked only in
swarm mode: in compose the section runs in about a second, so a carrier
there would add a render without saving one.

In a swarm drill of web-app-prometheus up to the known NFS probe-write
failure, "Resolve the NFS server backend flavor" went from 29.6 s to
0.07 s, "Probe-mount the NFS export" from 26.4 s to 0.4 s, "Mount
cluster-shared infinito state on controller" from 23.7 s to 0.5 s and
"Ensure Docker-In-Docker DNS works" from 30.0 s to 3.6 s; the run to
that point took 768 s instead of 1735 s.

Closes #925

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 03:21:17 +02:00
kevinveenbirkenbachandClaude Opus 5.5 47565781ea style(python): import datetime as dt for ruff 0.17
ruff 0.17.0 adds `datetime` -> `dt` to the default aliases of
flake8-import-conventions, and the project selects ICN, so lint-python
failed with ICN001 in seven files.

The files now import `datetime as dt`. ruff's unsafe fix also renamed
two local names in utils/github/playwright_summary.py into the alias:
the `_to_iso_z` parameter and the parsed value in
`_parse_iso_timestamp`. Both are now called `moment`; with the bare fix,
nine tests of test_playwright_summary.py failed with UnboundLocalError
and AttributeError.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 01:24:31 +02:00