Three docstrings from #1173 claimed that ensure() leaves only the pinned
release on PATH or replaces whatever version PATH finds. It does neither:
it installs the release into /usr/local/bin and fails unless PATH then
resolves to it, so a distribution copy such as /usr/bin/shfmt stays on
disk behind it.
The test module and ensure() lose their docstrings; the module docstring
of the installer returns to the form its siblings use ("Install hadolint
via GitHub-release prebuilt binary.").
Reported by CodeRabbit on #1173.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`shfmt.ensure()` returned as soon as any `shfmt` was on PATH and tried
the distribution package before the release pinned in
roles/sys-lint/meta/services.yml. The Debian dev container therefore
formats with its packaged 3.8.0 (/usr/bin/shfmt), Fedora with its packaged
3.7.0, the CentOS one with the pinned v3.14.1 (/usr/local/bin/shfmt), and
they disagree on
scripts/tests/deploy/swarm/utils/unmount/nfs_mounts.sh: every gate in a
CentOS stack moved the `then` of the heredoc `if` below the terminator.
ensure() now follows ruff's installer: it reads `shfmt --version`, keeps
the binary only when it reports the pinned release, and otherwise installs
the release binary into /usr/local/bin. It fails when PATH still resolves
another version afterwards instead of formatting with it. The system
package path is gone.
nfs_mounts.sh is committed in the pinned release's format (byte-identical
to the v3.14.1 output from the CentOS container), so the first gate on any
distro leaves the tree clean.
The install-lint stamp now also depends on roles/sys-lint/meta/services.yml,
the pin file of actionlint, hadolint and shfmt, so a bumped pin reaches
environments whose stamp predates it; until now only pyproject.toml and
scripts/install/lint.sh invalidated it.
Fixes#1155
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
02_provision_inventory.sh copies the manager's host_vars to every node,
the backup host included, so a row with network=multi handed bkp-01
NETWORK_MODE=multi as well. The backup host is deployed from its own
backup.yml, which has no svc-swarm-node group and no Tor provider, so
tor_serves_host is false there and resolve_node_mode raised at the first
tls lookup: "NETWORK_MODE 'multi' needs svc-net-tor with a node onion on
this node" (01_constructor.yml, pixelfed swarm drill on the multi axis,
after a green deploy round and Playwright pass).
extend_inventory already prepares the backup host's own inventory after
the copy; it now also drops NETWORK_MODE from that host's host_vars, so
the backup host derives clearnet as a host without Tor does. The
remote-2-local pull reaches the manager over SSH and needs no onion.
Rows on clearnet or without a mode resolve as before; only the tor and
multi rows, which raised, change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
7e82ba6d71 moved web-app-bigbluebutton's bbb-pads to v1.5.11, the tag
https://raw.githubusercontent.com/bigbluebutton/docker/develop/repos/tags
names for the bbb-pads source. alangecker/bbb-docker-pads has no such
image (Docker Hub: newest v1.5.8), so every BigBlueButton deploy pulls a
missing tag and test-external fails "These pinned Docker images do not
exist in their registry (HTTP 404): web-app-bigbluebutton/bbb-pads".
bbb-pads goes back to v1.5.8. The version-source updater now knows the
image a `version` pin tags (pulled directly or as the base of a custom
build) and withholds a bump when the registry answers 404 for the new
tag; an indeterminate answer does not block it, the same line
test-external draws. Run against main, the updater now reports no
update for bbb-pads instead of v1.5.11.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`tarball_name()` picked the archive from `platform.machine()` alone, so
an end-of-life Node.js on Intel macOS would have unpacked a Linux build
into /usr/local, Homebrew's prefix there. The upstream install now
supports Linux only and stops with a message to install a supported
Node.js through the package manager everywhere else.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Debian trixie ships nodejs 20.19, whose release line reached its end of
life on 2026-04-30. `@mermaid-js/mermaid-cli` 12 and its puppeteer
declare `node >=22.13.0`, so `make install-lint` failed in every fresh
Debian stack container with `All providers failed for
chrome-headless-shell` (#1088).
`ensure_node_present()` still installs the distro package first, then
reads the release line of `node` on PATH and looks it up in the
nodejs/Release schedule. A line past its `end` date is replaced by the
newest LTS from nodejs.org/dist/index.json: SHASUMS256.txt is verified
with gpgv against the active Node.js release keys from
nodejs/release-keys, the tarball against its SHA-256 entry, and the
release is unpacked into /usr/local, after which `node` on PATH must be
that release. Nothing pins a major: the next LTS line is picked up the
same way. Ubuntu 26.04's Node.js 22, supported until 2027-04-30, stays.
`ensure_npm_present()` calls it first, so a container that already holds
the end-of-life node is upgraded too rather than skipped because npm
exists. The call runs once per process.
Closes#1088
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Declare `__all__` in every package `__init__.py` whose `PROJECT_ROOT`
is imported by its modules (163 packages), so the constant counts as
exported; list every public name where the package defines more.
- Empty the 15 package `__init__.py` files whose `PROJECT_ROOT` no module
imports.
- Declare the public API of `utils.github.variant.axes` in `__all__`,
re-exports included, instead of silencing them with `noqa: F401`; the
list matches the one `feature/store-i18n` already carries.
- Use `import unittest.mock as mock` in `test_applications.py`, as the
rest of the suite does.
- Spell `_NON_PRINTABLE` in `utils/yaml_bootstrap.py` as the equivalent
BMP-only positive class; it matches exactly the same 2111 code points
as the old negated class across U+0000..U+10FFFF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
baudolo 7.1.x writes volume generations with `rsync -aHAX`, so the
producing host keeps `user.*` extended attributes (OpenCloud keeps its
storage metadata in `user.oc.*`). Every later copy dropped them again,
because none of the other rsync calls passed `-X` (nor `-A`/`-H`):
- NFS export snapshots (svc-bkp-nfs-2-local `script.sh`),
- the backup host's pull (svc-bkp-remote-2-local `pull_specific_host.py`),
- the device sync (svc-bkp-local-2-device `script.py`),
- every recover into a live target (`utils/recovery/base.py`, used by the
volume, NFS and device recover scripts),
- the `recover full` and `recover remote` tree pulls.
A disaster recovery through the documented chain therefore restored
OpenCloud without `user.oc.*`, silently: the copies succeeded and the
dirval stamp does not cover attributes.
All of these now run `rsync -aHAX` (`-aHAXP` for the device sync). A
target filesystem without xattr support makes rsync exit non-zero as soon
as a source file carries attributes, so the hop fails loudly instead of
dropping them; there is no per-device opt-out. The single-image pull in
`recover remote` keeps `-a`. The provider's ssh-wrapper needs no change:
its glob accepts any server flags and its character guard allows them.
Two tests write a `user.oc.id` attribute and assert it survives the NFS
snapshot and the recover; with the old `-a` both fail with
`OSError: [Errno 61] No data available`.
Closes#1021
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workflows pin `runs-on: ubuntu-NN.NN`, so no job moves to a new Ubuntu
release behind our back, but nothing raised the pin either. A sixth job of
`🔄 Update versions` closes that: `utils/update/runner.py` reads the image
table of actions/runner-images, takes the newest x64 Ubuntu label whose
image carries no preview or beta badge, and raises every older bare
`runs-on: ubuntu-NN.NN` to it; arch-suffixed labels stay untouched. The job
opens the change as one PR over `.github/workflows`, so its CI run tests
the new image before it reaches main.
This commit carries the updater's first result: every job now runs on
ubuntu-26.04 (49 `runs-on`).
actionlint 1.7.12, the newest release, does not know `ubuntu-26.04` yet
(rhysd/actionlint#682, fix proposed in rhysd/actionlint#743), so
`.github/actionlint.yaml` declares the label until a release knows it.
act maps each `ubuntu-NN.NN` label the workflows use to the local platform
image, derived from `.github/workflows`, instead of a fixed list that did
not know ubuntu-26.04.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Combine main's WireGuard mesh axis with this branch's network axis in the CI
variant code. Every matrix row now carries both: network (clearnet, tor,
multi) from utils.github.variant.network and vpn (meshed or direct) from
utils.github.variant.vpn. Label parsing, which main moved to label.py, reads
the network glyph instead of the old tor flag and keeps the vpn glyph; the
selection token, plan table, matrix and trigger CLIs, the workflows and the
swarm routine forward both axes. The test of an incapable priority variant
now compares the set of network modes, since a swarm priority row also
covers both mesh arms.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
compose/cache.shared.override.yml attaches every worktree stack's
infinito service to the primary's network (infinito-nexus-core_default,
172.30.0.0/24) without an ipv4_address. The default network declared
no ip_range, so Docker IPAM handed out the lowest free address and the
worktree stacks took 172.30.0.2, .3, .8, .9 and, whenever the primary
container was down, .10, the primary's INFINITO_IP4. The primary's
`make compose-up` then failed with "Address already in use", and with
it `make quality-high` on main.
The default network now declares ip_range: ${INFINITO_IP_RANGE}, the
upper half of INFINITO_SUBNET. Every static address of the stack (.1,
.4 to .6, .10, .53) sits in the lower half, so a container without a
static address can no longer take one.
INFINITO_IP_RANGE is derived from the final INFINITO_SUBNET after
custom.env is applied, because a worktree pins its subnet there and
`make dotenv-force` regenerates without it in the environment. Existing
worktrees therefore get their own slot's range without a new pin.
A lint checks that every static IP in default.env and every
172.30.${slot}.N pin in scripts/system/worktree/up.sh lies below the
pool.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.
The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.
call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every cold render of the applications SPOT walks the templates of all
269 roles, and those templates call lookup('config'), lookup('tls'),
lookup('domain') and others. Each of these re-enters
get_merged_applications while the render guard is set, and each one
rebuilt the unrendered view from scratch: a deepcopy of all role
defaults plus the override merge, about 8 ms, never cached. A local
cold render of the development inventory made 351 such calls, 2.82 s
of a 6.86 s render.
The outer render now keeps a copy of its unrendered view and its cache
key on the render guard, taken before the inventory volumes are
popped, so it is the same view a nested call used to build. A nested
call with the same key returns it; any other key still builds its own.
The guard drops it when the render ends.
Measured on the same inventory: the nested calls took 0.07 s instead
of 2.82 s and the cold render 3.60 s instead of 6.86 s. The rendered
SPOT is unchanged: the old and new dumps differ in 69 user passwords,
which the development inventory regenerates on every read (two runs of
the old code differ in 68), and in one absolute checkout path, the only
other difference.
In CI each cold render took 40 to 77 s and the heaviest swarm cell of
#950 missed its budget by a few minutes (#984).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
utils/yaml_bootstrap.py promises that load_block(text) either equals
yaml.safe_load(text) or raises BootstrapYamlError. A fuzzer comparing
both found two inputs that broke the promise:
- str.splitlines() also breaks lines at \x0b, \x0c and \x1c to \x1e,
which PyYAML's reader rejects as non-printable. "\x1e/: R" came back
as {"/": "R"}, and a non-printable character inside a comment line was
skipped with the comment.
- An indented " ---" on line 1 was taken as the document marker, while
PyYAML reads it as a plain scalar and rejects the mapping that follows.
load_block now refuses any character outside PyYAML's own printable
class before it splits lines, and takes "---" as the document marker
only at column 0. The line breaks splitlines() still honours (\r, \r\n,
NEL, LS, PS) are line breaks for PyYAML too, so splitting stays as is.
The eight inputs join STRUCTURES in the divergence test; against the
previous reader, four of them diverge and fail it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The autoformat step of the gate wrapped the long assertions that
660ad2ff16 committed unformatted. No behaviour change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The frontend and its sso, logout, dashboard and prometheus consumers were
gated on 'svc-net-tor' not in group_names, written when a Tor node served
onion only. On a multi node that removed the clearnet filer vhost, so the
storage specs of every SeaweedFS consumer failed on a clearnet pick with
ERR_SSL_UNRECOGNIZED_NAME_ALERT on filer.<domain>.
lookup('network_served', <network>, *role_ids) answers whether the node's
resolved NETWORK_MODE serves a network and every named role is deployed.
The gates use it instead of restating the mode resolution and the provider
check, so the frontend now drops only on a tor-only node. is_explicit_truth
registers the lookup in GATED_LOOKUPS so the static closure still reads
those flags as dynamic, and the dynamic-flag and universal-logout checks ask
is_explicit_truth instead of matching "in group_names" themselves.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Some servers build every absolute URL from one site URL, so a second
network cannot be answered under its own origin. reachability.handover
keeps the siblings but turns each one outside the canonical network into a
302 to the same path on the canonical URL (basic and ws_generic vhosts).
The redirect target comes from the canonical domain, not the application
id, because lookup('tls', application_id) follows the network of the vhost
being rendered and would point the sibling at itself.
The flag also drops Onion-Location for the role (Tor Browser would bounce
between the two), skips those siblings in the CSP crawl, expects 302 from
the webserver health check and keeps Playwright on the canonical domain.
The key needs a nocheck reason like single_mode.
Mattermost opts in: it builds every OAuth callback from its one SiteURL, so
an onion login came back on the clearnet callback and failed with 400.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Since 778c18fb23 the carrier key holds the tokens file's mtime and size.
sys-token-store changed that stat on every inclusion (00_core.yml
touched the file) and on every persisted token, and nothing re-parked a
carrier afterwards. Every later task's fresh worker then rendered the
applications SPOT cold, 30 to 80 s each. On main fea6f31d9f (run
38812964308) seven heavy jobs logged 194 to 287 such tasks and ran out
of the 20400 s budget; the same jobs before 778c18fb23 logged 65 to 83,
all of them real waits.
- 00_core.yml touches the store with modification_time and access_time
preserved, so it only creates a missing file. In swarm the controller
sees the store over NFS with attribute caching, and the touch alone
made the second pass go cold without any write.
- write.yml re-parks the carrier when the persist or the controller
mirror changed the store. Both run once, so every host that ran the
include gets the same verdict.
- tasks/utils/setup/role.yml re-parks after every role on every host,
so hosts that did not run the writing role recover too. Where the
key still matches this serves the carried payload; measured 0.1 s per
role in a local compose deploy.
- The tokens file enters the key as the SHA-256 of its content,
memoised per inode, mtime and size, so a touch no longer changes the
key. A stat that still matches serves the memo (1.67 us against
1.41 us for the old stat signature), and a stale stat keeps key and
YAML cache on the same old content instead of pairing new content
with an old key.
- applications_carrier opens the tokens file before it computes the
key, so on the NFS-mounted controller the parked key reflects the
revalidated attributes rather than a cached stat that changes a
minute later.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On the Fedora, CentOS Stream and Manjaro workspace jobs install-lint
ran `puppeteer browsers install chrome-headless-shell --install-deps`
as root, and puppeteer rejected it with "apt-get does not seem to be
available". The installer only warned ("optional lint tool"), so
install-lint passed while mmdc had no browser.
--install-deps is now passed only where apt-get exists. Everywhere else
the libraries `ldd` reports missing are installed by soname: dnf and
yum through the RPM provide `<soname>()(64bit)`, pacman through the
first package `pacman -F usr/lib/<soname>` names (libgbm.so.1 is owned
by both mesa and mesa-amber, which conflict). browser_installed() also
requires every library to resolve, so the install-lint stamp stays
stale until the browser can actually load.
A failed npm install, a missing mmdc, a failed or timed-out browser
download and unresolved libraries now raise instead of warning.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Manjaro workspace job failed test-unit-ruby with "cannot load such
file -- minitest". Debian vendors minitest in libruby, but Arch ships it
as ruby-minitest and Fedora and CentOS Stream as rubygem-minitest, none
of which ruby pulls in. ensure_ruby_present now probes
`require "minitest"` and installs the distro package when the probe
fails, then fails loudly if minitest is still missing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
No conflicts across 121 incoming commits. The one file both sides changed is
tasks/stages/01_constructor.yml: upstream parks an applications carrier for
the swarm bootstrap roles after the add_host tasks, and keys the carrier on
the tokens file as well. This branch's mesh bootstrap opens and drops its own
block-scoped carrier before that point, so both keep working as written.
Upstream also brings ruff 0.17.0, distro-derived env keys, a scoped act outer
teardown, the stalwart relay rule, the onion apex redirect and image and
version updates; ruff 0.17 reformats nothing on this branch.
One oauth2-proxy applies one cookie-secure flag to every host it answers, so a secure clearnet instance drops the CSRF cookie of its plaintext onion sibling and the login loops. proxy_instances now yields one instance per network with its own compose service, sso or sso_onion host port and config file; each config takes cookie_secure, redirect_url and cookie_domains from its own network, and every vhost routes /oauth2/ to the instance of its network. The 29 gated roles gain a sso_onion port, and Bluesky heals the bind mount of every instance.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A process that renders network-bound URLs once, such as oauth2-proxy or the dashboard, can serve only one network, so a multi-network role runs one per network. network_instances lists those networks with a label, the first one unlabelled so a single-network role keeps its names, and labelled and labelled_file suffix service and file names with it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The onion pick only bypassed the SSO issuer, so a clearnet CDN, asset or dashboard host an onion page loads went through Tor, which the lab network cannot exit. browser_proxy_bypass derives the list from the deployed applications' domains plus the issuer; it is empty on a clearnet pick and on a tor-only node. Specs that passed their own proxy to newContext dropped the bypass and now inherit use.proxy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- browser_revision() reads revisions.js from the puppeteer-core that
node resolves for mmdc's puppeteer: puppeteer/node_modules first, then
the node_modules directories above it. When mermaid-cli pins another
puppeteer-core, npm nests puppeteer's own copy, and the sibling held a
different revision.
- A failed browser install removes only the directories of the pinned
revision before the retry. Removing the whole chrome-headless-shell
cache also deleted browsers that other puppeteer installs share.
Addresses the CodeRabbit review on #958.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Only the node that runs svc-net-tor had it in group_names, so every other swarm node resolved no node onion, served no onion siblings and scheduled no onion apex redirect. tor_serves_host also accepts a provider anywhere in a cluster of more than one svc-swarm-node; a single compose host keeps the old answer.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
