compose/01_minimal.sh disabled every variable service of the dashboard,
asset included, and ran both matrix rounds; round 1 of web-app-dashboard
pins the asset provider off as well. Without web-svc-asset the dashboard
renders its logo and favicon from the public
https://file.infinito.nexus/assets, so the workspace test depended on
that host: while it does not answer, Playwright's no-console-noise spec
fails on `net::ERR_CONNECTION_REFUSED` for the logo (main run
39684179395, ubuntu and centos).
The test now keeps the asset provider out of the disable set and pins
round 0, so the dashboard serves its branding from the deployment's own
file service. Round 1 stays covered by the regular dashboard matrix.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/install/filesystem.sh retried `dnf install -y zfs` three times
against the same metadata during the Fedora image build of main run
39684179395 (XMPP#0): every attempt listed pcp-*-7.2.2-1.fc44, every
mirror answered 404, "Failed to download packages" after 3 attempts.
Each retried dnf install now runs with --refresh, so every attempt
resolves against freshly fetched metadata instead of repeating the one
that lists packages no mirror serves anymore.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The package cache creates every Nexus proxy with
strictContentTypeValidation. A raw proxy has no type contract, and Nexus
cannot type a GPG keybox: on a fresh cache it answers the Node.js release
keyring (raw.githubusercontent.com/nodejs/release-keys/.../pubring.kbx)
with 404 "Content type could not be determined". The Node.js installer
from #1108 fetches that keyring, so on main (run 39668792538) Workspace
debian:13 fails in `make install-lint`, and Workspace ubuntu, centos and
manjaro fail in `test-external`, whose URL check reaches the same file
through the cache.
The four raw proxies (raw-githubusercontent, raw-codeload-github,
raw-packagist, raw-alpine) now share the storage block with validation
switched off. Every typed proxy (apt, pypi, npm, helm, rubygems, go, yum)
keeps strict validation. The installer's integrity check does not depend
on Nexus: it verifies the SHASUMS file with gpgv and the tarball with
SHA-256.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
diagnostics.sh runs under a 900-second deadline that the sweep deadline
can shorten further. The package manager section came last, after the NFS,
ganesha backtrace and controller probes, so a cut-off run would have lost
exactly the log this collection exists for. It now runs first.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Six Fedora jobs of main runs 39494396920 and 39590304501 (Postmarks#0,
OpenBao#1, Jenkins#1, Magento#0, Matrix#2, OpenResty#0) died in dnf with
`Failed to download packages`. Ansible's dnf5 module reports nothing
more: `failures: []`. Its file logger writes the download errors, with
mirror, URL and status, at debug level to /var/log/dnf5.log, but no
rescue bundle collects that file, so the cause of five of them stays
unknown.
The swarm stack diagnostics now append the last 3000 lines of
/var/log/dnf5.log and, for dnf4 hosts, /var/log/dnf.librepo.log of every
node. A node without them records the `tail` error and the capture goes
on.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A selection token may pin the mesh state as &vpn or &direct, but check_pins
had no arm for it. pinned_vpn_states intersects the pin with the states the
mode allows and only swarm meshes, so a token like @compose+clearnet&vpn left
an empty state list: the row produced no job at all and the run reported green
for a combination that never deployed. That is the outcome _reject exists to
prevent, and every sibling axis already refuses an impossible pin.
The new arm asks whether any candidate mode keeps a state rather than whether
the pin is reachable, so a held vpn axis still overrides the pin instead of
being refused. That behaviour has its own test and is unchanged.
An earlier check was removed with the compose-mesh revert of 2026-10-16
because it rested on meshable() and smallest_variant(), which that revert
deleted. This one reuses pinned_vpn_states, applies to the swarm-only axis the
revert restored, and covers both mesh states instead of only &vpn.
pin_vpn and vpn_mode now reach check_pins from all three callers. The run's
vpn axis reaches the validation job through call-orchestrator and
call-validation, which passed every other axis but this one. The priority
procedure and its skill state all eight axes a token carries, not six:
architecture and mesh were pinnable and, for architecture, already validated,
yet neither was documented.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`security-apparmor-teardown` ran `aa-teardown`, which unloads every
profile including dockerd's in-memory `docker-default`. dockerd only loads
that profile when it starts, so every container start afterwards that
requests it fails until dockerd restarts. Since #1067 runs the teardown on
the GitHub runners, BuildKit failed each RUN of the swarm node image three
times (run 39332343490, 39 of 40 swarm jobs):
runc run failed: unable to start container process: error during
container init: unable to apply apparmor profile: apparmor failed to
apply profile: write .../attr/apparmor/exec: no such file or directory
The teardown now removes each loaded profile by name through securityfs
and skips `docker-default` and child profiles, which leave with their
parent. Dev hosts gain the same: no dockerd restart is needed after
`make environment-bootstrap`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On ubuntu-26.04 runners the swarm deploy fails at svc-storage-nfs-client's
probe-write, right after a successful probe-mount, on every client node:
`Destination directory /run/svc-storage-nfs-client/probe does not exist`.
The rescue bundles cannot say why: ganesha writes to
/var/log/ganesha/ganesha.log, which the swarm diagnostics never collected,
and its NFS_V4 component logged nothing below EVENT.
Debug runs now log NFS_V4 at DEBUG, and the diagnostics append the tail
of ganesha.log to the nfs-boundary section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workflows pin `runs-on: ubuntu-NN.NN`, so no job moves to a new Ubuntu
release behind our back, but nothing raised the pin either. A sixth job of
`🔄 Update versions` closes that: `utils/update/runner.py` reads the image
table of actions/runner-images, takes the newest x64 Ubuntu label whose
image carries no preview or beta badge, and raises every older bare
`runs-on: ubuntu-NN.NN` to it; arch-suffixed labels stay untouched. The job
opens the change as one PR over `.github/workflows`, so its CI run tests
the new image before it reaches main.
This commit carries the updater's first result: every job now runs on
ubuntu-26.04 (49 `runs-on`).
actionlint 1.7.12, the newest release, does not know `ubuntu-26.04` yet
(rhysd/actionlint#682, fix proposed in rhysd/actionlint#743), so
`.github/actionlint.yaml` declares the label until a release knows it.
act maps each `ubuntu-NN.NN` label the workflows use to the local platform
image, derived from `.github/workflows`, instead of a fixed list that did
not know ubuntu-26.04.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On swarm nodes /var/lib/infinito is the ganesha NFSv4.2 state share, so
every backup repository landed on it. ganesha 6.5 refuses user.* extended
attributes, and baudolo 7.1.x copies volumes with `rsync -aHAX`, so the
volume backup of OpenCloud (user.oc.* metadata) failed with
`fsetxattr(... "user.oc.id") failed: Operation not supported (95)`.
After the state mount the client role now binds the node's own
/var/lib/infinito-backup over /var/lib/infinito/backup. Every consumer
keeps its path; the data stays on local disk with its attributes, and the
backup host pulls each provider over ssh as before. The bind carries
_netdev: without it, systemd-fstab-generator puts the bind into
local-fs.target below its NFS parent in remote-fs.target, and the boot
transaction cycles. The DR drill remounts the bind after its lazy remount
of the state share, because `umount -l` on the parent detaches it.
Closes#1012
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ubuntu-latest is rolling over to ubuntu-26.04, whose sudo is sudo-rs. It
ignores `-E` ("preserving the entire environment is not supported") and
hands the child only the explicit `env` assignments, so the swarm matrix
died at import with `KeyError: 'INFINITO_SRC_DIR'` before any inventory
existed (PR #1009, Nextcloud#0). Jobs that landed on ubuntu-24.04 kept
passing.
Every `sudo -E` now passes the caller's exported names through
`--preserve-env="$(compgen -e | paste -sd, -)"`, which both classic sudo
and sudo-rs honour. A lint rejects `sudo -E` and bare `--preserve-env`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/meta/resolve/diff/affected_roles.sh emits __ALL__ as soon as
one changed path outside roles/<role>/ is not a .md or .rst file. A
role fix that also adds its unit test under tests/unit/python/roles/
therefore deployed the whole matrix: PR #982 changed
roles/web-app-zammad/ plus
tests/unit/python/roles/web-app-zammad/files/test_search_allocation_precheck.py,
and its "Detect affected roles" job logged "Resolver output: __ALL__".
Paths under tests/ now join the .md/.rst skip-list. No role deploy
reads the top-level tests/ tree, and those suites run in their own
jobs whatever the whitelist says. A diff of only skipped paths keeps
falling back to __ALL__, as before. The same resolver feeds
effective_whitelist.sh, so the narrowing applies to every diff-driven
run.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.
The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.
call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
group_vars/all/00_general.yml pins ansible_python_interpreter to
/usr/local/bin/python3 for every host. Fact gathering is the first module
run, so nothing inside the play can create that path, and a managed server
that does not already carry it dies at 'Gathering Facts' with rc 127.
The published deploy recipe addresses exactly such a server:
ghcr.io/kevinveenbirkenbach/base-debian:latest carries /usr/bin/python3 and
no /usr/local/bin/python3, so a reader following the documented instructions
on a fresh Debian hits the same wall. The web-app-docs instruction replay is
what surfaced it, across four jobs of run 39040406916.
A pre_task links the discovered python3 into place through
ansible.builtin.script, which needs no interpreter on the target, and the
play then gathers facts explicitly. Dropping the pin instead would change
which interpreter every module runs under on every existing host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A role-level `placement: manager` pins every service of the role and keeps
every volume node-local, so a role needing one node-bound service had to give
up NFS for all of them. GitLab's gitaly is that case: its repositories volume
is node-local (`nfs: false`, gitaly does not support NFS) and a reschedule
left the data behind on the old node.
`deploy.yml.j2` now resolves `services.<service>.placement` first and keeps the
role-wide list as the fallback, so gitaly carries the constraint alone while
webservice, sidekiq, workhorse, shell, migrations and gitlabmcp stay
schedulable and `gitlab_shared`, `gitlab_uploads` and `gitlab_builds` keep
their NFS backing. Role placement still comes from the primary entry, which is
what drives that backing.
Two things had to come with it. The swarm drill decided that a role is
manager-pinned by grepping the whole `meta/services.yml`, so a per-service pin
would have skipped GitLab's drain and reschedule steps while six of its
services still run on workers; it now reads the primary entry. And
`resource_filter` returns a present-but-null value rather than the hard
default, which rendered `node.role == None`, a constraint no node satisfies.
`test_no_placement_manager.py` now inspects every `placement` value instead of
only `manager`, requires the justification for each, and rejects a value that
is not a swarm node role, since a misspelled one hangs the converge wait
rather than failing the deploy. Requirement 023 records the shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 🐳 Workspace (ubuntu:latest) job on main (run 38879556215, 1b36ef6120)
failed test-unit-python in the hook's second make test:
AssertionError: Regex didn't match: '^ localhost=ok/\d+ms$'
not found in ' localhost=ok/-1612931482679767779ms'
ubuntu:latest is Ubuntu 26.04, whose date is uutils coreutils. uutils
0.7.0 ignores the field width in %3N and prints all nine nanosecond
digits, so `date +%s%3N` yields a 19-digit value instead of epoch
milliseconds. Against main's sampler, uutils 0.7.0 date reports
localhost=ok/7081859ms, and an unpadded nanosecond value turns the
difference negative.
answer_once now takes both timestamps from /proc/uptime, which the
kernel prints with two decimals whatever date the image ships. The
sampler already reads /proc/net/udp, so it was Linux-only before. The
resolution is 10 ms; the same probe reports 10 to 20 ms with uutils
date on PATH.
The new test puts a date on PATH that does not understand %3N; main's
sampler dies on it, the fixed one does not call date for the latency.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Manjaro workspace job failed test-unit-ruby with "cannot load such
file -- minitest". Debian vendors minitest in libruby, but Arch ships it
as ruby-minitest and Fedora and CentOS Stream as rubygem-minitest, none
of which ruby pulls in. ensure_ruby_present now probes
`require "minitest"` and installs the distro package when the probe
fails, then fails loudly if minitest is still missing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Fedora and CentOS Stream workspace jobs failed TLS to pypi.org,
registry.npmjs.org and repo.packagist.org (CERTIFICATE_VERIFY_FAILED,
curl error 7). package-frontend/ca.sh put the frontend CA into the Arch
anchor dir /etc/ca-certificates/trust-source/anchors, which p11-kit on
the RedHat family never reads.
Probing for the anchor dir does not fix it: with-ca-trust.sh creates
both /etc/pki/ca-trust/source/anchors and
/etc/ca-certificates/trust-source/anchors on every update-ca-trust host,
so the probe in registry-ca.sh already picks the RedHat dir on a reused
Arch runner. Both scripts now install into both dirs, rebuild only when
one of them changed, and take CA_ROOT so unit tests run them against a
fake root.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
No conflicts across 121 incoming commits. The one file both sides changed is
tasks/stages/01_constructor.yml: upstream parks an applications carrier for
the swarm bootstrap roles after the add_host tasks, and keys the carrier on
the tokens file as well. This branch's mesh bootstrap opens and drops its own
block-scoped carrier before that point, so both keep working as written.
Upstream also brings ruff 0.17.0, distro-derived env keys, a scoped act outer
teardown, the stalwart relay rule, the onion apex redirect and image and
version updates; ruff 0.17 reformats nothing on this branch.
No conflicts. The 14 incoming commits fix the n8n gateway probe trigger under
n8n 2, let stalwart address bans expire, keep the Docker apt key when its
refresh is refused, leave eol roles out of image discovery, stop the release
backfill below the oldest buildable tag, and provision the mermaid browser
only in install-lint. None of them touches the mesh, swarm, NFS, network or
workflow files this branch changes.
The main merge joined two correct rules into one broken script. The
architecture gate republishes the highest version when :latest misses an
architecture, because the published install instructions pull the image
untagged and only a release of that version rewrites latest. The backfill
floor refuses any tag below v9.0.2, which cannot build from its own tree.
The gate returned before the floor was ever consulted, so an incomplete
:latest would have asked the backfill for a tag it cannot build. Both
paths now ask below_floor, which also replaces the comparison the loop
carried inline.
The picker's own test could not have caught it: its docker stub answers
tags without emitting a manifest, and architectures.sh reads an empty
manifest as every architecture missing, so the gate fired on every case
and printed the newest tag. The stub now serves the whole pool, which is
the state the floor cases assume anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
down_act_outer.sh removed every container named act-*, so starting or
releasing a swarm drill in one worktree killed the act container of a
drill another worktree had running.
workflow.sh now labels act's job containers with
infinito.worktree=${INFINITO_CONTAINER} through the default
ACT_CONTAINER_OPTIONS, and down_act_outer.sh removes only containers
carrying the own label. Both scripts refuse to run without
INFINITO_CONTAINER, so the filter never widens to every act container;
make exports BASH_ENV, which loads it from .env.
Worktrees without a custom.env share the default container name and
therefore still share one label among themselves.
Closes#933
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The loader copies INFINITO_DISTRO into a loader-local variable before it
compares the value with the one in .env. With INFINITO_DISTRO read directly in
that condition, `shellcheck -x scripts/meta/resolve/apps.sh` reported SC2153
for INFINITO_DISTROS, which apps.sh takes from the sourced .env; with the copy
it reports nothing, as on main.
The loader test checks that .env is unchanged with filecmp against the file it
was copied from, instead of reading it twice with Path.read_bytes, which
test_no_uncached_file_reads rejects.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
