`security-apparmor-teardown` ran `aa-teardown`, which unloads every
profile including dockerd's in-memory `docker-default`. dockerd only loads
that profile when it starts, so every container start afterwards that
requests it fails until dockerd restarts. Since #1067 runs the teardown on
the GitHub runners, BuildKit failed each RUN of the swarm node image three
times (run 39332343490, 39 of 40 swarm jobs):
runc run failed: unable to start container process: error during
container init: unable to apply apparmor profile: apparmor failed to
apply profile: write .../attr/apparmor/exec: no such file or directory
The teardown now removes each loaded profile by name through securityfs
and skips `docker-default` and child profiles, which leave with their
parent. Dev hosts gain the same: no dockerd restart is needed after
`make environment-bootstrap`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On ubuntu-26.04 runners the swarm deploy fails at svc-storage-nfs-client's
probe-write, right after a successful probe-mount, on every client node:
`Destination directory /run/svc-storage-nfs-client/probe does not exist`.
The rescue bundles cannot say why: ganesha writes to
/var/log/ganesha/ganesha.log, which the swarm diagnostics never collected,
and its NFS_V4 component logged nothing below EVENT.
Debug runs now log NFS_V4 at DEBUG, and the diagnostics append the tail
of ganesha.log to the nfs-boundary section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workflows pin `runs-on: ubuntu-NN.NN`, so no job moves to a new Ubuntu
release behind our back, but nothing raised the pin either. A sixth job of
`🔄 Update versions` closes that: `utils/update/runner.py` reads the image
table of actions/runner-images, takes the newest x64 Ubuntu label whose
image carries no preview or beta badge, and raises every older bare
`runs-on: ubuntu-NN.NN` to it; arch-suffixed labels stay untouched. The job
opens the change as one PR over `.github/workflows`, so its CI run tests
the new image before it reaches main.
This commit carries the updater's first result: every job now runs on
ubuntu-26.04 (49 `runs-on`).
actionlint 1.7.12, the newest release, does not know `ubuntu-26.04` yet
(rhysd/actionlint#682, fix proposed in rhysd/actionlint#743), so
`.github/actionlint.yaml` declares the label until a release knows it.
act maps each `ubuntu-NN.NN` label the workflows use to the local platform
image, derived from `.github/workflows`, instead of a fixed list that did
not know ubuntu-26.04.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On swarm nodes /var/lib/infinito is the ganesha NFSv4.2 state share, so
every backup repository landed on it. ganesha 6.5 refuses user.* extended
attributes, and baudolo 7.1.x copies volumes with `rsync -aHAX`, so the
volume backup of OpenCloud (user.oc.* metadata) failed with
`fsetxattr(... "user.oc.id") failed: Operation not supported (95)`.
After the state mount the client role now binds the node's own
/var/lib/infinito-backup over /var/lib/infinito/backup. Every consumer
keeps its path; the data stays on local disk with its attributes, and the
backup host pulls each provider over ssh as before. The bind carries
_netdev: without it, systemd-fstab-generator puts the bind into
local-fs.target below its NFS parent in remote-fs.target, and the boot
transaction cycles. The DR drill remounts the bind after its lazy remount
of the state share, because `umount -l` on the parent detaches it.
Closes#1012
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ubuntu-latest is rolling over to ubuntu-26.04, whose sudo is sudo-rs. It
ignores `-E` ("preserving the entire environment is not supported") and
hands the child only the explicit `env` assignments, so the swarm matrix
died at import with `KeyError: 'INFINITO_SRC_DIR'` before any inventory
existed (PR #1009, Nextcloud#0). Jobs that landed on ubuntu-24.04 kept
passing.
Every `sudo -E` now passes the caller's exported names through
`--preserve-env="$(compgen -e | paste -sd, -)"`, which both classic sudo
and sudo-rs honour. A lint rejects `sudo -E` and bare `--preserve-env`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/meta/resolve/diff/affected_roles.sh emits __ALL__ as soon as
one changed path outside roles/<role>/ is not a .md or .rst file. A
role fix that also adds its unit test under tests/unit/python/roles/
therefore deployed the whole matrix: PR #982 changed
roles/web-app-zammad/ plus
tests/unit/python/roles/web-app-zammad/files/test_search_allocation_precheck.py,
and its "Detect affected roles" job logged "Resolver output: __ALL__".
Paths under tests/ now join the .md/.rst skip-list. No role deploy
reads the top-level tests/ tree, and those suites run in their own
jobs whatever the whitelist says. A diff of only skipped paths keeps
falling back to __ALL__, as before. The same resolver feeds
effective_whitelist.sh, so the narrowing applies to every diff-driven
run.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.
The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.
call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
group_vars/all/00_general.yml pins ansible_python_interpreter to
/usr/local/bin/python3 for every host. Fact gathering is the first module
run, so nothing inside the play can create that path, and a managed server
that does not already carry it dies at 'Gathering Facts' with rc 127.
The published deploy recipe addresses exactly such a server:
ghcr.io/kevinveenbirkenbach/base-debian:latest carries /usr/bin/python3 and
no /usr/local/bin/python3, so a reader following the documented instructions
on a fresh Debian hits the same wall. The web-app-docs instruction replay is
what surfaced it, across four jobs of run 39040406916.
A pre_task links the discovered python3 into place through
ansible.builtin.script, which needs no interpreter on the target, and the
play then gathers facts explicitly. Dropping the pin instead would change
which interpreter every module runs under on every existing host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A role-level `placement: manager` pins every service of the role and keeps
every volume node-local, so a role needing one node-bound service had to give
up NFS for all of them. GitLab's gitaly is that case: its repositories volume
is node-local (`nfs: false`, gitaly does not support NFS) and a reschedule
left the data behind on the old node.
`deploy.yml.j2` now resolves `services.<service>.placement` first and keeps the
role-wide list as the fallback, so gitaly carries the constraint alone while
webservice, sidekiq, workhorse, shell, migrations and gitlabmcp stay
schedulable and `gitlab_shared`, `gitlab_uploads` and `gitlab_builds` keep
their NFS backing. Role placement still comes from the primary entry, which is
what drives that backing.
Two things had to come with it. The swarm drill decided that a role is
manager-pinned by grepping the whole `meta/services.yml`, so a per-service pin
would have skipped GitLab's drain and reschedule steps while six of its
services still run on workers; it now reads the primary entry. And
`resource_filter` returns a present-but-null value rather than the hard
default, which rendered `node.role == None`, a constraint no node satisfies.
`test_no_placement_manager.py` now inspects every `placement` value instead of
only `manager`, requires the justification for each, and rejects a value that
is not a swarm node role, since a misspelled one hangs the converge wait
rather than failing the deploy. Requirement 023 records the shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 🐳 Workspace (ubuntu:latest) job on main (run 38879556215, 1b36ef6120)
failed test-unit-python in the hook's second make test:
AssertionError: Regex didn't match: '^ localhost=ok/\d+ms$'
not found in ' localhost=ok/-1612931482679767779ms'
ubuntu:latest is Ubuntu 26.04, whose date is uutils coreutils. uutils
0.7.0 ignores the field width in %3N and prints all nine nanosecond
digits, so `date +%s%3N` yields a 19-digit value instead of epoch
milliseconds. Against main's sampler, uutils 0.7.0 date reports
localhost=ok/7081859ms, and an unpadded nanosecond value turns the
difference negative.
answer_once now takes both timestamps from /proc/uptime, which the
kernel prints with two decimals whatever date the image ships. The
sampler already reads /proc/net/udp, so it was Linux-only before. The
resolution is 10 ms; the same probe reports 10 to 20 ms with uutils
date on PATH.
The new test puts a date on PATH that does not understand %3N; main's
sampler dies on it, the fixed one does not call date for the latency.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Manjaro workspace job failed test-unit-ruby with "cannot load such
file -- minitest". Debian vendors minitest in libruby, but Arch ships it
as ruby-minitest and Fedora and CentOS Stream as rubygem-minitest, none
of which ruby pulls in. ensure_ruby_present now probes
`require "minitest"` and installs the distro package when the probe
fails, then fails loudly if minitest is still missing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Fedora and CentOS Stream workspace jobs failed TLS to pypi.org,
registry.npmjs.org and repo.packagist.org (CERTIFICATE_VERIFY_FAILED,
curl error 7). package-frontend/ca.sh put the frontend CA into the Arch
anchor dir /etc/ca-certificates/trust-source/anchors, which p11-kit on
the RedHat family never reads.
Probing for the anchor dir does not fix it: with-ca-trust.sh creates
both /etc/pki/ca-trust/source/anchors and
/etc/ca-certificates/trust-source/anchors on every update-ca-trust host,
so the probe in registry-ca.sh already picks the RedHat dir on a reused
Arch runner. Both scripts now install into both dirs, rebuild only when
one of them changed, and take CA_ROOT so unit tests run them against a
fake root.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
No conflicts across 121 incoming commits. The one file both sides changed is
tasks/stages/01_constructor.yml: upstream parks an applications carrier for
the swarm bootstrap roles after the add_host tasks, and keys the carrier on
the tokens file as well. This branch's mesh bootstrap opens and drops its own
block-scoped carrier before that point, so both keep working as written.
Upstream also brings ruff 0.17.0, distro-derived env keys, a scoped act outer
teardown, the stalwart relay rule, the onion apex redirect and image and
version updates; ruff 0.17 reformats nothing on this branch.
No conflicts. The 14 incoming commits fix the n8n gateway probe trigger under
n8n 2, let stalwart address bans expire, keep the Docker apt key when its
refresh is refused, leave eol roles out of image discovery, stop the release
backfill below the oldest buildable tag, and provision the mermaid browser
only in install-lint. None of them touches the mesh, swarm, NFS, network or
workflow files this branch changes.
The main merge joined two correct rules into one broken script. The
architecture gate republishes the highest version when :latest misses an
architecture, because the published install instructions pull the image
untagged and only a release of that version rewrites latest. The backfill
floor refuses any tag below v9.0.2, which cannot build from its own tree.
The gate returned before the floor was ever consulted, so an incomplete
:latest would have asked the backfill for a tag it cannot build. Both
paths now ask below_floor, which also replaces the comparison the loop
carried inline.
The picker's own test could not have caught it: its docker stub answers
tags without emitting a manifest, and architectures.sh reads an empty
manifest as every architecture missing, so the gate fired on every case
and printed the newest tag. The stub now serves the whole pool, which is
the state the floor cases assume anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
down_act_outer.sh removed every container named act-*, so starting or
releasing a swarm drill in one worktree killed the act container of a
drill another worktree had running.
workflow.sh now labels act's job containers with
infinito.worktree=${INFINITO_CONTAINER} through the default
ACT_CONTAINER_OPTIONS, and down_act_outer.sh removes only containers
carrying the own label. Both scripts refuse to run without
INFINITO_CONTAINER, so the filter never widens to every act container;
make exports BASH_ENV, which loads it from .env.
Worktrees without a custom.env share the default container name and
therefore still share one label among themselves.
Closes#933
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The loader copies INFINITO_DISTRO into a loader-local variable before it
compares the value with the one in .env. With INFINITO_DISTRO read directly in
that condition, `shellcheck -x scripts/meta/resolve/apps.sh` reported SC2153
for INFINITO_DISTROS, which apps.sh takes from the sourced .env; with the copy
it reports nothing, as on main.
The loader test checks that .env is unchanged with filecmp against the file it
was copied from, instead of reading it twice with Path.read_bytes, which
test_no_uncached_file_reads rejects.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The release backfill dispatches the highest version tag whose images are
missing. Every tag from v15.0.0 down to v9.0.2 has its images now, so since
2026-10-05 it picks v9.0.1 every day, and each of the nine runs failed in
"Build missing CI images / Image" at
`entry.sh: line 19: infinito: command not found`. The v9.0.1 tree cannot
build its own image; c680be12ea fixed that build in v9.0.2, and a tag cannot
take the fix.
highest_release_tag.sh now stops at v9.0.2: no tag below it is picked, so the
backfill reports nothing missing instead of retrying v9.0.1. v9.0.0 has the
same Dockerfile, Makefile and scripts/docker as v9.0.1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
make lint runs install-lint once and then dispatches its fifteen targets
in parallel. scripts/lint/mermaid.sh provisioned chrome-headless-shell
inside that parallel phase with `npx puppeteer browsers install
--install-deps`, whose `apt-get satisfy` carries no lock timeout. In a
fresh stack container another target held the apt lock, the shared
libraries stayed missing, the script only warned, and all 289 Markdown
files then failed on libnspr4.so (#795).
utils/install/lint/mermaid_cli.py now provisions the browser right after
mmdc, inside the serial install-lint step, as root with --install-deps
and as a user without it, because puppeteer refuses to install libraries
without root. A failed install deletes the chrome-headless-shell cache
folder and retries once under a 900 second timeout per attempt; a cut-off
download otherwise leaves a folder without its executable, and puppeteer
refuses every later install of that build. The install-lint stamp counts
as fresh only while the browser executable exists, so a recreated
container provisions it again.
mermaid.sh no longer calls apt-get or npx. It renders one minimal diagram
first and, when that fails, exits with mmdc's output instead of reporting
every Markdown file.
Verified: tests/unit/python/scripts/lint/test_mermaid.py stubs mmdc,
npx and apt-get. With a browser that cannot load libnspr4.so the script
exits 1 with that line, before any Markdown file renders, and mmdc is the
only tool it calls. Against main's mermaid.sh the same test fails: it
calls apt-get and npx and reports the files one by one. The installer and
stamp cases pass, shellcheck is clean.
Not verified: a full make lint in a recreated stack container. The race
reproduced in one of two runs in the issue.
feature/store-i18n changed the provisioning in mermaid.sh with 2f5f52bd1a
and dc67bce5f0 (timeout, deleting a cut-off browser). Both behaviours
moved into mermaid_cli.py here, so its merge takes this version of
mermaid.sh.
Closes#795
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A compose deploy job labelled centos or arch built its stack image from
pkgmgr-debian and ran it as infinito_nexus_debian (#794). distros.sh
exports INFINITO_DISTRO per iteration and sources load.sh again, but the
loader returned early because INFINITO_ENV_LOADED and the .env mtime were
unchanged. A full reload would not have helped either: the debian values
were already exported, so the restore step kept them as caller intent.
load.sh now records the distro it loaded for. Another INFINITO_DISTRO
defeats the early return and drops INFINITO_CONTAINER,
INFINITO_PARENT_IMAGE and INFINITO_IMAGE, the three keys the env handlers
derive from the distro. When the requested distro also differs from the
one .env was generated for, the loader sources a set that
`python -m cli.meta.env --output` generates for it in a private temp
directory and leaves .env untouched, so a local multi-distro run does not
switch the checkout's distro. A caller value set for the loaded distro
still wins.
Verified: tests/unit/python/scripts/meta/env/test_load.py and the new
distros.sh case pass. Against main's loader both fail with the symptom of
the issue: INFINITO_DISTRO=centos next to infinito_nexus_debian and
pkgmgr-debian:stable, and distros.sh handing debian, arch and centos the
same debian parent. shellcheck is clean.
Consequence: CI jobs labelled arch, ubuntu, fedora or centos now build
and deploy that distro, so a failure that shows up there was masked
before.
feature/store-i18n carries 3f973eb137, which resolves the parent image
for scripts/image/push.sh on its own, and 05e4dbafcb, which inserts the
INFINITO_INSTANCE slot check into the block this commit restructures. The
merge keeps the slot check ahead of the distro check.
Closes#794
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
lint-playwright stages every role spec under a fresh mktemp directory and
runs `playwright test --list` there. Playwright keys its compilation cache
by source path and writes it to $TMPDIR/playwright-transform-cache-<uid>,
so with a new stage path per run the cache never hit and every run added a
full copy. In the dev container that tmpfs reached 12 GB and the quality
gate failed with "No space left on device".
PWTEST_CACHE_DIR now points into the stage dir, which the EXIT trap already
removes. Deploy-time Playwright runs are not affected; the export only
reaches this script's `npx playwright` calls.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The three-arm comparison on web-app-gitea#0 put the meshed arm at 414ms
per task over the direct arm, and the tunnel at 4% of that. What it could not
split is the rest: the arms differ both in connection plugin (docker against
ssh) and in escalation (root with no prompt against administrator with a
password sudo on every task).
A fixed set of module calls, timed either side of the transport switch with
become off and on, gives four numbers that separate the two. It costs about a
minute where a fourth deploy arm would have cost three hours, and it needs no
root login and no sudoers change.
INFINITO_SWARM_TRANSPORT_BENCH guards it, so a normal round times nothing.
The arm existed to answer one question: how much of the meshed arm's
cost is the connection plugin and how much is the tunnel. Runs 37592061960
(docker), 37753506661 (SSH on the underlay) and 37621556682 (SSH over
WireGuard) answered it on web-app-gitea#0, debian/ext4:
pass 1 5964s / 8511s / 8832s
pass 2 5319s / 7472s / 7705s
Leaving ansible_connection: docker costs +4700s, the tunnel +554s - 89.5%
against 10.5%. The answer is recorded; the scaffolding does not need to ship.
Reverts d792519725, 1b28f87b37, 89b8a8c8d1, 0470491e1a, 529c5ace43,
dd3d10a9a2, c20dd0e5f5, 458d8956e1, 63ade1e03b, d281555945 and 4adfb6e3d2.
95cfc8b7b7 is already superseded by bbfd82f89, which sets the profiling limit
upstream.
The drop and truncate helpers ended with `[[ $rc -ne 0 ]] && warn`,
which returns 1 when the command succeeded. The per-stack subshell runs
on the left of `||`, where set -e is suspended, so its status is that
of the last helper: every successful MariaDB drop printed "DB purge
failed (rc=1)" and the script exited 1. The helpers now end with an if
statement, which returns 0 when nothing failed and still warns when
something did.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
