`security-apparmor-teardown` ran `aa-teardown`, which unloads every
profile including dockerd's in-memory `docker-default`. dockerd only loads
that profile when it starts, so every container start afterwards that
requests it fails until dockerd restarts. Since #1067 runs the teardown on
the GitHub runners, BuildKit failed each RUN of the swarm node image three
times (run 39332343490, 39 of 40 swarm jobs):
runc run failed: unable to start container process: error during
container init: unable to apply apparmor profile: apparmor failed to
apply profile: write .../attr/apparmor/exec: no such file or directory
The teardown now removes each loaded profile by name through securityfs
and skips `docker-default` and child profiles, which leave with their
parent. Dev hosts gain the same: no dockerd restart is needed after
`make environment-bootstrap`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ubuntu-26.04 runner images load per-binary AppArmor profiles (wg-quick,
dig, who, fusermount3, ~150 upstream). Path-based attachment applies to
every unconfined task, so the profiles also attach inside our privileged
`apparmor=unconfined` containers. The upstream wg-quick profile only allows
`/usr/bin/ip`; on CentOS and Fedora nodes `ip` lives at `/usr/sbin/ip`, so
`svc-net-wireguard : Bring 'data' up` failed with rc=126:
/usr/bin/wg-quick: line 32: /usr/sbin/ip: Permission denied
apparmor="DENIED" operation="exec" profile="wg-quick" name="/usr/sbin/ip"
(run 39268090162, Fediwall#0, CentOS swarm VPN).
Dev hosts never see this: `environment-bootstrap` runs
`security-apparmor-teardown`. The runners skipped that preparation. The
deploy, workspace and DNS workflows now run the same target on the runner
host before any container starts (skipped under act, whose job container
cannot reach the host's policy), so CI matches the dev-host state for all
profiles, not only the ones seen so far.
A dedicated permissive profile for the containers was rejected: every
AppArmor host without that profile loaded would refuse to start them,
including dev hosts and act.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On ubuntu-26.04 runners the swarm deploy fails at svc-storage-nfs-client's
probe-write, right after a successful probe-mount, on every client node:
`Destination directory /run/svc-storage-nfs-client/probe does not exist`.
The rescue bundles cannot say why: ganesha writes to
/var/log/ganesha/ganesha.log, which the swarm diagnostics never collected,
and its NFS_V4 component logged nothing below EVENT.
Debug runs now log NFS_V4 at DEBUG, and the diagnostics append the tail
of ganesha.log to the nfs-boundary section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workflows pin `runs-on: ubuntu-NN.NN`, so no job moves to a new Ubuntu
release behind our back, but nothing raised the pin either. A sixth job of
`🔄 Update versions` closes that: `utils/update/runner.py` reads the image
table of actions/runner-images, takes the newest x64 Ubuntu label whose
image carries no preview or beta badge, and raises every older bare
`runs-on: ubuntu-NN.NN` to it; arch-suffixed labels stay untouched. The job
opens the change as one PR over `.github/workflows`, so its CI run tests
the new image before it reaches main.
This commit carries the updater's first result: every job now runs on
ubuntu-26.04 (49 `runs-on`).
actionlint 1.7.12, the newest release, does not know `ubuntu-26.04` yet
(rhysd/actionlint#682, fix proposed in rhysd/actionlint#743), so
`.github/actionlint.yaml` declares the label until a release knows it.
act maps each `ubuntu-NN.NN` label the workflows use to the local platform
image, derived from `.github/workflows`, instead of a fixed list that did
not know ubuntu-26.04.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`ubuntu-latest` moved from ubuntu-24.04 to ubuntu-26.04 on its own
schedule, in the middle of run 39166850273, and every job that landed on
the new image met a different OS: sudo-rs instead of sudo, no ip_tables
loaded, and now the NFS probe-write of svc-storage-nfs-client failing on
26.04 swarm nodes. The same branch went red without a code change.
Every `runs-on` now names `ubuntu-24.04`. GitHub still rebuilds that image
weekly, so patches and tool updates keep arriving; only the release jump
becomes a deliberate change of the label. The CodeQL job drops its dead
swift/macos-latest branch, since no swift language is in its matrix. act
already maps ubuntu-24.04 to the local platform image.
A lint rejects any job whose `runs-on` contains a `*-latest` label.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
#414 merged web-app-wazuh with a meta/networks.yml that carries only
`local`, and test_mesh_subnet_pool.py requires every stack role to ship
`meshes: []` so a compose host can join meshes through it. main's
test-lint has been red since d0698daa5e:
AssertionError: 'meshes' not found in {'local': {'subnet': '192.168.81.0/24'}}
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Generations a swarm node wrote to the NFS share before the node-local
bind stay under the export's backup/ directory, hidden from the nodes,
outside every cleanup and pull. No production swarm install carries any,
so the README states the manual removal instead of migrating.
Refs #1012
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two holes in the node-local backup bind, both reproduced with the role's
own tasks against a ganesha NFSv4.2 mount:
- The store was created with mode 0700. The role runs twice per deploy
(constructor svc-storage group, then 02_server), and between the runs
user-backup and the two cleanup roles converge the same inode to
root:root 0755. The second run then chmodded it to root:root 0700, the
backup user lost access (`ls` rc=2), and the backup host's pull would
list no repositories and exit 0. The store now uses 0755, the mode the
cleanup roles converge it to, so the second run changes nothing.
- A changed NFS mount line (options, version, server) makes
ansible.posix.mount fall back from `mount -o remount` to a plain
`umount`, which the child bind now blocks: `umount.nfs4: device is
busy`. A check-mode pass detects the pending change and unmounts the
bind first; the bind task mounts it again afterwards.
Refs #1012
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On swarm nodes /var/lib/infinito is the ganesha NFSv4.2 state share, so
every backup repository landed on it. ganesha 6.5 refuses user.* extended
attributes, and baudolo 7.1.x copies volumes with `rsync -aHAX`, so the
volume backup of OpenCloud (user.oc.* metadata) failed with
`fsetxattr(... "user.oc.id") failed: Operation not supported (95)`.
After the state mount the client role now binds the node's own
/var/lib/infinito-backup over /var/lib/infinito/backup. Every consumer
keeps its path; the data stays on local disk with its attributes, and the
backup host pulls each provider over ssh as before. The bind carries
_netdev: without it, systemd-fstab-generator puts the bind into
local-fs.target below its NFS parent in remote-fs.target, and the boot
transaction cycles. The DR drill remounts the bind after its lazy remount
of the state share, because `umount -l` on the parent detaches it.
Closes#1012
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On Fedora 44, dnf satisfies docker-ce's `(iptables-nft or iptables)`
requirement with iptables-legacy. The legacy backend needs the ip_tables
module, which ubuntu-26.04 runners no longer have loaded, and the compose
container cannot load it because it mounts no /lib/modules. The MSS clamp
task then failed with `modprobe: FATAL: Module ip_tables not found` and
`can't initialize iptables table 'mangle'` (Nextcloud#10, Gitea#0,
Jellyfin#0 on fedora). CentOS 10 already resolves to iptables-nft from the
same plan and deploys green on the same image.
Naming iptables-nft in the docker transaction makes Fedora use the
nf_tables backend as well.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ubuntu-latest is rolling over to ubuntu-26.04, whose sudo is sudo-rs. It
ignores `-E` ("preserving the entire environment is not supported") and
hands the child only the explicit `env` assignments, so the swarm matrix
died at import with `KeyError: 'INFINITO_SRC_DIR'` before any inventory
existed (PR #1009, Nextcloud#0). Jobs that landed on ubuntu-24.04 kept
passing.
Every `sudo -E` now passes the caller's exported names through
`--preserve-env="$(compgen -e | paste -sd, -)"`, which both classic sudo
and sudo-rs honour. A lint rejects `sudo -E` and bare `--preserve-env`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
builder.py imported utils.env.handlers.infinito.ip_range directly, while
ip_range imports EnvBuilder from builder for its annotations. CodeQL
reported that as py/unsafe-cyclic-import on PR #999. The import only
runs under TYPE_CHECKING, so nothing breaks at runtime, but builder now
takes ip_range from utils.env.handlers like ORDERED_HANDLERS, the path
every other handler module already uses.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
