diagnostics.sh runs under a 900-second deadline that the sweep deadline
can shorten further. The package manager section came last, after the NFS,
ganesha backtrace and controller probes, so a cut-off run would have lost
exactly the log this collection exists for. It now runs first.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Six Fedora jobs of main runs 39494396920 and 39590304501 (Postmarks#0,
OpenBao#1, Jenkins#1, Magento#0, Matrix#2, OpenResty#0) died in dnf with
`Failed to download packages`. Ansible's dnf5 module reports nothing
more: `failures: []`. Its file logger writes the download errors, with
mirror, URL and status, at debug level to /var/log/dnf5.log, but no
rescue bundle collects that file, so the cause of five of them stays
unknown.
The swarm stack diagnostics now append the last 3000 lines of
/var/log/dnf5.log and, for dnf4 hosts, /var/log/dnf.librepo.log of every
node. A node without them records the `tail` error and the capture goes
on.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On ubuntu-26.04 runners the swarm deploy fails at svc-storage-nfs-client's
probe-write, right after a successful probe-mount, on every client node:
`Destination directory /run/svc-storage-nfs-client/probe does not exist`.
The rescue bundles cannot say why: ganesha writes to
/var/log/ganesha/ganesha.log, which the swarm diagnostics never collected,
and its NFS_V4 component logged nothing below EVENT.
Debug runs now log NFS_V4 at DEBUG, and the diagnostics append the tail
of ganesha.log to the nfs-boundary section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workflows pin `runs-on: ubuntu-NN.NN`, so no job moves to a new Ubuntu
release behind our back, but nothing raised the pin either. A sixth job of
`🔄 Update versions` closes that: `utils/update/runner.py` reads the image
table of actions/runner-images, takes the newest x64 Ubuntu label whose
image carries no preview or beta badge, and raises every older bare
`runs-on: ubuntu-NN.NN` to it; arch-suffixed labels stay untouched. The job
opens the change as one PR over `.github/workflows`, so its CI run tests
the new image before it reaches main.
This commit carries the updater's first result: every job now runs on
ubuntu-26.04 (49 `runs-on`).
actionlint 1.7.12, the newest release, does not know `ubuntu-26.04` yet
(rhysd/actionlint#682, fix proposed in rhysd/actionlint#743), so
`.github/actionlint.yaml` declares the label until a release knows it.
act maps each `ubuntu-NN.NN` label the workflows use to the local platform
image, derived from `.github/workflows`, instead of a fixed list that did
not know ubuntu-26.04.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On swarm nodes /var/lib/infinito is the ganesha NFSv4.2 state share, so
every backup repository landed on it. ganesha 6.5 refuses user.* extended
attributes, and baudolo 7.1.x copies volumes with `rsync -aHAX`, so the
volume backup of OpenCloud (user.oc.* metadata) failed with
`fsetxattr(... "user.oc.id") failed: Operation not supported (95)`.
After the state mount the client role now binds the node's own
/var/lib/infinito-backup over /var/lib/infinito/backup. Every consumer
keeps its path; the data stays on local disk with its attributes, and the
backup host pulls each provider over ssh as before. The bind carries
_netdev: without it, systemd-fstab-generator puts the bind into
local-fs.target below its NFS parent in remote-fs.target, and the boot
transaction cycles. The DR drill remounts the bind after its lazy remount
of the state share, because `umount -l` on the parent detaches it.
Closes#1012
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ubuntu-latest is rolling over to ubuntu-26.04, whose sudo is sudo-rs. It
ignores `-E` ("preserving the entire environment is not supported") and
hands the child only the explicit `env` assignments, so the swarm matrix
died at import with `KeyError: 'INFINITO_SRC_DIR'` before any inventory
existed (PR #1009, Nextcloud#0). Jobs that landed on ubuntu-24.04 kept
passing.
Every `sudo -E` now passes the caller's exported names through
`--preserve-env="$(compgen -e | paste -sd, -)"`, which both classic sudo
and sudo-rs honour. A lint rejects `sudo -E` and bare `--preserve-env`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.
The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.
call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A role-level `placement: manager` pins every service of the role and keeps
every volume node-local, so a role needing one node-bound service had to give
up NFS for all of them. GitLab's gitaly is that case: its repositories volume
is node-local (`nfs: false`, gitaly does not support NFS) and a reschedule
left the data behind on the old node.
`deploy.yml.j2` now resolves `services.<service>.placement` first and keeps the
role-wide list as the fallback, so gitaly carries the constraint alone while
webservice, sidekiq, workhorse, shell, migrations and gitlabmcp stay
schedulable and `gitlab_shared`, `gitlab_uploads` and `gitlab_builds` keep
their NFS backing. Role placement still comes from the primary entry, which is
what drives that backing.
Two things had to come with it. The swarm drill decided that a role is
manager-pinned by grepping the whole `meta/services.yml`, so a per-service pin
would have skipped GitLab's drain and reschedule steps while six of its
services still run on workers; it now reads the primary entry. And
`resource_filter` returns a present-but-null value rather than the hard
default, which rendered `node.role == None`, a constraint no node satisfies.
`test_no_placement_manager.py` now inspects every `placement` value instead of
only `manager`, requires the justification for each, and rejects a value that
is not a swarm node role, since a misspelled one hangs the converge wait
rather than failing the deploy. Requirement 023 records the shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 🐳 Workspace (ubuntu:latest) job on main (run 38879556215, 1b36ef6120)
failed test-unit-python in the hook's second make test:
AssertionError: Regex didn't match: '^ localhost=ok/\d+ms$'
not found in ' localhost=ok/-1612931482679767779ms'
ubuntu:latest is Ubuntu 26.04, whose date is uutils coreutils. uutils
0.7.0 ignores the field width in %3N and prints all nine nanosecond
digits, so `date +%s%3N` yields a 19-digit value instead of epoch
milliseconds. Against main's sampler, uutils 0.7.0 date reports
localhost=ok/7081859ms, and an unpadded nanosecond value turns the
difference negative.
answer_once now takes both timestamps from /proc/uptime, which the
kernel prints with two decimals whatever date the image ships. The
sampler already reads /proc/net/udp, so it was Linux-only before. The
resolution is 10 ms; the same probe reports 10 to 20 ms with uutils
date on PATH.
The new test puts a date on PATH that does not understand %3N; main's
sampler dies on it, the fixed one does not call date for the latency.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
No conflicts across 121 incoming commits. The one file both sides changed is
tasks/stages/01_constructor.yml: upstream parks an applications carrier for
the swarm bootstrap roles after the add_host tasks, and keys the carrier on
the tokens file as well. This branch's mesh bootstrap opens and drops its own
block-scoped carrier before that point, so both keep working as written.
Upstream also brings ruff 0.17.0, distro-derived env keys, a scoped act outer
teardown, the stalwart relay rule, the onion apex redirect and image and
version updates; ruff 0.17 reformats nothing on this branch.
down_act_outer.sh removed every container named act-*, so starting or
releasing a swarm drill in one worktree killed the act container of a
drill another worktree had running.
workflow.sh now labels act's job containers with
infinito.worktree=${INFINITO_CONTAINER} through the default
ACT_CONTAINER_OPTIONS, and down_act_outer.sh removes only containers
carrying the own label. Both scripts refuse to run without
INFINITO_CONTAINER, so the filter never widens to every act container;
make exports BASH_ENV, which loads it from .env.
Worktrees without a custom.env share the default container name and
therefore still share one label among themselves.
Closes#933
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The three-arm comparison on web-app-gitea#0 put the meshed arm at 414ms
per task over the direct arm, and the tunnel at 4% of that. What it could not
split is the rest: the arms differ both in connection plugin (docker against
ssh) and in escalation (root with no prompt against administrator with a
password sudo on every task).
A fixed set of module calls, timed either side of the transport switch with
become off and on, gives four numbers that separate the two. It costs about a
minute where a fourth deploy arm would have cost three hours, and it needs no
root login and no sudoers change.
INFINITO_SWARM_TRANSPORT_BENCH guards it, so a normal round times nothing.
The arm existed to answer one question: how much of the meshed arm's
cost is the connection plugin and how much is the tunnel. Runs 37592061960
(docker), 37753506661 (SSH on the underlay) and 37621556682 (SSH over
WireGuard) answered it on web-app-gitea#0, debian/ext4:
pass 1 5964s / 8511s / 8832s
pass 2 5319s / 7472s / 7705s
Leaving ansible_connection: docker costs +4700s, the tunnel +554s - 89.5%
against 10.5%. The answer is recorded; the scaffolding does not need to ship.
Reverts d792519725, 1b28f87b37, 89b8a8c8d1, 0470491e1a, 529c5ace43,
dd3d10a9a2, c20dd0e5f5, 458d8956e1, 63ade1e03b, d281555945 and 4adfb6e3d2.
95cfc8b7b7 is already superseded by bbfd82f89, which sets the profiling limit
upstream.
A role sync runs the whole role, several hundred tasks, although a
design carrier usually lives in one task file. Iterating on that file
cost minutes per try.
`make compose-role-sync role=<app> variant=<idx> task=<file>` now runs
only that task file of the role against the running stack. The sync
playbook gets a third scope `task` next to `design` and `role`: it
places the shared services like a role sync, loads the token store and
the proxy handlers, includes the role with `tasks_from` and flushes
its handlers.
The task file gets the variables of the role and nothing an earlier
task of the role would have set, so it has to stand alone. The script
accepts a task path of plain path characters only, because the value
travels into a command line inside the container.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
make compose-entity-purge removed the containers, volumes, databases
and vhosts of an app but left its group in devices.yml. Inventory
provisioning only merges groups, and the webserver health unit builds
its domain list from the groups of the host, so every later full deploy
of another app ended with failed=1: the unit kept probing the purged
domains and the service terminator gave up after 25 retries.
The make target now passes retire to the purge wrapper. With it the
wrapper hands the inventory directory into the container, and the app
orchestrator removes the purged application ids from devices.yml of the
base inventory and of every round inventory next to it.
The deploy flows that purge before they deploy the same apps again
(purge=true, the purge between matrix rounds, the reinstall selection)
call the wrapper without retire and keep the groups, because an update
deploy needs the app in the inventory.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
make compose-role-sync runs only the role of one deployed application,
with its proxy vhost and injected snippets. It marks every shared
service of the stack as converged and skips the stack-wide service
stop. With pw= it stages the Playwright spec of the role again, renders
its env and runs it. make design-sync keeps re-rendering only the
shared CSS, the role stylesheet and the branding assets. Both run
through one playbook-sync.yml with a scope, which replaces
playbook-design.yml.
The host placement of the proxy and the shared services moves from the
constructor stage into tasks/utils/constructor/placement.yml, so the
role sync can include it.
base='<hex>' on both targets passes DESIGN_BASE_COLOR. The development
inventory references that variable for the design base color and keeps
#FFA500 as its default, so a deployed role can be rendered and checked
with another palette.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Narrowing the meshed drill to its backup half bought budget by giving
up the recovery legs over the mesh. The budget overrun is worth measuring
before it is designed around, so the pre-drill pass and the full teardown and
recover come back. Reverts a27b093c4.
DRILL_STAGE=backup returns once the marker has travelled from the live
export through the deployed units to the LUKS device, before step 6 tears the
stack down. The loop device is closed on the way out, which the full path does
inline at step 7. Default stays full, so the direct arm is unchanged.
A round that drills has to deploy twice, so the caller needs the answer
before it deploys anything. Answering here keeps the rule deciding which
applications have something to restore in the script that already owns it,
rather than restating it in the caller.
The verdict is printed rather than signalled by exit status: a probe that
dies on the way to the decision must not read as 'no drill' and leave the
caller skipping the deploy the drill depends on.
Fourteen real conflicts plus the 66 generated READMEs.
utils/cache/files.py and its test: both sides added a skip-dir rule, resolved
as a union so the root `build/` prune and the per-root walk key coexist.
plugins/lookup/database.py and postgres_libraries.py: main's added lookups kept,
with get_entity_name renamed to entity_name as this branch renamed it.
roles/web-app-{mailu,matrix}/meta/services.yml: main's `covers: [email]` and its
MAIL_PROVIDER-gated email entry, plus this branch's gitlab/jira/n8n/jitsi
consumers, which the merge base carried on neither side.
group_vars/all/17_resource.yml: this branch's lookup('resource', ...) refactor;
main's single added literal is already covered by _POSTGRES_MAX_LOCKS_PER_TRANSACTION.
docs/requirements/035-mcp-proxy-expansion.md: this branch moved the design into
ADR 0003, so the requirement keeps only the user story and the criteria. Main's
delta inside the moved block was two new roles, carried into the ADR audit:
web-app-stalwart as adapter_candidate, svc-net-tor-smtp as privileged_control_plane.
The no_surface total said 99 while its own subcategories summed to 112.
inventories/development/default.yml: biber keeps no explicit email. Both sides fixed
the same CI failure; this branch fixed it in _apply_account_defaults, which renders
biber@{{ DOMAIN_PRIMARY }} (utils/cache/users/__init__.py:68) — byte-identical to
the address main hardcoded, so the inventory line is a second copy of one fact.
roles/web-app-{gitlab,keycloak}/files/playwright/playwright.spec.js and
roles/web-app-gitlab/templates/playwright.env.j2: union of both import sets, with
requireDotenvValue kept over decodeDotenvQuotedValue for CANONICAL_DOMAIN because
an empty value must fail rather than match every substring.
The 66 role READMEs take this side: aceb5295ef moved Cosmos, Quick Setup and
Credits to the documentation build, and test_readme.py now rejects them in a
committed README. Main's prose edits inside them are carried over by hand (the
Mailu deprecation banner, Mailu -> Stalwart in erpnext and zammad), and the two
READMEs main added fresh are stripped of the three generated sections.
roles/web-app-sphinx/README.md stays deleted; 6eb9a59d14 merged that role into
web-app-docs. The root README's generated role table was regenerated from the
merged metadata.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The entity name keys a role's instance directory, its compose project and its container name, and was derived only by stripping the longest matching category prefix. Two roles under different categories whose names share a suffix therefore collapse onto one entity and overwrite each other's rendered stack, which no provides: can repair.
A literal entity_name in vars/main.yml now wins over the derivation. web-app-seaweedfs-console takes its shorter name back and declares the entity seaweedfs-console, which frees the seaweedfs key in its own services map for the engine consumer and removes the provides:, covers: and alias entries that stood in for it. A new lint pairs the declaration with the primary key it has to match, because a drift between them would drop the role from the provider set in silence.
Compared across all 288 roles, exactly one entity changes, and it restores what main already keeps on disk, so no instance directory moves.
The function and the filter lose their get_ prefix. Two lints read that prefix as the discriminator between the variable and the filter that fills it; both discriminate on the pipe now. Ansible's templar resolves a filter and a same-named variable independently, verified against ansible-core 2.20.2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One conflict, in the MCP exclusion list of requirement 035. Both sides
edited the same line: main added svc-net-tor-smtp, this branch replaced the
three svc-net-wireguard-{core,plain,firewalled} entries with the single
svc-net-wireguard it deploys. Resolved to the union of both — 24 entries —
which is what the merged tree actually holds.
Everything else merged clean and was checked for overlap rather than
assumed: no reference anywhere still names the three replaced roles;
matrix.py kept main's PRIMARY_APPS alongside the reordered round, and the
bootstrap pass does not need it, since only one template reads it and
group_vars defaults it empty; and main's ansible.cfg addition sits in
[defaults], so the ssh_args the mesh tool derives its own connection
settings from are untouched.
121 lines of restated code, step narration, section banners and rationale
prose, in workflow YAML, compose overrides, group_vars, lookup and action
plugins, role tasks, shell scripts and tests.
Three lines go the other way: where a reason has to survive it is now a
registered nocheck directive carrying it, not free prose. Two of them point at
an invariant further down the same file, the third records why an end-of-life
object store keeps no Playwright surface.
Scope was decided per line rather than per file. Classifying by "no additions"
would have swept roles/web-app-erpnext/tasks along with it, where the removed
lines are delegate_to, environment and cmd — a behaviour change that has no
business in a comment commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The instructions test brings up a machine per role, so it can only ever
replay one guide. Which one it picked was a side effect of ordering: the
lookup returned the first invokable peer the round listed, and on a CI row
that is the role under test only because the row's apps happen to read
<role>,web-app-docs.
It now reads APP_ID, which the deploy workflow already sets to the row's app
and which the swarm routine already documents as the app under test. A round
that carries no APP_ID, or one whose app publishes no Production block, draws
an invokable peer at random instead, so a local round covers a different
guide each time rather than always the first one the inventory lists.
Ansible runs inside the infinito container rather than on the runner, and
that exec passes an explicit allowlist, so the name arrives only where it is
forwarded. Two seams carry it: _FORWARDED_ENV_KEYS, which covers CI and the
local initialize and reinstall verbs, and the update verb's own docker exec,
which assembles its arguments without that CLI.
The pin is tolerant rather than strict. APP_ID is set on every deploy row and
not only on documented ones, so refusing an app that publishes no guide would
let a generic variable abort a deploy.
Measured: the draw is observed over 200 rounds rather than assumed, and the
forwarding is negative-controlled, removing the key and watching the new test
fail on that key alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The package cache was configured in six places at once: the nginx upstream
map, the consumer compose override, pip.conf, .npmrc, the apt sources and the
twenty ensure_proxy_repo calls that bootstrap Nexus each spelled the same
hostnames, repository names and suites by hand. A new upstream meant editing
five files and hoping none was missed, and the docs told contributors to do
exactly that.
A role now declares what it needs under cache: in its meta/networks.yml, and
everything else is derived from it. svc-cache-package holds the general
upstreams; web-svc-libretranslate holds its own. All nine derived files are
generated and gitignored, so a second spelling of a hostname cannot exist.
utils.cache.hosts reads the declarations without PyYAML, because the .env
generator reaches them before any dependency is installed; it raises rather
than truncate when a comment sits at column 0 inside the block, which would
otherwise drop every declaration below it in silence. utils.cache.records
owns the Nexus record format and the Ubuntu mirror resolution, so the dev
stack and the role provision from one rendering.
A build may only resolve a cached host it can verify: the plain-HTTP hosts
are hijacked for every image, and the TLS hosts only for an image whose
Dockerfile installs the frontend CA. Hijacking all of them would fail every
pip and npm build with CERTIFICATE_VERIFY_FAILED.
svc-cache-package deploys the cache and its TLS frontend by calling the same
shell scripts the dev stack calls, rather than reimplementing them as tasks.
The role is invokable through the new svc-cache category and inert until a
host carries it.
Verified: the twenty Nexus payloads, the fifteen nginx server blocks and all
seven package-manager configs are byte-identical to the hand-written files
they replace, and the new category adds one role to the deploy order while
preserving the relative order of the other 180.
The role has not been deployed yet.
SWARM_REQUIRED_SERVICES no longer force-keeps wireguard: with the axis off the
role is absent from the inventory, so there is no disabled-service conflict to
work around.
`roles/` set as inline code reads as a literal path, yet the directory
holds an index page the build resolves. Writing it as a link costs nothing
and gives the reader the page; leaving it as code hides it, which is how
this branch degraded six working references in one pass.
The lint strips markdown links from a line before scanning it, or it would
report the code span inside a finished link, and it skips fenced blocks. It
reads the index filenames from the same place the link lint does.
Two false-positive classes came out of the first run and shaped the rule. A
span of nothing but dots and slashes is a path word, not a path: '/' names a
filesystem in a table of mount points and '../' names the traversal an
argument must not perform. An absolute span is skipped because this
repository writes Keycloak group paths that way, and /roles/web-app-bluesky
is a group, not a folder.
35 spans across 12 files become links, and the documentation build resolves
every one of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill strips the services it cannot run without from the incoming disable
list, and wireguard was not among them. For an app whose own graph does not
reach it, the provision step disabled the service while extend_inventory had
already placed svc-net-wireguard on every swarm host, and the services
disabler rejected the pair:
service 'wireguard' is disabled, but provider role 'svc-net-wireguard' is
still active in .../devices.yml
svc-swarm-node declares the service required for exactly this reason, so the
drill has no more business disabling it than it has disabling node.
server() now resolves a translation endpoint in three steps: use the running
service, rebuild when the host has a GPU but no accelerated deploy left its
stamp, otherwise deploy. A plain run therefore costs nothing, and the rebuild
fires once, on the move from a CPU-built container to an accelerated one.
deploy() brings the runner up, waits for State.Running and deploys the role
inside it. The exec steps address `docker compose exec i18n` directly:
cli.administration.deploy.development exec hardcodes the `infinito` service
(compose.py:261), so every deploy through it silently ran in the main
container, where mailu already holds port 25 and postfix cannot start.
check=True stops a failing step instead of carrying on against the wrong
target.
The deploy router writes its pid to build/deploy.pid, which survives the exec
into the subscripts, and deploying() refuses a translation while that process
lives. The running deploy is left untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
