Commit Graph
14682 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5 93990bbf07 docs(claude): say that the mandated tee pipe throws the exit status away
The logging rule requires streaming through `… 2>&1 | tee /tmp/<name>.log`, and
that pipe reports tee's status, not make's. An agent following the rule as
written reads exit 0 from a run that ended in `Error 2`.

This happened twice on 2026-09-19. Both times `make test` printed
`FAILED TARGETS: test-lint` and the invocation returned 0; only reading the
table caught it, once on a genuine addon-schema violation that would otherwise
have been committed.

The rule now names the trade it makes, so the table is the verdict and the exit
code is not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 06:32:09 +02:00
kevinveenbirkenbachandClaude Opus 5 eef3c3be71 fix(lmstudio): bound the model fetch by size, disk and a fallback source
A wedged LM Studio daemon cost CI job 105558224377 three hours: three kills at
the global 60-minute `CONTAINER_EXEC_TIMEOUT`, for a 531 MB model that a healthy
node pulls in 27 seconds. Nothing in the path was sized against the thing being
fetched, and nothing checked whether it could land at all.

Size. Each model now declares `bytes`, taken from the value the deploy already
printed as "OK (531068224 bytes)", and the pre-pull derives its own timeout per
model: max(preload_floor_seconds, bytes / preload_min_bytes_per_second). The
floor of 600s protects a small model from a brief stall; the rate of 1 MB/s sits
twenty times below the 21.98 MB/s a healthy pull reached and seven thousand
times above the 142 B/s the wedged one managed, so it separates the two without
guessing. Evaluated against ansible-core's filters: 531 MB and 10 MB both land
on the floor, 4 GB gives 4000s.

Disk. The seed measures free space with `df -P -B1` before fetching and refuses
with a message naming both figures in GiB, the factor and the file to edit. The
factor is three because the node cache, the staging copy inside the container
and the models volume all exist at once during the import; that is measured, not
padding.

Fallback. A model may list `mirrors`, extra bases tried after the source. The
mirror task is a no-op once the source delivered, because get_url skips a dest
whose checksum already matches: measured at 145.24s for the first pass against
0.64s for the second. The list ships empty, because the obvious candidate is
not one: hf-mirror.com answers 206 but resolves to us.aws.cdn.hf.co, so it is a
front end for HuggingFace and adds no redundancy against a HuggingFace outage.
The mechanism is here; an independent copy is the operator's to name.

The external check grew with the declaration: `bytes` is compared against the
size the API publishes, and every declared mirror must still serve the pinned
file, or it is redundancy on paper only.

01_manager_ops.yml's rescue rationale claimed that restarting the container
clears a wedged transfer. Job 105558224377 shows the restart returning rc 0
without clearing it, so the claim now says what was measured instead.

Residual: the mirror path has never run in a deploy, since no mirror is
declared. Its loop expression is verified against ansible-core's subelements
over four shapes, including the zero-iteration case when nothing is mirrored.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 06:31:48 +02:00
kevinveenbirkenbachandClaude Opus 5 835584e2c9 test(addons): catch a pinned release that upstream retired
An addon may pin `config.archive` so the deploy fetches that release instead of
letting the app store choose. The pin is worth exactly as long as the artifact
outlives it: when upstream retires the release, every deploy of the role fails
at install time with a 404 and nothing in the failure names the declaration as
the cause.

That is not hypothetical. ONLYOFFICE listed onlyoffice 10.2.0 in the Nextcloud
appstore and never published the artifact, which made Nextcloud#4 red in both
runs 35329488812 and 35330645276 on the same commit. This check answers the
mirror image of that question: does the release we pinned still exist.

The split follows the sibling model-pin check. The declaration's shape is a
repository fact and is asserted; whether the archive is still served is a
network fact, so a timeout or a refused connection reports unverified rather
than failing, because a firewalled runner must not turn a good pin red. A 4xx
is an answer that contradicts the pin and does fail.

Negative control: pointing the pin at the retired v10.2.0 produces

    web-app-nextcloud.onlyoffice: .../v10.2.0/onlyoffice.tar.gz answered HTTP 404

so the check would have caught today's failure before it reached CI. The
unverified path is exercised too: with the sandbox blocking the host, the run
warns instead of failing.

Residual: this proves reachability, not compatibility. A pin whose max-version
sits below the deployed server major passes here and fails loudly later at
`app:enable`. Today the single pin is v10.1.2, which the store declares for
>=33.0.0 <35.0.0 against a role that deploys 34.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 06:30:36 +02:00
kevinveenbirkenbachandClaude Opus 5 30c3d699fd docs: name the workflows the prefixing sweep left behind
71888d86af moved twelve workflow files behind trigger prefixes and updated the
references GitHub validates. Three mentions in prose kept the old names: an echo
line, a file header and a lint's failure message. The same commit already fixed
the REST dispatch URL two lines below the echo it missed.

    roles/svc-runner/files/test/external.sh:36      test-runner-smoke.yml
    scripts/github/runner/cleanup_docker.sh:4       images-build-ci.yml
    tests/lint/repository/test_distros_spot.py:116  test-workspace.yml

None of them executes; the workflow-reference lint deliberately does not reach
them, because outside .github/workflows a bare *.yml collides with services.yml,
compose.yml and variants.yml. A repo-wide grep over all eleven legacy names now
returns nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 06:03:40 +02:00
kevinveenbirkenbachandClaude Opus 5 673de05a60 fix(inventory): write only the service toggles a role's contract declares
`disable=` wrote both `enabled: false` and `shared: false` for every role that
declares the service, but a role qualifies with any one of `enabled`, `shared`
or `provides`. compare_application_keys rejects any host_vars key the role
contract does not carry, so a role declaring only `enabled` came back as a
validation error and the deploy never reached Ansible:

    Missing default for web-svc-cdn: services.dashboard.shared
    Missing default for web-svc-mirror: services.dashboard.shared
    [ERROR] Inventory validation failed. Aborting deployment.

_declared_toggles reads the role's meta/services.yml and writes only what is
there; a role carrying neither toggle is skipped instead of gaining an empty
block. The module already held this principle: find_roles_with_service excludes
pure metadata entries so `disable` "doesn't add bogus enabled: false overrides",
and this applies the same rule per flag rather than per role.

Delta measured over the disable list the compose iteration procedure prescribes
(matomo, dashboard, prometheus, email, css): 407 of 409 role/service pairs are
untouched because they declare both toggles, exactly 2 stop receiving a
fabricated key, and 0 fall out of the disable entirely. The 2 are the pairs the
validator rejected. Across all 266 roles with a meta/services.yml, 1224 service
blocks declare `enabled`, 1190 declare `shared`, 34 declare `enabled` alone and
none declares `provides` alone, so nothing loses its disable.

Proven by the deploy sequence rather than by a test: the first local deploy
aborted at inventory validation, the second passed into the Ansible run with the
same disable list. Two regression tests cover both shapes, and the negative
control holds: hard-coding ["enabled", "shared"] fails the new test.

Without this, `disable=` is unusable in every local deploy, which the iteration
procedure requires on every run.

Residual: the generator was exercised through compose-deploy only, not through
roundtrip or swarm-zombie, which reach the same function by other call paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 06:00:56 +02:00
kevinveenbirkenbachandClaude Opus 5 9d940f4040 fix(nextcloud): install onlyoffice from a release that upstream published
ONLYOFFICE listed onlyoffice 10.2.0 in the Nextcloud appstore on 2026-09-18 at
10:16:31Z and never published the matching GitHub artifact; the tag itself does
not exist. `occ app:install` takes the newest compatible release, so it dies on
a 404 that no retry can heal. CI proved that twice independently: upstream run
35329488812 and fork run 35330645276 both fail Nextcloud#4 at that task after
six attempts, on the same commit, for the same reason.

The fault had three layers, each visible only once the one above it was gone.

Install. meta/addons/onlyoffice.yml pins `config.archive` to v10.1.2, whose
artifact answers 200 and which the appstore also lists for this server major.
01_install.yml stages that archive on the node and unpacks it into custom_apps
through a stdin pipe: the webroot is a named volume, so the node can only reach
the app directory through the container, and piping avoids both `container cp`
and a cleanup step that would have to run as a user which does not own the file.

Enable and configure needed no change at all. With the release already on disk
`occ app:install` reports "already installed", which nextcloud_install_status
has always mapped to runnable, so 02_enable.yml and 03_configure.yml proceed as
they always did. That is why this commit adds no branch to the plugin pipeline.

Update. `occ app:update --all` then dragged the pinned app back to the broken
10.2.0 and killed the deploy. occ offers no exclusion flag, but it does offer
`--showonly` and single app ids, so a pinned app is simply not updated. The old
`--all` call stays byte-identical and now carries a `when`: with nothing pinned,
thirteen of the fourteen variants take exactly the path they took before. This
layer cannot appear in CI at all, because it needs a second deploy with the app
already installed, and the install always died first.

The classifier test is inverted rather than deleted:
test_an_advertised_release_whose_artifact_is_gone_stays_fatal pins the decision
that a dead artifact fails loudly instead of degrading to a silent skip.

Verified by a local compose deploy of variant 4: ok=2508 changed=149 failed=0
and `✓ 36 tests/addons/onlyoffice.spec.js:12:1`, which drives the whole
JWT-authenticated round trip to the document server over its onion address, not
merely an enabled plugin. In the live container beforehand: `occ app:enable`
returned "onlyoffice 10.1.2 enabled", `occ app:install` returned "already
installed", and `occ app:update onlyoffice` reproduced the 404 that the pin now
avoids.

The loop expression was wrong before this commit and is fixed here: Jinja reads
'\1' as an octal escape, so the backreference produced chr(1) instead of an app
id. Evaluated against ansible-core's own filters over four shapes, the tail is
now cut instead of captured, which removes the escaping hazard entirely.

occ output is not localised: with default_language=de and a de_DE.UTF-8 locale
the strings are unchanged, so parsing them is as safe as the five markers the
role already relied on.

Residual: the per-app update branch has never run with a real item inside a
deploy. Its chain and `occ app:update <id>` are each verified separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 05:55:51 +02:00
kevinveenbirkenbachandClaude Opus 5 4e7f14a29d fix(lmstudio): let the model import answer its prompt and cross the volume
The seed path added in ee582762b0 fetched the model and never handed it over.
Two independent faults sat on top of each other; the second only became visible
once the first was gone.

First, the prompt. `lms import` keeps asking "Do you wish to continue? (Y/n)"
even with `--yes`, which suppresses only the warning about moving the file.
import.sh ran it through `container exec` without `-i`, so the question met a
closed stdin and the script died with 130, printing neither PRESENT:, ABSENT:
nor IMPORTED:. The sibling call in tasks/utils/preload.yml already passes `-i`.

Second, the move. `lms import` renames by default. `container cp` stages the
file in /tmp while MODELS_DIR lives on the mounted volume, so rename(2) crosses
a device boundary and cannot work:

    EXDEV: cross-device link not permitted, rename '/tmp/...gguf'
    -> '/root/.lmstudio/models/lmstudio-community/...'

Hence `--copy`, and hence the `rm` that clears the staged file copying leaves
behind. That path could therefore never have succeeded, which is why every CI
job since 2026-09-17 fetched 531068224 bytes, verified their sha256 and threw
them away.

Dropping `failed_when: false` is what makes the next failure visible: it
reported the abort as `ok` in 1.40s. The task name loses "best effort" for the
same reason.

Measured in a local compose deploy: IMPORTED:Qwen2.5-0.5B-Instruct-Q8_0.gguf,
then `lms get` returns rc 0 after 0.895s instead of a 60s network abort followed
by three 3600s timeout kills, both play recaps failed=0. The short-circuits still
hold: on the second pass the node cache check costs 0.64s instead of 145.24s and
import.sh prints PRESENT: before it copies anything.

Residual: rc 130 was measured on compose only. Some of the eight matrix rows
that pull svc-ai-lmstudio take the async swarm path, where the import is
unverified.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 05:46:58 +02:00
infinito-nexus-updater[bot] ade1b11861 update(images): update Docker image versions (web-app-socialhome) 2026-09-19 03:20:52 +00:00
infinito-nexus-updater[bot] e0f5295ccf update(images): update Docker image versions (web-app-prometheus) 2026-09-19 03:20:44 +00:00
infinito-nexus-updater[bot] cc7e774a89 update(images): update Docker image versions (web-app-peertube) 2026-09-19 03:20:36 +00:00
infinito-nexus-updater[bot] 80bd969b4e update(images): update Docker image versions (web-app-opentalk) 2026-09-19 03:20:32 +00:00
infinito-nexus-updater[bot] 147dc2707f update(images): update Docker image versions (web-app-keycloak) 2026-09-19 03:20:18 +00:00
infinito-nexus-updater[bot] b642e16e62 update(images): update Docker image versions (web-app-funkwhale) 2026-09-19 03:20:10 +00:00
infinito-nexus-updater[bot] 64267bea15 update(images): update Docker image versions (web-app-baserow) 2026-09-19 03:20:00 +00:00
infinito-nexus-updater[bot] afe2bed2a9 update(images): update Docker image versions (svc-ai-ollama) 2026-09-19 03:19:43 +00:00
evangelostsak 6a0cbb916b fix(swarm): give the controller the lab address for its NFS mount
The cluster mounts the export over the mesh; the act runner is not a member
and mounts it over the lab network. Both addresses now travel in the extras
rather than one serving two topologies.
2026-09-19 00:29:25 +02:00
evangelostsak 871ec84b15 fix(storage): mount the controller export on the controller's own address
The delegated controller mount used the cluster source, so once the NFS server
was addressed on the mesh it timed out: 'mount.nfs4: Connection timed out for
10.100.1.11'. The controller is deliberately not a mesh member, so that address
was never reachable from there.
2026-09-19 00:29:15 +02:00
evangelostsak dc57fd6ae2 feat(storage): expose the controller-facing NFS source
Defaults to the cluster source, so a deployment without a mesh is unchanged
and only one that separates the two planes has to set controller_server.
2026-09-19 00:29:06 +02:00
evangelostsak 452c7dc42e feat(storage): let the NFS source differ for the controller
The controller reaches the export on its own address. Once a VPN joins the
cluster and the controller is not a member, the server's mesh address is
unroutable from there, so one source cannot serve both.

Opt-in through a term, so every existing caller keeps the single-address
behaviour and only the controller mount asks for the other one.
2026-09-19 00:28:54 +02:00
evangelostsak 6dc4f89b86 fix(swarm): bring the mesh up before the swarm bootstrap
svc-swarm-node is staged explicitly in the constructor rather than reached
through the dependency walk, so the run_after edge declared against
svc-net-wireguard never ordered anything. Swarm initialised first, advertised
10.100.0.1, and docker refused: 'the address to advertise is not recognized as
a system address' -- the interface did not exist yet.

Bootstrapping WireGuard immediately before it puts the ordering where the
staging actually happens. The role keeps its own run-once guard, so the later
dependency-walked include is a no-op.
2026-09-18 23:41:15 +02:00
evangelostsak a5424b35b4 fix(swarm): ship ping in the test node image
The MTU probe skips itself where ping is absent, so without this the
VXLAN-inside-WireGuard check would silently never run on the NFS node -- the
one member whose path to the hub crosses the plane boundary.
2026-09-18 23:23:26 +02:00
evangelostsak c30ef534b1 fix(wireguard): prove the tunnel with the handshake, not with ping
The gate pinged its hub, and the NFS node ships no ping. The tunnel there was
healthy the whole time -- handshake 46 seconds old, 7.37 KiB in and 27.10 KiB
out -- and the deploy still failed, so the gate was reporting on its own
tooling rather than on the mesh.

wg reports the handshake directly: it needs nothing the host might not carry,
and it stays true where ICMP is filtered, which a tunnel gate must survive.
A peer that never completed one reports a literal 0.

The MTU probe still needs ping, because only a sized DF packet can show the
VXLAN-inside-WireGuard trap, so it is skipped rather than failed when ping is
absent. The handshake above already proves the tunnel; this proves the frame
size, and losing the second check is not worth failing a working deploy.
2026-09-18 23:22:49 +02:00
evangelostsak d3a990454f fix(swarm): address swarm and NFS on the mesh, not the underlay
The extras file pinned swarm.manager.advertise_addr and storage.nfs.server to
the lab addresses, and extras override the group_vars default. Every tunnel
came up and carried nothing: swarm kept advertising 192.168.244.10 and the
clients kept mounting 192.168.244.13, so the mesh existed beside the traffic it
was built for rather than under it.

Read from the host_vars the mesh writer produced earlier in the round rather
than recomputed, so the address allocation stays in one place, with the
underlay kept as the fallback for a round that has no mesh.

Caught in a drill whose gate had just gone green: tunnels healthy, handshakes
current, and swarm still on the underlay.
2026-09-18 20:19:09 +02:00
evangelostsak 7e45a85cbb fix(wireguard): gate on the direction the host can actually initiate
The gate had every host ping every peer it was configured for, which a hub can
never satisfy. Only the spoke carries an Endpoint; the hub learns a spoke's
address when that spoke dials in, so a hub probing its spokes is racing them.
The backup host settles it: deployed in a later play, its interface cannot
exist while the cluster play runs, so the manager's data mesh was unprovable by
construction.

Observed in a drill where both meshes were demonstrably healthy -- the workers
had handshakes and traffic on the swarm mesh, the NFS server on the data mesh
-- and the gate still failed on the manager and the NFS server.

A spoke now probes the hub and nothing else. That is the failure worth
catching: a spoke that cannot reach the hub has no mesh, while the hub's own
view converges as spokes arrive. The retry window widens because hosts
configure in parallel and a spoke can probe before the hub's interface
exists.
2026-09-18 19:01:12 +02:00
evangelostsak cd586979a1 fix(swarm): ship iproute2 in the test node image
wg-quick shells out to ip, and the NFS node has no iproute2: the swarm nodes
only carry it because Docker pulls it in, and that node runs no Docker. The
interface came up on the manager and both workers and failed on the NFS server
alone with 'ip: command not found'.

Installed for every node rather than only that one, so the image stops
depending on Docker to provide a base networking tool that a WireGuard member
needs directly.
2026-09-18 18:26:16 +02:00
evangelostsak 2b8be66a43 fix(swarm): ship wireguard-tools in the test node image
The image carries linux-image-rt-amd64 pending and unsatisfiable, so apt
refuses every install on a node with 'unmet dependencies' -- including one
whose own package resolves fine. The wireguard install therefore failed on
all four hosts for a reason that has nothing to do with WireGuard.

Installing at image build time avoids the broken state entirely: the same
place already pre-installs kmod, cryptsetup and e2fsprogs, which are exactly
this kind of host-level tooling the drill needs before any role runs. The role
keeps its own install for real hosts and now finds the binary already present
here, so the guard skips it.
2026-09-18 18:03:20 +02:00
evangelostsak 85d05e2407 fix(wireguard): probe the tooling by path, not by a shell builtin
The guard used ansible.builtin.command with 'command -v wg-quick'. The command
module execs without a shell and 'command' is a shell builtin, so there is no
binary to exec, the probe reported a non-zero return code on every host, and
the install ran everywhere the guard existed to prevent it.

Verifying the probe through swarm-exec hid the bug, because that path does go
through a shell. stat has no such gap: it needs no shell and answers the
question directly.
2026-09-18 17:38:40 +02:00
evangelostsak 424017291c fix(wireguard): skip the package install when the tooling is present
The swarm node images already ship wireguard-tools, and they also carry a
half-installed linux-image-rt-amd64. apt therefore refuses the install with
'unmet dependencies' even though it reports wireguard-tools is already the
newest version, so the role failed on all three nodes with nothing actually
missing.

Checking for wg-quick first keeps the role from doing work that is already
done, and stops an unrelated broken package state on the image from failing a
deploy the layer would otherwise complete.
2026-09-18 17:06:37 +02:00
evangelostsak 0cbdf65636 fix(swarm): place svc-net-wireguard on every mesh member
The role only reached the three swarm nodes, which pull it as a dependency.
The NFS and backup hosts are spokes of the data mesh and got no placement at
all, so they never brought an interface up while the writer had already given
them keys and addresses.

That fails in a misleading place: the inventory describes the mesh correctly
and the hub holds peers for both, so the first symptom is the manager's gate
reporting peers unreachable rather than anything pointing at placement.

Observed in a live drill, where the role was included for swarm-mgr-01 and the
two workers only.
2026-09-18 17:05:06 +02:00
kevinveenbirkenbachandClaude Opus 5 e7ade5fb27 docs(claude): make an agent pin a green run to the line that ran
A job conclusion aggregates "passed" and "did not run" into one colour, and
reading the colour turns an unexercised change into a claimed proof.

Measured on 2026-09-18 in run 35330645276: jobs 105558220731 (Fider#1) and
105558220741 (Fider#0) are both green. Fider#1 logs
"-  4 tests/playwright.spec.js:56:1 > fider: admin sso login", where the leading
dash is Playwright's skipped marker, because that variant has SSO off and the
spec opens with test.skip(!oidcEnabled, ...). Only Fider#0 logs the same spec as
"✓ ... (1.7s)" and actually exercised the fix.

The second rule covers the fork. Tags do not follow a fork and repository
variables differ, so a step gated on either never runs there: the release cron
is 25 of 25 green on the fork and 25 of 25 red upstream, because the fork holds
2 tags against upstream's 60+ and `if: steps.pick.outputs.tag != ''` can never
become true. In this checkout origin is infinito-nexus/core with
gh-resolved=base while pushDefault is the fork, so a bare `gh run list` shows
upstream while push runs land on the fork.

Neither rule is enforceable by a check, because "which line proves this fix" is
not machine-decidable. They are written where the agent reads them and recorded
in agent memory so they survive the file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:59:04 +02:00
kevinveenbirkenbachandClaude Opus 5 8f6c31056b test(variants): refuse two variants that declare the same override
Eight meta/variants.yml repeated their last entry verbatim. The six svc-* roles
held the same tor.enabled override twice; fusiondirectory and phpldapadmin held
a third entry identical to their second, nocheck comments included.

The matrix plans a row per entry, assigns it its own axes and spends a deploy
job on it, around a hundred minutes here. The repetition is invisible in the
plan table, which lists rows by role#variant, so the two read as different rows
and only their identical embeds and weight betray them.

test_variants_distinct.py compares the parsed payload rather than the file text,
so key order, indentation and comments do not hide a duplicate, and null
normalises to {} because a bare list item and an explicit {} declare the same
no-override entry. A role whose variants are told apart by something outside the
entry opts the file out with `# nocheck: variants-duplicate` and a reason; no
file needs it today.

Every deletion is the trailing entry, so the surviving indices keep their
meaning and nothing is renumbered. The vanishing tokens are svc-db-elasticsearch#1,
svc-db-memcached#1, svc-db-rabbitmq#1, svc-db-redis#1, svc-db-typesense#1,
svc-prx-openresty#1, web-app-fusiondirectory#2 and web-app-phpldapadmin#2;
`git grep` against the commit finds no consumer naming any of them.

Negative control executed: re-adding the redis duplicate fails the lint with
"svc-db-redis: variants #0, #1 declare the same override", removing it passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:57:24 +02:00
kevinveenbirkenbachandClaude Opus 5 57fbd3f37a fix(ci): dispatch the release workflow by the name it now has
cron-release-highest.yml ran `gh workflow run release-version.yml`, a file
renamed to call-release-version.yml by 71888d86af on 2026-08-07. GitHub does not
error on a stale name: it resolves it against the deleted workflow's tombstone
(upstream id 226809803, fork 248877824, both state=deleted) and answers HTTP
422. The step is the job's last, so the whole run went red. The last 25
scheduled upstream runs failed there, the streak reaching back to 2026-08-09.

This removes the 422. It does not make releases green: workflow 226809803 last
succeeded on 2026-07-08, and from 2026-07-09 through 2026-08-08 the dispatch
worked while every run still failed at "Build missing CI images (ubuntu)".
Expect that failure to surface next.

test_workflow_references.py was meant to survive exactly this rename and could
not: its pattern required a directory prefix, so a bare filename was invisible,
and it reported zero offenders against twenty real ones. It now also matches a
bare *.yml inside .github/workflows, where every such token names a sibling, and
it checks both manual-dispatch forms, `gh workflow run` and a REST POST to
actions/workflows/<file>/dispatches, for existence and for a workflow_dispatch
trigger. Outside that directory the prefix stays mandatory, because a bare name
collides with services.yml and compose.yml.

The nineteen README labels are the same sweep's leftovers in the mermaid
diagrams; the widened lint fails without them.

Negative controls executed per branch: reverting the dispatch name, and pointing
the REST call at a missing workflow and at a reusable-only one, each fails the
matching check, and restoring makes them pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:54:21 +02:00
evangelostsak 160b0eaff7 fix(inventory): route the whole pool through the hub from a spoke
A spoke's AllowedIPs for the hub was its own mesh subnet, so a worker on
10.100.0.0/24 had no route to the NFS server on 10.100.1.11 and could not
reach the plane it is deliberately not a member of -- which is the entire
reason the hub sits in both.

The symptom would have been misleading: every tunnel reports a healthy
handshake, the gate passes because it only pings peers the host is configured
for, and the failure surfaces later as an NFS mount that hangs. It points at
storage rather than at routing.

Spokes now route the declared pool through the hub while the hub keeps each
spoke pinned to a single address; routing the pool in both directions would
blackhole every member behind whichever spoke came first.

Found by running the writer against the prefixed five-host topology before the
deploy rather than during it.
2026-09-18 16:37:18 +02:00
kevinveenbirkenbachandClaude Opus 5 2567c5ece1 feat(ci): let a manual run set the chunk size the formula would derive
chunk_size() multiplied the runner slots a chunk may assume by the waves the
queue window affords, and nothing could say otherwise. The derivation is the
right default -- too few rows leaves runners idle, too many push a chunk's tail
past the queue cancel -- but a run that wants smaller chunks had to edit
default.env to get them.

INFINITO_CI_CHUNK_SIZE now takes precedence when it is greater than zero, and
zero keeps the derivation. The input travels the same path offset already
travels: entry-manual-steer.yml offers the field and maps empty to 0,
call-orchestrator.yml hands it to all four chunk blocks, call-test-deploy.yml
exports it as the env var. It is a string rather than a number for the reason
resume_from_chunk already documents: a workflow_dispatch number input reaches
the caller as a string, and a number-typed workflow_call input then makes
GitHub refuse the job before it exists.

Two places in call-test-deploy.yml set it, not one. The plan summary resolves
the chunk size itself, so exporting it only for the app resolution would print
a plan that disagrees with the matrix the run then deploys.

Measured against the live helper: 0 gives 80 rows over 3 chunks, 25 gives 25
over 4, and returning to 0 restores 80 and 3.

The gate found three faults in the first attempt and all three are in this
commit: the key sorted after INFINITO_CI_CONCURRENCY rather than before it, the
dispatch description ran to 81 characters against a limit of 80, and the
retrigger fixture did not carry the new input. That last one matters beyond the
test: it is the check that would otherwise let a retrigger drop the chunk size
silently. carried_inputs() itself needed nothing, since it derives from the
declared inputs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 11:25:12 +02:00
kevinveenbirkenbachandClaude Opus 5 55b5a3dd93 fix(fider): run the build that reads the private-network opt-in
0d1ba1777f set ALLOW_PRIVATE_NETWORK_TARGETS=true in July with the right
diagnosis: Fider's SSRF guard refuses the OAuth token exchange when the
Keycloak host resolves to an internal address, so login 500s and never
completes. The image pin stayed at v0.36.0, which does not read that variable,
so the fix has been inert ever since.

Read out of the binaries rather than the release notes: `stable` contains the
string ALLOW_PRIVATE_NETWORK_TARGETS, v0.36.0 contains it zero times. Upstream
added the opt-in on 2026-06-24 and the check now returns before the hostname is
resolved, which is what puts the OAuth token path under it as well.

CI showed this as three red specs on Fider#0, admin sso login, biber sso login
and the seaweedfs tenant-logo upload, all with a missing .c-menu-user. The
artifact says why the selector is missing: the page is Fider's own 500 screen,
and fider.log carries the cause four times over, `Token URL is not allowed:
This URL is not allowed because it targets a private or internal network
address (app/services/oauth/oauth.go:320)`. The selector was right; there was
no session to render.

The tag is `stable` and not a version because upstream published v0.36.1 on
GitHub on 2026-07-03 and never pushed the matching image. Docker Hub's newest
release tag remains v0.36.0, and `stable` was rebuilt three days after that
release, carrying the opt-in. The marker on the line records both that and the
condition for removing it.

This is a moving tag in a repository that has just finished pinning fifteen of
them, and the mirror copies by tag, so an existing copy of `stable` can go
quietly stale. That is the accepted cost of the only build upstream ships with
the fix, and the TODO is what closes it.

audit.md moves with it: the report is generated from the pins and the lint
fails when the committed copy differs from a fresh generation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 10:52:02 +02:00
evangelostsak ac0f08f181 docs(mcp): dispose of svc-net-wireguard as privileged control plane
The role alters host routing and forwarding, so it belongs with the roles that
require an audited operations gateway rather than general-purpose application
MCP. Absence is not a disposition, and the completeness lint enforces that.
2026-09-18 10:36:24 +02:00
evangelostsak 10d920e15b feat(swarm): write the meshes between inventory extension and deploy
Placed after extend_inventory, which is what creates the group memberships the
meshes resolve from and the sibling backup.yml the data mesh spans, and before
the deploy, which is the first step that needs a tunnel.

Both inventory files go in one invocation. A mesh written from only the cluster
inventory would silently omit the backup host, and nothing would notice until
the backup play could not reach the hub.
2026-09-18 10:36:14 +02:00
evangelostsak cccfc3362c feat(swarm): order swarm initialisation after the WireGuard layer
run_after plus the matching service entry is what puts the tunnel up and proven
before swarm binds to a mesh address it would otherwise not have.

The flag is literal rather than group-gated, against the usual convention:
WireGuard is mandatory for every swarm deploy and svc-swarm-node exists only in
one, so a dynamic flag would describe a state the deploy must never reach.
2026-09-18 10:36:05 +02:00
evangelostsak 6c2da2d111 docs(wireguard): document the role and how the mesh reaches the inventory
States the ordering the role depends on but cannot enforce: the mesh must be
written across every inventory of the deploy, including any sibling file
holding hosts the cluster inventory does not, before the play runs.

Also records the one operational trap: a peer with no recent handshake whose
config looks right means the inventory and the running interface disagree, and
the fix is to re-run the writer rather than edit /etc/wireguard, which the next
deploy overwrites.
2026-09-18 10:35:51 +02:00
evangelostsak cebb1935b9 feat(wireguard): fail the deploy at the gate rather than later
The layer is binary: either every swarm deploy proceeds over the mesh or none
does. The gate runs before swarm initialises and names the unreachable peer, so
a partition is reported where it happened instead of surfacing as an unrelated
role failing to reach a service.

Three separate checks, because they fail differently. An interface with no peer
means the inventory is incomplete. An unreachable peer means the mesh is
partitioned. A full-size payload that does not arrive means the MTU is wrong
for VXLAN inside WireGuard, which otherwise drops only large packets and reads
as flaky CI.
2026-09-18 10:35:35 +02:00
evangelostsak c6f2b9fc27 feat(wireguard): restart each interface rather than only starting it
wg-quick up on an already-up interface is a no-op, which would leave the old
keys live in the kernel after a rotation rewrote the config. Taking the
interface down first is what makes a rotated deploy actually take effect; the
down half is tolerated because it exits non-zero when nothing was up.
2026-09-18 10:35:25 +02:00
evangelostsak f595853d66 feat(wireguard): install, render and bring up the host's interfaces
Fails immediately when a swarm host carries no mesh, rather than installing
tooling and producing nothing: an absent mesh means the cross-host writer never
ran, and every later symptom would point somewhere else.

Forwarding is enabled only on the hub, the one member with peers in both
planes, and therefore the only host that has to route between them.

The template call keeps ansible.builtin.template deliberately: this is per-host
system config, and gating it on the stack host would leave every spoke without
an interface.
2026-09-18 10:35:00 +02:00
evangelostsak 533b18b02d feat(wireguard): load the role core behind the run-once and swarm guards 2026-09-18 10:34:49 +02:00
evangelostsak 773d7f01d1 feat(wireguard): render one interface per mesh the host belongs to
ListenPort is emitted only on the hub, since spokes dial out and never accept.
A spoke's peer entry for the hub carries an Endpoint plus a keepalive so the
tunnel survives NAT and re-establishes without traffic; the hub's entries for
its spokes carry neither, because it learns their addresses from the handshake.

AllowedIPs asymmetry is what makes the routing work: a spoke routes the whole
mesh subnet at the hub, which is how it reaches the far plane, while the hub
pins each spoke to a single address.
2026-09-18 10:34:16 +02:00
evangelostsak b5cc14c90d feat(wireguard): resolve this host's meshes from the inventory
The role plans nothing; it reads what the cross-host writer already agreed on,
so a host can only configure peers that pass saw.

The private keys are looked up one literal path per declared mesh rather than
by building the path from the loop variable. A dynamic config path cannot be
resolved by the lookup validator, so it could not be proven correct before a
deploy -- and an unresolvable credential surfaces as an interface that renders
with an empty PrivateKey and never handshakes.
2026-09-18 10:34:01 +02:00
evangelostsak af23a2798b feat(wireguard): declare the per-mesh private keys as plain credentials
Declared so the config-lookup validator can resolve the path before a deploy,
and plain so the per-host credential generator leaves the value to the
cross-host mesh pass: the public half has to appear in a different host's peer
list, which no single-host generator can arrange.

On a worker the data-mesh key stays unset, because only the manager, the NFS
server and the backup host are members of that plane.
2026-09-18 10:33:48 +02:00
evangelostsak b347ab841f feat(wireguard): declare the wireguard-tools package per distro
Userspace tooling only; the data path is the in-kernel module, which every
supported distro ships.
2026-09-18 10:33:22 +02:00
evangelostsak 26e299cc48 feat(wireguard): declare the service as host-mode and swarm-only
Host mode, because the interfaces are kernel objects rather than containers,
and enabled only when DEPLOYMENT_MODE is swarm so a compose deploy neither
installs the tooling nor runs the gate.
2026-09-18 10:02:53 +02:00
evangelostsak 6a35e0cc2d feat(wireguard): mark the role as having no consumer-facing upstream
WireGuard is a kernel protocol rather than a product with a landing page, so
the media requirement is suppressed the same way svc-storage-nfs-server does
for NFS.
2026-09-18 09:53:34 +02:00
evangelostsak 1063a0bde9 feat(wireguard): add the galaxy metadata for svc-net-wireguard 2026-09-18 09:53:15 +02:00