Swarm still fails hard on a host without a mesh. A compose host in the group
without one, such as the role's own CI row, prints why and changes nothing.
Tightening the retry condition was not enough. `failed_when: false` is
evaluated after the retry loop, so a task that runs out of attempts reports
`failed_when_result: false` and ends ok: the role carried on past a spoke
that never handshaked, and past a `wg show` that failed outright, whenever
the MTU probe was skipped.
The explicit failure is back, with the condition the wait now uses, so the
verdict is taken from the registered results rather than from the retry
outcome.
Measured on a three-case play, exit status of the run: a reported handshake
continues (0), '\t0' stops it (2), and a probe exiting non-zero with no
output stops it (2).
`wg show latest-handshakes` ran with `failed_when: false`, so a failed probe
left stdout empty. An empty string carries no '\t0' to match, the `until`
condition was satisfied on the first attempt, and the gate passed on a host
that never reported a handshake at all. The task meant to catch that could
not: Jinja's `default` replaces an undefined value, not an empty one, and
exhausted retries fail the task before a later task runs, so it was
unreachable in every case that mattered.
The wait now requires a zero exit code and a non-empty result as well as a
handshake, and the exhausted retry is the hard failure.
Verified with a local play over three cases: a reported handshake passes,
'\t0' fails, and a probe exiting non-zero with no output fails -- the last
of which passed under the previous condition.
wg-quick down on the interface Ansible is connected over kills the session
carrying the play, and the up that would restore it never runs. syncconf
applies new keys and peers to the live interface, so a rotated mesh converges
without dropping the connection that is rotating it.
Latent until now only because the deploy still runs over the underlay.
The install no longer skips when wg-quick happens to exist: the node image
stopped pre-baking it, so the role is what puts it there.
The tunnel and overlay MTUs are now measured with automtu against the hub's
underlay endpoint rather than hardcoded. Fixed values contradict the MTU
automtu already derives for the docker daemon, and the mesh addresses cannot
be probed because the interface those MTUs configure is not up yet.
Declaring meta/meshes.yml made the validator accept the generated block, and
handed every host both meshes whether or not it is a member. The role then
rendered an interface for a mesh with no address and died in the template:
'object of type list has no attribute 1', from splitting an empty subnet.
The failure pattern named the cause exactly -- the NFS server failed on the
swarm mesh, the workers on the data mesh, and the manager, which belongs to
both, passed.
An entry the writer filled in carries an address; a declared-but-unassigned one
does not. That is what now separates a real membership from the shape the
contract declares.
The gate pinged its hub, and the NFS node ships no ping. The tunnel there was
healthy the whole time -- handshake 46 seconds old, 7.37 KiB in and 27.10 KiB
out -- and the deploy still failed, so the gate was reporting on its own
tooling rather than on the mesh.
wg reports the handshake directly: it needs nothing the host might not carry,
and it stays true where ICMP is filtered, which a tunnel gate must survive.
A peer that never completed one reports a literal 0.
The MTU probe still needs ping, because only a sized DF packet can show the
VXLAN-inside-WireGuard trap, so it is skipped rather than failed when ping is
absent. The handshake above already proves the tunnel; this proves the frame
size, and losing the second check is not worth failing a working deploy.
The gate had every host ping every peer it was configured for, which a hub can
never satisfy. Only the spoke carries an Endpoint; the hub learns a spoke's
address when that spoke dials in, so a hub probing its spokes is racing them.
The backup host settles it: deployed in a later play, its interface cannot
exist while the cluster play runs, so the manager's data mesh was unprovable by
construction.
Observed in a drill where both meshes were demonstrably healthy -- the workers
had handshakes and traffic on the swarm mesh, the NFS server on the data mesh
-- and the gate still failed on the manager and the NFS server.
A spoke now probes the hub and nothing else. That is the failure worth
catching: a spoke that cannot reach the hub has no mesh, while the hub's own
view converges as spokes arrive. The retry window widens because hosts
configure in parallel and a spoke can probe before the hub's interface
exists.
The guard used ansible.builtin.command with 'command -v wg-quick'. The command
module execs without a shell and 'command' is a shell builtin, so there is no
binary to exec, the probe reported a non-zero return code on every host, and
the install ran everywhere the guard existed to prevent it.
Verifying the probe through swarm-exec hid the bug, because that path does go
through a shell. stat has no such gap: it needs no shell and answers the
question directly.
The swarm node images already ship wireguard-tools, and they also carry a
half-installed linux-image-rt-amd64. apt therefore refuses the install with
'unmet dependencies' even though it reports wireguard-tools is already the
newest version, so the role failed on all three nodes with nothing actually
missing.
Checking for wg-quick first keeps the role from doing work that is already
done, and stops an unrelated broken package state on the image from failing a
deploy the layer would otherwise complete.
The layer is binary: either every swarm deploy proceeds over the mesh or none
does. The gate runs before swarm initialises and names the unreachable peer, so
a partition is reported where it happened instead of surfacing as an unrelated
role failing to reach a service.
Three separate checks, because they fail differently. An interface with no peer
means the inventory is incomplete. An unreachable peer means the mesh is
partitioned. A full-size payload that does not arrive means the MTU is wrong
for VXLAN inside WireGuard, which otherwise drops only large packets and reads
as flaky CI.
wg-quick up on an already-up interface is a no-op, which would leave the old
keys live in the kernel after a rotation rewrote the config. Taking the
interface down first is what makes a rotated deploy actually take effect; the
down half is tolerated because it exits non-zero when nothing was up.
Fails immediately when a swarm host carries no mesh, rather than installing
tooling and producing nothing: an absent mesh means the cross-host writer never
ran, and every later symptom would point somewhere else.
Forwarding is enabled only on the hub, the one member with peers in both
planes, and therefore the only host that has to route between them.
The template call keeps ansible.builtin.template deliberately: this is per-host
system config, and gating it on the stack host would leave every spoke without
an interface.
The consolidated svc-net-wireguard role, its requirement document and the three
references that only existed to support it go away together: the lint ignore for
a load failure inside its tasks, the requirements index entry, and the tunnel
subnet variable nothing else consumed.
Left in place deliberately: the WireGuard mention in the svc-net category
description, which is upstream prose about networking roles in general rather
than anything this role owned.
Tighten verbose comments to one line each (no-long-comments) and remove
leftovers from the old mesh design: unused NODE_OCTET / WG_PORT and the
dead WIREGUARD_IMAGE/VERSION env asserts.
Replace the hardcoded 1400 client MTU with a pkgmgr automtu probe (path
MTU minus WireGuard overhead), falling back to the configured value when
probing is unavailable. Runs before the compose env is rendered so WG_MTU
reflects the derived value (decision 11).
Replaces the linuxserver-image harness with a deploy-driven one: 01_bootstrap (6 empty containers + make install + DinD), 02_registration (per-host dedicated inventory via 'inventory provision' + full-mesh wg0.conf), 03_handshake ('deploy dedicated --id svc-net-wireguard' per host + all-pairs check). Adds the role's inventory-driven config path (mounts a per-host wg0.conf). NAT-flavor coverage follows once the mesh is green.
Replaces mode+nat with a flavor list [server|client|nat]; declares the WireGuard internal subnet in group_vars/all/08_networks.yml; makes the masquerade task idempotent and applied on the deployed host (drops the DOCKER_IN_CONTAINER gate); bumps svc-net-wireguard and test-e2e-cli to lifecycle beta. (Revision 2 of req 026, PR #323 review.)
evangelostsak