Tightening the retry condition was not enough. `failed_when: false` is
evaluated after the retry loop, so a task that runs out of attempts reports
`failed_when_result: false` and ends ok: the role carried on past a spoke
that never handshaked, and past a `wg show` that failed outright, whenever
the MTU probe was skipped.
The explicit failure is back, with the condition the wait now uses, so the
verdict is taken from the registered results rather than from the retry
outcome.
Measured on a three-case play, exit status of the run: a reported handshake
continues (0), '\t0' stops it (2), and a probe exiting non-zero with no
output stops it (2).
`wg show latest-handshakes` ran with `failed_when: false`, so a failed probe
left stdout empty. An empty string carries no '\t0' to match, the `until`
condition was satisfied on the first attempt, and the gate passed on a host
that never reported a handshake at all. The task meant to catch that could
not: Jinja's `default` replaces an undefined value, not an empty one, and
exhausted retries fail the task before a later task runs, so it was
unreachable in every case that mattered.
The wait now requires a zero exit code and a non-empty result as well as a
handshake, and the exhausted retry is the hard failure.
Verified with a local play over three cases: a reported handshake passes,
'\t0' fails, and a probe exiting non-zero with no output fails -- the last
of which passed under the previous condition.
The gate pinged its hub, and the NFS node ships no ping. The tunnel there was
healthy the whole time -- handshake 46 seconds old, 7.37 KiB in and 27.10 KiB
out -- and the deploy still failed, so the gate was reporting on its own
tooling rather than on the mesh.
wg reports the handshake directly: it needs nothing the host might not carry,
and it stays true where ICMP is filtered, which a tunnel gate must survive.
A peer that never completed one reports a literal 0.
The MTU probe still needs ping, because only a sized DF packet can show the
VXLAN-inside-WireGuard trap, so it is skipped rather than failed when ping is
absent. The handshake above already proves the tunnel; this proves the frame
size, and losing the second check is not worth failing a working deploy.
The gate had every host ping every peer it was configured for, which a hub can
never satisfy. Only the spoke carries an Endpoint; the hub learns a spoke's
address when that spoke dials in, so a hub probing its spokes is racing them.
The backup host settles it: deployed in a later play, its interface cannot
exist while the cluster play runs, so the manager's data mesh was unprovable by
construction.
Observed in a drill where both meshes were demonstrably healthy -- the workers
had handshakes and traffic on the swarm mesh, the NFS server on the data mesh
-- and the gate still failed on the manager and the NFS server.
A spoke now probes the hub and nothing else. That is the failure worth
catching: a spoke that cannot reach the hub has no mesh, while the hub's own
view converges as spokes arrive. The retry window widens because hosts
configure in parallel and a spoke can probe before the hub's interface
exists.
The layer is binary: either every swarm deploy proceeds over the mesh or none
does. The gate runs before swarm initialises and names the unreachable peer, so
a partition is reported where it happened instead of surfacing as an unrelated
role failing to reach a service.
Three separate checks, because they fail differently. An interface with no peer
means the inventory is incomplete. An unreachable peer means the mesh is
partitioned. A full-size payload that does not arrive means the MTU is wrong
for VXLAN inside WireGuard, which otherwise drops only large packets and reads
as flaky CI.
evangelostsak