Tightening the retry condition was not enough. `failed_when: false` is
evaluated after the retry loop, so a task that runs out of attempts reports
`failed_when_result: false` and ends ok: the role carried on past a spoke
that never handshaked, and past a `wg show` that failed outright, whenever
the MTU probe was skipped.
The explicit failure is back, with the condition the wait now uses, so the
verdict is taken from the registered results rather than from the retry
outcome.
Measured on a three-case play, exit status of the run: a reported handshake
continues (0), '\t0' stops it (2), and a probe exiting non-zero with no
output stops it (2).
`wg show latest-handshakes` ran with `failed_when: false`, so a failed probe
left stdout empty. An empty string carries no '\t0' to match, the `until`
condition was satisfied on the first attempt, and the gate passed on a host
that never reported a handshake at all. The task meant to catch that could
not: Jinja's `default` replaces an undefined value, not an empty one, and
exhausted retries fail the task before a later task runs, so it was
unreachable in every case that mattered.
The wait now requires a zero exit code and a non-empty result as well as a
handshake, and the exhausted retry is the hard failure.
Verified with a local play over three cases: a reported handshake passes,
'\t0' fails, and a probe exiting non-zero with no output fails -- the last
of which passed under the previous condition.
The blanket info-media nocheck claimed WireGuard has no consumer-facing
upstream, which is wrong: wireguard.com is the project's own site and the
README table renders it as the role's link.
wg-quick down on the interface Ansible is connected over kills the session
carrying the play, and the up that would restore it never runs. syncconf
applies new keys and peers to the live interface, so a rotated mesh converges
without dropping the connection that is rotating it.
Latent until now only because the deploy still runs over the underlay.
The install no longer skips when wg-quick happens to exist: the node image
stopped pre-baking it, so the role is what puts it there.
The tunnel and overlay MTUs are now measured with automtu against the hub's
underlay endpoint rather than hardcoded. Fixed values contradict the MTU
automtu already derives for the docker daemon, and the mesh addresses cannot
be probed because the interface those MTUs configure is not up yet.
Enabling forwarding on the hub goes through ansible.posix.sysctl, which
shells out to the sysctl binary. Minimal RedHat and Arch images ship no
procps, so the hub failed with 'Failed to find required executable sysctl'
after every interface had already rendered.
The rotation gate wipes every generated credential so the next deploy has to
carry it. That is right for a per-host secret and wrong for this one: the mesh
keys are correlated across hosts, and the writer that mints them runs once per
round, before the first deploy. Wiped and not rewritten, the update pass would
render an interface with an empty PrivateKey.
rotatable: false says who owns them. The mesh rotates when the writer is asked
to rotate it, as one atomic pass over every member, which is the only way a
mesh can rotate without partitioning.
Declaring meta/meshes.yml made the validator accept the generated block, and
handed every host both meshes whether or not it is a member. The role then
rendered an interface for a mesh with no address and died in the template:
'object of type list has no attribute 1', from splitting an empty subnet.
The failure pattern named the cause exactly -- the NFS server failed on the
swarm mesh, the workers on the data mesh, and the manager, which belongs to
both, passed.
An entry the writer filled in carries an address; a declared-but-unassigned one
does not. That is what now separates a real membership from the shape the
contract declares.
Every key a host_vars file carries under applications.<role> must exist in that
role's meta tree, and the mesh is generated rather than authored, so the
inventory validator rejected it: 'Missing default for svc-net-wireguard:
meshes.swarm.address' and one line per key on every member.
It surfaced only at the rotation gate, which revalidates, so the first pass of
a round looked clean and the update pass aborted before any task ran.
meta/<topic>.yml maps to applications.<app>.<topic>, so declaring the shape in
meta/meshes.yml is enough. Values stay empty because the writer supplies them,
and peers is a list, which the validator does not walk.
The gate pinged its hub, and the NFS node ships no ping. The tunnel there was
healthy the whole time -- handshake 46 seconds old, 7.37 KiB in and 27.10 KiB
out -- and the deploy still failed, so the gate was reporting on its own
tooling rather than on the mesh.
wg reports the handshake directly: it needs nothing the host might not carry,
and it stays true where ICMP is filtered, which a tunnel gate must survive.
A peer that never completed one reports a literal 0.
The MTU probe still needs ping, because only a sized DF packet can show the
VXLAN-inside-WireGuard trap, so it is skipped rather than failed when ping is
absent. The handshake above already proves the tunnel; this proves the frame
size, and losing the second check is not worth failing a working deploy.
The gate had every host ping every peer it was configured for, which a hub can
never satisfy. Only the spoke carries an Endpoint; the hub learns a spoke's
address when that spoke dials in, so a hub probing its spokes is racing them.
The backup host settles it: deployed in a later play, its interface cannot
exist while the cluster play runs, so the manager's data mesh was unprovable by
construction.
Observed in a drill where both meshes were demonstrably healthy -- the workers
had handshakes and traffic on the swarm mesh, the NFS server on the data mesh
-- and the gate still failed on the manager and the NFS server.
A spoke now probes the hub and nothing else. That is the failure worth
catching: a spoke that cannot reach the hub has no mesh, while the hub's own
view converges as spokes arrive. The retry window widens because hosts
configure in parallel and a spoke can probe before the hub's interface
exists.
The guard used ansible.builtin.command with 'command -v wg-quick'. The command
module execs without a shell and 'command' is a shell builtin, so there is no
binary to exec, the probe reported a non-zero return code on every host, and
the install ran everywhere the guard existed to prevent it.
Verifying the probe through swarm-exec hid the bug, because that path does go
through a shell. stat has no such gap: it needs no shell and answers the
question directly.
The swarm node images already ship wireguard-tools, and they also carry a
half-installed linux-image-rt-amd64. apt therefore refuses the install with
'unmet dependencies' even though it reports wireguard-tools is already the
newest version, so the role failed on all three nodes with nothing actually
missing.
Checking for wg-quick first keeps the role from doing work that is already
done, and stops an unrelated broken package state on the image from failing a
deploy the layer would otherwise complete.
States the ordering the role depends on but cannot enforce: the mesh must be
written across every inventory of the deploy, including any sibling file
holding hosts the cluster inventory does not, before the play runs.
Also records the one operational trap: a peer with no recent handshake whose
config looks right means the inventory and the running interface disagree, and
the fix is to re-run the writer rather than edit /etc/wireguard, which the next
deploy overwrites.
The layer is binary: either every swarm deploy proceeds over the mesh or none
does. The gate runs before swarm initialises and names the unreachable peer, so
a partition is reported where it happened instead of surfacing as an unrelated
role failing to reach a service.
Three separate checks, because they fail differently. An interface with no peer
means the inventory is incomplete. An unreachable peer means the mesh is
partitioned. A full-size payload that does not arrive means the MTU is wrong
for VXLAN inside WireGuard, which otherwise drops only large packets and reads
as flaky CI.
wg-quick up on an already-up interface is a no-op, which would leave the old
keys live in the kernel after a rotation rewrote the config. Taking the
interface down first is what makes a rotated deploy actually take effect; the
down half is tolerated because it exits non-zero when nothing was up.
Fails immediately when a swarm host carries no mesh, rather than installing
tooling and producing nothing: an absent mesh means the cross-host writer never
ran, and every later symptom would point somewhere else.
Forwarding is enabled only on the hub, the one member with peers in both
planes, and therefore the only host that has to route between them.
The template call keeps ansible.builtin.template deliberately: this is per-host
system config, and gating it on the stack host would leave every spoke without
an interface.
ListenPort is emitted only on the hub, since spokes dial out and never accept.
A spoke's peer entry for the hub carries an Endpoint plus a keepalive so the
tunnel survives NAT and re-establishes without traffic; the hub's entries for
its spokes carry neither, because it learns their addresses from the handshake.
AllowedIPs asymmetry is what makes the routing work: a spoke routes the whole
mesh subnet at the hub, which is how it reaches the far plane, while the hub
pins each spoke to a single address.
The role plans nothing; it reads what the cross-host writer already agreed on,
so a host can only configure peers that pass saw.
The private keys are looked up one literal path per declared mesh rather than
by building the path from the loop variable. A dynamic config path cannot be
resolved by the lookup validator, so it could not be proven correct before a
deploy -- and an unresolvable credential surfaces as an interface that renders
with an empty PrivateKey and never handshakes.
Declared so the config-lookup validator can resolve the path before a deploy,
and plain so the per-host credential generator leaves the value to the
cross-host mesh pass: the public half has to appear in a different host's peer
list, which no single-host generator can arrange.
On a worker the data-mesh key stays unset, because only the manager, the NFS
server and the backup host are members of that plane.
Host mode, because the interfaces are kernel objects rather than containers,
and enabled only when DEPLOYMENT_MODE is swarm so a compose deploy neither
installs the tooling nor runs the gate.
WireGuard is a kernel protocol rather than a product with a landing page, so
the media requirement is suppressed the same way svc-storage-nfs-server does
for NFS.
The consolidated svc-net-wireguard role, its requirement document and the three
references that only existed to support it go away together: the lint ignore for
a load failure inside its tasks, the requirements index entry, and the tunnel
subnet variable nothing else consumed.
Left in place deliberately: the WireGuard mention in the svc-net category
description, which is upstream prose about networking roles in general rather
than anything this role owned.
Tighten verbose comments to one line each (no-long-comments) and remove
leftovers from the old mesh design: unused NODE_OCTET / WG_PORT and the
dead WIREGUARD_IMAGE/VERSION env asserts.
Reuse the proven client tunnel mechanics but restructure the e2e into 3
server/client pairs: the 3 debian nodes deploy in [server] flavor
(linuxserver generates each peer config), and each workstation joins via
that config in [client] flavor (wsdebian as [client, nat]). Verify each
client<->server tunnel, the NAT masquerade rule on the behind-NAT node,
and a sane auto-derived wg0 MTU.
Replace the hardcoded 1400 client MTU with a pkgmgr automtu probe (path
MTU minus WireGuard overhead), falling back to the configured value when
probing is unavailable. Runs before the compose env is rendered so WG_MTU
reflects the derived value (decision 11).
Endpoints used the outer node name (wg-e2e-<m>), which the wireguard
container cannot resolve from inside the peer node's inner dockerd, so no
handshakes formed (all 30 pairs down). Resolve each node's WGNET IP and
use it as the peer Endpoint, reached via the published relay port. Also
dump `wg show` per node before the connectivity check for debugging.
With the default server flavor, env.j2 sets PEERS/SERVERURL so the
linuxserver image runs in server mode and regenerates wg0.conf on
restart, wiping the rendered full-mesh peer set -> no handshakes (all 30
pairs unreachable). Deploy the mesh with WIREGUARD_FLAVOR=[client] so the
container honors the provided wg0.conf.
The DinD daemon.json hardcoded fuse-overlayfs, which fails copy-up on
images shipping an /etc/mtab symlink (e.g. linuxserver/wireguard) under
deeply-nested CI: "symlink /proc/mounts .../etc/mtab: file exists".
Expose SYS_SVC_CONTAINER_STORAGE_DRIVER (default fuse-overlayfs,
unchanged) and let the wireguard e2e deploy override it to vfs.
make install only sets up the venv + collections; the systemd dependency
comes from the distro package (built at image-build, not run here), so
/sbin/init never existed and the systemd re-create failed. Install
systemd (+ systemd-sysv / systemd-sysvcompat for /sbin/init) as a
bootstrap prerequisite on every node.
The pkgmgr bases have no systemd (no /sbin/init), so booting init failed;
systemd only arrives as an infinito package dependency during make
install. Two-phase: boot the bare base, run the per-node make install
(fresh environment), mask container-hostile first-boot units + seed a
machine-id, then commit and re-create the node with --entrypoint=/sbin/init
so systemd is PID 1 and the deploy can start docker.service like a host.
The pkgmgr base image's ENTRYPOINT runs the pkgmgr CLI, so passing
/sbin/init as a command arg never made systemd PID 1 (systemd never
became ready). Override with --entrypoint=/sbin/init so systemd (already
present in the base) boots as PID 1, and dump container logs on a
readiness timeout.
The role starts docker via systemd (docker.service), which raw debian/
manjaro/centos containers (PID1=sleep) cannot do. Boot the nodes from
Infinito's own fresh pkgmgr base images (systemd + pkgmgr, no infinito
baked) as systemd containers (cgroupns=host, cgroupfs rw, tmpfs /run),
so each node still runs its own make install + inventory + deploy and
the deploy can install/start docker exactly like a real host. Images are
pinned to 1.15.2 (never a moving tag); manjaro workstation -> arch base.
sys-svc-container configures the docker-ce apt repo via apt_repository,
which needs a gpg binary to handle the signing key; the minimal images
lack it (Either apt-key or gpg binary is required). Add gnupg/gnupg2 to
the per-distro prerequisites.
The -e container=docker env did not flip DOCKER_IN_CONTAINER: fact
gathering runs under become/sudo, which env_resets and drops `container`
before the setup module reads it. Force DOCKER_IN_CONTAINER=true as an
ansible extra-var (highest precedence) so the deploy takes the
container/DinD path and skips the host-only timezone task.
evangelostsak