The swarm output is byte-identical: the dev layout and the three-mesh example,
each with and without a controller, produced the same host_vars, wg.conf and
stdout before and after, with deterministic keys.
Under a svc-swarm-manager host the declarations stay the hub's own. Without
one, every host may declare the meshes, and hosts declaring different ones are
refused rather than merged, because a member would otherwise render a mesh
its peers do not know.
With no hub every joined host is a member, addressed from the same offsets a
spoke takes, and each host is primary on the first declared mesh it joined.
A topology may carry no hub. Its meshes then peer every member with every
other one, which is what a Compose deployment of several hosts needs: there
is no manager to route through.
The CLI reads the declarations and memberships from the host_vars it writes
to. --contract-file and --application-id go with them: there is no contract
left to write, and the application block is fixed by the mesh reader.
Each host's state moves from a map keyed by mesh name to a list under
svc-net-wireguard.networks.interfaces. The inventory validator does not walk
lists, so a deployment's own mesh names need no generated contract, and
every reader goes through utils.networks.mesh.
The declarations are the hub's svc-net-wireguard.networks.meshes, and a host
joins every mesh its roles list in their own networks.meshes, so two hosts
of one role may sit in different meshes. The hub is the svc-swarm-manager
host. A deployment narrows the pool in its own group_vars, otherwise the
platform's NETWORK_MESH_POOL applies. A mesh a role names but nobody
declared is refused rather than dropped.
members_of, primary_meshes and plan_mesh take the Topology instead of raw
inventory groups. The spokes come out deduplicated, sorted and without the
hub, as before.
MeshSpec loses hub_group, spoke_groups and spoke_group_prefixes: membership
is no longer part of a declaration but resolved from the inventory into a
Topology of one hub and the spokes of every mesh.
--controller-mesh defaulted to 'swarm', a name only the development
declarations use; any other deployment left the controller off the mesh. It
now defaults to the first declared mesh, an undeclared name fails instead of
being ignored, and the primary of every host is resolved over all declared
meshes before any is written.
Every spoke routed the whole pool through every mesh it belonged to. For a
spoke of two meshes that is two routes for one prefix: wg-quick refuses the
second with 'RTNETLINK answers: File exists' and deletes the interface. The
three-mesh example hits it on the swarm lab, where the NFS server is a spoke
of beta and of gamma.
Only the primary mesh routes the pool now; the others route their own subnet,
which the interface address already covers. Each entry records whether it is
the host's primary.
A host's primary mesh is the first declared mesh it belongs to. It is the one
interface a spoke of several meshes may route the pool through, and the
address the other planes reach the host on.
The guarantees were asserted against the declared swarm and data pair,
so they proved two meshes work rather than that the planner works. A
deployment declares its own, and the review asks for any number of them with
the nodes arranged however the client wants.
The same invariants now run over generated topologies: one to eight meshes,
three arrangements each, hosts dealt out so some belong to several meshes and
some to none. 2247 subtests, four seconds, and the planner needed no change to
pass them.
The arm existed to answer one question: how much of the meshed arm's
cost is the connection plugin and how much is the tunnel. Runs 37592061960
(docker), 37753506661 (SSH on the underlay) and 37621556682 (SSH over
WireGuard) answered it on web-app-gitea#0, debian/ext4:
pass 1 5964s / 8511s / 8832s
pass 2 5319s / 7472s / 7705s
Leaving ansible_connection: docker costs +4700s, the tunnel +554s - 89.5%
against 10.5%. The answer is recorded; the scaffolding does not need to ship.
Reverts d792519725, 1b28f87b37, 89b8a8c8d1, 0470491e1a, 529c5ace43,
dd3d10a9a2, c20dd0e5f5, 458d8956e1, 63ade1e03b, d281555945 and 4adfb6e3d2.
95cfc8b7b7 is already superseded by bbfd82f89, which sets the profiling limit
upstream.
keep_address left ansible_host alone, and the lab never sets it: the
docker connection uses inventory_hostname as the container name. Switching to
SSH without writing an address made ssh resolve 'web-app-gitea-swarm-mgr-01'
through DNS, and run 37617424998 died UNREACHABLE on all three nodes in 13
minutes.
switch_to_mesh now takes the addresses to dial. A member missing from the
mapping raises instead of falling back to its mesh address, because that
fallback would have produced the meshed arm under the other arm's name - a
wrong number rather than a failed run.
keep_address switches the connection, the user, the key and the ssh
args exactly as the mesh switch does, and leaves ansible_host where it was.
Against the other two transports that isolates the cost of leaving the local
connection from the cost of the tunnel.
The prose explaining why offsets are reused and how the hub keeps index
zero is recorded in requirement 037. A reader of plan.py needs the signature,
not the history.
Ansible appends -tt to any task it cannot pipeline when the task escalates,
and ansible.builtin.script is one of those, so the deploy over the mesh ran
those tasks on a pty. A pty ends every line with a carriage return, while
the container connection this transport replaces returned bare newlines. A
role that reads its own stdout therefore started failing for no reason other
than the deploy having moved onto the mesh: the Matomo bootstrap parsed a
perfectly good token as empty, because its regex anchors each line and a
trailing carriage return defeats that anchor.
The host the tool switches now asks for no tty, so the mesh returns the same
bytes the connection it replaced did, and no role has to learn about the
transport underneath it.
Taken from the connection plugin rather than guessed: the tty is added when
the task cannot pipeline and escalates, and the option carrying it is
exposed per host as ansible_ssh_use_tty.
The pass that runs over the mesh reaches every host by SSH where the
pass before it used the container connection, and it pays that difference
once per task. Measured in a container: an exec through a warm multiplexer
costs 7ms, a fresh connection 85ms, and 165ms once sudo asks for the
password every task demands. The pass runs thousands of tasks.
Two settings threw the multiplexer away. A persist window of 20s closes the
master whenever a host waits longer than that for its next task, which in a
linear play is most of them; and the named ControlPath points at a directory
nothing creates, so a master that cannot open its socket reconnects per task
without reporting it.
Neither is retuned for the repository. The tool that moves a host onto the
mesh now writes that host's own ssh_args, derived from the repository's
value so every other option is kept and only the window and the socket
differ. A host the mesh never touches keeps exactly what it had.
Confirmed against the built command: a switched host gets ControlPersist=600s
and a ControlPath under the directory Ansible creates, while a host beside it
keeps the repository's own.
The offset came from the spoke's position in a sorted list, so a host that
sorts earlier renumbered every member behind it. `wg syncconf` carries keys
and peers onto a live interface and never its address, so a renumbered member
keeps answering on the old address while the peers and the inventory have
moved to the new one -- and the address it gave up can be handed to a new
member while the old owner still holds it.
The planner now reuses the addresses already on disk and allocates only an
unseen member, from the lowest free offset. A value outside the subnet, a
duplicate, and the hub's own offset are refused, so a mirrored host_vars
cannot hand a spoke the hub's address. Rotation keeps addresses: it replaces
identities, not addresses.
The routing assertions re-planned the mesh and compared the result with
itself, so a writer that stored the mesh subnet in place of the routed range
would have passed; they now read the entry off disk, which is what the role
reads.
A host in two meshes has one private key, so the second mesh must reuse what
the first minted. The lookup prefers the entry of the mesh being written, which
after a rotation is the stale pre-rotation key, so the hub ended up advertising
one key on the swarm plane and another on the data plane while holding the
private half of only one. Its workers could never handshake again.
The run now carries what it decided: a host keyed while planning one mesh keeps
that key for every mesh after it.
The existing-key lookup walked the inventory hosts, and the controller joins
by name rather than through a group, so it was the one member whose key was
never found and always minted fresh. Every write then changed the hub's peer
list, and the converge re-keyed the swarm tunnel for no reason.
all_hosts already accounts for the controller two lines above; the lookup now
uses it, as the prune already does.
The mirror copies one host's file over every other so a rotation reaches all
of them, which is right for a shared secret and wrong for anything a
generator writes per host: the copy hands every node the source's identity.
--mirror-keep names the subtrees that belong to the destination and are put
back after the copy. Absent, the mirror behaves exactly as before.
--endpoint HOST=ADDRESS, repeatable. A topology whose inventory names do not
resolve on every member states the address once, where the mesh is written,
rather than every member having to be taught the names.
meta/users.yml states 'password: {{ ansible_become_password }}' to say the
account shares the become secret. Provisioning pinned a fresh random over it,
so the account was created with one secret while become sent another; the
lookup returns the expression as a literal, so skipping the pin would hand
the account the text instead. Resolve the reference against host_vars and pin
that value, vault node included.
Any deploy that reaches a host as the administrator over ssh hits this, not
just the mesh: the container connection escalates to root without a password,
which is why nothing caught it.
A declared password is a dict when the role asks for a policy and a string
when it names the value. Only the dict case needs a generated secret, but
both took one, so user-administrator's '{{ ansible_become_password }}' was
overwritten by a random pin: the account was created with one secret while
become sent another.
Nothing caught it because the container connection escalates to root without
a password. The first transport that logs in as the administrator and sudos
for real gets 'Incorrect sudo password' on every host.
The prune walks the hosts the inventory groups resolve to, but the controller
joins by name rather than through a group, so it was never visited. Its
mirrored host_vars kept the manager's data-mesh entry -- address 10.100.1.1,
is_hub true -- and the controller brought that interface up and enabled hub
forwarding for a mesh it is not a member of.
evangelostsak