It holds a mesh address and routes the pool, but it is the machine running
the play. Pointed at itself over ssh, every task a role delegates to it dials
its own tunnel address instead of staying local -- which is where the second
pass died, on the controller-side NFS mount, after running 189 tasks on the
manager and 176 on each worker over the mesh without a failure.
user-administrator creates the account from users.administrator.password,
which the development inventory states outright. The value ansible sends for
become is generated per host and never reaches the account, so every sudo
over the mesh was refused. Nothing noticed before because the container
connection escalates to root without a password.
The extras already carry that inventory's users, so the same value now rides
the deploy-facing twin as ansible_become_password.
The nodes are reached by container name over the docker connection, which
only the lab network resolves. The controller is not on it, so it is handed
the addresses default.env already pins for the five hosts.
The endpoint carries the address now, so nothing has to be taught a name.
Mutating the controller's /etc/hosts to make a config resolve was the lab
working around its own naming rather than the mesh stating what it dials.
--endpoint HOST=ADDRESS, repeatable. A topology whose inventory names do not
resolve on every member states the address once, where the mesh is written,
rather than every member having to be taught the names.
Added on a hypothesis the next CI run disproved: the index was never stale,
the candidate version was identical before and after. It fixed nothing and
buys a refresh every deploy already gets where it needs one.
meta/users.yml states 'password: {{ ansible_become_password }}' to say the
account shares the become secret. Provisioning pinned a fresh random over it,
so the account was created with one secret while become sent another; the
lookup returns the expression as a literal, so skipping the pin would hand
the account the text instead. Resolve the reference against host_vars and pin
that value, vault node included.
Any deploy that reaches a host as the administrator over ssh hits this, not
just the mesh: the container connection escalates to root without a password,
which is why nothing caught it.
A worker carries no web-app group, so it resolves to no host-type stage and
never ran user-administrator; the account existed on the manager alone. Once
the transport moves, Ansible logs into every node as that user, and the
workers refused the key outright.
A declared password is a dict when the role asks for a policy and a string
when it names the value. Only the dict case needs a generated secret, but
both took one, so user-administrator's '{{ ansible_become_password }}' was
overwritten by a random pin: the account was created with one secret while
become sent another.
Nothing caught it because the container connection escalates to root without
a password. The first transport that logs in as the administrator and sudos
for real gets 'Incorrect sudo password' on every host.
Records why the nodes converge before the controller joins, and reopens the
key-rotation criterion: the rotation did sever the run, so the box was ticked
on an assumption rather than on evidence.
The rotation gate mirrors one host's host_vars over every other, so the mesh
write that follows mints a fresh keypair for all but the mirror source. The
nodes kept running the keys of the first pass, so the hub turned the
controller away and its handshake gate failed the deploy.
Re-render every node before the controller joins, so the hub authorises the
set the inventory now holds.
It ran only the wireguard role already, so the same play converges the nodes
and admits the controller; only the inventory differs. The connection comes
from the inventory rather than the play, which is what made it controller-only.
Mirrors the hub's host_vars onto the controller the way the credential reset
does, then asserts the rerun leaves only the mesh the controller belongs to.
Without the fix the file keeps the hub's data entry, is_hub included.
Nodes resolve each other through the extra_hosts compose hands them; the
controller is not a container and got no such map, so wg setconf could not
resolve the hub and the swarm interface failed to come up. Pass the same
name-to-address pairs the compose file builds from the topology.
A peer's Endpoint is its inventory hostname. Where the controller cannot
resolve those names itself, the caller passes a name-to-address map and the
play writes it to /etc/hosts before the role renders the interface.
The prune walks the hosts the inventory groups resolve to, but the controller
joins by name rather than through a group, so it was never visited. Its
mirrored host_vars kept the manager's data-mesh entry -- address 10.100.1.1,
is_hub true -- and the controller brought that interface up and enabled hub
forwarding for a mesh it is not a member of.
The controller runs against a one-host inventory, so DEPLOYMENT_MODE derives to
compose and the role skipped itself. The controller then never got an interface,
while the transport switch still moved the inventory onto the mesh -- leaving
the second pass dialling tunnel addresses from a host that was not on the
tunnel, which is where it timed out.
apt installs recommends by default. wireguard-tools recommends the virtual
wireguard-modules, whose only providers are kernel images, so a plain install
pulls a realtime kernel into a container, fails there, and leaves the metapackage
broken -- after which every later apt call on that host exits 100, including the
retries of the task that caused it.
Every other apt call in the repository already passes --no-install-recommends;
the registry was the one path that did not.
The controller play runs the role on its own, without the constructor, so it
never reached the apt preparation the managed hosts get. It installs from the
same stripped image, so it needs the same index refresh before the role's
first package install.
The deploy images delete /var/lib/apt/lists at build time and the package
registry installs without update_cache, so the first apt task of a deploy
resolved against an empty index and picked whatever version the stale .deb
metadata named. Refresh once, where apt is already made ready, so every role
benefits rather than each one carrying its own refresh.
The role was the first thing to install a package after 'update' dist-upgrades
every node, and apt could no longer satisfy the dependencies -- not for
wireguard-tools, and not for iproute2 or iputils-ping either. Nothing between
the two installs anything, so the role was simply first to hit it.
Ahead of the upgrade is also where the mesh belongs: it is the transport the
rest of the deploy is meant to run over, so it comes up before anything else
touches the host. It stays ahead of the swarm bootstrap for the reason it
always was.
Listing it in the deploy inventory made it a target of playbook.yml, which
runs hosts: all with become: true -- so the controller took the whole host
bootstrap and died on the DNS check, which expects the test domain to resolve
on a managed node.
It is not a managed host. It runs the deploy, and it needs one interface, so
it gets a play that installs exactly that.
The discovery step builds its table with plan.py, not matrix.py, so passing
--vpn aborted the chunk before a runner was spent on it. The plan now takes
the flag and shows the state as a column, the way it already shows the onion.
evangelostsak
kevinveenbirkenbach
dependabot[bot]
Alejandro Roman