Whether a resolver answers on the docker bridge was inferred from the host's
own resolv.conf: all entries loopback was read as "a listener of ours is there,
and containers must reach it on the bridge". A systemd-resolved stub binds
127.0.0.53, NetworkManager's dnsmasq binds 127.0.0.1, and a node dnsmasq binds
both those and the bridge. All three look identical from resolv.conf and only
the third is reachable from a container.
So ordinary hosts were handed an address with nothing behind it. The
Instructions job died there: daemon.json carried "dns": ["10.208.0.1"] alone,
and the openresty build could not resolve dl-cdn.alpinelinux.org while the
daemon, resolving through the host, pulled the same base image fine. Two netns,
two outcomes.
e308e18923 had already fixed this once, with a reachability gate whose message
names the same URL. b09f2cbd82 dropped that gate and ce43263745 put the
resolv.conf inference in its place, so the guard was gone and only the missing
docker0 fact still hid it. 285ba1e76c removed that accident and the failure
surfaced.
svc-net-tor and NETWORK_CONTAINER_BRIDGE_RESOLVER now say it instead, the
latter false by default and true in the swarm lab, which provisions the dnsmasq
its nodes resolve through. Extra-vars are global, so the lab's NFS and backup
hosts carry it although no listener is installed there; the clearnet resolver
behind the bridge keeps them working.
The docker0 fact is gone from the address too. It is not merely late, it is
wrong while it is late: facts are read before the daemon config is applied, so
docker0 still holds docker's default and writing default-address-pools moves
the bridge out from under it. A swarm node shipped "dns": ["172.17.0.1", ...]
against a live 10.208.0.1, which is why its containers asked a dead address and
fell through to the public resolver, where infinito.example does not exist. The
pool answers before the daemon exists and answers the same afterwards, and it
is read per slice rather than per supernet, which differ once a pool does not
start on a slice boundary.
A list that would carry only the bridge now gets NETWORK_PUBLIC_DNS_RESOLVERS
behind it, so a declaration that turns out to be wrong costs a wasted first
query rather than all name resolution. An empty declaration raises.
Rendered from the repository's own variable files: the Instructions host yields
[] and the template omits the key, which is what de95a95c65 emitted while that
job was green; lab and tor nodes yield the live bridge ahead of the clearnet
resolver even with a stale docker0 fact present; an ordinary host yields only
its declared resolver.
The suite asserted the two behaviours this replaces, including one case added
with 285ba1e76c demanding the stale fact win. They are inverted, and a
cross-product over nameserver shapes, tor membership, clearnet and the docker0
fact now asserts that no shape ends up holding the bridge alone - the old suite
checked each shape by itself, which is how this passed through it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ce43263745 replaced the bridge literal with the clearnet resolver and left
the lookup to supply the bridge from ansible_facts.docker0. That fact
cannot be there on the run that needs it: facts are gathered at play
start, nothing in the role re-gathers them, and on a fresh node docker is
installed afterwards - so the render that writes daemon.json always sees
no docker0 and emits the clearnet resolver alone.
Measured on one node of the same job: docker0 is 10.208.0.1/24 while its
/etc/docker/daemon.json carries "dns": ["1.1.1.1"]. The bridge existed and
was missing from the config that had to name it.
The inner containers then asked a public resolver for a name only the node
dnsmasq serves, and the DinD check reported it accurately seven times:
Server: 1.1.1.1
** server can't find infinito.example: NXDOMAIN
(The '[container] CA trust certificate is not a regular file' line above it
is noise - container.py catches that and falls back to plain docker run.)
default-address-pools is where docker takes docker0 from, so the
declaration answers before the daemon exists and gives the same answer
after: subnet_gateway("10.208.0.0/12") is the 10.208.0.1 measured above. An
observed docker0 still wins, so a host whose bridge sits elsewhere keeps
its real address.
That also settles what the docstring flagged as unavoidable. The
fact-derived value appeared only from the second run on, so daemon.json
differed between passes and restarted the container runtime under a live
stack. Both passes now render the same bytes.
Only the swarm path showed it: 18 jobs died at the daemon start before
reaching this handler, and the six that got here in the next run are the
first to exercise it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
5345f1632 added the render step but not the re-trust, so it never worked.
ansible-core 2.19+ renders only strings carrying TrustedAsTemplate, and
the str() coercion on the line above drops the tag the extra-vars loader
attached. templar.template() then hands the expression back untouched,
the guard sees Jinja and raises:
The lookup plugin 'container_dns' failed: networks.internet.dns is
'{{ NETWORK_PUBLIC_DNS_RESOLVERS | first }}' and did not resolve to an
address.
Which is worse than the ParseAddr the guard was written to prevent, and
on the same 18 jobs: before, one bad daemon.json; after, every run that
reaches the daemon config, compose included, since
inventories/development/default.yml carries a lookup('env') there.
Re-tagging first is what proxy_pass, container_ports, container_volumes
and compose_volumes already do. Through the real lookup loader the field
now renders to ["10.208.0.1", "1.1.1.1"].
The unit stub was the reason none of this showed: it rendered any string,
so twenty tests passed against code that could not work in a play. It now
refuses an untagged string the way the templar does, which makes the four
tests covering this fail without the re-trust. Stub and production key off
the same private ansible module, so a move would degrade both at once -
production to silent identity, the stub to rendering anything. The added
class asserts that shared premise rather than trusting it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ce43263745 replaced the literal 172.17.0.1 under networks.internet.dns
with {{ NETWORK_PUBLIC_DNS_RESOLVERS | first }}, for a good reason: the
fixed address went stale whenever default-address-pools moved the
bridge. Nothing renders that field, though. The file arrives as
'-e @inventories/development/swarm.yml' and this lookup read it straight
out of variables, so the expression reached the daemon as its own source
text:
dockerd: unable to configure the Docker daemon with file
/etc/docker/daemon.json: invalid JSON:
ParseAddr("{{ NETWORK_PUBLIC_DNS_RESOLVERS | first }}"):
unable to parse IP
Then 'Start request repeated too quickly', and the node has no container
engine, which is where eight swarm jobs died - across centos and debian,
so it was never distro-specific. While the field held an address the
missing render step was invisible.
It now renders against the lookup's own templar, and refuses to emit a
value that still carries Jinja. A string that is not an address has no
business in daemon.json, and failing loudly beats leaving a node without
docker.
The swarm inventory carried the manager's docker bridge gateway as a literal under networks.internet.dns, so one field meant two things at once and the value went stale the moment default-address-pools moved the bridge. The daemon kept handing out the new address while the inventory kept naming the old one, and the DinD diagnostic probed that same literal, so it failed in step with the defect instead of catching it.
The inventory now declares only the clearnet resolver. The lookup hands containers the bridge whenever the host resolves solely through a loopback listener, which is how the simulated cluster nodes are wired and which docker strips from a container's resolv.conf. The diagnostic reads the gateway from the running daemon instead of being told it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bridge resolver comes from ansible_facts docker0, a fact that does not exist on a first deploy: the same role installs docker, and the daemon only starts on this very template's notify. Run 1 therefore renders without the bridge and run 2 with it. Where bridge and clearnet are the same address that is one entry against two - different bytes, so the template reports changed, the handler fires, and dockerd restarts underneath a live stack. In CI run 32119014832 that restart killed postgres on the manager at 12:42:26 and the deploy unravelled from there.
Deduplicating makes both renders collide on identical bytes. Order is preserved: the bridge must stay first or dnsmasq never sees the onion names.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Outside compose TOR_CONTAINER_DNS_HOST was empty, so the drop-in rendered no
listen-address and lookup('container_dns') handed containers the clearnet
resolver alone. A swarm stack host ran the egress router without a listener its
containers could reach, and .onion stayed unresolvable for every one of them
while compose resolved it fine. Ownership already governs the listener, the
upstream policy and the resolv.conf pin; the deploy mode only ever stood in for
it. With the mode clause gone both modes render the drop-in compose has been
running in production, and compose keeps its exact behaviour: where matched == 0
the old and the new expression are the same boolean.
In CI run 31927586060 the swarm node dnsmasq logged "ignoring query from
non-local network 172.18.0.16" seconds after a veth appeared and vanished on
docker0, and every container lookup timed out from then on. The health service
retried for 77 minutes against a resolver that no longer answered.
Measured on dnsmasq 2.93 before landing this: a listen-address disables
--local-service even when that flag comes from the init script; without a bind
directive dnsmasq takes the 0.0.0.0 and [::] wildcard pair, which would collide
with a systemd-resolved stub and occupy port 53 on every address; bind-dynamic
and bind-interfaces both hold their per-address listeners across 2000 veth
cycles; and the stock dnsmasq.conf on arch, fedora and centos carries no
bind-interfaces, so the drop-in's bind-dynamic cannot abort startup there.
Two expectations in tests/unit/plugins/lookup/test_container_dns.py still encode
the compose-only asymmetry and fail with this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
TOR_CONTAINER_DNS_HOST is empty in swarm mode by design, so no dnsmasq
listener appears on the docker bridge there. The lookup gated on the tor
group alone and wrote 172.17.0.1 into daemon.json anyway, handing every
nested container a resolver nothing answers on — nested image builds then
died resolving dl-cdn.alpinelinux.org.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drop a return-only temporary and state autoescape explicitly — the test
renders a dnsmasq config, where html escaping would corrupt the output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reaching an onion address required each role to point its own http client
at the tor socks proxy, which then carried clearnet traffic too — fatal
in the test network, which has no exit, and the reason nextcloud needed a
per-host exclusion list.
The egress router already redirects tor's virtual range to the TransPort
and dnsmasq already forwards .onion to tor's DNSPort; the piece missing
was reachability. dnsmasq listened on loopback only, which docker
discards, so containers never saw it. It now also answers on the docker
bridge, the daemon points there with the clearnet resolver kept as
fallback, and the role's own onion resolves to a container-reachable
backend instead of loopback.
Nextcloud drops its proxy block and the exclusion list; every other role
gets the same behaviour without configuring anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5