Three defects in the CLI replay of run 39186571513, independent of each other. docs#0 (117856696140) failed at tasks/main.yml with tar: cli/administration/deploy: file changed as we read it rc=1 The task archives the live checkout while the controller keeps running Python out of it. A __pycache__ appearing under a package directory moves THAT DIRECTORY's mtime, and a recursing tar re-stats directories after scanning them, so --exclude keeps the entry out of the archive but cannot stop the parent from changing underneath. Measured on a fixture: plain tar exits 1, --warning=no-file-changed silences the message and still exits 1, and the list-first form exits 0 with the three patterns absent and parents still ahead of their children under LC_ALL=C. tor#1 (118545536787) failed one layer deeper, at svc-net-tor/tasks/router.yml:67, 'Unable to start service dnsmasq'. The replay's CoreDNS shares the target's network namespace (network_mode: service:machine) and both server blocks declared :53 without bind, so it held the wildcard including 127.0.0.1:53 and dnsmasq, which binds listen-address=127.0.0.1 with bind-interfaces, could never start. Inference, not observation: machine.sh destroys the replay target on exit, so no journal for that dnsmasq exists in any artifact. The same config starts fine on the CI VM in the same job. That missing journal is the third defect. report_failed_units asked systemctl --failed, which does not list a unit parked in activating/auto-restart by a Restart= drop-in -- exactly where router.yml:59 had just put dnsmasq. The only unit it printed was an unrelated systemd-modules-load.service. Naming the states restores the diagnostic; the function still runs only on the failure path and returns 0 regardless, so nothing it reports can fail a test. Expect one more tor#1 failure after this: once dnsmasq starts, svc-net-tor rewrites the target's resolv.conf to 127.0.0.1 and drops the harness wildcard. Only firewall assertions follow and they need no DNS, so it is not pre-fixed here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Instruction replay
The CLI test of web-app-docs reads the Production block out of the running documentation site and executes it, so the published instructions are proven rather than reviewed.
Cast
| What it is | What runs there | |
|---|---|---|
| 🖥️ Stack host | the worker the outer play already runs on | test.sh, machine.sh, and a block that starts a container |
| 🎯 Target | a throwaway container, project docs-guide-<role> |
systemd and an ssh server, nothing else until the deploy lands; a host install also runs there |
| 🛰️ Resolver | a CoreDNS sidecar in the target's network namespace | answers every name under DOCS_TEST_DOMAIN with the target's own bridge address and forwards the rest |
| 🐳 Recipe container | started by the block's own docker run, on the stack daemon |
the deploy that the instructions tell a reader to run |
The target is what the instructions call "a managed server". It is deliberately bare: everything else it ends up with arrives through the deploy under test.
Its resolver is the one exception. DOCS_TEST_DOMAIN is the outer stack's own domain, and the stack daemon hands every bridge container the outer stack's CoreDNS, so without the sidecar the target resolves the domain it is deploying to the outer application container. The sidecar answers the whole domain with the target's bridge address, and with that address rather than 127.0.0.1 so that containers the deploy starts inside the target reach the same vhosts.
A role that ships no container stack publishes a host install instead of a recipe: a block that opens with git clone and configures the machine it runs on. The target is that machine, so the block is replayed inside it over the checkout mounted at DOCS_TEST_SRC_DIR, and the two clone lines are dropped so the replay reads the tree under test. A block that neither starts a container nor clones the repository fails the test.
Schema
flowchart TD
subgraph HOST["🖥️ stack host"]
direction TB
TEST["1️⃣ 🧪 test.sh<br>returns early unless DOCS_TEST_MODE=compose"]
EXTRACT["2️⃣ 📖 extract.py<br>reads the block off the live docs site"]
MIRRORED["🪞 mirror.sh<br>--mirror when CI configured a GHCR mirror"]
FILL["3️⃣ 🔑 ed25519 pair + placeholder fill<br>everything except HOST"]
MACHINE["4️⃣ 🚜 machine.sh"]
ADDR["6️⃣ 📍 the target's bridge address"]
DNS["7️⃣ 🛰️ Corefile.tmpl rendered, resolver up"]
SHAPE{"🔀 docker run or git clone"}
GUARD["9️⃣ 🚦 HOST fill + placeholder guard"]
BLOCK["🔟 ▶️ bash -e block.sh"]
end
subgraph TARGET["🎯 target container"]
direction TB
INIT["5️⃣ ⚙️ systemd, no engine<br>Dockerfile + init.sh"]
RESOLV["🛰️ /etc/resolv.conf names the sidecar<br>*.DOCS_TEST_DOMAIN answers with the bridge address"]
SSHD["8️⃣ 🔐 sshd.sh<br>ssh server + run key for root"]
HOSTRUN["▶️ bash -e block.sh in the mounted checkout<br>clone lines dropped"]
DEPLOYED["📦 docker-ce and the role"]
FAILED["⏹️ systemctl --failed + journalctl -xeu"]
end
RECIPE["🐳 docker run <deploy image><br>on the stack daemon"]
TEST --> EXTRACT --> MIRRORED --> FILL --> MACHINE
MACHINE -->|"🌉 compose up machine"| INIT
INIT -->|"⏳ await systemd"| ADDR
ADDR --> DNS
DNS -->|"🌉 compose up resolver"| RESOLV
RESOLV --> SHAPE
SHAPE -->|"🐳 recipe"| SSHD
SHAPE -->|"🏠 host install"| HOSTRUN
SSHD --> GUARD --> BLOCK
BLOCK --> RECIPE
RECIPE -->|"🔗 ssh as root"| DEPLOYED
HOSTRUN --> DEPLOYED
DEPLOYED -.->|"❌ non-zero replay"| FAILED
MACHINE -.->|"🧹 trap EXIT"| TARGET
classDef host fill:#e8f0fe,stroke:#1a73e8,color:#102a43;
classDef target fill:#e6f4ea,stroke:#137333,color:#0b2e13;
classDef pivot fill:#fef7e0,stroke:#b06000,color:#3d2600;
classDef recipe fill:#f3e8fd,stroke:#8430ce,color:#2a0b45;
class TEST,EXTRACT,MIRRORED,FILL,MACHINE,BLOCK host;
class INIT,RESOLV,SSHD,HOSTRUN,DEPLOYED,FAILED target;
class SHAPE,ADDR,DNS,GUARD pivot;
class RECIPE recipe;
🔵 stack host · 🟢 target · 🟣 the recipe's own container · 🟠 the steps that bridge them
Flow
| # | Step | Detail |
|---|---|---|
| 1️⃣ | 🧪 test.sh starts on the stack host |
returns early unless the deployment mode is compose |
| 2️⃣ | 📖 extract.py writes block.sh |
fetches the role's page over DOCS_TEST_BASE_URL; an empty block fails the test, and bash -n parses it before anything runs it |
| 🪞 | the GHCR mirror is attached | mirror.sh rewrites the block's provision call to write mirrors.yml and pass --mirror, so the replay pulls every application image from the mirror instead of the upstream registry. Outside CI the two GitHub variables are empty and the block is left exactly as published |
| 3️⃣ | 🔑 key pair and placeholders | an ed25519 pair per run under DOCS_TEST_WORK; every placeholder except HOST is substituted |
| 4️⃣ | 🚜 machine.sh brings the target up |
compose up machine, on the default bridge |
| 5️⃣ | ⚙️ the target boots | image built from Dockerfile + init.sh; MACHINE_BASE_IMAGE comes from INFINITO_PARENT_IMAGE in the checkout's .env |
| 6️⃣ | 📍 the bridge address is read | container inspect --type container; the bridge has no DNS, so it is an address and not a name |
| 7️⃣ | 🛰️ the resolver takes over the domain | Corefile.tmpl is rendered into DOCS_TEST_WORK with that address and the nameservers the target already had, compose up resolver starts CoreDNS in the target's network namespace, the target's /etc/resolv.conf is rewritten to name it, and a probe under DOCS_TEST_DOMAIN has to come back as the bridge address before the replay starts |
| 🔀 | the shape decides where the block runs | docker run on the stack host, git clone inside the target; steps 8️⃣ and 9️⃣ are the first of the two |
| 8️⃣ | 🔐 sshd.sh is piped in |
installs an ssh server, authorizes the run's public key for root, writes a PermitRootLogin prohibit-password drop-in |
| 9️⃣ | 🚦 HOST is filled and the guard runs |
the guard fails on any remaining <your- |
| 🔟 | ▶️ the block executes | the recipe shape runs it on the stack host, where its docker run starts the published deploy image on the stack daemon and reaches the target over ssh as root; the host install runs it inside the target over the mounted checkout. Either way the target receives docker-ce and the role, and its /var/lib/docker is a named volume, so the installed daemon writes to a real filesystem |
| ⏹️ | the evidence of a failed replay | a non-zero block prints systemctl --failed and a journalctl -xeu per failed unit from the target, because unit output never reaches the playbook stdout that is the only thing leaving it |
| 🧹 | teardown | compose down -v --rmi local from an EXIT trap |
Environment
| Variable | Source | Use |
|---|---|---|
DOCS_TEST_BASE_URL |
templates/test.env.j2 |
documentation site the block is read from |
DOCS_TEST_ROLE |
templates/test.env.j2 |
role whose page carries the instructions; names the compose project |
DOCS_TEST_MODE |
templates/test.env.j2 |
deployment mode; the test runs only for compose |
DOCS_TEST_DOMAIN |
templates/test.env.j2 |
fills <your-domain>; the zone the resolver answers with the target's bridge address |
DOCS_TEST_DNS |
templates/test.env.j2 |
fills <your-clearnet-resolver> |
DOCS_TEST_SRC_DIR |
templates/test.env.j2 |
checkout holding the generated .env; mounted into the target at the same path, where a host install replays from it |
DOCS_TEST_TIMEOUT |
templates/test.env.j2 |
wait for the on-demand documentation build |
DOCS_TEST_WORK |
test.sh |
holds the replay key pair and the rendered Corefile |
DOCS_TEST_REPLAY_PUBKEY |
test.sh |
public half, consumed by sshd.sh |
GITHUB_REPOSITORY |
templates/test.env.j2 |
names the GHCR repository the mirror lives under; empty outside CI, which leaves the block unmirrored |
GITHUB_REPOSITORY_OWNER |
templates/test.env.j2 |
names the GHCR namespace; empty outside CI |
INFINITO_GHCR_MIRROR_PREFIX |
default.env in DOCS_TEST_SRC_DIR |
path segment separating mirrored images from the repository's own |
Files
| File | Role |
|---|---|
test.sh |
entry point; extraction, key pair, placeholder fill |
extract.py |
pulls the Production block off the live documentation site |
mirror.sh |
points the extracted block's provision call at the GHCR image mirror when CI configured one |
machine.sh |
target lifecycle, wildcard resolution, HOST fill, guard, replay, failed-unit report, teardown |
compose.yml |
the target service on the default bridge with a named volume at /var/lib/docker, and the resolver service in the target's network namespace |
Corefile.tmpl |
the resolver's configuration; ${DOCS_TEST_DOMAIN}, ${DOCS_TEST_TARGET_IP} and ${DOCS_TEST_UPSTREAM} are substituted into DOCS_TEST_WORK/Corefile |
Dockerfile |
builds the target from the distribution base image |
init.sh |
installs systemd into that image, and deliberately no container engine |
sshd.sh |
ssh server and the run's authorized key, piped into the running target |