The ansible flavor's runner starts its own docker daemon, and that daemon
pulled every MDAD image straight from Docker Hub and ghcr.io, past the
registry cache the stack's own daemon uses. MDAD pulls with
`pull: always`, so every bootstrap spent the shared anonymous Docker Hub
quota; locally a single run exhausted it and died with
"toomanyrequests" even for images already on disk.
The role now reads the proxy of the stack's docker.service. When there
is one, it mounts a docker.service drop-in into the runner with the same
HTTP_PROXY, HTTPS_PROXY and NO_PROXY, and pins the proxy host through
container_extra_hosts. When the stack's daemon trusts the registry
cache's MITM CA, the drop-in fetches that CA from the proxy and runs
update-ca-certificates before the inner daemon starts. Without a proxy
on the stack's daemon nothing changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
From triage of run 33923190220. Each fix is independently complete.
penpot: ansible templates delegate_to before the task's merged when, and AnsibleUndefinedVariable escapes get_delegated_vars_and_hostname. PENPOT_BACKEND_HOST_NODE is set only by the resolve include, so the bootstrap task hard-failed on every host that include skipped instead of skipping with it. A block-level when cannot prevent this; move the tasks into a dynamic include.
matrix: the MDAD reconfigure looked for volume matrix_matrix_mdad_state (real name matrix_mdad_state) and container matrix-mdad (real name matrix). Both if-bodies were skipped while the task still reported changed, so a rotated S3 secret never reached synapse and every media write got SignatureDoesNotMatch. Move the shell into its own script and pass the real names through the environment. Also add no_log to the vars.yml render, whose diff printed 16 secrets in cleartext.
n8n: when the settings file and N8N_ENCRYPTION_KEY disagree, oclif's findCommand throws for every command, so 'n8n export:credentials' is unreachable in exactly the state that would need it. The re-key is one-way; pin it rotatable: false.
swarm-drill: distros.sh sees only rc=124 and cannot tell a wedged deploy from one the sweep deadline cut short. Say so where the governor caps it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bootstrap.sh re-cloned all 31 unauthenticated GitHub repos on every bootstrap,
including warm re-runs where roles/galaxy was already fully populated. The
install is now keyed on the sha256 of requirements.yml: the stamp is removed
before the install and written only after it, `--force` stays inside the branch
so a changed requirements.yml still refreshes every role, and set -euo pipefail
still aborts on a galaxy failure. Nothing is masked and no retry is added.
The stamp lives in roles/galaxy/, not in the matrix_mdad_state volume, so it
shares one lifecycle with the roles it describes. The volume-vs-bind-mount split
is the same cross-domain state coupling that 5d6d1c7ca blames for this run's
matrix failure, where the bootstrap gate read its marker from a volume and the
synapse unit from the container rootfs.
This is not a fix for the HTTP 503 that failed web-app-nextcloud 3 in run
30781723102: that job ran a cold container, which always clones. It removes
re-clones on redeploys of a persistent host. The 503's mechanism is still open -
the observed ~31.8s per clone is flat and payload-independent, and neither
runner bandwidth nor host load explains it.
PYTHONUNBUFFERED=1 line-buffers ansible's stdout in the unit. Without it every
progress line carries one flush timestamp at process exit, which is why the 667s
of the failing install cannot be split into per-clone times.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolves the runner Dockerfile to the lint-standard Acquire::Retries=3
and keeps the collabora TODO wording; hardens the merged tree for the
stricter lints (apt retries in the tor Dockerfile, shell-or-true
markers, comment cleanup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A transient Debian/Ubuntu mirror fetch aborts a docker build layer with exit 100 (it killed the web-app-nextcloud swarm openldap build while the same image built fine elsewhere). New lint under tests/lint/docker/dockerfile/apt/ flags any RUN with apt-get update/install that omits Acquire::Retries; retrofit all 15 apt Dockerfiles with -o Acquire::Retries=3. Both apt lints move into the apt/ subpackage. dnf retries by default and apk has no equivalent flag, so both stay out of scope.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Since the inner mdad dockerd actually starts (storage-driver fix), its
default docker0 claims 172.17.0.1 inside the container netns and
shadows exactly the resolver address the swarm inventory hands to every
container via networks.internet.dns, so the bootstrap's ansible-galaxy
died on 'Could not resolve host: github.com' one second after boot.
Declare the nested-docker bridge ranges as a network SPOT
(NETWORK_NESTED_DOCKER_BIP / _ADDRESS_POOLS, deliberately outside
172.16.0.0/12 and away from the 192.168.244.0/24 lab net) and render
the runner's daemon.json from it at staging time instead of shipping a
static file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The mdad runner pinned storage-driver overlay2, but on swarm test nodes
the container's /var/lib/docker sits on overlayfs and the kernel rejects
overlay-on-overlay upperdirs, so the inner dockerd exited, the bootstrap
oneshot failed by dependency and Synapse never came up (proven by the
rescue-diagnostics journal of run 29356373746). Drop the pin so dockerd
auto-detects and ship fuse-overlayfs as the nested fallback; real hosts
keep overlay2.
Replace the 30min blind HTTP poll with a bootstrap terminal-state wait:
fail fast with diagnostics when the inner docker daemon or the bootstrap
playbook fails, probe Synapse only after the oneshot completed, and run
the journal dump on every failure path instead of behind MODE_DEBUG.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Synapse on an onion node could never complete SSO: it has no Tor, so the
http://auth.<onion> issuer was unresolvable server-side (discovery/jwks/
token), and Synapse hard-rejects non-https issuers on top.
- runner/hostpin.{sh,service}: pin the onion auth host to the runner's real
default gateway (node OpenResty) from /proc/net/route, Before=docker.service
so nested containers see it from their first request; host-gateway is
unusable in nested DinD (resolves to the inner docker0)
- Synapse bind-mounts the runner's /etc/hosts; token iss stays the onion
issuer the browser uses
- bootstrap.service: PassEnvironment (systemd units do not inherit container
env; MATRIX_MDAD_PLAYBOOK_TAGS only worked via its fallback default)
- vars.yml.j2: skip_verification on onion (Synapse's escape hatch for
issuers without https; Tor provides transport security per RFC 7686)
- 02_render_inventory: drop the MDAD bootstrap marker + bounce the runner on
inventory changes — mode=update reconfigures the nested stack at all now
- MATRIX_SERVER_NAME follows matrix_domain onto the onion so MXIDs built from
it (DM spec, bridge configs) reference users that exist
- specs: shared Firefox onion fixture (16 files), gotoElementApp reload-retry
against Element's Tor-triggered init race, seaweedfs spec gates on the
filer frontend, env uses canonical_url consumer=
Verified in-container: SSO 302 to onion Keycloak, admin+biber OIDC login,
E2EE DM admin<->biber, CSP — all green over Tor.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DiD runner ran as a separate compose service matrix-mdad while the app entity is matrix, so the swarm overlay upstream needed a service_key override to reach matrix_matrix-mdad. Name the service, container_name and entity all matrix (MATRIX_MDAD_SERVICE, MATRIX_MDAD_CONTAINER, services.matrix.name) so the overlay DNS is matrix and resolve_upstream returns matrix:8008 by default. Drop the webserver_proxy_service_key override; keep proxy_internal_port (synapse 8008, element 8080).
Also drop the now-dead matrix-mdad meta entry (resource_filter resolves to the entity, so its cpus/mem were never read), flavor-gate the volume mounts (mdad to ansible, synapse/element to compose) so the ansible flavor no longer declares compose-only volumes/configs, and have diagnostics.sh read the container name from the MATRIX_CONTAINER env instead of hardcoding it. Renderer + unit validated: swarm matrix:8008/8080, compose 127.0.0.1:8021/8022.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The matrix-mdad runner's Type=oneshot bootstrap could run before the venv python runtime settled in the privileged systemd DiD container, hit a transient 'ModuleNotFoundError: _posixsubprocess', and die permanently with no retry (oneshot units never restart) — leaving Synapse undeployed and the host probe failing indefinitely. Add a readiness guard that retries importing _posixsubprocess/subprocess/ssl/ctypes (30 x 2s) before running ansible-galaxy/ansible-playbook, aborting cleanly only if the runtime never becomes importable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When matrix-mdad is recreated, the systemd unit dir is fresh but the
state volume keeps the marker. ConditionPathExists then skips bootstrap
and Synapse never gets a unit. Drop the condition, gate skip on marker
AND matrix-synapse.service existence, and remove a stale marker on
mismatch so bootstrap re-runs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Local V1 deploy diagnostic showed docker.service inside matrix-mdad
failing repeatedly:
iptables -t nat -N DOCKER: iptables v1.8.11 (legacy):
can't initialize iptables table 'nat': Table does not exist
The runner Dockerfile forced iptables-legacy via update-alternatives;
on a Debian 13 host with nf_tables as the kernel default the legacy
backend can't see the nat table, so the inner dockerd never starts,
matrix-mdad-bootstrap.service stays in 'dependency failed' state, and
the Synapse loopback probe times out for the full retry budget.
Drop the override and let Debian's default (iptables-nft, which talks
to nftables natively) be selected. The host's nf_tables stack is then
addressable from inside the DiD container.
Tests: 186/186 green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Failing CI run 27010233874 (job 79718652049) and the local DiD test
both timed out at `probe Synapse client API on host loopback` with
`Connection reset by peer` for the full 140s retry window. Workflow
rootcause inspection of the matrix-mdad container surface identified
two compounding issues:
1. `matrix_mdad_systemd:/etc/systemd/system` named volume shadowed the
image-baked `multi-user.target.wants/matrix-mdad-bootstrap.service`
enable symlink. First-boot's empty named volume masks the image
content so systemd never sees the unit as enabled — bootstrap never
runs, MDAD never installs, Synapse never starts. Smoking gun.
2. 60×2s = 140s probe window was an order of magnitude too short for a
cold MDAD bootstrap (galaxy install + image pulls + ansible-playbook
setup-all on a fresh container). Even with the unit enabled the
probe would race the bootstrap.
Changes:
- services.yml.j2: drop `matrix_mdad_systemd:/etc/systemd/system`
mount; MDAD-written units live in the container's writable layer
alongside the image's enable symlinks. Add `matrix_mdad_state:
/var/lib/matrix-mdad` so the bootstrap marker survives container
restarts. Normalize federation port publish to `DOCKER_BIND_HOST`.
- compose.yml.j2: rename declared volume `matrix_mdad_systemd` ->
`matrix_mdad_state`. Inside the `matrix_flavor == 'ansible'` branch
only — compose flavor untouched.
- bootstrap.service: drop `network-online.target` (never fires in a
container without systemd-networkd-wait-online). Add
`ExecStartPre=/usr/bin/docker info` for fail-fast if the inner
dockerd never came up.
- bootstrap.sh: log markers + per-step timing so the systemd journal
inside matrix-mdad shows a clear timeline.
- 04_proxy_wiring.yml: 60×2s -> 180×5s = 900s; on probe failure under
MODE_DEBUG, run a new diagnostics.sh script (container logs +
systemctl --failed + journalctl + docker.service status) so the next
iteration has actionable output instead of a silent retry stack.
make test: 186/186 green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- bootstrap.sh / bootstrap.service: systemd doesn't inherit Dockerfile
ENV PATH, so the oneshot couldn't find ansible-galaxy. Export PATH in
the script and set Environment=PATH in the unit explicitly.
- 04_proxy_wiring.yml: rewrite the inner-docker admin-register exec as
shell scalar with `# nocheck: raw-docker` (the inner docker CLI talks
to matrix-mdad's DiD daemon; not host docker).
- vars/main.yml: drop now-unused MATRIX_MDAD_POSTGRES_* + MATRIX_REDIS_*
vars (DiD uses bundled postgres + valkey).
make test: 186/186 green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
matrix-mdad is now a long-running compose service with systemd PID 1
and its own bundled docker daemon. MDAD writes its systemd unit files
and runs matrix-synapse/coturn/element/bridges/jitsi/element-call
entirely INSIDE that container via the inner docker daemon. Nothing
leaks to the host beyond standard /opt/compose/matrix/ role state.
Container layout:
- /sbin/init (systemd) as PID 1
- bundled docker.io daemon (inner DinD)
- bootstrap.service oneshot: runs `ansible-playbook setup.yml` once on
first boot; marker at /var/lib/matrix-mdad/bootstrap.done
- Named volumes: matrix_mdad_docker (inner /var/lib/docker),
matrix_mdad_systemd (/etc/systemd/system), matrix_mdad_matrix (/matrix)
- /sys/fs/cgroup:rw + cgroup: host for systemd-in-container
- Outer compose publishes synapse/element/federation ports to host loopback
mdad-vars.yml.j2:
- postgres_enabled: true (MDAD bundled postgres inside DiD)
- valkey_enabled: true (bundled redis)
- exim_relay_enabled: true when no outbound Mailu
- host_bind_port: 0.0.0.0 inside container so outer compose port-publish
can forward to host loopback
Dropped (no longer needed):
- 02a_database.yml — MDAD provisions postgres inside the DiD container
- 00_cleanup_mdad_residue.yml — no host residue to clean
- ensure rshared mount task — no host bind-propagation needed
- container wait matrix-mdad-runner — service is long-running not one-shot
- matrix_synapse_container_additional_networks_custom — DiD has its own
internal docker network
- matrix_synapse_container_extra_arguments CA mount — bundled CA inside
- coturn_container_additional_volumes — anonymous volumes live in inner
docker daemon, outer hlth-volumes can't see them
04_proxy_wiring.yml:
- admin user creation now uses two-hop exec: outer `container exec
matrix-mdad` -> inner `docker exec matrix-synapse register_new_matrix_user`
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
