fix(matrix): pass the CA bundle into the container synapse binds it from

The mdad stack runs its own Docker daemon inside the matrix container, so a
bind source in `matrix_synapse_container_extra_arguments` is resolved in that
container's mount namespace, not on the host. `vars.yml.j2` mounts
`CA_TRUST.bundle_host` while `services.yml.j2` passed only
`CA_TRUST.cert_host`, and those are two different files: the root CA and the
bundle beside it. The bundle therefore never existed where synapse looked.

`docker create` refuses the unit before Matrix logs anything of its own. From
the rescue artifact of job 102465251843 in run 34348821577, the nested
journal at containers/matrix/nested/.../journal.txt:5018:

    matrix-synapse[5612]: Error response from daemon: invalid mount config for
    type "bind": bind source path does not exist:
    /etc/infinito.nexus/ca/ca-bundle.crt

Five start attempts, the same line each time, and the only error in the 872 KB
journal. matrix-heisenbridge follows it down as a dependency, `systemctl
restart` returns rc=1, the mdad bootstrap fails and the rescue task ends the
play. Not an OOM (exit 1, not 137, `OOMKilled: false`, 9.7 GB free), not disk
pressure, not a failed pull, and 16 % of the job budget spent.

b5fcd535de is where the asymmetry began: it moved this mount from `cert_host`
to `bundle_host` for a good reason -- SSL_CERT_FILE and REQUESTS_CA_BUNDLE
replace the trust store rather than extend it -- and left the passthrough on
`cert_host`. So the bundle stays; what was missing is the line that carries it
in, at the same absolute path the nested daemon asks for.

The test that comes with it reads both templates as text and requires every
CA_TRUST path synapse binds to be passed through at that same path. Rendering
would need a full Ansible context, and the invariant is about what the two
files say to each other rather than about the values. It fails against HEAD,
where `bundle_host` is bound and only `cert_host` is passed. It also fails
loudly if the regex ever stops matching how the extra arguments are built,
rather than quietly guarding nothing.

The cause is proven to the byte; the effect is not observed. `docker create`
never succeeded, so a second synapse defect behind this one would still be
invisible. A local `make compose-deploy apps=web-app-matrix mode=update` with
TLS_MODE=self_signed, grepping the nested journal for "Started
matrix-synapse.service", settles both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kevinveenbirkenbachandClaude Opus 5 committed 2026-09-10 01:56:17 +02:00
1 parent 8434a31611
commit 2acf91e186
2 files changed
+71

No files matched your search

@@ -28,6 +28,7 @@
- {{ lookup('volume', application_id, 'matrix_mdad_matrix').name }}:{{ lookup('volume', 'web-app-matrix', 'matrix_mdad_matrix').target }}
{% if (TLS_MODE | lower) == 'self_signed' %}
- {{ CA_TRUST.cert_host }}:{{ CA_TRUST.cert_host }}:ro
- {{ CA_TRUST.bundle_host }}:{{ CA_TRUST.bundle_host }}:ro
{% endif %}
- /sys/fs/cgroup:/sys/fs/cgroup:rw
tmpfs:
@@ -0,0 +1,70 @@
"""Every CA path synapse binds must reach the matrix container at that path.
The mdad stack runs its own Docker daemon inside the matrix container, so a
bind source in ``matrix_synapse_container_extra_arguments`` is resolved in that
container's mount namespace rather than on the host. A path the compose service
never passes through therefore does not exist where synapse looks for it, and
``docker create`` refuses the unit before anything is logged about Matrix.
Run 34348821577 lost a job to exactly that: ``vars.yml.j2`` mounted
``CA_TRUST.bundle_host`` while ``services.yml.j2`` passed only
``CA_TRUST.cert_host``, and synapse crash-looped on "bind source path does not
exist". The templates are read as text because rendering them needs a full
Ansible context, and the invariant is about what the two files say to each
other, not about the values.
"""
from __future__ import annotations
import re
import unittest
from pathlib import Path
PROJECT_ROOT: Path = Path(__file__).resolve().parents[5]
_MATRIX = PROJECT_ROOT / "roles/web-app-matrix/templates/flavor/ansible"
_BOUND_BY_SYNAPSE = re.compile(r"src='?\s*~?\s*CA_TRUST\.(\w+)")
_PASSED_THROUGH = re.compile(
r"\{\{\s*CA_TRUST\.(\w+)\s*\}\}\s*:\s*\{\{\s*CA_TRUST\.(\w+)\s*\}\}"
)
class TestCaTrustPassthrough(unittest.TestCase):
def test_every_ca_path_synapse_mounts_is_passed_through_at_the_same_path(
self,
) -> None:
wanted = set(_BOUND_BY_SYNAPSE.findall((_MATRIX / "vars.yml.j2").read_text()))
self.assertTrue(
wanted,
"no CA_TRUST bind source found in vars.yml.j2 — the regex no longer "
"matches how synapse's extra arguments are built, so this test has "
"stopped guarding anything",
)
services = (_MATRIX / "services.yml.j2").read_text()
same_path = {
source
for source, target in _PASSED_THROUGH.findall(services)
if source == target
}
missing = sorted(wanted - same_path)
self.assertFalse(
missing,
"synapse binds CA_TRUST."
+ ", CA_TRUST.".join(missing)
+ " inside the nested daemon, but services.yml.j2 does not pass "
"it into the matrix container at the same absolute path. The "
"nested daemon resolves the source in the container's namespace, "
"so docker create fails with 'bind source path does not exist' "
"and matrix-synapse never starts. Add "
+ " and ".join(
f"`- {{{{ CA_TRUST.{name} }}}}:{{{{ CA_TRUST.{name} }}}}:ro`"
for name in missing
)
+ " to the self_signed block.",
)
if __name__ == "__main__": # pragma: no cover
unittest.main()