fix(ca): extend the container trust store instead of replacing it

SSL_CERT_FILE, CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE replace the trust store;
only NODE_EXTRA_CA_CERTS extends it. compose_ca.py pointed all four at the
deployment own root CA, so every OpenSSL and rustls client in every container
was blind to every public certificate. Measured here, one variable, same URL:

    unset                          curl https://registry.npmjs.org/ -> 200
    a single unrelated CA          -> 000 unable to get local issuer certificate
    that CA + the system bundle    -> 200

PeerTube was the one process that hit it: its plugin install fetches from
registry.npmjs.org at runtime through a client that honours SSL_CERT_FILE, and
it failed identically in compose and in swarm with

    invalid peer certificate: UnknownIssuer

Everything else reaches its artefacts at image build time, or through Node,
whose NODE_EXTRA_CA_CERTS is additive and therefore kept working.

The replacement was never a posture. e7e8cf8c03 calls these variables an
"env-based CA trust fallback", f1ea036bd2 adds them "instead of falling back to
a plain docker run without CA trust", e79183f96a wires the extra CA in "so npm
traffic through the cache validates". Every commit framed them as adding trust.

sys-ca-selfsigned now builds ca-bundle.crt beside the root CA on every node,
from that node own files: the stack host mints the CA and workers fetch it, so
there is no cross-node dependency. The bundle is the root CA concatenated with
the first present CA_BUNDLE_CANDIDATES entry, the SPOT fc0881ff33 introduced.
The cat argument list is assembled in Jinja, so a host with no public bundle
emits the root CA alone, which is exactly the previous behaviour, rather than an
empty argument that would fail. bundle_host joins the existing stray-directory
sweep and the build runs after it.

compose_ca.py uses the bundle only when the file exists and is non-empty, so a
host that has not built one keeps the previous behaviour instead of binding a
missing path that Docker would materialise as a directory. The php ini, msmtprc
and the Matrix flavour may name the container path with no fallback because the
bundle is written at task 99 and the php ini at task 123: no host can carry the
ini without the bundle.

CA_TRUST_CERT and NODE_EXTRA_CA_CERTS stay on the single certificate. keytool,
occ security:certificates:import and update-ca-certificates each take one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

group_vars/all/02_tls.yml

@@ -26,6 +26,8 @@ CA_ROOT:
     CN: "{{ (DOMAIN_PRIMARY ~ ' Root CA') if ((DOMAIN_PRIMARY ~ ' Root CA') | length <= 64) else DOMAIN_PRIMARY }}"
 CA_TRUST:
   cert_host: "{{ CA_ROOT.cert_host }}"
+  bundle_host: "{{ [CA_ROOT.base_dir, 'ca-bundle.crt'] | path_join }}"
+  inject_bundle_container: "/tmp/infinito/ca/ca-bundle.crt"
   cert_container: "/etc/ssl/certs/infinito-ca.pem"
   trust_name: "{{ DOMAIN_PRIMARY | lower }}"
   wrapper_host: "{{ BIN_CA_TRUST }}"

roles/sys-ca-selfsigned/files/shell/with-ca-trust.sh

@@ -36,14 +36,20 @@ log "Sanitized trust name: $name"
 
 installed=0
 
-export SSL_CERT_FILE="$CA_TRUST_CERT"
-export REQUESTS_CA_BUNDLE="$CA_TRUST_CERT"
-export CURL_CA_BUNDLE="$CA_TRUST_CERT"
+if [ -n "${CA_TRUST_BUNDLE:-}" ] && [ -r "${CA_TRUST_BUNDLE}" ]; then
+  _ca_trust_verify_file="$CA_TRUST_BUNDLE"
+else
+  _ca_trust_verify_file="$CA_TRUST_CERT"
+fi
+
+export SSL_CERT_FILE="$_ca_trust_verify_file"
+export REQUESTS_CA_BUNDLE="$_ca_trust_verify_file"
+export CURL_CA_BUNDLE="$_ca_trust_verify_file"
 export NODE_EXTRA_CA_CERTS="$CA_TRUST_CERT"
 
 if [ -n "${CA_TRUST_CERT_EXTRA:-}" ] && [ -r "${CA_TRUST_CERT_EXTRA}" ]; then
   if combined="$(mktemp -t ca-trust-combined.XXXXXX 2>/dev/null)" &&
-    cat "$CA_TRUST_CERT" "$CA_TRUST_CERT_EXTRA" > "$combined" 2>/dev/null
+    cat "$_ca_trust_verify_file" "$CA_TRUST_CERT_EXTRA" > "$combined" 2>/dev/null
   then
     export SSL_CERT_FILE="$combined"
     export REQUESTS_CA_BUNDLE="$combined"

roles/sys-ca-selfsigned/tasks/00_core.yml

@@ -66,6 +66,13 @@
       Generated certificate does not appear to be a CA certificate (CA:TRUE missing).
       Please inspect {{ CA_ROOT.cert_host }}.
 
+- name: "🔎 CA | locate the distribution's public CA bundle"
+  become: true
+  ansible.builtin.stat:
+    path: "{{ item }}"
+  loop: "{{ CA_BUNDLE_CANDIDATES }}"
+  register: _ca_public_bundle_candidates
+
 - name: "🔎 CA | stat host CA-trust bind-mount sources"
   become: true
   ansible.builtin.stat:
@@ -73,6 +80,7 @@
   loop:
     - "{{ BIN_CA_TRUST }}"
     - "{{ CA_TRUST.php_ini_host }}"
+    - "{{ CA_TRUST.bundle_host }}"
   register: _ca_trust_bind_sources
 
 - name: "🧹 CA | remove stray directory shadowing a CA-trust bind source"
@@ -87,6 +95,21 @@
     - item.stat.exists
     - item.stat.isdir
 
+- name: "📄 CA | build the trust bundle its readers name with no fallback"
+  become: true
+  ansible.builtin.shell: >-
+    cat {{ ([CA_ROOT.cert_host] + (_ca_public_bundle_candidates.results | selectattr('stat.exists') | map(attribute='item') | list)[:1]) | map('quote') | join(' ') }}
+    > {{ CA_TRUST.bundle_host | quote }}
+  changed_when: true
+
+- name: "🔐 CA | enforce permissions (trust bundle)"
+  become: true
+  ansible.builtin.file:
+    path: "{{ CA_TRUST.bundle_host }}"
+    owner: root
+    group: root
+    mode: "0644"
+
 - name: "📦 CA | install trust helper script (SPOT) on host"
   become: true
   ansible.builtin.copy:

roles/sys-ca-selfsigned/templates/ca-trust-php.ini.j2

@@ -1,2 +1,2 @@
-openssl.cafile={{ CA_TRUST.inject_cert_container }}
-curl.cainfo={{ CA_TRUST.inject_cert_container }}
+openssl.cafile={{ CA_TRUST.inject_bundle_container }}
+curl.cainfo={{ CA_TRUST.inject_bundle_container }}

roles/sys-svc-compose-ca/files/python/compose_ca.py

@@ -516,6 +516,21 @@ def _cache_frontend_ca_host() -> str:
     return ""
 
 
+def _trust_bundle_host(ca_host: str) -> str:
+    """Host path of the CA bundle sys-ca-selfsigned builds beside the root CA, or ''.
+
+    SSL_CERT_FILE, CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE replace the trust
+    store rather than extend it, so a container pointed at the root CA alone
+    cannot verify a public certificate. Returning '' when the bundle is absent
+    keeps a host that has not built one yet on the previous behaviour instead
+    of binding a missing path, which Docker would materialise as a directory.
+    """
+    bundle = Path(ca_host).parent / "ca-bundle.crt"
+    if bundle.is_file() and bundle.stat().st_size > 0:
+        return str(bundle)
+    return ""
+
+
 def render_override(
     services: dict[str, Any],
     service_to_compose_cmd: dict[str, list[str]],
@@ -552,6 +567,7 @@ def render_override(
     """
     out_services: dict[str, Any] = {}
     extra_ca_host = _cache_frontend_ca_host()
+    bundle_host = _trust_bundle_host(ca_host)
 
     image_meta = gather_image_meta(
         sorted(
@@ -586,6 +602,12 @@ def render_override(
             "CURL_CA_BUNDLE": ca_container,
             "NODE_EXTRA_CA_CERTS": ca_container,
         }
+        if bundle_host:
+            bundle_container = str(Path(ca_container).parent / Path(bundle_host).name)
+            volumes.append(f"{bundle_host}:{bundle_container}:ro")
+            environment["SSL_CERT_FILE"] = bundle_container
+            environment["REQUESTS_CA_BUNDLE"] = bundle_container
+            environment["CURL_CA_BUNDLE"] = bundle_container
         if extra_ca_host:
             extra_ca_container = str(Path(ca_container).parent / "ca-trust-extra.crt")
             volumes.append(f"{extra_ca_host}:{extra_ca_container}:ro")

roles/sys-svc-mail-msmtp/templates/msmtprc.conf.j2

@@ -8,7 +8,7 @@ auth                {{ email.auth_mechanism }}
 tls_starttls        {{ email.start_tls | on_off }}
 tls                 {{ email.tls | on_off }}
 {% if ca_injected %}
-tls_trust_file      {{ CA_TRUST.inject_cert_container }}
+tls_trust_file      {{ CA_TRUST.inject_bundle_container }}
 {% endif %}
 tls_host_override   {{ email.host }}
 account             system_email_no_reply

roles/web-app-matrix/templates/flavor/ansible/vars.yml.j2

@@ -62,7 +62,7 @@ matrix_livekit_jwt_service_container_port: 8030
 {% set _synapse_extra_args = [] %}
 {% if (TLS_MODE | lower) == 'self_signed' %}
 {%   set _ = _synapse_extra_args.extend([
-        '--mount type=bind,src=' ~ CA_TRUST.cert_host ~ ',dst=' ~ MATRIX_CA_MOUNT_PATH ~ ',ro',
+        '--mount type=bind,src=' ~ CA_TRUST.bundle_host ~ ',dst=' ~ MATRIX_CA_MOUNT_PATH ~ ',ro',
         '-e SSL_CERT_FILE=' ~ MATRIX_CA_MOUNT_PATH,
         '-e REQUESTS_CA_BUNDLE=' ~ MATRIX_CA_MOUNT_PATH,
      ]) %}

tests/unit/python/roles/sys-svc-compose-ca/files/test_compose_ca.py

@@ -1,6 +1,7 @@
 # tests/unit/python/roles/sys-svc-compose-ca/files/test_compose_ca_inject.py
 import importlib.util
 import json
+import tempfile
 import unittest
 from pathlib import Path
 from unittest.mock import patch
@@ -875,6 +876,53 @@ class TestComposeCaInject(unittest.TestCase):
             ["sh", "-lc", 'exec "$$CHESS_ENTRYPOINT_INT"'],
         )
 
+    def test_render_override_points_the_replacing_vars_at_the_bundle(self):
+        """A host that has a CA bundle gets the verifying vars pointed at it.
+
+        SSL_CERT_FILE, CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE replace the trust
+        store, so leaving them on the single root CA is what stops a container
+        from reaching any public HTTPS endpoint. NODE_EXTRA_CA_CERTS is
+        additive and stays on the root CA, and CA_TRUST_CERT stays single
+        because the anchor installers take one certificate.
+        """
+        services = {"svc": {"image": "img:1"}}
+        service_to_cmd = {"svc": ["docker", "compose", "-p", "p", "-f", "compose.yml"]}
+
+        with tempfile.TemporaryDirectory() as tmp:
+            ca_host = Path(tmp) / "root-ca.crt"
+            ca_host.write_text("root\n")
+            (Path(tmp) / "ca-bundle.crt").write_text("root\npublic\n")
+
+            with (
+                patch.object(self.m, "ensure_image_available"),
+                patch.object(self.m, "docker_image_inspect"),
+                patch.object(self.m, "docker_image_has_bin_sh"),
+            ):
+                doc = self.m.render_override(
+                    services,
+                    service_to_cmd,
+                    cwd=Path("/tmp"),
+                    env={},
+                    ca_host=str(ca_host),
+                    wrapper_host="/host/with-ca-trust.sh",
+                    ca_container=CA_CERT_CONTAINER,
+                    wrapper_container=CA_WRAPPER_CONTAINER,
+                    trust_name="infinito.local",
+                    wrap=False,
+                )
+
+            out = doc["services"]["svc"]
+            bundle_container = str(Path(CA_CERT_CONTAINER).parent / "ca-bundle.crt")
+            self.assertIn(
+                f"{Path(tmp) / 'ca-bundle.crt'}:{bundle_container}:ro", out["volumes"]
+            )
+            for key in ("SSL_CERT_FILE", "CURL_CA_BUNDLE", "REQUESTS_CA_BUNDLE"):
+                self.assertEqual(out["environment"].get(key), bundle_container)
+            self.assertEqual(
+                out["environment"].get("NODE_EXTRA_CA_CERTS"), CA_CERT_CONTAINER
+            )
+            self.assertEqual(out["environment"].get("CA_TRUST_CERT"), CA_CERT_CONTAINER)
+
     def test_render_override_no_wrapper_emits_env_and_mounts_only(self):
         """
         render_override(wrap=False): swarm path must emit only volumes + environment,

tests/unit/python/roles/sys-svc-mail-msmtp/test_msmtprc_ca_trust.py

@@ -2,9 +2,11 @@
 
 The rendered file is read inside a container while the template is rendered on
 the Ansible target, so naming a host-resolved bundle path measured the wrong
-machine. Only two answers are correct: the CA file the compose override mounts
-into every service of a self-signed app, or nothing at all - msmtp then falls
-back to its own ``system`` default, which ``with-ca-trust.sh`` curates.
+machine. Only two answers are correct: the trust bundle the compose override
+mounts into every service of a self-signed app, or nothing at all - msmtp then
+falls back to its own ``system`` default, which ``with-ca-trust.sh`` curates.
+The bundle rather than the bare root CA, because ``tls_trust_file`` replaces
+the trust store and a relay outside the deployment presents a public chain.
 
 The conditions are Jinja and compile with a plain environment plus a ``bool``
 filter shim, since ``bool`` is an Ansible filter.
@@ -24,6 +26,7 @@ TEMPLATE = (
     PROJECT_ROOT / "roles" / "sys-svc-mail-msmtp" / "templates" / "msmtprc.conf.j2"
 )
 CA_CONTAINER = "/tmp/infinito/ca/root-ca.crt"  # nocheck: S108 - CA_TRUST.inject_cert_container, a container path
+CA_BUNDLE_CONTAINER = "/tmp/infinito/ca/ca-bundle.crt"  # nocheck: S108 - CA_TRUST.inject_bundle_container, a container path
 EMAIL = {
     "timeout": 30,
     "auth_mechanism": "PLAIN",
@@ -63,7 +66,10 @@ def _render(*, in_container: bool, ca_injected: bool, **extra) -> str:
         lookup=lookup,
         sys_svc_mail_msmtp_in_container=in_container,
         MODE_DEBUG=False,
-        CA_TRUST={"inject_cert_container": CA_CONTAINER},
+        CA_TRUST={
+            "inject_cert_container": CA_CONTAINER,
+            "inject_bundle_container": CA_BUNDLE_CONTAINER,
+        },
         **extra,
     )
 
@@ -80,7 +86,7 @@ class TestMsmtprcCaTrust(unittest.TestCase):
         rendered = _render(
             in_container=True, ca_injected=True, application_id="web-app-moodle"
         )
-        self.assertEqual(_trust_line(rendered), CA_CONTAINER)
+        self.assertEqual(_trust_line(rendered), CA_BUNDLE_CONTAINER)
 
     def test_a_container_without_the_injected_ca_names_nothing(self):
         rendered = _render(
@@ -97,7 +103,9 @@ class TestMsmtprcCaTrust(unittest.TestCase):
             extra = {"application_id": "web-app-moodle"} if in_container else {}
             rendered = _render(in_container=in_container, ca_injected=injected, **extra)
             trust = _trust_line(rendered)
-            self.assertIn(trust, (None, CA_CONTAINER), f"host path leaked: {trust}")
+            self.assertIn(
+                trust, (None, CA_BUNDLE_CONTAINER), f"host path leaked: {trust}"
+            )
 
 
 if __name__ == "__main__":