SSL_CERT_FILE, CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE replace the trust store;
only NODE_EXTRA_CA_CERTS extends it. compose_ca.py pointed all four at the
deployment own root CA, so every OpenSSL and rustls client in every container
was blind to every public certificate. Measured here, one variable, same URL:
unset curl https://registry.npmjs.org/ -> 200
a single unrelated CA -> 000 unable to get local issuer certificate
that CA + the system bundle -> 200
PeerTube was the one process that hit it: its plugin install fetches from
registry.npmjs.org at runtime through a client that honours SSL_CERT_FILE, and
it failed identically in compose and in swarm with
invalid peer certificate: UnknownIssuer
Everything else reaches its artefacts at image build time, or through Node,
whose NODE_EXTRA_CA_CERTS is additive and therefore kept working.
The replacement was never a posture. e7e8cf8c03 calls these variables an
"env-based CA trust fallback", f1ea036bd2 adds them "instead of falling back to
a plain docker run without CA trust", e79183f96a wires the extra CA in "so npm
traffic through the cache validates". Every commit framed them as adding trust.
sys-ca-selfsigned now builds ca-bundle.crt beside the root CA on every node,
from that node own files: the stack host mints the CA and workers fetch it, so
there is no cross-node dependency. The bundle is the root CA concatenated with
the first present CA_BUNDLE_CANDIDATES entry, the SPOT fc0881ff33 introduced.
The cat argument list is assembled in Jinja, so a host with no public bundle
emits the root CA alone, which is exactly the previous behaviour, rather than an
empty argument that would fail. bundle_host joins the existing stray-directory
sweep and the build runs after it.
compose_ca.py uses the bundle only when the file exists and is non-empty, so a
host that has not built one keeps the previous behaviour instead of binding a
missing path that Docker would materialise as a directory. The php ini, msmtprc
and the Matrix flavour may name the container path with no fallback because the
bundle is written at task 99 and the php ini at task 123: no host can carry the
ini without the bundle.
CA_TRUST_CERT and NODE_EXTRA_CA_CERTS stay on the single certificate. keytool,
occ security:certificates:import and update-ca-certificates each take one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
roles/sys-svc-mail-msmtp/tasks/ca_bundle.yml stat'ed candidate bundle paths on
the Ansible target and wrote the winner into msmtprc. Moodle and WordSpress then
copied that file into a container, where the named path is a different file - or
none. Only update-ca-certificates populates a container's system bundle, and
that runs solely from the with-ca-trust.sh entrypoint, which compose_ca.py sets
only when 'docker image inspect' succeeded. On a fresh host the image is built
minutes later, so it never ran: job 94965457602 of run 31864448027 shows
Config.Entrypoint still moodle-entrypoint.sh while the CA sits mounted beside it,
and msmtp reporting 'The certificate issuer is unknown'.
An explicit tls_trust_file also outranks SSL_CERT_FILE, so the env injection the
override always performs could not compensate.
Name CA_TRUST.inject_cert_container when the CA is actually injected, and emit
no line at all otherwise - msmtp's own 'system' default is the right answer for
a letsencrypt relay, and it is what the container's stock bundle verifies. This
also removes a latent defect on letsencrypt: a RedHat host would have written
/etc/pki/... into a Debian container.
The removal of the probing apparatus is taken from 9a72d01eb on
fork/feature/ai-agent-employees, which diagnosed the wrong-machine measurement
correctly. Its own conclusion - drop the line entirely and rely on the trust
store - does not hold where the wrapper never runs, and in swarm it never does:
handlers/swarm.yml passes wrapper=False by design.
The three-term predicate behind all of this was hand-copied into four places.
lookup('ca_injected', application_id) is now its single source; the swarm and
compose gates and sys-svc-compose-ca's own var read it too.
compose_ca.py finally says when it degrades to mount-plus-env, naming the service
and the image it could not inspect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One template serves two consumers with opposite needs. sys-svc-mail-msmtp
renders it to /root/.msmtprc, which the host itself reads, and web-app-wordpress
and web-app-moodle render the same file into their config directory, from where
it is mounted at /etc/msmtprc inside a container. resolve_smtp_host returned the
loopback for both, which is right for the first and meaningless for the second:
inside a container 127.0.0.1 is that container's own loopback and nothing is
listening on it.
WordPress 0 of run 31658270172 died on it. The deploy converged and then
"⏳ Wait for the delivery marker the healthcheck writes" retried sixteen times
before `assertion: wordpress_smoke_mail.rc == 0` evaluated to false. The reason
is in the rescue bundle rather than the log, identically in all three replicas
(containers/wordpress_wordpress.{1,2,3}.*.inspect.json):
msmtp: cannot connect to 127.0.0.1, port 587: Connection refused
msmtp: could not send mail (account default from /etc/msmtprc)
Those containers report Status: healthy with FailingStreak: 0, which is by
design since ebdadd91b decoupled the smoke mail from liveness -- the probe ends
on curl and the delivery assertion carries detection. The assertion was right;
the address it complained about was ours.
smtp_host gains in_container=, and the template passes it from a variable each
of the three call sites sets explicitly. There is deliberately no default: a
fourth consumer must fail loudly rather than silently inherit a loopback that
does not apply to it, which is the very defect being fixed. grep across roles,
utils and plugins finds exactly those three consumers today.
The alias needs pinning to resolve, and container_extra_hosts is the only
emitter of extra_hosts. It gained host_alias=, which appends
host.docker.internal at whatever reaches the host in the effective mode by
reusing _gateway_address -- host-gateway under compose, the stack host's routable
address under swarm. Hardcoding host-gateway at the call site would repeat the
mistake 7b1839681 already paid for: that alias resolves per node and misses the
one that publishes.
Measured, not assumed:
- all four resolver branches: host 127.0.0.1, container host.docker.internal,
bare infra node the manager, outside the rig the configured relay unchanged
- both branches again through a real ansible lookup call, because run() at
first did not forward the kwarg and Python-level testing would have missed
a silently ignored parameter
- the pin emits `host.docker.internal:host-gateway` under compose and
`host.docker.internal:<node address>` under swarm
- the destination exists under swarm: sockets-tcp.txt of all three nodes in
the same rescue bundle shows `LISTEN 0 4096 *:587 users:(("dockerd",…))`, so
the submission port is published on every interface of the node -- which is
why the container's own loopback missed it and the host alias will not
- and under compose by construction: mailu publishes submission on
MAILU_IP4_PUBLIC (templates/compose.yml.j2:12), which is
DOCKER_PUBLIC_BIND_HOST, which is 0.0.0.0 inside the rig
(group_vars/all/00_general.yml:46)
- 11 existing smtp_host unit tests stay green; make test-lint stays at 328
The moodle half fixes nothing that is currently red, and that is the point:
moodle renders the same broken file into two services but its healthcheck only
opens a TCP connection to the FPM port, so Moodle 0 of run 31658241728 passed
with mail that could never leave the container. It breaks when Moodle actually
sends -- a password reset, a forum subscription -- and no job would say so.
Not verified: that the relay accepts the message once reached. The port is
proven reachable, delivery is not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Friendica 0 and WordPress 0 of run 31543714113 never delivered their startup mail. An onion relay has no TLS certificate, so the resolver forces tls=false - and both port and auth hung off that flag: port collapsed to 25 and auth to off. Port 25 is the inbound MX and offers no SMTP AUTH at all, so the message reached Mailu unauthenticated with a local envelope sender and rspamd rejected it 554 5.7.1. Port and auth now hang off `external` instead: 465 with TLS, 587 without, and auth wherever a relay is external.
Moving the port alone would only relocate the failure. msmtp refuses to pick PLAIN or LOGIN itself on a cleartext link and aborts with EX_UNAVAILABLE, so the mechanism has to be named. The new `auth_mechanism` key resolves to off, on, or plain and sits in RESOLUTION_ORDER right after auth, whose result plus tls it reads.
It is one derived value rather than two template expressions because the two failing jobs run different msmtp instances: WordPress renders sys-svc-mail-msmtp/templates/msmtprc.conf.j2 through WORDPRESS_MSMTP_SRC and mounts it at /etc/msmtprc, while Friendica generates its own from SMTP_AUTH via the image script, whose line 32 writes `auth $smtp_auth` verbatim. Both now read the same key.
Verified against the shipped Mailu 2024.06 front rather than assumed. EHLO on 587 under TLS_FLAVOR=notls answers `250-AUTH PLAIN LOGIN`, so PLAIN is offered on the cleartext link; dovecot renders disable_plaintext_auth=no in that flavor. PORT_587 is set because socrate lists 587 outside PORTS_REQUIRING_TLS, unlike 465, which notls skips - so 587 is the only submission port available there. smtp and submission bind identically to MAILU_IP4_PUBLIC, so the path to 587 is the one that already reached 25.
This supersedes the premise of 563da9e5b, which disabled AUTH because the resolver had. That commit quoted a different msmtp error - "the server does not support authentication" - which was port 25 having none, not the relay refusing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The template carried a three-way conditional across three {% set %} lines,
which was already at the edge of readability. Move the decision into
lookup('smtp_host') as plain control flow and pin all six cases in tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nodes outside the mesh (nfs, backup) can neither resolve the onion
mail host (no tor resolver) nor reach a locally published SMTP port;
hlth-msmtp died with status=68/NOHOST. The manager's ingress publishes
the mail ports on every node IP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nodes outside the mesh (nfs, backup) can neither resolve the onion
mail host (no tor resolver) nor reach a locally published SMTP port;
hlth-msmtp died with status=68/NOHOST. The manager's ingress publishes
the mail ports on every node IP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
msmtprc's tls_trust_file is now overridable per consumer: the moodle and
wordpress containers are Debian-based, so the host-distro default CA path
does not exist inside them; both pin /etc/ssl/certs/ca-certificates.crt.
MOODLE_SSLPROXY was hardcoded true, which 500s behind the http-only onion
vhost ("Must use https address in wwwroot when ssl proxy enabled"); derive
it from the central TLS lookup like the other roles.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
moodle mail failed two ways in a swarm cluster. (1) moodle's build_context
pulled the FULL sys-svc-mail-msmtp role early, which rendered /root/.msmtprc on
the manager before Mailu had created the no-reply token and tripped the role's
run-once flag, so the manager kept an empty-password msmtprc and the health
check failed 'authentication method PLAIN needs a password'. Pull only
tasks_from: ca_bundle so the manager re-renders on the normal post-token pass.
(2) msmtprc forced host=127.0.0.1 for any in-container host, but the NFS server
sits outside the swarm mesh and has no local Mailu, so it hit 'connect to
127.0.0.1:465 refused'. Narrow the 127.0.0.1 branch to compose or a real
svc-swarm-node AND to when Mailu is actually deployed (email.external); else use
email.host, and map the NFS node's test DNS to the manager IP. Also import the
shared self-signed Root CA for the relay's TLS trust.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI run 26721196005 failed at:
TASK [sys-service : sys-service | Fail after shared service diagnostics]
msg: Unable to start service hlth-msmtp.9.0.2.infinito.nexus.service.
Two breakages in the start sequence:
1. `sys-ctl-hlth-msmtp/files/script.sh` called `hostname` in `$(...)`.
The CI deploy containers ship without that binary, so `set -eu`
aborted before msmtp ever saw stdin. Fall back to `${HOSTNAME:-...}`
then `uname -n` then a literal "unknown".
2. `sys-svc-mail-msmtp/templates/msmtprc.conf.j2` hard-coded the trust
file path per RedHat family — the file was missing on the deploy
image and msmtp aborted with `Error while reading file. exit=66`.
Probe a candidate list at deploy time (`SYS_SVC_MAIL_MSMTP_CA_BUNDLE_CANDIDATES`),
pick the first existing path, surface it as
`sys_svc_mail_msmtp_ca_bundle`, and install `ca-certificates`
ahead of the probe.
`make test` 5/5 green (186 OK).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two related cleanups extracted from the in-flight req-013 work because
they touch shared config layers (filter plugins, group_vars/all,
unrelated role templates) and deserve their own atomic commit so the
req-013 PR stays focused on the joomla/bluesky integration.
1. `on_off` filter (plugins/filter/on_off.py + unit tests)
Replaces the verbose `{{ x | bool | ternary('on', 'off') }}` idiom
that nginx, msmtp, unbound etc. need for their config syntax.
Coercion mirrors Ansible's `bool` filter; unknown strings raise
so templates can't silently mis-render. Migrated call sites:
- roles/sys-svc-mail-msmtp/templates/msmtprc.conf.j2 (4x)
- roles/sys-svc-proxy/templates/location/html.conf.j2
2. `NETWORK_PUBLIC_DNS_RESOLVERS` SPOT (group_vars/all/08_networks.yml)
New canonical list of public DNS resolvers used wherever a role
needs runtime external-hostname lookups (nginx `resolver`
directives, unbound `forward-addr`, etc.). Default
`[1.1.1.1, 8.8.8.8]`. Migrated call sites:
- roles/web-app-mailu/templates/unbound.conf.j2 (forward-zone loop)
The IPv6 toggle in any nginx `resolver` directive comes from the
existing `NETWORK_IPV6_ENABLED` flag in the same file.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Moves the SYSTEM_EMAIL_* resolution logic out of group_vars/all/03_email.yml
into a dedicated Ansible lookup plugin, so every role template reads outgoing
mail settings via lookup('email') (global) or lookup('email', application_id)
(per-app). Per-app compose.services.email is merged on top of the resolved
defaults; SYSTEM_EMAIL_* variables remain usable as an inventory escape hatch.
Also splits the email docs: administrator-facing configuration stays under
docs/administration/configuration/, plugin contract and extension rules move
to docs/contributing/design/services/email.md. services.md is relocated to
services/base.md so the services/ subtree can host per-service design pages.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Detect container context and automatically use 127.0.0.1 as SMTP host
when running inside Docker, while preserving the external mail host
for non-containerized setups. Add tls_host_override to keep correct
TLS/SNI behavior when connecting locally.
This aligns mail routing with CI/container networking constraints
without
https://chatgpt.com/share/697d20b2-11c0-800f-aacc-57977c798162
- Replace DEFAULT_SYSTEM_EMAIL merge with flat SYSTEM_EMAIL_* variables
- Update templates across apps to new variable names
- Fix msmtp health check to use -t and local username fallback
- Improve postfix localhost config (localdomain, mydestination, timeouts)
- Extend sys-hostname to ensure /etc/hosts mapping once
https://chatgpt.com/share/695e8f39-0694-800f-b91c-a74f4b78d7c8
- introduce configurable mail timeout for msmtp and postfix
- switch health-check mails to local recipients instead of external addresses
- consolidate postfix setup into sys-svc-mail-smtp role
- add proper alias deployment and database generation (lmdb)
- fix relay access denied by enabling local delivery and aliases
- remove deprecated sys-postfix role to avoid configuration conflicts
https://chatgpt.com/share/695e2301-eaf0-800f-8e98-8c602c50d55c
- Migrate Mailu credentials from legacy mailu_token to tokens['web-app-mailu']
- Update all affected application templates to use the new token structure
- Adjust inventory validation to no longer special-case mailu_token
- Minor typing and formatting cleanups in inventory creation and tests
https://chatgpt.com/share/69554915-87c8-800f-afd5-12118dbb1107
- Switch SMTP port dynamically: 465 with TLS, 25 without TLS
- Disable SMTP AUTH automatically when TLS/STARTTLS is not enabled
- Bind Mailu SMTP ports to 0.0.0.0 in Docker-in-Docker environments
- Fix boolean handling in msmtp and Mobilizon SMTP templates
- Ensure plain SMTP (no TLS, no AUTH) works reliably in CI/test setups
https://chatgpt.com/share/69540941-279c-800f-a2b1-79a189c2bdcc
Refactored mail-related roles to support running Infinito.Nexus on nodes without a dedicated mail server:
- Introduced sys-svc-mail as central mail orchestration role.
- Split msmtp handling into sys-svc-mail-msmtp.
- Added sys-svc-mail-smtp to provide a localhost-only Postfix relay when Mailu is not present.
- Updated alert/health roles to use the new mail orchestration.
- Avoid installing postfix inside containers via IS_CONTAINER guard.
- Adjusted WordPress role to use the new msmtp template path.
This allows lightweight deployments (e.g. Raspberry Pi, robots, edge nodes) to send mail via localhost without requiring a full Mailu stack.
ChatGPT discussion: https://chatgpt.com/share/6931edf1-cb98-800f-9e3c-a62d69ccb223
