Under SSO the role tried to let applications relay without SMTP AUTH by
widening the relay rule to the private networks. The rule compared the
peer address to the text of a CIDR, which never matches, so an
application could reach local mailboxes but got "550 5.1.2 Relay not
allowed" for every external recipient. Repairing the comparison would
have opened a relay, because published ports, Tor and the swarm mesh all
hand outside clients an address inside those networks.
Stalwart checks app passwords before it asks the external directory, so
a bot can authenticate while the directory is OIDC. With SSO enabled the
role now creates one app password per bot, acting for that account as
the recovery admin, and stores it in the token store the SMTP password
is already read from. A bot gets a new one when its entry is deleted in
Stalwart or its token is removed.
The relay rule is back to authenticated sessions only. The lookup loses
its SSO relay branch and the role its submission_via_relay flag, so an
SSO deployment submits on 465 with AUTH like every other one.
The role's CLI test now fails when an unauthenticated session is given
an external recipient, or when the bot credential is refused one.
Closes#793
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Friendica 0 and WordPress 0 of run 31543714113 never delivered their startup mail. An onion relay has no TLS certificate, so the resolver forces tls=false - and both port and auth hung off that flag: port collapsed to 25 and auth to off. Port 25 is the inbound MX and offers no SMTP AUTH at all, so the message reached Mailu unauthenticated with a local envelope sender and rspamd rejected it 554 5.7.1. Port and auth now hang off `external` instead: 465 with TLS, 587 without, and auth wherever a relay is external.
Moving the port alone would only relocate the failure. msmtp refuses to pick PLAIN or LOGIN itself on a cleartext link and aborts with EX_UNAVAILABLE, so the mechanism has to be named. The new `auth_mechanism` key resolves to off, on, or plain and sits in RESOLUTION_ORDER right after auth, whose result plus tls it reads.
It is one derived value rather than two template expressions because the two failing jobs run different msmtp instances: WordPress renders sys-svc-mail-msmtp/templates/msmtprc.conf.j2 through WORDPRESS_MSMTP_SRC and mounts it at /etc/msmtprc, while Friendica generates its own from SMTP_AUTH via the image script, whose line 32 writes `auth $smtp_auth` verbatim. Both now read the same key.
Verified against the shipped Mailu 2024.06 front rather than assumed. EHLO on 587 under TLS_FLAVOR=notls answers `250-AUTH PLAIN LOGIN`, so PLAIN is offered on the cleartext link; dovecot renders disable_plaintext_auth=no in that flavor. PORT_587 is set because socrate lists 587 outside PORTS_REQUIRING_TLS, unlike 465, which notls skips - so 587 is the only submission port available there. smtp and submission bind identically to MAILU_IP4_PUBLIC, so the path to 587 is the one that already reached 25.
This supersedes the premise of 563da9e5b, which disabled AUTH because the resolver had. That commit quoted a different msmtp error - "the server does not support authentication" - which was port 25 having none, not the relay refusing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Remove the inline/trailing comments added across the Tor egress-router, onion-domain, mailu-notls and vault round-trip changes, per the repo no-comments convention. Comment-only: no code behaviour changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The email lookups tls default keyed on DOMAIN_PRIMARY, which can stay clearnet while Mailus canonical domain is turned into a .onion by the Tor onion injection. TLS therefore stayed on, but Mailu has no public certificate for the .onion (LE cannot sign it) and disabled TLS, so the SMTP handshake to the local front failed and the msmtp health check errored. Key the .onion detection on the resolved mail host instead (host is now resolved before tls in RESOLUTION_ORDER); an onion mail host relays in plaintext, which Tor already authenticates and encrypts end-to-end.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- svc-net-tor: expose SSH (22) + Mailu SMTP/submission/IMAP (25/465/587/143/993)
as HiddenServicePorts on the node onion (TOR_ONION_SSH_ENABLED, 19_tor.yml).
- email lookup: relay plaintext (tls off, port 25) for .onion mail domains —
no public TLS cert for a .onion; onion transport is already authenticated.
- CA root CN: drop the " Root CA" suffix when a 62-char v3 onion + suffix would
overflow the X.509 commonName 64-char cap (02_tls.yml).
- Playwright onion compat: HSTS is HTTPS-only (RFC 6797), absent on the http
onion vhost. New shared helper expectHstsWhenTls() gates the assertion on the
reached scheme; wired into web-svc-cdn spec.
- rerun-spec.sh: route the inner-loop browser through the Tor SOCKS proxy when
the staged env targets a .onion, mirroring the deploy-time runner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
email.py's environment folds TLS_ENABLED into its "external" base, so a non-external relay (no mailu deployed) under TLS resolved host to the mailu domain while auth/tls/from stayed in localhost mode: an unreachable, inconsistent msmtprc that aborted the sys-ctl-hlth-msmtp unit and the deploy (chess/bridgy-fed/hugo/legal). Gate host on external directly, like tls/auth/from, so a mailu-less deploy targets the loopback postfix that sys-svc-mail-smtp deploys under the same email.external gate. Adds a guarding unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Widen the openresty fan-in overlay /28 -> /24 (the /28 exhausted VIPs under mailu's services, deadlocking the stack). Pre-create NFS backing dirs for every volume of unpinned roles (placement decides, not the legacy per-volume nfs flag). Make email 'external' cluster-wide so swarm workers relay through the central mailu instead of a localhost root sender. Constructor: auto-place svc-prx-openresty, gate out unconsumed central engines, and load the svc-dns role group.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`DIR_COMPOSITIONS` resolves to a Jinja expression in swarm mode
(`{{ DIR_VAR_LIB ~ "/compose/" if DEPLOYMENT_MODE == "swarm" else "/opt/compose/" }}`),
but Ansible passes variables un-templated into lookups. container, database,
and compose_file_args now call `templar.template()` when the raw value still
contains `{{`, so the resolved path reaches `get_docker_paths()`.
email's `_render` switches to `_templar_render_best_effort` (TrustedAsTemplate
for Ansible 2.19+) and loops up to 4 passes: nested inventory expressions like
`DOMAIN_PRIMARY="{{ lookup('env','INFINITO_DOMAIN') }}"` need multiple
substitution rounds before reaching the literal domain.
Drops a stale `import textwrap` in test_depends_on.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`,
`requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from
non-Markdown source files. Comments anchored to requirement numbers
rot when requirements are renumbered or rewritten and conflate
present-state behaviour (which the comment should describe) with the
audit trail of how the code got there (which belongs in commit
messages, PR descriptions, and the requirement file itself).
* Document the rule in `docs/contributing/documentation.md` under
"Comments". Code MUST describe the *why* in present-state language;
contract enforcement lives in tests, not folklore.
* Clean 254 source files across `roles/`, `tests/`, `plugins/`,
`cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root
`eslint.config.js`. Pure comment / docstring / log-message edits;
no behavioural change.
* Add `tests/lint/repository/test_no_req_references_in_code.py` to
enforce the rule going forward across `.js`, `.py`, `.yml`,
`.yaml`, `.j2`, and `.sh` files. The legacy
`test_no_stale_requirement_refs.py` is exempt as the only file
whose purpose is to detect these very tokens.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds ``[tool.ruff.lint]`` with ``select = ["E", "F", "I", "B", "UP"]``
and ``target-version = "py311"``. ``E501`` is ignored because
``ruff format`` already reflows what it can; the remaining long lines
are unsplittable string literals / URLs.
Auto-fixed (2753 issues, behavior-preserving):
- UP006/UP035: ``List[X]`` → ``list[X]``, ``Dict[X]`` → ``dict[X]``,
``from typing import …`` redirected where superseded by builtins.
- UP045: ``Optional[X]`` → ``X | None``.
- UP010/UP009/UP015: dropped Python-2-era pragmas (``from __future__
import absolute_import, division, print_function``,
``# -*- coding: utf-8 -*-``), redundant ``open(..., "r")`` mode.
- UP004: dropped implicit ``class X(object):``.
- UP022/UP008/UP032: stdlib idiom modernization (``capture_output``,
super-call simplification, etc.).
- B905: ``zip(...)`` → ``zip(..., strict=False)`` (preserves prior
behavior; switching to ``strict=True`` is a separate semantic
choice not made here).
- I001: import-block ordering.
Hand-fixed (66 issues):
- B904 × 54: appended ``from <bound>`` (when the enclosing ``except``
binds a name) or ``from None`` (when it doesn't) so chained
exceptions stay distinguishable from errors-during-exception-
handling. Applied via a small AST-aware script that walked back to
the enclosing ``except`` clause and matched the binding.
- B017 × 4: narrowed broad ``assertRaises(Exception)`` to the actual
``AnsibleError`` / ``AnsibleFilterError`` raised by the code under
test.
- B023 × 2: bound captured loop variables via default args
(``def _record(name, *, _path_real=path_real, ...)``) so each
closure gets its own snapshot.
- B007 × 2: unused loop control variables renamed to ``_`` /
rewritten to ``.values()``.
- UP007 × 2: ``Union[X, Y]`` → ``X | Y`` (plus dropping the now-unused
``from typing import Union``).
- B018 × 1: Odoo addon manifests are bare dict expressions at module
level by design — suppressed with an inline ``# noqa: B018`` and
rationale comment.
- UP031 × 1: printf-style format → f-string.
The 65 ``# noqa: …`` markers introduced by the auto-fixer all carry
real ruff codes (``B018``, ``E402``, ``F401``, etc.), so
``test_noqa_only_ruff_codes`` stays green.
Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files already formatted
- ``tests.lint.repository.{test_project_root_import,
test_no_direct_yaml_calls, test_noqa_only_ruff_codes}`` and
``tests.unit.utils.annotations.test_suppress`` → 20 tests OK
- 1621 unit tests → all pass
- Integration suite: 44 failures (pre-existing baseline, none from
this change), 1 skipped.
Going forward, the new rules block regressions on E (style), F
(pyflakes), I (import order), B (likely-bug patterns), and UP
(modernization to py311) for any new code.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
email._compute('domain') returned variables['DOMAIN_PRIMARY'] verbatim. When
DOMAIN_PRIMARY held an unrendered template (e.g. the group_vars default
"{{ lookup('env','DOMAIN') | default('infinito.localhost', true) }}"), the raw
expression leaked into rendered templates. In discourse's containers/discourse.yml
the literal "{{ ..." opened a YAML flow mapping and made Ruby Psych fail with
"did not find expected ',' or '}' while parsing a flow mapping", aborting
`./launcher rebuild discourse` in CI.
Route every _compute() result through the same _render() helper that already
covered SYSTEM_EMAIL_* overrides, and add a regression test that exercises a
templated DOMAIN_PRIMARY default.
Moves the SYSTEM_EMAIL_* resolution logic out of group_vars/all/03_email.yml
into a dedicated Ansible lookup plugin, so every role template reads outgoing
mail settings via lookup('email') (global) or lookup('email', application_id)
(per-app). Per-app compose.services.email is merged on top of the resolved
defaults; SYSTEM_EMAIL_* variables remain usable as an inventory escape hatch.
Also splits the email docs: administrator-facing configuration stays under
docs/administration/configuration/, plugin contract and extension rules move
to docs/contributing/design/services/email.md. services.md is relocated to
services/base.md so the services/ subtree can host per-service design pages.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
