Stalwart closes a connection that exceeds its per-IP connection budget
without a greeting. msmtp reports that as "the server sent an empty reply"
with exit 76, and the mail health check treated exit 76 as permanent. One
such drop, in a second in which Alertmanager flushed its notifications,
failed a deploy although the same path accepted a mail one second later.
The health check now retries exit 76 when msmtp names the empty reply,
within the ten attempts it already gives to exit 69, 75 and 124. Every
other protocol error still ends the check at once.
The alarm mail was sent once, so the alarm about a failed unit was lost to
the same drop. It now makes three attempts ten seconds apart for the
transient cases of the health check and still reports the last error.
The limit on the server is left alone. Exempting the node's own senders by
source address would also exempt Tor clients and, in swarm, every external
client. That question is tracked in #793.
Closes#790
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mailu meters each sender login at an undeclared 200 messages per day. The
platform's no-reply identity carries both the deploy-time msmtp health check
and every service notification, so a day of deploys exhausts the quota and
from then on every further deploy fails on that health check rather than on
anything being wrong. Counted across eight runs: six failures each in the four
runs before the exemption, none in the four after. The limit was still active
an hour after the last attempt and the unit's timer only fires daily, so it
was not going to clear on its own.
The rejection comes from Mailu's sender-rate endpoint, not from its auth rate
limit. Postfix wraps the reply as "SASL login name rejected", which reads like
an authentication failure and is not one. MESSAGE_RATELIMIT_EXEMPTION is
checked before the counter is touched, so an exempt sender consumes nothing,
and it is Mailu's own mechanism for exactly this case. Raising the limit
instead would move the boundary for real user mail as well.
The value is lowercased because Mailu lowercases the configured set but
compares it against the raw sender from postfix, so a deployment whose domain
carries uppercase would silently miss the exemption.
The health check also misread the rejection. Every 4yz reply is transient per
RFC 5321, but the retry branch only matched 454 and the temporary
authentication wording, so a 450 aborted the deploy on its first attempt. It
now retries on any 4yz, which costs a longer wait when a 4xx condition is
persistent and keeps 5xx immediately terminal.
Its pacing is widened from twelve attempts twelve seconds apart to ten thirty
seconds apart. That was originally reasoned against AUTH_RATELIMIT and is not
what fixed the failure; it stands because a health check has no reason to
authenticate five times a minute.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
While rspamd's auth backend warms up during a deploy, mailu answers
'454 Temporary authentication failure'. msmtp maps every auth failure to
exit 77 (EX_NOPERM), which the health script treated as a permanent fault
and failed on immediately, so whichever node's check raced the warmup
window failed the deploy (fediwall swarm: only wrk-01, while the manager
and wrk-02 passed with the same config). Capture msmtp stderr and retry
exit 77 only on the transient 454 reply, still failing fast on a
permanent 5xx.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The liveness check sent one mail and failed the deploy on a transient 451 'try again later' emitted while mailu (postfix/rspamd) was restarting. Retry a fixed 12 times (12s apart) ONLY on transient exit codes (69 EX_UNAVAILABLE, 75 EX_TEMPFAIL, 124 timeout); any other code (config/usage/permanent reject) exits immediately with that exact code instead of being masked by retries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CI run 26721196005 failed at:
TASK [sys-service : sys-service | Fail after shared service diagnostics]
msg: Unable to start service hlth-msmtp.9.0.2.infinito.nexus.service.
Two breakages in the start sequence:
1. `sys-ctl-hlth-msmtp/files/script.sh` called `hostname` in `$(...)`.
The CI deploy containers ship without that binary, so `set -eu`
aborted before msmtp ever saw stdin. Fall back to `${HOSTNAME:-...}`
then `uname -n` then a literal "unknown".
2. `sys-svc-mail-msmtp/templates/msmtprc.conf.j2` hard-coded the trust
file path per RedHat family — the file was missing on the deploy
image and msmtp aborted with `Error while reading file. exit=66`.
Probe a candidate list at deploy time (`SYS_SVC_MAIL_MSMTP_CA_BUNDLE_CANDIDATES`),
pick the first existing path, surface it as
`sys_svc_mail_msmtp_ca_bundle`, and install `ca-certificates`
ahead of the probe.
`make test` 5/5 green (186 OK).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cluster 2a (hlth-msmtp failed=65 in universal single-app deploys):
the health-check script sent "To: administrator" but the administrator
Unix user is only provisioned in server-pass, so postfix rejected the
recipient. Replace the inline Jinja conditional with a role-local
lookup plugin that picks administrator.email (external), "administrator"
(local + user exists) or "root" (local + missing).
Side change: SIM105 cleanup in utils/install/lint/__main__.py.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5.5