fix(sys-ctl-hlth-msmtp): retry mailu's transient 454 auth failure

While rspamd's auth backend warms up during a deploy, mailu answers
'454 Temporary authentication failure'. msmtp maps every auth failure to
exit 77 (EX_NOPERM), which the health script treated as a permanent fault
and failed on immediately, so whichever node's check raced the warmup
window failed the deploy (fediwall swarm: only wrk-01, while the manager
and wrk-02 passed with the same config). Capture msmtp stderr and retry
exit 77 only on the transient 454 reply, still failing fast on a
permanent 5xx.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

roles/sys-ctl-hlth-msmtp/files/script.sh

@@ -1,38 +1,32 @@
 #!/bin/bash
-# set -u, not -e: the retry loop inspects msmtp's exit code explicitly, so a
-# non-zero send must not abort the script on its own.
 set -u
 : "${MAIL_RECIPIENT:?MAIL_RECIPIENT not set}"
 : "${MAIL_TIMEOUT:?MAIL_TIMEOUT not set}"
 HOST="${HOSTNAME:-$(uname -n 2>/dev/null || echo unknown)}"
 
-# The mail stack (mailu postfix/rspamd) briefly answers 451 "try again later"
-# while it (re)starts during a deploy. Retry only those transient failures; any
-# other exit code is a real fault and is propagated immediately, unmasked.
 attempts=12
 delay=12
 
 for attempt in $(seq 1 "${attempts}"); do
-  {
-    echo "To: ${MAIL_RECIPIENT}"
-    echo "Subject: ${HOST} is alive"
-    echo
-    echo "Host ${HOST} reports at $(date): I'm alive."
-  } | timeout "${MAIL_TIMEOUT}"s msmtp -t
+  msmtp_err="$(
+    {
+      echo "To: ${MAIL_RECIPIENT}"
+      echo "Subject: ${HOST} is alive"
+      echo
+      echo "Host ${HOST} reports at $(date): I'm alive."
+    } | timeout "${MAIL_TIMEOUT}"s msmtp -t 2>&1 1>/dev/null
+  )"
   rc=$?
   [ "${rc}" -eq 0 ] && exit 0
 
-  # Retry ONLY on transient codes:
-  #   69  EX_UNAVAILABLE - server temporarily unreachable or refused the
-  #                        envelope; this is the 451 "try again later" we hit
-  #                        while mailu is (re)starting.
-  #   75  EX_TEMPFAIL    - msmtp's code for a temporary 4xx SMTP reply.
-  #   124               - `timeout` killed the send (it exceeded MAIL_TIMEOUT;
-  #                        the server hung instead of answering).
-  # Anything else (e.g. 64 EX_USAGE, 78 EX_CONFIG, permanent 5xx rejects) is a
-  # real error: exit now with that exact code instead of masking it with retries.
   case "${rc}" in
     69 | 75 | 124) ;;
+    77)
+      case "${msmtp_err}" in
+        *454* | *"Temporary authentication failure"*) ;;
+        *) exit "${rc}" ;;
+      esac
+      ;;
     *) exit "${rc}" ;;
   esac