b067d21ef shipped two ways to kill a dedicated MariaDB, both reproduced.
The file was rendered 0640 under a play that runs as root, while
stack_host_template otherwise defaults to 0644. Compose emulates a config as a
plain host bind and ignores uid, gid and mode on the reference, so the engine,
which does not run as root, could not open the file. mariadbd treats an
unopenable init file as fatal and aborts every start. Dropping the explicit mode
puts it back on 0644, next to the env file that carries the identical password.
On a fresh data directory the entrypoint starts a temporary server with the same
arguments, so the init file ran before the database was ever set up. Giving root
a password there breaks the entrypoint's own readiness probe, which connects
without one, and the schema and grants are then never created. The CREATE USER
had the same problem from the other side: the entrypoint's own CREATE USER
carries no IF NOT EXISTS and the script runs under set -e.
What remains is the one statement the failure actually needs: the application
user, guarded by IF EXISTS so it is a no-op on the first start and a
realignment on every later one. That is the identity the health check
authenticates as.
Root keeps the password its data directory was initialised with. It did so
before this feature existed, nothing in the repository authenticates as root
against a dedicated engine, and the account is reachable only from inside the
app's own stack network. Realigning it needs a gate that can tell the
entrypoint's temporary server apart from the real one, which this does not have.
The GRANT is gone with it: rotating a credential changes passwords, not
privileges.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An engine of its own keeps its passwords in its data directory, written once
when that directory was initialised. When the deploy rotates the credential the
new value reaches the container's environment and never the engine, so the
health check authenticates with a password nothing has ever accepted. The
application user and root come from the same value, so nothing is left that
could log in and change it either.
mariadbd reads --init-file on every start, before it opens the socket, and runs
the statements with full privileges. That is the one point where a password can
be reset without knowing the previous one.
The SQL is rendered next to the env file that already holds the same secret in
plaintext, so it adds no new exposure. It reaches the container as a config
rather than a bind mount because a swarm task may be scheduled on a node the
rendered path does not exist on; the config name carries a hash of the file
content, so a rotated password produces a new name and swarm replaces the
immutable old one.
The guard is mariadb-and-not-shared, which matches the `local` guard the service
template already renders on, so the block and the top-level entry appear
together or not at all.
3831c8f804 fixed the other half of this failure: matomo, moodle and nextcloud
keep the same password in a config file on their data volume. Same rotation,
opposite end of the connection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5