The ansible flavor's runner starts its own docker daemon, and that daemon
pulled every MDAD image straight from Docker Hub and ghcr.io, past the
registry cache the stack's own daemon uses. MDAD pulls with
`pull: always`, so every bootstrap spent the shared anonymous Docker Hub
quota; locally a single run exhausted it and died with
"toomanyrequests" even for images already on disk.
The role now reads the proxy of the stack's docker.service. When there
is one, it mounts a docker.service drop-in into the runner with the same
HTTP_PROXY, HTTPS_PROXY and NO_PROXY, and pins the proxy host through
container_extra_hosts. When the stack's daemon trusts the registry
cache's MITM CA, the drop-in fetches that CA from the proxy and runs
update-ca-certificates before the inner daemon starts. Without a proxy
on the stack's daemon nothing changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A service entity can list canonical domain keys under `domains`; the
web health expectations skipped them only while the service was
disabled. A consumer whose service is enabled but shared does not serve
its own domain for it either, since the provider serves the service on
its own domain, so the probe failed with an unrecognised TLS name.
Domains of an enabled and shared service are now skipped as well.
The only other entity with `domains` (web-app-seaweedfs frontend) is
never shared, so its behaviour is unchanged.
Refs #1165
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Prosody keeps an account's first password. The image's init registers
focus and jvb only once, so after a credential rotation it prints
"User exists" and keeps the old passwords, while jicofo and the
videobridge start with the new ones. Jicofo then fails with
"SASLError using SCRAM-SHA-1: not-authorized" and no conference can
start.
After compose up the role now deletes and re-registers both accounts
with their current passwords, through the container_address lookup so
it works in compose and swarm.
Found by the full-cycle gate of #1165, whose second pass runs after a
credential rotation.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The mod_auth_ldap2 override redeclares VirtualHost "meet.jitsi" with
authentication = "ldap2". It was staged whenever LDAP was enabled, also
with SSO, where JITSI_AUTH_TYPE is internal and ENABLE_AUTH is false:
the web client then logs in anonymously, Prosody offers no matching
mechanism, and every conference join fails with
"connfail[no-auth-mech]" and reloads every 17 seconds. The role's own
spec only reaches the prejoin lobby, so it never noticed.
The override is now staged only when JITSI_AUTH_TYPE is ldap, and an
override left by an earlier deploy is removed otherwise.
Found by the new Matrix Jitsi spec against the shared Jitsi (#1165).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The resume check stats the pending marker after 04_files.yml has
written it, so it fired in every run that changed a compose file,
logged that the last compose up never completed, and notified
compose-build on top of the compose-up the change had already
notified. A PR CI compose deploy showed it for every role
("Resume the compose up a failed run left pending => changed" right
after "Mark compose up pending => changed").
The marker task now registers its result, and the resume task skips a
marker that this run wrote. A marker a failed run left behind is
unchanged when the file task meets it, so it is still resumed.
Refs #1169
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The pending-up marker followed compose.yml, .env and the compose override
but not the role's optional Dockerfile. A run that changed only the
Dockerfile and failed before `compose up` left no marker, so the next run
neither rebuilt nor recreated the containers. The marker now also follows
the Dockerfile task.
Resuming a pending marker notified only compose-up. The compose pull
skips buildable services (`--ignore-buildable`) and only compose-build
builds them, so a resumed up started the previous image. The resume task
now notifies compose-build as well; without a build section in the
project, `compose build` builds nothing.
Reported by CodeRabbit on #1174.
Refs #1169
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The ansible flavor turned on MDAD's Jitsi and Element Call stacks and the
Heisenbridge IRC bridge by default, while the role publishes only
Synapse's client and federation ports and Element, and the front proxy
wires only the Element and Synapse domains. Jitsi's videobridge, Element
Call's LiveKit media ports and their web domains are reachable from
nowhere, so the call buttons cannot connect while the stacks still cost
memory and CPU, and hosts moving from the compose flavor gain an IRC
bridge they never asked for.
`services.matrix.conferencing.{element_call,jitsi}` and
`services.matrix.plugins.heisenbridge` now default to false, matching
what production hosts already set per host. The README states the
defaults, including the bridge list, which still claimed several
bridges were on.
The Element Call and bridge-roster Playwright specs skip while the flags
are off. Wiring the ports, proxy entries, domains and certificates stays
open in #1165.
Refs #1165
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
is_database() compared the whole repository part of the image reference
against postgres, mariadb, redis, memcached and mongo. Every reference
with a registry or namespace prefix, every derivative image and every
locally built `<engine>_custom` image therefore counted as application
data and went to the mass storage. On a local stack the central
databases report `postgres_custom:18` and `openldap_custom:13.7-slim`,
both classified False and moved to the HDD.
The check now takes the last path component of the reference, drops
digest, tag and a `_custom` suffix, and compares it against the engines
the svc-db-* providers and the app-local databases run: elasticsearch,
mariadb, memcached, mongo, mysql, openldap, opensearch, postgis,
postgres, qdrant, rabbitmq, redis, typesense, valkey.
Volumes the script already symlinked stay where they are; a database
volume misplaced on the HDD has to be moved back by hand.
Fixes#1163
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Three docstrings from #1173 claimed that ensure() leaves only the pinned
release on PATH or replaces whatever version PATH finds. It does neither:
it installs the release into /usr/local/bin and fails unless PATH then
resolves to it, so a distribution copy such as /usr/bin/shfmt stays on
disk behind it.
The test module and ensure() lose their docstrings; the module docstring
of the installer returns to the form its siblings use ("Install hadolint
via GitHub-release prebuilt binary.").
Reported by CodeRabbit on #1173.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CentOS jobs also die on "Failed to download metadata for repo 'epel':
Cannot download repomd.xml ... All mirrors were tried" (feature/store-i18n
run 39658756830, Baserow#0 and Nextcloud#8 aarch64): the failing host's
librepo log shows mirrors that answered and then failed the checksum the
metalink expects, the stale-metalink signature already pinned away for
the CentOS Stream repos, plus TLS errors from two mirrors.
centos_mirror.sh now also rewrites the EPEL repo files: each section's
own commented `#baseurl=https://download.example/pub/epel/...` template
becomes the baseurl on dl.fedoraproject.org and the Fedora metalink line
goes. Using the file's template keeps the path variables of whichever
epel-release is installed (`$releasever${stream:+s}` on CentOS Stream,
`releasever_minor` in newer releases). A file whose metalinks lack that
template stops the script with an error instead of losing its source.
The Dockerfile runs the script again right after
scripts/install/filesystem.sh. Its first run precedes the EPEL install
(filesystem.sh adds epel-release for btrfs-progs), so the image shipped an
unpinned epel.repo, and the failing call, sys-svc-ssh's OpenSSH install,
runs before roles/update/tasks/centos.yml pins anything.
On CentOS Stream the template resolves to epel/10s/, which on
dl.fedoraproject.org serves the same aarch64 repomd.xml as epel/10.4/, the
only path the epel-10s metalink lists.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With the psql command tags gone the OIDC plugin task stays `ok`, but PeerTube
was still force-recreated on every deploy: "Back-fill column defaults so
bootstrap INSERT keeps NOT NULL" runs an unconditional `ALTER TABLE …
SET DEFAULT`, database_query reports every non-SELECT as changed, and the
task notifies compose-up, which runs `compose up -d --force-recreate`.
A SELECT on information_schema.columns now lists the bootstrap columns
(`nodeVersion`, `nodeABIVersion`) that still have no default, and the ALTER
with its compose-up notification runs only when that list is not empty. A
fresh database still gets the defaults before PeerTube's bootstrap INSERT
retries; every later deploy skips the task. The unused register of the
ALTER task is gone, and both read-only checks now live under
files/sql/check-application/ as the sibling-prefix rule requires.
Refs #1164
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With `-tA` psql drops headers and the row-count footer but still prints
the command tag of every write statement. A guarded
`UPDATE … RETURNING` that matched nothing therefore came back as
`rows: [['UPDATE 0']]`, and web-app-peertube's oidc/enable.yml, which
judges its change by the row count, reported `changed` and restarted
PeerTube on every deploy after the first.
The PostgreSQL rows flags are now `-qtA`. Probed against a running
postgres container:
-tA -> CREATE TABLE / INSERT 0 1 / UPDATE 0 / 3 / UPDATE 1
-qtA -> 3
SELECT output and the MariaDB path (`--batch --skip-column-names`, no
status line) are unchanged.
Fixes#1164
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A deploy that renders a changed compose.yml, .env or compose override
notifies the compose-up handler. When the run fails before that handler
completes, for example because the image pull hits Docker Hub's
toomanyrequests, the notification is lost. The next run renders the same
files (changed=false), and the fallback in tasks/utils/up.yml notifies
compose-up only when no container of the stack runs, so the stack keeps
its previous images and configuration while the deploy ends with
failed=0 (reproduced with web-app-akaunting 3.1.8 -> 3.1.21).
04_files.yml now writes `.compose-up-pending` into the instance
directory whenever one of those three files changed in compose mode. The
compose-up handler removes it right after `compose up` succeeds, so a
failed pull or up leaves it in place, and up.yml notifies compose-up
whenever it finds the marker on a later run.
Comparing `compose config --hash` with the containers' config-hash
labels was ruled out: on an up-to-date stack the two differ for every
service, so it would force-recreate on every deploy.
Fixes#1169
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`shfmt.ensure()` returned as soon as any `shfmt` was on PATH and tried
the distribution package before the release pinned in
roles/sys-lint/meta/services.yml. The Debian dev container therefore
formats with its packaged 3.8.0 (/usr/bin/shfmt), Fedora with its packaged
3.7.0, the CentOS one with the pinned v3.14.1 (/usr/local/bin/shfmt), and
they disagree on
scripts/tests/deploy/swarm/utils/unmount/nfs_mounts.sh: every gate in a
CentOS stack moved the `then` of the heredoc `if` below the terminator.
ensure() now follows ruff's installer: it reads `shfmt --version`, keeps
the binary only when it reports the pinned release, and otherwise installs
the release binary into /usr/local/bin. It fails when PATH still resolves
another version afterwards instead of formatting with it. The system
package path is gone.
nfs_mounts.sh is committed in the pinned release's format (byte-identical
to the v3.14.1 output from the CentOS container), so the first gate on any
distro leaves the tree clean.
The install-lint stamp now also depends on roles/sys-lint/meta/services.yml,
the pin file of actionlint, hadolint and shfmt, so a bumped pin reaches
environments whose stamp predates it; until now only pyproject.toml and
scripts/install/lint.sh invalidated it.
Fixes#1155
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every swarm deploy of PR #1114 stopped at the sys-svc-compose handler that
reloads the guard after `docker stack deploy`:
[ERROR]: Using 'ansible.builtin.include_role' as a handler is not supported.
Origin: roles/svc-swarm-node/tasks/utils/ingress_guard/refresh.yml:9:3
The handler includes refresh.yml, and ansible-core refuses a role include
anywhere below a handler (playbook/helpers.py: `use_handlers` propagates
into included task files). The join path ran the same file as a plain task
and passed, which is why the guard loaded before the join and failed only
after the first stack deploy (CI run 39754950746: SuiteCRM#2 on CentOS and
the workspace job's MariaDB swarm drill on Fedora).
refresh.yml now includes svc-net-firewall's 02_fragment.yml as a task file.
The fragment's naming (directory, source, file, table) lives in the role's
vars/main.yml, which a task include does not load, so refresh.yml reads that
file through include_vars under its own name and hands the four values to the
include. The vars file stays the single definition, and the namespaced load
cannot shadow FIREWALL_FRAGMENT_ROLE for a later include of the role.
Refs #1089
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
manage.py create-profile and update-profile parse their options with
argparse, and `-p <value>` reads a value that starts with `-` as the next
option. A generated password with a leading dash therefore fails both
commands with exit 2 ("argument -p/--password: expected one argument")
before they reach the backend, and the bootstrap task retries until it
gives up. On main this broke the guide run of Penpot#2 in run
39668792538: the Docker events of the rescue snapshot show eight
create-profile execs ending with exit 2 while the backend was healthy
and its prepl received no call.
Without -p, manage.py reads the password with getpass, which falls back
to stdin when the exec has no TTY. Both tasks now feed the password to
`container exec -i` on stdin. This also keeps the password out of the
exec command line, which Docker writes into its event stream and
therefore into the rescue diagnostics artifacts.
The password-quote lint masked only block `stdin: |` / `stdin: >`
values and flagged the inline `stdin: "{{ PENPOT_ADMIN_PASSWORD }}"` as
an unquoted password in a shell command. stdin never reaches a shell, so
the lint now masks every `stdin:` form; an unquoted password in `cmd`
is still reported.
Fixes#1139
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
