Element v1.12.33 (automated bump a8fba45d60, on main) ships element-web
#35364, which turns force_verification on by default. From the second
OIDC login of an account on, Element stops on "Confirm your digital
identity" with no skip button, so the Matrix SSO specs that log the same
account in twice (test-dm-admin-biber.js, test-design.js) never reach the
signed-in UI: "administrator: authenticated Element UI (user menu / room
list / welcome) must render" fails after 120 s (fork run 40625679270,
Matrix#2 and the co-deployed Moodle#2).
services.element.force_verification now carries the platform default,
true as upstream, and both flavors render it (the compose element config
and the MDAD element configuration extension). Only variants 0 and 2,
the SSO rounds, set it to false. Production keeps forced verification.
Cherry-picked from e042c9dbf9 on feature/store-i18n. main has no Element
branding block yet, so the MDAD flavor carries only the force_verification
key in its configuration extension.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An earlier commit on this branch turned the three off because the ansible flavor ran
them without a published port, proxy entry, domain or certificate. They
now stay on and are wired end to end.
Element Call: MDAD's LiveKit server and JWT service get their own
entities in meta/services.yml with resource limits, local ports and a
public media port (TCP and UDP). The runner container publishes them
through container_ports, and the Synapse vhost proxies /livekit-server/
(websocket) and /livekit-jwt-service/ to them. The JWT service checks
each OpenID token against the user's homeserver over federation, and
federation was not reachable in this flavor: Synapse served it only on
a plain-HTTP listener that the runner published directly on 8448.
Synapse now serves the client and federation resources on one listener,
OpenResty serves the Synapse vhost on 443 and as default_server on the
federation port, the direct 8448 publish is gone, and the role writes
/.well-known/matrix/server as the compose flavor does. With a
self-signed CA the JWT container mounts the CA bundle as Synapse does.
Jitsi: a new services.jitsi entity follows the shared-service contract.
It is enabled in the ansible flavor off tor nodes. When web-app-jitsi is
in the inventory, Element points at that shared Jitsi and MDAD's own
Jitsi stays off. Otherwise MDAD runs its own Jitsi on jitsi.<domain>
with a front-proxy vhost, its web port published to the runner and its
videobridge on a public UDP port. The entity lists the jitsi domain, so
the web health probe skips it whenever Matrix does not serve it. The
jitsi_*_container_network overrides were dropped: MDAD reads none of
them, Jitsi always ran on its own network. Variant 0 covers the shared
case, variant 1 the internal one. On an onion-primary deploy both call
stacks stay off, so Element never advertises a clearnet call surface.
MDAD's Prosody keeps an account's first password, so after a credential
rotation jicofo failed SASL with the new one. The role now derives the
jicofo and videobridge passwords from the macaroon secret and recreates
both accounts on every deploy.
Heisenbridge stays on by default. It runs in the nested host network,
where MDAD's container-name URLs do not resolve, so its homeserver URL
and the appservice URL in its registration point at 127.0.0.1.
A bootstrap that a previous deploy left failed (for example after a
Docker Hub rate limit) is now reset and restarted, so a mode=update
deploy converges instead of failing on the stale unit state.
Playwright proves each path: the RTC spec fetches a LiveKit token from
the JWT service and opens the SFU signalling websocket, the Jitsi spec
checks Element's preferred domain and joins a conference (signing in
through Keycloak first when the shared Jitsi sits behind SSO), and the
Heisenbridge spec opens a DM with the bot and waits for its answer to
"help".
Closes#1165
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The ansible flavor's runner starts its own docker daemon, and that daemon
pulled every MDAD image straight from Docker Hub and ghcr.io, past the
registry cache the stack's own daemon uses. MDAD pulls with
`pull: always`, so every bootstrap spent the shared anonymous Docker Hub
quota; locally a single run exhausted it and died with
"toomanyrequests" even for images already on disk.
The role now reads the proxy of the stack's docker.service. When there
is one, it mounts a docker.service drop-in into the runner with the same
HTTP_PROXY, HTTPS_PROXY and NO_PROXY, and pins the proxy host through
container_extra_hosts. When the stack's daemon trusts the registry
cache's MITM CA, the drop-in fetches that CA from the proxy and runs
update-ca-certificates before the inner daemon starts. Without a proxy
on the stack's daemon nothing changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The role advertised `matrix-synapse:9000` to Prometheus in both flavors.
That container only exists in the compose flavor. In the ansible flavor,
which is the default, Synapse runs inside the `matrix` container and had
no metrics listener, so the scrape target was down on every deployment
that pairs Matrix with Prometheus.
The ansible flavor now enables the Synapse metrics listener on 9100 and
the declaration names port and service per flavor, so the target is
`matrix:9100` there and stays `matrix-synapse:9000` in the compose
flavor.
This is the metrics part of b3e0dce578 on `feature/store-i18n`, without
the addon switch that branch introduces around it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The list form could publish a host:container pair and nothing else, so thirteen blocks stayed hand-written: the ones needing a protocol suffix, a port range, the swarm long syntax with mode host, or a container-port-only publish. Each of those duplicated numbers the role already declares in meta/services.yml, which is the drift the lookup exists to remove.
The lookup keeps the list form unchanged for its 93 existing call sites and gains a dict form: transport emits one item per protocol, container 'same' serves the services that cannot remap (STUN/TURN, media relays), mode 'host' emits the long block an edge role needs in swarm so the client address survives the hop, publish 'ephemeral' emits the container port alone, and key names the block for Discourse's launcher file, whose expose: list is what becomes docker run -p there. A declared {start, end} mapping renders as start-end on both sides, so a relay range needs no term of its own.
svc-prx-openresty had no ports block at all and bound 80 and 443 as literals in its template, while torrc.j2 carried a third literal 80 for the hidden service. None of the three referenced the others. openresty now declares the ports, the compose template reads them, and the onion's port 80 comes from ports.onion.http through TOR_ONION_EXTRA_PORTS.
Three blocks stay raw, each with the reason it cannot route through one call: keycloak's sso_proxy maps a published port owned by the consuming role to a container port owned by web-app-keycloak, and matrix's ansible flavor takes element on 8080 and federation on 8048 from the upstream playbook rather than from this repo's ports.internal.
The enforcing lint scanned only files with compose in the name and only double-quoted list items, so nine of the thirteen blocks were invisible to it. It now reads every role template that can carry a ports block, catches unquoted items, protocol suffixes, ranges, flow style and the long syntax, and tracks the block by the indentation of its key: keying on the next line closed the block on anything shaped like a key, which target, published, protocol and mode all are. Under expose: only a real mapping counts, because there the single-segment form documents a container port and publishes nothing.
Verified by rendering all 110 container_ports call sites through the real templar, the real lookup and the real merged applications view.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sys-svc-compose-ca writes the CA material for exactly lookup('ca_injected', application_id), which asks three questions per application: has a domain, TLS enabled, mode self_signed. The lookup exists against this drift and says so in its own docstring: anything that reads the mounted file must ask the same three or it points at a path that was never mounted.
The matrix templates asked the global TLS_MODE instead. On an onion deployment TLS is off per application while TLS_MODE stays self_signed, so run 34653754175 logged 'CA trust N/A for web-app-matrix (TLS off/not self-signed) mark done' and the templates rendered the binds anyway.
The rest follows from the timing in that run: the matrix container was created at 05:49:35, the bundle was first written at 05:58:45 by sys-ca-selfsigned. At create time the bind pointed at a path that did not exist, which docker materialises as a directory, and the later docker restart matrix from 02_render_inventory.yml:22 failed on 'error mounting /etc/infinito.nexus/ca/ca-bundle.crt ... not a directory'.
MATRIX_CA_INJECTED holds the predicate once in vars/main.yml, beside MATRIX_CA_MOUNT_PATH, and both templates read it: the outer container bind and the synapse arguments for the nested daemon. The test pins both links of the chain, that the constant is derived from the lookup and that both if lines use the constant, so redefining it from TLS_MODE cannot pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
304144ce67 probed each bridge's healthcheck at /_matrix/mau/ready. mautrix
sets Ready only after Synapse has pinged the appservice (connector.go
starts the listener, then ensureConnection, then AS.Ready = true), and a
swarm task with a healthcheck stays "starting" and receives no VIP traffic
until it is healthy. The ping could never arrive: Synapse logged 150
ConnectionRefused per bridge, each bridge exited FTL 13, and the stack
never converged (run 34615516009, Matrix#2, job 103327034630).
The healthcheck now probes /_matrix/mau/live, which answers once the
listener is up, before the ping. A deploy task after the Synapse wait asks
every bridge for /_matrix/mau/ready from inside Synapse, over the same
name Synapse pings, so a bridge that never becomes ready still fails the
deploy without gating swarm routing.
The ChatGPT bot is rendered only once its access token exists. Its account
and token are created by stack_host_provisioning after the first stack
deploy, so an earlier bot task failed its login with M_FORBIDDEN until the
converge wait called it stuck. The re-render after provisioning adds it
with a valid token.
Verified in a compose run of variant 2: the readiness task got 200 from
meta, signal, slack and whatsapp on the first attempt. Tests pin the live
probe and the token guard and fail against the previous template.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every mautrix bridge generated its own registration with -g on start,
with fresh random tokens. Synapse had to read that file from a shared
directory, be restarted afterwards, and on swarm the services had to be
removed first so the registrations could be wiped (93735fd533). The
tokens changed on every redeploy and the deploy depended on a wait for
files a container writes.
Each bridge now has as_token and hs_token declared in meta/secrets.yml
and a fixed appservice_port in its addon. The bridge config reads the
tokens through lookup('config'), and registration.yaml.j2 renders the
registration from that config in the upstream -g format, so both sides
are known before anything starts and stay identical across redeploys.
The files reach the containers as secrets: matrix_bridge_mounts offers a
stack view (top-level definitions, content-hash named by compose_volumes),
a synapse view (one registration per bridge under /registrations, mode
0444) and a bridge view (the config at /config/config.yaml). The bridges
run as the synapse uid with an explicit --config and --no-update, keep
their scratch /data on a 16m tmpfs so hlth-volumes finds no anonymous
volume, and depend on a healthy synapse. Their healthcheck is
/_matrix/mau/ready, which the bridge only reports after whoami with the
as_token and an appservice ping carrying the hs_token, so a healthy
bridge proves the token pair.
The compose tasks lose the compose down, the swarm service rm block, the
registration wait, the permission fixes and both synapse restarts; this
reverts the workaround of 93735fd533.
Verified on the local compose stack: four bridges ready, synapse pings
answered, registration and config sha256 unchanged across a redeploy,
test-bridge-roster and 12 matrix Playwright specs green. A token
rotation (new pair in both files, compose up --force-recreate as the
handler runs it) came back healthy; a mismatched pair left the bridge
unhealthy. The swarm-shaped stack file passes docker stack config with
the 0444 secret modes, the 16 MiB tmpfs and content-hash secret names.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mdad stack runs its own Docker daemon inside the matrix container, so a
bind source in `matrix_synapse_container_extra_arguments` is resolved in that
container's mount namespace, not on the host. `vars.yml.j2` mounts
`CA_TRUST.bundle_host` while `services.yml.j2` passed only
`CA_TRUST.cert_host`, and those are two different files: the root CA and the
bundle beside it. The bundle therefore never existed where synapse looked.
`docker create` refuses the unit before Matrix logs anything of its own. From
the rescue artifact of job 102465251843 in run 34348821577, the nested
journal at containers/matrix/nested/.../journal.txt:5018:
matrix-synapse[5612]: Error response from daemon: invalid mount config for
type "bind": bind source path does not exist:
/etc/infinito.nexus/ca/ca-bundle.crt
Five start attempts, the same line each time, and the only error in the 872 KB
journal. matrix-heisenbridge follows it down as a dependency, `systemctl
restart` returns rc=1, the mdad bootstrap fails and the rescue task ends the
play. Not an OOM (exit 1, not 137, `OOMKilled: false`, 9.7 GB free), not disk
pressure, not a failed pull, and 16 % of the job budget spent.
b5fcd535de is where the asymmetry began: it moved this mount from `cert_host`
to `bundle_host` for a good reason -- SSL_CERT_FILE and REQUESTS_CA_BUNDLE
replace the trust store rather than extend it -- and left the passthrough on
`cert_host`. So the bundle stays; what was missing is the line that carries it
in, at the same absolute path the nested daemon asks for.
The test that comes with it reads both templates as text and requires every
CA_TRUST path synapse binds to be passed through at that same path. Rendering
would need a full Ansible context, and the invariant is about what the two
files say to each other rather than about the values. It fails against HEAD,
where `bundle_host` is bound and only `cert_host` is passed. It also fails
loudly if the regex ever stops matching how the extra arguments are built,
rather than quietly guarding nothing.
The cause is proven to the byte; the effect is not observed. `docker create`
never succeeded, so a second synapse defect behind this one would still be
invisible. A local `make compose-deploy apps=web-app-matrix mode=update` with
TLS_MODE=self_signed, grepping the nested journal for "Started
matrix-synapse.service", settles both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
start_interval is the cadence while start_period runs, and only while it runs.
Once the window closes the probe falls back to interval, so a service that
becomes ready just after it pays a whole tick anyway and the cadence never
reached it. Six of the seven flavors were shaped that way, Curl worst: a thirty
second window against a one minute interval.
Measured across 151 converge waits in four job logs, 63 sit exactly on a tick
boundary, 36 in the thirty second step and 27 in the sixty second one.
Attributed by application, web-app-seaweedfs is the largest at 1501s over
sixteen waits with a median of 57.7s: it probes with curl and fell straight into
the gap. web-app-zammad probes with connect, whose window is fifteen minutes,
and its 154s median is real boot time with nothing to recover.
So the window moves to ten minutes and the steady state interval to two, and the
flavor overrides that sat below them go: Curl, Wget and Connect lose interval
1m, Http its twenty second window, Nc its ten. Connect keeps fifteen minutes.
Neither number costs anything while a container works. start_period delays
nothing, it only stops failures counting, and interval no longer touches
convergence at all since that hangs on the window alone; it is steady state
load, which drops from 120 probes an hour per service to 30.
Forty six role files follow. Twenty seven declarations go that repeat the new
default or undercut it, an undercut window being the defect itself. Eight
windows longer than ten minutes keep their value and now say why. The literal
blocks and the three Dockerfiles this repository builds spell every timing out,
because outside the lookup an omission takes docker's value and not the
platform's.
A fourth lint rule keeps the shape: start_period must outlast one interval,
checked against the effective values, so raising interval without the window
cannot quietly bring the cliff back.
Two faults in the lint surfaced while building it and are fixed here. It
compared timings as strings, so 600s and 10m read as different values, and it
matched the healthcheck key exactly, so a line carrying a trailing nocheck was
skipped whole, which is how the two swarm-init partials went unexamined.
One regression, bounded. A container that starts and never turns healthy is
neither rejected nor errored, so stack_ready.sh only calls it fatal after three
killed tasks, and each cycle now spans the longer window. That reaches twenty
minutes rather than six, capped there by swarm_stack_converge_retries at 600
with a two second delay, and it lands only on runs that were failing anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A healthcheck written outside the container_healthcheck lookup inherits nothing
from utils/docker/healthcheck, and docker offers nowhere to put a replacement.
daemon.json carries storage, logging, DNS, registries and runtimes, and no
health key at all, so an omitted timing lands on docker's own value rather than
the platform's. The two disagree where it matters most: docker starts a
container with no start period, which leaves start_interval no window to act in.
That is why the hand-written blocks never gained the cadence.
The three images this repository builds itself now carry it. openldap and
postgres also drop to the platform interval, and bluesky gains the start window
it never had:
svc-db-openldap/files/Dockerfile --interval=30s ... --start-interval=5s
svc-db-postgres/templates/Dockerfile.j2 --interval=30s ... --start-interval=5s
web-app-bluesky/.../Dockerfile --start-period=30s --start-interval=5s
Six literal blocks move to the platform interval as well: magento, redis,
elasticsearch, matrix and both sys-svc-rdbms partials. Since start_interval
exists, interval no longer touches convergence and is steady state load alone.
Sixteen values that stay where they are now name the mechanism that keeps them
there. Two are worth reading. web-app-kix overrides start_period and nothing
else on purpose, letting the image's own HEALTHCHECK supply test, interval,
timeout and retries; filling those in would have overwritten what upstream tuned.
The memcached and redis swarm-init partials probe every second and deliberately
run with no start window, which is why start_interval is absent rather than
forgotten.
The rest name their probe: occ boots the whole Nextcloud stack, tor re-parses
its torrc instead of asking a running service, opensearch is still forming a
cluster, matrix waits on a file another step writes to a share.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
42ebd9ea71 gave the healthcheck SPOT start_interval, the cadence docker uses
while start_period is still running. A block a template writes out itself
bypasses the SPOT and never gained it. Measured across the repository: 46 such
blocks, 40 of them already carrying start_period, not one carrying
start_interval.
Without it a probe fires only every interval, so a service ready after two
seconds is reported healthy at the next tick and a stack converges no faster
than its coarsest quantisation. That lands in sys-svc-compose's swarm converge
wait, 1551.7s per CI job over 38 stack deploys.
Twenty five of the blocks move into meta/services.yml and render through
container_healthcheck. Each keeps an explicit test rather than picking a flavor:
a flavor would replace the command instead of carrying it across.
The move is lossless by construction. probes.py block() writes
overrides.get(key, default) for interval, timeout, retries, start_period and
start_interval, so a declaration repeating the literal values renders the same
block plus the missing cadence. Checked per block against the old values: the
only difference is start_interval 5s, nothing changed, nothing lost.
Twenty one stay literal. Seventeen interpolate a template variable, whose value
resolves in a scope meta/services.yml does not share, so moving them would be a
rewrite rather than a transcription. Four interpolate nothing and still cannot
move: svc-db-memcached and svc-db-redis probe every second in their swarm
service partial, where a 5s start cadence would report healthy later than the
block already does, and the sys-svc-rdbms and keycloak sso_proxy partials render
for the consuming role's application_id, so their declaration has no single
meta/services.yml to live in. Each carries a nocheck naming its reason.
All twenty one gain start_interval. Three also gain start_period, without which
the key does nothing at all, since it only applies while the start period runs.
That widens their failure tolerance rather than narrowing it: bluesky from 90 to
120 seconds, magento from 50 to 80, the mariadb partial from 180 to 210.
pixelfed spelled its probe as a bare string and now spells it CMD-SHELL. Docker
reads both as a shell command; only the YAML shape changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SSL_CERT_FILE, CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE replace the trust store;
only NODE_EXTRA_CA_CERTS extends it. compose_ca.py pointed all four at the
deployment own root CA, so every OpenSSL and rustls client in every container
was blind to every public certificate. Measured here, one variable, same URL:
unset curl https://registry.npmjs.org/ -> 200
a single unrelated CA -> 000 unable to get local issuer certificate
that CA + the system bundle -> 200
PeerTube was the one process that hit it: its plugin install fetches from
registry.npmjs.org at runtime through a client that honours SSL_CERT_FILE, and
it failed identically in compose and in swarm with
invalid peer certificate: UnknownIssuer
Everything else reaches its artefacts at image build time, or through Node,
whose NODE_EXTRA_CA_CERTS is additive and therefore kept working.
The replacement was never a posture. e7e8cf8c03 calls these variables an
"env-based CA trust fallback", f1ea036bd2 adds them "instead of falling back to
a plain docker run without CA trust", e79183f96a wires the extra CA in "so npm
traffic through the cache validates". Every commit framed them as adding trust.
sys-ca-selfsigned now builds ca-bundle.crt beside the root CA on every node,
from that node own files: the stack host mints the CA and workers fetch it, so
there is no cross-node dependency. The bundle is the root CA concatenated with
the first present CA_BUNDLE_CANDIDATES entry, the SPOT fc0881ff33 introduced.
The cat argument list is assembled in Jinja, so a host with no public bundle
emits the root CA alone, which is exactly the previous behaviour, rather than an
empty argument that would fail. bundle_host joins the existing stray-directory
sweep and the build runs after it.
compose_ca.py uses the bundle only when the file exists and is non-empty, so a
host that has not built one keeps the previous behaviour instead of binding a
missing path that Docker would materialise as a directory. The php ini, msmtprc
and the Matrix flavour may name the container path with no fallback because the
bundle is written at task 99 and the php ini at task 123: no host can carry the
ini without the bundle.
CA_TRUST_CERT and NODE_EXTRA_CA_CERTS stay on the single certificate. keytool,
occ security:certificates:import and update-ca-certificates each take one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ChatGPT bridge got its homeserver URL from container_address, which
answers "what do I docker exec into". Under swarm that is a shell subshell,
and it was being written into a compose value, so docker stack config refused
the whole stack before anything deployed:
invalid interpolation format for
services.matrix-chatgpt-bot.environment.MATRIX_HOMESERVER_URL
container_service answers the other question, "what is this service
addressable as", and answers it per mode: the declared container name under
compose, <stack>_<key> under swarm. Compose therefore keeps exactly the name
it already used; only the branch that was broken changes.
Escaping the $ would have satisfied the parser and shipped a literal $(...)
into the bridge's environment, and the declared container name is emitted
through compose_only, so under swarm it has no DNS record at all. Neither is
a fix.
Name the resolved value once. The same lookup was already spelled out a
second time for the swarm restart, so both now read MATRIX_SYNAPSE_SERVICE.
Rename MATRIX_MDAD_SERVICE to MATRIX_MDAD_SERVICE_KEY while the vocabulary is
in hand: every one of its uses passes it as a service key, one of them as the
key of a rendered compose service, and _SERVICE now means the resolved form.
The role's four synapse identifiers finally say what they are - the declared
name, the addressable service, the exec target, and the key.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bridge container was in a restart loop and never joined a room, so no
Matrix surface could reach the gateway. Four distinct causes, in the order
they surfaced.
MATRIX_ACCESS_TOKEN came from a credential declared with algorithm: plain,
a random string Synapse would never accept, so it resolved empty. With no
token, matrix-chatgpt-bot runs its token-fetch path, prints "Set
MATRIX_ACCESS_TOKEN to above token" and exits after 0.6s. Log the bot in
against Synapse and hand the container the token it actually issued.
compose.yml is written before the containers start, but the bot account only
exists after the Synapse health check further down the play, so the file has
to be rendered a second time once the token exists.
E2EE is disabled. The bridge uploads device keys that Synapse 1.159.0
rejects with `M_INVALID_PARAM: device_keys must not be null`, after which it
backs off forever and never completes a sync. This is a deliberate tradeoff:
bot DMs are no longer end-to-end encrypted, and the alternative is upstream
(a newer bot or an older Synapse). It hangs on MATRIX_CHATGPT_ENCRYPTION and
is one variable to flip back. The spec exercises an unencrypted room anyway,
because an E2EE timeline is opaque to an API-only client.
rc_login and rc_joins get a burst of 30. Synapse's defaults (login burst 3)
are too tight for a bridge that re-authenticates on every deploy, and the
restart loop drove them into M_LIMIT_EXCEEDED and later 429 (rc_joins.local).
This widens the burst allowance while leaving the sustained rate at 0.17/s.
The stack-host work moves into utils/stack_host_provisioning.yml so the
IS_STACK_HOST condition appears once instead of on two consecutive siblings.
Verified live: token present, container up rather than restarting, and
test-litellm-chatgpt.js passing. Not covered: a bot that meets an unjoinable
invite still crashes; 26 stale test rooms had to be cleared by hand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Matrix was the one role that already shipped an AI credential and pointed it at
the vendor, and four separate defects kept it from reaching the gateway. Each
one hid the next, so fixing fewer than all four still leaves a surface that
reports green and answers nothing.
The schema rejected the gateway by construction.
`chatgpt_bridge_openai_api_key` validated `^sk-[a-zA-Z0-9]{40,}$` -- OpenAI's own
key format -- with algorithm `plain`. A LiteLLM virtual key cannot satisfy that
pattern, so no amount of configuration could have worked. It is `sk_prefixed`
with a length check now, and a `litellm_api_key` entry joins it in the same
shape the other eight consumers use.
The bridge reached Synapse over the public ingress. MATRIX_HOMESERVER_URL was
MATRIX_SYNAPSE_URL, an https URL on the canonical domain, so every login left
the container network, crossed openresty and came back. A 502 taken while
Synapse was still starting still consumes an rc_login.address slot, and once
that bucket empties the bridge dies on M_LIMIT_EXCEEDED and restarts into the
same wall: measured at 139 restarts on the local stack. MATRIX_SYNAPSE_LOCAL_URL
addresses the container directly over plain http on the internal port.
MATRIX_SYNAPSE_URL stays public where it has to be -- public_baseurl, Element,
and the mautrix bridges, which are a separate concern and untouched here.
Nothing ordered the bridge behind Synapse. Synapse itself depends on the mautrix
bridges, but matrix-chatgpt-bot is not one of them, so it was free to start
first and spend its login quota on a homeserver that was not listening. It now
waits for `service_healthy`.
The specs could not read their own configuration. templates/playwright.env.j2
renders MATRIX_PLUGINS_JSON through dotenv_quote, and both
test-litellm-chatgpt.js and test-bridge-roster.js called JSON.parse on the raw
value. The parse returned a string rather than an object, so `plugins.chatgpt`
was undefined: the litellm spec aborted after 0.99s claiming the plugin was off,
and the roster spec quietly downgraded itself to a skip. Both decode through
decodeDotenvQuotedValue first.
00_core.yml verifies the shared gateway contract before dispatching to either
flavor, so an empty virtual key or a base URL that resolves outside the
deployment fails ahead of any container work.
Verified: the decode fix by execution -- the spec previously died at 0.99s on
plugins.chatgpt and now runs through to the invite poll at line 202. The
contract assertions and the credential ran ok in a variant-2 deploy, which also
exercises the schema change end to end.
Not verified: the local URL and the health gate have not deployed yet, so
whether they break the crash loop is still open. Behind them sits a separate
defect this commit does not address -- matrix-synapse's process does not trust
the platform CA (openresty logs TLS alert 48 from its address) and
RegularPolicyForHTTPS caches one SSL context for the process lifetime, which
breaks every outbound HTTPS call Synapse makes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mount pipeline was split across two layers: compose_volumes and
container_volumes lived as filter modules that ansible's filter loader
rejected on every start ("has no attribute 'FilterModule'", two warnings
per run), while the same-named lookups wrapped them thinly. The filter
bodies now live inside their lookups (the tor_socks precedent), the
filter files are gone, and the render tests moved next to the lookup
tests.
As lookups they see the templating context, so the application_id term
every call site passed back to its own role is gone too: both lookups
read application_id from context and fail loudly when it is missing.
107 templates and nfs_prep.yml drop the term; the calls collapse to
lookup('compose_volumes') and lookup('container_volumes', service_name).
The container_volumes call is now canonical - four leading spaces,
service_name in the service position, | indent(4) - enforced by a new
lint (test_container_volumes_call_shape). Normalizing the 85 literal
call sites surfaced that erpnext keyed its meta/volumes.yml mounts by
container name instead of compose service key, the only role to do so;
its meta now uses the service keys. wordpress's column-0 call, which
rendered its first line at top level, is indented like every other.
Template logic that only restated meta/volumes.yml is deleted: mailu fed
compose_volumes an extra_volumes dict whose eight names were looked up
from the very meta file compose_volumes reads itself; zammad's init and
railsserver extra mounts were the volumes.yml entries minus the two
services, which the meta now lists (the when-gated OIDC binds included).
Conditionality that meta cannot express stays in code, but in role-local
lookup plugins instead of template loops: matrix_bridge_mounts renders
the per-bridge registration mounts, nextcloud_extra_volumes the
feature-gated whiteboard/recording volumes (type: volume entries are
declared unconditionally by compose_volumes, so gating must happen
before it).
Parity is proven, not assumed: every mechanically rewritten template is
byte-identical to HEAD after the transformation regexes (95/95), and the
semantic roles render identically through the real functions against
HEAD's meta - zammad across six sso/tls polarities, all seven erpnext
services, mailu's top level. make test-lint passes complete.
Also: suitecrm's compose template pins container_extra_hosts with
host_alias (the SOCKS proxy URL host.docker.internal never resolved in
compose+tor), and the five disabled mautrix bridge addons use the
empty-default config lookup so templating the applications dict stops
emitting 125 errors per batch in every CI job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A container without a healthcheck is accepted the moment its process starts. The restore drill asks the host's health service for its verdict, so a service that came back empty passes as long as it is running - which is how an empty volume survives a drill unnoticed.
Twenty-one services get a probe and thirteen get a marker naming why none is possible: a one-shot client that exits, a build image with no service port, a media bridge on UDP. The flavor comes from each image's own layer history, read from the registry through the mirror, so no image had to be pulled to find out whether it ships curl, wget or a shell.
Two entries carry more weight than the rest. The central Postgres, which every replay of the recover chain lands in, had no probe at all; it gets the pg_isready line that sys-svc-rdbms already runs for the per-app engine. And the matrix runner starts systemd within a second while its inner daemon is still pulling every image of the deployment, so it counted as healthy long before anything served - it now waits for Synapse to answer.
No probe replaces one: nineteen of the twenty-one images ship no HEALTHCHECK, and the two that do already carried a declaration before this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/volumes.yml says where a volume lands inside its container. Wherever a
second file spelled the same path out, the two drifted independently: the mount
moves, the consumer keeps pointing at the old place, and nothing fails until the
process looks for content that is not there. e031b7f61 fixed exactly this for
web-app-moodle; this generalises it and makes the rule hold.
The lookup gains what a caller needs. `targets` maps each mounting service to
its container path, and `target` is the one path an entry mounts at -- present
only when every mount agrees. An entry that lands at different paths per service
omits `target` rather than picking one, and the caller names the service:
"{{ lookup('volume', application_id, 'prometheus_config').target }}"
"{{ lookup('volume', 'web-app-prometheus', 'host_sys').targets['node-exporter'] }}"
66 literals across 48 role files now derive. Replacement only happened where the
path stood alone -- /srv/app inside /srv/app/data is a different path -- and
every rewritten file was re-parsed, Jinja and YAML alike. Three sites the tool
refused rather than guess were done by hand: zammad's path sat in a Jinja list
literal, seaweedfs' inside a quoted string feeding a filter, and openresty's
/var/run/docker.sock is not a mount target at all but the socket on the host,
which only happens to spell the same string. That one carries the single
nocheck.
tests/lint/repository/spot/test_volumes_path_single_spot.py keeps it that way.
roles/*/files/ is out of scope by construction: it is copied verbatim into an
image and executed there, so a Dockerfile or entrypoint has no Ansible to
evaluate a lookup with.
One defect this found, and it was in the tooling rather than the roles: deciding
ambiguity AFTER filtering short paths made web-app-prometheus' host_sys look
single-valued -- it mounts /sys for cadvisor and /host/sys for node-exporter,
and /sys is below the length filter. The template would have rendered an
undefined `.target`. Both the converter and the rule now decide ambiguity over
every target and filter afterwards, and that call reads targets['node-exporter'].
Verified by resolving rather than by reading: all 70 accesses in the diff were
run against the real registry and every one yields a non-empty path equal to the
literal it replaced. make test-lint: 328 passed.
Not verified: no Ansible render, so a call site whose surrounding context cannot
build the applications registry would still fail at deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extra_hosts is the one compose key a service may not carry twice. The onion SSO
back-channel pin was emitted by a Jinja partial that six roles included, while
six other services wrote the key by hand; nothing stopped a service from
receiving both. YAML then either rejects the document or keeps one list and
drops the other, silently.
The partial becomes plugins/lookup/container_extra_hosts.py, and every pin now
goes through it. Callers add their own entries with extra_hosts=[...], the way
container_volumes already takes extra_volumes=, and the lookup merges them after
the automatic pin and deduplicates in first-seen order. With nothing to emit it
returns an empty string.
The swarm address reasoning from 7b1839681 is preserved and now written down in
the plugin rather than only in that commit: the pin must NOT be host-gateway in
swarm, because that alias resolves per node while openresty publishes :80 with
mode: host under node.role == manager. It carries the stack host's address,
which reaches every task because the compose file is rendered once with the
manager as inventory host. The mode is read as compose_mode_force or
DEPLOYMENT_MODE, not the bare cluster mode, so a role with an override does not
lose its pin and its overlay at once.
Two roles kept a pin that now collides with the automatic one, and both are
resolved the same way -- the hand-written entry stays for clearnet and yields
the .onion case to the lookup, which picks the address per mode. In
web-app-mailu the three conditional entries move into a role-local
mailu_extra_hosts lookup exposed as MAILU_EXTRA_HOSTS, because an eight-line
conditional expression in the template argument is worse than the block it
replaced. It returns its entries as separate results, so vars/main.yml uses
query(); plain lookup() would join them into one comma-separated string.
tests/lint/ansible/jinja/test_compose_extra_hosts_use_lookup.py fails on any
literal extra_hosts: key in a role template, so the invariant survives the next
author who needs a pin. It follows test_compose_ports_use_container_ports_lookup:
the raw construct is forbidden, the lookup is not mandated per service -- a
service with no pins is correct without it, and the swarm branch raises when
facts are absent, which would turn a no-op into a failure surface for containers
that never speak OIDC.
Equivalence checked by rendering rather than by reading: under the trim_blocks +
lstrip_blocks the stack-host action forces for compose.yml.j2, both former call
site shapes (indented, and matrix's column 0) emit byte-identical text to the
new form when the block applies. The no-op case differs -- a whitespace-only
line where the include emitted none -- and yaml.safe_load returns the same
document for both.
31 unit tests; make test-lint 325 passed. Does not touch web-app-espocrm, whose
missing pin is a separate claim and gets its own commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml became meta/secrets.yml, but its content still landed at
applications.<app>.credentials — the one meta file whose payload did not sit
under a root named after the file. Every other topic obeys that rule:
meta/volumes.yml goes to volumes, meta/rbac.yml to rbac. Credentials now go to
secrets.credentials.
The path exists exactly once, in utils/manager/credential_key.py, and
OVERRIDE_SECTION is derived from its two segments rather than spelled beside
them, so the dotted form used by --set and the nested form used by the
inventory cannot drift. Six consumers read it: the loader, the generator, the
CLI override parser, the two emit helpers, and the four lookup validators.
--set now accepts exactly one key shape,
applications.<app>.secrets.credentials.<key>. The three short forms are
rejected instead of guessed at: the same short key names different credentials
on different apps. That also retired primary_app_id, which only existed to
disambiguate them.
Three defects surfaced while wiring this up. _apply_one_role_schema still wrote
to the old root, so an app's own credentials would have moved while its
providers' stayed behind. web-app-pgadmin reads through
lookup('applications', ...).credentials.<key>, an access path no regex over
lookup('config', ...) could see, and would have deployed with an empty postgres
password. And _META_NON_GENERIC_TOPICS still listed 'schema' — a bare topic
name, invisible to any path search — so the generic file-root loop assigned the
raw schema under secrets and the follow-up write landed inside the cached
document, truncating keycloak's four credentials to one for every later
reader.
tests/integration/ansible/lookups/config/test_credentials_roundtrip.py pins both
directions of the seam and the cache invariant that last defect broke.
Existing inventories still carry the pre-rename root; migrating them is
deliberately out of scope here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven include templates under sys-svc-container decided how a container was
probed, and they read their inputs from template-local `{% set %}` scope.
That scope leaks: a value set for one service silently reached every service
below it, which is how taiga's front and gateway ended up probing a port
declared for a different service, and how bluesky's frontend probed a PDS
endpoint.
Probes now live in utils/docker/healthcheck.py, one class per flavor behind a
PROBES registry, and each service declares its own:
healthcheck:
flavor: curl
path: health/ready
start_period: 20m
`port_key` picks another entry under ports.internal for the services that
expose their probe elsewhere (keycloak on management, nextcloud talk on
signaling). Without a flavor the entry must carry an explicit test argv, so a
missing declaration fails loudly instead of rendering a probe against nothing.
Rendering HEAD's templates against the new output confirms 49 of 56 call
sites are byte-identical; the remaining seven are covered by an Ansible run.
Three probes change on purpose: erpnext's backend gains a check it never had,
bluesky's frontend drops the leaked PDS path, and msmtp_curl moves from a
hardcoded localhost:80 to the service's own port on 127.0.0.1 like every
other flavor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The OIDC --add-host derived the gateway by string surgery on the subnet,
replacing a trailing .0/<prefix> with .1. That is correct for every subnet this
repo declares, so this changes no behaviour: checked old against new across all
98 role subnets, zero differences.
What it buys is that the derivation is now reachable from tests, that an
unusable subnet fails at render time with the role and the value named instead
of reaching Docker as a malformed address, and that a subnet not ending in .0
gets its real gateway rather than being passed through unchanged - a /25 now
yields .129 instead of an invalid IP.
Not exercised: the --add-host path runs only in a matrix deploy with OIDC
enabled, which this change was not run against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
On a fresh swarm cluster the bridges wrote registration.yaml as uid 1337, synapse (991) crash-looped on PermissionError, and the stack converge gate sits before the ansible chmod tasks, so the deploy could never turn green; swarm also ignores the service_healthy depends_on that serializes this in compose mode. The mautrix entrypoint chowns /data to $UID and drops privileges via su-exec, so pinning the bridge UID/GID to the synapse uid makes every registration owner-readable the moment it exists. The synapse uid moves into vars as the single source for the compose env and the volume chown.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three layers of one chain. The CI verifier polled the service matrix_matrix, which no variant creates (the ansible flavor deploys a node-local MDAD runner, the compose flavor names its services matrix_synapse and friends); workload: node-local routes it to the established skip like bigbluebutton. Under swarm the stack deploy is a notified handler, so the registration wait started before any bridge existed and killed the play with the handler still pending; a flush before the wait deploys the stack first, and is a no-op in compose mode. Synapse resolves the OIDC issuer through two docker layers where both 127.0.0.1 and host-gateway point at the wrong loopback; the issuer now maps to the runner bridge's node gateway (local subnet .1), emitted only when OIDC is enabled so a no-auth setup keeps its previous arguments, proven by the full Playwright suite against the live cluster.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Upstream matrix-docker-ansible-deploy (pinned ref: master) renamed all
bridge variables to the matrix_bridge_<name>_ prefix; its config sanity
check now hard-fails on the old matrix_<name>_ names, killing the
mdad-bootstrap service so Synapse never starts and the loopback probe
exhausts its retries.
Rename all 10 emitted bridge _enabled vars to matrix_bridge_* (verified
1:1 against the upstream CHANGELOG rename map), drop the redundant
default(false) (services.matrix.plugins always defines every key), and
drop the dead matrix_mautrix_gitter_enabled (no such role upstream).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ansible/MDAD flavor runs matrix as a node-local compose stack, so its local network declares force_bridge when the flavor is ansible; swarm components (network create, prometheus native metrics) then treat it as node-local bridge instead of demanding an overlay. The bridge-enable vars are renamed to the matrix_bridge_* names the pinned MDAD ref (master) expects; the old unprefixed names were silently ignored. All bridge flags and the flavor are read without defaults since the role config defines every key, so a missing key fails loudly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Synapse on an onion node could never complete SSO: it has no Tor, so the
http://auth.<onion> issuer was unresolvable server-side (discovery/jwks/
token), and Synapse hard-rejects non-https issuers on top.
- runner/hostpin.{sh,service}: pin the onion auth host to the runner's real
default gateway (node OpenResty) from /proc/net/route, Before=docker.service
so nested containers see it from their first request; host-gateway is
unusable in nested DinD (resolves to the inner docker0)
- Synapse bind-mounts the runner's /etc/hosts; token iss stays the onion
issuer the browser uses
- bootstrap.service: PassEnvironment (systemd units do not inherit container
env; MATRIX_MDAD_PLAYBOOK_TAGS only worked via its fallback default)
- vars.yml.j2: skip_verification on onion (Synapse's escape hatch for
issuers without https; Tor provides transport security per RFC 7686)
- 02_render_inventory: drop the MDAD bootstrap marker + bounce the runner on
inventory changes — mode=update reconfigures the nested stack at all now
- MATRIX_SERVER_NAME follows matrix_domain onto the onion so MXIDs built from
it (DM spec, bridge configs) reference users that exist
- specs: shared Firefox onion fixture (16 files), gotoElementApp reload-retry
against Element's Tor-triggered init race, seaweedfs spec gates on the
filer frontend, env uses canonical_url consumer=
Verified in-container: SSO 302 to onion Keycloak, admin+biber OIDC login,
E2EE DM admin<->biber, CSP — all green over Tor.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a lint that rejects hardcoded container_port literals and the
host-published ports.local port; container_port is the port the container
listens on, so it must resolve from
lookup('config', application_id, 'services.<svc>.ports.internal.<name>')
or a constant built that way. Bring 42 roles into compliance: move each
literal into services.<svc>.ports.internal.<name> and reference it via the
lookup, using entity_name/service_name where they equal the service key.
Values are preserved (mattermost/prometheus pointed at ports.local with the
same value as internal). Drop the dead container_port from fider and
prometheus, and the narrative comments the edits pulled into the comment
lint's changed-file scope.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mdad runner bind-mounts a manager-only host path (MATRIX_MDAD_DIR), but the ansible-flavor service template emitted no deploy/placement block, so swarm scheduled it onto workers where the path is absent and every task was Rejected ("bind source path does not exist"). Include deploy.yml.j2 (as the compose-flavor already does) so the role's default_placement: manager renders. Swarm-only; compose unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`docker stack deploy` rejects `cgroup: host` ("Additional property cgroup is not
allowed"), aborting the swarm deploy. Emit it via the compose_only lookup so the
key is dropped in swarm and kept in compose.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The matrix-mdad runner image's compose build.context pointed at the Ansible controller's role tree ({{ application_id | abs_role_path_by_application_id }}/files/flavor/ansible/runner). On swarm the pre-deploy 'compose build' handler runs on the manager (delegate_to STACK_HOST run_once), where roles/web-app-matrix/files/ does not exist, so buildx failed: 'unable to prepare context: path ... not found'. Stage the four runner build-context files onto STACK_HOST via stack_host_copy and point the build context at that host-local instance dir, mirroring svc-db-openldap. sys-stk-backend's flush is deferred so the build runs after the stage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Take the host IP as an optional third element of each term
([service, protocol, host_ip]) instead of an ip= kwarg, and build the whole
ports: block from declared ports.local/public (host) + ports.internal
(container). Convert every compose template across 84 roles, folding the
remaining raw var/numeric/specific-ip mappings (including mailu's per-interface
mail ports bound to MAILU_IP4_PUBLIC). Multi-term calls render one term per line.
Add the compose-ports-must-use-container-ports lint: it forbids any raw
"<host>:<published>:<container>" publish item under a ports: block and requires
every term to declare ports.internal. Remove
tests/lint/roles/test_swarm_internal_ports.py, now subsumed.
Fix a container_depends_on rendering regression. The lookup self-indented
(default indent=4), so the added 4-space placement + | indent(4) triple-indented
the block. Flip the lookup default to column 0 (consistent with
container_networks), make every call use the 4-space + | indent(4) form, and add
a lint enforcing it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Legacy mautrix-python facebook/instagram bridges are EOL and fail at runtime (appservice bot masquerade 403). Replace both with the bridgev2 mautrix-meta addon: config, schema credential, variant-2 wiring, and a profile spec with a positive @metabot probe plus an un-bridged-MXID negative control.
playwright.env.j2 now emits MAUTRIX_<bridge>_ADDON_ENABLED only for bridges that are actually deployed, so each bridge spec runs iff its bridge is up (closes the skip-as-green gap) and is telegram-drop aware.
Verified: compose variant-2 deploy failed=0; @metabot registers on the live Synapse; bridge suite green (meta/signal/whatsapp/slack), telegram skips (no API creds).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two bugs kept every mautrix bridge from working: (1) homeserver.yaml.j2 gated app_service_config_files on '{% if MATRIX_BRIDGES | bool %}', but MATRIX_BRIDGES is a list and Ansible's bool filter on a list is always False, so synapse loaded zero appservices and rejected every bridge's as_token (401). Use '| length > 0'. (2) the legacy facebook/instagram bridges interpolated the DB password into the postgres:// URL unencoded; a password with a reserved char crashed mautrix-python with 'Invalid URL: port can't be converted to integer'. URL-encode it like the bridgev2 bridges. Verified by a clean redeploy that recreated the bridge containers and regenerated registrations from the templates: all 5 bridges + synapse healthy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mounting the not-yet-existing per-bridge registration.yaml into synapse made docker create it as a directory, which crash-looped the bridge ('registration.yaml: is a directory'). Mount the existing bridge directory (mautrix/<bridge> -> /data/mautrix-<bridge>) read-only instead and point app_service_config_files at <dir>/registration.yaml, so the bridge generates the file normally and synapse loads it after the restart.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The bridgev2 config migration carried over the upstream example-config's analytics block, whose url pointed at https://api.segment.io/v1/track. The token is already null (telemetry off), but the live external URL made the external-URL-reachability test fail (HTTP 400 from the segment tracking endpoint). Null the url so no external telemetry endpoint is referenced.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-ups after the bridgev2 config migration, each verified in the live DiD stack: (1) URL-encode the bridge DB password in the postgres uri (the credential contains '/', which the stricter bridgev2 Go URL parser rejected as an invalid port) so signal/whatsapp/slack connect to postgres and generate their registration; (2) drop the telegram bridge from MATRIX_BRIDGES when its Telegram api_id/api_hash credentials are absent (telegram requires them at startup and crash-loops without — unlike the other bridges — blocking synapse); (3) give the bridge healthcheck a 5m start_period so the registration generation has grace before it is marked unhealthy.
(4) Two-phase synapse start: drop the synapse->bridges service_healthy depends_on (a single 'compose up -d' did not respect the start_period and aborted with 'dependency ... unhealthy' before the bridges generated registration) and instead restart synapse after the Ansible 'wait for registration files' step so it loads the freshly written appservice registrations. The bridges generate registration independently of synapse, so synapse no longer needs to block on their health.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The mautrix bridge images (dock.mau.dev/mautrix/<name>:latest) moved to the bridgev2/megabridge architecture, but the role's config templates were still the legacy mautrix-go format. On boot the bridgev2 binary aborts with 'Legacy bridge config detected, but hacky network config migrator is not set', crash-loops, never writes registration.yaml, stays unhealthy, and holds synapse in Created (depends_on service_healthy) — failing the matrix deploy in CI run 28044348006.
Rewrite the 4 cutover bridges' configs to the bridgev2 layout (top-level database/network/matrix/double_puppet/encryption), preserving every templated value: homeserver address/domain, the postgres DB uri with its credential lookup, the appservice id/port/bot, the permissions map, and the raw go-template guards. The default entrypoint still generates registration.yaml, so no compose/registration changes are needed. facebook/instagram/imessage stay on the legacy format (their images are legacy-frozen and deploy healthy). Render + YAML validated; actual bridge boot is verified by CI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
At HEAD the synapse service emitted the mautrix-* bridge dependencies as 6-space keys with no depends_on parent, so they nested under healthcheck: and 'docker compose config' rejected them (services.synapse.healthcheck additional properties 'mautrix-*' not allowed, rc=1) — which made the sys-svc-compose CA-trust handler exit rc=2 and failed the matrix deploy in CI run 28018750880.
Route the bridge gating through the container_depends_on lookup's extra= argument instead of a hand-written depends_on block, so the bridges merge with any db/redis/sso deps into a SINGLE depends_on mapping. Fixes the invalid-compose failure and removes the latent duplicate-depends_on risk a hand-written block plus a non-empty container_depends_on would produce. Render-verified for bridges-present and bridges-empty.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A role's vars/main.yml is auto-loaded into play scope and persists, so proxy
override vars (webserver_*) leaked into the central front-proxy render of OTHER
roles (re-evaluated against the wrong application_id) — silently applying one
role's value elsewhere, or hard-failing when the lazy lookup hit a role lacking
the key (the seaweedfs<->matrix client_max_body_size crash).
- Templates (sys-svc-proxy html/upload/ws_generic, matrix synapse homeserver)
resolve client_max_body_size from the role's own server config
(lookup('config', application_id, 'server.client_max_body_size', '100m')).
- Move every role-level webserver_* out of vars/main.yml into task-scoped
include params (9 roles); they apply only to that role's proxy and can't leak.
- sys-stk-front-proxy resets webserver_* to safe defaults via task-scoped vars
on the render tasks, so a leaked value from any other source is neutralized;
a role's own include params (#20) still override the reset (precedence-tested,
direct + nested).
- Add test_no_leaky_proxy_role_vars lint forbidding webserver_* in vars/main.yml
(suppressable via '# nocheck: leaky-proxy-var').
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reconcile main's object-store feature and pre-swarm deploy tasks with the branch's compose/networks/volumes SPOT and swarm lints.
objstore (seaweedfs/minio): standalone compose + embedded service templates migrated to the container_image/container_volumes/container_networks/compose_volumes/compose_networks/depends_on lookups; meta/volumes.yml on the canonical dict schema with s3.json and nginx.conf as type:config; embedded services made swarm-compatible via compose_only/compose_restart.
deploy tasks (erpnext, opentalk, seaweedfs, shopware, snipe-it, wordpress, yourls): container exec routed through the container_address resolver and gated on IS_STACK_HOST, shell module plus stack_host_template, no chdir or compose-verb forms.
Also removed orphaned vars, restored MAILU_INSTANCE_DIR, freed the seaweedfs subnet (192.168.206.0/24), stripped literal quotes from the nextcloud/bookwyrm env templates, and routed main's objstore compose_volumes tests through dir_var_lib.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Synapse's default rc_login (burst 3, ~0.17/s) is too strict for the SSO e2e suite, where consecutive persona sign-ins drain the burst and leave later sign-ins cycling consent <-> M_LIMIT_EXCEEDED until the test is torn down (Matrix avatar never uploaded). Set matrix_synapse_rc_login with a higher burst via the MDAD ansible flavor vars.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
