045ef095f2 taught the generic redirect vhost a Matrix location and let
web-app-matrix include web-opt-rdr-domains for its server name on its
own. The first put one app's contract into a shared role and made every
redirect source (www, aliases, asset, css) advertise the Matrix
delegation; the second bypassed current_play_redirect_domains and could
overwrite a vhost the server name already had, such as the Synapse one.
Every redirect vhost now serves `/.well-known/` from `<well-known>/<domain>/`
and hands a missing file or the bare directory to `location /` via
`error_page 403 404 = /`, so it still answers 301 with the original
request URI. web-app-matrix only writes its delegation file into the
directory of its server name.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The LiveKit JWT service checks each OpenID token over federation
against the user's homeserver. Server discovery found no delegation:
https://<server_name>/.well-known/matrix/server hit the redirect vhost
of web-opt-rdr-domains, which answered 301 to the dashboard and then
404, so the lookup fell back to <server_name>:8448. In compose OpenResty
runs in the host network and answers there; in swarm it publishes 80
and 443 only, the dial was refused and Element Call's token exchange
returned 401 M_UNAUTHORIZED (main run 40017166046, Nextcloud#12).
The role also wrote the delegation to `<well-known>/matrixserver`,
because the file path joined the directory and `server` without a
separator.
The file now lands at `<well-known>/matrix/server`, and the redirect
vhost serves `/.well-known/matrix/` from the shared well-known
directory, so discovery delegates to the Synapse domain on 443 in both
modes. Both flavors share tasks/utils/well_known.yml, which also
includes web-opt-rdr-domains for the server name when no play-wide
redirect claims it, so the delegation has a vhost there too.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An earlier commit on this branch turned the three off because the ansible flavor ran
them without a published port, proxy entry, domain or certificate. They
now stay on and are wired end to end.
Element Call: MDAD's LiveKit server and JWT service get their own
entities in meta/services.yml with resource limits, local ports and a
public media port (TCP and UDP). The runner container publishes them
through container_ports, and the Synapse vhost proxies /livekit-server/
(websocket) and /livekit-jwt-service/ to them. The JWT service checks
each OpenID token against the user's homeserver over federation, and
federation was not reachable in this flavor: Synapse served it only on
a plain-HTTP listener that the runner published directly on 8448.
Synapse now serves the client and federation resources on one listener,
OpenResty serves the Synapse vhost on 443 and as default_server on the
federation port, the direct 8448 publish is gone, and the role writes
/.well-known/matrix/server as the compose flavor does. With a
self-signed CA the JWT container mounts the CA bundle as Synapse does.
Jitsi: a new services.jitsi entity follows the shared-service contract.
It is enabled in the ansible flavor off tor nodes. When web-app-jitsi is
in the inventory, Element points at that shared Jitsi and MDAD's own
Jitsi stays off. Otherwise MDAD runs its own Jitsi on jitsi.<domain>
with a front-proxy vhost, its web port published to the runner and its
videobridge on a public UDP port. The entity lists the jitsi domain, so
the web health probe skips it whenever Matrix does not serve it. The
jitsi_*_container_network overrides were dropped: MDAD reads none of
them, Jitsi always ran on its own network. Variant 0 covers the shared
case, variant 1 the internal one. On an onion-primary deploy both call
stacks stay off, so Element never advertises a clearnet call surface.
MDAD's Prosody keeps an account's first password, so after a credential
rotation jicofo failed SASL with the new one. The role now derives the
jicofo and videobridge passwords from the macaroon secret and recreates
both accounts on every deploy.
Heisenbridge stays on by default. It runs in the nested host network,
where MDAD's container-name URLs do not resolve, so its homeserver URL
and the appservice URL in its registration point at 127.0.0.1.
A bootstrap that a previous deploy left failed (for example after a
Docker Hub rate limit) is now reset and restarted, so a mode=update
deploy converges instead of failing on the stale unit state.
Playwright proves each path: the RTC spec fetches a LiveKit token from
the JWT service and opens the SFU signalling websocket, the Jitsi spec
checks Element's preferred domain and joins a conference (signing in
through Keycloak first when the shared Jitsi sits behind SSO), and the
Heisenbridge spec opens a DM with the bot and waits for its answer to
"help".
Closes#1165
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The ansible flavor's runner starts its own docker daemon, and that daemon
pulled every MDAD image straight from Docker Hub and ghcr.io, past the
registry cache the stack's own daemon uses. MDAD pulls with
`pull: always`, so every bootstrap spent the shared anonymous Docker Hub
quota; locally a single run exhausted it and died with
"toomanyrequests" even for images already on disk.
The role now reads the proxy of the stack's docker.service. When there
is one, it mounts a docker.service drop-in into the runner with the same
HTTP_PROXY, HTTPS_PROXY and NO_PROXY, and pins the proxy host through
container_extra_hosts. When the stack's daemon trusts the registry
cache's MITM CA, the drop-in fetches that CA from the proxy and runs
update-ca-certificates before the inner daemon starts. Without a proxy
on the stack's daemon nothing changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
b3ce43a596 set no_log on every task of two included files, which
test_include_uniform_apply_keyword rejects: a keyword every included task
repeats belongs in the include's apply block. The Cloudflare development
mode tasks and the ChatGPT bridge token tasks now receive no_log through
apply at their include sites. The matrix include had set no_log on the
include task itself, which never reaches the included tasks.
Two loops over MATRIX_BRIDGES were unmasked: the bridge folder creation
and the readiness gate. A loop result prints the whole item, and every item
carries the bridge's database_password, so both printed it at -v whenever
MASK_CREDENTIALS_IN_LOGS is on. The credential lint cannot see this because
the loop variable's name carries no secret marker.
Verified: test_include_uniform_apply_keyword, the credential masking lint,
the task lints and the matrix unit tests pass, and ansible-lint is clean on
the five files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
At -vvv Ansible prints each module argument and result, so a password in
a command line, a request body, a header or a fact lands in the deploy
log. Nothing enforced masking outside the MCP task files, and no check
looked at parents at all.
A new lint, test_credential_tasks_are_masked, finds tasks whose printed
parts (module arguments, environment, loop) reference an identifier ending
in password, secret, token, api_key or private_key, or
secrets.credentials. A task passes when it or an enclosing block sets
no_log, when every file that pulls it in does so through include_tasks
with apply.no_log or import_tasks with no_log (transitively), when the
secret only reaches a parameter the module masks itself, or when the
module masks its values. no_log on an include_tasks task itself does not
count, because it does not reach the included tasks. The rule can be
suppressed per task with `# nocheck: credential-masking`, now listed in
the suppression catalog, and the lint tests its parent and suppression
logic on synthetic cases.
It found 39 unmasked tasks in 19 roles, all real: join tokens, client
secrets, API tokens in Cloudflare headers, database passwords in
command lines, the ChatGPT bot login body and more. Each now sets
no_log from MASK_CREDENTIALS_IN_LOGS. The sys-dns-hetzner-rdns alias
hetzner_no_log, which only restated that switch, is gone.
Not covered yet: template and copy tasks whose rendered file carries a
secret print it as a diff under --diff.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
304144ce67 probed each bridge's healthcheck at /_matrix/mau/ready. mautrix
sets Ready only after Synapse has pinged the appservice (connector.go
starts the listener, then ensureConnection, then AS.Ready = true), and a
swarm task with a healthcheck stays "starting" and receives no VIP traffic
until it is healthy. The ping could never arrive: Synapse logged 150
ConnectionRefused per bridge, each bridge exited FTL 13, and the stack
never converged (run 34615516009, Matrix#2, job 103327034630).
The healthcheck now probes /_matrix/mau/live, which answers once the
listener is up, before the ping. A deploy task after the Synapse wait asks
every bridge for /_matrix/mau/ready from inside Synapse, over the same
name Synapse pings, so a bridge that never becomes ready still fails the
deploy without gating swarm routing.
The ChatGPT bot is rendered only once its access token exists. Its account
and token are created by stack_host_provisioning after the first stack
deploy, so an earlier bot task failed its login with M_FORBIDDEN until the
converge wait called it stuck. The re-render after provisioning adds it
with a valid token.
Verified in a compose run of variant 2: the readiness task got 200 from
meta, signal, slack and whatsapp on the first attempt. Tests pin the live
probe and the token guard and fail against the previous template.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every mautrix bridge generated its own registration with -g on start,
with fresh random tokens. Synapse had to read that file from a shared
directory, be restarted afterwards, and on swarm the services had to be
removed first so the registrations could be wiped (93735fd533). The
tokens changed on every redeploy and the deploy depended on a wait for
files a container writes.
Each bridge now has as_token and hs_token declared in meta/secrets.yml
and a fixed appservice_port in its addon. The bridge config reads the
tokens through lookup('config'), and registration.yaml.j2 renders the
registration from that config in the upstream -g format, so both sides
are known before anything starts and stay identical across redeploys.
The files reach the containers as secrets: matrix_bridge_mounts offers a
stack view (top-level definitions, content-hash named by compose_volumes),
a synapse view (one registration per bridge under /registrations, mode
0444) and a bridge view (the config at /config/config.yaml). The bridges
run as the synapse uid with an explicit --config and --no-update, keep
their scratch /data on a 16m tmpfs so hlth-volumes finds no anonymous
volume, and depend on a healthy synapse. Their healthcheck is
/_matrix/mau/ready, which the bridge only reports after whoami with the
as_token and an appservice ping carrying the hs_token, so a healthy
bridge proves the token pair.
The compose tasks lose the compose down, the swarm service rm block, the
registration wait, the permission fixes and both synapse restarts; this
reverts the workaround of 93735fd533.
Verified on the local compose stack: four bridges ready, synapse pings
answered, registration and config sha256 unchanged across a redeploy,
test-bridge-roster and 12 matrix Playwright specs green. A token
rotation (new pair in both files, compose up --force-recreate as the
handler runs it) came back healthy; a mismatched pair left the bridge
unhealthy. The swarm-shaped stack file passes docker stack config with
the 0444 secret modes, the 16 MiB tmpfs and content-hash secret names.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every deploy of the compose flavour wipes /opt/compose/matrix/mautrix/ and
re-renders each bridge config with placeholder tokens, so the bridges generate
new registrations on their next start and synapse has to load them. Compose
gets that ordering from the `compose down` right before the wipe. Swarm had no
counterpart: the bridge services kept running on the deleted bind-mount
directories and never regenerated anything, while synapse was restarted by the
flush and found no registration.
Job 102705652096 of run 34421944328 (swarm, variant 2) hits it on the
credential-rotation redeploy, which changes homeserver.yaml, a swarm config,
and therefore restarts synapse every time:
FileNotFoundError: [Errno 2] No such file or directory: '/data/mautrix-meta/registration.yaml'
stuck: matrix_synapse
From 05:44:56 dockerd reported "current working directory is outside of
container mount namespace root" for the running slack and signal bridges, and
mautrix-meta, started at 03:50:50, only initialised again at 05:53:11, when its
healthcheck finally replaced it; stack_ready.sh had given up at its 180s grace
long before.
Before the wipe, swarm mode now removes the synapse and bridge services a
previous deploy left running, once from the stack host, so the flush's
`docker stack deploy` creates them together, exactly as on a first deploy
(which converged in about 32s in the same job). Removing only the bridges would
not be enough: a synapse that is not recreated keeps the old tokens, and the
new bridges would be refused before the synapse restart at the end of the file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
From triage of run 33923190220. Each fix is independently complete.
penpot: ansible templates delegate_to before the task's merged when, and AnsibleUndefinedVariable escapes get_delegated_vars_and_hostname. PENPOT_BACKEND_HOST_NODE is set only by the resolve include, so the bootstrap task hard-failed on every host that include skipped instead of skipping with it. A block-level when cannot prevent this; move the tasks into a dynamic include.
matrix: the MDAD reconfigure looked for volume matrix_matrix_mdad_state (real name matrix_mdad_state) and container matrix-mdad (real name matrix). Both if-bodies were skipped while the task still reported changed, so a rotated S3 secret never reached synapse and every media write got SignatureDoesNotMatch. Move the shell into its own script and pass the real names through the environment. Also add no_log to the vars.yml render, whose diff printed 16 secrets in cleartext.
n8n: when the settings file and N8N_ENCRYPTION_KEY disagree, oclif's findCommand throws for every command, so 'n8n export:credentials' is unreachable in exactly the state that would need it. The re-key is one-way; pin it rotatable: false.
swarm-drill: distros.sh sees only rc=124 and cannot tell a wedged deploy from one the sweep deadline cut short. Say so where the governor caps it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moving the statement into files/sql left it on stdin as a quoted scalar, and
the password-quoting guard strips stdin sub-blocks only when they are block
scalars. A quoted one falls through and is scanned as if it were shell, where
any expression naming a password must carry the quote filter - and the file is
called realign_user_password.sql.
realign_synapse_password.yml:24: In shell tasks, password expressions must
include '| quote': {{ lookup('file', 'sql/realign_user_password.sql') }}
Quoting it would paste literal quotes into the SQL, and renaming the file
would only hide the token from the pattern. A folded scalar is the shape the
guard already understands, and it holds whatever the file is called.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ChatGPT bridge got its homeserver URL from container_address, which
answers "what do I docker exec into". Under swarm that is a shell subshell,
and it was being written into a compose value, so docker stack config refused
the whole stack before anything deployed:
invalid interpolation format for
services.matrix-chatgpt-bot.environment.MATRIX_HOMESERVER_URL
container_service answers the other question, "what is this service
addressable as", and answers it per mode: the declared container name under
compose, <stack>_<key> under swarm. Compose therefore keeps exactly the name
it already used; only the branch that was broken changes.
Escaping the $ would have satisfied the parser and shipped a literal $(...)
into the bridge's environment, and the declared container name is emitted
through compose_only, so under swarm it has no DNS record at all. Neither is
a fix.
Name the resolved value once. The same lookup was already spelled out a
second time for the swarm restart, so both now read MATRIX_SYNAPSE_SERVICE.
Rename MATRIX_MDAD_SERVICE to MATRIX_MDAD_SERVICE_KEY while the vocabulary is
in hand: every one of its uses passes it as a service key, one of them as the
key of a rendered compose service, and _SERVICE now means the resolved form.
The role's four synapse identifiers finally say what they are - the declared
name, the addressable service, the exec target, and the key.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The realign statement was passed through psql -c with :'hash' and :'name'
placeholders. psql expands those only while reading a script; under -c the
string goes to the server untouched, so postgres answered
ERROR: syntax error at or near ":"
LINE 1: UPDATE users SET password_hash = :'hash' WHERE name = :'name...
on every attempt, in both matrix rows of the run. The statement now arrives on
stdin through -f -, where the lexer does substitute, so the values stay
parameters instead of being pasted into SQL.
Move the statement to files/sql and read it with lookup('file', ...), the way
listmonk already provisions its MCP identity. That also makes the script form
structural: the statement now lives where only psql's script lexer reaches it,
so folding it back into -c is no longer a one-character edit.
The exec target was mode-blind as well. lookup('database', ..., 'container')
resolves the same value in every mode, while 'address' is the one that becomes
the resolver subshell under swarm; the shell evaluates it, so the quote filter
goes with it.
This alone does not turn the role green: the swarm rows die earlier, when
docker stack config rejects a compose value carrying the same kind of resolver
subshell. That is a separate defect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
register_new_matrix_user only creates. On PASS 2 of a matrix-deploy round the administrator and the ChatGPT bridge bot already exist carrying their PASS-1 password, the call returns 'User ID already taken', and both tasks tolerate that. chatgpt_bridge_token.yml then logs in with the rotated password against an account that still holds the old one and burns ten retries, which is where job 99670614406 of run 33445831564 ended.
A new realign_synapse_password.yml hashes the declared password with Synapse's own hash_password and writes it into the users table, addressing the engine through lookup('database', application_id, 'container') the way realign_dedicated_db.yml already does. It runs for both accounts and only when the registration found them, so a first deploy never reaches it.
The hash is handed to psql as a variable and used as :'hash', so it never reaches the SQL parser as text, and it is passed through Ansible's quote filter, so the bcrypt dollar segments survive the shell. Verified by running the rendered command against a stub that prints its argv: hash=$2b$12$abc/def$ghi arrives as a single unsplit argument.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bridge container was in a restart loop and never joined a room, so no
Matrix surface could reach the gateway. Four distinct causes, in the order
they surfaced.
MATRIX_ACCESS_TOKEN came from a credential declared with algorithm: plain,
a random string Synapse would never accept, so it resolved empty. With no
token, matrix-chatgpt-bot runs its token-fetch path, prints "Set
MATRIX_ACCESS_TOKEN to above token" and exits after 0.6s. Log the bot in
against Synapse and hand the container the token it actually issued.
compose.yml is written before the containers start, but the bot account only
exists after the Synapse health check further down the play, so the file has
to be rendered a second time once the token exists.
E2EE is disabled. The bridge uploads device keys that Synapse 1.159.0
rejects with `M_INVALID_PARAM: device_keys must not be null`, after which it
backs off forever and never completes a sync. This is a deliberate tradeoff:
bot DMs are no longer end-to-end encrypted, and the alternative is upstream
(a newer bot or an older Synapse). It hangs on MATRIX_CHATGPT_ENCRYPTION and
is one variable to flip back. The spec exercises an unencrypted room anyway,
because an E2EE timeline is opaque to an API-only client.
rc_login and rc_joins get a burst of 30. Synapse's defaults (login burst 3)
are too tight for a bridge that re-authenticates on every deploy, and the
restart loop drove them into M_LIMIT_EXCEEDED and later 429 (rc_joins.local).
This widens the burst allowance while leaving the sustained rate at 0.17/s.
The stack-host work moves into utils/stack_host_provisioning.yml so the
IS_STACK_HOST condition appears once instead of on two consecutive siblings.
Verified live: token present, container up rather than restarting, and
test-litellm-chatgpt.js passing. Not covered: a bot that meets an unjoinable
invite still crashes; 26 stale test rooms had to be cleared by hand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Matrix was the one role that already shipped an AI credential and pointed it at
the vendor, and four separate defects kept it from reaching the gateway. Each
one hid the next, so fixing fewer than all four still leaves a surface that
reports green and answers nothing.
The schema rejected the gateway by construction.
`chatgpt_bridge_openai_api_key` validated `^sk-[a-zA-Z0-9]{40,}$` -- OpenAI's own
key format -- with algorithm `plain`. A LiteLLM virtual key cannot satisfy that
pattern, so no amount of configuration could have worked. It is `sk_prefixed`
with a length check now, and a `litellm_api_key` entry joins it in the same
shape the other eight consumers use.
The bridge reached Synapse over the public ingress. MATRIX_HOMESERVER_URL was
MATRIX_SYNAPSE_URL, an https URL on the canonical domain, so every login left
the container network, crossed openresty and came back. A 502 taken while
Synapse was still starting still consumes an rc_login.address slot, and once
that bucket empties the bridge dies on M_LIMIT_EXCEEDED and restarts into the
same wall: measured at 139 restarts on the local stack. MATRIX_SYNAPSE_LOCAL_URL
addresses the container directly over plain http on the internal port.
MATRIX_SYNAPSE_URL stays public where it has to be -- public_baseurl, Element,
and the mautrix bridges, which are a separate concern and untouched here.
Nothing ordered the bridge behind Synapse. Synapse itself depends on the mautrix
bridges, but matrix-chatgpt-bot is not one of them, so it was free to start
first and spend its login quota on a homeserver that was not listening. It now
waits for `service_healthy`.
The specs could not read their own configuration. templates/playwright.env.j2
renders MATRIX_PLUGINS_JSON through dotenv_quote, and both
test-litellm-chatgpt.js and test-bridge-roster.js called JSON.parse on the raw
value. The parse returned a string rather than an object, so `plugins.chatgpt`
was undefined: the litellm spec aborted after 0.99s claiming the plugin was off,
and the roster spec quietly downgraded itself to a skip. Both decode through
decodeDotenvQuotedValue first.
00_core.yml verifies the shared gateway contract before dispatching to either
flavor, so an empty virtual key or a base URL that resolves outside the
deployment fails ahead of any container work.
Verified: the decode fix by execution -- the spec previously died at 0.99s on
plugins.chatgpt and now runs through to the invite poll at line 202. The
contract assertions and the credential ran ok in a variant-2 deploy, which also
exercises the schema change end to end.
Not verified: the local URL and the health gate have not deployed yet, so
whether they break the crash loop is still open. Behind them sits a separate
defect this commit does not address -- matrix-synapse's process does not trust
the platform CA (openresty logs TLS alert 48 from its address) and
RegularPolicyForHTTPS caches one SSL context for the process lifetime, which
breaks every outbound HTTPS call Synapse makes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A handler name was doing two jobs at once: the line an operator reads in the
play output, and the identifier a `notify:` binds to. The two pull in opposite
directions -- the log wants "restart the SSH service, with diagnostics", the
binding wants a token that survives quoting, YAML flow style and a shell
round-trip -- and a name that changes for readability silently changes which
handler runs.
Every handler that a notify reached by name now carries a `listen:` with that
exact wording, so the binding is explicit before anything is renamed. The
topics themselves become identifiers: `compose up` reads `compose-up`,
`Restart NFS ganesha` reads `restart-nfs-ganesha`, 22 of them across both
sides of every notify. Where the topic is assembled at runtime, both branches
moved together: the conditional in sys-service carries two literals, and the
LDAP renderer interpolates `import-{{ folder }}-ldif-files`, which is what
makes those handlers reachable at all.
Freed from the binding, the names take the emoji prefix every other task line
already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/volumes.yml says where a volume lands inside its container. Wherever a
second file spelled the same path out, the two drifted independently: the mount
moves, the consumer keeps pointing at the old place, and nothing fails until the
process looks for content that is not there. e031b7f61 fixed exactly this for
web-app-moodle; this generalises it and makes the rule hold.
The lookup gains what a caller needs. `targets` maps each mounting service to
its container path, and `target` is the one path an entry mounts at -- present
only when every mount agrees. An entry that lands at different paths per service
omits `target` rather than picking one, and the caller names the service:
"{{ lookup('volume', application_id, 'prometheus_config').target }}"
"{{ lookup('volume', 'web-app-prometheus', 'host_sys').targets['node-exporter'] }}"
66 literals across 48 role files now derive. Replacement only happened where the
path stood alone -- /srv/app inside /srv/app/data is a different path -- and
every rewritten file was re-parsed, Jinja and YAML alike. Three sites the tool
refused rather than guess were done by hand: zammad's path sat in a Jinja list
literal, seaweedfs' inside a quoted string feeding a filter, and openresty's
/var/run/docker.sock is not a mount target at all but the socket on the host,
which only happens to spell the same string. That one carries the single
nocheck.
tests/lint/repository/spot/test_volumes_path_single_spot.py keeps it that way.
roles/*/files/ is out of scope by construction: it is copied verbatim into an
image and executed there, so a Dockerfile or entrypoint has no Ansible to
evaluate a lookup with.
One defect this found, and it was in the tooling rather than the roles: deciding
ambiguity AFTER filtering short paths made web-app-prometheus' host_sys look
single-valued -- it mounts /sys for cadvisor and /host/sys for node-exporter,
and /sys is below the length filter. The template would have rendered an
undefined `.target`. Both the converter and the rule now decide ambiguity over
every target and filter afterwards, and that call reads targets['node-exporter'].
Verified by resolving rather than by reading: all 70 accesses in the diff were
run against the real registry and every one yields a non-empty path equal to the
literal it replaced. make test-lint: 328 passed.
Not verified: no Ansible render, so a call site whose surrounding context cannot
build the applications registry would still fail at deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml became meta/secrets.yml, but its content still landed at
applications.<app>.credentials — the one meta file whose payload did not sit
under a root named after the file. Every other topic obeys that rule:
meta/volumes.yml goes to volumes, meta/rbac.yml to rbac. Credentials now go to
secrets.credentials.
The path exists exactly once, in utils/manager/credential_key.py, and
OVERRIDE_SECTION is derived from its two segments rather than spelled beside
them, so the dotted form used by --set and the nested form used by the
inventory cannot drift. Six consumers read it: the loader, the generator, the
CLI override parser, the two emit helpers, and the four lookup validators.
--set now accepts exactly one key shape,
applications.<app>.secrets.credentials.<key>. The three short forms are
rejected instead of guessed at: the same short key names different credentials
on different apps. That also retired primary_app_id, which only existed to
disambiguate them.
Three defects surfaced while wiring this up. _apply_one_role_schema still wrote
to the old root, so an app's own credentials would have moved while its
providers' stayed behind. web-app-pgadmin reads through
lookup('applications', ...).credentials.<key>, an access path no regex over
lookup('config', ...) could see, and would have deployed with an empty postgres
password. And _META_NON_GENERIC_TOPICS still listed 'schema' — a bare topic
name, invisible to any path search — so the generic file-root loop assigned the
raw schema under secrets and the follow-up write landed inside the cached
document, truncating keycloak's four credentials to one for every later
reader.
tests/integration/ansible/lookups/config/test_credentials_roundtrip.py pins both
directions of the seam and the cache invariant that last defect broke.
Existing inventories still carry the pre-rename root; migrating them is
deliberately out of scope here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The staging task listed three files by hand while the runner Dockerfile
COPYs five. 0a581f662 added hostpin.sh and hostpin.service together with
their two COPY lines and did not extend the list, so the build failed
deterministically:
#18 ERROR: failed to calculate checksum of ref ...: "/hostpin.sh": not found
failed to solve: failed to compute cache key
and compose pull retried it ten times -- a missing file no retry can heal.
Both files and the defect are branch-local: neither exists on origin/main.
Copy the directory instead of enumerating it. The per-file modes the list
carried were fiction anyway: every source is 100644 in git, including
bootstrap.sh which the list published as 0755, and the Dockerfile chmods
0755 on both scripts itself, so the staged mode never reached the image.
The point is not the two missing entries. A list that must be updated
whenever a file is added is silent exactly when it is forgotten -- here it
went wrong inside a single commit that touched the Dockerfile and the
directory at the same time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ActiveState=failed` only says the oneshot exited non-zero, and the unit runs
two phases: `ansible-galaxy install -r requirements.yml --force` and then
`ansible-playbook setup.yml` (bootstrap.sh:39-44, under set -euo pipefail).
Claiming the playbook failed sends the reader looking for an inner PLAY RECAP.
In run 30781723102 the web-app-nextcloud 3 job died in the first phase, at role
21 of 31, on a git clone that got HTTP 503. The container journal logs
"starting ansible-galaxy install" and never reaches "galaxy install done" or
"starting ansible-playbook setup.yml", so there was no recap to find.
Wording only; the branch, its condition and the diagnostics are unchanged. The
cause of that 503 is still open.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repo's async convention is fire and forget: ASYNC_POLL is 0 whenever
ASYNC_ENABLED is true, and the swarm test lane forces it true through
extra vars. A task dispatched that way returns a stub, so a changed_when
or failed_when reading its own register evaluates against nothing and can
never report changed and never fail. Several tasks carried a
"not ASYNC_ENABLED and" prefix in front of those conditions, added to
avoid touching undefined rc and stdout, which turned the guard off
entirely rather than adapting it.
That is how the nextcloud plugin configuration stayed broken unseen: the
batch reported ok after 0.8s while its delegated commands kept running,
the next task read the half written configuration, and no failure in any
of them could surface.
Every async in the repo was decided individually rather than swept. Where
nothing consumes the result, the async stays and a one line rationale now
records what does not read it. Where a later task, a handler or a browser
does consume it, either the async goes or the result is collected.
Collecting is what the new reaper does. It waits for a whole batch, then
reports every broken job at once instead of aborting on the first, so one
run shows whether a single item or the whole set is broken. Deciding what
counts as broken lives in a filter plugin, because the distinction is
subtle enough to deserve tests: a module reports failure in msg, a command
in rc and stdout, an idempotent API answers "already exists" and means
success, and a job that cannot be read at all comes back marked finished
with no outcome, which would otherwise read as green.
Timing cost is roughly one to two minutes on a deploy measured in hours.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_task_file_numbering.py: numbered task
files use exactly two leading digits, a numbered *_core.yml must be
00_core.yml, per directory the numbers are unique/consecutive and start
at 01 (00 with a core file), numbered includes appear in ascending
order, and shared helpers live in tasks/utils/ instead of the numbered
chain.
Apply it across every role: rename NN_core.yml to 00_core.yml with a
consecutive down-shift of the siblings, move letter-suffix children into
a subdirectory named after their parent task, relocate helpers included
from more than one place into tasks/utils/, and update every reference
(include_tasks, path_absolute, .ansible-lint-ignore, requirement docs).
Legacy narration comments in the touched files are deleted; the upstream
xwiki.properties reference block opts out via nocheck. The unused
MAILU_IP6_PUBLIC var (only ever named in now-removed comments) is dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolves the runner Dockerfile to the lint-standard Acquire::Retries=3
and keeps the collabora TODO wording; hardens the merged tree for the
stricter lints (apt retries in the tor Dockerfile, shell-or-true
markers, comment cleanup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The flavor:compose `synapse.conf` template builds its proxy target from proxy_service_key + proxy_internal_port, which were never supplied, so the synapse homeserver domain proxied to an empty upstream (variant 2 only). Supply synapse + MATRIX_SYNAPSE_CONTAINER_PORT, matching the sibling front-proxy include.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The compose-flavor front-proxy wiring gave the element and synapse vhosts only their local port, so in swarm mode resolve_upstream fell back to services.matrix.ports.internal.http (the ansible-flavor MDAD service has no ports) and aborted. It now passes each vhost its own service key and internal port, mirroring the ansible flavor, so the proxy resolves matrix_element and matrix_synapse on the overlay; MATRIX_ELEMENT_CONTAINER_PORT is added as its source.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On a fresh swarm cluster the bridges wrote registration.yaml as uid 1337, synapse (991) crash-looped on PermissionError, and the stack converge gate sits before the ansible chmod tasks, so the deploy could never turn green; swarm also ignores the service_healthy depends_on that serializes this in compose mode. The mautrix entrypoint chowns /data to $UID and drops privileges via su-exec, so pinning the bridge UID/GID to the synapse uid makes every registration owner-readable the moment it exists. The synapse uid moves into vars as the single source for the compose env and the volume chown.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The bridges write their registration.yaml into directories they create with a restrictive mode, and the role only chmodded the file; synapse, running as its unprivileged user with the directory bind-mounted read-only, crash-looped on PermissionError and the stack never converged. The directories now get mode 0755 before the file chmod and the synapse restart.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three layers of one chain. The CI verifier polled the service matrix_matrix, which no variant creates (the ansible flavor deploys a node-local MDAD runner, the compose flavor names its services matrix_synapse and friends); workload: node-local routes it to the established skip like bigbluebutton. Under swarm the stack deploy is a notified handler, so the registration wait started before any bridge existed and killed the play with the handler still pending; a flush before the wait deploys the stack first, and is a no-op in compose mode. Synapse resolves the OIDC issuer through two docker layers where both 127.0.0.1 and host-gateway point at the wrong loopback; the issuer now maps to the runner bridge's node gateway (local subnet .1), emitted only when OIDC is enabled so a no-auth setup keeps its previous arguments, proven by the full Playwright suite against the live cluster.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The seed include re-passed the caller's loop vars as database_name/username/password: '{{ database_name }}' etc., shadowing each var with a template that resolves to itself; ansible-core 2.19 detects the recursion and dies on first access inside the seed assert. include_tasks inherits the enclosing scope, so the bridge loop's vars flow through without the re-pass; the seed's assert still sees them defined and non-empty.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Since the inner mdad dockerd actually starts (storage-driver fix), its
default docker0 claims 172.17.0.1 inside the container netns and
shadows exactly the resolver address the swarm inventory hands to every
container via networks.internet.dns, so the bootstrap's ansible-galaxy
died on 'Could not resolve host: github.com' one second after boot.
Declare the nested-docker bridge ranges as a network SPOT
(NETWORK_NESTED_DOCKER_BIP / _ADDRESS_POOLS, deliberately outside
172.16.0.0/12 and away from the 192.168.244.0/24 lab net) and render
the runner's daemon.json from it at staging time instead of shipping a
static file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The mdad runner pinned storage-driver overlay2, but on swarm test nodes
the container's /var/lib/docker sits on overlayfs and the kernel rejects
overlay-on-overlay upperdirs, so the inner dockerd exited, the bootstrap
oneshot failed by dependency and Synapse never came up (proven by the
rescue-diagnostics journal of run 29356373746). Drop the pin so dockerd
auto-detects and ship fuse-overlayfs as the nested fallback; real hosts
keep overlay2.
Replace the 30min blind HTTP poll with a bootstrap terminal-state wait:
fail fast with diagnostics when the inner docker daemon or the bootstrap
playbook fails, probe Synapse only after the oneshot completed, and run
the journal dump on every failure path instead of behind MODE_DEBUG.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- matrix 02_render_inventory: the mdad-state volume does not exist yet on a
fresh install, so `container volume inspect` errored the reconfigure task
(`no such volume`) on the row-1 baseline. Guard the marker drop behind the
inspect succeeding; the runner bounce still runs if the container is up.
- test_coverage: read services.yml via utils.cache.files.read_text so the
no-uncached-file-reads lint passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- _shared.js: wrap the boot-poll delay in resolveTimeout (raw-timeout lint)
- 02_render_inventory: docker -> container wrapper (engine-abstraction lint)
- seaweedfs: api becomes the primary canonical so the deploy readiness
probe hits a served vhost when the frontend (filer/master) is disabled
- seaweedfs frontend: exempt from variant pinning — the onion gate is a
hard invariant, no variant may force the frontend on a tor node
- test_coverage: add the `# nocheck: variants-coverage` exempt (same
mechanism as bond-baseline) for such invariant flags
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Synapse on an onion node could never complete SSO: it has no Tor, so the
http://auth.<onion> issuer was unresolvable server-side (discovery/jwks/
token), and Synapse hard-rejects non-https issuers on top.
- runner/hostpin.{sh,service}: pin the onion auth host to the runner's real
default gateway (node OpenResty) from /proc/net/route, Before=docker.service
so nested containers see it from their first request; host-gateway is
unusable in nested DinD (resolves to the inner docker0)
- Synapse bind-mounts the runner's /etc/hosts; token iss stays the onion
issuer the browser uses
- bootstrap.service: PassEnvironment (systemd units do not inherit container
env; MATRIX_MDAD_PLAYBOOK_TAGS only worked via its fallback default)
- vars.yml.j2: skip_verification on onion (Synapse's escape hatch for
issuers without https; Tor provides transport security per RFC 7686)
- 02_render_inventory: drop the MDAD bootstrap marker + bounce the runner on
inventory changes — mode=update reconfigures the nested stack at all now
- MATRIX_SERVER_NAME follows matrix_domain onto the onion so MXIDs built from
it (DM spec, bridge configs) reference users that exist
- specs: shared Firefox onion fixture (16 files), gotoElementApp reload-retry
against Element's Tor-triggered init race, seaweedfs spec gates on the
filer frontend, env uses canonical_url consumer=
Verified in-container: SSO 302 to onion Keycloak, admin+biber OIDC login,
E2EE DM admin<->biber, CSP — all green over Tor.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the role, its test tree, the primary entity key
(volume-2-local), the derived vars (BKP_VOLUME_2_LOCAL_*,
SYS_SERVICE_BACKUP_VOLUME_2_LOCAL), the run-once fact, and every
group_names consumer flag, include path, doc, and lint reference. The
container_backup service key and the on-disk backup-docker-to-local
tree keep their names: both are contracts with existing consumer
entries, snapshot trees, and the baudolo pull tooling.
Operators must rename the svc-bkp-container-2-local inventory group and
disable the previous versioned units on already-provisioned hosts; the
pre-deploy snapshot glob derives its base from the new role name and
will not find them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In swarm mode the container_address lookup returns a $(/usr/bin/resolve-container-id ...) fragment that must expand at exec time; piping it through Jinja quote hands docker the literal string and every exec fails with 'No such container' (broke all three snipe-it CI variants). Remove the quote at the seven snipe-it sites and the live matrix/openproject sites, convert openproject's argv exec to shell so the fragment can expand, split the matrix bridge restart per deployment mode with a manager gate, and mark the compose-only or eval-contract sites with nocheck. A new lint (container-address-quote) bans the pattern repo-wide.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a 'timeout' lookup plugin that returns round(base * TIMEOUT_FACTOR)
(group_vars default 1, so value-identical to the old literals). One knob
in group_vars stretches every wait in the stack for a slow uplink:
TIMEOUT_FACTOR: 2 doubles them all.
- plugins/lookup/timeout.py + unit tests (identity, factor scaling,
string coercion, error paths).
- group_vars/all/17_resource.yml: TIMEOUT_FACTOR: 1.
- Convert all 34 literal 'timeout:' task keywords / module args across 25
task+handler files to "{{ lookup('timeout', <base>) | int }}".
- tests/lint/repository/yaml/test_timeout_uses_lookup.py enforces the
lookup in every task/handler timeout (opt-out: # nocheck: timeout-lookup).
- swarm.yml pre-deploy build handler: add the missing timeout (same
network-build hang vector as compose.yml's Build compose).
- Drop 13 pre-existing narration comments in the touched files (comment
lint enforces whole-file on any edited file).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The DiD runner ran as a separate compose service matrix-mdad while the app entity is matrix, so the swarm overlay upstream needed a service_key override to reach matrix_matrix-mdad. Name the service, container_name and entity all matrix (MATRIX_MDAD_SERVICE, MATRIX_MDAD_CONTAINER, services.matrix.name) so the overlay DNS is matrix and resolve_upstream returns matrix:8008 by default. Drop the webserver_proxy_service_key override; keep proxy_internal_port (synapse 8008, element 8080).
Also drop the now-dead matrix-mdad meta entry (resource_filter resolves to the entity, so its cpus/mem were never read), flavor-gate the volume mounts (mdad to ansible, synapse/element to compose) so the ansible flavor no longer declares compose-only volumes/configs, and have diagnostics.sh read the container name from the MATRIX_CONTAINER env instead of hardcoding it. Renderer + unit validated: swarm matrix:8008/8080, compose 127.0.0.1:8021/8022.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce plugins/lookup/path_absolute.py as the single source for the
repo-root anchor: lookup('path_absolute', 'roles/...') returns an absolute
path under playbook_dir, joining every term. Replace all hand-built
playbook_dir path constructions across .yml/.yaml and .j2 templates with it:
[playbook_dir, '<rel>'] | path_join -> lookup('path_absolute', '<rel>')
playbook_dir ~ '/<rel>' -> lookup('path_absolute', '<rel>')
{{ playbook_dir }}/<rel> -> lookup('path_absolute', '<rel>')
Unrelated repo_path symbols in cli/meta/roles/applications/resolution and the
single bare discover_playwright_roles(playbook_dir) call are left untouched.
Add tests/lint/ansible/test_playbook_dir_via_lookup.py: fails the build if any
of the three construction forms reappears in .yml/.yaml/.j2, pointing callers
at the lookup. Unit test covers the plugin.
Touching these files pulled their pre-existing narration comments into
test_comments_valid scope; clean them (delete restatement/banners, keep real
trip-wires as Workaround/Upstream/Invariant markers). Drop caution, important
and pitfall from the comment-lint marker vocabulary and remove the comments
that used them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ansible-flavor stages the matrix-mdad runner build context into MATRIX_MDAD_RUNNER_BUILD_DIR via stack_host_copy, but nothing created that directory first, so the copy failed 'Destination directory ... does not exist'. Ensure the dir (delegate_to STACK_HOST, run_once) before the copy loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The matrix-mdad runner image's compose build.context pointed at the Ansible controller's role tree ({{ application_id | abs_role_path_by_application_id }}/files/flavor/ansible/runner). On swarm the pre-deploy 'compose build' handler runs on the manager (delegate_to STACK_HOST run_once), where roles/web-app-matrix/files/ does not exist, so buildx failed: 'unable to prepare context: path ... not found'. Stage the four runner build-context files onto STACK_HOST via stack_host_copy and point the build context at that host-local instance dir, mirroring svc-db-openldap. sys-stk-backend's flush is deferred so the build runs after the stage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DEPLOYMENT_MODE is binary (swarm / compose), so a negated `!= 'swarm'` gate
always means `== 'compose'`. Replace every negated gate across roles/ and
group_vars with the positive form for readability, and add a lint
(tests/lint/ansible/variables/test_no_deployment_mode_negation.py) that forbids
`DEPLOYMENT_MODE !=` in .yml/.yaml/.j2 with a `# nocheck:
deployment-mode-negation` escape.
The compose-verb / chdir / jinja-gate lint guidance messages now suggest the
positive form too; the gate matcher (utils/annotations/task_gate.py) and its
test inputs keep recognising the legacy `!= 'swarm'` so a nocheck-marked
exception is still detected as compose-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-creating an empty registration.yaml made the bridges skip registration generation ('appservice.as_token not configured') and left synapse broken. Reverts to the two-phase state where synapse + element + NC deploy healthy; the bridge registration/directory chicken-and-egg needs proper pre-generation (bridge -g one-off) as a follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
