Four files outside the healthcheck work, all flagged by make quality-high.
The shopware marker sat on the first line of a value the YAML formatter wraps, where the lint never looked; it now sits on the last line, beside the trailing comment the formatter keeps there. The two ci meta tests get ClassVar for a mutable class attribute and unpacking instead of list concatenation.
The keycloak realm-import test asked jinja2 for an Environment without naming autoescape. It stays False - the template renders JSON, and HTML escaping corrupts the values - and the rule is silenced the way 9a9c2f239 already silences it for the dotenv filter test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A container without a healthcheck is accepted the moment its process starts. The restore drill asks the host's health service for its verdict, so a service that came back empty passes as long as it is running - which is how an empty volume survives a drill unnoticed.
Twenty-one services get a probe and thirteen get a marker naming why none is possible: a one-shot client that exits, a build image with no service port, a media bridge on UDP. The flavor comes from each image's own layer history, read from the registry through the mirror, so no image had to be pulled to find out whether it ships curl, wget or a shell.
Two entries carry more weight than the rest. The central Postgres, which every replay of the recover chain lands in, had no probe at all; it gets the pg_isready line that sys-svc-rdbms already runs for the per-app engine. And the matrix runner starts systemd within a second while its inner daemon is still pulling every image of the deployment, so it counted as healthy long before anything served - it now waits for Synapse to answer.
No probe replaces one: nineteen of the twenty-one images ship no HEALTHCHECK, and the two that do already carried a declaration before this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A container without a healthcheck counts as healthy the moment its process starts. That is how a service which came back empty passes a restore drill: the drill asks the host's health service, and the host says yes.
The image side is read from the registry rather than by pulling: the manifest names the config blob, the blob carries the HEALTHCHECK. The answer is three-valued on purpose. A mapping means the image ships a probe, an empty mapping means it definitely ships none, and None means the question could not be answered. Only the empty mapping is evidence; reading None as absent turns one network hiccup into a hundred false findings.
Probes address the GHCR mirror first and fall back upstream only for images the mirror does not carry, and a second test requires mirror plus cache to answer more often than upstream, so the fallback cannot quietly become the rule and spend the anonymous rate limit again.
Detection spans the four places a probe can come from: the services.yml declaration, a literal block in a template, a partial pulled in by include or lookup('template'), and a HEALTHCHECK in a Dockerfile the role builds its own image from. Without the last three the sweep reported nextcloud, every redis sidecar and openldap as gaps they are not.
The opposite direction gets its own lint: a declared healthcheck that no template renders is dead configuration. Fixing it surfaced that the shared resolver silently dropped every service_name call site, because it accepted only quoted literals and a role var resolves to a bare key. And the literal-protocol lint could never be suppressed on a wrapped value, since autoformat moves the marker to the value's last line while the offence is reported on its first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A databases.csv row with database = '*' makes the backup dump a whole instance with pg_dumpall. The recover chain had no path for that file: it sorted cluster dumps out and printed SKIP, so a host whose engine is dumped that way could not be restored at all, and the drill refused to certify it.
Cluster dumps now go back through 'baudolo-restore cluster' with the instance's superuser. Three details decide the shape. A cluster row states '*' where a single-database row names a database, so credentials_of would file every instance under one key and keep the first, hence a lookup keyed by instance. Which applications must be down is stated by the dump's own connect lines and nowhere else, and reading them needs a real identifier parser: a quoted name may hold spaces and psql options precede the target. And the consumers of a cluster sit in the compose project of its engine, not in projects named after each database, so both are checked before anything is dropped.
The version gate moves out. baudolo compares the dump's header against the running server before its own pre-clean, so the copy here was a second implementation of the same rule, free to drift from the one that actually gates the replay. The test asserts the delegation negatively: the replay must not query the engine version itself. The pin follows, and baudolo-restore joins the command assertion.
The docker probing splits into utils/recovery/docker.py, which keeps the replay module under the size limit and gives the two halves one job each. Calls go through the module rather than through imported names, or a test patching the new module would not reach them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill removed each volume and re-created it. Docker refuses to remove a volume any container still references, and a container outside compose is only stopped here, not removed, so discourse's launcher container aborted the run at that point: after the host was already down and part of the volumes were gone. Re-creating a removed volume also drops its driver options, handing an NFS- or bind-backed volume back as a plain local one.
Volumes are emptied in place instead, from inside a throwaway container, and the emptiness is proven in the same call. Nothing has to be re-created, so a launcher container starts again exactly as it was built.
The engine that receives a replay is started by the service name on its own compose label. The previous fixed name existed for no engine on the host: the central Postgres serves project postgres from volume postgres_data, bigbluebutton serves project bigbluebutton from service postgres. Neither is called database, so the drill died at 'up -d database' after the teardown and before any restore.
A cluster dump yields one marker per database it recreates, or a replay that brought back only part of an instance would still pass. The drill runs once, in the sync pass; repeating it in the async pass would only tear the host down twice for the same answer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cosmos generator linked a consumer to its provider only through an
'<role>' in group_names reference or an explicit provides: name. A hard
dependency has neither -- pinning the provider on unconditionally is
exactly what removes the Jinja reference the generator was reading -- so
web-svc-asset's bond to web-svc-file vanished from the diagram the moment
it became binding. Match a service key against the provider's own primary
key from the registry, in both directions.
Services the role turns off drew a solid 1:1 edge, claiming a mandatory
relationship for something that carries nothing at deploy time. They now
render as a red 0..0 edge; edge merging gained a real precedence so a
provider consumed through two keys stays live when only one is off.
Named colour, not hex: a '#' opens an entity code in the mermaid lexer and
turns the whole diagram into a parse error on GitHub.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The asset role copies its files into web-svc-file's webroot and redirects
its own domain onto web-svc-file's, yet declared the provider as
group-conditional. The minimal matrix variant pinned it off, so the round
deployed asset alone: the redirect target had no vhost, openresty rejected
the SNI, and the CSP health check failed the whole deploy after 26 retries.
Making the flags literal takes the key out of the variant matrix, which
variants-static-override requires -- a variant may not override a literal
base flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backup generation stores databases as sql dumps, not as a file tree
(--dump-only-sql), and the recover chain only ever restored trees:
`git grep sql HEAD -- cli/administration/recover utils/recovery` was
empty. An operator running `recover full` after a real loss got their
files back, their databases untouched, and an exit code of 0.
Adds the missing leg as a fifth recoverer, planned right after the
volume steps so the dumps land once the volumes are back. Three
properties make it a restore rather than an attempt:
- It refuses while a consumer of that database runs. `--empty`
pre-cleans the schema in one psql session and replays in the next; a
booting consumer recreates it in between and the dump's own CREATE
TABLE fails. That is what turned CI red with `relation
"ar_internal_metadata" already exists`.
- The engine is read, not guessed: the service key in the consuming
role's meta/services.yml, the same value that fed the backup seed.
The two previous call sites each sniffed it from the container image
and could disagree.
- The dump's own header states which server it came out of, and that is
compared against the running engine before the pre-clean drops
anything. Forward across a major version is the upgrade path and
stays allowed; backward is refused. The version deliberately does not
travel in databases.csv: that file is written at seed time and would
state the version deployed back then, and a generation copied to
another host carries its header along while a csv row stays behind.
Mind the header formats, they are not symmetric - postgres puts
`Dumped from database version` on line 7, while mariadb's line 2 opens
with mariadb-dump's own version and only the tab-separated `Server
version` line names the server. Both are pinned as test fixtures,
captured from postgres:17-alpine and mariadb:11 themselves.
baudolo is untouched: nothing beyond the credentials it already carries
is read from databases.csv, and no upstream format changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Outside compose TOR_CONTAINER_DNS_HOST was empty, so the drop-in rendered no
listen-address and lookup('container_dns') handed containers the clearnet
resolver alone. A swarm stack host ran the egress router without a listener its
containers could reach, and .onion stayed unresolvable for every one of them
while compose resolved it fine. Ownership already governs the listener, the
upstream policy and the resolv.conf pin; the deploy mode only ever stood in for
it. With the mode clause gone both modes render the drop-in compose has been
running in production, and compose keeps its exact behaviour: where matched == 0
the old and the new expression are the same boolean.
In CI run 31927586060 the swarm node dnsmasq logged "ignoring query from
non-local network 172.18.0.16" seconds after a veth appeared and vanished on
docker0, and every container lookup timed out from then on. The health service
retried for 77 minutes against a resolver that no longer answered.
Measured on dnsmasq 2.93 before landing this: a listen-address disables
--local-service even when that flag comes from the init script; without a bind
directive dnsmasq takes the 0.0.0.0 and [::] wildcard pair, which would collide
with a systemd-resolved stub and occupy port 53 on every address; bind-dynamic
and bind-interfaces both hold their per-address listeners across 2000 veth
cycles; and the stock dnsmasq.conf on arch, fedora and centos carries no
bind-interfaces, so the drop-in's bind-dynamic cannot abort startup there.
Two expectations in tests/unit/plugins/lookup/test_container_dns.py still encode
the compose-only asymmetry and fail with this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
container_address resolves a container id for exec and emits a $(...) subshell in swarm; setup/oidc.yml passes the value through shlex.quote, so the substitution never ran and client.php wrote the literal text into token_endpoint and userinfo_endpoint. The deploy stayed green and every admin SSO login failed at the code-to-token exchange. container_service is the maintained lookup for the addressable form and yields keycloak in compose, keycloak_keycloak in swarm.
init.sh no longer waits on the S3 endpoint: c5a149b11 unmounted the config it guarded, while a hard fail there still took web, worker and scheduler with it. The admin-bundle check now runs unconditionally — it was skipped exactly when objstore was on, i.e. in the only variant with SSO, although the bundles are local either way. The two objstore env vars lost their last consumer with it.
no_log now sits on the two tasks that carry the admin password and the OIDC client secret; handing it through include_tasks apply: does not censor them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The egress router scraped /etc/resolv.conf once and persisted whatever it found. Inside a container that is 127.0.0.11, docker's embedded resolver, which answered REFUSED to recursive queries in CI run 31927586060 and left the node with no clearnet DNS. The upstream now comes from networks.internet.dns, the same source lookup('container_dns') already gives the docker daemon, and an assert demands it before the role touches the host.
no-resolv, the clearnet server= lines and the /etc/resolv.conf pin now hang on TOR_DNSMASQ_OWNS_LISTENER. A dnsmasq the role does not own keeps its upstreams and the node keeps its resolver; only the onion routing is added there. Without that split a swarm node would end up with no-resolv, no upstream, and its resolver pinned at itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role's only login coverage was the shared biber flow, and that flow can
only drive OIDC: personas/biber.js authenticates solely when the URL contains
`openid-connect/auth`. SAML redirects to /protocol/saml and the ldap variant
uses SuiteCRM's own form, so neither path could be exercised — across every
persona helper there are 15 hits for `openid-connect` and none for `saml`.
Both specs assert through /session-status rather than by hunting a logout
control in the DOM. That is what the whole SSO change is about — whether
SuiteCRM owns a session or only the edge does — and it does not race an
asynchronously rendering Angular surface, which is exactly where the 2026-08-08
run tore.
Three timing decisions are load-bearing, each from a defect found while
reviewing the first draft:
- The ACS response is awaited explicitly instead of polling the URL.
performKeycloakLogin only checks that page.url() CONTAINS the canonical
domain, and php-saml puts the SP's own URL in RelayState, so Keycloak's login
URL already contains it: a URL wait returns before the assertion is posted.
- The `POST /login` response is awaited explicitly in the ldap spec. SuiteCRM 8
logs in by XHR with no document navigation, so waitForLoadState resolves while
the request is still in flight.
- /session-status is requested exactly once, after the login completed.
SecurityController's unauthenticated branch calls $session->invalidate(),
so a probe fired mid-login destroys the session it reports on. The first draft
did all three wrongly and would have gone red against a WORKING deployment.
The ldap spec clicks #login-button because the login form declares no
(ngSubmit); Enter only worked through HTML implicit submission.
Verified in the production Playwright image against a protocol stub that
replays the choreography (fake IdP with a Keycloak-shaped form, auto-posted
assertion, XHR login): saml.spec.js 2 passed, ldap.spec.js 2 passed. The
counter-runs matter more — against an app that does not redirect to Keycloak the
SAML spec fails with its own message, and against an app with no login form the
ldap spec times out on #login-button, so neither is vacuously green. Also
verified: collection of all 9 tests, the three service-gate combinations, and a
missing credential failing loudly instead of skipping.
playwright.env.j2's persona rationale said biber has no account when SSO is off.
He does — LDAP_AUTO_CREATE mints one on first bind. He is blocked because the
shared flow cannot drive that login, which is what ldap.spec.js now covers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role sat behind the oauth2-proxy, which authenticates at the edge and hands
the app nothing: SuiteCRM kept serving its own Angular login after the Keycloak
round trip, so no session ever existed and the app had no logout control. CI run
31927586060 is the only run in which the role's Playwright spec ever executed,
and it failed exactly there - "no in-app logout control reachable on the current
authenticated surface" at personas/utils/logout.js.
SuiteCRM 8.9.1 ships nbgrp/onelogin-saml-bundle and treats AUTH_TYPE=saml as a
first-class firewall in config/packages/security.php:249, so the app can own its
session without a line of PHP from us. flavor: saml turns the proxy off by
itself - is_proxy_gated is `enabled AND flavor == 'oauth2'` - which also drops
the redis session store and the auth_request clamp in the vhost. The role
declares no allowed_groups, so no edge ACL is lost.
Four settings are not cosmetic; each is a failure reproduced against a live
Keycloak 26.7.1 until the login completed:
- SAML_COMPRESS_REQUESTS: the HTTP-Redirect binding requires DEFLATE, and
Keycloak rejects an uncompressed AuthnRequest with a ZipException.
- SAML_USE_ATTRIBUTE_FRIENDLY_NAME=false: the bundle defaults to matching
attributes by friendlyName, which Keycloak's mappers do not set, so not a
single attribute arrives.
- APP_DEBUG is forced to 0 whenever SAML is on: public/legacy/modules/Users/
User.php:944 reads $_POST['ut'] unguarded while creating the user, and with
debug on Symfony turns that warning into an exception the authenticator
reports as a failed login - the account is created and the FIRST login of
every new user fails anyway.
- SAML_SP_ENTITY_ID must equal the Keycloak client's clientId.
The IdP signing certificate is read from the realm descriptor at boot rather
than templated in, so a key rotation needs no redeploy. It uses the same public
issuer URL the browser is redirected to - one address, not two - with the SOCKS
proxy espocrm established for onion issuers. An unreachable IdP or a descriptor
without a certificate aborts the start; both paths are verified, as is the skip
when the variable is absent on LDAP/native deployments.
Verified end to end locally against Keycloak 26.7.1, MariaDB 12.3 and SuiteCRM
8.9.1 from the upstream tarball: session-status reports active:true for the
Keycloak user, auto-created on the first attempt against an empty users table.
web-app-keycloak/TODO.md listed this role for the trusted-header bridge. Nothing
of it was implemented, and SuiteCRM exposes no header entry point at all
(REMOTE_USER / X-Forwarded-User / X-Auth-Request appear nowhere in core/ or
public/legacy), so it moves to the same exclusion as jitsi: use the native
mechanism the app already ships.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The realm carried exactly one client, the shared OIDC one, and no SAML client
existed anywhere in the repo - `flavor: saml` was in the resolver's enum with
zero consumers. Apps that cannot speak OIDC therefore had no way to
authenticate against Keycloak at all; they could only be fronted by the
oauth2-proxy, which authenticates at the edge and leaves the app itself without
a session.
KEYCLOAK_SAML_APPS collects every deployed app whose services.sso.flavor is
saml. It filters stepwise - value.services, then .sso, then .flavor, each
guarded by `defined` - because Ansible's undefined is a Marker(StrictUndefined,
Tripwire): resolving that chain in one selectattr raises for every app without
an sso block instead of yielding undefined, which would have broken the realm
render for every deployment in the repo.
The include is unconditional and the loop emits a leading comma per client, so
the realm needs no branch on the list being empty. That matters: an Ansible
`>-` value that failed to convert would be a non-empty string and a truthiness
guard would then splice a broken clients array.
The client's shape is not guesswork - every attribute comes from a failure
reproduced against a live Keycloak 26.7.1 until a SuiteCRM SAML login completed:
defaultClientScopes is emptied because the built-in role_list scope emits one
Role attribute per realm role and the SP rejects the assertion with "Found an
Attribute element with duplicated Name"; saml.client.signature is false because
the SP does not sign its AuthnRequests; the username mapper's attribute name
comes from OIDC.ATTRIBUTES.USERNAME rather than a hardcoded claim.
The new test renders the complete realm - every include, the role's own
ldap_groups_filter, StrictUndefined - and parses it as JSON for zero, one and
several SAML apps. The file had no test before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A command substitution inside an argument does not trip set -e, so a failing password generator would have left --password= and Shopware would have prompted for it interactively; on a non-TTY that kills init and with it every service gated on service_completed_successfully. The value is now generated in its own assignment.
privileges.php could not tell a missing user from an already-set flag, since both report zero affected rows. It now verifies the account exists, which is the only place a silently failed admin provisioning can surface: admin_user.sh swallows both of its commands with || true.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mount restored in 574c8954f would have broken every asset URL. plugins/lookup/objstore.py:226 builds public_url as domain/bucket without the per-filesystem root, so public, theme, sitemap and asset all advertise the same URL while their objects live under different roots. shopware.filesystem.asset feeds the administration bundle's script URLs, so the admin would have 404'd on clearnet and over tor alike.
Re-mounting requires objstore to expose a root-aware URL and web-app-shopware to whitelist the S3 origin in meta/csp.yml.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Shopware's in-app logout leaves the browser on Keycloak's end-session page, which asks 'Do you want to log out?' because the request carries no id_token_hint. The spec waited 60s for a password field that page never renders. The shared persona helper already handles this for inAppLogout; it is now exported so the role-local spec can use it too.
Evidence: the error-context of job 95119330653 shows heading "Logging out" and button "Logout" where the login form was expected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AdminOpenAuth clears the admin flag and the ACL roles on every login that no role-assignment rule matched (UserResolver::getUserChangeSet). The Playwright login therefore demoted the provisioned administrator, the admin backend rendered empty, and the next provisioning run answered 403 FRAMEWORK__MISSING_PRIVILEGE_ERROR. The client now carries a rule with userBecomeAdmin, and a separate step re-asserts the flag for accounts an earlier login already demoted, which no console command can do.
discoveryDocumentUrl stays empty because a successful discovery assigns the document over the explicit endpoints and moves the back channel onto the .onion host cURL refuses to resolve.
system:install --basic-setup provisions admin/shopware with a published password; that account is locked on first install. shopware.yaml is mounted again after being rendered but unmounted since the mounts moved into meta/volumes.yml. files/ is split into php/ and shell/ trees.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Variant 1 and 2 pin services.sso.enabled false, so templates/env.j2 skips
the whole OIDC block and with it OC_EXCLUDE_RUN_SERVICES=idp. OpenCloud
then starts its built-in IDP, and lico's bootstrap rejects a non-https iss
unconditionally (bootstrap/bootstrap.go: "invalid iss value, URL must start
with https://"). An onion has no TLS, so OC_URL is http, the single-binary
runtime fatals on every start, the service sits at 0/1, openresty cannot
resolve the upstream and the guest persona gets a 502 instead of a page.
Pin tor.enabled false in those two variants. utils.github.variant.axes
tor_capable reads exactly that literal false and never schedules them
behind the onion; variant 0 keeps the onion and still rotates through both
states, so the SSO-less branch stays covered on clearnet rather than being
dropped entirely.
The sso-variant-tor opt-out is role-level because the lint offers no
per-variant granularity, so it also stops enforcing "an SSO variant must
pin the onion" for this role. That partially reverts e16eb5728 here: its
rule presumes the onion is possible for every variant, which for OpenCloud
without an external IdP it is not.
The variants-coverage marker on the tor key goes with it. It existed only
because tor was true in every variant and the false polarity had nowhere
to live; it lives in variant 1 and 2 now and test_coverage passes without
the suppression.
Not addressed: a tor node deployed without Keycloak still crash-loops, and
nothing fails loudly to say why.
Verified: tests/lint/ansible/roles/meta plus tests/integration/roles/meta
101 passed / 534 subtests, tests/lint/repository/test_comments_valid
passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role runs two database configurations side by side: Doctrine gets everything
bundled in DATABASE_URL including the port, while the legacy installer gets
single options. The port was missing from that list, so the legacy half fell back
to 3306 while Doctrine used the configured port - a split brain that only shows
once the two disagree. LegacyInstallCommand has defined the option all along
(db_port, shortcut Z, VALUE_REQUIRED, not required, default '').
Verified against the real installer in a throwaway image (php:8.2-apache with the
role's extension set, the v8.9.1 tarball, all three composer runs, no frontend
build) and a live MariaDB 12.3:
--help lists -Z, --db_port=DB_PORT
-Z 3306 check-db-connection done, "DB credentials ok",
install completes, public/legacy/config.php carries
'db_port' => '3306'
-Z 3307, same host check-db-connection fails with
SQLSTATE[HY000] [2002] Connection refused
The last run is the one that matters: 3306 is also the built-in default, so only
a port nothing listens on proves the passed value is honoured rather than ignored.
Both runs started from a freshly dropped and recreated schema.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The LDAP bullet promised a config_override.php. A grep over the whole role finds
that filename in no file but that README line - what the role actually renders is
ldap.yaml.j2 into extensions/<software>/config/services/ldap/, and that file maps
LDAP attributes onto SuiteCRM user fields rather than switching authentication on.
AUTH_TYPE=ldap in env.j2 does the switching.
The SSO bullet claimed the role does not implement OIDC login flows, while the
Persona contract opt-outs section 60 lines below explains that biber reaches the
app solely through the oauth2-proxy and therefore has no account at all in the
sso: false variants. Both cannot be true. meta/services.yml declares
sso.flavor: oauth2 with origin.host suitecrm, i.e. a proxy that terminates the
Keycloak OIDC flow in front of the app. The wording dates back to the commit that
replaced the role's own OIDC wiring with that proxy and described the result from
SuiteCRM's point of view instead of the role's.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/volumes.yml mounts ldap.yaml only when services.ldap.enabled, and
compose_volumes drops the whole configs.ldap_config entry when no mount
survives - but the task rendered the file unconditionally, so the ldap-off
variant wrote a host file nothing references. Both sides now share the
condition.
The mechanism was stranded too. render_replicated_templates exists to copy a
rendered file to the swarm peers, and those branches hang off
`not (IS_STACK_HOST | bool)`. Since 1da3c2eda this task file is included only
`when: IS_STACK_HOST | bool`, and include_tasks is dynamic, so for every other
host the file never exists - the peer branches became unreachable. Nothing was
lost by that: type: config renders as configs.<name>.file in both modes, and in
swarm `docker stack deploy` reads it on STACK_HOST and distributes it over raft.
php.ini one task above is the identical case and already used plain
stack_host_template, so this makes the file internally consistent and drops an
include_tasks, a slurp and a loop over a one-element list.
The notify stays: role_templates_expanded defaulted it to compose-up, which is
now spelled out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SuiteCRM 8.9.1's config/packages/doctrine.yaml binds only url: '%env(DATABASE_URL)%'
and hardcodes driver: pdo_mysql; its server_version key is commented out. So
DATABASE_DRIVER and DATABASE_SERVER_VERSION never reached a consumer.
DATABASE_SERVER_VERSION=10.11 was the worse of the two: it read like a
maintained coupling to the database while svc-db-mariadb has moved to 12.3, so
the next reader either believes a stale number or "fixes" a decoration. Doctrine
resolves the server version on connect.
SUITECRM_MAINTENANCE had an intact lookup chain and no destination - the image
is php:8.2-apache, not a Bitnami image with a maintenance flag, and neither the
entrypoint nor anything else in the repo reads it. All three links go together,
otherwise the lookup breaks.
TRUSTED_HOSTS rendered as ^crm\\.example\\.com$: Jinja collapses the four
backslashes to two, and docker passes env_file values through byte for byte, so
the pattern demanded a literal backslash in the Host header and could never
match. It stays (unlike the other two) because TRUSTED_HOSTS is standard Symfony
that a future SuiteCRM may well wire up, and a wrongly valued knob turns into a
400 on the day it does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
59de61ec4 sized the stale-lock window at 1800s but left the waiter's budget at
300 tries of sleep 5, i.e. 1500s. A replica that restarts promptly after a
leader was killed mid-install therefore gives up 300s before it is ever allowed
to break that leader's lock, and only recovers on the next restart. The budget
goes to 420 tries so it always outlives the window it is waiting on. The
`continue` after the stale reap also skipped the sleep, so a persistently
failing rmdir spun hot without ever reaching the timeout; the loop now falls
through to the sleep and retries the mkdir from there.
The seed stamp stored the literal "seeded", so bumping app_version produced a
new image whose tree was never copied into the volume - silently running old
code forever. It now stores the version, warns loudly when image and tree
disagree, and adopts a legacy "seeded" stamp without noise. It deliberately
does NOT re-seed: new code on an unmigrated schema is worse than old code that
works, and there is no suitecrm:app:upgrade step in this role yet. That needs
SUITECRM_APP_VERSION in the runtime image, which the final stage did not carry.
59de61ec4 also established that root-owned trees serve permanent 500s
(live-verified there: chown turned GET / from 500 to 200) but chowned only the
two cache dirs. The installer runs as root and leaves config.php, custom/ and
upload/ behind as well, so public/legacy gets the same treatment once, on the
boot that installed. Scoped to public/legacy on purpose: ${APP_DIR}/extensions
holds a read-only config mount that a recursive chown would abort the container
on under set -e.
healthcheck.html was written on every boot and requested by nobody - the curl
probe hits / because meta/services.yml declares no path - so it goes, and
exec "$@" replaces the hardcoded apache2-foreground that made the Dockerfile's
CMD a lie.
Verified by running the unmodified script in a throwaway php:8.2-cli image with
a stubbed bin/console, seven scenarios: first boot with a read-only config
mount below extensions/ (chown survives it, CMD is reached), restart, legacy
stamp adoption, version mismatch warning, stale lock reclaim, missing
SUITECRM_APP_VERSION aborting legibly, and the non-leader path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d87cb4a51 set COMPOSER_ALLOW_SUPERUSER=1 for the whole builder stage, which
enables plugins for all four composer invocations while only the autoload dump
needs one. Two consequences of that blast radius: public/legacy/composer.json
allow-lists wikimedia/composer-merge-plugin, so the legacy install would start
running a plugin it never ran before, and PluginManager throws
PluginBlockedException non-interactively for a plugin missing from
allow-plugins, so a future app_version bump could abort the build at the very
first composer install instead of never evaluating that condition at all.
The three installs go back to running with plugins disabled, exactly as they
did in the builds CI already proved green. Only dump-autoload gets the flag:
DumpAutoloadCommand calls setRunScripts(true) unconditionally, so it always
dispatches POST_AUTOLOAD_DUMP, which is the event symfony/runtime's plugin
writes vendor/autoload_runtime.php from.
Proven by an A/B build in the role's own shape (php image, composer binary
copied in, open_basedir set to defeat Composer's docker detection): after
`composer install` without the flag `test ! -f vendor/autoload_runtime.php`
passes, and a single `COMPOSER_ALLOW_SUPERUSER=1 composer dump-autoload` makes
`test -f` pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Composer disables plugins when it runs as root without
COMPOSER_ALLOW_SUPERUSER; the env var is the first short-circuit of that
check in Application.php. The builder copies the composer binary out of
composer:2, and a binary copy leaves that image's own ENV behind, so every
composer call in the stage ran with plugins off. symfony/runtime writes
vendor/autoload_runtime.php from its ComposerPlugin, so the file was never
generated and the test -f guard after the seed COPY failed the build. Both
SuiteCRM swarm jobs of run 31927586060 died there, after the retry handler
and the no-cache rebuild.
e6fb78af4 removed the other blocker, where --no-scripts suppressed the
post-autoload-dump dispatch; this removes the one behind it. Composer skips
the root check entirely inside a container (Platform::isDocker), which is
why the defect only ever surfaces in a build layer and never in a run.
Proven by an A/B build in the same shape: php image, copied composer
binary, open_basedir set to defeat the docker detection. Without the
variable the build fails with "The symfony/runtime plugin was not loaded as
plugins are disabled"; with it the guard passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The i8pa* family had no auto counterpart to i8maal, so seeing the whole plan
meant hanging an empty argument on i8papr's --priority flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A clone and a row an earlier one already embeds prove nothing the sweep does
not already prove, but both still took a slot. Drop them when the regular line
is chunked, so the budget buys unique coverage only, and mark them in the plan
table with a status of their own instead of letting them read as 'beyond the
budget'.
A priority row is exempt: an operator who pins a clone asked for that deploy,
and dropping it would be the silent withdrawal 6605c7ef9 just removed.
The cut sits in chunks_of, not in entries_of, so the plan still lists every
candidate while no chunk can see one. 55 of 254 rows are cut on this branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A bare KC_HOSTNAME combined with KC_HOSTNAME_STRICT=false lets Keycloak derive scheme and port from the incoming request. The browser reaches the realm through the proxy on port 80 while container-to-container calls reach it on keycloak:8080, so the same realm mints two different issuers and userinfo rejects the front-channel token with 'Invalid token issuer'.
OIDC.URL carries the scheme, which Keycloak uses verbatim. It is also the value group_vars/all/11_oidc.yml:19 already hands every client as the expected issuer, so server and clients now agree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit glued a literal http:// onto a lookup inside the shell cmd, which test_literal_protocol_with_lookup rejects: the protocol must come from the tls lookup rather than be guessed. Here the exception is real -- this is a container-to-container hop that must bypass the canonical URL, because cURL refuses the .onion name the tls lookup returns -- so the value moves into SHOPWARE_OIDC_INTERNAL_BASE where the suppression sits on the offending line as an actual YAML comment.
The first attempt put the nocheck inside the cmd folded block, where # is not a comment: YAML folds it into the string, so the deploy would have run container exec with a dozen words of prose as arguments while the lint reported green. Reading the folded scalar back is what caught it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The keycloak provider derives every endpoint from one auth-server-url, so both the browser redirect and the server-side calls pointed at the onion. cURL refuses .onion outright under RFC 7686 -- it never opens a socket -- so discovery failed, the authorization endpoint stayed empty and the login button redirected to itself.
The generic open_id_connect provider takes the endpoints individually. The authorization endpoint keeps the onion, which is the address the browser reaches through Tor; token and userinfo point at the internal service name, which cURL accepts. discoveryDocumentUrl still names the onion and therefore fails, which is what leaves the explicit endpoints in place -- discoverWellKnown overwrites them whenever it succeeds, and Keycloak advertises unusable values under every address it is asked through.
Verified in the running container before any redeploy: patching the live client row to this shape makes GET /admin/open-auth/<id>/remote answer 307 to the Keycloak authorization endpoint instead of to itself, and the browser now completes the Keycloak login -- the spec fails afterwards on the admin menu rather than on the login form. The payload was checked against a stub first and matches the live patch byte for byte.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two calls of call-validation.yml activate disjoint halves of it -- the
entry passes pr_scope and gets the branch-prefix and merge checks, the
orchestrator passes the selection and gets the selection check plus the fork
wait -- but both rendered as a bare '✅ Validation', reading like the same gate
run twice.
Name the calling jobs after the half they activate, and take the full history
only on the path that needs it: main_merged.sh resolves a merge base, the
selection check shells out to no git at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
INFINITO_DISCOVERY_SORT is the single source of truth for the discovery
order, but 'asc clone' lived in sort_spec() and was concatenated in front of
it, so the declared value never showed the order a run actually takes.
Declare the key where the rest of the order is declared and let sort_spec
return the value verbatim. test_clones_sort_last_ahead_of_the_declared_spec
now guards the declaration instead of the concatenation, and covered rows
sort directly behind the clone key so they land at the tail of the ranking.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
i8papr takes the priority line as its argument and runs it through
cli.meta.ci.plan, the same build_sweep the deploy jobs discover through, so a
selection can be checked against what CI would actually deploy before the run
is triggered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The regular line was queried with selection.names(leading) as its blacklist,
and names() yields role ids only -- so pinning one variant in the priority
line withdrew every variant of that role from the sweep. Run 31927586060
deployed nextcloud only as #10 and #5; the other twelve variants existed
nowhere.
Blacklist only the tokens that name no variant, and drop the pinned
(role, variant) pairs from the regular rows afterwards. The variant-level cut
cannot go into the query filter: its tokenizer has no '#', so an axis token
aborts the parse.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
env_setting returns a non-empty process value, else the default.env value,
else KeyError -- an empty process override falls back through the strip check
in parser.py. INFINITO_DISCOVERY_SORT is declared non-empty, so the spec the
ternary guarded against could never be falsy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ASYNC pass fires a second backup run when it finds only one generation.
Generation directories are named by the second, so on a host with little to
copy both runs landed in the same one: baudolo reused the finished generation,
rsync --delete overwrote it, and the stamp guard failed the unit afterwards.
CI job 95119330633 died exactly there.
Wait out the second before the extra run, and take baudolo 3.4.3, which claims
the generation directory exclusively and aborts before the first write instead
of reporting the damage after it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bin/console user:update does not exist. The installed shopware/core ships exactly UserCreateCommand, UserChangePasswordCommand and UserListCommand. Both call sites hid the failure: one with || true, the other with || true and 2>/dev/null on top.
rename_admin_user.sh consisted of nothing else. It found the default admin user, printed Renaming: admin -> administrator and then did nothing, so the message was a false statement on every deploy. Script and task are removed. In ensure/admin_user.sh the user:update --email call goes too; user:create already passes --email and that works -- the administrator in the running stack carries administrator@infinito.example, read back through the admin API.
This is behaviour-identical: the removed calls never did anything. What it does change is visibility -- the Shopware default user admin with user@example.com survives every install, which is exactly what the rename was meant to prevent. Shopware offers no console command for that, so closing it needs the admin API and a decision about the account; that is deliberately not part of this commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AdminOpenAuth ships heptacom.admin_open_auth.oidc.http_client with protocols: [https]. An onion carries no TLS -- Tor encrypts the circuit itself -- so guzzle aborted OIDC discovery inside CurlFactory before opening a socket. Discovery never ran, getAuthorizationEndpoint stayed empty, and getLoginUrl returned a bare query string that the browser resolved against the current URL: the admin login button redirected to itself until the test timed out on a blank page. No request ever reached the reverse proxy, and the only trace was a warning in the container-local var/log/dev.log.
A project-level config/packages template redefines that one service id. It grants http only when the resolved provider URL is itself plaintext, so a clearnet deployment keeps the https-only guarantee untouched; the condition keys on the actual scheme rather than on the tor flag, because the scheme is what guzzle rejects. Discovery, token, userinfo and jwks all flow through this single client, so one override suffices; base_http_client and saml.http_client are left alone. Deployment and mount mirror framework.yaml.
Four independent skeptics attacked the earlier claim that this was a hard third-party constraint and refuted it: the pin is an ordinary Symfony service id, Shopware loads project configuration after plugin bundles, and config/packages/http_discovery.yaml demonstrably compiles its services block into the running container. Both branches of the template condition were rendered through Ansible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard sat at line 61, between TestChunksOf and TestOffsetIndex, because 0f450722b appended the offset tests behind an already-present guard. unittest.main() does not return, so running the file directly executed only the classes defined above it and TestOffsetIndex never ran. Under pytest the placement made no difference, which is why the suite stayed green and the drift stayed invisible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
plugin:install carried --clearCache, so the cache was emptied while the AdminOpenAuth client row did not exist yet. Whatever the plugin had cached about having no configuration survived the only clear the deploy performed. The clear moves into its own task behind the provisioning step, which is the first moment there is anything worth caching.
This does NOT make variant 0 green, and the ordering was verified not to: after deploying it, GET /admin/open-auth/<id>/remote still answers 307 to itself and the reverse proxy still logs zero discovery requests. The blocker sits one layer down -- the plugin pins its Guzzle clients to protocols: [https] in its own DI definition while an onion service runs without TLS, so the discovery request is rejected before it reaches the network. Shopware records that as a warning in var/log/dev.log: the scheme http is not allowed by the protocols request option.
The ordering is still correct on its own terms: once that blocker is resolved, clearing before the write would be a no-op again and the failure would return as a follow-up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The collector walked every line of a vars: block and treated each key it met
as a variable passed to the include, whatever its depth. In
vars:
role_templates:
- src_glob: "config/*.j2"
dest_dir: "{{ NEXTCLOUD_HOST_CONF_ADD_PATH }}"
mode: "0644"
exactly one variable is passed. src_glob, dest_dir and mode are fields of its
value, and demanding a Jinja appearance for each of them is a question the data
cannot answer.
It went unnoticed while every such field happened to occur in some template
somewhere. Moving the role-templates expansion into a lookup ended that:
dest_dir is now read in Python, the lint stopped seeing it, and a correct
refactor came out as a lint failure naming a variable that is very much used.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The new role-templates lookup read its paths with os.path, which the repo's
ruff config rejects (PTH113, PTH119). autoformat is the first step of the
quality gate, so those two lines stopped the gate before a single test ran --
the whole suite was unreachable over a filesystem call spelled the old way.
Behaviour is unchanged: Path(...).is_file() and Path(...).name answer exactly
what os.path.isfile and os.path.basename answered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four findings the quality gate raised against the offset and resume work, all
of them mine.
INFINITO_CI_OFFSET sat next to INFINITO_CI_CHUNK, where it reads well and
sorts wrong, and no handler claimed it -- so `make dotenv` would never have
materialised the default and every run outside CI would have resolved the
offset from an unset variable. It is in ASC order now and passes through the
static handler.
runs.py crossed 500 lines. The three helpers that translate job titles back
into selection tokens move to selections.py, which is a seam rather than a
quota: reading a finished run and dispatching a new one is one subject,
turning a job title into `role#variant@mode+tor` is another, and only the
latter is what lets a retrigger replay exactly what failed.
The sshd handler still told operators to look for a handler named
'sshd restart' after the topic became sshd-restart, and three lines from the
offset commit were past the format width.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
render_replicated_templates.yml built its expanded spec list in a 20-line Jinja set_fact, and both it and _render_one_template.yml carried their own copy of the same five defaults. The two drifted: a46348b6c changed the notify default to the compose-up listen topic in _render_one_template.yml and missed the identical line in its sibling, which kept notifying a handler named with a space. Ansible resolves handler names only when they fire, so nothing complained until a role took the src_glob path -- web-app-nextcloud variant 3 then died 49 minutes into a CI run with "The requested handler compose up was not found", after the rescue bundle had filled up with unrelated network diagnostics.
The expansion moves to plugins/lookup/role_templates_expanded.py, where DEFAULTS holds owner, group, mode, condition and notify once and applies them to both the glob-expanded and the passed-through branch. render_replicated_templates.yml stays as the wrapper every consumer includes by absolute path, so no call site changes; it shrinks to the include_tasks loop and no longer leaks _role_templates_expanded into the play namespace. The twelve now-dead defaults in _render_one_template.yml are removed, so a missing key fails loudly instead of falling through to a second definition.
The plugin lives in plugins/lookup rather than in the role, because role-local plugin directories are only on the search path while that role runs and the wrapper is included from magento, n8n, matrix, nextcloud and others; sys-stk-full does not include sys-svc-compose as a role and no consumer declares it as a dependency.
Verified through real Ansible, not only unit tests: the lookup resolves by name, returns a list of dicts that loop can iterate, builds dest from an actual templates directory, and keeps an explicit empty notify empty -- the shape magento relies on to suppress the handler. Twelve unit tests pin the contract, including both falsy cases and parity with fileglob, which drops directories and sorts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A retrigger started the regular line at the head again, redeploying the exact
window the source run had already judged. The red rows come back on the
priority line anyway, so that window was runner time spent on questions with
an answer.
The offset is computed now: the last row of the leading run of rows the source
run reached, as a selection token. Green or red does not matter -- what matters
is that the row ran. The scan stops at the first gap, because a hole inside the
window means that row was filtered out, not that the run got further, and
resuming past it would skip everything behind it.
The ranking is recomputed under the configuration the retrigger carries, since
that is the list the offset resolves against. The priority line is left out of
that computation on purpose: it only blacklists rows from the regular query,
and a token in it that no longer resolves would abort the very helper whose job
is to save time.
offset therefore joins priority as an input the retrigger decides rather than
carries. The integration test forced that: it asserts every carried input
arrives unchanged, and offset no longer did. The category was wrong, not the
test.
Costs a full local discovery per retrigger. There is no cheaper answer -- jobs
inside a chunk are sorted by name, so how far a run got cannot be read off the
job list without the ranking.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The regular rows began wherever (sweep * capacity) % count pointed. On run 210
that was offset 68: chunk 0 opened at candidate 69, the coverage-first head of
the list went undeployed, and no two runs could be compared because the window
moved with the run number. A run that looked wrong turned out to be merely
rotated, and finding that out meant recomputing the arithmetic by hand.
The rotation is gone. Chunk 0 reads the ranking from the top unless an offset
says otherwise, and the offset is an input now, threaded from the manual form
through the orchestrator and the deploy call down to the matrix builder. The
plan renderer takes the same flag, so the table shows the window the run will
take.
The offset is a row count or a selection token. A token is the honest form:
the ranking shifts whenever a variant appears, so 'offset 84' means something
different tomorrow while 'web-app-nextcloud#5' still means the same row. Axes
the token leaves open match anything, so a bare role name starts at its first
row and a pasted job label starts at exactly the job it names. A token that
matches no regular row aborts rather than silently starting at the head an
operator meant to skip.
The run title carries it as ⏩ followed by the value, and only when one was
asked for, so a rotated-looking run says so in its own name and the retrigger
reads it back from there when a job log has expired.
resume_from_chunk leaves the manual form with it. Skipping whole chunks was
the coarse way to say 'carry on further down'; the offset says it exactly and
keeps the rest of the chain intact. The orchestrator input stays for the other
entry points.
What this gives up is automatic tail coverage. Rows beyond the budget are
reached by moving the offset, widening the budget or narrowing the selection,
not by waiting for consecutive sweeps to walk the list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge assertion was a job nobody depended on. ci-orchestrator needed the
scope, the branch prefix and the affected roles -- not it -- so a PR whose base
was not merged in went on to spend five distros on a deploy sweep and reported
a single red job beside a chain that had already finished.
Validation is now a reusable workflow every entry calls, and what it checks
follows the event and the inputs rather than the caller: a pull request gets
its branch prefix and merge base asserted, a run carrying a selection gets that
selection checked, a fork PR waits here for the privileged run's images. The
wait and its gate move in with it, so the orchestrator no longer keeps a second
copy of that logic. Every downstream job depends on the gate.
The selection check is what the last two nights needed. cli.meta.ci.validate
resolves every token against the branch that would deploy it -- role, variant,
mode, onion state -- and reports all offenders at once. Until now the matrix
builder raised on the first bad token from inside a chunk job, so a stale
priority line cost one full CI round per entry and only ever showed the next
problem after the previous one was fixed. Measured on a real list: three
unusable tokens, one pass.
A bare role name that matches nothing is a warning, not an error: the
diff-derived whitelist legitimately names roles the lifecycle envelope filters
out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
