Zammad#2 of run 33923190220 hung three times for exactly 300 seconds and then raised AI::Provider::ResponseError. The rescue artifacts narrowed it only so far: ollama logged no generation during any of the three attempts, litellm had been up since 08:28:46 with zero restarts, and the deploy's own provider ping had passed from this very container minutes earlier at 10.84s. The path worked and then stopped, and nothing in the suite asked again at test time, so the evidence ends at 'somewhere between two swarm nodes'.
The probe asks from the position that failed. It runs through container exec in the rails container rather than in a fresh container on the same overlay, because an attachment a new container gets is not evidence that a long-lived one still has it, and that distinction is exactly what the artifacts could not make.
It asks the gateway for its model list, which needs no credential, so no token reaches the log and any HTTP status still proves the socket carried a request and an answer. REFUSED separates a dead listener from TIMEOUT's black hole, and both separate either from a gateway that answers while the consumer mishandles it.
Executed: all three verdicts reproduced against a live listener, a dead port and a socket that accepts and never replies; the budget is honoured at 10.1s wall for a stated 10s after splitting it across the connect and read phases, which each took the full value before; ruby and net/http are present in ghcr.io/zammad/zammad:7.1.2.
This does not make Zammad#2 green. It observes at test time, not during the assistant request, so a path that breaks after the probe still passes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
13f5ca6d95 put upstream_timeout above every consumer budget on the reasoning that the gateway should only fire once each caller had already given up. That enumerated the consumers and missed a proxy inside one of them: Zammad ships its own nginx with a fixed proxy_read_timeout of 300 in location /, and the role mounts no override. At 420 that hop always won, so a wedged backend would reach the caller as an opaque HTML 504 rather than a gateway error carrying a message.
240 sits below the tightest hop in the path. The front proxy allows 900s and the only tighter timeouts in the repository are the 3s and 5s of the Prometheus healthz routes, which are not on the AI path. Measured on qwen2.5:0.5b, the model the matrix serves: a generation bounded at max_output_tokens 512 finishes in under five seconds, so no legitimate answer under the configured cap can approach the new ceiling.
This does NOT fix Zammad#2 of run 33923190220, and an earlier draft of this message claimed it would. The rescue artifacts say otherwise: ollama logged no generation at all during the three attempts at 09:00, 09:06 and 09:11, litellm had been up since 08:28:46 with zero restarts, and each attempt hung for exactly 300 seconds before rails raised AI::Provider::ResponseError from custom_open_ai.rb:40. The gateway never saw the request, so no gateway timeout can act on it. The same path had worked minutes earlier, warm-up 2.69s and the deploy's provider ping 10.84s from inside the rails container. Rails and litellm share the litellm overlay but sit on different swarm nodes, so that failure belongs to cross-node overlay connectivity, not to this setting.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nextcloud#12 of run 33923190220 failed with 'objects before: 4, after: 4, matched: 0'. The bucket held objects already, so the S3 path works; nothing new arrived. The check could not say why, because the action set the file input and moved on: the save button was optional, and if it was not found within two seconds nobody clicked it, the upload never happened, and the failure still read as an object-store problem.
42449ebf0a had fixed a real bug there, isVisible ignores its timeout and never waits, but replaced it with a two second waitFor. Element renders that button after the file is chosen, which is not always inside two seconds on a loaded runner.
Wait for the media upload itself instead. The response wait is armed before the file is set, so it covers whichever interaction triggers the POST, and it gives the flow time rather than racing it. A future failure now separates the two causes it used to merge: Element never POSTed, or Synapse answered with a status.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Listmonk seeds the administrator from LISTMONK_ADMIN_PASSWORD only while the users table is still empty, and its CLI exposes install, upgrade and config flags but nothing for users. The swarm matrix rotates credentials between the sync and the async pass, so the async pass typed a password the app had never stored; in run 33923190220 listmonk#1 answered 'Invalid login or password'.
The task above already reaches into the same table for email and password_login, so this joins it rather than inventing a second path. Ansible's password_hash cannot produce the bcrypt: passlib 1.7.4 against bcrypt 4.x raises on any input in this toolchain. pgcrypto does it in the database instead, and it is a trusted extension since PostgreSQL 13, so the owner creates it without superuser rights; the control file in postgis/postgis:17-3.5 confirms trusted = true.
Idempotence comes from re-hashing with the stored salt: crypt(:'pw', password) reproduces the stored hash when the password is unchanged, the row falls out of the WHERE, and psql reports UPDATE 0.
Verified end to end against listmonk v6.2.0 on that same postgres image: the app seeds users.password as 60-character bcrypt, the first run reports UPDATE 1 and the second UPDATE 0, and the stored hash then verifies against the rotated password and no longer against the install one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The task above this one is named 'Flush handlers to make DB available before password reset' and was the last line in the file: the reset it flushed for did not exist. The upstream entrypoint sets the password, but inside its install branch next to create-admin-user, so it only ever runs when the app is first installed. The swarm matrix rotates credentials between the sync and the async pass, and the async pass then typed a password EspoCRM had never been told about; in run 33923190220 espocrm#1 re-rendered the login form with the credential still filled in.
The invocation mirrors the entrypoint's own: bin/command is literally 'include __DIR__ . "/../command.php"', so php command.php set-password is the same program, and SetPassword reads through IO::readSecretLine, which is fopen('php://stdin') plus fgets and needs no terminal. Executed against the image: the stty -echo call complains on stderr without a tty and fgets still returns the piped value. The image's working directory is /var/www/html, so the relative path resolves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SEMAPHORE_ADMIN_PASSWORD resolves to the web-app-semaphore-breakglass user password, and the deploy hands it to Semaphore only when the account is first created. The swarm matrix rotates credentials between the sync and the async pass, and --except administrator exempts the literal user key 'administrator', not this one, so the async pass typed a password the app had never been told about. In run 33923190220 semaphore#1 failed with 'Incorrect login or password' after 15.6 minutes of onion-scaled retries.
The realign runs ungated by SSO: the break-glass account exists precisely for the case where the OIDC path is unavailable. Both account operations now share one stack-host block instead of repeating the condition.
Verified end to end against semaphore v2.19.12 on postgres: the CLI accepts --password as a non-interactive flag, change-by-login reports the user changed, the stored hash moves, and it then verifies against the rotated password and no longer against the seeded one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two of them lived inside docs/ next to the pages they render, which is why both had grown a __file__ upward walk to find the repository root, a sys.path insert to import from utils, and a nocheck for each. A generator parked among documents reads as documentation and drops out of the checks that apply to code.
Moving them surfaced six real violations that the old location had hidden: yaml.safe_load instead of the shared cache, two uncached read_text calls, five hard-coded role paths instead of ROLE_FILE_META_SERVICES and ROLE_FILE_META_MCP, two make targets without an alias, and a requirement reference in code. All six are fixed at the root, none with a suppression, and both generators lose their project-root exemptions because a package supplies PROJECT_ROOT.
cli/build/readme joins them so the three generators that write into docs/ sit together. A new lint keeps docs/ to .md, .html, .rst and .nocheck; it enumerates through git ls-files so untracked __pycache__ is not its business. Checked against HEAD, it names exactly the three .py files that moved.
026-unified-addon-syntax stated that the matrix generator stays under docs/requirements/. The lint makes that impossible, so the requirement now records where the generator actually lives.
Verified: every one of the twelve modules under the new tree imports, no old module path survives anywhere in the repository, 482 lint tests and 1246 cli unit tests pass, and both generators run without drift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adopts infinito-nexus/core#568. The composer.json hunk is byte-identical to that PR; vendor/ and composer.lock are gitignored, so it is the only file the PR touches.
The docstring in utils/install/php.py names the constraint verbatim and would have gone stale, which dependabot cannot see.
Verified rather than assumed: phpunit 13.3.2 requires php >=8.4.1 where 11.x asked for >=8.2, and the project container runs 8.4.24. A partial update fails because the lock pins the transitive packages, so resolving needs -W, which carries fourteen packages along including php-code-coverage 11.0.12 to 14.3.2. The suite then runs green under the new major: 26 tests, 56 assertions, no deprecation and no warning, with phpunit.xml setting failOnWarning and failOnRisky.
Open: ensure_php_toolchain takes whatever php the distro ships and states no minimum, so a matrix row on php below 8.4.1 breaks at composer install rather than at a named check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The marker sat above image:, and the check reads it relative to the version: key, either on that line or on the one directly above. Neither matched, so the role declared a pin and still drew an Outdated warning on every run. A warning does not fail the run, so the dead suppressor stayed invisible.
Negative control against the committed file: is_suppressed_at reports same-line=False and line-above=False at HEAD, True after the move. A repo-wide scan with the same helper finds no other version key whose suppressor fails to apply; the seven that looked misplaced use the same-line form, which is equally valid.
Moved rather than bumped: the marker's own reason is that this is the base of a self-built stdlib-only image, and svc-db-qdrant pins the identical 3.13.15-slim for the identical reason. Raising one would split the two adapters.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
confluence 10.2.16 to 10.2.18, erpnext v16.34.1 to v16.34.2, espocrm 10.0.7 to 10.0.8, hermes v2026.7.20 to v2026.9.7, mattermost 11.10.1 to 11.11.0, nextcloud context_agent 2.7.0 to 2.8.0, seaweedfs 4.45 to 4.46, coturn 4.17.2 to 4.18.0. The automated updater opens one PR per role and had not reached these eight; playwright and elasticsearch were already current in the tree.
audit.md follows rather than leads: the MCP report carries a version column and test_mcp_audit_report compares the committed file against a fresh generation, so hermes and mattermost move there too. Regenerated with audit.gen.py as that test instructs. Note that make docs does not run this generator, so the report only tracks reality when somebody calls it by hand.
Verified by the docker-version check itself: eleven Outdated warnings before, zero after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The persona decided the login outcome from whether a password field was still visible, so a flatly refused credential and a broken auth chain produced the same sentence: 'administrator did NOT reach an authenticated surface'. In run 33923190220 listmonk answered 'Invalid login or password' and semaphore 'Incorrect login or password' within a second, and neither reached the report. Each job then spent three onion-scaled retries on it; semaphore burned 15.6 minutes on a login refused immediately.
Read the first visible alert line and quote it in the failure when it reads like a rejection, otherwise keep the existing text verbatim. The matcher is checked against both messages above and against two negative controls, so a page that merely says 'Loading campaigns' still falls through to the generic wording.
This names a cause, it does not fix one. Listmonk, EspoCRM and semaphore seed the administrator password at install and never re-apply it, so the rotation gate between the sync and async pass leaves the app on the old value. That is per-role work on each app's own CLI.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
da729cecec reads .Options.device when a volume declares a bind, on the premise that only compose volumes carry no .Options. A swarm node can hold the volume object without them anyway: in run 33923190220 the DR drill probe reported 'exposes no backing store for kix_kix_config' on the node that held it. The script then falls back to .Mountpoint, the node-local _data that is empty while the stack is down, finds no config there and exits 0.
Pass the inventory's own path as PATCH_DATA_DIR, the same DIR_VAR_LIB/<volume> that compose_volumes.py bakes into driver_opts.device, and try it when the volume-derived path holds nothing.
The second half is the asymmetry that hid this: a config present but unmatched fails loudly since 2c4e0e18b1, while a config found nowhere passed silently. The silent case is the dangerous one. KIX kept the pre-rotation password, its backend crash-looped on exit 255, and the converge gate only gave up 44 retries later. A total miss now prints SKIPPED with both paths it tried, and stays non-fatal so a first install still works.
The suite covered three of the seven consuming roles; it now covers all seven, plus the fallback and the named miss.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
e41be38cb replaced the ansible-vault subprocess with in-process VaultLib and imported ansible at module level. That pulled it into the chain cli.administration.deploy.development -> credentials_generator -> utils/manager/inventory.py -> utils/handler/vault.py, which the CI runner host resolves without ansible installed. Before that commit the module imported only subprocess and yaml, so the import cost nothing.
test_ci_host_dependencies names both remedies: declare the dependency in [project.dependencies], or make the import lazy. Lazy is right here because ansible serves one branch of this module and the declaration would be paid for on every CI job. _vault is already a cached_property, so the import runs once per handler instance and only when something actually encrypts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
VaultHandler.encrypt_string started one ansible-vault process for every single credential. The swarm matrix' rotation gate between PASS 1 and PASS 2 drives 113 application subprocesses over 855 credential keys, so the step cost 1078s in run 33923190220 while the green baseline never ran it at all. Measured per value: 226ms via the CLI, 3.3ms via VaultLib, so 855 keys drop from 193s to 2.8s.
The snippet layout is a contract: encrypt_leaves splits the returned lines and strips the body indent, and existing inventories were written by the CLI. Verified byte-compatible (same 'name: !vault |' header, same 10-space body indent, same $ANSIBLE_VAULT;1.1;AES256 marker), and ansible-vault decrypt reads back what the new path writes.
Reading the password file delegates to ansible's own FileVaultSecret rather than approximating it: ansible strips, then decrypts a vaulted password file, then rejects an empty secret. A hand-rolled .strip() covered only the first of those. _vault stays lazy because callers construct VaultHandler with a password file that need not exist yet.
The tests replace two subprocess.run mocks that asserted against invented output. They now pin the layout encrypt_leaves depends on, and pin the guarantee from 052cea02e2 (values starting with a dash), which landed without a test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The connectivity canary fetched https://pypi.org/simple/ with the suite's
ten-second read timeout. That path is the complete PyPI index: measured at
45 MB here, it times out long before it finishes, and the suite then aborts
with "This host cannot reach the infrastructure the repository depends on"
even though pypi.org answers 200. _probe_url uses requests.get, so the whole
body is downloaded to decide a yes/no question.
https://pypi.org/simple/ansible/ answers the same question against the same
host and service in 199 KB and 0.41s, and ansible is a declared dependency in
pyproject.toml, so the canary still sits on a path this repository actually
depends on.
The guard keeps its teeth. 7b5c462af2 added it because a networkless run had
reported green over 244 unanswered probes, 110 of them real 404s; an
unreachable pypi.org still fails this canary and still aborts the suite.
Only the volume needed to establish reachability changes.
The other two canaries were measured and left alone: registry.npmjs.org
returns 2 bytes in 0.25s and github.com is likewise small.
make test-external went from FAILED TARGETS with exit 2 to 11 passed.
Fediwall#1 of run 33923190220 dies in flowise's MCP registry reconcile:
"FAILED authorizing web-app-prometheus: 400 ... Failed to connect to Custom
MCP server: SSE error: request to http://prometheusmcp:8080/mcp failed,
reason: connect ECONNREFUSED", thirty retries apart. web-app-prometheus sits
in that round's preload closure, so discovery offers it, but the log carries
no web-app-prometheus task at all: the role never ran and its sidecar never
served.
Discovery cannot see that. endpoint_unreachable rejects a provider whose
declaration lacks a port or path, a static check; nothing probes whether the
provider is listening. The platform answers runtime absence with
MCP_RECONCILE_STRICT instead, and 001aeda0b1 states the rule: a rejection
"stays non-fatal in the app pass, where a provider may still be waiting", and
"becomes fatal at the reconciliation stage, where every provider has already
run". sys-svc-mcp-reconcile sets the flag true around its loop and false after;
the group_vars default is false.
flowise's registry task never consulted it, so its own early pass failed the
play on a provider that the later convergence would have reached. It now fails
only when strict, which is exactly the reconciliation stage. The retries stay,
so a provider that appears late in the same pass is still picked up, and the
password and identity tasks above keep failing hard because they address
flowise's own container rather than a foreign one.
Two client roles that never had this problem show the shape: web-app-hermes
and web-app-openclaw call the same file from their own 00_core.yml and carry
no retries/until at all. An earlier attempt in this series deleted the call
from flowise and openwebui as a duplicate of the central stage; fifteen roles
make that call, hermes and openclaw among them, so the duplication is the
convention and that change was withdrawn.
Decidim#1 of run 33923190220 fails on "Wait for web container to be healthy"
after twenty attempts with stdout: unhealthy. The same job passes that wait
twice before it: the container is healthy after roughly thirty seconds on the
first pass and after about three and a half minutes on the second, and on the
third it never gets there inside the five minutes the task allows. At 19:01:41
every decidim container is created fresh, so each pass migrates against a
database the previous pass left behind.
1518512fcf set start_period to 180s from a measurement on an empty database,
where migrate took ~37s and replicas reached healthy at 133s and 123s. That
number was right for what it measured. A third redeploy is a different
scenario, and pass 2 already needed ~210s, past the window.
start_period moves to 10m, joining the six roles that already use it, so swarm
stops reaping a task that is still migrating. The converge wait moves from
twenty to sixty retries at fifteen seconds, giving 900s over a 600s window; it
only costs that time when the container genuinely never comes up.
The healthcheck itself is not at fault. The rescue diagnostics captured
FailingStreak 0 with exit code 0 on curl -f http://127.0.0.1:3000/up, so the
probe works once the app serves. An earlier attempt in this series excluded
/up from HostAuthorization and force_ssl on the theory that the probe was
being refused; passes 1 and 2 pass with that initializer in place, which
refutes it, and that change is dropped.
Prometheus#2 of run 33923190220 fails at tasks/00_core.yml with "Destination
directory /etc/nginx/conf.d/global does not exist". The role wrote its global
nginx config before loading the stack that creates that directory.
sys-stk-backend, which ran just above the write, loads rdbms, objstore, engine
and compose. No webserver. The tree comes from the other end of the file:
sys-stk-front-proxy pulls sys-stk-front-base, which pulls
sys-svc-webserver-https, which includes sys-svc-webserver-core, whose
manager_setup includes 03_directories.yml and only then writes its own
default_server.conf into the same directory. That include sat thirty-six lines
below the write.
Moving the write below sys-stk-front-proxy follows the order webserver-core
already uses for its own file in that directory. It also puts the write ahead
of a flush that can act on it: notify restart-openresty previously fired
before the proxy had been loaded at all.
The ordering only bites when prometheus is the first role in the round to
touch the nginx tree. Prometheus#2 is exactly that case, its preload closure
holding web-app-prometheus alone; where prometheus rides along in a larger
round another role's front stack has already created the directory, which
Fediwall#1 shows with a changed /etc/nginx/conf.d/global/.
This does not address Fediwall#1. Prometheus deploys cleanly there and the
ECONNREFUSED that flowise hits on prometheusmcp:8080 stays unexplained.
collaboraoffice.com/code/ answers 301 and lands on
collaboraonline.com/code/, which answers 200. Both probed. The link now
names the target directly instead of arriving through the redirect.
Documentation only. This fixes no job in run 33923190220 and is not part of
the triage; it rides along because the file was already open.
.claude/settings.json still allowlists WebFetch for the old domain. That is
operator-local config and is left alone.
Nextcloud#2 of run 33923190220 probes the MCP surface three times and gets
"could not reach http://contextagentmcp:8080/mcp: <urlopen error [Errno 111]
Connection refused>" before failing on both workers. Connection refused, not
an unresolved name: the sidecar's address existed and nothing served it.
meta/services.yml derives both halves of that surface from the same switch,
context_agent and contextagentmcp each carrying
enabled: "{{ lookup('config', 'web-app-nextcloud', 'mcp.enabled') | bool }}".
mcp.enabled is itself derived, from peer presence rather than by hand:
utils/roles/applications/mcp.py renders "'peer' in group_names or ..." over
the consumers the role admits, and for nextcloud that is hermes, openclaw and
openwebui. Variant 2 is the openwebui variant, so the switch is true, the
sidecar comes up, and only the app behind it stayed down.
It stayed down because the variant pinned it off by hand. That pin is an
orphan. a2be4f081a introduced context_agent paired with an mcp pin in each
variant and kept the two in step; 06bc856c32 lifted mcp out of the services
map when it became a role topic; ffdef28da3 then dropped the hand-maintained
mcp values entirely, calling them "wrong in both directions", and replaced
them with the peer derivation. The mirrors it left behind have contradicted
that derivation ever since.
Removing them changes exactly one variant. Evaluated across all fourteen:
variants 0 and 1 keep their explicit enabled: true and are untouched, variant
2 gains the agent its sidecar needs, and variants 3 through 13 admit none of
the three peers, so the derivation yields false and they behave as before.
Eleven of the twelve removals were writing a value the derivation already
produced.
The budget objection does not hold either. collect_role_resources counts any
enabled that is not literally false, so a rendered template counts as on;
test_resource_budget therefore charges the agent's 512m to every variant that
no longer pins it off, and still passes.
merge/svc-net-tor still carries all fourteen declarations and will reintroduce
the orphans if it lands.
utils/roles/guide.py picks "compose" if the role ships a stack and "host"
otherwise. The compose branch runs make compose-up, which starts the coredns
service from compose.yml, so the replay resolves the deployment's own names.
The host branch runs this script, a bare container on the default bridge with
no resolver anywhere.
That only started to matter with 4986e80dd4, which threaded DOMAIN into the
README replay so it deploys under INFINITO_DOMAIN instead of inheriting
infinito.localhost. Before it, sys-svc-nss resolved *.localhost through
nss-myhostname and no DNS server was needed; after it, a stackless role asks
for infinito.test and gets nothing.
web-svc-html shows the whole chain in run 33923190220. Its guide replay logs
"Verify 'html.infinito.test' is served after the reload" with [Errno -2] Name
or service not known, sys-ctl-hlth-webserver probes every NGINX domain and
exits 1 through script.py's `return 1 if errors else 0`, systemd refuses to
start hlth-webserver, and sys-service-terminator fails the play after 25
attempts. The same run's Fider replay, a compose-mode role, answers the
identical task with real HTTP headers.
Rather than rebuild the resolver by hand, the script now brings up the very
service the compose branch uses and reads the network back off the container,
so subnet, gateway, MTU, image, Corefile and command all stay in compose.yml.
The Corefile is rendered through CoreDNSCorefileRenderer so the script does
not depend on an earlier phase having produced it. coredns declares no
depends_on, so the infinito service stays down and INFINITO_IP4 is free for
the guide container; the run's own log shows the CI phase removing
infinito-nexus-core_default seventeen seconds before the replay starts.
-v /srv is the second failure in the same class. svc-storage-nfs-server dies
with "mount.nfs4 ... reason given by server: No such file or directory" on a
host reporting Storage Driver: overlay2, because name_to_handle_at answers
EOPNOTSUPP there and the server builds no export root. A real volume under
the export path takes it off the overlay.
Nextcloud#1 of run 33923190220 fails two admin surfaces:
addons/whiteboard.spec.js:28 waiting for the settings/admin/whiteboard shell
and addons/integration_openai.spec.js:32 waiting for #openai_prefs on
settings/admin/ai. Both time out with "element(s) not found" after two
retries.
The cause is not the apps. In the sociallogin flavor the Keycloak login
provisions a second Nextcloud account whose uid is <provider>-<keycloak sub>,
while the role only ever puts the local administrator account into the admin
group, so the browser session that the specs drive is an ordinary user and
every /settings/admin/* route renders nothing. sso_oidc_plugin.py returns
"oidc_login" if ldap_enabled else "sociallogin", so this only bites where LDAP
is off: variants on oidc_login with oidc_login_proxy_ldap land on the
LDAP-backed account that 07_admin_identities.yml adds to admin instead.
The provider asked for scope "openid" with an empty groupsClaim, so no group
information reached the plugin at all and no mapping could have worked. It now
requests the RBAC scope, reads the claim, and maps the application's
administrator group onto Nextcloud's built-in admin group.
Every link is in place already. RBAC.GROUP.CLAIM is "groups", the Keycloak
client scope carries the same name and sits in defaultClientScopes, and its
mapper sets full.path true, so the claim carries /roles/<app>/<role> and the
leading slash in the mapping key matches. rbac_group_path accepts
'administrator' without a declaration because it is the implicit admin role,
and build_realm_rbac_groups adds that group to every RBAC application, so the
group the key names is really provisioned. Upstream reads both keys:
ProviderService maps groups_claim to groupsClaim and group_mapping to
groupMapping, and applies the mapping by group id.
Adminship is necessary for those two specs but may not be sufficient: both
assertions also claim their app is installed and enabled.
92c45e2471 moved Open WebUI off the model server onto the gateway and renamed
test-ollama-chat.js to test-litellm-chat.js in the same commit, git records it
as R073. `git ls-tree` on that commit shows only the successor in the
directory, and both OLLAMA_* variables left playwright.env.j2 with it.
fork/feature/svc-net-tor still carried the pre-refactor file. Merging it in
7162b7bf4c restored test-ollama-chat.js and OLLAMA_PRELOAD_MODEL without a
matching add: `git log --diff-filter=A` on that path knows only the original
creation from 06aa25c095. A scan of all 916 merges reachable from HEAD finds
exactly one other path resurrected the same way, .github/pull_request_template.md,
which is not a role file and is left alone here.
Deleting it removes no coverage. The file never executed: it does not match
the collector's `**/*.@(spec|test).js` and no aggregator required it, and the
JUnit report from openwebui#1 lists thirteen tests with no `ollama chat`
among them, not even as a skip. Four of its five assertions are byte-identical
in the registered successor, which runs green in openwebui#1 and #2.
The fifth is deliberately weaker there: the successor takes the first
non-embedding model from /api/models instead of pinning OLLAMA_PRELOAD_MODEL,
so a green run proves the gateway serves a chat model rather than that Ollama
specifically does. That was the refactor's stated intent, exercising the chain
rather than a fixture, and reviving the file would route a spec straight at a
model server that a lint exists to keep roles away from.
OLLAMA_PRELOAD_MODEL goes with it: the deleted spec was its only reader. The
plural OLLAMA_PRELOAD_MODELS in svc-ai-ollama is a different variable and
stays.
playwright.config.js collects `**/*.@(spec|test).js`, so a module named
test-<something>.js runs only when its role's playwright.spec.js requires it.
Three roles wrote theirs as standalone specs with a top-level test() instead:
the MCP guest guards of n8n and wordpress, and the OIDC login of hermes. All
three have been dead files since they were added, so every MCP surface was
believed to be probed by a hostile guest and hermes' oauth2-proxy round trip
to be covered while none of those assertions had ever executed.
Each moves to the exports.register shape its siblings use and is appended to
its aggregator. Nothing else changes: `git diff -w` on this commit is the
test.use line becoming the register wrapper plus one require per aggregator,
and no assertion is touched. The module-level
test.use({ignoreHTTPSErrors: true}) goes away because all three aggregators
already set it for the tests they register, and a test.use inside register()
would not apply anyway. registerMcpDisabledState moves inside register() so
the switched-off case comes with it, which the addon specs do not cover since
they skip when the addon is off.
Registering them also subjects them to test.beforeEach(shared.beforeEach).
That is safe here: wordpress sets a viewport and asserts env presence, n8n
does the same and clears cookies, hermes is a no-op. None authenticates, so
the guest guards stay guests.
wordpress is the only one of the three with a green job to lose, and its
status assertion is already proven there: addons/mcp-adapter.spec.js posts to
the same endpoint with maxRedirects 0 and asserts >= 400, it matches the
collector glob, and WordPress#1 passes. The permission callback runs at the
transport before the JSON-RPC method is parsed, so initialize is refused
exactly like tools/list.
A fourth dead module, openwebui's test-ollama-chat.js, gates on a service no
role renders and follows separately.
checkmk crash-looped in swarm on run 33923190220, 82 convergence attempts,
with PermissionError [Errno 13] on '/omd/sites/cmk/tmp/nagvis' raised from
omdlib/tmpfs.py:294 while OMD populated the site's tmp skeleton.
The mount is not the problem and must stay. Upstream's docker-entrypoint.sh
runs `omd create --no-tmpfs` and the comment above it states the site tmp
directory "might have been pre-created by docker when the --tmpfs option is
used", so supplying that tmpfs externally is the intended pattern, not a
workaround.
The mode was the problem. container_volumes coerces a declared mode with
int(text, 8), so "01777" reaches the stack file as the decimal 1023. Docker
29.7.2 reads that as raw mode bits and mounts 1777; the CI hosts run 28.0.4
and produced a directory the owning uid could not write. 0o1023 leaves the
owner with no permission bits at all, which is exactly the failure.
Docker's own default is already 1777, sticky and world-writable, which is
what a site tmp needs, so the declaration could only ever match it or break
it. Dropping the key restores the default. Measured on 29.7.2 in both
`docker compose` and `docker stack deploy`: a mode-less tmpfs mounts 1777 and
uid 1000, the uid `omd create -u 1000` gives the site, creates nagvis in it.
The validator now rejects tmpfs 'mode' outright rather than checking it is
octal. An octal check cannot help when the encoding itself does not survive
to the container the same way on every engine version. merge/svc-net-tor
still carries the old declaration and will fail this check if it lands.
The gateway registered ollama and lmstudio with an rpm cap and nothing
else, so a single request could generate without an output limit and hold
a worker for as long as the backend kept producing tokens. Requests per
minute bound how often a caller may ask, not how much one answer costs.
max_output_tokens caps the answer itself. upstream_timeout is the backstop
for a wedged backend, not a latency budget, so it sits above every
consumer's own budget: the longest wait on the AI path is 360s
(discourse-ai.spec.js), with 300s and 240s behind it, and TIMEOUT_FACTOR
defaults to 1. At 420s the gateway can only fire once every caller has
already given up, which keeps a slow generation from being reported as a
gateway error instead of the slow answer it is.
openrouter/auto stays unbounded here. It never carried an rpm cap either,
and its limits are the provider's rather than ours, so giving it one is a
separate decision from closing the local-backend hole.
mcp/vhost.conf.j2 wrapped its whole body in `{% if not mcp_acl_gated %}`.
That skipped the serve block, which was the intent, but it also skipped the
withdrawal loop underneath it. An ACL-gated vhost therefore rendered no
`location` for its MCP paths at all, so requests fell through to the
application's generic location and the endpoint was reachable behind
nothing but the SSO proxy ACL.
Folding the gate into the `mcp_served` expression keeps the serve block
gated while letting the withdrawal loop run in every case. With an empty
`mcp_served` both declared paths differ from it, so both are withdrawn with
`return 404`.
The behaviour is latent today: only moodle and wordpress declare
`exposure: public`, and both carry `flavor: oidc`, so `is_proxy_gated` is
false and the branch is never taken. This closes the hole before a role
combines public exposure with an oauth2 proxy flavor.
test_mcp_vhost_exposure.py could not reach the branch: its lookup stub
returned "" for every non-config kind, so mcp_acl_gated was always falsy.
It now resolves is_proxy_gated and oauth2_acl, and gained the bool filter
that the internal-exposure cases never needed because `mcp.exposure ==
'public'` short-circuits before it. Rendering the pre-fix template under
gating yields zero withdrawals; the new one yields two.
Commit 1cd9578b8 moved the OIDC plugin version out of a hardcoded string
into meta/addons/auth_oidc.yml so the update pipeline tracks it. That
declaration is service-coupled (enabled reads services.sso.enabled), which
obliges the role to ship an integration hook and a per-addon Playwright
spec. Both were missing, so test_addons_integration_tasks and
test_addons_playwright_spec went red.
The hook has no work to do: files/build/install/auth-oidc.sh bakes the
plugin into the image and tasks/04_oidc.yml configures it, included by
tasks/01_manager_ops.yml on the same services.sso.enabled flag. It records
where the wiring actually lives so the coupling stays greppable.
The spec does not follow the pretix shape of hunting a login-page link,
because tasks/04_oidc.yml sets alternateloginurl and /login/index.php
bounces straight to the plugin. It asserts the bounce instead: the login
page must land on the Keycloak openid-connect authorization endpoint
carrying a client_id and a redirect_uri pointing back at auth/oidc.
Rendering the addon flags into playwright.env.j2 is what makes that spec
real. Without lookup('addon_env_flags'), AUTH_OIDC_ADDON_ENABLED is
undefined, skipUnlessAddonEnabled skips forever and the spec is a check
that can never fail.
download.moodle.org returns 403 to GitHub-hosted runners while serving the identical artefact elsewhere: 12 failures across three runners over twelve hours, each 40-66ms after TLS, with curl's four retries and Ansible's three exhausted. The GitHub tag archive is content-equivalent - both tarballs extracted and diffed over 39312 entries, the only delta being githash.php, which nothing in the 4.5.11 tree includes - and needs --strip-components=1 because it roots at moodle-4.5.11/ rather than moodle/.
auth-oidc.sh was the next landmine, two Dockerfile lines behind the fixed one: an anonymous api.github.com tag query, 60 requests per hour per runner IP, no retry, -s swallowing the rate-limit body under set -e, and a floating plugin version as the build input. It now reads a new meta/addons/auth_oidc.yml, which declares bridges: [sso] and update.monitored so the version is watched by the update pipeline instead of frozen.
MOODLE_BRANCH and MOODLE_TARBALL_FILENAME die with the old URL; the latter was already dead, demanded only by its own :?required guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
