i8-quality drives `make quality`; the gate that has to be green before a pull
request is `make quality-high`, which appends every lint target and runs about
four times as long. Nothing routed an operator there.
The skill leaves the fix loop to test-fix and pins only what differs: the run
goes to the background with its output teed to /tmp so the operator gets a
tail -f line, nothing else may run beside it because the wall-clock budgets
fail under self-made load, and the verdict comes from the two per-target tables
rather than the pipe exit code, which is unreliable here. docs and autoformat
write to the tree, so an unexpected diff after a pass is a finding.
Suppression is not a fix: a lint hit is resolved by deleting the offending
construct, and any nocheck marker needs the operator's explicit approval.
The cheatsheet row is not decoration - test_cheatsheet_is_skill_index requires
every i8- skill directory to appear in that table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bluesky web service bound only deploy.yml.j2, never the container base
template, so compose rendered no restart key for it and Docker applied its own
`no`. When sys-svc-container reconfigured the daemon mid-deploy and restarted
it, pds and broker came back on `unless-stopped` and web stayed down; openresty
answered 502 and all three Playwright specs failed against it.
The lookup is a no-op under swarm, where deploy.restart_policy already carries
the intent, so this only fills the compose gap.
The compose-structure lint now refuses a service body that declares no restart
policy at all. A body that only includes deploy.yml.j2 renders a policy for
swarm and nothing for compose, which reads as deliberate in the template and is
invisible until a daemon restart. Bluesky's web was the only service in the
tree with that shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_prefix_groups_belong_in_subfolders flagged three groups added by this change
set: a shared delimiter-bounded prefix is a folder in disguise, so the group moves
into a folder named after the prefix and the prefix leaves the file names.
sql/select/ai-{bot-user,service}.sql.j2 -> sql/select/ai/{bot-user,service}.sql.j2
files/provision_{ai,mcp}.py -> files/provision/{ai,mcp}.py
files/ai_engine_{configure,endpoint}.php -> files/ai_engine/{configure,endpoint}.php
All six are recorded as renames, so history follows them. Every reference moves
with them: four templated query paths in mattermost's ai_gateway.yml, the two
lookup('file', ...) calls in n8n, the two in wordpress, and SCRIPT_PATH plus the
module docstring in the n8n unit test. lookup('file', ...) resolves relative to
the role's files/ directory, so the new argument is provision/mcp.py rather than
a repo-relative path.
Worth recording why this survived every gate: make quality was green on each run
this session, including immediately before the commits that introduced these
files. The lint enumerates through git ls-files, and the files were untracked, so
it could not see them. Committing them is what made them visible. A green gate
taken before new files are tracked says nothing about the gate after.
Verified: repository lints go from 88 passed with 1 failure to 89 passed, the
unit suite passes at 4448, and a repo-wide grep for every old path returns
nothing.
Not verified: no deploy has run. The lookup('file', ...) arguments and the
templated SQL paths resolve at play time, so a wrong path would surface as a task
failure rather than a test failure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The requirement corrects two rows against what the pinned releases actually
expose: Moodle 4.5 configures aiprovider_openai per action rather than through a
single base URL, and n8n 1.95.3 reads its endpoint from a stored credential,
after N8N_INSTANCE_AI_MODEL_URL turned out not to exist in that release. espocrm
and peertube move into the exclusion table with measured evidence rather than
vendor claims, and the variant-coverage section records which round exercises the
gateway.
Criteria 2, 5, 8 and 9 are ticked. Criteria 3, 4, 6 and 7 stay open on purpose:
the wiring is complete across all nine confirmed roles, but only zammad's spec
has been observed answering a prompt through the gateway, and the last deploy
ended before Playwright ran. Criterion 1 is still open because homeassistant,
xwiki and baserow have not been probed.
The 027 generator gains litellm as a framework key and litellm and n8n on the
web-app axis, so every consumer's gateway edge appears in the matrix. Its
regenerated output is committed alongside it; the generator writes the file in
place rather than to stdout, and a fresh run reproduces the committed matrix
byte-for-byte. Touching the generator also brought its comment history into the
lint's scope, so about thirty section banners and narration lines go with it.
The espocrm README carries the exclusion evidence -- at tag 10.0.4 only Crm ships
under application/Espo/Modules, the config metadata declares no AI key, and the
release ZIP leaves custom/Espo/Modules empty -- plus four steps for enabling the
commercial Intelligence extension once licensed. A version bump does not help:
upstream documents that extension as requiring 10.0.3 or greater, so newer cores
host it rather than absorb it. The provider field names are only discoverable by
unpacking the purchased artifact, and the note says so rather than guessing.
The root README and the litellm README are generated; both now show the gateway
among the dependencies of its eight consumers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scripts/tests/e2e/rerun-spec.sh opens with
reports_base="${INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR:?source scripts/meta/env/load.sh or run via make}"
and `make compose-playwright` runs it inside the development container, where
nothing carried that variable across the boundary. The documented target failed
on its first line. The image injects no BASH_ENV or profile entry, so compose's
environment block is the only channel in.
That target re-stages a role's specs from the repo and reruns them against the
live stack without redeploying -- seconds instead of hours -- so a broken entry
point costs far more than its size suggests. It is what proved the zammad AI
spec green today, reached only by supplying the variable by hand.
Only the reports dir is forwarded. INFINITO_PLAYWRIGHT_STAGE_BASE_DIR is
deliberately NOT, although fork/feature/svc-net-tor carries cca47b471a which adds
both. The two are not symmetric. REPORTS is a Jinja path_join over DIR_VAR_LIB,
which awk cannot parse, so the :? hard-fail is real and forwarding it is
necessary. STAGE is the bare literal /tmp/test-e2e-playwright, and the script's
fallback parses it from the same vars/main.yml the Ansible stager reads, deriving
repo_root from its own path -- so the fallback cannot fail anywhere the script
can run, and Ansible never reads the env var at all.
Forwarding STAGE would therefore add no capability while inserting a
higher-precedence source for a value whose only authority is that literal.
EnvBuilder.setdefault resolves `existing or value` against os.environ, and
Makefile exports BASH_ENV=load.sh which sources .env into every recipe, so a
stale value pins itself against the SPOT; `make dotenv-force` exists precisely to
break that pin. The divergence would surface as rerun-spec.sh's "missing staging
dir (run deploy first)", an error whose instruction cannot fix it.
MERGE NOTE: reconciling with cca47b471a needs a deliberate one-line resolution --
drop their STAGE line -- and no lint guards this in either direction, so it will
silently return otherwise.
Verified: the failure was reproduced, and the same command succeeded once the
variable was supplied, isolating the missing passthrough. At runtime the
container now shows REPORTS set and STAGE absent, with the fallback carrying
STAGE correctly.
Not verified: the repaired target has not been run, because picking up a new
environment entry needs the container recreated and that would destroy the stack
state this session built. Unfixed in either variant: the REPORTS :? still aborts
on DiD nodes reached through node.sh, which forwards no environment at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bd84664e8 moved debian and arch to a cached Packages index but left ubuntu
on packages.ubuntu.com, one HTTP request per package. That endpoint is the
same anti-bot territory: 96 requests behind a 2-per-host semaphore, and a
single timeout fails the target. Three workspace jobs of one run died that
way on different packages (tor/traceroute, msmtp), which is a lottery, not
a defect.
The debian loader is now generic over mirror, path, pockets and components,
so ubuntu answers from the same 12-request bulk fetch through the existing
_get retry layer. It also probes the pockets a default install resolves
against: ubuntu ships 0ad only in noble-updates, and the release-pocket
index alone reports it absent. The old HTML probe called it present because
the "not available" page does not match the "no such package" regex, so the
per-package route was reporting a false positive as well.
Verified: 96/96 identical verdicts old vs new across every ubuntu probe,
454 probes and 23 inconclusive unchanged, and test-external green inside
the container through the package-cache frontend (8 passed in 108s, against
1025s and red in CI).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role drove Jellyfin's first-run wizard as soon as the container existed, but
created is not serving: Jellyfin initialises its database and web host after
start, so the wizard POSTs raced that initialisation. Losing the race did not
merely fail the step, it could half-apply and consume the one-shot wizard on an
instance that was never configured.
The auth flow now polls the container's own healthcheck, up to five minutes,
before touching the wizard. Three details are deliberate: the Go template is
wrapped in {% raw %} so Jinja does not try to resolve .State.Health.Status and
fail at render rather than at run; the inspect is delegated to the hosting node,
because under swarm the container is not on the controller; and changed_when is
false, since a readiness probe reports state and never changes it.
Not verified: jellyfin was not in the last deploy's role set, so this has not
run. The gate is also only as good as the image's healthcheck -- if that reports
healthy on "HTTP responds" rather than "setup endpoints ready", this narrows the
race without closing it. The 60x5s bound is a choice, not a measurement against
Jellyfin's cold start on a loaded box.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A membership written in LAM crosses two systems before any consumer sees it:
Keycloak reads it from LDAP through this federation provider, then the consumer
syncs its groups claim on the next login. With cachePolicy DEFAULT the first hop
serves a cached federated user, so a grant is invisible to Keycloak -- and
therefore to every OIDC consumer -- until the entry expires. The user
re-authenticates and still does not hold the role they were just given.
NO_CACHE makes every lookup hit LDAP, so the membership lands at the next
authentication. This is the other half of 54aced186: that change had to make its
spec poll rather than read once precisely because of this staleness.
The cost is explicit: this removes a cache from the authentication path, so every
login and group check now performs an LDAP round trip. On this platform
svc-db-openldap sits on the container network and the round trip is cheap, but on
a deployment with a slow or remote directory it would be felt at every sign-in.
Choosing NO_CACHE over a bounded EVICT_DAILY or MAX_LIFESPAN is a judgement, not
a measurement: a grant that silently fails to apply is worse than a slower login.
The better design is an explicit cache invalidation when a grant is written,
which would keep both freshness and caching; it is not built here.
Not verified: no deploy has run with this, and the login-latency cost was not
measured before or after. I also did not confirm this template is the only place
a cache policy reaches the federation provider -- a second component or a runtime
override could still cache.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
checkmk's entrypoint runs `omd create --no-tmpfs`, so the site's tmp directory
is never mounted and the site cannot come up cleanly. Declaring it needs a tmpfs
with sticky, world-writable permissions, which the volume layer could not
express.
container_volumes built the tmpfs options as a hard-coded single key:
if mount.get("size") is not None:
tmpfs_entry["tmpfs"] = {"size": mount["size"]}
so a `mode` in the declaration was accepted by the schema and silently dropped by
the renderer. It now accumulates size and mode and attaches the key only when
something is set, coercing mode to the int docker expects. mounts.py validates
both the bind and tmpfs spellings through _is_octal_mode and names the expected
shape in the error, so `1777` or `0o1777` fails at render with a readable message
instead of producing the wrong bits.
Capability and consumer land together deliberately. Split apart, checkmk's
mode: "01777" would pass validation, be dropped at render, and the tmp directory
would come up without the sticky bit while everything reported success.
checkmk declares site_tmp as a tmpfs at the site's tmp path with size 512m and
mode 01777. The site name moves into services.checkmk.site, and CHECKMK_SITE
reads it through lookup('config', ...) instead of hard-coding "cmk", so the mount
target and the variable cannot drift.
MERGE NOTE: fork/feature/svc-net-tor carries acc9e25c0f, which deletes
plugins/filter/container_volumes.py and moves it to plugins/lookup/. That
relocated copy has the identical defect at lines 141-142 -- the same hard-coded
{"size": ...} -- and its _coerce_mode_int is wired only to the config-ref path.
It is not a competing fix, but this change will disappear at merge unless it is
re-applied in the lookup.
Verified: the unit suite passes, 4448 tests, including new coverage of the mode
path in test_container_volumes.py.
Not verified: checkmk has not been watched starting with this mount. Docker's
tmpfs mode also interacts with the container umask, so the effective bits on the
created directory were not inspected live. The new validation is fail-closed: a
role carrying a non-octal mode would now error at render rather than be ignored,
and while the suite is green, I did not grep every CHECKMK_SITE consumer for one
still assuming the literal "cmk".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Belongs with 8c3d6fcde and was left out of it.
The fake n8n API gains a credentials store, answers GET /rest/credentials, and
records every write. Recording the writes is the point: observing only the final
workflow body would pass a run that created the credential twice, or never, as
long as the node ended up referencing something. The assertions cover creation
when none exists and match-by-name when one does, and REFERENCE pins the
{id, name} shape the trigger node has to carry.
Verified: the full unit suite passes with it, 4448 tests.
Not verified: this exercises a fake, so it proves the provisioner issues the
right calls in the right order, not that n8n accepts them. The real
"500 No authentication data defined on node!" is still unobserved as fixed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ENABLE_OAUTH_ROLE_MANAGEMENT does two things and only one is obvious. Besides
mapping the administrator group onto the Open WebUI admin role, it turns
OAUTH_ALLOWED_ROLES into a sign-in gate: a user whose groups claim is non-empty
and matches neither the allowed list nor the admin list is refused outright.
Upstream defaults that list to the literals `user,admin`, and Keycloak emits
claims shaped /roles/<app>/<role>, which can never match them. A user with no
groups therefore signed in fine, and the moment they received their first `mcp`
grant their claim became non-empty, matched nothing, and they were refused. The
grant meant to give access was what removed it.
The env sets OAUTH_ALLOWED_ROLES=*. The grant itself stays the real gate, since
BYPASS_ADMIN_ACCESS_CONTROL=false keeps it binding on administrators too. Stated
plainly, this does widen a control: every authenticated platform user now passes
Open WebUI's role gate. A list matching the actual role paths would be stricter
and is not built here.
The LAM grant spec is rewritten to drive the real path rather than assert around
it. Three behaviours it now handles: LAM renders per-value add and clear controls
as anchors with an empty alt, so they cannot be found by accessible name and are
located structurally; a membership written in LAM crosses Keycloak federation and
then an Open WebUI login sync before it is observable, so the check polls instead
of reading once; and the sign-out step clears localStorage rather than only
cookies, because Open WebUI keeps its session token there and a cookie-only reset
left the old session live while the assertion passed.
Not verified: openwebui was not in the last deploy's role set, so neither the env
change nor the rewritten spec has run. The spec is also now genuinely
re-authenticating, which is more correct and will surface auth flakiness the
previous version masked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mcpTrigger node forces authentication to bearerAuth, because the upstream
`none` option would publish an unauthenticated SSE endpoint on the role's own
vhost. Setting that parameter is only half the contract: the node must also
reference an httpBearerAuth credential holding the token. Without one the
trigger answers every request to <path>/sse with
500 No authentication data defined on node!
before the SSE transport is ever opened, while the workflow itself looks
correctly configured and reports active.
The provisioning chain gains a credential leg between the API key and the
workflow: POST /rest/login for the owner cookie, /rest/api-keys for the
deterministic managed key, /rest/credentials for the bearer, then the public
/api/v1 routes for the workflow. The credential is upserted under the
deterministic name in N8N_MCP_CREDENTIAL, which is why that constant landed with
the AI bundle: the variable has to exist before this references it.
The workflow body is now pushed on every run, including while the workflow is
active. Previously a node that had drifted from the definition stayed broken
behind its own active flag, because the reconciler saw an existing active
workflow and left it alone. Pushing unconditionally lets a wrong node heal.
Verified: the /rest credential route with the owner cookie is exercised by the
sibling provision_ai.py, which ran in a variant-2 round, so the mechanism this
depends on works even though this call has not been observed.
Not verified: no MCP request against n8n has been seen answering 200 since the
change. Two consequences are deliberate but real -- an operator's intentional
edit to the managed workflow is now overwritten on every deploy, not just a
drifted one, and because n8n never returns encrypted credential data, a token
that drifted from N8N_MCP_TOKEN cannot be detected, only overwritten.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moodle 303-redirects every request whose Host is not $CFG->wwwroot, and an MCP
client dialling the internal service name cannot follow that redirect from the
container network. The previous accommodation was on the wrong side: the probe
sent -H 'Host: <public domain>' and -H 'X-Forwarded-Proto: https' and pinned
mcp_probe_host to MOODLE_DOMAIN, so it asserted the endpoint works when spoofed
and said nothing about how a real client reaches it.
Both headers and the probe host are gone. Acceptance moves into config.php,
which reads MOODLE_INTERNAL_HOST -- rendered from MOODLE_MCP_SERVICE_KEY -- and,
when the incoming HTTP_HOST matches it, rewrites HTTP_HOST and SERVER_NAME to
the wwwroot host before Moodle's redirect logic runs. The comparison splits on
':' so a port cannot defeat the match, and a request from any other host is
untouched, leaving the canonical-host protection intact for real traffic.
None of that could have taken effect on a deployed instance. The bootstrap seeds
the code directory with `cp -an`, which never clobbers, so an existing volume
keeps its first-run config.php forever and every later config change is silently
discarded. moodle_refresh_config now copies the deployment-managed config.php on
every start, which is what makes the rewrite above reachable at all.
Not verified: no MCP call has been observed succeeding against the internal name
since the change. This is also the same failure class that ended the last
deploy on another role, where the client already shared the provider's network
and had the right alias registered, yet the DNS search suffix still resolved the
single-label name to the public ingress. If that applies here too, the Host fix
is correct and the request still never arrives -- and removing the spoofing
headers will surface it as a failure where the probe previously passed.
Two further consequences worth stating: config.php is now overwritten on every
start, so an operator edit inside the container is discarded; and the HTTP_HOST
rewrite runs in the hot path of every page load, narrow though its condition is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mcp_servers hands every client the bare URL http://<service_key>:<port><path>,
which resolves only when the client shares a network with the provider. The
overlay declared modes: [swarm] with neither a topology nor a consumer kind, so
under compose _compute_attachments produced no attachment at all.
Nothing failed, which is why this survived deploys. The docker daemon ships
dns-search: [DOMAIN_PRIMARY], so the unresolvable service name was completed
into a public FQDN, the wildcard record answered with the ingress address, and
every MCP tool call for the Apache container went out over the internet and back
in through the reverse proxy. The calls succeeded; only the route was wrong.
The overlay now runs in both modes, creates shared_net, and admits mcp_client so
each role named in mcp.allowed_consumers is attached. aliases becomes
proxy_aliases, because the public-domain alias belongs on the proxy leg rather
than on the shared network.
auth changes from app_password to basic_auth: WordPress application passwords are
presented over HTTP Basic, and the adapter needs the transport, not the
credential's product name. web-app-openwebui leaves allowed_consumers.
The mu-plugin filters wp_is_application_passwords_available. Core gates them on
is_ssl(), which is false on the internal hop because TLS terminates at the proxy
and no X-Forwarded-Proto reaches wp-config.php, so the service account could not
authenticate. The filter reads the site's own canonical scheme, leaving the gate
closed on a genuinely plain-HTTP site.
The new lint fails any role declaring mcp.endpoint whose overlay has no topology
or does not admit mcp_client, with a nocheck opt-out in the networks file head.
This role is what motivated it.
Measured while writing this: from a throwaway container sharing a provider's
network, `nslookup <service>` returned <service>.infinito.example -> the ingress
address, while the service answered 401 immediately from inside its own
container. That is this failure mode observed live.
The same measurement bounds the claim. In that case the client already shared
the network AND the container alias was registered, and the search suffix still
won, so attachment may be necessary without being sufficient. The WordPress
endpoint has not been re-probed since this change, and the lint encodes
co-residency as the whole condition, which that observation puts in doubt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deletions only, no line added. The rationale each of these carried is already in
the commit message of the change that introduced it, which is where it belongs.
Three of the twenty Exception blocks this change set accumulated had already been
committed when the audit ran, so they were invisible to it: an audit that
enumerates git status stops seeing a file the moment it is committed. Two further
details are worth recording because they are what let these survive.
The purge pattern (^|\{)#\s*Exception: does not match an indented comment, so a
block nested inside YAML never matched while the report claimed none remained.
Anchoring it as ^\s*(#|//|--|\{#)\s*Exception: raised the count from 12 to 17.
One of the three was written as a paragraph inside a Python docstring rather than
as a comment. Docstrings are an allowed kind, so no comment grep matches them and
the repository's own comment lint passes over them; using the allowed kind to
carry the banned label is the same rule-gaming in a syntax the checks do not
cover.
Verified the deletion removed nothing else: the diff contains zero insertions,
every touched YAML file still parses, and the unit suite and comment lint are
green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The context_agent ExApp calls back into the same php-fpm pool it is being
proxied from: its UserAuthMiddleware resolves the bearer against
/ocs/v2.php/cloud/user while a child is already parked forwarding the MCP
request out to the ExApp. Every in-flight MCP request therefore occupies TWO
children. Below twice mcp.limits.concurrent_requests the two legs deadlock, and
because the ExApp's callback carries no timeout the pool never recovers -- which
is why the Context Agent sat unhealthy rather than merely slow.
services.nextcloud.performance.php.fpm now sizes the pool explicitly:
max_children 16, start_servers 4, min/max spare 4/8. request_terminate_timeout
is 900s and must not drop below the proxy's fastcgi_read_timeout; lower kills a
child while nginx is still waiting on it, equal means only work nginx has
already abandoned gets reclaimed.
The readiness path was spending the same scarce workers. /healthz/ready had no
location of its own, so it fell through to `location /`, was rewritten to
/index.php/healthz/ready and booted a full Nextcloud request on a php-fpm child
once per probe interval against a pool of five. It is an exact-match location
returning 200 from nginx now, with access_log off, and never reaches PHP.
The pool file is rendered through render_replicated_templates.yml so swarm peers
receive it too, and mounted from meta/volumes.yml at a path that sorts after
zz-docker.conf, which reopens the [www] section last -- mount it earlier and the
image's own values win while the deploy looks correct.
The context_agent healthcheck moves from a hand-written probe in compose.yml.j2
to the shared container_healthcheck lookup, declared as flavor curl on path
heartbeat with start_period 20m. The ExApp installs dependencies on first boot
and legitimately outlives a 60s start period, which is the other half of why it
reported unhealthy.
Not verified: none of this has deployed. max_children 16 is derived from twice
concurrent_requests rather than measured against real memory use, and the
mount-ordering requirement is read from the image layout rather than confirmed
against a merged pool config in a running container -- if it sorts wrong the
whole change is inert while appearing correct. request_terminate_timeout 900s
also means a genuinely stuck request now holds a child for fifteen minutes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The api_key write was guarded by `(nextcloud_openai_litellm_api_key |
default('')) | length > 0`, so an empty credential SKIPPED the task. Nextcloud
kept whatever integration_openai api_key it already held, or none, and the
deploy reported success -- exactly the silently unauthenticated AI surface
requirement 036 forbids.
The three tasks now sit in one block gated on nextcloud_openai_use_litellm,
which also satisfies the repeated-when lint, and the shared consumer contract
runs first. That contract is what turns the empty-key case from a skip into a
failure, and it also asserts the base URL stays inside the deployment, which
this role never checked.
The key lookup drops its '' fourth argument and is strict again. Ordering
matters: the contract fails first with a readable message, so the strict lookup
is never the thing that raises.
Nextcloud was the last of the nine confirmed roles without the shared contract.
The occ call keeps the space form, `--value '{{ ... }}'`, deliberately. A
foreign commit on fork/feature/svc-net-tor (bbf4ca09cc) converts that to
`--value={{ ... | quote }}` because Symfony ArgvInput reads a token beginning
with a dash as an option. Adopting it here was considered and rejected on
measurement: shlex.quote leaves an sk-prefixed value completely unquoted, so
both forms emit an identical argv token and the change is inert at this call
site; git merge-file yields one conflict hunk either way, because the two sides
differ in the variable name rather than the quoting; and ansible's quote filter
maps None to '', which would turn a null credential into `--value=` -- an empty
write that exits 0 and matches changed_when, defeating the very guarantee this
commit adds. The equals form is still worth having where values are minted
externally, notably integration_openproject_provision.yml and
integration_suitecrm_provision.yml, and belongs in its own change covering
every call site rather than the one where the failure cannot fire.
Not verified: this role has not deployed since the change. On the branch where
Open WebUI is present, nextcloud_openai_use_litellm is false and no contract
runs at all; traffic stays in-cluster because Open WebUI is itself
gateway-backed, but criterion 4 is not asserted on that path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AI Agent nodes read their endpoint from a stored credential, not from the
environment, so the gateway base URL and this role's virtual key have to land in
n8n's credential store. files/provision_ai.py logs in as the owner and upserts an
openAiApi credential under the deterministic name infinito:litellm.
The write goes through the internal /rest routes because that is the only place
the pinned 1.95.3 exposes credential creation: the public /api/v1 key this role
already mints for MCP carries workflow scopes and cannot create a credential.
That is why this is a second script rather than another call inside
provision_mcp.py.
n8n encrypts credential data at rest and never returns it, so there is nothing
to compare against and the managed credential is overwritten rather than
reconciled. The script prints CHANGED or OK so the task can report change state
without pretending to have diffed a value it cannot read.
env.j2 sets only N8N_AI_ENABLED. An earlier revision of this work templated
N8N_INSTANCE_AI_MODEL_URL, which does not exist in 1.95.3 -- it was invented
rather than measured, and has been removed along with the requirement table row
that repeated it. The one real AI environment variable in that release,
N8N_AI_ASSISTANT_BASE_URL, expects n8n's own assistant service rather than an
OpenAI-compatible endpoint, so the credential store is the only usable surface.
vars/main.yml also carries N8N_MCP_CREDENTIAL_NAME. It belongs to the MCP work
rather than to this change, but a file cannot be split across commits and this
order is the safe one: the MCP tasks land later and find the constant already
present, where the reverse would reference a variable that does not exist yet.
Verified: the spec runs rather than skipping -- it appears in the variant-2
JUnit as "administrator: the managed openAiApi credential answers a prompt
through the in-cluster gateway".
Not verified: I confirmed it executed, not its verdict. The owner-cookie login
is also unexercised on an sso-only variant, and because the credential is
rewritten every run, an operator's manual edit is replaced without notice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
WordPress has no AI surface in core, so the requirement applies to whichever
connector the role pins rather than leaving the choice to a site administrator.
The pinned connector is ai-engine, whose `custom` engine type takes an
OpenAI-compatible base URL and an API key per environment.
AI Engine keeps every setting in a single mwai_options array and its REST route
replaces that array wholesale, so the deploy merges instead of assigning:
ai_engine_configure.php filters ai_envs by id, re-appends the managed
environment, and leaves every other environment and API key untouched. A
straight write would delete configuration this deploy never owned.
Both scripts are fed to `wp eval-file -` over stdin with their input passed as
an environment variable. The first attempt used `wp eval` with the code inline,
which cannot survive the quoting: a YAML folded scalar hands Jinja a literal
backslash sequence and PHP fails to parse. That was reproduced in the container,
along with the detail that `wp eval-file -` requires the opening <?php tag,
before the approach was changed. Passing the payload as base64 in an env var
means no shell quoting touches either the code or the virtual key.
ai_engine_endpoint.php reads back the endpoint AI Engine actually stored for the
managed environment, so the assertion compares the plugin's own state rather
than the payload the deploy just sent.
Not verified: the configure script has never run against a live AI Engine
install, so the merge is reasoned from the option shape rather than observed,
and addons/ai-engine.spec.js has executed zero times because the addon bridge
gate resolved svc-ai-litellm against the Playwright role list and skipped it.
ai-engine is third-party, so a change to its option schema would leave this
writing a shape the plugin no longer reads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The extension is declared as an addon gated on services.litellm.enabled and
bridged to litellm, so it exists only where the gateway is deployed.
VisualEditorPlus is declared alongside it because AIEditingAssistant's
extension.json hard-requires it and the mediawiki image does not bundle it;
without it the AI extension fails to load, which makes the two one feature
rather than two.
02_extensions.yml previously ran only when OIDC was enabled, since OIDC was the
only extension consumer. Its gate now also fires on litellm, otherwise a
litellm-only variant installs no extensions at all and the AI surface is
silently absent from a deploy that reports success.
utils/ai_gateway.yml verifies the shared contract and then reads the running
instance back with `maintenance/run.php getConfiguration --format=json`,
asserting that the active provider is open-ai, that the connection URL is the
in-cluster gateway, and that the model is the one the gateway serves. It reads
MediaWiki's own resolved configuration rather than the template that produced
it, so a template that renders correctly but never lands still fails. The
include runs under IS_STACK_HOST because the read goes through the container.
$wgHTTPTimeout is raised to 120s. It is global to every MediaWiki outbound
request and core defaults to 25s, which is below a cold local model's time to
first token, so the first prompt after a deploy would abort. The trade-off is
deliberate and applies to all outbound calls, not only inference.
MEDIAWIKI_EXT_NAMES derives the loaded extension list from the materialised
addons mapping filtered on enabled, so LocalSettings.php.j2 loads exactly what
the variant turned on and there is no second list to keep in sync.
Not verified: none of this has completed a run. AIEditingAssistant.spec.js has
executed zero times, because the addon bridge gate resolved svc-ai-litellm
against the Playwright role list and skipped it, and no log yet shows the
getConfiguration assertion passing. Widening the 02_extensions.yml gate is a
real behaviour change for litellm-only variants, which have never installed
extensions before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Discourse AI is configured through an LlmModel row rather than settings, so the
addon upserts one via rails: provider open_ai, the gateway's chat-completions
URL, the model the gateway serves, and this role's virtual key. It then binds
that row to ai_default_llm_model and to the bot, because a model that exists
but is bound to nothing serves no surface.
llm_models.url is POSTed verbatim, so DISCOURSE_LITELLM_CHAT_ENDPOINT carries
the full /v1/chat/completions path rather than a base URL. Every other consumer
in this requirement takes a base and appends; getting this one wrong produces a
404 at prompt time, long after a green deploy.
Three things had to be cleared before any of that could work.
Discourse refuses to call a private address unless the host is in
allowed_internal_hosts, and it fails as an opaque provider error rather than as
a blocked request. The script appends the gateway host to that list and leaves
existing entries alone, which matters because the OIDC bootstrap in
config.yml.j2 appends the discovery host to the same pipe-separated setting.
Both sides check membership before adding, so they compose instead of
clobbering each other.
The launcher-managed container sits outside the compose graph and does not join
the gateway network on its own. handlers/stack_host.yml adds
DISCOURSE_LITELLM_NETWORK to its network list and config.yml.j2 passes the
matching --network flag, both only while litellm is enabled.
The bootstrap aborted before the plugin could install at all. after_code cloned
every enabled plugin unconditionally, but a bundled plugin already sits in
$home/plugins, git exits non-zero on the clone, and the whole bootstrap stops.
discourse-ai is bundled in current images, so it now skips plugins whose source
is bundled.
mem_limit goes from 2g to 4g. `rails r` boots a second Rails inside the
container already running unicorn and sidekiq, and at 2g the kernel kills it
mid-run: measured in deploy-036-11 and again in deploy-ai-v2-7, where the task
died with "Killed" and the deploy ended at exit 137. Upstream documents 4 GB as
Discourse's own minimum, so this corrects an under-provisioned limit rather than
working around the task. The variant resource budget still passes.
Verified: the OOM with the exact task and exit code across two independent runs,
the bundled-plugin abort, and that the budget test holds at 4g.
Not verified: the registration has never completed, because every attempt so far
died at the OOM first. The eight-fact read-back assertion and
addons/discourse-ai.spec.js are both entirely unexercised, the spec because the
addon bridge gate had it skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moodle 4.5 ships aiprovider_openai in core, where every action carries its own
endpoint rather than one shared base URL, so the provider is configured per
action. aiprovider_openaicompatible would be the natural fit but requires
Moodle 5.0, which the role does not pin, so the per-action endpoints are the
only surface available on this branch. MOODLE_LITELLM_SETTINGS carries that as
a list of component/name/value triples and also enables the courseassist and
editor placements: a configured provider with no enabled placement exposes no
AI surface to any user, so both halves are required for the feature to exist.
An earlier revision set a global `enableai`, which does not exist in 4.5.
tasks/utils/ai_gateway.yml verifies the shared gateway contract, writes each
setting through admin/cli/cfg.php, then purges caches. Without the purge Moodle
keeps answering from its config cache and the new endpoint appears to have been
ignored.
The egress rule is the half that is easy to miss. Moodle's cURL security helper
blocks private address ranges outright, so a correctly configured in-cluster
endpoint is still refused, and it surfaces as a generic provider error rather
than as a blocked request. 01_manager_ops.yml therefore writes
curlsecurityblockedhosts and curlsecurityallowedport, appending the gateway's
port only while litellm is enabled. This deliberately narrows a security
control: 172.16/12 and 192.168/16 become reachable so the container network
resolves, while 10.0.0.0/8, link-local and loopback stay blocked. The README
records the trade-off.
MOODLE_EGRESS_NEWLINE is double-quoted on purpose. Moodle stores both settings
one entry per line; a single-quoted YAML scalar hands Jinja a literal
backslash-n, the whole list collapses into one unparseable token, and the net
effect is that every port is blocked rather than the intended two. Verified in
the container with negative controls: keycloak and litellm reachable,
10.208.0.5 and 127.0.0.1 still refused.
The spec opens a site-administrator session -- through Keycloak when sso is on,
because 04_oidc.yml rewrites every account to auth: oidc and no password
reaches it then, and through the native form otherwise -- asserts the endpoint
is http on an undotted host that is not Moodle's own domain, then calls
aiplacement_editor_generate_text and requires a non-empty completion. That last
step is a prompt actually answered through the gateway, not a config read-back.
Verified: the deploy-time read-back assertion ran ok in a variant-2 round and
the spec executed rather than skipping. The egress newline defect and its fix
were both measured in-container.
Not verified: I confirmed the spec ran but not that its final assertion passed,
and the cfg.php loop is changed_when: false, so the deploy log cannot show
whether a setting actually moved.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous contract could not fail. Its guard and its closing assertion both
keyed on apiURL -- the one field the merge itself writes -- so once any run had
set the URL, every later run skipped the merge on a false_condition and still
reported all assertions passed. The live row shows what that hides: an apiKey of
`sk-` followed by 48 literal `a` characters, which the gateway answers 401 to,
and defaultModel `llama3:latest` while the round's gateway serves only
`smollm2:135m`. The merge task is `skipping` in every deploy log on this host.
The read now compares the whole desired service and bot with jsonb equality and
answers `converged` or `stale`, so a drifted key or model is drift the guard can
see. jsonb equality is key-order independent, which keeps the check from
depending on how Jinja happened to serialise the dict. Both SQL templates render
from the same two role vars, so read and write cannot disagree about what
correct means. The `jsonb_typeof(...) = 'array'` guards stay: before the first
write those keys exist as JSON null, coalesce does not fire on JSON null, and
jsonb_array_elements then raises "cannot extract elements from a scalar".
A row edit cannot create the bot account, which is why the Playwright spec 404s
on user `agent`. In mattermost-plugin-agents v2.5.1 the config lives in an
in-memory atomic pointer; bots.EnsureBots() is the only path that calls
pluginAPI.Bot.Create, and it runs from OnActivate and from the config update
listener. Writing straight into agents_confighistory reaches neither. The role's
only reload lever was `notify: container recreate`, which never fired -- the
container is unchanged with RestartCount 0 while the config row was written
inside it. The plugin is now cycled explicitly through mmctl, and the enable step
moves into tasks/utils/agents/enable.yml so the MCP path and the AI path share
one definition.
The closing gate is deliberately not another read of what we wrote: it polls the
bots-to-users join until the agent account exists. That table is written by the
plugin and never by this role, so it cannot pass by echoing the deploy's own
work. The reload fires on stale config OR a missing bot, which covers the state
this stack is actually in, where the config matched but no account existed.
Verified: the stale row, the absent bot account and the always-skipping merge are
all measured on the live stack; the EnsureBots path was read from upstream v2.5.1
source rather than inferred; every changed task file parses.
Not verified: none of this has executed yet. Specifically open are whether
`MATTERMOST_LITELLM_SERVICE | to_json` renders nested Jinja inside a dict var,
whether the converged/stale SQL is valid against live Postgres, and whether a
disable/enable cycle triggers EnsureBots on this version.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Matrix was the one role that already shipped an AI credential and pointed it at
the vendor, and four separate defects kept it from reaching the gateway. Each
one hid the next, so fixing fewer than all four still leaves a surface that
reports green and answers nothing.
The schema rejected the gateway by construction.
`chatgpt_bridge_openai_api_key` validated `^sk-[a-zA-Z0-9]{40,}$` -- OpenAI's own
key format -- with algorithm `plain`. A LiteLLM virtual key cannot satisfy that
pattern, so no amount of configuration could have worked. It is `sk_prefixed`
with a length check now, and a `litellm_api_key` entry joins it in the same
shape the other eight consumers use.
The bridge reached Synapse over the public ingress. MATRIX_HOMESERVER_URL was
MATRIX_SYNAPSE_URL, an https URL on the canonical domain, so every login left
the container network, crossed openresty and came back. A 502 taken while
Synapse was still starting still consumes an rc_login.address slot, and once
that bucket empties the bridge dies on M_LIMIT_EXCEEDED and restarts into the
same wall: measured at 139 restarts on the local stack. MATRIX_SYNAPSE_LOCAL_URL
addresses the container directly over plain http on the internal port.
MATRIX_SYNAPSE_URL stays public where it has to be -- public_baseurl, Element,
and the mautrix bridges, which are a separate concern and untouched here.
Nothing ordered the bridge behind Synapse. Synapse itself depends on the mautrix
bridges, but matrix-chatgpt-bot is not one of them, so it was free to start
first and spend its login quota on a homeserver that was not listening. It now
waits for `service_healthy`.
The specs could not read their own configuration. templates/playwright.env.j2
renders MATRIX_PLUGINS_JSON through dotenv_quote, and both
test-litellm-chatgpt.js and test-bridge-roster.js called JSON.parse on the raw
value. The parse returned a string rather than an object, so `plugins.chatgpt`
was undefined: the litellm spec aborted after 0.99s claiming the plugin was off,
and the roster spec quietly downgraded itself to a skip. Both decode through
decodeDotenvQuotedValue first.
00_core.yml verifies the shared gateway contract before dispatching to either
flavor, so an empty virtual key or a base URL that resolves outside the
deployment fails ahead of any container work.
Verified: the decode fix by execution -- the spec previously died at 0.99s on
plugins.chatgpt and now runs through to the invite poll at line 202. The
contract assertions and the credential ran ok in a variant-2 deploy, which also
exercises the schema change end to end.
Not verified: the local URL and the health gate have not deployed yet, so
whether they break the crash loop is still open. Behind them sits a separate
defect this commit does not address -- matrix-synapse's process does not trust
the platform CA (openresty logs TLS alert 48 from its address) and
RegularPolicyForHTTPS caches one SSL context for the process lifetime, which
breaks every outbound HTTPS call Synapse makes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Zammad 6.5.0 has no AI surface to configure: neither `ai_provider` nor
`ai_provider_config` exists in that line, so the role could not point Smart
Assist anywhere. 7.1.2 is the newest version whose contract is still that
setting pair; from 7.2 the provider moves into an `AI::ProviderConnection`
record and the shape changes, so the bump lands here rather than on latest.
Elasticsearch is unaffected: 7.1.2 accepts >= 7.8, < 10 and the role's 8.13.4
pin stays.
The provider is written by piping files/scripts/apply/ai_provider.rb into
`rails r -` over stdin, with url, model and token passed as environment
variables instead of interpolated into Ruby source, so no quoting path can
mangle a virtual key. Order inside the script is load-bearing and documented
there: `ai_provider` is the boolean "a provider is configured" and its
validator reads `ai_provider_config`, so the config hash has to land first or
the flag is refused. `UserInfo.current_user_id` is set because `Setting.set`
needs an acting user.
This is the strongest evidence in the requirement, because it is not a
read-back of something the deploy just wrote: `Setting.set("ai_provider_config")`
runs `AI::Provider::CustomOpenAI.ping!` upstream, so the write itself is a live
round trip. An unreachable gateway or a rejected key fails the task.
Two consequences of the bump ride along, because the version does not work
without them.
`/api/v1/getting_started` answers 403 from Zammad 7.0 on, so the Playwright
helper could no longer lift a csrf token there; it uses `/api/v1/signshow`,
the unauthenticated seed. Its old fallback read `csrf_token` from a JSON body
that endpoint never returns, so the failure surfaced as a confusing throw
rather than the 403 it was.
env.j2 drops the hand-built DATABASE_URL in favour of POSTGRESQL_OPTIONS
`?pool=`. Zammad composes its DSN from the POSTGRESQL_* parts itself, so the
explicit URL both duplicated and competed with them while leaving the
connection pool at its default.
The spec decodes ZAMMAD_LITELLM_BASE_URL and ZAMMAD_LITELLM_MODEL through
decodeDotenvQuotedValue. Without it the values arrive still wrapped in the
literal quotes dotenv_quote adds and the strict comparison can never match in
compose mode -- measured as expected `"\"http://litellm:4000/v1\""` against
received `http://litellm:4000/v1`.
Verified: the spec passes against the live stack (1 passed, via
scripts/tests/e2e/rerun-spec.sh --grep=Assist, which restages the spec from the
repo), and both the contract and the Smart Assist write ran ok in a variant-2
deploy. `make quality` green on test-lint, test-unit, test-integration,
test-external.
Not covered: the no-gateway path (ZAMMAD_LITELLM_ENABLED false) has not run
since the bump, and Zammad features this deploy does not exercise -- email
channel, LDAP sync, ticket workflows -- are unproven on 7.1.2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A role that enables its native AI surface can reach the gateway with an empty
credential, or point its base URL at something that resolves outside the
cluster, and still deploy green. Both are silent: the surface simply answers
from somewhere else, or not at all. This adds the shared contract that makes
either state fail the deploy, so a consuming role inherits one definition of
"inside the deployment" instead of restating it.
The key assertion refuses an empty credentials.litellm_api_key while
services.litellm is enabled. Without it the gateway leg is unauthenticated and
nothing in the run says so.
The URL assertion is four independent checks, each able to fail on its own:
the scheme is plain http, the hostname carries no dot, the hostname is not the
consuming role's own domain, and the port is the gateway's internal port rather
than its published one. The undotted check is the load-bearing one. The
container dns-search suffix completes a dotted host into the public FQDN, so
`litellm.infinito.example` leaves through the ingress while looking local in
the config; only a bare service name stays on the container network. The
own-domain check catches a role that points its AI surface back at itself, and
the internal-port check catches a config that would work only via the host
bind.
An earlier revision of this file compared two values that both came from the
same lookup and therefore could never fail. Every check here is falsifiable
against the value the role actually deploys.
group_vars carries the SPOT the contract measures against.
LITELLM_OPENAI_BASE_LOCAL_URL appends the OpenAI-compatible /v1 suffix once,
rather than in each role. LITELLM_CHAT_MODEL derives the model from whichever
backend is deployed -- an ollama preload model with embedding models rejected,
else lmstudio, else openrouter -- resolved through `lookup('deployment').running`
so it reflects what is running rather than what the inventory could contain.
Contract and vars land before the roles that include them; the reverse order
would commit roles referencing a file that does not exist yet.
Verified: the contract's assertions ran green at runtime for matrix,
mattermost, moodle and zammad. Zammad's is the strongest evidence, since
`Setting.set('ai_provider_config')` triggers upstream
`AI::Provider::CustomOpenAI.ping!` and therefore a live gateway round trip.
`make quality` green on test-lint, test-unit, test-integration, test-external.
Not covered: nothing yet forces a role that declares services.litellm to
include this contract, so a future consumer can still skip it silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both halves of the Playwright harness used "ships a Playwright suite" as a
proxy for "is deployed". The proxy is structurally wrong in opposite
directions, and each error hid itself.
discover_playwright_roles.py finds roles by globbing
`templates/playwright.env.j2`, so a role without that template can never
appear in TEST_E2E_PLAYWRIGHT_APPS no matter how deployed it is.
addon_env_flags resolved its bridge-partner check against exactly that list,
so every addon declaring `bridges: [litellm]` (5) or `bridges: [ldap]` (4)
gated itself off: svc-ai-litellm and svc-db-openldap ship no suite. The
affected specs reported as skipped, which is indistinguishable from passing
in a `failed=0` PLAY RECAP -- discourse-ai, AIEditingAssistant and ai-engine
have never once executed. Measured: `ls roles/svc-ai-litellm/files/playwright`
does not exist and the role is absent from the runner's "Playwright-enabled
roles" line, while `grep svc-ai-litellm ~/inventories/localhost-2/devices.yml`
returns a group entry. Two sources, opposite verdicts.
The gate now reads `group_names`, the round's actual closure. It stays
fail-open when that cannot be resolved: `_resolve_deployed_roles` returns None
for an empty set, so contexts without an inventory skip bridge gating exactly
as before rather than flipping every bridged addon false. The change is
monotone -- TEST_E2E_PLAYWRIGHT_APPS is the discovered roles intersected with
`running`, and `running` is a subset of `groups`, so a flag can only move
false -> true and no spec that ran before is newly skipped.
The second half is the mirror image. test_e2e_playwright_only_roles defaulted
to `lookup('deployment').groups`, the whole round closure, so `apps=` deploying
three roles still ran every spec in the round; specs for roles left on stale
state then failed and masked the result of the deploy that actually ran. It is
`.running` now: the whitelist when a targeted deploy sets one, the inventory
groups otherwise, leaving full runs unchanged.
The two lookups deliberately pull in opposite directions. Addon gating asks
whether the partner exists anywhere in the deployment and needs the full
closure; run scoping asks what was just deployed and needs the subset.
Conflating them was the shared defect.
defaults/main.yml keeps the same knobs and values apart from only_roles; its
mid-file narration moved into a Param header so the comments policy passes on
a file that is now touched.
Verified: 23 unit tests including a guard that fails on the old path
(group_names unread -> gating skipped -> true, where the test demands false),
and `make quality` green on test-lint, test-unit, test-integration and
test-external, read from the per-target table rather than the exit code.
Not yet verified live: the three unskipped specs have still never run, and
`group_names` is per-host, so a swarm delegate whose membership differs from
the stack host is unexercised.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two arch swarm jobs in fork run 32667447816 died in the ganesha AUR
install at 07:41Z, 42s apart, on two different network errors. All three
EXTERNAL_FETCH attempts ran and lost - the ~40s window is sized for
mirror pools where a bad member rotates out in seconds, but the AUR is
one origin host: when it flaps, it flaps for minutes. A sibling job
fetched the same package successfully 2.5 minutes earlier, and the
module (kewlfft.aur 0.13.0) hardcodes the AUR URLs with no timeout or
mirror knob, so a wider client-side window is the only available layer.
6x60s rides out a multi-minute flap and costs its 5 minutes only when
the AUR is actually down; every mirror-pool source keeps the tighter
EXTERNAL_FETCH.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The index-based probe (d71f34464) checks names against what stable
actually ships, and three declarations never did: stable carries no
'firefox' binary (only firefox-esr), openjdk-11 left after bullseye, and
the Qt GUI ships as 'wireshark' rather than wireshark-qt. Each id has
exactly one consumer (dev-java role, admin-network and dev workstation
bundles), and installing any of the three on Debian stable failed with
'unable to locate package' - these can only go from broken to working.
default-jdk follows the distro default like fedora's
java-latest-openjdk-devel already does; the registry never agreed on a
major across distros (Arch pins 11, RedHat 21). Ubuntu keeps 'firefox'
via a distro override because noble ships no firefox-esr.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The per-package endpoints are anti-bot territory: packages.debian.org shed
request-level 503s for >13h (2 CI runs and 2 local quality gates red, 7
resp. 91 probes unverified), and archlinux.org connection resets are the
chronic class 00784105f already fought. One Packages.xz/.db fetch per
distro answers every probe from a CDN meant for machines, through the
existing _get retry/semaphore layer; the index is cached per run behind a
lock, and an unreadable index stays a loud unknown.
Debian goes over http because the dev container hijacks deb.debian.org
onto the package-cache frontend, which serves the name on port 80 only -
the same transport the stack's own apt list uses. The stricter index also
exposed three registry declarations that never existed in Debian stable;
they are fixed in the follow-up commit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The coherence step wrapped a blocking 'systemctl try-restart' in a 120s
budget, one second short of the wedge-recovery cycle the unit performs
when the liburcu-bp startup wedge (f10a4a181) fires - observed in run
32657210518, where the Akaunting drill died on rc=124 while ganesha came
back healthy 4s later. 240s covers even the uncapped 123s cycle; with the
abort grace capped at 10s (64b39e227) one cycle is ~43s.
The '|| systemctl try-restart nfs-server' fallback also masked exactly
this failure: nfs-server.service is 'inactive dead' on a ganesha node,
try-restart on an inactive unit is a documented no-op exiting 0, so a
larger budget alone would have turned a SIGKILLed ganesha into a green
drill. Dispatching on the installed unit file keeps the kernel-NFS branch
and lets a real restart failure surface: a wedge leg now fails loud after
one ~43s cycle instead of passing on self-healing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sys-daemon sets DefaultTimeoutAbortSec=90s host-wide, and it governs the
phase after the SIGABRT that a wedged ganesha (the liburcu-bp futex wedge
from f10a4a181) blocks. One probe-fail -> SIGTERM -> SIGABRT -> SIGKILL ->
restart cycle therefore took 20+10+90+2+1 = ~123s; f10a4a181 sized its
gates on ~32s per cycle, assuming the SIGABRT kills. Observed in run
32657210518 (Akaunting swarm drill, fedora): the drill's 120s try-restart
budget expired one second before the SIGKILL landed.
TimeoutAbortSec=10 cuts the cycle to ~43s. The wedged process never dumps
core, so the shorter grace costs nothing there; a genuinely aborting
ganesha may truncate its core dump - the documented trade-off of a short
TimeoutAbortSec. Without sys-daemon the abort phase already fell back to
TimeoutStopSec=10, so standalone behavior is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
