fix(ai): resolve the gateway backend from the inventory, not from one host

LITELLM_BACKEND_ROLES read lookup('deployment').groups, which deployment.py
fills from group_names. That variable is host-scoped: it lists only the groups
the host currently being processed belongs to.

On a swarm round the manager carries svc-ai-ollama, svc-ai-lmstudio and
svc-ai-litellm while the workers carry three groups and none of them is an AI
backend. Both branches of LITELLM_CHAT_MODEL therefore missed on a worker and,
with OPENROUTER_API_KEY empty, the model resolved to the empty string.

Only one task made that visible. Every stateful task in the gateway include
carries delegate_to STACK_HOST plus run_once, so it templates in the manager's
scope and wrote the correct model into the database. The assert carries
neither, so it re-templated in each worker's own scope and failed there while
the manager passed.

The manager-only membership is not a placement decision. The swarm test
inventory is provisioned with a single --host MGR, so every role of the round's
closure gets that one host, and extend_inventory then widens only
svc-swarm-node, svc-storage-nfs-client, svc-storage-nfs-server and the round's
own application. The three AI roles declare no placement at all; they are
manager-only because nothing widened them.

Add an inventory selector next to the host-scoped one, built from the play-wide
groups magic var, skipping groups without hosts and Ansible's implicit all and
ungrouped, sorted so the cache key is stable. The key takes it too, or two
hosts with equal group_names but different inventories would share an entry.

The blast radius is two readers, both in 16_ai.yml, and both ask only whether
svc-ai-ollama or svc-ai-lmstudio is present. On the manager the new value gains
exactly svc-storage-nfs-server and svc-bkp-nfs-2-local, neither of which either
reader tests, so the manager's answer does not move and only the workers change,
to match it.

This is the third step on the same line. 46a16127fc set .running; b149ad0d22
corrected that to .groups because a targeted apps= round leaves only the
command-line roles in the whitelist. .groups solves the targeted round and not
the multi-host one, because group_names stays host-scoped either way.

Measured in a two-host play with a real group_vars file and the real lookup:
the worker resolves ollama false through group_names and true through the new
selector, the manager is true both ways, and the assert that failed in CI
passes on both hosts.

Requirement 036 named the host-scoped backend selection as the thing that has
to change and deferred it. That residual is closed, so the paragraph now
describes the selector in use and why the two rejected ones are wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/requirements/036-native-ai-gateway-integration.md

@@ -71,7 +71,7 @@ Excluded, with the reason recorded so the question is not reopened:
 - [x] No credential validation in `meta/secrets.yml` encodes a vendor key format, so a gateway-issued virtual key satisfies every AI credential the platform mints.
 - [x] `docs/requirements/027-integration-matrix.md` lists the AI surface of every role in the confirmed table.
 
-Consumers that must name a model (`web-app-moodle`, `web-app-matrix`) read `LITELLM_CHAT_MODEL`, which picks the first non-embedding preload model of `svc-ai-ollama` and mirrors the branch order of the gateway's own `config.yaml.j2`. The consumer resolves that branch from `lookup('deployment').groups`, the round's deployed closure, which is what `group_names` carries. `.running` is wrong here: a targeted `apps=` round sets it to the whitelist, and `svc-ai-ollama` is pulled in as a dependency rather than named on the command line, so both backend branches fall through to the `openrouter/auto` vendor default. The gateway then answers `Invalid model name passed in model=openrouter/auto` with HTTP 400 and no consumer can complete a prompt. On a multi-host round where the gateway and a backend land on different hosts they can diverge, and the gateway's host-scoped backend selection is what has to change; that belongs to [031](031-llm-gateway-model-backends.md).
+Consumers that must name a model (`web-app-moodle`, `web-app-matrix`) read `LITELLM_CHAT_MODEL`, which picks the first non-embedding preload model of `svc-ai-ollama` and mirrors the branch order of the gateway's own `config.yaml.j2`. The consumer resolves that branch from `lookup('deployment').inventory`, every group the round's inventory populates, read from the play-wide `groups` magic var. `.running` is wrong here: a targeted `apps=` round sets it to the whitelist, and `svc-ai-ollama` is pulled in as a dependency rather than named on the command line, so both backend branches fall through to the `openrouter/auto` vendor default. The gateway then answers `Invalid model name passed in model=openrouter/auto` with HTTP 400 and no consumer can complete a prompt. `.groups` is wrong for the mirror-image reason: it carries `group_names`, which is host-scoped, so on a multi-host round a worker that hosts neither the gateway nor a backend resolves an empty model while the manager resolves the right one.
 
 ### Which variant exercises the gateway
 

group_vars/all/16_ai.yml

@@ -11,7 +11,7 @@ LMSTUDIO_BASE_LOCAL_URL: "http://{{ lookup('config','svc-ai-lmstudio', 'services
 
 LITELLM_OPENAI_BASE_LOCAL_URL: "{{ LITELLM_BASE_LOCAL_URL }}/v1"
 
-LITELLM_BACKEND_ROLES: "{{ lookup('deployment').groups }}"
+LITELLM_BACKEND_ROLES: "{{ lookup('deployment').inventory }}"
 
 LITELLM_CHAT_MODEL: "{{ (lookup('config', 'svc-ai-ollama', 'services.ollama.preload_models') | reject('search', 'embed') | list | first) if 'svc-ai-ollama' in LITELLM_BACKEND_ROLES
   else ('lmstudio/default' if 'svc-ai-lmstudio' in LITELLM_BACKEND_ROLES else ('openrouter/auto' if ((OPENROUTER_API_KEY | string) | length > 0) else '')) }}"

plugins/lookup/deployment.py

@@ -10,6 +10,8 @@ _STATUS_CACHE: dict[tuple, dict[str, list[str]]] = {}
 
 _EPHEMERAL_RUNTIMES = frozenset({"act", "github"})
 
+_IMPLICIT_GROUPS = frozenset({"all", "ungrouped"})
+
 
 def _reset_cache_for_tests() -> None:
     _STATUS_CACHE.clear()
@@ -34,8 +36,13 @@ class LookupModule(LookupBase):
 
         whitelist = _coerce_to_list(vars_.get("APPLICATIONS_WHITELIST"))
         groups = _coerce_to_list(vars_.get("group_names"))
+        inventory = sorted(
+            name
+            for name, hosts in (vars_.get("groups") or {}).items()
+            if hosts and name not in _IMPLICIT_GROUPS
+        )
         runtime = str(vars_.get("RUNTIME", "")).strip().lower()
-        key = (tuple(whitelist), tuple(groups), runtime)
+        key = (tuple(whitelist), tuple(groups), tuple(inventory), runtime)
 
         cached = _STATUS_CACHE.get(key)
         if cached is not None:
@@ -47,6 +54,7 @@ class LookupModule(LookupBase):
             "whitelist": whitelist,
             "running": list(running),
             "groups": groups,
+            "inventory": inventory,
             "deployed": deployed,
             "runtime": runtime,
             "all": list(list_invokable_app_ids()),

tests/unit/python/plugins/lookup/test_deployment.py

@@ -22,7 +22,15 @@ class TestDeploymentLookup(unittest.TestCase):
         self.assertEqual(len(result), 1)
         self.assertEqual(
             set(result[0].keys()),
-            {"whitelist", "running", "groups", "deployed", "runtime", "all"},
+            {
+                "whitelist",
+                "running",
+                "groups",
+                "inventory",
+                "deployed",
+                "runtime",
+                "all",
+            },
         )
 
     @patch("plugins.lookup.deployment.list_invokable_app_ids")
@@ -88,6 +96,72 @@ class TestDeploymentLookup(unittest.TestCase):
         self.assertEqual(result[0]["running"], [])
         self.assertEqual(result[0]["all"], ["a"])
 
+    @patch("plugins.lookup.deployment.list_invokable_app_ids")
+    def test_inventory_is_play_wide_not_host_scoped(self, mock_list):
+        mock_list.return_value = []
+        worker = {
+            "group_names": ["web-app-mattermost"],
+            "groups": {
+                "svc-ai-litellm": ["mgr-01"],
+                "svc-ai-ollama": ["mgr-01"],
+                "web-app-mattermost": ["mgr-01", "wrk-01"],
+            },
+        }
+        result = LookupModule().run([], variables=worker)
+        self.assertEqual(result[0]["groups"], ["web-app-mattermost"])
+        self.assertEqual(
+            result[0]["inventory"],
+            ["svc-ai-litellm", "svc-ai-ollama", "web-app-mattermost"],
+        )
+
+    @patch("plugins.lookup.deployment.list_invokable_app_ids")
+    def test_inventory_skips_groups_without_hosts(self, mock_list):
+        mock_list.return_value = []
+        result = LookupModule().run(
+            [],
+            variables={
+                "group_names": [],
+                "groups": {"has-hosts": ["h1"], "empty": [], "also-empty": None},
+            },
+        )
+        self.assertEqual(result[0]["inventory"], ["has-hosts"])
+
+    @patch("plugins.lookup.deployment.list_invokable_app_ids")
+    def test_inventory_drops_the_implicit_groups(self, mock_list):
+        mock_list.return_value = []
+        result = LookupModule().run(
+            [],
+            variables={
+                "group_names": ["web-app-mattermost"],
+                "groups": {
+                    "all": ["mgr-01", "wrk-01"],
+                    "ungrouped": ["mgr-01"],
+                    "web-app-mattermost": ["mgr-01", "wrk-01"],
+                },
+            },
+        )
+        self.assertEqual(result[0]["inventory"], ["web-app-mattermost"])
+
+    @patch("plugins.lookup.deployment.list_invokable_app_ids")
+    def test_inventory_empty_without_the_magic_var(self, mock_list):
+        mock_list.return_value = []
+        result = LookupModule().run([], variables={"group_names": ["a"]})
+        self.assertEqual(result[0]["inventory"], [])
+
+    @patch("plugins.lookup.deployment.list_invokable_app_ids")
+    def test_cache_key_separates_hosts_with_equal_group_names(self, mock_list):
+        mock_list.return_value = []
+        first = LookupModule().run(
+            [],
+            variables={"group_names": ["a"], "groups": {"a": ["h1"]}},
+        )
+        second = LookupModule().run(
+            [],
+            variables={"group_names": ["a"], "groups": {"a": ["h1"], "b": ["h2"]}},
+        )
+        self.assertEqual(first[0]["inventory"], ["a"])
+        self.assertEqual(second[0]["inventory"], ["a", "b"])
+
     @patch("plugins.lookup.deployment.list_invokable_app_ids")
     def test_caches_same_input(self, mock_list):
         mock_list.return_value = ["a"]