test(zammad): ask from the rails container whether the AI gateway still answers

Zammad#2 of run 33923190220 hung three times for exactly 300 seconds and then raised AI::Provider::ResponseError. The rescue artifacts narrowed it only so far: ollama logged no generation during any of the three attempts, litellm had been up since 08:28:46 with zero restarts, and the deploy's own provider ping had passed from this very container minutes earlier at 10.84s. The path worked and then stopped, and nothing in the suite asked again at test time, so the evidence ends at 'somewhere between two swarm nodes'.

The probe asks from the position that failed. It runs through container exec in the rails container rather than in a fresh container on the same overlay, because an attachment a new container gets is not evidence that a long-lived one still has it, and that distinction is exactly what the artifacts could not make.

It asks the gateway for its model list, which needs no credential, so no token reaches the log and any HTTP status still proves the socket carried a request and an answer. REFUSED separates a dead listener from TIMEOUT's black hole, and both separate either from a gateway that answers while the consumer mishandles it.

Executed: all three verdicts reproduced against a live listener, a dead port and a socket that accepts and never replies; the budget is honoured at 10.1s wall for a stated 10s after splitting it across the connect and read phases, which each took the full value before; ruby and net/http are present in ghcr.io/zammad/zammad:7.1.2.

This does not make Zammad#2 green. It observes at test time, not during the assistant request, so a path that breaks after the probe still passes it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

roles/web-app-zammad/files/test/ai_gateway.sh

@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+# Can the rails container still reach the AI gateway when the tests run?
+#
+# The probe runs where the consumer runs rather than in a fresh container on the
+# same overlay, because an attachment that a new container gets is not evidence
+# that a long-lived one still has it. It asks for the model list, which needs no
+# credential: any HTTP status proves the socket carried a request and an answer,
+# so REFUSED separates a dead listener from TIMEOUT's black hole, and both
+# separate either from a gateway that answers and a consumer that mishandles it.
+#
+# Env (rendered into test.env from templates/test.env.j2):
+#   ZAMMAD_AI_PROBE_ENABLED    whether this round serves the AI path at all
+#   ZAMMAD_AI_GATEWAY_URL      the OpenAI-compatible base the role configured
+#   ZAMMAD_AI_PROBE_CONTAINER  the rails container to probe from
+#   ZAMMAD_AI_PROBE_TIMEOUT    seconds to wait before calling it a black hole
+set -euo pipefail
+
+: "${ZAMMAD_AI_PROBE_ENABLED:?}"
+: "${ZAMMAD_AI_GATEWAY_URL?}"
+: "${ZAMMAD_AI_PROBE_CONTAINER?}"
+: "${ZAMMAD_AI_PROBE_TIMEOUT:?}"
+
+if [ "${ZAMMAD_AI_PROBE_ENABLED}" != "true" ]; then
+  echo "SKIPPED the AI gateway is not served in this round"
+  exit 0
+fi
+
+if [ -z "${ZAMMAD_AI_GATEWAY_URL}" ] || [ -z "${ZAMMAD_AI_PROBE_CONTAINER}" ]; then
+  echo "FAILED the AI path is enabled but the gateway URL or the rails container is empty" >&2
+  exit 1
+fi
+
+probe=$(
+  cat <<'RUBY'
+require "net/http"
+require "uri"
+uri = URI.parse("#{ENV.fetch('URL')}/models")
+timeout = Integer(ENV.fetch("TIMEOUT"))
+http = Net::HTTP.new(uri.host, uri.port)
+http.use_ssl = uri.scheme == "https"
+phase = [timeout / 2, 1].max
+http.open_timeout = phase
+http.read_timeout = phase
+started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+begin
+  response = http.request(Net::HTTP::Get.new(uri))
+  elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started
+  puts format("ANSWERED %s in %.1fs", response.code, elapsed)
+rescue Errno::ECONNREFUSED
+  puts "REFUSED nothing is listening"
+rescue Net::OpenTimeout, Net::ReadTimeout
+  elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started
+  puts format("TIMEOUT no answer within %.0fs", elapsed)
+rescue StandardError => e
+  puts "ERROR #{e.class}"
+end
+RUBY
+)
+
+# nocheck: container-exec-resolver  address resolved by the caller and passed in
+verdict="$(container exec -i \
+  -e "URL=${ZAMMAD_AI_GATEWAY_URL}" \
+  -e "TIMEOUT=${ZAMMAD_AI_PROBE_TIMEOUT}" \
+  "${ZAMMAD_AI_PROBE_CONTAINER}" ruby -e "${probe}")"
+
+echo "${verdict}"
+
+case "${verdict}" in
+ANSWERED*) exit 0 ;;
+*)
+  echo "FAILED the rails container cannot reach the AI gateway, so a slow or failing" >&2
+  echo "       assistant request is a connectivity fault and not a model or timeout one" >&2
+  exit 1
+  ;;
+esac

roles/web-app-zammad/files/test/test.sh

@@ -1,6 +1,8 @@
 #!/usr/bin/env bash
-# CLI test for web-app-zammad: the MCP endpoint honours its declared contract.
+# CLI test for web-app-zammad: the MCP endpoint honours its declared contract,
+# and the rails container can still reach the AI gateway when the tests run.
 set -euo pipefail
 
 here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 bash "${here}/shared/mcp/run.sh"
+bash "${here}/ai_gateway.sh"

roles/web-app-zammad/templates/test.env.j2

@@ -1 +1,5 @@
 {{ lookup('template', lookup('path_absolute', 'roles/test-e2e-cli/templates/mcp/test.env.j2')) }}
+ZAMMAD_AI_GATEWAY_URL={{ ZAMMAD_LITELLM_BASE_URL if ZAMMAD_LITELLM_ENABLED | bool else '' }}
+ZAMMAD_AI_PROBE_CONTAINER={{ ZAMMAD_RAILS_CONTAINER_ADDRESS }}
+ZAMMAD_AI_PROBE_ENABLED={{ ZAMMAD_LITELLM_ENABLED | bool | lower }}
+ZAMMAD_AI_PROBE_TIMEOUT={{ lookup('timeout', 30) | int }}