Files
computer-playbook/.github/workflows/test-environment.yml
T
kevinveenbirkenbachandClaude Opus 5 aee6d8dc4a ci(environment): capture rescue diagnostics on failure
The Environment matrix was the only CI path that produced no artifact at
all: no Playwright reports, no container state, no container logs. When
all five jobs of run 30254851245 failed on a dead prometheus backend, the
job log carried the 502 and nothing else, so the reason the container was
not serving could not be determined from CI at all.

Reuse the existing rescue-diagnostics SPOT instead of adding a second
capture path. Three things make it work here:

- the job mounts the runner docker socket, so `make compose-up` creates
  infinito_nexus_debian on the runner daemon and it outlives the `--rm` of
  the orchestrator container;
- the app stack sits one level deeper inside that container's own dockerd,
  and container.py recurses into any running container carrying python3
  and a container runtime;
- nothing tears the stack down on failure: 00_orchestrator.sh runs under
  `set -euo pipefail` with no trap, and 10_teardown.sh is the only
  `make compose-down` in the suite and never reached.

The image ref needs a slug because artifact names cannot contain `/` or
`:`, and the matrix values are refs like quay.io/centos/centos:latest.
`|| true` mirrors dedicated.sh: container.py always returns 1 by design so
a failing pipeline stays failing.

Probed locally against a running infinito_nexus_debian: recursion reaches
containers/infinito_nexus_debian/nested/, per-container inspect and logs
are captured with State.ExitCode, State.OOMKilled and RestartCount intact,
24 containers in roughly 20 seconds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 23:25:50 +02:00

113 lines
3.7 KiB
YAML

name: "💻 Test: Development Environment"
on:
workflow_dispatch:
workflow_call:
permissions:
actions: write
contents: read
packages: read
jobs:
test-environment:
name: 🧑‍💻 Environment (${{ matrix.dev_runtime_image }})
runs-on: ubuntu-latest
timeout-minutes: 345
strategy:
fail-fast: false
matrix:
dev_runtime_image:
- manjarolinux/base
- debian:bookworm
- ubuntu:latest
- fedora:latest
- quay.io/centos/centos:latest
env:
DEV_RUNTIME_IMAGE: ${{ matrix.dev_runtime_image }}
NIX_CONFIG: |
access-tokens = github.com=${{ secrets.GITHUB_TOKEN }}
INFINITO_CACHE_REGISTRY_MAX_SIZE: 16g
INFINITO_CACHE_REGISTRY_HOST_PATH: /mnt/cache/registry/mirror
INFINITO_CACHE_REGISTRY_CA_HOST_PATH: /mnt/cache/registry/ca
INFINITO_CACHE_PACKAGE_BLOBSTORE_MAX: 8g
INFINITO_CACHE_PACKAGE_HEAP: 1g
INFINITO_CACHE_PACKAGE_DIRECT_MEM: 1g
INFINITO_CACHE_PACKAGE_HOST_PATH: /mnt/cache/package/data
steps:
- name: 🧹 Free disk space
uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
with:
tool-cache: true
android: true
dotnet: true
haskell: true
large-packages: true
docker-images: true
swap-storage: false
- name: ⬇️ Checkout repository
uses: actions/checkout@v7.0.0
- name: "📥 Load rescue-diagnostics SPOT into job env"
shell: bash
run: |
set -euo pipefail
grep '^INFINITO_RESCUE_DIAGNOSTICS_BASE=' default.env >> "$GITHUB_ENV"
echo "INFINITO_ENV_SLUG=$(printf '%s' "${DEV_RUNTIME_IMAGE}" | tr '/:' '--')" >> "$GITHUB_ENV"
- name: ▶️ Run dev environment routine
shell: bash
run: |
set -euo pipefail
HOST_WORKDIR="$(pwd -P)"
docker run --rm \
-e GITHUB_ACTIONS=true \
-e GITHUB_REPOSITORY="${{ github.repository }}" \
-e GITHUB_REPOSITORY_OWNER="${{ github.repository_owner }}" \
-e GITHUB_SHA="${{ github.sha }}" \
-e DEV_RUNTIME_IMAGE \
-e INFINITO_DISTRO=debian \
-e INFINITO_IMAGE=infinito-debian \
-e INFINITO_PULL_POLICY=never \
-e INFINITO_BUILD=1 \
-e NIX_CONFIG \
-e INFINITO_CACHE_REGISTRY_MAX_SIZE \
-e INFINITO_CACHE_REGISTRY_HOST_PATH \
-e INFINITO_CACHE_REGISTRY_CA_HOST_PATH \
-e INFINITO_CACHE_PACKAGE_BLOBSTORE_MAX \
-e INFINITO_CACHE_PACKAGE_HEAP \
-e INFINITO_CACHE_PACKAGE_DIRECT_MEM \
-e INFINITO_CACHE_PACKAGE_HOST_PATH \
--network host \
-v "${HOST_WORKDIR}:${HOST_WORKDIR}" \
-v "/mnt/docker:/var/lib/docker" \
-v "/var/run/docker.sock:/var/run/docker.sock" \
-w "${HOST_WORKDIR}" \
"${DEV_RUNTIME_IMAGE}" \
bash -lc '
bash ./scripts/tests/environment/00_orchestrator.sh
'
- name: 🩺 Capture rescue diagnostics
if: failure()
timeout-minutes: 15
shell: bash
env:
INFINITO_RESCUE_DIAGNOSTICS_DIR: ${{ env.INFINITO_RESCUE_DIAGNOSTICS_BASE }}/${{ env.INFINITO_ENV_SLUG }}
run: |
python3 utils/diagnostics/container.py \
"${INFINITO_ENV_SLUG}" "environment suite failure" || true
- name: ⬆️ Upload rescue diagnostics
if: always()
uses: actions/upload-artifact@v7
with:
name: rescue-diagnostics-environment-${{ env.INFINITO_ENV_SLUG }}
path: ${{ env.INFINITO_RESCUE_DIAGNOSTICS_BASE }}/
if-no-files-found: ignore