computer-playbook/roles/test-e2e-cli/defaults/main.yml at fix/dev-python/python-upper-bound - computer-playbook - Infinito.Nexus Code Hub

kevinveenbirkenbach/computer-playbook

0 Fork 0

mirror of https://github.com/kevinveenbirkenbach/computer-playbook.git synced 2026-11-06 02:54:26 +00:00

Files

T

kevinveenbirkenbachandClaude Fable 5 d7e38514c9 fix(ci): bound every hang path that drove jobs into the 6h cancel

27 jobs of run 29147179243 burned their full 360-min budget. Three
systemic layers, all addressed here:

- The CLI e2e task ran the staged test.sh unbounded, and test.sh blocks
  in 'systemctl start' on a oneshot backup unit with no JobTimeout; on a
  wedged dockerd or a stalled NFS mount both freeze silently for hours.
  The task now runs the script under 'timeout TEST_E2E_CLI_TIMEOUT'
  (60 min via the timeout lookup, so TIMEOUT_FACTOR scales it) and the
  unit starts are bounded by BKP_TEST_HEALTH_TIMEOUT.
- The converge probe re-ran the full per-service diagnostics (service ps,
  service logs, inspect, journalctl) on EVERY retry, so 600 nominal
  2-second attempts cost 25-33s each and silently outlived the step
  timeout. The block is gone (the workflow-level container.py rescue
  snapshot already collects it recursively on failure) and every docker
  call is timeout-bounded. Same removal in the swarm collect script,
  which keeps only the NFS and render evidence container.py does not see.
- Playwright-collect, rescue diagnostics and Cleanup had no
  timeout-minutes, and teardown.sh calls docker against D-state nodes,
  so a failed job hung there for hours until the job cap killed it
  without artifacts. Every step is bounded now, every teardown call is
  timeout-wrapped, and the deploy steps (swarm 270, compose 300) leave
  room for diagnostics and cleanup below the 360-min cap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-07-12 04:06:17 +02:00

8 lines

221 B

YAML

Raw Permalink Blame History

 ---
 TEST_E2E_CLI_STAGE_BASE_DIR: "/tmp/test-e2e-cli"
 TEST_E2E_CLI_TIMEOUT: "{{ lookup('timeout', 3600) | int }}"
 allowed_applications: []
 TEST_E2E_CLI_ONLY_ROLES: "{{ allowed_applications }}"
 TEST_E2E_CLI_SKIP_ROLES: []