mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
chore(ci): drop the cancelled-run rerun automation
The hourly sweep fought the rest of the CI control plane instead of helping it: it preempted itself (17 of the last 29 runs concluded cancelled 1-2s after their successor was created, for 8s of work behind a 74-80min queue wait) and shipped MAX_ATTEMPTS=0, which the script reads as no ceiling while its own README promised the opposite. The 24h age cutoff could not bound it either, because every rerun refreshes the run's updatedAt and so keeps it inside the window. Removes the workflow, the two scripts it was the sole consumer of, its unit test, and the CI_RERUN_* documentation that would otherwise point at a deleted file. The workflow is disabled in both origin and fork so no sweep fires before this lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
8aaac027d2
commit
3a1ef70cbb
8 files changed
-276
No files matched your search
@@ -1,42 +0,0 @@
|
||||
name: "⏰ Rerun: Cancelled Branch Runs"
|
||||
|
||||
concurrency:
|
||||
group: rerun-cancelled
|
||||
cancel-in-progress: true
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: "0 * * * *"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
actions: write
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
rerun:
|
||||
name: 🔁 Rerun the failed jobs of cancelled runs
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
|
||||
steps:
|
||||
- name: ⬇️ Checkout repository
|
||||
uses: actions/checkout@v7.0.0
|
||||
|
||||
- name: ⏱️ Decide whether this hour is due
|
||||
id: due
|
||||
if: ${{ github.event_name == 'schedule' }}
|
||||
shell: bash
|
||||
env:
|
||||
INTERVAL_MINUTES: ${{ vars.CI_RERUN_INTERVAL_MINUTES || '60' }}
|
||||
run: bash scripts/github/rerun/due.sh
|
||||
|
||||
- name: 🔁 Restart the failed jobs per branch
|
||||
if: ${{ steps.due.outputs.due != 'false' }}
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
REPOSITORY: ${{ github.repository }}
|
||||
MAX_ATTEMPTS: ${{ vars.CI_RERUN_MAX_ATTEMPTS || '0' }}
|
||||
MAX_AGE_HOURS: ${{ vars.CI_RERUN_MAX_AGE_HOURS || '24' }}
|
||||
run: bash scripts/github/rerun/cancelled_branch_runs.sh
|
||||
@@ -13,9 +13,6 @@ Repository variables are set under **Settings → Secrets and variables → Acti
|
||||
| `CI_RUN_ON_MAIN` | [entry-push-latest.yml](../../../../../.github/workflows/entry-push-latest.yml) | Pushes to `main` skip CI | `true` to run CI on `main` pushes too |
|
||||
| `CI_ENABLE_AUTO_UPDATES` | [cron-update.yml](../../../../../.github/workflows/cron-update.yml), [entry-pr-open-dependabot-close.yml](../../../../../.github/workflows/entry-pr-open-dependabot-close.yml) | Update jobs skipped; Dependabot PRs auto-closed | `true` to allow update PRs (workflow-driven and Dependabot) |
|
||||
| `INFINITO_PLAYWRIGHT_KEEP` | [call-test-deploy-compose.yml](../../../../../.github/workflows/call-test-deploy-compose.yml) | Playwright keeps trace, screenshot and video only when a test fails | `true` to keep them for every test (passing runs included) |
|
||||
| `CI_RERUN_INTERVAL_MINUTES` | [cron-rerun-cancelled.yml](../../../../../.github/workflows/cron-rerun-cancelled.yml) | Sweeps for cancelled runs every full hour | Minutes between two sweeps, rounded up to whole hours |
|
||||
| `CI_RERUN_MAX_ATTEMPTS` | [cron-rerun-cancelled.yml](../../../../../.github/workflows/cron-rerun-cancelled.yml) | Revives a run however often it is cancelled | Attempt count at which a run is left alone; `0` for no ceiling |
|
||||
| `CI_RERUN_MAX_AGE_HOURS` | [cron-rerun-cancelled.yml](../../../../../.github/workflows/cron-rerun-cancelled.yml) | Ignores runs untouched for more than 24 hours | Age in hours beyond which a cancelled run is left alone |
|
||||
|
||||
## `CI_CANCEL_IN_PROGRESS` 🛑
|
||||
|
||||
@@ -187,26 +184,3 @@ INFINITO_PLAYWRIGHT_KEEP: ${{ vars.INFINITO_PLAYWRIGHT_KEEP }}
|
||||
| `true` | Trace / screenshot / video kept for every test ✓ |
|
||||
| any other value | Trace / screenshot / video kept only on failure ✓ |
|
||||
|
||||
## `CI_RERUN_*` 🔁
|
||||
|
||||
Three variables tune [cron-rerun-cancelled.yml](../../../../../.github/workflows/cron-rerun-cancelled.yml), the sweep that restarts the failed jobs of a branch whose latest run was cancelled.
|
||||
|
||||
| Variable | Unset | Effect when set |
|
||||
|---|---|---|
|
||||
| `CI_RERUN_INTERVAL_MINUTES` | `60` | Minutes between two sweeps |
|
||||
| `CI_RERUN_MAX_ATTEMPTS` | `0` | A run at or above this attempt count is left alone; `0` means no ceiling, a run is revived however often it is cancelled |
|
||||
| `CI_RERUN_MAX_AGE_HOURS` | `24` | A run untouched for longer than this is left alone, so a sweep does not resurrect abandoned branches |
|
||||
|
||||
**How the interval works:**
|
||||
|
||||
A cron expression cannot read a repository variable, so the workflow is triggered every full hour and a gate lets only the hours that open the configured window through:
|
||||
|
||||
```yaml
|
||||
INTERVAL_MINUTES: ${{ vars.CI_RERUN_INTERVAL_MINUTES || '60' }}
|
||||
```
|
||||
|
||||
The interval is rounded up to whole hours and counted from 00:00 UTC, so `60` sweeps every hour, `180` at 00:00, 03:00, 06:00 and so on, and `1440` once a day at 00:00. One hour is the floor: anything below it sweeps hourly. Anything above 24 hours collapses to a daily sweep. Manual dispatch skips the gate entirely and always sweeps.
|
||||
|
||||
The gate reads the hour, not the minute, so a scheduled run that GitHub starts a few minutes late still counts for its hour.
|
||||
|
||||
**On the missing ceiling:** with `CI_RERUN_MAX_ATTEMPTS` unset, a run that is cancelled again after every revival is revived again, indefinitely. `CI_RERUN_MAX_AGE_HOURS` does not bound this, because each new attempt refreshes the run's `updatedAt` and so keeps it inside the age window. Set the variable to a positive number if a run should eventually be given up on.
|
||||
@@ -88,7 +88,6 @@ Trigger column legend: **auto** = fires automatically (push, pull_request, sched
|
||||
| [cron-cleanup-stale.yml](../../../../../.github/workflows/cron-cleanup-stale.yml): `⏰ Cleanup: Stale Repository Data` | Marks and closes stale issues and PRs, deletes inactive branches, and prunes old GHCR CI image versions. | auto (`schedule`: daily 00:00 UTC), manual | none |
|
||||
| [cron-update.yml](../../../../../.github/workflows/cron-update.yml): `⏰ Update: Versions` | Updates Docker image versions and other pinned dependencies; opens an update PR via a GitHub App installation token so PR-lifecycle events fire on the resulting PR. Both jobs are gated behind the `CI_ENABLE_AUTO_UPDATES` repository variable, see [configuration.md](configuration.md). Requires the `BOT_APP_CLIENT_ID` and `BOT_APP_PRIVATE_KEY` repository secrets, see [secrets.md](secrets.md). | auto (`push` to `main`, `schedule`: daily 00:30 UTC), manual | none |
|
||||
| [entry-pr-open-dependabot-close.yml](../../../../../.github/workflows/entry-pr-open-dependabot-close.yml): `🚫 Dependabot: Close while auto-updates disabled` | Auto-closes Dependabot PRs while `CI_ENABLE_AUTO_UPDATES` is not set to `true`, so Dependabot honours the same gate as the workflow-driven update jobs. See [configuration.md](configuration.md). | auto (`pull_request_target`: opened, reopened) | none |
|
||||
| [cron-rerun-cancelled.yml](../../../../../.github/workflows/cron-rerun-cancelled.yml): `⏰ Rerun: Cancelled Branch Runs` | Sweeps every branch: when its latest run ended `cancelled`, the failed jobs are rerun via `gh run rerun --failed` while the successful ones are kept, so a concurrency cancellation costs a resume instead of a full pipeline. The attempt ceiling, the age cutoff, and the sweep interval come from `CI_RERUN_MAX_ATTEMPTS`, `CI_RERUN_MAX_AGE_HOURS` and `CI_RERUN_INTERVAL_MINUTES`, see [configuration.md](configuration.md). Sweeps hourly by default, with no attempt ceiling. A new trigger cancels a sweep that is still running. Manual dispatch always sweeps, ignoring the interval. | auto (`schedule`: hourly, gated to the configured interval), manual | none |
|
||||
|
||||
## Changing workflows ✍️
|
||||
|
||||
|
||||
@@ -1,11 +0,0 @@
|
||||
# Run Reruns 🔁
|
||||
|
||||
Shell helpers that restart the unfinished parts of GitHub Actions workflow runs that were cancelled rather than completed.
|
||||
|
||||
## Scope 📋
|
||||
|
||||
This directory contains helpers invoked from scheduled workflows. They call the GitHub Actions API to rerun the failed jobs of a cancelled run, never the run as a whole, so the work that already succeeded is kept. A run is a candidate only while it is recent and below its attempt ceiling; both bounds are passed in by the calling workflow, because a sweep without them would resurrect abandoned branches and retry a permanently cancelled run forever.
|
||||
|
||||
Helpers MUST NOT rerun a run whose conclusion is `failure`. A real defect has to be fixed, not retried, and the [cancel](../cancel/README.md) helpers exist precisely so that a cancellation means "interrupted", not "broken".
|
||||
|
||||
For the workflow catalog that drives these calls see [workflows.md](../../../docs/contributing/tools/github/actions/workflows.md).
|
||||
@@ -1,74 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Restart the failed jobs of every branch whose latest workflow run ended
|
||||
# cancelled.
|
||||
#
|
||||
# Inputs via env:
|
||||
# GH_TOKEN token carrying actions:write on REPOSITORY
|
||||
# REPOSITORY owner/repo
|
||||
# MAX_ATTEMPTS leave runs alone once they reached this attempt count;
|
||||
# 0 keeps reviving a run for as long as it stays in range
|
||||
# MAX_AGE_HOURS leave runs alone whose last update is older than this
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
: "${GH_TOKEN:?Missing GH_TOKEN}"
|
||||
: "${REPOSITORY:?Missing REPOSITORY}"
|
||||
: "${MAX_ATTEMPTS:?Missing MAX_ATTEMPTS}"
|
||||
: "${MAX_AGE_HOURS:?Missing MAX_AGE_HOURS}"
|
||||
|
||||
if ! command -v gh >/dev/null 2>&1; then
|
||||
echo "ERROR: gh CLI not found." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v jq >/dev/null 2>&1; then
|
||||
echo "ERROR: jq not found." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cutoff="$(date -u -d "${MAX_AGE_HOURS} hours ago" +%Y-%m-%dT%H:%M:%SZ)"
|
||||
|
||||
echo "Rerunning cancelled runs updated after ${cutoff}, below attempt ${MAX_ATTEMPTS}"
|
||||
|
||||
branches="$(
|
||||
gh api --paginate \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
"/repos/${REPOSITORY}/branches?per_page=100" |
|
||||
jq -r '.[].name'
|
||||
)"
|
||||
|
||||
while read -r branch; do
|
||||
[[ -n "${branch}" ]] || continue
|
||||
|
||||
runs="$(
|
||||
gh run list \
|
||||
--repo "${REPOSITORY}" \
|
||||
--branch "${branch}" \
|
||||
--limit 20 \
|
||||
--json databaseId,conclusion,status,attempt,updatedAt
|
||||
)"
|
||||
|
||||
active="$(jq -r 'map(select(.status != "completed")) | length' <<<"${runs}")"
|
||||
if [[ "${active}" != "0" ]]; then
|
||||
echo "${branch}: ${active} run(s) still active, leaving the cancelled one alone"
|
||||
continue
|
||||
fi
|
||||
|
||||
run_id="$(
|
||||
jq -r --arg cutoff "${cutoff}" --argjson max "${MAX_ATTEMPTS}" '
|
||||
.[0]
|
||||
| select(.conclusion == "cancelled")
|
||||
| select($max == 0 or .attempt < $max)
|
||||
| select(.updatedAt > $cutoff)
|
||||
| .databaseId
|
||||
' <<<"${runs}"
|
||||
)"
|
||||
|
||||
[[ -n "${run_id}" ]] || continue
|
||||
|
||||
echo "${branch}: rerunning the failed jobs of cancelled run ${run_id}"
|
||||
if ! gh run rerun "${run_id}" --repo "${REPOSITORY}" --failed; then
|
||||
echo "${branch}: run ${run_id} could not be rerun, likely already restarted"
|
||||
fi
|
||||
done <<<"${branches}"
|
||||
@@ -1,28 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Emit due=true on GITHUB_OUTPUT when the current hour opens an interval window
|
||||
# counted from 00:00 UTC. Intervals are rounded up to whole hours.
|
||||
#
|
||||
# Inputs via env:
|
||||
# INTERVAL_MINUTES wanted interval between two sweeps
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
: "${INTERVAL_MINUTES:?Missing INTERVAL_MINUTES}"
|
||||
: "${GITHUB_OUTPUT:?Missing GITHUB_OUTPUT}"
|
||||
|
||||
hours="$(((INTERVAL_MINUTES + 59) / 60))"
|
||||
|
||||
if ((hours < 1)); then
|
||||
hours=1
|
||||
fi
|
||||
|
||||
hour="$((10#$(date -u +%H)))"
|
||||
|
||||
if ((hour % hours == 0)); then
|
||||
echo "$(date -u +%H:%M) UTC opens the ${hours}h window, sweeping"
|
||||
echo "due=true" >>"${GITHUB_OUTPUT}"
|
||||
else
|
||||
echo "$(date -u +%H:%M) UTC is inside the ${hours}h window, skipping"
|
||||
echo "due=false" >>"${GITHUB_OUTPUT}"
|
||||
fi
|
||||
Whitespace-only changes.
@@ -1,94 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from utils.cache.files import PROJECT_ROOT, read_text
|
||||
|
||||
SCRIPT = PROJECT_ROOT / "scripts" / "github" / "rerun" / "cancelled_branch_runs.sh"
|
||||
|
||||
|
||||
def _run_record(database_id: int, conclusion: str, status: str) -> dict:
|
||||
return {
|
||||
"databaseId": database_id,
|
||||
"conclusion": conclusion,
|
||||
"status": status,
|
||||
"attempt": 1,
|
||||
"updatedAt": "2026-08-10T15:46:00Z",
|
||||
}
|
||||
|
||||
|
||||
class TestCancelledBranchRuns(unittest.TestCase):
|
||||
def _run(self, runs: list[dict]) -> tuple[subprocess.CompletedProcess, list[str]]:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
stub_bin = Path(tmp) / "bin"
|
||||
stub_bin.mkdir()
|
||||
rerun_log = Path(tmp) / "rerun.log"
|
||||
gh = stub_bin / "gh"
|
||||
gh.write_text(
|
||||
"#!/bin/sh\n"
|
||||
'if [ "$1" = "api" ]; then printf \'[{"name":"b"}]\\n\'; exit 0; fi\n'
|
||||
'if [ "$1" = "run" ] && [ "$2" = "list" ]; then'
|
||||
f" cat {json.dumps(str(Path(tmp) / 'runs.json'))}; exit 0; fi\n"
|
||||
'if [ "$1" = "run" ] && [ "$2" = "rerun" ]; then'
|
||||
f' echo "$3" >>{json.dumps(str(rerun_log))}; exit 0; fi\n'
|
||||
"exit 1\n"
|
||||
)
|
||||
gh.chmod(0o755)
|
||||
(Path(tmp) / "runs.json").write_text(json.dumps(runs))
|
||||
|
||||
env = dict(os.environ)
|
||||
env["PATH"] = f"{stub_bin}:{env['PATH']}"
|
||||
env["GH_TOKEN"] = "t"
|
||||
env["REPOSITORY"] = "o/r"
|
||||
env["MAX_ATTEMPTS"] = "0"
|
||||
env["MAX_AGE_HOURS"] = "999999"
|
||||
|
||||
proc = subprocess.run(
|
||||
["bash", str(SCRIPT)],
|
||||
env=env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
reruns = read_text(str(rerun_log)).split() if rerun_log.exists() else []
|
||||
return proc, reruns
|
||||
|
||||
def test_revives_the_newest_cancelled_run_when_the_branch_is_idle(self) -> None:
|
||||
proc, reruns = self._run(
|
||||
[
|
||||
_run_record(2, "cancelled", "completed"),
|
||||
_run_record(1, "success", "completed"),
|
||||
]
|
||||
)
|
||||
self.assertEqual(proc.returncode, 0, proc.stderr)
|
||||
self.assertEqual(reruns, ["2"])
|
||||
|
||||
def test_leaves_the_branch_alone_while_a_run_is_still_active(self) -> None:
|
||||
proc, reruns = self._run(
|
||||
[
|
||||
_run_record(3, "cancelled", "completed"),
|
||||
_run_record(2, "", "pending"),
|
||||
]
|
||||
)
|
||||
self.assertEqual(proc.returncode, 0, proc.stderr)
|
||||
self.assertEqual(reruns, [])
|
||||
self.assertIn("still active", proc.stdout)
|
||||
|
||||
def test_a_queued_run_counts_as_active_too(self) -> None:
|
||||
proc, reruns = self._run(
|
||||
[
|
||||
_run_record(4, "cancelled", "completed"),
|
||||
_run_record(3, "", "queued"),
|
||||
]
|
||||
)
|
||||
self.assertEqual(proc.returncode, 0, proc.stderr)
|
||||
self.assertEqual(reruns, [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user