chore(ci): drop the cancelled-run rerun automation

The hourly sweep fought the rest of the CI control plane instead of
helping it: it preempted itself (17 of the last 29 runs concluded
cancelled 1-2s after their successor was created, for 8s of work behind
a 74-80min queue wait) and shipped MAX_ATTEMPTS=0, which the script
reads as no ceiling while its own README promised the opposite. The 24h
age cutoff could not bound it either, because every rerun refreshes the
run's updatedAt and so keeps it inside the window.

Removes the workflow, the two scripts it was the sole consumer of, its
unit test, and the CI_RERUN_* documentation that would otherwise point
at a deleted file. The workflow is disabled in both origin and fork so
no sweep fires before this lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kevinveenbirkenbachandClaude Opus 5 committed 2026-08-11 20:20:20 +02:00
1 parent 8aaac027d2
commit 3a1ef70cbb
8 files changed
-276

No files matched your search

@@ -1,42 +0,0 @@
name: "⏰ Rerun: Cancelled Branch Runs"
concurrency:
group: rerun-cancelled
cancel-in-progress: true
on:
schedule:
- cron: "0 * * * *"
workflow_dispatch:
permissions:
actions: write
contents: read
jobs:
rerun:
name: 🔁 Rerun the failed jobs of cancelled runs
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: ⬇️ Checkout repository
uses: actions/checkout@v7.0.0
- name: ⏱️ Decide whether this hour is due
id: due
if: ${{ github.event_name == 'schedule' }}
shell: bash
env:
INTERVAL_MINUTES: ${{ vars.CI_RERUN_INTERVAL_MINUTES || '60' }}
run: bash scripts/github/rerun/due.sh
- name: 🔁 Restart the failed jobs per branch
if: ${{ steps.due.outputs.due != 'false' }}
shell: bash
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REPOSITORY: ${{ github.repository }}
MAX_ATTEMPTS: ${{ vars.CI_RERUN_MAX_ATTEMPTS || '0' }}
MAX_AGE_HOURS: ${{ vars.CI_RERUN_MAX_AGE_HOURS || '24' }}
run: bash scripts/github/rerun/cancelled_branch_runs.sh
@@ -13,9 +13,6 @@ Repository variables are set under **Settings → Secrets and variables → Acti
| `CI_RUN_ON_MAIN` | [entry-push-latest.yml](../../../../../.github/workflows/entry-push-latest.yml) | Pushes to `main` skip CI | `true` to run CI on `main` pushes too |
| `CI_ENABLE_AUTO_UPDATES` | [cron-update.yml](../../../../../.github/workflows/cron-update.yml), [entry-pr-open-dependabot-close.yml](../../../../../.github/workflows/entry-pr-open-dependabot-close.yml) | Update jobs skipped; Dependabot PRs auto-closed | `true` to allow update PRs (workflow-driven and Dependabot) |
| `INFINITO_PLAYWRIGHT_KEEP` | [call-test-deploy-compose.yml](../../../../../.github/workflows/call-test-deploy-compose.yml) | Playwright keeps trace, screenshot and video only when a test fails | `true` to keep them for every test (passing runs included) |
| `CI_RERUN_INTERVAL_MINUTES` | [cron-rerun-cancelled.yml](../../../../../.github/workflows/cron-rerun-cancelled.yml) | Sweeps for cancelled runs every full hour | Minutes between two sweeps, rounded up to whole hours |
| `CI_RERUN_MAX_ATTEMPTS` | [cron-rerun-cancelled.yml](../../../../../.github/workflows/cron-rerun-cancelled.yml) | Revives a run however often it is cancelled | Attempt count at which a run is left alone; `0` for no ceiling |
| `CI_RERUN_MAX_AGE_HOURS` | [cron-rerun-cancelled.yml](../../../../../.github/workflows/cron-rerun-cancelled.yml) | Ignores runs untouched for more than 24 hours | Age in hours beyond which a cancelled run is left alone |
## `CI_CANCEL_IN_PROGRESS` 🛑
@@ -187,26 +184,3 @@ INFINITO_PLAYWRIGHT_KEEP: ${{ vars.INFINITO_PLAYWRIGHT_KEEP }}
| `true` | Trace / screenshot / video kept for every test ✓ |
| any other value | Trace / screenshot / video kept only on failure ✓ |
## `CI_RERUN_*` 🔁
Three variables tune [cron-rerun-cancelled.yml](../../../../../.github/workflows/cron-rerun-cancelled.yml), the sweep that restarts the failed jobs of a branch whose latest run was cancelled.
| Variable | Unset | Effect when set |
|---|---|---|
| `CI_RERUN_INTERVAL_MINUTES` | `60` | Minutes between two sweeps |
| `CI_RERUN_MAX_ATTEMPTS` | `0` | A run at or above this attempt count is left alone; `0` means no ceiling, a run is revived however often it is cancelled |
| `CI_RERUN_MAX_AGE_HOURS` | `24` | A run untouched for longer than this is left alone, so a sweep does not resurrect abandoned branches |
**How the interval works:**
A cron expression cannot read a repository variable, so the workflow is triggered every full hour and a gate lets only the hours that open the configured window through:
```yaml
INTERVAL_MINUTES: ${{ vars.CI_RERUN_INTERVAL_MINUTES || '60' }}
```
The interval is rounded up to whole hours and counted from 00:00 UTC, so `60` sweeps every hour, `180` at 00:00, 03:00, 06:00 and so on, and `1440` once a day at 00:00. One hour is the floor: anything below it sweeps hourly. Anything above 24 hours collapses to a daily sweep. Manual dispatch skips the gate entirely and always sweeps.
The gate reads the hour, not the minute, so a scheduled run that GitHub starts a few minutes late still counts for its hour.
**On the missing ceiling:** with `CI_RERUN_MAX_ATTEMPTS` unset, a run that is cancelled again after every revival is revived again, indefinitely. `CI_RERUN_MAX_AGE_HOURS` does not bound this, because each new attempt refreshes the run's `updatedAt` and so keeps it inside the age window. Set the variable to a positive number if a run should eventually be given up on.
@@ -88,7 +88,6 @@ Trigger column legend: **auto** = fires automatically (push, pull_request, sched
| [cron-cleanup-stale.yml](../../../../../.github/workflows/cron-cleanup-stale.yml): `⏰ Cleanup: Stale Repository Data` | Marks and closes stale issues and PRs, deletes inactive branches, and prunes old GHCR CI image versions. | auto (`schedule`: daily 00:00 UTC), manual | none |
| [cron-update.yml](../../../../../.github/workflows/cron-update.yml): `⏰ Update: Versions` | Updates Docker image versions and other pinned dependencies; opens an update PR via a GitHub App installation token so PR-lifecycle events fire on the resulting PR. Both jobs are gated behind the `CI_ENABLE_AUTO_UPDATES` repository variable, see [configuration.md](configuration.md). Requires the `BOT_APP_CLIENT_ID` and `BOT_APP_PRIVATE_KEY` repository secrets, see [secrets.md](secrets.md). | auto (`push` to `main`, `schedule`: daily 00:30 UTC), manual | none |
| [entry-pr-open-dependabot-close.yml](../../../../../.github/workflows/entry-pr-open-dependabot-close.yml): `🚫 Dependabot: Close while auto-updates disabled` | Auto-closes Dependabot PRs while `CI_ENABLE_AUTO_UPDATES` is not set to `true`, so Dependabot honours the same gate as the workflow-driven update jobs. See [configuration.md](configuration.md). | auto (`pull_request_target`: opened, reopened) | none |
| [cron-rerun-cancelled.yml](../../../../../.github/workflows/cron-rerun-cancelled.yml): `⏰ Rerun: Cancelled Branch Runs` | Sweeps every branch: when its latest run ended `cancelled`, the failed jobs are rerun via `gh run rerun --failed` while the successful ones are kept, so a concurrency cancellation costs a resume instead of a full pipeline. The attempt ceiling, the age cutoff, and the sweep interval come from `CI_RERUN_MAX_ATTEMPTS`, `CI_RERUN_MAX_AGE_HOURS` and `CI_RERUN_INTERVAL_MINUTES`, see [configuration.md](configuration.md). Sweeps hourly by default, with no attempt ceiling. A new trigger cancels a sweep that is still running. Manual dispatch always sweeps, ignoring the interval. | auto (`schedule`: hourly, gated to the configured interval), manual | none |
## Changing workflows ✍️
-11
View File
@@ -1,11 +0,0 @@
# Run Reruns 🔁
Shell helpers that restart the unfinished parts of GitHub Actions workflow runs that were cancelled rather than completed.
## Scope 📋
This directory contains helpers invoked from scheduled workflows. They call the GitHub Actions API to rerun the failed jobs of a cancelled run, never the run as a whole, so the work that already succeeded is kept. A run is a candidate only while it is recent and below its attempt ceiling; both bounds are passed in by the calling workflow, because a sweep without them would resurrect abandoned branches and retry a permanently cancelled run forever.
Helpers MUST NOT rerun a run whose conclusion is `failure`. A real defect has to be fixed, not retried, and the [cancel](../cancel/README.md) helpers exist precisely so that a cancellation means "interrupted", not "broken".
For the workflow catalog that drives these calls see [workflows.md](../../../docs/contributing/tools/github/actions/workflows.md).
@@ -1,74 +0,0 @@
#!/usr/bin/env bash
#
# Restart the failed jobs of every branch whose latest workflow run ended
# cancelled.
#
# Inputs via env:
# GH_TOKEN token carrying actions:write on REPOSITORY
# REPOSITORY owner/repo
# MAX_ATTEMPTS leave runs alone once they reached this attempt count;
# 0 keeps reviving a run for as long as it stays in range
# MAX_AGE_HOURS leave runs alone whose last update is older than this
set -euo pipefail
: "${GH_TOKEN:?Missing GH_TOKEN}"
: "${REPOSITORY:?Missing REPOSITORY}"
: "${MAX_ATTEMPTS:?Missing MAX_ATTEMPTS}"
: "${MAX_AGE_HOURS:?Missing MAX_AGE_HOURS}"
if ! command -v gh >/dev/null 2>&1; then
echo "ERROR: gh CLI not found." >&2
exit 1
fi
if ! command -v jq >/dev/null 2>&1; then
echo "ERROR: jq not found." >&2
exit 1
fi
cutoff="$(date -u -d "${MAX_AGE_HOURS} hours ago" +%Y-%m-%dT%H:%M:%SZ)"
echo "Rerunning cancelled runs updated after ${cutoff}, below attempt ${MAX_ATTEMPTS}"
branches="$(
gh api --paginate \
-H "Accept: application/vnd.github+json" \
"/repos/${REPOSITORY}/branches?per_page=100" |
jq -r '.[].name'
)"
while read -r branch; do
[[ -n "${branch}" ]] || continue
runs="$(
gh run list \
--repo "${REPOSITORY}" \
--branch "${branch}" \
--limit 20 \
--json databaseId,conclusion,status,attempt,updatedAt
)"
active="$(jq -r 'map(select(.status != "completed")) | length' <<<"${runs}")"
if [[ "${active}" != "0" ]]; then
echo "${branch}: ${active} run(s) still active, leaving the cancelled one alone"
continue
fi
run_id="$(
jq -r --arg cutoff "${cutoff}" --argjson max "${MAX_ATTEMPTS}" '
.[0]
| select(.conclusion == "cancelled")
| select($max == 0 or .attempt < $max)
| select(.updatedAt > $cutoff)
| .databaseId
' <<<"${runs}"
)"
[[ -n "${run_id}" ]] || continue
echo "${branch}: rerunning the failed jobs of cancelled run ${run_id}"
if ! gh run rerun "${run_id}" --repo "${REPOSITORY}" --failed; then
echo "${branch}: run ${run_id} could not be rerun, likely already restarted"
fi
done <<<"${branches}"
-28
View File
@@ -1,28 +0,0 @@
#!/usr/bin/env bash
#
# Emit due=true on GITHUB_OUTPUT when the current hour opens an interval window
# counted from 00:00 UTC. Intervals are rounded up to whole hours.
#
# Inputs via env:
# INTERVAL_MINUTES wanted interval between two sweeps
set -euo pipefail
: "${INTERVAL_MINUTES:?Missing INTERVAL_MINUTES}"
: "${GITHUB_OUTPUT:?Missing GITHUB_OUTPUT}"
hours="$(((INTERVAL_MINUTES + 59) / 60))"
if ((hours < 1)); then
hours=1
fi
hour="$((10#$(date -u +%H)))"
if ((hour % hours == 0)); then
echo "$(date -u +%H:%M) UTC opens the ${hours}h window, sweeping"
echo "due=true" >>"${GITHUB_OUTPUT}"
else
echo "$(date -u +%H:%M) UTC is inside the ${hours}h window, skipping"
echo "due=false" >>"${GITHUB_OUTPUT}"
fi
Whitespace-only changes.
@@ -1,94 +0,0 @@
from __future__ import annotations
import json
import os
import subprocess
import tempfile
import unittest
from pathlib import Path
from utils.cache.files import PROJECT_ROOT, read_text
SCRIPT = PROJECT_ROOT / "scripts" / "github" / "rerun" / "cancelled_branch_runs.sh"
def _run_record(database_id: int, conclusion: str, status: str) -> dict:
return {
"databaseId": database_id,
"conclusion": conclusion,
"status": status,
"attempt": 1,
"updatedAt": "2026-08-10T15:46:00Z",
}
class TestCancelledBranchRuns(unittest.TestCase):
def _run(self, runs: list[dict]) -> tuple[subprocess.CompletedProcess, list[str]]:
with tempfile.TemporaryDirectory() as tmp:
stub_bin = Path(tmp) / "bin"
stub_bin.mkdir()
rerun_log = Path(tmp) / "rerun.log"
gh = stub_bin / "gh"
gh.write_text(
"#!/bin/sh\n"
'if [ "$1" = "api" ]; then printf \'[{"name":"b"}]\\n\'; exit 0; fi\n'
'if [ "$1" = "run" ] && [ "$2" = "list" ]; then'
f" cat {json.dumps(str(Path(tmp) / 'runs.json'))}; exit 0; fi\n"
'if [ "$1" = "run" ] && [ "$2" = "rerun" ]; then'
f' echo "$3" >>{json.dumps(str(rerun_log))}; exit 0; fi\n'
"exit 1\n"
)
gh.chmod(0o755)
(Path(tmp) / "runs.json").write_text(json.dumps(runs))
env = dict(os.environ)
env["PATH"] = f"{stub_bin}:{env['PATH']}"
env["GH_TOKEN"] = "t"
env["REPOSITORY"] = "o/r"
env["MAX_ATTEMPTS"] = "0"
env["MAX_AGE_HOURS"] = "999999"
proc = subprocess.run(
["bash", str(SCRIPT)],
env=env,
capture_output=True,
text=True,
check=False,
)
reruns = read_text(str(rerun_log)).split() if rerun_log.exists() else []
return proc, reruns
def test_revives_the_newest_cancelled_run_when_the_branch_is_idle(self) -> None:
proc, reruns = self._run(
[
_run_record(2, "cancelled", "completed"),
_run_record(1, "success", "completed"),
]
)
self.assertEqual(proc.returncode, 0, proc.stderr)
self.assertEqual(reruns, ["2"])
def test_leaves_the_branch_alone_while_a_run_is_still_active(self) -> None:
proc, reruns = self._run(
[
_run_record(3, "cancelled", "completed"),
_run_record(2, "", "pending"),
]
)
self.assertEqual(proc.returncode, 0, proc.stderr)
self.assertEqual(reruns, [])
self.assertIn("still active", proc.stdout)
def test_a_queued_run_counts_as_active_too(self) -> None:
proc, reruns = self._run(
[
_run_record(4, "cancelled", "completed"),
_run_record(3, "", "queued"),
]
)
self.assertEqual(proc.returncode, 0, proc.stderr)
self.assertEqual(reruns, [])
if __name__ == "__main__":
unittest.main()