Commit Graph
620 Commits
Author SHA1 Message Date
Alejandro Roman 1c80c2413d Merge remote-tracking branch 'upstream/main' into feature/self-host-runners 2026-06-29 11:36:40 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e55001e849 feat(update): one PR per role and compute diffs from latest main
The roles updaters (Docker image versions, repository refs) opened a
single combined PR for all role changes, and the jobs checked out the
trigger commit (actions/checkout default), so updates were computed
against a stale tree and could re-propose already-merged changes.

- Check out 'ref: main' in all update jobs so the diff is always
  computed against the latest main tip at run time.
- Add open_role_prs.sh: split the working-tree changes per role and
  open one isolated branch/PR per changed role, reusing open_pr.sh for
  the dedup/push/PR logic (new UPDATE_DEDUPE_PREFIX hook scopes the
  duplicate check per role; default preserves prior single-PR behavior
  for the skills job).
- Wire the Docker and repository-refs jobs to open_role_prs.sh; skills
  stays a single PR.

Branch: update/<type>-<role>-<date>-<fp>; commit/title gain a (<role>)
suffix and a Scope line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 19:58:54 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0df75d83a6 fix(ci): retry buildx setup against transient Docker Hub timeouts
The 'Set up Docker Buildx' step boots BuildKit, which pulls
moby/buildkit from registry-1.docker.io. A transient Docker Hub
timeout there had no retry (only the build step did), so the whole
build job failed at setup.

Pre-pull the BuildKit image with a 7x/20s retry before the buildx
setup, and pin the action to that exact tag via driver-opts. Once the
image is local, buildx falls back to it if the bootstrap pull fails
('pulling failed, using local image'), so a flaky Docker Hub no longer
breaks setup. Applied to both images-build-ci.yml and
release-version.yml.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 19:58:53 +02:00
kevinveenbirkenbachandClaude Opus 4.8 293f70d785 feat(scripts): make GHCR CI image cleanup report its actions
The cleanup step ran silently, so a no-op run was indistinguishable
from a broken one. Add a config banner, per-deletion lines, and a
per-package plus grand-total summary with a kept-reason breakdown, and
an explicit message when nothing matches. Add DRY_RUN support.

Deletion rule: delete a version only when it carries a ci-* tag, has no
semver (v*) or latest tag, and is older than MAX_AGE_DAYS. Released
images and untagged multi-arch child layers are never touched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 02:47:32 +02:00
Alejandro Roman 17d6ac904f Merge remote-tracking branch 'upstream/main' into feature/self-host-runners
# Conflicts:
#	.github/workflows/test-deploy-universal.yml
#	.github/workflows/test-deploy-workstation.yml
2026-06-27 23:45:38 +02:00
kevinveenbirkenbachandClaude Opus 4.8 8a586ad373 feat(lint): validate generated distro changelogs + harden the mirror generator
Adds a lint-packages target/workflow that parses the generated packaging metadata with each ecosystem's native parser (dpkg-parsechangelog, rpmspec, bash -n for the PKGBUILD), failing on any parser stderr so warnings cannot slip through. Wired into the make lint fan-out and ci-orchestrator's lint gate; an install group provisions the parsers (best-effort, cross-distro).

Hardens package_mirror so the generated changelogs parse without warnings: every RPM body line gets a '- ' continuation prefix (a column-0 '*' was read as a new %changelog entry -> 'bad date'), and the archived-releases notice is folded into the oldest kept entry instead of trailing after the final entry (free text there made dpkg-parsechangelog warn). Regenerates packaging/debian/changelog and the Fedora spec accordingly.

Adds a test-lint check (test_package_changelog_completeness) asserting both package changelogs list every CHANGELOG.md + docs/changelog release and that their inline window matches CHANGELOG.md, plus unit regressions for the RPM continuation prefix and the Debian footer fold.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-27 21:36:30 +02:00
kevinveenbirkenbach f1c3beee68 ci: add repository cleanup workflow 2026-06-27 21:36:30 +02:00
kevinveenbirkenbachandClaude Opus 4.8 f3fcaa8004 feat(ci): sync main from source repository before CI scope discovery
entry-push-latest.yml gains a sync-main job that force-pushes the current repository's main from a source repo (default infinito-nexus/core) before CI policy and deploy discovery derive scope from origin/main. The source is configurable and can be disabled via CI_SYNC_MAIN_SOURCE_REPOSITORY; the same-repository case and disabled values skip the sync.

The sync-main checkout uses fetch-depth: 0 and the push is --force to avoid shallow-update and non-fast-forward rejections. Docs warn that main is force-overwritten and must not be used as a working branch on synced repositories.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-27 18:32:29 +02:00
Alejandro Roman a77b596271 remove async pass in local 2026-06-24 18:31:23 +02:00
Alejandro Roman dcb07255c0 fix: INFINITO_REPO_ROOT must be empty in default.env; hardcode container path in dedicated.sh 2026-06-24 18:31:03 +02:00
Alejandro Roman cdff6c1323 fix: remove inline INFINITO_* defaults from compose.yml and dedicated.sh 2026-06-24 18:31:02 +02:00
Alejandro Roman 0153b5cac1 fix CI failures after rebase 2026-06-24 18:30:58 +02:00
Alejandro Roman c038d147a3 fix runner issue in DinD + addeed docs for CLI runners 2026-06-24 18:30:46 +02:00
Alejandro Roman dd4fb1c5d8 test-e2e-cli structure 2026-06-24 18:30:46 +02:00
Alejandro Roman 3e06ee738f fix cleaner + revert changes ot of scope 2026-06-24 18:30:46 +02:00
Alejandro Roman 8fd23c6062 fix cleaner 2026-06-24 18:30:45 +02:00
Alejandro Roman 7dfba97555 add retries for self hosted slower servers 2026-06-24 18:30:44 +02:00
Alejandro Roman 968882bcb2 revert(ci): restore original split_runner_apps.sh — pinning shifted app boundaries 2026-06-24 18:30:44 +02:00
Alejandro Roman c34d9bf477 fix(svc-runner): pin svc-runner to GitHub-hosted runners; remove sys-svc-container dep 2026-06-24 18:30:43 +02:00
Alejandro Roman aff4a7d158 refactor(svc-runner): fix comments on PR 2026-06-24 18:30:43 +02:00
Alejandro Roman e6866faa08 refactor(svc-runner): modify test 2026-06-24 18:30:42 +02:00
Alejandro Roman 83975bd8d4 refactor(svc-runner): fix new test to pass CI 2026-06-24 18:30:41 +02:00
Alejandro Roman fecdb8cc0a refactor(svc-runner): fix test for runenr 2026-06-24 18:30:41 +02:00
Alejandro Roman 958409cb85 fix(svc-runners): fix prune not working 2026-06-24 18:30:40 +02:00
Alejandro Roman b42a493865 fix(svc-runners): volume pruen before deploy to aovid access denied 2026-06-24 18:30:40 +02:00
Alejandro Roman 1ea5345205 fix(svc-runners):add more time for slow specs server 2026-06-24 18:30:39 +02:00
Alejandro Roman f2998b8f8c fix(svc-runners):fix -T issue 2026-06-24 18:30:39 +02:00
Alejandro Roman c8bc1692dd fix(svc-runners):fix multiplier for sel hosted 2026-06-24 18:30:38 +02:00
Alejandro Roman d45d894feb fix(svc-runners): multiply retries for slower self hosted runners only 2026-06-24 18:30:38 +02:00
Alejandro Roman 5567a996c3 fix(svc-runners): cleaning unsued contaienrs, imprve CPU use and cache 2026-06-24 18:30:37 +02:00
Alejandro Roman a61e5a9ac0 fix(svc-runners): fix issues with tmpfs access for self hosted 2026-06-24 18:30:36 +02:00
Alejandro Roman 5698d9e708 fix(svc-runners): improve runner speed, use RAm isntead of disk 2026-06-24 18:30:35 +02:00
Alejandro Roman 8cab0f04da fix(svc-runners): isolate parallel runners (ports, inventory, docker volume, dns) 2026-06-24 18:30:34 +02:00
Alejandro Roman 7d28cbfbf3 feat(svc-runners): fix CI conflicts 2026-06-24 18:30:33 +02:00
Alejandro Roman 2bd4c07e0b feat(svc-runners): fix container finisher conflicts 2026-06-24 18:30:32 +02:00
Alejandro Roman bcf9e97fda feat(svc-runners): fix container conflicts 2026-06-24 18:30:32 +02:00
Alejandro Roman 764a1adf30 feat(svc-runners): add the logic to have x numbers of runners self hosted ad connected to GIthub, allowing to have 20 runners from gh + x self hosted 2026-06-24 18:30:31 +02:00
kevinveenbirkenbachandClaude Opus 4.8 a9072d251c fix(ci): use venv python in discover step so variant-bundle expander has deps
The deploy-matrix discovery ran `python3 -m utils.github.variant_bundles`
with the runner system python, which lacks PyYAML, so importing get_variants
failed (ModuleNotFoundError: No module named yaml). Source
scripts/meta/env/python.sh to use the venv `make install` populates (pip
install -e . puts PyYAML there), matching how the rest of the toolchain
resolves PYTHON.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 14:17:54 +02:00
kevinveenbirkenbachandClaude Opus 4.8 35c2ff5f26 feat(deploy): runner-split variant bundles + unified variant selector
Thread a single `variant` selector (CSV) end-to-end so CI can split a
role's matrix deploy across runners.

- variant_select.py (SPOT, shared by init + deploy): one --variant taking
  a CSV (2 pins one round, 0,1,2 is a runner bundle, empty = full matrix);
  replaces the old separate --variant/--variants.
- planner.filter_plan_to_variants filters the plan to a round subset.
- CI discovery (utils/github/variant_bundles.py) splits roles with more
  than INFINITO_VARIANT_BUNDLE_SIZE (default 3) variants into bundles of
  consecutive indices, one matrix runner each (5 -> 0,1,2 + 3,4), threaded
  via the variant env. test-deploy-{local,universal,workstation,server}.yml
  switch to matrix.include with a comma-free variant_slug for artifact/job
  names (GitHub expressions have no replace()).
- runtime_budget honours INFINITO_VARIANT_TIME_OVERFLOW (cut|fail) when a
  runner cannot finish its slice within INFINITO_MAX_RUNTIME.

Unit tests + docs (variants.md, deploy.md) included.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-21 14:52:50 +02:00
kevinveenbirkenbachandClaude Opus 4.8 a2176018a1 fix(test-e2e-playwright): rerun-spec stages addons/ and fixtures/ subdirs
The inner-loop rerunner copied only top-level *.js and personas/, never the role's addons/ (or fixtures/) tree, so addon-spec edits were invisible and every rerun silently used the stale deploy-time addon specs. Mirror the deploy-time stager so addon coverage reruns against the edited repo copy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-21 06:03:07 +02:00
kevinveenbirkenbachandClaude Opus 4.8 4efe7b0dfc refactor(mirror): extract the wait-for-mirrors logic into a cli module
scripts/meta/wait/mirrors.sh embedded its logic as a `python - <<PY` heredoc
plus a bash skopeo/retry loop, and the heredoc imported iter_role_images from
utils.docker.image_discovery — a module that does not exist (it is
utils.docker.image.discovery). That raised ModuleNotFoundError, so the process
substitution yielded zero refs and the script logged "Nothing to wait for" and
exited 0, silently turning the fork-PR "wait for mirrored images" safeguard
into a no-op.

Move the whole thing into cli/contributing/mirror/wait/ (sibling to sync and
resolver): an argparse entrypoint that enumerates role images via
iter_role_images, computes each GHCR destination via GHCRProvider, and polls
GHCRProvider.tag_exists with --attempts/--sleep-seconds until all are present
or it times out. mirrors.sh becomes a thin env->args wrapper invoking
python -m cli.contributing.mirror.wait, matching scripts/meta/mirror/sync_missing.sh.

Behaviour is identical to the previous bash, minus the broken import.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 11:30:35 +02:00
kevinveenbirkenbachandClaude Opus 4.8 61477fb55d refactor(mirror): extract the wait-for-mirrors logic into a cli module
scripts/meta/wait/mirrors.sh embedded its logic as a `python - <<PY` heredoc
plus a bash skopeo/retry loop, and the heredoc imported iter_role_images from
utils.docker.image_discovery — a module that does not exist (it is
utils.docker.image.discovery). That raised ModuleNotFoundError, so the process
substitution yielded zero refs and the script logged "Nothing to wait for" and
exited 0, silently turning the fork-PR "wait for mirrored images" safeguard
into a no-op.

Move the whole thing into cli/contributing/mirror/wait/ (sibling to sync and
resolver): an argparse entrypoint that enumerates role images via
iter_role_images, computes each GHCR destination via GHCRProvider, and polls
GHCRProvider.tag_exists with --attempts/--sleep-seconds until all are present
or it times out. mirrors.sh becomes a thin env->args wrapper invoking
python -m cli.contributing.mirror.wait, matching scripts/meta/mirror/sync_missing.sh.

Behaviour is identical to the previous bash, minus the broken import.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 13:42:38 +02:00
kevinveenbirkenbach 85c1ed1a6e Merge branch 'main' into feature/web-app-zammad
# Conflicts:
#	CHANGELOG.md
#	packaging/arch/PKGBUILD
#	packaging/debian/changelog
#	packaging/fedora/infinito-nexus.spec
#	pyproject.toml
#	roles/sys-svc-compose/README.md
#	scripts/docker/entry.sh
#	scripts/github/update/image_versions.sh
#	scripts/github/update/repository_refs.sh
2026-05-29 11:18:28 +02:00
kevinveenbirkenbachandClaude Opus 4.7 f286ce54dd fix(update/open_pr): scope dedup to changed files
The previous tree-hash dedup compared whole-commit trees, so unrelated
drift on main (e.g. a manual edit to any roles/* file) caused the same
auto-update delta to appear "new" each daily cron and a fresh PR to open.

Now: fingerprint only the files that this run actually committed
(git diff --name-only HEAD~1 HEAD), both locally and on each candidate
open PR with the same UPDATE_BRANCH_PREFIX. If any open PR carries the
identical content for those files, skip push + PR creation entirely.

Branch name now carries the fingerprint's short hash so parallel PRs
of the same class can coexist (the close-superseded loop is dropped).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-29 10:37:21 +02:00
kevinveenbirkenbachandClaude Opus 4.7 c680be12ea fix(image/build): propagate INFINITO_VENV_DIR to docker build
scripts/image/build.sh drives 'make build' and the test-environment
matrix's dev-image build, but missed the new INFINITO_VENV_DIR build
arg. Dockerfile's ARG resolved to empty, ENV set INFINITO_VENV_DIR=""
in the image, and entry.sh's assert tripped on first --compile.

Adds the assert and the matching --build-arg, parallel to the
INFINITO_SRC_DIR plumbing already in this file and to the equivalent
plumbing in scripts/image/push.sh.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-28 22:33:15 +02:00
kevinveenbirkenbachandClaude Opus 4.7 00c4246c3a refactor(env): introduce INFINITO_VENV_DIR SPOT, fix dev image build
Adds INFINITO_VENV_DIR=/opt/venvs/infinito to default.env (registered
in passthrough's STATIC_KEYS so make dotenv propagates it to .env) and
threads it through the build chain: compose.yml forwards it as a build
arg; scripts/image/push.sh asserts and --build-args it; the Dockerfile
declares the matching ARG/ENV and derives PYTHON/PIP/PATH from the
variable. scripts/meta/env/python.sh now prefers the value when set
before its fallback discovery so the SPOT is effective in env
resolution.

Fixes the dev image build regression where 'infinito --version' in
scripts/docker/entry.sh exited 127. Root cause: the Dockerfile RUN
SHELL uses bash -lc, which sources /etc/profile and resets PATH to
the Debian default, dropping ${INFINITO_VENV_DIR}/bin from it. The
fix invokes the script via its absolute path "${INFINITO_VENV_DIR}/bin/infinito"
so the post-install version check no longer depends on a PATH that
the login-shell init has just clobbered.

scripts/lint/wrapper.sh's existing PATH-reset note now references the
variable instead of the literal path. test-install-make.yml and
test-install-pkgmgr.yml read INFINITO_VENV_DIR from default.env up
front, pass it into their pkgmgr containers, and source the venv
activate script via the variable.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-28 22:12:19 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a561aa76b5 ci(push): skip push CI when an open PR already covers the branch
push_ci_policy.sh now hard-fails on missing GH_TOKEN/GITHUB_REPOSITORY
and, for non-main branches, calls gh pr list to detect whether the
push is already covered by an open pull request. If so, emits
should_run=false (skip_reason=pr-ci-covers-this-push) so the push
workflow exits cleanly and the pull_request workflow handles the
actual CI run.

To absorb the bot's push -> gh-pr-create latency on update/* and
alert-autofix-* branches, the PR check retries up to 10x with 2s
sleeps before falling through to a real CI run. Human branches
(feature/**, hotfix/**, fix/**, chore/**) check once with no delay.

entry-push-latest.yml's run-policy step now forwards GH_TOKEN so the
gh call works against the repo.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-28 21:06:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 af628847bb fix(update/open_pr): delete head branch when closing superseded PRs
The previous gh pr close call left the bot's daily update branch
behind on the remote, so stale update/<prefix>-YYYYMMDD refs
accumulated indefinitely. --delete-branch removes the head ref on
close. Scope-safe because the loop only iterates branches matching
${UPDATE_BRANCH_PREFIX}-.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-28 21:06:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 806a3e93db ci(pr-flow): gate env-matrix on cross-cutting diffs, dedup resolution
Adds a detect-affected-roles job to entry-pull-request-change.yml that
runs the affected-roles resolver upfront on host python (via make
install) and emits whitelist + roles_only. ci-orchestrator skips
test-development when roles_only=true; done job tolerates the skip.
entry-manual derives roles_only from its whitelist input.

affected_roles.sh now invokes the resolver host-side only (compose-exec
fallback removed; it was a no-op in every existing caller because
compose was never up there, always emitting __ALL__ fail-safe). The
PR-flow gate consolidates what test-deploy-* used to recompute per
job; downstream effective_whitelist.sh now consumes the pre-resolved
whitelist verbatim.

Test fixture switched from a fake docker binary to a fake PYTHON
binary and clears BASH_ENV so the bash-startup env loader does not
reset PYTHON.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-28 21:06:34 +02:00
kevinveenbirkenbach 84bd56e72c Merge branch 'main' into feature/req-21-sso-flavor-migration 2026-05-28 18:07:28 +02:00