Commit Graph
12 Commits
Author SHA1 Message Date
Environment TestandClaude Sonnet 4.6 eebb3b9788 refactor(utils): move utils/docker/image_*.py into utils/docker/image/ subpackage
- Move image_discovery.py → utils/docker/image/discovery.py
- Move image_ref.py → utils/docker/image/ref.py
- Add utils/docker/image/__init__.py
- Update all imports and references across cli/, tests/, utils/, and docs/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 19:00:02 +02:00
Environment TestandClaude Sonnet 4.6 1530788dbe feat(mirror): add repository segment to GHCR image path
Image names now follow {owner}/{repository}/mirror/{registry}/{name}
instead of {owner}/mirror/{registry}/{name}.

- GHCRProvider: add `repository` param + `add_args()`/`from_args()` classmethods
  as SPOT for CLI argument registration and provider construction
- sync, resolver: use GHCRProvider.add_args() / from_args() (no duplication)
- Workflows: resolve repository name via scripts/meta/resolve/repository/name.sh
  and pass --ghcr-repository to sync and publish
- mirrors.sh: require and forward GHCR_REPOSITORY env var
- deploy/development/mirrors.py: derive repository from GITHUB_REPOSITORY
- Tests: update expected paths, argv and env fixtures

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 00:20:41 +02:00
Environment TestandClaude Sonnet 4.6 317287983a fix(mirror): resolve merge conflicts with fix/playwright-pull
Keep new multi-registry logic (feature/pre-commit):
- Merge is_mirrorable_image(), _detect_registry(), image_source() and
  canonical_image_name() into utils/docker/image_discovery.py
- iter_role_images() now scans both config/main.yml and vars/main.yml
  and uses image_source() for full registry-prefixed URLs
- Delete cli/mirror/util.py (superseded by utils/docker/image_discovery)
- model.py re-exports ImageRef from utils.docker.image_discovery
- sync/__main__.py: revert fix/playwright-pull's if/else src URL to
  simple `src = f"docker://{img.source}"` (source already contains
  the full registry prefix)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 23:44:58 +02:00
Environment TestandClaude Sonnet 4.6 3f19ec1df4 fix(mirror): extend registry support and harden publish visibility handling
Extend image mirroring to cover quay.io and ghcr.io in addition to Docker Hub,
and make the publish step non-fatal when the token lacks write:packages scope.

cli/mirror/util.py:
- Add _detect_registry(), _strip_registry_prefix(), is_mirrorable_image()
  to support Docker Hub, quay.io, ghcr.io, and mcr.microsoft.com
- Update canonical_image_name() to strip any known registry prefix so that
  quay.io/keycloak/keycloak -> keycloak/keycloak (mirror/keycloak-keycloak)
- Add image_source() returning full registry-prefixed pull reference
  (docker.io/library/postgres:16, quay.io/keycloak/keycloak:latest, …)
- iter_role_images() now uses is_mirrorable_image + image_source, picking up
  the 17 previously skipped quay.io/ghcr.io images

cli/mirror/sync/__main__.py:
- Use docker://{img.source} directly (no hardcoded docker.io/ prefix)

cli/mirror/publish/__main__.py:
- Detect HTTP 404 on PATCH as _InsufficientTokenError (no retry, graceful
  GHA warning + exit 0) instead of retrying 5x and failing the workflow
- Use pkg.get("url") from listing response as canonical PATCH target URL
- Use safe="/" in urllib.parse.quote so mirror/name stays as literal path
- Do not retry on non-transient 4xx errors (400/401/403)

tests/unit/cli/mirror/test_sync.py:
- Update ImageRef.source to new docker.io/library/… format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 22:27:11 +02:00
Environment TestandClaude Sonnet 4.6 9ea2ad6ff1 refactor(docker): introduce utils/docker/image_discovery as single SPOT for image enumeration
- Move iter_role_images + all helpers + ImageRef from cli/mirror/util.py and
  cli/mirror/model.py into utils/docker/image_discovery.py; add source_file field
  to ImageRef to distinguish config/main.yml vs vars/main.yml entries
- Delete cli/mirror/util.py; update all callers to import from utils.docker.image_discovery
- Move utils into subpackages: utils/docker/ (image_ref, paths_utils, hadolint) and
  utils/roles/ (dependency_resolver, resource_validation); update all imports
- Reorganise tests into subpackages: tests/lint/docker/dockerfile/, tests/integration/docker/,
  tests/unit/utils/docker/, tests/unit/utils/roles/; fix parents[N] offsets after relocation
- tests/lint/docker/test_image_versions.py now uses iter_role_images (same SPOT as mirror)
- Add vars/main.yml images.<name>.{image,version} support for non-application roles
  (test-e2e-playwright, sys-ctl-hlth-csp); eliminate hardcoded duplicates in defaults/tasks
- Add web-app-matrix matrix-chatgpt-bot and web-app-mini-qr mini-qr entries in config/main.yml;
  replace hardcoded image strings in vars/templates with lookup('config', ...) references
- Fix sync/__main__.py log line to use correct registry prefix for non-Docker-Hub images
- Update docs/contributing/code/role/dockerfile.md with image declaration rules and SPOT reference

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 19:32:31 +02:00
kevinveenbirkenbachandClaude Sonnet 4.6 7c8783c623 fix(mirror): use GHCR_PAT for package visibility API and remove ensure_public from skip path
Root cause: GITHUB_TOKEN in Actions is repo-scoped and cannot change package
visibility in user/org namespaces — all 58 mirror operations failed with 404.

- Use GHCR_PAT (which has write:packages scope) as GITHUB_TOKEN in the mirror
  step, falling back to GITHUB_TOKEN if GHCR_PAT is not set
- Remove ensure_public() call from the already-exists skip path (wrong place:
  existing images were already made public when first mirrored)
- Update test to assert ensure_public is not called for skipped images

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-08 09:45:38 +02:00
kevinveenbirkenbachandClaude Sonnet 4.6 b3a4a81e01 fix(mirror): remove redundant mirror job and publish images as public
- Remove redundant `images-mirror-missing` job from entry-pull-request-change.yml
  (mirroring already runs via ci-orchestrator for same-repo PRs)
- Set GHCR package visibility to public after each successful mirror via GitHub API
- Add `ensure_public` hook to base class (no-op for other providers)
- Also call `ensure_public` for already-existing images to fix any that are still private
- Pass GITHUB_TOKEN explicitly to the mirror step
- Add unit tests for _set_public (user/org endpoint fallback) and sync skip-path

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-08 07:45:33 +02:00
kevinveenbirkenbach 8a81630391 Added newline at file and POSIX compatible 2026-02-04 19:49:56 +01:00
kevinveenbirkenbach c0cd4b8eda feat(ci,mirror): add branch-scoped mirror of missing images and gate deploys on it
- Introduce reusable workflow `mirror-images-missing.yml` to mirror only missing
  Docker Hub images to GHCR using `--only-missing`
- Scope mirror concurrency per branch using `github.ref_name`
- Rename full mirror workflow to `mirror-images-all.yml` for clarity
- Integrate missing-image mirror into CI orchestrator and gate server/universal
  deploys on successful mirror completion
- Add tag existence checks via `skopeo inspect` to avoid redundant uploads
- Keep mirror runs best-effort to avoid blocking CI on transient registry issues

https://chatgpt.com/share/698141e9-0844-800f-9c4f-fecd468ede73
2026-02-03 01:31:30 +01:00
kevinveenbirkenbach 4949b45f44 feat(mirror): add rate-limited image sync with concurrency protection
- add optional --images-per-hour throttle to mirror sync
- enforce safe default (50 images/hour) in CI
- prevent overlapping mirror runs via workflow concurrency
- make workflow robust for schedule, dispatch, and workflow_call
- fix GHCR namespace fallback to repository owner

https://chatgpt.com/share/6980be4f-d5d4-800f-a9ad-2aaba637897e
2026-02-02 16:09:57 +01:00
kevinveenbirkenbach e7fdaa1d10 fix(mirror): iterate all images and fail at end on mirror errors
- mirror sync now continues on individual skopeo failures
- collect and summarize failed images
- return non-zero exit code only after all images were processed
- make GitHub workflow respect mirror sync exit status

https://chatgpt.com/share/6980b558-5fdc-800f-911e-48bbc35a461f
2026-02-02 15:45:27 +01:00
kevinveenbirkenbach 221c1c67aa feat(cli): add flexible docker image mirror resolver and sync system
Introduce a modular mirror subsystem with:
- canonical ImageRef model
- shared discovery & normalization logic for Docker Hub images
- pluggable registry providers (GHCR, Gitea-ready)
- inventory resolver for mirrored image paths
- skopeo-based mirror sync implementation

Design follows SRP/KISS principles and allows future registry backends.

https://chatgpt.com/share/6971e6cf-be18-800f-9419-589fd3e22c3d
2026-01-22 09:58:48 +01:00