Adds a container-shaped heuristic so non-shared compose.services entries
that declare neither resource fields (mem_*/pids_limit/cpus) nor
container-identifying fields (image/name/version/container) are treated
as feature flags and excluded from the resource table. Keeps the
keycloak recaptcha toggle and similar booleans out of the per-role
footprint. Adds unit coverage for the helper and recursion behavior.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a CLI that reports per-container resource budgets (mem_reservation,
mem_limit, pids_limit, cpus) for an Ansible role, recursively following
enabled shared services via the service registry. Aggregation rules:
mem_* and pids_limit are summed; cpus is MAX (shared across containers,
not additive). Includes text + JSON output, cycle protection, and unit
tests covering parsing helpers, aggregation, recursion, rendering, and
end-to-end CLI invocation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds inventory consistency guard that fails fast when SERVICES_DISABLED
conflicts with an already-deployed inventory, and ensures the env var is
forwarded through the dedicated runner, development exec command, and
local test scripts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolve conflicts:
- docs/agents/action/iteration.md: keep both new bullets (SERVICES_DISABLED
proposal + make test requirement)
- docs/contributing/design/services.md: deleted (renamed to
docs/contributing/code/services.md in this branch)
- group_vars/all/20_services.yml: keep deletion from this branch (req-001)
- tests/lint/ansible/roles/test_service_core_first_task_run_once.py:
use build_service_registry_from_roles_dir (req-001) with repo_root()
from main for robust path resolution
Fix broken relative links caused by rename and deletion of files:
- docs/contributing/design/architecture.md: services.md → ../code/services.md
- docs/contributing/design/exceptions.md: load_frontend_service.yml →
load_service.yml
- docs/requirements/001-service-registry-refactoring.md: fix paths for
unit.md, integration.md, principles.md, documentation.md, AGENTS.md
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Delete group_vars/all/20_services.yml; all service metadata now lives
in each role's config/main.yml under compose.services.<name>
- Rename generic 'database' service key to explicit 'mariadb'/'postgres'
across all app roles; remove legacy type: field
- Add utils/service_registry.py and plugins/lookup/service_registry.py
as the single source of truth for service discovery
- Add utils/database_service.py to centralize DB service resolution
- Delete service_load_enabled filter and service_should_load lookup;
replaced by service lookup plugin
- Update sys-utils-service-loader to run in constructor stage and use
generic load_service.yml; remove load_frontend_service.yml
- Add _service_loader_preloading guard to sys-svc-compose,
sys-svc-webserver-core and sys-ctl-alm-compose to prevent
health-checks/alarms from firing during preload phase
- Replace manual load_app.yml calls for shared DB/Mailu with assert-
based preload validation in sys-svc-rdbms and sys-svc-mail
- Update networks.yml.j2 templates to resolve LDAP/AI networks via
service_registry instead of hardcoded role names
- Migrate all tests to new service discovery API; delete obsolete tests
- Update docs/contributing/code/services.md to reflect role-local model
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds cli.mirror.cleanup for deleting GHCR container packages by prefix
and visibility via the GitHub Packages REST API. Supports dry-run mode.
Requires a classic PAT with delete:packages scope (GITHUB_TOKEN from
gh auth or GitHub Actions does not include this scope).
Also adds unit tests and a usage guide at
docs/contributing/flow/security/mirror-cleanup.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces GHCR_PAT-based authentication with GITHUB_TOKEN across all
mirror and build workflows. Packages pushed from a public repository
with GITHUB_TOKEN are automatically linked to the repository and
inherit its public visibility — no publish step required.
Removes cli.mirror.publish (used a non-existent GitHub REST API
endpoint for changing package visibility), the associated unit tests,
and the GHCR_PAT debug helper. Cleans up all GHCR_PAT/GHCR_USERNAME
references and rewrites the GHCR authentication documentation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace safe="/" with safe="" so package names like
mirror/docker.io/openresty/openresty are encoded as a single URL path
segment (mirror%2Fdocker.io%2Fopenresty%2Fopenresty). With literal
slashes GitHub's router matched only "mirror" as the package name → 404.
- _set_public now returns bool; main() only prints "✓ set to public" and
increments updated when True. Previously a 404-skip returned None which
still triggered the success path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Old packages (pre-rename) have no versions left in GHCR and return 404
on PATCH. This caused an early exit that skipped all remaining packages.
404 means the package is gone; actual token scope failures return 401/403.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Image names now follow {owner}/{repository}/mirror/{registry}/{name}
instead of {owner}/mirror/{registry}/{name}.
- GHCRProvider: add `repository` param + `add_args()`/`from_args()` classmethods
as SPOT for CLI argument registration and provider construction
- sync, resolver: use GHCRProvider.add_args() / from_args() (no duplication)
- Workflows: resolve repository name via scripts/meta/resolve/repository/name.sh
and pass --ghcr-repository to sync and publish
- mirrors.sh: require and forward GHCR_REPOSITORY env var
- deploy/development/mirrors.py: derive repository from GITHUB_REPOSITORY
- Tests: update expected paths, argv and env fixtures
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Keep new multi-registry logic (feature/pre-commit):
- Merge is_mirrorable_image(), _detect_registry(), image_source() and
canonical_image_name() into utils/docker/image_discovery.py
- iter_role_images() now scans both config/main.yml and vars/main.yml
and uses image_source() for full registry-prefixed URLs
- Delete cli/mirror/util.py (superseded by utils/docker/image_discovery)
- model.py re-exports ImageRef from utils.docker.image_discovery
- sync/__main__.py: revert fix/playwright-pull's if/else src URL to
simple `src = f"docker://{img.source}"` (source already contains
the full registry prefix)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add `registry` field to `ImageRef` (default `"docker.io"`)
- Populate `registry` from `_detect_registry()` in `iter_role_images()`
- `GHCRProvider.image_base()` and `GiteaProvider.image_base()` now produce
`{prefix}/{registry}/{name}` paths (e.g. `mirror/docker.io/nextcloud`,
`mirror/quay.io/keycloak/keycloak`) instead of flattening slashes to dashes
- Fix `mirror()` in both providers: use `docker://{image.source}` directly
instead of always prepending `docker://docker.io/`, which broke quay.io/ghcr.io images
- Update tests to reflect new naming convention and add quay.io test case
- Add missing __init__.py files in test subdirectories
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Keep HEAD's registry-aware src construction in sync/__main__.py
(docker://docker.io/{source} for Docker Hub, docker://{source} otherwise)
to stay compatible with image_discovery.py's ImageRef.registry field
- Accept deletion of cli/mirror/util.py (content moved to utils/docker/image_discovery.py)
- Revert test_sync.py source format to library/nextcloud:31.0.0 (HEAD format)
- Retain publish/__main__.py improvements from feature/pre-commit:
graceful 404 handling, no blind retries on client errors, pkg_url usage
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extend image mirroring to cover quay.io and ghcr.io in addition to Docker Hub,
and make the publish step non-fatal when the token lacks write:packages scope.
cli/mirror/util.py:
- Add _detect_registry(), _strip_registry_prefix(), is_mirrorable_image()
to support Docker Hub, quay.io, ghcr.io, and mcr.microsoft.com
- Update canonical_image_name() to strip any known registry prefix so that
quay.io/keycloak/keycloak -> keycloak/keycloak (mirror/keycloak-keycloak)
- Add image_source() returning full registry-prefixed pull reference
(docker.io/library/postgres:16, quay.io/keycloak/keycloak:latest, …)
- iter_role_images() now uses is_mirrorable_image + image_source, picking up
the 17 previously skipped quay.io/ghcr.io images
cli/mirror/sync/__main__.py:
- Use docker://{img.source} directly (no hardcoded docker.io/ prefix)
cli/mirror/publish/__main__.py:
- Detect HTTP 404 on PATCH as _InsufficientTokenError (no retry, graceful
GHA warning + exit 0) instead of retrying 5x and failing the workflow
- Use pkg.get("url") from listing response as canonical PATCH target URL
- Use safe="/" in urllib.parse.quote so mirror/name stays as literal path
- Do not retry on non-transient 4xx errors (400/401/403)
tests/unit/cli/mirror/test_sync.py:
- Update ImageRef.source to new docker.io/library/… format
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Move iter_role_images + all helpers + ImageRef from cli/mirror/util.py and
cli/mirror/model.py into utils/docker/image_discovery.py; add source_file field
to ImageRef to distinguish config/main.yml vs vars/main.yml entries
- Delete cli/mirror/util.py; update all callers to import from utils.docker.image_discovery
- Move utils into subpackages: utils/docker/ (image_ref, paths_utils, hadolint) and
utils/roles/ (dependency_resolver, resource_validation); update all imports
- Reorganise tests into subpackages: tests/lint/docker/dockerfile/, tests/integration/docker/,
tests/unit/utils/docker/, tests/unit/utils/roles/; fix parents[N] offsets after relocation
- tests/lint/docker/test_image_versions.py now uses iter_role_images (same SPOT as mirror)
- Add vars/main.yml images.<name>.{image,version} support for non-application roles
(test-e2e-playwright, sys-ctl-hlth-csp); eliminate hardcoded duplicates in defaults/tasks
- Add web-app-matrix matrix-chatgpt-bot and web-app-mini-qr mini-qr entries in config/main.yml;
replace hardcoded image strings in vars/templates with lookup('config', ...) references
- Fix sync/__main__.py log line to use correct registry prefix for non-Docker-Hub images
- Update docs/contributing/code/role/dockerfile.md with image declaration rules and SPOT reference
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace templated Dockerfiles with plain Dockerfiles for web-app-littlejs,
web-app-openproject, web-app-roulette-wheel, and web-app-wordpress.
Variables are passed as Docker ARG without defaults, keeping config/main.yml
and vars/main.yml as the single source of truth. Build args are declared
in compose.yml.j2 directly after the build.yml.j2 lookup. Image and version
for roulette-wheel and littlejs are added to config/main.yml and exposed
via vars/main.yml.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move _set_public/ensure_public out of GHCRProvider into a new
cli.mirror.publish module that lists all non-public mirror packages
via the GitHub API and sets them public in bulk. The workflow runs
this as a separate step after mirroring. Mirror sync itself no longer
touches visibility.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Move utils/gha/annotations.py → utils/annotations/message.py
- Move scripts/gha/summarize_annotations.py → utils/annotations/summarize.py
- Keep utils/gha/annotations.py as compatibility shim
- Update all imports across codebase to utils.annotations.message
- Wire tee + summary step into test-code-lint, unit, and integration workflows
- Add unit tests for utils/annotations/summarize (20 tests)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add utils/gha/annotations.py with warning/error/notice helpers and *_each
variants that emit one annotation per list item
- Use warning() in GHCRProvider._set_public for missing GHCR_PAT case
- Expose GHCR_PAT_SET env var in mirror step to distinguish missing-PAT from
real API failures
- Add unit tests for all annotation functions
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/users/{username}/packages does not support PATCH — it is a read-only endpoint
for public data. The correct endpoint for updating visibility of the authenticated
user's packages is /user/packages (without username).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A freshly pushed package may not be immediately visible in the GitHub
Packages API. Add up to 5 retries with 10s delay before failing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause: GITHUB_TOKEN in Actions is repo-scoped and cannot change package
visibility in user/org namespaces — all 58 mirror operations failed with 404.
- Use GHCR_PAT (which has write:packages scope) as GITHUB_TOKEN in the mirror
step, falling back to GITHUB_TOKEN if GHCR_PAT is not set
- Remove ensure_public() call from the already-exists skip path (wrong place:
existing images were already made public when first mirrored)
- Update test to assert ensure_public is not called for skipped images
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove redundant `images-mirror-missing` job from entry-pull-request-change.yml
(mirroring already runs via ci-orchestrator for same-repo PRs)
- Set GHCR package visibility to public after each successful mirror via GitHub API
- Add `ensure_public` hook to base class (no-op for other providers)
- Also call `ensure_public` for already-existing images to fix any that are still private
- Pass GITHUB_TOKEN explicitly to the mirror step
- Add unit tests for _set_public (user/org endpoint fallback) and sync skip-path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces a SPOT for all frontend service loading in sys-front-inj-all/tasks/01_services.yml,
driven by a new generic `service` lookup plugin that resolves services by key or role name
and returns {id, role, enabled, shared, needed}. Needed = enabled AND shared (direct or
transitive). The canonical key→role mapping moves to group_vars/all/20_services.yml
(previously scattered hardcoded rules). Removes per-role service loads from bigbluebutton,
nextcloud, and dashboard. Renames 01_dependencies.yml→01_services.yml and
08_services.yml→08_daemons.yml. Adds flush_handlers after each service load.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Scans roles_dir to find every role whose config/main.yml defines the
disabled service, then creates or updates the compose.services entry in
host_vars for each matching role — including roles not yet present in
applications. roles_dir is now a required parameter (no Optional fallback).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- deploy.py: use find_provider_roles() to map service names (e.g. 'matomo')
to provider app IDs (e.g. 'web-app-matomo') before filtering deploy_ids;
forward SERVICES_DISABLED into the container via extra_env
- container.sh: rename INSPECT_CMD to CMD for consistency; fix indentation
- Makefile: expose CMD variable on exec target for non-interactive use
- environment.sh: source scripts/meta/env/all.sh for INVENTORY_DIR; add
inventory verification after SERVICES_DISABLED deploy using make exec
- variables.md: document CMD variable
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add scripts/meta/resolve/image/local.sh as SPOT for the local Docker
image tag (<repo-basename>/<distro>). Wire it into Makefile build
targets (IMAGE_TAG), compose.py (_base_env sets INFINITO_IMAGE), and
compose.yml (no default fallback). Fix build.sh to accept a pre-set
IMAGE_TAG without requiring REPO_PREFIX. Add make build + build-no-cache
to environment.sh test flow with correct section ordering.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
compose.exec runs inside the container, so host env vars are not
inherited. Pass SERVICES_DISABLED via -e so services_disabler.py
can suppress the correct entries when creating the inventory.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add services_disabler.py: parse SERVICES_DISABLED, disable services in host_vars,
remove provider roles from inventory
- Wire apply_services_disabled_from_env into cli/create/inventory/cli.py after deploy
- Add unit tests for all services_disabler functions
- Update hardware.md and variables.md with SERVICES_DISABLED documentation
- Update environment.sh: deploy dashboard with SERVICES_DISABLED=matomo first,
assert dashboard reachable + matomo not reachable, then deploy matomo and assert
reachable; extract assert_http_status / assert_http_status_not helpers
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Makes clear that multiple space- or comma-separated app ids are
supported. Renames the env var, CLI flag (--app → --apps), and all
references in scripts, workflows, and documentation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolves deps for each app individually and merges the results,
enabling APP="web-app-taiga web-app-pixelfed make deploy-fresh-purged-app".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
