Commit Graph
295 Commits
Author SHA1 Message Date
kevinveenbirkenbach 8a81630391 Added newline at file and POSIX compatible 2026-02-04 19:49:56 +01:00
kevinveenbirkenbach bc38347798 refactor(ci-deploy): split per-distro deploy, add two-pass async test and hard cleanup
- Replace legacy app.sh/distros.sh orchestration with explicit all_distros.sh + dedicated_distro.sh
- Deploy each app sequentially across distros against a fresh stack
- Run two deploy passes per distro:
  - Pass 1 with ASYNC_ENABLED=false
  - Pass 2 with ASYNC_ENABLED=true against the same stack
- Remove deprecated cli.deploy.development run orchestrator
- Extend init command with --vars to inject inventory overrides
- Enforce debug mode for CI deploys
- Add hard Docker cleanup after each distro (containers, networks, volumes, build cache; keep images)
- Emit disk and docker usage before and after deploy for observability

https://chatgpt.com/share/6981c7d4-642c-800f-8438-9e2349ac53fb
2026-02-03 11:02:51 +01:00
kevinveenbirkenbach c0cd4b8eda feat(ci,mirror): add branch-scoped mirror of missing images and gate deploys on it
- Introduce reusable workflow `mirror-images-missing.yml` to mirror only missing
  Docker Hub images to GHCR using `--only-missing`
- Scope mirror concurrency per branch using `github.ref_name`
- Rename full mirror workflow to `mirror-images-all.yml` for clarity
- Integrate missing-image mirror into CI orchestrator and gate server/universal
  deploys on successful mirror completion
- Add tag existence checks via `skopeo inspect` to avoid redundant uploads
- Keep mirror runs best-effort to avoid blocking CI on transient registry issues

https://chatgpt.com/share/698141e9-0844-800f-9c4f-fecd468ede73
2026-02-03 01:31:30 +01:00
kevinveenbirkenbach 9a84fafec5 feat(inventory): add mirror_policy to control mirror image overrides
- Default policy (if_missing): manual image/version win, mirror fills blanks only
- Add mirror_policy=force to always overwrite image/version
- Add mirror_policy=skip to ignore mirror completely
- Treat empty/whitespace values as missing
- Preserve non-image/version service keys
- Extend unit tests to cover all mirror policy scenarios

https://chatgpt.com/share/6981036a-7e40-800f-b43e-13c66cc55fac
2026-02-02 21:04:53 +01:00
kevinveenbirkenbach 734f013179 ci: wire GHCR mirror envs into deploy workflows and inventory init
- Add required GHCR mirror environment variables to CI workflows
- Generate mirrors.yml during development inventory init on CI
- Introduce mirrors helper module with strict env validation
- Minor formatting cleanup in mirror providers
`

https://chatgpt.com/share/6980ff2e-8f18-800f-85e1-79d56a2027f5
2026-02-02 20:46:41 +01:00
kevinveenbirkenbach 7faf87cc68 fix(mirror): remove invalid skopeo flag and add recompress fallback for GHCR
- drop unsupported --dest-no-reuse-blobs flag
- add retry with forced recompression to avoid GHCR blob reuse errors
- print skopeo stdout/stderr on failure for easier debugging

https://chatgpt.com/share/6980f7ca-2aa0-800f-ae64-60081aa88f1c
2026-02-02 20:14:49 +01:00
kevinveenbirkenbach a70a033864 feat(inventory): add --mirror support to override docker images from mirrors file
- add --mirror flag to inventory creator
- resolve mirrors.yml automatically or via explicit path
- apply mirror overrides to applications.*.docker.services.* (image/version only)
- keep existing host_vars keys untouched
- add unit tests for mirror override behavior

https://chatgpt.com/share/6980cdb1-1e5c-800f-b25f-a1af757e4239
2026-02-02 17:15:30 +01:00
kevinveenbirkenbach 647e93ff18 mirror: disable blob reuse for skopeo copy to GHCR
https://chatgpt.com/share/6980c6ff-3a6c-800f-8c95-6acdbfebfbfd
2026-02-02 16:47:04 +01:00
kevinveenbirkenbach 4949b45f44 feat(mirror): add rate-limited image sync with concurrency protection
- add optional --images-per-hour throttle to mirror sync
- enforce safe default (50 images/hour) in CI
- prevent overlapping mirror runs via workflow concurrency
- make workflow robust for schedule, dispatch, and workflow_call
- fix GHCR namespace fallback to repository owner

https://chatgpt.com/share/6980be4f-d5d4-800f-a9ad-2aaba637897e
2026-02-02 16:09:57 +01:00
kevinveenbirkenbach e7fdaa1d10 fix(mirror): iterate all images and fail at end on mirror errors
- mirror sync now continues on individual skopeo failures
- collect and summarize failed images
- return non-zero exit code only after all images were processed
- make GitHub workflow respect mirror sync exit status

https://chatgpt.com/share/6980b558-5fdc-800f-911e-48bbc35a461f
2026-02-02 15:45:27 +01:00
kevinveenbirkenbach 5c499def02 Fixed collabora service for bbb
https://chatgpt.com/share/698062c6-0000-800f-928d-03fd39e7d38c
2026-02-02 09:38:33 +01:00
kevinveenbirkenbach 42e809ae4f feat(cli): add callorder analyzer for tasks/groups role execution
Introduce a CLI tool to inspect and analyze role call order per group file,
supporting marker-based splitting, role filtering, and effected-group views.

https://chatgpt.com/share/697f5565-2b44-800f-999c-dbb2fb8ec449
2026-02-01 14:29:44 +01:00
kevinveenbirkenbach 546bf78cd4 feat(services,tls): register simpleicons as shared service for auto resolution
- Add simpleicons → web-svc-simpleicons to service resolver
- Ensure simpleicons is transitively included via docker.services flags
- Fix missing TLS SAN generation for icons.infinito.example
- Clean up unused service resolution modules

https://chatgpt.com/share/697eaf81-5500-800f-aec9-8902601a0124
2026-02-01 02:42:14 +01:00
kevinveenbirkenbach 8bff1b984d refactor(jinja): add strict recursive renderer and use in tls/compose lookups
Introduce module_utils/jinja_strict.py to render nested Jinja-in-vars with hard-fail semantics. Refactor cert_plan, compose_f_args, and compose_ca_inject_cmd to use render_strict so no "{{ ... }}" leaks into generated paths/commands, and add unit tests.

https://chatgpt.com/share/697b8e1b-54fc-800f-b895-7161f00ebd51
2026-01-29 17:42:28 +01:00
kevinveenbirkenbach ddb210b294 feat(users): add firstname/lastname to generated users and LDAP entries
- Generate firstname and lastname fields in CLI user setup
- Add meaningful default description including primary domain
- Pass firstname/lastname explicitly to OpenLDAP givenName/sn
- Remove LDAP-side fallbacks to item.key to ensure consistent user data

https://chatgpt.com/share/6978d573-a8c4-800f-8670-c9344f594a3a
2026-01-27 16:10:26 +01:00
kevinveenbirkenbach cbd4a4dc93 Fixed path 2026-01-25 11:25:59 +01:00
kevinveenbirkenbach c86d9820f5 fix(dev-deploy): add missing when_down arg in restart
restart delegates to up.handler(), which expects args.when_down.
Add when_down=False to the restart -> up args shim to avoid AttributeError.

https://chatgpt.com/share/69752530-c570-800f-b82c-f4bc10231292
2026-01-24 21:01:45 +01:00
kevinveenbirkenbach fae7b82243 feat(security): introduce strong_password filter backed by ValueGenerator
- add strong_password Jinja2 filter as thin wrapper around ValueGenerator
- enforce password policy (upper, lower, digit, special, min length)
- replace insecure ascii-only password lookup with strong_password filter
- update CLI setup users and unit tests accordingly
- harden inventory sample password example

https://chatgpt.com/share/69722c7e-ebe0-800f-a9ab-a3e21b43269c
2026-01-22 14:56:05 +01:00
kevinveenbirkenbach b334a5c98d fix(cli): ensure project root on sys.path when running cli/__main__.py directly
- Prevent accidental import of installed "cli" package from site-packages
- Make `python3 cli/__main__.py ...` behave identical to `python -m cli ...`
- Add env.development to .dockerignore to match .gitignore

https://chatgpt.com/share/69720fd6-a1c8-800f-b1e2-6909aa63643a
2026-01-22 13:13:01 +01:00
kevinveenbirkenbach d2694566e5 feat(runtime): add development runtime marker and unified runtime detection
- introduce env.development marker for local dev runtime
- add cli.meta.runtime module with library API and CLI entrypoint
- detect runtime via ACT/GitHub/marker with explicit RUNTIME override
- use env.development as docker-compose env override
- pass RUNTIME=dev into development inventory generation
- add make setup-development helper

https://chatgpt.com/share/69720fd6-a1c8-800f-b1e2-6909aa63643a
2026-01-22 12:53:42 +01:00
kevinveenbirkenbach 4b712c22cb feat(compose): load optional env.local as override for env.ci
https://chatgpt.com/share/697200be-e96c-800f-a8d3-b89b75b6e37c
2026-01-22 11:49:31 +01:00
kevinveenbirkenbach 221c1c67aa feat(cli): add flexible docker image mirror resolver and sync system
Introduce a modular mirror subsystem with:
- canonical ImageRef model
- shared discovery & normalization logic for Docker Hub images
- pluggable registry providers (GHCR, Gitea-ready)
- inventory resolver for mirrored image paths
- skopeo-based mirror sync implementation

Design follows SRP/KISS principles and allows future registry backends.

https://chatgpt.com/share/6971e6cf-be18-800f-9419-589fd3e22c3d
2026-01-22 09:58:48 +01:00
kevinveenbirkenbach 4cabf905b4 Deleted deprecated classes 2026-01-22 09:38:11 +01:00
kevinveenbirkenbach 6af034a0a0 feat(dev-deploy): stream compose/ansible output with CI-aware TTY and tail buffers
https://chatgpt.com/share/69712357-7548-800f-9b51-7fd4c4207bef
2026-01-21 20:04:48 +01:00
kevinveenbirkenbach 5e097c02f1 fix(shellcheck): address lint findings across scripts
- openldap: use find instead of ls for module path detection
- docker-volumes health: robust whitelist parsing + quote vars
- btrfs health: quote path
- docker CLI installer: silence SC2031 false-positive for /etc/os-release vars
- test-dns: use loop counter
- chess: require APP_KEY_FILE env
- espocrm: log normalized seed flags to avoid unused vars
- shopware: disable SC2016 for php -r blocks (PHP vars in single quotes)
- fix script: add shebang + strict mode

https://chatgpt.com/share/69712026-7288-800f-b1d0-701e31754362
2026-01-21 19:51:12 +01:00
kevinveenbirkenbach bdd6587bcc fix(ci): show deploy stdout/stderr and pass inventory_dir in run orchestrator
- Capture and print deploy stdout/stderr on failure (development/deploy.py)
- Forward INVENTORY_DIR into init and deploy handlers (development/run.py)
- Add missing when_down flag for up handler
- Print full traceback on orchestrator failure for CI visibility

https://chatgpt.com/share/6970fcf5-5860-800f-89ae-0d601405489b
2026-01-21 17:21:02 +01:00
kevinveenbirkenbach 33ab4b61dc feat(dev-deploy): extract CoreDNS Corefile rendering into dedicated Python class
- Move Corefile rendering logic into CoreDNSCorefileRenderer
- Add atomic rendering via envsubst with validation and preview
- Enforce strict error handling for missing files, invalid paths and permissions
- Integrate renderer into development compose workflow
- Rename DNS make targets for consistency (dns-setup / dns-remove)
- Rename test-act target to test-act-all

https://chatgpt.com/share/6970e503-e818-800f-8724-d791a3abfb33
2026-01-21 15:38:54 +01:00
kevinveenbirkenbach a565344ed5 Solved shellcheck 2026-01-21 00:25:35 +01:00
kevinveenbirkenbach f08993049c feat(inventory): add dynamic inventory resolver and propagate INVENTORY_DIR everywhere
- add scripts/inventory/resolve.sh to compute INVENTORY_DIR based on runtime context (act/github/local)
- resolve INVENTORY_DIR dynamically in Makefile via resolve.sh
- enforce INVENTORY_DIR usage across all local deploy/test scripts
- propagate INVENTORY_DIR into container and development CLI commands
- remove hardcoded /etc/inventories paths

https://chatgpt.com/share/69700d69-b270-800f-ad50-d9ff486378b9
2026-01-21 00:18:57 +01:00
kevinveenbirkenbach 0a4040d14d Lint 2026-01-20 22:00:49 +01:00
kevinveenbirkenbach d22dc4748b **refactor(cli): switch development dispatcher to argparse subcommands**
* Replace runpy-based nested CLI dispatch with argparse subcommand hub
* Auto-register development commands via `add_parser()` and execute `_handler`
* Fix `python -m cli.deploy.development <cmd>` resolution (e.g. `up`, `down`, `run`)

https://chatgpt.com/share/696fdea6-b208-800f-8b55-c4205c567aa2
2026-01-20 20:59:24 +01:00
kevinveenbirkenbach 50816243ed Solved linting hints 2026-01-20 20:50:50 +01:00
kevinveenbirkenbach d35e932e0c refactor(cli): switch development CLI to dynamic dispatcher and fix test paths
- Replace static argparse-based CLI in cli.deploy.development with dynamic
  module dispatcher compatible with nested command discovery.
- Ensure cli.deploy.development is executable via python -m and root dispatcher.
- Fix unit test path resolution to work locally and in CI.
- Make docker-* Makefile targets depend on install.
- Simplify test runner script and keep Nix flake config compatibility.

This aligns the CLI architecture with integration test discovery
(cli/**/__main__.py) and removes the need for manual subcommand registration.

https://chatgpt.com/share/696fdbbc-b6d8-800f-a8b5-067aa7e38706
2026-01-20 20:46:20 +01:00
kevinveenbirkenbach 4e0d09e773 refactor(dev-orchestrator): centralize image build and add up --when-down SPOT
- Introduce cli.deploy.development build subcommand as single build SPOT
- Remove legacy compose.build_infinito implementation
- Make up use build-missing logic via build handler
- Add --when-down flag to up (skip startup if stack already running)
- Simplify restart/run to reuse up handler
- Ensure tests use up --when-down to avoid stack interference
- Make test targets depend on build-missing

https://chatgpt.com/share/696fd3c9-99f0-800f-a0ab-33a91058419a
2026-01-20 20:13:02 +01:00
kevinveenbirkenbach bbea43f669 refactor(ci,tests): migrate discovery and local deploy to development compose stack
- Replace legacy ci scripts with meta build-test-matrix and resolve-invokable-roles
- Switch log tailing to scripts/meta/log-tail.sh
- Rename local DNS helpers to scripts/dns/*
- Run ci-deploy-discover via development docker-up
- Replace deploy-local-full with development compose orchestrator
- Ensure INFINITO_DISTRO is passed explicitly for local full deploy

https://chatgpt.com/share/696f9ccd-e4d8-800f-9c67-9c071188ecd0
https://chatgpt.com/share/696facf5-b2fc-800f-8a33-3352dd193c54
2026-01-20 17:27:09 +01:00
kevinveenbirkenbach f52a012829 feat(cli): replace test deploy runner with development CLI (up/down/stop/restart/run)
https://chatgpt.com/share/696f9ccd-e4d8-800f-9c67-9c071188ecd0
2026-01-20 16:18:31 +01:00
kevinveenbirkenbach 811501d85f refactor(ci): move compose down logic into dedicated down module with safe /mnt/docker cleanup
https://chatgpt.com/share/696f4386-fbbc-800f-bdde-f1e6c4145c84
2026-01-20 12:54:54 +01:00
kevinveenbirkenbach 5cfa73941d **fix(cli): remove roles_root argument and resolve roles path internally**
https://chatgpt.com/share/696e3d20-276c-800f-87a7-a7052989ffe4
2026-01-19 19:10:20 +01:00
kevinveenbirkenbach c92aff464b feat(ci): add min_storage validation and CI filtering with warnings
- Add sufficient_storage CLI for validating docker.services.<entity>.min_storage
- Introduce role_resource_validation utility with human-friendly size parsing
- Filter CI app discovery by available runner storage (default: 12GB)
- Emit GitHub Actions warnings for:
  - missing min_storage metadata (treated as 0GB)
  - roles exceeding available runner storage
- Add BigBlueButton min_storage metadata (50GB)
- Add unit tests for validation logic and CLI behavior
- Add humanfriendly dependency for size parsing

https://chatgpt.com/share/696e097b-1038-800f-b8fc-23e9dadfa021
2026-01-19 11:37:28 +01:00
kevinveenbirkenbach daab131b71 ci: auto-detect storage constraints and enable aggressive docker cleanup
- add storage detection utility for CI (DockerRootDir free space check)
- automatically set STORAGE_CONSTRAINED in cli.deploy.test when disk < 100 GiB
- forward STORAGE_CONSTRAINED into generated CI inventory via --vars
- introduce global STORAGE_CONSTRAINED default in group_vars
- run aggressive docker prune handler only in storage constrained mode
- mark CI inventory environment as development

https://chatgpt.com/share/696d6a11-2d90-800f-83e8-a30f6229e2e0
2026-01-19 00:17:31 +01:00
kevinveenbirkenbach 72929e83fc feat(shopware,credentials): add strong OpenSearch admin password generation
- add strong_password generator with special characters
- introduce opensearch_initial_admin_password credential
- wire OpenSearch admin password via env template
- decouple OpenSearch admin password from Shopware admin user
- minor formatting cleanup in service resolver
2026-01-18 23:33:19 +01:00
kevinveenbirkenbach 160a3a3233 feat(services): resolve shared onlyoffice provider role
- add onlyoffice enabled+shared → web-svc-onlyoffice service resolution
- enable shared onlyoffice for Nextcloud
- fix missing trailing newlines

https://chatgpt.com/share/696d54a1-9dbc-800f-8507-088bad56bc48
2026-01-18 22:50:45 +01:00
kevinveenbirkenbach e11dd60714 feat(services): resolve coturn as shared provider and refactor BBB coturn handling
- add coturn enabled+shared → web-svc-coturn service resolution
- refactor BigBlueButton coturn logic to distinguish local vs shared coturn
- introduce BBB_COTURN_LOCAL_ENABLED flag
- split TURN secrets into local and shared variants
- update dependencies, env and compose templates accordingly
- enable shared coturn in BigBlueButton and Nextcloud configs

https://chatgpt.com/share/696d54a1-9dbc-800f-8507-088bad56bc48
2026-01-18 22:46:02 +01:00
kevinveenbirkenbach 7a1fb02554 test(cli): add unit tests for combined + services resolution (services edges, desktop, cycles)
- add role_config_path support and services edges in combined resolver/tree
- add services resolver mapping with desktop enabled rule (non-shared)
- add unittest coverage for repo_paths, yaml parsing, services mapping/BFS, and combined tree cycles
- update tests to reflect new stable path expectations and tree output

https://chatgpt.com/share/696cb80f-e8f4-800f-b062-e47985b4b074
2026-01-18 11:37:41 +01:00
kevinveenbirkenbach cf216014e3 refactor(cli): introduce combined role resolution and reorganize meta resolvers
- add cli.meta.applications.resolution.combined to resolve run_after + app-only dependencies
- add optional --tree output for combined resolution
- add cli.meta.applications.resolution.dependencies (app-only) resolver
- move run_after resolver to cli.meta.applications.resolution.run_after
- update test deploy dependency resolver to use combined module
- add unit tests for combined/dependencies/run_after resolvers
- minor cleanup: remove stray header comment, fix newline in vars, tidy YAML formatting

https://chatgpt.com/share/696ca531-9644-800f-98c0-bae58c0aff41
2026-01-18 10:17:26 +01:00
kevinveenbirkenbach fa422ed7fe cli: allow comma-separated application IDs for --id flag
- Support space-separated, comma-separated and mixed --id values
- Normalize and deduplicate application IDs before validation
- Fix Makefile to pass APP without quoting
- Add unittest coverage for ID normalization

https://chatgpt.com/share/696be0c2-816c-800f-8b07-3c5cd0304ac5
2026-01-17 20:19:13 +01:00
kevinveenbirkenbach 357a9bda83 fix(users): normalize reserved flag handling and align CLI/tests
- Always emit reserved flag in generated users (default: false)
- Do not override existing user definitions via --reserved-usernames
- Normalize description default to empty string
- Align unit tests with new reserved semantics
- Fix YAML quoting and trailing newline issues
- Clean up minor formatting and config inconsistencies

https://chatgpt.com/share/696bb761-0048-800f-9cf7-aebd3fda75be
2026-01-17 17:22:50 +01:00
kevinveenbirkenbach fd983bc69c refactor(ci,network): make DNS, IP and domain configuration fully env-driven
- Introduce env.ci as single source of truth for CI networking
- Make docker-compose, CoreDNS and Docker daemon fully env-driven
- Render CoreDNS Corefile from template via envsubst
- Replace hardcoded IPs/domains with environment variables
- Integrate env-based networking into Ansible defaults
- Remove static Corefile and daemon.json

This makes DNS and networking deterministic and reproducible across
host, CoreDNS, infinito runner and Docker-in-Docker.

https://chatgpt.com/share/696a3dda-5c6c-800f-8a16-47c0c604e1e3
2026-01-16 14:30:14 +01:00
kevinveenbirkenbach a06f395640 Replaced -i with --id 2026-01-16 00:47:34 +01:00
kevinveenbirkenbach 505af3e2ad cli: make --log require explicit directory and create parents automatically
- Change --log flag to require explicit log directory (--log <LOG_DIR>)
- Automatically create log directory with parents=True
- Harden permissions best-effort (0700)
- Update CLI help text accordingly
- Adjust Makefile test targets to pass explicit log directory
- Ignore logs/ via .gitignore and .dockerignore
- Refactor flag parsing to correctly detect deploy with --log
- Add dedicated logging unit tests
- Update existing tests for new log semantics
- Remove deprecated -i alias for --id in deploy command

https://chatgpt.com/share/6969680a-dbf8-800f-aafb-3d32d6212ba0
2026-01-15 23:19:31 +01:00