Commit Graph
64 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 4.7 a8a117c86e refactor(yaml): route every YAML touchpoint through utils.cache.yaml + lint
New lint guard
==============
tests/lint/repository/test_no_direct_yaml_calls.py AST-walks the
production tree (utils/, cli/, plugins/, filter_plugins/,
lookup_plugins/, roles/, scripts/, tasks/) and flags any direct
`yaml.safe_load` / `yaml.safe_dump` / `yaml.load` / `yaml.dump` /
`yaml.safe_load_all` / `yaml.dump_all` call. Aliased imports are
tracked: `import yaml as Y` ⇒ `Y.safe_load(...)` is flagged;
`from yaml import safe_load` ⇒ bare `safe_load(...)` is flagged.

Per-line opt-out: `# noqa: direct-yaml` on the call's physical line
(or any line spanned by a multi-line call). No central exemption
list — every legitimate exception lives next to its call with a
human-readable rationale in the surrounding comment.

Cache module additions
======================
* `dump_yaml_str(data)` — symmetric to `safe_dump(data)` with no file
  path. Used for stdout output and filter return values where the
  cache cannot help but the API needs to stay consistent.
* `load_yaml_str(text)` — symmetric to `safe_load(text)`. Used when
  YAML comes from an HTTP body, a mutated buffer, or a mock-patched
  source whose content does not correspond to a stable on-disk file.

Migrated production call sites (33 → 0 direct-yaml violations)
==============================================================
* utils/{entity_name_utils, valid_deploy_id, invokable, service_registry,
  domains/list, applications/{config, in_group_deps},
  docker/image/discovery, meta/{port_bands, scan},
  roles/{dependency_resolver, meta_lookup, resource_validation}}
* cli/{build/{graph, inventory/full/command}, create/inventory/{host_vars,
  mirror_overrides}, fix/vars_main_files, meta/{applications/in_group_deps,
  callorder}, mirror/resolver, validate/inventory}
* plugins/{filter/{compose_volumes, get_all_application_ids, get_role,
  invokable_paths, role_path_by_app_id}, lookup/applications_current_play}
* roles/{svc-db-postgres/filter_plugins/split_postgres_connections,
  web-app-bigbluebutton/filter_plugins/compose_mods,
  web-app-dashboard/lookup_plugins/docker_cards}

Behaviour adjustments
=====================
* utils/roles/dependency_resolver — `safe_load_all` (multi-doc) →
  single-doc `load_yaml_any` wrapped in a list. Ansible task files are
  single-doc by convention; the downstream multi-doc loop stays intact
  for forward compatibility.
* roles/sys-svc-compose-ca/files/compose_ca.py — KEEPS direct
  yaml.safe_load / yaml.safe_dump (annotated with noqa). The script is
  copied to the target host via `ansible.builtin.copy` and the target
  has no `utils/` package on PYTHONPATH.

noqa-annotated direct-yaml call sites (with rationale)
======================================================
* utils/cache/yaml.py — IS the cache; would recurse.
* utils/handler/yaml.py — vault-aware Loader subclass that wraps
  \!vault scalars in VaultScalar; cache-by-path collapses the wrapping.
* cli/encrypt/inventory/__main__.py — vault-aware Dumper subclass that
  emits \!vault literal blocks; the cached dump_yaml writes plain YAML
  which is the wrong output here.
* roles/sys-svc-compose-ca/files/compose_ca.py — runtime-deployed.
* tasks/utils/migrate_meta_layout.py — once-off migration script.

Validation
==========
* Unit tests: 1573 passed.
* Lint:       56 passed (incl. new test_no_direct_yaml_calls).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 13:10:14 +02:00
kevinveenbirkenbachandClaude Opus 4.7 b3de2a4d7c refactor(meta): land req-011 meta/info.yml + utils/cache cleanup
req-011 — descriptive role-level metadata
  * Move logo / homepage / video / display out of
    meta/main.yml.galaxy_info into a new project-owned
    roles/<role>/meta/info.yml (file-root convention, no `info:` wrapper).
  * 78 role meta/main.yml files migrated via ruamel round-trip
    (key-order / quoting / inline-comments preserved). Roles with none
    of the four fields are left untouched (no empty info.yml created).
  * web-app-dashboard/lookup_plugins/docker_cards.py reads logo +
    display from meta/info.yml; description / galaxy_tags continue to
    come from galaxy_info (Galaxy-spec fields).
  * docker_cards.py now routes ALL its file reads through the
    process-wide caches: load_yaml_any for vars/main.yml,
    meta/main.yml, meta/info.yml; read_text from utils.cache.files for
    README.md. The orphan `import yaml` is gone.

New utils/cache/files (renamed from tests/utils/fs)
  * The process-wide filesystem cache (read_text, iter_project_files,
    iter_project_files_with_content, _all_project_files) is not
    test-specific; it now lives under utils/cache/files.py next to the
    other process-wide caches. 34 consumer imports rewritten across
    tests/ and docs/. The empty tests/utils/ package is removed.
  * New unit suite tests/unit/utils/cache/test_files.py — 12 cases
    covering read_text, iter_project_files (skip-dirs, extensions,
    exclude_tests, exclude_dirs, walk-cache invariants), the combined
    iterator, non-UTF-8 silent skip, and _reset.

utils/cache redundancy cleanup
  * applications.py + users.py + cli/create/inventory/services_disabler.py
    drop their direct yaml.safe_load calls and their private
    _load_yaml_mapping/_load_yaml_variant_list copies; everything routes
    through utils.cache.yaml.load_yaml / load_yaml_any. The dead
    utils.cache.base._load_yaml_mapping is removed.
  * users.py:_load_store_users (tokens.yml read) likewise migrated.
  * _load_yaml_variant_list moves out of base.py to applications.py
    (its only consumer) and is renamed _load_variants_overrides — also
    routes through the cached loader now.
  * utils.cache.yaml._reset_cache_for_tests is renamed _reset (kept as
    alias) for parity with the per-domain reset helpers.
  * utils.cache.files grows a _reset() that clears its two LRU caches.
  * utils.cache.__init__._reset_cache_for_tests now orchestrates ALL
    six caches (applications, users, domains, base, yaml, files) so
    tests get clean state across cache layers — needed because
    applications/users now share the yaml cache and would otherwise
    serve stale parses across reset boundaries.
  * utils.cache.files imports PROJECT_ROOT from base instead of
    redefining it.
  * tests/unit/utils/cache/test_data.py + test_base.py: obsolete
    TestLoadYamlMapping / TestLoadYamlVariantList classes deleted
    (covered by test_yaml.py + test_variants.py).
  * README.md updated for the new module list (files, yaml) and the
    six-module orchestrator.

Galaxy schema lint
  * tests/lint/repository/test_role_meta_main_galaxy_schema.py: split
    galaxy_info field requirements into hard-required {author,
    description, license} and soft-recommended {min_ansible_version,
    platforms}. Soft-recommended -> emit a GitHub Actions ::warning::
    annotation via utils.annotations.message.warning (titled
    "meta/main.yml galaxy_info" with file= pointing at the offender),
    no test failure. Rationale: min_ansible_version + platforms are
    only enforced by Galaxy when *publishing*, which Infinito.Nexus
    does not do.
  * Galaxy lint goes green (was 226 offenders before req-011 +
    license_url/repository/documentation strip; was 62 before this
    relaxation; now 0 hard offenders, 62 soft warnings).
  * roles/svc-opt-keyboard-color/meta/main.yml: backfill `license`
    (only role missing the hard-required field).

New lint
  * tests/lint/repository/test_role_meta_info.py — enforces meta/info.yml
    file-root convention, the four-field allowlist (logo, homepage,
    video, display), and shape (logo.class string, homepage/video
    non-empty strings, display bool). Also fails if any
    meta/main.yml.galaxy_info still carries one of the four migrated
    fields.

Docs
  * docs/contributing/design/services/layout.md: new section
    "Descriptive Role-Level Metadata — meta/info.yml" with allowed-
    fields table, optionality rule, materialised path, and consumer
    note. File-Layout table grows the meta/info.yml row;
    meta/main.yml row gets the explicit "no logo/homepage/video/display"
    note. req-011 added to header and Related Pages.
  * utils/cache/README.md: files.py + yaml.py rows added; six-module
    reset orchestrator documented.

Validation
  * Unit:        1573 passed.
  * Lint:        55 passed.
  * Integration: 161 passed (1 skipped) via pre-commit hook.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 12:27:17 +02:00
kevinveenbirkenbachandClaude Opus 4.7 ed04dad5d5 refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).

req-008 — role meta layout
  * Move every role's metadata under meta/<topic>.yml with file-root
    convention (no wrapping key matching the basename):
      - schema/main.yml + config/main.yml.credentials → meta/schema.yml
      - users/main.yml                                → meta/users.yml
      - config/main.yml.server                        → meta/server.yml
      - config/main.yml.rbac                          → meta/rbac.yml
      - config/main.yml.compose.services + every other top-level key
        of config/main.yml inlined under <entity>.<key>
                                                      → meta/services.yml
      - config/main.yml.compose.volumes               → meta/volumes.yml
  * Drop the legacy schema/, users/, config/ directories from every role.
  * meta/schema.yml: support nested credential keys, optional `default:`
    field (Jinja literal, not rendered/validated/regenerated), implicit
    `algorithm: plain`.
  * Materialised paths drop the `compose.` prefix:
      compose.services.<e>.<…> → services.<e>.<…>
      compose.volumes.<k>      → volumes.<k>
      <former-top-level>.<…>   → services.<entity_name>.<former-top-level>.<…>
  * Update every Python consumer (utils/cache, utils/manager, utils/applications,
    plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
    and task file to the new paths.
  * No legacy fallback. Repository-wide grep for `compose.services.` /
    `compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
    is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
  * Add tests/lint/repository/test_role_meta_layout.py guarding against the
    old paths returning, and an explicit lint guard that meta/variants.yml
    files do not carry a `compose:` wrapper.

req-009 — per-role networks/ports
  * Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
    dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
    dns_resolver}.
  * Delete group_vars/all/09_ports.yml entirely; ports now live on the
    service entity that exposes them at
    roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
  * Rename + int-cast every former top-level `services.<entity>.port`
    field to `services.<entity>.ports.inter` (single int).
  * Always-category-keyed maps for ports.local / ports.public; relay
    spans use {start, end} integers.
  * Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
    of truth for the per-<scope>.<category> band ranges, consumed by both
    the new CLI helpers and the lint test.
  * New CLI helpers (cli meta ports suggest, cli meta networks suggest)
    walk the live role tree gap-first then increment-fallback, with
    deterministic output and non-zero exit on capacity overflow.
  * cli create role consumes both helpers when scaffolding a new role.
  * Lint extensions: bare-int port shape rejection, host-bound port
    collision check (single ints + relay span integers, inter excluded),
    band-membership check with documented allow-list for the legacy
    web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.

req-010 — run_after / lifecycle migration
  * Move galaxy_info.run_after and galaxy_info.lifecycle out of
    meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
    lifecycle}; <primary_entity> is the value returned by
    get_entity_name(role_name).
  * For multi-entity roles whose primary entity has no compose service
    (web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
    gen roles), create a dedicated metadata-only top-level entry under
    <primary_entity>.
  * Introduce utils/roles/meta_lookup.py with get_role_run_after /
    get_role_lifecycle that resolve <primary_entity> once and read from
    meta/services.yml; degrade gracefully when meta/services.yml is
    absent OR the field is absent; raise MetaServicesShapeError when the
    file exists but is malformed.
  * Update every consumer (utils/service_registry, utils/invokable,
    cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
    lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
    cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
    init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
    use the helper.
  * Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
    values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
    run_after/lifecycle never on a non-primary entity; empty run_after
    lists rejected.

Also folded into this change set
  * services_disabler now writes `services.<svc>.{enabled,shared}` into
    host_vars instead of the old `compose.services.<svc>.{...}` path; the
    discover/find pair walks meta/services.yml and only treats entries
    carrying enabled/shared/provides as provider declarations (so
    metadata-only entries from req-010 do not create spurious provider
    rows).
  * service_registry skips `run_after` deps whose target was filtered
    out via SERVICES_DISABLED instead of failing the whole load.
  * cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
    consume the new applications.<role>.services.<svc>.image shape
    (mirror.md updated to match).
  * utils/domains/list.build_applications_from_roles now reads
    meta/server.yml directly (it had been silently returning {} since
    config/main.yml stopped existing).
  * utils/roles/resource_validation.filter_roles_by_min_storage reads
    meta/services.yml.<entity>.min_storage instead of the legacy
    compose.services path.
  * web-app-bluesky pre-clones the social-app build context BEFORE
    sys-svc-compose so the build context is on disk by the time
    sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
  * 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
    web-app-wordpress, web-app-keycloak) lose their `compose: services:`
    wrapper so deep-merge over meta/services.yml lands on the new
    file-root shape.

Validation
  * Unit tests:        1571 passed.
  * Lint tests:        50 passed.
  * Integration tests: 161 passed (1 skipped).
  * Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
    set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
    discourse,mattermost}.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 09:58:26 +02:00
kevinveenbirkenbach 12dba01156 refactor(utils): consolidate cache modules into utils/cache/ package
Both in-process cache layers — the cached YAML loader and the cached
runtime-data getters — now live next to each other under
`utils/cache/`. The package gains a README that documents the
"orthogonal layers, no third cache" rule so future contributors hit
the right primitive on the first look.

Moves:
- `utils/yaml_cache.py` -> `utils/cache/yaml.py`
- `utils/runtime_data.py` -> `utils/cache/data.py`
- `tests/unit/utils/test_yaml_cache.py` -> `tests/unit/utils/cache/test_yaml.py`
- `tests/unit/utils/test_runtime_data.py` -> `tests/unit/utils/cache/test_data.py`
- `tests/unit/utils/test_variants.py` -> `tests/unit/utils/cache/test_variants.py`

New:
- `utils/cache/__init__.py` (empty per project convention)
- `utils/cache/README.md` describing the two layers, when to reach for
  which, lifetime, test-only helpers, and design guidelines (RFC 2119
  keywords, emoji-after-headline, no self-SPOT declaration, no
  slash-in-link-text).
- `tests/unit/utils/cache/__init__.py`

Mechanical updates:
- All imports across cli/, plugins/lookup/, tests/, and roles/ rewritten
  from `utils.yaml_cache` -> `utils.cache.yaml` and
  `utils.runtime_data` -> `utils.cache.data` (40+ files).
- `REPO_ROOT` parents-index in the moved tests bumped from `parents[3]`
  to `parents[4]` to reflect the deeper nesting.
- Six broken markdown links in docs/ updated to the new module paths
  (test_markdown_links lint test stays green).
- Two test files that used the `from utils import runtime_data` form
  now use `from utils.cache import data as runtime_data` so the local
  alias remains.

Tests: 848 passed across unit + lint suites (1 pre-existing collection
error in test_host_vars.py unrelated to this change).
2026-04-25 17:17:14 +02:00
kevinveenbirkenbach 35b47a2048 perf(cli): cache YAML loads in-process via utils/yaml_cache
Many CLI tools walk the same `roles/<role>/<config|vars|meta>/...yml`
files multiple times during a single invocation (dependency
resolution, service resolution, role introspection, ...). Each direct
`yaml.safe_load(path.read_text())` re-parses YAML and re-touches disk;
for `infinito create inventory` over the ~600-role tree this dominates
wall-clock time.

This change introduces `utils/yaml_cache.py`, a small in-process,
per-absolute-path memoising loader, and migrates the three existing
ad-hoc loaders to delegate to it.

`utils/yaml_cache.py` exposes:
- `load_yaml(path, *, default_if_missing=_MISSING)`: cached read,
  dict only at the YAML root, raises `FileNotFoundError` by default
  for missing files (callers can opt into a default value).
- `dump_yaml(path, data)`: write-through that evicts the cached entry,
  so a CLI tool that edits a file mid-run sees the new content on the
  next read.
- `invalidate(path)`: explicit per-path drop for the rare cases where
  an external process touches the file.
- `_reset_cache_for_tests()` for unit-test isolation.

Cache semantics are documented in the module docstring; the key point
is that the same dict instance is returned on cache hit (no deep
copy), so callers that want to mutate the result MUST copy first.
This trades that constraint for ~5x fewer disk reads during the hot
resolution loops without measurably increasing memory.

Migrated wrappers (each keeps its existing public surface + domain
error type so call sites stay untouched):

- `cli/create/inventory/yaml_io.py` — `load_yaml(path)` /
  `dump_yaml(path, data)`. The legacy "missing file -> {}" contract
  stays via `default_if_missing={}`. Non-mapping roots still surface
  as `SystemExit`.
- `cli/meta/applications/resolution/combined/yaml_utils.py` —
  `load_yaml_file(path)` re-raises errors as `CombinedResolutionError`.
- `cli/meta/applications/resolution/services/yaml_utils.py` —
  `load_yaml_file(path)` re-raises errors as `ServicesResolutionError`.

Tests: `tests/unit/utils/test_yaml_cache.py` covers cache hit,
path-resolution sharing, missing-file behaviour with and without the
default, non-mapping rejection, write-through eviction, and explicit
invalidation. All existing unit + lint test suites stay green.

Phase 2 (auditing the remaining direct `yaml.safe_load` call sites in
`cli/` and migrating the hot ones) is intentionally out of scope here
to keep the diff focused; this commit is the prerequisite SPOT.
2026-04-25 17:17:14 +02:00
kevinveenbirkenbachandClaude Opus 4.7 6d53214ea0 fix(services-disabler): resolve providers via service registry
find_provider_roles now uses build_service_registry_from_roles_dir, the
same resolver the SERVICES_DISABLED pre-check uses. Previously it matched
only roles with a literal compose.services.<svc>.image entry, so services
exposed via compose.services.<entity>.provides (e.g. mailu → email) were
invisible to the init phase. The pre-check then flagged the unresolved
provider role as a conflict and aborted every fresh-purged deploy that
disabled "email".

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-21 18:40:38 +02:00
kevinveenbirkenbach 50e4cb8422 chore(group_vars/all): renumber files to close 04/05 gap
- Rename 06..19 → 04..17 so the sequence is dense and gap-free.
- Update hard-coded references in cli/, tests/, and PR template.
- Add README.md describing scope, load order, and naming rules
  per docs/contributing/documentation.md standards.
2026-04-19 15:55:52 +02:00
kevinveenbirkenbachandClaude Sonnet 4.6 3cfdc4bdf7 feat(services-disabler): propagate SERVICES_DISABLED through deploy stack and add conflict detection
Adds inventory consistency guard that fails fast when SERVICES_DISABLED
conflicts with an already-deployed inventory, and ensures the env var is
forwarded through the dedicated runner, development exec command, and
local test scripts.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14 19:36:16 +02:00
Environment TestandClaude Sonnet 4.6 aa33256cde feat(mirror): split resolver output, move images to defaults, add lookup plugin
- resolver: output both applications: (config/main.yml compose images) and
  images: (defaults/main.yml images) keyed by source_file
- image_discovery: scan defaults/main.yml instead of vars/main.yml for images:
- mirror_overrides: write mirror URLs to applications.*.compose.services.*
  (mirror_policy-aware) and images_overrides.*.* (always merged, if_missing)
- roles/sys-ctl-hlth-csp, roles/test-e2e-playwright: move images: from
  vars/main.yml to defaults/main.yml for correct Ansible variable precedence
- plugins/lookup/image.py: new Ansible lookup plugin that prefers
  images_overrides over role-local images defaults
- docs: add artefact/image.md and artefact/mirror.md as SPOTs; add
  code/role/image.md contributor guide; link from ci.md
- tests: update test_util, test_mirror_overrides; add test_resolver,
  test_publish (404/401 coverage), test_image (lookup plugin)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 11:17:42 +02:00
kevinveenbirkenbachandClaude Sonnet 4.6 18779de223 feat(services-disabler): set enabled/shared false for all roles defining service
Scans roles_dir to find every role whose config/main.yml defines the
disabled service, then creates or updates the compose.services entry in
host_vars for each matching role — including roles not yet present in
applications. roles_dir is now a required parameter (no Optional fallback).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-30 02:02:36 +02:00
kevinveenbirkenbachandClaude Sonnet 4.6 9d863f1ef4 feat(inventory): add SERVICES_DISABLED env var support and environment test flow
- Add services_disabler.py: parse SERVICES_DISABLED, disable services in host_vars,
  remove provider roles from inventory
- Wire apply_services_disabled_from_env into cli/create/inventory/cli.py after deploy
- Add unit tests for all services_disabler functions
- Update hardware.md and variables.md with SERVICES_DISABLED documentation
- Update environment.sh: deploy dashboard with SERVICES_DISABLED=matomo first,
  assert dashboard reachable + matomo not reachable, then deploy matomo and assert
  reachable; extract assert_http_status / assert_http_status_not helpers

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 15:00:24 +02:00
kevinveenbirkenbach 2b0ed2eacf Merge branch 'feature/pixelfed-playwright' into feature/taiga-ldap 2026-03-28 09:22:30 +01:00
kevinveenbirkenbach 67fadb57b8 refactor: normalize inventory filename to devices 2026-03-28 05:40:13 +01:00
kevinveenbirkenbach 2f0edbb2fc Fix oauth2 proxy cookie secret generation 2026-03-26 01:15:08 +01:00
kevinveenbirkenbach 189267496d Rename module_utils package to utils 2026-03-21 02:05:39 +01:00
kevinveenbirkenbach 3c8f99d53a Removed legacy args 2026-02-10 11:56:21 +01:00
kevinveenbirkenbach b78ab29218 Renamed config entry from docker to compose
https://chatgpt.com/share/69870f54-d708-800f-9870-e9ae5a4bff44
2026-02-07 11:09:36 +01:00
kevinveenbirkenbach e1e1f3eb11 chore(vars): rename PATH_* to DIR_/BIN_/FILE_ conventions
https://chatgpt.com/share/6985cf49-3c74-800f-af97-f68b1969ab01
2026-02-06 12:23:46 +01:00
kevinveenbirkenbach 8a81630391 Added newline at file and POSIX compatible 2026-02-04 19:49:56 +01:00
kevinveenbirkenbach 9a84fafec5 feat(inventory): add mirror_policy to control mirror image overrides
- Default policy (if_missing): manual image/version win, mirror fills blanks only
- Add mirror_policy=force to always overwrite image/version
- Add mirror_policy=skip to ignore mirror completely
- Treat empty/whitespace values as missing
- Preserve non-image/version service keys
- Extend unit tests to cover all mirror policy scenarios

https://chatgpt.com/share/6981036a-7e40-800f-b43e-13c66cc55fac
2026-02-02 21:04:53 +01:00
kevinveenbirkenbach a70a033864 feat(inventory): add --mirror support to override docker images from mirrors file
- add --mirror flag to inventory creator
- resolve mirrors.yml automatically or via explicit path
- apply mirror overrides to applications.*.docker.services.* (image/version only)
- keep existing host_vars keys untouched
- add unit tests for mirror override behavior

https://chatgpt.com/share/6980cdb1-1e5c-800f-b25f-a1af757e4239
2026-02-02 17:15:30 +01:00
kevinveenbirkenbach 8bff1b984d refactor(jinja): add strict recursive renderer and use in tls/compose lookups
Introduce module_utils/jinja_strict.py to render nested Jinja-in-vars with hard-fail semantics. Refactor cert_plan, compose_f_args, and compose_ca_inject_cmd to use render_strict so no "{{ ... }}" leaks into generated paths/commands, and add unit tests.

https://chatgpt.com/share/697b8e1b-54fc-800f-b895-7161f00ebd51
2026-01-29 17:42:28 +01:00
kevinveenbirkenbach 8bf2b4c1e1 Ruff 2026-01-12 17:50:43 +01:00
kevinveenbirkenbach 198a2fe3cf fix(ansible): enforce system python interpreter and install python-ldap via OS packages
- Stop auto-injecting ansible_python_interpreter from PYTHON env for localhost
- Set ansible_python_interpreter globally to /usr/bin/python3
- Add interpreter to sample inventory
- Remove role-local interpreter overrides
- Switch python-ldap installation to distro package (python-ldap / python3-ldap)
- Drop pip-based python-ldap installation to avoid venv/libldap mismatches

https://chatgpt.com/share/6964335a-29cc-800f-ae65-6a35c7cd2559
2026-01-12 14:48:53 +01:00
kevinveenbirkenbach 66d50e2dde feat(inventory): resolve shared providers transitively and support schema-less roles
- Allow credentials generator to run for roles without schema/main.yml so
  transitive config-based generation still works.
- Make InventoryManager resilient to missing YAML by defaulting to {} and
  only loading schema when schema/main.yml exists.
- Add recursive shared-provider include resolution based on docker.services
  config and apply provider schemas/special rules transitively.
- Require database_password injection only when database.enabled=true and shared=true.
- Update unit/integration tests to create required files for .exists() checks and
  validate transitive provider role credential generation.

https://chatgpt.com/share/69639d93-4c10-800f-bc6c-8e5b9fb75088
2026-01-11 13:54:37 +01:00
kevinveenbirkenbach e4baa6d7ab Ruff 2026-01-11 01:01:44 +01:00
kevinveenbirkenbach 39f7f8fbc7 feat(create-credentials): generate shared provider app credentials from role schemas
Extend credentials generator to include schemas for shared provider roles
(OpenLDAP, Keycloak, Matomo, and svc-db-<type>) based on docker.services.*.shared
flags in role config. Credentials are now emitted/added for all involved
applications while preserving existing inventory formatting, and CLI overrides
can target specific apps via applications.<app>.credentials.<key>.
2026-01-11 00:00:29 +01:00
kevinveenbirkenbach 9d95eea3b0 fix(inventory): pin ansible_python_interpreter for localhost to avoid auto-discovery issues
Ensure that localhost inventories always define ansible_python_interpreter when using
ansible_connection=local. This prevents Ansible from falling back to interpreter
auto-discovery (interpreter_python=auto_silent), which could select unexpected Python
binaries inside containers (e.g. /usr/sbin/python3.13) and break module execution.

The interpreter is now pinned to $PYTHON if set, otherwise to sys.executable.

https://chatgpt.com/share/69628603-3e18-800f-a6f9-2b367728a9ff
2026-01-10 18:01:20 +01:00
kevinveenbirkenbach 0f7add0f1a Unify Mailu token handling via users[*].tokens and clean up inventory validation
- Migrate Mailu credentials from legacy mailu_token to tokens['web-app-mailu']
- Update all affected application templates to use the new token structure
- Adjust inventory validation to no longer special-case mailu_token
- Minor typing and formatting cleanups in inventory creation and tests

https://chatgpt.com/share/69554915-87c8-800f-afd5-12118dbb1107
2025-12-31 17:02:20 +01:00
kevinveenbirkenbach 533a1235fe Implement SPOT-based authorized_keys management for users
* Move SSH key handling to users.<name>.authorized_keys (list-only) and write keys via shared user role
* Enforce at least one SSH key for administrator user
* Wrap backup user keys with restricted command wrapper before writing authorized_keys
* Remove legacy inventory file copy and CLI flag/function for administrator authorized_keys
* Extend user setup generator to include authorized_keys field
* Rename administrator scripts path to root scripts path and update dependent variables
* Update inventory sample and docs to reflect new SSH key workflow

https://chatgpt.com/share/695510b8-23f4-800f-b5db-877b07a1469b
2025-12-31 13:01:45 +01:00
kevinveenbirkenbach ea04035e12 feat(config): rename SSL_ENABLED to TLS_ENABLED across stack
* Replace global `SSL_ENABLED` flag with `TLS_ENABLED`
* Update web protocol/ports (http/https, ws/wss) to use `TLS_ENABLED`
* Switch cert-related roles and nginx TLS header gating to `TLS_ENABLED`
* Update app env templates (BookWyrm, Confluence, Jira, Magento, OpenProject, WordPress, etc.) to use `TLS_ENABLED`
* Adjust inventory generator to write `TLS_ENABLED` and update CLI help text accordingly
* Update unit tests for inventory vars overrides to expect `TLS_ENABLED`

https://chatgpt.com/share/695306ef-a798-800f-b118-e07b525e1811
2025-12-29 23:55:20 +01:00
kevinveenbirkenbach 1e5ebbb05f feat(inventory): add --vars-file support for YAML-based host_vars overrides
- add --vars-file CLI option to load host_vars overrides from YAML
- merge vars-file overrides before JSON --vars (JSON takes precedence)
- implement deep-merge logic for YAML vars files
- update CI deploy script to use vars file instead of inline JSON
- add unit test for apply_vars_overrides_from_file
- remove obsolete MASK_CREDENTIALS_IN_LOGS TODO entry

https://chatgpt.com/share/6952e51d-6c50-800f-8764-31dc8c609cdb
2025-12-29 21:31:05 +01:00
kevinveenbirkenbach bf4af7c823 Renamed domain variables 2025-12-29 20:37:47 +01:00
kevinveenbirkenbach d2e88b6a8c feat(cli): eliminate runpy warnings for inventory + container entrypoints
* Remove legacy **main** re-export wrappers to avoid `runpy` "already in sys.modules" warnings
* Introduce `cli.deploy.container.command` as the stable implementation module
* Reduce `cli.deploy.container.__main__` to a minimal entrypoint delegating to `command.main`
* Re-export container helpers from package `__init__` without importing `__main__`
* Update unit tests to import/patch the new command module and keep CLI dispatch stable

https://chatgpt.com/share/69484a22-60dc-800f-a23d-068d7c45fe37
2025-12-21 20:27:26 +01:00
kevinveenbirkenbachandCopilot Autofix powered by AI efe7429c8b Potential fix for code scanning alert no. 121: Empty except
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-21 11:41:45 +01:00
kevinveenbirkenbach 6eb35ef450 refactor(cli): remove wildcard re-exports from __init__ wrappers
Replace `from .__main__ import *` with controlled re-exports based on `__all__`
to avoid namespace pollution and improve static analysis.

Public APIs remain unchanged while making re-export behavior explicit and
lint-friendly across all CLI subpackages.

https://chatgpt.com/share/6947c1d6-23d0-800f-9910-008ee277213c
2025-12-21 10:45:51 +01:00
kevinveenbirkenbach 59a68ff940 fix(inventory): run credentials generator via module instead of file path
Switch credentials generation to `python -m cli.create.credentials` to avoid
fragile file path execution after package migration and ensure the correct
interpreter and environment are used.

https://chatgpt.com/share/69471879-cd18-800f-a390-efa23d9d1813
2025-12-20 22:43:15 +01:00
kevinveenbirkenbach a354992f27 refactor(create-inventory): modularize CLI, stabilize path handling, add unit tests
- Split create-inventory logic into focused modules (cli, project, filters, host_vars, credentials, etc.)
- Replace fragile path construction with python -m module execution
- Add robust project root detection
- Centralize password generation
- Improve test coverage with per-module unit tests
- Preserve existing inventory and host_vars behavior

https://chatgpt.com/share/69471157-79c0-800f-8dec-25c0f8e38f03
2025-12-20 22:12:44 +01:00
kevinveenbirkenbach 23ce1e50b2 fix(inventory): correct project root detection to avoid duplicate cli path
Fix detect_project_root() to return the actual repository root instead of the
cli directory. This prevents incorrect path construction like `cli/cli/...`
when invoking build inventory scripts.

https://chatgpt.com/share/6946fe09-8730-800f-8eca-2503df61d9ae
2025-12-20 20:50:26 +01:00
kevinveenbirkenbach 2e6d973839 fix(build): align CLI module invocations and stabilize include generation
- Switch Makefile targets to direct module execution (cli.build.*)
- Remove obsolete flags and normalize tree invocation
- Minor formatting cleanups for readability and test stability
- Keep inventory and include paths consistent after package layout migration

https://chatgpt.com/share/6946eac4-eb48-800f-becb-f08420ab03a6
2025-12-20 19:28:14 +01:00
kevinveenbirkenbach 102735dba4 Renamed inventory.py to inventory/__main__.py 2025-12-20 18:30:20 +01:00
kevinveenbirkenbach 993ed8addd Renamed Makefile *.py includes 2025-12-20 18:28:37 +01:00
kevinveenbirkenbach 5c6afd2cf6 p2pkg executed 2025-12-20 18:25:37 +01:00
kevinveenbirkenbachandCopilot Autofix powered by AI aa586e0e2c Potential fix for code scanning alert no. 43: Unnecessary lambda
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-18 16:28:02 +01:00
kevinveenbirkenbach a38f36837f style: reformat code for readability and lint compliance 2025-12-18 14:08:01 +01:00
kevinveenbirkenbachandCopilot Autofix powered by AI 18ae1b2838 Potential fix for code scanning alert no. 40: Jinja2 templating with autoescape=False
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-12-18 12:42:33 +01:00
kevinveenbirkenbach 39b52f4449 fix(security): create vault password file with restrictive permissions
https://chatgpt.com/share/6943d45b-fc7c-800f-8742-413bde1b1516
2025-12-18 11:15:38 +01:00
kevinveenbirkenbach 7123841fb4 style: apply ruff autoformat 2025-12-17 16:37:21 +01:00
kevinveenbirkenbach 176c8799ce lint(ruff): fix E402 and related import issues
- Fix E402 by removing sys.path hacks and ensuring imports are at module top level
- Replace ad-hoc imports with proper package imports (filter_plugins, lookup_plugins, module_utils, cli)
- Make lookup plugins importable via package paths (e.g. lookup_plugins.application_gid)
- Clean up test imports to rely on installed package layout instead of sys.path manipulation
- Partially fix F821 caused by missing/invalid imports

This establishes a consistent, ruff-compliant import architecture across CLI, filters, plugins, and tests.
2025-12-17 14:36:29 +01:00
kevinveenbirkenbach 73f7c00acd Solved ruff E741 2025-12-17 13:54:51 +01:00