mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-08 11:46:35 +00:00
fix/dev-python/python-upper-bound
61
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  kevinveenbirkenbachandClaude Opus 4.7
|
e5de4e085a |
refactor(meta): wire services.<key>.{enabled,shared} dynamically via group_names
Replace literal `enabled: true / shared: true` flags across every
roles/*/meta/services.yml with a Jinja `"{{ '<role>' in group_names }}"`
form so the per-role services map becomes the single source of truth
for shared-service co-deployment, on a per-host basis. Inventory build
already runs through the templar before reaching the static `is True`
check, so runtime behaviour is unchanged; the new form only opens up
static-analysis and CLI tooling to the group-aware semantics.
Mechanism + lint:
- New integration test tests/integration/roles/meta/test_services_
dynamic_flags.py enforces the dynamic shape per top-level service
entry. Registered service keys must reference the registry-resolved
provider role; covered keys (e.g. oauth2 -> web-app-keycloak via
the new `covers:` block) reference the covering role.
- Two suppression placements documented in docs/contributing/actions/
testing/suppression.md: per-flag inline `# nocheck: dynamic-flag`
for databases (`enabled: true` stays literal, `shared` resolves
dynamically) and block-level above the key for entries whose both
flags legitimately stay literal (`css`, `javascript`, role-local
`redis`, `recaptcha`, `hcaptcha`, role-local containers with
`shared: false`).
Registry + helpers:
- utils/service_registry.py: new `is_explicit_truth(value)` helper
accepts `True` literal OR a Jinja string carrying `in group_names`.
Adopted by `resolve_service_dependency_roles_from_config`,
`ServicesResolver._is_enabled_shared`, `test_web_app_email_
integration`, `test_prometheus`, `test_universal_logout`, and the
run_after-explicit guard so static analysis keeps resolving deps.
- New `covers:` field on a primary service entity declares
transitive equivalents (web-app-keycloak.covers: [oauth2]).
Exposed via `build_role_to_covered_keys` and
`build_covered_key_to_role`, replacing the previously hard-coded
`_TRANSITIVE_SERVICE_EQUIVALENTS` / `_SPECIAL_REFERENCE_ROLE` maps
in the run_after and dynamic-flag tests.
Bulk migration:
- All 80 roles/*/meta/services.yml files migrated to the dynamic
form. DB consumer keys (mariadb, postgres, ldap) keep
`enabled: true # nocheck: dynamic-flag` literal with a dynamic
`shared`. Block-nocheck applied to css/javascript/recaptcha/
hcaptcha and role-local containers (e.g. mailu's admin/imap/smtp/
antispam/antivirus/oletools/webdav/fetchmail/webmail/resolver/
front, nextcloud's talk).
- web-app-prometheus's primary entity now declares
`provides: prometheus` + dynamic flags so the registry recognises
it as a service provider. Consequence: tasks/01_core.yml split out
with `- include_tasks: utils/once/flag.yml` as the first task and
tasks/main.yml reduced to the standard run_once-guarded loader.
- Seven minimal no-login apps (web-app-hugo, -bridgy-fed, -chess,
-littlejs, -navigator, -mini-qr, -mig) gained an explicit
`logout: { enabled: false, shared: false }` opt-out (with
`# nocheck: dynamic-flag` block-marker) to satisfy the universal-
logout schema check.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbach
|
d239b2ab19 | Merge branch 'main' into feature/web-app-fediwall | ||
|
kevinveenbirkenbachandClaude Opus 4.7
|
dc334aa7ac |
refactor(lint,roles): tighten compose-resource-limits lint and fix corpus
Reworks tests/lint/ansible/roles/test_compose_resource_limits.py from
warn-only-on-every-role to a strict failure scoped to actual
compose-service declarations:
1. Only roles that ship a `templates/compose.yml.j2` are scanned.
Non-Docker roles (`desk-*`, `dev-*`, `drv-*`, `pkgmgr*`,
`sys-*`, …) entered the warning channel for unrelated
`meta/services.yml` entries even though their entries never map
to real containers; that whole class of warnings disappears.
2. Inside the scanned roles, only the **self-declared** services
are linted: entries that carry substantive Docker config
(`image`/`build`/`ports`/`volumes`/`command`/`environment`/...)
OR a primary entity backed by a `files/Dockerfile` /
`templates/Dockerfile.j2`. Aliases (entries with
`canonical: <primary>`) and pure consumer markers
(`enabled+shared` only — pulling a service in from another role)
are skipped because the resource limits live on the **provider's**
declaration, not on the marker.
3. Failures are hard `self.fail` again so regressions block CI.
Concretely: `web-app-mailu/meta/services.yml.{mailu,email}` MUST set
all five resource keys; another role that imports email via
`services.email.enabled+shared: true` is a marker and is exempt.
Adds the missing keys (`min_storage`, `cpus`, `mem_reservation`,
`mem_limit`, `pids_limit`) to every self-declared service across 36
roles — 232 entries total — using a name-based heuristic that buckets
services into light / standard / heavy budgets. Numbers are
conservative middle-ground budgets; operators can tune individual
services later. Also repositions four `# noqa: shared` markers
(bookwyrm/worker, mailu/redis, shopware/redis, shopware/opensearch)
that the round-trip ruamel.yaml dump shifted into the wrong service
block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
07ada34155 |
feat(req-013): promote 6 web-* roles to beta with full per-role baselines
Implements requirement docs/requirements/013-alpha-to-beta-promotion.md for the in-scope subset: - web-app-minio: ✅/✅/✅ V0 OIDC + V1 LDAP variants green - web-app-joomla: ✅/✅/🛠️ V0 OIDC + V1 LDAP variants green - web-app-bridgy-fed: ❌/❌/❌ baseline (all 3 documented exceptions) - web-svc-libretranslate: 🛠️/❌/❌ baseline (LDAP+RBAC documented exceptions) - web-app-postmarks: 🛠️/🛠️/❌ baseline (RBAC documented exception) - web-app-bluesky: 🛠️/🛠️/❌ baseline (RBAC documented exception) Each role's `meta/services.yml.<entity>.lifecycle` is now `beta`. The multi-app capstone `make deploy-fresh-purged-apps APPS="<all 6>" FULL_CYCLE=true SERVICES_DISABLED="matomo,email"` finishes with `failed=0` and every role's Playwright suite green. Per-role highlights: * MinIO. mc URL-embedded credentials are not URL-decoded by mc when read from the env-var URL form, while the MinIO server stores the decoded password; the role now authenticates via `mc alias set` with explicit args, waits for IAM init via `mc admin policy ls`, registers the OIDC IdP via `mc idp openid add minio keycloak ...` (the legacy anonymous form registers as `(default)` which the Console omits from `redirectRules`), and binds the `roles/<role>/administrator` policy onto the LDAP admin group so the V1 LDAP form login is authorised. * Bridgy-fed. Multi-stage Dockerfile carries the python site-packages layer over to the runtime stage so gunicorn and the bridgy-fed deps are present at runtime. Removes the colon from `meta/main.yml.galaxy_info.description` so the dashboard config rendering does not produce invalid YAML. * Joomla. Adds `oidc:` enable + `ldap:` block in meta/services.yml, matrix-deploy variants for the two auth flows, and an extended Playwright env exposing OIDC issuer, biber persona, canonical domain, and the `<service>_SERVICE_ENABLED` gate flags. * LibreTranslate. Adds `container_port=5000`, drops the `dmbs_excl.yml.j2` depends_on partial that produced an empty `'': condition: service_healthy` map, and pins `LT_LOAD_ONLY=en,de` so the deploy gate's readiness probe does not race the upstream argos-translate model mirror. * Postmarks. Built from scratch as a full role (was a stub): tasks, Node-based Dockerfile that clones ckolderup/postmarks, compose template, env with SESSION_SECRET / ADMIN_KEY pulled from meta/schema.yml credentials, server.yml with subnet, Playwright env + spec. * Bluesky. Adds `files/playwright.spec.js` and `templates/playwright.env.j2` for the deploy gate. The Keycloak event-listener bridge to PDS `com.atproto.server.createAccount` remains to be built in follow-up work; until then the spec exercises baseline reachability and the OIDC/LDAP/RBAC gaps stay documented in `roles/web-app-bluesky/README.md` per lifecycle.md. Shared infrastructure: * `roles/test-e2e-playwright/tasks/run_one.yml`. The wait-for-app probe now accepts 4xx as proof of life; the prior `<400` boundary rejected legitimate 404 landing surfaces (bridgy-fed serves federation endpoints under specific paths, not `/`). 5xx remains the only signal of an unhealthy app. README.md SSO/RBAC exception sections per lifecycle.md (`docs/contributing/design/services/lifecycle.md`): * web-app-bluesky — RBAC exception * web-app-bridgy-fed — full SSO/RBAC exception * web-app-postmarks — RBAC exception * web-svc-libretranslate — LDAP + RBAC exception Out of scope for this commit (deferred to follow-up requirements): * Bluesky Keycloak event-listener SPI for PDS account auto- provisioning (needs Java SPI build pipeline + JAR mount + realm wiring). * LibreTranslate sidecar oauth2-proxy on the UI subpath only (must not gate the API endpoints). * Postmarks sidecar oauth2-proxy in front of the web UI. * Joomla community OIDC plugin install + group-mapping config. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
b3de2a4d7c |
refactor(meta): land req-011 meta/info.yml + utils/cache cleanup
req-011 — descriptive role-level metadata
* Move logo / homepage / video / display out of
meta/main.yml.galaxy_info into a new project-owned
roles/<role>/meta/info.yml (file-root convention, no `info:` wrapper).
* 78 role meta/main.yml files migrated via ruamel round-trip
(key-order / quoting / inline-comments preserved). Roles with none
of the four fields are left untouched (no empty info.yml created).
* web-app-dashboard/lookup_plugins/docker_cards.py reads logo +
display from meta/info.yml; description / galaxy_tags continue to
come from galaxy_info (Galaxy-spec fields).
* docker_cards.py now routes ALL its file reads through the
process-wide caches: load_yaml_any for vars/main.yml,
meta/main.yml, meta/info.yml; read_text from utils.cache.files for
README.md. The orphan `import yaml` is gone.
New utils/cache/files (renamed from tests/utils/fs)
* The process-wide filesystem cache (read_text, iter_project_files,
iter_project_files_with_content, _all_project_files) is not
test-specific; it now lives under utils/cache/files.py next to the
other process-wide caches. 34 consumer imports rewritten across
tests/ and docs/. The empty tests/utils/ package is removed.
* New unit suite tests/unit/utils/cache/test_files.py — 12 cases
covering read_text, iter_project_files (skip-dirs, extensions,
exclude_tests, exclude_dirs, walk-cache invariants), the combined
iterator, non-UTF-8 silent skip, and _reset.
utils/cache redundancy cleanup
* applications.py + users.py + cli/create/inventory/services_disabler.py
drop their direct yaml.safe_load calls and their private
_load_yaml_mapping/_load_yaml_variant_list copies; everything routes
through utils.cache.yaml.load_yaml / load_yaml_any. The dead
utils.cache.base._load_yaml_mapping is removed.
* users.py:_load_store_users (tokens.yml read) likewise migrated.
* _load_yaml_variant_list moves out of base.py to applications.py
(its only consumer) and is renamed _load_variants_overrides — also
routes through the cached loader now.
* utils.cache.yaml._reset_cache_for_tests is renamed _reset (kept as
alias) for parity with the per-domain reset helpers.
* utils.cache.files grows a _reset() that clears its two LRU caches.
* utils.cache.__init__._reset_cache_for_tests now orchestrates ALL
six caches (applications, users, domains, base, yaml, files) so
tests get clean state across cache layers — needed because
applications/users now share the yaml cache and would otherwise
serve stale parses across reset boundaries.
* utils.cache.files imports PROJECT_ROOT from base instead of
redefining it.
* tests/unit/utils/cache/test_data.py + test_base.py: obsolete
TestLoadYamlMapping / TestLoadYamlVariantList classes deleted
(covered by test_yaml.py + test_variants.py).
* README.md updated for the new module list (files, yaml) and the
six-module orchestrator.
Galaxy schema lint
* tests/lint/repository/test_role_meta_main_galaxy_schema.py: split
galaxy_info field requirements into hard-required {author,
description, license} and soft-recommended {min_ansible_version,
platforms}. Soft-recommended -> emit a GitHub Actions ::warning::
annotation via utils.annotations.message.warning (titled
"meta/main.yml galaxy_info" with file= pointing at the offender),
no test failure. Rationale: min_ansible_version + platforms are
only enforced by Galaxy when *publishing*, which Infinito.Nexus
does not do.
* Galaxy lint goes green (was 226 offenders before req-011 +
license_url/repository/documentation strip; was 62 before this
relaxation; now 0 hard offenders, 62 soft warnings).
* roles/svc-opt-keyboard-color/meta/main.yml: backfill `license`
(only role missing the hard-required field).
New lint
* tests/lint/repository/test_role_meta_info.py — enforces meta/info.yml
file-root convention, the four-field allowlist (logo, homepage,
video, display), and shape (logo.class string, homepage/video
non-empty strings, display bool). Also fails if any
meta/main.yml.galaxy_info still carries one of the four migrated
fields.
Docs
* docs/contributing/design/services/layout.md: new section
"Descriptive Role-Level Metadata — meta/info.yml" with allowed-
fields table, optionality rule, materialised path, and consumer
note. File-Layout table grows the meta/info.yml row;
meta/main.yml row gets the explicit "no logo/homepage/video/display"
note. req-011 added to header and Related Pages.
* utils/cache/README.md: files.py + yaml.py rows added; six-module
reset orchestrator documented.
Validation
* Unit: 1573 passed.
* Lint: 55 passed.
* Integration: 161 passed (1 skipped) via pre-commit hook.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
0ab817a839 |
refactor(roles): drop non-Galaxy license_url / repository / documentation from meta/main.yml
These three sub-keys lived under `galaxy_info:` purely as a workaround because Galaxy silently ignores unknown keys (same pattern as the `run_after` / `lifecycle` migration in req-010). The new lint `tests/lint/repository/test_role_meta_main_galaxy_schema.py` rejects them, and no consumer in `plugins/`, `utils/`, `cli/`, `filter_plugins/` or `roles/` reads any of them, so the strip is safe. Round-trip via ruamel.yaml so the remaining `galaxy_info:` block keeps its key order, quoting style and inline comments. 224 files touched. Remaining lint offenders after this commit are still present (`logo`, `homepage`, `video`, plus a handful of roles missing required Galaxy fields like `min_ansible_version` / `platforms` / `license`); those are a separate decision and not folded into this change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
ed04dad5d5 |
refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbach
|
5ceaa4bba7 | Downgraded minio to pre-alpha | ||
|
kevinveenbirkenbach
|
c104867735 | Changed License to Infinito.Nexus Community License (Non-Commercial) | ||
|
kevinveenbirkenbach
|
dd729fd35d | Introduced lifecycle into meta/main.yml roles | ||
|
kevinveenbirkenbach
|
5d1210d651 |
feat(ai): introduce dedicated AI roles and wiring; clean up legacy AI stack
• Add svc-ai category under roles and load it in constructor stage
• Create new 'svc-ai-ollama' role (vars, tasks, compose, meta, README) and dedicated network
• Refactor former AI stack into separate app roles: web-app-flowise and web-app-openwebui
• Add web-app-minio role; adjust config (no central DB), meta (fa-database, run_after), compose networks include, volume key
• Provide user-focused READMEs for Flowise, OpenWebUI, MinIO, Ollama
• Networks: add subnets for web-app-openwebui, web-app-flowise, web-app-minio; rename web-app-ai → svc-ai-ollama
• Ports: rename ai_* keys to web-app-openwebui / web-app-flowise; keep minio_api/minio_console
• Add group_vars/all/17_ai.yml (OLLAMA_BASE_LOCAL_URL, OLLAMA_LOCAL_ENABLED)
• Replace hardcoded include paths with path_join in multiple roles (svc-db-postgres, sys-service, sys-stk-front-proxy, sys-stk-full-stateful, sys-svc-webserver, web-svc-cdn, web-app-keycloak)
• Remove obsolete web-app-ai templates/vars/env; split Flowise into its own role
• Minor config cleanups (CSP flags to {}, central_database=false)
https://chatgpt.com/share/68d15cb8-cf18-800f-b853-78962f751f81
|