mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
fix/dev-python/python-upper-bound
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
e16eb5728d |
test(lint): drop the minimal variant's exemption from the onion
72fd9c870 required the onion in every variant but exempted variant 1, on the
argument that the minimal round is the one place the stack runs without it. That
argument does not hold: the onion is part of the deployment under test, not a
feature bolted onto the full shape. A round that skips it is not a smaller
deployment, it is a different one.
So the exemption is gone and 95 variant-1 entries pin the onion on. Two
exclusions remain, both read from data rather than a comment: a role with no tor
block has no onion to pin, and a role whose meta/services.yml pins tor.enabled to
a literal false has already declared it has none.
That put the rule in conflict with test_coverage, which requires both polarities
of every dynamic flag: with tor on everywhere, the false side has no variant left
to live in. 94 roles take the exemption that contract already provides for
exactly this - "flags whose group-derived value is a hard invariant that variants
must never override", the same one
|
||
|
|
ab72cff929 |
test(lint): require the onion in every variant, not just SSO ones
|
||
|
|
8a812c20ec |
fix(variants): pin the dev password where the role reads it
A variant entry is baked into host_vars raw, not deep-merged, so its top-level
key IS the inventory path. keycloak and lam still pinned their Playwright
administrator_password under credentials, while
|
||
|
|
c2fa2da88e |
style: apply make autoformat (whitespace only)
Indentation/whitespace normalization emitted by make autoformat across role meta/services.yml, meta/variants.yml and a task file. No semantic changes (verified: git diff --ignore-all-space is empty for every file). Kept separate from the Tor egress-router feature. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
1f60e2b3fa |
feat(variants): tor variant coverage for keycloak/lam/matrix/wordpress/rdr-www
Add the tor.enabled true/false rounds (shared:false + nocheck on the disabled round) so the dynamic tor flag is covered on these roles, and reorder each variant services-first so the tor nocheck registers. Each variant keeps its exact keys (credentials / addons / server.domains preserved, values unchanged) — mapping key order only; no cross-variant moves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
21d81ccdc2 | Merge branch 'main' into feature/req-21-sso-flavor-migration | ||
|
|
29377e1bd3 |
fix(cache/applications): stop variant-0 service leak via base-config defaults
`_build_application_defaults` returned variant 0 of every role with a
meta/variants.yml, which then deep-merged with the variant-N override
that build_dev_inventory bakes into the inventory. Any service flag
that variant N did not explicitly disable carried through from
variant 0 into the runtime applications map. The most visible casualty
was svc-bkp-container-2-local variant 1 (postgres + mariadb only),
where gitea + keycloak leaked in via variant 0, sys-service-loader
pulled in svc-db-openldap via keycloak.services.ldap, and the deploy
crashed in sys-svc-compose/04_files.yml on openldap's env.j2 with
"credentials is defined in schema at roles/svc-db-openldap/meta/schema.yml
but not set in application 'svc-db-openldap'".
Defaults now come from each role's variant-free base config
(meta/<topic>.yml plus schema-defaults, no variants overlay). Variants
stay accessible via get_variants for any caller that explicitly needs
a specific variant. Inventory-level applications.<app> overrides win
exactly as before.
Supporting changes that keep the variant matrix consistent under the
new defaults model:
- New lint guard tests/integration/roles/meta/variants/
test_non_baseline_explicit_disables.py: for every dynamic
service-key declared in a role's meta/services.yml that variant 0
pins to literal true, every non-baseline variant MUST explicitly
pin the same key to either literal true or to literal
enabled: false + shared: false. Statically-enabled keys are exempt
(same rule as test_auth_isolation.py).
- roles/web-app-{fusiondirectory,lam,nextcloud,phpldapadmin}/meta/
variants.yml: explicit `false` pins for dashboard / email / logout /
matomo / prometheus / talk in restrictive variants so deep-merge no
longer hands them the deleted variant-0 inheritance.
- roles/web-app-moodle/templates/playwright.env.j2: gate the LAM
block on `lookup('config', application_id, 'services.lam.enabled',
false)`. LAM is a transitive helper, not a moodle-owned service, so
the default-bearing lookup tolerates the missing path when the
variant does not deploy LAM.
- tests/unit/utils/cache/test_variants.py: rename the legacy
"always variant zero" coverage to TestMergedApplicationsUsesBaseConfig
and add test_get_application_defaults_omits_variant_zero_only_keys
as the regression guard for the leak this commit fixes.
- docs/contributing/artefact/files/role/variants.md +
docs/contributing/design/variants.md: variant 0 now manually
enumerates every dynamic services.yml key with
enabled: true + shared: true; `{}` is only permitted when the role
has no dynamic-enabled service-key.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
|
24d4ab8e4a |
refactor(sso)\!: collapse oauth2 + oidc into unified services.sso block (req 21)
Implements docs/requirements/021-sso-flavor-migration.md atomically:
* Schema: every role's meta/services.yml replaces the parallel
`oauth2:` / `oidc:` top-level blocks with a single `sso:` block
carrying `enabled`, `shared`, and a `flavor` discriminator
(`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
entirety; its 5 sidecar templates move under
roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
`is_oidc_native`) and the oauth2 sub-values
(`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
Three lookup plugins consume the resolver: `sso` (single-app
predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
and plugins/filter/compose_volumes.py all gate on
`lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
`enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
→ `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
`OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
`oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
`OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
`SSO_SERVICE_ENABLED`; Playwright gating helpers
(isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
`flavor: oauth2` with their existing `oauth2.acl.blacklist`
(/user/login) and `oauth2.acl.whitelist` (federation + discovery
endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
(tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
test_acl_mutual_exclusive.py) are deleted per Decision #10;
test_oauth2_contract.py is renamed and rewritten to
test_sso_contract.py, asserting that `flavor: oauth2` roles carry
`oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
(web-app integration) now requires the unified `sso:` block with an
explicit flavor or a `# nocheck: sso` opt-out; new
tests/lint/repository/no_legacy_sso_paths/ guards the source tree
against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
docs/contributing/actions/testing/suppression.md are updated to the
unified SSO naming.
Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).
BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
|
6344c3c970 |
style(meta/variants): normalize YAML header and comment indentation
Same scope as
|
||
|
|
96e6a3daa5 |
feat(web-svc-css): introduce canonical owner of the css service flag
Decouples the CSS service concept from its underlying transport
(currently `web-svc-cdn`) so the flag becomes opt-in via inventory
group membership instead of staying literal-`true` everywhere.
- `roles/web-svc-css/`: new role. Marker / wrapper role that owns
the `css` entry in the service registry, declares `web-svc-cdn` as
the upstream that serves the actual bytes, and 301-redirects its
own canonical domain to the cdn domain so health probes return a
valid response.
- `roles/web-svc-cdn/meta/services.yml`: drop the `css` block — the
canonical owner is now web-svc-css; cdn keeps `javascript`.
- 51 consumer roles (web-app-* and web-svc-{coturn,file,html,
libretranslate,logout,onlyoffice}) migrated their `css:` block from
literal `{ enabled: true, shared: true }` to dynamic
`{ enabled: "{{ 'web-svc-css' in group_names }}", shared: ... }`,
with matching matrix coverage added to each role's
`meta/variants.yml` (true-pinning + false-pinning entries) so the
dynamic-flag variant lint stays green.
- 41 `# nocheck: playwright-service-flag` rationale strings updated
from the now-incorrect "canonical:cdn alias of static-asset
provider" to "css consumer, no Playwright surface (provider:
web-svc-css)".
Bundled refactor: `roles/web-svc-asset/tasks/01_core.yml` now
includes `web-opt-rdr-domains` directly with a single-entry mapping
to redirect its canonical domain to web-svc-file, instead of having
that wiring live in the global `tasks/stages/01_constructor.yml`
constructor. The `add_redirect_if_group` filter
(`plugins/filter/redirect_filters.py`) and its unit test are dropped
as dead code — both call sites are now role-local.
Dashboard's consumption of the new pattern is intentionally not part
of this commit and follows separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
32b641dfec |
fix(web-app-{fusiondirectory,lam,phpldapadmin}): pin ldap.shared:true
Three roles whose persistence layer IS LDAP — they cannot run without
it — had `services.ldap.shared` rendered dynamically as
`'svc-db-openldap' in group_names`, and their `meta/variants.yml`
explicitly pinned `ldap.{enabled,shared}: false` in the "everything-
off" variant. The combination meant the matrix-deploy variant-1
deploy actually tore LDAP out from under FusionDirectory / LAM /
phpLDAPadmin, leaving the upstream containers unhealthy and the
matrix-job red.
Fix the dependency declaration to match physical reality: pin
`services.ldap.{enabled,shared}: true` in each role's `meta/
services.yml` (with `# nocheck: dynamic-flag` rationale comments), and
drop every `ldap:` override from `meta/variants.yml` so each variant
inherits the pinned values. fusiondirectory's variants.yml already
carried an explanatory comment ("The LDAP marker stays on across both
variants because LDAP IS the storage backend") that contradicted the
actual variant body; align the body to the comment.
The split between "LDAP IS the storage" (storage-backend) and "LDAP
is an optional federation source" (consumers like Keycloak,
Nextcloud, Gitea, …) is intentional — only the storage-backend
roles get the pin; federation consumers keep the dynamic
`'svc-db-openldap' in group_names` resolution.
Local FULL_CYCLE v0 for `web-app-fusiondirectory` runs green with
this change. v1 + v2 fail later on an unrelated oauth2-proxy gating
issue (502 from openresty when oauth2 is disabled) and are tracked
separately; the ldap fix is a prerequisite for that work, not a
solution to it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
c52e4269bf |
chore(tooling): integrate mbake (Makefile linter/formatter) + working-tree drift sync
Makefile lint integration: * scripts/install/lint.sh: replace the prior checkmake binary install with ensure_mbake (mbake is pip-installable, parses our Makefile correctly whereas checkmake bailed on .ONESHELL: / .SHELLFLAGS: / ifneq). * scripts/lint/makefile.sh: new runner — `mbake format --check Makefile` reports formatting drift and missing .PHONY declarations. * scripts/lint/autoformat.sh: add a two-pass `mbake format Makefile` (mbake's formatter is not strictly idempotent on the first pass). * Makefile: split the multi-line `.PHONY:` block into 18 single-line `.PHONY:` declarations so mbake's parser can validate it; add the previously-missing `install-python-dev` and `build-dependency` PHONY targets; add `lint-makefile` aggregate-and-direct target. * .pre-commit-config.yaml: new make-lint-makefile hook scoped to `^Makefile$`. * .github/workflows/lint-makefile.yml: new CI workflow (pip-install mbake + make lint-makefile). * .github/workflows/ci-orchestrator.yml: wire lint-makefile into lint-gate.needs. Misc: * .gitignore: add roles/importer_result.json (tooling artefact written by galaxy-importer when it runs in roles/). Plus working-tree drift across roles/, group_vars/, inventories/, and tasks/ that accumulated from the make-test setup phase regenerating generated artefacts during recent test runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
9e07353682 |
fix(roles): satisfy test_oidc_oauth2_ldap_variants_cover_auth_matrix in 25 roles
Two coupled deltas to ``meta/variants.yml`` across every role whose
``meta/services.yml`` declares ``oidc`` / ``oauth2`` / ``ldap`` via
the dynamic ``"… in group_names"`` shape:
1. **Pair-polarity backfill** in 14 roles whose existing all-true /
all-false reference variants only set ``shared`` (relying on
``services.yml``'s literal ``enabled: true # nocheck:
dynamic-flag``). The auth-coverage rule is stricter: it counts a
pair only when BOTH ``enabled`` AND ``shared`` are explicitly
pinned in the same variant entry. Added the missing ``enabled:
true`` (true-variant) / ``enabled: false`` (false-variant) for
``ldap`` (and ``oidc`` in friendica/gitea/bookwyrm). Touched
roles: web-app-{friendica, funkwhale, fusiondirectory, keycloak,
kix, lam, nextcloud, openproject, opentalk, phpldapadmin,
snipe-it, suitecrm, taiga} and web-svc-xmpp.
2. **LDAP-only third variant** in 25 roles where both an auth
service (``oidc`` / ``oauth2``) AND ``ldap`` are dynamic. The
variant pins ``ldap.enabled: true`` together with ``enabled:
false`` on every dynamic auth service in the same entry, so the
matrix exercises the LDAP-only branch in isolation from OIDC —
the explicit Playwright LDAP-coverage gap tracked by
docs/requirements/018-playwright-ldap-coverage.md. Touched roles:
web-app-{akaunting, baserow, bluesky, bookwyrm, discourse,
espocrm, flowise, friendica, funkwhale, fusiondirectory, gitea,
jenkins, kix, lam, minio, nextcloud, odoo, opencloud, openproject,
opentalk, openwebui, phpldapadmin, snipe-it, suitecrm, taiga}.
Mechanically applied via a one-off ruamel.yaml round-trip script
(comments and key order on existing variants survive); the new
ldap-only entries are a flat ``services:`` block at the end of each
file.
Verification: ``test_oidc_oauth2_ldap_variants_cover_auth_matrix``
passes; ``test_every_dynamic_flag_has_true_and_false_variant``
still passes; full integration suite is now ``172 tests, 0 failures,
1 skip`` — first time the suite is fully green on this branch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
4da44ac36a |
fix(variants): pin administrator_password on variant 1 of keycloak + lam
Variant 1 inherited the empty default credential, so the matrix-deploy round 2 couldn't authenticate as the administrator. Carry the same ``administrator_password`` literal as variant 0 so both rounds share the same credentials and the Playwright admin scenarios stay deterministic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
0cc48be9ae |
test(integration): variants coverage + no-self-reference guards for services.yml
Two new integration lints under tests/integration/roles/meta/:
* test_variants_coverage.py — every services.yml flag declared via
the dynamic ``"{{ '<role>' in group_names }}"`` form MUST be
exercised by meta/variants.yml: at least one variant entry pins
it to literal ``true`` AND at least one pins it to ``false``.
Pairs may share variant entries — the test only checks the union
of all variants covers both polarities for every dynamic flag.
* test_services_no_self_reference.py — a role's own primary entity
(matching ``get_entity_name(role)``) MUST NOT use the dynamic form
for ``enabled`` / ``shared``. Both MUST be literal ``true``: the
role's own services.yml is loaded only when the role itself is in
group_names, so a ``'<self>' in group_names`` Jinja is tautological.
The dynamic-flag test now skips the role's own entity to avoid a
contradiction with this new guard.
Self-reference fixes (Jinja → literal ``true: true``) on 19 roles:
svc-ai-ollama, svc-db-{mariadb, openldap, postgres}, web-app-{
dashboard, discourse, keycloak, mailu, matomo, prometheus},
web-svc-{asset, cdn, collabora, coturn, file, libretranslate, logout,
onlyoffice, simpleicons}.
Variants coverage rollout (62 new + 17 merged meta/variants.yml):
* Roles without a pre-existing variants.yml get a new file with two
entries — one pinning every dynamic flag true, one pinning every
dynamic flag false. No ``- {}`` baseline (matrix runs only the
two coverage variants).
* Roles with a pre-existing variants.yml (akaunting/baserow/bluesky/
bookwyrm/flowise/fusiondirectory/jenkins/jira/joomla/keycloak/lam/
minio/phpmyadmin/postmarks/wordpress/web-svc-xmpp) keep their
matrix variants (OIDC vs LDAP, Single-Site vs Multisite, dev creds
+ TOTP-disabled realm, autologin, etc.); the all-true coverage is
deep-merged into variant 0 and the all-false coverage into
variant 1, so each role still ships exactly two matrix rounds.
* svc-ai-ollama keeps its single CI image-shrink variant; after the
self-reference fix it has no remaining dynamic flags so no extra
coverage entry is required.
* svc-db-{mariadb, openldap, postgres} drop variants.yml entirely
for the same reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
d97ed7a9e8 |
feat(deploy): per-role matrix-deploy variants with folder-per-round model
Introduces per-role `meta/variants.yml` for declaring multi-variant
deploys (e.g. WordPress Single-Site vs. Multisite) and reshapes the
development deploy CLI to materialise one inventory folder per variant
and iterate them with co-located sync+async passes.
Why:
- The previous single-inventory + Pass-1-then-Pass-2 split deployed each
variant twice and ran the async pass against a host left in a
different variant's state, masking real bugs.
- Inventory and runtime concerns were tangled: variant selection lived
as a runtime extra-var and required cross-stage coordination.
How:
- `roles/<role>/meta/variants.yml` (file format docs at
docs/contributing/artefact/files/role/variants.md): YAML list of
deep-merge overrides on top of `config/main.yml`. Empty entry `{}` is
variant 0 (= legacy payload). 6 roles get their dev overrides moved
here from inventories/dev.yml; WordPress declares 2 variants.
- `utils/runtime_data.py` gains `_build_variants` / `get_variants` and
drops the retired `_active_variants` runtime selector. Loader always
returns variant 0 as the default; non-zero variants reach
`get_merged_applications` as inventory-level `applications.<app>`
overrides baked at init time.
- New `cli/deploy/development/inventory.py`: `DevInventorySpec`,
`plan_dev_inventory_matrix` (pure planner shared between init and
deploy), `build_dev_inventory_matrix` (materialises one folder per
round at `<dir>-<R>`, or `<dir>` when single-variant). Single-folder
paths preserve full backward compat.
- `cli/deploy/development/deploy.py`: matrix loop with per-round
cleanup (only apps whose variant changed AND that are about to
re-deploy), `--variant <idx>` (or `VARIANT=` env) to pin a single
round, and `--full-cycle` (or `FULL_CYCLE=true`) which interleaves
sync + async per round so the async re-deploy targets exactly the
host state the matching sync deploy just produced.
- `cli/deploy/development/exec.py` gains repeatable `--env KEY=VAL` so
bash callers can inject per-run context into in-container helpers
without inlining heredocs.
- `scripts/meta/env/inventory.sh` exports `INVENTORY_VARS_FILE`
(Python mirror in `cli.deploy.development.common.DEV_INVENTORY_VARS_FILE`;
test_inventory_vars_file_spot.py locks them against drift).
- `make deploy-fresh-purged-apps` and `scripts/tests/deploy/ci/dedicated.sh`
switch to a single init + single deploy `--full-cycle`. The
`make deploy-reuse-*` bash scripts honour `VARIANT=` by suffixing
`INVENTORY_DIR`/`INVENTORY_FILE`.
- The 5 `scripts/tests/deploy/local/deploy/*.sh` scripts now delegate
their in-container heredocs to dedicated helpers in
`scripts/tests/deploy/local/utils/`, removing the bash-in-bash
quoting hell.
Tests:
- `tests/unit/utils/test_variants.py` (renamed from
`test_application_variants.py`): loader edge cases, variant-zero
default, retired-`_active_variants` guard.
- `tests/unit/cli/deploy/development/test_inventory.py` (new): planner,
variant baking, builder, `filter_plan_to_variant`, `DevInventorySpec`.
- `tests/unit/cli/deploy/development/test_matrix_deploy.py` (new):
per-round deploy filter, cleanup decisions, `--variant` pinning,
`--full-cycle` per-round interleave.
- `tests/unit/cli/deploy/development/test_inventory_vars_file_spot.py`
(new): drift guard for the bash <-> Python literal pair.
- `tests/unit/cli/deploy/development/test_exec.py` extended for `--env`.
Docs:
- `docs/contributing/artefact/files/role/variants.md` (new): file format.
- `docs/contributing/design/matrix-deploy.md` (new): runtime mechanism
(folder layout, --variant, --full-cycle, cleanup logic).
- `docs/contributing/actions/deploy.md` (new): contributor dev-deploy
flow (make targets, --variant pinning).
- `docs/administration/deploy.md` slimmed to admin-only (was mixed
admin+dev), cross-links to the new contributor doc.
- `docs/agents/action/iteration/role.md` keeps only agent-iteration
recipes; mechanism explanation moved out and linked.
- `docs/contributing/documentation.md` adds the "no self-SPOT
declaration" rule; `applications.md` renamed its SoT header for
consistency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|