mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
fix/dev-python/python-upper-bound
26
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  kevinveenbirkenbachandClaude Opus 5
|
328113e7c3 |
refactor(ai): declare the litellm consumer contract once
Fourteen roles repeated the same two lines to say they use the gateway: a
`shared` flag gated on group_names, and an `enabled` flag conjoining that gate
with LITELLM_CHAT_MODEL_SERVED. Both are now single-sourced.
`shared` becomes literal true. The flag means the provider is a shared instance
rather than one deployed per consumer, and a central gateway is always that;
the dynamic form only said "shared if deployed", which `enabled` already
decides. The 50 shared pins under `litellm:` leave meta/variants.yml with it.
The tree had already noticed: 20 of those 50 lines carried
`# nocheck: variants-static-override`, two of them with copy-pasted text about
container_backup. This narrows
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
4164f37981 |
fix(mcp): give every MCP round an identity provider to sign in with
An MCP client spec signs in through the client's own OIDC button, which redirects to Keycloak. Every provider put its MCP pairing in the lean variant, and that variant pinned sso off - so the round that exists to prove the pairing was the one round without an identity provider. Run 33574056480 shows what that costs: openwebui's MCP specs reached _shared.js:131 and died on "openwebui OIDC sign-in button must be visible on the auth page". The rule now demands two things of the variant that carries the consumers: the provider pins sso on, and each admitted consumer, in the variant that round actually pulls in, does not pin it off. The second half matters because a round deploys every dependency at its own index and falls back to variant 0 only when the dependency has fewer, so the client shipping alongside is that one and not the client's variant 0. A role that declares no sso, or pins it to a literal false because upstream speaks no OIDC, is exempt from a flag it cannot have. Twenty roles moved to satisfy it. Ten already had a second sso: false pin and only needed the flip. Six could host the pairing in variant 0, which already has sso on, and dropped back to two variants. Six kept a third lean variant to carry the false polarity variants-coverage wants. moodle and mattermost took swaps instead: moodle's variant 1 is the mandated LDAP-only branch, and mattermost was already at the three-variant cap with sso on in its variant 2. nextcloud also turns context_agent on in that round, the ExApp that serves the endpoint - with it off the round proved nothing it was built to prove. The rule moves to tests/lint/ansible/services/ because it reads meta YAML and executes nothing, which is where integration.md says such a check belongs, and its suppression key joins the catalog it was never listed in. nextcloud_exapp_reregister becomes nextcloud_exapp_renew: mcp-secret-masking matches `register:` unanchored, so the substring in the old name read as a task that prints a secret. register.yml already masks the two tasks that carry one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
5bf4d4d8ac |
refactor(variants)!: let an MCP round decide for itself what it carries
The mcp-consumer-variant guard demanded three things of a provider's MCP
round: every admitted consumer enabled, each of them pinned `services: null`
so it dragged no dependencies along, and no other service left enabled. Two
of those were the rule legislating a round's contents rather than checking
that the pairing gets proven at all.
Keep the first and drop the rest. What else the round carries is the role's
own call, so the tor/litellm exemption, the datastore exemption and the
`allowed` set they fed have no reader left and go with them.
Strip the 63 `services: null` pins the rule forced across 20 variants, all
written bare as `services:`. The topic-emptying capability behind them
stays: apply_topic still normalises None to {} for the inventory payload,
and the unit tests still cover it. Only the uses and the mandate are gone.
That leaves overridden_providers scanning a repository with nothing left to
find, so its three assertions passed over an empty dict and would have
stayed green however the function broke. Rebuild them on a synthetic role
tree, which also covers the case the repository scan never held: a variant
naming a plain flag is not an override.
Removing the pins changes what deploys. Measured across all 20 variants: 18
identical, prometheus#1 grows 23 to 31 roles, moodle#2 grows 31 to 32 and
crosses the memory budget until the following commit shrinks it back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
f1635e6a59 |
feat(variants): let a variant dictate the config of the providers it pulls in
A round deploys a provider once, so the role that pulls it in is the only place that can say how lean it has to be. A variant entry may now carry any application config topic under a service it depends on: a mapping deep-merges, so naming one key changes that key alone, and null empties the topic. Empty was the case that needed a marker of its own, because merging an empty mapping is a no-op and would leave exactly the services the round was trying to shed. The topic vocabulary and the merge semantics live in utils.roles.applications.topics, which is also where resources.py now gets them rather than spelling the same null-or-merge out a second time. Collecting who demands what stays in the inventory tree, refuses two roles demanding different values for one provider in the same round, and skips a role's own entity entry so the existing services.<own>.users declarations are not read as overrides. The CI set-cover learns the same fact. Its greedy cover retires a role's variant-0 row when an earlier row of another role embeds it, on the stated premise that embedding pulls the provider's variant 0. A variant that dictates that provider's config breaks the premise from the other side: whatever ran, it was not the provider as the provider declares it. Such a pair now covers nothing, so the retired row keeps its job, its artefact and its verdict. Measured on the current tree: 20 override pairs, 41 covered rows before and after, so nothing is revoked today. The rule bites the moment an override lands in a row that covers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
30e43cce88 |
refactor(services): state a refused pairing as off instead of gating it
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
ffdef28da3 |
refactor(mcp)!: derive a surface's presence from its peers
enabled and shared were hand-maintained in 24 roles and wrong in both directions. Both now resolve from the mcp_consumer edges already declared in meta/services.yml: a server takes the consumers it admits, a client takes the servers that admit it, and the pair renders as "'peer' in group_names or ...", or False when no peer is admitted at all. setdefault keeps meta/mcp.yml and meta/variants.yml as the override channel, so pinning either key still wins. The resolved block replaces the mapping instead of being written into it. _build_role_base_config parks the YAML cache's own object under mcp, so an in-place setdefault injected the derived keys into every later raw read and made test_mcp_schema pass alone but fail in a directory run. The derivation keys on peer presence, never on a peer's own value: a re-entrant read under _RENDER_GUARD returns the unrendered dict. Four lints now ask the resolver instead of reading meta/mcp.yml, which they could not do while the key was hand-written. That is what exposed the eleven missing hostile-path specs the previous commit closed. Carries one change that belongs to the WordPress probe but shares a file with this one: the tool allowlist moves to the hyphenated names the adapter actually serves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
83c1c729de |
refactor(mcp)!: derive admission from the client
allowed_consumers was twenty hand-kept lists of which fourteen were identical, so every new client meant an edit in every provider. A client now declares services.<own-entity>.mcp_consumer: true once and a provider refuses one with mcp_consumer: false, which is also what tells a deliberate exclusion apart from a forgotten entry. The group_names-gated consumer entries give the inventory resolver the co-deploy edge meta/mcp.yml never had: one of sixty-nine matrix rounds carried an MCP client before, twenty-three role/round pairs do now. bond 0.5 keeps the clients out of round 0 so the host memory budget still holds. BREAKING CHANGE: meta/mcp.yml no longer accepts allowed_consumers; declare services.<entity>.mcp_consumer on the client instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
7162b7bf4c |
Merge fork/feature/svc-net-tor into feature/ai-agent-employees
834 commits from the Tor onion-service branch against 299 of ours, from merge base |
||
|
kevinveenbirkenbachandClaude Opus 5
|
0421ebeb16 |
feat(wordpress): bind the AI Engine connector to the platform gateway
WordPress has no AI surface in core, so the requirement applies to whichever connector the role pins rather than leaving the choice to a site administrator. The pinned connector is ai-engine, whose `custom` engine type takes an OpenAI-compatible base URL and an API key per environment. AI Engine keeps every setting in a single mwai_options array and its REST route replaces that array wholesale, so the deploy merges instead of assigning: ai_engine_configure.php filters ai_envs by id, re-appends the managed environment, and leaves every other environment and API key untouched. A straight write would delete configuration this deploy never owned. Both scripts are fed to `wp eval-file -` over stdin with their input passed as an environment variable. The first attempt used `wp eval` with the code inline, which cannot survive the quoting: a YAML folded scalar hands Jinja a literal backslash sequence and PHP fails to parse. That was reproduced in the container, along with the detail that `wp eval-file -` requires the opening <?php tag, before the approach was changed. Passing the payload as base64 in an env var means no shell quoting touches either the code or the virtual key. ai_engine_endpoint.php reads back the endpoint AI Engine actually stored for the managed environment, so the assertion compares the plugin's own state rather than the payload the deploy just sent. Not verified: the configure script has never run against a live AI Engine install, so the merge is reasoned from the option shape rather than observed, and addons/ai-engine.spec.js has executed zero times because the addon bridge gate resolved svc-ai-litellm against the Playwright role list and skipped it. ai-engine is third-party, so a change to its option schema would leave this writing a shape the plugin no longer reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
e16eb5728d |
test(lint): drop the minimal variant's exemption from the onion
72fd9c870 required the onion in every variant but exempted variant 1, on the
argument that the minimal round is the one place the stack runs without it. That
argument does not hold: the onion is part of the deployment under test, not a
feature bolted onto the full shape. A round that skips it is not a smaller
deployment, it is a different one.
So the exemption is gone and 95 variant-1 entries pin the onion on. Two
exclusions remain, both read from data rather than a comment: a role with no tor
block has no onion to pin, and a role whose meta/services.yml pins tor.enabled to
a literal false has already declared it has none.
That put the rule in conflict with test_coverage, which requires both polarities
of every dynamic flag: with tor on everywhere, the false side has no variant left
to live in. 94 roles take the exemption that contract already provides for
exactly this - "flags whose group-derived value is a hard invariant that variants
must never override", the same one
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
ab72cff929 |
test(lint): require the onion in every variant, not just SSO ones
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
46baeeda5b |
feat(lint): let variant 1 decide the addons that install everywhere
An addon's enabled is either dynamic -- a Jinja expression resolving per deploy
against a service flag, an API credential, the SSO plugin choice or group_names
-- or static, a literal boolean. A dynamic addon already follows its variant:
switch the service off and the expression yields false. A statically-on addon
does not. It installs in every round, so each non-baseline variant pays install
time for plugins it never exercises, and variant 0 stops being the only
maximum-footprint shape that meta/variants.yml exists to document.
web-app-nextcloud carried 22 of them across nine variants. They are now pinned
off once in variant 1 behind a YAML anchor, and variants 2-8 alias it, so the
set has one spot -- the same shape meta/volumes.yml and the
volumes-target-single-spot lint already prescribe.
test_addons_variant_pinning.py holds the rule for every role that ships
meta/addons/ and declares more than one variant:
1. an addon whose base enabled is literal true MUST be pinned enabled: false
in variant 1; a literal false is out of scope, the pin would override
nothing,
2. an addon whose base enabled is a Jinja expression MUST NOT be pinned in
any variant -- that replaces the expression with a constant and decouples
it from the service it follows,
3. every later variant MUST carry the same set, in practice by aliasing
variant 1. Variant-specific pins beyond that set stay allowed, so a
variant may still switch a statically-off addon on (web-app-matrix does
exactly that for its bridges in variant 2).
Measured on the real role through utils.cache.applications.get_variants: all
nine variants keep 51 addon keys, variant 0 keeps its 22 statics on, variants
1-8 hold none, and every dynamic entry keeps its Jinja string -- xwiki still
resolves true only where services.xwiki.enabled is. The 51 keys matter: the
override merges per addon instead of replacing the map, which is what an
earlier lookup-based attempt at this got wrong.
Each rule was broken on purpose and seen to fail: calendar pinned true, xwiki
pinned (reported for variants 1-8, which also proves the alias resolves), and
variant 5's alias removed (22 findings, exactly the required set).
web-app-discourse is exempted per addon with a reason. Its plugin set is
compiled into the image from services.discourse.plugins, and meta/addons/ is a
second declaration of the same thing; an addon-level disable would still build
the plugin and only flip the gate flag, saving no time while silencing the
spec. web-app-wordpress is not exempt: vars/main.yml:16 builds the plugin list
from the merged map and 07_manager_ops.yml:45 gates the install loop on
enabled, so its variant 1 genuinely skips a wp plugin install. Its README
claimed activitypub was "always enabled"; that is corrected rather than left to
contradict the file.
Not verified: no deploy has run. The saving is derived from the install loops
being gated, not observed, and the WordPress Multisite variant losing
activitypub rests on the same reading plus its skipUnlessAddonEnabled gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
5aca44ee8c |
fix(wordpress): pin the email service across the variants and register its flag
|
||
|
kevinveenbirkenbach
|
06bc856c32 |
refactor(mcp): make MCP a role topic instead of a service entry
The block sat under services but never behaved like a service. A service has exactly one providing role, declared with provides, so the registry can map the key to that role and consumers bind their flag to its presence. No role declares provides: mcp. There are seventeen providers and the pairing is done by allowed_consumers inside the block itself. Every rule that enforces the service model therefore needed an exemption for it: a shared-provider suppression in fourteen roles, a playwright-service-flag suppression, and two freshly added exemptions in the dynamic-flag and variants-static-override checks. Four independent rules, four escape hatches, one cause. They are all gone now, together with a bond key that only the resource budgeter reads and only for entries under services, and a wordpress addon bridge that pointed at a service block that no longer exists. The block moves to meta/mcp.yml. The applications loader maps meta/<name>.yml to applications.<app>.<name> generically, so the move needs no wiring; the file takes the same shape as meta/rbac.yml. Every config path follows, the variant overrides are lifted out of their services map, and the vocabulary module leaves the services package because it no longer describes one. roles_with_service gains an explicit topic parameter rather than an implicit fallback, so nothing silently resolves against the wrong container. Files that are otherwise unrelated to this move are included where the committed tree would otherwise keep pointing at a path that no longer resolves: thirteen vars files, eleven playwright env templates and one task file. |
||
|
kevinveenbirkenbachandClaude Fable 5
|
d88f272f9f |
Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
3875eb2cf9 |
feat(variants): pin container_backup per matrix variant
Variant 0 pins the backup consumer flag enabled so the matrix baseline exercises the in-group branch; every other variant pins it disabled (shared: false carries the static-override suppression since the base value stays literally true). The matrix and wordpress variant entries move their services mapping to the list-item head so the static-override line parser resolves the override paths; yaml semantics are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
8b1110d571 |
style: apply autoformat reformatting (no functional change)
make autoformat normalised inline-comment spacing (` #` -> ` #`) in role meta/services.yml + meta/variants.yml and re-wrapped a few Python files (ruff line-length). Pure formatting — verified via git diff -w (whitespace-only) and diff inspection (ruff line-wraps only, no logic change). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
1f60e2b3fa |
feat(variants): tor variant coverage for keycloak/lam/matrix/wordpress/rdr-www
Add the tor.enabled true/false rounds (shared:false + nocheck on the disabled round) so the dynamic tor flag is covered on these roles, and reorder each variant services-first so the tor nocheck registers. Each variant keeps its exact keys (credentials / addons / server.domains preserved, values unchanged) — mapping key order only; no cross-variant moves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
24d4ab8e4a |
refactor(sso)\!: collapse oauth2 + oidc into unified services.sso block (req 21)
Implements docs/requirements/021-sso-flavor-migration.md atomically:
* Schema: every role's meta/services.yml replaces the parallel
`oauth2:` / `oidc:` top-level blocks with a single `sso:` block
carrying `enabled`, `shared`, and a `flavor` discriminator
(`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
entirety; its 5 sidecar templates move under
roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
`is_oidc_native`) and the oauth2 sub-values
(`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
Three lookup plugins consume the resolver: `sso` (single-app
predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
and plugins/filter/compose_volumes.py all gate on
`lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
`enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
→ `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
`OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
`oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
`OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
`SSO_SERVICE_ENABLED`; Playwright gating helpers
(isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
`flavor: oauth2` with their existing `oauth2.acl.blacklist`
(/user/login) and `oauth2.acl.whitelist` (federation + discovery
endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
(tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
test_acl_mutual_exclusive.py) are deleted per Decision #10;
test_oauth2_contract.py is renamed and rewritten to
test_sso_contract.py, asserting that `flavor: oauth2` roles carry
`oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
(web-app integration) now requires the unified `sso:` block with an
explicit flavor or a `# nocheck: sso` opt-out; new
tests/lint/repository/no_legacy_sso_paths/ guards the source tree
against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
docs/contributing/actions/testing/suppression.md are updated to the
unified SSO naming.
Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).
BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
d03db7f73e |
test(web-svc-css): own consumer CSS-indexing assertion + close variants/playwright gates
Adds the web-svc-css provider Playwright spec that mirrors the
dashboard/prometheus SPOT-provider pattern: emits CSS_TARGET_ROLES_JSON
via roles_with_service('css') and asserts every consumer's canonical
surface indexes a <link rel="stylesheet"> pointing at the shared CSS
asset host (CSS_BASE_URL / CDN_BASE_URL). The baseline test exercises
CDN reachability first so per-consumer failures unambiguously point
at the consumer's sys-front-inj-css wiring, not at the provider.
Suppresses the per-role playwright env-services-match guard above each
consumer's `css:` key with the canonical `# nocheck:
playwright-service-flag — verified by web-svc-css provider spec`
marker. web-svc-css/meta/services.yml gets the self-provider nocheck
on `css:` and `provider role; no consumer surface to gate` nochecks on
`cdn:` / `matomo:`.
Adds css.{enabled,shared} true/false pinning to every variants.yml
that declared the dynamic flag without coverage so
test_every_dynamic_flag_has_true_and_false_variant passes. Unifies
the css service declaration across espocrm / mediawiki / openproject
(was static enabled: false, shared: true) so the matrix matches the
rest of the consumer fleet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
c55ad69c41 |
refactor: drop requirement-file references from code, add doc rule + lint
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`, `requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from non-Markdown source files. Comments anchored to requirement numbers rot when requirements are renumbered or rewritten and conflate present-state behaviour (which the comment should describe) with the audit trail of how the code got there (which belongs in commit messages, PR descriptions, and the requirement file itself). * Document the rule in `docs/contributing/documentation.md` under "Comments". Code MUST describe the *why* in present-state language; contract enforcement lives in tests, not folklore. * Clean 254 source files across `roles/`, `tests/`, `plugins/`, `cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root `eslint.config.js`. Pure comment / docstring / log-message edits; no behavioural change. * Add `tests/lint/repository/test_no_req_references_in_code.py` to enforce the rule going forward across `.js`, `.py`, `.yml`, `.yaml`, `.j2`, and `.sh` files. The legacy `test_no_stale_requirement_refs.py` is exempt as the only file whose purpose is to detect these very tokens. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
2a023ca6a1 |
chore(lint): autoformat strict-failure semantics + claude.sh rename
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
0cc48be9ae |
test(integration): variants coverage + no-self-reference guards for services.yml
Two new integration lints under tests/integration/roles/meta/:
* test_variants_coverage.py — every services.yml flag declared via
the dynamic ``"{{ '<role>' in group_names }}"`` form MUST be
exercised by meta/variants.yml: at least one variant entry pins
it to literal ``true`` AND at least one pins it to ``false``.
Pairs may share variant entries — the test only checks the union
of all variants covers both polarities for every dynamic flag.
* test_services_no_self_reference.py — a role's own primary entity
(matching ``get_entity_name(role)``) MUST NOT use the dynamic form
for ``enabled`` / ``shared``. Both MUST be literal ``true``: the
role's own services.yml is loaded only when the role itself is in
group_names, so a ``'<self>' in group_names`` Jinja is tautological.
The dynamic-flag test now skips the role's own entity to avoid a
contradiction with this new guard.
Self-reference fixes (Jinja → literal ``true: true``) on 19 roles:
svc-ai-ollama, svc-db-{mariadb, openldap, postgres}, web-app-{
dashboard, discourse, keycloak, mailu, matomo, prometheus},
web-svc-{asset, cdn, collabora, coturn, file, libretranslate, logout,
onlyoffice, simpleicons}.
Variants coverage rollout (62 new + 17 merged meta/variants.yml):
* Roles without a pre-existing variants.yml get a new file with two
entries — one pinning every dynamic flag true, one pinning every
dynamic flag false. No ``- {}`` baseline (matrix runs only the
two coverage variants).
* Roles with a pre-existing variants.yml (akaunting/baserow/bluesky/
bookwyrm/flowise/fusiondirectory/jenkins/jira/joomla/keycloak/lam/
minio/phpmyadmin/postmarks/wordpress/web-svc-xmpp) keep their
matrix variants (OIDC vs LDAP, Single-Site vs Multisite, dev creds
+ TOTP-disabled realm, autologin, etc.); the all-true coverage is
deep-merged into variant 0 and the all-false coverage into
variant 1, so each role still ships exactly two matrix rounds.
* svc-ai-ollama keeps its single CI image-shrink variant; after the
self-reference fix it has no remaining dynamic flags so no extra
coverage entry is required.
* svc-db-{mariadb, openldap, postgres} drop variants.yml entirely
for the same reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
f3ca1d41c9 |
fix(web-app-wordpress): correct discourse-integration order, DRY plugin enables, explicit per-variant discourse toggle
services.yml:
- Add web-app-discourse to wordpress.run_after, so the wp-discourse
plugin install in tasks/plugins/wp-discourse.yml finds the discourse
container running when its `container exec discourse rake
api_key:create_master[...]` and friends fire.
- Make wp-discourse.enabled and daggerhart-openid-connect-generic.enabled
read their flag from services.{discourse,oidc}.enabled via
`lookup('config', application_id, ...)`, replacing the inline
conditional / hardcoded literal. Single source of truth per
shared-service flag, matching the pattern in
roles/web-app-gitea/meta/services.yml.
- Replace the literal `discourse.enabled: false` baseline with
`"{{ 'web-app-discourse' in group_names }}"`. The inventory-builder
ignores Jinja-string `enabled` values (strict `is True` check in
utils/service_registry.py:203), so this is a play-time fallback only:
if a variant does not override the flag and the operator still adds
web-app-discourse to the inventory groups, the conditional resolves
to true at runtime so the wp-discourse plugin enables itself.
variants.yml:
- Variant 0 (canonical Single-Site): set
`services.discourse.enabled: true` so the Service-Registry pulls
web-app-discourse in via the shared-service auto-include path and
the matrix-deploy round exercises the wp-discourse round-trip.
- Variant 1 (Multisite): set `services.discourse.enabled: false` so
the Multisite Playwright scenarios run isolated from the Discourse
edge cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
ed04dad5d5 |
refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
d97ed7a9e8 |
feat(deploy): per-role matrix-deploy variants with folder-per-round model
Introduces per-role `meta/variants.yml` for declaring multi-variant
deploys (e.g. WordPress Single-Site vs. Multisite) and reshapes the
development deploy CLI to materialise one inventory folder per variant
and iterate them with co-located sync+async passes.
Why:
- The previous single-inventory + Pass-1-then-Pass-2 split deployed each
variant twice and ran the async pass against a host left in a
different variant's state, masking real bugs.
- Inventory and runtime concerns were tangled: variant selection lived
as a runtime extra-var and required cross-stage coordination.
How:
- `roles/<role>/meta/variants.yml` (file format docs at
docs/contributing/artefact/files/role/variants.md): YAML list of
deep-merge overrides on top of `config/main.yml`. Empty entry `{}` is
variant 0 (= legacy payload). 6 roles get their dev overrides moved
here from inventories/dev.yml; WordPress declares 2 variants.
- `utils/runtime_data.py` gains `_build_variants` / `get_variants` and
drops the retired `_active_variants` runtime selector. Loader always
returns variant 0 as the default; non-zero variants reach
`get_merged_applications` as inventory-level `applications.<app>`
overrides baked at init time.
- New `cli/deploy/development/inventory.py`: `DevInventorySpec`,
`plan_dev_inventory_matrix` (pure planner shared between init and
deploy), `build_dev_inventory_matrix` (materialises one folder per
round at `<dir>-<R>`, or `<dir>` when single-variant). Single-folder
paths preserve full backward compat.
- `cli/deploy/development/deploy.py`: matrix loop with per-round
cleanup (only apps whose variant changed AND that are about to
re-deploy), `--variant <idx>` (or `VARIANT=` env) to pin a single
round, and `--full-cycle` (or `FULL_CYCLE=true`) which interleaves
sync + async per round so the async re-deploy targets exactly the
host state the matching sync deploy just produced.
- `cli/deploy/development/exec.py` gains repeatable `--env KEY=VAL` so
bash callers can inject per-run context into in-container helpers
without inlining heredocs.
- `scripts/meta/env/inventory.sh` exports `INVENTORY_VARS_FILE`
(Python mirror in `cli.deploy.development.common.DEV_INVENTORY_VARS_FILE`;
test_inventory_vars_file_spot.py locks them against drift).
- `make deploy-fresh-purged-apps` and `scripts/tests/deploy/ci/dedicated.sh`
switch to a single init + single deploy `--full-cycle`. The
`make deploy-reuse-*` bash scripts honour `VARIANT=` by suffixing
`INVENTORY_DIR`/`INVENTORY_FILE`.
- The 5 `scripts/tests/deploy/local/deploy/*.sh` scripts now delegate
their in-container heredocs to dedicated helpers in
`scripts/tests/deploy/local/utils/`, removing the bash-in-bash
quoting hell.
Tests:
- `tests/unit/utils/test_variants.py` (renamed from
`test_application_variants.py`): loader edge cases, variant-zero
default, retired-`_active_variants` guard.
- `tests/unit/cli/deploy/development/test_inventory.py` (new): planner,
variant baking, builder, `filter_plan_to_variant`, `DevInventorySpec`.
- `tests/unit/cli/deploy/development/test_matrix_deploy.py` (new):
per-round deploy filter, cleanup decisions, `--variant` pinning,
`--full-cycle` per-round interleave.
- `tests/unit/cli/deploy/development/test_inventory_vars_file_spot.py`
(new): drift guard for the bash <-> Python literal pair.
- `tests/unit/cli/deploy/development/test_exec.py` extended for `--env`.
Docs:
- `docs/contributing/artefact/files/role/variants.md` (new): file format.
- `docs/contributing/design/matrix-deploy.md` (new): runtime mechanism
(folder layout, --variant, --full-cycle, cleanup logic).
- `docs/contributing/actions/deploy.md` (new): contributor dev-deploy
flow (make targets, --variant pinning).
- `docs/administration/deploy.md` slimmed to admin-only (was mixed
admin+dev), cross-links to the new contributor doc.
- `docs/agents/action/iteration/role.md` keeps only agent-iteration
recipes; mechanism explanation moved out and linked.
- `docs/contributing/documentation.md` adds the "no self-SPOT
declaration" rule; `applications.md` renamed its SoT header for
consistency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|