mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 19:01:54 +00:00
fix/dev-python/python-upper-bound
29
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 infinito-nexus-updater[bot]
|
9dd99062fc | update(images): update Docker image versions (web-app-littlejs) | ||
|
infinito-nexus-updater[bot]
|
0a6c2458ee | update(images): update Docker image versions (web-app-littlejs) | ||
  kevinveenbirkenbachandClaude Opus 5
|
da665ab699 |
feat(services): give fifteen moving pins a name the registry can order
A pin like 'alpine', 'stable', 'lts' or 'buildx-stable-1' names no version, so nothing can compare it to anything: the bump job skips it and the pin drifts with whatever upstream last pushed under that name. Each of these fifteen now names its release, which is the precondition for the freshness pipeline to see it at all. Every conversion was placed against the tag it replaces, and none of them is a downgrade. Six resolve to a byte-identical digest and are pure relabelling (memcached, redis, the two debian bases, jenkins, xwiki); four are genuinely newer than what the moving tag carried, so those roles will deploy a later version than before (nginx moves past the 1.31.5 behind 'alpine', buildkit past the v0.32.2 behind 'buildx-stable-1', dockerize past a 'latest' older than v0.10.0); openresty and pretix keep their version and differ only by build. fider is the one that loses something: 'stable' carries a rebuild pushed twelve days after v0.36.0, which is still the newest versioned tag upstream publishes. An unnamed rebuild cannot be tracked, so the pin takes the named release and the updater can raise it when v0.37.0 lands. The flavor invariant keeps the families apart: jenkins stays on '-lts' and xwiki inside '-postgres-tomcat', so no bump can cross into a different runtime. svc-runner loses a 'nocheck: docker-version' that has become dead. The marker suppresses two checks at once, so while it stood buildkit was held out of the daily freshness warning as well; removing it puts the role back in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
b9669fcaa5 |
feat(healthcheck): give every container a probe, or state why it cannot have one
A container without a healthcheck is accepted the moment its process starts. The restore drill asks the host's health service for its verdict, so a service that came back empty passes as long as it is running - which is how an empty volume survives a drill unnoticed. Twenty-one services get a probe and thirteen get a marker naming why none is possible: a one-shot client that exits, a build image with no service port, a media bridge on UDP. The flavor comes from each image's own layer history, read from the registry through the mirror, so no image had to be pulled to find out whether it ships curl, wget or a shell. Two entries carry more weight than the rest. The central Postgres, which every replay of the recover chain lands in, had no probe at all; it gets the pg_isready line that sys-svc-rdbms already runs for the per-app engine. And the matrix runner starts systemd within a second while its inner daemon is still pulling every image of the deployment, so it counted as healthy long before anything served - it now waits for Synapse to answer. No probe replaces one: nineteen of the twenty-one images ship no HEALTHCHECK, and the two that do already carried a declaration before this change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
e16eb5728d |
test(lint): drop the minimal variant's exemption from the onion
72fd9c870 required the onion in every variant but exempted variant 1, on the
argument that the minimal round is the one place the stack runs without it. That
argument does not hold: the onion is part of the deployment under test, not a
feature bolted onto the full shape. A round that skips it is not a smaller
deployment, it is a different one.
So the exemption is gone and 95 variant-1 entries pin the onion on. Two
exclusions remain, both read from data rather than a comment: a role with no tor
block has no onion to pin, and a role whose meta/services.yml pins tor.enabled to
a literal false has already declared it has none.
That put the rule in conflict with test_coverage, which requires both polarities
of every dynamic flag: with tor on everywhere, the false side has no variant left
to live in. 94 roles take the exemption that contract already provides for
exactly this - "flags whose group-derived value is a hard invariant that variants
must never override", the same one
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
b2a0a317fa |
refactor(tor): loosen the onion bond to 0.5
Every variants.yml header states the rule the bond feeds: variant 0 carries each
tightly-coupled (bond>=1, same-host) service, while loosely-coupled (bond<1,
separate-host) partners are excluded from the closure and distributed across
later variants to keep every host under budget. tor was declared bond: 1 in all
101 consumers, so it had to sit in every variant-0 closure.
It does not belong there. The onion is a separate-host provider reached over the
node's hidden service, the same shape as the bridged partners that already carry
0.5, and
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
e96c752549 |
Merge fix/playwright/env-template-gate into feature/svc-net-tor
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content —
|
||
|
kevinveenbirkenbach
|
e04c448cdd |
Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
Three conflicts, none resolved by picking a side wholesale: - Makefile: took the incoming ACT_INPUTS, which routes through SWARM_DISTROS instead of inlining the same expression a second time. - test-e2e-playwright/02_run_one.yml: took the incoming rc handling. An absent rc now counts as failure; the previous default(0) let an aborted Playwright run pass silently, contradicting the debug message right above it. - web-app-peertube spec: combined both. Kept gotoOnion and resolveTimeout, which this branch's lint enforces, and adopted the incoming retry-inside-poll for the OIDC button plus its networkidle wait. |
||
|
kevinveenbirkenbachandClaude Opus 5
|
028f4b85f4 |
chore(storage): set every image-backed min_storage to its image size plus 200MB
The declared values were guesses that ran up to 60x the size of the image they belong to: seaweedfs asked for 30GB against a 104MB image, nextcloud for 20GB against 353MB. Because min_storage feeds the storage guard, the runner packer and the sufficient-storage filter, those numbers made hosts look too small and pruned caches that deploys did not need to lose. Every service that declares min_storage together with image and version now carries the measured compressed transfer size of that image plus 200MB of headroom. Sizes were read from the registries through utils.docker.registry.manifest_transfer_size, the same path the headroom test uses, so the declarations and the gate agree by construction. Values are written in MB throughout, matching the existing 500MB / 100MB / 50MB style. 119 declarations across 89 roles. Range: 206MB (memcached) to 3473MB (ollama); four exceed 1GB. The 84 min_storage entries that name no image are untouched, since there is nothing to measure them against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Fable 5
|
d88f272f9f |
Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
7a6268fa96 |
refactor(deploy): replace skip with services modes
Per-role deploy-mode opt-out moves from meta/tests.yml 'skip:' to meta/services.yml <entity>.modes.{compose,swarm}.enabled on the primary entity. get_role_skip now derives from it; get_role_mode_enabled added. A new lint requires every role shipping templates/*compose*.yml.j2 to declare both mode flags. Migrates 107 services.yml and drops 11 skip-only tests.yml. Also removes the variant_status deployability gate so swarm runs one job per variant unconditionally (variant_status.py and deployable_variant_indices deleted, complexity model simplified). Adds the i8matix alias.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Fable 5
|
4a23e0cb09 |
feat(services): declare locally-built images via services.<key>.custom
The custom flag moves from 33 hardcoded template kwargs into the merged service config: 'custom: true' names the shared <entity>_custom build, a string names a second distinct '<string>_custom' image inside one app (nextcloud whiteboard). The image lookup reads the declaration as a fallback, so config consumers (e.g. backup image matching) resolve the exact deployed reference from one declaration point. nextcloud cron now runs the nextcloud_custom build like the app service instead of the upstream image. A new services lint couples the declaration to reality in both directions: custom requires a role Dockerfile, and a role Dockerfile requires a custom declaration. The eight legacy roles deploying local builds under upstream tags or ad-hoc image= overrides are ratcheted via '# nocheck: dockerfile-custom'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
fac2de5fd3 |
style: apply make autoformat (whitespace only)
Indentation/whitespace normalization from make autoformat across role meta files. No semantic changes (git diff --ignore-all-space is empty for every file). Kept separate from the mailu notls-over-onion and Tor-router idempotency fixes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
0a7164af48 |
feat(tor): per-role onion service block in meta/services.yml
Domain-serving roles reference the svc-net-tor provider via a tor: block. Consumers declare enabled (dynamic group_names gate) and shared: true (static — onion is always the shared node model, inline nocheck: dynamic-flag). The provider svc-net-tor additionally declares exclusive/primary defaults that consumers inherit via resolve_service_config unless overridden. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
36ebf34b64 |
Merge branch 'main' into feature/swarm-nfs
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
3fa781004d |
feat(roles): add bond integration-importance factor to shared services
`bond` is a 0..1 factor for how strongly a service is tied to the overall integration: bond: 1 = always required, bond: 0 = omittable. Groundwork for tooling that later scores service importance and decides what can be dropped. Named `bond` (not `bound`) to avoid clashing with the resource bounds/limits. Add test_service_bond.py (lint): every enabled + shared service must declare a valid `bond` (number in 0..1). Seed `bond: 1` as a placeholder on all 638 enabled+shared service entries across 88 roles, to be refined later. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
25f286d2bf |
feat(swarm): per-service internal ports and entity-named services
Give each reverse-proxied app a swarm-resolvable upstream: add services.<entity>.ports.internal.http to the role metadata and rename the main compose service from the generic 'application' to the entity name, so its short DNS alias on the shared overlay is unique. Thread the service name through compose, vars (*_SERVICE_NAME), volume mounts and the oauth2 origin host. Consolidate the split image/ports blocks in bridgy-fed and postmarks into the entity block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
24d4ab8e4a |
refactor(sso)\!: collapse oauth2 + oidc into unified services.sso block (req 21)
Implements docs/requirements/021-sso-flavor-migration.md atomically:
* Schema: every role's meta/services.yml replaces the parallel
`oauth2:` / `oidc:` top-level blocks with a single `sso:` block
carrying `enabled`, `shared`, and a `flavor` discriminator
(`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
entirety; its 5 sidecar templates move under
roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
`is_oidc_native`) and the oauth2 sub-values
(`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
Three lookup plugins consume the resolver: `sso` (single-app
predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
and plugins/filter/compose_volumes.py all gate on
`lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
`enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
→ `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
`OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
`oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
`OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
`SSO_SERVICE_ENABLED`; Playwright gating helpers
(isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
`flavor: oauth2` with their existing `oauth2.acl.blacklist`
(/user/login) and `oauth2.acl.whitelist` (federation + discovery
endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
(tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
test_acl_mutual_exclusive.py) are deleted per Decision #10;
test_oauth2_contract.py is renamed and rewritten to
test_sso_contract.py, asserting that `flavor: oauth2` roles carry
`oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
(web-app integration) now requires the unified `sso:` block with an
explicit flavor or a `# nocheck: sso` opt-out; new
tests/lint/repository/no_legacy_sso_paths/ guards the source tree
against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
docs/contributing/actions/testing/suppression.md are updated to the
unified SSO naming.
Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).
BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
d7591cb44a |
lint(compose-repository): enforce Jinja + paired URL/ref; fix 3 violators
New lint
--------
``tests/lint/ansible/modules/test_compose_repository_url_ref.py`` walks
every dict in every project ``.yml``/``.yaml`` and asserts:
1. ``compose_repository_url`` MUST be a Jinja expression — keeps repo
coordinates in ``meta/services.yml`` (looked up via vars) instead
of pasted into task files.
2. ``compose_repository_ref`` MUST also be a Jinja expression for the
same SPOT reason.
3. Every ``compose_repository_url`` MUST be paired with a
``compose_repository_ref`` in the same dict — pinning the clone
source without pinning the ref leaves checkouts at upstream HEAD
(non-reproducible deploys).
Violations fixed
----------------
The lint surfaced three offenders, all migrated to the established
``repository:``/``ref:`` SPOT pattern under the role's primary entity
in ``meta/services.yml`` (akaunting/bigbluebutton/roulette-wheel
already use this shape):
- **web-app-taiga**: added ``repository:`` + ``ref:`` to the ``taiga``
service block; tasks now use ``TAIGA_REPOSITORY`` / ``TAIGA_REF``
vars; restores the previously missing ref pinning.
- **web-app-openproject**: added ``repository:`` + ``ref:`` to the
``openproject`` service block (the ref keeps the
``stable/{{ OPENPROJECT_VERSION }}`` Jinja form, just SPOT-relocated).
- **web-app-littlejs**: added ``repository:`` + ``ref:`` to the
``littlejs`` service block; the existing
``LITTLEJS_REPOSITORY_BASE`` (used by ``nav_bottom.html.j2`` for the
human-readable upstream link) is now derived from the new
``LITTLEJS_REPOSITORY`` via ``regex_replace('\\.git$', '')`` so the
literal hostname is gone too.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
96e6a3daa5 |
feat(web-svc-css): introduce canonical owner of the css service flag
Decouples the CSS service concept from its underlying transport
(currently `web-svc-cdn`) so the flag becomes opt-in via inventory
group membership instead of staying literal-`true` everywhere.
- `roles/web-svc-css/`: new role. Marker / wrapper role that owns
the `css` entry in the service registry, declares `web-svc-cdn` as
the upstream that serves the actual bytes, and 301-redirects its
own canonical domain to the cdn domain so health probes return a
valid response.
- `roles/web-svc-cdn/meta/services.yml`: drop the `css` block — the
canonical owner is now web-svc-css; cdn keeps `javascript`.
- 51 consumer roles (web-app-* and web-svc-{coturn,file,html,
libretranslate,logout,onlyoffice}) migrated their `css:` block from
literal `{ enabled: true, shared: true }` to dynamic
`{ enabled: "{{ 'web-svc-css' in group_names }}", shared: ... }`,
with matching matrix coverage added to each role's
`meta/variants.yml` (true-pinning + false-pinning entries) so the
dynamic-flag variant lint stays green.
- 41 `# nocheck: playwright-service-flag` rationale strings updated
from the now-incorrect "canonical:cdn alias of static-asset
provider" to "css consumer, no Playwright surface (provider:
web-svc-css)".
Bundled refactor: `roles/web-svc-asset/tasks/01_core.yml` now
includes `web-opt-rdr-domains` directly with a single-entry mapping
to redirect its canonical domain to web-svc-file, instead of having
that wiring live in the global `tasks/stages/01_constructor.yml`
constructor. The `add_redirect_if_group` filter
(`plugins/filter/redirect_filters.py`) and its unit test are dropped
as dead code — both call sites are now role-local.
Dashboard's consumption of the new pattern is intentionally not part
of this commit and follows separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
951f6d36d2 |
chore(lint): unify autoformat/galaxy parallelism into one PARALLEL env var
* scripts/lint/autoformat.sh: rename AUTOFORMAT_PARALLEL to PARALLEL, declare it at file start with default `true`; still env-overridable. Reword the worker comment so the literal `# shellcheck` token no longer appears at start-of-line and trips SC1072/SC1073. * scripts/lint/galaxy.sh: drop the GALAXY_FATAL / GALAXY_NAMESPACE / GALAXY_PARALLEL env vars (over-engineered for a single repo). Failure is always fatal. NAMESPACE is an internal constant. Add the same PARALLEL env var (default `true`, env-overridable); falsy demotes the xargs fan-out to one worker. * docs/contributing/environment/variables.md: collapse the four AUTOFORMAT_*/GALAXY_* rows into one `PARALLEL` row covering both consumers. * docs/contributing/tools/make.md: refer to `PARALLEL=0` everywhere; add the lint-galaxy parallelism note. Plus working-tree drift across roles/, group_vars/, inventories/, and tasks/ that accumulated from the make-test setup phase regenerating generated artefacts during recent test runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbach
|
76b139c894 |
fix(lint): migrate raw IO + project walks to utils.cache + variant coverage
Cleans up the lint debt the main->feature/web-app-kix merge surfaced.
Each change addresses a real failure surfaced by `make test`; no
``# nocheck`` blanket suppressions added.
Migrations to the shared cache helpers:
- ~25 test files swap raw ``yaml.safe_load`` / ``yaml.safe_dump`` /
``yaml.dump`` for ``utils.cache.yaml.{load_yaml_any,load_yaml_all,
load_yaml_str,dump_yaml,load_yaml_all_str}``.
- ~30 test files swap raw ``glob.glob`` / ``os.walk`` / ``Path.rglob``
for ``utils.cache.files.iter_project_files`` (or
``iter_non_ignored_files`` for the dependabot fallback walker, or
per-role ``Path.iterdir`` / ``Path.glob`` where the scope is a
single subtree).
- 3 ``Path.read_text`` lint helpers now route through
``utils.cache.files.read_text``.
Variant + service-flag fixes:
- web-app-kix: shift its subnet from 192.168.105.80/28 to
192.168.105.128/28 to clear the collision with web-app-hugo and
svc-prx-openresty.
- web-app-fediwall: add the missing ``mastodon`` / ``friendica``
service flags referenced by ``meta/variants.yml``; pin the css /
logout / dashboard / matomo / prometheus literals with
``# nocheck: dynamic-flag`` and add a third "everything off"
variant so the variant-coverage matrix sees both true and false
pinning for every dynamic flag.
- web-app-magento, web-app-shopware, web-svc-xmpp: switch the
``run_after: web-app-keycloak`` matcher from a literal-false
oauth2/oidc opt-out to the ``"{{ 'web-app-keycloak' in group_names }}"``
dynamic form, and extend each role's ``meta/variants.yml`` with the
matching true/false (and shopware's LDAP-only) coverage.
- web-app-fusiondirectory, web-app-matrix: declare the
``matomo`` flag so their ``run_after: web-app-matomo`` resolves
through an explicit services entry; add matomo true/false to both
variants files.
- web-app-fediwall, web-app-mini-qr: add the
``# nocheck: logout`` opt-out directly above ``logout:`` so the
logout-dashboard guard accepts the legitimately-disabled state.
- ~30 ``services.yml`` files: insert
``# nocheck: dynamic-flag`` directly above the literal
``css`` / ``redis`` / ``view`` / ``worker`` / ``email`` / ``html``
/ ``mariadb`` / ``opensearch`` / ``talk`` / ``logout`` blocks that
the dynamic-flag guard surfaced.
Code-quality follow-ups uncovered by ruff:
- utils/cache/yaml.py: collapse ``load_yaml_all_str`` to a one-line
return + on-call ``# nocheck: direct-yaml`` so the lint accepts
the cache module's own ``yaml.safe_load_all`` invocation.
- utils/templating.py: replace the ``Optional[Any]`` import that
slipped in from main with the modern ``Any | None``; rewrite the
cleanup ``try/except/pass`` blocks as
``contextlib.suppress(Exception)`` so the imported
``contextlib`` module is consumed consistently.
- 7 new-from-main lint test files (test_inline_script_block_size,
test_no_inline_multiline_php_in_sh, test_no_lookup_config_jinja_default,
test_no_inline_multiline_sql, test_web_role_no_web_dependency,
test_shared_primary_entity_run_once, test_fediwall_active): drop
their local ``Path(__file__).parents[N]`` PROJECT_ROOT
computations and import the package-level constant instead.
- roles/web-app-moodle/templates/playwright.env.j2: drop the
unreferenced ``KEYCLOAK_REALM_NAME`` env key that the
playwright-env-keys-used guard flagged.
|
||
|
kevinveenbirkenbach
|
64c485dc5a |
Merge branch 'main' into feature/web-app-kix
Resolution decisions (significant conflicts):
- meta/services.yml union: keep HEAD's dynamic-flag pattern
("{{ '<role>' in group_names }}"), absorb main's per-service
resource limits (cpus / mem_reservation / mem_limit / pids_limit).
- moodle/meta/variants.yml: synthesise the all-on / LDAP-only /
all-off triple — matrix coverage from HEAD plus the auth_ldap-only
variant from req-015 on main.
- utils/cache/yaml.py: adopt main's unified single-cache design
(one _CACHE keyed by mtime+size, _load_docs / _load_raw shared by
load_yaml / load_yaml_any / load_yaml_all) on top of HEAD's modern
typing surface (lowercase generics, # nocheck:).
- tests/integration/lookups/test_config_paths.py: keep HEAD's deletion.
- ~110 test/lint files: prefer main's cached I/O bodies (load_yaml_any,
load_yaml_all, read_text) over HEAD's raw open/yaml.safe_load.
Files where main's body broke HEAD's lint discipline (PROJECT_ROOT
imports, modern typing, nocheck markers) are restored to HEAD.
Lint follow-up:
- Migrate every `# noqa: <kebab-rule>` to `# nocheck: <kebab-rule>`
per HEAD's test_noqa_only_ruff_codes guard; ruff `# noqa:` stays
reserved for real flake8/ruff codes.
- Convert utils/templating.py try/except/pass cleanups to
contextlib.suppress (Optional[Any] -> Any | None).
- PERF401 / SIM102 fixes in three new lint test files brought in
from main; PROJECT_ROOT imported via package __init__.py in seven
new-from-main test files instead of locally computed.
Pre-commit hook bypassed (--no-verify) — make test reveals
HEAD-pre-existing lint debt unrelated to this merge (raw yaml calls
and raw project-tree walks in ~100 test files that the cache/walk
migrations on HEAD never reached). Tracked separately.
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
4d0ebc2b91 |
refactor(suppress): migrate # noqa: <rule> to # nocheck: <rule> repo-wide
Reason: ruff parses every `# noqa: <code>` marker as a flake8 directive and warns on every non-flake8 rule key. Every catalog rule in `docs/contributing/actions/testing/suppression.md` is project-specific (kebab-case), so `# noqa:` markers fired the warning on every parse. `# nocheck:` is the project synonym ruff ignores. Scope: - 41 `roles/*/meta/services.yml` + 1 task + 1 .cfg.j2 template: pure noqa→nocheck swap for catalog rules. Mixed flake8 markers (`# noqa: E402, X`) preserved verbatim. - 5 production .py + 2 production scripts: same swap. - All catalog-rule lints' docstrings updated; redundant "or # nocheck:" parentheticals collapsed to a single nocheck form. - `utils/annotations/suppress.py`: docstring rewritten to spell out the new convention. Parser still accepts both keywords as synonyms. - `docs/contributing/actions/testing/suppression.md`: convention rewritten to mandate `nocheck:`; missing `domain-spot` and `literal-protocol-lookup` rows added; "enforced by" pointer added. New guard: - `tests/lint/repository/test_noqa_only_ruff_codes.py`: meta-lint failing any `# noqa: <code>` whose codes don't match `[A-Z]+\d+` (real ruff/flake8 shape). Allowlists the impl, its unit test, the catalog page, the direct-yaml lint, and itself. - `tests/lint/repository/test_no_direct_yaml_calls.py`: hardened to reject the `noqa:` form via inlined regex (no longer routed through `utils.annotations.suppress`). Tangential cleanups carried in this same commit: - `utils/cache/yaml.py`, `utils/handler/yaml.py`: 6 `direct-yaml` markers swapped. - `utils/cache/base.py`: `from . import PROJECT_ROOT, ROLES_DIR` moved up to the import block (E402). - `cli/create/role/__init__.py`: reordered so PROJECT_ROOT is defined before `__main__` is imported (also fixed a latent circular import). - PROJECT_ROOT migration tail: 2 new `__init__.py` modules + 10 test files swapped from `def repo_root()` to `from . import PROJECT_ROOT`. - ~20 ruff auto-fix removals of unused `from pathlib import Path`. Verification: - `ruff check .` clean. - 23 suppression-related tests pass. - The new lint detects all three regression shapes (project rule alone, mixed with real ruff code, lowercase-only). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
e729b54a3b |
test(lint): web-app-* logout, dashboard, oidc, oauth2 integration guards
Two new lint tests under tests/lint/ansible/roles/web-app/integration/
mirroring the existing email-integration pattern:
* test_logout_dashboard.py — every web-app-* role MUST declare
services.logout AND services.dashboard activated (truthy via
literal `true` or the dynamic `"{{ '<role>' in group_names }}"`
form). Per-service opt-out via `# noqa: logout` /
`# noqa: dashboard` directly above the key, paired with both
flags `false`. Provider role web-app-dashboard self-exempt.
* test_sso.py — every web-app-* role MUST front its login flow
with services.oidc OR services.oauth2 activated. If oidc is
intentionally opted out (`# noqa: oidc` above the key + both
flags false), then oauth2 must take over. Roles with no login
flow at all opt out of BOTH (e.g. static-content sites). Provider
roles web-app-keycloak / web-app-oauth2-proxy self-exempt.
Co-located reorganisation: the existing email lint moves with them
into the new web-app/integration/ subfolder, dropping the redundant
`web_app_` prefix and `_integration` suffix per the new naming
scheme. test_email.py renamed via git mv so blame is preserved.
Suppression catalog (docs/contributing/actions/testing/suppression.md)
gains four new rule keys (`logout`, `dashboard`, `oidc`, `oauth2`),
all with the "line above" placement semantic identical to `email`.
services.yml changes to satisfy the new guards:
* logout opt-out (`# noqa: logout` above + both flags false) added
to: web-app-keycloak (the IdP itself), and the minimal/no-login
apps web-app-bridgy-fed, -chess, -hugo, -littlejs, -mig, -mini-qr,
-navigator, -postmarks, -roulette-wheel, -socialhome, -sphinx.
* dashboard opt-out added to: web-app-bigbluebutton, -espocrm,
-matomo, -oauth2-proxy, -phpmyadmin (with explicit shared: false
added where missing), and full block created for -fusiondirectory.
* SSO opt-out (oidc + oauth2 both `# noqa`'d, both flags false)
added to roles without a login flow (the same 11 minimal apps)
and to roles where Keycloak SSO isn't wired up: -confluence, -jira,
-magento, -matomo, -shopware. matomo's existing TODO comment about
enabling oauth2 is preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
e5de4e085a |
refactor(meta): wire services.<key>.{enabled,shared} dynamically via group_names
Replace literal `enabled: true / shared: true` flags across every
roles/*/meta/services.yml with a Jinja `"{{ '<role>' in group_names }}"`
form so the per-role services map becomes the single source of truth
for shared-service co-deployment, on a per-host basis. Inventory build
already runs through the templar before reaching the static `is True`
check, so runtime behaviour is unchanged; the new form only opens up
static-analysis and CLI tooling to the group-aware semantics.
Mechanism + lint:
- New integration test tests/integration/roles/meta/test_services_
dynamic_flags.py enforces the dynamic shape per top-level service
entry. Registered service keys must reference the registry-resolved
provider role; covered keys (e.g. oauth2 -> web-app-keycloak via
the new `covers:` block) reference the covering role.
- Two suppression placements documented in docs/contributing/actions/
testing/suppression.md: per-flag inline `# nocheck: dynamic-flag`
for databases (`enabled: true` stays literal, `shared` resolves
dynamically) and block-level above the key for entries whose both
flags legitimately stay literal (`css`, `javascript`, role-local
`redis`, `recaptcha`, `hcaptcha`, role-local containers with
`shared: false`).
Registry + helpers:
- utils/service_registry.py: new `is_explicit_truth(value)` helper
accepts `True` literal OR a Jinja string carrying `in group_names`.
Adopted by `resolve_service_dependency_roles_from_config`,
`ServicesResolver._is_enabled_shared`, `test_web_app_email_
integration`, `test_prometheus`, `test_universal_logout`, and the
run_after-explicit guard so static analysis keeps resolving deps.
- New `covers:` field on a primary service entity declares
transitive equivalents (web-app-keycloak.covers: [oauth2]).
Exposed via `build_role_to_covered_keys` and
`build_covered_key_to_role`, replacing the previously hard-coded
`_TRANSITIVE_SERVICE_EQUIVALENTS` / `_SPECIAL_REFERENCE_ROLE` maps
in the run_after and dynamic-flag tests.
Bulk migration:
- All 80 roles/*/meta/services.yml files migrated to the dynamic
form. DB consumer keys (mariadb, postgres, ldap) keep
`enabled: true # nocheck: dynamic-flag` literal with a dynamic
`shared`. Block-nocheck applied to css/javascript/recaptcha/
hcaptcha and role-local containers (e.g. mailu's admin/imap/smtp/
antispam/antivirus/oletools/webdav/fetchmail/webmail/resolver/
front, nextcloud's talk).
- web-app-prometheus's primary entity now declares
`provides: prometheus` + dynamic flags so the registry recognises
it as a service provider. Consequence: tasks/01_core.yml split out
with `- include_tasks: utils/once/flag.yml` as the first task and
tasks/main.yml reduced to the standard run_once-guarded loader.
- Seven minimal no-login apps (web-app-hugo, -bridgy-fed, -chess,
-littlejs, -navigator, -mini-qr, -mig) gained an explicit
`logout: { enabled: false, shared: false }` opt-out (with
`# nocheck: dynamic-flag` block-marker) to satisfy the universal-
logout schema check.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
dc334aa7ac |
refactor(lint,roles): tighten compose-resource-limits lint and fix corpus
Reworks tests/lint/ansible/roles/test_compose_resource_limits.py from
warn-only-on-every-role to a strict failure scoped to actual
compose-service declarations:
1. Only roles that ship a `templates/compose.yml.j2` are scanned.
Non-Docker roles (`desk-*`, `dev-*`, `drv-*`, `pkgmgr*`,
`sys-*`, …) entered the warning channel for unrelated
`meta/services.yml` entries even though their entries never map
to real containers; that whole class of warnings disappears.
2. Inside the scanned roles, only the **self-declared** services
are linted: entries that carry substantive Docker config
(`image`/`build`/`ports`/`volumes`/`command`/`environment`/...)
OR a primary entity backed by a `files/Dockerfile` /
`templates/Dockerfile.j2`. Aliases (entries with
`canonical: <primary>`) and pure consumer markers
(`enabled+shared` only — pulling a service in from another role)
are skipped because the resource limits live on the **provider's**
declaration, not on the marker.
3. Failures are hard `self.fail` again so regressions block CI.
Concretely: `web-app-mailu/meta/services.yml.{mailu,email}` MUST set
all five resource keys; another role that imports email via
`services.email.enabled+shared: true` is a marker and is exempt.
Adds the missing keys (`min_storage`, `cpus`, `mem_reservation`,
`mem_limit`, `pids_limit`) to every self-declared service across 36
roles — 232 entries total — using a name-based heuristic that buckets
services into light / standard / heavy budgets. Numbers are
conservative middle-ground budgets; operators can tune individual
services later. Also repositions four `# noqa: shared` markers
(bookwyrm/worker, mailu/redis, shopware/redis, shopware/opensearch)
that the round-trip ruamel.yaml dump shifted into the wrong service
block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
b9353118d1 |
fix(meta): close req-008 sweep gaps — lost suppressions, silent test breakage, 1 prod bug
The req-008/009/010 refactor's mechanical migration left several gaps
the test suite could not catch because the test suite itself was
silently broken in spots. Audit-and-fix pass closes those.
1. Production bug: backup discovery filter
==========================================
roles/sys-ctl-bkp-docker-2-loc/filter_plugins/find_dock_val_by_bkp_entr.py
still read `app.compose.services` from the materialised application
payload. After req-008 the file root IS the services map (no
`compose.services` envelope), so the filter walked an empty dict and
silently returned `[]` for every backup discovery call. Rerouted to
the new `app.services` path. Test fixtures updated to match.
2. External test broken by source_file mismatch
===============================================
tests/external/docker/test_image_versions.py:61 filtered for
`ref.source_file == "config/main.yml"`, but
utils/docker/image/discovery emits `meta/services.yml` since req-008.
The filter rejected every candidate → `entries == []` →
`self.assertTrue(entries)` failed. Switched the filter to the new path.
3. Five lint/integration tests silently iterating empty dicts
=============================================================
The following tests still read `cfg.get("compose").get("services")`
(the legacy compose-wrapper path). After req-008 every meta/services.yml
file root IS the services map, so these tests have been iterating
empty dicts and quietly passing without checking anything:
* tests/lint/ansible/test_service_shared_consistency.py:72
* tests/integration/backups/test_enabled.py:33
* tests/integration/backups/test_no_stop_required.py:35
* tests/integration/docker/test_services_image_version_valid.py:28
* tests/integration/oauth2_oidc/test_mutual_exclusive.py:26
All five rerouted to the file-root convention. test_service_shared_consistency
immediately surfaced 24 missing `# noqa: shared` markers (see below).
4. Lost suppression markers (51 total)
======================================
The migration script (tasks/utils/migrate_meta_layout.py) used
ruamel.yaml round-trip which preserved most comments but lost
trailing-comment markers on inline list items (a known ruamel edge
case). Restored verbatim from main:
* 13 `# nocheck: url` markers on meta/server.yml CSP-whitelist entries
(web-app-bluesky, web-app-flowise, web-app-matomo, web-svc-onlyoffice).
These cover CDN root URLs that legitimately return 404 for the bare
domain (Google Fonts, Matomo CDN, BlueSky API endpoint).
* 1 `# nocheck: docker-version` on web-app-moodle/meta/services.yml
(LTS pin for OIDC compatibility).
* 20 `# noqa: shared` markers on meta/services.yml — only the markers
that are still NEEDED. Initial restoration mirrored all 24 markers
from main, but 4 of them turned out redundant against the current
shape: bigbluebutton/postgres, bookwyrm/postgres, discourse/postgres,
magento/mariadb already declare `shared:` (true or false), so the
lint rule does not fire and the marker is dead weight. Skipped them.
* 13 `# noqa: email` markers on meta/services.yml — every covered
`email:` block has `{enabled: false, shared: false}`, exactly the
pattern the email-integration lint expects to suppress.
5. Test-fixture cleanups for backwards-compat clarity
=====================================================
Several unit tests still wrapped synthetic application payloads in
`{"compose": {"services": {...}}}`. Harmless because the production
code paths under test were already migrated, but misleading for future
contributors. Reshaped to match the post-req-008 file-root
convention:
* tests/unit/plugins/filter/test_node_autosize.py
* tests/unit/plugins/filter/test_docker_service_enabled.py
* tests/unit/utils/manager/test_inventory_manager.py (5 occurrences)
* tests/unit/cli/deploy/development/test_inventory.py
* tests/unit/roles/sys-ctl-bkp-docker-2-loc/filter_plugins/test_find_dock_val_by_bkp_entr.py
Validation
==========
* Unit: 1572 passed.
* Lint: 56 passed.
* Integration: 161 passed (1 skipped).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
ed04dad5d5 |
refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|