mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
fix/dev-python/python-upper-bound
145
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  kevinveenbirkenbachandClaude Opus 4.8
|
70ad33bc13 |
fix(web-app-peertube): repair the SeaweedFS object-store e2e test for PeerTube v8.2.1
The test timed out: the publish-button locator no longer matched the v8.2.1 manage-video wizard, and the inline 78-byte MP4 stub was structurally invalid, so PeerTube rejected the upload as "Video file unreadable" and left the Save button disabled. - test-seaweedfs.js: target the v8.2.1 save control (.save-button > button), upload PeerTube's real video_short.mp4 fixture, and assert a version-tolerant saved marker. - add fixture roles/web-app-peertube/files/playwright/fixtures/video_short.mp4. - test-e2e-playwright: stage role-local files/playwright/fixtures/ next to the tests (the *.js glob ships neither binaries nor subdirectories), so specs load a fixture via path.join(__dirname, "fixtures", <name>). - docs: document the fixtures/ folder and the per-scenario test-*.js split in the playwright spec contract and the runner README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
16a4298dbb |
refactor(ci): rename test-deploy-server to test-deploy-compose
Use whale (🐳) for the per-app matrix job to mirror the swarm workflow's bee (🐝) pattern, and magnifying glass (🔍) for the discover step. Keep the workflow header on the existing 📦 prefix to stay consistent with the other test-deploy-* siblings. Rename the workflow file and align all references: - ci-orchestrator.yml job key + uses: path - artifact names: inventory-server-* -> inventory-compose-*, playwright-server-* -> playwright-compose-* - scripts/github/inventory/zip_compose_for_upload.sh output path - 3 docs files INFINITO_DEPLOY_TYPE: server stays - that key is wired to the invokable-rule registry in utils/roles/validation/invokable.py and is a separate refactor concern. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
kevinveenbirkenbach
|
84bd56e72c | Merge branch 'main' into feature/req-21-sso-flavor-migration | ||
|
kevinveenbirkenbachandClaude Opus 4.7
|
8e293bd921 |
fix(compose-deploy): stop persisting runtime routing knobs into .env
Root cause of step-08 deploy failures across all 4 test-development
distros in run 26572559126: `make compose-deploy mode=reinstall
apps=web-app-dashboard INFINITO_SERVICES_DISABLED=matomo` (step 07)
made make export the cmdline var into the recipe's shell env, which
reached `runner.py` → `make setup` → `make dotenv`. The dotenv
generator's `setdefault` then baked the runtime-only value into the
on-disk `.env`. Step 08 sourced that .env, filtered `web-app-matomo`
out of its primary apps, and exited with "All primary apps disabled".
Fix:
- Drop the 7 routing knobs (apps, mode, purge, bundles, disable,
full_cycle, variant) from `passthrough.STATIC_KEYS` and from
`default.env`. They are per-invocation; .env is the persistent SPOT
and must not capture them.
- Switch every consumer from `os.environ["KEY"]` strict access to
`os.environ.get("KEY", "")` so the missing-key path stops at "no
filter" instead of a KeyError.
- Rename the env-var surface to match the make-alias surface 1:1
(lowercase): INFINITO_APPS → apps, INFINITO_SERVICES_DISABLED →
disable, INFINITO_BUNDLES → bundles, INFINITO_FULL_CYCLE →
full_cycle, INFINITO_VARIANT → variant, INFINITO_DEPLOY_MODE → mode,
INFINITO_PURGE_ENTITIES → purge. The make alias was the user-facing
name in every doc; routing-knob env vars now use the same name so
the cmdline form, the recipe env-prefix, and the consumer reads
all line up.
- Collapse the Makefile `compose-deploy` recipe to just the two
knobs whose alias differs from the env name (`type` →
INFINITO_DEPLOY_TYPE, `debug` → INFINITO_DEBUG). GNU Make
auto-exports cmdline variables into the recipe's shell env, so the
identical-name `$(if $(apps),apps="$(apps)")` lines were pure
noise.
- Drop `disable: ${disable}` from compose.yml. The dev CLI builds
`extra_env` per `docker exec` call and forwards `disable` via
`-e`, so the container env never needs the value frozen at
compose-up time.
- Update the 3 actual leak call sites
(07/08/09_deploy_*.sh) to the make-alias form
(`disable=matomo`, `full_cycle=true`, `variant=1`).
- Add `SIM112` to ruff's ignore set: the lowercase env names are
the deliberate make/env parity, not a typo.
Repro before fix: `INFINITO_SERVICES_DISABLED=matomo make dotenv`
wrote `INFINITO_SERVICES_DISABLED=matomo` to `.env`. After fix:
`disable=matomo make dotenv` leaves `.env` untouched.
Tests: `make test` and `make autoformat` green (2711 tests, 8 lint
sub-targets).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
kevinveenbirkenbach
|
4187237ed0 |
docs(req-21): rename stale oidc/oauth2 service-gate refs to sso
Three docs still referenced the pre-collapse service keys:
* docs/contributing/artefact/files/role/playwright.specs.js.md — biber
and administrator auth-chain diagrams listed
`skipUnlessServiceEnabled('oidc' | 'oauth2' | 'ldap')`; the unified gate
is `'sso' | 'ldap'`.
* docs/contributing/design/role/services/base.md — the `lookup('service',
...)` example block carried the now-invalid `'oidc'` key; replaced
with `'sso'`.
* docs/requirements/019-playwright-meta-services-parity.md — the
prometheus, matomo, and mailu rows still spelled
`safeSkipUnlessEnabled("oauth2"|"oidc")` and
`isServiceEnabled("oauth2")`; all rewritten to `"sso"`.
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
29377e1bd3 |
fix(cache/applications): stop variant-0 service leak via base-config defaults
`_build_application_defaults` returned variant 0 of every role with a
meta/variants.yml, which then deep-merged with the variant-N override
that build_dev_inventory bakes into the inventory. Any service flag
that variant N did not explicitly disable carried through from
variant 0 into the runtime applications map. The most visible casualty
was svc-bkp-container-2-local variant 1 (postgres + mariadb only),
where gitea + keycloak leaked in via variant 0, sys-service-loader
pulled in svc-db-openldap via keycloak.services.ldap, and the deploy
crashed in sys-svc-compose/04_files.yml on openldap's env.j2 with
"credentials is defined in schema at roles/svc-db-openldap/meta/schema.yml
but not set in application 'svc-db-openldap'".
Defaults now come from each role's variant-free base config
(meta/<topic>.yml plus schema-defaults, no variants overlay). Variants
stay accessible via get_variants for any caller that explicitly needs
a specific variant. Inventory-level applications.<app> overrides win
exactly as before.
Supporting changes that keep the variant matrix consistent under the
new defaults model:
- New lint guard tests/integration/roles/meta/variants/
test_non_baseline_explicit_disables.py: for every dynamic
service-key declared in a role's meta/services.yml that variant 0
pins to literal true, every non-baseline variant MUST explicitly
pin the same key to either literal true or to literal
enabled: false + shared: false. Statically-enabled keys are exempt
(same rule as test_auth_isolation.py).
- roles/web-app-{fusiondirectory,lam,nextcloud,phpldapadmin}/meta/
variants.yml: explicit `false` pins for dashboard / email / logout /
matomo / prometheus / talk in restrictive variants so deep-merge no
longer hands them the deleted variant-0 inheritance.
- roles/web-app-moodle/templates/playwright.env.j2: gate the LAM
block on `lookup('config', application_id, 'services.lam.enabled',
false)`. LAM is a transitive helper, not a moodle-owned service, so
the default-bearing lookup tolerates the missing path when the
variant does not deploy LAM.
- tests/unit/utils/cache/test_variants.py: rename the legacy
"always variant zero" coverage to TestMergedApplicationsUsesBaseConfig
and add test_get_application_defaults_omits_variant_zero_only_keys
as the regression guard for the leak this commit fixes.
- docs/contributing/artefact/files/role/variants.md +
docs/contributing/design/variants.md: variant 0 now manually
enumerates every dynamic services.yml key with
enabled: true + shared: true; `{}` is only permitted when the role
has no dynamic-enabled service-key.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
ee01f331a8 |
docs(variants): document variant-aware credentials generation
design/variants.md gains a "Credentials Generation" section that explains both flows the credential machinery can take: - CI/CD path: planner picks the round's variants -> build_dev_inventory emits --app-variants -> provision CLI forwards --variant -> credentials CLI hands the index to InventoryManager, which resolves the root role's variants.yml overlay so resolve_schema_includes_recursive sees the variant-merged services map and discovers every shared provider that the variant enables. - Standalone path: any caller that omits --app-variants / --variant gets variant=None, InventoryManager skips the overlay, and shared- provider discovery walks the base meta config only. Also adds four reference-table rows for the involved files and fixes two unrelated wording issues in the surrounding text: - "every dep round R needs MUST be redeployed" -> grammatical fix. - "historical payload" / "rename the file back" -> drop historical framing per documentation.md's current-state-only rule. artefact/files/role/variants.md gains a short "Credentials Interaction" section that anchors back to the design doc so a role author setting services.<key>.enabled+shared in a variant knows that the credential generator will pick the provider up. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
aacca6025d |
docs(playwright-specs): require dedicated LDAP scenario per persona, drop contradictory variant-pinning sentence
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
5196283f6c |
refactor(test-e2e-playwright): bake @playwright/test into a shared node_modules
Per-role Playwright runs no longer call `npm install` against registry.npmjs.org on every invocation. A new `01_setup.yml` runs once per host/version, populates `_shared/node_modules/` inside the Playwright container, and stamps the installed version in `.installed-version`. Each `02_run_one.yml` mounts that tree read-only at `/e2e/node_modules` and the container command becomes `npx --no-install playwright test`. This eliminates the npmjs.org TLS-handshake flake that took out the keycloak playwright in a recent variant-2 update, saves ~4 s per role, and gives version bumps a single rebuild point. Three retry layers cover transient registry failures: Ansible (3× with 5 s delay), npm fetch (5× with 10-60 s backoff), and image pull (3×). Stage-base / shared-dir / stamp / node_modules paths live as SPOT constants in `vars/main.yml` so call sites never reconstruct them. Renames `tasks/run_one.yml` → `tasks/02_run_one.yml` (and adds `01_setup.yml`) to make the include order visible by filename; updates cross-references in `AGENTS.md`, the two `docs/contributing/...` pages, `dump_disk_diagnostics.sh`, and the lint test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
b3c519b914 |
refactor(images): route role image refs through lookup('config', ...) so the GHCR mirror takes effect
`lookup('image', '<role>', '<svc>', 'ref')` only consulted
`images_overrides.*` plus role defaults, which the mirror pipeline
never populates — the pipeline writes into
`applications.<role>.services.<svc>.{image,version}`. As a result,
`sys-svc-container`'s `BUSYBOX_IMAGE` / `NODE_IMAGE`,
`sys-ctl-hlth-csp`'s `HEALTH_CSP_IMAGE`, and `test-e2e-playwright`'s
`TEST_E2E_PLAYWRIGHT_IMAGE` always resolved to the raw upstream ref
and hit Docker Hub directly. In CI this surfaced as
`toomanyrequests: ...unauthenticated pull rate limit` on
`busybox:1.37`.
Switches every productive `lookup('image', ...)` to
`lookup('config', '<role>', 'services.<svc>.image|version')` — the
same path templates like `web-app-oauth2-proxy` already use, which
resolves through the merged `applications.*` map and so transparently
picks up the GHCR mirror override written by the inventory creator.
Removes the now-unused `plugins/lookup/image.py` plugin and its unit
test (zero remaining consumers across roles, scripts, and tests).
Rewrites the image-origin / image-mirror / dockerfile docs to teach
the `lookup('config', ...)` pattern, dropping references to the
deleted plugin and the dead `images_overrides.*` variable. Doc edits
follow the `documentation.md` rules (no em dashes, one sentence per
line, no historical phrasing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
c02229b4be |
refactor(make): align targets to domain prefixes (compose-, network-, etc.)
Apply a consistent naming convention to the Make target inventory so
every target is grouped by its domain:
- compose-* (compose-up/down/restart/stop/exec/inner-run)
- network-* (network-dns-setup/remove, network-ipv6-disable/restore,
network-refresh, network-trust-ca)
- security-* (security-apparmor-restore/teardown)
- diagnose-* (diagnose-network, diagnose-disk-usage)
- fix-* (fix-chmod, fix-dockerignore)
- clean-* (clean-cache, clean-pycache-dirs)
- install-* (install-agent)
- meta-* (meta-list, meta-tree, meta-mig)
Drop the stale debug-network target and reintroduce its functionality
as diagnose-network. Cascade renames through:
- Makefile (.PHONY entries, prereqs, recursive $(MAKE) calls)
- contributor docs, agent-iteration docs, role-level docs
- bundle README, environment scripts, install/lint/test wrappers
- AGENTS.md, CLAUDE.md, compose/cache.override.yml, .github/workflows
Historical entries under docs/requirements/ and changelog/ stay
untouched as audit logs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
13c93ad6d5 |
refactor(compose): drop the ci profile and its --profile ci arg everywhere
compose.yml no longer hides services behind \`profiles: ["ci"]\`. The profile existed solely as a no-op safety net against a bare \`docker compose up\` in the repo root, but the top-level compose.yml is already explicitly scoped to local dev / CI testing (production goes through the Ansible roles under roles/<role>/templates/), so the safety net is redundant with the documented scope. Every \`--profile ci\` callsite collapses: - compose.yml: \`profiles: ["ci"]\` removed from coredns + infinito - Profile.args() deleted; Profile keeps is_ci() and registry_cache_active() for the cache-override gating - compose.py / down.py: profile.args() splat removed from the docker-compose command builder; down.py drops its now-unused Profile import - 5 shell wrappers (install, lint, tests/code, meta/resolve/apps, meta/resolve/diff/affected_roles): plain \`docker compose\` instead of \`docker compose --profile ci\` - tests/unit/.../test_profile.py: TestProfileArgs class removed - tests/unit/.../test_compose.py: two cmd-shape asserts updated to reflect the missing --profile pair Docs: - docs/contributing/artefact/files/compose.yml.md: the \"every service MUST be behind a profiles: entry\" rule is gone - docs/contributing/actions/debugging/ci.md: the \`docker compose ... --profile ci config\` debug snippet drops the flag; while touching the file the two em-dash clause separators in the reproduction-profiles shell block are rewritten to colons (documentation.md em-dash rule), and one list-item in compose.yml.md is collapsed to one sentence per line for consistency with its siblings Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
12a56e449d |
refactor(env): move env/default.env to repo-root default.env and drop env/
With env.development and env/ci.env both gone, env/default.env was the
last inhabitant of the directory. Hoisting it to the repo root removes
the empty-shell indirection: the SPOT is now [default.env](default.env),
sitting next to .env (its generated companion) and other repo-root
configuration artefacts like compose.yml, Dockerfile, and Makefile.
Path updates land in:
- generator: cli/meta/env/__main__.py (REPO_ROOT / "default.env")
- workflow: .github/workflows/test-install-make.yml (awk reads from
default.env, error message updated in lockstep)
- lint tests: test_used_keys_declared.py and test_infinito_prefix.py
switched from globbing env/*.env to targeting default.env directly;
test_default_env_keys_handled.py points at the new path
- drift test: test_inventory_vars_file_spot.py whitelist + STATIC_ENV
- smoke test: tests/integration/meta/env/test_dotenv_generator.py
STATIC_ENV + every docstring/error message
Doc-string and prose references in utils/env/*, scripts/*,
cli/administration/deploy/development/{common,coredns}.py,
Makefile, Dockerfile, and every doc page that mentioned env/default.env
are rewritten in the same pass so future readers see one path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
07a9440ac7 |
refactor(env): fold ci.env into default.env so INFINITO_IP4/DNS/DOMAIN reach Ansible
The four ci.env keys (INFINITO_DNS_IP, INFINITO_DOMAIN, INFINITO_DOMAIN_RE,
INFINITO_IP4) were only honored by docker compose --env-file, so processes
that sourced .env via load.sh saw INFINITO_IP4 as unset. As a result
networks.internet.ip4 in the Ansible playbook fell through to the
127.0.0.1 default and the DinD DNS-check handler failed on the deploy
server matrix.
Moving the defaults into env/default.env makes them flow through the same
.env / load.sh / os.environ chain as every other INFINITO_* key, so
\${INFINITO_IP4} in compose.yml, lookup('env', 'INFINITO_IP4') in
group_vars, and envsubst over the CoreDNS Corefile template all see the
same value deterministically.
Side fix: utils/env/parser.py now undoes the writer's \\\\/\\" escaping for
quoted values, closing a round-trip gap that surfaced once
INFINITO_DOMAIN_RE (the only value containing a literal backslash) started
flowing through the generator.
All --env-file env/ci.env call sites (compose.py, down.py, install/lint/
test wrappers, resolve scripts, test-dns workflow + script) are removed.
coredns.py now reads .env via parse_static_env, docs and unit/integration
tests follow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
b278cd8ccd |
refactor(entry.sh): switch INFINITO_COMPILE{,_SILENCE} env-vars to flags
scripts/docker/entry.sh's two compile-control reads (INFINITO_COMPILE
and INFINITO_COMPILE_SILENCE) crashed during Dockerfile RUN because
that context has no env-pipeline: env/default.env exists in the repo
but no make/BASH_ENV/load.sh chain materialises it into the running
shell. The bare reads under `set -u` triggered unbound-variable exit
even though the keys were registered in default.env + passthrough.
Move build/silence control from env to script flags so entry.sh is
self-contained at bootstrap time. Each flag now triggers its action
directly inside the parse loop -- no COMPILE/SILENCE state vars:
--compile -> run_local_build (streaming)
--compile-silent -> run_local_build >/dev/null 2>&1
-- -> end flag parsing, rest is the exec target
Drop INFINITO_COMPILE + INFINITO_COMPILE_SILENCE from the whole
registry surface: env/default.env, passthrough.STATIC_KEYS, the
infinito_compile handler (gone entirely), compose.yml's env
injection block, the dotenv-generator baseline test, the
build_env unit-test fixture, and the now-no-op `-e INFINITO_COMPILE=0`
in test-install-{make,pkgmgr} workflows.
Dockerfile RUN switches to `entry.sh --compile -- true`; runtime
ENTRYPOINT receives no flags so the default (no-compile) holds and
exec hands off to systemd via the CMD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
eb4bcd23a8 |
refactor(env/spot): forbid .get() defaults + cache namespace rename
Three new lint tests close gaps in the INFINITO_* SPOT contract:
* tests/lint/filesystem/env/test_default_env_keys_handled.py
Every default.env key must be wired through a handler.
* tests/lint/filesystem/env/test_used_keys_declared.py
Every INFINITO_* in .sh/.py must live in env/{default,ci}.env or
a handler; per-line `# nocheck:` suppresses runtime-only refs.
* tests/lint/filesystem/python/test_no_infinito_get_defaults.py
Forbid ANY default in `.get("INFINITO_*", ...)` -- including the
empty-string fallback (which silently masks an unset key). The
nocheck check walks every physical line the call spans so an
autoformat-split call still honours the marker.
Cache namespace flattens to a single prefix:
INFINITO_PACKAGE_CACHE_* -> INFINITO_CACHE_PACKAGE_*
INFINITO_REGISTRY_CACHE_* -> INFINITO_CACHE_REGISTRY_*
The five previously driver-injected paths (APT_LIST, NPMRC,
PIP_CONF, REGISTRY_PROXY_CONF, FRONTEND_CA_FILE) move into
env/default.env and passthrough.STATIC_KEYS; the
`cache_env_overrides()` helper and its two call sites are gone
along with the test-fixture override for FRONTEND_CA_DIR.
SPOT-strict rewrites:
- 19 sites: `something.get("INFINITO_X", "")` -> subscript
`something["INFINITO_X"]` (loud KeyError when env loader did
not run; all keys live in env/default.env).
- 2 handler-internal bootstrap reads suppressed with
`# nocheck: handler-bootstrap-read`.
- 30 runtime-only / driver-injected / test-fixture references
annotated with `# nocheck: <reason>` per-line.
- Unit tests that previously removed SPOT-guaranteed keys via
`{k: v ... if k \!= KEY}` or `clear=True` switch to setting
the key to "" explicitly, matching the new contract.
Side fixes:
- INFINITO_DNS_DOMAIN + INFINITO_DNS_HOSTS_FILE move from
shell-side defaults in common.sh to env/default.env and
passthrough.STATIC_KEYS.
- dns_systemd_is_operational() drops the dead
INFINITO_DNS_FORCE_HOSTS_FALLBACK override branch (the
variable was never set anywhere).
- INFINITO_WORKER_FETCH switches from `os.environ.get(KEY, "1")`
to `os.environ[KEY]`.
- scripts/tests/code/wrapper.sh exec args move into a bash
array so `# nocheck:` on INFINITO_TEST_TYPE can live on its
own element line (SC2215 fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
22cf6f38f1 |
refactor(env+make): rename static.env→default.env + sort + lint gates
Rename env/static.env → env/default.env across the entire repo
(string label, doc links, Python file-path constants) — the
`parse_static_env*` function names stay as a stable API surface.
Sort all `*.env` / `*.env.j2` entries alphabetically (52 in
default.env, 4 in ci.env, 46 of 49 j2 templates; 3 templates
with Jinja2 `{% %}` control flow are skipped because conditional
blocks would tear apart under a naive sort). Sort all 83
Makefile target blocks alphabetically; .PHONY group headers
stay in place.
Collapse the two remaining multi-line Makefile recipe comments
(dotenv-force, test-signed) to single-line WHY comments.
Add four lint tests under tests/lint/filesystem/ to keep the
invariants enforced going forward:
- env/test_alphabetical_sort.py
- env/test_single_line_comments.py (# nocheck: markers exempt)
- makefile/test_alphabetical_targets.py
- makefile/test_single_line_comments.py (# nocheck: markers exempt)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
9a578952c3 |
feat(env): SPOT env-pipeline + INFINITO_* namespace
Replace 8 bash env modules with a single Python generator
(cli.meta.env) that materialises env/static.env + runtime context
into a gitignored .env. Each variable lives in its own handler
under utils/env/handlers/; defaults carry inline one-line `# ...`
comments.
Rename to the INFINITO_* schema:
- workflow inputs: APPS, SERVICES_DISABLED, CMD, DEBUG, WHITELIST,
BUNDLES, FULL_CYCLE, VARIANT, RUN_FLAGS, LIMIT_HOST, PARALLEL
- inventory: INVENTORY_DIR / INVENTORY_FILE / HOST_VARS_FILE,
PW_FILE -> INVENTORY_PASSWORD_FILE
- venv: VENV_BASE / VENV_FALLBACK / VENV_NAME
- new SPOT defaults: COMPILE_SILENCE, CONTAINER_NO_CA,
WAIT_HEALTH_TIMEOUT_S, PACKAGE_CACHE_FRONTEND_INIT_IMAGE,
promoted IMAGE_TAG from GHA-only to general passthrough
compose.yml hardening:
- inline \${VAR:-default} removed; strict \${VAR:?...} everywhere
- new integration lint blocks regressions
Tooling:
- scripts/install/all.sh owns the stamp-file install logic
(Makefile target delegates)
- scripts/meta/env/load.sh is the single shell entry point;
8 bash modules deleted
- new shell lint: no INFINITO_* defaults outside env/static.env
Tests:
- 93 new unit tests for utils/env/{builder,parser,runtime,writer}
- restructure tests/lint into repository/ (file-content rules) vs
filesystem/{shell,python,env}/ (filesystem-shape rules)
- drift + smoke integration tests against the generator
Docs:
- variables.md is now a pointer; per-variable docs live next to
the key in env/static.env / env/ci.env
- compose.yml.md drops the Default column
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
f9147e4486 |
refactor(scripts/github): split helpers into responsibility subfolders
Group the shell helpers under scripts/github/ into five subfolders that match how the workflow files consume them: runner/ host-runner hygiene (disk, swap, Docker daemon, diagnostics) cancel/ run cancellation on PR-close and branch-delete events resolve/ derivation of workflow inputs and outputs from repo state update/ scheduled Update: Versions pipeline release/ release-time gates Drop the now-redundant subfolder prefix from each filename, e.g. dump_runner_state.sh becomes dump_state.sh and cancel_pull_request_runs.sh becomes pull_request_runs.sh. Each subfolder carries a README that describes its scope per documentation.md rules; the top-level README points at the subfolders without inventorying files. Propagate the new paths through ten workflow files, three docs (workflow.md, pipeline.md, secrets.md), and two tests. Fix a real runtime bug in enlarge_swap.sh, which still referenced the pre-rename dump_runner_state.sh path. Extend the inline-block-size lint hint to describe the new subfolder convention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
ab99f5dbc0 |
refactor(meta/services): rename ports.inter to ports.internal.<category>
The internal container port slot under <entity>.ports is now a
category-keyed map (matching the local/public shape) instead of a
single int. "internal" replaces "inter" because the latter is
ambiguous in English (could read as "between").
Affected:
- 8 role meta/services.yml (matrix, ollama, mig, shopware, oauth2-proxy,
gitea, listmonk, sphinx) and every caller (vars, templates, group_vars).
- web-app-matrix: 8008 hardcoded across compose.yml.j2, homeserver.yaml.j2,
tasks/01_docker.yml, and 7 mautrix bridges now reads
MATRIX_SYNAPSE_CONTAINER_PORT from services.synapse.ports.internal.http.
- web-app-matrix: admin username/password lookups consolidated as
MATRIX_ADMINISTRATOR_{USERNAME,PASSWORD} in vars/main.yml, sourced from
lookup('users', 'administrator').
- Tooling: cli/meta/ports/suggest, utils/meta/scan, integration scan,
migrate_meta_layout/role_builder updated to the new key name.
- Docs: design/role/services/layout.md, tools/ports-suggest.md,
requirements 009/021, image/origin.md, PR template.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
16cab61397 |
fix(image-lookup): require explicit role_id to prevent wrong-role resolution
The `image` lookup inferred `role_id` from `role_name` when only
`service_name` was passed. Inference broke lazy re-evaluation: a default
like `HEALTH_CSP_IMAGE: "{{ lookup('image', 'csp-checker', 'ref') }}"`
defined in sys-ctl-hlth-csp got re-evaluated inside sys-service's
systemctl.service.j2 template (via `system_service_tpl_exec_start`),
where `role_name='sys-service'`. The lookup then searched
`roles/sys-service/meta/services.yml` for `csp-checker` and aborted with
`no image mapping found for role 'sys-service' service 'csp-checker'`.
CI run 25976825584 surfaced this on six apps (web-app-{bridgy-fed,chess,
mig,hugo,mini-qr,postmarks}) whose deploy reached sys-svc-webserver-core
outside the service-loader preload context, which is what actually
triggers the sys-ctl-hlth-csp -> sys-service include chain.
Make `role_id` mandatory in the plugin signature, remove `_infer_role_id`,
and update the three callsites that relied on inference
(sys-ctl-hlth-csp, sys-svc-container, test-e2e-playwright) to pass the
owning role id explicitly. A migration guard catches the removed
`(service_name, want)` form and emits a clear error instead of the
confusing "missing services.yml" file-not-found message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
c2c532c8e6 |
refactor(playwright): centralize dashboard-tile mechanics in web-app-dashboard SPOT; move specs into files/playwright/ subdir
Architecture
============
- New SPOT helper `roles/web-app-dashboard/files/playwright/dashboard-card-flow.js` (`runDashboardCardScenario(page, context, target)`): per `roles_with_service('dashboard')` target, asserts tile visible → click loads in `#main iframe` (outer URL gets `?iframe=` + iframe src matches canonical_domain) → header "Tab" button pops the embed into a fresh browser tab.
- `web-app-dashboard/files/playwright/playwright.spec.js` parametrised loop calls the helper for every `DASHBOARD_TARGET_ROLES_JSON` entry — the single owner of dashboard-tile mechanics across the whole codebase.
Spec relocation
===============
- 71 `roles/<role>/files/playwright.spec.js` → `roles/<role>/files/playwright/playwright.spec.js` (`git mv`, history preserved). odoo + pixelfed show as D+A in `git status` because of substantial content rewrites; coverage retained.
- New SPOT constant `ROLE_FILE_PLAYWRIGHT_SPEC` in `utils/roles/mapping.py`, registered as optional on `ROLE_TYPE_APPLICATION` (disallowed elsewhere, mirrors `ROLE_FILE_META_VARIANTS` pattern).
- `roles/test-e2e-playwright/tasks/run_one.yml`: staging task replaced with `with_fileglob` over `files/playwright/*.js` — roles can ship companion helpers (like `dashboard-card-flow.js`) next to the spec without further wiring.
Spec migrations (no more dashboard-iframe path in app specs)
============================================================
13 specs rewritten so the persona/SSO journey starts at the app URL directly, never through the dashboard tile or iframe:
* fider, mailu, mattermost, nextcloud, odoo, taiga: iframe-wrap removed (no more `frameLocator("#main iframe")`)
* bigbluebutton, discourse, kix, matrix, openwebui, peertube: dashboard goto preamble removed
* pixelfed: ~270 LOC of iframe-aware helpers/login/logout collapsed to direct-URL flow
Title cleanup
=============
- 64 specs: `"<persona>: dashboard → app → universal logout"` / `"administrator: dashboard → prometheus → app → universal logout"` → `"<persona>: app → universal logout"`.
- 13 specs: `"<persona>: dashboard to <app> ..."` → `"<persona>: <app> ..."`.
Dead-code cleanup
=================
- `dashboardBaseUrl` consts + beforeEach assertions removed from 11 specs (no functional uses; personas helper handles env validation).
- `DASHBOARD_BASE_URL=` env-template lines removed from fider, mattermost, odoo, taiga, pixelfed, kix.
- `DASHBOARD_SERVICE_ENABLED=` env-template lines removed (pixelfed retained for the still-gated spec).
- Stale "loads inside the iframe" / "FrameLocator parameter" comments cleaned in fider, odoo, prometheus, nextcloud.
Tooling
=======
- 5 lint Python files + 1 integration test now use `from utils.roles.mapping import ROLE_FILE_PLAYWRIGHT_SPEC`.
- `eslint.config.js` glob pattern + `roles/test-e2e-playwright/meta/main.yml` description + `roles/test-e2e-playwright/README.md` updated to new path.
- 25 markdown/docstring references in `docs/`, `roles/**/README.md`, `roles/**/TODO.md`, persona-helper docstrings updated to new path.
- `docs/contributing/artefact/files/role/playwright.specs.js.md`: file-placement section now documents the `files/playwright/` convention + `ROLE_FILE_PLAYWRIGHT_SPEC` SPOT constant; dashboard SPOT delegation explicit at 3 places.
Net diff: 124 files, +1142/-1564 (−422 LOC).
Integration suite: 183/183 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
98457f98fd |
refactor(update): unify version bumpers under utils/update + cli/contributing/update + scripts/github/update
Folds the parallel Docker-image and git-ref version-update plumbing
into a single subsystem so the auto-update workflow, the external
freshness tests, and the unit tests all share one module layout.
utils/
update/
base.py # semver primitives shared by both backends
docker.py # OCI registry lookups + version: line rewriter
repository.py # git ls-remote walker + ref: line rewriter
cli/contributing/update/
docker/ # python -m cli.contributing.update.docker
repository/ # python -m cli.contributing.update.repository
scripts/github/update/
image_versions.sh
repository_refs.sh
Tests mirror the same shape under tests/{unit,external}/update/{docker,repository}/.
Also widens the Docker bump scope to all roles (previously gated on
`web-*`) so non-web roles like sys-svc-container get bumped too;
pins Bookwyrm's python image at 3.11-bookworm via
`# nocheck: docker-version` with a TODO until upstream supports
Python >= 3.12.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
4ac7a95ec6 |
refactor(tests): split overflowing folders into thematic subdirs
Reorganises four folders that exceeded the 12-item readability limit: - tests/lint/repository/ (27 -> 11): structure/, naming/, documentation/, python/, dependencies/, shell/, yaml/, network/, settings/ - tests/integration/ (25 -> 7): ansible/, infrastructure/, iam/, meta/ (24 sibling dirs grouped under 4 super-categories) - tests/integration/roles/ (13 -> 10): naming/, playwright/, dashboard/ - tests/unit/utils/ (15 -> 12): certs/ for 3 cert/TLS test files Renames 7 test files whose name duplicated the new folder name (e.g. test_no_direct_yaml_calls.py -> yaml/test_no_direct_calls.py, test_dashboard_integration_scope.py -> dashboard/test_integration_scope.py). Adjusts parents[N] in moved __init__.py files, fixes __file__-based path resolution in 8 tests, updates self-path whitelists, and uses absolute imports for PROJECT_ROOT re-exports (ruff TID252). Adds README.md to each new subfolder following the existing scope template. Updates all docs links (suppression.md, base.md, email.md, playwright spec/parity docs, 005-/019- requirements) to new paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
9cefcba516 |
fix(cache): lower default freshness window to 6 days (< apt Valid-Until)
`INFINITO_PACKAGE_CACHE_MAX_AGE_MIN` was 129600 (90 days), 13× longer than the 7-day `Valid-Until` window Debian and Ubuntu generate by default in their apt `Release` files. Nexus held cached `Release` manifests for 90 days, so once the upstream `Valid-Until` elapsed the manifest stayed in the cache for another ~83 days and every Docker build that ran `apt-get update` against the package-cache aborted with E: Release file for http://deb.debian.org/debian/dists/.../InRelease is expired (invalid since 5h 30min 54s). Drop the default to 8640 minutes (6 days). Stays one day below the 7-day `Valid-Until` so Nexus always re-validates the cached `Release` against upstream before apt would see it expired. The single value still applies to every proxy repo (apt, yum, pypi, npm, gem, go, helm, raw-*), but the `Valid-Until` enforcement only exists in apt — for the other repos the change merely makes the cache 84 days fresher instead of staler, no regression. Update both docs sites that document the default: - docs/contributing/artefact/files/compose.yml.md (env-var table) - docs/contributing/environment/cache.md (operator narrative) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
576fad84ff |
refactor(images): move role images from defaults/main.yml to meta/services.yml
Consolidates image declarations under a single schema: every role's
`<svc>.{image,version}` lives in `meta/services.yml` alongside compose
service entries. The `image` lookup now reads from disk per role,
`iter_role_images` collapses to a single source, and the mirror chain
(resolver + apply_mirror_overrides) drops the dead `images:` branch.
`test-e2e-playwright` resolves `TEST_E2E_PLAYWRIGHT_IMAGE` directly via
`lookup('image', 'playwright', 'ref')` in vars/main.yml; the runtime
set_fact + override-empty-string dance is gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
7018b3c991 |
refactor(playwright): share CSP header/meta/violation helpers
Nine role specs (bigbluebutton, dashboard, discourse, keycloak,
matomo, matrix, moodle, openwebui, wordpress) each carried inline
copies of the same six CSP helpers and the `EXPECTED_CSP_DIRECTIVES`
constant. The bodies were semantically identical; only message
wording, brace style, and one captured-field-set differed.
Changes
-------
- Extends `roles/test-e2e-playwright/files/personas/utils/csp.js`
(already home to `assertCspInjections`) with
`installCspViolationObserver`, `readCspViolations`,
`parseCspHeader`, `assertCspResponseHeader`,
`assertCspMetaParity`, `expectNoCspViolations`, and the
`EXPECTED_CSP_DIRECTIVES` constant. The helpers are re-exported
through `personas/utils/index.js` so specs that
`require("./personas")` get them via the existing aggregator.
- Strips the inline definitions from the nine role specs and amends
each spec's `require("./personas")` destructuring with whichever
helpers the body still references.
Drift handling
--------------
- moodle's reduced violation-record shape (only `violatedDirective`
+ `blockedURI`) is replaced by the canonical 5-field record.
Moodle's downstream consumer asserts `.toEqual([])` on the array
only, so the extra fields are diagnostic-only — no behaviour
change.
- dashboard's 2-arg `expectNoCspViolations(page, label)` call site
is updated to the canonical 3-arg form
`expectNoCspViolations(page, null, label)`. The shared helper
guards `if (diagnostics && Array.isArray(diagnostics.cspRelated))`
before checking the diagnostics arm, so passing `null` keeps the
exact pre-migration behaviour for that call site.
- bigbluebutton and matrix had subtly different failure-message text
in `assertCspResponseHeader` and `expectNoCspViolations`. The
canonical wording is more descriptive; passing tests are
unaffected, failing tests get clearer diagnostics.
Doc
---
`docs/contributing/artefact/files/role/playwright.specs.js.md` is
extended in two places: the invariants section names the full CSP
test surface and pins the "import from `./personas`, no inline copy"
rule, and the per-service catalogue's CSP / injectors row lists the
helpers by name.
Verification
------------
- 419 passed / 1 skipped / 0 failed across `tests/lint/`,
`tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors, 283 warnings (unchanged from
baseline).
- `markdownlint-cli2`: 0 errors on the touched doc.
- 0 inline definitions of any extracted helper / constant remain in
any role spec.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
122c82910b |
refactor(playwright): share decodeDotenvQuotedValue + normalizeBaseUrl
Both helpers were inlined verbatim in 71 of the role-local
`playwright.spec.js` files (52 with the compact form, 19 with the
verbose form, plus one parameter-name variant). All bodies were
semantically identical.
Changes
-------
- New `roles/test-e2e-playwright/files/personas/utils/dotenv.js` holds
the canonical pair (`decodeDotenvQuotedValue` and `normalizeBaseUrl`),
re-exported through `personas/utils/index.js` so specs that
`require("./personas")` get them via the existing aggregator.
- Every role spec drops its inline copies and pulls the helpers from
`./personas` instead. Specs that only reference one of the two
carry only that one in the destructuring (no unused-import noise).
Doc
---
`docs/contributing/artefact/files/role/playwright.specs.js.md` no
longer carries the "once a third spec needs them, MUST be promoted"
directive: the helpers live at the shared path and inline copies are
forbidden.
Verification
------------
- 419 passed / 1 skipped / 0 failed across `tests/lint/`,
`tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors, 283 warnings (unchanged from
baseline; no new unused-imports introduced).
- 0 inline `function decodeDotenvQuotedValue|normalizeBaseUrl`
definitions remain in any role spec.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
db2a2d9638 |
refactor(playwright): move cross-service tests to provider specs
Cross-service surface assertions for dashboard, prometheus and matomo
no longer run in every consumer role's persona. They are owned by the
provider's own spec, parameterised once over a `<NAME>_TARGET_ROLES_JSON`
manifest rendered into the provider's `templates/playwright.env.j2` via
the new `lookup('roles_with_service', '<svc>')` Ansible lookup.
Changes
-------
- New `plugins/lookup/roles_with_service.py` enumerates roles that
declare `services.<svc>.{enabled, shared}` truthy in the merged
applications config, returning `[{id, canonical_domain}, …]`.
- `roles/web-app-{dashboard,prometheus,matomo}/files/playwright.spec.js`
parse `<NAME>_TARGET_ROLES_JSON` and parameterise the per-consumer
assertion (dashboard tile reachability, prometheus `up=1`, matomo
SitesManager registration). The matomo spec gains an explicit
biber-denial test (request body validated against admin DOM markers
per the body-marker rule); the prometheus admin-reach test drops
the dashboard-iframe path in favour of direct goto.
- Persona helpers `roles/test-e2e-playwright/files/personas/{admin,
biber,guest}.js` enter at `${APP_BASE_URL}/` directly. The
`clickRoleTileFromDashboard` step and the `assertPrometheus*` /
`assertMatomo*` cross-service probes are gone; `utils/{dashboard,
prometheus,matomo}.js` are deleted. `utils/csp.js` switches to
literal `process.env.<KEY>` reads so the env-keys-used parity guard
recognises matomo's CSP injector consumption.
- 42 consumer-role `templates/playwright.env.j2` templates drop
`DASHBOARD_BASE_URL`, `DASHBOARD_SERVICE_ENABLED`, `PROMETHEUS_BASE_URL`,
`PROMETHEUS_SERVICE_ENABLED`. `MATOMO_BASE_URL` and
`MATOMO_SERVICE_ENABLED` stay because the per-role CSP injection
helper still gates on matomo's tracker host.
- `tests/integration/roles/test_playwright_env_services_match.py`
globally exempts `dashboard` and `prometheus` from the
services.yml-to-env parity rule.
- `tests/unit/plugins/lookup/test_roles_with_service.py` adds 14
unit tests for the lookup, including an explicit-contract test
that pins the over-inclusive behaviour on raw Jinja strings (the
lookup expects a templar-rendered payload).
- Docs: `docs/requirements/019-playwright-meta-services-parity.md`
rewords Rules 1, 3, 12 and adds Rule 13 describing the SPOT shift.
`docs/contributing/artefact/files/role/playwright.specs.js.md`
reflects the new persona shape and per-service catalogue, and is
cleaned per `docs/contributing/documentation.md` (no SPOT self-
labels, no req-references, em dashes replaced with complete
sentences, multi-clause lines split onto separate lines, broken
anchor `#two-personas-fixed-flow-` corrected).
Verification
------------
- 418 passed / 1 skipped / 0 failed across `tests/lint/`,
`tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors.
- `markdownlint-cli2`: 0 errors on all touched docs.
- `make autoformat`: clean (FURB171 single-item-set inequality fixed).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
672c61e37e |
test(playwright/personas): hard-fail on un-executable persona journeys
The persona-flow runners now fail loudly when biber / administrator cannot reach an authenticated surface on the role under test. The previous runtime auto-detection silently `test.skip(...)`'d, hiding real regressions (broken OIDC mapping, removed logout button, misconfigured oauth2-proxy, drifted UI selectors) behind a green deploy. Roles where the persona legitimately has no access (admin-only roles, auth-provider roles) MUST declare an explicit env opt-out `PERSONA_BIBER_BLOCKED=true` / `PERSONA_ADMINISTRATOR_BLOCKED=true` in `templates/playwright.env.j2` with a documented rationale; without the flag the helper fails the test with a diagnostic naming the last-seen URL. A new lint guard (`test_persona_strict_mode.py`) hard-fails any future `test.skip(...)` in the persona helpers that is not guarded by an allowed env opt-out / auth-less-collapse / service-gate condition, and any `status === 200` branch in the prometheus / matomo deny-check helpers that does not validate role-specific body markers. Req 019 Rule 11 (strict-skip contract) and Rule 12 (200-body validation) plus a matching "Strict failure on un-executable persona cases" section in `playwright.specs.js.md` document the contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
0b14ca4d09 |
test(lint): forbid stub bodies in persona/contract playwright tests
Adds a lint that rejects empty/tautological/TODO-marked persona and contract test bodies in roles/<role>/files/playwright.spec.js, plus the spec-contract doc updates (playwright.specs.js.md, req-017/018/019) and the integration-test parity-guard tweak that articulate it. Reason: previous batch updates produced "contract: <svc> gate is wired" stubs whose bodies only proved what the gate itself already proved; the lint catches both the title shape and the body shape so a rename is not enough. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
f6865c6464 |
docs(req-019): tighten persona contract + autonomous closure procedure
Two coupled doc changes that turn req 019 from "the matrix" into a self-contained, agent-runnable rollout plan. playwright.specs.js.md (the per-role spec contract): - Persona scenarios are MUST-include with `<persona>: <flow>` naming; additional scenarios MAY be added on top. - New invariants for browser-context isolation, post-login assertion hierarchy, admin authorisation hierarchy, and `ignoreHTTPSErrors`. - New trigger entries for CSP baseline, cross-role round-trip hosting, and variant-specific scenarios. - New sub-sections "Test-budget overrides" (when `test.setTimeout`) and "Cleanup and teardown" (try/finally, idempotent, bounded). - Env-contract spelled out the minimum-required env-var set and the promote-to-shared-helper rule for `decodeDotenvQuotedValue` / `normalizeBaseUrl`. - OIDC-over-LDAP preference: when both are enabled, the persona flow uses OIDC; the LDAP-bind path lives in a separate scenario exercised by the LDAP-only matrix variant. req 019: - Vision rephrased: the role's single spec file is the unit of coverage; (role, included-service) pair MUST be exercised by ≥1 gated step inside it. - Persona scenarios are mandatory tree-wide (Rule 3); web-svc-* and the listed auth-less web-app-* roles MAY collapse to a single baseline. - New "Closure procedure" section with: Required reading, Autonomy (no questions, no SERVICES_DISABLED, ignore unrelated failures), Closure vocabulary (flag-closure vs role-closure), Per-role flow (now incl. meta/variants.yml adjustment as step 5), Pattern transfer (post-deploy edits to all later roles, single deploy per role), Preserving existing tests (purely additive, rename-only-when-flow-matches), Iteration order (dependency-aware within tier; explicit auth-less list), Resumability (drift matrix IS the progress marker), Commits (single commit at end). - Per-service catalogue dropped as a duplicate; pointed at the page that owns it. - Stale test-path references updated to reflect the `tests/lint/ansible/roles/web-app/playwright/` move. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
256e027525 |
docs: drop stale requirement refs + reflow soft-wraps to one sentence per line
Removes 67 live references to docs/requirements/<NNN>-*.md files across
43 contributor docs, role READMEs, code comments, task files, and tests.
Requirement docs are initial-creation-only artefacts; live references
freeze a snapshot that rots once the per-feature SPOT (a contributor
doc, a code module, a lint test) takes over the live behaviour.
In the same pass, fixes documentation.md violations on the touched
files: drops em-dashes used as thought breaks, removes "SPOT for ..."
self-labels, strips historical phrasing ("the legacy", "previously",
"used to", "is no longer used", "matching the historical"), and
reflows soft-wrapped sentences so each sentence stays on one logical
line.
The `test_inventory_vars_file_spot.py` whitelist for the literal
inventory path drops the docs/requirements/003 entry and instead
skips docs/requirements/ entirely; future requirement docs that
mention the literal path are then ignored automatically.
Verified by tests/lint/repository/test_no_stale_requirement_refs.py
(green tree-wide).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
79409ebf10 |
test(tooling): integrate markdownlint-cli2, ansible-lint, galaxy-importer + fix the resulting baseline
Tooling integration
* add .markdownlint-cli2.jsonc, .ansible-lint, .ansible-lint-ignore
configs (permissive day-1: cosmetic rules disabled, semantic ones on)
* extend scripts/install/lint.sh with generic install_pip_pkg helper +
ensure_ansible_lint / ensure_galaxy_importer / ensure_markdownlint_cli2
ensure_* functions (pip / npm with sudo / --user / --break-system-packages
fallbacks). install_ansible_tools now also pulls ansible-lint and
galaxy-importer; new `markdown` install group.
* extend scripts/lint/ansible.sh to run ansible-lint after the existing
syntax-check.
* add scripts/lint/markdown.sh and scripts/lint/galaxy.sh runners. The
galaxy runner parallelizes via xargs (-P=$(nproc)), prints OK/FAIL per
role, and treats `bare-name dependency` errors as success since our
roles/ tree is not a Galaxy collection layout. All other Galaxy schema
errors still fail.
* Makefile: new lint-markdown and lint-galaxy targets, both wired into
`make lint`.
* CI: new workflows lint-markdown.yml and lint-galaxy.yml; lint-ansible.yml
now installs ansible-core + ansible-lint; ci-orchestrator wires both
new lint workflows into lint-gate.
Galaxy schema fixes
* shorten the canonical `galaxy_info.company` block from three lines to
two ("Kevin Veen-Birkenbach\nhttps://www.veen.world", 45 chars) so it
stays under Galaxy's hard 50-char limit. Updated everywhere: 229 role
meta/main.yml plus _CANONICAL_COMPANY in
tests/lint/repository/test_role_meta_main_galaxy_schema.py.
* shorten 19 over-long role descriptions to <=255 chars.
* add stub READMEs for sys-svc-compose-ca and sys-utils-service-loader.
Markdown baseline fixes
* markdownlint-cli2 --fix run repo-wide (formatting normalization
across CHANGELOG, role READMEs, docs).
* manual MD041 fixes: prepend H1 in 15 files (CHANGELOG, LICENSE,
TODO.md helpers, role README templates, etc.).
* MD025 / MD003 / MD001 / MD032 manual fixes for heading-style and
list-spacing edge cases.
* role README content polish: "## Single Point of Truth (SPOT)" headings
renamed to "## User data resolution" (D3 self-label rule).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
a26bfc4e94 |
refactor(cli): reorg admin/contributing tree + per-folder help system
Module moves (with imports, paths, argv literals, scripts, docs, and
parents[N] formulas updated in lockstep):
- cli/deploy -> cli/administration/deploy
- cli/validate/inventory -> cli/administration/inventory/validate
- cli/vault -> cli/administration/secrets/vault
- cli/administration/inventory/encrypt -> cli/administration/secrets/seal
- cli/mirror -> cli/contributing/mirror
The new cli/administration/secrets cluster collocates the whole-file
ansible-vault wrapper next to the inventory-aware per-field sealer.
The user-facing CLI subcommand paths follow the package layout, e.g.
"infinito administration deploy dedicated <inv>" and
"infinito administration inventory validate <inv>".
Help system redesign in cli/core/{discovery,help,app}.py:
- Discovery now distinguishes runnable commands (directories with
__main__.py) from category folders (directories that only group
further commands). Empty support packages are filtered out.
- Each category folder MAY carry a README.md; its H1 heading is used
as the title and its first paragraph as the one-line description in
the help output.
- "infinito" / "infinito --help" prints only the top-level entries
(one level deep). "infinito <category>" navigates into a folder and
shows its sub-folders and commands one level deep, with the README
title as the page heading and the invocation as a dim subtitle.
- New "--tree [N]" flag walks the tree under the (optional) path
prefix recursively. N caps the depth (1-based); omit N for an
unbounded walk. The output uses Unicode tree connectors, an emoji
per entry (folder for categories, gear for commands), a globally
aligned description column, and multi-line wrapped descriptions
with tree-continuity bars on continuation lines.
- Category and command listings use the full invocation
("infinito meta callorder") rather than the bare leaf name, so the
output is copy-pasteable.
Convention doc + 17 CLI READMEs:
- New docs/contributing/artefact/files/cli/readme_md.md describes the
H1+paragraph layout the help system reads, and how it differs from
role/docs READMEs.
- README.md added for cli/{administration, build, contributing, meta}
and their immediate sub-folders, plus cli/meta/roles/applications
and its resolution sub-folder. Each follows the "title with trailing
emoji + single descriptive paragraph" convention.
Mirror resolver/sync description strings (the help system reads them
from argparse --help output) added so they no longer surface as
"options:" in the global index. cli/mirror/cleanup already had a
description and is unchanged.
Fixes uncovered by the move:
- import unittest.mock added to four dedicated-deploy tests so
unittest.mock.patch resolves under stdlib unittest discovery
(Python 3.13 no longer surfaces unittest.mock through "import
unittest" alone).
- parents[N] index in the new
cli/administration/inventory/validate/__init__.py corrected from
3 to 4 to match the deeper layout.
- test_paths.INVENTORY_VALIDATOR_PATH assertion and test_runner
substring assertions updated to match the new validator path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
a080dc4f98 |
refactor(cli): inventory tools under administration/, drop role scaffolder + dead SSO indirection
Move:
- cli/build/inventory/full -> cli/administration/inventory/devices
- cli/create/inventory -> cli/administration/inventory/provision
- cli/create/credentials -> cli/administration/inventory/credentials
- cli/encrypt/inventory -> cli/administration/inventory/encrypt
Delete:
- cli/create/role + templates/roles/ (stale scaffolder)
- roles/web-app-baserow/templates/javascript.js.j2 + dead sso_warning snippet
- js_application_name across akaunting/baserow/opencloud/opentalk/taiga,
javascript service disabled for baserow + taiga
Wire-through:
- argv lists in cli/deploy/{container,development}/* + 2 shell scripts
now invoke 'infinito administration inventory provision'
- subprocess module names + docs (contributing/* + administration/deploy.md)
- discovery resolver test + folder-item-limit whitelist updated
- ruff fixes in devices/command.py (RET504 inline + Path.open over open)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
7d7301614d |
docs(changelog): per-release archive CLI + release docs + SPOT cleanup
Add cli/contributing/changelog/archive: a CLI that trims CHANGELOG.md
to the last N entries (default 7) and writes every older release to
its own file under docs/changelog/ named <padded-semver>-<date>.md
(7.0.0 → 007.000.000-2026-05-08.md). The padded form makes a directory
listing already chronological, and the index in CHANGELOG.md is
rebuilt from the archive directory listing on every run, so it never
drifts away from what is on disk. The tool is idempotent and never
overwrites an existing archive file.
Run the migration once to split the 44-entry CHANGELOG.md into 7
inline entries plus 37 archived per-release files, with the index at
the bottom of CHANGELOG.md linking each archive in descending order.
Add docs/contributing/actions/release.md as the contributor-facing
release procedure (versioning, active window vs. archive, cutting a
release end-to-end, CI hand-off on tag push) and add a docs README
under docs/changelog/ that explains the archive directory layout and
filename schema. Cross-link release.md from pipeline.md and add
"releases" to the actions/ scope description.
Documentation cleanup pass on every page touched in the previous
mapping refactor:
- Remove "is the SPOT for X" / "Single Point of Truth" self-labels
across docs/contributing/, docs/agents/, and docs/changelog/. The
rule in documentation.md forbids self-declared SPOT status; status
is conferred by readers cross-linking, not by the page asserting
it. Rephrase to descriptive phrasing ("This page covers X",
"describes X", "lists X") throughout.
- Drop the orphaned docs/contributing/tools/actions.md duplicate
(101 lines paralleling tools/github/actions/workflows.md, with no
inbound links).
- Fix em-dashes used as thought breaks in dockerfile.md, origin.md,
agents/style.css.md, and suppression.md per documentation.md.
- Unwrap soft-wrapped sentences in iam/common.md, iam/ldap.md,
iam/oidc.md, image/origin.md, requirements.md, and dockerfile.md
so each logical line carries one full sentence.
Add a unit test for the archive CLI under
tests/unit/cli/contributing/changelog/archive/test_main.py covering
_pad_version, _split_into_entries, idempotency, dry-run, no-overwrite
of existing archives, and the multi-run index rebuild.
Quote `# noqa:` in CHANGELOG.md prose with backticks so the
noqa-only-ruff-codes lint does not parse "for" as a non-ruff code.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
286408fa91 |
refactor(roles): role-mapping SPOT under utils/roles + cli inspector + generalist lint
Move utils/roles/applications/files.py to utils/roles/mapping.py and add a data-driven role-type detector in utils/roles/type.py. Each entry in ROLE_FILES carries per-role-type allowed/mandatory flags plus marker entries that drive get_role_types(); a role MAY belong to several types (e.g. an application that also ships a system service). Add cli/meta/roles/mapping with files / types / markers subcommands so contributors can query the schema in text or JSON. Move cli/meta/applications/ to cli/meta/roles/applications/ and adjust PROJECT_ROOT parents indices accordingly. Move docs/contributing/design/services/ to docs/contributing/design/role/services/ and document the mapping in docs/contributing/design/role/mapping.md per documentation.md. Replace the narrow test_app_only_files with a generalist tests/integration/roles/meta/test_mapping.py that walks every role and every ROLE_FILES entry and reports forbidden files, missing mandatory files, forbidden entries, and missing mandatory entries in one structured fail report. meta/info.yml and meta/users.yml are now allowed across every role type. Create the seven mandatory files surfaced by the generalist test (README.md for desk-keepassxc, desk-obs, web-svc-cdn following readme_md.md, plus tasks/main.yml stubs for svc-db-memcached, svc-db-redis, sys-svc-proxy, web-app-socialhome). Audit and rewrite touched markdown for documentation.md compliance: replace em-dashes with fluent sentences, drop SPOT self-labels, fix all broken relative links introduced by the services/ move. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
9524c50928 |
feat(ci): diff-driven app whitelist for deploy tests
Narrow the test-deploy-{server,universal,workstation} matrix to roles
actually impacted by the branch's diff against origin/main, expanded
transitively over run_after + dependencies + shared services.
Why: full-matrix deploy CI on every PR is wasteful; explicit per-PR
whitelisting is error-prone. The resolver derives the affected set
automatically while remaining fail-safe.
Behaviour (in resolve_effective_whitelist.sh precedence order):
- whitelist input "__ALL__" -> empty whitelist (force full deploy)
- whitelist input non-empty -> verbatim
- whitelist input empty -> diff vs origin/main
- empty diff or any path outside roles/<role>/ -> __ALL__
- all paths under roles/<role>/ -> reverse closure
Fail-safe: any seed that is not modellable in CombinedResolver
(no application_id and not referenced by any role's run_after) and
any resolver runtime error MUST trigger a fall-back to __ALL__.
The closure never silently shrinks the deploy matrix.
test-deploy-local.yml is intentionally exempt (local/manual dispatch
keeps explicit whitelist semantics).
Documented in pipeline.md and workflows.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
eb45f0ffaf |
feat(cache): inner-build override + TTL env-var + SPOT docs
Extend the local cache stack with three independent capabilities: 1. Inner-`dockerd` build coverage for HTTP-only package mirrors. `roles/sys-svc-compose/files/compose.py` (the runner's `compose` wrapper) auto-generates a transient `compose.cache.override.yml` per app when `INFINITO_PACKAGE_CACHE_FRONTEND_IP` is exported. The override pins `build.extra_hosts` for `deb.debian.org`, `archive.ubuntu.com`, and `security.ubuntu.com` to the frontend. `pull.py` now delegates to the wrapper so build/pull operations see the same `-f` set the wrapper auto-detects. nginx in the frontend gains a port-80 listener with path rewrites for those hostnames, and `package.sh` registers two new Nexus proxies (`apt-debian-security`, `apt-ubuntu-security`) for the security suites whose upstream URL prefix differs from the main archive. Alpine is intentionally excluded from the inner-build extra_hosts: 3.18+ defaults to HTTPS, and inner-`dockerd` build containers do not import the frontend CA. Routing apk through the frontend would trip TLS verification. 2. General cache freshness env-var. `scripts/meta/env/cache/package.sh` exports `INFINITO_PACKAGE_CACHE_MAX_AGE_MIN` (default `129600`, i.e. 90 days). The bootstrap helper consumes it for every Nexus proxy's `contentMaxAge`, `metadataMaxAge`, and `negativeCache.timeToLive`, replacing the previous 1440-minute literal. Lower the value when chasing fast-moving upstreams. 3. SPOT cache documentation at `docs/contributing/environment/cache.md`. `compose.yml.md`, `variables.md`, the per-service READMEs, and `requirements/012-...` now link to it instead of restating the stack's behaviour. Net documentation lines drop substantially while the cache page becomes the single user-facing reference. Also: code-comment cleanup per `docs/contributing/documentation.md` guidelines across all touched .py / .sh / .yml files. Multi-paragraph docstrings, banner separators, and restate-the-obvious comments are removed; one-line WHY comments remain where the rationale is not already visible in the code. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
334c915598 |
feat(cache): transparent TLS frontend + override-only gating
Add a TLS-terminating reverse proxy in front of the Sonatype Nexus 3 OSS
pull-through cache so the `infinito` runner can hit upstream
package-manager URLs (pypi.org, registry.npmjs.org, rubygems.org,
repo.packagist.org, proxy.golang.org, dl-cdn.alpinelinux.org,
files.pythonhosted.org, raw.githubusercontent.com, codeload.github.com)
unchanged and still flow through Nexus.
Mechanism:
* scripts/docker/cache/package-frontend-certs.sh — host-side helper,
idempotently generates a dedicated CA + per-hostname leaf certs.
* compose/package-cache-frontend/upstreams.conf — nginx server-blocks
that match SNI per hostname and reverse-proxy onto the matching
Nexus repo path.
* compose/cache.override.yml `infinito.extra_hosts` — DNS-hijack so
the runner's clients resolve each upstream onto the frontend's
static IP.
* scripts/docker/cache/package-frontend-ca.sh — runner-side helper,
drops the CA into the system trust store post-startup so apt/pip/
curl/etc. accept TLS against the hijacked hostnames.
* cli/deploy/development/compose.py — orchestrates cert gen before
compose-up, CA install after health.
Scope today: runner-only. App-service Dockerfile builds executed by
the inner dockerd are NOT routed through the frontend; that requires
build-time CA injection and is deferred.
Also extends scripts/docker/cache/package.sh with five additional
Nexus proxies registered for transparent use: gem-proxy, go-proxy,
yum-rocky, yum-fedora, raw-packagist, raw-alpine, raw-codeload-github.
Refactor: move all cache services + runner-side cache mounts/depends_on/
extra_hosts from compose.yml into compose/cache.override.yml. Base
compose.yml now describes only the bare CI/runtime stack (coredns +
infinito). The override file is the single gate for the cache stack;
`profiles: ["cache"]` attributes and `--profile cache` flag emission
were redundant once file inclusion controls existence and removed
accordingly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
170b16c59d |
docs(design): rename matrix-deploy.md to variants.md, add inventory.md, sync cross-links
Two design docs under docs/contributing/design/: - variants.md (renamed from matrix-deploy.md): explains the folder-per-round wrapper that consumes per-role meta/variants.yml and iterates variants at deploy time. Headings normalised to sentence-case; the "legacy unsuffixed inventory path" wording rewritten to drop the historical-context phrasing forbidden by documentation.md. - inventory.md (new): explains how a single round's inventory is assembled out of meta/services.yml plus the variant payloads, which inclusion mechanisms exist (operator --apps, run_after walk, shared-service auto-include via strict `is True`), what lives in the inventory file vs. host_vars/<host>.yml, and when Jinja actually gets rendered (only at play runtime, via utils.cache.applications._render_with_templar). Worked example documents the WordPress + Discourse wiring as a reference case. Cross-links updated wherever they pointed at matrix-deploy.md or its concept name as a file: - docs/agents/action/iteration/role.md - docs/contributing/actions/deploy.md - docs/contributing/artefact/files/plugins/lookup/applications.md - docs/contributing/artefact/files/role/variants.md - scripts/tests/deploy/local/deploy/reuse-kept-app.sh (path comment) Plain-text occurrences of "matrix-deploy" as a concept term inside Python comments, log messages, and test docstrings stay as-is; only file-link references are migrated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
b34131ca1d |
feat(package-cache): add Sonatype Nexus 3 OSS pull-through cache (req-012)
Brings up the package-cache compose service alongside the existing
registry-cache, gated by the same `cache` profile. Bootstrap helper
(scripts/docker/cache/package.sh) auto-rotates the admin password from
the well-known default, accepts the Sonatype CE EULA (3.71+ blocks
artifact downloads with HTTP 403 until /v1/system/eula is POSTed —
disclaimer must be sent verbatim with its unicode quotes), enables
anonymous read, creates one default blobstore, and registers the MVP
proxy set: apt-debian, apt-ubuntu, pypi-proxy, npm-proxy, helm-bitnami,
raw-githubusercontent. The runner gets pip.conf / .npmrc /
sources.list snippets bind-mounted in (each defaulting to /dev/null
when the cache profile is inactive) so the deploy is byte-identical
with cache off.
Cache plumbing:
- scripts/meta/env/cache/{registry,package}.sh: per-cache env defaults,
hostname-derived admin password seed (uses `hostname` command, not
$HOSTNAME shell var, so the value is identical across interactive
shells, non-interactive subshells, and Python subprocesses).
- scripts/docker/cache/{registry-ca,package}.sh: container-side
bootstrap helpers, both moved under scripts/docker/cache/.
- scripts/system/cache/clean.sh + `make cache-clean`: wipes both
/var/cache/infinito/core/cache/{registry,package} dirs and stops
the cache containers.
- cli/deploy/development/compose.py: Profile gate exports the three
new INFINITO_PACKAGE_CACHE_* client-snippet paths and runs the
bootstrap helper after the package-cache reports healthy.
Compose service uses sonatype/nexus3:3.91.1-ubi pinned, runs as
root (UBI image refuses to chown a fresh bind-mount), and forces
NEXUS_SECURITY_RANDOMPASSWORD=false so the bootstrap can log in with
admin/admin123 before rotating; the random-password mode otherwise
auto-completes onboarding silently and leaves no readable secret on
disk for an automated bootstrap to pick up.
Tests: tests/unit/cli/deploy/development/test_compose.py grows two
new assertions (cache services listed in `up` args when local; build
tests pin CI=true to retain ci-only assertions).
Closes req-012.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
9af3055d80 |
feat(registry-cache): move env defaults into registry_cache.sh, mandate in compose.yml
The three INFINITO_REGISTRY_CACHE_* variables previously carried their
defaults in compose.yml as `${VAR:-...}`, which baked /tmp paths and
a static 32g cache size into the YAML. Move the defaults into a new
shell module that is sourced via `scripts/meta/env/all.sh` and tighten
compose.yml to require the variables strictly via `${VAR:?...}`.
Why
* Adapt cache size to the host. Hard-coding 32g hurts hosts where
`/tmp` is a small tmpfs (typical Linux: ~half of RAM); the cache
would either be capped above the partition or starve other tmpfs
users. Computing the default as half the free disk space at the
cache path leaves a usable budget for the rest of the system on
small hosts and grows on large ones.
* Single source of values. With defaults in shell, the compose YAML
no longer carries duplicated tunables; `compose.yml` just consumes
what the env layer provides. Same pattern as the existing
INFINITO_CONTAINER strict consumption.
Changes
* New scripts/meta/env/registry_cache.sh exports the three vars.
CACHE_MAX_SIZE = `df --output=avail -B1G "$path" / 2`, walking up to
the closest existing ancestor when the cache path itself does not
exist yet (it is created on first start by `bind:
create_host_path: true`). Minimum 1g, fallback 2g if `df` fails.
* scripts/meta/env/all.sh sources the new module so the Makefile-driven
flow picks it up automatically.
* compose.yml: three `${VAR:-default}` -> `${VAR:?Source scripts/meta/env/registry_cache.sh ...}`
so the absence of the env layer fails fast with a precise message.
* Stale block comment above the registry-cache service (CI-only,
/tmp-defaults, etc.) collapsed to a one-line pointer to the README,
per documentation.md "default to no comments / explain only the
non-obvious why".
* docs/contributing/environment/variables.md and
docs/contributing/artefact/files/compose.yml.md describe the new
defaults source and the half-free-disk computation.
Verified
`bash -c 'source scripts/meta/env/all.sh && docker compose --profile ci --profile cache config'`
resolves the three vars (CACHE_MAX_SIZE: 3g on a host with 7G free
in /tmp). Without sourcing, compose fails with the exact
"Source scripts/meta/env/registry_cache.sh ..." message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
94936f198b |
feat(compose): gate registry-cache to local dev only via Profile class
Move the registry-cache pull-through proxy out of the always-on `ci` profile into its own `cache` profile, and select it based on the runtime environment instead of statically. Why The proxy only earns its keep when image pulls amortize across many runs on the same host. That holds on developer machines (cross-run image dedup, persistent host disk) but not on GitHub-hosted runners (fresh disk per job), where the proxy startup overhead is net loss without the corresponding payoff. How * Compose: split `registry-cache` into its own `cache` profile; `infinito.depends_on.registry-cache` gets `required: false` so Compose does not auto-start the proxy via the dependency when the profile is inactive; the systemd drop-in bind-mount source is gated by the new `INFINITO_REGISTRY_CACHE_PROXY_CONF` env var with `/dev/null` default, so dockerd loads an empty drop-in (no `HTTP_PROXY`, no `ExecStartPre`) when the proxy is not running and pulls images directly. * New `cli/deploy/development/profile.py` `Profile` class is the SPOT for which compose profiles activate. KISS + SRP: `is_ci()` reads the three CI signals (`GITHUB_ACTIONS`, `RUNNING_ON_GITHUB`, `CI`), `registry_cache_active()` is its strict inverse, and `args()` returns the matching `--profile ...` flag list. * `Compose` and `down.py` consume the same `Profile` so up/down cannot drift on which services are part of the stack. * Documentation under `compose/registry-cache/README.md`, `docs/contributing/artefact/files/compose.yml.md` and `docs/contributing/environment/variables.md` describes the new toggle and the `INFINITO_REGISTRY_CACHE_PROXY_CONF` env var. Pre-existing documentation.md violations in those files (one SPOT self-label, one em dash, two path-as-link-text instances) corrected in the same pass per the fix-nearby rule. * Tests: 14 new `Profile` unit tests in `test_profile.py` plus two scenario tests in `test_compose.py` that lock down the `--profile ci` (CI runner) vs `--profile ci --profile cache` (local) argument shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
1dd8d3f5c0 |
feat(compose): add pull-through registry-cache + reorg compose/ subdirs
Introduces a `registry-cache` service (rpardini/docker-registry-proxy) in the `ci` profile that MITM-caches every outbound docker registry pull made by `infinito`'s inner dockerd, drastically cutting bandwidth on re-deploys, parallel CI shards, and local fresh-purges. Wiring (all bind-mounted via compose.yml, no Dockerfile or entry.sh staging, so edits hot-reload on next container start): - CA bundle at /opt/registry-cache-ca (read-only), - systemd drop-in at /etc/systemd/system/docker.service.d/registry-cache-proxy.conf, - registry-cache-ca.sh at /usr/local/bin/registry-cache-ca.sh. The drop-in registers registry-cache-ca.sh as ExecStartPre. Compose gates `infinito` on `registry-cache` `service_healthy` so the CA is on disk before dockerd boots; the install script supports both Debian (`update-ca-certificates`) and Arch/Fedora (`update-ca-trust extract`) trust stores, detected by command availability. Also reorganizes compose/ into per-service subdirectories (compose/coredns/, compose/registry-cache/) with README.md per directory following the documentation guidelines, adds scripts/docker/README.md, and nests the registry-cache host paths under /tmp/infinito/core/ (mirror/ for blobs, ca/ for the MITM CA bundle). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
ed04dad5d5 |
refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
3f2021b07d |
refactor: collapse INFINITO_DISTRO/INFINITO_CONTAINER to a single SPOT and align all consumers
Follow-up to d002796a5. The formula `infinito_nexus_${INFINITO_DISTRO}` is
now defined in exactly one place — `scripts/meta/env/defaults.sh:18`,
exported outside the load-once guard so it tracks INFINITO_DISTRO across
matrix iterations (caller re-sources defaults.sh after distro changes).
Every other site reads INFINITO_CONTAINER strictly from env; nothing
re-derives the formula, no lazy/colon-dash fallbacks, no per-call defaults.
Production code aligned:
- cli/deploy/development/common.py: add resolve_container() (strict env
reader, raises SystemExit on unset). make_compose() calls it as an
early-fail guard so a missing defaults.sh source surfaces here instead
of as a confusing compose.yml interpolation error later.
- cli/deploy/development/deploy.py: stop re-deriving from compose.distro;
use resolve_container().
- compose.yml: container_name reads ${INFINITO_CONTAINER:?...} strictly,
so callers that bypass defaults.sh fail fast with a clear hint.
- scripts/meta/resolve/image/local.sh: drop ${INFINITO_DISTRO:-debian}
fallback; require INFINITO_DISTRO via :? error.
- scripts/tests/deploy/ci/dedicated.sh: drop local _container fallback.
- scripts/tests/deploy/local/deploy/fresh-kept-app.sh: drop redundant
${INFINITO_DISTRO:-debian} reassignment after sourcing env/all.sh
(defaults.sh already defaults the value).
- scripts/tests/deploy/local/deploy/fresh-purged-app.sh: drop
target_container fallback; INFINITO_CONTAINER is already in env.
- scripts/tests/deploy/local/exec/container.sh: drop the lazy
${INFINITO_CONTAINER:=infinito_nexus_...} default; defaults.sh source
in env/all.sh already guarantees the value.
- scripts/tests/dns.sh: source defaults.sh so callers only need to set
INFINITO_DISTRO (the implicit dependency was previously a footgun).
- scripts/system/tls/trust/{wsl2,linux}.sh: same — source defaults.sh,
drop the ${INFINITO_CONTAINER:-infinito_nexus_debian} fallback in
linux.sh.
- .github/workflows/test-dns.yml: drop redundant
INFINITO_CONTAINER: infinito_nexus_${{ matrix.distro }} from per-step
env blocks (3 sites). The env now flows from INFINITO_DISTRO via the
defaults.sh sourced inside the scripts.
- tests/unit/cli/deploy/development/test_matrix_deploy.py: replace the
module-level os.environ.setdefault('INFINITO_CONTAINER', 'infinito_nexus_arch')
shim — which itself encoded the formula — with a module-level patch of
resolve_container that returns an opaque test-fixture string. Drop the
now-unused distro="arch" field from the _args() Namespace fixture
(deploy.handler stopped consuming it after the --distro arg removal).
Documentation aligned:
- docs/contributing/environment/variables.md: rewrite the INFINITO_CONTAINER
row to spell out the SPOT contract — set automatically by defaults.sh,
never re-derived elsewhere, callers re-source defaults.sh after
changing INFINITO_DISTRO mid-process.
- docs/contributing/artefact/files/compose.yml.md: replace the
INFINITO_DISTRO row (which described the now-removed
`infinito_nexus_${INFINITO_DISTRO:-debian}` interpolation) with an
INFINITO_CONTAINER row noting the strict ${INFINITO_CONTAINER:?...} read.
Verified:
- Formula derivation: 1 production hit (the SPOT in defaults.sh:18).
- INFINITO_CONTAINER fallback patterns (:-/:=): 0.
- --distro CLI arg refs: 0 (one comment explaining the removal).
- make test green: 1553 unit + 168 integration tests, no skipped suites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
9c35909752 |
refactor(utils/cache): split data.py into base/applications/users/domains
The 934-line `utils/cache/data.py` monolith mixed cross-cutting helpers
(constants, yaml-loaders, deep-merge, signatures, render machinery)
with three independent domain caches (applications, users, domains).
Per the KISS/SPOT/DRY guidelines, split it along domain boundaries so
each file has a single, well-scoped responsibility:
utils/cache/__init__.py — package; owns _reset_cache_for_tests()
orchestrator
utils/cache/base.py — cross-cutting helpers: constants,
yaml-loaders, _deep_merge, cache-key +
content-fingerprint signatures, the
cross-domain re-entry guard, and the
templar-render machinery (the only
ansible-coupled symbol, lazy-imports
its dependency)
utils/cache/applications.py — variants + defaults +
get_merged_applications. Strictly
ansible-free at import time so the
GitHub Actions runner-host CLI path
(cli.deploy.development.init →
plan_dev_inventory_matrix →
get_variants) keeps working without
ansible installed
utils/cache/users.py — user defs, token store hydration,
alias materialization,
get_user_defaults, get_merged_users
utils/cache/domains.py — get_merged_domains, derived from the
merged applications view
utils/cache/data.py — DELETED. Façade is gone; the package
exports nothing the domain modules
don't already
Backward-compat: the facade was deleted (per operator request) and all
36+ consumers — 18 plugins, 2 CLI entry points, 9 test files, 4 doc
files, plus a role-internal lookup_plugin — were rewired to import
directly from the owning domain module. `_reset_cache_for_tests` is
now exposed at `from utils.cache import _reset_cache_for_tests`.
Tests added (per-module, focused):
tests/unit/utils/cache/test_base.py (31 cases)
tests/unit/utils/cache/test_applications.py ( 9 cases)
tests/unit/utils/cache/test_users_module.py (21 cases; suffix
disambiguates from
the existing
tests/unit/plugins/
lookup/test_users.py)
tests/unit/utils/cache/test_domains.py ( 6 cases)
The ansible-block invariants from CI runs 24934007615 and 24935979190
are now pinned via subprocess isolation: each ansible-less import test
spawns a fresh `python3 -c '…'` with a meta_path filter denying every
`ansible*` import and asserts the import + invocation succeed in that
clean process. The earlier in-process sys.modules-manipulation shims
fragmented module identity and broke under Python 3.13 inside the
infinito container; subprocess isolation has neither problem.
Doc-style audit (per docs/contributing/documentation.md):
- Removed em-dash from utils/cache/README.md (split into two
sentences)
- Updated stale `utils.runtime_data` reference in
docs/contributing/design/matrix-deploy.md
- Removed two SPOT self-declarations
(docs/agents/files/group_vars/applications.md and
docs/contributing/artefact/files/plugins/lookup/users.md) per
the documentation rule that SPOT status is conferred by
cross-links, not by the page asserting it about itself
Empirical proof:
- All four domain modules import without ansible (subprocess
isolation tests)
- get_variants(roles_dir=ROLES_DIR) returns 99 apps in the
ansible-blocked subprocess
- 142 cache tests + 1085 wider-suite tests pass; the only
remaining failures (test_host_vars free-function fixture bug,
test_runtime_performance timing flake) are pre-existing and
unrelated to this refactor (verified via `git stash`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbach
|
12dba01156 |
refactor(utils): consolidate cache modules into utils/cache/ package
Both in-process cache layers — the cached YAML loader and the cached runtime-data getters — now live next to each other under `utils/cache/`. The package gains a README that documents the "orthogonal layers, no third cache" rule so future contributors hit the right primitive on the first look. Moves: - `utils/yaml_cache.py` -> `utils/cache/yaml.py` - `utils/runtime_data.py` -> `utils/cache/data.py` - `tests/unit/utils/test_yaml_cache.py` -> `tests/unit/utils/cache/test_yaml.py` - `tests/unit/utils/test_runtime_data.py` -> `tests/unit/utils/cache/test_data.py` - `tests/unit/utils/test_variants.py` -> `tests/unit/utils/cache/test_variants.py` New: - `utils/cache/__init__.py` (empty per project convention) - `utils/cache/README.md` describing the two layers, when to reach for which, lifetime, test-only helpers, and design guidelines (RFC 2119 keywords, emoji-after-headline, no self-SPOT declaration, no slash-in-link-text). - `tests/unit/utils/cache/__init__.py` Mechanical updates: - All imports across cli/, plugins/lookup/, tests/, and roles/ rewritten from `utils.yaml_cache` -> `utils.cache.yaml` and `utils.runtime_data` -> `utils.cache.data` (40+ files). - `REPO_ROOT` parents-index in the moved tests bumped from `parents[3]` to `parents[4]` to reflect the deeper nesting. - Six broken markdown links in docs/ updated to the new module paths (test_markdown_links lint test stays green). - Two test files that used the `from utils import runtime_data` form now use `from utils.cache import data as runtime_data` so the local alias remains. Tests: 848 passed across unit + lint suites (1 pre-existing collection error in test_host_vars.py unrelated to this change). |