SPOT: declare the var-lib base once in default.env as INFINITO_DIR_VAR_LIB; the env build derives INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR from it; reference it everywhere (utils/paths.py, scripts via ${...:?}, tests, the dev inventory's DIR_VAR_LIB injection) instead of hardcoding the literal. group_vars keeps the literal default; the env-lookup lives in the dev inventory per the no-ansible-env-lookups lint rule. pull_specific_host.py --folder is now required and the role passes {{ DIR_BACKUPS }}.
Swarm e2e: collect_playwright_reports.sh docker-cp's the reports off the manager plus a workflow upload step, so swarm Playwright traces reach CI artifacts (was compose-only).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add plugins/lookup/playwright_workers.py computing a conservative parallel-worker count (CPU/RAM-derived, app-under-test concurrency-capped, reduced on CI) and wire test_e2e_playwright_workers + test_e2e_playwright_fully_parallel + test_e2e_playwright_test_timeout into playwright.config.js (read from env) and 02_run_one.yml (passed via -e). fullyParallel stays false so parallelism is file-level only and intra-file ordering assumptions stay safe. Raise run_timeout to 3600 for addon-heavy roles (nextcloud ~50 specs).
Split the role's variables by Ansible precedence and intent: derived/fixed constants live in vars/main.yml (UPPERCASE, high precedence, not operator-tuned), operator-tunable knobs live in defaults/main.yml (lowercase, overridable). VARIANT_INDEX stays UPPERCASE with a single definition. Update references across tasks/templates/inventory/docs accordingly. Includes a unit test for playwright_workers and an autoformat pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolves CI swarm rc=69 ("msmtp: needs a password"): controller-side
Ansible lookups (lookup('email').password etc.) couldn't see the
tokens.yml that Mailu writes target-side, because the controller has
its own filesystem in swarm mode.
Mount NFS-shared /var/lib/infinito on every swarm host AND on the
controller during deploy (in-play delegate_to: localhost task at end
of svc-storage-nfs-server role). State subdir exported separately
with no_root_squash + wildcard CIDRs so root-owned writes succeed
and the controller can mount from outside the svc-swarm IP set.
Single-writer guarantee on FILE_TOKENS via delegate_to: STACK_HOST +
run_once on sys-token-store/{01_core,write}.yml. Backup roles
(svc-bkp-container-2-local, sys-ctl-cln-bkps, sys-ctl-cln-faild-bkps,
user-backup/03_permissions_folders, three direct callers of
04_seed-database-to-backup.yml) gated on IS_STACK_HOST to prevent
N-host write races on the shared NFS backup tree.
Fail-fast assertion in svc-storage-nfs-client + 05_controller_mount
when pre-existing local /var/lib/infinito would be hidden by mount-
over. Operator must migrate or wipe before re-deploying.
SPOTs: SWARM_LAB_NETWORK in workflow env block, storage.nfs.state_subdir
in group_vars/all/15_storage.yml, DIR_VAR_LIB unchanged. UPPER_CASE
constants in NFS roles. Cleanup script umounts controller view before
container teardown to prevent stale-mount leak between matrix jobs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The MODE_DEBUG-gated DinD DNS handler in sys-svc-container spawned a
busybox container with `--dns 1.1.1.1` and expected the mocked
infinito.example to resolve to 127.0.0.1, which Cloudflare obviously
cannot do. Run an in-cluster dnsmasq inside mgr-01 instead of patching
the symptom out:
- Stop and mask systemd-resolved so port 53 is free.
- Bootstrap /etc/resolv.conf with 1.1.1.1 so apt-get can fetch dnsmasq.
- Install dnsmasq with bind-dynamic, no-resolv, address=/${INFINITO_DOMAIN}/127.0.0.1
and 1.1.1.1/8.8.8.8 as upstream forwarders; bind-dynamic catches docker0
when the Ansible-installed dockerd creates it later.
- Switch /etc/resolv.conf to 127.0.0.1 and verify the chain via
`getent ahosts ${INFINITO_DOMAIN}` so the script fails fast on misconfig.
- Pin mgr-01's docker0 to 172.17.0.1/16 via daemon.json `bip`, then point
the swarm.yml networks.internet.dns at that gateway so DinD-spawned
containers reach dnsmasq via their default route.
Also re-stage three pure-formatting tightenings ansible-lint made to
existing `template: # nocheck` lines (double space -> single space).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI failures from run 26987751848:
* `cache-read` (3 hits): annotate the `Path.read_text()` calls in
utils/tests/swarm/{write_extras,extend_inventory}.py — every one
reads a file the same script just wrote, so the cache helper would
serve stale content. Per `# nocheck: cache-read` carve-out.
* `project-root-import`: annotate the `parents[6]` in the swarm
closure test loader. The file-path loader bypasses the package
__init__ to avoid pulling in the heavy dedicated CLI imports.
* No-unittest-tests: tests in test_closure.py inherited only via
`_BaseClosureCase` (which itself extends `unittest.TestCase`), but
the AST lint walks bases literally and does not follow indirect
inheritance. Switch each `Test*` class to multi-inherit
`(_BaseClosureCase, unittest.TestCase)` so the discovery check
recognises them as runnable.
* `hardcoded-dns-resolver`: the per-block head suppression in
inventories/development/swarm.yml was insufficient; the lint scans
per-line and needs the marker on the actual `dns: "1.1.1.1"` line.
Verified locally: 16 of the previously-failing tests now pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`-e @<file>` replaces sibling dicts wholesale; the extras file only
spelled networks.internet.dns, so the loopback ip4/ip6 from
group_vars/all/08_networks.yml were dropped at deploy time and
templates dereferencing networks.internet.ip4 (e.g. web-app-bigbluebutton
env.j2 EXTERNAL_IPv4) failed with "object of type 'dict' has no
attribute 'ip4'".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New ``cli.administration.deploy.swarm`` package wraps the strict
``cli.administration.deploy.dedicated`` runner with a layered closure
so swarm deploys never require operators (or test scripts) to
enumerate every infra role:
effective --id = seed (operator --id, else inventory_role_groups)
∪ dep-walk(seed via applications_if_group_and_all_deps)
∪ default-placement safety net
The dep-walker is the same SPOT
(``utils.roles.applications.in_group_deps.applications_if_group_and_all_deps``)
the runtime constructor consumes via ``lookup('applications_current_play')``,
so the CLI-side closure and the in-play role set stay in sync.
The dedicated CLI keeps strict explicit-only semantics: ``main`` was
split into ``parse_args`` + ``run_from_args`` so the swarm wrapper
can share parser, mode wiring, validator and runner verbatim and
only mutate ``args.id``.
Meta-side: ``svc-db-postgres`` gains ``default_placement: manager``
(semantic symmetry with mariadb). ``default_placement`` is now a
PLACEMENT hint only ("if deployed, run on the manager"); deploy
decisions come from the dep-walk. Apps that declare ``mariadb:``
pull svc-db-mariadb; apps that declare ``postgres:`` pull
svc-db-postgres; apps that declare neither get no central DB.
Shared utils extracted so the four consumers stay aligned:
* ``utils.inventory.groups.inventory_has_group``: YAML + INI static
inventory group probe (extracted from ValidDeployId).
* ``utils.roles.meta_lookup.{get_role_default_placement,
iter_roles_with_default_placement}``: single source of truth for
``default_placement`` discovery (lookup plugin is now a thin wrapper).
Test pipeline (``scripts/tests/deploy/swarm/*``):
* Heredoc-Python shells converted to proper modules under
``utils.tests.swarm.*`` and invoked via ``python -m``:
- extend_inventory.py (was 08_extend_inventory.sh)
- write_extras.py (was 09_write_extras.sh dynamic half)
- derive_includes.py (replaces 06's hardcoded includes array)
- write_hosts_entries.py (replaces 02's hardcoded /etc/hosts heredoc;
derives every canonical + www-redirect domain from APP_ID via
canonical_domains_map + generate_all_domains, so the test
works for any app in the matrix without per-app maintenance)
* 01_create_network.sh deleted (single docker command inlined in workflow).
* 11_deploy.sh deleted (swarm CLI now called directly in workflow).
* 09_write_extras.sh: static half moved to inventories/development/swarm.yml.
* Remaining shell scripts renumbered 01-13 without gaps.
* 08_wait_converge.sh: case block on DB_DEP replaced with single
conditional (container name == DB_DEP).
* 06_provision_inventory.sh: hardcoded includes array replaced with
`mapfile -t includes < <(python -m utils.tests.swarm.derive_includes)`.
* Workflow YAML step references updated to new script names + module
invocations.
New tests (39 covering the closure surface):
* tests/unit/cli/administration/deploy/swarm/test_closure.py (14)
* tests/unit/utils/inventory/test_groups.py (11)
* tests/unit/utils/roles/test_meta_lookup.py (+3 for default_placement)
* tests/unit/utils/roles/validation/test_deploy_id.py (unchanged
8 tests still pass after refactor onto the shared util).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of step-08 deploy failures across all 4 test-development
distros in run 26572559126: `make compose-deploy mode=reinstall
apps=web-app-dashboard INFINITO_SERVICES_DISABLED=matomo` (step 07)
made make export the cmdline var into the recipe's shell env, which
reached `runner.py` → `make setup` → `make dotenv`. The dotenv
generator's `setdefault` then baked the runtime-only value into the
on-disk `.env`. Step 08 sourced that .env, filtered `web-app-matomo`
out of its primary apps, and exited with "All primary apps disabled".
Fix:
- Drop the 7 routing knobs (apps, mode, purge, bundles, disable,
full_cycle, variant) from `passthrough.STATIC_KEYS` and from
`default.env`. They are per-invocation; .env is the persistent SPOT
and must not capture them.
- Switch every consumer from `os.environ["KEY"]` strict access to
`os.environ.get("KEY", "")` so the missing-key path stops at "no
filter" instead of a KeyError.
- Rename the env-var surface to match the make-alias surface 1:1
(lowercase): INFINITO_APPS → apps, INFINITO_SERVICES_DISABLED →
disable, INFINITO_BUNDLES → bundles, INFINITO_FULL_CYCLE →
full_cycle, INFINITO_VARIANT → variant, INFINITO_DEPLOY_MODE → mode,
INFINITO_PURGE_ENTITIES → purge. The make alias was the user-facing
name in every doc; routing-knob env vars now use the same name so
the cmdline form, the recipe env-prefix, and the consumer reads
all line up.
- Collapse the Makefile `compose-deploy` recipe to just the two
knobs whose alias differs from the env name (`type` →
INFINITO_DEPLOY_TYPE, `debug` → INFINITO_DEBUG). GNU Make
auto-exports cmdline variables into the recipe's shell env, so the
identical-name `$(if $(apps),apps="$(apps)")` lines were pure
noise.
- Drop `disable: ${disable}` from compose.yml. The dev CLI builds
`extra_env` per `docker exec` call and forwards `disable` via
`-e`, so the container env never needs the value frozen at
compose-up time.
- Update the 3 actual leak call sites
(07/08/09_deploy_*.sh) to the make-alias form
(`disable=matomo`, `full_cycle=true`, `variant=1`).
- Add `SIM112` to ruff's ignore set: the lowercase env names are
the deliberate make/env parity, not a typo.
Repro before fix: `INFINITO_SERVICES_DISABLED=matomo make dotenv`
wrote `INFINITO_SERVICES_DISABLED=matomo` to `.env`. After fix:
`disable=matomo make dotenv` leaves `.env` untouched.
Tests: `make test` and `make autoformat` green (2711 tests, 8 lint
sub-targets).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move the deploy router into the compose- domain prefix to match the
rest of the local stack lifecycle (compose-up/down/exec/...). Reorder
its block alphabetically between compose-down and compose-entity-purge
to keep `make help` and the alphabetical-targets lint happy.
Update every active call site: contributor docs, agent-iteration docs,
role-level Administration/README/TODO files, the bundles README, the
test-environment scripts, the in-repo deploy helpers and main.sh, and
the server stage task. The two wildcard prose references (`make
deploy-*` in CLAUDE.md and testing/playwright.md) become single
`make compose-deploy` mentions since no other deploy-* target exists.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the seven separate deploy-* make targets with a single `make
deploy` recipe that maps short Make variables to the canonical
INFINITO_* environment variables and dispatches via the new
scripts/tests/deploy/local/deploy/main.sh router. main.sh and all
sub-scripts continue to read the long-form names, so Ansible passthrough
and container forwarding stay unchanged.
Short Make CLI now exposes apps, mode, purge, type, bundles, disabled,
and full_cycle as aliases. The router picks the verb folder
(initialize, reinstall, update) and scope (all, selection) based on the
chosen mode plus the presence of INFINITO_APPS or INFINITO_BUNDLES.
Also folds in the global INFINITO_TEST_DEPLOY_TYPE -> INFINITO_DEPLOY_TYPE
rename across default.env, passthrough handler, dotenv test, CI deploy
workflows, discovery scripts, and the server stage task.
Active contributor and agent-iteration docs, role-level Administration
and README files, the bundles README, and the deploy reset helper are
all retargeted to the new short CLI. docs/requirements/ entries stay
untouched as historical audit logs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
\`lookup('env', 'INFINITO_*')\` calls in group_vars / roles couple
deployment-agnostic Ansible code to default.env, which only exists in
dev/CI. Production deploys never source .env, so the lookups silently
fall back to their group_vars defaults and the inventory's intent gets
lost on the way (see the IP4 host_vars-overlay incident fixed in
8980b00b1).
Migrated keys (group_vars/roles -> inventory):
- DOMAIN_PRIMARY (00_general.yml)
- BUILDKIT_ENABLED (00_general.yml)
- CA_CONTAINER_ENABLED (02_tls.yml)
- defaults_networks.internet.{ip4,ip6,dns} (08_networks.yml)
- TEST_E2E_PLAYWRIGHT_KEEP (roles/test-e2e-playwright/defaults/main.yml)
group_vars and the role default now carry literal fallback values;
the dev/CI inventory at \`inventories/development/default.yml\` pulls
the live values from .env via straight \`lookup('env', 'INFINITO_*')\`
calls without any \`| default(...)\` filter, because load.sh guarantees
the keys are set whenever .env exists.
The convention is enforced by the new lint test
\`tests/lint/filesystem/env/test_no_ansible_env_lookups.py\`, which
scans group_vars/, tasks/, roles/, and playbook.yml for any
\`lookup('env', 'INFINITO_*')\` against a default.env key and fails
with a pointer to the dev inventory as the injection site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace 8 bash env modules with a single Python generator
(cli.meta.env) that materialises env/static.env + runtime context
into a gitignored .env. Each variable lives in its own handler
under utils/env/handlers/; defaults carry inline one-line `# ...`
comments.
Rename to the INFINITO_* schema:
- workflow inputs: APPS, SERVICES_DISABLED, CMD, DEBUG, WHITELIST,
BUNDLES, FULL_CYCLE, VARIANT, RUN_FLAGS, LIMIT_HOST, PARALLEL
- inventory: INVENTORY_DIR / INVENTORY_FILE / HOST_VARS_FILE,
PW_FILE -> INVENTORY_PASSWORD_FILE
- venv: VENV_BASE / VENV_FALLBACK / VENV_NAME
- new SPOT defaults: COMPILE_SILENCE, CONTAINER_NO_CA,
WAIT_HEALTH_TIMEOUT_S, PACKAGE_CACHE_FRONTEND_INIT_IMAGE,
promoted IMAGE_TAG from GHA-only to general passthrough
compose.yml hardening:
- inline \${VAR:-default} removed; strict \${VAR:?...} everywhere
- new integration lint blocks regressions
Tooling:
- scripts/install/all.sh owns the stamp-file install logic
(Makefile target delegates)
- scripts/meta/env/load.sh is the single shell entry point;
8 bash modules deleted
- new shell lint: no INFINITO_* defaults outside env/static.env
Tests:
- 93 new unit tests for utils/env/{builder,parser,runtime,writer}
- restructure tests/lint into repository/ (file-content rules) vs
filesystem/{shell,python,env}/ (filesystem-shape rules)
- drift + smoke integration tests against the generator
Docs:
- variables.md is now a pointer; per-variable docs live next to
the key in env/static.env / env/ci.env
- compose.yml.md drops the Default column
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- new utils.inventory.bundle_apps resolver aggregates and dedupes role
groups from inventories/bundles/{servers,workstations}/<name>/inventory.yml
- new make targets deploy-bundles (fresh-purged) and redeploy-bundles
(reuse-kept), driven by BUNDLES=... env, wrapped by
scripts/tests/deploy/local/deploy/{bundles,redeploy-bundles}.sh
- unit tests for the resolver under tests/unit/utils/inventory/
- document the new targets in docs/contributing/tools/make.md,
docs/contributing/actions/testing.md, inventories/bundles/README.md
- collapse multi-line Makefile comments to one-liners for consistency
- disable nextcloud bbb plugin auto-enable (commented out)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause: upstream friendica-addons/ldapauth's `ldap_createaccount`
returns the raw `User::create()` result (`['user' => UserRow, 'password' => '']`),
but the `authenticate` hook only sets `authenticated = 1` when
`$user['uid']` is non-empty — never true for the autocreate path. First-
login LDAP users (e.g. biber) were inserted into `friendica.user` but
the session was never started, so /network kept serving the login form.
Encoded as an idempotent in-container patch task in `tasks/02_ldap.yml`
+ `files/patch_ldapauth_autocreate.sh` (reports PATCHED on first run,
ALREADY on subsequent runs). Runs whenever LDAP is wired, gated by
FRIENDICA_LDAP_ENABLED.
Other changes in the same stack:
- meta/services.yml: drop keycloakpassword addon (oauth2-proxy in front
owns SSO); ldap.enabled now follows svc-db-openldap group presence so
the variant matrix can disable it without a static override.
- tasks/01_core.yml + tasks/06_user_bootstrap.yml: pre-create the
administrator only in LDAP-disabled variants — when LDAP is wired,
ldapauth materialises the row on first admin login.
- files/playwright.spec.js: variant-aware spec replacing the parked
test.skip cases. Three tests now run: guest flow (always), admin
native login (picks double-login vs direct based on
OAUTH2_SERVICE_ENABLED), biber + admin side-by-side in isolated
contexts. PERSONA_* still blocked — friendica has no header-trusted
autocreate path.
- templates/playwright.env.j2: expose BIBER_* + service-gate flags.
- inventories/development/default.yml: biber email aligned with the
oauth2-proxy `email_domains = infinito.example` filter so biber's
Keycloak login isn't 403'd at the proxy.
- TODO.md: SAML migration outlined as the long-term replacement for
the double-login (currently second hop silently bypasses Keycloak 2FA).
Drive-by: trim double-space before `# nocheck` in
roles/sys-aur/meta/services.yml.
Verified on VARIANT=0 (oauth2 + ldap + oidc all enabled): three
playwright tests green, failed=0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Picked up by the make-test setup phase regenerating generated artefacts
during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh: rename AUTOFORMAT_PARALLEL to PARALLEL,
declare it at file start with default `true`; still env-overridable.
Reword the worker comment so the literal `# shellcheck` token no
longer appears at start-of-line and trips SC1072/SC1073.
* scripts/lint/galaxy.sh: drop the GALAXY_FATAL / GALAXY_NAMESPACE /
GALAXY_PARALLEL env vars (over-engineered for a single repo). Failure
is always fatal. NAMESPACE is an internal constant. Add the same
PARALLEL env var (default `true`, env-overridable); falsy demotes the
xargs fan-out to one worker.
* docs/contributing/environment/variables.md: collapse the four
AUTOFORMAT_*/GALAXY_* rows into one `PARALLEL` row covering both
consumers.
* docs/contributing/tools/make.md: refer to `PARALLEL=0` everywhere;
add the lint-galaxy parallelism note.
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Makefile lint integration:
* scripts/install/lint.sh: replace the prior checkmake binary install with
ensure_mbake (mbake is pip-installable, parses our Makefile correctly
whereas checkmake bailed on .ONESHELL: / .SHELLFLAGS: / ifneq).
* scripts/lint/makefile.sh: new runner — `mbake format --check Makefile`
reports formatting drift and missing .PHONY declarations.
* scripts/lint/autoformat.sh: add a two-pass `mbake format Makefile`
(mbake's formatter is not strictly idempotent on the first pass).
* Makefile: split the multi-line `.PHONY:` block into 18 single-line
`.PHONY:` declarations so mbake's parser can validate it; add the
previously-missing `install-python-dev` and `build-dependency` PHONY
targets; add `lint-makefile` aggregate-and-direct target.
* .pre-commit-config.yaml: new make-lint-makefile hook scoped to
`^Makefile$`.
* .github/workflows/lint-makefile.yml: new CI workflow (pip-install mbake
+ make lint-makefile).
* .github/workflows/ci-orchestrator.yml: wire lint-makefile into
lint-gate.needs.
Misc:
* .gitignore: add roles/importer_result.json (tooling artefact written by
galaxy-importer when it runs in roles/).
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* .pre-commit-config.yaml: add 6 scoped lint hooks (lint-action,
lint-ansible, lint-python, lint-shellcheck, lint-markdown, lint-galaxy)
with `files:` patterns so a markdown-only commit no longer triggers
ansible-lint or galaxy-importer. Aggregate make-test, make-test-lint
(md-only branch) and pre-push make-test-external are kept.
* Makefile: drop `lint` from the `make test` dependency chain. Per-file-
type lint stages now live exclusively in pre-commit; running them again
from `make test` would just duplicate work.
* scripts/lint/autoformat.sh: add markdownlint-cli2 --fix and
ansible-lint --fix passes alongside ruff/shfmt/shellcheck.
* roles/svc-db-openldap/meta/main.yml: collapse the wrapped description
into a single line (still under Galaxy's 255-char limit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tooling integration
* add .markdownlint-cli2.jsonc, .ansible-lint, .ansible-lint-ignore
configs (permissive day-1: cosmetic rules disabled, semantic ones on)
* extend scripts/install/lint.sh with generic install_pip_pkg helper +
ensure_ansible_lint / ensure_galaxy_importer / ensure_markdownlint_cli2
ensure_* functions (pip / npm with sudo / --user / --break-system-packages
fallbacks). install_ansible_tools now also pulls ansible-lint and
galaxy-importer; new `markdown` install group.
* extend scripts/lint/ansible.sh to run ansible-lint after the existing
syntax-check.
* add scripts/lint/markdown.sh and scripts/lint/galaxy.sh runners. The
galaxy runner parallelizes via xargs (-P=$(nproc)), prints OK/FAIL per
role, and treats `bare-name dependency` errors as success since our
roles/ tree is not a Galaxy collection layout. All other Galaxy schema
errors still fail.
* Makefile: new lint-markdown and lint-galaxy targets, both wired into
`make lint`.
* CI: new workflows lint-markdown.yml and lint-galaxy.yml; lint-ansible.yml
now installs ansible-core + ansible-lint; ci-orchestrator wires both
new lint workflows into lint-gate.
Galaxy schema fixes
* shorten the canonical `galaxy_info.company` block from three lines to
two ("Kevin Veen-Birkenbach\nhttps://www.veen.world", 45 chars) so it
stays under Galaxy's hard 50-char limit. Updated everywhere: 229 role
meta/main.yml plus _CANONICAL_COMPANY in
tests/lint/repository/test_role_meta_main_galaxy_schema.py.
* shorten 19 over-long role descriptions to <=255 chars.
* add stub READMEs for sys-svc-compose-ca and sys-utils-service-loader.
Markdown baseline fixes
* markdownlint-cli2 --fix run repo-wide (formatting normalization
across CHANGELOG, role READMEs, docs).
* manual MD041 fixes: prepend H1 in 15 files (CHANGELOG, LICENSE,
TODO.md helpers, role README templates, etc.).
* MD025 / MD003 / MD001 / MD032 manual fixes for heading-style and
list-spacing edge cases.
* role README content polish: "## Single Point of Truth (SPOT)" headings
renamed to "## User data resolution" (D3 self-label rule).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Module moves (with imports, paths, argv literals, scripts, docs, and
parents[N] formulas updated in lockstep):
- cli/deploy -> cli/administration/deploy
- cli/validate/inventory -> cli/administration/inventory/validate
- cli/vault -> cli/administration/secrets/vault
- cli/administration/inventory/encrypt -> cli/administration/secrets/seal
- cli/mirror -> cli/contributing/mirror
The new cli/administration/secrets cluster collocates the whole-file
ansible-vault wrapper next to the inventory-aware per-field sealer.
The user-facing CLI subcommand paths follow the package layout, e.g.
"infinito administration deploy dedicated <inv>" and
"infinito administration inventory validate <inv>".
Help system redesign in cli/core/{discovery,help,app}.py:
- Discovery now distinguishes runnable commands (directories with
__main__.py) from category folders (directories that only group
further commands). Empty support packages are filtered out.
- Each category folder MAY carry a README.md; its H1 heading is used
as the title and its first paragraph as the one-line description in
the help output.
- "infinito" / "infinito --help" prints only the top-level entries
(one level deep). "infinito <category>" navigates into a folder and
shows its sub-folders and commands one level deep, with the README
title as the page heading and the invocation as a dim subtitle.
- New "--tree [N]" flag walks the tree under the (optional) path
prefix recursively. N caps the depth (1-based); omit N for an
unbounded walk. The output uses Unicode tree connectors, an emoji
per entry (folder for categories, gear for commands), a globally
aligned description column, and multi-line wrapped descriptions
with tree-continuity bars on continuation lines.
- Category and command listings use the full invocation
("infinito meta callorder") rather than the bare leaf name, so the
output is copy-pasteable.
Convention doc + 17 CLI READMEs:
- New docs/contributing/artefact/files/cli/readme_md.md describes the
H1+paragraph layout the help system reads, and how it differs from
role/docs READMEs.
- README.md added for cli/{administration, build, contributing, meta}
and their immediate sub-folders, plus cli/meta/roles/applications
and its resolution sub-folder. Each follows the "title with trailing
emoji + single descriptive paragraph" convention.
Mirror resolver/sync description strings (the help system reads them
from argparse --help output) added so they no longer surface as
"options:" in the global index. cli/mirror/cleanup already had a
description and is unchanged.
Fixes uncovered by the move:
- import unittest.mock added to four dedicated-deploy tests so
unittest.mock.patch resolves under stdlib unittest discovery
(Python 3.13 no longer surfaces unittest.mock through "import
unittest" alone).
- parents[N] index in the new
cli/administration/inventory/validate/__init__.py corrected from
3 to 4 to match the deeper layout.
- test_paths.INVENTORY_VALIDATOR_PATH assertion and test_runner
substring assertions updated to match the new validator path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces per-role `meta/variants.yml` for declaring multi-variant
deploys (e.g. WordPress Single-Site vs. Multisite) and reshapes the
development deploy CLI to materialise one inventory folder per variant
and iterate them with co-located sync+async passes.
Why:
- The previous single-inventory + Pass-1-then-Pass-2 split deployed each
variant twice and ran the async pass against a host left in a
different variant's state, masking real bugs.
- Inventory and runtime concerns were tangled: variant selection lived
as a runtime extra-var and required cross-stage coordination.
How:
- `roles/<role>/meta/variants.yml` (file format docs at
docs/contributing/artefact/files/role/variants.md): YAML list of
deep-merge overrides on top of `config/main.yml`. Empty entry `{}` is
variant 0 (= legacy payload). 6 roles get their dev overrides moved
here from inventories/dev.yml; WordPress declares 2 variants.
- `utils/runtime_data.py` gains `_build_variants` / `get_variants` and
drops the retired `_active_variants` runtime selector. Loader always
returns variant 0 as the default; non-zero variants reach
`get_merged_applications` as inventory-level `applications.<app>`
overrides baked at init time.
- New `cli/deploy/development/inventory.py`: `DevInventorySpec`,
`plan_dev_inventory_matrix` (pure planner shared between init and
deploy), `build_dev_inventory_matrix` (materialises one folder per
round at `<dir>-<R>`, or `<dir>` when single-variant). Single-folder
paths preserve full backward compat.
- `cli/deploy/development/deploy.py`: matrix loop with per-round
cleanup (only apps whose variant changed AND that are about to
re-deploy), `--variant <idx>` (or `VARIANT=` env) to pin a single
round, and `--full-cycle` (or `FULL_CYCLE=true`) which interleaves
sync + async per round so the async re-deploy targets exactly the
host state the matching sync deploy just produced.
- `cli/deploy/development/exec.py` gains repeatable `--env KEY=VAL` so
bash callers can inject per-run context into in-container helpers
without inlining heredocs.
- `scripts/meta/env/inventory.sh` exports `INVENTORY_VARS_FILE`
(Python mirror in `cli.deploy.development.common.DEV_INVENTORY_VARS_FILE`;
test_inventory_vars_file_spot.py locks them against drift).
- `make deploy-fresh-purged-apps` and `scripts/tests/deploy/ci/dedicated.sh`
switch to a single init + single deploy `--full-cycle`. The
`make deploy-reuse-*` bash scripts honour `VARIANT=` by suffixing
`INVENTORY_DIR`/`INVENTORY_FILE`.
- The 5 `scripts/tests/deploy/local/deploy/*.sh` scripts now delegate
their in-container heredocs to dedicated helpers in
`scripts/tests/deploy/local/utils/`, removing the bash-in-bash
quoting hell.
Tests:
- `tests/unit/utils/test_variants.py` (renamed from
`test_application_variants.py`): loader edge cases, variant-zero
default, retired-`_active_variants` guard.
- `tests/unit/cli/deploy/development/test_inventory.py` (new): planner,
variant baking, builder, `filter_plan_to_variant`, `DevInventorySpec`.
- `tests/unit/cli/deploy/development/test_matrix_deploy.py` (new):
per-round deploy filter, cleanup decisions, `--variant` pinning,
`--full-cycle` per-round interleave.
- `tests/unit/cli/deploy/development/test_inventory_vars_file_spot.py`
(new): drift guard for the bash <-> Python literal pair.
- `tests/unit/cli/deploy/development/test_exec.py` extended for `--env`.
Docs:
- `docs/contributing/artefact/files/role/variants.md` (new): file format.
- `docs/contributing/design/matrix-deploy.md` (new): runtime mechanism
(folder layout, --variant, --full-cycle, cleanup logic).
- `docs/contributing/actions/deploy.md` (new): contributor dev-deploy
flow (make targets, --variant pinning).
- `docs/administration/deploy.md` slimmed to admin-only (was mixed
admin+dev), cross-links to the new contributor doc.
- `docs/agents/action/iteration/role.md` keeps only agent-iteration
recipes; mechanism explanation moved out and linked.
- `docs/contributing/documentation.md` adds the "no self-SPOT
declaration" rule; `applications.md` renamed its SoT header for
consistency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements requirements 005, 006, and 007.
005: per-application Keycloak `group-ldap-mapper`s anchor each app's
role groups under `/roles/<application_id>/<role_name>` (and, for
tenant-aware apps, `/roles/<application_id>/<tenant>/<role_name>`).
The flat-CN LDAP layout keeps each CN globally unique so Keycloak's
group-tree resolver never collides; per-app mappers use the LDAP
`description` as the visible Keycloak group name to keep the path
free of redundant prefixes. Migration purges legacy pre-005 entries
and forces a Keycloak `logout-all` so stale tokens drop pre-005
group paths.
006: per-service `<SERVICE>_SERVICE_ENABLED` env flags and the
`service-gating.js` helper let role specs `skipUnlessServiceEnabled`
cleanly when a shared service is missing or `SERVICES_DISABLED` is
set; `MODE_CI` markers (`INFINITO_MAKE_DEPLOY`, `GITHUB_ACTIONS`,
`ACT`) propagate through the deploy CLI into the test stage.
007: WordPress->Discourse round-trip Playwright scenario gated on the
`web-app-discourse` deployment.
WordPress Playwright runs 5/5 green on the single-site flow; the
multisite + Discourse scenarios are correctly skipped when their
service prerequisites are absent. The `rbac_group_path` lookup is the
single sanctioned producer for OIDC group paths and is wired into
every consumer (yourls, lam, minio, openwebui, pgadmin, phpldapadmin,
phpmyadmin, prometheus, xwiki).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switch SYS_SVC_DOCKER_DNS_TEST_BUSYBOX_IMAGE / _NODE_IMAGE to the
role-local `images:` pattern. Consumers now resolve through
`lookup('image', <service>, 'ref')`, which picks up mirrored GHCR
paths from `images_overrides` automatically. Drops the hardcoded
rate-limit workaround from inventories/dev.yml.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrates large parts of the legacy util-* role structure to inventory-driven
bundles and standardizes AUR installation via central system roles.
- Adds `web-svc-legal`, `web-svc-asset`, and `web-app-dashboard` to all server bundles.
- Removes `util-srv-corporate-identity` entirely.
- Introduces new workstation bundle inventories:
`admin`, `admin-network`, `browser`, `design`, `dev-*`, `game-*`, `office`.
- Uses bundle variables `SYS_PACKAGES` and `SYS_AUR_PACKAGES` as the primary
installation mechanism.
- Removes/replaces obsolete util-desk/util-dev roles where fully covered.
- Renames dev roles:
`util-desk-dev-core` -> `dev-core`,
`util-desk-dev-python` -> `dev-python`,
`util-desk-dev-arduino` -> `dev-arduino`.
- Adds missing `application_id` vars for renamed roles.
- Removes the entire `util` section from `roles/categories.yml`.
- Replaces direct `kewlfft.aur.aur` calls in desktop/driver roles with
a centralized wrapper role.
- Introduces new role `sys-aur-install` as AUR installation wrapper.
- Migrates `dev-yay` to `sys-aur` and adds `SYS_AUR_PACKAGES: []` default.
- Extends constructor stage:
auto-load `sys-package` when `SYS_PACKAGES` is set,
and `sys-aur` + `sys-aur-install` when `SYS_AUR_PACKAGES` is set.
- Cleans up stage loading:
removes util-srv from `02_server.yml`,
reduces workstation loading to `desk-roles.yml` + `drv-non-free`.
- Updates integration/unit tests for the new invokable and include behavior.
- Integration test now accepts static `include_tasks` references.
- Fixes recursion risk between `sys-aur` and `sys-aur-install` by setting
`run_once_sys_aur` at the beginning of `sys-aur/tasks/01_core.yml`.
BREAKING CHANGE:
- `util-*` categories and bundle paths are no longer part of stage loading.
- Multiple `util-desk-*`, `util-dev-*`, and `util-srv-*` roles were removed.
- Deployment groups must be migrated to the new bundle/role naming.
