derive_includes applies the same shared:true flip as force_shared_db, so
variant rounds that pin a consumer's DB to shared:false keep the central
provider (and its generated credentials) in the include set the deploy
will actually load. extend_inventory derives its manager-placed groups
from the same variant-aware closure instead of a base-config walk, which
re-added pruned provider groups without credentials. matrix.py passes the
round's variant map to the extend step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Inventory include sets ignored the active variant: swarm
derive_includes walked the variant-free base config (dynamic service
flags count as enabled) and the compose CombinedResolver followed
run_after as an include edge. Providers a variant pins to false
(e.g. mobilizon variant 1 sso) were still deployed.
- derive_includes: swap every app listed in INFINITO_APP_VARIANTS to
its variant config, index 0 included (nextcloud variant 0 disables
11 partner services)
- CombinedResolver: follow_run_after flag; include resolution passes
False so run_after stays a pure ordering hint
- drop dead deps.py chain (resolve_deploy_ids_for_*) and unused
resolve_with_edges; rewiring them would resurrect variant-blind
run_after-following includes
- inventory.md: document the real include pipeline and
is_explicit_truth semantics
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lifecycle envelope for test-deploy discovery is now driven by
INFINITO_LIFECYCLES (default: alpha beta rc stable) instead of a
hardcoded constant. apps.sh forwards it to the complexity CLI via a
new --lifecycles flag (comma- or whitespace-separated); the model
falls back to the built-in default when unset.
Manual CI (entry-manual) gains two override fields: 'lifecycles' to
override the envelope per run, and 'modes' (default 'swarm compose')
to pick which deploy jobs run. ci-orchestrator gates test-deploy-compose
and test-deploy-swarm on the selected modes and threads lifecycles into
both. The new key is registered in the passthrough handler so it lands
in the generated .env.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Embedded per-app DBs (shared:false) resolve to mariadb-on-NFS with no placement pin in swarm, which InnoDB does not support (locking, split-brain); embedded DBs are a compose-only pattern. After provisioning each swarm-test round's inventory, force shared:true on every svc-db-* provider service so consumers route to the central provider.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
autoformat reformats (settings sort, complexity/model.py, test_database_query.py, keycloak vars wrap); shorten akaunting swarm task name to <=80 chars; delete narration comments from keycloak vars; remove 'invariant' from the comment linter's valid marker set. make autoformat + make test verified green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drive the test-deploy matrix discovery through the complexity report instead of the type CLI: apps.sh selects the mode's tested roles via the 'compose'/'swarm' column (which already bakes in invokable + tested-lifecycle + per-mode skip), orders them coverage-first, and hard-caps the cumulative per-mode runner budget.
- complexity gains --deploy-mode {compose,swarm} (makes 'bundles'/'jobs' the runner count for that mode: compose packs variants into size/storage bundles, swarm runs one per deployable variant) and --max-jobs N (keep only the leading rows whose cumulative jobs stay below N).
- apps.sh emits whole role names; variant_bundles/output_apps.sh expand them into the matrix unchanged. Manual INFINITO_WHITELIST becomes a 'name %% {…}' filter clause applied before the cut, so a manual run only counts the apps it actually invokes.
- INFINITO_DEPLOY_MODE is now required; INFINITO_MAX_JOBS, INFINITO_DISCOVERY_SORT and INFINITO_REQUIRED_STORAGE move to default.env (the SPOT) + the passthrough handler, so no shell-side defaults remain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the 'meta roles applications complexity' report:
- --sort takes comma-separated multi-key clauses with per-key direction (e.g. 'desc embeds, asc weight'); replaces --order.
- --variant lists one row per meta/variants.yml variant; --format adds yaml.
- New columns: weight (was total), random, id, row, variants, bundles, jobs, lifecycle, covered_by, compose, swarm.
- covered_by marks the first green (1-based id) different-role predecessor that embeds the row; variant>0 never covered.
- bundles is the compose bundle/job count (whole-role) or 1 (per variant); jobs is the running sum.
- compose/swarm reflect the CI test-deploy skip logic (invokable + tested lifecycle + per-mode skip + swarm deployable variant).
- --filter is a boolean expression DSL (%% == \!= < > <= >=, and/or/xor/not, parens, set literals) over row fields.
Add shared SPOTs: variant_bundles.compose_bundle_counts and variant_status.deployable_variant_indices, reused by the swarm/compose matrix. Update roundtrip.sh/md to --sort "desc weight".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
failed_roles for the swarm/docker scope only flags a role whose job in that mode actually ran and was not green; total_state is green when every mode that ran is green. A role with no job in a mode (a host driver that never deploys in swarm) is no longer counted as failed for that mode, so 'ci trigger --failed swarm' / 'ci status --failed swarm' stop listing drv-* and other roles that simply do not run in swarm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The merged subset feature failed test-lint: a .py under scripts/, a local parents[N] PROJECT_ROOT, a direct yaml.safe_load, an uncached read, and narration comments.
- move scripts/meta/resolve/pr/subset_roles.py -> cli.meta.ci.subset_roles (scripts/ is .sh-only); dispatcher invokes it via python -m
- parse via utils.cache.yaml.load_yaml_str; import utils.PROJECT_ROOT
- test invokes the module via -m against real roles; nocheck the tempfile read
- strip narration comments; docs document HOW not WHY
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The merged subset feature failed test-lint: a .py under scripts/, a local parents[N] PROJECT_ROOT, a direct yaml.safe_load, an uncached read, and narration comments.
- move scripts/meta/resolve/pr/subset_roles.py -> cli.meta.ci.subset_roles (scripts/ is .sh-only); dispatcher invokes it via python -m
- parse via utils.cache.yaml.load_yaml_str; import utils.PROJECT_ROOT
- test invokes the module via -m against real roles; nocheck the tempfile read
- strip narration comments; docs document HOW not WHY
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conflicts resolved:
- .claude/.gitignore: union of ignore patterns
- plugins/lookup/rbac_group_path.py: drop both unused conflicting imports
(body fetches apps via lookup_loader)
- roles/sys-svc-compose/handlers/main.yml: keep HEAD's refactor (compose-up
handlers live in compose.yml, included per DEPLOYMENT_MODE)
- utils/cache/base.py: keep HEAD's merge_with_defaults/FILE_TOKENS imports
Post-merge fixes for main's newly-enabled lints:
- pyproject.toml: grandfather 5 files under the BLE001 ratchet
- justify 6 pre-existing security # noqa markers (S108/S311/S701)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The --failed swarm/compose/total filters keyed the deploy-job regex on the
literal words "Compose"/"Swarm", which no real CI job carries. Job names are
"<glyph> <app> <variant>" (e.g. "... / test-deploy-swarm / 🐝 web-app-x 0,1"),
so the regex matched nothing and the CLI reported "Nothing failed" for runs
full of failures. app_for_job in the role-runtime path had the identical bug
(returned the shard token instead of the app id).
Match on the 🐳/🐝 glyph instead, tolerate the orchestrator caller prefix and
the " <variant>" shard suffix, and route both consumers through one parser
(runs.app_of_job). Centralize the naming as SPOTs: MODE_GLYPHS in runs.py and
ORCHESTRATOR_PREFIX plus a workflow-sourced deploy_job_name() helper in tests,
so fixtures render real names from the workflow YAML and cannot drift back to
job names GitHub never emits.
Add an integration test that renders names from test-deploy-{compose,swarm}.yml
and asserts the parser extracts app+mode for both modes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the csp, domains, and networks blocks out of roles/*/meta/server.yml
into dedicated meta files, each mapping to its own top-level applications key
via the recursive meta-walk:
csp: -> meta/csp.yml -> applications.<app>.csp
domains: -> meta/domains.yml -> applications.<app>.domains
networks: -> meta/networks.yml -> applications.<app>.networks
Residual nginx/proxy keys (status_codes, client_max_body_size, locations,
ip6_enabled, config_upstream_url) stay in server.yml; an emptied server.yml is
deleted. 98 server.yml -> 249 new files, 74 deleted.
- Consumers repointed applications.<app>.server.{csp,domains,networks} ->
applications.<app>.{csp,domains,networks} across templates, vars, tasks,
lookups and filters.
- Infra readers migrated: canonical_domains_map, application_domain_index,
build_ldap_role_entries, service-registry overlay discovery,
roles_with_service, and the file-based readers utils/domains/list.py +
utils/meta/scan.py (read the now-deleted server.yml directly).
- Detection + mapping: _META_TOPICS gains csp/domains/networks;
ROLE_FILE_META_{CSP,DOMAINS,NETWORKS} constants + ROLE_FILES entries.
- New self-expiring lint guard (test_server_meta_no_split_fields) fails if a
server.yml still carries a split field; auto-fails from 2026-12-28 as a
delete reminder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve the 80 open CodeQL/code-scanning alerts on this branch.
Security-sensitive fixes:
- cli/administration/deploy/development: break the unsafe import cycle by
extracting the env/arg helpers into a leaf module `env.py`, giving the
load-time DAG compose -> common -> deps -> env.
- web-app-bluesky login-broker server.js: prototype-pollution guard on
cookie parsing, CR/LF + control-char log sanitisation, HTML-escaped
exception text, and a ReDoS-bounded handle regex.
- web-app-baserow SSO: harden the open-redirect `next` handling.
- network diagnose probes: require TLS >= 1.2.
- actionlint installer: validate tar members against path traversal.
- decidim/apt-purge lint regexes: rewrite to avoid catastrophic backtracking.
- 6x Playwright URL regexes anchored; crs-k/stale-branches pinned to a commit SHA.
Mechanical cleanups: unused imports/globals/locals, empty-except comments,
import-and-import-from, implicit-string-concat, multiple-definition,
unreachable/useless-assignment.
False positives are intentionally left in place (Django settings patches,
Odoo __manifest__, side-effect imports, cross-module symbols).
Recurrence guards:
- import-linter contract (pyproject + tests/lint) locks the dev-deploy
package acyclic; a new cycle now fails CI instead of only CodeQL.
- new lint test requires every bandit (S###) `noqa` to carry a justification.
- BLE001 (blind-except) added to ruff as a ratchet (tests/roles exempt,
current production offenders grandfathered).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The merged main renamed the inventory variant filter (filter_plan_to_variant -> filter_plan_to_variants) and added the shared variant_select selector. Adapt cli.administration.deploy.swarm.matrix to consume the variant CSV bundle via variant_select (add_variant_args + apply_variant_filter) instead of its bespoke single-int parser, so it iterates exactly the runner's bundle of rounds and the stale import no longer breaks.
Mirror test-deploy-compose.yml in the swarm workflow: pass the variant bundle to the orchestrator (re-passed through sudo so env_reset can't drop it), show the variant in the job name, and suffix the playwright artifact with variant_slug. Also singularise the discover job names in both workflows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a lint-packages target/workflow that parses the generated packaging metadata with each ecosystem's native parser (dpkg-parsechangelog, rpmspec, bash -n for the PKGBUILD), failing on any parser stderr so warnings cannot slip through. Wired into the make lint fan-out and ci-orchestrator's lint gate; an install group provisions the parsers (best-effort, cross-distro).
Hardens package_mirror so the generated changelogs parse without warnings: every RPM body line gets a '- ' continuation prefix (a column-0 '*' was read as a new %changelog entry -> 'bad date'), and the archived-releases notice is folded into the oldest kept entry instead of trailing after the final entry (free text there made dpkg-parsechangelog warn). Regenerates packaging/debian/changelog and the Fedora spec accordingly.
Adds a test-lint check (test_package_changelog_completeness) asserting both package changelogs list every CHANGELOG.md + docs/changelog release and that their inline window matches CHANGELOG.md, plus unit regressions for the RPM continuation prefix and the Debian footer fold.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
format_debian_entry indented only the first body line per entry, leaving the rest at column 0. dpkg's changelog parser treats un-indented change lines as badly formatted, loses the entry boundary and reports 'cannot parse maintainer email address ""', failing the .deb build in build-ci-images. Every non-empty body line is now indented by two spaces.
trim_and_archive additionally dropped the '# Changelog' H1 preamble on every rewrite (it rebuilt content from the version entries only), which would break the MD041 markdown lint on the next trim. The preamble before the first version header is now preserved.
Regenerated packaging/debian/changelog and packaging/fedora/infinito-nexus.spec from CHANGELOG.md (untrimmed). Also adds the missing YAML document-start to web-app-pihole/tasks/main.yml.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reorder the clean-container-owned / clean-pycache-dirs / clean-stale-nfs Makefile targets into ascending order, set the executable bit on the swarm NFS cleanup scripts, and opt the CI-logs analyze and download-test file reads out of the cached-read lint (a runtime read of a user-supplied log, and a temp fixture written inside the same test).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
python -m cli.meta.ci.logs.download fetches a run's job logs and artefacts in parallel, with conclusion filters (-s/-f/-c/-k), a jobs.json/summary.tsv manifest, per-request jitter to dodge API rate limits, and idempotent re-runs. python -m cli.meta.ci.logs.analyze clusters the downloaded logs by their root failing task.
Rename the agent CI-triage page debug/pipeline.md to debug/ci.md, point its download/inspect step at the new CLI, and reference the tooling from the contributing CI-debugging page. Add unit tests for both commands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the single utils/github/profile/role_summary.py script with a
focused CLI package split per responsibility (model, logparse, csvio,
render, github, sources). Output format is selectable via --format
(table default, plus markdown, csv, json); source is auto-detected as an
Ansible log, a role-runtimes CSV, or a GitHub Actions run/job URL (CSV
artifacts downloaded via gh).
The compose workflow now writes the CSV as a downloadable artifact and
appends the per-variant markdown to the step summary, and hard-fails
when the log is missing or carries no profile_roles data so a broken
metrics capture can no longer pass silently.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_gh ran gh with check=True and capture_output=True, so on failure the gh error message (e.g. 'Not Found (HTTP 404)' when dispatching to a repo that lacks the workflow) was buried inside the CalledProcessError and only the command repr showed in the traceback. Check the return code explicitly, write gh's stderr through to our stderr, and raise SystemExit(returncode) so the real error is visible and the CLI exits with gh's non-zero code cleanly instead of a traceback. Keeps the explicit check=False for ruff PLW1510.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collect and aggregate each service's min_storage (summed like mem) and surface it as a column in the detail, per-variant and per-role views, the JSON output and the filter/order fields.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
meta roles applications ressources gains two summary modes alongside the existing per-service detail table: (1) no --role: one row per role, each showing its heaviest variant (mem_reservation/mem_limit/pids_limit = the max per-variant sum, cpus = max), or the --variant N footprint of every role. (2) --role X without --variant: one row per variant of X (its sum/max). (3) --role X --variant N still renders the detailed per-service table - now the only way to reach it. --format/--order/--filter/--depth/--unshared apply in every mode. The table/JSON renderers are generalized (render_summary_text/json with a key_field) and a variant query accessor is added.
Useful for the CI runner-budget question: the heaviest-variant footprint per role/variant is now directly queryable. Tests cover all three modes (detail/all-roles/per-variant).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The required_by coverage verifier treated every flat 'required_by: {categories/roles}' as required in BOTH deploy modes. But sys-ctl-rpr-container-hard/soft repair compose compositions (DIR_COMPOSITIONS) and gate their body to compose; declaring required_by:[web] flat made the verifier expect them on swarm too, where they correctly skip, so web-opt-rdr-domains swarm CI failed 'required role(s) did not execute'. Let required_by nest under 'compose:'/'swarm:'; the verifier infers the deploy mode from the role set, mirroring DEPLOYMENT_MODE's own groups['svc-swarm-node'] definition, and enforces only the matching mode.
Migrate all 21 existing flat declarations to mirror under both modes (behavior unchanged) and mark sys-ctl-rpr-container-* compose-only. role_has_required_by reads both forms. New lint test_required_by_mode_keys requires the explicit compose:/swarm: form so the flat-vs-gate contradiction can't recur.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The required_by coverage verifier (_role_body_executed) reads ANSIBLE_LOG_PATH, but the swarm and compose deploy point that at a log FILE, which ansible prefixes every line with '<ts> p=<pid> u=<user> n=<name> <LVL>| '. That prefix defeated both the line-anchored non-skip status match and the TASK-header block split, so every required role (e.g. sys-ctl-cln-bkps under any svc-bkp deploy) false-positived as 'did not execute' although it actually ran on the manager. Strip the prefix before matching; reproduced locally with a minimal multi-host include and verified the file slice flips while raw-stdout matching is unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
--run URL selects the run whose results determine the --failed apps;
without it the latest deploy run on the branch is used. Repo + run id are
parsed from the URL.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the unused shebangs (EXE001), mark the PASS emoji constant noqa
S105, annotate the test fixture as ClassVar, and use reserved example.com
hosts / a nocheck marker for the placeholder run URLs so the
url-reachability test stops flagging them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add 'infinito administration deploy ci status' and '... trigger' for
inspecting and re-running the compose/swarm deploy-test matrix of a
branch's CI run.
status: per-role docker (compose) and swarm result plus an aggregated
total column (green only when both modes are green), a trailing 'run'
column with the job URLs, --failed [total|swarm|compose] filtering, and
table/string output. Default reads the latest deploy run on the current
branch; --url reads a specific run.
trigger: dispatches entry-manual.yml for the current branch -- all roles
by default, only the last run's failures with --failed [scope], or an
explicit --apps list.
The target repo is resolved from the branch's tracking remote (a fork),
not gh's default repo. Covered by unit tests for run parsing, status
rendering, and trigger dispatch logic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a `skip: [compose, swarm]` field on a role's meta/services.yml primary entity
so an app can opt out of the test-deploy matrix per deploy mode, replacing the
hardcoded exclude regex in apps.sh (the "TODO: Remove exclude condition"). The
filter threads through both discovery paths: the CI workflows pass
INFINITO_DEPLOY_MODE -> apps.sh -> `cli meta roles applications type
--skip-mode`, and roundtrip.sh honours the same get_role_skip SPOT per app/mode.
Apply skip: [compose, swarm] to svc-storage-nfs-server and svc-storage-nfs-client
(NFS storage is swarm-only and set up via the cluster inventory, not the matrix)
and web-opt-rdr-www (the previously regex-excluded app).
Covered by get_role_skip and skip_mode-filter unit tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two pre-existing ruff findings in the complexity reporter (from an earlier
refactor) blocked `make autoformat` / `make test`: model.py used a bare
hashlib.sha1 for a non-security cache key (S324) and render.py built a list with
a for-append loop (PERF401). Pass usedforsecurity=False and switch to
list.extend.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Split the single complexity __main__.py into graph (discovery + BFS),
model (ComplexityRow + scoring), render (table/json/string) and cli
(argparse + main); __main__ stays a thin entry point.
Add a 'base' column (sha1 of the role's name unioned with its embedded
services, sorted) so roles in the same mutual-embedding cluster share a
key, and a 'siblings' column listing the rest of that cluster. --unique
keeps one role per base. roundtrip's default app list now passes
--unique to skip redundant same-cluster deploys.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
