Commit Graph
1953 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 4.8 18cf9b9ada fix(matomo): inject tracker into web-svc consumers on swarm re-injection
The matomo bootstrap re-injects early-rendered web-svc consumers (such as
web-svc-logout) after obtaining the admin token, but in swarm this never ran:
roles_with_service filters by the manager's group_names, which omits
worker-placed web-svc services, so the re-injection loop was empty and the
logout page shipped without the matomo tracker.

- roles_with_service: add a scope kwarg; scope='all' bypasses the per-host
  group_names filter (the merged applications config is already
  deployment-scoped, so iterating all of it equals the deployment's consumers).
- 05_bootstrap: enumerate consumers with scope='all', stat each compose project
  dir, and re-inject only deployed ones. Avoids a fatal chdir into an undeployed
  consumer's missing /opt/compose/<entity>/ that scope='all' would otherwise hit.
- sys-front-inj-matomo: resolve matomo_effective_token (override else
  lookup('users')); the disable-gate and the SitesManager getSitesId/addSite
  calls use it, so the re-injection passes the fresh bootstrap token and bypasses
  the play-start-stale controller-side users cache that left matomo disabled.

scope defaults to 'host' and the override defaults to empty, so every existing
caller and the per-consumer injection path are unchanged. Validated green in
both compose and swarm round 1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 16:37:56 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6ce6780575 refactor(meta): split server.yml into csp/domains/networks.yml
Extract the csp, domains, and networks blocks out of roles/*/meta/server.yml
into dedicated meta files, each mapping to its own top-level applications key
via the recursive meta-walk:

  csp:      -> meta/csp.yml      -> applications.<app>.csp
  domains:  -> meta/domains.yml  -> applications.<app>.domains
  networks: -> meta/networks.yml -> applications.<app>.networks

Residual nginx/proxy keys (status_codes, client_max_body_size, locations,
ip6_enabled, config_upstream_url) stay in server.yml; an emptied server.yml is
deleted. 98 server.yml -> 249 new files, 74 deleted.

- Consumers repointed applications.<app>.server.{csp,domains,networks} ->
  applications.<app>.{csp,domains,networks} across templates, vars, tasks,
  lookups and filters.
- Infra readers migrated: canonical_domains_map, application_domain_index,
  build_ldap_role_entries, service-registry overlay discovery,
  roles_with_service, and the file-based readers utils/domains/list.py +
  utils/meta/scan.py (read the now-deleted server.yml directly).
- Detection + mapping: _META_TOPICS gains csp/domains/networks;
  ROLE_FILE_META_{CSP,DOMAINS,NETWORKS} constants + ROLE_FILES entries.
- New self-expiring lint guard (test_server_meta_no_split_fields) fails if a
  server.yml still carries a split field; auto-fails from 2026-12-28 as a
  delete reminder.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 22:45:05 +02:00
kevinveenbirkenbachandClaude Opus 4.8 238bc285c7 refactor(lookups): reach merged config via the lookup-loader SPOT
Consumer lookup plugins now reach the merged config through the Ansible
lookup loader instead of importing get_merged_applications /
get_merged_domains from utils.cache. The merge stays one cached
implementation surfaced via the applications/config/domains/users lookups;
consumers call lookup_loader.get(...).run() (the compose_networks idiom).

- get_merged_applications walks the whole meta/ tree recursively
  (meta/**/*.yml to applications.<app>.<path>, addons and any depth) and
  folds the raw meta/volumes.yml subtree onto applications.<app>.volumes
  after the templar render so the Jinja source: strings stay unrendered.
- 31 consumer lookups rerouted; providers (applications/config/
  applications_current_play/users/domains) and the raw-volume accessors
  (volume/nginx) keep the cache import behind # nocheck: lookup-cache-import.
- New lint test_no_cache_imports_in_lookups enforces the contract.
- nfs_flavor is now the flavor SPOT (svc-storage-nfs-server + nfs_client_src
  resolve via lookup('nfs_flavor')); removed the unused STORAGE_NFS_FLAVOR
  group_var. Extracted extract_default_credentials to
  utils/cache/schema_credentials.py to keep applications.py under the cap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 18:35:06 +02:00
kevinveenbirkenbachandClaude Opus 4.8 5e90c9cfd0 fix(nfs lookups): render via variables + read the applications var
self._templar.template('{{ var }}') no-ops on untrusted strings in ansible 2.19+, so the nfs_* lookups emitted literal Jinja into /etc/exports and exportfs failed (the fsid=0 export never came up). The base lookups now read resolved inputs from 'variables'. nfs_flavor/nfs_client_src read the flavor from the applications variable (the variant-baked config provisioning bakes under applications:), not a re-merge of the role files via get_merged_applications. Adds a real-Templar regression test that fails on the old code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 18:35:06 +02:00
kevinveenbirkenbachandClaude Opus 4.8 8cdf9d2f6d fix(web-app-dashboard): swarm simpleicons probe + tile iframe capability
Simpleicons: the reachability probe ran inside the controller-side Jinja filter, which has no route to the node's published port in swarm, so every card silently fell back to Font Awesome. Move the probe to a STACK_HOST-delegated uri_retry step and make the filter pure assembly. Compose output is byte-identical.

Tile reachability: add a per-card services.dashboard.iframe capability (default = enabled) so apps whose SPA top-redirects out of an embed (Keycloak admin console, iframe:false) open in a new tab; roles_with_service emits the flag into the target manifest and the spec branches on it. For iframe:false targets the spec also asserts the HTTP layer permits framing (no blocking X-Frame-Options, CSP frame-ancestors allows the dashboard) so the flag can never mask a real header block. Fix the Tab-button locator and reveal it from fullscreen before asserting.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 18:35:06 +02:00
kevinveenbirkenbachandClaude Opus 4.8 fc77d9d265 fix(security): resolve all open code-scanning alerts and guard against recurrence
Resolve the 80 open CodeQL/code-scanning alerts on this branch.

Security-sensitive fixes:
- cli/administration/deploy/development: break the unsafe import cycle by
  extracting the env/arg helpers into a leaf module `env.py`, giving the
  load-time DAG compose -> common -> deps -> env.
- web-app-bluesky login-broker server.js: prototype-pollution guard on
  cookie parsing, CR/LF + control-char log sanitisation, HTML-escaped
  exception text, and a ReDoS-bounded handle regex.
- web-app-baserow SSO: harden the open-redirect `next` handling.
- network diagnose probes: require TLS >= 1.2.
- actionlint installer: validate tar members against path traversal.
- decidim/apt-purge lint regexes: rewrite to avoid catastrophic backtracking.
- 6x Playwright URL regexes anchored; crs-k/stale-branches pinned to a commit SHA.

Mechanical cleanups: unused imports/globals/locals, empty-except comments,
import-and-import-from, implicit-string-concat, multiple-definition,
unreachable/useless-assignment.

False positives are intentionally left in place (Django settings patches,
Odoo __manifest__, side-effect imports, cross-module symbols).

Recurrence guards:
- import-linter contract (pyproject + tests/lint) locks the dev-deploy
  package acyclic; a new cycle now fails CI instead of only CodeQL.
- new lint test requires every bandit (S###) `noqa` to carry a justification.
- BLE001 (blind-except) added to ruff as a ratchet (tests/roles exempt,
  current production offenders grandfathered).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 15:42:03 +02:00
kevinveenbirkenbachandClaude Opus 4.8 9942dbdbcf fix(svc-storage-nfs-server): fsid=0 v4 root + nfs_* lookup SPOT
Root cause of the kernel-flavor swarm ENOENT: no export carried fsid=0, so the NFSv4 mount of the nested /srv/nfs/infinito-state sub-export had no pseudo-root anchor and the server returned NFS4ERR_NOENT on the PUTROOTFH+LOOKUP walk. crossmnt could not fix it (it only crosses a boundary mid-walk, never sets the starting filehandle). Kernel now exports the state dir itself at fsid=0 and clients mount the bare v4 root ':/' (zero LOOKUPs). ganesha keeps Pseudo=Path and v3 mounts the full path. The self-bind is kept so fsid=0 rides a real mount boundary (stable st_dev) under overlayfs DinD.

The NFS derivations move to per-value lookup plugins (nfs_flavor, nfs_client_src, nfs_fstype, nfs_mount_opts, nfs_state_path) backed by pure helpers in utils/storage/nfs.py with unit tests; group_vars STORAGE_NFS_* become thin lookup wrappers. The dev/act->ganesha flavor rule stays the single SPOT in the role config; nfs_flavor reads the merged value so client and server agree.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 15:27:53 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6559ca0b84 fix(web-app-prometheus): resolve swarm scrape targets + healthz via a SPOT lookup
In swarm the sibling exporters (alertmanager, cadvisor, node-exporter, blackbox) and the healthz backend are unreachable on the compose container_name / host.docker.internal / 127.0.0.1 addresses, so prometheus marked every target down and /healthz/ready returned 502. Add a scrape_target lookup emitting <host>:<port> per mode (swarm tasks.<entity>_<service>:<internal_port>, compose container_name/compose_host), reading the internal port from services.<svc>.ports.internal; declare those ports on the four exporters; route both healthz probes through the proxy_pass lookup via one set variable. Compose output stays byte-identical.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 15:27:53 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e9299d0871 fix(sys-svc-compose): skip 'compose up' when the project declares no services
A launcher-managed app (discourse) renders an empty services map, so 'docker compose up' fails with 'no service selected'. The wrapper now returns no command for an 'up' against a zero-service project (main() exits 0); a compose that fails to parse is not treated as serviceless, so real errors still surface.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 07:39:24 +02:00
Alejandro Roman 17d6ac904f Merge remote-tracking branch 'upstream/main' into feature/self-host-runners
# Conflicts:
#	.github/workflows/test-deploy-universal.yml
#	.github/workflows/test-deploy-workstation.yml
2026-06-27 23:45:38 +02:00
kevinveenbirkenbach 0798a21b2a Merge branch 'main' into feature/swarm-nfs 2026-06-27 21:52:19 +02:00
kevinveenbirkenbachandClaude Opus 4.8 8a586ad373 feat(lint): validate generated distro changelogs + harden the mirror generator
Adds a lint-packages target/workflow that parses the generated packaging metadata with each ecosystem's native parser (dpkg-parsechangelog, rpmspec, bash -n for the PKGBUILD), failing on any parser stderr so warnings cannot slip through. Wired into the make lint fan-out and ci-orchestrator's lint gate; an install group provisions the parsers (best-effort, cross-distro).

Hardens package_mirror so the generated changelogs parse without warnings: every RPM body line gets a '- ' continuation prefix (a column-0 '*' was read as a new %changelog entry -> 'bad date'), and the archived-releases notice is folded into the oldest kept entry instead of trailing after the final entry (free text there made dpkg-parsechangelog warn). Regenerates packaging/debian/changelog and the Fedora spec accordingly.

Adds a test-lint check (test_package_changelog_completeness) asserting both package changelogs list every CHANGELOG.md + docs/changelog release and that their inline window matches CHANGELOG.md, plus unit regressions for the RPM continuation prefix and the Debian footer fold.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-27 21:36:30 +02:00
kevinveenbirkenbachandClaude Opus 4.8 36ebf34b64 Merge branch 'main' into feature/swarm-nfs
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 17:13:55 +02:00
kevinveenbirkenbachandClaude Opus 4.8 4313fa7375 fix(sys-svc-compose): inject CA trust into swarm deploys + use compose_file_args
In swarm, OmniAuth/OpenSSL (decidim OIDC discovery) and Node (bluesky-broker -> PDS) could not trust Keycloak's internal self-signed CA: docker stack deploy never received compose.ca.override.yml, and the override's $$-escaped entrypoint wrapper is swarm-incompatible (stack deploy does not un-escape it). Add a --no-wrapper override mode (env + CA mount only: SSL_CERT_FILE/CURL_CA_BUNDLE/NODE_EXTRA_CA_CERTS) generated on the stack host for self-signed apps, and build the stack-config/stack-deploy --compose-file list via compose_file_args (new --compose-file flag kwarg) instead of hardcoding compose.yml -- which also stops swarm silently dropping a role's own compose.override.yml. Compose untouched: self-signed-gated, no-wrapper is swarm-only, lookup defaults to -f.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 00:50:53 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0c0ec562f5 fix(database_query): resolve the swarm container id before exec
The module execs 'container exec <container>' as an argv list (no shell), using config["container"] = the literal "postgres" in shared mode, which is not a swarm container name -> 'No such container: postgres' for peertube and ~12 other callers. The database lookup already provides a swarm-aware config["address"] = "$(resolve-container-id <stack> <key>)"; expand it here (shlex-split, run the resolver as argv) to a concrete id when present. Compose (address == container) is byte-identical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 22:02:17 +02:00
kevinveenbirkenbachandClaude Opus 4.8 504324d528 refactor(ci/variant-bundles): lower default storage cap to 350GB from a single source
Set DEFAULT_MAX_STORAGE to 350GB (was 400GB) and make it the single source of truth: the INFINITO_VARIANT_BUNDLE_MAX_STORAGE env handler's setdefault and the resolve_max_storage error message now reference DEFAULT_MAX_STORAGE instead of repeating the literal. Update the default-resolution unit test accordingly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 20:37:35 +02:00
kevinveenbirkenbachandClaude Opus 4.8 9f669c985b fix(tests): nocheck the CI-logs placeholder run URL so test-external skips it
The example _RUN_URL in test_download.py lacked the '# nocheck: url' marker its siblings carry, so test_public_http_urls_are_reachable probed github.com/o/r/actions/runs/123 -> HTTP 404 and failed the external suite.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 17:20:52 +02:00
kevinveenbirkenbachandClaude Opus 4.8 cf765ee00c refactor: single /var/lib SPOT + swarm Playwright artifact upload
SPOT: declare the var-lib base once in default.env as INFINITO_DIR_VAR_LIB; the env build derives INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR from it; reference it everywhere (utils/paths.py, scripts via ${...:?}, tests, the dev inventory's DIR_VAR_LIB injection) instead of hardcoding the literal. group_vars keeps the literal default; the env-lookup lives in the dev inventory per the no-ansible-env-lookups lint rule. pull_specific_host.py --folder is now required and the role passes {{ DIR_BACKUPS }}.

Swarm e2e: collect_playwright_reports.sh docker-cp's the reports off the manager plus a workflow upload step, so swarm Playwright traces reach CI artifacts (was compose-only).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 17:01:09 +02:00
kevinveenbirkenbachandClaude Opus 4.8 4085c2344b chore: clear repo-lint failures surfaced by make test
Reorder the clean-container-owned / clean-pycache-dirs / clean-stale-nfs Makefile targets into ascending order, set the executable bit on the swarm NFS cleanup scripts, and opt the CI-logs analyze and download-test file reads out of the cached-read lint (a runtime read of a user-supplied log, and a temp fixture written inside the same test).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 17:01:08 +02:00
kevinveenbirkenbachandClaude Opus 4.8 9c029870fe feat(cli): add CI-logs download and analyze tooling
python -m cli.meta.ci.logs.download fetches a run's job logs and artefacts in parallel, with conclusion filters (-s/-f/-c/-k), a jobs.json/summary.tsv manifest, per-request jitter to dodge API rate limits, and idempotent re-runs. python -m cli.meta.ci.logs.analyze clusters the downloaded logs by their root failing task.

Rename the agent CI-triage page debug/pipeline.md to debug/ci.md, point its download/inspect step at the new CLI, and reference the tooling from the contributing CI-debugging page. Add unit tests for both commands.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 17:01:08 +02:00
kevinveenbirkenbachandClaude Opus 4.8 5ff862bef8 refactor(node-sizing): convert filter to lookup and extract shared to_bytes util
Replace 'lookup(applications) | node_max_old_space_size(app, service)' with the 'lookup(node_max_old_space_size, app, service)' lookup form (it resolves the app config itself), dropping the boilerplate at all 4 call sites (nextcloud whiteboard, peertube x2, suitecrm). Extract the shared decimal to_bytes size parser into utils/sizes.py, used by the lookup and the peertube install_mem_limit filter (removing the cross-plugin import).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 05:45:29 +02:00
kevinveenbirkenbachandClaude Opus 4.8 a1c062f6c6 fix(sys-svc-compose): centralize empty-services handling so launcher-based apps validate
base.yml.j2 now captures the sidecar service includes and emits 'services: {}' when none render, instead of a bare 'services:' which docker compose rejects ('services must be a mapping'). A launcher-based app with all backends shared (web-app-discourse) has no compose services because it runs its container via ./launcher, so its compose was previously invalid. web-app-discourse drops its local _has_local_sidecar gate and just includes base. compose_ca.py (ca-inject) no-ops -- writes an empty {'services': {}} override and exits 0 -- instead of dying with 'No services found after merging' when the merged compose has no services, since there is nothing to CA-inject and the downstream compose up/ps tolerate an empty stack; add test_main_empty_services_writes_noop_override. .ansible-lint-ignore: add the 04_login resolve_host_cid path_join include (a load-failure false-positive that strict ansible-lint profile:min flags). make test is green (lint, integration render, unit).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 03:10:07 +02:00
kevinveenbirkenbachandClaude Opus 4.8 40052446dc refactor(ci): move role-runtime summary into cli.meta.ci.roles.runtime
Replace the single utils/github/profile/role_summary.py script with a
focused CLI package split per responsibility (model, logparse, csvio,
render, github, sources). Output format is selectable via --format
(table default, plus markdown, csv, json); source is auto-detected as an
Ansible log, a role-runtimes CSV, or a GitHub Actions run/job URL (CSV
artifacts downloaded via gh).

The compose workflow now writes the CSV as a downloadable artifact and
appends the per-variant markdown to the step summary, and hard-fails
when the log is missing or carries no profile_roles data so a broken
metrics capture can no longer pass silently.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 03:10:07 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0319d49d26 fix(ci): capture role runtimes from deploy stdout, split per variant
The "Role runtimes" step read /tmp/ansible-<app>.log produced by a
docker cp out of the infinito container. That cp silently failed
(2>/dev/null || true) because the ansible log was not at the expected
in-container path, so role_summary reported "log not found".

The profile_roles table is already on the deploy step's stdout, so tee
it into the file role_summary reads and drop the broken cp (it also
raced the tee on the same path).

role_summary now segments the log by the matrix-deploy round/pass
markers and emits one per-role table per variant round; falls back to a
single combined table when no markers are present.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 03:10:07 +02:00
kevinveenbirkenbachandClaude Opus 4.8 bc76fe0dff feat(ci/variant-bundles): storage-aware job split and explicit variant CSV
Open a new CI runner bundle as soon as it would exceed INFINITO_VARIANT_BUNDLE_SIZE variants OR INFINITO_VARIANT_BUNDLE_MAX_STORAGE (default 400GB) cumulative min_storage, so storage-heavy variants are not stacked onto one runner that then runs too long. Every job now carries its explicit variant CSV in the name; empty only for roles without variants.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 03:03:31 +02:00
kevinveenbirkenbachandClaude Opus 4.8 4b4316871e feat(cli): per-variant and all-role resource summaries for the ressources CLI
meta roles applications ressources gains two summary modes alongside the existing per-service detail table: (1) no --role: one row per role, each showing its heaviest variant (mem_reservation/mem_limit/pids_limit = the max per-variant sum, cpus = max), or the --variant N footprint of every role. (2) --role X without --variant: one row per variant of X (its sum/max). (3) --role X --variant N still renders the detailed per-service table - now the only way to reach it. --format/--order/--filter/--depth/--unshared apply in every mode. The table/JSON renderers are generalized (render_summary_text/json with a key_field) and a variant query accessor is added.

Useful for the CI runner-budget question: the heaviest-variant footprint per role/variant is now directly queryable. Tests cover all three modes (detail/all-roles/per-variant).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-26 03:03:30 +02:00
kevinveenbirkenbachandClaude Opus 4.8 44e798423f test(sys-svc-compose): route swarm_registry_sync fixture through dump_yaml
The no-direct-yaml lint (tests.lint.repository.yaml.test_no_direct_calls) flags any .py calling yaml.safe_load/safe_dump directly. The swarm_registry_sync regression test built its temp compose.yml with yaml.safe_dump; route it through utils.cache.yaml.dump_yaml instead and drop the now-unused yaml import. Includes the ruff line-wrap of the _sync helper signature. make test (incl test-lint) is green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:15:40 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e82fdfe76e fix(sys-svc-compose): de-prefix locally_built so build-siblings are not pulled
A service that builds an image locally produces a <prefix>X image; sibling services may reference that same already-prefixed image without their own build:. swarm_registry_sync compared the de-prefixed upstream against a locally_built set holding the PREFIXED form, so the siblings missed the membership check and fell through to docker pull X for a local-only image with no registry upstream (espocrm/bookwyrm/shopware 'pull access denied'). Store locally_built de-prefixed so it matches upstream.

Add test_swarm_registry_sync.py covering the build-sibling skip, the real-upstream pull, the already-in-registry skip and the empty-prefix short-circuit; the first test fails on the pre-fix logic.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:15:39 +02:00
kevinveenbirkenbachandClaude Opus 4.8 48753de136 feat(deploy-verify): scope required_by coverage to deploy modes (compose/swarm)
The required_by coverage verifier treated every flat 'required_by: {categories/roles}' as required in BOTH deploy modes. But sys-ctl-rpr-container-hard/soft repair compose compositions (DIR_COMPOSITIONS) and gate their body to compose; declaring required_by:[web] flat made the verifier expect them on swarm too, where they correctly skip, so web-opt-rdr-domains swarm CI failed 'required role(s) did not execute'. Let required_by nest under 'compose:'/'swarm:'; the verifier infers the deploy mode from the role set, mirroring DEPLOYMENT_MODE's own groups['svc-swarm-node'] definition, and enforces only the matching mode.

Migrate all 21 existing flat declarations to mirror under both modes (behavior unchanged) and mark sys-ctl-rpr-container-* compose-only. role_has_required_by reads both forms. New lint test_required_by_mode_keys requires the explicit compose:/swarm: form so the flat-vs-gate contradiction can't recur.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 14:59:57 +02:00
kevinveenbirkenbachandClaude Opus 4.8 210c6e551c fix(deploy-verify): strip the ansible log_path line prefix before the required-role check
The required_by coverage verifier (_role_body_executed) reads ANSIBLE_LOG_PATH, but the swarm and compose deploy point that at a log FILE, which ansible prefixes every line with '<ts> p=<pid> u=<user> n=<name> <LVL>| '. That prefix defeated both the line-anchored non-skip status match and the TASK-header block split, so every required role (e.g. sys-ctl-cln-bkps under any svc-bkp deploy) false-positived as 'did not execute' although it actually ran on the manager. Strip the prefix before matching; reproduced locally with a minimal multi-host include and verified the file slice flips while raw-stdout matching is unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 05:43:31 +02:00
kevinveenbirkenbachandClaude Opus 4.8 5dd5c421aa feat(cli): add --run to deploy ci trigger
--run URL selects the run whose results determine the --failed apps;
without it the latest deploy run on the branch is used. Repo + run id are
parsed from the URL.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 05:43:31 +02:00
kevinveenbirkenbachandClaude Opus 4.8 099887f016 fix(container_ports): resolve an unrendered application_id via the templar
When sys-stk-full renders a shared service (web-svc-logout, web-svc-collabora)
it restores application_id="{{ sys_stk_full_application_id }}"; that reaches the
container_ports lookup as the raw, un-templated string (the lookup reads it from
`variables`, not as a templated positional arg), so it raised "Application ID
'{{ sys_stk_full_application_id }}' not found" and broke the swarm deploy of
every app pulling in those shared services. Template it through self._templar,
the way container_depends_on already resolves the vars it reads. Add a
regression test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 05:43:31 +02:00
kevinveenbirkenbachandClaude Opus 4.8 52e601b38d fix(cli): satisfy lint + url-reachability for deploy ci
Drop the unused shebangs (EXE001), mark the PASS emoji constant noqa
S105, annotate the test fixture as ClassVar, and use reserved example.com
hosts / a nocheck marker for the placeholder run URLs so the
url-reachability test stops flagging them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 05:43:30 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0f921a69be feat(cli): add deploy ci status + trigger commands
Add 'infinito administration deploy ci status' and '... trigger' for
inspecting and re-running the compose/swarm deploy-test matrix of a
branch's CI run.

status: per-role docker (compose) and swarm result plus an aggregated
total column (green only when both modes are green), a trailing 'run'
column with the job URLs, --failed [total|swarm|compose] filtering, and
table/string output. Default reads the latest deploy run on the current
branch; --url reads a specific run.

trigger: dispatches entry-manual.yml for the current branch -- all roles
by default, only the last run's failures with --failed [scope], or an
explicit --apps list.

The target repo is resolved from the branch's tracking remote (a fork),
not gh's default repo. Covered by unit tests for run parsing, status
rendering, and trigger dispatch logic.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 05:43:30 +02:00
kevinveenbirkenbachandClaude Opus 4.8 594109018a refactor(compose): publish every container port through the container_ports lookup
Take the host IP as an optional third element of each term
([service, protocol, host_ip]) instead of an ip= kwarg, and build the whole
ports: block from declared ports.local/public (host) + ports.internal
(container). Convert every compose template across 84 roles, folding the
remaining raw var/numeric/specific-ip mappings (including mailu's per-interface
mail ports bound to MAILU_IP4_PUBLIC). Multi-term calls render one term per line.

Add the compose-ports-must-use-container-ports lint: it forbids any raw
"<host>:<published>:<container>" publish item under a ports: block and requires
every term to declare ports.internal. Remove
tests/lint/roles/test_swarm_internal_ports.py, now subsumed.

Fix a container_depends_on rendering regression. The lookup self-indented
(default indent=4), so the added 4-space placement + | indent(4) triple-indented
the block. Flip the lookup default to column 0 (consistent with
container_networks), make every call use the 4-space + | indent(4) form, and add
a lint enforcing it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 03:30:11 +02:00
kevinveenbirkenbachandClaude Opus 4.8 9e732ac9fe feat(test-deploy): declarative per-mode skip to opt apps out of discovery
Add a `skip: [compose, swarm]` field on a role's meta/services.yml primary entity
so an app can opt out of the test-deploy matrix per deploy mode, replacing the
hardcoded exclude regex in apps.sh (the "TODO: Remove exclude condition"). The
filter threads through both discovery paths: the CI workflows pass
INFINITO_DEPLOY_MODE -> apps.sh -> `cli meta roles applications type
--skip-mode`, and roundtrip.sh honours the same get_role_skip SPOT per app/mode.

Apply skip: [compose, swarm] to svc-storage-nfs-server and svc-storage-nfs-client
(NFS storage is swarm-only and set up via the cluster inventory, not the matrix)
and web-opt-rdr-www (the previously regex-excluded app).

Covered by get_role_skip and skip_mode-filter unit tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 21:55:38 +02:00
kevinveenbirkenbachandClaude Opus 4.8 76fac1fbfe fix(lint): filesystem-lint compliance for default.env, Makefile, clean_all.sh
Pre-existing test-lint failures surfaced while greening the suite:
- default.env: move the INFINITO_SWARM_* block into ASC order (it sat before
  INFINITO_LIMIT/LINT instead of after INFINITO_SUBNET) and trim the
  INFINITO_SWARM_TEST_LABEL comment under the 140-char cap.
- Makefile: order act-swarm-clean before act-swarm-down and fold the
  act-swarm-clean comment block to one description line plus Note: markers.
- scripts/tests/deploy/swarm/clean_all.sh: add the executable bit.
- tests/unit/plugins/filter/test_ruby.py: autoformat reflow.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 18:31:36 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6a0ab683df fix(web-app-gitlab): fold GITLAB_OMNIBUS_CONFIG to one line for the swarm env-file
A swarm docker stack deploy env-file value must be a single physical line; the
multi-line GITLAB_OMNIBUS_CONFIG was read as separate whitespace-named env
entries and aborted the deploy. Add a depth-aware ruby_single_line filter that
collapses the Ruby program to one line (statement-boundary newlines at bracket
depth 0 become ';', newlines inside an open bracket or a string become a space
so hash/array literals like the OIDC omniauth_providers stay valid) and apply it
on the swarm branch only; compose keeps the multi-line value unchanged.

Covered by tests/unit/plugins/filter/test_ruby.py (10 cases incl. a realistic
omnibus config, escaped quotes and nested literals).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 18:31:36 +02:00
kevinveenbirkenbachandClaude Opus 4.8 ac63feeea7 fix(web-app-keycloak): re-resolve live CID on swarm reschedule (kcadm TOCTOU)
The role cached keycloak's swarm container ID into KEYCLOAK_LOCAL_CID and reused it for every 'container exec <CID> kcadm.sh'. In swarm the realm-import scale-up reschedules keycloak, so the cached CID dies mid-block; the next kcadm exec aborts with 'No such container' and the module falls into the create branch, surfacing as 'Failed to create Keycloak object' on Update Client settings (15 swarm apps).

Make kcadm resilient: keycloak_kcadm_update.py re-resolves a fresh healthy CID via the swarm service label and retries on 'No such container' (bounded). Quiesce churn first: 05b waits for replicas X/X stable across 3 consecutive polls before re-resolving (stability poll extracted to files/wait_replicas_stable.sh). Add --filter health=healthy to CID resolution in 04_login and 05b. The 26.6.3 pin is unrelated. Swarm-gated; compose runs the module exactly once as before.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 18:31:35 +02:00
kevinveenbirkenbachandClaude Opus 4.8 bc1184dd10 refactor(complexity-cli): split modules, add base/siblings + --unique
Split the single complexity __main__.py into graph (discovery + BFS),
model (ComplexityRow + scoring), render (table/json/string) and cli
(argparse + main); __main__ stays a thin entry point.

Add a 'base' column (sha1 of the role's name unioned with its embedded
services, sorted) so roles in the same mutual-embedding cluster share a
key, and a 'siblings' column listing the rest of that cluster. --unique
keeps one role per base. roundtrip's default app list now passes
--unique to skip redundant same-cluster deploys.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 18:31:35 +02:00
Alejandro Roman 39ad3dc65e refactor test.sh and last comments 2026-06-24 18:31:10 +02:00
Alejandro Roman 09995d134f small refactor fixes 2026-06-24 18:31:05 +02:00
Alejandro Roman 187a954350 fix CI failures after rebase 2026-06-24 18:30:58 +02:00
Alejandro Roman 0153b5cac1 fix CI failures after rebase 2026-06-24 18:30:58 +02:00
Alejandro Roman 4119086c9d Revert "add missing var for multipliers since is not hardcoded anymore"
This reverts commit 9cb13b1fb774aac0dadd121159ba74718aa3ae55.
2026-06-24 18:30:45 +02:00
Alejandro Roman df481a8c22 add missing var for multipliers since is not hardcoded anymore 2026-06-24 18:30:44 +02:00
Alejandro Roman 7dfba97555 add retries for self hosted slower servers 2026-06-24 18:30:44 +02:00
Alejandro Roman aff4a7d158 refactor(svc-runner): fix comments on PR 2026-06-24 18:30:43 +02:00
Alejandro Roman fecdb8cc0a refactor(svc-runner): fix test for runenr 2026-06-24 18:30:41 +02:00
Alejandro Roman 764a1adf30 feat(svc-runners): add the logic to have x numbers of runners self hosted ad connected to GIthub, allowing to have 20 runners from gh + x self hosted 2026-06-24 18:30:31 +02:00