Commit Graph
153 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5.5 50ee46ed22 refactor(provision): group the services disabler modules in a services package
test_prefix_grouped_files flagged cli/administration/inventory/provision/services_disabler.py and services_disabled_conflicts.py, which share the 'services' prefix since the conflict check moved into its own module. Both now live in cli/administration/inventory/provision/services/ as disabler.py and disabled_conflicts.py; every importer, the purge script, the BLE001 per-file ignore and the requirement 037 link follow the new path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 12:23:07 +01:00
kevinveenbirkenbachandClaude Opus 5.5 c3e2114e91 Merge branch 'fix/web-app-wordpress/permalink-options' into feature/store-i18n
Conflicts resolved: the inventory validator keeps store-i18n's custom-role
declaration skip and gains the open-subtree check; the suppression catalogue
keeps both new rows; WORDPRESS_PERMALINK_STRUCTURE gives way to the options
map. WORDPRESS_OPTIONS_OWNED_ELSEWHERE also lists the store-i18n writers the
merge brought along: the design logos (WORDPRESS_DESIGN_LOGOS keys) and the
freedomtranslate options; two custom-role validator tests pass open_paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 11:35:18 +01:00
kevinveenbirkenbachandClaude Opus 5.5 46e0bd1367 feat(web-app-wordpress): manage WordPress options from services.yml and pin the old permalink structure
Deploys since 197d74416d rewrote the blog's permalink structure to
/%postname%/, so links of the form /blog/YYYY/MM/DD/<slug>/ answer 404
(docs/security/user.md:24 fails the URL check).

services.wordpress.options maps WordPress option names to values; the role
reads each option, writes only the ones that drift and flushes the rewrite
rules after a write. permalink_structure is pinned to
/blog/%year%/%monthnum%/%day%/%postname%/, the value WordPress itself sets
for a subdirectory network. A guard refuses options the role writes
elsewhere, secret-named keys and values WordPress would not read back
unchanged.

The inventory validator accepts any child key under a services.yml key that
carries '# nocheck: inventory-open-subtree <reason>'; services.wordpress.options
carries it, everything else stays strict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 11:30:31 +01:00
kevinveenbirkenbachandClaude Opus 5.5 e9779c4c7b refactor(provision): move the disable/inventory conflict check into its own module
services_disabler.py grew past the 500-line limit with the dependents step
(gate pass 16, test_python_file_size). The conflict check moves unchanged to
services_disabled_conflicts.py; the YAML helper it shares becomes public
(load_yaml_mapping_tolerant) and the two importers follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 11:24:38 +01:00
kevinveenbirkenbachandClaude Opus 5.5 1ed3fd11da fix(provision): disable the services gated on a provider the row disables
A tor-only CI row disables web-svc-coturn, which no onion can serve, but the
disabler switched off only service entries keyed coturn. Nextcloud's talk
entry is gated on 'web-svc-coturn' in group_names and variant 0 pins it true,
so Talk stayed on without a TURN server and 01_fullstack.yml's assert stopped
every Nextcloud#0 tor row.

When disable resolves to a provider role P, every entry whose base enabled is
exactly the presence test of P is written false as well, and the reconcile
releases those entries once P is no longer disabled. Negated, compound and
variable gates are left alone. Across all 755 planner row states this changes
one value an app sees: talk on Nextcloud#0 tor rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 10:41:09 +01:00
kevinveenbirkenbachandClaude Opus 5.5 c8625c4592 fix(deploy): pass list extra vars as compact JSON
ansible-playbook splits each -e value through parse_kv on whitespace, so json.dumps' default ', ' separator broke PRIMARY_APPS=["a", "b"] into a raw parameter and the list never reached the play. Dump extra vars with separators=(',', ':') in the development deploy and the swarm matrix deploy, and test that a list extra var survives parse_kv.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:33:12 +01:00
kevinveenbirkenbachandClaude Opus 5.5 167cc22194 feat(inventory): fail validation on required users without a pinned password
A user the host's roles need but the inventory gives no password falls back
to a generated password that renders anew on every read, so password
converges set values nobody knows (echoserver and galaxyserver stopped at
web-app-matomo for that after passing validation). The validator now asks
provision's required_user_policies for each host's groups, merges the
host's users with Ansible's replace precedence and fails before the deploy,
naming the users and the provision command. Presence only: vaulted values
count, nothing is decrypted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 22:09:10 +01:00
kevinveenbirkenbach 5cb5acae6d Merge branch 'fix/lint/resource-budget-honours-pins' into feature/store-i18n 2026-10-31 13:27:42 +01:00
kevinveenbirkenbachandClaude Opus 5.5 76fc6992c4 fix(variants): apply a variant pin only while its carrier is in the round
Pins of every role applied at every round index, so a role the round did not deploy could pull a partner into a heavier variant (web-app-discourse#1 measured 35.46 GB against the 32 GB reservation cap). resolve_round now starts every role on its clamped index and recomputes the include and the pins in force until the assignment settles, raising VariantPinError on a cycle. The planner resolves each round once (39.1 s to 16.0 s for all rounds), affected judges every (role, index) with the round the planner deploys for it, and the resource budget lint measures the pin-aware rounds. The stale web-app-matomo pin above the primary-only docs entry is dropped, and test_variant_pin_fixed_point.py guards every planned round against unsettled pins and pins above a disabled entry. docs/contributing/artefact/files/role/variants.md explains the settle loop with diagrams.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:45:53 +01:00
kevinveenbirkenbachandClaude Opus 5.5 49387850dc fix(deploy): check the deploy extra before running ansible-playbook
The dedicated deploy launched whatever ansible-playbook PATH found. A stale pkgmgr venv without Pillow crashed the galaxyserver deploy after 4h23m at the first branding task. The runner now verifies every distribution of the pyproject deploy extra in the running interpreter before make setup, exits naming the missing ones with the install command, and runs the ansible-playbook next to that interpreter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 11:56:54 +01:00
kevinveenbirkenbachandClaude Opus 5.5 74d99201e0 feat(cache): share package and tool caches across stacks and release swarm clusters after rescue
Parallel compose slots, swarm lab clusters and GUI lanes each kept their own pip, npm and pacman downloads, and a failed swarm run left its cluster (about 25 GB of node docker data plus the rescue bundle in the act container) on the root filesystem until someone ran make swarm-down. INFINITO_CACHE_SHARED_HOST_PATH (default <DIR_CACHE>/shared) is now mounted read-write at the same path into every compose slot and every swarm lab node, with PIP_CACHE_DIR and npm_config_cache below it; empty, the renders stay byte-identical to before. On a host whose os-release ID is exactly arch running an arch stack, INFINITO_CACHE_HOST_PACMAN_PATH mounts the host pacman cache read-only and scripts/docker/cache/pacman.sh makes it the first CacheDir with <shared>/pacman/ second, so new downloads land in the shared folder and the host cache stays untouched; on Manjaro and every other host pacman uses the shared folder alone.

make swarm-zombie without keep=true now copies the act job's rescue diagnostics and Playwright artefacts to the host and then releases the lab nodes, the lab network, the NFS export volume and the act outer container, on success and on failure; when the copy fails the nodes still go, the act container stays and the run exits naming make swarm-down. make roundtrip passes keep to every swarm step, make clean-cache removes the shared folder, and the disc-space cleanup skips pacman cache directories it cannot write. apt and dnf keep their per-container caches behind the package-cache proxy, and no docker data root is shared between daemons. Requirement 065.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 22:04:38 +01:00
kevinveenbirkenbachandClaude Opus 5.5 46962b5110 Merge branch 'feature/updater' into feature/store-i18n
Brings in feature/updater 8c14873225: lockdown-probe with the Bluesky
broker key-rotation fix (0a6b6f9130) and the occ-only trusted_proxies
(406998f43a), the ERPNext server_scripts switch, and main with the
hotfixes #1290 (BBB apps-akka READ_COMMITTED), #1291 (FusionDirectory
PHP_MEMORY_LIMIT) and #1293 (Matrix variant 1 force_verification false).

Conflict resolutions:
- web-app-erpnext: create_site.yml takes updater's include of
  restart_frappe.yml; that file's compose/swarm gates use
  IS_COMPOSE_MODE / IS_SWARM_MODE instead of DEPLOYMENT_MODE literals.
- web-app-mailu, web-app-stalwart vars: store-i18n's RUF from
  users.platform_administrator plus the new RUA vars; dmarc.rua_user is
  platform_administrator (on store-i18n `administrator` is the host
  account), so RUA and RUF share the app mailbox. READMEs follow.
- web-app-stalwart 09_dns-records.yml: store-i18n's block form with the
  alias-domain MX task, plus rua=mailto in the DMARC record.
- web-app-pretix: store-i18n's source build stays; the version bump
  lands as source_version v2026.9.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 10:16:23 +01:00
kevinveenbirkenbachandClaude Opus 5.5 828515483b fix(roles): skip the agent harness directory when copying a local custom role
A custom role with a local source (`source.url: /opt/src/custom`) is copied with shutil.copytree, which skipped only .git and __pycache__. A dev checkout that Claude Code has worked in carries an empty `.claude/.cc-writes` (mode 0700, ignored by the global gitignore), unreadable for the runner uid, so `infinito administration deploy dedicated` aborted before the play with "Custom roles refused: ... [Errno 13] Permission denied". A git source never carries that directory, because it is staged through `git archive`.

The copy now skips `.claude` as well. The ignore list has one consumer, the local branch of stage(); git sources are unchanged. A new test creates `.claude/.cc-writes` at mode 000: before the fix it fails in shutil with the reported Permission denied, after it all 102 custom-role tests pass. Reported by the GUI lane on efd72474a6.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 08:23:20 +01:00
kevinveenbirkenbachandClaude Opus 5.5 c855efc30a refactor(web-app-dashboard): hide a tile through services.dashboard.enabled instead of display
services.dashboard.enabled already says whether a role takes part in the dashboard: roles_with_service lists only roles with enabled and shared as tile-reachability targets, the CSP filter adds the dashboard to frame-ancestors only when it is set, and docker_cards uses it as the card's iframe flag. A separate display key duplicated that switch and left a role with enabled false still rendering a card. docker_cards now skips the card when services.dashboard.enabled is false (a role without a dashboard binding keeps its card, and a value that is not a boolean fails loudly), so display, its validator exception and its roles_with_service filter go again; base.md documents enabled as the per-host switch.

An inventory hides a tile with applications.<role>.services.dashboard.enabled: false, which also drops the dashboard framing of that role. Unit tests cover false, true, a missing binding, a rendered string and a non-boolean; forcing the helper to always show the card fails four of them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 19:00:46 +01:00
kevinveenbirkenbachandClaude Opus 5.5 3ef937a9d8 feat(web-app-dashboard): move the tile switch to services.dashboard.display
The dashboard read `display` from the role's meta/info.yml, which no inventory can override, so a host could not hide a tile. docker_cards now skips a card when `services.dashboard.display` in the merged applications is false, and absent still means shown; a non-bool value fails loudly. roles_with_service drops the same role from the dashboard tile-reachability targets so the Playwright spec does not wait for a hidden tile, and the inventory validator accepts `services.dashboard.display` for every role that declares `services.dashboard`, since no role default carries it. `display` leaves the meta/info.yml allowlist, web-app-jellyfin drops its redundant `display: true`, and base.md and layout.md document the new switch.

Real lookups with web-app-gitea set to display false render the Dashboard and Jellyfin cards and only jellyfin as a tile target, and all three without the override; the nine tests for the new behaviour fail against the old code.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 17:46:03 +01:00
kevinveenbirkenbachandClaude Opus 5.5 c9e7f950d9 Merge branch 'feature/updater' into feature/store-i18n
68 conflicts, resolved per hunk; resolutions agreed with the Updater
session were applied as agreed:

- svc-db-postgres (superuser password script and core) and
  sys-svc-rdbms realign_dedicated_db: updater's argv/stdin redesign with
  expand_argument_vars: false, nested into store-i18n's block structure.
- svc-db-redis 01_init: store-i18n's ACL password on stdin.
- sys-svc-proxy forwarded.conf.j2: store-i18n's handover-aware file,
  without the forwarded_base include (Host and X-Forwarded-Proto would
  be sent twice); forwarded_base serves the inline consumers, e.g.
  web-app-seaweedfs nginx.conf.j2.
- basic.conf.j2 and web-app-openproject proxy.conf.j2: issuer redirect,
  then handover, else updater's location/sso_acl.conf.j2; the
  server-level webserver_extra_configuration hook stays.
- web-app-keycloak: updater's admin-REST settings step and renumbered
  update steps; update/01_client.yml removed; store-i18n's design step
  and LDAP block kept; a73435cc9d arrived through rename detection.
- web-app-nextcloud OpenProject provisioning: updater's rewrite, all
  credentials on the Ruby runner's stdin.
- web-app-semaphore: store-i18n's canonical web root (a1d7835863).
- Role READMEs: no generated Cosmos/Architecture/Quick Setup/Credits
  sections; desk-* paths resolved to the dsk-* renames.

Integration fixes for what merged cleanly but did not fit together:

- web-app-matrix ansible vars: drop updater's second
  matrix_client_element_configuration_extension_json, which silently
  overrode store-i18n's design-aware value.
- url.origin instead of a stripped url.base in 19 places store-i18n
  added since 44aaa68924, same lookup targets (the double-slash lint).
- Package batch (18ea75049e, cb4d340de9): installs the deploy-start
  batch covers are removed (caffeine, cursor, blender, npm in dsk-gnt-*,
  with shared references), the rest carry batch: false and a nocheck
  with the verified reason (PyPI fallback, before the batch, vendor
  repository, configured flavor only, addon gate); dev-npm references
  the root npm id.
- sys-token-store write_many.yml: store-i18n's change-gated block and
  IS_SWARM_MODE.
- web-app-openproject: IS_COMPOSE_MODE in 02_migrate/run.yml; the
  seeded-admin lock (ccbf8f168d) ported into 04_admin/provision.yml and
  covered by the admin-state digest.
- Bucket policy (3b4ac40df5 and follow-ups) moved to web-svc-seaweedfs,
  where store-i18n's object store lives; the S3 admin secret travels on
  stdin instead of a task-level environment.
- act workflow.sh: no Exception-labelled comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 02:28:16 +01:00
kevinveenbirkenbachandClaude Opus 5.5 06a899df0d fix(inventory): pin the passwords of every bot and account holder, deployed role or not
Stalwart, Mailu and LDAP provision every bot and account holder of the whole users catalogue, but the generator pinned only the users of the resolved roles. The bots of undeployed roles (no-reply, blackhole, bounce, newsletter, helpdesk, contact) fell back to strong_password, got a new secret on every deploy, rewrote the token store six times per run and left email.password consumers on a stale value. The generator now also pins every catalogue user holding an account or the bot role unless a role states its password outright, and reads policies independent of declaration order; existing pins are never overwritten.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 01:31:15 +01:00
kevinveenbirkenbachandClaude Opus 5.5 d92375922d feat(roles): pull custom roles from inventory-declared sources
An inventory group web-app-cst-<name> with applications.<id>.custom: true and
a pinned source {url, version, path} is pulled into roles/ before the play,
read raw (Jinja, Ansible tags and credential URLs refused), placed through a
staging directory under an exclusive lock, and owned by a marker that names
its own directory, so a pull or roles-clean never replaces or removes a
directory it did not place and a failed pull leaves roles/ as it was.
Directory inventories, host lists and non-UTF-8 files behave as Ansible
reads them; generated test deploys pass --skip-pull. make roles-pull,
roles-clean and roles list, INFINITO_EXTRA_MOUNTS for local sources, the
web-app-cst category, the reserved prefix lint and a central
repository_files() that also yields the gitignored custom roles let every
lint scan them like in-tree roles. Supersedes the uncommitted 045 External
Roles draft and reuses its fetch, sync and mount design.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 00:02:58 +01:00
kevinveenbirkenbachandClaude Opus 5.5 764b69267c fix(recover): restore an NFS generation onto the export base it was taken from
svc-bkp-nfs-2-local snapshots the export base (/srv/nfs), so a
generation's files/ root already holds infinito-state/. Without an
explicit target, `recover nfs` and `recover full` restored it onto
/srv/nfs/infinito-state: the tree came out as
infinito-state/infinito-state/<volume>, and `rsync --delete` wiped every
live volume directory and the backup root under the old target.

- paths.NFS_EXPORT_BASE = get_export_base(), the function behind
  lookup('nfs_export_base') that also feeds the backup unit's source,
  replaces NFS_EXPORT_STATE. NfsRecoverer defaults to it; an explicit
  `snapshot:/abs` target still wins, a remote target is still prefixed
  with its host. `recover full` inherits the default.
- recover_drill.sh seeds its marker at files/infinito-state/<volume>/
  and asserts it under the export base, so the drill can no longer pass
  on a nested layout.
- The help examples, schema.yml and the CLI README show the export base;
  the documented subtree example would have nested too and now names the
  matching snapshot subtree.

Reproduced on a temp tree with the role's real script.sh and recover.py
and the CLI's argv: the old default loses every live volume and the
backup root, the new one restores in place. The new tests fail 6 of 6
against the old default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 17:56:26 +01:00
kevinveenbirkenbachandClaude Opus 5.5 68b0dfd508 feat(ci): drop the dot between the category glyphs and the role title
Job and run titles read `🐳🧅…𓏴𓏛·Nextcloud#2`; the `·` between the
hieroglyph block and the title only costs width. RoleDisplayName.encode
now joins the two directly (`𓏴𓏛Nextcloud#2`). Spaces inside a README
heading still become `·`, which keeps a display name one token in the
space-separated whitelist and priority inputs.

decode and decode_list still map the dotted form of older runs back to
the role id, so report_failures, the run tooling and selection tokens
pasted from an earlier run keep working. Old and new names cannot
collide: a glyph block holds only hieroglyphs and a title starts with a
letter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 17:16:18 +01:00
kevinveenbirkenbachandClaude Opus 5.5 78678d220f fix(inventory/validate): keep the keys after a vaulted value
The validator cut every !vault block with a regex that ran up to the next line at column 0, so the keys after a vaulted password vanished, a whole user among them. utils.cache.yaml has parsed the !vault tag itself since c4556fe72d, so the loader now hands the file to it unchanged and a vaulted value loads as its ciphertext. An inventory-only user that brings its own username no longer warns as unknown.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-28 15:33:49 +01:00
kevinveenbirkenbachandClaude Opus 5.5 91022ad37a fix(ci): let only the primary app's declared variant bring the guide host
Every role's base services carry `docs.enabled: "{{ 'web-app-docs' in
group_names }}"`, and the static closure counts any "in group_names" flag
as possibly true (is_explicit_truth). A round therefore pulled web-app-docs
in through any member whose active variant did not pin docs off, and a
round hands every role its index or clamps it to variant 0, where 84 roles
declare docs on. The README replay and everything it drags in (dashboard,
matomo, keycloak, prometheus with its MCP partners) landed in 365 of 469
rounds instead of one per role, which ran CI run 40625679270 out of
resources: Listmonk#0 memory, OpenProject#0 I/O in the DR drill,
Nextcloud#0/#2 budget (4310-4539 s in the replay each, 60 s over),
BookWyrm#0 drill disk, Matrix#1/#3 pids in the replay; none of those
primaries declares docs in the failing variant.

The guide host now stays on only where the primary app's own variant
declares it (declares_guide, the predicate instructions.py already used);
every other role has it switched off (guide_off), in the planner's round
overrides and in the swarm include set. The host_vars bake drops a docs
edge whenever the round does not deploy web-app-docs. Measured for
listmonk, openproject, translate and mariadb: web-app-docs is deployed in
exactly each primary's declared guide round, swarm and planner agree in
all 12 rounds, and rounds shed up to ten apps (mariadb#1 19 -> 9).

test_guide_rides_the_primary_only pins that on real role data (it fails
with the rule disabled); test_variant_pin_required leaves the guide host
out of its clone comparison and no longer counts the docs key as a
pairing edge, because the replay no longer follows a partner's index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 12:49:51 +01:00
kevinveenbirkenbachandClaude Opus 5.5 4eb4c0fe22 fix(inventory): bake the variant a pin forces, as the round's include already does
services_overrides_for_round applied the # variant-pin: markers when it resolved a round's include set, while plan_dev_inventory_matrix built round_variants, which the inventory bake, the deploy CLI and the swarm matrix read, by clamping the round index alone. Once docs variant 0 pinned web-app-dashboard#reciprocal, round 2 of a sweep resolved dashboard variant 1 (SSO off, no web-app-keycloak in the inventory) but baked variant 0 (SSO on): Keycloak came up without its own bake, defaulted totp_enabled to true, and every dashboard login stopped at CONFIGURE_TOTP while the Keycloak card was missing (11 jobs of run 40042690056).

active_variants_for_round holds the clamp-plus-pins logic once; services_overrides_for_round and the planner both call it, and the planner keeps the primary apps' indices on top.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 19:18:53 +02:00
kevinveenbirkenbachandClaude Opus 5.5 685ec790eb fix(inventory): count the disable= record as read by the deploy
0327114c44 made the provisioning CLI keep the keys of the last `disable=` run in host_vars under _INFINITO_SERVICES_DISABLED and read them back on the next run. The inventory validator from 285ce910d2 only collects keys that group_vars, role defaults or vars declare, that templates guard, or that plugins read by name, so a key read through a constant in cli/ counted as unread, and every local compose-deploy with disable= aborted with "Inventory validation failed" before the play (repro: make compose-deploy mode=reinstall apps=web-app-keycloak disable=matomo variant=0).

read_variables() now includes DISABLED_SERVICES_RECORD from the disabler module, so the name has one source. An inventory carrying the record validates; an unknown key next to it is still reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 15:18:24 +02:00
kevinveenbirkenbachandClaude Opus 5.5 5725fe6c20 fix(cache): keep the vault tag when utils.cache.yaml loads a !vault block
utils.cache.yaml registered its own !vault constructor that returned a plain str, and it overrides the VaultScalar constructor of utils.handler.vault whenever it registers last. Every writer that loads a host_vars file through the cached loader, changes one key and dumps it back then re-emitted each credential as an untagged quoted string, and Ansible handed the $ANSIBLE_VAULT block itself to the roles.

08680e933d and c88647b3a1 moved tor_node.py and network_mode.py to the round-trip helpers. The shared loader now returns a VaultScalar, so every yaml_io writer keeps the tag regardless of import order. VaultScalar subclasses str, so the other readers of the cached loader see the same value.

test_a_round_trip_keeps_the_vault_tag fails against the previous constructor: re-registering the plain-str constructor on CSafeLoader and SafeLoader drops '!vault |' from the dumped file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 12:23:45 +02:00
kevinveenbirkenbachandClaude Opus 5 c88647b3a1 fix(inventory): keep vault tags when provisioning the network mode
apply_network_mode_from_env loaded the whole host_vars file through
utils.cache.yaml, set one scalar and wrote everything back. That loader's
!vault constructor returns a plain string, so every credential in the file
was re-emitted untagged and Ansible handed the raw ciphertext to the roles
instead of the secret. It only runs when the row sets `network`, which is
every multi-network round.

Run 40032069040 on the fork shows what that costs: the generated host_vars
of a failing job carries 0 `!vault` tags against 56 `$ANSIBLE_VAULT`
headers, mariadb then refuses root with 1045 on a freshly initialised
database, and the openldap bind fails with result 49.

08680e933d fixed the same class in tor_node.py, the sibling writer in this
package. This one writes through the round-trip helpers too.

The existing test did not cover the case, which is why the sibling fix did
not stop it: test_preserves_existing_vault_credentials now fails against
the previous writer with
`'!vault |' not found in "... root_password: '$ANSIBLE_VAULT;1.1;AES256`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-24 01:24:35 +02:00
kevinveenbirkenbachandClaude Opus 5 acfc79db94 fix(merge): green the gate after the updater merge
Git conflicts only on lines both sides touched, so the feature/updater
merge dropped symbols and markers silently where one side alone had moved
them. Execution found them, not review.

- plugins/lookup/image.py bound get_entity_name while its test patches
  plugins.lookup.image.entity_name; an alias satisfies an importer but not
  mock.patch, which reads the module's own binding. The module now uses
  entity_name, the spelling 60+ call sites share. The alias stays for
  utils/update_check and plugins/filter/update, whose tests patch the old
  name.
- A download-source package resolves at its pinned URL, never in a
  distribution index, so the availability probe asked the wrong oracle and
  reported cursor absent. It now issues a HEAD against the declaration's
  URL. The CDN answers 403 rather than 404 for an object that is gone, so
  both count as absent.
- The validate suite still guarded the name "design", which moved into
  roles/web-svc-design/meta/services.yml in 86cefeff26.
- 193 meta/services.yml and meta/variants.yml declare why the guide edge
  is not a variant dimension of their role, and why docs rides each role's
  smallest round: the README replay is the costliest partner in the matrix.
  No new CI round comes out of it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-23 19:43:54 +02:00
kevinveenbirkenbach a04f2c9895 Merge branch 'feature/updater' into feature/store-i18n 2026-10-23 17:32:32 +02:00
kevinveenbirkenbachandClaude Opus 5.5 285ce910d2 feat(inventory): reject unread keys and a missing Cloudflare token before the deploy
The Keycloak stage on echoserver ran ten minutes before sys-svc-dns stopped
it with "API.cloudflare.api_token must be set": the inventory still kept the
token under the flat name CLOUDFLARE_API_TOKEN, which nothing reads any
more. The validator that runs before every deploy now catches that class.

- api.py reports a host that selects DNS_PROVIDER: cloudflare without
  API.cloudflare.api_token, with group_vars/all merged under the host_vars
- variables.py reports every top-level key of host_vars and group_vars that
  core does not read: not declared in group_vars or a role's defaults or
  vars, not tested with "is defined" or "| default(", not read by name in a
  plugin, and not a SYSTEM_EMAIL_<KEY> the email lookup builds; a flat
  credential name such as CLOUDFLARE_API_TOKEN names its API.<provider>.<key>
  path; ansible_* keys are exempt

A key that is read in a way the scan cannot see takes
"# nocheck: inventory-variable", documented in suppression.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 16:18:12 +02:00
kevinveenbirkenbachandClaude Opus 5 3cc0214c28 fix(deploy): stop the dedicated deploy from git-cleaning its own checkout
`MODE_CLEANUP` says what to tidy on the deployed system: prune unused Docker
resources, clear Mastodon caches, switch an LDAP user list from `present` to
`exact`. Eight roles read it that way. The dedicated runner read the same flag
and ran `make clean` in the control repository instead, which is
`git clean -fdX` over every ignored path there.

`inventories/production/` is one of those paths. The published host install
guide provisions into it and then deploys against `$INVENTORY/.password`, so
the deploy deleted the vault password it was about to use:

    Removing inventories/production/
    [ERROR] The vault password file .../inventories/production/.password was
    not found

Reproduced in CI on two unrelated rows, arm64/zfs/debian and
amd64/btrfs/centos, so it is neither architecture nor filesystem specific.

The coupling was a name collision, not a decision. 7e5990aa16 generated the
MODE_* flags from `01_modes.yml` and removed the legacy skip flags, and the
repository clean - which had had a flag of its own - was attached to the
generated `MODE_CLEANUP` because one now existed by that name. No other deploy
path does this; `container`, `development` and `swarm` never call `make` at
all.

`make clean` keeps its own entry points, `make setup-clean`, `make clean-sudo`
and `infinito --git-clean`, where a caller asks for it deliberately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-23 10:59:05 +02:00
kevinveenbirkenbachandClaude Opus 5 677a63931c Merge branch 'feature/network/multi-network-serving' into feature/store-i18n
Resolves 68 conflicted files. The governing rule was our structure with their
network semantics: the tor boolean becomes a network mode (clearnet | tor |
multi) throughout the variant layer, while this branch's architecture and mesh
axes, its extracted modules and its i18n content stay.

Where the two sides covered different ground, both are kept rather than one
picked:

- ca_injected answers a ca_client on the deployment's TLS mode OR on an SSO
  issuer that resolves self-signed. 3a3e9ed2c9 records why: an issuer is not
  an object store, so keeping only one side drops the other's coverage.
- get_merged_domains keeps the loud refusal when svc-net-tor is deployed
  without a provisioned node, now gated on their tor_serves_host; their
  node_onion_of returns an empty string there, which would have served every
  onion-exclusive app on clearnet in silence.
- sso_cookie_domains stays the single source for oauth2-proxy's cookie scope.
  Their per-network branch omitted the issuer domain on the onion arm, which
  breaks the callback leg.
- The clearnet CSP batch carries --clearnet-timeout again; their batch dict
  hardcoded zero while the role still passes the flag.
- The dashboard keeps its i18n content dir, image, version and design
  resolution; their side reduced the logo sources to the bare service
  provider values.

Their side wins where it is measurably better: the onion mirror rewrite is
gated per vhost rather than per app, the Playwright onion decision reads the
pass's own domain instead of the role's canonical one, and the matomo
reinjection and dashboard config render per network sibling.

Eight files were broken by the merge without ever carrying a conflict marker,
because git marks only the lines both sides touched:

- utils/cache/domains.py lost the onionize block that sat between two hunks
- roles/sys-front-inj-all/templates/body_filter.lua.j2 called it
- web-app-bluesky, keycloak sso_proxy: the get_entity_name filter was renamed
- trigger/__main__.py, playwright_coverage.py and its test: the deleted tor
  module
- cli/administration/deploy/ci/timings.py read Label.tor
- plugins/filter/csp_filters.py called _aligned_url with four of six arguments
- tests/unit/python/utils/cache/test_domains.py referenced a renamed class
- tests/.../sys-ctl-hlth-csp/files/test_script.py mocked a function main no
  longer calls

Verified: the variant layer (155 tests, 65 subtests), ca_injected (18),
the CI layer (175), the domains cache including the restored guard (32),
retrigger inputs (9), timings and the CSP health script (53). make
quality-high clears autoformat; its remaining findings are tracked separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-23 09:53:41 +02:00
kevinveenbirkenbachandClaude Opus 5.5 05135a7cee feat(svc-net-wireguard): let a compose deployment name a hub with roaming spokes
Without a swarm manager the mesh planner only built full meshes, whose
handshake gate waits for every peer, so laptops behind NAT could not join.
An inventory may now name a hub in networks.hub: the planner builds every
declared mesh around it, the hub forwards between its spokes, a spoke marked
networks.roaming gets no endpoint in the hub's peer list and dials the hub
with PersistentKeepalive, and the hub's gate skips roaming spokes while a
roaming spoke gates only on the hub. The mesh CLI takes the hub's public
address per host (--endpoint) and a vault password per host
(--host-vault-password-file). Full meshes and the swarm manager hub plan
exactly as before; NETWORK_MESH_POOL keeps the operator's subnet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 10:56:50 +02:00
kevinveenbirkenbachandClaude Opus 5.5 7ac49234ca test(inventory): read the width test's tempdir file without the cache
The file is written by the test itself, so the path-keyed read cache does
not fit; opt the read out of the uncached-read lint instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 09:47:59 +02:00
kevinveenbirkenbachandClaude Opus 5.5 46f32392b9 fix(inventory): stop the inventory writers from folding long lines
ruamel folds every scalar and flow collection past 80 columns, so each
run of the provision, credentials or mesh CLI broke SSH keys and alias
lists in host_vars over several lines. round_trip_yaml() now sets an
unlimited width, and the credentials CLI and the services disabler use it
instead of building their own handler.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 09:41:11 +02:00
kevinveenbirkenbach 396c937033 Merge branch 'main' into feature/store-i18n
# Conflicts:
#	cli/build/docs/integration_matrix/__init__.py
#	tests/lint/ansible/roles/meta/__init__.py
#	utils/github/variant/axes.py
2026-10-21 00:09:53 +02:00
kevinveenbirkenbachandClaude Opus 5 fbd4b3a18d refactor(meta): label categories with one hieroglyph
A deploy job name repeats the category path on every row of the matrix. One Egyptian hieroglyph costs four UTF-8 bytes where the two Chinese characters it replaces cost six. Unicode names each sign by its Gardiner code, so the opaque value stays identifiable without a comment per entry.

A label has to stay one codepoint: a sign written with a variation selector costs seven bytes and is worse than what it replaced. tests/integration/meta/categories/test_glyph.py enforces that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-20 23:17:19 +02:00
kevinveenbirkenbach fdb6e668b2 Merge branch 'main' into feature/network/multi-network-serving
# Conflicts:
#	utils/github/variant/axes.py
2026-10-20 22:56:45 +02:00
kevinveenbirkenbachandClaude Opus 5.5 6e776a5780 refactor(python): clear the open CodeQL code-quality alerts
- Declare `__all__` in every package `__init__.py` whose `PROJECT_ROOT`
  is imported by its modules (163 packages), so the constant counts as
  exported; list every public name where the package defines more.
- Empty the 15 package `__init__.py` files whose `PROJECT_ROOT` no module
  imports.
- Declare the public API of `utils.github.variant.axes` in `__all__`,
  re-exports included, instead of silencing them with `noqa: F401`; the
  list matches the one `feature/store-i18n` already carries.
- Use `import unittest.mock as mock` in `test_applications.py`, as the
  rest of the suite does.
- Spell `_NON_PRINTABLE` in `utils/yaml_bootstrap.py` as the equivalent
  BMP-only positive class; it matches exactly the same 2111 code points
  as the old negated class across U+0000..U+10FFFF.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 21:40:58 +02:00
kevinveenbirkenbach d0b45f8b2d Merge branch 'main' of github.com:infinito-nexus/core into feature/store-i18n
# Conflicts:
#	.github/workflows/call-images-build-ci.yml
#	.github/workflows/call-test-deploy.yml
#	.github/workflows/call-test-install-pkgmgr.yml
#	.github/workflows/call-test-workspace.yml
#	roles/svc-bkp-local-2-device/README.md
#	roles/svc-storage-nfs-client/README.md
#	roles/web-app-pixelfed/meta/services.yml
#	scripts/tests/deploy/act/workflow.sh
#	tests/unit/python/utils/recovery/test_base.py
2026-10-20 20:05:15 +02:00
kevinveenbirkenbachandClaude Opus 5.5 21eba1e88f fix(backup): carry hard links, ACLs and xattrs on every backup and recover hop
baudolo 7.1.x writes volume generations with `rsync -aHAX`, so the
producing host keeps `user.*` extended attributes (OpenCloud keeps its
storage metadata in `user.oc.*`). Every later copy dropped them again,
because none of the other rsync calls passed `-X` (nor `-A`/`-H`):

- NFS export snapshots (svc-bkp-nfs-2-local `script.sh`),
- the backup host's pull (svc-bkp-remote-2-local `pull_specific_host.py`),
- the device sync (svc-bkp-local-2-device `script.py`),
- every recover into a live target (`utils/recovery/base.py`, used by the
  volume, NFS and device recover scripts),
- the `recover full` and `recover remote` tree pulls.

A disaster recovery through the documented chain therefore restored
OpenCloud without `user.oc.*`, silently: the copies succeeded and the
dirval stamp does not cover attributes.

All of these now run `rsync -aHAX` (`-aHAXP` for the device sync). A
target filesystem without xattr support makes rsync exit non-zero as soon
as a source file carries attributes, so the hop fails loudly instead of
dropping them; there is no per-device opt-out. The single-image pull in
`recover remote` keeps `-a`. The provider's ssh-wrapper needs no change:
its glob accepts any server flags and its character guard allows them.

Two tests write a `user.oc.id` attribute and assert it survives the NFS
snapshot and the recover; with the old `-a` both fail with
`OSError: [Errno 61] No data available`.

Closes #1021

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 18:20:46 +02:00
kevinveenbirkenbach feb5630f19 Merge remote-tracking branch 'origin/main' into feature/network/multi-network-serving 2026-10-20 17:14:44 +02:00
kevinveenbirkenbachandClaude Opus 5.5 310663ae8c feat(compose): recreate the stack network without losing volumes or worktree stacks
When the IPAM config of a compose project's network changes (#999 added
`ip_range`), compose recreates the network on the next `up`. With a
foreign endpoint attached, as every worktree stack is on the primary's
network, compose first stops the project's own containers, then fails
with `error while removing network: ... has active endpoints`, and a
repeated `up` or `up --force-recreate` does not recover. On 2026-10-19
this took the shared registry/package cache and coredns down for every
local session.

`make compose-network-recreate` (alias `i8conere`) runs the sequence that
recovered it: detach every running container of another project from the
network, `docker compose down --remove-orphans` without `-v` so the
caches keep their volumes, the normal `up`, then attach the detached
containers again, which draw addresses from the dynamic pool. If `up`
fails, the still-detached containers are listed instead of reattached.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 13:45:37 +02:00
kevinveenbirkenbachandClaude Opus 5 5647a2d2bd style: take the line breaks autoformat wrote
cfe20f0726 and 4915a401d4 wrapped three statements more cautiously than
the formatter wants. Without this the lint phase rewrites the same diff
into the tree on every run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-19 03:41:36 +02:00
kevinveenbirkenbachandClaude Opus 5 4915a401d4 fix(ci): replay the mesh state of the row that failed
failed_selections reproduced mode, onion state, distro and architecture
but not the mesh axis. These tokens feed the priority line, where an
unpinned axis offers both states, so a failed direct swarm row came back
as itself and as a meshed sibling: two jobs to reproduce one.

Only a mode the axis covers takes the pin. pinned_vpn_states returns an
empty list for a compose row pinned to the mesh, and an empty list drops
the row from the matrix -- the class ff75343aba already had to fix once
for held runs.

The fixture had drifted from what the matrix emits: row_label built
titles with no mesh glyph while axes.assign writes one for every swarm
and compose row, so parse_label fell back to its False default and the
new &direct assertion would have been right by accident. row_label and
deploy_job_name now place the glyph under the same condition as
axes.assign, and a meshed row has a case of its own, which nothing
covered before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-19 03:25:39 +02:00
kevinveenbirkenbach 15930945fe Merge branch 'main' into feature/store-i18n 2026-10-19 01:54:37 +02:00
kevinveenbirkenbachandClaude Opus 5 67dd278d0c fix(deploy): forward the guide host into the container Ansible runs in
e17d182d0a gave web-app-docs a switch that turns the language assertion
off on a row where docs only rides along, and read it from the
guide_host environment variable. The deploy workflow sets that variable
on the runner, but a compose deploy runs Ansible inside the infinito
container, where only _FORWARDED_ENV_KEYS arrives. guide_host was not in
it, so the lookup resolved empty, the comparison against the row's own
id was true everywhere, and the switch never fired.

Run 39137606155 shows both halves: the passenger rows 117711219652 and
117711220306 carry guide_host: web-app-docs and each spent about 7000 s
of a 20400 s budget in that one assertion, while the docs row itself,
117711219660, carries an empty guide_host and therefore keeps running
it. Forwarding the variable preserves that split rather than silencing
the assertion everywhere.

APP_ID is in the tuple for the same reason and feeds the guide_role
lookup from the same workflow step.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-19 01:10:57 +02:00
kevinveenbirkenbachandClaude Opus 5.5 7f28564548 Merge branch 'main' into feature/network/multi-network-serving
Combine main's WireGuard mesh axis with this branch's network axis in the CI
variant code. Every matrix row now carries both: network (clearnet, tor,
multi) from utils.github.variant.network and vpn (meshed or direct) from
utils.github.variant.vpn. Label parsing, which main moved to label.py, reads
the network glyph instead of the old tor flag and keeps the vpn glyph; the
selection token, plan table, matrix and trigger CLIs, the workflows and the
swarm routine forward both axes. The test of an incapable priority variant
now compares the set of network modes, since a swarm priority row also
covers both mesh arms.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 00:15:05 +02:00
kevinveenbirkenbach da449f8894 Merge pull request #323 from evangelostsak/feature/svc-net-wireguard
feat(svc-net-wireguard): Dockerized WireGuard role + test-e2e-cli DinD e2e
2026-10-18 19:50:42 +02:00
kevinveenbirkenbach af24e1f920 Merge branch 'main' into feature/network/multi-network-serving 2026-10-18 15:28:01 +02:00
kevinveenbirkenbach 4a469e6476 Merge branch 'main' into feature/store-i18n 2026-10-18 13:20:06 +02:00