svc-bkp-nfs-2-local snapshots the export base (/srv/nfs), so a
generation's files/ root already holds infinito-state/. Without an
explicit target, `recover nfs` and `recover full` restored it onto
/srv/nfs/infinito-state: the tree came out as
infinito-state/infinito-state/<volume>, and `rsync --delete` wiped every
live volume directory and the backup root under the old target.
- paths.NFS_EXPORT_BASE = get_export_base(), the function behind
lookup('nfs_export_base') that also feeds the backup unit's source,
replaces NFS_EXPORT_STATE. NfsRecoverer defaults to it; an explicit
`snapshot:/abs` target still wins, a remote target is still prefixed
with its host. `recover full` inherits the default.
- recover_drill.sh seeds its marker at files/infinito-state/<volume>/
and asserts it under the export base, so the drill can no longer pass
on a nested layout.
- The help examples, schema.yml and the CLI README show the export base;
the documented subtree example would have nested too and now names the
matching snapshot subtree.
Reproduced on a temp tree with the role's real script.sh and recover.py
and the CLI's argv: the old default loses every live volume and the
backup root, the new one restores in place. The new tests fail 6 of 6
against the old default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Job and run titles read `🐳🧅…𓏴𓏛·Nextcloud#2`; the `·` between the
hieroglyph block and the title only costs width. RoleDisplayName.encode
now joins the two directly (`𓏴𓏛Nextcloud#2`). Spaces inside a README
heading still become `·`, which keeps a display name one token in the
space-separated whitelist and priority inputs.
decode and decode_list still map the dotted form of older runs back to
the role id, so report_failures, the run tooling and selection tokens
pasted from an earlier run keep working. Old and new names cannot
collide: a glyph block holds only hieroglyphs and a title starts with a
letter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every role's base services carry `docs.enabled: "{{ 'web-app-docs' in
group_names }}"`, and the static closure counts any "in group_names" flag
as possibly true (is_explicit_truth). A round therefore pulled web-app-docs
in through any member whose active variant did not pin docs off, and a
round hands every role its index or clamps it to variant 0, where 84 roles
declare docs on. The README replay and everything it drags in (dashboard,
matomo, keycloak, prometheus with its MCP partners) landed in 365 of 469
rounds instead of one per role, which ran CI run 40625679270 out of
resources: Listmonk#0 memory, OpenProject#0 I/O in the DR drill,
Nextcloud#0/#2 budget (4310-4539 s in the replay each, 60 s over),
BookWyrm#0 drill disk, Matrix#1/#3 pids in the replay; none of those
primaries declares docs in the failing variant.
The guide host now stays on only where the primary app's own variant
declares it (declares_guide, the predicate instructions.py already used);
every other role has it switched off (guide_off), in the planner's round
overrides and in the swarm include set. The host_vars bake drops a docs
edge whenever the round does not deploy web-app-docs. Measured for
listmonk, openproject, translate and mariadb: web-app-docs is deployed in
exactly each primary's declared guide round, swarm and planner agree in
all 12 rounds, and rounds shed up to ten apps (mariadb#1 19 -> 9).
test_guide_rides_the_primary_only pins that on real role data (it fails
with the rule disabled); test_variant_pin_required leaves the guide host
out of its clone comparison and no longer counts the docs key as a
pairing edge, because the replay no longer follows a partner's index.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
services_overrides_for_round applied the # variant-pin: markers when it resolved a round's include set, while plan_dev_inventory_matrix built round_variants, which the inventory bake, the deploy CLI and the swarm matrix read, by clamping the round index alone. Once docs variant 0 pinned web-app-dashboard#reciprocal, round 2 of a sweep resolved dashboard variant 1 (SSO off, no web-app-keycloak in the inventory) but baked variant 0 (SSO on): Keycloak came up without its own bake, defaulted totp_enabled to true, and every dashboard login stopped at CONFIGURE_TOTP while the Keycloak card was missing (11 jobs of run 40042690056).
active_variants_for_round holds the clamp-plus-pins logic once; services_overrides_for_round and the planner both call it, and the planner keeps the primary apps' indices on top.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0327114c44 made the provisioning CLI keep the keys of the last `disable=` run in host_vars under _INFINITO_SERVICES_DISABLED and read them back on the next run. The inventory validator from 285ce910d2 only collects keys that group_vars, role defaults or vars declare, that templates guard, or that plugins read by name, so a key read through a constant in cli/ counted as unread, and every local compose-deploy with disable= aborted with "Inventory validation failed" before the play (repro: make compose-deploy mode=reinstall apps=web-app-keycloak disable=matomo variant=0).
read_variables() now includes DISABLED_SERVICES_RECORD from the disabler module, so the name has one source. An inventory carrying the record validates; an unknown key next to it is still reported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
utils.cache.yaml registered its own !vault constructor that returned a plain str, and it overrides the VaultScalar constructor of utils.handler.vault whenever it registers last. Every writer that loads a host_vars file through the cached loader, changes one key and dumps it back then re-emitted each credential as an untagged quoted string, and Ansible handed the $ANSIBLE_VAULT block itself to the roles.
08680e933d and c88647b3a1 moved tor_node.py and network_mode.py to the round-trip helpers. The shared loader now returns a VaultScalar, so every yaml_io writer keeps the tag regardless of import order. VaultScalar subclasses str, so the other readers of the cached loader see the same value.
test_a_round_trip_keeps_the_vault_tag fails against the previous constructor: re-registering the plain-str constructor on CSafeLoader and SafeLoader drops '!vault |' from the dumped file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
apply_network_mode_from_env loaded the whole host_vars file through
utils.cache.yaml, set one scalar and wrote everything back. That loader's
!vault constructor returns a plain string, so every credential in the file
was re-emitted untagged and Ansible handed the raw ciphertext to the roles
instead of the secret. It only runs when the row sets `network`, which is
every multi-network round.
Run 40032069040 on the fork shows what that costs: the generated host_vars
of a failing job carries 0 `!vault` tags against 56 `$ANSIBLE_VAULT`
headers, mariadb then refuses root with 1045 on a freshly initialised
database, and the openldap bind fails with result 49.
08680e933d fixed the same class in tor_node.py, the sibling writer in this
package. This one writes through the round-trip helpers too.
The existing test did not cover the case, which is why the sibling fix did
not stop it: test_preserves_existing_vault_credentials now fails against
the previous writer with
`'!vault |' not found in "... root_password: '$ANSIBLE_VAULT;1.1;AES256`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Git conflicts only on lines both sides touched, so the feature/updater
merge dropped symbols and markers silently where one side alone had moved
them. Execution found them, not review.
- plugins/lookup/image.py bound get_entity_name while its test patches
plugins.lookup.image.entity_name; an alias satisfies an importer but not
mock.patch, which reads the module's own binding. The module now uses
entity_name, the spelling 60+ call sites share. The alias stays for
utils/update_check and plugins/filter/update, whose tests patch the old
name.
- A download-source package resolves at its pinned URL, never in a
distribution index, so the availability probe asked the wrong oracle and
reported cursor absent. It now issues a HEAD against the declaration's
URL. The CDN answers 403 rather than 404 for an object that is gone, so
both count as absent.
- The validate suite still guarded the name "design", which moved into
roles/web-svc-design/meta/services.yml in 86cefeff26.
- 193 meta/services.yml and meta/variants.yml declare why the guide edge
is not a variant dimension of their role, and why docs rides each role's
smallest round: the README replay is the costliest partner in the matrix.
No new CI round comes out of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Keycloak stage on echoserver ran ten minutes before sys-svc-dns stopped
it with "API.cloudflare.api_token must be set": the inventory still kept the
token under the flat name CLOUDFLARE_API_TOKEN, which nothing reads any
more. The validator that runs before every deploy now catches that class.
- api.py reports a host that selects DNS_PROVIDER: cloudflare without
API.cloudflare.api_token, with group_vars/all merged under the host_vars
- variables.py reports every top-level key of host_vars and group_vars that
core does not read: not declared in group_vars or a role's defaults or
vars, not tested with "is defined" or "| default(", not read by name in a
plugin, and not a SYSTEM_EMAIL_<KEY> the email lookup builds; a flat
credential name such as CLOUDFLARE_API_TOKEN names its API.<provider>.<key>
path; ansible_* keys are exempt
A key that is read in a way the scan cannot see takes
"# nocheck: inventory-variable", documented in suppression.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`MODE_CLEANUP` says what to tidy on the deployed system: prune unused Docker
resources, clear Mastodon caches, switch an LDAP user list from `present` to
`exact`. Eight roles read it that way. The dedicated runner read the same flag
and ran `make clean` in the control repository instead, which is
`git clean -fdX` over every ignored path there.
`inventories/production/` is one of those paths. The published host install
guide provisions into it and then deploys against `$INVENTORY/.password`, so
the deploy deleted the vault password it was about to use:
Removing inventories/production/
[ERROR] The vault password file .../inventories/production/.password was
not found
Reproduced in CI on two unrelated rows, arm64/zfs/debian and
amd64/btrfs/centos, so it is neither architecture nor filesystem specific.
The coupling was a name collision, not a decision. 7e5990aa16 generated the
MODE_* flags from `01_modes.yml` and removed the legacy skip flags, and the
repository clean - which had had a flag of its own - was attached to the
generated `MODE_CLEANUP` because one now existed by that name. No other deploy
path does this; `container`, `development` and `swarm` never call `make` at
all.
`make clean` keeps its own entry points, `make setup-clean`, `make clean-sudo`
and `infinito --git-clean`, where a caller asks for it deliberately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolves 68 conflicted files. The governing rule was our structure with their
network semantics: the tor boolean becomes a network mode (clearnet | tor |
multi) throughout the variant layer, while this branch's architecture and mesh
axes, its extracted modules and its i18n content stay.
Where the two sides covered different ground, both are kept rather than one
picked:
- ca_injected answers a ca_client on the deployment's TLS mode OR on an SSO
issuer that resolves self-signed. 3a3e9ed2c9 records why: an issuer is not
an object store, so keeping only one side drops the other's coverage.
- get_merged_domains keeps the loud refusal when svc-net-tor is deployed
without a provisioned node, now gated on their tor_serves_host; their
node_onion_of returns an empty string there, which would have served every
onion-exclusive app on clearnet in silence.
- sso_cookie_domains stays the single source for oauth2-proxy's cookie scope.
Their per-network branch omitted the issuer domain on the onion arm, which
breaks the callback leg.
- The clearnet CSP batch carries --clearnet-timeout again; their batch dict
hardcoded zero while the role still passes the flag.
- The dashboard keeps its i18n content dir, image, version and design
resolution; their side reduced the logo sources to the bare service
provider values.
Their side wins where it is measurably better: the onion mirror rewrite is
gated per vhost rather than per app, the Playwright onion decision reads the
pass's own domain instead of the role's canonical one, and the matomo
reinjection and dashboard config render per network sibling.
Eight files were broken by the merge without ever carrying a conflict marker,
because git marks only the lines both sides touched:
- utils/cache/domains.py lost the onionize block that sat between two hunks
- roles/sys-front-inj-all/templates/body_filter.lua.j2 called it
- web-app-bluesky, keycloak sso_proxy: the get_entity_name filter was renamed
- trigger/__main__.py, playwright_coverage.py and its test: the deleted tor
module
- cli/administration/deploy/ci/timings.py read Label.tor
- plugins/filter/csp_filters.py called _aligned_url with four of six arguments
- tests/unit/python/utils/cache/test_domains.py referenced a renamed class
- tests/.../sys-ctl-hlth-csp/files/test_script.py mocked a function main no
longer calls
Verified: the variant layer (155 tests, 65 subtests), ca_injected (18),
the CI layer (175), the domains cache including the restored guard (32),
retrigger inputs (9), timings and the CSP health script (53). make
quality-high clears autoformat; its remaining findings are tracked separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without a swarm manager the mesh planner only built full meshes, whose
handshake gate waits for every peer, so laptops behind NAT could not join.
An inventory may now name a hub in networks.hub: the planner builds every
declared mesh around it, the hub forwards between its spokes, a spoke marked
networks.roaming gets no endpoint in the hub's peer list and dials the hub
with PersistentKeepalive, and the hub's gate skips roaming spokes while a
roaming spoke gates only on the hub. The mesh CLI takes the hub's public
address per host (--endpoint) and a vault password per host
(--host-vault-password-file). Full meshes and the swarm manager hub plan
exactly as before; NETWORK_MESH_POOL keeps the operator's subnet.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The file is written by the test itself, so the path-keyed read cache does
not fit; opt the read out of the uncached-read lint instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ruamel folds every scalar and flow collection past 80 columns, so each
run of the provision, credentials or mesh CLI broke SSH keys and alias
lists in host_vars over several lines. round_trip_yaml() now sets an
unlimited width, and the credentials CLI and the services disabler use it
instead of building their own handler.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A deploy job name repeats the category path on every row of the matrix. One Egyptian hieroglyph costs four UTF-8 bytes where the two Chinese characters it replaces cost six. Unicode names each sign by its Gardiner code, so the opaque value stays identifiable without a comment per entry.
A label has to stay one codepoint: a sign written with a variation selector costs seven bytes and is worse than what it replaced. tests/integration/meta/categories/test_glyph.py enforces that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Declare `__all__` in every package `__init__.py` whose `PROJECT_ROOT`
is imported by its modules (163 packages), so the constant counts as
exported; list every public name where the package defines more.
- Empty the 15 package `__init__.py` files whose `PROJECT_ROOT` no module
imports.
- Declare the public API of `utils.github.variant.axes` in `__all__`,
re-exports included, instead of silencing them with `noqa: F401`; the
list matches the one `feature/store-i18n` already carries.
- Use `import unittest.mock as mock` in `test_applications.py`, as the
rest of the suite does.
- Spell `_NON_PRINTABLE` in `utils/yaml_bootstrap.py` as the equivalent
BMP-only positive class; it matches exactly the same 2111 code points
as the old negated class across U+0000..U+10FFFF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
baudolo 7.1.x writes volume generations with `rsync -aHAX`, so the
producing host keeps `user.*` extended attributes (OpenCloud keeps its
storage metadata in `user.oc.*`). Every later copy dropped them again,
because none of the other rsync calls passed `-X` (nor `-A`/`-H`):
- NFS export snapshots (svc-bkp-nfs-2-local `script.sh`),
- the backup host's pull (svc-bkp-remote-2-local `pull_specific_host.py`),
- the device sync (svc-bkp-local-2-device `script.py`),
- every recover into a live target (`utils/recovery/base.py`, used by the
volume, NFS and device recover scripts),
- the `recover full` and `recover remote` tree pulls.
A disaster recovery through the documented chain therefore restored
OpenCloud without `user.oc.*`, silently: the copies succeeded and the
dirval stamp does not cover attributes.
All of these now run `rsync -aHAX` (`-aHAXP` for the device sync). A
target filesystem without xattr support makes rsync exit non-zero as soon
as a source file carries attributes, so the hop fails loudly instead of
dropping them; there is no per-device opt-out. The single-image pull in
`recover remote` keeps `-a`. The provider's ssh-wrapper needs no change:
its glob accepts any server flags and its character guard allows them.
Two tests write a `user.oc.id` attribute and assert it survives the NFS
snapshot and the recover; with the old `-a` both fail with
`OSError: [Errno 61] No data available`.
Closes#1021
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When the IPAM config of a compose project's network changes (#999 added
`ip_range`), compose recreates the network on the next `up`. With a
foreign endpoint attached, as every worktree stack is on the primary's
network, compose first stops the project's own containers, then fails
with `error while removing network: ... has active endpoints`, and a
repeated `up` or `up --force-recreate` does not recover. On 2026-10-19
this took the shared registry/package cache and coredns down for every
local session.
`make compose-network-recreate` (alias `i8conere`) runs the sequence that
recovered it: detach every running container of another project from the
network, `docker compose down --remove-orphans` without `-v` so the
caches keep their volumes, the normal `up`, then attach the detached
containers again, which draw addresses from the dynamic pool. If `up`
fails, the still-detached containers are listed instead of reattached.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
cfe20f0726 and 4915a401d4 wrapped three statements more cautiously than
the formatter wants. Without this the lint phase rewrites the same diff
into the tree on every run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
failed_selections reproduced mode, onion state, distro and architecture
but not the mesh axis. These tokens feed the priority line, where an
unpinned axis offers both states, so a failed direct swarm row came back
as itself and as a meshed sibling: two jobs to reproduce one.
Only a mode the axis covers takes the pin. pinned_vpn_states returns an
empty list for a compose row pinned to the mesh, and an empty list drops
the row from the matrix -- the class ff75343aba already had to fix once
for held runs.
The fixture had drifted from what the matrix emits: row_label built
titles with no mesh glyph while axes.assign writes one for every swarm
and compose row, so parse_label fell back to its False default and the
new &direct assertion would have been right by accident. row_label and
deploy_job_name now place the glyph under the same condition as
axes.assign, and a meshed row has a case of its own, which nothing
covered before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
e17d182d0a gave web-app-docs a switch that turns the language assertion
off on a row where docs only rides along, and read it from the
guide_host environment variable. The deploy workflow sets that variable
on the runner, but a compose deploy runs Ansible inside the infinito
container, where only _FORWARDED_ENV_KEYS arrives. guide_host was not in
it, so the lookup resolved empty, the comparison against the row's own
id was true everywhere, and the switch never fired.
Run 39137606155 shows both halves: the passenger rows 117711219652 and
117711220306 carry guide_host: web-app-docs and each spent about 7000 s
of a 20400 s budget in that one assertion, while the docs row itself,
117711219660, carries an empty guide_host and therefore keeps running
it. Forwarding the variable preserves that split rather than silencing
the assertion everywhere.
APP_ID is in the tuple for the same reason and feeds the guide_role
lookup from the same workflow step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Combine main's WireGuard mesh axis with this branch's network axis in the CI
variant code. Every matrix row now carries both: network (clearnet, tor,
multi) from utils.github.variant.network and vpn (meshed or direct) from
utils.github.variant.vpn. Label parsing, which main moved to label.py, reads
the network glyph instead of the old tor flag and keeps the vpn glyph; the
selection token, plan table, matrix and trigger CLIs, the workflows and the
swarm routine forward both axes. The test of an incapable priority variant
now compares the set of network modes, since a swarm priority row also
covers both mesh arms.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The lane ran litellm but never republished its port, so a host-side probe hit the compose proxy on the same number and read 502. The lane could therefore never report ready, and the oracle suite skipped itself to a green exit instead of asking a judge. A handler reads the port from the role the way translate.py already does, the runner forwards it, and TOOLS_LANE_KEYS takes it up: compose_file_args layers the override in without checking its variables, so an .env written before the key existed would abort every compose command rather than merely drop the lane.
The gateway's master key is pinned in the lane inventory for the reason the System One key already is: the provisioner would otherwise generate it into a vault the checks cannot read back, and the oracle judge is not a deployed consumer with a virtual key of its own.
serving() stops treating reachability as readiness. A lane brought up before AI_LOCAL_MODELS named a different model keeps serving the old one, and a gateway holding a key these checks cannot use looks healthy from outside while refusing every call they make. It now asks /models with the bearer token and requires the configured alias, and requires the gateway to offer every language the catalogs name, because that advertisement is what the translation run waits on. Without this the lane waved itself through and no change to its roles ever shipped.
Two lane-scoped overrides come with it. The ollama role caps itself at four CPUs, which left the host at 361 percent while twenty cores idled, so the lane gives it the host count the way the engine entry above it already does. And the lane pins the router to its declared order: the decider may send a pair the engine serves to the chat model, which a batch over 183 catalogs cannot pay for. Deployments keep the decider.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.
The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.
call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Three faults, one mechanism each, that together let the oracle suite report
a verdict it never reached.
The address. lane_url handed out the runner's compose service name, but the
stack's containers resolve through the deployment's CoreDNS, which never
learns one: getent hosts tools answers nothing inside them, so every check
that asked the lane from there skipped itself. The bring-up now writes the
runner's address, picked as the one inside INFINITO_SUBNET because the
runner sits on every network it was ever brought up on, and the checks read
it from there. It is rewritten on both paths, because a restart gives the
runner a new one.
The readiness. serving judged the engine alone, so a lane whose engine make
i18n had already brought up counted as up while System One was down. It now
probes every role the caller names, which is why the runner republishes
System One's port: a container IP is unreachable from the host that decides
whether to deploy.
The verdict. A silent lane made the suite skip, and a skip exits green.
Only a deploy that is already running leaves the stack alone and exits 0
now; a crashed deploy and a deployed-but-silent role both exit 1.
The override's third :? guard is added to the gate that decides whether the
file is layered in at all, next to the one that was already missing. Without
it an .env written before these keys aborts every compose command, down to
compose-down, instead of dropping the lane. A test now keeps that list and
the override's guards in step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Covers the declarations read from every host, the refusal of differing ones,
every peer pinned to its /32, no hub anywhere, the host of one mesh holding no
peer of the other, and a second run leaving every file alone.
Covers per-role and per-host membership, a host two roles put into one mesh,
the hub never being a spoke, an undeclared mesh being refused, and the
example's members matching its table.
