0327114c44 made the provisioning CLI keep the keys of the last `disable=` run in host_vars under _INFINITO_SERVICES_DISABLED and read them back on the next run. The inventory validator from 285ce910d2 only collects keys that group_vars, role defaults or vars declare, that templates guard, or that plugins read by name, so a key read through a constant in cli/ counted as unread, and every local compose-deploy with disable= aborted with "Inventory validation failed" before the play (repro: make compose-deploy mode=reinstall apps=web-app-keycloak disable=matomo variant=0).
read_variables() now includes DISABLED_SERVICES_RECORD from the disabler module, so the name has one source. An inventory carrying the record validates; an unknown key next to it is still reported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
utils.cache.yaml registered its own !vault constructor that returned a plain str, and it overrides the VaultScalar constructor of utils.handler.vault whenever it registers last. Every writer that loads a host_vars file through the cached loader, changes one key and dumps it back then re-emitted each credential as an untagged quoted string, and Ansible handed the $ANSIBLE_VAULT block itself to the roles.
08680e933d and c88647b3a1 moved tor_node.py and network_mode.py to the round-trip helpers. The shared loader now returns a VaultScalar, so every yaml_io writer keeps the tag regardless of import order. VaultScalar subclasses str, so the other readers of the cached loader see the same value.
test_a_round_trip_keeps_the_vault_tag fails against the previous constructor: re-registering the plain-str constructor on CSafeLoader and SafeLoader drops '!vault |' from the dumped file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
apply_network_mode_from_env loaded the whole host_vars file through
utils.cache.yaml, set one scalar and wrote everything back. That loader's
!vault constructor returns a plain string, so every credential in the file
was re-emitted untagged and Ansible handed the raw ciphertext to the roles
instead of the secret. It only runs when the row sets `network`, which is
every multi-network round.
Run 40032069040 on the fork shows what that costs: the generated host_vars
of a failing job carries 0 `!vault` tags against 56 `$ANSIBLE_VAULT`
headers, mariadb then refuses root with 1045 on a freshly initialised
database, and the openldap bind fails with result 49.
08680e933d fixed the same class in tor_node.py, the sibling writer in this
package. This one writes through the round-trip helpers too.
The existing test did not cover the case, which is why the sibling fix did
not stop it: test_preserves_existing_vault_credentials now fails against
the previous writer with
`'!vault |' not found in "... root_password: '$ANSIBLE_VAULT;1.1;AES256`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Git conflicts only on lines both sides touched, so the feature/updater
merge dropped symbols and markers silently where one side alone had moved
them. Execution found them, not review.
- plugins/lookup/image.py bound get_entity_name while its test patches
plugins.lookup.image.entity_name; an alias satisfies an importer but not
mock.patch, which reads the module's own binding. The module now uses
entity_name, the spelling 60+ call sites share. The alias stays for
utils/update_check and plugins/filter/update, whose tests patch the old
name.
- A download-source package resolves at its pinned URL, never in a
distribution index, so the availability probe asked the wrong oracle and
reported cursor absent. It now issues a HEAD against the declaration's
URL. The CDN answers 403 rather than 404 for an object that is gone, so
both count as absent.
- The validate suite still guarded the name "design", which moved into
roles/web-svc-design/meta/services.yml in 86cefeff26.
- 193 meta/services.yml and meta/variants.yml declare why the guide edge
is not a variant dimension of their role, and why docs rides each role's
smallest round: the README replay is the costliest partner in the matrix.
No new CI round comes out of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Keycloak stage on echoserver ran ten minutes before sys-svc-dns stopped
it with "API.cloudflare.api_token must be set": the inventory still kept the
token under the flat name CLOUDFLARE_API_TOKEN, which nothing reads any
more. The validator that runs before every deploy now catches that class.
- api.py reports a host that selects DNS_PROVIDER: cloudflare without
API.cloudflare.api_token, with group_vars/all merged under the host_vars
- variables.py reports every top-level key of host_vars and group_vars that
core does not read: not declared in group_vars or a role's defaults or
vars, not tested with "is defined" or "| default(", not read by name in a
plugin, and not a SYSTEM_EMAIL_<KEY> the email lookup builds; a flat
credential name such as CLOUDFLARE_API_TOKEN names its API.<provider>.<key>
path; ansible_* keys are exempt
A key that is read in a way the scan cannot see takes
"# nocheck: inventory-variable", documented in suppression.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolves 68 conflicted files. The governing rule was our structure with their
network semantics: the tor boolean becomes a network mode (clearnet | tor |
multi) throughout the variant layer, while this branch's architecture and mesh
axes, its extracted modules and its i18n content stay.
Where the two sides covered different ground, both are kept rather than one
picked:
- ca_injected answers a ca_client on the deployment's TLS mode OR on an SSO
issuer that resolves self-signed. 3a3e9ed2c9 records why: an issuer is not
an object store, so keeping only one side drops the other's coverage.
- get_merged_domains keeps the loud refusal when svc-net-tor is deployed
without a provisioned node, now gated on their tor_serves_host; their
node_onion_of returns an empty string there, which would have served every
onion-exclusive app on clearnet in silence.
- sso_cookie_domains stays the single source for oauth2-proxy's cookie scope.
Their per-network branch omitted the issuer domain on the onion arm, which
breaks the callback leg.
- The clearnet CSP batch carries --clearnet-timeout again; their batch dict
hardcoded zero while the role still passes the flag.
- The dashboard keeps its i18n content dir, image, version and design
resolution; their side reduced the logo sources to the bare service
provider values.
Their side wins where it is measurably better: the onion mirror rewrite is
gated per vhost rather than per app, the Playwright onion decision reads the
pass's own domain instead of the role's canonical one, and the matomo
reinjection and dashboard config render per network sibling.
Eight files were broken by the merge without ever carrying a conflict marker,
because git marks only the lines both sides touched:
- utils/cache/domains.py lost the onionize block that sat between two hunks
- roles/sys-front-inj-all/templates/body_filter.lua.j2 called it
- web-app-bluesky, keycloak sso_proxy: the get_entity_name filter was renamed
- trigger/__main__.py, playwright_coverage.py and its test: the deleted tor
module
- cli/administration/deploy/ci/timings.py read Label.tor
- plugins/filter/csp_filters.py called _aligned_url with four of six arguments
- tests/unit/python/utils/cache/test_domains.py referenced a renamed class
- tests/.../sys-ctl-hlth-csp/files/test_script.py mocked a function main no
longer calls
Verified: the variant layer (155 tests, 65 subtests), ca_injected (18),
the CI layer (175), the domains cache including the restored guard (32),
retrigger inputs (9), timings and the CSP health script (53). make
quality-high clears autoformat; its remaining findings are tracked separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without a swarm manager the mesh planner only built full meshes, whose
handshake gate waits for every peer, so laptops behind NAT could not join.
An inventory may now name a hub in networks.hub: the planner builds every
declared mesh around it, the hub forwards between its spokes, a spoke marked
networks.roaming gets no endpoint in the hub's peer list and dials the hub
with PersistentKeepalive, and the hub's gate skips roaming spokes while a
roaming spoke gates only on the hub. The mesh CLI takes the hub's public
address per host (--endpoint) and a vault password per host
(--host-vault-password-file). Full meshes and the swarm manager hub plan
exactly as before; NETWORK_MESH_POOL keeps the operator's subnet.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The file is written by the test itself, so the path-keyed read cache does
not fit; opt the read out of the uncached-read lint instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ruamel folds every scalar and flow collection past 80 columns, so each
run of the provision, credentials or mesh CLI broke SSH keys and alias
lists in host_vars over several lines. round_trip_yaml() now sets an
unlimited width, and the credentials CLI and the services disabler use it
instead of building their own handler.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Declare `__all__` in every package `__init__.py` whose `PROJECT_ROOT`
is imported by its modules (163 packages), so the constant counts as
exported; list every public name where the package defines more.
- Empty the 15 package `__init__.py` files whose `PROJECT_ROOT` no module
imports.
- Declare the public API of `utils.github.variant.axes` in `__all__`,
re-exports included, instead of silencing them with `noqa: F401`; the
list matches the one `feature/store-i18n` already carries.
- Use `import unittest.mock as mock` in `test_applications.py`, as the
rest of the suite does.
- Spell `_NON_PRINTABLE` in `utils/yaml_bootstrap.py` as the equivalent
BMP-only positive class; it matches exactly the same 2111 code points
as the old negated class across U+0000..U+10FFFF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Combine main's WireGuard mesh axis with this branch's network axis in the CI
variant code. Every matrix row now carries both: network (clearnet, tor,
multi) from utils.github.variant.network and vpn (meshed or direct) from
utils.github.variant.vpn. Label parsing, which main moved to label.py, reads
the network glyph instead of the old tor flag and keeps the vpn glyph; the
selection token, plan table, matrix and trigger CLIs, the workflows and the
swarm routine forward both axes. The test of an incapable priority variant
now compares the set of network modes, since a swarm priority row also
covers both mesh arms.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Covers the declarations read from every host, the refusal of differing ones,
every peer pinned to its /32, no hub anywhere, the host of one mesh holding no
peer of the other, and a second run leaving every file alone.
Covers per-role and per-host membership, a host two roles put into one mesh,
the hub never being a spoke, an undeclared mesh being refused, and the
example's members matching its table.
The any-number-of-meshes test zipped the specs against the planned meshes
without strict=, which ruff reports as B905 and the lint-python job failed on.
Both sequences come from the same declarations, so strict=True also turns a
plan that drops a mesh into a failure instead of a silently shorter loop.
The guarantees were asserted against the declared swarm and data pair,
so they proved two meshes work rather than that the planner works. A
deployment declares its own, and the review asks for any number of them with
the nodes arranged however the client wants.
The same invariants now run over generated topologies: one to eight meshes,
three arrangements each, hosts dealt out so some belong to several meshes and
some to none. 2247 subtests, four seconds, and the planner needed no change to
pass them.
Provisioning merges into host_vars and never deletes, so the false
enabled/shared toggles that apply_services_disabled writes for a key stayed
in the round inventory after a later run dropped that key from disable=.
That covered the provider's own entry too: with seaweedfs once disabled,
web-app-seaweedfs kept shared: false, the service registry offered no
seaweedfs provider, container_networks attached no consumer, and BookWyrm
ran USE_S3=true against http://seaweedfs:8333 without a route.
reconcile_services_disabled_from_env records the keys of each disable run
in the host_vars key _INFINITO_SERVICES_DISABLED. On the next run it deletes,
in every application block, the enabled/shared toggles that are still
exactly false for a recorded key the current disable no longer lists, drops
a block left empty, and rewrites or removes the record. It runs right after
ensure_host_vars_file, before credentials and the variant bake, so a variant
that wants a toggle off writes it again. With an unchanged or empty disable
it does nothing. An inventory written before this change has no record, so
its older toggles are only reverted after one run with the old list.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`disable=lam` aborted the local deploy of web-app-openbao with "`disable`
conflicts with the current inventory state", and web-app-moodle carries the same
construction in two of its variants.
The writer and the conflict check disagreed on scope. apply_services_disabled
walked `find_roles_with_service`, the roles whose meta/services.yml declares the
key, so neither role was in that set for `lam` and nothing was written.
find_services_disabled_conflicts walks every deployed app's host_vars instead,
found the `lam` block the variant had baked, and refused the run. Dropping
`lam` from the list is not an answer either: the round then deploys web-app-lam
although ldap is off.
The writer now also flips a key it finds in host_vars under an app whose role
does not declare it, and only the flags that are already in that block. It never
creates a key or a flag, which is what `_declared_toggles` exists to prevent:
compare_application_keys rejects a host_vars key the role contract does not
carry. Overwriting a value the variant already wrote introduces nothing, so that
rule is untouched; test_skips_app_whose_role_does_not_define_service still
passes and pins it.
The alternative the TODO proposed, declaring `lam` in meta/services.yml with the
`'web-app-lam' in group_names` flags, is the worse trade: a consumer entry is
resolved through the service registry, so it would bind web-app-lam and its
whole closure into every round that carries openbao or moodle, for a key both
roles read only to gate a Playwright spec.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The arm existed to answer one question: how much of the meshed arm's
cost is the connection plugin and how much is the tunnel. Runs 37592061960
(docker), 37753506661 (SSH on the underlay) and 37621556682 (SSH over
WireGuard) answered it on web-app-gitea#0, debian/ext4:
pass 1 5964s / 8511s / 8832s
pass 2 5319s / 7472s / 7705s
Leaving ansible_connection: docker costs +4700s, the tunnel +554s - 89.5%
against 10.5%. The answer is recorded; the scaffolding does not need to ship.
Reverts d792519725, 1b28f87b37, 89b8a8c8d1, 0470491e1a, 529c5ace43,
dd3d10a9a2, c20dd0e5f5, 458d8956e1, 63ade1e03b, d281555945 and 4adfb6e3d2.
95cfc8b7b7 is already superseded by bbfd82f89, which sets the profiling limit
upstream.
One test asserts ansible_host lands on the given address, the other
that a missing member raises. Either would have caught the UNREACHABLE run
before it was dispatched.
The carriage returns a pty adds are invisible until a role parses its own
output, so the absence of the tty is asserted where the transport is
written rather than left to the next role that reads stdout.
The pass that runs over the mesh reaches every host by SSH where the
pass before it used the container connection, and it pays that difference
once per task. Measured in a container: an exec through a warm multiplexer
costs 7ms, a fresh connection 85ms, and 165ms once sudo asks for the
password every task demands. The pass runs thousands of tasks.
Two settings threw the multiplexer away. A persist window of 20s closes the
master whenever a host waits longer than that for its next task, which in a
linear play is most of them; and the named ControlPath points at a directory
nothing creates, so a master that cannot open its socket reconnects per task
without reporting it.
Neither is retuned for the repository. The tool that moves a host onto the
mesh now writes that host's own ssh_args, derived from the repository's
value so every other option is kept and only the window and the socket
differ. A host the mesh never touches keeps exactly what it had.
Confirmed against the built command: a switched host gets ControlPersist=600s
and a ControlPath under the directory Ansible creates, while a host beside it
keeps the repository's own.
The offset came from the spoke's position in a sorted list, so a host that
sorts earlier renumbered every member behind it. `wg syncconf` carries keys
and peers onto a live interface and never its address, so a renumbered member
keeps answering on the old address while the peers and the inventory have
moved to the new one -- and the address it gave up can be handed to a new
member while the old owner still holds it.
The planner now reuses the addresses already on disk and allocates only an
unseen member, from the lowest free offset. A value outside the subnet, a
duplicate, and the hub's own offset are refused, so a mirrored host_vars
cannot hand a spoke the hub's address. Rotation keeps addresses: it replaces
identities, not addresses.
The routing assertions re-planned the mesh and compared the result with
itself, so a writer that stored the mesh subnet in place of the routed range
would have passed; they now read the entry off disk, which is what the role
reads.
`config/main.yml` exists in no role: f65f4d60fb migrated the last one. A guard
against the string already exists, `TestNoLegacyPathReferences`, and it names
`config/main.yml` among its LEGACY_TOKENS. It scanned plugins, utils, cli,
roles, scripts, filter_plugins and lookup_plugins, which is every tree except
the two the stale references actually lived in.
So nine of them survived in `tests/`, and the guard reported green the whole
time. The code underneath was already correct everywhere: `test_csp_consistency`
and `test_role_wiring` iterate ROLE_FILE_META_SERVICES, and only their docstrings
and one assertion message still said `config/main.yml`. Had any of them globbed
the old path the test would have gone vacuous rather than red, which is the
failure mode worth naming here.
`tests` and `tasks` are now scanned. The targets come from
`utils/roles/mapping.py`, not from guesswork: `config/main.yml` is
`meta/services.yml`, `schema/main.yml` is `meta/secrets.yml` (confirmed at
`cli/administration/inventory/provision/reset.py:67`, which reads exactly that
file), and `users/main.yml` is `meta/users.yml`.
Scanning `tasks` exposed a second dead pointer. The migration tool became a
package on 2026-09-14, but EXEMPT_FILES still named `migrate_meta_layout.py`, a
file that no longer exists, so the exemption was inert and only invisible
because `tasks` went unscanned. Two dead pointers covering for each other. The
exemption is now a directory, so a new module in that package cannot fall out of
it. `test_no_direct_calls.py` named the same dead path in prose; that lint
allows through nocheck markers at the call site, so nothing was broken there.
`docs/administration/configuration/email.md` was the worst of them, because it
instructed operators rather than describing tests: it told them to write
`compose.services.email` into `roles/<application_id>/config/main.yml`. The
consumer, `plugins/lookup/email.py:255`, reads `entry["services"]["email"]` and
says in its own comment that there is no `compose.services` envelope. The role
snippet, the inventory snippet and the precedence list now match the reader.
`group_vars/all/TODO.md` pointed at the same dead file.
Verified by watching the guard fail, twice. Re-introducing `config/main.yml` in
a test file reports it; pointing the new directory exemption at a wrong path
reports `tasks/utils/migrate_meta_layout/credentials.py`, which proves both that
`tasks` is scanned and that the exemption is the only thing letting the
migration tool through. Restored, 11 passed.
The changelogs under `docs/changelog/` keep their mentions, and `.md` stays out
of the scanned suffixes for that reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Asserts the hub's two mesh entries carry the same key and that its spoke holds
it. A first attempt matched only the first entry and passed against the bug.
kevinveenbirkenbachandClaude Opus 5.5
evangelostsak