Covers the declarations read from every host, the refusal of differing ones,
every peer pinned to its /32, no hub anywhere, the host of one mesh holding no
peer of the other, and a second run leaving every file alone.
Covers per-role and per-host membership, a host two roles put into one mesh,
the hub never being a spoke, an undeclared mesh being refused, and the
example's members matching its table.
The any-number-of-meshes test zipped the specs against the planned meshes
without strict=, which ruff reports as B905 and the lint-python job failed on.
Both sequences come from the same declarations, so strict=True also turns a
plan that drops a mesh into a failure instead of a silently shorter loop.
The guarantees were asserted against the declared swarm and data pair,
so they proved two meshes work rather than that the planner works. A
deployment declares its own, and the review asks for any number of them with
the nodes arranged however the client wants.
The same invariants now run over generated topologies: one to eight meshes,
three arrangements each, hosts dealt out so some belong to several meshes and
some to none. 2247 subtests, four seconds, and the planner needed no change to
pass them.
The arm existed to answer one question: how much of the meshed arm's
cost is the connection plugin and how much is the tunnel. Runs 37592061960
(docker), 37753506661 (SSH on the underlay) and 37621556682 (SSH over
WireGuard) answered it on web-app-gitea#0, debian/ext4:
pass 1 5964s / 8511s / 8832s
pass 2 5319s / 7472s / 7705s
Leaving ansible_connection: docker costs +4700s, the tunnel +554s - 89.5%
against 10.5%. The answer is recorded; the scaffolding does not need to ship.
Reverts d792519725, 1b28f87b37, 89b8a8c8d1, 0470491e1a, 529c5ace43,
dd3d10a9a2, c20dd0e5f5, 458d8956e1, 63ade1e03b, d281555945 and 4adfb6e3d2.
95cfc8b7b7 is already superseded by bbfd82f89, which sets the profiling limit
upstream.
One test asserts ansible_host lands on the given address, the other
that a missing member raises. Either would have caught the UNREACHABLE run
before it was dispatched.
The carriage returns a pty adds are invisible until a role parses its own
output, so the absence of the tty is asserted where the transport is
written rather than left to the next role that reads stdout.
The pass that runs over the mesh reaches every host by SSH where the
pass before it used the container connection, and it pays that difference
once per task. Measured in a container: an exec through a warm multiplexer
costs 7ms, a fresh connection 85ms, and 165ms once sudo asks for the
password every task demands. The pass runs thousands of tasks.
Two settings threw the multiplexer away. A persist window of 20s closes the
master whenever a host waits longer than that for its next task, which in a
linear play is most of them; and the named ControlPath points at a directory
nothing creates, so a master that cannot open its socket reconnects per task
without reporting it.
Neither is retuned for the repository. The tool that moves a host onto the
mesh now writes that host's own ssh_args, derived from the repository's
value so every other option is kept and only the window and the socket
differ. A host the mesh never touches keeps exactly what it had.
Confirmed against the built command: a switched host gets ControlPersist=600s
and a ControlPath under the directory Ansible creates, while a host beside it
keeps the repository's own.
The offset came from the spoke's position in a sorted list, so a host that
sorts earlier renumbered every member behind it. `wg syncconf` carries keys
and peers onto a live interface and never its address, so a renumbered member
keeps answering on the old address while the peers and the inventory have
moved to the new one -- and the address it gave up can be handed to a new
member while the old owner still holds it.
The planner now reuses the addresses already on disk and allocates only an
unseen member, from the lowest free offset. A value outside the subnet, a
duplicate, and the hub's own offset are refused, so a mirrored host_vars
cannot hand a spoke the hub's address. Rotation keeps addresses: it replaces
identities, not addresses.
The routing assertions re-planned the mesh and compared the result with
itself, so a writer that stored the mesh subnet in place of the routed range
would have passed; they now read the entry off disk, which is what the role
reads.
Asserts the hub's two mesh entries carry the same key and that its spoke holds
it. A first attempt matched only the first entry and passed against the bug.
Mirrors the hub's host_vars onto the controller the way the credential reset
does, then asserts the rerun leaves only the mesh the controller belongs to.
Without the fix the file keeps the hub's data entry, is_hub included.
Pins the two failure modes the design turns on: a host with no mesh address is
left reachable rather than stranded, and an address is only read from the host
that owns it, because the swarm reset mirrors one node's host_vars over every
other.
Brings in 225 commits, including the merged web-app-openbao work and the
LMStudio loopback fix.
Seven conflicts:
- The legacy svc-net-wireguard-core/plain/firewalled roles this branch deleted
in e706170a0 were modified upstream. The deletion stands, which is the whole
premise of the branch.
- The MCP audit takes upstream's list plus svc-net-firewall, with the three
legacy wireguard entries collapsed into the single svc-net-wireguard: 23.
- web-app-openbao/README.md takes main's reviewed version wholesale.
- matrix.py takes both sides: upstream's app_id argument to _reset_credentials
and this branch's _write_mesh call after it.
The merge pushed matrix.py to 511 lines against the 500 cap, so the step
runner moved to utils/tests/swarm/run.py, which is self-contained and leaves
matrix.py at 427. Its test moved with it.
Adds the mirror case pruning has to survive and its inverse, that an owned mesh
is left untouched.
The existing assertions searched the file for 'swarm:' and 'data:', which the
credential keys mesh_private_key_swarm and mesh_private_key_data also match, so
they passed against a file whose mesh block was empty. They now read the mesh
names out of the parsed document.
The failure this covers is silent in the inventory and total on the cluster:
every node adopts the hub's key and address, every interface claims 10.100.0.1,
and nothing handshakes. Asserts the mirrored hosts read as unkeyed and that a
rewrite gives the three swarm members three distinct addresses again.
`disable=` wrote both `enabled: false` and `shared: false` for every role that
declares the service, but a role qualifies with any one of `enabled`, `shared`
or `provides`. compare_application_keys rejects any host_vars key the role
contract does not carry, so a role declaring only `enabled` came back as a
validation error and the deploy never reached Ansible:
Missing default for web-svc-cdn: services.dashboard.shared
Missing default for web-svc-mirror: services.dashboard.shared
[ERROR] Inventory validation failed. Aborting deployment.
_declared_toggles reads the role's meta/services.yml and writes only what is
there; a role carrying neither toggle is skipped instead of gaining an empty
block. The module already held this principle: find_roles_with_service excludes
pure metadata entries so `disable` "doesn't add bogus enabled: false overrides",
and this applies the same rule per flag rather than per role.
Delta measured over the disable list the compose iteration procedure prescribes
(matomo, dashboard, prometheus, email, css): 407 of 409 role/service pairs are
untouched because they declare both toggles, exactly 2 stop receiving a
fabricated key, and 0 fall out of the disable entirely. The 2 are the pairs the
validator rejected. Across all 266 roles with a meta/services.yml, 1224 service
blocks declare `enabled`, 1190 declare `shared`, 34 declare `enabled` alone and
none declares `provides` alone, so nothing loses its disable.
Proven by the deploy sequence rather than by a test: the first local deploy
aborted at inventory validation, the second passed into the Ansible run with the
same disable list. Two regression tests cover both shapes, and the negative
control holds: hard-coding ["enabled", "shared"] fails the new test.
Without this, `disable=` is unusable in every local deploy, which the iteration
procedure requires on every run.
Residual: the generator was exercised through compose-deploy only, not through
roundtrip or swarm-zombie, which reach the same function by other call paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A spoke's AllowedIPs for the hub was its own mesh subnet, so a worker on
10.100.0.0/24 had no route to the NFS server on 10.100.1.11 and could not
reach the plane it is deliberately not a member of -- which is the entire
reason the hub sits in both.
The symptom would have been misleading: every tunnel reports a healthy
handshake, the gate passes because it only pings peers the host is configured
for, and the failure surfaces later as an NFS mount that hangs. It points at
storage rather than at routing.
Spokes now route the declared pool through the hub while the hub keeps each
spoke pinned to a single address; routing the pool in both directions would
blackhole every member behind whichever spoke came first.
Found by running the writer against the prefixed five-host topology before the
deploy rather than during it.
Covers the entry point rather than the library: one invocation writes every
member across both inventory files, a missing host_vars directory fails instead
of silently writing nothing, and a rerun is a no-op end to end.
Asserted against real files in a tempdir, because the guarantees are about
files: which host_vars a secret may appear in, and whether a second run of an
unchanged mesh leaves them alone.
Two cases carry the security argument. A private key must reach exactly one
host_vars file, which is what keeps one host's inventory from impersonating
another, and it must never appear there in plaintext. A third covers the
inverse of the idempotence shortcut: a public key whose credential has gone
missing is worse than no key, because peers accept it while its owner cannot
use it, so that member is reported unkeyed and the pair is reminted together.
A worker is asserted NOT to receive the data mesh. Carrying it would peer the
worker directly with the NFS server and bypass the routing through the hub that
this topology depends on.
Every failure mode here is invisible at deploy time. A spoke missing from the
hub's peer list configures cleanly and never handshakes. A public key that does
not answer for its private half looks identical until traffic is attempted. A
partial rotation leaves the hub holding an old key while the spoke presents a
new one, and the handshake is refused with nothing logged on either side.
The load-bearing case is the hub filter: the manager is a member of
svc-swarm-node as well as the hub group, so left unfiltered it peers with
itself, which WireGuard accepts and which then routes the hub's own traffic
into its own tunnel.
The subnet guards are asserted against the shipped declaration too, so the
configuration proves its own collision checks rather than only the synthetic
ones.
make quality-high pass 75 rewrote it: a concatenation whose right operand is a
multi-line literal gets parenthesised, with the operator leading its own line.
The string itself is unchanged, so
test_narrowing_the_credentials_still_rotates_every_user keeps the declaration
that separates web-app-keycloak from web-app-a and still fails when the user
half of the rotation is narrowed along with the credentials.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The matrix gate rotates credentials between the two deploy passes so PASS 2
has to carry fresh values. It rotated every application block in the host_vars
file, and on a matrix inventory that is not the round. Provision applies its
--include as a strict intersection (filters.py:57-62), so it generates
credentials for at most derive_includes, while mirror_overrides.py adds a block
per mirror artefact carrying only image and version. Rotating those costs one
credential subprocess each for values provision never wrote.
matrix.py now passes --include derive_includes(app_id, variants=round_variants),
the same call 02_provision_inventory.sh:11 feeds provision, from the same
round_variants the provision step got one call earlier. The rotation set is
therefore a superset of what provision generated: every database_password still
rotates, because utils/manager/inventory.py:236-242 writes it under an id from
that same filtered list. Measured for web-app-wordpress: 35 ids.
The scope stops at the credentials. reset_credentials takes credential_ids
beside application_ids because the two halves shared one list, and narrowing it
narrowed _drop_user_passwords too: that resolves through
required_user_policies(roles_dir, application_ids), so 60 of 66 declared users
would have stopped rotating, root, backup and every mcp-* bot among them. The
user half keeps the full document list and is byte-identical to before.
The test fixture declared the same users in both roles, which made the
regression test pass against the regression. web-app-keycloak now declares a
user web-app-a does not, so the assertion has something to bite on.
The CLI aborts when --include matches no block instead of rotating nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
VaultHandler.encrypt_string started one ansible-vault process for every single credential. The swarm matrix' rotation gate between PASS 1 and PASS 2 drives 113 application subprocesses over 855 credential keys, so the step cost 1078s in run 33923190220 while the green baseline never ran it at all. Measured per value: 226ms via the CLI, 3.3ms via VaultLib, so 855 keys drop from 193s to 2.8s.
The snippet layout is a contract: encrypt_leaves splits the returned lines and strips the body indent, and existing inventories were written by the CLI. Verified byte-compatible (same 'name: !vault |' header, same 10-space body indent, same $ANSIBLE_VAULT;1.1;AES256 marker), and ansible-vault decrypt reads back what the new path writes.
Reading the password file delegates to ansible's own FileVaultSecret rather than approximating it: ansible strips, then decrypts a vaulted password file, then rejects an empty secret. A hand-rolled .strip() covered only the first of those. _vault stays lazy because callers construct VaultHandler with a password file that need not exist yet.
The tests replace two subprocess.run mocks that asserted against invented output. They now pin the layout encrypt_leaves depends on, and pin the guarantee from 052cea02e2 (values starting with a dash), which landed without a test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
65c72fc374 settled the convention: import unittest, and reach the submodule as import unittest.mock as mock, so a call site has one spelling of the name. test_onion.py and test_tor_socks_proxy.py still carried from unittest import mock; CodeQL reports them as py/import-and-import-from, the last two in the repository.
Same one-line change per file the sweep applied; mock.<attr> call sites untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shell-safe default drew from letters and digits only, and every user without declared accounts is an identity account, so kcadm set-password rejected hermes and openclaw with invalidPasswordMinSpecialCharsMessage as soon as keycloak entered the GitLab#1 round without LDAP (4164f37981) and the extras file stopped replacing the users map (03e42bae6d; run 33689701895). The generator now draws from letters, digits and -_. and guarantees upper case, lower case, a digit and one mark, never in the leading position.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
_drop_app_credentials walked applications[app].credentials, which
f713ba8350 moved to secrets.credentials. Nothing lives at the old address
any more - the run's inventory has twelve applications under
secrets.credentials and none above it - so the isinstance check never
matched and the function returned 0 every time. The rotation ran, reported
success and rotated no application secret at all.
It stayed hidden because the sibling half worked: _drop_user_passwords
reads document["users"], which did not move, so the total was never zero
and the call looked healthy.
That matters more than a dead branch usually would. Rotating between two
deploy passes is what turns the second pass into a test of the update
path, where a role has to push a new secret into the application and the
database it already created. With the drop silently skipped, every such
run has been re-deploying the values the first pass wrote.
The fixture pinned the pre-rename shape, so the test confirmed the bug
rather than catching it. Migrated to the real layout, with the address in
one helper instead of four assertions; against the old code it now fails
1 != 4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
evangelostsak
kevinveenbirkenbachandClaude Opus 5