The README said a templated pin moves whenever the assembled value
changes. newer_version() additionally rejects an assembled value that
sorts below the current pin, which
test_an_unchanged_or_older_templated_tag_reports_nothing covers.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An http_regex source read one document, so a tag whose parts upstream
keeps in two files could only carry the second part as a literal. Every
pin also fetched its document on its own: a failed request left one pin
of a release behind while the others moved.
url now takes a list whose bodies are searched as one text, and every
document is fetched once per run, so all pins that read it resolve against
the same snapshot. The HTTP reads move into utils/update/fetch.py.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
keep_address left ansible_host alone, and the lab never sets it: the
docker connection uses inventory_hostname as the container name. Switching to
SSH without writing an address made ssh resolve 'web-app-gitea-swarm-mgr-01'
through DNS, and run 37617424998 died UNREACHABLE on all three nodes in 13
minutes.
switch_to_mesh now takes the addresses to dial. A member missing from the
mapping raises instead of falling back to its mesh address, because that
fallback would have produced the meshed arm under the other arm's name - a
wrong number rather than a failed run.
A pin that names its own upstream in an update: block was still collected
by the image updater or, for a ref, by the repository updater. Those raise
it to the newest tag of its registry or repository, which for a component
of a release is not the release's value. A tag built from several upstream
values, such as <freeswitch>-<bigbluebutton>, could not be resolved at
all: the resolver takes one capture and keeps a pin on its suffix.
The image updater now skips a version and the repository updater a ref
that declares a source for itself. An http_regex source accepts a template
over the pattern's named groups, and a templated pin moves whenever the
assembled value changes, its suffix included.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One conflict, in the generated role overview table of README.md.
Both sides had edited the same block. Upstream dropped Redis from the pihole
row; this branch still listed it. Upstream also listed svc-net-wireguard-core,
-plain and -firewalled, which this branch removed in e706170a0b as superseded
by the unified svc-net-wireguard role.
Resolved by taking upstream's block and then running `make readme-index`, so
the table is generated from the roles that actually exist rather than
hand-merged: Redis is gone from pihole, the three legacy WireGuard roles stay
gone, and svc-net-wireguard is listed once. `make readme-index check=true`
reports the table up to date.
Git kept the role deletions silently and correctly - upstream has not touched
those three directories since the merge base, so there was no modify/delete
conflict to resolve.
keep_address switches the connection, the user, the key and the ssh
args exactly as the mesh switch does, and leaves ansible_host where it was.
Against the other two transports that isolates the cost of leaving the local
connection from the cost of the tunnel.
Of the 76 open code scanning alerts, 39 point at code that can change.
svc-ai-mcp-adapter:
- A path argument that is empty or made only of dots is refused.
Percent-encoding keeps a separator inside its segment, but "." and ".."
need none, so an upstream that resolves dot segments could answer from a
path the contract does not name.
- parse_duration splits on the units. The pattern it used before rescanned
a long run of digits from every start position.
Tests and the README generators:
- Every Jinja environment built with autoescape=False and a "noqa: S701"
uses select_autoescape now, as six test modules and
utils/templating/jinja.py already did. It escapes templates named
*.html, *.htm and *.xml, which none of these call sites loads.
- Dead code goes: a doubled "rel" block in the email lint, an unread
_SKIP_DIRS, and a module that _resolve returned to no caller.
- Two "except ...: pass" blocks become contextlib.suppress and
assertRaises.
- Nine test modules import unittest.mock one way instead of two, and one
lookup test imports its module once.
Closes#807
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The update bot opened a version PR for web-app-jira (#821) although the role is end of life: none of the updaters looked at the lifecycle. utils.update.base.is_maintained now answers that question once, and the collectors for image versions, declared version sources, pip pins, repository refs and the addon check skip a role whose lifecycle is eol.
The lint that requires every semver pin to be watched shares that scope, otherwise it would report the pins of the eol roles as unwatched. Every other stage, deprecated included, stays in the update scope.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The version-source resolver orders upstream versions by semver. A pin that
is not one, such as `latest` or a branch name, has no depth, so no candidate
ever matches it and the declaration never moves anything. Nothing reported
that: the watched-pins lint only looks at values that already are a semver.
test_version_keys now fails on an `update:` block whose pinned key does not
hold a semver. A monitored addon the resolver does not read is held to the
same rule as soon as it pins a `version`, because the catalog check skips a
non-semver pin just as silently. An addon without a `version` keeps tracking
the app default and is left alone.
The Stalwart README no longer asks for `webui_version` to be bumped by hand
next to the image tag. Its `update:` block names the upstream tags, the
server itself follows `releases/latest` of the bundle, and the resolver
plans v1.0.5 to v1.0.11 for it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The prose explaining why offsets are reused and how the hub keeps index
zero is recorded in requirement 037. A reader of plan.py needs the signature,
not the history.
No conflicts. Two files were touched by both sides and both kept each
side: the Makefile, where upstream retires the mcp-audit target and this
branch pins the throwaway cluster's vpn axis, and requirement 035, where
the role table keeps the single svc-net-wireguard row.
Incoming carries the fixes for two applications this branch's last run
failed on -- penpot probing its frontend without curl, and mobilizon's
postgres 18 migrations -- plus dependency bumps and a lint repair.
Gates on the merged tree: unit 6202, lint 523, integration 330,
ansible-lint clean.
The version-source updater only read meta/services.yml. An addon pinned in
meta/addons/<id>.yml stayed where it was: the ONLYOFFICE connector for
Nextcloud sat on v10.1.2, which does not install on Nextcloud 35, until
someone bumped it by hand.
The updater now reads addon files too. An addon declares its source in the
`update:` block it already has, beside monitored, catalog and upstream_id.
A monitored addon of the github-releases catalog needs nothing more: its
upstream_id names the repository whose tags are its versions. The
update-version-sources job opens the PRs, one per role.
Every pinned addon is wired:
- nextcloud onlyoffice carries its own version pin now. Its archive URL
stays a literal that carries the pin, because
test_pinned_addon_archives_available reads it as written
- wordpress mcp-adapter, moodle auth_oidc and jellyfin sso-authentication
follow the tags of their repository, moodle within its v4.5 series
- the three xwiki extensions read their Maven metadata
- pretix-oidc reads the PyPI simple index. The JSON API answers 404 behind
the package cache the test stack routes pypi.org through
- joomla's plugin is built here and jellyfin's ldap-authentication is bound
to the md5 of its archive; both are opted out with that reason
A bump rewrites the pin and the config.archive line together, and an
archive that does not carry the pin fails test_version_keys, so the two
cannot drift apart again. test_version_pins_are_watched covers addon pins
as well: a new one without a source or a reasoned opt-out fails the lint.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Docker updater watches image plus version, the repository updater
repository plus ref. A dozen pins fell outside both - an app_version, a
release, a tag in a registry neither speaks - so nothing ever told anyone
they had gone stale.
Such a pin now names its upstream next to itself in an `update:` block:
git_tags, registry_tags, npm, http_regex or a role-local script, with
`match` selecting a tag flavour and `strip` removing a prefix the pin does
not carry. utils/update/source.py resolves them beside the two existing
updaters, `python -m cli.contributing.update.source` bumps them, the
update-version-sources job opens one PR per role, and
tests/external/update/source warns while a bump is pending.
Two lints hold the rule: every semver pin is watched by one of the three
updaters or opted out with a reason, and a semver only ever sits behind
`version`, `<what>_version` or `ref` - which renames moodle's release to
app_version and simpleicons' local_version to local_tag, the latter being
the tag of a locally built image rather than an upstream version. A
declaration naming a key the entity does not carry, or a type the resolver
does not know, fails the lint instead of resolving nothing in silence.
Picked from feature/store-i18n (94d607fd0a) onto current main. Two things
differ from the original: svc-ai-lmstudio no longer carries an app_version
pin, so its declaration is dropped, and web-app-stalwart's webui_version,
added since, declares the git tags of the webui repository.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The report aggregated each role's meta/mcp.yml into one table, and its
freshness test failed on every image-version bump of an MCP-serving role,
because the updater edits services.yml and nothing regenerates the committed
report. The table added nothing the sources lack: scope and credential subject
were identical for every role, so it restated its inputs.
The generator, the report, its freshness test, make mcp-audit and the i8mcau
alias go, and make docs no longer calls the generator. The metadata stays in
each role's meta/mcp.yml, where test_mcp_schema.py enforces its shape, and
requirement 035 now says so. The classification and completeness lints stay,
because they check the lists in 035 against the roles, not the report.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One conflict, in the generated role table of README.md. Both sides changed
the same rows: upstream swapped WordPress's Mailu dependency for Stalwart,
while this branch replaced the three svc-net-wireguard-{core,plain,
firewalled} rows with the single svc-net-wireguard it deploys.
Resolved by regenerating rather than by hand, since make readme-index owns
that table and gates it with --check. The regenerated table carries both
sides and the check reports it up to date.
README.md was the only file both sides touched, and nothing in the tree
still names the three replaced roles.
Ansible appends -tt to any task it cannot pipeline when the task escalates,
and ansible.builtin.script is one of those, so the deploy over the mesh ran
those tasks on a pty. A pty ends every line with a carriage return, while
the container connection this transport replaces returned bare newlines. A
role that reads its own stdout therefore started failing for no reason other
than the deploy having moved onto the mesh: the Matomo bootstrap parsed a
perfectly good token as empty, because its regex anchors each line and a
trailing carriage return defeats that anchor.
The host the tool switches now asks for no tty, so the mesh returns the same
bytes the connection it replaced did, and no role has to learn about the
transport underneath it.
Taken from the connection plugin rather than guessed: the tty is added when
the task cannot pipeline and escalates, and the option carrying it is
exposed per host as ansible_ssh_use_tty.
lifecycle.md said a deprecated role is still covered by the test suite while the tested envelope already left it out. A deprecated role is now untested: the matrix-deploy MUST NOT exercise it and a failure in it MUST NOT block a release, while planned, pre-alpha and eol stay open to best-effort runs. The complexity legend says untested too.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One conflict, in the MCP exclusion list of requirement 035. Both sides
edited the same line: main added svc-net-tor-smtp, this branch replaced the
three svc-net-wireguard-{core,plain,firewalled} entries with the single
svc-net-wireguard it deploys. Resolved to the union of both — 24 entries —
which is what the merged tree actually holds.
Everything else merged clean and was checked for overlap rather than
assumed: no reference anywhere still names the three replaced roles;
matrix.py kept main's PRIMARY_APPS alongside the reordered round, and the
bootstrap pass does not need it, since only one template reads it and
group_vars defaults it empty; and main's ansible.cfg addition sits in
[defaults], so the ssh_args the mesh tool derives its own connection
settings from are untouched.
The pass that runs over the mesh reaches every host by SSH where the
pass before it used the container connection, and it pays that difference
once per task. Measured in a container: an exec through a warm multiplexer
costs 7ms, a fresh connection 85ms, and 165ms once sudo asks for the
password every task demands. The pass runs thousands of tasks.
Two settings threw the multiplexer away. A persist window of 20s closes the
master whenever a host waits longer than that for its next task, which in a
linear play is most of them; and the named ControlPath points at a directory
nothing creates, so a master that cannot open its socket reconnects per task
without reporting it.
Neither is retuned for the repository. The tool that moves a host onto the
mesh now writes that host's own ssh_args, derived from the repository's
value so every other option is kept and only the window and the socket
differ. A host the mesh never touches keeps exactly what it had.
Confirmed against the built command: a switched host gets ControlPersist=600s
and a ControlPath under the directory Ansible creates, while a host beside it
keeps the repository's own.
The offset came from the spoke's position in a sorted list, so a host that
sorts earlier renumbered every member behind it. `wg syncconf` carries keys
and peers onto a live interface and never its address, so a renumbered member
keeps answering on the old address while the peers and the inventory have
moved to the new one -- and the address it gave up can be handed to a new
member while the old owner still holds it.
The planner now reuses the addresses already on disk and allocates only an
unseen member, from the lowest free offset. A value outside the subnet, a
duplicate, and the hub's own offset are refused, so a mirrored host_vars
cannot hand a spoke the hub's address. Rotation keeps addresses: it replaces
identities, not addresses.
The routing assertions re-planned the mesh and compared the result with
itself, so a writer that stored the mesh subnet in place of the routed range
would have passed; they now read the entry off disk, which is what the role
reads.
A host in two meshes has one private key, so the second mesh must reuse what
the first minted. The lookup prefers the entry of the mesh being written, which
after a rotation is the stale pre-rotation key, so the hub ended up advertising
one key on the swarm plane and another on the data plane while holding the
private half of only one. Its workers could never handshake again.
The run now carries what it decided: a host keyed while planning one mesh keeps
that key for every mesh after it.
The existing-key lookup walked the inventory hosts, and the controller joins
by name rather than through a group, so it was the one member whose key was
never found and always minted fresh. Every write then changed the hub's peer
list, and the converge re-keyed the swarm tunnel for no reason.
all_hosts already accounts for the controller two lines above; the lookup now
uses it, as the prune already does.
The mirror copies one host's file over every other so a rotation reaches all
of them, which is right for a shared secret and wrong for anything a
generator writes per host: the copy hands every node the source's identity.
--mirror-keep names the subtrees that belong to the destination and are put
back after the copy. Absent, the mirror behaves exactly as before.
--endpoint HOST=ADDRESS, repeatable. A topology whose inventory names do not
resolve on every member states the address once, where the mesh is written,
rather than every member having to be taught the names.
meta/users.yml states 'password: {{ ansible_become_password }}' to say the
account shares the become secret. Provisioning pinned a fresh random over it,
so the account was created with one secret while become sent another; the
lookup returns the expression as a literal, so skipping the pin would hand
the account the text instead. Resolve the reference against host_vars and pin
that value, vault node included.
Any deploy that reaches a host as the administrator over ssh hits this, not
just the mesh: the container connection escalates to root without a password,
which is why nothing caught it.
A declared password is a dict when the role asks for a policy and a string
when it names the value. Only the dict case needs a generated secret, but
both took one, so user-administrator's '{{ ansible_become_password }}' was
overwritten by a random pin: the account was created with one secret while
become sent another.
Nothing caught it because the container connection escalates to root without
a password. The first transport that logs in as the administrator and sudos
for real gets 'Incorrect sudo password' on every host.
The prune walks the hosts the inventory groups resolve to, but the controller
joins by name rather than through a group, so it was never visited. Its
mirrored host_vars kept the manager's data-mesh entry -- address 10.100.1.1,
is_hub true -- and the controller brought that interface up and enabled hub
forwarding for a mesh it is not a member of.
The discovery step builds its table with plan.py, not matrix.py, so passing
--vpn aborted the chunk before a runner was spent on it. The plan now takes
the flag and shows the state as a column, the way it already shows the onion.
Named rather than resolved from a group: the controller is not a swarm member
and has no role group, and inventing one would put a non-role name into
group_names where every service and placement lookup would special-case it.
The first pass of a deploy reaches the hosts however it can, because no tunnel
exists yet and that pass is what creates one. Every pass after it connects
over the mesh, which is what makes the mesh load-bearing rather than merely
present: a broken tunnel then fails the deploy at the connection, and no role
can quietly fall back to the underlay it was supposed to stop using.
A separate step rather than a fact set during the play, because a play cannot
change the transport it is already running over. A host holding no address on
the mesh is left alone -- rewriting it would strand a host that was
reachable.
A host presents the same identity on every plane it belongs to, so the hub no
longer carries two identities and the writer has one credential to keep in
step. A member already keyed by an earlier mesh keeps that key instead of
minting a second one.
The pool is declared once under NETWORK_* and referenced from the mesh file.
Ansible resolves that at play time; this writer runs before the play, so it
follows the one hop rather than duplicating the value.
Brings in 225 commits, including the merged web-app-openbao work and the
LMStudio loopback fix.
Seven conflicts:
- The legacy svc-net-wireguard-core/plain/firewalled roles this branch deleted
in e706170a0 were modified upstream. The deletion stands, which is the whole
premise of the branch.
- The MCP audit takes upstream's list plus svc-net-firewall, with the three
legacy wireguard entries collapsed into the single svc-net-wireguard: 23.
- web-app-openbao/README.md takes main's reviewed version wholesale.
- matrix.py takes both sides: upstream's app_id argument to _reset_credentials
and this branch's _write_mesh call after it.
The merge pushed matrix.py to 511 lines against the 500 cap, so the step
runner moved to utils/tests/swarm/run.py, which is self-contained and leaves
matrix.py at 427. Its test moved with it.
An image bump in a provider redeployed every variant of every consumer,
including the variants that switch that provider off and therefore never
deploy the bumped image at all. The diff-derived whitelist now narrows each
affected role to the variants whose own round closure still reaches a seed,
emitted as `role#0,2` selection tokens. That closure is walked on the same
variant-merged services maps the matrix planner builds its rounds from, so the
narrowing follows a parent's variant choice across every generation of the
chain. It covers every auto-derived whitelist: the weekly update PRs,
Dependabot, and any other diff-driven run.
The narrowing never shrinks the matrix on partial information. A seed role
keeps every variant, because its own files changed. A role reached only
through a run_after edge keeps every variant too, because the round closure
does not follow that edge and a role whose hits are empty is therefore
unnarrowed rather than dropped.
A pinned token naming a role the discovery query does not return now warns
instead of aborting the run. The diff-derived whitelist legitimately names
roles the lifecycle envelope filters out, and once it pins variants such a
token would abort every chunk; role names are still checked against roles/
before the run starts, so a typo is caught one layer earlier.
The round's variant merge moves out of the deploy inventory package into
utils/roles/applications/variants.py. The CI selection has to build the same
maps, and importing the deploy package for them pulls in a module that reads
INFINITO_SRC_DIR at import time, which the runner host does not set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5.5
evangelostsak
Alejandro Roman