Commit Graph
102 Commits
Author SHA1 Message Date
evangelostsak 1c5ba164f2 Merge remote-tracking branch 'upstream/main' into feature/svc-net-wireguard
No conflicts across 121 incoming commits. The one file both sides changed is
tasks/stages/01_constructor.yml: upstream parks an applications carrier for
the swarm bootstrap roles after the add_host tasks, and keys the carrier on
the tokens file as well. This branch's mesh bootstrap opens and drops its own
block-scoped carrier before that point, so both keep working as written.

Upstream also brings ruff 0.17.0, distro-derived env keys, a scoped act outer
teardown, the stalwart relay rule, the onion apex redirect and image and
version updates; ruff 0.17 reformats nothing on this branch.
2026-10-15 21:27:11 +02:00
kevinveenbirkenbach c38f95457b Merge pull request #955 from infinito-nexus/fix/constructor/bootstrap-carrier/925
perf(constructor): carry the applications SPOT into the bootstrap roles
2026-10-15 17:54:46 +02:00
kevinveenbirkenbachandClaude Opus 5.5 33bb7e99e9 fix(lint): resolve puppeteer-core like node and keep other browser revisions
- browser_revision() reads revisions.js from the puppeteer-core that
  node resolves for mmdc's puppeteer: puppeteer/node_modules first, then
  the node_modules directories above it. When mermaid-cli pins another
  puppeteer-core, npm nests puppeteer's own copy, and the sibling held a
  different revision.
- A failed browser install removes only the directories of the pinned
  revision before the retry. Removing the whole chrome-headless-shell
  cache also deleted browsers that other puppeteer installs share.

Addresses the CodeRabbit review on #958.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 11:23:22 +02:00
kevinveenbirkenbach 706745162b Merge remote-tracking branch 'origin/main' into fix/lint/mermaid-browser-revision 2026-10-15 10:36:09 +02:00
kevinveenbirkenbachandClaude Opus 5.5 c74a31e83e fix(lint): install the browser revision mmdc's own puppeteer pins
install-lint provisioned chrome-headless-shell with `npx --yes
puppeteer browsers install`, which runs the latest puppeteer from the
registry. mmdc runs with the puppeteer npm installed next to it as a
peer dependency when mmdc itself was installed, and looks for exactly
the revision that puppeteer-core pins. Once puppeteer 25.14.0 was out,
npx installed 155.0.8059.79 while mmdc with puppeteer-core 25.13.0
wanted 155.0.8059.39, and lint-mermaid failed on every file. Because
browser_installed() accepted any chrome-headless-shell, a stack with
the wrong revision never repaired itself.

The browser is now installed through the puppeteer CLI next to mmdc,
which installs its puppeteer-core's revision, and only that revision,
read from puppeteer-core's revisions.js, counts as installed.

Closes #957

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 03:21:30 +02:00
kevinveenbirkenbachandClaude Opus 5.5 778c18fb23 perf(constructor): carry the applications SPOT into the bootstrap roles
The constructor parked no applications carrier between the merge block
and the token store, so every lookup in the svc-swarm-node, svc-storage,
svc-registry and svc-virt bootstrap rendered the SPOT cold, 25 to 30 s
each on a loaded host.

A carrier there was unsafe: in swarm, svc-storage-nfs-server mounts the
NFS export over the controller's DIR_VAR_LIB, so the tokens file the
render reads can appear in the middle of that section, and the cache key
did not notice. merged_applications_cache_key now includes the tokens
file's path, mtime and size, as the users cache key already did, so a
carrier rejects itself once the tokens change. A carrier parked after
the add_host tasks then serves the bootstrap roles. It is parked only in
swarm mode: in compose the section runs in about a second, so a carrier
there would add a render without saving one.

In a swarm drill of web-app-prometheus up to the known NFS probe-write
failure, "Resolve the NFS server backend flavor" went from 29.6 s to
0.07 s, "Probe-mount the NFS export" from 26.4 s to 0.4 s, "Mount
cluster-shared infinito state on controller" from 23.7 s to 0.5 s and
"Ensure Docker-In-Docker DNS works" from 30.0 s to 3.6 s; the run to
that point took 768 s instead of 1735 s.

Closes #925

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 03:21:17 +02:00
kevinveenbirkenbachandClaude Opus 5.5 47565781ea style(python): import datetime as dt for ruff 0.17
ruff 0.17.0 adds `datetime` -> `dt` to the default aliases of
flake8-import-conventions, and the project selects ICN, so lint-python
failed with ICN001 in seven files.

The files now import `datetime as dt`. ruff's unsafe fix also renamed
two local names in utils/github/playwright_summary.py into the alias:
the `_to_iso_z` parameter and the parsed value in
`_parse_iso_timestamp`. Both are now called `moment`; with the bare fix,
nine tests of test_playwright_summary.py failed with UnboundLocalError
and AttributeError.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 01:24:31 +02:00
evangelostsak 8386823b2a Merge remote-tracking branch 'upstream/main' into feature/svc-net-wireguard
No conflicts. The 14 incoming commits fix the n8n gateway probe trigger under
n8n 2, let stalwart address bans expire, keep the Docker apt key when its
refresh is refused, leave eol roles out of image discovery, stop the release
backfill below the oldest buildable tag, and provision the mermaid browser
only in install-lint. None of them touches the mesh, swarm, NFS, network or
workflow files this branch changes.
2026-10-15 00:38:14 +02:00
evangelostsak 365d78462e test(swarm-test): extras read the interfaces list 2026-10-14 15:03:42 +02:00
evangelostsak 514a4e11b1 test(swarm-test): the round reads and hands over a layout 2026-10-14 15:03:42 +02:00
evangelostsak 2f96b52580 test(swarm-test): extras bind swarm and NFS to the primary mesh 2026-10-14 11:23:23 +02:00
evangelostsak af74e70ac0 test(swarm-test): the round deploys the declarations it is pointed at
The transport tests pin INFINITO_SWARM_MESHES to the default, so an operator
.env pointing at an example does not change what they assert. A controller
link left by other declarations is removed before the controller joins, and
one outside the pool is left alone.
2026-10-14 11:23:23 +02:00
kevinveenbirkenbach 67b508cc5d Merge pull request #918 from infinito-nexus/fix/images/retired-role-discovery
fix(images): leave eol roles out of image discovery
2026-10-14 10:04:03 +02:00
kevinveenbirkenbachandClaude Opus 5.5 078d0a0682 fix(lint): provision mmdc's browser only when it is missing
install-lint provisioned chrome-headless-shell on every run, and `make lint`
runs install-lint once per lint target in parallel whenever its stamp is not
fresh. On CI the stamp never is (eslint lives in node_modules, not on PATH),
so one lint job provisioned 16 times (run 38455552981). As root, puppeteer's
--install-deps runs `apt-get satisfy` even for an installed browser, and two
of those at once fail on the dpkg lock; a failed provisioning then deleted the
shared browser cache while lint-mermaid could be rendering with it.

The browser is now provisioned only when it is not installed, and --force
drops its cache so `make install-lint-force` still reinstalls it with its
libraries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 08:56:59 +02:00
kevinveenbirkenbachandClaude Opus 5.5 fd9286b24f test(lint): mark the stub call log as written by the test
test_no_uncached_file_reads rejects Path.read_text in tests unless the file is
written in the test itself. The mermaid script test reads the call log its
stubs append to during the run, so that read carries `# nocheck: cache-read`.
The mermaid cli test is formatted with ruff.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 01:32:35 +02:00
kevinveenbirkenbachandClaude Opus 5.5 8a29ba0bfd fix(images): leave eol roles out of image discovery
The scheduled mirror job has failed every day since 2026-09-25 on the same two
images: quay.io/minio/minio and quay.io/minio/mc answer `unauthorized` to
anonymous pulls ("2/197 failed", run 38352569503). Both belong to
web-app-minio, an eol role outside the lifecycle envelope that default.env
tests, so no CI deploy pulls them.

iter_role_images now skips every role that is_maintained() reports as
unmaintained, the check the version updaters already apply to the same list.
On the tree the discovery yields 187 images instead of 192; the five it drops
belong to the four eol roles web-app-confluence, web-app-jira, web-app-minio
and web-app-phpldapadmin. The mirror sync, resolver and wait read this list,
and their descriptions and mirror.md now say so.

The roles directory is resolved first: is_maintained reads a relative path as
a role name and would keep every role.

475ab9b386 on feature/store-i18n takes the same approach with a lifecycle
reader of its own; main already has is_maintained for it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 01:19:28 +02:00
evangelostsak 93ab3f326c test(ci): hold both mesh arms inside every default sweep
Eight compose-or-swarm rows over eight sweeps: each sweep must put swarm rows
on the meshed and the direct arm, and no compose row may carry the mesh. Fails
on the previous wants_vpn, which held every swarm row of a sweep on one arm.
2026-10-14 00:27:17 +02:00
kevinveenbirkenbachandClaude Opus 5.5 c8cb7fda8c fix(lint): provision the mermaid browser in install-lint, not in the parallel lint run
make lint runs install-lint once and then dispatches its fifteen targets
in parallel. scripts/lint/mermaid.sh provisioned chrome-headless-shell
inside that parallel phase with `npx puppeteer browsers install
--install-deps`, whose `apt-get satisfy` carries no lock timeout. In a
fresh stack container another target held the apt lock, the shared
libraries stayed missing, the script only warned, and all 289 Markdown
files then failed on libnspr4.so (#795).

utils/install/lint/mermaid_cli.py now provisions the browser right after
mmdc, inside the serial install-lint step, as root with --install-deps
and as a user without it, because puppeteer refuses to install libraries
without root. A failed install deletes the chrome-headless-shell cache
folder and retries once under a 900 second timeout per attempt; a cut-off
download otherwise leaves a folder without its executable, and puppeteer
refuses every later install of that build. The install-lint stamp counts
as fresh only while the browser executable exists, so a recreated
container provisions it again.

mermaid.sh no longer calls apt-get or npx. It renders one minimal diagram
first and, when that fails, exits with mmdc's output instead of reporting
every Markdown file.

Verified: tests/unit/python/scripts/lint/test_mermaid.py stubs mmdc,
npx and apt-get. With a browser that cannot load libnspr4.so the script
exits 1 with that line, before any Markdown file renders, and mmdc is the
only tool it calls. Against main's mermaid.sh the same test fails: it
calls apt-get and npx and reports the files one by one. The installer and
stamp cases pass, shellcheck is clean.

Not verified: a full make lint in a recreated stack container. The race
reproduced in one of two runs in the issue.

feature/store-i18n changed the provisioning in mermaid.sh with 2f5f52bd1a
and dc67bce5f0 (timeout, deleting a cut-off browser). Both behaviours
moved into mermaid_cli.py here, so its merge takes this version of
mermaid.sh.

Closes #795

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:31:36 +02:00
evangelostsak 34b49036d7 Merge remote-tracking branch 'upstream/main' into feature/svc-net-wireguard 2026-10-12 17:52:36 +02:00
kevinveenbirkenbachandClaude Opus 5.5 1ba796d7f1 fix(ci): pull the images the runner uses itself from the GHCR mirror
The GHCR mirror covered role images only. The images the runner pulls on
its own went straight to Docker Hub: coredns from compose.yml, the cache
stack in compose/cache.override.yml and alpine:latest, the init image of
scripts/docker/cache/package-frontend/certs.sh. With the anonymous
githubactions quota spent, the guide step of Postmarks#0 (main run
37954203718) and Shopware#2 (#872) died on "toomanyrequests" while pulling
alpine:latest, after coredns had hit the same limit.

iter_runner_images (utils/docker/image/discovery.py) finds those images,
and the mirror sync and the fork-PR wait now cover them next to the role
images. On a real GitHub runner compose_file_args layers a generated
build/compose/runner-mirror.override.yml that points only the services of
the layered compose files at their mirror, and the certificate generator
gets the mirrored init image. Local runs and act are unchanged, and the
coredns literal stays in compose.yml for Dependabot. test-dns now waits for
images-mirror-missing like the deploy and workspace jobs, since it brings up
the stack too.

Verified with docker compose config on a simulated GitHub runner: the
service set is unchanged and only the runner images point at
ghcr.io/infinito-nexus/core/mirror/docker.io/..., and an existing role mirror
under that scheme pulls anonymously.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 10:37:41 +02:00
evangelostsak 5a83785b53 Merge remote-tracking branch 'origin/main' into feature/svc-net-wireguard 2026-10-09 12:00:07 +02:00
kevinveenbirkenbachandClaude Opus 5.5 5b722d840d style(update): reformat two test statements for ruff
ruff format rewraps the read-back line in test_repository.py, and ruff
check reports the nested with in test_fetch.py as SIM117. Both fail
lint-python.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 11:50:05 +02:00
kevinveenbirkenbachandClaude Opus 5.5 4cba7e7c2f feat(update): read a version source from several documents, once per run
An http_regex source read one document, so a tag whose parts upstream
keeps in two files could only carry the second part as a literal. Every
pin also fetched its document on its own: a failed request left one pin
of a release behind while the others moved.

url now takes a list whose bodies are searched as one text, and every
document is fetched once per run, so all pins that read it resolve against
the same snapshot. The HTTP reads move into utils/update/fetch.py.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 11:26:13 +02:00
kevinveenbirkenbachandClaude Opus 5.5 d0b09ec473 fix(update): keep a sourced ref from giving its line to the next entity
The repository updater assigns ref: lines to entities by position: every
entity that pins a value takes the next line carrying that value. A ref
that declares its own source was dropped before it took its line, so a
second entity pinning the same value was given the sourced entity's line,
and an update of it would have rewritten the wrong ref.

A sourced ref now takes its line first and is skipped afterwards, the
same order the nocheck marker already follows.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 11:26:12 +02:00
kevinveenbirkenbachandClaude Opus 5.5 1585e056fc feat(update): let a declared version source own the whole pin
A pin that names its own upstream in an update: block was still collected
by the image updater or, for a ref, by the repository updater. Those raise
it to the newest tag of its registry or repository, which for a component
of a release is not the release's value. A tag built from several upstream
values, such as <freeswitch>-<bigbluebutton>, could not be resolved at
all: the resolver takes one capture and keeps a pin on its suffix.

The image updater now skips a version and the repository updater a ref
that declares a source for itself. An http_regex source accepts a template
over the pattern's named groups, and a templated pin moves whenever the
assembled value changes, its suffix included.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 10:32:14 +02:00
evangelostsak e9929d07c3 Merge remote-tracking branch 'origin/main' into feature/svc-net-wireguard
One conflict, in the generated role overview table of README.md.

Both sides had edited the same block. Upstream dropped Redis from the pihole
row; this branch still listed it. Upstream also listed svc-net-wireguard-core,
-plain and -firewalled, which this branch removed in e706170a0b as superseded
by the unified svc-net-wireguard role.

Resolved by taking upstream's block and then running `make readme-index`, so
the table is generated from the roles that actually exist rather than
hand-merged: Redis is gone from pihole, the three legacy WireGuard roles stay
gone, and svc-net-wireguard is listed once. `make readme-index check=true`
reports the table up to date.

Git kept the role deletions silently and correctly - upstream has not touched
those three directories since the merge base, so there was no modify/delete
conflict to resolve.
2026-10-07 10:04:31 +02:00
kevinveenbirkenbach 0622cce916 Merge remote-tracking branch 'origin/main' into fix/code-scanning-alerts/807 2026-10-06 09:18:00 +02:00
kevinveenbirkenbachandClaude Opus 5.5 3610e7b9ef fix(code-scanning): resolve the alerts that describe real code
Of the 76 open code scanning alerts, 39 point at code that can change.

svc-ai-mcp-adapter:
- A path argument that is empty or made only of dots is refused.
  Percent-encoding keeps a separator inside its segment, but "." and ".."
  need none, so an upstream that resolves dot segments could answer from a
  path the contract does not name.
- parse_duration splits on the units. The pattern it used before rescanned
  a long run of digits from every start position.

Tests and the README generators:
- Every Jinja environment built with autoescape=False and a "noqa: S701"
  uses select_autoescape now, as six test modules and
  utils/templating/jinja.py already did. It escapes templates named
  *.html, *.htm and *.xml, which none of these call sites loads.
- Dead code goes: a doubled "rel" block in the email lint, an unread
  _SKIP_DIRS, and a module that _resolve returned to no caller.
- Two "except ...: pass" blocks become contextlib.suppress and
  assertRaises.
- Nine test modules import unittest.mock one way instead of two, and one
  lookup test imports its module once.

Closes #807

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 09:18:00 +02:00
kevinveenbirkenbachandClaude Opus 5.5 89b95d751c fix(update): stop bumping pins of eol roles
The update bot opened a version PR for web-app-jira (#821) although the role is end of life: none of the updaters looked at the lifecycle. utils.update.base.is_maintained now answers that question once, and the collectors for image versions, declared version sources, pip pins, repository refs and the addon check skip a role whose lifecycle is eol.

The lint that requires every semver pin to be watched shares that scope, otherwise it would report the pins of the eol roles as unwatched. Every other stage, deprecated included, stays in the update scope.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 01:09:10 +02:00
kevinveenbirkenbachandClaude Opus 5.5 4c955d080f feat(update): fail a declared update source whose pin is not a semver
The version-source resolver orders upstream versions by semver. A pin that
is not one, such as `latest` or a branch name, has no depth, so no candidate
ever matches it and the declaration never moves anything. Nothing reported
that: the watched-pins lint only looks at values that already are a semver.

test_version_keys now fails on an `update:` block whose pinned key does not
hold a semver. A monitored addon the resolver does not read is held to the
same rule as soon as it pins a `version`, because the catalog check skips a
non-semver pin just as silently. An addon without a `version` keeps tracking
the app default and is left alone.

The Stalwart README no longer asks for `webui_version` to be bumped by hand
next to the image tag. Its `update:` block names the upstream tags, the
server itself follows `releases/latest` of the bundle, and the resolver
plans v1.0.5 to v1.0.11 for it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 12:48:22 +02:00
kevinveenbirkenbachandClaude Opus 5.5 739c78bad1 feat(update): bump addon pins through their declared source
The version-source updater only read meta/services.yml. An addon pinned in
meta/addons/<id>.yml stayed where it was: the ONLYOFFICE connector for
Nextcloud sat on v10.1.2, which does not install on Nextcloud 35, until
someone bumped it by hand.

The updater now reads addon files too. An addon declares its source in the
`update:` block it already has, beside monitored, catalog and upstream_id.
A monitored addon of the github-releases catalog needs nothing more: its
upstream_id names the repository whose tags are its versions. The
update-version-sources job opens the PRs, one per role.

Every pinned addon is wired:

- nextcloud onlyoffice carries its own version pin now. Its archive URL
  stays a literal that carries the pin, because
  test_pinned_addon_archives_available reads it as written
- wordpress mcp-adapter, moodle auth_oidc and jellyfin sso-authentication
  follow the tags of their repository, moodle within its v4.5 series
- the three xwiki extensions read their Maven metadata
- pretix-oidc reads the PyPI simple index. The JSON API answers 404 behind
  the package cache the test stack routes pypi.org through
- joomla's plugin is built here and jellyfin's ldap-authentication is bound
  to the md5 of its archive; both are opted out with that reason

A bump rewrites the pin and the config.archive line together, and an
archive that does not carry the pin fails test_version_keys, so the two
cannot drift apart again. test_version_pins_are_watched covers addon pins
as well: a new one without a source or a reasoned opt-out fails the lint.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 02:50:48 +02:00
kevinveenbirkenbachandClaude Opus 5 81ab6e4ffc feat(update): let a version pin declare its own upstream source
The Docker updater watches image plus version, the repository updater
repository plus ref. A dozen pins fell outside both - an app_version, a
release, a tag in a registry neither speaks - so nothing ever told anyone
they had gone stale.

Such a pin now names its upstream next to itself in an `update:` block:
git_tags, registry_tags, npm, http_regex or a role-local script, with
`match` selecting a tag flavour and `strip` removing a prefix the pin does
not carry. utils/update/source.py resolves them beside the two existing
updaters, `python -m cli.contributing.update.source` bumps them, the
update-version-sources job opens one PR per role, and
tests/external/update/source warns while a bump is pending.

Two lints hold the rule: every semver pin is watched by one of the three
updaters or opted out with a reason, and a semver only ever sits behind
`version`, `<what>_version` or `ref` - which renames moodle's release to
app_version and simpleicons' local_version to local_tag, the latter being
the tag of a locally built image rather than an upstream version. A
declaration naming a key the entity does not carry, or a type the resolver
does not know, fails the lint instead of resolving nothing in silence.

Picked from feature/store-i18n (94d607fd0a) onto current main. Two things
differ from the original: svc-ai-lmstudio no longer carries an app_version
pin, so its declaration is dropped, and web-app-stalwart's webui_version,
added since, declares the git tags of the webui repository.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 02:50:48 +02:00
evangelostsak e456b1eee6 Merge remote-tracking branch 'origin/main' into feature/svc-net-wireguard
One conflict, in the MCP exclusion list of requirement 035. Both sides
edited the same line: main added svc-net-tor-smtp, this branch replaced the
three svc-net-wireguard-{core,plain,firewalled} entries with the single
svc-net-wireguard it deploys. Resolved to the union of both — 24 entries —
which is what the merged tree actually holds.

Everything else merged clean and was checked for overlap rather than
assumed: no reference anywhere still names the three replaced roles;
matrix.py kept main's PRIMARY_APPS alongside the reordered round, and the
bootstrap pass does not need it, since only one template reads it and
group_vars defaults it empty; and main's ansible.cfg addition sits in
[defaults], so the ssh_args the mesh tool derives its own connection
settings from are untouched.
2026-10-03 02:29:45 +02:00
evangelostsak ff75343aba fix(ci): stop a held mesh axis deleting priority rows
The pin narrowed the cross product after the axis was resolved, so a row
pinned against an enforced or disabled run lost every combination and left
the matrix entirely -- while a regular row with the same pin was simply moved
onto the held side. A priority row now takes the same precedence a regular
row already took: the run's axis first, the pin second.
2026-10-02 00:54:20 +02:00
evangelostsak f91f9bccdf test(swarm-test): cover the transport switch
A node is pointed at its mesh address; the controller keeps the connection it
runs the play with. The step had no test, and this entry has now bitten twice.
2026-09-28 08:52:50 +02:00
evangelostsak 02c56f6434 test(swarm-test): pin the become password to the account
Asserts it equals users.administrator.password and that it survives into the
deploy-facing twin, which strips the users block it comes from.
2026-09-27 23:56:22 +02:00
evangelostsak fb2fb0e55c test(ci): cover the mesh axis
A fully pinned priority row now pins the mesh state too, and a priority row
with more combinations than the distro pool has members wraps the rotation
instead of holding one distribution per combination.
2026-09-25 22:34:21 +02:00
evangelostsak 0a6700e378 Merge origin/main into feature/svc-net-wireguard
Brings in 225 commits, including the merged web-app-openbao work and the
LMStudio loopback fix.

Seven conflicts:

- The legacy svc-net-wireguard-core/plain/firewalled roles this branch deleted
  in e706170a0 were modified upstream. The deletion stands, which is the whole
  premise of the branch.
- The MCP audit takes upstream's list plus svc-net-firewall, with the three
  legacy wireguard entries collapsed into the single svc-net-wireguard: 23.
- web-app-openbao/README.md takes main's reviewed version wholesale.
- matrix.py takes both sides: upstream's app_id argument to _reset_credentials
  and this branch's _write_mesh call after it.

The merge pushed matrix.py to 511 lines against the 500 cap, so the step
runner moved to utils/tests/swarm/run.py, which is self-contained and leaves
matrix.py at 427. Its test moved with it.
2026-09-24 21:14:25 +02:00
Alejandro Roman 9df3287adf Merge remote-tracking branch 'upstream/main' into feature/role-stalwart-mail-swarm 2026-09-22 12:16:44 +04:00
kevinveenbirkenbachandClaude Opus 5 5b8465260f feat(ci): deploy only the variants a change reaches
An image bump in a provider redeployed every variant of every consumer,
including the variants that switch that provider off and therefore never
deploy the bumped image at all. The diff-derived whitelist now narrows each
affected role to the variants whose own round closure still reaches a seed,
emitted as `role#0,2` selection tokens. That closure is walked on the same
variant-merged services maps the matrix planner builds its rounds from, so the
narrowing follows a parent's variant choice across every generation of the
chain. It covers every auto-derived whitelist: the weekly update PRs,
Dependabot, and any other diff-driven run.

The narrowing never shrinks the matrix on partial information. A seed role
keeps every variant, because its own files changed. A role reached only
through a run_after edge keeps every variant too, because the round closure
does not follow that edge and a role whose hits are empty is therefore
unnarrowed rather than dropped.

A pinned token naming a role the discovery query does not return now warns
instead of aborting the run. The diff-derived whitelist legitimately names
roles the lifecycle envelope filters out, and once it pins variants such a
token would abort every chunk; role names are still checked against roles/
before the run starts, so a typo is caught one layer earlier.

The round's variant merge moves out of the deploy inventory package into
utils/roles/applications/variants.py. The CI selection has to build the same
maps, and importing the deploy package for them pulls in a module that reads
INFINITO_SRC_DIR at import time, which the runner host does not set.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 22:51:03 +02:00
Alejandro Roman 4b9a834546 Merge remote-tracking branch 'upstream/main' into feature/role-stalwart-mail-swarm 2026-09-21 03:25:20 +04:00
kevinveenbirkenbachandClaude Opus 5 9b436a3162 fix(cache): key the project-tree walk on the root it walked
_all_project_files carried @lru_cache(maxsize=1) and read PROJECT_ROOT from the
module global inside its body, so the root was not part of the cache key. The
first call decided the answer for the rest of the process: point PROJECT_ROOT at
a fixture afterwards and the walk still reports the real repository, or, in the
other order, a sweep after the fixture is torn down enumerates a directory that
no longer exists.

Both failure modes are silent. iter_project_files, iter_project_files_with_content
and iter_non_ignored_files all sit on this one walk, 202 files in the tree use
them, and a lint whose corpus collapses to zero paths reports zero offenders and
passes. Nothing raises.

The root is now a parameter and therefore the key, with room for four of them.
_reset() still clears it through cache_clear(), and the only other caller is
iter_project_files, which passes str(PROJECT_ROOT) explicitly.

Live impact today is none, and that is worth stating: exactly one test patches
PROJECT_ROOT in this module, and it calls _reset() in both setUp and
addCleanup, so the defect was fully masked. Every other patch site in the repo
targets PROJECT_ROOT in a different module. This is about the next test that
forgets, not about a failure anyone has seen.

A regression test covers it rather than a comment. Negative control: with the
key removed the new test fails on the second root with

    Items in the first set but not the second: '/tmp/.../first.txt'

and passes once the key is back; the file is byte-identical to its pre-probe
state afterwards.

The lint docstring that quoted the old cache shape is deleted rather than
updated, because a cross-module implementation reference goes stale the moment
that module changes, which is exactly what happened here.

Same class as c4556fe72d, which keyed the applications SPOT by the groups its
payload reads. That one was a live defect in utils/cache/base.py; this is the
latent twin one module over.

Residual: maxsize=4 is a choice, not a measurement. More than four roots alive
at once evicts the oldest, which re-walks rather than answers wrongly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 07:44:02 +02:00
kevinveenbirkenbachandClaude Opus 5 86f7d8c4f5 fix(install): keep the collection check inside the slice that runs it
The image runs this module from a layer that holds three files of utils:
__init__.py, install/__init__.py and install/collections.py, the slice
ef45d69dc3 cut so the source tree cannot invalidate the collection install.
67f5398985 established that the module therefore stays on the standard
library, and d4ae5cf9f0 broke it 15 minutes later by routing the reads
through utils.cache.files to satisfy the cache-read lint. The import raises
ModuleNotFoundError: No module named 'utils.cache' in that layer, which no
import inside a full checkout can show, because this repository installs
utils into site-packages.

Nothing turned red. The caller reads only the exit code, and the module's
contract is exit 1 with the missing collections on stdout, so a crash - exit
1, nothing on stdout - is indistinguishable from work to do. Every image
build since has reinstalled every collection with --force-with-deps, which
is the cost the check exists to avoid. The deploy job the traceback was
found in finished green.

The reads go back to Path.read_text with a cache-read marker each. The cache
was never earning its place here: read_text is an lru_cache keyed on the
path, and one run reads the requirements file once and one MANIFEST.json per
collection, eight distinct paths, none of them twice. Copying utils/cache
into the slice instead would work - the closure is three stdlib-only files -
but it binds a package with 51 commits in the last twelve months to the
layer that installs ansible, in front of a cache with no possible hit.

The caller now exits 3 when a non-zero status arrives with an empty stdout,
a state the contract cannot produce, so the next crash stops the install
instead of quietly reinstalling everything. all.sh runs under set -e and
touches the stamp only after ansible.sh returns, so the failure also leaves
the stamp stale rather than recording an install that did not happen.

The probe that would have caught this builds the slice in a tempdir from the
Dockerfile's own COPY lines and runs the module inside it. It has to pass
python -S: without it the interpreter resolves every submodule from the
installed checkout however the slice is built, and the test passes on a
module that cannot import in the image. Verified by injecting the import
back into the tempdir copy, which reproduces the traceback verbatim.

Delta measured by loading the committed module and this one side by side:
identical on both requirements files, against the real collections directory
and an empty one, on a missing file, on non-UTF-8 input and on a corrupt
MANIFEST.json. The one behavioural difference is that a file changed within
a process is now read fresh rather than from the memo.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 03:03:37 +02:00
kevinveenbirkenbachandClaude Opus 5 c4556fe72d fix(cache): key the applications SPOT by the groups its payload reads
A service flag may be written as `{{ '<role>' in group_names }}`; 114
meta/services.yml files do exactly that, and it is the only host-dependent
variable any of them uses. The rendered payload therefore differs per host
while the cache key did not: it was built from applications, users,
DOMAIN_PRIMARY and SYSTEM_EMAIL_DOMAIN alone, and in the failing job all
five host_vars files were byte-identical.

113113e553 moved the render out of every task on the premise that Ansible
forks a fresh worker per task, so the in-process cache is always cold. That
holds for workers and not for the strategy's main process, which renders a
task's name once for the first host of a batch. Anything cached there is
inherited by every worker forked afterwards, and applications.py consults
that process cache before the host's own carrier fact, so an inherited
entry outranks the correct answer.

Pretix#2 of run 35152776835 shows both sides in one task: `skipping:
[web-app-pretix-swarm-mgr-01]` against `included: dedicated.yml for
wrk-01, wrk-02`, eleven seconds before mgr-01 resolved its database to
`pretix_database`. The swarm was running the central `postgres_postgres`,
so the wait never converged and the job died after 600 attempts. Variants 0
and 1 pin the flag to a literal and both pass; only variant 2 inherits the
Jinja.

The groups join the key as a sorted comma-joined string rather than a
tuple, because carrier.py stringifies only one side of its key comparison.
A non-scalar component would make the carrier silently stop matching and
every lookup re-render.

domains derives from the applications view and inherits the same
dependency, so this closes a second instance that had not surfaced yet.
users reads group_names nowhere. No extra rendering results: group_names is
constant inside a worker, and the one process that sees several hosts is
the one the split is for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 21:12:42 +02:00
kevinveenbirkenbachandClaude Opus 5 c693f4f619 style(quality): apply what the gate wrote back into the tree
Nothing here was typed by hand. The docs generator rebuilt the MCP audit
table against pins that had already landed, so fider reads v0.36.0 instead
of stable, jenkins 2.568.3-lts instead of lts, pretix 2026.7.0 instead of
stable, and flowise 24-slim instead of 24-bookworm-slim. That is the table
catching up with da665ab69 and 10c5b35e8, reading the same declaration the
updater reads.

Autoformat wrapped three test files from the same series: one condition
pulled onto a single line, three assertEqual calls broken across lines.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 19:35:05 +02:00
kevinveenbirkenbachandClaude Opus 5 badfe72a4f test(update): build the release-group fixture from the role-file constant
test_file_spot failed the gate on the entry fixture, which spelled
meta/services.yml out by hand. The path now comes from utils.roles.mapping
as ROLE_FILE_META_SERVICES, following 4a0fca2bcb, which fixed the same
lint on the addon fixture next door.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 19:26:41 +02:00
kevinveenbirkenbachandClaude Opus 5 e14e73fba0 refactor(docker): let a moving tag declare itself with the marker it already has
A pin outside semver was given its own 'docker-ref' marker on the theory
that every class needs one. That holds for a digest and not for a tag. A
digest pin declares something about the registry, that bytes move under a
standing tag, which no other choice says. A tag outside semver declares
that upstream publishes nothing orderable, which is exactly what
'docker-version' already means and what minio and the act runner already
carry on their version lines.

Keeping both would have put '# nocheck: docker-version, docker-ref' on
every such line: two keys, two checks, one sentence said twice. 'docker-ref'
is dropped instead. It lived in this one mapping, was never entered in the
suppression catalog, and no check ever read it.

The semver entry goes with it, because a semver pin never needs a
declaration. The test that asserted three distinct rules is replaced by two
that pin the separation where it is real: a digest marker differs from a
tag marker, and a semver pin has no marker at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 17:47:02 +02:00
kevinveenbirkenbachandClaude Opus 5 e66005ab07 fix(update): move a role's images as one release or not at all
Seven roles pin several images at one shared version, because those images
are one product: the four Jitsi components, six GitLab CNG images, eight
BigBlueButton services, three Penpot, plus Mastodon, Funkwhale and
Nextcloud Talk. The bump job walks entries one by one and takes each
image's own newest tag, so a run landing between upstream's first and last
push raises the components already there and leaves the rest behind. What
deploys afterwards is a mixed set, and for Jitsi a mixed set breaks video.

The window is real rather than theoretical. Measured on Docker Hub,
'stable-9457' exists for jitsi/prosody, jitsi/jicofo and jitsi/jvb but not
for jitsi/web, one build out of sixty.

Candidates are now grouped by role and current version, and the group is
held to the newest tag the intersection of all their tag lists still
offers. A group with nothing newer in common yields no update at all rather
than a partial one. A lone entry passes through untouched, so nothing
changes for the large majority.

The key is deliberately the role and the version, not the image namespace.
A namespace rule does not survive the data: prom/* carries four components
at four different versions, opentalk/* three, and minio, taigaio and
markoshust each two. Grouping by namespace would hold back pins that are
correctly independent.

Five of the seven roles have carried this exposure all along; the channel
grammar in the previous commit added the sixth by making Jitsi orderable at
all. This closes all of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 17:45:28 +02:00
kevinveenbirkenbachandClaude Opus 5 23fcad5ddf feat(update): read a release line written before the number as a family
Upstreams that ship several channels in parallel put the channel first:
LiteLLM publishes 'main-v1.77.7-stable' next to 'main-v1.77.7-nightly',
Jitsi ships 'stable-9646'. The grammar only allowed a leading 'v', so both
shapes parsed as nothing at all, sat at depth 0 next to 'latest', and the
bump job never looked at them.

The numeric core now takes an optional leading channel, and the channel
folds into the flavor the same way the patch letter already does.
Everything that is not the number is family; only the number is order. That
is what keeps '-stable' away from '-nightly' and 'stable-9646' away from a
bare '9646', which would leave the channel and change what the role
deploys.

The change is additive. Only a tag starting with letters followed by a dash
can match the new group, and every such tag failed the grammar outright
before, so no tag that parses today parses differently. The regression case
pins the four shapes that already worked, and the new cases pin the two
directions that must not happen along with the tags that must stay refused:
'act-latest', 'bookworm-slim', 'main-latest' and 'buildx-stable-1' carry a
channel but no version.

Two pins in this repository become orderable as a result: the four Jitsi
components and the LiteLLM gateway.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 17:43:13 +02:00
kevinveenbirkenbachandClaude Opus 5 51dc06ad09 feat(update): order vendor patch counters and five-component versions
Two tag shapes in this repository name a fixed release yet failed the
semver gate, so the bump job skipped them and nothing could tell whether a
newer one existed: the Checkmk counter '2.4.0p32' and the Collabora
five-component '26.04.2.1.1'. Both returned depth 0 and fell into the
unorderable residue next to 'latest' and 'alpine'.

The numeric core now takes up to five components, and a counter written as
a letter plus a number is parsed into its own group. The letter joins the
flavor and the number joins the ordering key, which is what keeps
'2.4.0p32' in a different family from the four-component '2.4.0.32': a
bump between them would silently change what the role deploys. The number
in the key also makes p9 order below p10 instead of above it.

version_flavor reads the parse result positionally, so the wider tuple
leaves it untouched. Its contract does change for counter tags, which now
report the letter as their flavor; every shape the grammar already handled
keeps its classification, pinned by a regression case in the new test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:44:28 +02:00