Commit Graph
85 Commits
Author SHA1 Message Date
evangelostsak 6f24461720 refactor: drop rationale and narration from tests/unit/python/plugins/lookup/test_mesh_ranges.py 2026-10-16 19:10:01 +02:00
evangelostsak d480c2785e refactor: drop rationale and narration from tests/unit/python/plugins/lookup/test_mesh_address.py 2026-10-16 19:10:01 +02:00
evangelostsak 1c5ba164f2 Merge remote-tracking branch 'upstream/main' into feature/svc-net-wireguard
No conflicts across 121 incoming commits. The one file both sides changed is
tasks/stages/01_constructor.yml: upstream parks an applications carrier for
the swarm bootstrap roles after the add_host tasks, and keys the carrier on
the tokens file as well. This branch's mesh bootstrap opens and drops its own
block-scoped carrier before that point, so both keep working as written.

Upstream also brings ruff 0.17.0, distro-derived env keys, a scoped act outer
teardown, the stalwart relay rule, the onion apex redirect and image and
version updates; ruff 0.17 reformats nothing on this branch.
2026-10-15 21:27:11 +02:00
kevinveenbirkenbach 06a74e7658 Merge pull request #869 from infinito-nexus/fix/web-app-stalwart/relay-rule/793
fix(web-app-stalwart): authenticate bots with an app password under SSO
2026-10-15 15:42:19 +02:00
evangelostsak 01e8c1a75f test(lookup): mesh_address returns only the host's own primary address 2026-10-14 15:03:41 +02:00
kevinveenbirkenbachandClaude Opus 5.5 45732329b6 fix(lookup): keep the onion apex from redirecting to a clearnet homepage
On a Tor node current_play_redirect_domains schedules the bare node
onion as a redirect source targeting DOMAIN_HOMEPAGE. Without
web-app-dashboard, DOMAIN_HOMEPAGE falls back to SOFTWARE_DOMAIN
(infinito.nexus), so web-opt-rdr-domains rendered a 301 from
http://<node>.onion/ to http://infinito.nexus/: an onion visitor sent
to a clearnet host over plain HTTP.

The onion apex entry is now added only when DOMAIN_HOMEPAGE is an
onion host. Otherwise the onion apex gets no redirect and the default
server answers.

Closes #924

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 13:35:29 +02:00
evangelostsak 762513ec32 test(lookup): hold mesh_ranges to a rendered pool
The declared template must come back rendered, and one that stays a template
must fail rather than reach the export list.
2026-10-14 11:23:23 +02:00
evangelostsak 75fd3e9433 Merge remote-tracking branch 'upstream/main' into feature/svc-net-wireguard 2026-10-13 01:08:20 +02:00
kevinveenbirkenbachandClaude Opus 5.5 24d7872fde Merge remote-tracking branch 'origin/main' into HEAD
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-12 18:54:31 +02:00
kevinveenbirkenbachandClaude Opus 5.5 eda6249d7b fix(sys-svc-compose): probe provider networks by their exact name
The network probe used community.docker.docker_network_info, whose
get_network() scans GET /networks and returns the first entry whose Name
equals the requested name OR whose Id starts with it, in a list order that
changes between calls. In main run 38038541631 (Zammad#1, swarm) the
central redis overlay got the ID n8nwzb1q39m3u4k7awwwmt6yg, the probe for
the not yet created provider network "n8n" returned it as existing, the
create step was skipped, docker stack deploy attached hermes to the redis
overlay as "n8n", and the oauth2-proxy then reached the password-protected
central redis by its bare host name: "NOAUTH Authentication required".
The same scan can also return the ID-prefix twin of a network that does
exist, so comparing the returned name alone would turn that network into
"missing" and fail the create with "already exists".

Both probes now run `container network ls --filter name=<entity> --format
'{{json .}}'`, whose name filter never matches an ID. The new lookup
network_driver picks the entry named exactly after the entity and returns
its driver, '' when there is none; the drift checks, their messages and the
create steps all read that one entry. No docker module is left in the file,
so its two docker-module-uses-container-cli suppressions go too.

Verified: on the host daemon GET /networks/<id prefix> returns a network of
another name and docker network ls --filter name=<id prefix> returns
nothing. In ansible-playbook against real docker network ls output the
lookup returned the exact network's driver in 20 of 20 probes beside an
ID-prefix twin, an empty driver (create path) for a missing network and for
a name only contained in a sibling, and the drift path for a foreign
driver. Unit tests cover the same cases; with a substring match instead of
the exact name the two sibling cases fail.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-12 18:35:22 +02:00
evangelostsak 5a83785b53 Merge remote-tracking branch 'origin/main' into feature/svc-net-wireguard 2026-10-09 12:00:07 +02:00
kevinveenbirkenbach 799b835799 Merge remote-tracking branch 'origin/main' into fix/web-app-stalwart/relay-rule/793 2026-10-09 00:53:26 +02:00
kevinveenbirkenbachandClaude Opus 5.5 64677d4790 chore: regenerate role docs and apply autoformat after #881 and #873
`make quality-high` rewrote these files on every branch: the SuiteCRM
README and the root role table lacked the Redis service #881 added, the
session path test was not formatted, and an OpenTalk secret description
from #873 was not wrapped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 22:27:12 +02:00
kevinveenbirkenbach 3874f66ac3 Merge remote-tracking branch 'origin/main' into fix/web-app-stalwart/relay-rule/793 2026-10-08 19:38:56 +02:00
kevinveenbirkenbach 6e9b4e23dc Merge remote-tracking branch 'origin/main' into fix/web-app-stalwart/relay-rule/793 2026-10-08 19:27:27 +02:00
kevinveenbirkenbach badc2583eb Merge remote-tracking branch 'origin/main' into fix/web-app-suitecrm/redis-sessions/879 2026-10-08 19:25:24 +02:00
kevinveenbirkenbachandClaude Opus 5.5 3cefb01d71 fix(web-app-suitecrm): keep sessions in Redis so replicas share them
With three swarm replicas the PHP session files on the shared NFS volume
were read as empty under concurrent requests of one browser, so SuiteCRM
answered 403 on /api/graphql and dropped the session. Sessions now live in
Redis with session locking that waits like flock did; the phpredis version
is pinned in services.yml with its own update source, and the engine lookup
renders the save path.

Refs #879

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 18:31:19 +02:00
kevinveenbirkenbachandClaude Opus 5.5 08ef66d8de fix(web-app-stalwart): authenticate bots with an app password under SSO
Under SSO the role tried to let applications relay without SMTP AUTH by
widening the relay rule to the private networks. The rule compared the
peer address to the text of a CIDR, which never matches, so an
application could reach local mailboxes but got "550 5.1.2 Relay not
allowed" for every external recipient. Repairing the comparison would
have opened a relay, because published ports, Tor and the swarm mesh all
hand outside clients an address inside those networks.

Stalwart checks app passwords before it asks the external directory, so
a bot can authenticate while the directory is OIDC. With SSO enabled the
role now creates one app password per bot, acting for that account as
the recovery admin, and stores it in the token store the SMTP password
is already read from. A bot gets a new one when its entry is deleted in
Stalwart or its token is removed.

The relay rule is back to authenticated sessions only. The lookup loses
its SSO relay branch and the role its submission_via_relay flag, so an
SSO deployment submits on 465 with AUTH like every other one.

The role's CLI test now fails when an unauthenticated session is given
an external recipient, or when the bot credential is refused one.

Closes #793

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 08:57:22 +02:00
kevinveenbirkenbach b3563f4ea3 Merge remote-tracking branch 'origin/main' into fix/web-app-prometheus/blackbox-targets/792 2026-10-08 08:57:13 +02:00
kevinveenbirkenbachandClaude Opus 5 50b9f60c43 fix(prometheus): resolve the native-metrics scrape target per deployment mode
`native_metrics_target` was a Jinja filter, so it could not read
DEPLOYMENT_MODE: filters see no Ansible variables. It returned
`<services.<key>.name>:<port>` unconditionally, which is `matrix-synapse:9000`
for matrix. Compose serves that name; swarm does not, because `docker stack
deploy` ignores `container_name` and the alias never exists. This repository
says as much itself, through `lookup('compose_only', 'container_name', ...)`.
The target was therefore scraped and refused, and `playwright.spec.js:274`
reported `web-app-matrix: 1 matching target(s) reporting up != 1`, meaning one
matching target whose `up` value is not 1.

It becomes a lookup that delegates host resolution to `scrape_target`, which
was already mode-aware, so exactly one plugin knows the difference between the
two modes. The port has to be handed over explicitly: matrix exposes metrics on
9000 while `services.synapse.ports.internal.http` is 8008, so the existing
internal-port lookup cannot serve this case on its own.

`scrape_target` now refuses `port` together with an explicit `port_kind`.
Before, `port` won silently and the error text still named the
`ports.internal.<port_kind>` path that the call had never consulted.

`native_metrics_apps` drops an app that pins itself to compose through
`compose_mode_force` while the cluster runs swarm, because the scrape target
would then name the wrong object. Only the literal case: matrix declares that
override as a Jinja expression over its flavour, which this lookup cannot
resolve from the scraping role's variable scope, so variant 0 on swarm stays a
latent mismatch.

The two comment lines claiming `container_name:port` went with the change;
after it they state the opposite of what the code does.

Verified by mutation: with the swarm branch of `scrape_target` disabled, the
new test emits `matrix-synapse:9000`, the exact string CI reported.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-08 08:57:13 +02:00
evangelostsak e9929d07c3 Merge remote-tracking branch 'origin/main' into feature/svc-net-wireguard
One conflict, in the generated role overview table of README.md.

Both sides had edited the same block. Upstream dropped Redis from the pihole
row; this branch still listed it. Upstream also listed svc-net-wireguard-core,
-plain and -firewalled, which this branch removed in e706170a0b as superseded
by the unified svc-net-wireguard role.

Resolved by taking upstream's block and then running `make readme-index`, so
the table is generated from the roles that actually exist rather than
hand-merged: Redis is gone from pihole, the three legacy WireGuard roles stay
gone, and svc-net-wireguard is listed once. `make readme-index check=true`
reports the table up to date.

Git kept the role deletions silently and correctly - upstream has not touched
those three directories since the merge base, so there was no modify/delete
conflict to resolve.
2026-10-07 10:04:31 +02:00
kevinveenbirkenbach d287e55964 Merge remote-tracking branch 'origin/main' into fix/code-scanning-alerts/807 2026-10-07 01:51:19 +02:00
kevinveenbirkenbachandClaude Opus 5.5 45d956e5ea fix(sidecars): cap an engine's sidecar at its provider's size
A Redis sidecar whose consumer declares no services.redis.mem_limit
runs in a container that inherits the consumer's primary entity limit.
Since the lookup replaced the old filter, maxmemory followed that limit
and rose from 204 MiB to between 819 and 3276 MiB in 17 roles.

lookup('service_memory_mb', ..., provider=<role>) caps the result at
the same share of the provider's own limit when the application does
not declare one for that service. A declared limit is not capped, and a
container smaller than the provider keeps its own share. The Redis
sidecar passes provider='svc-db-redis': those 17 roles get 409 MiB, the
seven with a 256 MiB container stay at 204.

The lookup now rejects unknown options and a provider that is no role
or declares no limit for the service, so a misspelled option fails
instead of silently lifting the cap.

The memcached sidecar's literal -m 64 takes the same lookup and keeps
its value.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 01:30:01 +02:00
kevinveenbirkenbachandClaude Opus 5.5 e203d6f7ab feat(plugins): size memory parameters from the container limit with one lookup
lookup('service_memory_mb', application_id, service_name, share) returns
a share of a service's mem_limit in whole MiB. It resolves the limit the
way sys-svc-container resolves the container's own: the named service,
then the role's primary entity, then RESOURCE_MEM_LIMIT.

It replaces the three filters of memory_filters, each with its own unit
parser and its own idea of the limit: Redis maxmemory at 0.8 of the
limit, and the Confluence and Jira heap at 0.7 and 0.35.

The central Redis (409 MiB), Confluence (5734/2867) and Jira (4300/2150)
keep their values. A Redis sidecar without a mem_limit of its own does
not: the filter fell back to a fixed 256m, which gave 204 MiB, while the
lookup takes the limit that container really runs with, the primary
entity's. That moves 25 roles to between 409 and 3276 MiB.

The heap clamps of the old filters are gone: the 1024 MiB floor, limit
minus 1024, the 12288 cap and Xms bounded by mem_reservation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 23:05:26 +02:00
kevinveenbirkenbachandClaude Opus 5.5 3610e7b9ef fix(code-scanning): resolve the alerts that describe real code
Of the 76 open code scanning alerts, 39 point at code that can change.

svc-ai-mcp-adapter:
- A path argument that is empty or made only of dots is refused.
  Percent-encoding keeps a separator inside its segment, but "." and ".."
  need none, so an upstream that resolves dot segments could answer from a
  path the contract does not name.
- parse_duration splits on the units. The pattern it used before rescanned
  a long run of digits from every start position.

Tests and the README generators:
- Every Jinja environment built with autoescape=False and a "noqa: S701"
  uses select_autoescape now, as six test modules and
  utils/templating/jinja.py already did. It escapes templates named
  *.html, *.htm and *.xml, which none of these call sites loads.
- Dead code goes: a doubled "rel" block in the email lint, an unread
  _SKIP_DIRS, and a module that _resolve returned to no caller.
- Two "except ...: pass" blocks become contextlib.suppress and
  assertRaises.
- Nine test modules import unittest.mock one way instead of two, and one
  lookup test imports its module once.

Closes #807

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 09:18:00 +02:00
evangelostsak bb484ecb8f Merge remote-tracking branch 'upstream/main' into feature/svc-net-wireguard
One conflict, in the generated role table of README.md. Both sides changed
the same rows: upstream swapped WordPress's Mailu dependency for Stalwart,
while this branch replaced the three svc-net-wireguard-{core,plain,
firewalled} rows with the single svc-net-wireguard it deploys.

Resolved by regenerating rather than by hand, since make readme-index owns
that table and gates it with --check. The regenerated table carries both
sides and the check reports it up to date.

README.md was the only file both sides touched, and nothing in the tree
still names the three replaced roles.
2026-10-04 02:18:03 +02:00
kevinveenbirkenbachandClaude Opus 5.5 52fe3c9c2c style: apply make autoformat to the merged Stalwart and Tor files
Formatter output only: the parsed YAML and the Python AST of every touched file equal their previous versions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 19:57:30 +02:00
evangelostsak e456b1eee6 Merge remote-tracking branch 'origin/main' into feature/svc-net-wireguard
One conflict, in the MCP exclusion list of requirement 035. Both sides
edited the same line: main added svc-net-tor-smtp, this branch replaced the
three svc-net-wireguard-{core,plain,firewalled} entries with the single
svc-net-wireguard it deploys. Resolved to the union of both — 24 entries —
which is what the merged tree actually holds.

Everything else merged clean and was checked for overlap rather than
assumed: no reference anywhere still names the three replaced roles;
matrix.py kept main's PRIMARY_APPS alongside the reordered round, and the
bootstrap pass does not need it, since only one template reads it and
group_vars defaults it empty; and main's ansible.cfg addition sits in
[defaults], so the ssh_args the mesh tool derives its own connection
settings from are untouched.
2026-10-03 02:29:45 +02:00
kevinveenbirkenbach d90f1e8ed6 Merge pull request #756 from kevinveenbirkenbach/fix/svc-db-postgres/pgdg-bullseye-archived
fix(svc-db-postgres): build on postgres 18 since PGDG archived bullseye
2026-10-02 17:41:28 +02:00
kevinveenbirkenbach 30417d2515 fix(svc-db-postgres): build on the official multi-arch postgres 18 image
postgis/postgis publishes amd64 only, so every postgres consumer failed
on arm64. The official postgres image serves both architectures and
already ships the contrib extensions bloom, pg_trgm and unaccent.

PostGIS moves into the postgres_libraries registry and is installed from
PGDG (postgresql-18-postgis-3) only for a consumer that declares it as
an extension, which is mobilizon alone. apt update reads every source
list again, because the PostGIS package depends on GDAL from Debian
main, which a PGDG-only update left without an install candidate.

Postgres 18 keeps its cluster in /var/lib/postgresql/18/docker and its
entrypoint treats a volume at the former /var/lib/postgresql/data as an
old database, so the central and the dedicated data volume both mount at
/var/lib/postgresql. Existing 17 clusters are not migrated: their data
directory trips the entrypoint's old-data check and the container
refuses to start.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f78ebe46a4)
2026-10-01 22:19:09 +02:00
Alejandro Roman b160d07008 fix(lint): annotate ClassVar in test_email and reorder pixelfed playwright hooks 2026-09-26 19:27:31 +04:00
evangelostsak 78409cb642 test(packages): pin the per-manager install arguments
Covers the three cases that matter: apt drops recommends, another manager keeps
its defaults, and a concrete module is left untouched.
2026-09-26 16:19:57 +02:00
Alejandro Roman fadf36e0a6 fix(swarm-stalwart): fix CI issues 2026-09-26 18:05:56 +04:00
Alejandro Roman 866775924e fix(swarm-stalwart): Tor port conflicts and NFS localhost vs host issues 2026-09-23 13:30:07 +04:00
Alejandro Roman c8894b2dd2 Merge remote-tracking branch 'upstream/main' into feature/role-stalwart-mail-swarm 2026-09-22 22:33:13 +04:00
evangelostsak 603233f894 test(prometheus): pin that an unscraped role contributes no alert rules
Stubs native_metrics_apps and asserts the intersection, including the case the
refactor exists for: a role that ships a fragment and is deployed but is not
scraped must not reach the rule file.
2026-09-21 17:45:26 +02:00
Alejandro Roman 4b9a834546 Merge remote-tracking branch 'upstream/main' into feature/role-stalwart-mail-swarm 2026-09-21 03:25:20 +04:00
evangelostsak c762924675 Merge upstream/main into feature/web-app-openbao
Brings in Kevin's AI-agent work, the svc-net-firewall role and the update
tooling. One conflict, in the MCP proxy audit: both sides appended a role to
privileged_control_plane, ours web-app-openbao and upstream svc-net-firewall.
Taken as a union at 25, which is what test_mcp_audit_completeness requires -
neither role declares meta/mcp.yml, so the audit is the only place they are
classified.
2026-09-19 20:07:00 +02:00
kevinveenbirkenbachandClaude Opus 5 c693f4f619 style(quality): apply what the gate wrote back into the tree
Nothing here was typed by hand. The docs generator rebuilt the MCP audit
table against pins that had already landed, so fider reads v0.36.0 instead
of stable, jenkins 2.568.3-lts instead of lts, pretix 2026.7.0 instead of
stable, and flowise 24-slim instead of 24-bookworm-slim. That is the table
catching up with da665ab69 and 10c5b35e8, reading the same declaration the
updater reads.

Autoformat wrapped three test files from the same series: one condition
pulled onto a single line, three assertEqual calls broken across lines.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 19:35:05 +02:00
kevinveenbirkenbachandClaude Opus 5 ada2e4d311 fix(mirror): keep the origin prefix when an upstream redirects
The mirror proxies /<origin>/<path> to https://<origin>/<path> and relayed the upstream Location header untouched. A root-relative Location then resolved against the mirror instead of the origin, the prefix that names the origin was lost, the follow-up request matched no location block and fell into the catch-all 404. Chromium discards a 404 served for a script request, so the page lost the asset with net::ERR_BLOCKED_BY_ORB.

MIG job 105068912392 shows both halves, 28 times each: GET /unpkg.com/3d-force-graph/dist/3d-force-graph.min.js answered 302, then GET /3d-force-graph@1.80.0/dist/3d-force-graph.min.js answered 404. unpkg redirects an unversioned path to the resolved version and expresses the target as an absolute path.

The forms move into a redirect_sources filter so the template states the intent once and the list is unit-tested. Measured against a two-container nginx fixture: an absolute https Location, an absolute http one and a root-relative one all come back as /<origin>/target. A Location naming a third host stays untouched, which is unchanged behaviour and still leaks past the mirror.

A network-path form was written first and then dropped: nginx normalises such a target in return, and routing one through add_header emits two Location headers of which the proxy relays only the first, so the case could not be produced and the rule stayed unproven.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 12:21:41 +02:00
kevinveenbirkenbachandClaude Opus 5 b036f60039 refactor(container_ports): express the remaining published ports through the lookup
The list form could publish a host:container pair and nothing else, so thirteen blocks stayed hand-written: the ones needing a protocol suffix, a port range, the swarm long syntax with mode host, or a container-port-only publish. Each of those duplicated numbers the role already declares in meta/services.yml, which is the drift the lookup exists to remove.

The lookup keeps the list form unchanged for its 93 existing call sites and gains a dict form: transport emits one item per protocol, container 'same' serves the services that cannot remap (STUN/TURN, media relays), mode 'host' emits the long block an edge role needs in swarm so the client address survives the hop, publish 'ephemeral' emits the container port alone, and key names the block for Discourse's launcher file, whose expose: list is what becomes docker run -p there. A declared {start, end} mapping renders as start-end on both sides, so a relay range needs no term of its own.

svc-prx-openresty had no ports block at all and bound 80 and 443 as literals in its template, while torrc.j2 carried a third literal 80 for the hidden service. None of the three referenced the others. openresty now declares the ports, the compose template reads them, and the onion's port 80 comes from ports.onion.http through TOR_ONION_EXTRA_PORTS.

Three blocks stay raw, each with the reason it cannot route through one call: keycloak's sso_proxy maps a published port owned by the consuming role to a container port owned by web-app-keycloak, and matrix's ansible flavor takes element on 8080 and federation on 8048 from the upstream playbook rather than from this repo's ports.internal.

The enforcing lint scanned only files with compose in the name and only double-quoted list items, so nine of the thirteen blocks were invisible to it. It now reads every role template that can carry a ports block, catches unquoted items, protocol suffixes, ranges, flow style and the long syntax, and tracks the block by the indentation of its key: keying on the next line closed the block on anything shaped like a key, which target, published, protocol and mode all are. Under expose: only a real mapping counts, because there the single-segment form documents a container port and publishes nothing.

Verified by rendering all 110 container_ports call sites through the real templar, the real lookup and the real merged applications view.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-15 14:55:25 +02:00
kevinveenbirkenbachandClaude Opus 5 d7efffc31a fix(tor_ports): forward only the ports a role names under ports.onion
The public-port sweep never forwarded anything. It read tor.enabled from meta/services.yml on disk, where 104 of 105 roles declare it as the Jinja string "{{ 'svc-net-tor' in group_names }}", which _as_bool answers False for. Running HEAD's collect_public_ports the way HEAD's run() calls it, against the real roles directory with every role deployed, returns []. Only svc-net-tor reads True on disk, and it declares no public ports. The node onion therefore carried port 80 and port 22 and nothing else.

Reviving the sweep by reading the merged view instead would have been worse than leaving it dead: it publishes 16 ports across 7 roles, and declared is not published. openldap's ldaps 636 sits next to network.public false, Mailu's implicit-TLS ports are dropped by container_ports whenever TLS is off (which an onion deployment always is), and svc-net-tor's own SOCKS port would become an open proxy. So the source and the selection rule change together: read the variant-merged applications view, and forward only the categories a service names under ports.onion, resolving the number from ports.local when declared there and ports.public otherwise, which is the precedence container_ports publishes by.

collect_public_ports had no consumer outside this module and its own test. UDP_ONLY_CATEGORIES becomes public so the accompanying lint imports the list rather than restating it, and gains stun_turn and stun_turn_tls: those speak TCP, but the allocation TURN hands back is a UDP relay address an onion client can never receive on.

No role declares ports.onion yet, so the rendered HiddenServicePort set is unchanged by this commit alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-15 13:39:35 +02:00
kevinveenbirkenbachandClaude Opus 5 9a68f0eaaf fix(sso): read the OIDC gate's flags through the rendering config lookup
meta/addons/oidc_login.yml gates on lookup('sso_oidc_plugin'), which the applications render evaluates while it is still in flight. Under that re-entry guard the merged view hands out service flags as their own Jinja text, and bool("{{ ... }}") is True. The lookup read services.sso.enabled and services.ldap.enabled as enabled on a deployment with neither Keycloak nor OpenLDAP and announced oidc_login, so Nextcloud advertised a login hand-off to a provider that is not there.

lookup('config', ...) renders the value it returns, which is why the roughly ninety '| bool' sites in meta/ do not have this problem. 238bc285c7 established reaching the merged config through the lookup loader; this one had not been converted. utils.roles.applications.config.get is the raw accessor and does not render.

Measured against the real render: with only web-app-nextcloud deployed the addon set is now empty; adding Keycloak yields sociallogin; adding OpenLDAP as well yields oidc_login. That is the resolution order the lookup's own docstring documents. All 14 nextcloud variants pin both flags as literal booleans, so no CI-deployed variant changes.

The stub next to the plugin answered every lookup with the applications payload; it now mirrors lookup('config', app, path, default). A stub resolves a path but does not render it, so it cannot see this defect at all: the regression test renders for real.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-15 11:52:15 +02:00
kevinveenbirkenbachandClaude Opus 5 182ad13717 fix(api): let the SPOT merge survive a templar without the kwarg
15a6251d7 introduced the merge but tripped over three of the repository's own rules and over the lookup's existing contract test.

_templated called templar.template(value, fail_on_undefined=False). A templar whose signature is template(self, value) raises TypeError on that kwarg, the except swallowed it, and the value came back unrendered: the lookup returned ['{{ vault_cf }}'] where ['secret-cf'] was due. It now falls back to the positional call, the way utils/cache/base.py has always done it.

The SPOT read goes through load_yaml_any instead of yaml.safe_load, which tests/lint/repository/yaml/test_no_direct_calls.py requires so the file is parsed once per process. The module docstring drops the inventories/development/default.yml literal, which tests/unit/python/cli/administration/deploy/development/test_inventory_vars_file_spot.py whitelists by path.

test_missing_api_global_raises_unknown_key asserted exactly the behaviour that broke CI: a scope without API had to raise. It now asserts the fallback, and two new cases pin the edges. A provider the scope omits still resolves; a provider the declaration does not know still raises.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-15 03:50:02 +02:00
kevinveenbirkenbachandClaude Opus 5 ce8368019d perf(constructor): carry the applications SPOT through the merge block
113113e553 parks the carrier after the token store, 165 lines below the merge
block, so each of the block's six tasks still renders the applications SPOT
cold in its own forked worker. Measured in CI run wp-swarm-tor2 they cost
323.51s in the PASS 1 main play, 180.39s in the backup play and 479.79s in
PASS 2.

The cost is the render, not the work. Two tasks in that block share the exact
construct `defaults_X | combine(X | default({}, true), recursive=True)`: design
takes 0.247s and service_provider 75.94s. The difference is that
defaults_service_provider is dense with lookup('config'/'users'/'domains'/'tls')
and every one of them lands in get_merged_applications.

The carrier cannot simply move up. get_merged_applications serves the carrier
fact whenever the key matches, and the carrier action goes through that same
path, so a second carrier task would be handed the first one's payload instead
of rendering. merged_applications_cache_key reads neither group_names nor the
token file, and with a real templar 71 of 265 applications differ by
group_names alone, so a carrier left standing across add_host or the token
store would be served where a fresh render belongs.

So the block opens its own carrier and drops it again at the end. Nothing
outside the block changes: every task from sys-version to the stage-wide
carrier renders exactly as before.

Placement follows measurement. A per-key perturbation test over the whole
265-application payload, with the credential regeneration noise floor
subtracted and every role in group_names, finds exactly two reads of the seven
facts the block sets:

    .web-app-matomo.services.matomo.excluded_ips[0]   <- networks
    .web-app-wordpress.services.wordpress.title       <- service_provider

networks is the block's first task and service_provider its last, which is why
the carrier opens after the one and is dropped before the other takes effect.
The block's tasks already render before service_provider is merged today, so
the frozen payload is what they would have rendered anyway.

The action gains a `clear` argument so the playbook never spells the private
fact name. Parking the carrier serves a payload in 0.000s against a 3.417s cold
render; a wrong key and a cleared fact both force the render again, which is
what makes the drop task work.

Worth roughly 806s of the 20199s job, against a floor of 673.6s.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 18:10:40 +02:00
evangelostsak 912b6cda39 Merge infinito-nexus/core#454 (feature/ai-agent-employees) into feature/web-app-openbao
Only two files were touched by both sides. env.js merged automatically: the
incoming side owns recordVideoOptions and this branch had cherry-picked that
same function from it, so the result carries one definition. README.md is the
generated roles index and both sides append rows; resolved as a union, taking
the incoming side for the three rows it edits (OnlyOffice, Open WebUI, OpenClaw)
and keeping ours only for OpenBao, then re-sorted.

The incoming side adds lint rules and resources that this role then violated.
Five fixes, each the remedy the failing rule names:

* services.openbao.ports.local.http 8081 -> 8093; svc-ai-litellm now holds 8081.
* meta/networks.yml subnet 192.168.221.0/24 -> 192.168.222.0/24; svc-ai-mcp-adapter
  now holds 221, and the suggester only sub-allocates inside occupied /24s, so
  the next free block was taken directly.
* services.email bond 1 -> 0; the entry is disabled and the resource model skips
  it before reading the weight.
* README Quick Setup regenerated from the template it had drifted from.
* Credits link the implementer, matching how they are credited elsewhere.
* docs/requirements/035-mcp-proxy-expansion.md disposes of web-app-openbao as
  privileged_control_plane, which is what a secrets manager is, and the list
  count follows.

Verified against a worktree of #454 alone so the baseline is measured rather
than assumed: lint 501 passed with only the branch-name rule failing, which
fails identically on that baseline because the worktree branch is named
merge/*; unit 5665 passed with the same three host-environment failures the
baseline has; integration 324 passed, 0 failed.
2026-09-14 16:59:39 +02:00
kevinveenbirkenbachandClaude Opus 5 6e98ecf66f fix(hlth-csp): give onion vhosts a navigation budget Tor can meet
csp-checker navigates every URL with a 20000 ms budget until
domcontentloaded. Over the Tor SOCKS proxy that expires before a page of
module scripts finishes: run 34614940508 lost Mastodon#1 and OpenProject#1
to it, 26 of 27 attempts logging "Unable to reach http://<onion>/
(Navigation timeout of 20000 ms exceeded)" while sys-service-terminator
retried the unit for 35 minutes. No policy was ever read; the checker never
got that far.

csp-checker 2.3.0 takes --timeout, so script.py takes --onion-timeout and
hands it to the onion batch alone, before the -- separator that would make
it a URL. Clearnet keeps the image default. The budget is
HEALTH_CSP_TIMEOUT_MS (20000, the image default) times the tor flavor's
multiplier, 100000 ms on the public network.

The unit's TimeoutStartSec had to follow. It was 30 + 25 s per domain --
130 s for the two onion vhosts Mastodon checks, while a 100 s budget needs
up to 200 s, so systemd would have killed the check mid-run. The filter now
takes onion_per_domain_seconds and counts .onion domains at that rate;
without the argument it computes exactly what it did before, so
sys-ctl-hlth-webserver is unchanged. The Mastodon case goes from 130 s to
450 s, a clearnet-only play stays at 130 s.

Tests pin that --timeout is appended only when set and stays ahead of the
separator, that main budgets the onion batch alone, that the role passes
both --onion-timeout and the onion rate for the start budget (both guards
fail against the previous tasks file), and the filter's split rate.

Not covered: the filter still caps at 3600 s, which binds beyond 34 onion
domains in one round.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-12 00:18:00 +02:00
kevinveenbirkenbachandClaude Opus 5 641f7a40dd fix(nextcloud): keep the sociallogin uid within 64 chars on onion deploys
Sociallogin creates a Nextcloud user as "<provider name>-<keycloak sub>",
and the provider name was the Keycloak domain. On an onion variant that
domain alone is 67 chars, so every OIDC callback died with
InvalidArgumentException "Username is too long" and HTTP 500 at
/apps/sociallogin/custom_oidc/auth.<onion> (run 34615516009,
Nextcloud#1, 204 occurrences). The whiteboard and admin OIDC specs failed
on the same login.

This is the second derived identifier an onion domain overflows; the
kernel hostname is the first, clamped to 63 bytes by container_hostname
since the pretix and bookwyrm fixes. The domain-or-entity rule now lives in
one lookup, bounded_name(application_id, max_length): the primary domain
when it fits, otherwise the entity name. container_hostname delegates to it
at 63 and keeps its eight callers and its behaviour.

The provider name is bounded_name('web-app-keycloak', 64 - 1 - 36), the uid
limit minus the separator and the UUID sub. A short clearnet domain stays
the name, so existing users keep their uids; an onion deploy gets
"keycloak", which, unlike a hash of the domain, survives a rotated onion
address.

Tests cover the lookup's fit, boundary and fallback cases and its argument
validation, container_hostname's delegation at 63, and the provider name's
room for the sub.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 21:14:55 +02:00
evangelostsak 699823f6cb test(prometheus): cover the alert_rule_roles lookup 2026-09-11 18:35:13 +02:00
kevinveenbirkenbachandClaude Opus 5 9d0f618e23 refactor(sso): compute the app logout target once in the sso lookup
The previous commit wrote the same proxy-aware logout expression into
four javascript.js.j2 templates. The rule it encodes - sign out of
oauth2-proxy first when the app is proxy-gated, else go straight to the
provider - belongs where proxy-gating is already decided.

utils/roles/applications/services/sso.py gains logout_url(url,
is_proxy_gated) next to get_sso_config, with OAUTH2_SIGN_OUT_PATH as the
path constant; the rd value is quoted exactly as Jinja's urlencode
filter does it. The sso lookup exposes it as the logout_url term: it
renders OIDC through the templar, takes is_proxy_gated from the same
resolution, and refuses a context without OIDC. Prometheus, Checkmk,
Baserow and Jitsi now render lookup('sso', application_id, 'logout_url').

Behaviour is unchanged: the resolver tests pin the rd encoding against
jinja2's do_urlencode, and the lookup tests cover a proxy-gated app, an
OIDC-native app, a disabled SSO and a missing OIDC.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 13:06:19 +02:00