categories.yml moves from roles/ to the repository-root meta/, joining packages.yml. utils/roles/categories.py owns its location plus the tree-walking helpers that were stranded in entity/name.py, so no caller spells the path or duplicates the lookup.
meta/distros.yml becomes the SPOT for every distribution: os_family, Galaxy platform name, dev-environment base image, and the pkgmgr/pkgmgr-virgin/environment reference templates. It replaces seven scattered copies (DISTRO_FAMILY, VALID_DISTROS, _DISTROS, _CANONICAL_PLATFORMS, two shell arrays, a Makefile literal). INFINITO_DISTROS is no longer a static default but generated by utils/env/handlers/infinito/distros.py; the shell consumers read it through the env layer.
Drops dead CLI surface found along the way: --categories-file and --roles-dir on the provision CLI plus three unused parameters of generate_dynamic_inventory, and the positional roles_file of the invokable categories command.
tests/lint/repository/test_distros_spot.py pins the copies that cannot call the SPOT (workflow literals, docs table) and forbids composing a pkgmgr reference outside it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drop a return-only temporary and state autoescape explicitly — the test
renders a dnsmasq config, where html escaping would corrupt the output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reaching an onion address required each role to point its own http client
at the tor socks proxy, which then carried clearnet traffic too — fatal
in the test network, which has no exit, and the reason nextcloud needed a
per-host exclusion list.
The egress router already redirects tor's virtual range to the TransPort
and dnsmasq already forwards .onion to tor's DNSPort; the piece missing
was reachability. dnsmasq listened on loopback only, which docker
discards, so containers never saw it. It now also answers on the docker
bridge, the daemon points there with the clearnet resolver kept as
fallback, and the role's own onion resolves to a container-reachable
backend instead of loopback.
Nextcloud drops its proxy block and the exclusion list; every other role
gets the same behaviour without configuring anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both iteration guides and the nfs_flavor lookup docstring still described
the flavor as RUNTIME conditional, selecting ganesha only for dev and act
and kernel everywhere else. Since the per-variant switching was dropped,
meta/services.yml pins ganesha for every runtime and kernel is a
per-inventory override.
State that, name meta/services.yml as the single point of truth, and
record which os families ship ganesha.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A role with a Model Context Protocol surface declares one mcp: entry in
meta/services.yml. The vocabulary module owns every allowed value, a hard
lint validates each declaration, and roles_with_service gains an opt-in
direction= kwarg so a client role resolves its servers at deploy time.
The lookup stays backwards compatible: without direction= the entries
keep their original four keys, so the existing dashboard, prometheus and
matomo callers are untouched. With it, each entry additionally carries
transport, auth, auth_subject and an endpoint dict whose port is
resolved from the referenced service's ports.local or ports.internal.
The lint rejects Jinja on MCP-specific fields (only enabled/shared may
carry it) because the static resolver cannot evaluate a templated block,
and it verifies that endpoint.service_key names a service in the same
file whose ports.* actually contain endpoint.port_key.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Five helpers rescanned the whole roles/ tree on every call while their result is a pure function of the (static) tree, so callers that invoke them per-role/per-app/per-service turned the scan into O(N^2) filesystem ops. compute_application_gid was the measured hotspot: 254 rescans in one applications render (~25s of a 41s cold render). Memoize each tree-derived result with functools.cache keyed by the resolved roles_dir (the repo's existing idiom), returning tuples so the shared cache cannot be mutated; dependency_resolver uses an instance cache instead since functools.cache on a method would pin instances via self. Cold applications/domains render drops 41s -> ~8.5s (5.6x); behaviour is unchanged (GID/registry/placement/path values identical).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The asset lookup hardcoded https for the internal CDN origin, but
.onion vhosts are plaintext with no 443 listener, and the dashboard
mirror-rewrite sent even onion-local asset URLs through the mirror
(404). The adapter never loaded, so the OIDC code was never exchanged
and Login stayed visible after the round-trip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ruff toolchain was unpinned, so host venvs, the lint container and CI
installed different releases over time. That drift is why lint-python only
failed on CI: locally ruff 0.15.x preview-gated ISC004/FURB192 and (falsely)
fired S310 on urllib.request.Request, while CI's 0.16.0 has ISC004/FURB192
stable and drops the S310-on-Request false positive.
Pin ruff==0.16.0 in pyproject.toml's dev extras (Dependabot keeps it current)
and have utils.install.lint.ruff read that pin and reinstall whenever the
found version drifts, so every environment resolves the same ruff. Then fix
what 0.16.0 flags: parenthesise the implicit string concatenations (ISC004),
use min() over sorted()[0] (FURB192), drop the now-unused S310 noqa on the
Request() builders (RUF100), and delete the narration comments that the
comment lint surfaces in the touched files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_task_file_numbering.py: numbered task
files use exactly two leading digits, a numbered *_core.yml must be
00_core.yml, per directory the numbers are unique/consecutive and start
at 01 (00 with a core file), numbered includes appear in ascending
order, and shared helpers live in tasks/utils/ instead of the numbered
chain.
Apply it across every role: rename NN_core.yml to 00_core.yml with a
consecutive down-shift of the siblings, move letter-suffix children into
a subdirectory named after their parent task, relocate helpers included
from more than one place into tasks/utils/, and update every reference
(include_tasks, path_absolute, .ansible-lint-ignore, requirement docs).
Legacy narration comments in the touched files are deleted; the upstream
xwiki.properties reference block opts out via nocheck. The unused
MAILU_IP6_PUBLIC var (only ever named in now-removed comments) is dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add lookup('asset', app, pkg, path) and lookup('asset_host'): both resolve
the asset origin through one shared resolve_host, returning the internal
CDN domain when web-svc-cdn is deployed with flavor internal and
cdn.jsdelivr.net otherwise, so template URLs and CSP origins can never
diverge. Versions come from the role's committed files/package-lock.json.
Roles declare frontend dependencies in files/package.json; web-svc-cdn
aggregates every role lockfile, runs one npm ci per role in a node
container and mirrors the packages under /npm/<pkg>@<version>/ following
jsdelivr's path scheme, so internal and external URLs differ only by host.
Convert web-app-dashboard (keycloak-js 24.0.5 UMD) and web-app-littlejs
(bootstrap, fontawesome) to the asset lookup, drop the vendored keycloak
blob, and add a repository lint forbidding hardcoded external CDN hosts
in templates and CSP metas; pre-existing external CDN entries are checked
out with nocheck markers and a migration TODO.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A force_bridge app (matrix's MDAD flavor) deploys as a node-local compose stack and publishes its local port on its host, while openresty runs as an overlay task whose 127.0.0.1 is its own loopback; the readiness probe died on 502. The proxy_pass lookup now resolves networks.local.force_bridge (trust-tagged for the 2.19 templar, hard-failing on an unrendered template instead of silently falling back to loopback) and keys the decision on the real DEPLOYMENT_MODE, since such apps rewrite the effective mode to compose via compose_mode_force. The upstream becomes host.docker.internal:<local_port> rendered as a literal directive, because the request-time resolver cannot see /etc/hosts entries, and the swarm openresty service maps host.docker.internal to the host gateway like prometheus already does. Non-force_bridge renders stay byte-identical.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A native-metrics app on a node-local force_bridge network cannot be joined or resolved by a swarm-wide prometheus: the stack deploy fails with the network scope error and the scrape job would flap as permanently down. native_metrics_apps now drops those apps at the source (DEPLOYMENT_MODE swarm + networks.local.force_bridge), covering all four call sites (network precreate, compose service networks, external-networks block, scrape include) without per-template guards. DEPLOYMENT_MODE is templated without a default so a missing value fails loudly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
roles/categories.yml now declares the ordered 'stages' and a per-category 'stage' (plus run_after and bootstrap). utils/roles/stage.py resolves a role to its stage and exposes stage_groups(); the new stage_groups lookup feeds the 01_constructor / 02_server / 02_workstation / 03_destructor group loops in place of their hardcoded lists. A static invariant test proves the lookup reproduces the exact previous loops, so the rendered plays (and deploy behavior) are unchanged.
cli.meta.roles.order.run orders its main phase by stage as a run_after tiebreak. plugins/filter/invokable_paths tolerates the new sibling 'stages' key, and the invokable-roles wiring test recognizes stage_groups-driven wiring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Flatten a single list term in the service lookup so query(service, [ids]) does one applications load instead of one per id. Precompute the required service ids in sys-service-loader and gate the preload by set membership, collapsing about 30 applications loads into one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`lookup('container_hostname', app)` returns the app's canonical primary
domain when it fits sethostname(2)'s 63-byte cap, else the short entity
name. Onion subdomains (sub.<56-char>.onion) overflow that limit and make
`docker run` fail with "sethostname: invalid argument", killing container
init on onion nodes. Replaces the seven roles that hostname'd their
container on the raw domain (bookwyrm, bridgy-fed, confluence, jira,
postmarks, pretix, xwiki) with the encapsulated lookup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
svc-runner's CLI test runs a full nested reinstall deploy and needs
75+ minutes, so the uniform 3600s timeout killed it deterministically
in both modes. A new role_tests lookup reads roles/<id>/meta/tests.yml
(cli.timeout, scaled through the timeout lookup) with the existing
default as fallback, keeping the short hang-leash for every other CLI
test. svc-runner gets 4h and pins the nested deploy to variant 0; the
second matrix round only redeployed cdn+dashboard and added no
assertion value.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The custom flag moves from 33 hardcoded template kwargs into the merged
service config: 'custom: true' names the shared <entity>_custom build,
a string names a second distinct '<string>_custom' image inside one app
(nextcloud whiteboard). The image lookup reads the declaration as a
fallback, so config consumers (e.g. backup image matching) resolve the
exact deployed reference from one declaration point. nextcloud cron now
runs the nextcloud_custom build like the app service instead of the
upstream image.
A new services lint couples the declaration to reality in both
directions: custom requires a role Dockerfile, and a role Dockerfile
requires a custom declaration. The eight legacy roles deploying local
builds under upstream tags or ad-hoc image= overrides are ratcheted via
'# nocheck: dockerfile-custom'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New 'image' lookup is the SPOT for resolving <registry>/<image>:<version>
(swarm registry prefix in swarm mode, bare in compose; custom= accepts a
string base for a second distinct custom image inside one app).
container_image becomes a thin wrapper that forwards terms+kwargs and adds
the compose image: "<ref>" line, so all existing template call sites keep
their output byte-identical. The lookup-usage integration test now counts
lookup_loader.get() delegation as production usage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lookup('canonical_url', app_id[, key][, consumer=app]) resolves the web base
URL for an app or one of its named canonicals (element/synapse/filer),
following the onion-injected merged-domain map: .onion targets resolve to
http (Tor provides transport security), everything else follows TLS_ENABLED.
The consumer= kwarg returns '' unless the consuming app has the target's
service binding enabled, replacing the recurring
"(lookup(...) if lookup('config', app, 'services.X.enabled') else '')"
template branching.
- utils/cache/domains.py: inject the node onion into dict-form
(named-canonical) domain declarations, not just list/str forms
- keycloak redirect_uris: onion redirect URIs are http
- 13 playwright.env.j2: manual scheme branching -> canonical_url(consumer=)
- lint: forbid manual '://' concatenation and to_onion_url in role templates
- unit tests: 10 cases (primary/named x clearnet/onion, consumer gating)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
storage.nfs shrinks to the deployment-specific server address: export_base moves into svc-storage-nfs-server's services.yml (the server dictates where exports root) and the mount protocol version into svc-storage-nfs-client's services.yml (the client owns how it mounts). Both are read through utils.storage.nfs loaders by the nfs lookups, the env handler and 00_topology, eliminating the unrendered-variables class for these values; the group_vars assert shrinks accordingly and the swarm test extras stop restating role defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lookup plugins receive ansible variables unrendered, so routing the state subdir through a group_vars jinja value handed state_path the literal '{{ lookup(...) }}' string and the controller mount asked the server for /srv/nfs/{{ lookup('nfs_state_subdir') }} (Connection refused, nextcloud swarm roundtrip). Both nfs lookups now import STATE_SUBDIR directly; the intermediate lookup, the group_vars indirection and the unit-test fixture var that masked the gap are gone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A wedged DinD dockerd hung a single diagnostics docker exec until the 360-minute job timeout cancelled the job together with its artifact uploads (job 86013153099 sat 5h in service logs). Route every diagnostics exec through a timeout wrapper, tail-limit the log dumps, bound showmount, cap the collect step at 30 minutes, and dump the pending services' log tails once on the final converge failure. The NFS state path moves into a single source of truth: STATE_SUBDIR in utils/storage/nfs.py feeds the nfs_state_subdir lookup (deploy side) and the derived INFINITO_SWARM_NFS_STATE_PATH env var (harness side); the diagnostics script requires both path vars instead of defaulting, and the swarm inventory drops its redundant export_base restatement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename the meta/services.yml key across all roles, the roles_with_default_placement lookup (now roles_with_placement), the get_role_default_placement / iter_roles_with_default_placement helpers, and every consumer (swarm closure, compose_volumes filter, nfs_prep, container deploy template, constructor stage, extend_inventory) plus their tests and docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
compose_networks/container_networks read DEPLOYMENT_MODE directly, so a node-local compose-in-swarm role (compose_mode_force=compose, e.g. the matrix ansible flavor) rendered a swarm overlay a standalone compose container cannot attach to ("network matrix not found"). Honor compose_mode_force like the other mode-aware lookups.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The freeswitch build fetched libopusenc from launchpad.net, whose
librarian backend rate-limits CI (429s across 11 attempts); three more
build-time fetches carried the same risk. The maintainer publishes an
alangecker/* image for every buildable service, so compose_mods drops
build: wherever the service carries an image and compose pulls instead;
a guard task fails loudly if a future upstream service still builds.
Pin ref to the v3.0.23 release branch (the packaging repo has no tags)
and clone it without submodules via the new compose_repository_recursive
toggle: submodules only feed build contexts and DEV_MODE mounts, and
generate-compose falls back to the committed repos/tags file.
Declare the pulled images in meta/services.yml (updater-managed SPOT,
mirror pre-pull) and the named data volumes in meta/volumes.yml; the
override renders both through the standard container_image /
container_volumes / compose_volumes lookups with the service_name
pattern. Compose merges volumes by target, so the override's named
volumes verifiably replace the clone-local bind mounts (docker compose
config proven). This also heals a split-brain mount: apps-akka,
periodic and prometheus-exporter previously kept bind mounts while
their peers used the named volumes.
Teach container_image and compose_volumes to honour compose_mode_force
(the SPOT the other mode-aware lookups already read): bbb deploys as a
node-local compose stack in swarm, so it must get hub image refs and
plain local volumes there, not registry-prefixed refs and NFS
driver_opts. The unpinned-engine-volume lint learns the same two safe
conditions. Read the embedded Collabora port from web-svc-collabora's
services SPOT instead of hardcoding 9980.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Workers failed 'docker build: no Dockerfile': the build context, compose file, .env and runner binary were staged on the stack host only while every node deploys its own node-local runner stack. Build and push the image once on the manager (RUNNER_IMAGE_REF via the swarm registry, 04_image.yml) and let workers pull it; render compose.yml (template with lstrip_blocks, matching the replaced stack_host_template action), .env and the extracted run.sh on every node. Add a node_local kwarg to the compose_networks/container_networks/compose_restart/compose_only lookups so this node-local compose stack renders a bridge default network (workers cannot create overlays), keeps restart: unless-stopped (ephemeral runners died after one job) and keeps container_name under swarm; default is unchanged for every other role. Treat build/install.stamp as stale when the venv is missing and exclude ./build from the nested-src copy, closing the nested-e2e ruamel failure class. Regression tests for the node_local render paths; the network include tests now accept lookup kwargs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the inline/trailing comments added across the Tor egress-router, onion-domain, mailu-notls and vault round-trip changes, per the repo no-comments convention. Comment-only: no code behaviour changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The email lookups tls default keyed on DOMAIN_PRIMARY, which can stay clearnet while Mailus canonical domain is turned into a .onion by the Tor onion injection. TLS therefore stayed on, but Mailu has no public certificate for the .onion (LE cannot sign it) and disabled TLS, so the SMTP handshake to the local front failed and the msmtp health check errored. Key the .onion detection on the resolved mail host instead (host is now resolved before tls in RESOLUTION_ORDER); an onion mail host relays in plaintext, which Tor already authenticates and encrypts end-to-end.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a `tor_ports` lookup that collects every single-int `ports.public` entry
from the deployed roles' meta/services.yml and returns HiddenServicePort
mappings (`127.0.0.1:<port>`). UDP-only categories (relay, media) are excluded
since Tor forwards TCP streams only. Wire it into TOR_ONION_EXTRA_PORTS
(19_tor.yml) alongside the SSH port so mail (25/143/465/587/993), gitea/gitlab
SSH, ldaps, matrix federation and STUN/TURN control ports are reachable at
<node>.onion. Register Mailu's mail endpoints under ports.public and add the
matching PORT_BANDS entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- svc-net-tor: expose SSH (22) + Mailu SMTP/submission/IMAP (25/465/587/143/993)
as HiddenServicePorts on the node onion (TOR_ONION_SSH_ENABLED, 19_tor.yml).
- email lookup: relay plaintext (tls off, port 25) for .onion mail domains —
no public TLS cert for a .onion; onion transport is already authenticated.
- CA root CN: drop the " Root CA" suffix when a 62-char v3 onion + suffix would
overflow the X.509 commonName 64-char cap (02_tls.yml).
- Playwright onion compat: HSTS is HTTPS-only (RFC 6797), absent on the http
onion vhost. New shared helper expectHstsWhenTls() gates the assertion on the
reached scheme; wired into web-svc-cdn spec.
- rerun-spec.sh: route the inner-loop browser through the Tor SOCKS proxy when
the staged env targets a .onion, mirroring the deploy-time runner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Turn a NAT/edge node into a full Tor v3 onion node: DOMAIN_PRIMARY
becomes the node's .onion, so the whole stack (web, SSO, LDAP, CA)
resolves onion domains and is reachable over Tor without a public IP,
port forwarding, VPS or DynDNS.
- utils/tor_onion.py: offline v3 key mint (address + HiddenServiceDir
key files), verified against a known key->address test vector.
- cli .../inventory/onion: `init-env` pre-mints the node onion, writes
INFINITO_DOMAIN(+_RE) into .env and authoritative key files into
.onion-identity/hs (idempotent). The onion is a deploy-time input;
no in-deploy random minting (that would not match the pre-set domain).
- roles/svc-net-tor: Tor sidecar (Debian-built, network_mode: host),
installs the minted keys, torrc with one node HiddenService + a SOCKS
proxy restricted to loopback/DiD ranges. files/test.sh probes every
served onion domain over the Tor SOCKS.
- tls: .onion -> TLS off (onion v3 crypto is the transport auth; LE
cannot sign .onion) + app-level flavor override.
- harness/DNS: coredns resolves the onion zone via INFINITO_DOMAIN;
skip the clearnet DNS sanity check for onion; raise nginx
server_names_hash_bucket_size to 128 for long onion server_names.
- test-e2e-playwright: gated browser SOCKS proxy (PLAYWRIGHT_PROXY) +
navigation/action timeouts so specs reach .onion over Tor and fail
fast; the onion node points it at the Tor SOCKS.
- docs/requirements/031: full design.
Verified: dashboard + SSO (Keycloak) + 9 apps reachable over Tor
(HTTP 200 through the node hidden service). make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The sys-service-loader iterates the role-wide static service order, so it
asks the service lookup about keys (e.g. the css alias of cdn) that the
play-local registry built from applications does not contain. The lookup
raised and killed the bootstrap play on every distro.
Fall back to the registry read from roles_dir, the same source the
loader iterates, so every loader-visible key resolves with its provider
role and play-computed flags (false when no consumer is deployed).
Canonical aliases aggregate their primary key's consumer flags. A term
unknown to both registries still raises.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
