Commit Graph
14682 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 4.8 703380fba7 refactor(web-app-flowise): consume the shared gateway instead of a private sidecar
Flowise shipped its own LiteLLM container: own image, own port, own
config template, own volume, own master key. Every further consumer of
the model plane would have needed the same again.

The service entry becomes a plain consumer flag on svc-ai-litellm, so the
include resolver pulls the gateway into the round and the flag is false
when no gateway is deployed. The sidecar render task, its template, its
volume and the compose_handlers_flush dance that existed only to sequence
that render are all gone; 00_core.yml is back to the standard shape.

Two corrections ride along in env.j2. The endpoint keeps its /v1 suffix,
which the OpenAI-compatible surface requires. And Flowise now presents
its own virtual key from credentials.litellm_api_key rather than the
gateway master key, which previously handed a consumer full
administrative access to the gateway.

The whole block is conditional now, so without a gateway Flowise renders
no OPENAI_API_BASE at all instead of an address pointing nowhere.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 19:03:30 +02:00
kevinveenbirkenbachandClaude Opus 4.8 cc4f76ad66 feat(web-app-litellm): expose the gateway admin UI as a stackless vhost
The LiteLLM gateway ships its own admin UI, but svc-ai-litellm stays
headless: the UI is optional and public, the gateway is neither. Putting
both in one role would either make the UI mandatory or teach the gateway
domain logic it has no business knowing.

So this role runs no container at all. It loads sys-stk-front-proxy and
points domain, http_port and proxy_internal_port at svc-ai-litellm's
service entry. The third variable is what makes swarm work: compose
reaches the gateway through its published host port, swarm renders the
vhost against the internal port and routes over the overlay.

networks.local.force_bridge is set because a role without a container has
no overlay for openresty to join; the upstream resolves to
host.docker.internal and the gateway's published port instead.

provides: litellm_ui renames the registry entry, because
get_entity_name(web-app-litellm) collides with the key svc-ai-litellm
already owns and a duplicate aborts every service resolution repo-wide.

The coupling is one-way: the UI role pulls the gateway, the gateway knows
nothing about the UI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 19:00:32 +02:00
kevinveenbirkenbachandClaude Opus 4.8 ad743bf447 feat(svc-ai): add the LiteLLM gateway and the LM Studio backend
Consumers of the model plane speak one OpenAI-compatible endpoint and
carry one virtual key; which backend answers is the gateway's business.
svc-ai-litellm renders its model_list from the round's actual topology,
so an alias only exists when its backend role is in group_names, and it
emits an explicit empty list when no backend is present at all because
LiteLLM rejects a bare model_list: key.

Keys reach the gateway through the per-consumer credential rather than
being minted there: 01_keys.yml probes /key/info and only POSTs
/key/generate on 404, so reruns stay idempotent. The consumer list is
built in 00_core.yml, not inside the included file, because 01_keys.yml
delegates to the hosting node and delegate_to is templated before when.

The gateway declares ollama and lmstudio as consumer flags so the
network renderer attaches it to its own backends; without them it cannot
reach them.

svc-ai-lmstudio runs the headless llmster daemon behind expose only, so
it is reachable from the gateway but never bound on the host, matching
svc-ai-ollama.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 18:50:49 +02:00
kevinveenbirkenbachandClaude Opus 4.8 c2ac6b5aeb feat(svc-virt-kata): provision a kernel-isolating container runtime
Agent workloads execute model-generated code, so they need a kernel
boundary on top of the app's own sandboxing: Kata plus Firecracker where
hardware virtualisation is available, the gVisor userspace kernel
otherwise, never plain runc.

runsc is fetched as the upstream static release binary with its published
sha512 and the host architecture in the URL, because Debian ships only a
2024 build and Arch and Fedora ship none at all; the same path serves the
arm64 Raspberry Pi target.

select_runtime.yml is a reusable task file: it picks kata only when
/dev/kvm, the kata shim and a bare host all hold. /dev/kvm is visible
inside a Docker-in-Docker daemon through host passthrough, but the microVM
still cannot start there (fuse-overlayfs, no containerd block devices), so
DinD is excluded explicitly and degrades to runsc.

daemon.json gains a runtimes block gated on group membership. Under DinD
runsc additionally needs --network=host: its netstack cannot pluck
interfaces inside a nested netns and aborts with 'cannot run with network
enabled in root network namespace'. The flag keeps the syscall sandbox and
reaches the container netns through real sockets.

The role bootstraps before the service-loader preload, alongside
svc-storage and svc-registry, so the runtime is registered and the daemon
restarted before the first workload compose file renders.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 18:40:50 +02:00
kevinveenbirkenbachandClaude Opus 4.8 ab2ecf5d6d feat(services): add the MCP service block and its client discovery
A role with a Model Context Protocol surface declares one mcp: entry in
meta/services.yml. The vocabulary module owns every allowed value, a hard
lint validates each declaration, and roles_with_service gains an opt-in
direction= kwarg so a client role resolves its servers at deploy time.

The lookup stays backwards compatible: without direction= the entries
keep their original four keys, so the existing dashboard, prometheus and
matomo callers are untouched. With it, each entry additionally carries
transport, auth, auth_subject and an endpoint dict whose port is
resolved from the referenced service's ports.local or ports.internal.

The lint rejects Jinja on MCP-specific fields (only enabled/shared may
carry it) because the static resolver cannot evaluate a templated block,
and it verifies that endpoint.service_key names a service in the same
file whose ports.* actually contain endpoint.port_key.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 18:37:40 +02:00
kevinveenbirkenbachandClaude Opus 4.8 5d59d72afe fix(deploy): apply the disable orphan-prune on the deploy side too
init.py prunes the include set twice: it drops the roles named by disable
and then re-walks the dependency graph so providers that only existed to
serve a disabled role are dropped as well. deploy/cli.py filtered only
the named roles, so its deploy-id set stayed a superset of what the
inventory actually contains.

With disable=matomo,dashboard,prometheus,email,css the init side pruned
svc-db-redis, svc-db-mariadb, web-svc-cdn, web-svc-file, web-svc-asset
and web-svc-simpleicons; the deploy side still passed them to
validate_application_ids, which aborted the run with 'not present in
inventory' before any role deployed.

Both sides now call prune_orphans_after_disable with the same
round-merged services map, so the sets match. Document the two-sided
contract in the README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 18:13:03 +02:00
kevinveenbirkenbachandClaude Opus 4.8 7400a652e3 fix(sys-ca-selfsigned): write the combined CA bundle to a writable temp path
The CA-injection entrypoint wrapper built the combined bundle inside
/tmp/infinito, which is the root-owned mount root docker creates for the
read-only cert and wrapper binds. Any image whose entrypoint runs as a
non-root user (node, www-data) hit 'cannot create
/tmp/infinito/ca-trust-combined.crt: Permission denied' and never
started; web-app-openclaw looped on restart. Images starting as root
(s6-based ones like hermes) masked the defect.

mktemp -t resolves TMPDIR itself and always yields a path writable by
the running user, so the wrapper no longer creates a directory it does
not own.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 18:05:36 +02:00
kevinveenbirkenbachandClaude Opus 4.8 d0721d2e36 feat(credentials): add sk_prefixed value-generator algorithm
LiteLLM rejects user-supplied virtual keys that lack the sk- prefix
(HTTP 400 on POST /key/generate), so consumers of the shared gateway
need a credential that already carries it. Mirrors the existing
base64_prefixed_32 precedent: sk- plus 48 secure alphanumeric chars,
51 total, clearing the min_length: 50 validation the consumer schemas
declare.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 18:03:44 +02:00
kevinveenbirkenbachandClaude Opus 5 386f27e901 fix(web-app-gitea): hoist delegate_to to apply:
Commit 99bc9a94d gave every top-level task in cleanup/ldap.yml and cleanup/oidc.yml an identical delegate_to, tripping the include-uniform-apply-keyword lint and failing CI run 30160077828.

apply: on include_tasks wraps the included tasks in a Block, and delegate_to is Block-valid, so hoisting it to the include site is semantically identical: register still resolves on the original host, the per-task when: gate is unaffected, and the include-site when: does not leak into the block. Leaf-only keywords (failed_when/changed_when) stay per-task per 64eef3440.

Matches the established pattern in keycloak, mediawiki, mastodon, matomo, espocrm, collabora and shopware. setup/ldap.yml and setup/oidc.yml deliberately keep the per-task form, since their undelegated set_fact must not run on the hosting node.

make quality green: test-external, test-integration, test-lint, test-unit all pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 16:17:31 +02:00
kevinveenbirkenbach 98b6b3db8e Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
# Conflicts:
#	plugins/lookup/application_gid.py
2026-07-25 15:44:55 +02:00
kevinveenbirkenbachandClaude Opus 4.8 99bc9a94d9 fix(web-app-gitea): delegate ldap/oidc auth exec to host node
In swarm the ldap/oidc setup+cleanup 'container exec' tasks used
GITEA_CONTAINER_ADDRESS (a swarm-wide-resolved container id) and ran on
the manager without delegation. When gitea is scheduled on a worker,
'docker exec' is node-local, so the manager's daemon can't see that
container -> 'No such container', failing the deploy.

Bring these tasks to parity with the DB/manager-ops tasks (commit
6a5cf7b66): use GITEA_LOCAL_CID + delegate_to GITEA_HOST_NODE (facts
already set by resolve_host_cid.yml in 01_manager_ops before setup/
cleanup run), with the container-exec-resolver nocheck marker. Applies
to every exec incl. the id-lookup steps, else idempotency/cleanup
misfire on worker placement.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 15:31:13 +02:00
kevinveenbirkenbachandClaude Opus 4.8 62e763e3de test(perf): tighten cold-render budget to 60s with a true-cold reset
The cold-render warmup budget was 120s; after the rescan memoization the cold render dropped ~5x, making 120s far too loose to catch a caching regression. Halve it to 60s (validated under make quality's xdist load, where the render lands ~25s). Wire application_gid's functools.cache into applications._reset() so every perf-test setUp measures a genuinely cold render -- without it a warm GID cache from a prior test would let a warm-up pass 60s that a true cold render on a slower runner would miss, and the cache-ratio assertions could go order-dependent.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 14:35:35 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e543ca605d perf(cache): memoize redundant roles-tree rescans via functools.cache
Five helpers rescanned the whole roles/ tree on every call while their result is a pure function of the (static) tree, so callers that invoke them per-role/per-app/per-service turned the scan into O(N^2) filesystem ops. compute_application_gid was the measured hotspot: 254 rescans in one applications render (~25s of a 41s cold render). Memoize each tree-derived result with functools.cache keyed by the resolved roles_dir (the repo's existing idiom), returning tuples so the shared cache cannot be mutated; dependency_resolver uses an instance cache instead since functools.cache on a method would pin instances via self. Cold applications/domains render drops 41s -> ~8.5s (5.6x); behaviour is unchanged (GID/registry/placement/path values identical).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 14:35:35 +02:00
kevinveenbirkenbachandClaude Opus 4.8 01b4eabb14 build(pre-commit): prune orphan __pycache__-only dirs on post-checkout
Switching branches leaves __pycache__ directories behind after their source files vanish; the now-orphan parent dirs (no source __init__.py, only ignored bytecode) trip the init-files lint. Wire the existing 'make clean-pycache-dirs' cleanup into a post-checkout hook and add the stage to default_install_hook_types so 'pre-commit install' picks it up. Non-privileged: it removes host-owned orphans and warns-and-continues on container-owned ones.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 12:44:05 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0299e90057 fix(web-svc-cdn): keep test.env keys in ascending order
The alphabetical-sort lint requires env keys ascending; the --no-verify commit that added CDN_TEST_STACK_DIR appended it after CDN_TEST_WEB_ROOT (STACK < WEB). Reorder it before WEB_ROOT. The keys are independent assignments, so sourcing order is unchanged at runtime.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 12:44:04 +02:00
kevinveenbirkenbachandClaude Opus 4.8 911152d3b6 fix(web-app-gitea): justify replicas:1 with single-replica nocheck marker
The single-replica lint requires every 'replicas: 1' pin to carry a '# nocheck: single-replica <reason>' marker, since pinning to one replica forgoes horizontal scaling. The --no-verify commit that added the pin skipped that hook; add the marker with the reason (single-writer state on swarm-shared NFS corrupts under concurrent multi-node writers). No runtime change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 12:44:04 +02:00
kevinveenbirkenbachandClaude Opus 4.8 68c0ffa84e feat(ci-trigger): add --strict to retrigger only hard failures, plus i8cifo/i8pufo
The --failed selection treats anything not green (cancelled, timed-out, still-running) as a failure, so retriggering a partially-running or aborted run re-fires jobs that never actually failed. Add --failed --strict to select only roles with a hard failure conclusion, excluding cancelled/aborted and running. New aliases i8cifo (strict base, mirrors i8cifail) and i8pufo (sign-push then strict retrigger, mirrors i8pufail); i8pufail is unchanged. Verified against a live run: --failed matched 48 roles (8 failed + 40 still-running), --strict matched exactly the 8 hard failures.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 12:44:03 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6a5cf7b663 fix(web-app-gitea): pin single writer and delegate DB patches to the hosting node
In swarm the gitea service defaulted to replicas = group size (3), so three replicas on manager + both workers bind-mounted the same NFS-shared /data and concurrently rewrote app.ini with no cross-node locking, leaving it NUL-filled and failing migrate. Gitea is a single-writer app (external MySQL, repos + app.ini on disk), so pin it to one replica; it stays on the shared NFS volume and remains node-mobile for failover.

The DB-patch sed tasks ran manager-local via GITEA_CONTAINER_ADDRESS, which cannot reach a replica scheduled on a worker. Resolve the hosting node + container id first (the same resolve_host_cid idiom 01_manager_ops.yml already uses) and delegate each patch to GITEA_HOST_NODE, so the patches follow the single replica wherever swarm places it. Compose is unaffected: replicas is swarm-only and the resolver short-circuits to localhost.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 12:44:03 +02:00
kevinveenbirkenbachandClaude Opus 4.8 f373fa0ec8 fix(svc-storage-nfs-server): disable NLM/RQUOTA with valid ganesha option names
The ganesha template used two non-existent option names (NLM_Param, Enable_RPCSTATD) that ganesha ignores as 'Unknown parameter', so RQUOTA stayed enabled and bound port 875. When rpc.statd took 875 first (PID-deterministic) ganesha crash-looped with a FATAL bind error (exit 2), so NFS 2049 never served and controller mounts failed with 'Connection refused'. Use the real NFS_CORE_PARAM booleans Enable_NLM/Enable_RQUOTA (both default true) to actually turn the sidebands off; the stack is NFSv4-only so neither is used. Verified on nfs-ganesha 4.3-8ubuntu1: corrected config starts, binds only 2049, serves a v4 mount, and stays up under the exact 875-occupied condition that crashes the old config.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 12:44:03 +02:00
kevinveenbirkenbachandClaude Opus 4.8 097794302c fix(web-svc-cdn): assert external flavor by absent compose stack, not leftover npm
The external-flavor CLI test checked for an npm/ dir under the served web root, but that bind source survives the inter-round purge, so a prior internal-flavor round left it behind and every later external round failed a false positive. Check the CDN compose instance dir (directories.instance) instead: it is purged between rounds and never recreated by the front-proxy-only external deploy, so it is present iff the build stack is actually deployed -- the real invariant the test means to assert.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 12:44:02 +02:00
kevinveenbirkenbachandClaude Opus 4.8 4aede4deb9 ci(build): pre-pull syntax frontend with retry
Transient Docker Hub timeouts resolving docker/dockerfile:1 aborted two deploy jobs at '#2 resolve image config'. Pre-pull the frontend from the Dockerfile's syntax line with 3 bounded retries; fail soft so the build still resolves live if Docker Hub stays down. Real build errors remain unretried.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 12:44:02 +02:00
kevinveenbirkenbachandClaude Opus 4.8 52ac2e3a49 fix(web-app-postmarks): declare built image custom
The built image was tagged with its own FROM base ref (node:26-bookworm), so the CA-trust override — running before the build — inspected the stock base and baked its bare 'node' CMD over the image's 'node server.js': exit-0 restart loop, openresty 502, OIDC specs red. custom:true names the build postmarks_custom like every other build-based role, removing the collision.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 12:44:01 +02:00
kevinveenbirkenbachandClaude Fable 5 62cf9bd358 style(runner): mark rollback or-true with nocheck reasons
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 09:04:21 +02:00
kevinveenbirkenbachandClaude Fable 5 f334cccf18 fix(web-app-dashboard): load keycloak-js from onion-local CDN
The asset lookup hardcoded https for the internal CDN origin, but
.onion vhosts are plaintext with no 443 listener, and the dashboard
mirror-rewrite sent even onion-local asset URLs through the mirror
(404). The adapter never loaded, so the OIDC code was never exchanged
and Login stayed visible after the round-trip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 09:03:21 +02:00
kevinveenbirkenbachandClaude Fable 5 2475bd4d57 fix(e2e): always source reports base dir from .env
INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR is exported nowhere inside the
container, so make compose-playwright aborted at its env guard; .env
stays the single source of truth, no env-override caching.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 09:02:16 +02:00
kevinveenbirkenbachandClaude Fable 5 185e899ba3 fix(e2e): wait out async OIDC exchange in biber check
The login-still-visible probe sampled 2s right after the redirect,
before keycloak-js finished the code→token exchange over Tor; wait
Tor-scaled for the login control to disappear instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 02:07:04 +02:00
kevinveenbirkenbachandClaude Fable 5 58ea059757 docs(requirements): add 034 embodied robot platform
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 01:51:38 +02:00
kevinveenbirkenbachandClaude Fable 5 8343e154fb docs(requirements): add 033 home assistant MCP server role
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 01:51:27 +02:00
kevinveenbirkenbachandClaude Fable 5 f9b0ee7ca4 docs(requirements): add 032 firecracker agent employees
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 01:51:13 +02:00
kevinveenbirkenbachandClaude Fable 5 1fd80120e5 docs(requirements): add 031 LLM gateway and model backends
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 01:51:03 +02:00
kevinveenbirkenbachandClaude Fable 5 7d1d1bde79 docs(requirements): add MCP how-it-works diagram to 025
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 01:50:52 +02:00
kevinveenbirkenbach 0107915009 Merge remote-tracking branch 'fork/feature/swarm-nfs' into feature/svc-net-tor 2026-07-24 20:11:01 +02:00
kevinveenbirkenbachandClaude Opus 4.8 58a5f7e8ed refactor(web-app-matomo): move resolve helpers into utils/resolve/ subfolder
A shared 'resolve_' prefix on two sibling task files trips the
prefix-grouped-files lint (a delimiter-bounded prefix is a folder in
disguise). Move both into utils/resolve/ and drop the prefix:
resolve_config_host.yml -> resolve/config_host.yml,
resolve_host_cid.yml -> resolve/host_cid.yml. Update the include paths in
02_manager_ops, 03_install_path, 05_patch_database, 06_patch_trusted_hosts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 20:05:59 +02:00
kevinveenbirkenbachandClaude Opus 4.8 51dbc46876 ci(runner): relocate Docker data-root onto /mnt when roomier
Swarm's nested DinD-node daemons plus the backup DR drill (LUKS loop image,
per-node registry pulls) fill the ~50 GB root fs and trip the <6 GB abort
guard in utils/tests/swarm/matrix.py, while /mnt is a separate ~74 GB
partition sitting idle behind only a swapfile.

Add a runner step that moves the host Docker data-root to /mnt/docker when
/mnt is a separate, roomier partition (detection mirrors enlarge_swap.sh),
before any Docker usage. Best-effort: any failure rolls Docker back onto /
and the step still exits 0, so it can never turn a green deploy red. Wired
into the compose, swarm, and host deploy workflows.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 20:05:59 +02:00
kevinveenbirkenbachandClaude Opus 4.8 2a7b0a7f3f fix(web-app-matomo): patch config.ini.php on host bind path, not container exec
Swarm can replace the matomo task mid-patch; a container id resolved earlier
then points at a dead container and the config patch fails with
"container ... is not running" (observed in the swarm CI matrix deploy).

Patch config.ini.php directly on the matomo_data volume (state NFS export is
no_root_squash on both kernel and ganesha flavors), so the DB-credential (05)
and trusted-hosts (06) writes no longer depend on container liveness. Resolve
the host bind path once via a shared utils/resolve_config_host.yml.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 20:05:59 +02:00
kevinveenbirkenbachandClaude Opus 4.8 f4fa8adf16 feat(deploy): prune orphaned transitive deps on disable
disable= now drops providers left with no surviving parent, not only
the named provider roles. Compose (init) and swarm (derive_includes)
share one reachability cut: BFS from the surviving primaries over
dependency + service edges with disabled roles as cut-nodes. No-disable
deploys stay byte-identical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 20:05:59 +02:00
kevinveenbirkenbachandClaude Opus 4.8 21fe9e9841 fix(web-svc-cdn): mark cdn workload node-local so swarm converge gate skips
The swarm post-deploy verify gate (03_wait_converge) derives a serving
service name cdn_cdn and polls it until timeout. web-svc-cdn has no such
service: it ships only the cdn_build one-shot and serves the built npm
mirror through the shared manager-pinned openresty host bind. The gate
therefore failed the whole matrix-deploy step even though the deploy
itself was green.

workload: node-local sets HAS_SWARM_SERVICE=false so skip_if_no_swarm_service
skips converge/chaos for this role, matching the already-declared
placement: manager host-bind design. Test-context only; deploy and compose
paths do not read workload.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 20:05:58 +02:00
kevinveenbirkenbach 8488e373b7 Merge remote-tracking branch 'fork/feature/swarm-nfs' into feature/svc-net-tor
# Conflicts:
#	roles/sys-ca-selfsigned/files/with-ca-trust.sh
2026-07-24 19:42:19 +02:00
kevinveenbirkenbachandClaude Fable 5 5399b48482 feat(svc-net-tor): add a chutney flavor for private-net test deployments
The test variant (meta/variants.yml, variant 0) switches the tor
service to the chutney flavor: the image ships chutney and the
entrypoint boots a private Tor network inside the container (all nodes
on 127.0.0.1, so the design is identical under compose host-networking
and swarm). The entrypoint bootstraps from /opt/chutney (chutney
creates its net/ state CWD-relative), extracts the DirAuthority/
TestingTorNetwork lines into /run/tor/chutney-net.conf and fails fast
when that file ends up empty, since the node would otherwise silently
join the public network. torrc %includes the conf only for the chutney
flavor.

Descriptor publish and rendezvous now run at localhost speed: per-app
onion e2e suites dropped from minutes to seconds in the compose
validation. The test variant also raises the footprint to
2g/1.0cpu/512pids (measured: at 1g/0.25 the ~9 extra tor nodes pin the
caps and desc publishing stalls); meta/services.yml stays static on
the public flavor and the old production footprint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 19:31:16 +02:00
kevinveenbirkenbachandClaude Fable 5 4afc9b7702 chore(lint): settle the lint fallout of the swarm-nfs merge
Mark the merged best-effort '|| true' sites (guide EXIT trap, baseline
fetch fallback) with shell-or-true nocheck reasons, parenthesize the
implicitly concatenated message strings ISC004 enforces since the ruff
0.16 pin, and drop the regex narration comments the comment lint flags
on the now-changed test files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 19:13:30 +02:00
kevinveenbirkenbachandClaude Fable 5 cdd2bbf81c refactor(tasks): apply the 00_core numbering scheme to branch-local roles
svc-net-tor and web-svc-mirror predate the task-file numbering lint the
swarm-nfs merge brought in (a numbered *_core.yml must be 00_core.yml);
rename their 01_core.yml and retarget the main.yml includes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 19:00:42 +02:00
kevinveenbirkenbachandClaude Opus 4.8 466f04503b feat(web-svc-cdn): serve the npm mirror from a compose one-shot build service
Replace the ansible container-run plus copy publish with a proper compose
build service deployed via the role's own stack. The one-shot writes the
npm mirror straight into the node-local served CDN root that openresty
reads, so there is no DiND bind-layer mismatch and no separate publish step.

- meta/volumes.yml: build mounts (served root rw, staged manifests ro,
  build script as a docker config)
- templates/compose.yml.j2 + env.j2: one-shot build service (restart no,
  swarm condition none), CDN_WEB_ROOT and ROLES_DIR from the mount targets
- meta/services.yml: placement manager, replicas 1, resources, deploy modes
- meta/networks.yml: local subnet (the role now ships a stack)
- meta/variants.yml: internal and external flavor variants
- meta/tests.yml + test.env.j2 + files/test/test.sh: flavor-branching e2e
- tasks: internal flavor deploys the stack via sys-stk-full; drop the
  container-run and ansible-copy publish; remove the unused dev-git include
- vars: pin CDN_FLAVOR to the role id (scope-safe) and compute host paths
  directly (avoid the volume-lookup double-template)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 15:22:35 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e79183f96a feat(ca): trust package-cache frontend CA in build containers
The package-cache frontend MITMs the registry with a cert signed by an
independent CA, not the stack root CA. Wire that extra CA into all three
container entry points so npm traffic through the cache validates. Gated
on the CA file existing, so it is a no-op when the cache is inactive.

- container.py: mount the frontend CA and set CA_TRUST_CERT_EXTRA in the
  container-run wrapper
- with-ca-trust.sh: consume CA_TRUST_CERT_EXTRA (combined bundle for
  single-file readers plus a separate OS trust anchor)
- compose_ca.py: compose and swarm side injection (env-only when unwrapped)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 15:22:35 +02:00
kevinveenbirkenbach 1a0af0ce09 Merge remote-tracking branch 'fork/feature/swarm-nfs' into feature/svc-net-tor
# Conflicts:
#	roles/sys-ctl-mtn-cert-deploy/tasks/main.yml
#	roles/sys-svc-compose-ca/tasks/main.yml
#	roles/web-app-dashboard/templates/javascript/oidc.js.j2
#	roles/web-app-mastodon/tasks/01_overrides.yml
#	roles/web-app-peertube/templates/compose.yml.j2
#	scripts/github/guide/compose_deploy.sh
#	scripts/github/guide/host_deploy.sh
#	scripts/make/alias.sh
#	tests/lint/ansible/roles/test_service_core_first_task_run_once.py
2026-07-24 04:30:56 +02:00
kevinveenbirkenbach 7e1cd73cd5 chore(tmp): pre-stage settings.json for bind-mount-safe swarm-nfs merge 2026-07-24 04:21:29 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0a3570ecea feat(lint): require replicas:1 for scheduler services
A scheduler (cron, clock, celery beat, a framework scheduler process) enqueues
recurring jobs; running more than one replica fires the same jobs each tick,
causing duplicate work, lock contention, and unique-index collisions (e.g.
GoodJob's (cron_key, cron_at)). On swarm the default replica count is
len(hosts), so a scheduler with no explicit replicas silently fans out one per
node. Worker services (sidekiq, worker) are excluded: they scale horizontally.

Add tests/lint/ansible/services/test_scheduler_single_replica.py to enforce
replicas: 1 on any service whose key ends in cron/scheduler/beat/clock, with a
nocheck opt-out. Fix the two offenders it found, moodle cron and zammad
scheduler, and justify openproject cron with the required single-replica nocheck.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 04:06:39 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e9b0d5f76e fix(swarm-drill): give the reschedule convergence poll 240s for slow apps
The failover-drill assert polled service convergence for 60 iterations (120s).
OpenProject web boots in ~140s, so after 06_drain_worker rescheduled it the
replicas turned healthy ~20s after the poll gave up, and 07_assert_state.sh
reported 'Replicas not fully converged: 1/3' and exited 1 even though the stack
did converge moments later. The loop breaks early once converged, so raising the
ceiling to 120 iterations (240s) costs fast apps nothing and only widens the
window for heavy apps.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 04:06:39 +02:00
kevinveenbirkenbachandClaude Opus 4.8 d82e21a144 fix(web-svc-cdn): stage each manifest under its own role, not the reserved role_name
The asset-mirror loop derived the staging dir and gate from a task var named
role_name, but role_name is Ansible's reserved role-magic variable pinned to the
executing role (web-svc-cdn); the vars override was silently ignored. So every
found package-lock.json (web-app-dashboard, web-app-littlejs) evaluated the gate
as 'web-svc-cdn in groups' and staged into CDN_ASSET_STAGE/web-svc-cdn/, the
second overwriting the first. No per-role dir was created, so keycloak-js (only
in dashboard's lock) was never mirrored and the CDN served 404 for
/npm/keycloak-js@24.0.5/dist/keycloak.js, breaking dashboard OIDC and the shared
universal-logout e2e across every app.

Rename the loop var to manifest_role so the override takes effect: each manifest
stages under its real role dir and the gate 'manifest_role in groups' mirrors
only deployed asset providers. Drop the redundant CDN_FLAVOR default (always set
in vars/main.yml).

Verified: build_assets.sh over dashboard's manifest produces
npm/keycloak-js@24.0.5/dist/keycloak.js; an ansible repro shows role_name stays
'web-svc-cdn' while manifest_role resolves to 'web-app-dashboard'.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 04:06:38 +02:00
kevinveenbirkenbachandClaude Fable 5 48ecbde887 fix(svc-bkp-volume-2-local): disable compose/swarm test-deploy modes
Host-only backup role: the missing compose/swarm mode declarations
defaulted to enabled, so the roundtrip ran both — swarm produced an
empty deploy matrix and failed. Mirror the svc-bkp-secrets sibling.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 01:27:56 +02:00
kevinveenbirkenbachandClaude Opus 4.8 32ff7d9ba0 fix(web-app-mailu): correct the renumbered create_user_token include depth
user_token.yml moved to tasks/utils/manage/ during the task-file renumbering,
but its include of create_user_token.yml kept a single '../', resolving to the
non-existent tasks/utils/05_manage_user/ and collapsing to /opt/src/05_manage_user/
at deploy time. Every app that pulls in mailu (fusiondirectory, kix, odoo,
openproject, penpot, pgadmin, peertube, xwiki, logout) failed on it. Use '../../'
to reach tasks/05_manage_user/create_user_token.yml.

Add tests/lint/ansible/tasks/test_relative_includes_resolve.py to guard the
class: it parses every role tasks file and asserts each relative
include_tasks/import_tasks/include_vars path resolves through ansible's search
bases, so a moved task or vars file can no longer leave a dangling include that
only surfaces at deploy time.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 00:25:25 +02:00