834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Docker's built-in pools yield about thirty networks. The stack holds
seventy-two, so every further network creation failed with 'all predefined
address pools have been fully subnetted' — which is why web-app-nextcloud
never came up: it brings its own networks and was the first role to hit the
ceiling.
Declare the pools beside the nested daemon's in group_vars, so both docker
daemons read their allocation from one place, and render them from there.
The base moves to 10.208.0.0/12 because 172.16/12 is exhausted: a probe
subnet in 172.28 was refused as overlapping while one in 10.210 was
created.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, each resolved by combining both sides rather than picking one.
compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.
daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.
The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The template replaces daemon.json wholesale, so the data root the swarm lab puts on the chosen filesystem was dropped on every managed node. The rescue artifact of run 30688981182 shows nfs-server keeping it while mgr-01, wrk-01 and wrk-02 lost it, leaving those nodes on the container overlay instead of the prepared loop filesystem. Read the file back before rendering and re-emit the value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Archlinux entry in SYS_SVC_CONTAINER_DOCKER_FIREWALL_BACKEND_BY_OS_FAMILY landed in daemon.json unconditionally, and dockerd rejects `swarm init` with 'firewall-backend=nftables is incompatible with swarm mode'. Swarm only ever works on the iptables backend, which arch provides through iptables-nft, so the per-family pin now applies outside swarm mode only and compose deploys keep it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Archlinux entry in SYS_SVC_CONTAINER_DOCKER_FIREWALL_BACKEND_BY_OS_FAMILY landed in daemon.json unconditionally, and dockerd rejects `swarm init` with 'firewall-backend=nftables is incompatible with swarm mode'. Swarm only ever works on the iptables backend, which arch provides through iptables-nft, so the per-family pin now applies outside swarm mode only and compose deploys keep it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The daemon configuration named fuse-overlayfs as the storage driver for every
host it was rendered on, while the package that provides it is installed under
the opposite condition: only inside a nested container. A host that is not
docker-in-docker therefore received a daemon pointed at a graphdriver nothing
had installed, and dockerd does not start on one.
That path is reached by a bare host in the svc-virt-kata group, which is
exactly the shape the agent tier is meant to run on. It stayed invisible
because every deploy in this repository nests through the stack container, so
the condition that installs the package and the condition that renders the file
were never false at the same time.
fuse-overlayfs exists for the nested case, where overlay2 cannot stack on
overlay2. On a bare host the native default is both correct and faster, so the
key is now emitted only where it is needed and the JSON comma structure moves
with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reaching an onion address required each role to point its own http client
at the tor socks proxy, which then carried clearnet traffic too — fatal
in the test network, which has no exit, and the reason nextcloud needed a
per-host exclusion list.
The egress router already redirects tor's virtual range to the TransPort
and dnsmasq already forwards .onion to tor's DNSPort; the piece missing
was reachability. dnsmasq listened on loopback only, which docker
discards, so containers never saw it. It now also answers on the docker
bridge, the daemon points there with the clearnet resolver kept as
fallback, and the role's own onion resolves to a container-reachable
backend instead of loopback.
Nextcloud drops its proxy block and the exclusion list; every other role
gets the same behaviour without configuring anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`svc-virt-kata` is pulled in as a shared service of the agent roles, so it also
executes on hosts that are in no inventory group of that name. Gating the
runtime registration on `'svc-virt-kata' in group_names` therefore left such a
host with runsc installed and its node labelled kata-capable, while
`daemon.json` registered nothing and the daemon kept running everything under
runc: the exact isolation breach the label promises is closed. The stat probes
in `daemon_config.yml` decide instead, and they cannot be fooled by an
inventory that says otherwise.
Nothing else in the tree installs or references runsc, so a shim on disk is
proof that this role put it there.
The agent compose templates pin their runtime on the same `services.kata`
switch that already drives the swarm placement constraint, instead of pinning
unconditionally against a daemon config written under a different condition.
Both roles declare that key themselves, so the lookup is the strict form: a
deleted flag now aborts the render rather than silently starting the agent
without a sandbox.
The new unit test covers what nothing covered before, which is how the runc
breach went unnoticed: an absent shim is never registered, a default runtime is
always a registered one, the stack host keeps the shared kernel, compose never
sets a daemon-wide default, and registration follows the probe rather than the
inventory group.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`docker stack deploy` drops the per-service `runtime:` key, so a swarm worker
can only carry the sandbox as its daemon-wide `default-runtime`. That was never
set: the role reached the manager only, no node was labelled, and one of three
agent replicas executed model-generated code under plain runc.
Setting it from the base render is not possible either. The container role
writes `daemon.json` before the runtime binary exists, and dockerd refuses to
start when a runtime path does not resolve ("stat /usr/local/bin/runsc: no such
file or directory"). The role's own attempt to re-render afterwards was a silent
no-op, because `include_role: sys-svc-container` hits the
`run_once_sys_svc_container` guard.
Move the probes plus the render into `daemon_config.yml` and let the role
re-enter it via `tasks_from` after installing runsc. Only runtimes whose binary
the probe found are registered, and `default-runtime` is written only for a
present shim on a non-stack host, so the stack host keeps the shared kernel for
the platform's own services.
The placement constraint compares against a fixed literal because Docker
constraints have no presence test: label the node `kata-capable=true` rather
than with the runtime name, otherwise a Kata node fails a `== runsc`
comparison. Without a worker outside the stack host no node is eligible at all,
so refuse the deploy instead of leaving the service pending forever.
`select_runtime.yml` is gone: hermes and openclaw included it by absolute path
out of a foreign role, and one probe in the container role now serves every
consumer.
Verified live on a three node cluster: worker `DefaultRuntime=runsc`, both
workers labelled, manager unlabelled, service constrained, and the running
replica reports `runtime=runsc`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Agent workloads execute model-generated code, so they need a kernel
boundary on top of the app's own sandboxing: Kata plus Firecracker where
hardware virtualisation is available, the gVisor userspace kernel
otherwise, never plain runc.
runsc is fetched as the upstream static release binary with its published
sha512 and the host architecture in the URL, because Debian ships only a
2024 build and Arch and Fedora ship none at all; the same path serves the
arm64 Raspberry Pi target.
select_runtime.yml is a reusable task file: it picks kata only when
/dev/kvm, the kata shim and a bare host all hold. /dev/kvm is visible
inside a Docker-in-Docker daemon through host passthrough, but the microVM
still cannot start there (fuse-overlayfs, no containerd block devices), so
DinD is excluded explicitly and degrades to runsc.
daemon.json gains a runtimes block gated on group membership. Under DinD
runsc additionally needs --network=host: its netstack cannot pluck
interfaces inside a nested netns and aborts with 'cannot run with network
enabled in root network namespace'. The flag keeps the syscall sandbox and
reaches the container netns through real sockets.
The role bootstraps before the service-loader preload, alongside
svc-storage and svc-registry, so the runtime is registered and the daemon
restarted before the first workload compose file renders.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DR drill ran in every variant job because its gate checked the
job-local plan index, which is 0 in each of the single-variant CI jobs;
gate it on the absolute round index so the destructive drill and the
backup host deploy run once per app, in the variant-0 job. Guard the
matrix steps with a disk watchdog that aborts visibly below 6G free
instead of letting the Actions Worker die silently on ENOSPC with no
artifacts. Cap node journald (256M) and DiD container logs (10m x 2),
which were unbounded across five nodes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
svc-cache-registry -> svc-registry-cache
svc-docker-registry -> svc-registry-docker
svc-docker-swarm -> svc-swarm
svc-docker-swarm-manager -> svc-swarm-manager
categories.yml: drop svc.docker, add svc.registry. Deliberately no svc.swarm
category so get_entity_name('svc-swarm') stays 'swarm' instead of collapsing
to '' (which would break discover_role_services).
Entity keys updated to match the new svc.registry prefix:
services.cache-registry.* -> services.cache.*
services.registry.* -> services.docker.*
01_constructor: replace svc-docker bootstrap-loop entry with explicit
include_role of svc-swarm + bootstrap loop over [svc-registry, svc-storage].
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DinD DNS chain reworked so the Playwright container run --rm npm install
no longer EAI_AGAINs on registry.npmjs.org:
- 03_start_swarm_nodes.sh runs each DinD with --dns 1.1.1.1/8.8.8.8 and
pre-seeds /etc/docker/daemon.json. systemd-resolved still clobbers the
outer resolv.conf at boot, so the daemon-level DNS is the only thing
that survives for containers the inner daemon spawns.
- daemon.json.j2 omits the "dns" key when networks.internet.dns is empty
(default), so the rendered config no longer overwrites the pre-seeded
daemon.json with "dns": [""] and break inner-container resolution.
- 09_write_extras.sh sets networks.internet.dns to a public resolver for
the test rig and bakes RUNTIME via cli.meta.runtime (so e2e-playwright
actually runs in act/github instead of being skipped under "host").
07_provision_inventory.sh generates mirrors.yml via
cli.contributing.mirror.resolver and passes --mirror to the inventory
provisioner only when cli.meta.runtime == "github". act/local keeps
upstream pulls (no mirror infrastructure to point at).
12_wait_converge.sh and 15_assert_state.sh accept any N/N replica state
instead of hardcoded 1/1, so multi-host-replica deploys converge cleanly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New role that runs registry:2 in pull-through proxy mode on the swarm
manager. Worker nodes' daemon.json gets a registry-mirrors entry plus
the corresponding insecure-registries entry so plain-HTTP traffic to
the cache is permitted. The role's inventory group lists exactly one
host (manager) so the new inventory-driven topology model resolves
DEPLOYMENT_MODE to compose for the cache itself.
Within a single deploy run, the three swarm workers all pull the same
base images via the cache instead of hitting registry-1.docker.io
three separate times. Measured speedup on iter26 vs the no-cache iter21
baseline: pkgmgr/apt-install steps 2.4-2.8x faster, Docker image pulls
1.8-2.8x faster.
Cross-iter cache persistence is intentionally out of scope: act
recycles the DinD nodes between runs so the cache volume is wiped each
time. Cross-run reuse would require either bind-mounting the cache
from the host or pointing daemon.json at the dev compose stack's
registry-cache; both are tracked separately.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the host-scoped set_fact resolvers in sys-svc-compose/tasks/01_core.yml
and tasks/stages/01_constructor.yml with a single group_vars definition driven
by group_names. Uppercase the variable to match the SPOT convention and drop
every `| default('compose')` fallback now that the resolution is unconditional.
Role-var opt-out on a swarm host still works via Ansible's role-vars > group_vars
precedence.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a manager-pinned, NFS-backed `registry:2` service plus a transparent
retag/push/rewrite step so swarm workers can pull custom-built images from a
local registry, replacing the `docker save | docker load` pilot crutch.
Mechanism: after `compose build`, the new `swarm_registry_push.py` reads the
rendered compose.yml, retags each service that declares both `build:` and
`image:` with `<manager>:5000/<image>`, pushes the retag, and rewrites the
`image:` field in place. Subsequent `docker stack deploy --resolve-image
never` then references the registry-qualified name, so workers pull from the
cluster-local registry over HTTP via `insecure-registries`.
No per-role changes required — the rewrite is centralized in
sys-svc-compose handlers and skips services without a `build:` directive.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Arch Linux uses nftables by default. Docker's default iptables backend
conflicts with nftables rules, causing broken container networking and
port forwarding. Explicitly configuring nftables ensures both coexist correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
