update_config never set failure_action, so swarmkit's default applied: pause,
at a max_failure_ratio of zero. One failed task stops the rollout for good, and
nothing in the deploy resumes it.
That contradicts the restart_policy three lines below. The policy replaces the
task and the service ends up healthy, while the update status stays latched and
service_converged.sh reads that status before it looks at any task, so the
deploy reports a failure over a service that is running. The rescue artifact
shows the sequence: the old task shut down, its replacement tripped over a
containerd object runsc could not reap, and the third task was Running while
the job died on "update latched".
Four swarm jobs failed that way, all in the update pass, never in the first.
The convergence check is untouched and stays as strict as it was: a genuinely
broken spec fails every task and still trips "tasks not running yet".
What this does give up: the latch was also the only thing catching a service
that keeps crash-looping. At ratio zero any failure latched, so a task dying
and restarting forever was guaranteed to be reported. Now the rollout
completes, and the check samples task state at one moment - a task on its
fiftieth restart still reads as Running, and wait_converged's `until` needs a
single lucky sample. Restoring that detection needs the check to look at task
churn, which the service ps history already carries and the script prints only
on failure today. Not in this commit.
max_failure_ratio cannot substitute here: at replicas 1, the common case in
this repository, one failure is always the whole service.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Docker's built-in pools yield about thirty networks. The stack holds
seventy-two, so every further network creation failed with 'all predefined
address pools have been fully subnetted' — which is why web-app-nextcloud
never came up: it brings its own networks and was the first role to hit the
ceiling.
Declare the pools beside the nested daemon's in group_vars, so both docker
daemons read their allocation from one place, and render them from there.
The base moves to 10.208.0.0/12 because 172.16/12 is exhausted: a probe
subnet in 172.28 was refused as overlapping while one in 10.210 was
created.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extra_hosts is the one compose key a service may not carry twice. The onion SSO
back-channel pin was emitted by a Jinja partial that six roles included, while
six other services wrote the key by hand; nothing stopped a service from
receiving both. YAML then either rejects the document or keeps one list and
drops the other, silently.
The partial becomes plugins/lookup/container_extra_hosts.py, and every pin now
goes through it. Callers add their own entries with extra_hosts=[...], the way
container_volumes already takes extra_volumes=, and the lookup merges them after
the automatic pin and deduplicates in first-seen order. With nothing to emit it
returns an empty string.
The swarm address reasoning from 7b1839681 is preserved and now written down in
the plugin rather than only in that commit: the pin must NOT be host-gateway in
swarm, because that alias resolves per node while openresty publishes :80 with
mode: host under node.role == manager. It carries the stack host's address,
which reaches every task because the compose file is rendered once with the
manager as inventory host. The mode is read as compose_mode_force or
DEPLOYMENT_MODE, not the bare cluster mode, so a role with an override does not
lose its pin and its overlay at once.
Two roles kept a pin that now collides with the automatic one, and both are
resolved the same way -- the hand-written entry stays for clearnet and yields
the .onion case to the lookup, which picks the address per mode. In
web-app-mailu the three conditional entries move into a role-local
mailu_extra_hosts lookup exposed as MAILU_EXTRA_HOSTS, because an eight-line
conditional expression in the template argument is worse than the block it
replaced. It returns its entries as separate results, so vars/main.yml uses
query(); plain lookup() would join them into one comma-separated string.
tests/lint/ansible/jinja/test_compose_extra_hosts_use_lookup.py fails on any
literal extra_hosts: key in a role template, so the invariant survives the next
author who needs a pin. It follows test_compose_ports_use_container_ports_lookup:
the raw construct is forbidden, the lookup is not mandated per service -- a
service with no pins is correct without it, and the swarm branch raises when
facts are absent, which would turn a no-op into a failure surface for containers
that never speak OIDC.
Equivalence checked by rendering rather than by reading: under the trim_blocks +
lstrip_blocks the stack-host action forces for compose.yml.j2, both former call
site shapes (indented, and matrix's column 0) emit byte-identical text to the
new form when the block applies. The no-op case differs -- a whitespace-only
line where the include emitted none -- and yaml.safe_load returns the same
document for both.
31 unit tests; make test-lint 325 passed. Does not touch web-app-espocrm, whose
missing pin is a separate claim and gets its own commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
force_shared_db flipped every DB service to shared with no exemption, so
web-app-magento's deliberate embedded MariaDB 11.4 - declared shared:
false with a comment warning that no bot may move the major underneath
it - became dead config and setup:install hit the central engine's 12.3.2,
outside Magento's supported range. The run log names the override:
'swarm force-shared: web-app-magento.services.mariadb -> shared=true'.
Exempts a DB service whose role meta declares a literal shared: false AND
whose role is manager-pinned. The second condition is not decoration:
utils/storage/nfs.py keeps a manager-pinned role's volumes node-local, and
InnoDB over NFS is why the harness force-shares in the first place. Of the
two roles with a literal local DB, magento qualifies and bigbluebutton
(unpinned) does not, so its behaviour is unchanged.
derive_includes._force_shared_db_view documents itself as mirroring
force_shared_db, so it takes the same exemption - without it the dep walk
would compute magento's include closure against the central engine while
the deploy runs a local sidecar.
The sidecar carried no deploy block at all, so an exempted service would
have been free-floating with one task per node over a single-writer
datadir. The block now comes from a container_deploy lookup shared with
sys-svc-container/templates/deploy.yml.j2, which replaces the replicas
ternary and the duplicated placement constraint there. Its unit test
caught a real defect while being written: with compose_mode_force=swarm
over DEPLOYMENT_MODE=compose the delegated compose_replicas returned
empty, emitting a deploy: key with no child.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both sidecar templates rendered a swarm service with no deploy: block at
all, so the kata placement constraint never applied and a non-kata
workload could land on a kata node under runsc. They also ran without
resource limits under swarm.
Include deploy.yml.j2 from both templates. Guard the replica count with
an explicit override, because deploy.yml.j2 otherwise falls back to
compose_replicas (the group size): these caches have no clustering, so a
replica per node would leave a consumer's session missing from the other
instances. Swarm already defaults to one task without a deploy block, so
the forced value preserves the current behaviour.
The central path is untouched: svc-db-redis/compose.yml.j2 reaches
deploy.yml.j2 via sys-svc-container/base.yml.j2, where the override is
undefined and the original resource_filter expression applies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extra_hosts mapped auth.<onion> to host-gateway in both deploy modes, but
host-gateway is a per-node address while openresty publishes :80 with mode: host
under node.role == manager. Every replica scheduled on a worker therefore dialled
its own gateway, where nothing listens, and the server-side OIDC token exchange
died with 'Connection refused'. Penpot 0 of run 31340965955 measured it: 60 of
the ConnectException lines on wrk-02, 20 on wrk-01, zero on mgr-01.
In swarm the pin now resolves to the stack host's routable address -- the same
destination svc-net-tor already sends all hidden-service traffic to via
TOR_HS_BACKEND_HOST. That address reaches every task, because
plugins/action/stack_host_template.py skips any host whose IS_STACK_HOST is
false and IS_STACK_HOST is true only on a swarm manager: the compose file is
rendered once with the manager as inventory host, so ansible_facts carries the
manager's IP and swarm applies the same extra_hosts to every task.
Retargeting rather than dropping the pin, because a manager-pinned consumer
(the keycloak sso-proxy) is measurably working through this very line today.
The guard fa2f6eb7e removed would have deleted it for them too.
Keyed on compose_mode_force or DEPLOYMENT_MODE, not the bare cluster mode:
container_networks resolves attachments through the same effective mode, so a
role with an override must not lose its pin and its overlay at once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven include templates under sys-svc-container decided how a container was
probed, and they read their inputs from template-local `{% set %}` scope.
That scope leaks: a value set for one service silently reached every service
below it, which is how taiga's front and gateway ended up probing a port
declared for a different service, and how bluesky's frontend probed a PDS
endpoint.
Probes now live in utils/docker/healthcheck.py, one class per flavor behind a
PROBES registry, and each service declares its own:
healthcheck:
flavor: curl
path: health/ready
start_period: 20m
`port_key` picks another entry under ports.internal for the services that
expose their probe elsewhere (keycloak on management, nextcloud talk on
signaling). Without a flavor the entry must carry an explicit test argv, so a
missing declaration fails loudly instead of rendering a probe against nothing.
Rendering HEAD's templates against the new output confirms 49 of 56 call
sites are byte-identical; the remaining seven are covered by an Ansible run.
Three probes change on purpose: erpnext's backend gains a check it never had,
bluesky's frontend drops the leaked PDS path, and msmtp_curl moves from a
hardcoded localhost:80 to the service's own port on 127.0.0.1 like every
other flavor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, each resolved by combining both sides rather than picking one.
compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.
daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.
The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Read platform_arch from the role config and append a node.platform.arch
constraint where it is set, so a role whose image exists for one
architecture only cannot be scheduled onto a node that could never run
it. LM Studio ships x86_64 alone and is the first consumer.
The constraint lives with the other placement rules rather than in the
consuming role, because architecture binding is a property any role with
a platform-specific image can carry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The template replaces daemon.json wholesale, so the data root the swarm lab puts on the chosen filesystem was dropped on every managed node. The rescue artifact of run 30688981182 shows nfs-server keeping it while mgr-01, wrk-01 and wrk-02 lost it, leaving those nodes on the container overlay instead of the prepared loop filesystem. Read the file back before rendering and re-emit the value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tcp healthcheck ends its probe with `cat <&3 | grep -q 'HTTP/1'`, which
matches the status LINE. A backend answering HTTP/1.1 500 therefore passes the
probe exactly like a 200. Swarm's only mechanism for taking a task out of the
VIP is the healthcheck, so a Keycloak replica serving 500s stays in rotation
indefinitely, and with replicas: 3 it catches roughly every third request.
That is what failed run 30586213932's keycloak job. The Playwright artifact's
error context shows the sign-out assertion waiting on the login form while the
page rendered "We are sorry... An internal server error has occurred" on
/realms/master/protocol/openid-connect/auth, and the rescue artifact's container
log shows why the replica was broken: ISPN000196, failed to recover cluster
state after becoming coordinator, followed by 25 Infinispan TimeoutExceptions,
17 of them through StateTransferLockImpl. Eight tests failed across three roles
because web-app-dashboard and web-app-prometheus authenticate against the same
Keycloak.
The status grading lands in a new template rather than in tcp.yml.j2, because
that one has fifteen other consumers whose probe paths were not audited. They
keep the behaviour they have; only Keycloak moves. The new probe reads the
status line with head -n1 and accepts 2xx/3xx only.
Keycloak also moves from health/live to health/ready: liveness answers "the JVM
is up", which a broken replica satisfies, while readiness answers "requests can
be served", which is the distinction the VIP needs.
Readiness returns 503 through schema migration, so the strict probe needs a
start period that outlives it. Measured across all five replica logs of that
run, Keycloak reaches "started" in 13.0-13.2s including migration, so 180s is
roughly thirteen times the observed boot. The deploy gate that waits for
health=healthy allows 600 retries at 2s, i.e. 1200s, so the longer start period
consumes at most an eighth of its budget.
Tool availability verified in quay.io/keycloak/keycloak:26.7.0 (digest
0f198be292568439d700cdbfb893e69a6009bb43a94a06a945b1d3d506c76b13): bash, cat,
grep and head are all present, so the probe's head -n1 resolves.
This fixes detection, not the clustering fault behind it. Why the Infinispan
cluster fails to form is still unknown; a replica that never clusters is now
pulled from the VIP instead of serving errors, but if no replica clusters there
is nothing healthy to route to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Archlinux entry in SYS_SVC_CONTAINER_DOCKER_FIREWALL_BACKEND_BY_OS_FAMILY landed in daemon.json unconditionally, and dockerd rejects `swarm init` with 'firewall-backend=nftables is incompatible with swarm mode'. Swarm only ever works on the iptables backend, which arch provides through iptables-nft, so the per-family pin now applies outside swarm mode only and compose deploys keep it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Archlinux entry in SYS_SVC_CONTAINER_DOCKER_FIREWALL_BACKEND_BY_OS_FAMILY landed in daemon.json unconditionally, and dockerd rejects `swarm init` with 'firewall-backend=nftables is incompatible with swarm mode'. Swarm only ever works on the iptables backend, which arch provides through iptables-nft, so the per-family pin now applies outside swarm mode only and compose deploys keep it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The daemon configuration named fuse-overlayfs as the storage driver for every
host it was rendered on, while the package that provides it is installed under
the opposite condition: only inside a nested container. A host that is not
docker-in-docker therefore received a daemon pointed at a graphdriver nothing
had installed, and dockerd does not start on one.
That path is reached by a bare host in the svc-virt-kata group, which is
exactly the shape the agent tier is meant to run on. It stayed invisible
because every deploy in this repository nests through the stack container, so
the condition that installs the package and the condition that renders the file
were never false at the same time.
fuse-overlayfs exists for the nested case, where overlay2 cannot stack on
overlay2. On a bare host the native default is both correct and faster, so the
key is now emitted only where it is needed and the JSON comma structure moves
with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reaching an onion address required each role to point its own http client
at the tor socks proxy, which then carried clearnet traffic too — fatal
in the test network, which has no exit, and the reason nextcloud needed a
per-host exclusion list.
The egress router already redirects tor's virtual range to the TransPort
and dnsmasq already forwards .onion to tor's DNSPort; the piece missing
was reachability. dnsmasq listened on loopback only, which docker
discards, so containers never saw it. It now also answers on the docker
bridge, the daemon points there with the clearnet resolver kept as
fallback, and the role's own onion resolves to a container-reachable
backend instead of loopback.
Nextcloud drops its proxy block and the exclusion list; every other role
gets the same behaviour without configuring anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sandbox node runs gVisor as its daemon-wide default runtime and every
container scheduled there inherits it, so keycloak failed to start on one.
Swarm has no taints, which leaves the inverse placement constraint as the only
lever: sandbox consumers are pinned to the labelled nodes, everything else is
kept off them. Both halves were measured on a live cluster first, because a
docker constraint has no presence test and rejects an empty comparison value:
`!= true` does match a node without the label, and three replicas under it
avoided two labelled workers entirely.
That makes the set of labelled nodes load-bearing for the whole platform, and
it was decided by accident. Without an `svc-virt-kata` inventory group the
service loader runs the role on every host, so every worker becomes a sandbox
node and the rest of the platform is evacuated onto the stack host. The role
now refuses to run in swarm mode until the operator names those nodes.
With the group present the opposite gap opened: the provisioner puts every role
group on the single `--host` it is given, so the role lands on the manager
alone, the label task skips itself there, and no node ends up labelled. The old
assert only proved that a worker exists, which says nothing about where the
role ran, so the deploy passed the check and then hung until the convergence
wait gave up. The assert now reads the labels back from the stack host.
`docker node ls --filter label=` matches engine labels, not swarm node labels;
the filter has to be `node.label`.
The constraint itself is emitted only where a sandbox tier exists, so a deploy
without an agent role is untouched by any of this, and the lab dedicates a
single worker: labelling both would leave the manager as the only home for
every other service.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`svc-virt-kata` is pulled in as a shared service of the agent roles, so it also
executes on hosts that are in no inventory group of that name. Gating the
runtime registration on `'svc-virt-kata' in group_names` therefore left such a
host with runsc installed and its node labelled kata-capable, while
`daemon.json` registered nothing and the daemon kept running everything under
runc: the exact isolation breach the label promises is closed. The stat probes
in `daemon_config.yml` decide instead, and they cannot be fooled by an
inventory that says otherwise.
Nothing else in the tree installs or references runsc, so a shim on disk is
proof that this role put it there.
The agent compose templates pin their runtime on the same `services.kata`
switch that already drives the swarm placement constraint, instead of pinning
unconditionally against a daemon config written under a different condition.
Both roles declare that key themselves, so the lookup is the strict form: a
deleted flag now aborts the render rather than silently starting the agent
without a sandbox.
The new unit test covers what nothing covered before, which is how the runc
breach went unnoticed: an absent shim is never registered, a default runtime is
always a registered one, the stack host keeps the shared kernel, compose never
sets a daemon-wide default, and registration follows the probe rather than the
inventory group.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`docker stack deploy` drops the per-service `runtime:` key, so a swarm worker
can only carry the sandbox as its daemon-wide `default-runtime`. That was never
set: the role reached the manager only, no node was labelled, and one of three
agent replicas executed model-generated code under plain runc.
Setting it from the base render is not possible either. The container role
writes `daemon.json` before the runtime binary exists, and dockerd refuses to
start when a runtime path does not resolve ("stat /usr/local/bin/runsc: no such
file or directory"). The role's own attempt to re-render afterwards was a silent
no-op, because `include_role: sys-svc-container` hits the
`run_once_sys_svc_container` guard.
Move the probes plus the render into `daemon_config.yml` and let the role
re-enter it via `tasks_from` after installing runsc. Only runtimes whose binary
the probe found are registered, and `default-runtime` is written only for a
present shim on a non-stack host, so the stack host keeps the shared kernel for
the platform's own services.
The placement constraint compares against a fixed literal because Docker
constraints have no presence test: label the node `kata-capable=true` rather
than with the runtime name, otherwise a Kata node fails a `== runsc`
comparison. Without a worker outside the stack host no node is eligible at all,
so refuse the deploy instead of leaving the service pending forever.
`select_runtime.yml` is gone: hermes and openclaw included it by absolute path
out of a foreign role, and one probe in the container role now serves every
consumer.
Verified live on a three node cluster: worker `DefaultRuntime=runsc`, both
workers labelled, manager unlabelled, service constrained, and the running
replica reports `runtime=runsc`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Agent workloads execute model-generated code, so they need a kernel
boundary on top of the app's own sandboxing: Kata plus Firecracker where
hardware virtualisation is available, the gVisor userspace kernel
otherwise, never plain runc.
runsc is fetched as the upstream static release binary with its published
sha512 and the host architecture in the URL, because Debian ships only a
2024 build and Arch and Fedora ship none at all; the same path serves the
arm64 Raspberry Pi target.
select_runtime.yml is a reusable task file: it picks kata only when
/dev/kvm, the kata shim and a bare host all hold. /dev/kvm is visible
inside a Docker-in-Docker daemon through host passthrough, but the microVM
still cannot start there (fuse-overlayfs, no containerd block devices), so
DinD is excluded explicitly and degrades to runsc.
daemon.json gains a runtimes block gated on group membership. Under DinD
runsc additionally needs --network=host: its netstack cannot pluck
interfaces inside a nested netns and aborts with 'cannot run with network
enabled in root network namespace'. The flag keeps the syscall sandbox and
reaches the container netns through real sockets.
The role bootstraps before the service-loader preload, alongside
svc-storage and svc-registry, so the runtime is registered and the daemon
restarted before the first workload compose file renders.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The compose-only guard left swarm sso-proxies resolving auth.<onion>
through the embedded DNS, which races container startup and
intermittently hands back a loopback that dead-ends in the task netns
(token exchange refused). The static extra_hosts entry wins in both
modes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New roles/sys-svc-container/templates/env.j2 sets TZ={{ HOST_TIMEZONE }} and LANG=C.UTF-8. Every role env.j2 includes it at the top; it renders via stack_host_template, the same engine that resolves the includes in compose.yml.j2, so it works in both compose and swarm. TZ/LANG are removed from the individual role env.j2 so each value stays single-source.
New lint tests/lint/ansible/test_env_base_include.py verifies every role env.j2 includes the base and never re-sets a base-owned key (keys derived from the base file).
mattermost and gitea meta/volumes.yml drop the /etc/timezone host-path bind: swarm hard-rejects a task whose bind source is missing on the node (Debian-trixie nodes have no /etc/timezone). TZ now comes from the base env. /etc/localtime bind also dropped; the container timezone follows TZ.
Non-directive comments were removed from the touched env.j2 (banners, restatements, reference URLs, narration) so the changed files pass the comment-lint; only tool directives remain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The DR drill ran in every variant job because its gate checked the
job-local plan index, which is 0 in each of the single-variant CI jobs;
gate it on the absolute round index so the destructive drill and the
backup host deploy run once per app, in the variant-0 job. Guard the
matrix steps with a disk watchdog that aborts visibly below 6G free
instead of letting the Actions Worker die silently on ENOSPC with no
artifacts. Cap node journald (256M) and DiD container logs (10m x 2),
which were unbounded across five nodes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On an onion node the OIDC issuer (Keycloak) is a .onion; libcurl (oidc_login's Guzzle client) refuses to resolve .onion per RFC 7686, so the server-side OIDC backchannel (discovery/token/jwks) fails. Set Nextcloud's system proxy to socks5h://host.docker.internal:<tor-socks> and expose the host Tor SOCKS to the app container via extra_hosts, both gated on SSO being enabled and the Keycloak domain being an onion. No-op on clearnet nodes.
Hoist the repeated services.sso.enabled and Keycloak-domain lookups into NEXTCLOUD_SSO_ENABLED and NEXTCLOUD_OIDC_HOST constants (mirroring MATRIX_OIDC_HOST) and use them across the role.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
deploy.yml.j2 keyed on the cluster-wide DEPLOYMENT_MODE, so forced-compose
roles (web-app-matrix ansible flavor, web-app-bigbluebutton) rendered a
swarm deploy block into their node-local compose files; docker compose v2
honors deploy.resources.limits, capping Synapse at 512m and resetting
connections in the swarm nextcloud-3 job. Use the same
compose_mode_force-or-DEPLOYMENT_MODE resolution the compose_* and
container_* lookup plugins apply; the default() is required because
renders like the xwiki manager-ops re-render run outside the
sys-svc-compose scope where compose_mode/compose_mode_force are defined.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rename the meta/services.yml key across all roles, the roles_with_default_placement lookup (now roles_with_placement), the get_role_default_placement / iter_roles_with_default_placement helpers, and every consumer (swarm closure, compose_volumes filter, nfs_prep, container deploy template, constructor stage, extend_inventory) plus their tests and docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared tcp healthcheck grepped the response for the literal 'HTTP/1.1',
but odoo's werkzeug server (and port-ui) answer HTTP/1.0, so the probe never
matched despite a 200 response; swarm then killed the task as unhealthy and the
stack never converged. Grep for 'HTTP/1' so both 1.0 and 1.1 status lines pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the per-role service_replicas Jinja variable with a per-service
replicas: field in meta/services.yml. deploy.yml.j2 resolves it via
resource_filter (with entity fallback), the same path already used for
cpus/mem, so services.yml is the single source of truth for replica
overrides.
- 13 web roles migrated to per-service (or entity-level) replicas: 1
- matomo bootstrap -> replicas: 0, mastodon migrate -> replicas: 1
- minio/seaweedfs provider entries -> replicas: 1, which also pins the
central single-writer object-store providers instead of relying on the
topology default
- service_replicas removed entirely; dropped its dead reference from the
compose-structure lint allowlist and updated requirement 023 wording
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The tcp and wget healthcheck templates set no start_period (so it
defaulted to 0), and nc/http hardcoded it, meaning a role's
container_healthcheck_start_period was ignored for every probe except
curl. Odoo sets 5m but ran with a 0s grace. Make all templates honor the
configured value while keeping each template's previous default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sys-svc-container healthcheck templates (curl/wget/nc/tcp/http) now
derive container_port from services.<service_name>.ports.internal.http,
so a role no longer needs to set container_port only to feed the shared
healthcheck.
Add the container-port-set-unused lint that fails a role setting
container_port without using it elsewhere, then resolve every flagged
role: drop the redundant assignment where the derivation reproduces the
value, or mark it with nocheck where the port is non-http, a constant, or
keyed on a service the derivation cannot reach.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Jinja's indent filter leaves the first line untouched, so the filter
indent(4) wrapper rendered healthcheck: at column 0 and produced
invalid compose YAML. The template now ships pre-indented like its
sibling healthcheck templates; taiga drops the wrapper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every compose.yml.j2 now declares services as '{{ service_name }}:' with
its '{% set %}' knobs on two-space lines above the key, includes at four
spaces and if/for tags aligned to the content they wrap. The layout
renders correctly because stack_host_template enables lstrip_blocks for
compose.yml.j2; indent(4, True) in the container base and '{%+' guards
in whitespace-sensitive scalars keep the shared fragments identical
under both render modes. A new lint test enforces the skeleton.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The curl healthcheck include already emits start_period, and jenkins appended a second one (10m) right after the include, producing a duplicate mapping key so 'docker compose config' failed to parse (yaml: mapping key start_period already defined). Make the curl include's start_period overridable via container_healthcheck_start_period (default 30s, so the 26 other curl-include roles are unchanged) and have jenkins set it to 10m before the include instead of appending.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The curl healthcheck template lacked a start_period (unlike the http/nc ones), so slow-starting containers were marked unhealthy and rescheduled during boot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On swarm, friendica's manager-node replica failed its curl healthcheck under
warmup contention: with no start_period the first DB/migration warmup probes
counted, 3 misses in 3 minutes made swarm kill the task on a fixed cadence
(2/3 converge), while the worker replicas stayed healthy. Add start_period:
120s + retries: 5 + a 30s interval so the warmup window is not counted and a
loaded node tolerates a slow first probe.
This lets friendica (and the other curl-healthchecked PHP apps) converge
multi-replica without pinning to a single replica. start_period is a standard
healthcheck field honored identically by compose and swarm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DEPLOYMENT_MODE is binary (swarm / compose), so a negated `!= 'swarm'` gate
always means `== 'compose'`. Replace every negated gate across roles/ and
group_vars with the positive form for readability, and add a lint
(tests/lint/ansible/variables/test_no_deployment_mode_negation.py) that forbids
`DEPLOYMENT_MODE !=` in .yml/.yaml/.j2 with a `# nocheck:
deployment-mode-negation` escape.
The compose-verb / chdir / jinja-gate lint guidance messages now suggest the
positive form too; the gate matcher (utils/annotations/task_gate.py) and its
test inputs keep recognising the legacy `!= 'swarm'` so a nocheck-marked
exception is still detected as compose-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In swarm the matomo_bootstrap service inherited compose_replicas() (topology default = node count), so multiple installer replicas ran the Matomo web-installer against the shared central DB at once and raced on table creation ('[install] Please fix the following errors'). The bootstrap is a one-shot (restart_policy: no) that must run exactly once. Add a per-service replicas override: deploy.yml.j2 honours a 'service_replicas' var when defined (else the topology default, unchanged for every other service), and matomo's compose sets service_replicas=1 for the bootstrap only. The main matomo service is rendered before the set and keeps the topology default.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In github CI the whole manager-pinned stack (mailu + central infra) lands on one ~16GB runner; the cumulative deploy.resources.reservations.memory exceeds the node, so swarm rejects placement with 'no suitable node ... insufficient resources on 1 node' (mailu_admin -> bluesky). Swarm reservation accounting counts physical RAM, not swap, so the enlarged runner swap cannot help. Gate the reservations block behind RUNTIME \!= 'github' (RUNTIME is baked into the inventory from detect_runtime, confirmed =github in CI); host/dev/act keep full reservations for QoS/spread.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Load openresty via the sys-svc-compose network/routine.yml helper (same
pattern as the db/ollama roles) so the shared `openresty` overlay is created
attachable + swarm-scoped before the stack deploys; emit the top-level
networks block via compose_networks.
- Add an opt-in `order:` to the swarm update_config and set it to `stop-first`
for openresty: it publishes 80/443 in host mode, so a default rolling update
deadlocks (the new task cannot bind ports the old task still holds).
- Relax the network-include lint to accept a single compose_networks lookup in
a host-mode template (openresty is mode-conditional: host in compose, overlay
in swarm).
- Whitelist openresty's path_join network/routine.yml include in
.ansible-lint-ignore (same dynamic-include pattern as the db/ollama roles).
Runtime-confirmed (swarm, attempt 32): openresty created its overlay, attached
consumers, reached backends, and redeployed cleanly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Convert the two registry-driven networks templates into pure-Python
rendering functions plus thin Ansible lookup plugins:
roles/sys-svc-compose/templates/networks.yml.j2 -> {{ lookup('compose_networks') }}
roles/sys-svc-container/templates/networks.yml.j2 -> {{ lookup('container_networks') | indent(4) }}
The rendering logic lives in utils/networks/render.py as pure-Python
functions that take a registry + closure-style config/database lookups,
making the logic directly unit-testable without a Jinja stub harness.
The lookup plugins are thin wrappers that resolve application_id,
DEPLOYMENT_MODE, and swarm.network.encryption from the templar, build
the service_registry, and forward to the render functions.
Migrate 233 include lines across 85 production templates. Convert
web-app-matrix's hand-coded MDAD networks block (default + conditional
postgres + conditional openldap) to the generic lookup; this adds an
openresty attachment when services.sso.enabled (harmless extra
interface, pinned in test_matrix_mdad_consumer_attaches_postgres_ldap_sso).
Mark local-DB / redis sidecars (sys-svc-rdbms/templates/services/,
svc-db-redis/templates/service.yml.j2) with `# nocheck: networks-literal`
because they intentionally attach to default-only.
Extend the lint to scan ALL roles/*/templates/**/*.yml.j2 (not only
compose.yml.j2) and to forbid the legacy include form; the marker
above is the documented escape hatch.
Coverage:
- tests/unit/utils/networks/test_render.py (30 cases incl. encrypted=false literal pin)
- tests/unit/plugins/lookup/test_compose_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/unit/plugins/lookup/test_container_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/integration/infrastructure/compose/test_networks_render.py (10 snapshot scenarios)
- tests/integration/infrastructure/compose/test_network_includes.py (3 lint rules)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the hand-coded if-chains for db/ldap/ollama/proxy shared overlays
with a single registry-driven loop in sys-svc-{compose,container}/networks.yml.j2.
Each provider declares its overlay metadata under meta/server.yml.networks.overlay;
discover_role_services passes the block through verbatim into the service_registry.
Schema (modes/topology/consumer/aliases/proxy_resolvable) with implicit defaults:
- consumer.kind defaults to services_flags
- consumer.key defaults to <provides or entity_name>
- consumer.flags defaults to [enabled, shared]
- aliases defaults to [entity_name] (shared_net) or [] (default_net)
Openresty becomes a generic alias-collecting proxy: web-app-keycloak declares
a proxy_resolvable beacon with its public auth.<domain> alias; openresty sweeps
the registry at render time and harvests all proxy_resolvable aliases onto its
default-net overlay. Openresty no longer hardcodes keycloak's domain.
Lock the new behaviour with 9 render-snapshot tests covering provider self-render,
consumer attachment, beacon sweep, default-derivation, and compose-mode fallback.
Replace the Ansible-only `_entry.get('canonical') is not defined` filter with
`'canonical' not in _entry` so the templates also render correctly under vanilla
Jinja2 (test harness).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The mapping from compose-style `docker_restart_policy` (`no`,
`on-failure[:N]`, `always`, `unless-stopped`) to swarm's
`deploy.restart_policy.condition` (`none`, `on-failure`, `any`) was
seven lines of inline Jinja `{% set / if / elif / else %}` in
`roles/sys-svc-container/templates/deploy.yml.j2`. That made the
template noisy at the call site, hid the typo-safety logic from
unit tests, and forced every reviewer to re-derive the mapping table
from a chain of conditionals.
Move the mapping into a dedicated filter plugin so the call site is
a single readable expression and the mapping is exercised by a
focused test suite.
- plugins/filter/swarm_restart_condition.py: pure function, no Jinja
context dependency (the input is the only thing that matters).
Strips + str-casts the input so YAML-literal slop like `" no "`
or a numeric `0` does not defeat the mapping. Output is always a
member of the closed swarm condition set `{none, on-failure, any}`;
unknown / typo inputs map to `any` (safe default -- a misconfigured
long-running service still runs and the operator notices the typo
via the literal `condition: any` in the rendered file, instead of
silently degrading to `none` which would skip respawning a real
service).
- tests/unit/plugins/filter/test_swarm_restart_condition.py: 12
cases. Direct mapping per branch, `on-failure:N` suffix form
(suffix dropped because swarm carries max-attempts in a separate
`max_attempts` key), None / empty / whitespace fallback,
stringification of non-string input, strip semantics, an explicit
unknown-value test that pins the safe-default rationale, and a
property test over twelve inputs that asserts the output is always
inside the closed swarm condition set. A second test class
verifies the filter is registered under the exact name
`swarm_restart_condition` so a typo in the FilterModule shim does
not silently fall back to the identity filter.
- roles/sys-svc-container/templates/deploy.yml.j2: thirteen lines of
inline Jinja (`{% set _drp = … %}` + four `{% set
_swarm_restart_condition = … %}` branches) collapse to a single
filter expression:
condition: {{ docker_restart_policy
| default(DOCKER_RESTART_POLICY)
| swarm_restart_condition }}
The runtime mapping is identical to the pre-refactor inline
version (verified by the new unit tests against the same input
buckets), so nothing changes for existing roles.
