update_config never set failure_action, so swarmkit's default applied: pause,
at a max_failure_ratio of zero. One failed task stops the rollout for good, and
nothing in the deploy resumes it.
That contradicts the restart_policy three lines below. The policy replaces the
task and the service ends up healthy, while the update status stays latched and
service_converged.sh reads that status before it looks at any task, so the
deploy reports a failure over a service that is running. The rescue artifact
shows the sequence: the old task shut down, its replacement tripped over a
containerd object runsc could not reap, and the third task was Running while
the job died on "update latched".
Four swarm jobs failed that way, all in the update pass, never in the first.
The convergence check is untouched and stays as strict as it was: a genuinely
broken spec fails every task and still trips "tasks not running yet".
What this does give up: the latch was also the only thing catching a service
that keeps crash-looping. At ratio zero any failure latched, so a task dying
and restarting forever was guaranteed to be reported. Now the rollout
completes, and the check samples task state at one moment - a task on its
fiftieth restart still reads as Running, and wait_converged's `until` needs a
single lucky sample. Restoring that detection needs the check to look at task
churn, which the service ps history already carries and the script prints only
on failure today. Not in this commit.
max_failure_ratio cannot substitute here: at replicas 1, the common case in
this repository, one failure is always the whole service.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
force_shared_db flipped every DB service to shared with no exemption, so
web-app-magento's deliberate embedded MariaDB 11.4 - declared shared:
false with a comment warning that no bot may move the major underneath
it - became dead config and setup:install hit the central engine's 12.3.2,
outside Magento's supported range. The run log names the override:
'swarm force-shared: web-app-magento.services.mariadb -> shared=true'.
Exempts a DB service whose role meta declares a literal shared: false AND
whose role is manager-pinned. The second condition is not decoration:
utils/storage/nfs.py keeps a manager-pinned role's volumes node-local, and
InnoDB over NFS is why the harness force-shares in the first place. Of the
two roles with a literal local DB, magento qualifies and bigbluebutton
(unpinned) does not, so its behaviour is unchanged.
derive_includes._force_shared_db_view documents itself as mirroring
force_shared_db, so it takes the same exemption - without it the dep walk
would compute magento's include closure against the central engine while
the deploy runs a local sidecar.
The sidecar carried no deploy block at all, so an exempted service would
have been free-floating with one task per node over a single-writer
datadir. The block now comes from a container_deploy lookup shared with
sys-svc-container/templates/deploy.yml.j2, which replaces the replicas
ternary and the duplicated placement constraint there. Its unit test
caught a real defect while being written: with compose_mode_force=swarm
over DEPLOYMENT_MODE=compose the delegated compose_replicas returned
empty, emitting a deploy: key with no child.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both sidecar templates rendered a swarm service with no deploy: block at
all, so the kata placement constraint never applied and a non-kata
workload could land on a kata node under runsc. They also ran without
resource limits under swarm.
Include deploy.yml.j2 from both templates. Guard the replica count with
an explicit override, because deploy.yml.j2 otherwise falls back to
compose_replicas (the group size): these caches have no clustering, so a
replica per node would leave a consumer's session missing from the other
instances. Swarm already defaults to one task without a deploy block, so
the forced value preserves the current behaviour.
The central path is untouched: svc-db-redis/compose.yml.j2 reaches
deploy.yml.j2 via sys-svc-container/base.yml.j2, where the override is
undefined and the original resource_filter expression applies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Read platform_arch from the role config and append a node.platform.arch
constraint where it is set, so a role whose image exists for one
architecture only cannot be scheduled onto a node that could never run
it. LM Studio ships x86_64 alone and is the first consumer.
The constraint lives with the other placement rules rather than in the
consuming role, because architecture binding is a property any role with
a platform-specific image can carry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sandbox node runs gVisor as its daemon-wide default runtime and every
container scheduled there inherits it, so keycloak failed to start on one.
Swarm has no taints, which leaves the inverse placement constraint as the only
lever: sandbox consumers are pinned to the labelled nodes, everything else is
kept off them. Both halves were measured on a live cluster first, because a
docker constraint has no presence test and rejects an empty comparison value:
`!= true` does match a node without the label, and three replicas under it
avoided two labelled workers entirely.
That makes the set of labelled nodes load-bearing for the whole platform, and
it was decided by accident. Without an `svc-virt-kata` inventory group the
service loader runs the role on every host, so every worker becomes a sandbox
node and the rest of the platform is evacuated onto the stack host. The role
now refuses to run in swarm mode until the operator names those nodes.
With the group present the opposite gap opened: the provisioner puts every role
group on the single `--host` it is given, so the role lands on the manager
alone, the label task skips itself there, and no node ends up labelled. The old
assert only proved that a worker exists, which says nothing about where the
role ran, so the deploy passed the check and then hung until the convergence
wait gave up. The assert now reads the labels back from the stack host.
`docker node ls --filter label=` matches engine labels, not swarm node labels;
the filter has to be `node.label`.
The constraint itself is emitted only where a sandbox tier exists, so a deploy
without an agent role is untouched by any of this, and the lab dedicates a
single worker: labelling both would leave the manager as the only home for
every other service.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`docker stack deploy` drops the per-service `runtime:` key, so a swarm worker
can only carry the sandbox as its daemon-wide `default-runtime`. That was never
set: the role reached the manager only, no node was labelled, and one of three
agent replicas executed model-generated code under plain runc.
Setting it from the base render is not possible either. The container role
writes `daemon.json` before the runtime binary exists, and dockerd refuses to
start when a runtime path does not resolve ("stat /usr/local/bin/runsc: no such
file or directory"). The role's own attempt to re-render afterwards was a silent
no-op, because `include_role: sys-svc-container` hits the
`run_once_sys_svc_container` guard.
Move the probes plus the render into `daemon_config.yml` and let the role
re-enter it via `tasks_from` after installing runsc. Only runtimes whose binary
the probe found are registered, and `default-runtime` is written only for a
present shim on a non-stack host, so the stack host keeps the shared kernel for
the platform's own services.
The placement constraint compares against a fixed literal because Docker
constraints have no presence test: label the node `kata-capable=true` rather
than with the runtime name, otherwise a Kata node fails a `== runsc`
comparison. Without a worker outside the stack host no node is eligible at all,
so refuse the deploy instead of leaving the service pending forever.
`select_runtime.yml` is gone: hermes and openclaw included it by absolute path
out of a foreign role, and one probe in the container role now serves every
consumer.
Verified live on a three node cluster: worker `DefaultRuntime=runsc`, both
workers labelled, manager unlabelled, service constrained, and the running
replica reports `runtime=runsc`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
deploy.yml.j2 keyed on the cluster-wide DEPLOYMENT_MODE, so forced-compose
roles (web-app-matrix ansible flavor, web-app-bigbluebutton) rendered a
swarm deploy block into their node-local compose files; docker compose v2
honors deploy.resources.limits, capping Synapse at 512m and resetting
connections in the swarm nextcloud-3 job. Use the same
compose_mode_force-or-DEPLOYMENT_MODE resolution the compose_* and
container_* lookup plugins apply; the default() is required because
renders like the xwiki manager-ops re-render run outside the
sys-svc-compose scope where compose_mode/compose_mode_force are defined.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rename the meta/services.yml key across all roles, the roles_with_default_placement lookup (now roles_with_placement), the get_role_default_placement / iter_roles_with_default_placement helpers, and every consumer (swarm closure, compose_volumes filter, nfs_prep, container deploy template, constructor stage, extend_inventory) plus their tests and docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the per-role service_replicas Jinja variable with a per-service
replicas: field in meta/services.yml. deploy.yml.j2 resolves it via
resource_filter (with entity fallback), the same path already used for
cpus/mem, so services.yml is the single source of truth for replica
overrides.
- 13 web roles migrated to per-service (or entity-level) replicas: 1
- matomo bootstrap -> replicas: 0, mastodon migrate -> replicas: 1
- minio/seaweedfs provider entries -> replicas: 1, which also pins the
central single-writer object-store providers instead of relying on the
topology default
- service_replicas removed entirely; dropped its dead reference from the
compose-structure lint allowlist and updated requirement 023 wording
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In swarm the matomo_bootstrap service inherited compose_replicas() (topology default = node count), so multiple installer replicas ran the Matomo web-installer against the shared central DB at once and raced on table creation ('[install] Please fix the following errors'). The bootstrap is a one-shot (restart_policy: no) that must run exactly once. Add a per-service replicas override: deploy.yml.j2 honours a 'service_replicas' var when defined (else the topology default, unchanged for every other service), and matomo's compose sets service_replicas=1 for the bootstrap only. The main matomo service is rendered before the set and keeps the topology default.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In github CI the whole manager-pinned stack (mailu + central infra) lands on one ~16GB runner; the cumulative deploy.resources.reservations.memory exceeds the node, so swarm rejects placement with 'no suitable node ... insufficient resources on 1 node' (mailu_admin -> bluesky). Swarm reservation accounting counts physical RAM, not swap, so the enlarged runner swap cannot help. Gate the reservations block behind RUNTIME \!= 'github' (RUNTIME is baked into the inventory from detect_runtime, confirmed =github in CI); host/dev/act keep full reservations for QoS/spread.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Load openresty via the sys-svc-compose network/routine.yml helper (same
pattern as the db/ollama roles) so the shared `openresty` overlay is created
attachable + swarm-scoped before the stack deploys; emit the top-level
networks block via compose_networks.
- Add an opt-in `order:` to the swarm update_config and set it to `stop-first`
for openresty: it publishes 80/443 in host mode, so a default rolling update
deadlocks (the new task cannot bind ports the old task still holds).
- Relax the network-include lint to accept a single compose_networks lookup in
a host-mode template (openresty is mode-conditional: host in compose, overlay
in swarm).
- Whitelist openresty's path_join network/routine.yml include in
.ansible-lint-ignore (same dynamic-include pattern as the db/ollama roles).
Runtime-confirmed (swarm, attempt 32): openresty created its overlay, attached
consumers, reached backends, and redeployed cleanly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mapping from compose-style `docker_restart_policy` (`no`,
`on-failure[:N]`, `always`, `unless-stopped`) to swarm's
`deploy.restart_policy.condition` (`none`, `on-failure`, `any`) was
seven lines of inline Jinja `{% set / if / elif / else %}` in
`roles/sys-svc-container/templates/deploy.yml.j2`. That made the
template noisy at the call site, hid the typo-safety logic from
unit tests, and forced every reviewer to re-derive the mapping table
from a chain of conditionals.
Move the mapping into a dedicated filter plugin so the call site is
a single readable expression and the mapping is exercised by a
focused test suite.
- plugins/filter/swarm_restart_condition.py: pure function, no Jinja
context dependency (the input is the only thing that matters).
Strips + str-casts the input so YAML-literal slop like `" no "`
or a numeric `0` does not defeat the mapping. Output is always a
member of the closed swarm condition set `{none, on-failure, any}`;
unknown / typo inputs map to `any` (safe default -- a misconfigured
long-running service still runs and the operator notices the typo
via the literal `condition: any` in the rendered file, instead of
silently degrading to `none` which would skip respawning a real
service).
- tests/unit/plugins/filter/test_swarm_restart_condition.py: 12
cases. Direct mapping per branch, `on-failure:N` suffix form
(suffix dropped because swarm carries max-attempts in a separate
`max_attempts` key), None / empty / whitespace fallback,
stringification of non-string input, strip semantics, an explicit
unknown-value test that pins the safe-default rationale, and a
property test over twelve inputs that asserts the output is always
inside the closed swarm condition set. A second test class
verifies the filter is registered under the exact name
`swarm_restart_condition` so a typo in the FilterModule shim does
not silently fall back to the identity filter.
- roles/sys-svc-container/templates/deploy.yml.j2: thirteen lines of
inline Jinja (`{% set _drp = … %}` + four `{% set
_swarm_restart_condition = … %}` branches) collapse to a single
filter expression:
condition: {{ docker_restart_policy
| default(DOCKER_RESTART_POLICY)
| swarm_restart_condition }}
The runtime mapping is identical to the pre-refactor inline
version (verified by the new unit tests against the same input
buckets), so nothing changes for existing roles.
- roles/sys-svc-container/templates/deploy.yml.j2: render
`placement.constraints: node.role == manager` in the swarm
`deploy:` block whenever the application_id is one of the roles
whose `meta/services.yml` declares `default_placement: manager`
(resolved via the `roles_with_default_placement` lookup). Pins
state-sensitive services (mariadb, postgres, openldap, openresty,
registries) to the manager so their bind-source / local volume
expectations hold; app roles without the marker stay free to land
on workers.
- roles/sys-svc-compose/handlers/main.yml: add `retries: 10 / delay: 6
/ until: succeeded` to the `swarm: retag built images for local
registry and push` handler. On a cold deploy the local registry task
may still be settling when the push fires; retrying for up to 60s
lets the push outlast the registry's warmup instead of aborting the
whole stack deploy on a single transient connect-refused.
Add the compose_replicas lookup plugin as the single source of truth for
swarm replica calculation. Default emits `replicas: len(groups[application_id])`
(topology-driven); override term forces an explicit value. In compose mode
the lookup returns the empty string because docker-compose ignores
deploy.replicas.
Refactor deploy.yml.j2 to use the lookup. The previous service_replicas
default in defaults/main.yml is no longer needed and is dropped.
Add a lint test that walks every roles/*/templates/compose*.yml.j2 and
verifies each service entry under the top-level services: declares
replica handling via one of three valid paths:
1. Include sys-svc-container/templates/base.yml.j2 (full inheritance).
2. Include sys-svc-container/templates/deploy.yml.j2 directly (escape
hatch for services that own their restart/logging/env_file and
therefore can't use base.yml.j2 without duplicate-key conflicts).
3. Call lookup('compose_replicas', ...) explicitly for handcrafted
deploy: blocks.
Fix the 7 web-app compose templates flagged by the new lint (bluesky,
dashboard, magento, matrix, moodle, nextcloud, opentalk) by including
deploy.yml.j2 directly and setting service_name as a SPOT before each
service entry, so the service name in the yaml key and the resource_filter
lookups in deploy.yml.j2 both flow from one variable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Default service_replicas to (groups[application_id] | length) instead of
hard-coded 1. Single-host inventories still get replicas=1; 3-host swarm
inventories now get replicas=3 for every stateless service that renders
the deploy: block. Stateful services (DBs etc.) are unaffected because
deploy.yml.j2 gates the block on DEPLOYMENT_MODE == 'swarm' and stateful
services run as plain compose.
Simplify deploy.yml.j2 (the | default fallback in the template is now
redundant) and drop the explicit replicas:3 + SHOPWARE_WORKER_REPLICAS
plumbing from web-app-shopware - the new default covers it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DEPLOYMENT_MODE is now derived per role from group membership:
swarm iff len(groups[application_id]) > 1, else compose. The cluster
infrastructure flag SWARM_CLUSTER_AVAILABLE is true whenever any host
sits in svc-docker-swarm, so DB and proxy services render attachable
overlay networks even when their own group lists only one host.
All service_is_stateful flags are removed across svc-db-mariadb,
svc-db-postgres, svc-db-openldap, svc-prx-openresty, svc-docker-registry
and svc-db-mariadb's compose template; the inventory shape now expresses
the same intent. Cluster-wide handlers (swarm build/push/validate/deploy,
external network creation, DB healthcheck) gain run_once + delegate_to
the manager so they execute exactly once regardless of how many hosts
the role's play targets. CA-injection and compose validate are gated to
compose-mode only; their compose-config strictness breaks when the
deploy block coexists with container_name under swarm.
Dead per-host swarm-resolve infrastructure is removed: swarm_resolve_
container.sh, resolve_container.yml, resolve_self.yml, the MariaDB-
specific swarm_mariadb_cid.sh, the cross-host template replication
step, and the _swarm_retries / _swarm_delay defaults. With run_once +
delegate_to: manager replacing the old "render every template on every
swarm peer" pattern, every consumer becomes a noop on workers and
collapses to the manager's one-shot path.
The web-app-nextcloud memcache override mapping flips from the legacy
"single" sentinel to "compose" (semantically equivalent under the new
model). Requirement 023 is updated to reflect the new model and drops
the now-irrelevant per-role opt-out / placement-constraint clauses.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`service_is_stateful` is set unconditionally to `false` in
group_vars/all/18_swarm.yml; per-role overrides in vars/main.yml win by
Ansible precedence. The `| default(false)` fallback is no longer reachable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the host-scoped set_fact resolvers in sys-svc-compose/tasks/01_core.yml
and tasks/stages/01_constructor.yml with a single group_vars definition driven
by group_names. Uppercase the variable to match the SPOT convention and drop
every `| default('compose')` fallback now that the resolution is unconditional.
Role-var opt-out on a swarm host still works via Ansible's role-vars > group_vars
precedence.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the svc-docker-swarm, svc-docker-swarm-manager, svc-storage-nfs-server
and svc-storage-nfs-client roles, the compose_volumes filter NFS support,
the test-deploy-swarm-nfs CI workflow with its swarm-nfs-pilot script set,
and the deploy_mode-aware compose/swarm wiring across mediawiki, mariadb
and sys-svc-* helpers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
