mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
kevinveenbirkenbachandClaude Opus 5
d80cad478a
Under swarm an unpinned role's volumes are rewritten onto the shared NFS
export while its services run one task per node. A role that declares no
RDBMS keeps its state in-process, so several replicas then write the same
SQLite file, Mnesia store or search index over NFS and corrupt it. This
already cost a deploy: openclaw ran three replicas against one shared
export and its state directory answered "file is not a database".
The existing unpinned-no-engine-volume lint states the right rule but
detects state through a hardcoded path list, so app-specific paths slip
through and each case gets pinned reactively after it fails. Add a lint
that joins three facts the roles already declare: the volume is NFS
backed (asked of the production swarm_nfs_backed, not re-derived), it is
mounted read-write, and the role runs no RDBMS. Share-safe payloads carry
a justification marker.
Replicas resolve service-first and fall back to the primary entity, and
a role whose swarm mode is disabled is skipped; both cases exist in tree.
Also correct the requirement and the design doc, which still described
the per-volume opt-in model that b09598dc8 replaced with placement
derived backing. Requirement 023 contradicted itself: line 76 already
described the implemented behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
145 lines
6.8 KiB
Markdown
145 lines
6.8 KiB
Markdown
# Deployment Modes 🚢
|
||
|
||
Infinito.Nexus supports more than one backend for rendering and running an
|
||
application role's containers. This page documents the abstraction that
|
||
keeps per-role declarations identical across backends so adding a new
|
||
backend is additive, not a rewrite.
|
||
|
||
## Trajectory
|
||
|
||
```
|
||
compose → swarm → kubernetes
|
||
```
|
||
|
||
- **compose** is the default. Single host, local docker volumes, no
|
||
cluster awareness. Every role works in this mode unchanged.
|
||
- **swarm** (v1) is opt-in per host via Ansible group membership.
|
||
Adds: multi-host cluster, overlay networks, NFS-backed shared
|
||
volumes, rolling updates, placement constraints.
|
||
- **kubernetes** (future) is reserved as a third render backend
|
||
consuming the same per-role inputs.
|
||
|
||
Adding `kubernetes` MUST be additive — a new render path that consumes
|
||
the same per-role declarations (`replicas`, `update_config`, `placement`,
|
||
volumes, networks). It MUST NOT require touching every role.
|
||
|
||
## Mode selection trigger
|
||
|
||
The trigger is **group membership**, not a per-role flag.
|
||
|
||
| Host's `group_names` contains | Resolved `DEPLOYMENT_MODE` |
|
||
| ---------------------------------- | -------------------------- |
|
||
| `svc-swarm-node` | `swarm` |
|
||
| anything else | `compose` |
|
||
|
||
`DEPLOYMENT_MODE` is defined at a single point of truth in
|
||
[group_vars/all/18_swarm.yml](../../../group_vars/all/18_swarm.yml) as
|
||
`{{ 'swarm' if 'svc-swarm-node' in group_names else 'compose' }}`.
|
||
|
||
Refinements:
|
||
|
||
- **Per-role opt-OUT** on a swarm host: a role MAY override
|
||
`DEPLOYMENT_MODE: compose` in its `vars/main.yml` to stay on the
|
||
compose path even when the host is in `svc-swarm-node`. Role vars
|
||
win over `group_vars/all` by Ansible precedence.
|
||
- **Per-role opt-IN** on a non-swarm host: explicitly rejected. Swarm
|
||
mode is exclusively triggered by group membership. The per-role
|
||
`DEPLOYMENT_MODE` override only opts OUT on a swarm host, never opts
|
||
IN on a non-swarm host.
|
||
|
||
## Manager identification
|
||
|
||
Among the hosts in `svc-swarm-node`, the cluster manager is identified
|
||
by additional membership in `svc-swarm-manager`. v1 scope:
|
||
exactly ONE host. Workers are derived as
|
||
`svc-swarm-node − svc-swarm-manager` — no duplicated list in
|
||
`group_vars`.
|
||
|
||
A deploy with zero or more than one manager fails at
|
||
inventory-validation time, never at runtime.
|
||
|
||
## Reverse proxy under swarm
|
||
|
||
[svc-prx-openresty](../../../roles/svc-prx-openresty/) remains the
|
||
reverse proxy in BOTH modes. Under swarm it is rendered as a Swarm
|
||
service pinned to the manager node via
|
||
`deploy.placement.constraints: node.role == manager`.
|
||
|
||
This decision is **load-bearing**: openresty owns the
|
||
`sys-front-inj-*` Lua body-rewriting pipeline (CSP-meta stripping,
|
||
Matomo / CSS / JavaScript / Dashboard / Logout snippet injection).
|
||
Migrating to Traefik in v1 would silently drop that pipeline — it has
|
||
no equivalent in Traefik without a non-trivial Yaegi-plugin port.
|
||
Traefik migration is tracked as a Future Extension once edge-HA
|
||
becomes worth the porting effort.
|
||
|
||
Upstream resolution under swarm goes through the overlay network via
|
||
the `tasks.<service>` DNS form, returning every healthy task IP. The
|
||
per-app `upstream` block stays under openresty's control, so
|
||
HTTP-aware load balancing (`proxy_next_upstream`, future weighting,
|
||
future sticky sessions) is uniform across both modes.
|
||
|
||
## Volume model
|
||
|
||
Storage backing is derived from the role's placement, not declared per
|
||
volume. An unpinned role's volumes are rendered onto the shared export; a
|
||
`placement: manager` role keeps them node-local.
|
||
|
||
| Mode | `storage.backend` | Role placement | Rendered as |
|
||
| ------- | ----------------- | -------------- | ---------------------------------------- |
|
||
| compose | any | any | local docker volume (today's shape) |
|
||
| swarm | `nfs` | unpinned | NFS driver block (multi-node accessible) |
|
||
| swarm | `nfs` | `manager` | local docker volume, node-local |
|
||
| swarm | `local` | any | local docker volume, node-local |
|
||
|
||
A volume whose payload cannot tolerate NFS semantics opts out with
|
||
`nfs: false` plus a justification, which keeps engine state local without
|
||
pinning the whole role. The deploy MUST NOT silently lose data on
|
||
reschedule.
|
||
|
||
State that several replicas would write to the same file needs more than
|
||
node-local storage: pair `nfs: false` with a single replica or a manager
|
||
pin, or externalise the state into a database. The
|
||
`embedded-state-on-nfs` lint enforces this for roles that run no RDBMS of
|
||
their own.
|
||
|
||
## Network model
|
||
|
||
Under swarm, every per-role `driver: bridge` network is rendered as
|
||
`driver: overlay` with `attachable: true`. Cross-node service-to-service
|
||
traffic depends on overlay networks — bridge networks are node-local and
|
||
would break the mesh.
|
||
|
||
Overlay-network encryption (`driver_opts: { encrypted: "true" }`) is
|
||
ON by default for east-west traffic confidentiality. Disable via
|
||
`swarm.network.encryption: false` only for performance-sensitive
|
||
deployments with alternative segmentation.
|
||
|
||
## Stack granularity
|
||
|
||
One `docker stack` per role, consistent with today's
|
||
one-compose-project-per-role model. Granular `deploy / update / rm`
|
||
per role is preserved. A single global stack across all Infinito
|
||
services is explicitly rejected — it would couple update windows
|
||
and rollback granularity across unrelated apps.
|
||
|
||
## v1 scope summary
|
||
|
||
| Capability | v1 | Future |
|
||
| --------------------- | --------------------------------------------- | ----------------------------------------- |
|
||
| Cluster managers | Single | Raft-quorum HA (3+) |
|
||
| Edge proxy | openresty pinned to manager (single replica) | Traefik / multi-replica edge HA |
|
||
| CI runner | GitHub-hosted DinD | Self-hosted via `svc-runner` (014) |
|
||
| NFS for databases | Out of scope (DB stays local + pinned) | Per-engine NFS-tuning requirement |
|
||
| Secrets | env-file (existing) | `docker secret` (Raft-encrypted) |
|
||
| Volume migration | Greenfield (manual rsync documented) | Automated migration |
|
||
| LB strategy | Round-robin via `tasks.<service>` upstream | Weighted / sticky / active healthchecks |
|
||
| Kubernetes backend | Out of scope | Third render backend (additive) |
|
||
|
||
## See also
|
||
|
||
- [svc-swarm-node](../../../roles/svc-swarm-node/)
|
||
- [svc-storage-nfs-server](../../../roles/svc-storage-nfs-server/)
|
||
- [svc-storage-nfs-client](../../../roles/svc-storage-nfs-client/)
|
||
- [svc-prx-openresty](../../../roles/svc-prx-openresty/)
|