svc-cache-registry -> svc-registry-cache
svc-docker-registry -> svc-registry-docker
svc-docker-swarm -> svc-swarm
svc-docker-swarm-manager -> svc-swarm-manager
categories.yml: drop svc.docker, add svc.registry. Deliberately no svc.swarm
category so get_entity_name('svc-swarm') stays 'swarm' instead of collapsing
to '' (which would break discover_role_services).
Entity keys updated to match the new svc.registry prefix:
services.cache-registry.* -> services.cache.*
services.registry.* -> services.docker.*
01_constructor: replace svc-docker bootstrap-loop entry with explicit
include_role of svc-swarm + bootstrap loop over [svc-registry, svc-storage].
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
inventories/development/swarm.yml carries only swarm-mode-specific
overrides and omits cross-cutting cred seeds (OIDC.CLIENT.SECRET,
biber, administrator password). Until now the swarm CI provision step
did not pass --vars-file, so default.yml was never merged into the
generated host_vars; the first deploy that preloaded web-app-mailu as
a shared email service then hit object of type 'dict' has no attribute
'SECRET' when sys-svc-compose rendered roles/web-app-mailu/templates/
env.j2. Affected jobs in CI run 27006276488: web-app-chess,
web-app-espocrm, web-app-funkwhale, web-app-prometheus.
Match the server-CI pattern (scripts/tests/deploy/local/reset/
inventory.sh and apps/initialize/all.sh both pass --vars-file
${INFINITO_INVENTORY_VARS_FILE}). 05_provision_inventory.sh now does
the same: the env loader sourced via _context.sh already exports
INFINITO_INVENTORY_VARS_FILE in the script shell, so the path
resolves through the SPOT.
write_extras.py re-reads default.yml users dict and patches the
runtime SSH key onto admin authorized_keys before emitting the
extras file. Without this, the deploy-time -e @swarm-nfs-extras.yml
would REPLACE the host_vars users entry (ansible later-wins on
top-level dicts) and wipe biber + the admin password the
--vars-file just baked in.
Two distinct breakages that let the workflow fail before ever reaching
the ansible deploy:
* `python3 -m utils.tests.swarm.*` (and friends) ran with the
actions-setup interpreter that has no project deps, so the imports
of `plugins.filter.canonical_domains_map` blew up on `from ansible.errors`.
The swarm preamble scripts now source `scripts/meta/env/load.sh` so
PATH picks up the venv `make install` populated, and the workflow itself
uses `install-force` (under `act --bind` the build/install.stamp is the
host's and would short-circuit the in-container install) plus pushes
`\${VENV}/bin` onto GITHUB_PATH for downstream steps.
* `ansible-playbook -e @file` replaces top-level dicts rather than
deep-merging them. The runtime-only extras file was dropping
`storage.nfs.{export_base,version}`, `storage.backend`, and
`swarm.{registry.port,network}` that the static
`inventories/development/swarm.yml` ships. Re-state those keys
in write_extras.py so the merge ends up with everything the roles need.
The script's three lines were redundant: \`pip install -e .\` and the
ansible-galaxy collection step are already covered by \`make install\`
(now landed in the swarm matrix preamble), and \`PYTHON=python3 bash
scripts/setup.sh\` is exactly what \`make setup\` invokes. Replace the
step with a direct \`make setup\` call.
Renumber the remaining scripts to keep the sequence lückenlos:
06_provision_inventory.sh -> 05_provision_inventory.sh
07_debug_inventory.sh -> 06_debug_inventory.sh
08_wait_converge.sh -> 07_wait_converge.sh
09_seed_content.sh -> 08_seed_content.sh
10_drain_worker.sh -> 09_drain_worker.sh
11_assert_state.sh -> 10_assert_state.sh
12_collect_diagnostics.sh -> 11_collect_diagnostics.sh
13_cleanup.sh -> 12_cleanup.sh
Workflow YAML updated to the new names. No script content changed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New ``cli.administration.deploy.swarm`` package wraps the strict
``cli.administration.deploy.dedicated`` runner with a layered closure
so swarm deploys never require operators (or test scripts) to
enumerate every infra role:
effective --id = seed (operator --id, else inventory_role_groups)
∪ dep-walk(seed via applications_if_group_and_all_deps)
∪ default-placement safety net
The dep-walker is the same SPOT
(``utils.roles.applications.in_group_deps.applications_if_group_and_all_deps``)
the runtime constructor consumes via ``lookup('applications_current_play')``,
so the CLI-side closure and the in-play role set stay in sync.
The dedicated CLI keeps strict explicit-only semantics: ``main`` was
split into ``parse_args`` + ``run_from_args`` so the swarm wrapper
can share parser, mode wiring, validator and runner verbatim and
only mutate ``args.id``.
Meta-side: ``svc-db-postgres`` gains ``default_placement: manager``
(semantic symmetry with mariadb). ``default_placement`` is now a
PLACEMENT hint only ("if deployed, run on the manager"); deploy
decisions come from the dep-walk. Apps that declare ``mariadb:``
pull svc-db-mariadb; apps that declare ``postgres:`` pull
svc-db-postgres; apps that declare neither get no central DB.
Shared utils extracted so the four consumers stay aligned:
* ``utils.inventory.groups.inventory_has_group``: YAML + INI static
inventory group probe (extracted from ValidDeployId).
* ``utils.roles.meta_lookup.{get_role_default_placement,
iter_roles_with_default_placement}``: single source of truth for
``default_placement`` discovery (lookup plugin is now a thin wrapper).
Test pipeline (``scripts/tests/deploy/swarm/*``):
* Heredoc-Python shells converted to proper modules under
``utils.tests.swarm.*`` and invoked via ``python -m``:
- extend_inventory.py (was 08_extend_inventory.sh)
- write_extras.py (was 09_write_extras.sh dynamic half)
- derive_includes.py (replaces 06's hardcoded includes array)
- write_hosts_entries.py (replaces 02's hardcoded /etc/hosts heredoc;
derives every canonical + www-redirect domain from APP_ID via
canonical_domains_map + generate_all_domains, so the test
works for any app in the matrix without per-app maintenance)
* 01_create_network.sh deleted (single docker command inlined in workflow).
* 11_deploy.sh deleted (swarm CLI now called directly in workflow).
* 09_write_extras.sh: static half moved to inventories/development/swarm.yml.
* Remaining shell scripts renumbered 01-13 without gaps.
* 08_wait_converge.sh: case block on DB_DEP replaced with single
conditional (container name == DB_DEP).
* 06_provision_inventory.sh: hardcoded includes array replaced with
`mapfile -t includes < <(python -m utils.tests.swarm.derive_includes)`.
* Workflow YAML step references updated to new script names + module
invocations.
New tests (39 covering the closure surface):
* tests/unit/cli/administration/deploy/swarm/test_closure.py (14)
* tests/unit/utils/inventory/test_groups.py (11)
* tests/unit/utils/roles/test_meta_lookup.py (+3 for default_placement)
* tests/unit/utils/roles/validation/test_deploy_id.py (unchanged
8 tests still pass after refactor onto the shared util).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drop the required_by declaration on svc-cache-registry so the
post-deploy validator no longer enforces its execution for every
svc-docker / web-app deploy. The pull-through cache is a performance
optimization, not a correctness requirement; operators that omit it
from inventory should not see a deploy fail.
Compose-mode auto-place now scopes svc-prx-openresty to plays that
actually carry a web-* role (any group whose name matches ^web-),
so svc-only deploys do not get an orphan front-proxy. The match is
intentionally on the web-* prefix family so new web-<sub>-<entity>
categories pick up the auto-place rule without an allowlist edit.
The swarm-NFS test still wants the cache for the pull-through speedup,
so svc-cache-registry is listed explicitly in 07_provision_inventory.sh
alongside the swarm bootstrap roles.
Fixes test-deploy-server / Server web-app-mediawiki failing with
"required role(s) did not execute for deploy: svc-cache-registry".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DinD DNS chain reworked so the Playwright container run --rm npm install
no longer EAI_AGAINs on registry.npmjs.org:
- 03_start_swarm_nodes.sh runs each DinD with --dns 1.1.1.1/8.8.8.8 and
pre-seeds /etc/docker/daemon.json. systemd-resolved still clobbers the
outer resolv.conf at boot, so the daemon-level DNS is the only thing
that survives for containers the inner daemon spawns.
- daemon.json.j2 omits the "dns" key when networks.internet.dns is empty
(default), so the rendered config no longer overwrites the pre-seeded
daemon.json with "dns": [""] and break inner-container resolution.
- 09_write_extras.sh sets networks.internet.dns to a public resolver for
the test rig and bakes RUNTIME via cli.meta.runtime (so e2e-playwright
actually runs in act/github instead of being skipped under "host").
07_provision_inventory.sh generates mirrors.yml via
cli.contributing.mirror.resolver and passes --mirror to the inventory
provisioner only when cli.meta.runtime == "github". act/local keeps
upstream pulls (no mirror infrastructure to point at).
12_wait_converge.sh and 15_assert_state.sh accept any N/N replica state
instead of hardcoded 1/1, so multi-host-replica deploys converge cleanly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Roles that declare default_placement: manager in meta/services.yml are now
auto-added to their own inventory group when the operator did not list them
explicitly. Both the runtime constructor add_host loop and the CLI
pre-flight inventory extender use the new roles_with_default_placement
lookup so swarm and CLI-validation stay consistent.
In compose-mode (no svc-docker-swarm-manager group), svc-prx-openresty
auto-joins the first play host. run_once + ansible_play_hosts | first
keeps that deterministic in multi-host compose deploys so the openresty
vhost view does not fragment.
The svc-cache-registry / svc-docker-registry entries previously hardcoded
in scripts/tests/deploy/swarm/08_extend_inventory.sh fall out of the new
dynamic loop; net inventory is unchanged.
ansible_play_hosts joins the magic-var allowlist in
test_variable_definitions so the constructor add_host passes integration
lint.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The install + update + admin maintenance commands all run as throwaway
container instances against the custom image, with the live service's
bind mounts attached. The previous container exec-against-static-name
pattern stops working in swarm where the container name is a
per-replica swarm task id and the task itself can be reaped by the
healthcheck mid-update.
container_name is gated under swarm because docker-compose's strict
validation rejects it once replicas > 1. Cross-host config replication
in 01_configuration.yml is removed entirely: the play now runs natively
on every host in the role's group, so the LocalSettings.php / oidc.php
/ debug.php templates render in place per host.
Test inventory adds the pilot role to all three simulated swarm hosts,
which is what flips DEPLOYMENT_MODE to swarm under the new
inventory-driven model.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New role that runs registry:2 in pull-through proxy mode on the swarm
manager. Worker nodes' daemon.json gets a registry-mirrors entry plus
the corresponding insecure-registries entry so plain-HTTP traffic to
the cache is permitted. The role's inventory group lists exactly one
host (manager) so the new inventory-driven topology model resolves
DEPLOYMENT_MODE to compose for the cache itself.
Within a single deploy run, the three swarm workers all pull the same
base images via the cache instead of hitting registry-1.docker.io
three separate times. Measured speedup on iter26 vs the no-cache iter21
baseline: pkgmgr/apt-install steps 2.4-2.8x faster, Docker image pulls
1.8-2.8x faster.
Cross-iter cache persistence is intentionally out of scope: act
recycles the DinD nodes between runs so the cache volume is wiped each
time. Cross-run reuse would require either bind-mounting the cache
from the host or pointing daemon.json at the dev compose stack's
registry-cache; both are tracked separately.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
iter11/iter12/iter14 each hung nfs-server's `apt-get update` in TCP
CLOSE_WAIT against apt-cache:3142 for 5-60 minutes, even with
Pipeline-Depth=0 + No-Keep-Alive + Retries=3 + Timeout=30. Pulling once
into the cache and fanning out to four nodes saves ~4 minutes of
bootstrap when it works, but every run that hit the hang lost 20+
minutes — net negative.
Remove the workflow cache restore step, the 02b_start_apt_cache.sh
launcher, the apt.conf.d/01-cacher write in 05_bootstrap_nodes.sh, and
the apt-cache cleanup in 17_cleanup.sh. Each node now talks directly
to the upstream mirrors. Slower-but-reliable beats faster-but-stuck.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
iter11 hung nfs-server's `apt-get update` for 16+ minutes in CLOSE_WAIT
against apt-cache:3142 even with Pipeline-Depth=0 — the cache closed
each TCP slot but apt's HTTP method held onto them via keep-alive and
never noticed the half-close.
Set `Acquire::http::No-Keep-Alive "true"` so every apt request opens a
fresh socket and tears it down on completion (no shared state to leak),
plus a 30 s `Acquire::http::Timeout` so any future stuck fetch errors
out fast enough for the Retries=3 backoff to actually kick in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
In CI the workflow runs as the unprivileged `runner` user, so
`scripts/install/package.sh`'s `apt-get update && apt-get install
build-essential debhelper dpkg-dev rsync` aborted with `E: Could not open
lock file /var/lib/apt/lists/lock - open (13: Permission denied)` before
the .deb was ever produced. Local act runs as root inside
catthehacker/ubuntu so we never hit it.
Detect non-root in 05_bootstrap_nodes.sh and prepend `sudo -E` to the
build invocation. Inside the swarm-node containers package.sh still
runs as root and never goes through sudo (those exec calls are
unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Leaving the apt-cacher-ng proxy enabled past the .deb-deps install made the
later ansible-driven apt tasks (sys-svc-ssh, update, ...) hang on TCP
CLOSE_WAIT against the cache even with Pipeline-Depth=0: the connections
the proxy closed upstream were never picked up by ansible's apt method on
the next invocation. The cache only pays for itself during the parallel
4-node .deb-deps fetch anyway — afterwards the proxy is pure friction.
Remove /etc/apt/apt.conf.d/01-cacher at the end of provision_node so
every ansible apt task hits the upstream mirrors directly, undisturbed
by the cache's keep-alive semantics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jrei/systemd-ubuntu:24.04 has been exiting with code 255 in ~0.4s on
ubuntu-24.04 GHA runners since image build 20260525.161, producing zero
stdout/stderr (the docker-logs fallback from d7e859041 captured nothing).
`--privileged` alone no longer disables AppArmor + seccomp on the host
runner image, so systemd can't take ownership of its cgroup / signal mask
and PID 1 aborts before it logs anything.
Add the three flags that, in combination, restore the same isolation
posture systemd-in-docker has historically had:
--cgroupns=host let systemd see the host cgroup tree
(otherwise the unified hierarchy is
remapped under /sys/fs/cgroup/system
and pid 1 refuses to manage it)
--security-opt seccomp=unconfined drop the default seccomp filter that
now blocks io_uring/clone3 calls
systemd-255 uses during init
--security-opt apparmor=unconfined explicit override; --privileged is
supposed to imply this but the new
runner image's docker.service-level
profile re-applies it on top
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
iter4 hung swarm-wrk-02's `apt-get update` for 5+ minutes in TCP CLOSE_WAIT
against the apt-cacher-ng container: 3 sockets to apt-cache:3142 were closed
upstream, but apt's http method kept reusing pipelined slots and never
detected EOF.
`Acquire::http::Pipeline-Depth "0"` forces apt to one request per connection,
which is the documented workaround for apt-cacher-ng (see acng manual
section 4 "Common Apt Configuration Problems"). Bundled defaults:
* `No-Cache "false"` lets validated cache hits short-circuit IMS checks
* `Retries "3"` makes transient proxy errors recoverable
Switched the inline apt.conf write from a fragile single-line `printf
> file` to a heredoc piped through `docker exec -i`, since the new content
spans multiple lines.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pilot's /tmp/swarm-nfs-extras.yml is passed to ansible via -e @file
which makes its top-level keys (including swarm:) fully *replace* the
group_vars/all/18_swarm.yml swarm dict — silently dropping swarm.registry
and breaking the daemon.json.j2 + handler chain with
"object of type 'dict' has no attribute 'registry'" once iter3 hit the
Configure docker daemon storage driver task on the workers.
09_write_extras.sh now emits swarm.registry.host=${MGR} (swarm-mgr-01,
docker-DNS-resolvable from the workers) and port=5000, matching the
group_vars defaults.
Bundled: the lint-python EXE001 violation on the new
roles/sys-svc-compose/files/swarm_registry_push.py — the shebang was
present but the file was not chmod +x.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a single apt-cacher-ng container (sameersbn/apt-cacher-ng:3.7.4-20220421)
on the swarm-lab network and routes every node's apt traffic through it via
/etc/apt/apt.conf.d/01-cacher. With four nodes provisioning the same
infinito-nexus.deb deps in parallel (~200 MB / node), the cache turns the
second-through-fourth fetch into a hot serve from the local proxy.
The workflow restores ${{ runner.temp }}/apt-cacher-ng across runs via
actions/cache@v4 (keyed on packaging/debian/control), so on warm GHA runs
the cache is also populated for the first node — driving bootstrap from
~11 min toward sub-3 min once the cache is hot. The cache restore step is
guarded by `if: \!env.ACT` so act runs still exercise the cold path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`docker exec journalctl` fails when the container is already dead (which
is the actual failure mode: jrei/systemd-ubuntu:24.04 exits with code 255
~0.4s after start on the current GHA runner image). `docker logs` works
on stopped containers and surfaces what systemd printed before it gave up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a manager-pinned, NFS-backed `registry:2` service plus a transparent
retag/push/rewrite step so swarm workers can pull custom-built images from a
local registry, replacing the `docker save | docker load` pilot crutch.
Mechanism: after `compose build`, the new `swarm_registry_push.py` reads the
rendered compose.yml, retags each service that declares both `build:` and
`image:` with `<manager>:5000/<image>`, pushes the retag, and rewrites the
`image:` field in place. Subsequent `docker stack deploy --resolve-image
never` then references the registry-qualified name, so workers pull from the
cluster-local registry over HTTP via `insecure-registries`.
No per-role changes required — the rewrite is centralized in
sys-svc-compose handlers and skips services without a `build:` directive.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous 60s window flaked on GHA matrix runs because jrei/systemd-ubuntu:24.04 cold-starts D-Bus + targets slowly under parallel load. 03_start_swarm_nodes.sh additionally let downstream steps run against a non-ready node instead of failing.
Both scripts now exit non-zero with journalctl + State diagnostics on timeout. Override via SWARM_PILOT_SYSTEMD_TIMEOUT for local debug.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror the test-deploy-server iteration scaffold: discover + matrix on
the same app list (INFINITO_DEPLOY_TYPE=server), one swarm-deploy run
per app. Pilot scripts derive per-app context from APP_ID + role meta
(ENTITY, SERVICE_NAME, DB_DEP, CUSTOM_IMAGE_REPO, PRIMARY_NFS_VOLUME)
so they adapt to roles other than mediawiki.
- .github/workflows/test-deploy-swarm-nfs.yml -> test-deploy-swarm.yml
- scripts/tests/deploy/swarm-nfs-pilot/ -> scripts/tests/deploy/swarm/
- new _context.sh helper sourced by 07/08/11–16
- 07_provision_inventory + 08_extend_inventory + 11_deploy build their
--include list from DB_DEP + APP_ID
- 12_wait_converge skips DB readiness when DB_DEP=none
- 13_seed_content seeds NFS marker only when meta/volumes.yml carries an
nfs-flagged volume
- 14_drain_worker skips custom-image distribution when role has no
build context
- 15_assert_state asserts marker survival only when an NFS volume exists
- ci-orchestrator wires the renamed job + passes whitelist through
- naming-pattern alignment across test-deploy-{server,universal,workstation,swarm}
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
