Resolves CI swarm rc=69 ("msmtp: needs a password"): controller-side
Ansible lookups (lookup('email').password etc.) couldn't see the
tokens.yml that Mailu writes target-side, because the controller has
its own filesystem in swarm mode.
Mount NFS-shared /var/lib/infinito on every swarm host AND on the
controller during deploy (in-play delegate_to: localhost task at end
of svc-storage-nfs-server role). State subdir exported separately
with no_root_squash + wildcard CIDRs so root-owned writes succeed
and the controller can mount from outside the svc-swarm IP set.
Single-writer guarantee on FILE_TOKENS via delegate_to: STACK_HOST +
run_once on sys-token-store/{01_core,write}.yml. Backup roles
(svc-bkp-container-2-local, sys-ctl-cln-bkps, sys-ctl-cln-faild-bkps,
user-backup/03_permissions_folders, three direct callers of
04_seed-database-to-backup.yml) gated on IS_STACK_HOST to prevent
N-host write races on the shared NFS backup tree.
Fail-fast assertion in svc-storage-nfs-client + 05_controller_mount
when pre-existing local /var/lib/infinito would be hidden by mount-
over. Operator must migrate or wipe before re-deploying.
SPOTs: SWARM_LAB_NETWORK in workflow env block, storage.nfs.state_subdir
in group_vars/all/15_storage.yml, DIR_VAR_LIB unchanged. UPPER_CASE
constants in NFS roles. Cleanup script umounts controller view before
container teardown to prevent stale-mount leak between matrix jobs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
12_collect_diagnostics.sh starts with `set +e` before sourcing
_context.sh, which immediately re-enables `set -euo pipefail` for the
remainder of the script. The first `docker exec ... mount | grep -i nfs`
that finds no nfs mount returned rc=1, pipefail propagated it, and
errexit aborted the script halfway through. GitHub Actions reported a
red tab on the diagnostic step next to the actual failing deploy step.
Move `set +e`/`set +o pipefail`/`set +u` after the source so the
post-failure helper continues through every probe even when individual
commands return non-zero, and the trailing `exit 0` still pins the
script's return code to success.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Loading default.yml via `-e @"${INFINITO_INVENTORY_VARS_FILE}"` (commit
a3c75bde7) made the dev vars available on workers but at extra-vars
precedence (level 22), which overrode the `set_fact` merge in
tasks/stages/01_constructor.yml that reconciles defaults_oidc with the
inventory-supplied OIDC. Workers and mgr-01 ended up with OIDC =
{CLIENT: {SECRET: ...}} only, so the templates that resolved
OIDC.CLIENT.LOGOUT_URL (sys-front-inj-logout etc.) aborted with "dict
has no attribute 'LOGOUT_URL'".
Mirror the provisioned host_vars/${MGR}.yml to host_vars/${WRK1}.yml and
host_vars/${WRK2}.yml at the end of 05_provision_inventory.sh instead.
Same default.yml source via the existing INFINITO_INVENTORY_VARS_FILE
SPOT, same content on every swarm host, but at host_vars precedence
(level 9) so the role-level merge wins on access. Removes the
now-redundant `-e` line from the workflow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
12_collect_diagnostics.sh now ends with an explicit `exit 0` so the
diagnostic step never propagates a non-zero rc to GitHub Actions.
zip_swarm_for_upload.sh switches from `set -euo pipefail` to `set +e`
plus a trailing `exit 0` for the same reason. Keeps the workflow's red
tab on the actual failing step instead of doubling up on the post-mortem
collectors.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add `# nocheck: hardcoded-dns-resolver` on the two `server=` lines
inside the dnsmasq heredoc in 02_start_swarm_nodes.sh so the
test_no_hardcoded_dns_resolvers lint guard recognises the literals as
intentional (dnsmasq parses `#` as a comment, so the marker is
inert at runtime).
Re-stage three pure-formatting tightenings ansible-lint applied to
existing `template: # nocheck` lines (double space -> single space)
in sys-daemon, sys-svc-mail-msmtp and web-app-moodle.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The MODE_DEBUG-gated DinD DNS handler in sys-svc-container spawned a
busybox container with `--dns 1.1.1.1` and expected the mocked
infinito.example to resolve to 127.0.0.1, which Cloudflare obviously
cannot do. Run an in-cluster dnsmasq inside mgr-01 instead of patching
the symptom out:
- Stop and mask systemd-resolved so port 53 is free.
- Bootstrap /etc/resolv.conf with 1.1.1.1 so apt-get can fetch dnsmasq.
- Install dnsmasq with bind-dynamic, no-resolv, address=/${INFINITO_DOMAIN}/127.0.0.1
and 1.1.1.1/8.8.8.8 as upstream forwarders; bind-dynamic catches docker0
when the Ansible-installed dockerd creates it later.
- Switch /etc/resolv.conf to 127.0.0.1 and verify the chain via
`getent ahosts ${INFINITO_DOMAIN}` so the script fails fast on misconfig.
- Pin mgr-01's docker0 to 172.17.0.1/16 via daemon.json `bip`, then point
the swarm.yml networks.internet.dns at that gateway so DinD-spawned
containers reach dnsmasq via their default route.
Also re-stage three pure-formatting tightenings ansible-lint made to
existing `template: # nocheck` lines (double space -> single space).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous commit's default.yml load turned MODE_DEBUG (and therefore
MODE_ASSERT) on for the swarm CI, which activates the DNS check in
tasks/stages/01_constructor.yml:12 that resolves
test.{{ DOMAIN_PRIMARY }}. The pipeline failed on all four hosts
(mgr-01, wrk-01, wrk-02, nfs-server) because:
1. write_hosts_entries did not emit test.<domain> at all.
2. 02_start_swarm_nodes wrote /etc/hosts only on the manager.
Add test.<domain> to the emitted set and write the resulting block to
mgr + both workers + nfs-server so the assertion succeeds everywhere
the play targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add 08_verify_reachable.sh between wait_converge (07) and seed_content
(09, renumbered from 08): TCP-connect via the manager's local
routing-mesh entry confirms incoming traffic reaches a live task. For
multi-replica services, additionally probe each node that hosts a
Running task to verify the mesh works locally on every task-bearing
node.
Skips services without published ports (tag-only roles, internal-only
services). Logs replica count and port list for debugging.
Renumber 08-12 -> 09-13 to keep the sequence contiguous.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
svc-cache-registry -> svc-registry-cache
svc-docker-registry -> svc-registry-docker
svc-docker-swarm -> svc-swarm
svc-docker-swarm-manager -> svc-swarm-manager
categories.yml: drop svc.docker, add svc.registry. Deliberately no svc.swarm
category so get_entity_name('svc-swarm') stays 'swarm' instead of collapsing
to '' (which would break discover_role_services).
Entity keys updated to match the new svc.registry prefix:
services.cache-registry.* -> services.cache.*
services.registry.* -> services.docker.*
01_constructor: replace svc-docker bootstrap-loop entry with explicit
include_role of svc-swarm + bootstrap loop over [svc-registry, svc-storage].
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
inventories/development/swarm.yml carries only swarm-mode-specific
overrides and omits cross-cutting cred seeds (OIDC.CLIENT.SECRET,
biber, administrator password). Until now the swarm CI provision step
did not pass --vars-file, so default.yml was never merged into the
generated host_vars; the first deploy that preloaded web-app-mailu as
a shared email service then hit object of type 'dict' has no attribute
'SECRET' when sys-svc-compose rendered roles/web-app-mailu/templates/
env.j2. Affected jobs in CI run 27006276488: web-app-chess,
web-app-espocrm, web-app-funkwhale, web-app-prometheus.
Match the server-CI pattern (scripts/tests/deploy/local/reset/
inventory.sh and apps/initialize/all.sh both pass --vars-file
${INFINITO_INVENTORY_VARS_FILE}). 05_provision_inventory.sh now does
the same: the env loader sourced via _context.sh already exports
INFINITO_INVENTORY_VARS_FILE in the script shell, so the path
resolves through the SPOT.
write_extras.py re-reads default.yml users dict and patches the
runtime SSH key onto admin authorized_keys before emitting the
extras file. Without this, the deploy-time -e @swarm-nfs-extras.yml
would REPLACE the host_vars users entry (ansible later-wins on
top-level dicts) and wipe biber + the admin password the
--vars-file just baked in.
Two distinct breakages that let the workflow fail before ever reaching
the ansible deploy:
* `python3 -m utils.tests.swarm.*` (and friends) ran with the
actions-setup interpreter that has no project deps, so the imports
of `plugins.filter.canonical_domains_map` blew up on `from ansible.errors`.
The swarm preamble scripts now source `scripts/meta/env/load.sh` so
PATH picks up the venv `make install` populated, and the workflow itself
uses `install-force` (under `act --bind` the build/install.stamp is the
host's and would short-circuit the in-container install) plus pushes
`\${VENV}/bin` onto GITHUB_PATH for downstream steps.
* `ansible-playbook -e @file` replaces top-level dicts rather than
deep-merging them. The runtime-only extras file was dropping
`storage.nfs.{export_base,version}`, `storage.backend`, and
`swarm.{registry.port,network}` that the static
`inventories/development/swarm.yml` ships. Re-state those keys
in write_extras.py so the merge ends up with everything the roles need.
The script's three lines were redundant: \`pip install -e .\` and the
ansible-galaxy collection step are already covered by \`make install\`
(now landed in the swarm matrix preamble), and \`PYTHON=python3 bash
scripts/setup.sh\` is exactly what \`make setup\` invokes. Replace the
step with a direct \`make setup\` call.
Renumber the remaining scripts to keep the sequence lückenlos:
06_provision_inventory.sh -> 05_provision_inventory.sh
07_debug_inventory.sh -> 06_debug_inventory.sh
08_wait_converge.sh -> 07_wait_converge.sh
09_seed_content.sh -> 08_seed_content.sh
10_drain_worker.sh -> 09_drain_worker.sh
11_assert_state.sh -> 10_assert_state.sh
12_collect_diagnostics.sh -> 11_collect_diagnostics.sh
13_cleanup.sh -> 12_cleanup.sh
Workflow YAML updated to the new names. No script content changed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New ``cli.administration.deploy.swarm`` package wraps the strict
``cli.administration.deploy.dedicated`` runner with a layered closure
so swarm deploys never require operators (or test scripts) to
enumerate every infra role:
effective --id = seed (operator --id, else inventory_role_groups)
∪ dep-walk(seed via applications_if_group_and_all_deps)
∪ default-placement safety net
The dep-walker is the same SPOT
(``utils.roles.applications.in_group_deps.applications_if_group_and_all_deps``)
the runtime constructor consumes via ``lookup('applications_current_play')``,
so the CLI-side closure and the in-play role set stay in sync.
The dedicated CLI keeps strict explicit-only semantics: ``main`` was
split into ``parse_args`` + ``run_from_args`` so the swarm wrapper
can share parser, mode wiring, validator and runner verbatim and
only mutate ``args.id``.
Meta-side: ``svc-db-postgres`` gains ``default_placement: manager``
(semantic symmetry with mariadb). ``default_placement`` is now a
PLACEMENT hint only ("if deployed, run on the manager"); deploy
decisions come from the dep-walk. Apps that declare ``mariadb:``
pull svc-db-mariadb; apps that declare ``postgres:`` pull
svc-db-postgres; apps that declare neither get no central DB.
Shared utils extracted so the four consumers stay aligned:
* ``utils.inventory.groups.inventory_has_group``: YAML + INI static
inventory group probe (extracted from ValidDeployId).
* ``utils.roles.meta_lookup.{get_role_default_placement,
iter_roles_with_default_placement}``: single source of truth for
``default_placement`` discovery (lookup plugin is now a thin wrapper).
Test pipeline (``scripts/tests/deploy/swarm/*``):
* Heredoc-Python shells converted to proper modules under
``utils.tests.swarm.*`` and invoked via ``python -m``:
- extend_inventory.py (was 08_extend_inventory.sh)
- write_extras.py (was 09_write_extras.sh dynamic half)
- derive_includes.py (replaces 06's hardcoded includes array)
- write_hosts_entries.py (replaces 02's hardcoded /etc/hosts heredoc;
derives every canonical + www-redirect domain from APP_ID via
canonical_domains_map + generate_all_domains, so the test
works for any app in the matrix without per-app maintenance)
* 01_create_network.sh deleted (single docker command inlined in workflow).
* 11_deploy.sh deleted (swarm CLI now called directly in workflow).
* 09_write_extras.sh: static half moved to inventories/development/swarm.yml.
* Remaining shell scripts renumbered 01-13 without gaps.
* 08_wait_converge.sh: case block on DB_DEP replaced with single
conditional (container name == DB_DEP).
* 06_provision_inventory.sh: hardcoded includes array replaced with
`mapfile -t includes < <(python -m utils.tests.swarm.derive_includes)`.
* Workflow YAML step references updated to new script names + module
invocations.
New tests (39 covering the closure surface):
* tests/unit/cli/administration/deploy/swarm/test_closure.py (14)
* tests/unit/utils/inventory/test_groups.py (11)
* tests/unit/utils/roles/test_meta_lookup.py (+3 for default_placement)
* tests/unit/utils/roles/validation/test_deploy_id.py (unchanged
8 tests still pass after refactor onto the shared util).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drop the required_by declaration on svc-cache-registry so the
post-deploy validator no longer enforces its execution for every
svc-docker / web-app deploy. The pull-through cache is a performance
optimization, not a correctness requirement; operators that omit it
from inventory should not see a deploy fail.
Compose-mode auto-place now scopes svc-prx-openresty to plays that
actually carry a web-* role (any group whose name matches ^web-),
so svc-only deploys do not get an orphan front-proxy. The match is
intentionally on the web-* prefix family so new web-<sub>-<entity>
categories pick up the auto-place rule without an allowlist edit.
The swarm-NFS test still wants the cache for the pull-through speedup,
so svc-cache-registry is listed explicitly in 07_provision_inventory.sh
alongside the swarm bootstrap roles.
Fixes test-deploy-server / Server web-app-mediawiki failing with
"required role(s) did not execute for deploy: svc-cache-registry".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add tests/lint/ansible/jinja/test_no_ansible_facts_in_templates.py: under
swarm-mode the rendered template lands once on the primary manager and is
slurp-distributed to every other node, so any per-host fact reference
(ansible_hostname, ansible_default_ipv4, ansible_fqdn, ansible_nodename,
ansible_machine_id, ansible_eth0, ansible_all_ipv*_addresses) would
silently produce wrong output on the other nodes. The lint blocks them
at PR time.
scripts/tests/deploy/act/workflow.sh forwards --action-offline-mode +
--action-cache-path so repeated local act runs do not refetch the actions
catalog.
roles/sys-ctl-cln-anon-volumes/tasks/01_core.yml quoting cleanup applied
by autoformat (single to double quotes, no semantic change).
.claude/settings.json adds Workflow to permissions.allow so the
multi-agent Workflow tool no longer prompts per invocation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DinD DNS chain reworked so the Playwright container run --rm npm install
no longer EAI_AGAINs on registry.npmjs.org:
- 03_start_swarm_nodes.sh runs each DinD with --dns 1.1.1.1/8.8.8.8 and
pre-seeds /etc/docker/daemon.json. systemd-resolved still clobbers the
outer resolv.conf at boot, so the daemon-level DNS is the only thing
that survives for containers the inner daemon spawns.
- daemon.json.j2 omits the "dns" key when networks.internet.dns is empty
(default), so the rendered config no longer overwrites the pre-seeded
daemon.json with "dns": [""] and break inner-container resolution.
- 09_write_extras.sh sets networks.internet.dns to a public resolver for
the test rig and bakes RUNTIME via cli.meta.runtime (so e2e-playwright
actually runs in act/github instead of being skipped under "host").
07_provision_inventory.sh generates mirrors.yml via
cli.contributing.mirror.resolver and passes --mirror to the inventory
provisioner only when cli.meta.runtime == "github". act/local keeps
upstream pulls (no mirror infrastructure to point at).
12_wait_converge.sh and 15_assert_state.sh accept any N/N replica state
instead of hardcoded 1/1, so multi-host-replica deploys converge cleanly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Roles that declare default_placement: manager in meta/services.yml are now
auto-added to their own inventory group when the operator did not list them
explicitly. Both the runtime constructor add_host loop and the CLI
pre-flight inventory extender use the new roles_with_default_placement
lookup so swarm and CLI-validation stay consistent.
In compose-mode (no svc-docker-swarm-manager group), svc-prx-openresty
auto-joins the first play host. run_once + ansible_play_hosts | first
keeps that deterministic in multi-host compose deploys so the openresty
vhost view does not fragment.
The svc-cache-registry / svc-docker-registry entries previously hardcoded
in scripts/tests/deploy/swarm/08_extend_inventory.sh fall out of the new
dynamic loop; net inventory is unchanged.
ansible_play_hosts joins the magic-var allowlist in
test_variable_definitions so the constructor add_host passes integration
lint.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The install + update + admin maintenance commands all run as throwaway
container instances against the custom image, with the live service's
bind mounts attached. The previous container exec-against-static-name
pattern stops working in swarm where the container name is a
per-replica swarm task id and the task itself can be reaped by the
healthcheck mid-update.
container_name is gated under swarm because docker-compose's strict
validation rejects it once replicas > 1. Cross-host config replication
in 01_configuration.yml is removed entirely: the play now runs natively
on every host in the role's group, so the LocalSettings.php / oidc.php
/ debug.php templates render in place per host.
Test inventory adds the pilot role to all three simulated swarm hosts,
which is what flips DEPLOYMENT_MODE to swarm under the new
inventory-driven model.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New role that runs registry:2 in pull-through proxy mode on the swarm
manager. Worker nodes' daemon.json gets a registry-mirrors entry plus
the corresponding insecure-registries entry so plain-HTTP traffic to
the cache is permitted. The role's inventory group lists exactly one
host (manager) so the new inventory-driven topology model resolves
DEPLOYMENT_MODE to compose for the cache itself.
Within a single deploy run, the three swarm workers all pull the same
base images via the cache instead of hitting registry-1.docker.io
three separate times. Measured speedup on iter26 vs the no-cache iter21
baseline: pkgmgr/apt-install steps 2.4-2.8x faster, Docker image pulls
1.8-2.8x faster.
Cross-iter cache persistence is intentionally out of scope: act
recycles the DinD nodes between runs so the cache volume is wiped each
time. Cross-run reuse would require either bind-mounting the cache
from the host or pointing daemon.json at the dev compose stack's
registry-cache; both are tracked separately.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
iter11/iter12/iter14 each hung nfs-server's `apt-get update` in TCP
CLOSE_WAIT against apt-cache:3142 for 5-60 minutes, even with
Pipeline-Depth=0 + No-Keep-Alive + Retries=3 + Timeout=30. Pulling once
into the cache and fanning out to four nodes saves ~4 minutes of
bootstrap when it works, but every run that hit the hang lost 20+
minutes — net negative.
Remove the workflow cache restore step, the 02b_start_apt_cache.sh
launcher, the apt.conf.d/01-cacher write in 05_bootstrap_nodes.sh, and
the apt-cache cleanup in 17_cleanup.sh. Each node now talks directly
to the upstream mirrors. Slower-but-reliable beats faster-but-stuck.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
iter11 hung nfs-server's `apt-get update` for 16+ minutes in CLOSE_WAIT
against apt-cache:3142 even with Pipeline-Depth=0 — the cache closed
each TCP slot but apt's HTTP method held onto them via keep-alive and
never noticed the half-close.
Set `Acquire::http::No-Keep-Alive "true"` so every apt request opens a
fresh socket and tears it down on completion (no shared state to leak),
plus a 30 s `Acquire::http::Timeout` so any future stuck fetch errors
out fast enough for the Retries=3 backoff to actually kick in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
In CI the workflow runs as the unprivileged `runner` user, so
`scripts/install/package.sh`'s `apt-get update && apt-get install
build-essential debhelper dpkg-dev rsync` aborted with `E: Could not open
lock file /var/lib/apt/lists/lock - open (13: Permission denied)` before
the .deb was ever produced. Local act runs as root inside
catthehacker/ubuntu so we never hit it.
Detect non-root in 05_bootstrap_nodes.sh and prepend `sudo -E` to the
build invocation. Inside the swarm-node containers package.sh still
runs as root and never goes through sudo (those exec calls are
unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Leaving the apt-cacher-ng proxy enabled past the .deb-deps install made the
later ansible-driven apt tasks (sys-svc-ssh, update, ...) hang on TCP
CLOSE_WAIT against the cache even with Pipeline-Depth=0: the connections
the proxy closed upstream were never picked up by ansible's apt method on
the next invocation. The cache only pays for itself during the parallel
4-node .deb-deps fetch anyway — afterwards the proxy is pure friction.
Remove /etc/apt/apt.conf.d/01-cacher at the end of provision_node so
every ansible apt task hits the upstream mirrors directly, undisturbed
by the cache's keep-alive semantics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jrei/systemd-ubuntu:24.04 has been exiting with code 255 in ~0.4s on
ubuntu-24.04 GHA runners since image build 20260525.161, producing zero
stdout/stderr (the docker-logs fallback from d7e859041 captured nothing).
`--privileged` alone no longer disables AppArmor + seccomp on the host
runner image, so systemd can't take ownership of its cgroup / signal mask
and PID 1 aborts before it logs anything.
Add the three flags that, in combination, restore the same isolation
posture systemd-in-docker has historically had:
--cgroupns=host let systemd see the host cgroup tree
(otherwise the unified hierarchy is
remapped under /sys/fs/cgroup/system
and pid 1 refuses to manage it)
--security-opt seccomp=unconfined drop the default seccomp filter that
now blocks io_uring/clone3 calls
systemd-255 uses during init
--security-opt apparmor=unconfined explicit override; --privileged is
supposed to imply this but the new
runner image's docker.service-level
profile re-applies it on top
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
iter4 hung swarm-wrk-02's `apt-get update` for 5+ minutes in TCP CLOSE_WAIT
against the apt-cacher-ng container: 3 sockets to apt-cache:3142 were closed
upstream, but apt's http method kept reusing pipelined slots and never
detected EOF.
`Acquire::http::Pipeline-Depth "0"` forces apt to one request per connection,
which is the documented workaround for apt-cacher-ng (see acng manual
section 4 "Common Apt Configuration Problems"). Bundled defaults:
* `No-Cache "false"` lets validated cache hits short-circuit IMS checks
* `Retries "3"` makes transient proxy errors recoverable
Switched the inline apt.conf write from a fragile single-line `printf
> file` to a heredoc piped through `docker exec -i`, since the new content
spans multiple lines.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pilot's /tmp/swarm-nfs-extras.yml is passed to ansible via -e @file
which makes its top-level keys (including swarm:) fully *replace* the
group_vars/all/18_swarm.yml swarm dict — silently dropping swarm.registry
and breaking the daemon.json.j2 + handler chain with
"object of type 'dict' has no attribute 'registry'" once iter3 hit the
Configure docker daemon storage driver task on the workers.
09_write_extras.sh now emits swarm.registry.host=${MGR} (swarm-mgr-01,
docker-DNS-resolvable from the workers) and port=5000, matching the
group_vars defaults.
Bundled: the lint-python EXE001 violation on the new
roles/sys-svc-compose/files/swarm_registry_push.py — the shebang was
present but the file was not chmod +x.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a single apt-cacher-ng container (sameersbn/apt-cacher-ng:3.7.4-20220421)
on the swarm-lab network and routes every node's apt traffic through it via
/etc/apt/apt.conf.d/01-cacher. With four nodes provisioning the same
infinito-nexus.deb deps in parallel (~200 MB / node), the cache turns the
second-through-fourth fetch into a hot serve from the local proxy.
The workflow restores ${{ runner.temp }}/apt-cacher-ng across runs via
actions/cache@v4 (keyed on packaging/debian/control), so on warm GHA runs
the cache is also populated for the first node — driving bootstrap from
~11 min toward sub-3 min once the cache is hot. The cache restore step is
guarded by `if: \!env.ACT` so act runs still exercise the cold path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ACT_INPUTS="key=value [key2=value2]" expands to repeated --input flags so
make act-workflow can drive workflow_dispatch-style inputs without falling
back to a manual act invocation. Needed for the swarm pilot, where the
discover job's `whitelist` input gates the per-app matrix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`docker exec journalctl` fails when the container is already dead (which
is the actual failure mode: jrei/systemd-ubuntu:24.04 exits with code 255
~0.4s after start on the current GHA runner image). `docker logs` works
on stopped containers and surfaces what systemd printed before it gave up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a manager-pinned, NFS-backed `registry:2` service plus a transparent
retag/push/rewrite step so swarm workers can pull custom-built images from a
local registry, replacing the `docker save | docker load` pilot crutch.
Mechanism: after `compose build`, the new `swarm_registry_push.py` reads the
rendered compose.yml, retags each service that declares both `build:` and
`image:` with `<manager>:5000/<image>`, pushes the retag, and rewrites the
`image:` field in place. Subsequent `docker stack deploy --resolve-image
never` then references the registry-qualified name, so workers pull from the
cluster-local registry over HTTP via `insecure-registries`.
No per-role changes required — the rewrite is centralized in
sys-svc-compose handlers and skips services without a `build:` directive.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous 60s window flaked on GHA matrix runs because jrei/systemd-ubuntu:24.04 cold-starts D-Bus + targets slowly under parallel load. 03_start_swarm_nodes.sh additionally let downstream steps run against a non-ready node instead of failing.
Both scripts now exit non-zero with journalctl + State diagnostics on timeout. Override via SWARM_PILOT_SYSTEMD_TIMEOUT for local debug.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror the test-deploy-server iteration scaffold: discover + matrix on
the same app list (INFINITO_DEPLOY_TYPE=server), one swarm-deploy run
per app. Pilot scripts derive per-app context from APP_ID + role meta
(ENTITY, SERVICE_NAME, DB_DEP, CUSTOM_IMAGE_REPO, PRIMARY_NFS_VOLUME)
so they adapt to roles other than mediawiki.
- .github/workflows/test-deploy-swarm-nfs.yml -> test-deploy-swarm.yml
- scripts/tests/deploy/swarm-nfs-pilot/ -> scripts/tests/deploy/swarm/
- new _context.sh helper sourced by 07/08/11–16
- 07_provision_inventory + 08_extend_inventory + 11_deploy build their
--include list from DB_DEP + APP_ID
- 12_wait_converge skips DB readiness when DB_DEP=none
- 13_seed_content seeds NFS marker only when meta/volumes.yml carries an
nfs-flagged volume
- 14_drain_worker skips custom-image distribution when role has no
build context
- 15_assert_state asserts marker survival only when an NFS volume exists
- ci-orchestrator wires the renamed job + passes whitelist through
- naming-pattern alignment across test-deploy-{server,universal,workstation,swarm}
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the svc-docker-swarm, svc-docker-swarm-manager, svc-storage-nfs-server
and svc-storage-nfs-client roles, the compose_volumes filter NFS support,
the test-deploy-swarm-nfs CI workflow with its swarm-nfs-pilot script set,
and the deploy_mode-aware compose/swarm wiring across mediawiki, mariadb
and sys-svc-* helpers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause of step-08 deploy failures across all 4 test-development
distros in run 26572559126: `make compose-deploy mode=reinstall
apps=web-app-dashboard INFINITO_SERVICES_DISABLED=matomo` (step 07)
made make export the cmdline var into the recipe's shell env, which
reached `runner.py` → `make setup` → `make dotenv`. The dotenv
generator's `setdefault` then baked the runtime-only value into the
on-disk `.env`. Step 08 sourced that .env, filtered `web-app-matomo`
out of its primary apps, and exited with "All primary apps disabled".
Fix:
- Drop the 7 routing knobs (apps, mode, purge, bundles, disable,
full_cycle, variant) from `passthrough.STATIC_KEYS` and from
`default.env`. They are per-invocation; .env is the persistent SPOT
and must not capture them.
- Switch every consumer from `os.environ["KEY"]` strict access to
`os.environ.get("KEY", "")` so the missing-key path stops at "no
filter" instead of a KeyError.
- Rename the env-var surface to match the make-alias surface 1:1
(lowercase): INFINITO_APPS → apps, INFINITO_SERVICES_DISABLED →
disable, INFINITO_BUNDLES → bundles, INFINITO_FULL_CYCLE →
full_cycle, INFINITO_VARIANT → variant, INFINITO_DEPLOY_MODE → mode,
INFINITO_PURGE_ENTITIES → purge. The make alias was the user-facing
name in every doc; routing-knob env vars now use the same name so
the cmdline form, the recipe env-prefix, and the consumer reads
all line up.
- Collapse the Makefile `compose-deploy` recipe to just the two
knobs whose alias differs from the env name (`type` →
INFINITO_DEPLOY_TYPE, `debug` → INFINITO_DEBUG). GNU Make
auto-exports cmdline variables into the recipe's shell env, so the
identical-name `$(if $(apps),apps="$(apps)")` lines were pure
noise.
- Drop `disable: ${disable}` from compose.yml. The dev CLI builds
`extra_env` per `docker exec` call and forwards `disable` via
`-e`, so the container env never needs the value frozen at
compose-up time.
- Update the 3 actual leak call sites
(07/08/09_deploy_*.sh) to the make-alias form
(`disable=matomo`, `full_cycle=true`, `variant=1`).
- Add `SIM112` to ruff's ignore set: the lowercase env names are
the deliberate make/env parity, not a typo.
Repro before fix: `INFINITO_SERVICES_DISABLED=matomo make dotenv`
wrote `INFINITO_SERVICES_DISABLED=matomo` to `.env`. After fix:
`disable=matomo make dotenv` leaves `.env` untouched.
Tests: `make test` and `make autoformat` green (2711 tests, 8 lint
sub-targets).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
INFINITO_CMD is a per-invocation argument, not a config value — drop it
from default.env, the passthrough STATIC_KEYS, and the dotenv-generator
test, and rename to lowercase `cmd` in Makefile targets, exec scripts,
internal callers, and user-facing docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Renames the three in-container deploy helpers out of `utils/` (where
they lived next to unrelated bash libs) into a `container/` subtree
that mirrors the host-side `apps/` layout one-to-one:
utils/fresh-kept-all-init-and-deploy.sh -> container/initialize/all.sh
utils/reuse-kept-all-deploy.sh -> container/update/all.sh
utils/reuse-kept-app-deploy.sh -> container/update/selection.sh
The `fresh-/reuse-/kept-` prefixes were overloaded ("kept" was
universal, "fresh/reuse" duplicated the verb already encoded in
initialize/update). New names use the same verbs + scope as the host
wrappers, so the host/container halves of each flow share a name.
Updates the three host wrappers under apps/{initialize,update}/ to
point at the new in-container paths and modernizes their
cache-retry keys (`fresh-kept-all` -> `initialize-all`, `reuse-all` ->
`update-all`, `reuse-<app>` -> `update-<app>`).
Adds READMEs for container/, container/initialize/, container/update/
following the project documentation conventions (H1 + trailing emoji,
RFC 2119 keywords, env-var tables, cross-links to the matching host
wrappers) and extends deploy/README.md with a pointer to the new
subtree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CI `📊 Top role runtimes` step parses `$ANSIBLE_LOG_PATH` on the
runner, but ansible-playbook actually runs inside the dev container via
`docker compose exec` — and the env var was never forwarded there, so
the log file never materialised. Even when set, the log would have
lived inside the container, invisible to the runner-side parser.
Forwards `ANSIBLE_LOG_PATH` through `compose.exec`'s `extra_env` so
ansible-playbook writes the log at the configured path inside the
container, then `docker cp`s it back to the runner from the cleanup
trap before the stack is torn down.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`entry.sh true` skips `run_local_build` because `true` is treated as
the exec target rather than a flag, so the in-container reinstall step
never ran. `--compile -- true` triggers the build first and then runs
`true` as a no-op exec target.
Catches refactor drift across `.sh`, `Makefile` recursive calls, and
GitHub workflows. Includes fixes for the 5 bugs the new test surfaced
on first run: `make -C ROOT down`/`up` in the network stack refresh
helper (missed during the compose-* rename) and four
`make compose-deploymode=…` / `compose-deployapps=…` typos in
07/08/09_deploy_*.sh plus three docstring/echo references
(missing space after `compose-deploy`, introduced by the
deploy → compose-deploy sed-rename).
Move the deploy router into the compose- domain prefix to match the
rest of the local stack lifecycle (compose-up/down/exec/...). Reorder
its block alphabetically between compose-down and compose-entity-purge
to keep `make help` and the alphabetical-targets lint happy.
Update every active call site: contributor docs, agent-iteration docs,
role-level Administration/README/TODO files, the bundles README, the
test-environment scripts, the in-repo deploy helpers and main.sh, and
the server stage task. The two wildcard prose references (`make
deploy-*` in CLAUDE.md and testing/playwright.md) become single
`make compose-deploy` mentions since no other deploy-* target exists.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the seven separate deploy-* make targets with a single `make
deploy` recipe that maps short Make variables to the canonical
INFINITO_* environment variables and dispatches via the new
scripts/tests/deploy/local/deploy/main.sh router. main.sh and all
sub-scripts continue to read the long-form names, so Ansible passthrough
and container forwarding stay unchanged.
Short Make CLI now exposes apps, mode, purge, type, bundles, disabled,
and full_cycle as aliases. The router picks the verb folder
(initialize, reinstall, update) and scope (all, selection) based on the
chosen mode plus the presence of INFINITO_APPS or INFINITO_BUNDLES.
Also folds in the global INFINITO_TEST_DEPLOY_TYPE -> INFINITO_DEPLOY_TYPE
rename across default.env, passthrough handler, dotenv test, CI deploy
workflows, discovery scripts, and the server stage task.
Active contributor and agent-iteration docs, role-level Administration
and README files, the bundles README, and the deploy reset helper are
all retargeted to the new short CLI. docs/requirements/ entries stay
untouched as historical audit logs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Split the local deploy scripts into a verb-based hierarchy under
scripts/tests/deploy/local/deploy/apps/{initialize,reinstall,update}/
plus scripts/tests/deploy/local/deploy/bundles/, with each verb folder
exposing all.sh (full discovery) and selection.sh (one or more apps via
INFINITO_APPS). The Makefile targets retain their existing names and now
dispatch to the new paths. The reinstall and bundles flows take over the
down + up cycle that used to be a Make prerequisite, and bundles/fresh.sh
now correctly delegates to apps/reinstall/selection.sh (the previous
delegate path was broken after the move). Each new directory carries a
README.md documenting scope, entry points, and when to use which verb.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause of the DinD DNS-check failure on test-deploy-server
(networks.internet.ip4 falling through to 127.0.0.1 despite
INFINITO_IP4=172.30.0.10 reaching every link of the env pipeline):
cli/administration/inventory/provision/cli.py declared --ip4 with a
hardcoded default of "127.0.0.1" and ensure_host_vars_file wrote that
value unconditionally into host_vars/<host>.yml. group_vars'
defaults_networks.internet.ip4 resolves "172.30.0.10" via
\`lookup('env', 'INFINITO_IP4')\`, but tasks/stages/01_constructor.yml
merges \`defaults_networks | combine(networks | default({}, true))\` so
host_vars wins. INFINITO_IP4 looked broken end-to-end; the breakage
actually lived in the provision-CLI host_vars overlay.
INFINITO_DNS_IP and INFINITO_DOMAIN propagated correctly because
the provision CLI had no --dns and the --primary-domain write was
None-gated and never invoked.
While here, --ssl-disabled and --primary-domain are removed too:
neither had any effect in the dev flow. provision/cli.py applies
\`--vars-file inventories/development/default.yml\` AFTER
ensure_host_vars_file via apply_vars_overrides_from_file, whose
_deep_update_commented_map unconditionally overwrites
(\`target[key] = value\`). The dev vars-file always set
\`TLS_ENABLED: true\`, so \`--ssl-disabled\` (which forced
\`TLS_ENABLED: false\` in host_vars) was systematically reverted to
true. Dev/CI ran with TLS enabled the whole time. Inventory stays at
\`TLS_ENABLED: true\` to preserve that effective behaviour.
Removed from cli/administration/inventory/provision/cli.py:
- --ip4, --ip6: defaulted to 127.0.0.1 / ::1, shadowed group_vars
- --primary-domain: never set by any caller, None-gated write
- --ssl-disabled: always re-overwritten by --vars-file later in the
same provision pass
Removed from cli/administration/inventory/provision/host_vars.py the
matching writes in ensure_host_vars_file; the function now only sets
\`ansible_connection: local\` for localhost-class hosts. Defaults for
DOMAIN_PRIMARY / TLS_ENABLED / networks.internet.{ip4,ip6} live in
group_vars (env-driven) and the inventory vars-file (deployment-wide
flags); host_vars stops duplicating them.
Callers cleaned up:
- cli/administration/deploy/development/inventory/builder.py
- cli/administration/deploy/container/command.py
- scripts/tests/deploy/local/utils/fresh-kept-all-init-and-deploy.sh
- scripts/tests/deploy/local/reset/inventory.sh
Tests + docs follow:
- tests/unit/cli/administration/inventory/provision/test_host_vars.py
now asserts host_vars no longer carries baked DOMAIN_PRIMARY /
TLS_ENABLED / networks defaults
- docs/administration/deploy.md drops --ssl-disabled from the
provision example
- docs/contributing/design/inventory.md re-scopes the host_vars
column: the DOMAIN_PRIMARY / TLS_ENABLED / networks.internet.{ip4,ip6}
values arrive via --vars / --vars-file overlays, otherwise group_vars
provides the env-driven default
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
With env.development and env/ci.env both gone, env/default.env was the
last inhabitant of the directory. Hoisting it to the repo root removes
the empty-shell indirection: the SPOT is now [default.env](default.env),
sitting next to .env (its generated companion) and other repo-root
configuration artefacts like compose.yml, Dockerfile, and Makefile.
Path updates land in:
- generator: cli/meta/env/__main__.py (REPO_ROOT / "default.env")
- workflow: .github/workflows/test-install-make.yml (awk reads from
default.env, error message updated in lockstep)
- lint tests: test_used_keys_declared.py and test_infinito_prefix.py
switched from globbing env/*.env to targeting default.env directly;
test_default_env_keys_handled.py points at the new path
- drift test: test_inventory_vars_file_spot.py whitelist + STATIC_ENV
- smoke test: tests/integration/meta/env/test_dotenv_generator.py
STATIC_ENV + every docstring/error message
Doc-string and prose references in utils/env/*, scripts/*,
cli/administration/deploy/development/{common,coredns}.py,
Makefile, Dockerfile, and every doc page that mentioned env/default.env
are rewritten in the same pass so future readers see one path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes a stack of env-pipeline regressions left over from the recent
INFINITO_* namespace refactor by:
* DOMAIN_PRIMARY now reads INFINITO_DOMAIN strictly (was: legacy DOMAIN
env var → defaulted to infinito.localhost, which never resolved
because CoreDNS serves the INFINITO_DOMAIN value, infinito.example).
* group_vars/all/08_networks.yml IPs (ip4/ip6/dns) now read
INFINITO_IP4 / INFINITO_IP6 / INFINITO_DNS_IP.
* PLAYWRIGHT_KEEP_ALL renamed to INFINITO_PLAYWRIGHT_KEEP end-to-end
(env-builder, role default, run_one.yml docker exec, playwright.config.js,
lint allowlist, docs anchor, CI workflow vars).
* PKGMGR_IMAGE renamed to INFINITO_PARENT_IMAGE; Dockerfile ARG drops
the inline default so env/default.env is the only SPOT.
* INFINITO_SRC_DIR centralized: Dockerfile ARG (no default) + ENV;
compose forwards it as build-arg, volume mount path, and container env;
every Python CLI + shell helper + CI workflow now reads it strictly
(os.environ[...] / "${INFINITO_SRC_DIR:?...}") instead of the literal
/opt/src/infinito.
* New entries in env/default.env: INFINITO_IP6=::1, INFINITO_PARENT_IMAGE,
INFINITO_PLAYWRIGHT_KEEP=false. All three propagated through
utils/env/handlers/passthrough.STATIC_KEYS and (where consumed in the
container) through compose.yml's environment block.
Validated end-to-end with `make test` (2360 tests across lint, lint-*,
test-lint, test-external, test-unit, test-integration).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pulls VENV/PYTHON/PIP/PYTHONPATH out of the dotenv handler pipeline
into scripts/meta/python/detect.sh. load.sh now sources detect.sh and
prepends ${VENV}/bin to PATH above the INFINITO_ENV_LOADED guard, so
every BASH_ENV re-entry re-resolves PYTHON the moment a venv
materialises mid-chain — the same fix as b79ed2869 but at the SPOT,
which retires the reload-trick from ansible.sh, python.sh and
pre-commit.sh.
Also renames INVENTORY_VARS_FILE → INFINITO_INVENTORY_VARS_FILE under
the new env-prefix lint, drops the 5 dead INFINITO_ENV_*_LOADED unsets
left over from the multi-loader era, and adds a lint guarding the
INFINITO_ prefix invariant on env/*.env keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new lint tests close gaps in the INFINITO_* SPOT contract:
* tests/lint/filesystem/env/test_default_env_keys_handled.py
Every default.env key must be wired through a handler.
* tests/lint/filesystem/env/test_used_keys_declared.py
Every INFINITO_* in .sh/.py must live in env/{default,ci}.env or
a handler; per-line `# nocheck:` suppresses runtime-only refs.
* tests/lint/filesystem/python/test_no_infinito_get_defaults.py
Forbid ANY default in `.get("INFINITO_*", ...)` -- including the
empty-string fallback (which silently masks an unset key). The
nocheck check walks every physical line the call spans so an
autoformat-split call still honours the marker.
Cache namespace flattens to a single prefix:
INFINITO_PACKAGE_CACHE_* -> INFINITO_CACHE_PACKAGE_*
INFINITO_REGISTRY_CACHE_* -> INFINITO_CACHE_REGISTRY_*
The five previously driver-injected paths (APT_LIST, NPMRC,
PIP_CONF, REGISTRY_PROXY_CONF, FRONTEND_CA_FILE) move into
env/default.env and passthrough.STATIC_KEYS; the
`cache_env_overrides()` helper and its two call sites are gone
along with the test-fixture override for FRONTEND_CA_DIR.
SPOT-strict rewrites:
- 19 sites: `something.get("INFINITO_X", "")` -> subscript
`something["INFINITO_X"]` (loud KeyError when env loader did
not run; all keys live in env/default.env).
- 2 handler-internal bootstrap reads suppressed with
`# nocheck: handler-bootstrap-read`.
- 30 runtime-only / driver-injected / test-fixture references
annotated with `# nocheck: <reason>` per-line.
- Unit tests that previously removed SPOT-guaranteed keys via
`{k: v ... if k \!= KEY}` or `clear=True` switch to setting
the key to "" explicitly, matching the new contract.
Side fixes:
- INFINITO_DNS_DOMAIN + INFINITO_DNS_HOSTS_FILE move from
shell-side defaults in common.sh to env/default.env and
passthrough.STATIC_KEYS.
- dns_systemd_is_operational() drops the dead
INFINITO_DNS_FORCE_HOSTS_FALLBACK override branch (the
variable was never set anywhere).
- INFINITO_WORKER_FETCH switches from `os.environ.get(KEY, "1")`
to `os.environ[KEY]`.
- scripts/tests/code/wrapper.sh exec args move into a bash
array so `# nocheck:` on INFINITO_TEST_TYPE can live on its
own element line (SC2215 fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename env/static.env → env/default.env across the entire repo
(string label, doc links, Python file-path constants) — the
`parse_static_env*` function names stay as a stable API surface.
Sort all `*.env` / `*.env.j2` entries alphabetically (52 in
default.env, 4 in ci.env, 46 of 49 j2 templates; 3 templates
with Jinja2 `{% %}` control flow are skipped because conditional
blocks would tear apart under a naive sort). Sort all 83
Makefile target blocks alphabetically; .PHONY group headers
stay in place.
Collapse the two remaining multi-line Makefile recipe comments
(dotenv-force, test-signed) to single-line WHY comments.
Add four lint tests under tests/lint/filesystem/ to keep the
invariants enforced going forward:
- env/test_alphabetical_sort.py
- env/test_single_line_comments.py (# nocheck: markers exempt)
- makefile/test_alphabetical_targets.py
- makefile/test_single_line_comments.py (# nocheck: markers exempt)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
