Resolves CI swarm rc=69 ("msmtp: needs a password"): controller-side
Ansible lookups (lookup('email').password etc.) couldn't see the
tokens.yml that Mailu writes target-side, because the controller has
its own filesystem in swarm mode.
Mount NFS-shared /var/lib/infinito on every swarm host AND on the
controller during deploy (in-play delegate_to: localhost task at end
of svc-storage-nfs-server role). State subdir exported separately
with no_root_squash + wildcard CIDRs so root-owned writes succeed
and the controller can mount from outside the svc-swarm IP set.
Single-writer guarantee on FILE_TOKENS via delegate_to: STACK_HOST +
run_once on sys-token-store/{01_core,write}.yml. Backup roles
(svc-bkp-container-2-local, sys-ctl-cln-bkps, sys-ctl-cln-faild-bkps,
user-backup/03_permissions_folders, three direct callers of
04_seed-database-to-backup.yml) gated on IS_STACK_HOST to prevent
N-host write races on the shared NFS backup tree.
Fail-fast assertion in svc-storage-nfs-client + 05_controller_mount
when pre-existing local /var/lib/infinito would be hidden by mount-
over. Operator must migrate or wipe before re-deploying.
SPOTs: SWARM_LAB_NETWORK in workflow env block, storage.nfs.state_subdir
in group_vars/all/15_storage.yml, DIR_VAR_LIB unchanged. UPPER_CASE
constants in NFS roles. Cleanup script umounts controller view before
container teardown to prevent stale-mount leak between matrix jobs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
12_collect_diagnostics.sh starts with `set +e` before sourcing
_context.sh, which immediately re-enables `set -euo pipefail` for the
remainder of the script. The first `docker exec ... mount | grep -i nfs`
that finds no nfs mount returned rc=1, pipefail propagated it, and
errexit aborted the script halfway through. GitHub Actions reported a
red tab on the diagnostic step next to the actual failing deploy step.
Move `set +e`/`set +o pipefail`/`set +u` after the source so the
post-failure helper continues through every probe even when individual
commands return non-zero, and the trailing `exit 0` still pins the
script's return code to success.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Loading default.yml via `-e @"${INFINITO_INVENTORY_VARS_FILE}"` (commit
a3c75bde7) made the dev vars available on workers but at extra-vars
precedence (level 22), which overrode the `set_fact` merge in
tasks/stages/01_constructor.yml that reconciles defaults_oidc with the
inventory-supplied OIDC. Workers and mgr-01 ended up with OIDC =
{CLIENT: {SECRET: ...}} only, so the templates that resolved
OIDC.CLIENT.LOGOUT_URL (sys-front-inj-logout etc.) aborted with "dict
has no attribute 'LOGOUT_URL'".
Mirror the provisioned host_vars/${MGR}.yml to host_vars/${WRK1}.yml and
host_vars/${WRK2}.yml at the end of 05_provision_inventory.sh instead.
Same default.yml source via the existing INFINITO_INVENTORY_VARS_FILE
SPOT, same content on every swarm host, but at host_vars precedence
(level 9) so the role-level merge wins on access. Removes the
now-redundant `-e` line from the workflow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
12_collect_diagnostics.sh now ends with an explicit `exit 0` so the
diagnostic step never propagates a non-zero rc to GitHub Actions.
zip_swarm_for_upload.sh switches from `set -euo pipefail` to `set +e`
plus a trailing `exit 0` for the same reason. Keeps the workflow's red
tab on the actual failing step instead of doubling up on the post-mortem
collectors.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add `# nocheck: hardcoded-dns-resolver` on the two `server=` lines
inside the dnsmasq heredoc in 02_start_swarm_nodes.sh so the
test_no_hardcoded_dns_resolvers lint guard recognises the literals as
intentional (dnsmasq parses `#` as a comment, so the marker is
inert at runtime).
Re-stage three pure-formatting tightenings ansible-lint applied to
existing `template: # nocheck` lines (double space -> single space)
in sys-daemon, sys-svc-mail-msmtp and web-app-moodle.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The MODE_DEBUG-gated DinD DNS handler in sys-svc-container spawned a
busybox container with `--dns 1.1.1.1` and expected the mocked
infinito.example to resolve to 127.0.0.1, which Cloudflare obviously
cannot do. Run an in-cluster dnsmasq inside mgr-01 instead of patching
the symptom out:
- Stop and mask systemd-resolved so port 53 is free.
- Bootstrap /etc/resolv.conf with 1.1.1.1 so apt-get can fetch dnsmasq.
- Install dnsmasq with bind-dynamic, no-resolv, address=/${INFINITO_DOMAIN}/127.0.0.1
and 1.1.1.1/8.8.8.8 as upstream forwarders; bind-dynamic catches docker0
when the Ansible-installed dockerd creates it later.
- Switch /etc/resolv.conf to 127.0.0.1 and verify the chain via
`getent ahosts ${INFINITO_DOMAIN}` so the script fails fast on misconfig.
- Pin mgr-01's docker0 to 172.17.0.1/16 via daemon.json `bip`, then point
the swarm.yml networks.internet.dns at that gateway so DinD-spawned
containers reach dnsmasq via their default route.
Also re-stage three pure-formatting tightenings ansible-lint made to
existing `template: # nocheck` lines (double space -> single space).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous commit's default.yml load turned MODE_DEBUG (and therefore
MODE_ASSERT) on for the swarm CI, which activates the DNS check in
tasks/stages/01_constructor.yml:12 that resolves
test.{{ DOMAIN_PRIMARY }}. The pipeline failed on all four hosts
(mgr-01, wrk-01, wrk-02, nfs-server) because:
1. write_hosts_entries did not emit test.<domain> at all.
2. 02_start_swarm_nodes wrote /etc/hosts only on the manager.
Add test.<domain> to the emitted set and write the resulting block to
mgr + both workers + nfs-server so the assertion succeeds everywhere
the play targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The stamp_dir already gets 0777 so any uid can land the stamp on a
shared host, but `touch ${STAMP}` requires write access to the file
itself. When the first run was sandboxed (uid 65534) and the second is
the real user, the second touch failed and pre-commit's `make test`
re-triggered a full `make install`. Matching the dir's permissions on
the file closes that gap.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Use whale (🐳) for the per-app matrix job to mirror the swarm
workflow's bee (🐝) pattern, and magnifying glass (🔍) for the
discover step. Keep the workflow header on the existing 📦 prefix to
stay consistent with the other test-deploy-* siblings.
Rename the workflow file and align all references:
- ci-orchestrator.yml job key + uses: path
- artifact names: inventory-server-* -> inventory-compose-*,
playwright-server-* -> playwright-compose-*
- scripts/github/inventory/zip_compose_for_upload.sh output path
- 3 docs files
INFINITO_DEPLOY_TYPE: server stays - that key is wired to the
invokable-rule registry in utils/roles/validation/invokable.py and is
a separate refactor concern.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add 08_verify_reachable.sh between wait_converge (07) and seed_content
(09, renumbered from 08): TCP-connect via the manager's local
routing-mesh entry confirms incoming traffic reaches a live task. For
multi-replica services, additionally probe each node that hosts a
Running task to verify the mesh works locally on every task-bearing
node.
Skips services without published ports (tag-only roles, internal-only
services). Logs replica count and port list for debugging.
Renumber 08-12 -> 09-13 to keep the sequence contiguous.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
svc-cache-registry -> svc-registry-cache
svc-docker-registry -> svc-registry-docker
svc-docker-swarm -> svc-swarm
svc-docker-swarm-manager -> svc-swarm-manager
categories.yml: drop svc.docker, add svc.registry. Deliberately no svc.swarm
category so get_entity_name('svc-swarm') stays 'swarm' instead of collapsing
to '' (which would break discover_role_services).
Entity keys updated to match the new svc.registry prefix:
services.cache-registry.* -> services.cache.*
services.registry.* -> services.docker.*
01_constructor: replace svc-docker bootstrap-loop entry with explicit
include_role of svc-swarm + bootstrap loop over [svc-registry, svc-storage].
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
inventories/development/swarm.yml carries only swarm-mode-specific
overrides and omits cross-cutting cred seeds (OIDC.CLIENT.SECRET,
biber, administrator password). Until now the swarm CI provision step
did not pass --vars-file, so default.yml was never merged into the
generated host_vars; the first deploy that preloaded web-app-mailu as
a shared email service then hit object of type 'dict' has no attribute
'SECRET' when sys-svc-compose rendered roles/web-app-mailu/templates/
env.j2. Affected jobs in CI run 27006276488: web-app-chess,
web-app-espocrm, web-app-funkwhale, web-app-prometheus.
Match the server-CI pattern (scripts/tests/deploy/local/reset/
inventory.sh and apps/initialize/all.sh both pass --vars-file
${INFINITO_INVENTORY_VARS_FILE}). 05_provision_inventory.sh now does
the same: the env loader sourced via _context.sh already exports
INFINITO_INVENTORY_VARS_FILE in the script shell, so the path
resolves through the SPOT.
write_extras.py re-reads default.yml users dict and patches the
runtime SSH key onto admin authorized_keys before emitting the
extras file. Without this, the deploy-time -e @swarm-nfs-extras.yml
would REPLACE the host_vars users entry (ansible later-wins on
top-level dicts) and wipe biber + the admin password the
--vars-file just baked in.
Two distinct breakages that let the workflow fail before ever reaching
the ansible deploy:
* `python3 -m utils.tests.swarm.*` (and friends) ran with the
actions-setup interpreter that has no project deps, so the imports
of `plugins.filter.canonical_domains_map` blew up on `from ansible.errors`.
The swarm preamble scripts now source `scripts/meta/env/load.sh` so
PATH picks up the venv `make install` populated, and the workflow itself
uses `install-force` (under `act --bind` the build/install.stamp is the
host's and would short-circuit the in-container install) plus pushes
`\${VENV}/bin` onto GITHUB_PATH for downstream steps.
* `ansible-playbook -e @file` replaces top-level dicts rather than
deep-merging them. The runtime-only extras file was dropping
`storage.nfs.{export_base,version}`, `storage.backend`, and
`swarm.{registry.port,network}` that the static
`inventories/development/swarm.yml` ships. Re-state those keys
in write_extras.py so the merge ends up with everything the roles need.
The script's three lines were redundant: \`pip install -e .\` and the
ansible-galaxy collection step are already covered by \`make install\`
(now landed in the swarm matrix preamble), and \`PYTHON=python3 bash
scripts/setup.sh\` is exactly what \`make setup\` invokes. Replace the
step with a direct \`make setup\` call.
Renumber the remaining scripts to keep the sequence lückenlos:
06_provision_inventory.sh -> 05_provision_inventory.sh
07_debug_inventory.sh -> 06_debug_inventory.sh
08_wait_converge.sh -> 07_wait_converge.sh
09_seed_content.sh -> 08_seed_content.sh
10_drain_worker.sh -> 09_drain_worker.sh
11_assert_state.sh -> 10_assert_state.sh
12_collect_diagnostics.sh -> 11_collect_diagnostics.sh
13_cleanup.sh -> 12_cleanup.sh
Workflow YAML updated to the new names. No script content changed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New ``cli.administration.deploy.swarm`` package wraps the strict
``cli.administration.deploy.dedicated`` runner with a layered closure
so swarm deploys never require operators (or test scripts) to
enumerate every infra role:
effective --id = seed (operator --id, else inventory_role_groups)
∪ dep-walk(seed via applications_if_group_and_all_deps)
∪ default-placement safety net
The dep-walker is the same SPOT
(``utils.roles.applications.in_group_deps.applications_if_group_and_all_deps``)
the runtime constructor consumes via ``lookup('applications_current_play')``,
so the CLI-side closure and the in-play role set stay in sync.
The dedicated CLI keeps strict explicit-only semantics: ``main`` was
split into ``parse_args`` + ``run_from_args`` so the swarm wrapper
can share parser, mode wiring, validator and runner verbatim and
only mutate ``args.id``.
Meta-side: ``svc-db-postgres`` gains ``default_placement: manager``
(semantic symmetry with mariadb). ``default_placement`` is now a
PLACEMENT hint only ("if deployed, run on the manager"); deploy
decisions come from the dep-walk. Apps that declare ``mariadb:``
pull svc-db-mariadb; apps that declare ``postgres:`` pull
svc-db-postgres; apps that declare neither get no central DB.
Shared utils extracted so the four consumers stay aligned:
* ``utils.inventory.groups.inventory_has_group``: YAML + INI static
inventory group probe (extracted from ValidDeployId).
* ``utils.roles.meta_lookup.{get_role_default_placement,
iter_roles_with_default_placement}``: single source of truth for
``default_placement`` discovery (lookup plugin is now a thin wrapper).
Test pipeline (``scripts/tests/deploy/swarm/*``):
* Heredoc-Python shells converted to proper modules under
``utils.tests.swarm.*`` and invoked via ``python -m``:
- extend_inventory.py (was 08_extend_inventory.sh)
- write_extras.py (was 09_write_extras.sh dynamic half)
- derive_includes.py (replaces 06's hardcoded includes array)
- write_hosts_entries.py (replaces 02's hardcoded /etc/hosts heredoc;
derives every canonical + www-redirect domain from APP_ID via
canonical_domains_map + generate_all_domains, so the test
works for any app in the matrix without per-app maintenance)
* 01_create_network.sh deleted (single docker command inlined in workflow).
* 11_deploy.sh deleted (swarm CLI now called directly in workflow).
* 09_write_extras.sh: static half moved to inventories/development/swarm.yml.
* Remaining shell scripts renumbered 01-13 without gaps.
* 08_wait_converge.sh: case block on DB_DEP replaced with single
conditional (container name == DB_DEP).
* 06_provision_inventory.sh: hardcoded includes array replaced with
`mapfile -t includes < <(python -m utils.tests.swarm.derive_includes)`.
* Workflow YAML step references updated to new script names + module
invocations.
New tests (39 covering the closure surface):
* tests/unit/cli/administration/deploy/swarm/test_closure.py (14)
* tests/unit/utils/inventory/test_groups.py (11)
* tests/unit/utils/roles/test_meta_lookup.py (+3 for default_placement)
* tests/unit/utils/roles/validation/test_deploy_id.py (unchanged
8 tests still pass after refactor onto the shared util).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drop the required_by declaration on svc-cache-registry so the
post-deploy validator no longer enforces its execution for every
svc-docker / web-app deploy. The pull-through cache is a performance
optimization, not a correctness requirement; operators that omit it
from inventory should not see a deploy fail.
Compose-mode auto-place now scopes svc-prx-openresty to plays that
actually carry a web-* role (any group whose name matches ^web-),
so svc-only deploys do not get an orphan front-proxy. The match is
intentionally on the web-* prefix family so new web-<sub>-<entity>
categories pick up the auto-place rule without an allowlist edit.
The swarm-NFS test still wants the cache for the pull-through speedup,
so svc-cache-registry is listed explicitly in 07_provision_inventory.sh
alongside the swarm bootstrap roles.
Fixes test-deploy-server / Server web-app-mediawiki failing with
"required role(s) did not execute for deploy: svc-cache-registry".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add tests/lint/ansible/jinja/test_no_ansible_facts_in_templates.py: under
swarm-mode the rendered template lands once on the primary manager and is
slurp-distributed to every other node, so any per-host fact reference
(ansible_hostname, ansible_default_ipv4, ansible_fqdn, ansible_nodename,
ansible_machine_id, ansible_eth0, ansible_all_ipv*_addresses) would
silently produce wrong output on the other nodes. The lint blocks them
at PR time.
scripts/tests/deploy/act/workflow.sh forwards --action-offline-mode +
--action-cache-path so repeated local act runs do not refetch the actions
catalog.
roles/sys-ctl-cln-anon-volumes/tasks/01_core.yml quoting cleanup applied
by autoformat (single to double quotes, no semantic change).
.claude/settings.json adds Workflow to permissions.allow so the
multi-agent Workflow tool no longer prompts per invocation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both deploy workflows now ship the rendered inventory for the matrix
entry as a zip artifact so failed runs can be inspected without
re-running the provisioner. test-deploy-swarm.yml zips /tmp/inv directly;
test-deploy-server.yml docker-cps /root/inventories from every running
infinito_nexus_* container into a staging dir first because the
inventory lives inside the compose container, not on the runner.
Both steps are if: always() to capture artifacts on failure too, but
gated on \!env.ACT because act-local does not provide
ACTIONS_RUNTIME_TOKEN and would otherwise fail the upload step.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DinD DNS chain reworked so the Playwright container run --rm npm install
no longer EAI_AGAINs on registry.npmjs.org:
- 03_start_swarm_nodes.sh runs each DinD with --dns 1.1.1.1/8.8.8.8 and
pre-seeds /etc/docker/daemon.json. systemd-resolved still clobbers the
outer resolv.conf at boot, so the daemon-level DNS is the only thing
that survives for containers the inner daemon spawns.
- daemon.json.j2 omits the "dns" key when networks.internet.dns is empty
(default), so the rendered config no longer overwrites the pre-seeded
daemon.json with "dns": [""] and break inner-container resolution.
- 09_write_extras.sh sets networks.internet.dns to a public resolver for
the test rig and bakes RUNTIME via cli.meta.runtime (so e2e-playwright
actually runs in act/github instead of being skipped under "host").
07_provision_inventory.sh generates mirrors.yml via
cli.contributing.mirror.resolver and passes --mirror to the inventory
provisioner only when cli.meta.runtime == "github". act/local keeps
upstream pulls (no mirror infrastructure to point at).
12_wait_converge.sh and 15_assert_state.sh accept any N/N replica state
instead of hardcoded 1/1, so multi-host-replica deploys converge cleanly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Roles that declare default_placement: manager in meta/services.yml are now
auto-added to their own inventory group when the operator did not list them
explicitly. Both the runtime constructor add_host loop and the CLI
pre-flight inventory extender use the new roles_with_default_placement
lookup so swarm and CLI-validation stay consistent.
In compose-mode (no svc-docker-swarm-manager group), svc-prx-openresty
auto-joins the first play host. run_once + ansible_play_hosts | first
keeps that deterministic in multi-host compose deploys so the openresty
vhost view does not fragment.
The svc-cache-registry / svc-docker-registry entries previously hardcoded
in scripts/tests/deploy/swarm/08_extend_inventory.sh fall out of the new
dynamic loop; net inventory is unchanged.
ansible_play_hosts joins the magic-var allowlist in
test_variable_definitions so the constructor add_host passes integration
lint.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The install + update + admin maintenance commands all run as throwaway
container instances against the custom image, with the live service's
bind mounts attached. The previous container exec-against-static-name
pattern stops working in swarm where the container name is a
per-replica swarm task id and the task itself can be reaped by the
healthcheck mid-update.
container_name is gated under swarm because docker-compose's strict
validation rejects it once replicas > 1. Cross-host config replication
in 01_configuration.yml is removed entirely: the play now runs natively
on every host in the role's group, so the LocalSettings.php / oidc.php
/ debug.php templates render in place per host.
Test inventory adds the pilot role to all three simulated swarm hosts,
which is what flips DEPLOYMENT_MODE to swarm under the new
inventory-driven model.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New role that runs registry:2 in pull-through proxy mode on the swarm
manager. Worker nodes' daemon.json gets a registry-mirrors entry plus
the corresponding insecure-registries entry so plain-HTTP traffic to
the cache is permitted. The role's inventory group lists exactly one
host (manager) so the new inventory-driven topology model resolves
DEPLOYMENT_MODE to compose for the cache itself.
Within a single deploy run, the three swarm workers all pull the same
base images via the cache instead of hitting registry-1.docker.io
three separate times. Measured speedup on iter26 vs the no-cache iter21
baseline: pkgmgr/apt-install steps 2.4-2.8x faster, Docker image pulls
1.8-2.8x faster.
Cross-iter cache persistence is intentionally out of scope: act
recycles the DinD nodes between runs so the cache volume is wiped each
time. Cross-run reuse would require either bind-mounting the cache
from the host or pointing daemon.json at the dev compose stack's
registry-cache; both are tracked separately.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
iter11/iter12/iter14 each hung nfs-server's `apt-get update` in TCP
CLOSE_WAIT against apt-cache:3142 for 5-60 minutes, even with
Pipeline-Depth=0 + No-Keep-Alive + Retries=3 + Timeout=30. Pulling once
into the cache and fanning out to four nodes saves ~4 minutes of
bootstrap when it works, but every run that hit the hang lost 20+
minutes — net negative.
Remove the workflow cache restore step, the 02b_start_apt_cache.sh
launcher, the apt.conf.d/01-cacher write in 05_bootstrap_nodes.sh, and
the apt-cache cleanup in 17_cleanup.sh. Each node now talks directly
to the upstream mirrors. Slower-but-reliable beats faster-but-stuck.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
iter11 hung nfs-server's `apt-get update` for 16+ minutes in CLOSE_WAIT
against apt-cache:3142 even with Pipeline-Depth=0 — the cache closed
each TCP slot but apt's HTTP method held onto them via keep-alive and
never noticed the half-close.
Set `Acquire::http::No-Keep-Alive "true"` so every apt request opens a
fresh socket and tears it down on completion (no shared state to leak),
plus a 30 s `Acquire::http::Timeout` so any future stuck fetch errors
out fast enough for the Retries=3 backoff to actually kick in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
In CI the workflow runs as the unprivileged `runner` user, so
`scripts/install/package.sh`'s `apt-get update && apt-get install
build-essential debhelper dpkg-dev rsync` aborted with `E: Could not open
lock file /var/lib/apt/lists/lock - open (13: Permission denied)` before
the .deb was ever produced. Local act runs as root inside
catthehacker/ubuntu so we never hit it.
Detect non-root in 05_bootstrap_nodes.sh and prepend `sudo -E` to the
build invocation. Inside the swarm-node containers package.sh still
runs as root and never goes through sudo (those exec calls are
unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Leaving the apt-cacher-ng proxy enabled past the .deb-deps install made the
later ansible-driven apt tasks (sys-svc-ssh, update, ...) hang on TCP
CLOSE_WAIT against the cache even with Pipeline-Depth=0: the connections
the proxy closed upstream were never picked up by ansible's apt method on
the next invocation. The cache only pays for itself during the parallel
4-node .deb-deps fetch anyway — afterwards the proxy is pure friction.
Remove /etc/apt/apt.conf.d/01-cacher at the end of provision_node so
every ansible apt task hits the upstream mirrors directly, undisturbed
by the cache's keep-alive semantics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jrei/systemd-ubuntu:24.04 has been exiting with code 255 in ~0.4s on
ubuntu-24.04 GHA runners since image build 20260525.161, producing zero
stdout/stderr (the docker-logs fallback from d7e859041 captured nothing).
`--privileged` alone no longer disables AppArmor + seccomp on the host
runner image, so systemd can't take ownership of its cgroup / signal mask
and PID 1 aborts before it logs anything.
Add the three flags that, in combination, restore the same isolation
posture systemd-in-docker has historically had:
--cgroupns=host let systemd see the host cgroup tree
(otherwise the unified hierarchy is
remapped under /sys/fs/cgroup/system
and pid 1 refuses to manage it)
--security-opt seccomp=unconfined drop the default seccomp filter that
now blocks io_uring/clone3 calls
systemd-255 uses during init
--security-opt apparmor=unconfined explicit override; --privileged is
supposed to imply this but the new
runner image's docker.service-level
profile re-applies it on top
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
iter4 hung swarm-wrk-02's `apt-get update` for 5+ minutes in TCP CLOSE_WAIT
against the apt-cacher-ng container: 3 sockets to apt-cache:3142 were closed
upstream, but apt's http method kept reusing pipelined slots and never
detected EOF.
`Acquire::http::Pipeline-Depth "0"` forces apt to one request per connection,
which is the documented workaround for apt-cacher-ng (see acng manual
section 4 "Common Apt Configuration Problems"). Bundled defaults:
* `No-Cache "false"` lets validated cache hits short-circuit IMS checks
* `Retries "3"` makes transient proxy errors recoverable
Switched the inline apt.conf write from a fragile single-line `printf
> file` to a heredoc piped through `docker exec -i`, since the new content
spans multiple lines.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pilot's /tmp/swarm-nfs-extras.yml is passed to ansible via -e @file
which makes its top-level keys (including swarm:) fully *replace* the
group_vars/all/18_swarm.yml swarm dict — silently dropping swarm.registry
and breaking the daemon.json.j2 + handler chain with
"object of type 'dict' has no attribute 'registry'" once iter3 hit the
Configure docker daemon storage driver task on the workers.
09_write_extras.sh now emits swarm.registry.host=${MGR} (swarm-mgr-01,
docker-DNS-resolvable from the workers) and port=5000, matching the
group_vars defaults.
Bundled: the lint-python EXE001 violation on the new
roles/sys-svc-compose/files/swarm_registry_push.py — the shebang was
present but the file was not chmod +x.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a single apt-cacher-ng container (sameersbn/apt-cacher-ng:3.7.4-20220421)
on the swarm-lab network and routes every node's apt traffic through it via
/etc/apt/apt.conf.d/01-cacher. With four nodes provisioning the same
infinito-nexus.deb deps in parallel (~200 MB / node), the cache turns the
second-through-fourth fetch into a hot serve from the local proxy.
The workflow restores ${{ runner.temp }}/apt-cacher-ng across runs via
actions/cache@v4 (keyed on packaging/debian/control), so on warm GHA runs
the cache is also populated for the first node — driving bootstrap from
~11 min toward sub-3 min once the cache is hot. The cache restore step is
guarded by `if: \!env.ACT` so act runs still exercise the cold path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ACT_INPUTS="key=value [key2=value2]" expands to repeated --input flags so
make act-workflow can drive workflow_dispatch-style inputs without falling
back to a manual act invocation. Needed for the swarm pilot, where the
discover job's `whitelist` input gates the per-app matrix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`docker exec journalctl` fails when the container is already dead (which
is the actual failure mode: jrei/systemd-ubuntu:24.04 exits with code 255
~0.4s after start on the current GHA runner image). `docker logs` works
on stopped containers and surfaces what systemd printed before it gave up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a manager-pinned, NFS-backed `registry:2` service plus a transparent
retag/push/rewrite step so swarm workers can pull custom-built images from a
local registry, replacing the `docker save | docker load` pilot crutch.
Mechanism: after `compose build`, the new `swarm_registry_push.py` reads the
rendered compose.yml, retags each service that declares both `build:` and
`image:` with `<manager>:5000/<image>`, pushes the retag, and rewrites the
`image:` field in place. Subsequent `docker stack deploy --resolve-image
never` then references the registry-qualified name, so workers pull from the
cluster-local registry over HTTP via `insecure-registries`.
No per-role changes required — the rewrite is centralized in
sys-svc-compose handlers and skips services without a `build:` directive.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous 60s window flaked on GHA matrix runs because jrei/systemd-ubuntu:24.04 cold-starts D-Bus + targets slowly under parallel load. 03_start_swarm_nodes.sh additionally let downstream steps run against a non-ready node instead of failing.
Both scripts now exit non-zero with journalctl + State diagnostics on timeout. Override via SWARM_PILOT_SYSTEMD_TIMEOUT for local debug.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror the test-deploy-server iteration scaffold: discover + matrix on
the same app list (INFINITO_DEPLOY_TYPE=server), one swarm-deploy run
per app. Pilot scripts derive per-app context from APP_ID + role meta
(ENTITY, SERVICE_NAME, DB_DEP, CUSTOM_IMAGE_REPO, PRIMARY_NFS_VOLUME)
so they adapt to roles other than mediawiki.
- .github/workflows/test-deploy-swarm-nfs.yml -> test-deploy-swarm.yml
- scripts/tests/deploy/swarm-nfs-pilot/ -> scripts/tests/deploy/swarm/
- new _context.sh helper sourced by 07/08/11–16
- 07_provision_inventory + 08_extend_inventory + 11_deploy build their
--include list from DB_DEP + APP_ID
- 12_wait_converge skips DB readiness when DB_DEP=none
- 13_seed_content seeds NFS marker only when meta/volumes.yml carries an
nfs-flagged volume
- 14_drain_worker skips custom-image distribution when role has no
build context
- 15_assert_state asserts marker survival only when an NFS volume exists
- ci-orchestrator wires the renamed job + passes whitelist through
- naming-pattern alignment across test-deploy-{server,universal,workstation,swarm}
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the svc-docker-swarm, svc-docker-swarm-manager, svc-storage-nfs-server
and svc-storage-nfs-client roles, the compose_volumes filter NFS support,
the test-deploy-swarm-nfs CI workflow with its swarm-nfs-pilot script set,
and the deploy_mode-aware compose/swarm wiring across mediawiki, mariadb
and sys-svc-* helpers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous tree-hash dedup compared whole-commit trees, so unrelated
drift on main (e.g. a manual edit to any roles/* file) caused the same
auto-update delta to appear "new" each daily cron and a fresh PR to open.
Now: fingerprint only the files that this run actually committed
(git diff --name-only HEAD~1 HEAD), both locally and on each candidate
open PR with the same UPDATE_BRANCH_PREFIX. If any open PR carries the
identical content for those files, skip push + PR creation entirely.
Branch name now carries the fingerprint's short hash so parallel PRs
of the same class can coexist (the close-superseded loop is dropped).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
scripts/image/build.sh drives 'make build' and the test-environment
matrix's dev-image build, but missed the new INFINITO_VENV_DIR build
arg. Dockerfile's ARG resolved to empty, ENV set INFINITO_VENV_DIR=""
in the image, and entry.sh's assert tripped on first --compile.
Adds the assert and the matching --build-arg, parallel to the
INFINITO_SRC_DIR plumbing already in this file and to the equivalent
plumbing in scripts/image/push.sh.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds INFINITO_VENV_DIR=/opt/venvs/infinito to default.env (registered
in passthrough's STATIC_KEYS so make dotenv propagates it to .env) and
threads it through the build chain: compose.yml forwards it as a build
arg; scripts/image/push.sh asserts and --build-args it; the Dockerfile
declares the matching ARG/ENV and derives PYTHON/PIP/PATH from the
variable. scripts/meta/env/python.sh now prefers the value when set
before its fallback discovery so the SPOT is effective in env
resolution.
Fixes the dev image build regression where 'infinito --version' in
scripts/docker/entry.sh exited 127. Root cause: the Dockerfile RUN
SHELL uses bash -lc, which sources /etc/profile and resets PATH to
the Debian default, dropping ${INFINITO_VENV_DIR}/bin from it. The
fix invokes the script via its absolute path "${INFINITO_VENV_DIR}/bin/infinito"
so the post-install version check no longer depends on a PATH that
the login-shell init has just clobbered.
scripts/lint/wrapper.sh's existing PATH-reset note now references the
variable instead of the literal path. test-install-make.yml and
test-install-pkgmgr.yml read INFINITO_VENV_DIR from default.env up
front, pass it into their pkgmgr containers, and source the venv
activate script via the variable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
push_ci_policy.sh now hard-fails on missing GH_TOKEN/GITHUB_REPOSITORY
and, for non-main branches, calls gh pr list to detect whether the
push is already covered by an open pull request. If so, emits
should_run=false (skip_reason=pr-ci-covers-this-push) so the push
workflow exits cleanly and the pull_request workflow handles the
actual CI run.
To absorb the bot's push -> gh-pr-create latency on update/* and
alert-autofix-* branches, the PR check retries up to 10x with 2s
sleeps before falling through to a real CI run. Human branches
(feature/**, hotfix/**, fix/**, chore/**) check once with no delay.
entry-push-latest.yml's run-policy step now forwards GH_TOKEN so the
gh call works against the repo.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous gh pr close call left the bot's daily update branch
behind on the remote, so stale update/<prefix>-YYYYMMDD refs
accumulated indefinitely. --delete-branch removes the head ref on
close. Scope-safe because the loop only iterates branches matching
${UPDATE_BRANCH_PREFIX}-.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a detect-affected-roles job to entry-pull-request-change.yml that
runs the affected-roles resolver upfront on host python (via make
install) and emits whitelist + roles_only. ci-orchestrator skips
test-development when roles_only=true; done job tolerates the skip.
entry-manual derives roles_only from its whitelist input.
affected_roles.sh now invokes the resolver host-side only (compose-exec
fallback removed; it was a no-op in every existing caller because
compose was never up there, always emitting __ALL__ fail-safe). The
PR-flow gate consolidates what test-deploy-* used to recompute per
job; downstream effective_whitelist.sh now consumes the pre-resolved
whitelist verbatim.
Test fixture switched from a fake docker binary to a fake PYTHON
binary and clears BASH_ENV so the bash-startup env loader does not
reset PYTHON.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of step-08 deploy failures across all 4 test-development
distros in run 26572559126: `make compose-deploy mode=reinstall
apps=web-app-dashboard INFINITO_SERVICES_DISABLED=matomo` (step 07)
made make export the cmdline var into the recipe's shell env, which
reached `runner.py` → `make setup` → `make dotenv`. The dotenv
generator's `setdefault` then baked the runtime-only value into the
on-disk `.env`. Step 08 sourced that .env, filtered `web-app-matomo`
out of its primary apps, and exited with "All primary apps disabled".
Fix:
- Drop the 7 routing knobs (apps, mode, purge, bundles, disable,
full_cycle, variant) from `passthrough.STATIC_KEYS` and from
`default.env`. They are per-invocation; .env is the persistent SPOT
and must not capture them.
- Switch every consumer from `os.environ["KEY"]` strict access to
`os.environ.get("KEY", "")` so the missing-key path stops at "no
filter" instead of a KeyError.
- Rename the env-var surface to match the make-alias surface 1:1
(lowercase): INFINITO_APPS → apps, INFINITO_SERVICES_DISABLED →
disable, INFINITO_BUNDLES → bundles, INFINITO_FULL_CYCLE →
full_cycle, INFINITO_VARIANT → variant, INFINITO_DEPLOY_MODE → mode,
INFINITO_PURGE_ENTITIES → purge. The make alias was the user-facing
name in every doc; routing-knob env vars now use the same name so
the cmdline form, the recipe env-prefix, and the consumer reads
all line up.
- Collapse the Makefile `compose-deploy` recipe to just the two
knobs whose alias differs from the env name (`type` →
INFINITO_DEPLOY_TYPE, `debug` → INFINITO_DEBUG). GNU Make
auto-exports cmdline variables into the recipe's shell env, so the
identical-name `$(if $(apps),apps="$(apps)")` lines were pure
noise.
- Drop `disable: ${disable}` from compose.yml. The dev CLI builds
`extra_env` per `docker exec` call and forwards `disable` via
`-e`, so the container env never needs the value frozen at
compose-up time.
- Update the 3 actual leak call sites
(07/08/09_deploy_*.sh) to the make-alias form
(`disable=matomo`, `full_cycle=true`, `variant=1`).
- Add `SIM112` to ruff's ignore set: the lowercase env names are
the deliberate make/env parity, not a typo.
Repro before fix: `INFINITO_SERVICES_DISABLED=matomo make dotenv`
wrote `INFINITO_SERVICES_DISABLED=matomo` to `.env`. After fix:
`disable=matomo make dotenv` leaves `.env` untouched.
Tests: `make test` and `make autoformat` green (2711 tests, 8 lint
sub-targets).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Docker entry script's `[docker-infinito] Installed:` banner used
to call `pkgmgr version infinito`; after the pkgmgr removal that line
became `infinito --version || true`, which printed
`Error: command '--version' not found.` to stderr because the CLI's
argparse-style dispatcher treated `--version` as an unknown
subcommand.
Hook a real top-level flag into cli.core.app.parse_flags that prints
`infinito <version>` from importlib.metadata (single source of truth:
pyproject.toml's `version` field; falls back to `unknown` when the
package isn't pip-installed) and exits 0. entry.sh drops the
`|| true` since the call is now well-defined.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Gives Playwright sidecars a stable name for the docker host so role specs
can reach services whose port is bound on the host (e.g. Mailu's SMTP
submission on 465/587 in the Zammad email-channel test) without
attaching to that service's docker network.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`update.yml` jobs (Docker image versions, repository refs) shelled
straight into `python3 -m cli.contributing.update.{docker,repository}`
without sourcing the project dotenv. After eb4bcd23a removed the
`.get(..., default)` fallback in resolve_max_fetch_workers(),
KeyError: 'INFINITO_WORKER_FETCH' surfaced on every run.
Both wrapper scripts now `make dotenv` (idempotent) and `set -a; . .env;
set +a` before exec'ing the updater, restoring the SPOT env chain on
CI runners that don't go through compose.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
