On Fedora 44, dnf satisfies docker-ce's `(iptables-nft or iptables)`
requirement with iptables-legacy. The legacy backend needs the ip_tables
module, which ubuntu-26.04 runners no longer have loaded, and the compose
container cannot load it because it mounts no /lib/modules. The MSS clamp
task then failed with `modprobe: FATAL: Module ip_tables not found` and
`can't initialize iptables table 'mangle'` (Nextcloud#10, Gitea#0,
Jellyfin#0 on fedora). CentOS 10 already resolves to iptables-nft from the
same plan and deploys green on the same image.
Naming iptables-nft in the docker transaction makes Fedora use the
nf_tables backend as well.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A freshly built dev container installs ansible-lint unpinned (26.10.1,
yamllint 1.38.0), and its --fix pass in make autoformat now drops the
trailing blank line of meta/packages.yml and
roles/sys-svc-container/meta/packages.yml and adds the missing document
start to roles/svc-net-wireguard/files/test/compose.yml (from #323).
Parsed YAML is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
No conflicts. The 14 incoming commits fix the n8n gateway probe trigger under
n8n 2, let stalwart address bans expire, keep the Docker apt key when its
refresh is refused, leave eol roles out of image discovery, stop the release
backfill below the oldest buildable tag, and provision the mermaid browser
only in install-lint. None of them touches the mesh, swarm, NFS, network or
workflow files this branch changes.
A local deploy stopped four times in two days in "Ensure Docker apt key
and sources are consistent" (#849). apt_repo.sh downloads the Docker GPG
key on every run, and when download.docker.com refused the connection,
curl's four attempts were over within about six seconds and the script
returned 1, although the key from an earlier run was installed and is
all the following apt operations need.
install_key now fails only when no key is installed. With a key in
place, a refused download prints a warning and keeps it. A downloaded
key that differs still replaces the installed one, and the 0644 mode is
enforced in every case. The keyring path became an argument:
ensure-key-and-sanitize passes the keyring it already hands to
sanitize_sources, so the role's call still resolves to
/etc/apt/keyrings/docker.asc, and a test can point the script at a
temporary keyring.
Verified: tests/unit/python/roles/sys-svc-container/files/test_apt_repo.py
stubs curl and sudo. A refused download keeps the installed key and ends
with changed=0, a refused download without a key exits 1, and a new key
replaces the installed one. A copy of the script whose keep branch always
errors, the old behaviour, fails the first test. Two local deploys ran
the new script with rc 0 and an empty stderr and left the installed key
untouched. shellcheck is clean.
Not verified live: the refused download itself, which needs
download.docker.com to refuse a connection during a deploy. A host
without a key still needs the download and still fails when it is
refused.
Closes#849
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolving the merge I kept this branch's SYS_SVC_CONTAINER_STORAGE_DRIVER on the
grounds that it routes a value through a variable and nothing superseded it. The
unit suite disagrees: upstream added TestDaemonStorageDriver, which renders
daemon.json.j2 against a context that deliberately does not define the variable
and asserts the literal, so six tests failed with UndefinedError.
Nothing ever set the variable -- not even the wireguard role this branch removed
-- so it was a knob with no consumer, and upstream has since pinned the literal
with a test. Both files now match upstream exactly.
Brings this pre-swarm branch up to the current swarm-era codebase: 3681 commits
across 7221 files. Our side contributed almost nothing by comparison, so the
conflict surface stayed at the 17 files both sides had touched.
Resolutions:
- The 15 legacy svc-net-wireguard-{core,firewalled,plain} files conflicted as
modify/delete and rename/delete (upstream renamed 01_core.yml to 00_core.yml).
All resolved as deletions: this branch is starting wireguard over. Five more
files of those roles came back silently because upstream had not touched them
since the merge base, and were removed after the merge.
- sys-svc-container/templates/daemon.json.j2 took upstream's structure, which
adds default-address-pools and json-file log limits and makes storage-driver
conditional on DOCKER_IN_CONTAINER. Our unconditional version would have set
fuse-overlayfs on bare-metal hosts too, so theirs is the correct semantics;
only the value now routes through SYS_SVC_CONTAINER_STORAGE_DRIVER.
- sys-svc-container/vars/main.yml keeps that variable, which upstream never had
and nothing supersedes, and adopts upstream's removal of the MTU comment.
Follow-up the merge itself required: the root README role table was regenerated
and the privileged_control_plane list in requirement 035 dropped from 24 to 21,
both of which still enumerated the deleted roles.
sys-pip-install passes package_name straight to ansible.builtin.pip and defaults package_state to latest, so a bare name pulled whatever PyPI served that day and nothing in the repository recorded which version a host ended up with. svc-bkp-volume-2-local already pins this way.
A version inside the name is compatible with state=latest: the pip module rejects only the separate version parameter, its guard reads 'state == latest and version is not None', so this resolves as pip install -U automtu==1.1.1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
update_config never set failure_action, so swarmkit's default applied: pause,
at a max_failure_ratio of zero. One failed task stops the rollout for good, and
nothing in the deploy resumes it.
That contradicts the restart_policy three lines below. The policy replaces the
task and the service ends up healthy, while the update status stays latched and
service_converged.sh reads that status before it looks at any task, so the
deploy reports a failure over a service that is running. The rescue artifact
shows the sequence: the old task shut down, its replacement tripped over a
containerd object runsc could not reap, and the third task was Running while
the job died on "update latched".
Four swarm jobs failed that way, all in the update pass, never in the first.
The convergence check is untouched and stays as strict as it was: a genuinely
broken spec fails every task and still trips "tasks not running yet".
What this does give up: the latch was also the only thing catching a service
that keeps crash-looping. At ratio zero any failure latched, so a task dying
and restarting forever was guaranteed to be reported. Now the rollout
completes, and the check samples task state at one moment - a task on its
fiftieth restart still reads as Running, and wait_converged's `until` needs a
single lucky sample. Restoring that detection needs the check to look at task
churn, which the service ps history already carries and the script prints only
on failure today. Not in this commit.
max_failure_ratio cannot substitute here: at replicas 1, the common case in
this repository, one failure is always the whole service.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
container_dns decides per host whether containers get the docker bridge
or the clearnet resolver, and this host gets the clearnet one: daemon.json
carries 172.30.0.53, and nothing listens on the bridge gateway. The probe
read that gateway from the running daemon and forced the test container
onto it with --dns, so it answered 'Connection refused' against a daemon
that was resolving correctly, and the deploy died in the infrastructure
phase before any role ran.
Dropping the override lets the container take what every app container
takes. That measures the real path in both topologies - the cluster nodes
where the daemon does hand out the bridge, and here.
ce43263745 replaced a stale inventory literal with this runtime lookup
and kept its assumption that the gateway is the resolver at all.
The swarm inventory carried the manager's docker bridge gateway as a literal under networks.internet.dns, so one field meant two things at once and the value went stale the moment default-address-pools moved the bridge. The daemon kept handing out the new address while the inventory kept naming the old one, and the DinD diagnostic probed that same literal, so it failed in step with the defect instead of catching it.
The inventory now declares only the clearnet resolver. The lookup hands containers the bridge whenever the host resolves solely through a loopback listener, which is how the simulated cluster nodes are wired and which docker strips from a container's resolv.conf. The diagnostic reads the gateway from the running daemon instead of being told it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
must_exist accepted any existing path, so a directory that dockerd auto-created where a bind source was missing passed the guard. The wrapper was then mounted over the container path and set as --entrypoint, and every image that has an ENTRYPOINT died with rc 126 'is a directory: permission denied' — seven identical retries, because the directory persists. Requiring a regular file routes that state into the existing 'CA injection disabled' fallback, which the same suite already exercises green when the wrapper is simply absent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Docker's built-in pools yield about thirty networks. The stack holds
seventy-two, so every further network creation failed with 'all predefined
address pools have been fully subnetted' — which is why web-app-nextcloud
never came up: it brings its own networks and was the first role to hit the
ceiling.
Declare the pools beside the nested daemon's in group_vars, so both docker
daemons read their allocation from one place, and render them from there.
The base moves to 10.208.0.0/12 because 172.16/12 is exhausted: a probe
subnet in 172.28 was refused as overlapping while one in 10.210 was
created.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A container without a healthcheck is accepted the moment its process starts. The restore drill asks the host's health service for its verdict, so a service that came back empty passes as long as it is running - which is how an empty volume survives a drill unnoticed.
Twenty-one services get a probe and thirteen get a marker naming why none is possible: a one-shot client that exits, a build image with no service port, a media bridge on UDP. The flavor comes from each image's own layer history, read from the registry through the mirror, so no image had to be pulled to find out whether it ships curl, wget or a shell.
Two entries carry more weight than the rest. The central Postgres, which every replay of the recover chain lands in, had no probe at all; it gets the pg_isready line that sys-svc-rdbms already runs for the per-app engine. And the matrix runner starts systemd within a second while its inner daemon is still pulling every image of the deployment, so it counted as healthy long before anything served - it now waits for Synapse to answer.
No probe replaces one: nineteen of the twenty-one images ship no HEALTHCHECK, and the two that do already carried a declaration before this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cosmos generator linked a consumer to its provider only through an
'<role>' in group_names reference or an explicit provides: name. A hard
dependency has neither -- pinning the provider on unconditionally is
exactly what removes the Jinja reference the generator was reading -- so
web-svc-asset's bond to web-svc-file vanished from the diagram the moment
it became binding. Match a service key against the provider's own primary
key from the registry, in both directions.
Services the role turns off drew a solid 1:1 edge, claiming a mandatory
relationship for something that carries nothing at deploy time. They now
render as a red 0..0 edge; edge merging gained a real precedence so a
provider consumed through two keys stays live when only one is off.
Named colour, not hex: a '#' opens an entity code in the mermaid lexer and
turns the whole diagram into a parse error on GitHub.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A handler name was doing two jobs at once: the line an operator reads in the
play output, and the identifier a `notify:` binds to. The two pull in opposite
directions -- the log wants "restart the SSH service, with diagnostics", the
binding wants a token that survives quoting, YAML flow style and a shell
round-trip -- and a name that changes for readability silently changes which
handler runs.
Every handler that a notify reached by name now carries a `listen:` with that
exact wording, so the binding is explicit before anything is renamed. The
topics themselves become identifiers: `compose up` reads `compose-up`,
`Restart NFS ganesha` reads `restart-nfs-ganesha`, 22 of them across both
sides of every notify. Where the topic is assembled at runtime, both branches
moved together: the conditional in sys-service carries two literals, and the
LDAP renderer interpolates `import-{{ folder }}-ldif-files`, which is what
makes those handlers reachable at all.
Freed from the binding, the names take the emoji prefix every other task line
already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extra_hosts is the one compose key a service may not carry twice. The onion SSO
back-channel pin was emitted by a Jinja partial that six roles included, while
six other services wrote the key by hand; nothing stopped a service from
receiving both. YAML then either rejects the document or keeps one list and
drops the other, silently.
The partial becomes plugins/lookup/container_extra_hosts.py, and every pin now
goes through it. Callers add their own entries with extra_hosts=[...], the way
container_volumes already takes extra_volumes=, and the lookup merges them after
the automatic pin and deduplicates in first-seen order. With nothing to emit it
returns an empty string.
The swarm address reasoning from 7b1839681 is preserved and now written down in
the plugin rather than only in that commit: the pin must NOT be host-gateway in
swarm, because that alias resolves per node while openresty publishes :80 with
mode: host under node.role == manager. It carries the stack host's address,
which reaches every task because the compose file is rendered once with the
manager as inventory host. The mode is read as compose_mode_force or
DEPLOYMENT_MODE, not the bare cluster mode, so a role with an override does not
lose its pin and its overlay at once.
Two roles kept a pin that now collides with the automatic one, and both are
resolved the same way -- the hand-written entry stays for clearnet and yields
the .onion case to the lookup, which picks the address per mode. In
web-app-mailu the three conditional entries move into a role-local
mailu_extra_hosts lookup exposed as MAILU_EXTRA_HOSTS, because an eight-line
conditional expression in the template argument is worse than the block it
replaced. It returns its entries as separate results, so vars/main.yml uses
query(); plain lookup() would join them into one comma-separated string.
tests/lint/ansible/jinja/test_compose_extra_hosts_use_lookup.py fails on any
literal extra_hosts: key in a role template, so the invariant survives the next
author who needs a pin. It follows test_compose_ports_use_container_ports_lookup:
the raw construct is forbidden, the lookup is not mandated per service -- a
service with no pins is correct without it, and the swarm branch raises when
facts are absent, which would turn a no-op into a failure surface for containers
that never speak OIDC.
Equivalence checked by rendering rather than by reading: under the trim_blocks +
lstrip_blocks the stack-host action forces for compose.yml.j2, both former call
site shapes (indented, and matrix's column 0) emit byte-identical text to the
new form when the block applies. The no-op case differs -- a whitespace-only
line where the include emitted none -- and yaml.safe_load returns the same
document for both.
31 unit tests; make test-lint 325 passed. Does not touch web-app-espocrm, whose
missing pin is a separate claim and gets its own commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
force_shared_db flipped every DB service to shared with no exemption, so
web-app-magento's deliberate embedded MariaDB 11.4 - declared shared:
false with a comment warning that no bot may move the major underneath
it - became dead config and setup:install hit the central engine's 12.3.2,
outside Magento's supported range. The run log names the override:
'swarm force-shared: web-app-magento.services.mariadb -> shared=true'.
Exempts a DB service whose role meta declares a literal shared: false AND
whose role is manager-pinned. The second condition is not decoration:
utils/storage/nfs.py keeps a manager-pinned role's volumes node-local, and
InnoDB over NFS is why the harness force-shares in the first place. Of the
two roles with a literal local DB, magento qualifies and bigbluebutton
(unpinned) does not, so its behaviour is unchanged.
derive_includes._force_shared_db_view documents itself as mirroring
force_shared_db, so it takes the same exemption - without it the dep walk
would compute magento's include closure against the central engine while
the deploy runs a local sidecar.
The sidecar carried no deploy block at all, so an exempted service would
have been free-floating with one task per node over a single-writer
datadir. The block now comes from a container_deploy lookup shared with
sys-svc-container/templates/deploy.yml.j2, which replaces the replicas
ternary and the duplicated placement constraint there. Its unit test
caught a real defect while being written: with compose_mode_force=swarm
over DEPLOYMENT_MODE=compose the delegated compose_replicas returned
empty, emitting a deploy: key with no child.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both sidecar templates rendered a swarm service with no deploy: block at
all, so the kata placement constraint never applied and a non-kata
workload could land on a kata node under runsc. They also ran without
resource limits under swarm.
Include deploy.yml.j2 from both templates. Guard the replica count with
an explicit override, because deploy.yml.j2 otherwise falls back to
compose_replicas (the group size): these caches have no clustering, so a
replica per node would leave a consumer's session missing from the other
instances. Swarm already defaults to one task without a deploy block, so
the forced value preserves the current behaviour.
The central path is untouched: svc-db-redis/compose.yml.j2 reaches
deploy.yml.j2 via sys-svc-container/base.yml.j2, where the override is
undefined and the original resource_filter expression applies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extra_hosts mapped auth.<onion> to host-gateway in both deploy modes, but
host-gateway is a per-node address while openresty publishes :80 with mode: host
under node.role == manager. Every replica scheduled on a worker therefore dialled
its own gateway, where nothing listens, and the server-side OIDC token exchange
died with 'Connection refused'. Penpot 0 of run 31340965955 measured it: 60 of
the ConnectException lines on wrk-02, 20 on wrk-01, zero on mgr-01.
In swarm the pin now resolves to the stack host's routable address -- the same
destination svc-net-tor already sends all hidden-service traffic to via
TOR_HS_BACKEND_HOST. That address reaches every task, because
plugins/action/stack_host_template.py skips any host whose IS_STACK_HOST is
false and IS_STACK_HOST is true only on a swarm manager: the compose file is
rendered once with the manager as inventory host, so ansible_facts carries the
manager's IP and swarm applies the same extra_hosts to every task.
Retargeting rather than dropping the pin, because a manager-pinned consumer
(the keycloak sso-proxy) is measurably working through this very line today.
The guard fa2f6eb7e removed would have deleted it for them too.
Keyed on compose_mode_force or DEPLOYMENT_MODE, not the bare cluster mode:
container_networks resolves attachments through the same effective mode, so a
role with an override must not lose its pin and its overlay at once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every conflicted hunk resolves onto the swarm-nfs side, because on all six
files it is the strict superset: it carries the display-name codec
(utils/roles/display.py) the CI role selections now encode through, and the
sequencing glyph move from 910748ec4. svc-net-tor holds no unique content on
any of them -- runs.py even had non-conflicted hunks already calling `codec`,
so keeping its side would have left a NameError.
The lint fallout is settled in the same commit, as 4afc9b770 did for the
previous merge: the shell-or-true lint lives on svc-net-tor (c505ce746) and
swarm-nfs brings two unmarked `|| true` sites with it. Both get a nocheck
reason rather than a rewrite, because neither swallows anything -- the
slaptest call is the measured exception of c4bbbba93 and has_cn_config decides
right below it, and the dockerd restart is followed by an is-active poll that
fatals. In both the exit code was never the authority.
make quality green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One conflict, in the category catalogue: this branch added a virt category for
the sandboxed container runtimes, the other did not.
Keeping our side alone would have closed the conflict and broken the build. The
same merge introduces a test requiring every category to declare a hanzi label,
unique among its siblings, and virt predates that rule and carries none. It now
declares 沙箱, checked against its ten siblings for a collision.
The first version asked dockerd which containers it still knew and treated
the rest as orphaned. That predicate can never fire in time. dockerd keeps
reporting an orphan as running for minutes, then synthesises an exit it never
verified, so the container is either still listed as running or already listed
as exited, and `docker ps -a` covers both. On one captured manager 17 of 18
orphaned scopes carried an Exited container that was still listed, so they
were skipped; the reap found nothing and the unconditional dockerd restart
that followed took the node's tasks down for no gain.
A restarted containerd re-adopts none of the shims it left behind: across 13
observed restarts, 96 of 96 were leaked and the counts match the orphaned
scopes node by node. Age is therefore the exact criterion, and it holds the
moment the damage happens rather than minutes later. It also drops the docker
CLI from the script entirely, so a wedged daemon can no longer hang the play.
Around that: the stop tolerates systemctl's "no such unit" because the scope
vanishing is the wanted outcome, both calls are bounded so a hung unit cannot
block forever, each stopped scope is verified gone, and dockerd is polled
until it is active instead of trusting the restart's return code. A unit that
systemd no longer knows answers 0 for its start timestamp, which would sort
before every real one, so that answer is treated as unknown rather than old.
The tests drive the real script against a stubbed systemctl and cover the
selection, both filters, and every failure branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A containerd restart under a live dockerd orphans the container shims. The
new containerd does not re-adopt them, dockerd keeps stale task handles and
force-deletes the container records while the processes stay alive and keep
their file locks, so every replacement task is blocked by its own
predecessor: mariadb aborts on the Aria and InnoDB locks, slapd returns
LDAP_OTHER from the LMDB reader lock, and an exec into a healthcheck-less
service fails with "task not found".
Package versions do not reveal that state, because containerd restarts into
the version it was upgraded to and dockerd is not upgraded at all. The
signal is the unit start order: containerd entering active later than
dockerd means it came up underneath it.
Reaping is done on the systemd scopes rather than through the engine,
because dockerd has already dropped the records and its own force-kill
cannot reach the surviving processes.
Runs one node at a time: a package upgrade restarts containerd on every node
of the cluster at once, and simultaneous dockerd restarts can cost the swarm
managers their quorum.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven include templates under sys-svc-container decided how a container was
probed, and they read their inputs from template-local `{% set %}` scope.
That scope leaks: a value set for one service silently reached every service
below it, which is how taiga's front and gateway ended up probing a port
declared for a different service, and how bluesky's frontend probed a PDS
endpoint.
Probes now live in utils/docker/healthcheck.py, one class per flavor behind a
PROBES registry, and each service declares its own:
healthcheck:
flavor: curl
path: health/ready
start_period: 20m
`port_key` picks another entry under ports.internal for the services that
expose their probe elsewhere (keycloak on management, nextcloud talk on
signaling). Without a flavor the entry must carry an explicit test argv, so a
missing declaration fails loudly instead of rendering a probe against nothing.
Rendering HEAD's templates against the new output confirms 49 of 56 call
sites are byte-identical; the remaining seven are covered by an Ansible run.
Three probes change on purpose: erpnext's backend gains a check it never had,
bluesky's frontend drops the leaked PDS path, and msmtp_curl moves from a
hardcoded localhost:80 to the service's own port on 127.0.0.1 like every
other flavor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, each resolved by combining both sides rather than picking one.
compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.
daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.
The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Read platform_arch from the role config and append a node.platform.arch
constraint where it is set, so a role whose image exists for one
architecture only cannot be scheduled onto a node that could never run
it. LM Studio ships x86_64 alone and is the first consumer.
The constraint lives with the other placement rules rather than in the
consuming role, because architecture binding is a property any role with
a platform-specific image can carry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The check asserted the A record equals networks.internet.ip4, which answers a different question: under which public address the deployment answers. The two coincide in compose and diverge in the swarm lab, where dnsmasq maps the domain to the manager, so job 91358007229 failed on three healthy nodes that had resolved the name correctly.
Assert that an address came back instead. The SERVFAIL guard is unchanged, and the role no longer reads networks.internet.ip4 at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make quality wrapped the conditional NODE_IMAGE expression introduced in 823c6f4ee; the rendered value is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The check starts a container with the advertised resolver, which is exactly the path a pre-deploy build takes, but MODE_DEBUG gated it off in the swarm drill: that path loads only inventories/development/swarm.yml, which never sets it. Run 30688981182 skipped the check at 07:31 and lost the openldap_custom build at 07:44 on the one node whose dnsmasq held no UDP socket for the advertised address.
Pass NODE_IMAGE empty in swarm mode instead: node:26-alpine is 63.3 of the 65.5 MB pulled per node, and its getaddrinfo assertion is a second opinion on a SERVFAIL check the busybox half already performs. NODE_IMAGE stays mandatory to set, so omitting it remains an error while empty is a stated decision.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible's run_once inspects the first host and generalises to the whole play, so a cluster where a single node's dnsmasq never bound the advertised resolver still reads as healthy. The property is per node, and forks=25 covers them concurrently. Also require a non-empty resolver, since 'container run --dns ""' cannot work.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The template replaces daemon.json wholesale, so the data root the swarm lab puts on the chosen filesystem was dropped on every managed node. The rescue artifact of run 30688981182 shows nfs-server keeping it while mgr-01, wrk-01 and wrk-02 lost it, leaving those nodes on the container overlay instead of the prepared loop filesystem. Read the file back before rendering and re-emit the value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tcp healthcheck ends its probe with `cat <&3 | grep -q 'HTTP/1'`, which
matches the status LINE. A backend answering HTTP/1.1 500 therefore passes the
probe exactly like a 200. Swarm's only mechanism for taking a task out of the
VIP is the healthcheck, so a Keycloak replica serving 500s stays in rotation
indefinitely, and with replicas: 3 it catches roughly every third request.
That is what failed run 30586213932's keycloak job. The Playwright artifact's
error context shows the sign-out assertion waiting on the login form while the
page rendered "We are sorry... An internal server error has occurred" on
/realms/master/protocol/openid-connect/auth, and the rescue artifact's container
log shows why the replica was broken: ISPN000196, failed to recover cluster
state after becoming coordinator, followed by 25 Infinispan TimeoutExceptions,
17 of them through StateTransferLockImpl. Eight tests failed across three roles
because web-app-dashboard and web-app-prometheus authenticate against the same
Keycloak.
The status grading lands in a new template rather than in tcp.yml.j2, because
that one has fifteen other consumers whose probe paths were not audited. They
keep the behaviour they have; only Keycloak moves. The new probe reads the
status line with head -n1 and accepts 2xx/3xx only.
Keycloak also moves from health/live to health/ready: liveness answers "the JVM
is up", which a broken replica satisfies, while readiness answers "requests can
be served", which is the distinction the VIP needs.
Readiness returns 503 through schema migration, so the strict probe needs a
start period that outlives it. Measured across all five replica logs of that
run, Keycloak reaches "started" in 13.0-13.2s including migration, so 180s is
roughly thirteen times the observed boot. The deploy gate that waits for
health=healthy allows 600 retries at 2s, i.e. 1200s, so the longer start period
consumes at most an eighth of its budget.
Tool availability verified in quay.io/keycloak/keycloak:26.7.0 (digest
0f198be292568439d700cdbfb893e69a6009bb43a94a06a945b1d3d506c76b13): bash, cat,
grep and head are all present, so the probe's head -n1 resolves.
This fixes detection, not the clustering fault behind it. Why the Infinispan
cluster fails to form is still unknown; a replica that never clusters is now
pulled from the VIP instead of serving errors, but if no replica clusters there
is nothing healthy to route to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Archlinux entry in SYS_SVC_CONTAINER_DOCKER_FIREWALL_BACKEND_BY_OS_FAMILY landed in daemon.json unconditionally, and dockerd rejects `swarm init` with 'firewall-backend=nftables is incompatible with swarm mode'. Swarm only ever works on the iptables backend, which arch provides through iptables-nft, so the per-family pin now applies outside swarm mode only and compose deploys keep it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Archlinux entry in SYS_SVC_CONTAINER_DOCKER_FIREWALL_BACKEND_BY_OS_FAMILY landed in daemon.json unconditionally, and dockerd rejects `swarm init` with 'firewall-backend=nftables is incompatible with swarm mode'. Swarm only ever works on the iptables backend, which arch provides through iptables-nft, so the per-family pin now applies outside swarm mode only and compose deploys keep it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The daemon configuration named fuse-overlayfs as the storage driver for every
host it was rendered on, while the package that provides it is installed under
the opposite condition: only inside a nested container. A host that is not
docker-in-docker therefore received a daemon pointed at a graphdriver nothing
had installed, and dockerd does not start on one.
That path is reached by a bare host in the svc-virt-kata group, which is
exactly the shape the agent tier is meant to run on. It stayed invisible
because every deploy in this repository nests through the stack container, so
the condition that installs the package and the condition that renders the file
were never false at the same time.
fuse-overlayfs exists for the nested case, where overlay2 cannot stack on
overlay2. On a bare host the native default is both correct and faster, so the
key is now emitted only where it is needed and the JSON comma structure moves
with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reaching an onion address required each role to point its own http client
at the tor socks proxy, which then carried clearnet traffic too — fatal
in the test network, which has no exit, and the reason nextcloud needed a
per-host exclusion list.
The egress router already redirects tor's virtual range to the TransPort
and dnsmasq already forwards .onion to tor's DNSPort; the piece missing
was reachability. dnsmasq listened on loopback only, which docker
discards, so containers never saw it. It now also answers on the docker
bridge, the daemon points there with the clearnet resolver kept as
fallback, and the role's own onion resolves to a container-reachable
backend instead of loopback.
Nextcloud drops its proxy block and the exclusion list; every other role
gets the same behaviour without configuring anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the raw package modules at 76 call sites with package_install, moves
each role's distribution mapping into its own meta/packages.yml, and turns the
inventory PACKAGES variable from source-keyed literal names into a list of
declared package ids. The lint that forbids the raw modules is enabled here,
because the migration burns it down from 76 offenders to zero.
Removes sys-aur, sys-aur-install and sys-package. The AUR build user, the
passwordless pacman rule and the inventory-driven install are part of the plan
the action plugin executes, so the roles no longer carry them.
Declaring the mappings surfaced breakage the raw modules had hidden: on Debian
the `which` binary comes from debianutils, jrnl is in no Debian or Ubuntu
suite, chromium-browser is absent from Debian stable, JDK 11 is unobtainable on
RedHat, wireguard had no RedHat branch at all, qbittorrent was built from the
AUR although Arch ships it in extra, and desk-docker installed a Docker variant
that conflicts with the one sys-svc-container installs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sandbox node runs gVisor as its daemon-wide default runtime and every
container scheduled there inherits it, so keycloak failed to start on one.
Swarm has no taints, which leaves the inverse placement constraint as the only
lever: sandbox consumers are pinned to the labelled nodes, everything else is
kept off them. Both halves were measured on a live cluster first, because a
docker constraint has no presence test and rejects an empty comparison value:
`!= true` does match a node without the label, and three replicas under it
avoided two labelled workers entirely.
That makes the set of labelled nodes load-bearing for the whole platform, and
it was decided by accident. Without an `svc-virt-kata` inventory group the
service loader runs the role on every host, so every worker becomes a sandbox
node and the rest of the platform is evacuated onto the stack host. The role
now refuses to run in swarm mode until the operator names those nodes.
With the group present the opposite gap opened: the provisioner puts every role
group on the single `--host` it is given, so the role lands on the manager
alone, the label task skips itself there, and no node ends up labelled. The old
assert only proved that a worker exists, which says nothing about where the
role ran, so the deploy passed the check and then hung until the convergence
wait gave up. The assert now reads the labels back from the stack host.
`docker node ls --filter label=` matches engine labels, not swarm node labels;
the filter has to be `node.label`.
The constraint itself is emitted only where a sandbox tier exists, so a deploy
without an agent role is untouched by any of this, and the lab dedicates a
single worker: labelling both would leave the manager as the only home for
every other service.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`svc-virt-kata` is pulled in as a shared service of the agent roles, so it also
executes on hosts that are in no inventory group of that name. Gating the
runtime registration on `'svc-virt-kata' in group_names` therefore left such a
host with runsc installed and its node labelled kata-capable, while
`daemon.json` registered nothing and the daemon kept running everything under
runc: the exact isolation breach the label promises is closed. The stat probes
in `daemon_config.yml` decide instead, and they cannot be fooled by an
inventory that says otherwise.
Nothing else in the tree installs or references runsc, so a shim on disk is
proof that this role put it there.
The agent compose templates pin their runtime on the same `services.kata`
switch that already drives the swarm placement constraint, instead of pinning
unconditionally against a daemon config written under a different condition.
Both roles declare that key themselves, so the lookup is the strict form: a
deleted flag now aborts the render rather than silently starting the agent
without a sandbox.
The new unit test covers what nothing covered before, which is how the runc
breach went unnoticed: an absent shim is never registered, a default runtime is
always a registered one, the stack host keeps the shared kernel, compose never
sets a daemon-wide default, and registration follows the probe rather than the
inventory group.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
