engine returns password='' for a non-shared redis. The old template rendered gitlab_rails['redis_password']='' (an empty Ruby string, which is truthy), so GitLab omnibus sent an AUTH with an empty password to a passwordless redis and failed at boot. Render Ruby nil (no AUTH) when the password is empty; the with-password path (ruby_dq) is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Swarm converge deadlocked because the curl -f healthcheck never passed on an uninstalled wiki (HTTP 500 on an empty DB), so the task never became a running replica and the post-converge schema install never ran. Use the tcp liveness healthcheck (passes on any HTTP/1 response) so convergence gates on container liveness; the ansible install and playwright e2e assert application readiness.
The OIDC extension flow execed the container from STACK_HOST via the manager-only resolve-container-id helper, which yields a non-exec-able remote container id when the task lands on a worker. Route extension install, composer, the sanity check and the DB update through the resolve_host_cid SPOT (MEDIAWIKI_HOST_NODE + MEDIAWIKI_LOCAL_CID, delegate_to the hosting node). Running the update one-shot on the hosting node reuses the real NFS-backed named volume instead of auto-creating an empty local one, and keeps it isolated from the live task cgroup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fpm/cron occ-status healthcheck rendered a 30s start_period, far shorter
than the (image-serialized) first-boot init, so swarm killed every task with
exit 137 mid-init and the stack never converged. Give both healthchecks a 10m
start_period and raise swarm_stack_converge_retries to 600.
The nextcloud image already serializes first boot across replicas (its own
init lock), so unpin fpm from replicas: 1 and let it run on every node; keep
cron pinned as a genuine singleton. For correct multi-instance runtime, follow
the Nextcloud caching recommendation: memcache.local => APCu in both modes,
memcache.distributed and memcache.locking => Redis in swarm, backed by an
explicit redis connection array (which also fixes the RedisException at occ
upgrade, where OC\Memcache\Redis previously fell back to 127.0.0.1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The shared tcp healthcheck grepped the response for the literal 'HTTP/1.1',
but odoo's werkzeug server (and port-ui) answer HTTP/1.0, so the probe never
matched despite a 200 response; swarm then killed the task as unhealthy and the
stack never converged. Grep for 'HTTP/1' so both 1.0 and 1.1 status lines pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The container-exec-resolver lint only saw the inline shell form, so the
`command: argv: [container, exec, <name>]` list form (two separate lines)
slipped through, which is how checkmk's bare name reached swarm. Extend the
scanner to parse argv sequences via yaml.compose_all with node line numbers,
and annotate the nine legitimate bare-name execs (mariadb recovery container,
matomo compose-only debug, probe-gated nextcloud partner provisioning, pixelfed
runtime-resolved CID, wordpress launcher-managed discourse) with
# nocheck: container-exec-resolver.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every `container exec {{ ODOO_CONTAINER_ADDRESS }}` (DB init, module install,
update) ran on the manager against the bare compose name. Under swarm the odoo
task sits on a worker, so the manager-local exec silently no-op'd behind
failed_when: false and auth_oauth was never installed, surfacing later as
"relation auth_oauth_provider does not exist". Resolve the hosting node + local
container id, delegate the execs there, drop failed_when: false in favour of
real error detection (rc + 'invalid module names'), and make the /web/health
wait SIGPIPE-safe with a per-attempt timeout. Restarts split into compose
(container restart) and swarm (service update --force); DB init extracted to
01a_bootstrap_schema.yml so the parent block stays within the size limit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
configure-auth.sh ran `container exec/restart jellyfin` with the bare compose
name, which does not exist under docker stack deploy. Split the script into
install and configure phases (JELLYFIN_PHASE), target the runtime-resolved CID
(JELLYFIN_CT), and drive restarts from the task (container restart in compose,
container service update --force in swarm). The phase flow moves into
01a_auth.yml so the parent block stays within the size limit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The OIDC admin provisioning ran `container exec {{ SEMAPHORE_NAME }}` with the
bare compose name, which does not exist under docker stack deploy. Resolve the
hosting node + local container id and delegate the execs there. The three
tasks that shared the IS_STACK_HOST + SEMAPHORE_OIDC_ENABLED guard move into
02_oidc.yml, included once under that condition.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The readiness wait and the SSO/LDAP configure step ran `container exec
{{ CHECKMK_NAME }}` with the bare compose name. Under docker stack deploy the
task container is auto-suffixed (and may sit on a worker), so the exec failed
with 'No such container: checkmk' on every retry. Resolve the hosting node and
local container id via resolve_host_cid and delegate both execs to that node;
compose keeps the bare name as a no-op.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two stacked defects broke keycloak object updates on the swarm-delegated
cross-node path:
- get_merged_users cached the templar-less raw_users payload (fetched by
get_merged_applications) under the same key as templar-backed lookups, so
every later lookup('users') in the process returned UNRENDERED values. The
literal '{{ 42 | strong_password }}' / '{{ ansible_become_password }}' then
reached kcadm and swarmkit, failing the realm password policy and the stack
env validation. Key the cache on templar presence so rendered and raw
payloads never share an entry.
- The cross-node recovery block re-included itself with a self-referential
'kc_update_attempt' template var, which Ansible 2.19 expands recursively
('Recursive loop detected: maximum recursion depth exceeded') on the second
attempt. Advance the counter with an eager set_fact so each level reads a
concrete int.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rewrite the operator-drafted additions in machine-coherent MUST language:
15-minute polling of newly failed jobs, 99% root-cause certainty before
fixing, no commit/push/ask commands, and a verified-fix completion gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
12_swarm ran 'make swarm-zombie' which invoked act, but act was never installed in the runtime container, giving 'act: command not found' (Error 127). Add an on-demand installer (scripts/install/act.sh) fronted by 'make install-act' (plus 'make install-act-update' to force the latest), and make the act-based targets (act-workflow, swarm-zombie) depend on it. The static release binary runs on all five default distros; verified on debian, ubuntu, fedora, centos, manjaro.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the per-role service_replicas Jinja variable with a per-service
replicas: field in meta/services.yml. deploy.yml.j2 resolves it via
resource_filter (with entity fallback), the same path already used for
cpus/mem, so services.yml is the single source of truth for replica
overrides.
- 13 web roles migrated to per-service (or entity-level) replicas: 1
- matomo bootstrap -> replicas: 0, mastodon migrate -> replicas: 1
- minio/seaweedfs provider entries -> replicas: 1, which also pins the
central single-writer object-store providers instead of relying on the
topology default
- service_replicas removed entirely; dropped its dead reference from the
compose-structure lint allowlist and updated requirement 023 wording
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dump docker service inspect (UpdateStatus plus the resolved TaskTemplate)
and docker node ls, and widen the docker journal window to 10 minutes, so
a non-converging stack reveals rolling-update loops, node flaps, and the
desired-state changes that the previous 3-minute service-ps-only dump did
not capture.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The tcp and wget healthcheck templates set no start_period (so it
defaulted to 0), and nc/http hardcoded it, meaning a role's
container_healthcheck_start_period was ignored for every probe except
curl. Odoo sets 5m but ran with a 0s grace. Make all templates honor the
configured value while keeping each template's previous default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OIDC front plugin builder hardcoded FROM node:16-alpine. Source it
from services.front.image / services.front.version so the base is
config-driven and reachability-checked, and pin taiga-events/protected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace base_image: ubuntu:22.04 with image: ubuntu / version: "22.04"
so the runner's base image is discoverable and reachability-checked;
RUNNER_BASE_IMAGE now composes the two.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The broker set image: bluesky-broker (a local build name) and carried
the pullable base in base_image/base_version, so the base was never
version-checked. Move the base to image: node / version: 22-alpine and
source the build args from it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace version: latest with the newest stable semver tag across 21
roles (node pinned to the 24 LTS where a rolling latest broke the
build). The weekly update workflow now inspects these pins and proposes
bumps via PR instead of silently tracking a moving tag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a generic Docker Registry v2 client (WWW-Authenticate bearer-token
challenge) so tag listing and reachability work for any registry (Docker
Hub, GHCR, MCR, Quay, GitLab/opencode), and route the version updater
through it so quay.io and opencode.de images auto-update like the rest.
Add an external test that hard-fails when a pinned image:version tag is
unreachable (404) and warns on incomplete image/version pairs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Checkmk's OMD site crashes at boot with fcntl.flock raising OSError
[Errno 9] Bad file descriptor: NFSv4 ignores the nolock mount option and
routes flock() to the server. Force flock() client-local on v4 so
single-writer apps (checkmk, openldap) boot; POSIX locks stay
server-side. NFSv3 keeps nolock (statd-free, flock already local).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sys-svc-container healthcheck templates (curl/wget/nc/tcp/http) now
derive container_port from services.<service_name>.ports.internal.http,
so a role no longer needs to set container_port only to feed the shared
healthcheck.
Add the container-port-set-unused lint that fails a role setting
container_port without using it elsewhere, then resolve every flagged
role: drop the redundant assignment where the derivation reproduces the
value, or mark it with nocheck where the port is non-http, a constant, or
keyed on a service the derivation cannot reach.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The web container's listen port (8080) was defined as
services.openproject.ports.internal.web on the front/proxy service
block, so that service carried both http (80, the proxy) and web (8080),
and the web service itself had no ports. Move it to
services.web.ports.internal.http and read it via the web service key.
Same value, nothing else referenced the old key; the proxy still reaches
the backend by hostname (APP_HOST) on the image's default port.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Byproducts of `make autoformat`: add YAML document-start markers to
svc-runner/test-e2e-cli meta files and rewrap the jellyfin galaxy_info
description. No semantic change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
HSTS (Strict-Transport-Security) is an HTTPS-only header: a browser ignores it
over plain http (RFC 6797) and OpenResty only emits it on the TLS vhost. On an
onion node TLS is off and every domain is served over http, so the inline
`expect(headers["strict-transport-security"]).toBeTruthy()` in 33 specs failed
with `Received: undefined`.
Route all 33 through the shared expectHstsWhenTls(headers, baseUrl, label)
helper (personas/utils/tls.js), which requires HSTS over https and asserts its
absence over http. On a TLS node the base URL is https, so the assertion is
identical to before; on an onion node the same spec now passes. HSTS logic
lives in one place (SPOT) instead of 34 inline copies.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The key/torrc tasks notified the sys-svc-compose "compose up" handler, which is
only registered when this role's later include_role of sys-svc-compose runs.
When svc-net-tor is loaded through sys-service-loader, its per-service
`meta: flush_handlers` fires before that include, so the notification resolves
to an unregistered handler and the play aborts ("handler compose up not found").
Register the key/torrc results instead and force-recreate the Tor compose
project inline when either changed (sys-svc-compose only recreates on
compose/env change, and the bind-mounted torrc is re-read at startup). No
cross-role handler dependency, so loader ordering can no longer break it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a `tor_ports` lookup that collects every single-int `ports.public` entry
from the deployed roles' meta/services.yml and returns HiddenServicePort
mappings (`127.0.0.1:<port>`). UDP-only categories (relay, media) are excluded
since Tor forwards TCP streams only. Wire it into TOR_ONION_EXTRA_PORTS
(19_tor.yml) alongside the SSH port so mail (25/143/465/587/993), gitea/gitlab
SSH, ldaps, matrix federation and STUN/TURN control ports are reachable at
<node>.onion. Register Mailu's mail endpoints under ports.public and add the
matching PORT_BANDS entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sys-svc-compose-ca (required_by web) and sys-ctl-mtn-cert-deploy
(required_by web-app-mailu) were skipped at their include sites via a
TLS/self_signed `when:`. On a TLS-off node (e.g. .onion) the skipped
include_role leaves no role-named TASK block, so the post-deploy
required_by validator (cli/meta/roles/services/called) reports the role
as "did not execute" and exits 2.
Move the applicability decision into each role: an always-run debug marker
when not applicable (leaving the non-skip TASK block the validator needs)
plus the real CA/cert work gated behind the same condition. Clearnet
self_signed nodes still run the real work; TLS-off nodes pass with the
marker. Adds sys-svc-compose-ca/vars/main.yml (_compose_ca_applicable SPOT).
Also flush the deploy phase-header prints so they stream in order instead
of being buffered until after a failure (which misattributed the failing
pass).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The between-rounds vhost purge computed which `servers/{http,https}/<domain>.conf`
files to remove from the domain, falling back to the hardcoded default
`infinito.example` when the DOMAIN env was unset. On a node whose
DOMAIN_PRIMARY differs (e.g. a .onion node, where DOMAIN_PRIMARY is driven
by INFINITO_DOMAIN), the purge computed non-existent filenames and removed
nothing, leaving stale vhosts. The next matrix-deploy round's health probe
then hit those orphaned vhosts as HTTP 502.
Fall back to INFINITO_DOMAIN (the dev harness' domain SPOT) before the
default so the purge targets the real vhost files on any node. Precedence:
explicit arg > DOMAIN > INFINITO_DOMAIN > default. Covered by 4 new tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The docker journal/events dump added for convergence-failure debugging
tripped the repo's raw-docker lint: 'docker events' is not in its
allowlist, and the lint also matches the literal string in the echo
messages. Call the container wrapper ('container events', which passes
through to docker) and drop 'docker events' from the surrounding echoes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a lint that rejects hardcoded container_port literals and the
host-published ports.local port; container_port is the port the container
listens on, so it must resolve from
lookup('config', application_id, 'services.<svc>.ports.internal.<name>')
or a constant built that way. Bring 42 roles into compliance: move each
literal into services.<svc>.ports.internal.<name> and reference it via the
lookup, using entity_name/service_name where they equal the service key.
Values are preserved (mattermost/prometheus pointed at ports.local with the
same value as internal). Drop the dead container_port from fider and
prometheus, and the narrative comments the edits pulled into the comment
lint's changed-file scope.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The convergence probe only echoed the non-converged service name, so a
healthy-but-churning task (rescheduled with no error) could not be
diagnosed from the CI log. On the failing path emit 'docker service ps
--no-trunc' (reschedule ERROR column), 'journalctl -u docker' and
'docker events' for the last 3 min. The converged path exits before this,
so successful deploys are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
03_upgrade.yml invoked NEXTCLOUD_DOCKER_EXEC_OCC via the 'command'
module, but in swarm the container address is a $(resolve-container-id
...) substitution that only a shell evaluates. 'command' passed it
literally, so docker reported 'No such container: $(...)' and the deploy
failed. Switch the seven tasks to 'shell', matching 02_config.yml and
every other nextcloud task file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous commit read 'producer_rc=${PIPESTATUS[0]}' then
'consumer_rc=${PIPESTATUS[1]}', but the first scalar assignment resets
PIPESTATUS to a one-element array, so the second read hit an unbound
index and 'set -u' aborted the collector with exit 1 - failing every
swarm job whose deploy and tests actually passed (jenkins, pixelfed).
Snapshot the whole array in one statement, then index it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
While rspamd's auth backend warms up during a deploy, mailu answers
'454 Temporary authentication failure'. msmtp maps every auth failure to
exit 77 (EX_NOPERM), which the health script treated as a permanent fault
and failed on immediately, so whichever node's check raced the warmup
window failed the deploy (fediwall swarm: only wrk-01, while the manager
and wrk-02 passed with the same config). Capture msmtp stderr and retry
exit 77 only on the transient 454 reply, still failing fast on a
permanent 5xx.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Swarm config and secret objects are single files, so 'docker stack deploy'
aborts with "is a directory" when a 'type: config'/'type: secret' mount
targets a path ending in '/' (the nextcloud infinito_conf bug). Add a lint
that scans meta volumes files and flags such mounts, with a
'config-secret-directory-target' nocheck escape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
