Collapse the per-distro DNS test matrix into a single job that runs each
triggered distro in its own systemd Docker-in-Docker container, built from
that distro's prebuilt CI image, all in parallel. Each container installs
and starts a local docker daemon, brings the dev stack up and runs the
existing dns.sh suite unchanged. The local daemon makes the dev compose
bind-mounts resolve against the baked repo, and the per-container network
namespace keeps the static CoreDNS IP and subnet from colliding across
distros. GITHUB_ACTIONS=true and INFINITO_BUILD=0 pin the deploy to the
pull-and-run path (no cache stack, no local image build); dig is installed
unconditionally so arch keeps the host-query assertions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A transient Debian/Ubuntu mirror fetch aborts a docker build layer with exit 100 (it killed the web-app-nextcloud swarm openldap build while the same image built fine elsewhere). New lint under tests/lint/docker/dockerfile/apt/ flags any RUN with apt-get update/install that omits Acquire::Retries; retrofit all 15 apt Dockerfiles with -o Acquire::Retries=3. Both apt lints move into the apt/ subpackage. dnf retries by default and apk has no equivalent flag, so both stay out of scope.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
meta/services.yml carried a '# nocheck: single run-once seeder' marker whose token did not match the single-replica rule, so the seeder replicas:1 pin read as unjustified. Correct the token to single-replica (reason unchanged). test_plan.py is an autoformat reflow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The provision include used a templated path, which the container-exec lint cannot follow, so its argv-form 'container exec' into the OpenProject container read as ungated and would fire on swarm workers (resolver exits 64). Switch to a static include the lint follows and carry IS_STACK_HOST on the include when, so workers skip the whole coupling as a unit. Also picks up the autoformat reflow of the diagnostics daemon-journal capture.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A prior deploy on a reused host leaves DIR_VAR_LIB ephemerally mounted (no fstab, but live). When the recomputed client src differs (flavor path or server IP), state: ephemeral refuses to override it and retries to failure. Read the live source with findmnt and unmount only when it is a non-empty differing src, then mount; a correctly-mounted src is left for the module to remount.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The repair services (rpr-container-soft/hard) emit a syslog 'infinito-kill' marker before they down/restart containers; the rescue collector captures containers/_kill-markers.txt (plus a 6h docker/containerd daemon journal) so a failure artifact names which service stopped containers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A missing bind-mount source is auto-created as a directory, so the trust helper / php.ini wrapper copy nests inside it and the entrypoint execs a directory. Stat the two SPOT sources and delete any that is a dir before the copy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Nested-attr access on the lazy vars-lookup dict (_cert.files.cert, _tls.domains.primary) raises under Ansible 2.19+. Use the want-path lookup form so the vars hold resolved scalars.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
3.1.3 wraps the postgres dump replay in a single transaction, preventing the concurrent-writer duplicate-key abort that failed the swarm restore drill for the discourse-backed nextcloud/wordpress variants.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mattermost and openproject integration probes plus the openproject provisioning execs addressed the partner by its bare compose container name. Under swarm the running task name differs, so container inspect found nothing, the partner-running probe returned false, and the bidirectional OAuth provisioning was skipped. Route them through the container_address lookup (bare name in compose, resolve-container-id subshell in swarm) so the probe resolves and provisioning fires in both deploy modes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collapse the 5-distro matrix in the make and pkgmgr install-test workflows
into a single job that runs every distro in parallel inside extracted
scripts, cutting 5 runners down to 1 per workflow. Each distro installs
against its own copy of the checkout so the concurrent in-container builds
never race on the shared tree. The distro list is read from the
INFINITO_DISTROS SPOT in default.env instead of being hardcoded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the plan table out of the standalone orchestrator Plan job into each
deploy mode's discover job (swarm/compose/host), so the table shows as a
run-summary spot on the orchestrator that will run it. Drop the redundant
Priority column since the Triggered column already carries the star.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The swarm front-proxy pointed at openproject_web on the entity's ports.internal.http (80), which the Apache proxy sidecar and its compose publish mapping use, but the Puma web service listens on 8080, so OpenResty got a 502. Add services.openproject.proxy.internal_port=8080 and resolve front_proxy_internal_port per-app from services.<entity>.proxy.internal_port (mirroring how service_key already resolves), so it stays entity-scoped and cannot leak into co-located apps like the reverted proxy_internal_port var did. Compose is unaffected (resolve_upstream uses the local published port there).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The seeder (./docker/prod/seeder) is a run-once job, but with no replicas set it resolved to the swarm default (3). Concurrent seeders race on one Postgres (duplicate-key), fail and restart, so the stack never converges (not converged: openproject_seeder). Pin replicas: 1; it seeds once and exits, leaving a Shutdown+Complete task that stack_ready.sh counts as converged. web/worker stay at the default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Embedded Solr holds an exclusive Lucene index lock and the distribution-flavor install is not concurrency-safe, so multiple replicas over one Postgres and one NFS Solr index dir collide (xwikidoc_pkey duplicate key, LockObtainFailedException) and the superadmin probe gets HTTP 500 from an uninitialized replica. Pin replicas: 1 like jellyfin/mastodon/collabora; swarm reschedule keeps failover via the shared NFS state. Raise mem_limit 2g to 3g for the single JVM's working set.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The guide deploys to a .localhost target, where letsencrypt cannot issue (Cloudflare DNS-01 needs a real domain). Default TLS_MODE=self_signed in the generated Production block so the guide uses the working self-signed provider. Document every deploy parameter as a top-level shell var (APP, HOST, TLS_MODE, SSH_PUBLIC_KEY) instead of an inline placeholder.
The compose guide now extracts and runs the whole README block, sourcing params from it (no more -e injection); HOST is forced to localhost. Skip checkout/login under ACT so the guide workflow is iterable locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The tls resolver no longer exposes files (that moved to the cert lookup), so the letsencrypt provider's _tls.files.cert/key raised 'object of type dict has no attribute files'. Resolve cert/key from _cert via lookup('cert', application_id), the same paths sys-front-tls renders into the nginx ssl config. Declare _tls/_cert in vars/main.yml (lazily resolved) instead of set_fact.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ci trigger (--failed) and status commands matched deploy jobs only by
the compose/swarm glyph, so host-mode jobs (svc-storage-nfs-server etc.)
were invisible and re-triggering a failed run never picked them up. Add
host to the mode set and read the mode glyphs from the symbol glossary
(single source of truth) instead of hardcoding them. Align the host deploy
job titles to the glossary glyph 💻 (were a rogue 🖥️ absent from the
glossary) so they match, with a leading space like compose/swarm.
Priority (⭐) and regular (🔁) jobs were already both discovered (the match
ignores the line prefix); host now works in both. Past runs titled 🖥️ will
not match; a fresh run does.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reverts b9a5aa816. It rewrote the Mattermost OAuth provisioning to
'container exec ... bash -s', but the mattermost-team-edition:11.9.0 image
has no shell at all (verified: 'stat /bin/sh: no such file'), so the task
fails rc127 in BOTH swarm and compose (run 29715516546, web-app-nextcloud 3,
jobs 88269803451 + 88270969917). Restores the previous provisioning form.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Back to the pre-swarm-nfs-fix baseline:
- 5c2c8f531 + 985ec7eab: the swarm writer-quiesce DR drill. Deep-dive
attribution of CI run 29715516546 found it the dominant failure source
(17 of 23 jobs): [9b/9] handed-off generation mismatch, [6/9] quiesce
stack-rm network race, [4/9] remote-2-local pull stall, compose discourse
restore duplicate key.
- d6f980574: openproject swarm ingress to web:8080. Carried a leaking
proxy_internal_port lookup (no per-app default) that broke the seaweedfs
dependency front-proxy ('Key web not found'); the bypass-the-Apache-proxy
approach was never verified against downloads/X-Sendfile.
- 1dd153bad: xwiki replicas:1 + mem 3g. Did not resolve the xwiki superadmin
failure (still red in run 29715516546), so it is unverified.
Re-approach each from a clean baseline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up corrections to b449f8157 (swarm DR drill):
- export test.env in 06/07 via set -a so the child db restore/probe
scripts inherit BKP_TEST_DATABASES_CSV and BKP_TEST_RESTORE_BIN
instead of aborting at their :? guards before any restore runs
- move db_probe.sh/db_restore.sh into a db/ subfolder (probe.sh,
restore.sh) and update all callers, so they stop tripping the
prefix-grouped-files structure lint
- define TEST_E2E_SWARM_DRILL in group_vars/all and drop the now
redundant default() from test.env.j2 (satisfies the undefined-var
and spot-default lints together)
- fix a mermaid parse error in the drill README: a ; in a
sequenceDiagram note was read as a statement separator
make quality-high green (test + lint phases, 15 targets, 0 failures).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Defer Swarm SQL replay to the disposable DR drill so shared writers such as Discourse are stopped before the destructive restore. Verify the selected generation with before/after database probes while preserving the Compose restore path.
In Swarm the front proxy targeted the Apache proxy service (:80), which
was unreachable on the overlay, so OpenResty returned connection refused
and the probe got 'No server is currently available'. Puma web was
healthy on :8080 the whole time. Set proxy_internal_port to the web
service internal port (8080) in Swarm so sys-stk-full proxies straight
to Puma over overlay DNS, bypassing the Apache hop; Compose keeps the
published-port Apache path (empty proxy_internal_port). Document the
compose-vs-swarm routing split in the role README.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Embedded Solr requires an exclusive Lucene index lock and the
distribution-flavor install is not concurrency-safe, so multiple
replicas sharing one Postgres DB and one NFS Solr index dir collide
(xwikidoc_pkey duplicate key, LockObtainFailedException) and the
superadmin probe gets HTTP 500 from an uninitialized replica. Pin to
replicas: 1 like jellyfin/mastodon/collabora; swarm reschedule keeps
failover.
Also raise mem_limit 2g to 3g: even an uncontended replica sits at
1.694GiB/2GiB (84.68%) at rest from the JVM working set, so 2g leaves
too little headroom (jellyfin runs 3g too). pids_limit stays 1024: a
single replica used only 224 threads, the 1021/1024 hits were purely
the contention storm that replicas: 1 removes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The README Production block went provision -> deploy with no admin-key step, so user-administrator failed early (authorized_keys empty) for both the Guide CI jobs and any real user copying the README. Add a visible 'provision --vars' step that sets users.administrator.authorized_keys (merged into host_vars) in both Production variants of the role README template, regenerate all role READMEs, and replace the invisible CI payload splice with a visible substitution of the documented <your-ssh-public-key> placeholder.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
allowed_ips was derived only from svc-swarm-node member IPs, so single-node/host mode (empty group) produced an empty list and the 'at least one allowed IP' assert hard-failed on localhost. Fall back to the host's own default_ipv4 (or 127.0.0.1) when the swarm-node list is empty; swarm mode keeps the member IPs unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Floating collabora off the manager (c81d78ee) added delegate_to: COLLABORA_HOST_NODE to two inline coolconfig tasks. Ansible templates delegate_to on every play host before when: IS_STACK_HOST skips the workers, and the fact is only set by the manager-only resolve_host_cid include, so the workers fail with 'COLLABORA_HOST_NODE' is undefined (collabora-0 swarm). Move the two tasks into 02_coolconfig.yml and pull them via include_tasks; a dynamic include is not materialised on skipped hosts, so delegate_to only templates where the fact exists.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The p88f3f6b etherpad image ships ep_redis_publisher@0.0.3 whose retry() calls logger.error() with no logger in scope, so any redis reconnect throws ReferenceError and kills etherpad, which fails the shared hlth-container gate (observed on bbb-0 swarm and, collaterally, the nextcloud bundle that pulls the same image). Vendor the plugin index.js verbatim from the image plus a console-backed logger shim, and bind it read-only over the plugin file via meta/volumes.yml + stack_host_copy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The flavor:compose `synapse.conf` template builds its proxy target from proxy_service_key + proxy_internal_port, which were never supplied, so the synapse homeserver domain proxied to an empty upstream (variant 2 only). Supply synapse + MATRIX_SYNAPSE_CONTAINER_PORT, matching the sibling front-proxy include.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The upgrade ran via `container exec` into a live-serving odoo: serving backends hold RowShareLock while the migration DDL wants AccessExclusiveLock, so `-u all` and the workers deadlock. The install task now pauses crons, touches a marker and restarts; the entrypoint runs `-u all --stop-after-init` as the sole DB actor under a pg advisory lock, marker-gated, before serving. The marker path lives in one source (vars/main.yml), fed to the container env and the touch task.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
swarm-exec reaches a DinD node; reaching a swarm service container (moodle,
openldap) needed a nested docker exec. Add swarm-app-exec, which resolves one
running replica on the node and runs a base64-encoded command inside it, so
live in-container inspection stays on a sanctioned make helper. Adding a
second exec_* script tripped the prefix-group lint, so exec_node.sh moves to
exec/node.sh and the new one lands at exec/service.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An onion mail hostname (mail.<56-char v3>.onion = 67 chars) overflows the
X.509 commonName ASN.1 limit (64), so openssl req aborts and the stack never
starts. Carry the full hostname in subjectAltName (no length cap, and the only
field modern TLS validates) and fall back the CN to DOMAIN_PRIMARY when the
hostname exceeds 64 chars, matching the CA cert convention in group_vars/all/02_tls.yml.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
write_dotenv truncated .env in place, so a concurrent make dotenv (parallel
make test regenerates the shared file once per target) could expose a torn
half-written file to a sibling reader. Write a per-pid sibling temp with the
normal umask-respecting mode, then atomically rename it over .env.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
INFINITO_TEST_TYPE is a per-invocation label the Makefile sets differently
per parallel test target (external/integration/lint/unit), not a persistent
default. Registering it as a static passthrough key wrote a per-target-varying
value into the shared .env that make test regenerates concurrently, which
tears the file for the env-file schema lint. Drop it from default.env and the
passthrough handler and suppress the run.sh reference with the
makefile-supplied nocheck that lines 15 and 27 already use, hoisting the junit
path into a variable so the continuation line can carry it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Autoformat collapses now-fitting multi-line calls in the runtime model,
render helper and their unit tests. Format-only, AST-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
