GitLab's migration on its dedicated postgres sidecar failed with "ERROR: out of shared memory" and the hint to raise max_locks_per_transaction. The default of 64 locks per transaction cannot cover a schema that locks thousands of tables and indexes in one transaction: a reproduction on postgres:18 failed at table 7453 with the default and held 20000 tables at 256.
The value sits next to the other postgres tuning constants in group_vars/all/17_resource.yml and reaches both the central svc-db-postgres command and the per-app sidecar. The sidecar's new command keeps the image entrypoint and only adds the flag. The shared lock table grows to about 256 x max_connections entries, roughly 30 MB at the 400-connection cap.
Ported from 7690d2ca6d on feature/store-i18n, which kept the constant in the resource lookup that main no longer has.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
postgis/postgis publishes amd64 only, so every postgres consumer failed
on arm64. The official postgres image serves both architectures and
already ships the contrib extensions bloom, pg_trgm and unaccent.
PostGIS moves into the postgres_libraries registry and is installed from
PGDG (postgresql-18-postgis-3) only for a consumer that declares it as
an extension, which is mobilizon alone. apt update reads every source
list again, because the PostGIS package depends on GDAL from Debian
main, which a PGDG-only update left without an install candidate.
Postgres 18 keeps its cluster in /var/lib/postgresql/18/docker and its
entrypoint treats a volume at the former /var/lib/postgresql/data as an
old database, so the central and the dedicated data volume both mount at
/var/lib/postgresql. Existing 17 clusters are not migrated: their data
directory trips the entrypoint's old-data check and the container
refuses to start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f78ebe46a4)
start_interval is the cadence while start_period runs, and only while it runs.
Once the window closes the probe falls back to interval, so a service that
becomes ready just after it pays a whole tick anyway and the cadence never
reached it. Six of the seven flavors were shaped that way, Curl worst: a thirty
second window against a one minute interval.
Measured across 151 converge waits in four job logs, 63 sit exactly on a tick
boundary, 36 in the thirty second step and 27 in the sixty second one.
Attributed by application, web-app-seaweedfs is the largest at 1501s over
sixteen waits with a median of 57.7s: it probes with curl and fell straight into
the gap. web-app-zammad probes with connect, whose window is fifteen minutes,
and its 154s median is real boot time with nothing to recover.
So the window moves to ten minutes and the steady state interval to two, and the
flavor overrides that sat below them go: Curl, Wget and Connect lose interval
1m, Http its twenty second window, Nc its ten. Connect keeps fifteen minutes.
Neither number costs anything while a container works. start_period delays
nothing, it only stops failures counting, and interval no longer touches
convergence at all since that hangs on the window alone; it is steady state
load, which drops from 120 probes an hour per service to 30.
Forty six role files follow. Twenty seven declarations go that repeat the new
default or undercut it, an undercut window being the defect itself. Eight
windows longer than ten minutes keep their value and now say why. The literal
blocks and the three Dockerfiles this repository builds spell every timing out,
because outside the lookup an omission takes docker's value and not the
platform's.
A fourth lint rule keeps the shape: start_period must outlast one interval,
checked against the effective values, so raising interval without the window
cannot quietly bring the cliff back.
Two faults in the lint surfaced while building it and are fixed here. It
compared timings as strings, so 600s and 10m read as different values, and it
matched the healthcheck key exactly, so a line carrying a trailing nocheck was
skipped whole, which is how the two swarm-init partials went unexamined.
One regression, bounded. A container that starts and never turns healthy is
neither rejected nor errored, so stack_ready.sh only calls it fatal after three
killed tasks, and each cycle now spans the longer window. That reaches twenty
minutes rather than six, capped there by swarm_stack_converge_retries at 600
with a two second delay, and it lands only on runs that were failing anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A healthcheck written outside the container_healthcheck lookup inherits nothing
from utils/docker/healthcheck, and docker offers nowhere to put a replacement.
daemon.json carries storage, logging, DNS, registries and runtimes, and no
health key at all, so an omitted timing lands on docker's own value rather than
the platform's. The two disagree where it matters most: docker starts a
container with no start period, which leaves start_interval no window to act in.
That is why the hand-written blocks never gained the cadence.
The three images this repository builds itself now carry it. openldap and
postgres also drop to the platform interval, and bluesky gains the start window
it never had:
svc-db-openldap/files/Dockerfile --interval=30s ... --start-interval=5s
svc-db-postgres/templates/Dockerfile.j2 --interval=30s ... --start-interval=5s
web-app-bluesky/.../Dockerfile --start-period=30s --start-interval=5s
Six literal blocks move to the platform interval as well: magento, redis,
elasticsearch, matrix and both sys-svc-rdbms partials. Since start_interval
exists, interval no longer touches convergence and is steady state load alone.
Sixteen values that stay where they are now name the mechanism that keeps them
there. Two are worth reading. web-app-kix overrides start_period and nothing
else on purpose, letting the image's own HEALTHCHECK supply test, interval,
timeout and retries; filling those in would have overwritten what upstream tuned.
The memcached and redis swarm-init partials probe every second and deliberately
run with no start window, which is why start_interval is absent rather than
forgotten.
The rest name their probe: occ boots the whole Nextcloud stack, tor re-parses
its torrc instead of asking a running service, opensearch is still forming a
cluster, matrix waits on a file another step writes to a share.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
42ebd9ea71 gave the healthcheck SPOT start_interval, the cadence docker uses
while start_period is still running. A block a template writes out itself
bypasses the SPOT and never gained it. Measured across the repository: 46 such
blocks, 40 of them already carrying start_period, not one carrying
start_interval.
Without it a probe fires only every interval, so a service ready after two
seconds is reported healthy at the next tick and a stack converges no faster
than its coarsest quantisation. That lands in sys-svc-compose's swarm converge
wait, 1551.7s per CI job over 38 stack deploys.
Twenty five of the blocks move into meta/services.yml and render through
container_healthcheck. Each keeps an explicit test rather than picking a flavor:
a flavor would replace the command instead of carrying it across.
The move is lossless by construction. probes.py block() writes
overrides.get(key, default) for interval, timeout, retries, start_period and
start_interval, so a declaration repeating the literal values renders the same
block plus the missing cadence. Checked per block against the old values: the
only difference is start_interval 5s, nothing changed, nothing lost.
Twenty one stay literal. Seventeen interpolate a template variable, whose value
resolves in a scope meta/services.yml does not share, so moving them would be a
rewrite rather than a transcription. Four interpolate nothing and still cannot
move: svc-db-memcached and svc-db-redis probe every second in their swarm
service partial, where a 5s start cadence would report healthy later than the
block already does, and the sys-svc-rdbms and keycloak sso_proxy partials render
for the consuming role's application_id, so their declaration has no single
meta/services.yml to live in. Each carries a nocheck naming its reason.
All twenty one gain start_interval. Three also gain start_period, without which
the key does nothing at all, since it only applies while the start period runs.
That widens their failure tolerance rather than narrowing it: bluesky from 90 to
120 seconds, magento from 50 to 80, the mariadb partial from 180 to 210.
pixelfed spelled its probe as a bare string and now spells it CMD-SHELL. Docker
reads both as a shell command; only the YAML shape changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An engine of its own keeps its passwords in its data directory, written once
when that directory was initialised. When the deploy rotates the credential the
new value reaches the container's environment and never the engine, so the
health check authenticates with a password nothing has ever accepted. The
application user and root come from the same value, so nothing is left that
could log in and change it either.
mariadbd reads --init-file on every start, before it opens the socket, and runs
the statements with full privileges. That is the one point where a password can
be reset without knowing the previous one.
The SQL is rendered next to the env file that already holds the same secret in
plaintext, so it adds no new exposure. It reaches the container as a config
rather than a bind mount because a swarm task may be scheduled on a node the
rendered path does not exist on; the config name carries a hash of the file
content, so a rotated password produces a new name and swarm replaces the
immutable old one.
The guard is mariadb-and-not-shared, which matches the `local` guard the service
template already renders on, so the block and the top-level entry appear
together or not at all.
3831c8f804 fixed the other half of this failure: matomo, moodle and nextcloud
keep the same password in a config file on their data volume. Same rotation,
opposite end of the connection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
force_shared_db flipped every DB service to shared with no exemption, so
web-app-magento's deliberate embedded MariaDB 11.4 - declared shared:
false with a comment warning that no bot may move the major underneath
it - became dead config and setup:install hit the central engine's 12.3.2,
outside Magento's supported range. The run log names the override:
'swarm force-shared: web-app-magento.services.mariadb -> shared=true'.
Exempts a DB service whose role meta declares a literal shared: false AND
whose role is manager-pinned. The second condition is not decoration:
utils/storage/nfs.py keeps a manager-pinned role's volumes node-local, and
InnoDB over NFS is why the harness force-shares in the first place. Of the
two roles with a literal local DB, magento qualifies and bigbluebutton
(unpinned) does not, so its behaviour is unchanged.
derive_includes._force_shared_db_view documents itself as mirroring
force_shared_db, so it takes the same exemption - without it the dep walk
would compute magento's include closure against the central engine while
the deploy runs a local sidecar.
The sidecar carried no deploy block at all, so an exempted service would
have been free-floating with one task per node over a single-writer
datadir. The block now comes from a container_deploy lookup shared with
sys-svc-container/templates/deploy.yml.j2, which replaces the replicas
ternary and the duplicated placement constraint there. Its unit test
caught a real defect while being written: with compose_mode_force=swarm
over DEPLOYMENT_MODE=compose the delegated compose_replicas returned
empty, emitting a deploy: key with no child.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The entrypoint initialises on a server that listens on the unix socket only, so pg_isready without -h reports healthy while the database is still being built -- visible in run 30699990912 as socket-only at 14:07:37.425 against socket+TCP at 14:07:42.174. Probe 127.0.0.1 instead, which the bootstrap server does not serve.
pg_isready keeps using PQping, so the check stays immune to pg_hba changes as a psql probe would not be. The compose healthcheck gains the start_period it never had, so the stricter probe cannot eat into its 60s retry budget on a cold start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dedicated database branch reused the provider's <db>_custom tag,
which only exists where the central db stack happened to build it; a
cold runner deploying a central-db-disabled variant (nextcloud 1) hit
'pull access denied for mariadb_custom' and the aborted pull dragged
the remaining service pulls down with it. Pin the sidecar to the
upstream reference via the image= override: the CI mirror overrides
rewrite exactly that config path, so CI pulls the ghcr mirror and local
deploys pull the hub. mariadb's custom image is a bare passthrough and
postgres' baked healthcheck is shadowed by the compose-level
healthcheck, so the sidecar behavior is unchanged; the postgres
extensions branch keeps its local build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Convert the two registry-driven networks templates into pure-Python
rendering functions plus thin Ansible lookup plugins:
roles/sys-svc-compose/templates/networks.yml.j2 -> {{ lookup('compose_networks') }}
roles/sys-svc-container/templates/networks.yml.j2 -> {{ lookup('container_networks') | indent(4) }}
The rendering logic lives in utils/networks/render.py as pure-Python
functions that take a registry + closure-style config/database lookups,
making the logic directly unit-testable without a Jinja stub harness.
The lookup plugins are thin wrappers that resolve application_id,
DEPLOYMENT_MODE, and swarm.network.encryption from the templar, build
the service_registry, and forward to the render functions.
Migrate 233 include lines across 85 production templates. Convert
web-app-matrix's hand-coded MDAD networks block (default + conditional
postgres + conditional openldap) to the generic lookup; this adds an
openresty attachment when services.sso.enabled (harmless extra
interface, pinned in test_matrix_mdad_consumer_attaches_postgres_ldap_sso).
Mark local-DB / redis sidecars (sys-svc-rdbms/templates/services/,
svc-db-redis/templates/service.yml.j2) with `# nocheck: networks-literal`
because they intentionally attach to default-only.
Extend the lint to scan ALL roles/*/templates/**/*.yml.j2 (not only
compose.yml.j2) and to forbid the legacy include form; the marker
above is the documented escape hatch.
Coverage:
- tests/unit/utils/networks/test_render.py (30 cases incl. encrypted=false literal pin)
- tests/unit/plugins/lookup/test_compose_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/unit/plugins/lookup/test_container_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/integration/infrastructure/compose/test_networks_render.py (10 snapshot scenarios)
- tests/integration/infrastructure/compose/test_network_includes.py (3 lint rules)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).
* 25 unit tests for the lookup cover compose vs swarm prefix, custom
flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
fails on any raw `image:` line in roles/*/templates/*.j2 unless
nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
`{{ lookup('container_image', application_id, '<service_key>') }}`
form. 26 roles use `custom=True` (mariadb, postgres, openldap,
jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
into consuming roles where application_id evaluates to the outer
role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
test_build_requires_image) recognise the standalone lookup as an
implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
the registry-prefixed image string because the custom var is now
bare.
Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace per-key `compose_container_name` lookup with a generic
`compose_only(key, value)` SPOT that emits `key: "<value>"` only in
compose mode and nothing in swarm. Same gating logic now serves
container_name (157 call sites) and pull_policy (17 call sites)
without duplicate plugin code.
Why this consolidation:
- The earlier `build-key` lint rule was misdirected. Swarm tolerates
`build:` with a harmless warning when `image:` is pre-pushed (via
the existing handlers/swarm_registry_push.py flow in
roles/sys-svc-compose). The actual swarm-rejected key is
`pull_policy:` ("Additional property pull_policy is not allowed"),
and 10 templates were silently broken because they declared inline
`pull_policy: never` outside the existing `{% if DEPLOYMENT_MODE
\!= 'swarm' %}` gate in roles/sys-svc-container/templates/build.yml.j2.
- A single generic lookup is cleaner than N per-key wrappers when
the gating logic is identical (emit-key-or-nothing).
Lint changes in tests/lint/ansible/jinja/test_mode_incompatible_keys.py:
- New `pull-policy-key` rule, regex `^\s*pull_policy\s*:`, remediation
points at compose_only.
- Existing `container-name` rule keeps its raw-key regex; remediation
now also points at compose_only.
- Broken-split / broken-embed regexes rekeyed on the `compose_only`
lookup name (was `compose_container_name`).
- `build-key` rule and its 12 nocheck annotations across templates
removed.
Coordinated changes:
- 86 templates rewritten (157 container_name calls migrated;
17 inline pull_policy lines replaced).
- 1 redundant `{% if DEPLOYMENT_MODE \!= 'swarm' %}` guard dropped in
build.yml.j2 (the lookup self-gates).
- 12 unit tests pin the compose_only contract.
- Catalog (suppression.md) updated: `pull-policy-key` row added,
`container-name` row remediation updated.
- Old `plugins/lookup/compose_container_name.py` and its test module
deleted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mirror compose_container_name pattern: plugins/lookup/compose_restart.py
emits `restart: <policy>` only in compose mode. In swarm the key is
silently ignored (replaced by deploy.restart_policy) and produces
"Ignoring unsupported options" warnings at docker stack deploy time.
Sweep 19 raw restart: sites across 10 templates to the no-arg form
`{{ lookup('compose_restart') }}` (the lookup falls back to
DOCKER_RESTART_POLICY from ansible-vars).
base.yml.j2 keeps the explicit-arg form
`{{ lookup('compose_restart', docker_restart_policy | default(DOCKER_RESTART_POLICY)) }}`
so the 5 templates that override the policy via Jinja-scope
`{% set docker_restart_policy = ... %}` (erpnext, zammad, shopware,
matomo, openproject) retain their override path. Jinja-scope bindings
are visible only in the rendering template, never in the Ansible
variable context that the lookup sees, so the caller must resolve them.
Generalize the container-name lint into a data-driven multi-rule
scanner at tests/lint/ansible/jinja/test_mode_incompatible_keys.py
covering both container-name and restart-key. Adding a new rule is one
ModeRule row sharing the file-walk / suppression / line-grouping
scaffolding.
Also gate roles/svc-db-redis/templates/service.yml.j2's
resource.yml.j2 include with DEPLOYMENT_MODE \!= 'swarm'. Mirrors
base.yml.j2's existing pattern; prevents cpus / mem_limit /
mem_reservation / pids_limit from leaking into swarm output where
deploy.resources.* in deploy.yml.j2 already handles them.
11 unit tests pin the compose_restart lookup contract. Both rules
registered in docs/contributing/actions/testing/suppression.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
container_name is illegal in swarm mode when replicas > 1 (docker rejects
with "services.deploy.replicas: can't set container_name and X as container
name must be unique"). All compose.yml.j2 templates now route through a
single lookup that suppresses the key in swarm mode and emits it in compose
mode.
SPOT: plugins/lookup/compose_container_name.py
Reads DEPLOYMENT_MODE from the templating context, returns
'container_name: "<name>"' under compose and an empty string under swarm.
Call sites read:
{{ lookup('compose_container_name', MY_CONTAINER) }}
No explicit DEPLOYMENT_MODE pass-through needed.
Sweep: 82 templates rewritten across roles/web-app-*, roles/web-svc-*,
roles/svc-*, plus 4 partial includes (sso_proxy, redis, mariadb, postgres).
For container names assembled from multiple Jinja fragments the args use
tilde concat (X ~ '-' ~ Y) so the lookup receives a single resolved string
and the call stays a single well-formed Jinja expression.
Lint: tests/lint/ansible/jinja/test_no_raw_container_name.py blocks
(a) raw 'container_name:' YAML keys in any roles/*/templates/**/*.j2,
(b) split-Jinja regression where the outer braces close mid-call:
{{ lookup('compose_container_name', A }}-{{ B) }},
(c) embedded-Jinja regression inside a string argument:
lookup(..., "{{ X }}_suffix") (removed in ansible-core 2.23).
The (b) and (c) regexes accept either single- or double-quoted lookup
name and scan the full file so multi-line shapes also surface. Suppress
per-site with '# nocheck: container-name' on the same or preceding line.
Tests: tests/unit/plugins/lookup/test_compose_container_name.py pins the
lookup contract: compose vs swarm output, whitespace tolerance, default-to
compose when DEPLOYMENT_MODE is absent, AnsibleError on missing or excess
terms, non-string name coercion.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixes Cluster 3a ca-inject YAML parse: when a role variant disables its
DB (enabled=false), the depends_on template emitted ` :` because
the existing `not shared` guard fired for a non-existing DB host.
The new derived flag mirrors `service.required` (= enabled AND shared)
and is consumed in:
- depends_on/dbms_base.yml.j2, dmbs_excl.yml.j2 (the actual fix)
- sys-svc-rdbms postgres/mariadb service templates
- web-app-discourse, web-app-snipe-it (latent same-pattern defensives)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Generalises the postgres image build to be driven by a declarative
extension→library registry instead of a hardcoded POSTGRES_VECTOR_ENABLED
flag. Same Dockerfile.j2 now serves both the central svc-db-postgres
image (libraries aggregated across all deployed consumers) and per-
consumer images for shared:false postgres consumers whose extensions
need libraries not in the postgis/postgis base (e.g. pgvector).
postgres.yml.j2 switches between image: and build: based on whether
custom libraries are needed. New postgres_libraries lookup filters
base-image extensions out and returns recipes for the rest. Dedicated
task path extracted into dedicated.yml + dedicated_postgres.yml so the
block-with-when size limit stays satisfied and the postgres-type check
happens once.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
sys-svc-rdbms now renders an extensions.sql into a per-consumer
.initdb.d/ directory and bind-mounts it into the dedicated postgres
container at /docker-entrypoint-initdb.d/. CREATE EXTENSION runs as
POSTGRES_USER (= bootstrap superuser) on first initdb so non-trusted
extensions succeed. Same declarative source (service_extensions
lookup, services.yml extensions field) drives both shared:true
(02_init.yml loop) and shared:false (initdb.d) paths. database lookup
exposes a new initdb_dir field; service_extensions lookup loses its
shared:true guard (shared:false is now a real path). Refactored
repeated lookups in sys-svc-rdbms tasks into vars/main.yml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Rename all docker-compose.yml templates to compose.yml
- Update override and CA override filenames accordingly
- Adjust compose wrapper, lookup plugins, filters and path helpers
- Update roles, tasks, handlers and scripts to use compose CLI
- Rewrite documentation and admin guides to reference compose
- Update unit and integration tests to expect compose.yml
- Keep internal variable keys for backward compatibility where needed
https://chatgpt.com/share/69870272-b168-800f-84c3-6417398e441f
- Replace database_application_id with database_consumer_id
- Use explicit consumer context for database resolution
- Fix recursive Jinja variable loop in sys-svc-rdbms
- Decouple discourse database network from global database_type
https://chatgpt.com/share/696bd40d-ebfc-800f-9ba8-1e6be919a629
This change removes --network=host usage, introduces DOCKER_IN_CONTAINER
for reliable environment detection, and replaces hardcoded localhost
bindings with an explicit DOCKER_BIND_HOST abstraction to avoid nftables
issues in CI/DiD environments.
https://chatgpt.com/share/69497616-7b00-800f-8fa3-ddc6eb9c5e98
Includes:
- Rewrite of test-deploy workflow to use isolated inner dockerd with privileged mode.
- Switch logging drivers to 'json-file' when IS_CONTAINER=true for compatibility with non-systemd CI runners.
- Adjust Dockerfile to install docker CLI and simplify package setup.
- Improve inventory creation and deploy steps for CI stability.
- Fully compatible with Ansible 2.20 variable handling.
Conversation reference:
https://chatgpt.com/share/6930e285-9604-800f-aad8-7a81c928548c
- Unified Jinja2 variable spacing in tasks and templates
- Introduced database_image and database_version variables in vars/database.yml
- Updated mariadb.yml.j2 and postgres.yml.j2 to use {{ database_image }}:{{ database_version }}
- Ensured env file paths and includes are consistent
- Prepared support for versioned database images (needed for Magento deployment)
Ref: https://chatgpt.com/share/68b96a9d-c100-800f-856f-cd23d1eda2ed
