fix(web-app-zammad): size Elasticsearch disk watermarks to the disk and cap them in CI and dev

The bundled Elasticsearch ran with its default watermarks, which require
up to 150 GB free for the high watermark on a large disk. On a 1.8 TB
development host with about 112 GB free it assigned no shard, zammad-init
never finished and the compose-up handler timed out after 1800 seconds.

meta/services.yml now declares the watermarks under search.watermark:
85 / 90 / 95 % of the disk with a max_headroom cap of 200 / 150 / 100 GB,
which is Elasticsearch's own default, so production keeps its behaviour.
Every entry in meta/variants.yml lowers only the caps to 10 / 5 / 2 GB;
variants reach development and CI inventories, while inventory provision
applies them only when --app-variants names one. compose.yml.j2 passes the
six values to the container through ZAMMAD_SEARCH_WATERMARK.

Measured on the development host with 119 GB free: Elasticsearch reports
85% / 90% / 95% with max_headroom 10gb / 5gb / 2gb from variant 0, keeps no
persistent or transient override, and allocates the new primary shards.
TODO.md held only this defect and is removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

roles/web-app-zammad/README.md

@@ -94,7 +94,7 @@ Solid `1:1` edges are fixed relationships; dashed `0..1` edges are conditional (
 - **LDAP federation:** When `svc-db-openldap` is present, Zammad authenticates and provisions accounts against the central LDAP.
 - **Mail-to-ticket:** When `web-app-stalwart` is present, the `helpdesk` mailbox is auto-provisioned and Zammad polls it to create tickets from incoming mail.
 - **Server-name alias:** `zammad.helpdesk.{{ DOMAIN_PRIMARY }}` is a true vhost alias of `helpdesk.{{ DOMAIN_PRIMARY }}` (not a 301 redirect).
-- **Bundled Elasticsearch:** Search engine ships with the role until a central `svc-db-elasticsearch` exists.
+- **Bundled Elasticsearch:** Search engine ships with the role until a central `svc-db-elasticsearch` exists. Its disk watermarks follow the disk size (85 / 90 / 95 % used, requiring at most 200 / 150 / 100 GB free, `services.search.watermark`); the CI and development variants lower those caps to 10 / 5 / 2 GB.
 - **Search index per Elasticsearch major:** The index lives in a directory named after the Elasticsearch major inside the `zammad_search` volume. A new major starts on an empty directory and `zammad-init` rebuilds the index from the database on the same deploy.
 - **Wizard bypass:** First deploy seeds `auto_wizard.json` so no manual setup UI step is required.
 

roles/web-app-zammad/TODO.md

@@ -1,3 +0,0 @@
-# TODO
-
-- The bundled Elasticsearch (`search` and `elasticsearch` in `meta/services.yml`, image `docker.elastic.co/elasticsearch/elasticsearch:8.13.4`, service block in `templates/compose.yml.j2`) runs with the default disk watermarks, so on a host whose data disk has less than 150 GB free it assigns no shard and the role does not deploy. Measured on a 1.8 TB disk with 111.9 GB free (93.8 % used): `GET _cluster/allocation/explain` named the decider `disk_threshold` with `cluster.routing.allocation.disk.watermark.high.max_headroom=150gb`, `GET _cluster/health` reported `red` with 0 active primary and 52 unassigned shards, every index creation of `rake zammad:searchindex:rebuild` in `zammad-init` took 30 seconds, the reload that follows logged `Unable to send AI::TextTool.find(5).search_index_update_backend` (Elasticsearch answered 503 `unavailable_shards_exception`, `primary shard is not active Timeout: [1m]`) once per record every 75 seconds, `init` never completed, and the `compose up` handler of the deploy timed out after 1800 seconds with `failed=1`. Options: set `cluster.routing.allocation.disk.threshold_enabled=false` or absolute watermarks in the environment of the `elasticsearch` service, or check the free space before `compose up` and fail with that message.

roles/web-app-zammad/meta/services.yml

@@ -135,6 +135,14 @@ elasticsearch:
   pids_limit: 512
 init:
   name: zammad-init
+  watermark:
+    low: "85%"
+    high: "90%"
+    flood_stage: "95%"
+    max_headroom:
+      low: 200gb
+      high: 150gb
+      flood_stage: 100gb
   cpus: "1.0"
   mem_reservation: 256m
   mem_limit: 1g

roles/web-app-zammad/meta/variants.yml

@@ -36,6 +36,12 @@
       shared: true
     redis:
       shared: true
+    search:
+      watermark:
+        max_headroom:
+          low: 10gb
+          high: 5gb
+          flood_stage: 2gb
     litellm:
       enabled: false
     tor:
@@ -83,6 +89,12 @@
       shared: false # nocheck: variants-static-override backup shared stays literally true
     redis:
       shared: false # nocheck: variants-static-override backup shared stays literally true
+    search:
+      watermark:
+        max_headroom:
+          low: 10gb
+          high: 5gb
+          flood_stage: 2gb
     litellm:
       enabled: false
     tor:
@@ -133,6 +145,12 @@
       shared: false # nocheck: variants-static-override backup shared stays literally true
     redis:
       shared: true
+    search:
+      watermark:
+        max_headroom:
+          low: 10gb
+          high: 5gb
+          flood_stage: 2gb
     tor:
       enabled: true
     openwebui:

roles/web-app-zammad/templates/compose.yml.j2

@@ -56,6 +56,12 @@
       - xpack.security.enabled=false
       - ES_JAVA_OPTS=-Xms{{ ZAMMAD_SEARCH_HEAP_MB }}m -Xmx{{ ZAMMAD_SEARCH_HEAP_MB }}m
       - path.data={{ ZAMMAD_SEARCH_DATA_DIR }}
+      - cluster.routing.allocation.disk.watermark.low={{ ZAMMAD_SEARCH_WATERMARK.low }}
+      - cluster.routing.allocation.disk.watermark.high={{ ZAMMAD_SEARCH_WATERMARK.high }}
+      - cluster.routing.allocation.disk.watermark.flood_stage={{ ZAMMAD_SEARCH_WATERMARK.flood_stage }}
+      - cluster.routing.allocation.disk.watermark.low.max_headroom={{ ZAMMAD_SEARCH_WATERMARK.max_headroom.low }}
+      - cluster.routing.allocation.disk.watermark.high.max_headroom={{ ZAMMAD_SEARCH_WATERMARK.max_headroom.high }}
+      - cluster.routing.allocation.disk.watermark.flood_stage.max_headroom={{ ZAMMAD_SEARCH_WATERMARK.max_headroom.flood_stage }}
     {{ lookup('container_volumes', service_name) | indent(4) }}
     {{ lookup('container_extra_hosts') | indent(4) }}
     {{ lookup('container_networks') | indent(4) }}

roles/web-app-zammad/vars/main.yml

@@ -23,6 +23,7 @@ ZAMMAD_RAILS_PORT: "{{ lookup('config', application_id, 'services.' ~ ZAMMAD_RAI
 ZAMMAD_WEBSOCKET_PORT: "{{ lookup('config', application_id, 'services.' ~ ZAMMAD_WEBSOCKET_SERVICE ~ '.ports.internal.http') }}"
 ZAMMAD_SEARCH_PORT: "{{ lookup('config', application_id, 'services.' ~ ZAMMAD_SEARCH_SERVICE ~ '.ports.internal.http') }}"
 ZAMMAD_SEARCH_HEAP_MB: "{{ lookup('service_memory_mb', application_id, ZAMMAD_SEARCH_SERVICE, 0.25) }}"
+ZAMMAD_SEARCH_WATERMARK: "{{ lookup('config', application_id, 'services.search.watermark') }}"
 ZAMMAD_SEARCH_MAJOR: "{{ (lookup('config', application_id, 'services.search.version') | string).split('.') | first }}"
 ZAMMAD_SEARCH_DATA_DIR: "{{ [lookup('volume', application_id, 'zammad_search').target, ZAMMAD_SEARCH_MAJOR] | path_join }}"