fix(openproject): wait for the schema before starting web and worker

Swarm ignores depends_on, so web and worker start while the seeder is still migrating and die on ActiveRecord::PendingMigrationError. The artifact records 12 failed attempts across the cluster before any replica survived, and all three survivors landed on the node with the fewest failures - one against five and seven - which is how the drain drill lost the worker task it asserts on.

The app services now start through files/shell/await_migrations.sh, which polls a read-only gate and then execs the upstream entrypoint. The gate never migrates, so replicas cannot race each other the way an `until db:migrate` loop does. The seeder keeps its job: the artifact shows it applying 465 migration lines and then seeding, so the wait is aimed at work that really happens during the deploy.

The wait is bounded because the gate is ambiguous. Measured against openproject/openproject:17: `bundle exec rails db:abort_if_pending_migrations` takes 11 s and exits 1 both for pending migrations and for an unreachable database (ActiveRecord::DatabaseConnectionError). An unbounded loop would spin forever whenever the database is down. Past the deadline the entrypoint starts regardless, so the worst case stays the crash loop that already exists, never a service that refuses to come up.

The container path is declared once, in meta/volumes.yml; the second mount takes it by YAML anchor and compose reads it back through lookup(volume). Both were verified through the projects own loader, not only yaml.safe_load.

Verified by running the script in that image with no database and no workdir override: it polls, reports the deadline and starts the entrypoint. Not verified: the success branch, since no migrated database was available, and no live deploy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

roles/web-app-openproject/files/shell/await_migrations.sh

@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+# Wait for the seeder's migrations, then exec the app entrypoint.
+#
+# Swarm ignores depends_on, so web and worker would start while the seeder is
+# still migrating and die on ActiveRecord::PendingMigrationError. The gate is
+# read-only -- it never migrates, so concurrent replicas cannot race.
+#
+# The wait is bounded on purpose. `db:abort_if_pending_migrations` exits 1 both
+# for pending migrations and for an unreachable database (measured against
+# openproject/openproject:17), so an unbounded loop would spin forever whenever
+# the database is down. Past the deadline the entrypoint starts regardless,
+# which is exactly today's behaviour: the worst case stays the crash loop that
+# already exists, never a service that refuses to come up.
+#
+# Argv:
+#   $1 deadline_seconds -- how long to wait before starting anyway
+#   $2 poll_seconds     -- sleep between probes
+#   $3.. entrypoint     -- the upstream start command to exec
+set -euo pipefail
+
+deadline_seconds="${1:?deadline_seconds required}"
+poll_seconds="${2:?poll_seconds required}"
+shift 2
+[ "$#" -gt 0 ] || {
+	echo "await_migrations: entrypoint required" >&2
+	exit 2
+}
+
+deadline=$(($(date +%s) + deadline_seconds))
+while [ "$(date +%s)" -lt "$deadline" ]; do
+	if bundle exec rails db:abort_if_pending_migrations >/dev/null 2>&1; then
+		echo "await_migrations: schema is current, starting $1"
+		exec "$@"
+	fi
+	sleep "$poll_seconds"
+done
+
+echo "await_migrations: deadline of ${deadline_seconds}s reached, starting $1 regardless" >&2
+exec "$@"

roles/web-app-openproject/meta/volumes.yml

@@ -13,6 +13,15 @@ data:
       target: /var/openproject/assets
     - service: seeder
       target: /var/openproject/assets
+await_migrations:
+  type: config
+  source: "{{ OPENPROJECT_AWAIT_MIGRATIONS_ABS }}"
+  mode: "0555"
+  mounts:
+    - service: web
+      target: &await_migrations_target /usr/local/bin/await_migrations.sh
+    - service: worker
+      target: *await_migrations_target
 pgdata_dummy:
   # Exception: mount only exists to prevent OpenProject images from creating
   # anonymous volumes for the unused embedded postgres data path.

roles/web-app-openproject/tasks/01_manager_ops.yml

@@ -13,6 +13,13 @@
     - compose up
     - compose build
 
+- name: "⏳ Transferring await_migrations.sh to {{ OPENPROJECT_AWAIT_MIGRATIONS_ABS }}"
+  stack_host_copy:
+    src: shell/await_migrations.sh
+    dest: "{{ OPENPROJECT_AWAIT_MIGRATIONS_ABS }}"
+    mode: "0555"
+  notify: compose up
+
 - name: "📁 create {{ OPENPROJECT_DUMMY_DIR }}"
   file:
     path: "{{ OPENPROJECT_DUMMY_DIR }}"

roles/web-app-openproject/templates/compose.yml.j2

@@ -32,7 +32,7 @@ x-op-app: &app
   {{ service_name }}:
     <<: *app
     {% include 'roles/sys-svc-container/templates/base.yml.j2' %}
-    command: "./docker/prod/web"
+    command: ["{{ OPENPROJECT_AWAIT_MIGRATIONS_TARGET }}", "{{ OPENPROJECT_MIGRATION_WAIT_SECONDS }}", "{{ OPENPROJECT_MIGRATION_POLL_SECONDS }}", "./docker/prod/web"]
     {{ lookup('compose_only', 'container_name', OPENPROJECT_WEB_CONTAINER) }}
     {{ lookup('container_networks') | indent(4) }}
     {{ lookup('container_depends_on', application_id, extra={
@@ -46,7 +46,7 @@ x-op-app: &app
   {{ service_name }}:
     <<: *app
     {% include 'roles/sys-svc-container/templates/base.yml.j2' %}
-    command: "./docker/prod/worker"
+    command: ["{{ OPENPROJECT_AWAIT_MIGRATIONS_TARGET }}", "{{ OPENPROJECT_MIGRATION_WAIT_SECONDS }}", "{{ OPENPROJECT_MIGRATION_POLL_SECONDS }}", "./docker/prod/worker"]
     {{ lookup('compose_only', 'container_name', OPENPROJECT_WORKER_CONTAINER) }}
     {{ lookup('container_networks') | indent(4) }}
     {{ lookup('container_depends_on', application_id, extra={

roles/web-app-openproject/vars/main.yml

@@ -9,6 +9,10 @@ OPENPROJECT_REF: "{{ lookup('config', application_id, 'services.openproject.ref'
 OPENPROJECT_WEB_SERVICE: "web"
 OPENPROJECT_WEB_CONTAINER: "{{ lookup('config', application_id, 'services.' ~ OPENPROJECT_WEB_SERVICE ~ '.name') }}"
 OPENPROJECT_WEB_CONTAINER_ADDRESS: "{{ lookup('container_address', application_id, OPENPROJECT_WEB_SERVICE) }}"
+OPENPROJECT_MIGRATION_WAIT_SECONDS: 600
+OPENPROJECT_MIGRATION_POLL_SECONDS: 10
+OPENPROJECT_AWAIT_MIGRATIONS_ABS: "{{ [ lookup('container', application_id, 'directories.config'), 'await_migrations.sh'] | path_join }}"
+OPENPROJECT_AWAIT_MIGRATIONS_TARGET: "{{ (lookup('volume', application_id, 'await_migrations').mounts | first).target }}"
 OPENPROJECT_SEEDER_SERVICE: "seeder"
 OPENPROJECT_SEEDER_CONTAINER: "{{ lookup('config', application_id, 'services.' ~ OPENPROJECT_SEEDER_SERVICE ~ '.name') }}"
 OPENPROJECT_CRON_CONTAINER: "{{ lookup('config', application_id, 'services.cron.name') }}"