fix(web-app-moodle): gate the ownership walks on a settled bootstrap

01_manager_ops.yml waits for the moodle container to report healthy and then runs through to the ownership walks, but compose.yml.j2 gives a healthcheck to moodle alone. cron runs the same entrypoint against the same shared code volume and is not covered by that wait, so a bootstrap could still be filling and tearing down .bootstrap.stage while the walks traversed it. That is the 5089 'cannot access' lines and the closing 'Stale file handle' of job 116229890655.

5c3749c437 and 7bc5822421 serialized bootstraps against each other, append-only lock and a sentinel recheck when flock returns EIO across NFS clients. Neither serializes the deploy against a bootstrap, and no condition expressed 'the tree is final' for the deploy to wait on. The walks now wait for the state a finished bootstrap leaves behind: the stage gone and the sentinel holding the expected release. That observes the outcome rather than the lock, so it holds whether or not flock serializes on the mount.

The sentinel path moves to vars/main.yml and reaches the image as an ARG and ENV, like the stage, so the deploy and the entrypoint read one definition instead of two.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

roles/web-app-moodle/files/Dockerfile

@@ -23,6 +23,7 @@ ARG MOODLE_DATA_DIR
 ARG MOODLE_LOCAL_CACHE_DIR
 ARG MOODLE_SOURCE_DIR
 ARG MOODLE_BOOTSTRAP_STAGE
+ARG MOODLE_BOOTSTRAP_SENTINEL
 ARG MOODLE_RUNTIME_USER
 ARG MOODLE_OIDC_ENABLED
 ARG MOODLE_OIDC_PLUGIN_RELPATH
@@ -45,6 +46,7 @@ ENV MOODLE_CODE_DIR=${MOODLE_CODE_DIR} \
     MOODLE_LOCAL_CACHE_DIR=${MOODLE_LOCAL_CACHE_DIR} \
     MOODLE_SOURCE_DIR=${MOODLE_SOURCE_DIR} \
     MOODLE_BOOTSTRAP_STAGE=${MOODLE_BOOTSTRAP_STAGE} \
+    MOODLE_BOOTSTRAP_SENTINEL=${MOODLE_BOOTSTRAP_SENTINEL} \
     MOODLE_RUNTIME_USER=${MOODLE_RUNTIME_USER} \
     MOODLE_OIDC_PLUGIN_RELPATH=${MOODLE_OIDC_PLUGIN_RELPATH} \
     MOODLE_MCP_PLUGIN_RELPATH=${MOODLE_MCP_PLUGIN_RELPATH} \

roles/web-app-moodle/files/build/moodle-entrypoint.sh

@@ -17,11 +17,11 @@ set -euo pipefail
 : "${MOODLE_LOCAL_CACHE_DIR:?required}"
 : "${MOODLE_SOURCE_DIR:?required}"
 : "${MOODLE_BOOTSTRAP_STAGE:?required}"
+: "${MOODLE_BOOTSTRAP_SENTINEL:?required}"
 : "${MOODLE_RUNTIME_USER:?required}"
 : "${MOODLE_VERSION_FILE:?required}"
 : "${MOODLE_RELEASE:?required}"
 
-MOODLE_BOOTSTRAP_SENTINEL="${MOODLE_CODE_DIR}/.bootstrap.done"
 MOODLE_BOOTSTRAP_LOCK="${MOODLE_CODE_DIR}/.bootstrap.lock"
 
 mkdir -p "${MOODLE_DATA_DIR}"

roles/web-app-moodle/tasks/03_ownership.yml

@@ -9,8 +9,15 @@
 # because only Moodle FPM (running as MOODLE_RUNTIME_USER) needs to
 # read/write user-uploaded content.
 #
-# The walks prune MOODLE_BOOTSTRAP_STAGE: an entrypoint on the shared volume
-# tears that directory down while they run, so walking it hits vanished paths.
+- name: "⏳ Wait until no container is still bootstrapping the shared code volume"
+  shell: >
+    container exec --user root {{ MOODLE_CONTAINER_ADDRESS }}
+    sh -c 'test ! -e {{ MOODLE_BOOTSTRAP_STAGE }} && test "$(cat {{ MOODLE_BOOTSTRAP_SENTINEL }} 2>/dev/null)" = "{{ MOODLE_RELEASE }}"'
+  register: moodle_bootstrap_settled
+  until: moodle_bootstrap_settled.rc == 0
+  retries: 120
+  delay: 5
+  changed_when: false
 
 - name: "✅ Ensure ownership is correct on Moodle code dir"
   shell: "container exec --user root {{ MOODLE_CONTAINER_ADDRESS }} find {{ MOODLE_CODE_DIR }} -path {{ MOODLE_BOOTSTRAP_STAGE }} -prune -o -exec chown -h {{ MOODLE_RUNTIME_USER_GROUP }} {} +"

roles/web-app-moodle/templates/compose.yml.j2

@@ -15,6 +15,7 @@
         MOODLE_LOCAL_CACHE_DIR: "{{ MOODLE_LOCAL_CACHE_DIR }}"
         MOODLE_SOURCE_DIR: "{{ MOODLE_SOURCE_DIR }}"
         MOODLE_BOOTSTRAP_STAGE: "{{ MOODLE_BOOTSTRAP_STAGE }}"
+        MOODLE_BOOTSTRAP_SENTINEL: "{{ MOODLE_BOOTSTRAP_SENTINEL }}"
         MOODLE_RUNTIME_USER: "{{ MOODLE_RUNTIME_USER }}"
         MOODLE_OIDC_ENABLED: "{{ MOODLE_OIDC_ENABLED | bool | ternary('1','0') }}"
         MOODLE_OIDC_PLUGIN_RELPATH: "{{ MOODLE_OIDC_PLUGIN_RELPATH }}"

roles/web-app-moodle/vars/main.yml

@@ -9,6 +9,7 @@ MOODLE_LOCAL_CACHE_DIR: "{{ lookup('volume', application_id, 'localcache').mount
 MOODLE_SOURCE_DIR: "/opt/moodle-source"
 MOODLE_ENTRYPOINT_PATH: "/usr/local/bin/moodle-entrypoint.sh"
 MOODLE_BOOTSTRAP_STAGE: "{{ [MOODLE_CODE_DIR, '.bootstrap.stage'] | path_join }}"
+MOODLE_BOOTSTRAP_SENTINEL: "{{ [MOODLE_CODE_DIR, '.bootstrap.done'] | path_join }}"
 
 MOODLE_BUILD_DIR: "build"
 MOODLE_BUILD_FILES:

tests/unit/python/roles/web-app-moodle/files/build/test_moodle_entrypoint.py

@@ -31,6 +31,7 @@ class TestMoodleEntrypoint(unittest.TestCase):
         self.source = self.root / "source"
         self.code = self.root / "code"
         self.stage = self.code / ".bootstrap.stage"
+        self.sentinel = self.code / ".bootstrap.done"
         self.bin_dir = self.root / "bin"
         self.bin_dir.mkdir()
         self.real_cp = shutil.which("cp")
@@ -45,6 +46,7 @@ class TestMoodleEntrypoint(unittest.TestCase):
             MOODLE_LOCAL_CACHE_DIR=str(self.root / "localcache"),
             MOODLE_SOURCE_DIR=str(self.source),
             MOODLE_BOOTSTRAP_STAGE=str(self.stage),
+            MOODLE_BOOTSTRAP_SENTINEL=str(self.sentinel),
             MOODLE_RUNTIME_USER=pwd.getpwuid(os.getuid()).pw_name,
             MOODLE_VERSION_FILE="public/version.php",
         )