mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 19:01:54 +00:00
kevinveenbirkenbachandClaude Opus 5
cce3d48e65
The volume mounted at /mattermost/plugins was an NFS export shared by every replica, so a redeploy had one task remove the tree while the outgoing task still held the plugin binary open. Measured on task .1.ixph4yrvwfbo7mgghcbdsyreh: 13:41:15.658 [outgoing] plugin exited mattermost-ai 13:41:17.280 [incoming] Removing local installation of managed plugin before sync 13:41:17.304 [incoming] Failed to remove ... unlinkat plugins/mattermost-ai/server/dist: directory not empty 13:41:24.227 [incoming] Failed to install prepackaged plugin ... destination already exists 14:28:23.936 [incoming] Access to route for non-existent plugin mattermost-ai NFS replaces a still-open file with a .nfsXXXX entry on unlink, so unlinkat on the parent returns ENOTEMPTY. The removal fails, the directory survives, the reinstall cannot move into it, and that node answers 404 on the plugin routes for the rest of its life while the service VIP keeps sending it traffic. On a local filesystem unlinking an open file is legal, so the failure cannot arise. Four of six replicas in the same run had no overlapping predecessor and came up clean.becd8f1577fixed the sibling volume and deliberately left this one shared, on the grounds that uploaded plugin archives live in it. That reason does not hold. Upstream server/channels/app/plugin.go lists bundles from fileStorePluginFolder, reads them with fileReader and calls installPluginLocally after removePluginLocally, which is the code path that emits the log line above; the local directory is the copy and the file store is the original. In this deployment the file store holds no plugin at all: every replica logs "No plugins to sync from the file store" and the bundle comes from /mattermost/prepackaged_plugins inside the image. So plugins and client-plugins are the two halves of one per-node extraction. The activation error names bundle_path plugins/mattermost-ai while failing to copy into client/plugins, and splitting their lifetimes is what left one half node local and the other shared. They now share a lifetime. Two consequences worth stating. The flag only takes effect on a volume that does not exist yet, because docker does not recreate a named volume when driver_opts change and stack deploy --prune prunes services; a cluster already running the NFS stack needs the volume removed per node. And the worker copies leave the export's backup path for svc-bkp-volume-2-local, which runs manager-only, so they are covered by nothing. The content is extracted per node from the image on every boot, which is what the nfs-false-data-loss exemption argues. The README promised persistence through both plugin volumes. That has been untrue for client-plugins sincebecd8f1577, so it now says what the volumes are and where an uploaded bundle actually persists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
40 lines
1.2 KiB
YAML
40 lines
1.2 KiB
YAML
---
|
|
config:
|
|
type: volume
|
|
name: mattermost_config
|
|
mounts:
|
|
- service: mattermost
|
|
target: /mattermost/config
|
|
|
|
data:
|
|
type: volume
|
|
name: mattermost_data
|
|
mounts:
|
|
- service: mattermost
|
|
target: /mattermost/data
|
|
|
|
logs:
|
|
type: volume
|
|
name: mattermost_logs
|
|
mounts:
|
|
- service: mattermost
|
|
target: /mattermost/logs
|
|
|
|
plugins:
|
|
type: volume
|
|
name: mattermost_plugins
|
|
# nocheck: nfs-false-data-loss Reason: extracted per node from the image's prepackaged bundles, the file store holds none; shared over NFS a redeploy unlinks the tree while the outgoing task still holds the plugin binary open, silly-rename leaves it non-empty, and the reinstall then fails destination-already-exists and the node serves 404
|
|
nfs: false
|
|
mounts:
|
|
- service: mattermost
|
|
target: /mattermost/plugins
|
|
|
|
client-plugins:
|
|
type: volume
|
|
name: mattermost_client_plugins
|
|
# nocheck: nfs-false-data-loss Reason: unpacked from the server-side plugin bundle at every activation; shared, replicas starting at once copy into one directory and the loser then serves 404 on the plugin routes
|
|
nfs: false
|
|
mounts:
|
|
- service: mattermost
|
|
target: /mattermost/client/plugins
|