The default is what varies between Docker versions, not the explicit value.
Measured by starting each engine and reading the mount back:
engine tmpfs without mode tmpfs with mode 1777
29.7.2 1777 root honoured
28.5.2 755 root 1777 root
A checkmk site runs as uid 1000, so a root-owned 755 tmp makes omdlib fail
`mkdir /omd/sites/cmk/tmp/apache` and the container restarts forever. That is
exactly what run 34348821577 shows, on runners reporting Docker 28.0.4.
f3eba39cae removed `mode: "01777"` here after measuring on 29.7.2 alone, where
the default happens to match, and so replaced the mechanism that behaves the
same everywhere with the one that does not. The mode comes back.
The validator loses the ban with it. Its stated reason, that `docker stack
deploy` drops the key, is a different claim from the one the removing commit
made, and neither survives the measurement above for the compose path this
failure came from; it now checks the type and nothing more.
One thing this does not settle. The note this repository kept on the earlier
attempt records run 33923190220 failing the same way on 28.0.4 while the mode
WAS set. That contradicts the 28.5.2 row and did not reproduce, so either the
patch releases differ or that diagnosis was wrong. The next run decides it:
read the engine version out of the rescue artifact and check whether the site
tmp comes up 1777.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
checkmk crash-looped in swarm on run 33923190220, 82 convergence attempts,
with PermissionError [Errno 13] on '/omd/sites/cmk/tmp/nagvis' raised from
omdlib/tmpfs.py:294 while OMD populated the site's tmp skeleton.
The mount is not the problem and must stay. Upstream's docker-entrypoint.sh
runs `omd create --no-tmpfs` and the comment above it states the site tmp
directory "might have been pre-created by docker when the --tmpfs option is
used", so supplying that tmpfs externally is the intended pattern, not a
workaround.
The mode was the problem. container_volumes coerces a declared mode with
int(text, 8), so "01777" reaches the stack file as the decimal 1023. Docker
29.7.2 reads that as raw mode bits and mounts 1777; the CI hosts run 28.0.4
and produced a directory the owning uid could not write. 0o1023 leaves the
owner with no permission bits at all, which is exactly the failure.
Docker's own default is already 1777, sticky and world-writable, which is
what a site tmp needs, so the declaration could only ever match it or break
it. Dropping the key restores the default. Measured on 29.7.2 in both
`docker compose` and `docker stack deploy`: a mode-less tmpfs mounts 1777 and
uid 1000, the uid `omd create -u 1000` gives the site, creates nagvis in it.
The validator now rejects tmpfs 'mode' outright rather than checking it is
octal. An octal check cannot help when the encoding itself does not survive
to the container the same way on every engine version. merge/svc-net-tor
still carries the old declaration and will fail this check if it lands.
checkmk's entrypoint runs `omd create --no-tmpfs`, so the site's tmp directory
is never mounted and the site cannot come up cleanly. Declaring it needs a tmpfs
with sticky, world-writable permissions, which the volume layer could not
express.
container_volumes built the tmpfs options as a hard-coded single key:
if mount.get("size") is not None:
tmpfs_entry["tmpfs"] = {"size": mount["size"]}
so a `mode` in the declaration was accepted by the schema and silently dropped by
the renderer. It now accumulates size and mode and attaches the key only when
something is set, coercing mode to the int docker expects. mounts.py validates
both the bind and tmpfs spellings through _is_octal_mode and names the expected
shape in the error, so `1777` or `0o1777` fails at render with a readable message
instead of producing the wrong bits.
Capability and consumer land together deliberately. Split apart, checkmk's
mode: "01777" would pass validation, be dropped at render, and the tmp directory
would come up without the sticky bit while everything reported success.
checkmk declares site_tmp as a tmpfs at the site's tmp path with size 512m and
mode 01777. The site name moves into services.checkmk.site, and CHECKMK_SITE
reads it through lookup('config', ...) instead of hard-coding "cmk", so the mount
target and the variable cannot drift.
MERGE NOTE: fork/feature/svc-net-tor carries acc9e25c0f, which deletes
plugins/filter/container_volumes.py and moves it to plugins/lookup/. That
relocated copy has the identical defect at lines 141-142 -- the same hard-coded
{"size": ...} -- and its _coerce_mode_int is wired only to the config-ref path.
It is not a competing fix, but this change will disappear at merge unless it is
re-applied in the lookup.
Verified: the unit suite passes, 4448 tests, including new coverage of the mode
path in test_container_volumes.py.
Not verified: checkmk has not been watched starting with this mount. Docker's
tmpfs mode also interacts with the container umask, so the effective bits on the
created directory were not inspected live. The new validation is fail-closed: a
role carrying a non-octal mode would now error at render rather than be ignored,
and while the suite is green, I did not grep every CHECKMK_SITE consumer for one
still assuming the literal "cmk".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Exclusion was a property of the image: --images-no-backup-required resolves through volume_is_fully_ignored, which skips a volume only when every container using it is ignored. A container holding a derived tree beside state that must be kept could not express the exclusion at all.
matrix_mdad_docker is that case - the data root of a nested docker daemon, sharing one container with matrix_mdad_matrix and matrix_mdad_state, which carry the actual Matrix data. Copying it is not merely wasteful: overlay2 encodes a deletion both as a character-device whiteout and as a trusted.overlay.* xattr, and no rsync in the chain carries -X, so the copy cannot be restored faithfully. The whiteouts also cannot be written onto an overlayfs destination at all, which is what kept the web-app-matrix DR drill red for 58 minutes a round.
meta/volumes.yml gains a 'backup:' key, validated like 'nfs:'. The backup_volume lookup resolves every 'backup: false' entry to its pinned docker name and feeds baudolo's --volumes-no-backup-required. A lint requires the decision to be written down for any volume mounted at /var/lib/docker or /var/lib/containers, so the next docker-in-docker role cannot repeat this silently.
Requires backup-docker-to-local 3.3.0, which carries the flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
compose_volumes emitted a top-level configs:/secrets: entry for every declared
config/secret, but container_volumes already gates each per-service reference on
its mount's when:. So a config whose only mount was gated off (e.g. LDAP
disabled) became an orphan: a top-level object no service references, whose file:
source points at a file the role never renders (same flag). Under docker stack
deploy that source is read at deploy time and the deploy fails; under compose it
was a harmless unused entry.
Gate the top-level emission on the same mount_when_passes container_volumes uses,
so a top-level config/secret exists iff a service actually references it. To share
the predicate, promote the private _mount_when_passes from container_volumes into
utils/roles/applications/mounts and import it in both filters; the bare except
moves with it (inline noqa), so container_volumes no longer needs its BLE001
per-file-ignore.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5