`docker stack deploy` ignores `depends_on`, so in swarm `init` started
even when the `precheck` service exited 1. The stack then looked exactly
as it did before #982: `init` hung, and `stack_ready.sh` reported "not
converged" until all 600 retries of the convergence handler ran out.
The single failed `precheck` task never turned fatal, because
`has_fatal_task` needs three errors.
`init` now runs `search_allocation_precheck.sh` in front of
`zammad-init` as well. In compose the `precheck` service still stops
`compose up` before `init` starts. In swarm, `init` keeps
`restart: on-failure`, so a refusing Elasticsearch makes its tasks fail
again and again. After three errors past `swarm_stack_fatal_grace`
(180 s), `has_fatal_task` marks the service stuck and the deploy stops
with the precheck output in the task report.
Reported by CodeRabbit on #982.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On a data disk with less free space than the high disk watermark asks
for, Elasticsearch creates indices but assigns none of their primary
shards. `zammad-init` then reports every search index update as failed
and never exits, so `compose up` waits on
`service_completed_successfully` until its 1800 s task timeout and the
deploy fails without naming the cause (#850). Elasticsearch's default
high watermark is an absolute 150 GB headroom on a large disk.
A new one-shot `precheck` service runs `search_allocation_precheck.sh`
with `restart: "no"`, and `init` depends on it with
`service_completed_successfully`:
- it waits a bounded time for `_cluster/health`;
- it creates a one-shard, zero-replica probe index with
`wait_for_active_shards=1`;
- once Elasticsearch allocates it, it deletes the probe and exits 0;
- otherwise it prints the creation answer and the answer of
`_cluster/allocation/explain` and exits 1.
The check sits in its own service because `init` keeps
`restart: on-failure`: wrapped into `init`, the failing check was
restarted seven times within eight minutes on a local deploy while
`compose up` kept waiting. Inside the stack, a `restart: "no"`
dependency that exits 1 made `compose up` fail after one second with
`service "pre" didn't complete successfully: exit 1`.
Every curl call carries --connect-timeout and --max-time, so a hanging
request cannot outlast the bounded wait. The image's entrypoint passes
any non-`zammad-*` command through `exec "$@"` (bin/docker-entrypoint of
7.1.2), and the image ships `curl`. The watermark values themselves come
with feature/store-i18n (b82591ba9d); this change covers inventories
without a variant cap.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The probe scraped a directory listing under dl.fedoraproject.org, which
redirects to whichever mirror is near the caller. A mirror that does not
carry the directory answers 404, and 404 is not retried, so every EPEL
package came back uncheckable and the suite failed on runners whose nearest
mirror is incomplete -- twice in a row, while the same URLs answer 200 from
elsewhere.
mdapi already backs the Fedora probe and serves the epel10 branch from a
single host. Routing EPEL through it removes the mirror roulette and keeps
the verdict sharp: a name it does not know is still reported absent rather
than unknown.
Checked against the live branch: ShellCheck and keepassxc resolve, an
invented name reports absent, and the suite passes.
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.
The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.
call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every cold render of the applications SPOT walks the templates of all
269 roles, and those templates call lookup('config'), lookup('tls'),
lookup('domain') and others. Each of these re-enters
get_merged_applications while the render guard is set, and each one
rebuilt the unrendered view from scratch: a deepcopy of all role
defaults plus the override merge, about 8 ms, never cached. A local
cold render of the development inventory made 351 such calls, 2.82 s
of a 6.86 s render.
The outer render now keeps a copy of its unrendered view and its cache
key on the render guard, taken before the inventory volumes are
popped, so it is the same view a nested call used to build. A nested
call with the same key returns it; any other key still builds its own.
The guard drops it when the render ends.
Measured on the same inventory: the nested calls took 0.07 s instead
of 2.82 s and the cold render 3.60 s instead of 6.86 s. The rendered
SPOT is unchanged: the old and new dumps differ in 69 user passwords,
which the development inventory regenerates on every read (two runs of
the old code differ in 68), and in one absolute checkout path, the only
other difference.
In CI each cold render took 40 to 77 s and the heaviest swarm cell of
#950 missed its budget by a few minutes (#984).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
OpenSSF Scorecard reports no fuzzer integration (code scanning alert
507, issue #853). tests/fuzz/_yaml_bootstrap.py is an atheris harness
that feeds arbitrary input to utils.yaml_bootstrap.load_block and fails
whenever the reader accepts a document that load_yaml_str parses
differently or rejects. That is the contract the module's docstring
states, and the existing property test only samples it. The leading
underscore keeps unittest discovery from importing it, because atheris
is only installed with the fuzz extra.
cron-security-fuzz.yml runs the harness for 15 minutes every Sunday and
on demand, on ubuntu-latest, and uploads crashing inputs as the
fuzz-crashes artifact. atheris 3.1.0 ships x86_64 manylinux wheels for
CPython 3.12 to 3.14 only, so it lives in its own `fuzz` extra and
stays out of every other install path. tests/fuzz/ is not part of
`make test`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
utils/yaml_bootstrap.py promises that load_block(text) either equals
yaml.safe_load(text) or raises BootstrapYamlError. A fuzzer comparing
both found two inputs that broke the promise:
- str.splitlines() also breaks lines at \x0b, \x0c and \x1c to \x1e,
which PyYAML's reader rejects as non-printable. "\x1e/: R" came back
as {"/": "R"}, and a non-printable character inside a comment line was
skipped with the comment.
- An indented " ---" on line 1 was taken as the document marker, while
PyYAML reads it as a plain scalar and rejects the mapping that follows.
load_block now refuses any character outside PyYAML's own printable
class before it splits lines, and takes "---" as the document marker
only at column 0. The line breaks splitlines() still honours (\r, \r\n,
NEL, LS, PS) are line breaks for PyYAML too, so splitting stays as is.
The eight inputs join STRUCTURES in the divergence test; against the
previous reader, four of them diverge and fail it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The bundled Elasticsearch ran with its default watermarks. On a large
disk the high watermark's max_headroom demands 150 GB free, so a 1.8 TB
development host with about 56 to 112 GB free assigned no shard,
zammad-init never finished and the compose-up handler timed out after
1800 seconds (#850).
meta/services.yml declares the watermarks on the search service under
search.watermark: 85 / 90 / 95 % of the disk with max_headroom caps of
200 / 150 / 100 GB, which are Elasticsearch's own defaults, so a
production inventory keeps today's behaviour. Every entry in
meta/variants.yml lowers only the caps to 10 / 5 / 2 GB. Variants reach
development and CI inventories, and a production inventory only when
inventory provision is called with --app-variants;
utils/cache/applications.py
deep-merges them onto the base config, so low, high and flood_stage stay
on the base values. compose.yml.j2 passes all six settings to the
container through ZAMMAD_SEARCH_WATERMARK.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5.5
evangelostsak
infinito-nexus-updater[bot]