On a data disk with less free space than the high disk watermark asks
for, Elasticsearch creates indices but assigns none of their primary
shards. `zammad-init` then reports every search index update as failed
and never exits, so `compose up` waits on
`service_completed_successfully` until its 1800 s task timeout and the
deploy fails without naming the cause (#850). Elasticsearch's default
high watermark is an absolute 150 GB headroom on a large disk.
A new one-shot `precheck` service runs `search_allocation_precheck.sh`
with `restart: "no"`, and `init` depends on it with
`service_completed_successfully`:
- it waits a bounded time for `_cluster/health`;
- it creates a one-shard, zero-replica probe index with
`wait_for_active_shards=1`;
- once Elasticsearch allocates it, it deletes the probe and exits 0;
- otherwise it prints the creation answer and the answer of
`_cluster/allocation/explain` and exits 1.
The check sits in its own service because `init` keeps
`restart: on-failure`: wrapped into `init`, the failing check was
restarted seven times within eight minutes on a local deploy while
`compose up` kept waiting. Inside the stack, a `restart: "no"`
dependency that exits 1 made `compose up` fail after one second with
`service "pre" didn't complete successfully: exit 1`.
Every curl call carries --connect-timeout and --max-time, so a hanging
request cannot outlast the bounded wait. The image's entrypoint passes
any non-`zammad-*` command through `exec "$@"` (bin/docker-entrypoint of
7.1.2), and the image ships `curl`. The watermark values themselves come
with feature/store-i18n (b82591ba9d); this change covers inventories
without a variant cap.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The bundled Elasticsearch ran with its default watermarks. On a large
disk the high watermark's max_headroom demands 150 GB free, so a 1.8 TB
development host with about 56 to 112 GB free assigned no shard,
zammad-init never finished and the compose-up handler timed out after
1800 seconds (#850).
meta/services.yml declares the watermarks on the search service under
search.watermark: 85 / 90 / 95 % of the disk with max_headroom caps of
200 / 150 / 100 GB, which are Elasticsearch's own defaults, so a
production inventory keeps today's behaviour. Every entry in
meta/variants.yml lowers only the caps to 10 / 5 / 2 GB. Variants reach
development and CI inventories, and a production inventory only when
inventory provision is called with --app-variants;
utils/cache/applications.py
deep-merges them onto the base config, so low, high and flood_stage stay
on the base values. compose.yml.j2 passes all six settings to the
container through ZAMMAD_SEARCH_WATERMARK.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The heaps of the central Elasticsearch, Zammad's Elasticsearch and
Magento's OpenSearch follow their container through the lookup, so far
at half of the limit. That doubled them to 2048, 1024 and 1024 MiB.
Xms equals Xmx, so each container claims that heap from its start, and
the central one locks it in memory.
A quarter of the limit gives 1024, 512 and 512 MiB, the sizes these
engines ran with before. The heap still follows the limit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The central Elasticsearch, Zammad's Elasticsearch and Magento's
OpenSearch ran with a literal heap beside a mem_limit four times its
size: 1g in 4g, 512m in 2g and 512m in 2g. The heap is now half of the
limit through lookup('service_memory_mb'), which makes it 2048, 1024
and 1024 MiB and lets it follow the limit when that changes.
Xms equals Xmx, so these containers hold that much more memory from
the start.
Zammad's memcached size and Magento's PHP_MEMORY_LIMIT take the same
lookup and keep their values, 256 and 768M.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Elasticsearch 9 does not start on a data directory written by 8.13. It
accepts one from 8.19 only. On a deployment that already ran 8.13.4 the
new pin left the search container in a restart loop ("cannot upgrade a
node from version [8.13.4] directly to version [9.5.5]"), zammad-init
kept waiting for it and Zammad did not come up.
The index is derived data. zammad-init rebuilds it from the database
whenever it is missing. path.data now points to a directory named after
the Elasticsearch major inside the zammad_search volume, so a new major
starts on an empty directory and the same deploy rebuilds the index. The
data of the previous major stays in the volume.
The REST ticket test of the spec now also requires the search to return
the ticket it has just created.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Run 34421944328 Zammad#1 (swarm) failed its CLI contract with
"UNREACHABLE read call zammad_list_organizations: the provider answered
nothing". The adapter's upstream_url named nginx by its container_name,
zammad-nginx, which only compose registers; the templates set that
field through compose_only and swarm drops it. The daemon journal shows
the A and AAAA lookups for zammad-nginx failing exactly across the CLI
window, and nginx logged no request from the adapter's network.
ZAMMAD_MCP_UPSTREAM_HOST now takes ZAMMAD_NGINX_SERVICE, the service
key that resolves in both modes and the same way env.j2 already reaches
railsserver, websocket and elasticsearch. Only admitted MCP clients
(flowise, hermes, n8n, openclaw, openwebui) join the shared zammad
network and none of them runs a service keyed nginx, so the short name
is unambiguous there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Thirteen of sixteen MCP sidecar entries carried no enabled at all, so
collect_role_resources fell back to _looks_like_container and billed every
one of them as running. Three roles already gated theirs on mcp.enabled;
the rest now do the same, gitea twice because it is the only role with a
vendor container behind the adapter.
The fifteen adapter roles then read services.<key>.enabled instead of
mcp.enabled, so the compose block and the task gates follow the services
entry rather than running beside it on a second path. The six plugin and
native roles keep reading the topic: they ship no sidecar entry to point at.
The three-level resolution is not new. KEYCLOAK_LOGOUT_PANEL_ENABLED has
the same shape and depth and gates a task the same way, and the same value
reaches JAVASCRIPT_SERVICE_ENABLED, where service-gating.js throws on
anything but "true" or "false" — an unresolved chain would have been
killing keycloak's Playwright run since 575a5a87ac.
This does not change what the resource model counts: _is_enabled reads the
unrendered Jinja and still treats it as on. What it buys is one switch
instead of two, and a declaration that matches it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Zammad 6.5.0 has no AI surface to configure: neither `ai_provider` nor
`ai_provider_config` exists in that line, so the role could not point Smart
Assist anywhere. 7.1.2 is the newest version whose contract is still that
setting pair; from 7.2 the provider moves into an `AI::ProviderConnection`
record and the shape changes, so the bump lands here rather than on latest.
Elasticsearch is unaffected: 7.1.2 accepts >= 7.8, < 10 and the role's 8.13.4
pin stays.
The provider is written by piping files/scripts/apply/ai_provider.rb into
`rails r -` over stdin, with url, model and token passed as environment
variables instead of interpolated into Ruby source, so no quoting path can
mangle a virtual key. Order inside the script is load-bearing and documented
there: `ai_provider` is the boolean "a provider is configured" and its
validator reads `ai_provider_config`, so the config hash has to land first or
the flag is refused. `UserInfo.current_user_id` is set because `Setting.set`
needs an acting user.
This is the strongest evidence in the requirement, because it is not a
read-back of something the deploy just wrote: `Setting.set("ai_provider_config")`
runs `AI::Provider::CustomOpenAI.ping!` upstream, so the write itself is a live
round trip. An unreachable gateway or a rejected key fails the task.
Two consequences of the bump ride along, because the version does not work
without them.
`/api/v1/getting_started` answers 403 from Zammad 7.0 on, so the Playwright
helper could no longer lift a csrf token there; it uses `/api/v1/signshow`,
the unauthenticated seed. Its old fallback read `csrf_token` from a JSON body
that endpoint never returns, so the failure surfaced as a confusing throw
rather than the 403 it was.
env.j2 drops the hand-built DATABASE_URL in favour of POSTGRESQL_OPTIONS
`?pool=`. Zammad composes its DSN from the POSTGRESQL_* parts itself, so the
explicit URL both duplicated and competed with them while leaving the
connection pool at its default.
The spec decodes ZAMMAD_LITELLM_BASE_URL and ZAMMAD_LITELLM_MODEL through
decodeDotenvQuotedValue. Without it the values arrive still wrapped in the
literal quotes dotenv_quote adds and the strict comparison can never match in
compose mode -- measured as expected `"\"http://litellm:4000/v1\""` against
received `http://litellm:4000/v1`.
Verified: the spec passes against the live stack (1 passed, via
scripts/tests/e2e/rerun-spec.sh --grep=Assist, which restages the spec from the
repo), and both the contract and the Smart Assist write ran ok in a variant-2
deploy. `make quality` green on test-lint, test-unit, test-integration,
test-external.
Not covered: the no-gateway path (ZAMMAD_LITELLM_ENABLED false) has not run
since the bump, and Zammad features this deploy does not exercise -- email
channel, LDAP sync, ticket workflows -- are unproven on 7.1.2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mount pipeline was split across two layers: compose_volumes and
container_volumes lived as filter modules that ansible's filter loader
rejected on every start ("has no attribute 'FilterModule'", two warnings
per run), while the same-named lookups wrapped them thinly. The filter
bodies now live inside their lookups (the tor_socks precedent), the
filter files are gone, and the render tests moved next to the lookup
tests.
As lookups they see the templating context, so the application_id term
every call site passed back to its own role is gone too: both lookups
read application_id from context and fail loudly when it is missing.
107 templates and nfs_prep.yml drop the term; the calls collapse to
lookup('compose_volumes') and lookup('container_volumes', service_name).
The container_volumes call is now canonical - four leading spaces,
service_name in the service position, | indent(4) - enforced by a new
lint (test_container_volumes_call_shape). Normalizing the 85 literal
call sites surfaced that erpnext keyed its meta/volumes.yml mounts by
container name instead of compose service key, the only role to do so;
its meta now uses the service keys. wordpress's column-0 call, which
rendered its first line at top level, is indented like every other.
Template logic that only restated meta/volumes.yml is deleted: mailu fed
compose_volumes an extra_volumes dict whose eight names were looked up
from the very meta file compose_volumes reads itself; zammad's init and
railsserver extra mounts were the volumes.yml entries minus the two
services, which the meta now lists (the when-gated OIDC binds included).
Conditionality that meta cannot express stays in code, but in role-local
lookup plugins instead of template loops: matrix_bridge_mounts renders
the per-bridge registration mounts, nextcloud_extra_volumes the
feature-gated whiteboard/recording volumes (type: volume entries are
declared unconditionally by compose_volumes, so gating must happen
before it).
Parity is proven, not assumed: every mechanically rewritten template is
byte-identical to HEAD after the transformation regexes (95/95), and the
semantic roles render identically through the real functions against
HEAD's meta - zammad across six sso/tls polarities, all seven erpnext
services, mailu's top level. make test-lint passes complete.
Also: suitecrm's compose template pins container_extra_hosts with
host_alias (the SOCKS proxy URL host.docker.internal never resolved in
compose+tor), and the five disabled mautrix bridge addons use the
empty-default config lookup so templating the applications dict stops
emitting 125 errors per batch in every CI job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/volumes.yml says where a volume lands inside its container. Wherever a
second file spelled the same path out, the two drifted independently: the mount
moves, the consumer keeps pointing at the old place, and nothing fails until the
process looks for content that is not there. e031b7f61 fixed exactly this for
web-app-moodle; this generalises it and makes the rule hold.
The lookup gains what a caller needs. `targets` maps each mounting service to
its container path, and `target` is the one path an entry mounts at -- present
only when every mount agrees. An entry that lands at different paths per service
omits `target` rather than picking one, and the caller names the service:
"{{ lookup('volume', application_id, 'prometheus_config').target }}"
"{{ lookup('volume', 'web-app-prometheus', 'host_sys').targets['node-exporter'] }}"
66 literals across 48 role files now derive. Replacement only happened where the
path stood alone -- /srv/app inside /srv/app/data is a different path -- and
every rewritten file was re-parsed, Jinja and YAML alike. Three sites the tool
refused rather than guess were done by hand: zammad's path sat in a Jinja list
literal, seaweedfs' inside a quoted string feeding a filter, and openresty's
/var/run/docker.sock is not a mount target at all but the socket on the host,
which only happens to spell the same string. That one carries the single
nocheck.
tests/lint/repository/spot/test_volumes_path_single_spot.py keeps it that way.
roles/*/files/ is out of scope by construction: it is copied verbatim into an
image and executed there, so a Dockerfile or entrypoint has no Ansible to
evaluate a lookup with.
One defect this found, and it was in the tooling rather than the roles: deciding
ambiguity AFTER filtering short paths made web-app-prometheus' host_sys look
single-valued -- it mounts /sys for cadvisor and /host/sys for node-exporter,
and /sys is below the length filter. The template would have rendered an
undefined `.target`. Both the converter and the rule now decide ambiguity over
every target and filter afterwards, and that call reads targets['node-exporter'].
Verified by resolving rather than by reading: all 70 accesses in the diff were
run against the real registry and every one yields a non-empty path equal to the
literal it replaced. make test-lint: 328 passed.
Not verified: no Ansible render, so a call site whose surrounding context cannot
build the applications registry would still fail at deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml became meta/secrets.yml, but its content still landed at
applications.<app>.credentials — the one meta file whose payload did not sit
under a root named after the file. Every other topic obeys that rule:
meta/volumes.yml goes to volumes, meta/rbac.yml to rbac. Credentials now go to
secrets.credentials.
The path exists exactly once, in utils/manager/credential_key.py, and
OVERRIDE_SECTION is derived from its two segments rather than spelled beside
them, so the dotted form used by --set and the nested form used by the
inventory cannot drift. Six consumers read it: the loader, the generator, the
CLI override parser, the two emit helpers, and the four lookup validators.
--set now accepts exactly one key shape,
applications.<app>.secrets.credentials.<key>. The three short forms are
rejected instead of guessed at: the same short key names different credentials
on different apps. That also retired primary_app_id, which only existed to
disambiguate them.
Three defects surfaced while wiring this up. _apply_one_role_schema still wrote
to the old root, so an app's own credentials would have moved while its
providers' stayed behind. web-app-pgadmin reads through
lookup('applications', ...).credentials.<key>, an access path no regex over
lookup('config', ...) could see, and would have deployed with an empty postgres
password. And _META_NON_GENERIC_TOPICS still listed 'schema' — a bare topic
name, invisible to any path search — so the generic file-root loop assigned the
raw schema under secrets and the follow-up write landed inside the cached
document, truncating keycloak's four credentials to one for every later
reader.
tests/integration/ansible/lookups/config/test_credentials_roundtrip.py pins both
directions of the seam and the cache invariant that last defect broke.
Existing inventories still carry the pre-rename root; migrating them is
deliberately out of scope here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine roles reached their MCP surface as the deployment administrator or with
an account password kept beside the account in credentials. A Moodle site
admin bypasses every capability check, a Nextcloud app password is not
scopable, a Gitea token reads every private repository, and a GitLab token on
root reads every private project. Each role now declares its own principal in
meta/users.yml and provisions it, so the exposed surface is what the service
role permits. jellyfin, zammad and snipe-it kept their account but took its
password from a credential named token; they read the registry too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A user password reaches its application through Ansible, a shell, a container
exec -e and finally the runtime's own environment reader. The generator seeded
it from the same alphabet as the become password, punctuation included, and
that does not survive the chain intact.
Three roles had already worked around it by declaring a separate alphanumeric
credential for an identity that their meta/users.yml already declares, and
saying so in the description. Removing those without fixing the generator would
have reintroduced exactly the bug they were built against, so the generator is
fixed first and all three now read the user's own password. At length 64 the
smaller alphabet costs nothing worth measuring, and the become password keeps
the old one because it travels a different path.
Zammad additionally used the bot's password for its wizard-bypass admin, one
secret for two users. Each has its own declaration and now its own value.
A test asserts the generated password is alphanumeric, so a future change to
the alphabet fails there rather than inside a Ruby ENV.fetch in a container.
Ten applications with no MCP server of their own get one without a line of code
inside them: qdrant, checkmk, fider, jellyfin, listmonk, pretix, prometheus,
snipe-it, zammad and libretranslate.
The sidecar is confined rather than trusted. It runs read-only, drops every
capability, forbids privilege escalation, publishes no port and joins only its
own provider's network, carrying neither that provider's alias nor the consumer
networks the provider itself joined. Its tool surface is a JSON contract, not
code: which tools exist, which upstream path each maps to and what shape the
arguments take. Adding a tool is a line of JSON, not a new image.
The image is built locally from the audited adapter source with the base image
as a build argument, so all ten share one reviewed build instead of ten upstream
dependencies. Each role provisions its own read-only identity in the language
its application speaks, and every adapter include is guarded by the role's MCP
flag, so switching it off leaves no container behind.
Container-name vars derived their value inconsistently (hard-coded literals, entity_name composites) instead of the role-owned SPOT. Add a lint requiring every *_CONTAINER var to resolve through lookup('config', app, 'services.<x>.name'), the bare entity_name form, or the engine lookup; give each multi-container worker a name: in meta/services.yml (with realistic resource limits) and switch its var to the lookup. Extend the engine lookup with a shared-aware 'container' field (central provider container when shared, embedded <entity>-<engine> otherwise) and gate zammad's embedded memcached sidecar on not-shared so its container_name never collides with the central memcached.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
resource_filter now falls back service_name -> entity_name -> hard
default, so entity-level limits keep applying to per-service compose
entries. Compose service names drop the entity prefix (kix, jitsi,
zammad, bluesky); the docker container_name keeps it via compose_only.
A new lint forbids '<entity>-'-prefixed service names.
Service names and internal ports are declared once: vars/main.yml
holds <ROLE>_<SERVICE>_SERVICE constants (container names derived via
entity_name), meta/services.yml declares ports.internal consumed
through config lookups. sso-proxy upstreams stack-qualify the origin
host in swarm because short aliases are ambiguous on shared overlays.
Stale proxy.service_key entries fixed for erpnext/shopware/nextcloud.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).
* 25 unit tests for the lookup cover compose vs swarm prefix, custom
flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
fails on any raw `image:` line in roles/*/templates/*.j2 unless
nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
`{{ lookup('container_image', application_id, '<service_key>') }}`
form. 26 roles use `custom=True` (mariadb, postgres, openldap,
jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
into consuming roles where application_id evaluates to the outer
role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
test_build_requires_image) recognise the standalone lookup as an
implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
the registry-prefixed image string because the custom var is now
bare.
Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a mode-aware `container_address` lookup plugin and a
`resolve-container-id` helper script that resolve swarm task IDs at
exec time (compose mode returns bare names unchanged). Migrates ~25
roles + ~28 task files from `command:` to `shell:` so the lookup's
`$(...)` subshell evaluates correctly, and extends `database` lookup
with an `address` key for cross-role DB exec.
Three new lint tests enforce the contract: every container exec
target goes through the resolver, lives behind an IS_STACK_HOST gate
(walks the include graph one hop deep), and uses `shell:` rather than
`command:` when the target is a Jinja reference. Existing nocheck
suppressions on bare-name DB exec sites (nextcloud redis, listmonk,
moodle OIDC, mailu) are replaced with the lookup-based form.
Fixes the systemd Environment= injection in sys-ctl-cln-disc-space
(it never expanded $(...)) by passing STACK / SERVICE_KEY parts and
resolving inside the script. Extracts shopware admin and mediawiki
extension install bodies into dedicated .sh files invoked via env.
Rewrites password-bearing shell tasks to either `| quote` directly or
the `-e VAR=... | quote` + `sh -c '... ${VAR} ...'` env-bridge pattern.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- meta/volumes.yml: declare zammad_storage + zammad_search named volumes
- vars/main.yml: ZAMMAD_STORAGE_VOLUME / ZAMMAD_SEARCH_VOLUME constants reading from meta
- compose.yml.j2: extra_volumes uses the constants instead of empty dicts
- tasks/01_core.yml: 4 host-rendered init files (auto_wizard.json, zz_omniauth_allow_get.rb, telegram_helper_stub.rb, zz_oidc_group_role_mapping.rb) migrate from inline template: tasks to the central render_replicated_templates.yml helper so the bind-mounted host paths resolve on every swarm peer
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure whitespace reflow applied by `make autoformat` — column-aligned
mapping values were collapsed to single-space alignment so the files
match the project-wide YAML formatter output. No behavior change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrates the role from the pre-v9.0.0 legacy oidc/oauth2 top-level
service blocks to the unified services.sso shape with flavor: oidc.
Drops the playwright.spec.js require for the test-alias-domain module
that was deleted in 82491791e, and removes the README link to the
TODO.md file dropped in c48386eee. Archives the in-flight 022 zammad
requirement doc into the README index since the role itself is now
landed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop test-mail-to-ticket Playwright spec + its playwright.env keys; mail/SMTP
integration is mostly testing Mailu, not Zammad, and required either a /28
subnet resize or an email-lookup-plugin shared-mode branch to actually pass.
- Set auth_third_party_auto_link_at_inital_login=true in apply_oidc_settings.rb
so the first OIDC sign-in of an LDAP-synced user links to the existing record
by email instead of returning 422. Drops the conditional LDAP user_filter
workaround and the LDAP_OIDC_COEXISTS env wiring that propped it up.
- Replace the RUBYOPT-loaded Telegram constant shim with a bind-mounted
TelegramHelper stub that cuts the require 'telegram/bot' chain (the gem's
Zeitwerk bypasses Kernel#require intercepts under bootsnap).
- Strip verbose docblocks across vars/tasks/templates/scripts/playwright; keep
only WHY-non-obvious notes (upstream gem mismatches, security defaults, the
intentional inital-typo).
- Fix broken Docker Compose CLI docs URL in sys-svc-compose/README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User identity SPOT
- meta/users.yml gains zammad-api-bot (Basic-auth REST API bot) and
zammad-wizard-bypass-admin (auto_wizard.json setup-bypass admin).
- vars/main.yml reads identity (login/email/firstname/lastname) via
lookup('users', …); password stays in credentials/ to guarantee the
shell-safe alphanumeric form needed for the Ansible→bash→docker-exec
→Ruby ENV chain.
- auto_wizard.json.j2, apply_api_bot.rb, tasks/01_core.yml: consume the
lookup-resolved identity.
Boot reliability
- templates/zz_telegram_constant_shim.rb.j2: pre-loaded via RUBYOPT;
pre-declares the Telegram::Bot namespace and intercepts Kernel#require
to refuse telegram/* loads, sidestepping the broken telegram-bot-ruby
2.4.0 gem that crashes Zammad 6.5.0 boot. Telegram channel is inert.
- env.j2 sets RUBYOPT=-r/opt/zammad/zz_telegram_constant_shim.rb.
- tasks/01_core.yml stages the shim; compose.yml.j2 bind-mounts it into
zammad-init/rails/websocket/scheduler/nginx.
- tasks/01_core.yml adds an explicit AutoWizard.setup post-init step
(upstream zammad-init does NOT auto-process auto_wizard.json), which
is what V2 (no-auth) and V3 (LDAP-only) need to skip the setup wizard.
LDAP source wire-up (TODO #1)
- files/scripts/apply_ldap_source.rb creates an LdapSource record
pointing at svc-db-openldap (inetOrgPerson + groupOfNames mapping).
- tasks/01_core.yml runs it when services.ldap.enabled is true.
OIDC group→role mapping (TODO #3)
- templates/zz_oidc_group_role_mapping.rb.j2 patches Zammad's
User.create_from_hash / update_from_hash to translate the RBAC
group-path claim (e.g. /roles/web-app-zammad/administrator) into
the matching Zammad role (Admin / Agent / Customer fallback).
- apply_oidc_settings.rb pins uid_field=preferred_username so OIDC
matches the human-readable Keycloak username, NOT the sub UUID.
- compose.yml.j2 bind-mounts the initializer when OIDC is enabled.
Test coverage
- new test-login-via-ldap.js (driven only in LDAP-only V3).
- test-alias-domain.js accepts 2xx-direct OR 301-to-canonical
(sys-svc-proxy renders aliases as 301 sources, not server-name
aliases — server.yml updated to canonical=helpdesk + alias=
zammad.helpdesk).
- test-mail-to-ticket.js: stage-level SMTP trace logging; reaches
Mailu via host.docker.internal:465.
- test-ticket-reply-as-agent.js: scoped contenteditable lookup to
.article-add to avoid hitting the ticket-title field.
- _shared.js: signInAsApiBot drives /api/v1/signin via the server-side
APIRequestContext (shared cookie jar with the browser), unblocking
the SPA-OIDC interlock.
SQL outsourcing
- templates/sql/grant_createdb.sql.j2: the CREATEDB grant now lives in
a Jinja-templated .sql file; tasks/01_core.yml consumes it via
lookup('template', …). Role name is rendered into the SQL by Jinja
(NOT via database_query named_args), because psql parameter binding
quotes string values as literals (`'zammad'`) which is invalid SQL
for ALTER ROLE.
Misc
- meta/server.yml: canonical=helpdesk.<domain>, aliases=[zammad.helpdesk.<domain>]
to match the 022 requirement (Decision #1).
- TODO.md drained — all seven items either landed or are explicitly
documented as out-of-scope / brittle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Rename credential basic_auth_password → api_bot_password and var
ZAMMAD_ADMINISTRATOR_PASSWORD → ZAMMAD_API_BOT_PASSWORD to reflect
that the value is the role-internal bot's local password, not the
administrator's (reused for the wizard-bypass admin only because both
need the same shell-safe alphanumeric form).
- Introduce ORGANIZATION constant (defaults to SOFTWARE_NAME) and migrate
organization-string consumers (mattermost team, shopware shop title,
dashboard login/logout descriptions, simpleicons "provided by",
user-administrator description/lastname).
- Add HOST_LL_CC_TAG (BCP-47 form, e.g. en-gb) and use in auto_wizard.
- Add NETWORK_TRUSTED_PROXY_CIDRS SPOT and consume from suitecrm,
shopware and zammad reverse-proxy configs.
- Extract ZAMMAD_WEB_PROTOCOL via lookup('tls', …, 'protocols.web')
instead of inline 'https' if … else 'http'.
- Use database_query for CREATEDB grant so it works on shared+non-shared
databases; back the SQL with files/sql/grant_createdb.sql SPOT.
- Read administrator firstname/lastname from users lookup in
auto_wizard.json.j2 instead of hardcoding.
- Drop redundant elasticsearch: block from web-app-zammad services.yml
(no consumer reads it).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 7 new Playwright scenarios covering helpdesk-core workflows and
infrastructure regressions:
- test-csp-headers — CSP response + meta-tag parity
- test-alias-domain — server-name alias serves 2xx, not 30x
- test-rest-api-ticket-create — POST /api/v1/tickets via Basic auth
- test-ticket-create-as-customer — biber SPA → fill form → submit
- test-ticket-reply-as-agent — API-seeded ticket; admin posts a reply
- test-websocket-realtime — open ticket; API appends article;
expect realtime push into the open SPA
- test-mail-to-ticket — raw-socket SMTP to Mailu helpdesk
mailbox; poll Zammad until ticket
appears
Auto-configures Zammad's Email channel (inbound IMAP + outbound SMTP)
against Mailu via a new post-bootstrap rails-runner step. The Ruby for
both bootstrap steps lives in standalone files and is piped through
`container exec -i ... bundle exec rails r -`; the previous wrapper
shell scripts (`apply_*.sh`) are removed in favour of inline `shell:`
calls in tasks/01_core.yml. Env vars are forwarded into the container
via `container exec -e VAR ...` so the Ruby uses `ENV.fetch` and stays
value-free / portable.
vars/main.yml grows three constants:
- ZAMMAD_OMNIAUTH_INIT_CONTAINER_PATH — bind-mount target inside the
container (was hard-coded four times in compose.yml.j2)
- ZAMMAD_HELPDESK_USER / _EMAIL / _PASS — derived from the `helpdesk`
mail-bot user provisioned via meta/users.yml
- ZAMMAD_MAIL / ZAMMAD_EMAIL_ENABLED — derived from the central email
lookup + the services.email gate
playwright.env.j2 grows MAIL_SMTP_HOST/PORT/USER/PASS + HELPDESK_EMAIL
for the SMTP injection in the mail-to-ticket scenario, sorted ASC per
the env-key lint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Playwright spec split into per-test partials, mirroring the web-app-kix
layout (lint-conform: top-level `playwright.spec.js` requires the
partials and shared module):
- `files/playwright/_shared.js` — env vars, `signInViaZammadOidc`,
`zammadLogout`, `beforeEach`
- `files/playwright/test-landing.js`
- `files/playwright/test-login-administrator.js`
- `files/playwright/test-login-biber.js`
- `files/playwright/test-guest-persona.js`
Variable dedup in `vars/main.yml`:
- `ZAMMAD_VOLUMES_DIR` — single base path
- `ZAMMAD_AUTO_WIZARD_HOST_PATH` — derived
- `ZAMMAD_OMNIAUTH_INIT_HOST_PATH` — derived (was repeated 4x in compose)
- `ZAMMAD_OIDC_ENABLED` — single source for the OIDC gate
All consumers in `tasks/01_core.yml`, `templates/compose.yml.j2`,
`templates/auto_wizard.json.j2`, `templates/playwright.env.j2` now read
the constants instead of inlining the `lookup(...)` expressions.
Drive-by: `meta/main.yml` description narrowed to describe Zammad itself
(role-specific text moved out of the upstream-facing galaxy_info field).
`make autoformat` clean, `make test` green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds roles/web-app-zammad/ — a brand-new web-app role deploying Zammad as
the Infinito.Nexus helpdesk with direct OpenID Connect SSO against the
shared Keycloak client. Tracks requirement 022.
Stack: ghcr.io/zammad/zammad 6.5.0 (init / railsserver / websocket /
scheduler / nginx, all driven by upstream command modes), bundled
elasticsearch 8.13.4 + bundled memcached (no central svc role exists),
central svc-db-postgres reused via lookup('database', application_id).
Variant matrix mirrors web-app-kix: V1 oidc+ldap, V2 all-false, V3 ldap-only.
Bootstrap flow (tasks/01_core.yml):
- Stage auto_wizard.json into the role's volumes dir AFTER sys-stk-full
creates the directory; bind-mount it into zammad-init so the upstream
rake zammad:setup:auto_wizard bypasses the setup UI on first boot.
- Stage zz_omniauth_allow_get.rb initializer + bind-mount into rails /
websocket / scheduler / nginx, since the Zammad SPA's SSO link is a
plain GET and omniauth 2.x forbids GET by default.
- ALTER ROLE <zammad-db-user> WITH CREATEDB so Zammad's unconditional
`rake db:create` doesn't trip on insufficient privilege when the
database is pre-created by svc-db-postgres.
- Apply post-bootstrap settings via apply_oidc_settings.sh
(rails-runner inside zammad-railsserver): fqdn, http_type so Zammad
emits the correct redirect_uri to Keycloak; auth_openid_connect +
auth_openid_connect_credentials so the SSO button works.
Workarounds embedded in compose.yml.j2:
- zammad-nginx command override that copies /docker-entrypoint.sh to
/tmp and patches `grep "nameserver"` to `grep -E "^nameserver"`,
because Docker writes comment lines like `# Overrides: …` into
/etc/resolv.conf that the upstream grep matches as substrings,
yielding an invalid `resolver … Overrides: …` directive nginx rejects.
- DATABASE_URL pre-populated in env.j2 so post-bootstrap
`container exec rails r` calls (which bypass the entrypoint) can
load Rails without database.yml.
Playwright (files/playwright/playwright.spec.js, single-file lint-conform
layout): landing-reachable + administrator OIDC + biber OIDC + guest
public-landing scenarios. Personas skip-correctly via
skipUnlessServiceEnabled('oidc') under V2 / V3.
V1 end-to-end green: admin + biber Playwright pass through the full
Keycloak SSO round-trip. V2 (all-false) and V3 (ldap-only) green on
baseline reinstall with full_cycle=true.
Follow-ups indexed in roles/web-app-zammad/TODO.md: Zammad LdapSource
auto-create for V3, mail channel auto-config + end-to-end mail-to-ticket
test, Keycloak-group-to-Zammad-role mapping (Zammad doesn't ship a
built-in claim-to-role translator — currently all OIDC logins default to
Customer per Decision #5 fallback), Playwright assertion for the
`zammad.helpdesk.*` server-name alias.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
