Commit Graph
31 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5.5 3fb237a143 fix(search): keep the search heaps at the size they had
The heaps of the central Elasticsearch, Zammad's Elasticsearch and
Magento's OpenSearch follow their container through the lookup, so far
at half of the limit. That doubled them to 2048, 1024 and 1024 MiB.
Xms equals Xmx, so each container claims that heap from its start, and
the central one locks it in memory.

A quarter of the limit gives 1024, 512 and 512 MiB, the sizes these
engines ran with before. The heap still follows the limit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 01:32:05 +02:00
kevinveenbirkenbachandClaude Opus 5.5 a7db89ac0b fix(search): give the search engines half of their container as heap
The central Elasticsearch, Zammad's Elasticsearch and Magento's
OpenSearch ran with a literal heap beside a mem_limit four times its
size: 1g in 4g, 512m in 2g and 512m in 2g. The heap is now half of the
limit through lookup('service_memory_mb'), which makes it 2048, 1024
and 1024 MiB and lets it follow the limit when that changes.

Xms equals Xmx, so these containers hold that much more memory from
the start.

Zammad's memcached size and Magento's PHP_MEMORY_LIMIT take the same
lookup and keep their values, 256 and 768M.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 23:31:14 +02:00
kevinveenbirkenbachandClaude Opus 5 61bf7a9ce3 refactor(web-app-magento): name the docroot and the install marker
The container docroot was spelled out in four shell invocations and twice in the nginx vhost, where both sides must agree or nginx serves an empty server block — the 403 this role hit. A single MAGENTO_ROOT now feeds both, and the install guard's mountpoint expression moves next to it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 06:56:01 +02:00
kevinveenbirkenbachandClaude Opus 5 d997d564b9 fix(web-app-magento): make the seaweedfs variant install and serve
Variant 0 had never run before: it is the only variant with sso and remote storage enabled, and every layer on that path was broken.

setup:install passed --remote-storage-path-style, but ConfigOptionsList writes remote_storage/config/path-style while AwsS3Factory reads path_style. The hyphenated key is dead, so the SDK addressed the bucket virtual-hosted-style and resolved magento.seaweedfs, which has no DNS record. files/remote-storage.php writes the key the factory actually reads, after the install; a PUT of storage.flag through Magento's own AWS SDK confirms path-style reaches the gateway.

The install guard named app/etc/env.php, which setup:install writes long before it finishes, so a failed install skipped the install forever after. It now guards on a marker the task writes only on success, and clears a stale env.php on the path where no install ever completed.

nginx includes the vhost as a glob, so on a fresh volume it starts before composer creates the file and serves an empty server block. The reload was gated on the vhost file changing, which is false when a previous run already wrote it; it now also fires when the install ran.

Magento ships no OIDC, so the oauth2-proxy round trip authenticates at the edge without creating a Magento session and no in-app logout exists. Both personas are declared blocked accordingly, biber unconditionally and the administrator under sso.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 06:44:16 +02:00
kevinveenbirkenbachandClaude Opus 5 dd457083da fix(magento): get the install through, no Adobe keys needed
Five independent blockers, none of which the role could get past.

1. Credentials. create-project pulled the metapackage from repo.magento.com,
which needs Adobe Marketplace keys; without them the bootstrap got HTTP 401
after 40 minutes of stack build. meta/schema.yml declared both keys with
validation ^.*$, so an empty value passed inventory validation silently.
mirror.mage-os.org serves the same packages unauthenticated and hosts the
dists itself - verified on the deploy's own network path, HTTP 200 for
project-community-edition-2.4.8.zip. Repository, package and version now
live in meta/services.yml, which also makes a later switch to the Mage-OS
distribution proper a data change rather than a code change.

2. Premature handler flush. The config renders notified 'compose up', so the
handler chain ran before sys-stk-full had rendered compose.yml and the CA
handler died on the missing file. Worse, the render had already created the
instance directory, so the repo's own guard for exactly this
(Fail: instance dir missing at handler flush) no longer tripped.

3. Anonymous volumes. Both markoshust images declare VOLUME /var/www while
the named volume mounted one level deeper at /var/www/html, so docker created
an anonymous volume per container and sys-ctl-hlth-volumes refused to start.
The mount moves to /var/www, where the image declares it; the image ships an
empty html/ that a fresh named volume initialises from.

4. Unescaped expansions. setup:install passed --db-host=\"$MYSQL_HOST\" and
friends, but only the quotes were escaped for the outer shell, not the
dollars. The DiD shell expanded them to empty, Magento fell back to localhost,
took that as a unix socket and reported SQLSTATE[HY000] [2002]. 18 expansions.

5. A non-resumable bootstrap. create-project needs an empty target and cannot
be retried, yet it carried the dependency download - and mirror.mage-os.org
runs nginx with the default http2_max_requests=1000, so composer's ~490
requests on one connection hit GOAWAY mid-transfer at the same package twice
in a row (curl error 18, not the flake it looked like). It is now
create-project --no-install plus a retryable composer install, which resumes
from the cache; measured succeeding in the live container after the abort.
The creates: guards follow: composer.json marks the skeleton, bin/magento the
finished install - it only exists once dependencies are in.

MAGENTO_COMPOSER_SPEC folds package and version because autoformat broke the
create-project line inside a Jinja expression; it rendered correctly by luck.

Not fixed here: the base URL renders with a double slash, and the lifecycle
stays pre-alpha - the role reaches setup:install now but has not yet served
a page.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 10:47:40 +02:00
kevinveenbirkenbachandClaude Opus 5 dc08bc6212 fix(magento): make nginx and php-fpm actually serve
The role could not deliver anything. The markoshust nginx image ships a
default vhost built for a different topology: fastcgi points at
unix:/sock/phpfpm.sock, port 8000 only returns 301 to https, the real
server sits on 8443 with a self-signed cert, and it references an upstream
host 'phpfpm' that does not exist here - so nginx aborted at config load
with 'host not found in upstream' and the container never started.
php-fpm listened on that same unix socket, because its
/usr/local/etc/php-fpm.conf carries no include line and php-fpm.d/*.conf
is therefore dead config.

Both are now replaced through type: config mounts. The role never had any
- that absence is why every template it previously rendered was inert.

The vhost stays minimal and includes Magento's own nginx.conf, copied from
nginx.conf.sample by the bootstrap. Its location blocks carry the .php
whitelist, the /static/version rewrites and the /media to get.php fallback
that a hand-written vhost has to reimplement - the previous attempt did
not, and would have executed every .php under pub/.

PHP moves from 8.5 to 8.4: Magento 2.4.8 requires ~8.2.0||~8.3.0||~8.4.0
and the bootstrap passes no platform-requirement override, so
composer create-project failed before nginx ever mattered.

nginx waits for the php socket before starting, for the reason given in
the erpnext frontend commit: the wildcard search domain turns a DNS miss
into a routable wrong answer, and nginx pins upstreams at config load.

Also: the oauth2 origin port named 8080, where nothing listens, and the
healthcheck proved only that some listener existed - it stayed green on a
redirect loop and on 502.

Measured in a rebuild of the role topology against Magento's real
nginx.conf.sample: HTTP 200 on / and /index.php, healthy healthcheck, and
the counter-test - without the gate and with php down, nginx exits 1.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 10:47:39 +02:00
kevinveenbirkenbachandClaude Opus 5 e62e695876 chore(magento): drop the never-wired config layer
01_runtime_conf.yml rendered a nginx.conf and a php-fpm pool override that
never reached a container: no task included the file, and meta/volumes.yml
declares no mount for either output. Verified against HEAD in a second
checkout - both templates were already dead there, so removing them changes
runtime behaviour by exactly nothing.

Their four path variables go with them, as do MAGENTO_PHP_HOST,
MAGENTO_PHP_PORT and MAGENTO_NGINX_PORT, whose only consumers were the
assert and the two templates.

Closing the numbering gap renames 02_setup.yml to 01_setup.yml, the name it
could not take while 01_runtime_conf.yml existed.

This does not make the role work - it does not work today either. The
markoshust nginx image cannot even start under this topology
(host not found in upstream "phpfpm"), and its php-fpm listens on a unix
socket rather than TCP 9000. Both are addressed separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 10:47:39 +02:00
kevinveenbirkenbachandClaude Fable 5 cf32840586 feat(lint): require *_PORT vars to resolve via lookup, migrate literals
New lint rejects literal port numbers in roles/*/vars/main.yml: the port
belongs in meta/services.yml (services.<svc>.ports.<scope>.<name>) where
band governance and stack rendering read it; vars resolve it via
lookup('config', ...). Migrated the 16 remaining literals across
elasticsearch, registry-docker, erpnext, flowise, funkwhale, magento,
moodle, nextcloud, penpot and shopware.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 03:15:53 +02:00
kevinveenbirkenbachandClaude Opus 4.7 0ffbb73d72 refactor(images): central container_image lookup + migrate all roles
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).

* 25 unit tests for the lookup cover compose vs swarm prefix, custom
  flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
  fails on any raw `image:` line in roles/*/templates/*.j2 unless
  nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
  longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
  `{{ lookup('container_image', application_id, '<service_key>') }}`
  form. 26 roles use `custom=True` (mariadb, postgres, openldap,
  jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
  odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
  fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
  into consuming roles where application_id evaluates to the outer
  role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
  test_build_requires_image) recognise the standalone lookup as an
  implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
  the registry-prefixed image string because the custom var is now
  bare.

Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-16 00:17:11 +02:00
kevinveenbirkenbachandClaude Opus 4.7 6e49fe74fb feat(volumes): canonical dict-of-dicts meta schema + lookup('volume')
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.

What's in this commit (210 paths):

Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
  top-level volumes:/configs:/secrets: section and per-service mount
  blocks from the canonical meta. container_volumes uses the SEMANTIC
  name as per-service mount source so it resolves against compose_volumes'
  top-level dict key (docker_name as source produced 'undefined volume'
  on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
  volume(application_id, name) returns a normalised single entry;
  volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
  _CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
  raw Jinja source: strings never enter the templar walk (root-cause of
  the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
  list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
  both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
  compose-only validate+reload behind a single when-block; swarm uses
  service-update instead (bind-mounts of a FILE pin source inode at
  task-create time, so atomic write-tmp+rename leaves the running task
  on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
  lookup('volume', application_id) | dict2items so NFS subdir
  pre-creation works under the new cache split.

Tests (12)
- New strict lints flagging inline mount sections in compose templates,
  single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
  call sites — all volume-mount surfaces MUST flow through container_volumes
  / compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
  accept both the legacy and the new lookup('volume', app, 'X').name forms
  during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
  nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
  alert_rules.yml bind mount.

Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
  with lookup('container_volumes', application_id, '<service>') |
  indent(N) per service + a single trailing lookup('compose_volumes',
  application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
  legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
  application_id, 'X').Y across 9 roles

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-14 19:08:46 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a4cb5b89e1 feat(swarm): swarm-aware container exec via container_address lookup
Adds a mode-aware `container_address` lookup plugin and a
`resolve-container-id` helper script that resolve swarm task IDs at
exec time (compose mode returns bare names unchanged). Migrates ~25
roles + ~28 task files from `command:` to `shell:` so the lookup's
`$(...)` subshell evaluates correctly, and extends `database` lookup
with an `address` key for cross-role DB exec.

Three new lint tests enforce the contract: every container exec
target goes through the resolver, lives behind an IS_STACK_HOST gate
(walks the include graph one hop deep), and uses `shell:` rather than
`command:` when the target is a Jinja reference. Existing nocheck
suppressions on bare-name DB exec sites (nextcloud redis, listmonk,
moodle OIDC, mailu) are replaced with the lookup-based form.

Fixes the systemd Environment= injection in sys-ctl-cln-disc-space
(it never expanded $(...)) by passing STACK / SERVICE_KEY parts and
resolving inside the script. Extracts shopware admin and mediawiki
extension install bodies into dedicated .sh files invoked via env.
Rewrites password-bearing shell tasks to either `| quote` directly or
the `-e VAR=... | quote` + `sh -c '... ${VAR} ...'` env-bridge pattern.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-10 03:46:28 +02:00
kevinveenbirkenbachandClaude Opus 4.7 5adf3011fe refactor(swarm-nfs): generalize swarm+NFS chain across all roles
Migrate the mediawiki-specific swarm patterns into shared sys-svc-compose
helpers so every role inherits swarm + NFS portability:

- meta/volumes.yml: dict schema with mandatory name|path + optional nfs
  block (uid/gid/mode); 47 roles migrated, 48 lookup consumers updated to
  volumes.<key>.{name,path}
- sys-svc-compose helpers: finalize.yml (flush + post_deploy),
  post_deploy.yml (swarm-resolve + DB-healthy-wait), nfs_prep.yml
  (pre-create NFS subdirs), swarm/resolve_self.yml (canonical
  <ENTITY>_CONTAINER fact), render_replicated_templates.yml +
  _render_one_template.yml (template render + swarm-peer replication,
  per-entry notify, fileglob support)
- 15 roles migrated to render_replicated_templates.yml helper
  (mediawiki, prometheus, kix, friendica, dashboard, nextcloud,
  suitecrm, matrix, flowise, openldap)
- 5 new lint tests pin the architecture: volumes schema, compose chain,
  volume-constants-are-lookups, template-render-centralization,
  meta-no-role-var-refs
- ROLE_FILE_TEMPL_COMPOSE SPOT constant + 7 tests migrated to use it
- Handler test scanner extended to detect nested notify in
  vars: role_templates: lists

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-02 18:57:35 +02:00
kevinveenbirkenbachandClaude Opus 4.7 2a023ca6a1 chore(lint): autoformat strict-failure semantics + claude.sh rename
* scripts/lint/autoformat.sh:
  - rename run_sort_claude_settings worker to run_claude (matches the
    renamed scripts/lint/claude.sh).
  - drop the EXIT trap that wiped status_dir; cleanup happens explicitly
    after `wait` returns, so a parallel worker that interrupts the
    parent never races a vanished status dir.
  - each worker now captures its tool's exit code and writes one of
    OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
    (markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
    real failures count as FAIL.
  - all workers run to completion regardless of individual tool failures
    (so the user always sees the full picture); after the summary, the
    script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
  (history preserved via git mv); update test_claude_settings_sorted.py
  fix-hint accordingly.

Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 01:30:49 +02:00
kevinveenbirkenbachandClaude Opus 4.7 ed04dad5d5 refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).

req-008 — role meta layout
  * Move every role's metadata under meta/<topic>.yml with file-root
    convention (no wrapping key matching the basename):
      - schema/main.yml + config/main.yml.credentials → meta/schema.yml
      - users/main.yml                                → meta/users.yml
      - config/main.yml.server                        → meta/server.yml
      - config/main.yml.rbac                          → meta/rbac.yml
      - config/main.yml.compose.services + every other top-level key
        of config/main.yml inlined under <entity>.<key>
                                                      → meta/services.yml
      - config/main.yml.compose.volumes               → meta/volumes.yml
  * Drop the legacy schema/, users/, config/ directories from every role.
  * meta/schema.yml: support nested credential keys, optional `default:`
    field (Jinja literal, not rendered/validated/regenerated), implicit
    `algorithm: plain`.
  * Materialised paths drop the `compose.` prefix:
      compose.services.<e>.<…> → services.<e>.<…>
      compose.volumes.<k>      → volumes.<k>
      <former-top-level>.<…>   → services.<entity_name>.<former-top-level>.<…>
  * Update every Python consumer (utils/cache, utils/manager, utils/applications,
    plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
    and task file to the new paths.
  * No legacy fallback. Repository-wide grep for `compose.services.` /
    `compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
    is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
  * Add tests/lint/repository/test_role_meta_layout.py guarding against the
    old paths returning, and an explicit lint guard that meta/variants.yml
    files do not carry a `compose:` wrapper.

req-009 — per-role networks/ports
  * Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
    dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
    dns_resolver}.
  * Delete group_vars/all/09_ports.yml entirely; ports now live on the
    service entity that exposes them at
    roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
  * Rename + int-cast every former top-level `services.<entity>.port`
    field to `services.<entity>.ports.inter` (single int).
  * Always-category-keyed maps for ports.local / ports.public; relay
    spans use {start, end} integers.
  * Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
    of truth for the per-<scope>.<category> band ranges, consumed by both
    the new CLI helpers and the lint test.
  * New CLI helpers (cli meta ports suggest, cli meta networks suggest)
    walk the live role tree gap-first then increment-fallback, with
    deterministic output and non-zero exit on capacity overflow.
  * cli create role consumes both helpers when scaffolding a new role.
  * Lint extensions: bare-int port shape rejection, host-bound port
    collision check (single ints + relay span integers, inter excluded),
    band-membership check with documented allow-list for the legacy
    web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.

req-010 — run_after / lifecycle migration
  * Move galaxy_info.run_after and galaxy_info.lifecycle out of
    meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
    lifecycle}; <primary_entity> is the value returned by
    get_entity_name(role_name).
  * For multi-entity roles whose primary entity has no compose service
    (web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
    gen roles), create a dedicated metadata-only top-level entry under
    <primary_entity>.
  * Introduce utils/roles/meta_lookup.py with get_role_run_after /
    get_role_lifecycle that resolve <primary_entity> once and read from
    meta/services.yml; degrade gracefully when meta/services.yml is
    absent OR the field is absent; raise MetaServicesShapeError when the
    file exists but is malformed.
  * Update every consumer (utils/service_registry, utils/invokable,
    cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
    lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
    cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
    init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
    use the helper.
  * Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
    values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
    run_after/lifecycle never on a non-primary entity; empty run_after
    lists rejected.

Also folded into this change set
  * services_disabler now writes `services.<svc>.{enabled,shared}` into
    host_vars instead of the old `compose.services.<svc>.{...}` path; the
    discover/find pair walks meta/services.yml and only treats entries
    carrying enabled/shared/provides as provider declarations (so
    metadata-only entries from req-010 do not create spurious provider
    rows).
  * service_registry skips `run_after` deps whose target was filtered
    out via SERVICES_DISABLED instead of failing the whole load.
  * cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
    consume the new applications.<role>.services.<svc>.image shape
    (mirror.md updated to match).
  * utils/domains/list.build_applications_from_roles now reads
    meta/server.yml directly (it had been silently returning {} since
    config/main.yml stopped existing).
  * utils/roles/resource_validation.filter_roles_by_min_storage reads
    meta/services.yml.<entity>.min_storage instead of the legacy
    compose.services path.
  * web-app-bluesky pre-clones the social-app build context BEFORE
    sys-svc-compose so the build context is on disk by the time
    sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
  * 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
    web-app-wordpress, web-app-keycloak) lose their `compose: services:`
    wrapper so deep-merge over meta/services.yml lands on the new
    file-root shape.

Validation
  * Unit tests:        1571 passed.
  * Lint tests:        50 passed.
  * Integration tests: 161 passed (1 skipped).
  * Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
    set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
    discourse,mattermost}.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 09:58:26 +02:00
kevinveenbirkenbach a55c99ef62 refactor(compose): replace docker lookup with container and unify compose paths
- Rename lookup plugin from `docker` to `container`
- Switch all lookups from `lookup('docker', ...)` to `lookup('container', ...)`
- Normalize compose file keys:
  - docker_compose → compose
  - docker_compose_override → compose_override
  - docker_compose_ca_override → compose_ca_override
- Centralize compose path resolution via docker_paths_utils
- Update compose_file_args to always derive paths from DIR_COMPOSITIONS
- Align all roles, templates, and vars with container/compose naming
- Update and rename unit tests accordingly

This prepares the codebase for engine-agnostic container backends
(e.g. Docker ↔ Podman) and removes legacy docker-specific assumptions.

https://chatgpt.com/share/6987a7cc-d374-800f-99ad-bb4c8f92c59e
2026-02-07 21:59:39 +01:00
kevinveenbirkenbach b78ab29218 Renamed config entry from docker to compose
https://chatgpt.com/share/69870f54-d708-800f-9870-e9ae5a4bff44
2026-02-07 11:09:36 +01:00
kevinveenbirkenbach 53e817fa22 refactor(compose): migrate from docker-compose.yml to compose.yml everywhere
- Rename all docker-compose.yml templates to compose.yml
- Update override and CA override filenames accordingly
- Adjust compose wrapper, lookup plugins, filters and path helpers
- Update roles, tasks, handlers and scripts to use compose CLI
- Rewrite documentation and admin guides to reference compose
- Update unit and integration tests to expect compose.yml
- Keep internal variable keys for backward compatibility where needed

https://chatgpt.com/share/69870272-b168-800f-84c3-6417398e441f
2026-02-07 10:14:21 +01:00
kevinveenbirkenbach 3fa7887dc8 Added whitespaces for better readability 2026-02-06 16:21:10 +01:00
kevinveenbirkenbach 5100c2c1ce refactor(config): replace get_app_conf filter with config lookup
Replace usages of the get_app_conf filter with the config lookup plugin
across roles and templates. Adjust quoting where required to keep YAML
and Jinja syntax valid. No functional changes intended.

https://chatgpt.com/share/69860678-cef8-800f-8402-7fe6231fdc2d
2026-02-06 16:19:14 +01:00
kevinveenbirkenbach 2d344fb129 refactor(domains): switch to domain() lookup across config and templates
Replace `domains | get_domain(...)` usage with `lookup('domain', ...)` throughout
group_vars, roles, and templates. Fix load_app domain resolution to use the
current application_id.

https://chatgpt.com/share/6985ef1e-11b8-800f-b231-52523055060b
2026-02-06 14:46:29 +01:00
kevinveenbirkenbach 3bb8c0ce30 refactor(tls): rename tls_resolve lookup to tls
The TLS lookup has been renamed from `tls_resolve` to `tls` to better
reflect its responsibility as a strict TLS state resolver.

This refactor updates all usages across the codebase, including:
- lookup plugins and internal helpers
- roles, vars, tasks, and templates
- tests, stubs, and error messages
- documentation and changelog references

Behavior, lookup API, and semantics remain unchanged.

https://chatgpt.com/share/6985e635-5a94-800f-b794-937e566ee308
2026-02-06 14:01:32 +01:00
kevinveenbirkenbach 8fe6b92e63 refactor(lookups): migrate nginx_paths, tls_resolve and cert_plan to strict positional want-path API
- Replace legacy want= keyword usage with mandatory positional want-path
- Align all nginx_paths, tls_resolve and cert_plan lookups to strict API
- Keep behavior unchanged while enforcing consistent lookup semantics

https://chatgpt.com/share/6985d642-b40c-800f-9cdf-ed54fc484a92
2026-02-06 13:06:09 +01:00
kevinveenbirkenbach 26a5b159e0 refactor(docker-lookup): replace want= syntax with path-based access
- Replace lookup('docker', ..., want='...') with lookup('docker', ..., 'path')
- Allow both top-level keys (directories/files) and dotted paths
- Simplify docker lookup API and reduce lazy dict propagation
- Update unit tests to reflect new lookup semantics

https://chatgpt.com/share/6984ccea-8290-800f-a6d7-ec0e94df95a0
2026-02-05 18:45:43 +01:00
kevinveenbirkenbach 56aea31667 refactor(docker): replace docker_compose vars with docker lookup across roles
Remove implicit docker_compose variable usage and resolve all docker paths
(files, directories, env) via lookup('docker', application_id).
This prevents scope leakage, centralizes path resolution, and makes docker
context deterministic across roles and handlers.

https://chatgpt.com/share/6984ccea-8290-800f-a6d7-ec0e94df95a0
2026-02-05 18:15:10 +01:00
kevinveenbirkenbach 0b9f71463c refactor(tls): replace legacy tls filters with strict tls_resolve lookup
- Remove deprecated filter_plugins/tls.py and all dependent filters
- Introduce strict lookup plugin lookup_plugins/tls_resolve.py as single TLS authority
- Migrate all roles, templates, vars, and tasks to use tls_resolve (protocols, ports, URLs, cert paths)
- Enforce explicit TLS context (domains, applications, TLS_ENABLED, TLS_MODE)
- Support letsencrypt and self_signed with SAN handling and per-app overrides
- Update TLS provider roles to assert resolver state instead of setting facts
- Add comprehensive unit tests for tls_resolve lookup

https://chatgpt.com/share/6972b49b-b074-800f-a323-dd9fe69d02eb
2026-01-23 00:36:46 +01:00
kevinveenbirkenbach 9fdb0cddd0 Replace global WEB_PROTOCOL / WEB_PORT / WEBSOCKET_PROTOCOL logic
with explicit per-application TLS resolution.

Key changes:
- Introduce sys-front-tls-layer as the single TLS abstraction
- Centralize certificate paths via TLS_CERT_FILE / TLS_KEY_FILE / TLS_CA_FILE
- Replace sys-svc-letsencrypt SSL templates with generic TLS layer includes
- Add tls_* Jinja filters (enabled, mode, protocol, port, websocket)
- Resolve TLS mode per application (letsencrypt | self_signed | off)
- Remove implicit defaults and WEB_* globals to fail fast on misconfiguration
- Support self-signed certificates with SAN awareness
- Make TLS behavior explicit and consistent across all roles

This decouples application logic from certificate providers and
allows switching TLS flavor without touching app roles.

https://chatgpt.com/share/69727c61-8e68-800f-a267-473a7e627978
2026-01-22 20:36:49 +01:00
kevinveenbirkenbach e2bf636ba9 Changed features structure 2026-01-10 11:13:29 +01:00
kevinveenbirkenbach d48a1b3c0a Solved missing variable bugs. Role is not fully implemented need to pause development on it for the moment 2025-09-05 17:07:15 +02:00
kevinveenbirkenbach 77e8e7ed7e Magento 2.4.8 refactor:
- Switch to split containers (markoshust/magento-php:8.2-fpm + magento-nginx:latest)
- Disable central DB; use app-local MariaDB and pin to 11.4
- Composer bootstrap of Magento in php container (Adobe repo keys), idempotent via creates
- Make setup:install idempotent; run as container user 'app'
- Wire OpenSearch (security disabled) and depends_on ordering
- Add credentials schema (adobe_public_key/adobe_private_key)
- Update vars for php/nginx/search containers + MAGENTO_USER
- Remove legacy docs (Administration.md, Upgrade.md)
Context: changes derived from our ChatGPT session about getting Magento 2.4.8 running with MariaDB 11.4.
Conversation: https://chatgpt.com/share/68b8dc30-361c-800f-aa69-88df514cb160
2025-09-04 12:45:03 +02:00
kevinveenbirkenbach 0e72dcbe36 feat(magento): switch to ghcr.io/alexcheng1982/docker-magento2:2.4.6-p3; update Compose/Env/Tasks/Docs
• Docs: updated to MAGENTO_VOLUME; removed Installation/User_Administration guides
• Compose: volume path → /var/www/html; switched variables to MAGENTO_*/MYSQL_*/OPENSEARCH_*
• Env: new variable set + APACHE_SERVERNAME
• Task: setup:install via docker compose exec (multiline form)
• Schema: removed obsolete credentials definition
Link: https://chatgpt.com/share/68b8dc30-361c-800f-aa69-88df514cb160
2025-09-04 02:25:49 +02:00
kevinveenbirkenbach 699a6b6f1e feat(web-app-magento): add Magento role + network/ports
- add role files (docs, vars, config, tasks, schema, templates)

- networks: add web-app-magento 192.168.103.208/28

- ports: add localhost http 8052

Conversation: https://chatgpt.com/share/68b8820f-f864-800f-8819-da509b99cee2
2025-09-03 20:00:01 +02:00