kevinveenbirkenbach/computer-playbook
1
0 Fork 0
mirror of https://github.com/kevinveenbirkenbach/computer-playbook.git synced 2026-11-10 12:47:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
14,682 Commits 14 Branches 79 Tags
fix/dev-python/python-upper-bound
Commit Graph
17 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5 b036f60039 refactor(container_ports): express the remaining published ports through the lookup 
The list form could publish a host:container pair and nothing else, so thirteen blocks stayed hand-written: the ones needing a protocol suffix, a port range, the swarm long syntax with mode host, or a container-port-only publish. Each of those duplicated numbers the role already declares in meta/services.yml, which is the drift the lookup exists to remove.

The lookup keeps the list form unchanged for its 93 existing call sites and gains a dict form: transport emits one item per protocol, container 'same' serves the services that cannot remap (STUN/TURN, media relays), mode 'host' emits the long block an edge role needs in swarm so the client address survives the hop, publish 'ephemeral' emits the container port alone, and key names the block for Discourse's launcher file, whose expose: list is what becomes docker run -p there. A declared {start, end} mapping renders as start-end on both sides, so a relay range needs no term of its own.

svc-prx-openresty had no ports block at all and bound 80 and 443 as literals in its template, while torrc.j2 carried a third literal 80 for the hidden service. None of the three referenced the others. openresty now declares the ports, the compose template reads them, and the onion's port 80 comes from ports.onion.http through TOR_ONION_EXTRA_PORTS.

Three blocks stay raw, each with the reason it cannot route through one call: keycloak's sso_proxy maps a published port owned by the consuming role to a container port owned by web-app-keycloak, and matrix's ansible flavor takes element on 8080 and federation on 8048 from the upstream playbook rather than from this repo's ports.internal.

The enforcing lint scanned only files with compose in the name and only double-quoted list items, so nine of the thirteen blocks were invisible to it. It now reads every role template that can carry a ports block, catches unquoted items, protocol suffixes, ranges, flow style and the long syntax, and tracks the block by the indentation of its key: keying on the next line closed the block on anything shaped like a key, which target, published, protocol and mode all are. Under expose: only a real mapping counts, because there the single-segment form documents a container port and publishes nothing.

Verified by rendering all 110 container_ports call sites through the real templar, the real lookup and the real merged applications view.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-15 14:55:25 +02:00
kevinveenbirkenbachandClaude Opus 5 72a7f39505 fix(sso-proxy): restart the swarm sidecar when its rendered config changes 
Snipe-IT#1 (run 34615516009, job 103327034508) failed every login in the
async pass with an oauth2-proxy 500. The sidecar log shows 24x "error
saving redis session: WRONGPASS invalid username-password pair". The swarm
matrix rotated 54 credentials before that pass, svc-db-redis re-rendered
its requirepass and redis_redis was recreated ("about an hour ago"), while
snipe-it_sso-proxy kept running from the first pass ("3 hours ago").

oauth2-proxy reads its config, redis password included, from a bind mount
at start. Its render task notifies compose-up, which in compose is
"compose up -d --force-recreate" and so recreates the sidecar, but in swarm
is "docker stack deploy", which only restarts a service whose spec changed.
A new file behind an unchanged bind mount is no spec change, so the sidecar
never read the rotated password. pihole#1 in the same run shows the compose
side: shared redis, 47 rotated values, the config re-rendered, compose up
recreated the sidecar, and the job passed.

The sidecar now carries a container label with the sha256 of its config,
rendered from the same template sso_proxy.yml writes, with
sso_proxy_application_id supplied through template_vars. A changed config
changes the label and with it the service spec, so stack deploy restarts
only the sidecar; an unchanged config keeps the hash and restarts nothing.
Compose behaviour is unchanged.

A guard test pins the label and requires it to hash the template
sso_proxy.yml renders; it fails against the previous template. The hash is
stable for equal input and changes with a rotated secret, checked with the
template lookup on ansible-core.

Not covered: other bind-mounted configs that embed rotated secrets have the
same swarm exposure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-11 23:00:36 +02:00
kevinveenbirkenbachandClaude Opus 5 96408f23a0 perf(healthcheck): let the start window outlast a real boot 
start_interval is the cadence while start_period runs, and only while it runs.
Once the window closes the probe falls back to interval, so a service that
becomes ready just after it pays a whole tick anyway and the cadence never
reached it. Six of the seven flavors were shaped that way, Curl worst: a thirty
second window against a one minute interval.

Measured across 151 converge waits in four job logs, 63 sit exactly on a tick
boundary, 36 in the thirty second step and 27 in the sixty second one.
Attributed by application, web-app-seaweedfs is the largest at 1501s over
sixteen waits with a median of 57.7s: it probes with curl and fell straight into
the gap. web-app-zammad probes with connect, whose window is fifteen minutes,
and its 154s median is real boot time with nothing to recover.

So the window moves to ten minutes and the steady state interval to two, and the
flavor overrides that sat below them go: Curl, Wget and Connect lose interval
1m, Http its twenty second window, Nc its ten. Connect keeps fifteen minutes.

Neither number costs anything while a container works. start_period delays
nothing, it only stops failures counting, and interval no longer touches
convergence at all since that hangs on the window alone; it is steady state
load, which drops from 120 probes an hour per service to 30.

Forty six role files follow. Twenty seven declarations go that repeat the new
default or undercut it, an undercut window being the defect itself. Eight
windows longer than ten minutes keep their value and now say why. The literal
blocks and the three Dockerfiles this repository builds spell every timing out,
because outside the lookup an omission takes docker's value and not the
platform's.

A fourth lint rule keeps the shape: start_period must outlast one interval,
checked against the effective values, so raising interval without the window
cannot quietly bring the cliff back.

Two faults in the lint surfaced while building it and are fixed here. It
compared timings as strings, so 600s and 10m read as different values, and it
matched the healthcheck key exactly, so a line carrying a trailing nocheck was
skipped whole, which is how the two swarm-init partials went unexamined.

One regression, bounded. A container that starts and never turns healthy is
neither rejected nor errored, so stack_ready.sh only calls it fatal after three
killed tasks, and each cycle now spans the longer window. That reaches twenty
minutes rather than six, capped there by swarm_stack_converge_retries at 600
with a two second delay, and it lands only on runs that were failing anyway.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-09 21:54:26 +02:00
kevinveenbirkenbachandClaude Opus 5 d0a8608bfa perf(healthcheck): give every probe the start cadence 42ebd9ea71 added 
42ebd9ea71 gave the healthcheck SPOT start_interval, the cadence docker uses
while start_period is still running. A block a template writes out itself
bypasses the SPOT and never gained it. Measured across the repository: 46 such
blocks, 40 of them already carrying start_period, not one carrying
start_interval.

Without it a probe fires only every interval, so a service ready after two
seconds is reported healthy at the next tick and a stack converges no faster
than its coarsest quantisation. That lands in sys-svc-compose's swarm converge
wait, 1551.7s per CI job over 38 stack deploys.

Twenty five of the blocks move into meta/services.yml and render through
container_healthcheck. Each keeps an explicit test rather than picking a flavor:
a flavor would replace the command instead of carrying it across.

The move is lossless by construction. probes.py block() writes
overrides.get(key, default) for interval, timeout, retries, start_period and
start_interval, so a declaration repeating the literal values renders the same
block plus the missing cadence. Checked per block against the old values: the
only difference is start_interval 5s, nothing changed, nothing lost.

Twenty one stay literal. Seventeen interpolate a template variable, whose value
resolves in a scope meta/services.yml does not share, so moving them would be a
rewrite rather than a transcription. Four interpolate nothing and still cannot
move: svc-db-memcached and svc-db-redis probe every second in their swarm
service partial, where a 5s start cadence would report healthy later than the
block already does, and the sys-svc-rdbms and keycloak sso_proxy partials render
for the consuming role's application_id, so their declaration has no single
meta/services.yml to live in. Each carries a nocheck naming its reason.

All twenty one gain start_interval. Three also gain start_period, without which
the key does nothing at all, since it only applies while the start period runs.
That widens their failure tolerance rather than narrowing it: bluesky from 90 to
120 seconds, magento from 50 to 80, the mariadb partial from 180 to 210.

pixelfed spelled its probe as a bare string and now spells it CMD-SHELL. Docker
reads both as a shell command; only the YAML shape changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-09 20:31:07 +02:00
kevinveenbirkenbachandClaude Opus 5 8c7593f43f refactor(container): make container_extra_hosts the only emitter of extra_hosts 
extra_hosts is the one compose key a service may not carry twice. The onion SSO
back-channel pin was emitted by a Jinja partial that six roles included, while
six other services wrote the key by hand; nothing stopped a service from
receiving both. YAML then either rejects the document or keeps one list and
drops the other, silently.

The partial becomes plugins/lookup/container_extra_hosts.py, and every pin now
goes through it. Callers add their own entries with extra_hosts=[...], the way
container_volumes already takes extra_volumes=, and the lookup merges them after
the automatic pin and deduplicates in first-seen order. With nothing to emit it
returns an empty string.

The swarm address reasoning from 7b1839681 is preserved and now written down in
the plugin rather than only in that commit: the pin must NOT be host-gateway in
swarm, because that alias resolves per node while openresty publishes :80 with
mode: host under node.role == manager. It carries the stack host's address,
which reaches every task because the compose file is rendered once with the
manager as inventory host. The mode is read as compose_mode_force or
DEPLOYMENT_MODE, not the bare cluster mode, so a role with an override does not
lose its pin and its overlay at once.

Two roles kept a pin that now collides with the automatic one, and both are
resolved the same way -- the hand-written entry stays for clearnet and yields
the .onion case to the lookup, which picks the address per mode. In
web-app-mailu the three conditional entries move into a role-local
mailu_extra_hosts lookup exposed as MAILU_EXTRA_HOSTS, because an eight-line
conditional expression in the template argument is worse than the block it
replaced. It returns its entries as separate results, so vars/main.yml uses
query(); plain lookup() would join them into one comma-separated string.

tests/lint/ansible/jinja/test_compose_extra_hosts_use_lookup.py fails on any
literal extra_hosts: key in a role template, so the invariant survives the next
author who needs a pin. It follows test_compose_ports_use_container_ports_lookup:
the raw construct is forbidden, the lookup is not mandated per service -- a
service with no pins is correct without it, and the swarm branch raises when
facts are absent, which would turn a no-op into a failure surface for containers
that never speak OIDC.

Equivalence checked by rendering rather than by reading: under the trim_blocks +
lstrip_blocks the stack-host action forces for compose.yml.j2, both former call
site shapes (indented, and matrix's column 0) emit byte-identical text to the
new form when the block applies. The no-op case differs -- a whitespace-only
line where the include emitted none -- and yaml.safe_load returns the same
document for both.

31 unit tests; make test-lint 325 passed. Does not touch web-app-espocrm, whose
missing pin is a separate claim and gets its own commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-14 02:02:28 +02:00
kevinveenbirkenbachandClaude Fable 5 fa2f6eb7e3 fix(sso): pin the onion OIDC issuer via host-gateway in swarm too 
The compose-only guard left swarm sso-proxies resolving auth.<onion>
through the embedded DNS, which races container startup and
intermittently hands back a loopback that dead-ends in the task netns
(token exchange refused). The static extra_hosts entry wins in both
modes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-22 15:24:08 +02:00
kevinveenbirkenbachandClaude Fable 5 d88f272f9f Merge branch 'feature/swarm-nfs' into feature/svc-net-tor 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-19 15:31:08 +02:00
kevinveenbirkenbachandClaude Opus 4.8 bb8cf66622 fix(web-app-keycloak/sso-proxy): only require provider CA for https OIDC issuers 
Over an onion-exclusive deployment the OIDC issuer is plaintext http (.onion has no TLS), so provider_ca_files pointed oauth2-proxy at a CA file that was never mounted into the sidecar — crashing it at startup (connection refused on the auth gate, no redirect to Keycloak). Gate provider_ca_files on an https issuer and mount the root CA into the sso-proxy container for the self-signed https case (the file was previously never mounted, a latent crash there too). Verified over onion: prometheus admin/biber/guest specs 5 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-08 18:20:30 +02:00
kevinveenbirkenbachandClaude Opus 4.8 544e4b83fc fix(networks): make objstore providers (seaweedfs/minio) shared-net providers 
The net refactor (366dc7b70/cb27ff375) migrated every shared-net provider (postgres/mariadb/openldap/ollama/openresty/keycloak) to the overlay registry but missed seaweedfs and minio, so an object-store consumer like baserow never joined the provider's docker network and its S3 client hit ECONNREFUSED on seaweedfs:8333 (the intended direct data-plane path).

Add networks.overlay to both engines so consumers attach, and complete the multi-service shared-net-provider contract: (1) render.py emits a bare auto-IPAM project default for an own shared_net provider (no name, no subnet) so the entity-named external net created by network/create.yml is not re-requested -- this caused 'invalid pool request: Pool overlaps' on the first attempt. (2) a provider_self_alias flag lets the provider's sidecars (proxy/sso-proxy) JOIN the shared net WITHOUT publishing the <entity> alias, so the name resolves only to the main container and baserow's :8333 never round-robins onto a port-less sidecar. (3) seaweedfs/minio 01_core pre-create the external <entity> net via network/create.yml (mirroring web-app-prometheus) so compose adopts it as the project default instead of building <entity>_default.

Verified: baserow compose e2e test-seaweedfs.js passes (upload stored in the bucket, 3.9s); new TestOwnSharedNetProvider locks the contract; full make test green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-20 00:44:38 +02:00
kevinveenbirkenbachandClaude Opus 4.7 cb27ff3756 refactor(networks): replace Jinja includes with compose_networks/container_networks lookup plugins 
Convert the two registry-driven networks templates into pure-Python
rendering functions plus thin Ansible lookup plugins:

  roles/sys-svc-compose/templates/networks.yml.j2  -> {{ lookup('compose_networks') }}
  roles/sys-svc-container/templates/networks.yml.j2 -> {{ lookup('container_networks') | indent(4) }}

The rendering logic lives in utils/networks/render.py as pure-Python
functions that take a registry + closure-style config/database lookups,
making the logic directly unit-testable without a Jinja stub harness.
The lookup plugins are thin wrappers that resolve application_id,
DEPLOYMENT_MODE, and swarm.network.encryption from the templar, build
the service_registry, and forward to the render functions.

Migrate 233 include lines across 85 production templates. Convert
web-app-matrix's hand-coded MDAD networks block (default + conditional
postgres + conditional openldap) to the generic lookup; this adds an
openresty attachment when services.sso.enabled (harmless extra
interface, pinned in test_matrix_mdad_consumer_attaches_postgres_ldap_sso).

Mark local-DB / redis sidecars (sys-svc-rdbms/templates/services/,
svc-db-redis/templates/service.yml.j2) with `# nocheck: networks-literal`
because they intentionally attach to default-only.

Extend the lint to scan ALL roles/*/templates/**/*.yml.j2 (not only
compose.yml.j2) and to forbid the legacy include form; the marker
above is the documented escape hatch.

Coverage:
  - tests/unit/utils/networks/test_render.py (30 cases incl. encrypted=false literal pin)
  - tests/unit/plugins/lookup/test_compose_networks.py (5 cases incl. lookup_loader arg-schema)
  - tests/unit/plugins/lookup/test_container_networks.py (5 cases incl. lookup_loader arg-schema)
  - tests/integration/infrastructure/compose/test_networks_render.py (10 snapshot scenarios)
  - tests/integration/infrastructure/compose/test_network_includes.py (3 lint rules)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-16 18:42:04 +02:00
kevinveenbirkenbachandClaude Opus 4.7 366dc7b705 refactor(overlay): registry-driven shared-overlay networks with implicit defaults 
Replace the hand-coded if-chains for db/ldap/ollama/proxy shared overlays
with a single registry-driven loop in sys-svc-{compose,container}/networks.yml.j2.
Each provider declares its overlay metadata under meta/server.yml.networks.overlay;
discover_role_services passes the block through verbatim into the service_registry.

Schema (modes/topology/consumer/aliases/proxy_resolvable) with implicit defaults:
- consumer.kind defaults to services_flags
- consumer.key defaults to <provides or entity_name>
- consumer.flags defaults to [enabled, shared]
- aliases defaults to [entity_name] (shared_net) or [] (default_net)

Openresty becomes a generic alias-collecting proxy: web-app-keycloak declares
a proxy_resolvable beacon with its public auth.<domain> alias; openresty sweeps
the registry at render time and harvests all proxy_resolvable aliases onto its
default-net overlay. Openresty no longer hardcodes keycloak's domain.

Lock the new behaviour with 9 render-snapshot tests covering provider self-render,
consumer attachment, beacon sweep, default-derivation, and compose-mode fallback.
Replace the Ansible-only `_entry.get('canonical') is not defined` filter with
`'canonical' not in _entry` so the templates also render correctly under vanilla
Jinja2 (test harness).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-16 18:42:03 +02:00
kevinveenbirkenbachandClaude Opus 4.7 b627e383a1 fix(keycloak-swarm): manager pin + LOCAL_CID + chdir + sso-proxy DNS/CA 
Make web-app-keycloak deployable in swarm without the manager-only
assumptions of the original compose-only shape.

Manager-side delegate scaffolding:
* tasks/02_manager_ops.yml: switch the `include_tasks` of
  02a_manager_ops_on_host.yml to the dict form so the `apply:
  delegate_to: KEYCLOAK_HOST_NODE` block actually attaches (free
  form raised `conflicting action statements: include_tasks, apply`
  in ansible 2.19).

Swarm bind sources that exist only on the manager:
* tasks/05b_realm_import_swarm.yml: pin the one-shot
  `<stack>_<service>_import` service to `node.role == manager` so
  the realm-import bind mount `/opt/compose/keycloak/volumes/import`
  resolves.
* templates/sso_proxy/container.yml.j2: same manager pin for the
  sso-proxy sidecar (its oauth2-proxy cfg bind source also lives
  only on the manager). Also add `extra_hosts` for the keycloak
  auth domain pointing at `host-gateway` so OIDC discovery reaches
  the host's openresty, and mount the self-signed CA when TLS_MODE
  is `self_signed`.
* templates/sso_proxy/oauth2-proxy-keycloak.cfg.j2: add
  `provider_ca_files` so oauth2-proxy validates the self-signed
  cert instead of failing OIDC discovery with TLS EOF.

CA mount path SPOT:
* group_vars/all/02_tls.yml: add `CA_TRUST.cert_container` for the
  in-container CA file path so the literal `/etc/ssl/certs/...` is
  defined once.
* roles/web-app-matrix/vars/main.yml: consume the new SPOT.

Container ID resolution in delegated tasks:
* vars/main.yml: add `KEYCLOAK_TARGET_CONTAINER` (= LOCAL_CID with
  fallback to KEYCLOAK_CONTAINER_ADDRESS) and recompose
  `KEYCLOAK_EXEC_CONTAINER` from it, replacing the inline default.
* tasks/update/05c_per_app_mappers.yml and
  tasks/update/ldap/per_app_mapper_one.yml: `KC_CONTAINER` env now
  resolves via the new SPOT, so the per-app-mapper helper scripts
  hit the actual swarm container ID on the delegated host node
  (compose mode keeps using the bare container_name via the
  fallback).

Delegated script paths:
* tasks/update/08_users_email_verified.yml and
  tasks/update/08_users_realm.yml: drop
  `chdir: "{{ role_path }}/files"` from `ansible.builtin.script`.
  role_path is a controller-side absolute path; on a delegated
  DinD swarm node that path doesn't exist and the script fails to
  start. Both scripts only consume env vars.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-16 13:34:06 +02:00
kevinveenbirkenbachandClaude Opus 4.7 0ffbb73d72 refactor(images): central container_image lookup + migrate all roles 
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).

* 25 unit tests for the lookup cover compose vs swarm prefix, custom
  flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
  fails on any raw `image:` line in roles/*/templates/*.j2 unless
  nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
  longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
  `{{ lookup('container_image', application_id, '<service_key>') }}`
  form. 26 roles use `custom=True` (mariadb, postgres, openldap,
  jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
  odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
  fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
  into consuming roles where application_id evaluates to the outer
  role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
  test_build_requires_image) recognise the standalone lookup as an
  implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
  the registry-prefixed image string because the custom var is now
  bare.

Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-16 00:17:11 +02:00
kevinveenbirkenbachandClaude Opus 4.7 587a3a1796 feat(swarm): consolidate container_name + pull_policy into generic compose_only lookup 
Replace per-key `compose_container_name` lookup with a generic
`compose_only(key, value)` SPOT that emits `key: "<value>"` only in
compose mode and nothing in swarm. Same gating logic now serves
container_name (157 call sites) and pull_policy (17 call sites)
without duplicate plugin code.

Why this consolidation:
- The earlier `build-key` lint rule was misdirected. Swarm tolerates
  `build:` with a harmless warning when `image:` is pre-pushed (via
  the existing handlers/swarm_registry_push.py flow in
  roles/sys-svc-compose). The actual swarm-rejected key is
  `pull_policy:` ("Additional property pull_policy is not allowed"),
  and 10 templates were silently broken because they declared inline
  `pull_policy: never` outside the existing `{% if DEPLOYMENT_MODE
  \!= 'swarm' %}` gate in roles/sys-svc-container/templates/build.yml.j2.
- A single generic lookup is cleaner than N per-key wrappers when
  the gating logic is identical (emit-key-or-nothing).

Lint changes in tests/lint/ansible/jinja/test_mode_incompatible_keys.py:
- New `pull-policy-key` rule, regex `^\s*pull_policy\s*:`, remediation
  points at compose_only.
- Existing `container-name` rule keeps its raw-key regex; remediation
  now also points at compose_only.
- Broken-split / broken-embed regexes rekeyed on the `compose_only`
  lookup name (was `compose_container_name`).
- `build-key` rule and its 12 nocheck annotations across templates
  removed.

Coordinated changes:
- 86 templates rewritten (157 container_name calls migrated;
  17 inline pull_policy lines replaced).
- 1 redundant `{% if DEPLOYMENT_MODE \!= 'swarm' %}` guard dropped in
  build.yml.j2 (the lookup self-gates).
- 12 unit tests pin the compose_only contract.
- Catalog (suppression.md) updated: `pull-policy-key` row added,
  `container-name` row remediation updated.
- Old `plugins/lookup/compose_container_name.py` and its test module
  deleted.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-05 02:24:50 +02:00
kevinveenbirkenbachandClaude Opus 4.7 3816e69638 feat(swarm): compose_restart lookup gates restart key per deploy mode 
Mirror compose_container_name pattern: plugins/lookup/compose_restart.py
emits `restart: <policy>` only in compose mode. In swarm the key is
silently ignored (replaced by deploy.restart_policy) and produces
"Ignoring unsupported options" warnings at docker stack deploy time.

Sweep 19 raw restart: sites across 10 templates to the no-arg form
`{{ lookup('compose_restart') }}` (the lookup falls back to
DOCKER_RESTART_POLICY from ansible-vars).

base.yml.j2 keeps the explicit-arg form
`{{ lookup('compose_restart', docker_restart_policy | default(DOCKER_RESTART_POLICY)) }}`
so the 5 templates that override the policy via Jinja-scope
`{% set docker_restart_policy = ... %}` (erpnext, zammad, shopware,
matomo, openproject) retain their override path. Jinja-scope bindings
are visible only in the rendering template, never in the Ansible
variable context that the lookup sees, so the caller must resolve them.

Generalize the container-name lint into a data-driven multi-rule
scanner at tests/lint/ansible/jinja/test_mode_incompatible_keys.py
covering both container-name and restart-key. Adding a new rule is one
ModeRule row sharing the file-walk / suppression / line-grouping
scaffolding.

Also gate roles/svc-db-redis/templates/service.yml.j2's
resource.yml.j2 include with DEPLOYMENT_MODE \!= 'swarm'. Mirrors
base.yml.j2's existing pattern; prevents cpus / mem_limit /
mem_reservation / pids_limit from leaking into swarm output where
deploy.resources.* in deploy.yml.j2 already handles them.

11 unit tests pin the compose_restart lookup contract. Both rules
registered in docs/contributing/actions/testing/suppression.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-05 02:24:50 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a0f0e3ac03 feat(swarm): SPOT lookup gates container_name emission per deploy mode 
container_name is illegal in swarm mode when replicas > 1 (docker rejects
with "services.deploy.replicas: can't set container_name and X as container
name must be unique"). All compose.yml.j2 templates now route through a
single lookup that suppresses the key in swarm mode and emits it in compose
mode.

SPOT: plugins/lookup/compose_container_name.py
  Reads DEPLOYMENT_MODE from the templating context, returns
  'container_name: "<name>"' under compose and an empty string under swarm.
  Call sites read:

      {{ lookup('compose_container_name', MY_CONTAINER) }}

  No explicit DEPLOYMENT_MODE pass-through needed.

Sweep: 82 templates rewritten across roles/web-app-*, roles/web-svc-*,
roles/svc-*, plus 4 partial includes (sso_proxy, redis, mariadb, postgres).
For container names assembled from multiple Jinja fragments the args use
tilde concat (X ~ '-' ~ Y) so the lookup receives a single resolved string
and the call stays a single well-formed Jinja expression.

Lint: tests/lint/ansible/jinja/test_no_raw_container_name.py blocks
  (a) raw 'container_name:' YAML keys in any roles/*/templates/**/*.j2,
  (b) split-Jinja regression where the outer braces close mid-call:
      {{ lookup('compose_container_name', A }}-{{ B) }},
  (c) embedded-Jinja regression inside a string argument:
      lookup(..., "{{ X }}_suffix") (removed in ansible-core 2.23).
  The (b) and (c) regexes accept either single- or double-quoted lookup
  name and scan the full file so multi-line shapes also surface. Suppress
  per-site with '# nocheck: container-name' on the same or preceding line.

Tests: tests/unit/plugins/lookup/test_compose_container_name.py pins the
lookup contract: compose vs swarm output, whitespace tolerance, default-to
compose when DEPLOYMENT_MODE is absent, AnsibleError on missing or excess
terms, non-string name coercion.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-05 02:24:50 +02:00
kevinveenbirkenbachandClaude Opus 4.7 24d4ab8e4a refactor(sso)\!: collapse oauth2 + oidc into unified services.sso block (req 21) 
Implements docs/requirements/021-sso-flavor-migration.md atomically:

* Schema: every role's meta/services.yml replaces the parallel
  `oauth2:` / `oidc:` top-level blocks with a single `sso:` block
  carrying `enabled`, `shared`, and a `flavor` discriminator
  (`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
  sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
  allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
  entirety; its 5 sidecar templates move under
  roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
  rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
  web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
  centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
  `is_oidc_native`) and the oauth2 sub-values
  (`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
  Three lookup plugins consume the resolver: `sso` (single-app
  predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
  consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
  plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
  and plugins/filter/compose_volumes.py all gate on
  `lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
  `enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
  → `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
  `OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
  `oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
  `OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
  `SSO_SERVICE_ENABLED`; Playwright gating helpers
  (isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
  safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
  is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
  `flavor: oauth2` with their existing `oauth2.acl.blacklist`
  (/user/login) and `oauth2.acl.whitelist` (federation + discovery
  endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
  (tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
  test_acl_mutual_exclusive.py) are deleted per Decision #10;
  test_oauth2_contract.py is renamed and rewritten to
  test_sso_contract.py, asserting that `flavor: oauth2` roles carry
  `oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
  (web-app integration) now requires the unified `sso:` block with an
  explicit flavor or a `# nocheck: sso` opt-out; new
  tests/lint/repository/no_legacy_sso_paths/ guards the source tree
  against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
  009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
  docs/contributing/actions/testing/suppression.md are updated to the
  unified SSO naming.

Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).

BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-23 12:59:59 +02:00