Problem:
Every compose service logs to journald through
sys-svc-container/templates/base.yml.j2. The daily journal health
check grepped the whole journal for "error", so routine application
errors from containers triggered the host alert (email/Telegram) and
buried real host problems.
Change:
- Read the last day of the journal as JSON (MESSAGE, CONTAINER_ID,
SYSLOG_IDENTIFIER, _SYSTEMD_UNIT) and skip every entry that carries
a CONTAINER_ID field. Docker daemon and system errors are still
reported.
- Report each match as one readable "source: message" line instead
of raw journal output.
- Stream the journalctl output line by line instead of buffering a
whole day of logs in memory.
- Exit 2 with journalctl's stderr if journalctl itself fails, instead
of silently reporting "no errors".
- Rewrite the check in Python (files/python/script.py, deployed by
sys-service like the other sys-ctl-hlth roles) and decode non-UTF-8
messages.
- Fix the success message, which claimed "All docker containers are
healthy".
Impact:
Container log errors no longer raise this host alert on any
installation. Container health stays covered by sys-ctl-hlth-container;
application log analysis belongs in a log platform such as Wazuh.
system_service_suppress_flush is unchanged until a deploy confirms
that no host errors remain.
Tests:
Unit tests run the script against a fake journalctl (8 cases):
container error ignored, daemon error reported, identifier fallback,
non-UTF-8 message, journalctl failure, many container lines streamed
past, clean journal passes. The core cases fail against the previous
script.
The manager and indexer nofile limits now live next to their other resource settings in services.yml and the compose template reads them from there, as requested in review. The values are unchanged.
The platform always sets CA_TRUST_NAME together with CA_TRUST_CERT, so the infinito-dev-ca default was dead code that could hide a missing setting. The entrypoint now fails with a clear message instead.
The security-analyst page never calls the roles API, so the response wait timed out in CI. The non-admin tiers now check that the user stays inside the Wazuh app, sees a non-empty page and sees neither the role list nor the create-role button. Failure messages include the page URL and text.
The indexer entrypoint is now only our CA-import block followed by exec of the image's own /entrypoint.sh, mounted at /infinito-entrypoint.sh with an explicit command. The certs entrypoint is our own script: it downloads the cert tool only from packages.wazuh.com and fails if that download fails, so there is no silent fallback to the dev bucket. The manager node name is passed in as WAZUH_MANAGER_NODE for both compose and swarm.
WAZUH_CERTS_TOOL_VERSION was a literal in vars/main.yml. It now reads the manager image version from meta/services.yml and keeps major.minor (4.14.6 gives 4.14), so it follows the Docker updater and needs no separate pin.
bcrypt_hash takes an optional salt_seed. Without a seed it keeps the random salt. With a seed the salt is derived from the seed and the secret, so the same password always gives the same hash. Wazuh passes a fixed seed for the indexer admin and the dashboard service hashes, so internal_users.yml no longer changes on every run and no longer triggers a restart.
The manager and the indexer do not set bootstrap.memory_lock, so memlock -1 is not needed and would let a leaking process pin all host RAM. nofile stays: it is a per-process limit and OpenSearch needs a high file-descriptor limit.
Wait for the security-dashboards-plugin roles API response and assert 200 for the admin tier and 403 for the other tiers, with the role list and create-role button checked on the page. Navigate with gotoOnion so the test works over Tor.
The Ace editor overlays its hidden textarea and intercepts pointer events, so the click never reached the input. Focus the editor and let a failure show up instead of swallowing it.
Put roles/test-e2e-playwright back to origin/main: the uppercase TEST_E2E_PLAYWRIGHT_WAIT_RETRIES/_DELAY constants, with no lowercase defaults and no per-role override. Wazuh's own override is removed, since nothing reads it any more. The longer wait was added to fix 502s that were really the swarm proxy pointing at the wrong service name.
Tor: use the shared tor block (bond 0.5, enabled when svc-net-tor is in the node's groups, tor pinned on in both variants) so a node serves one network, onion on a Tor node and clearnet otherwise.
SPOT and sizing: WAZUH_INDEXER_URL replaces four hand-built copies of the indexer URL, and the indexer heap is half of mem_limit instead of a hardcoded 1g (unchanged for the current 2g limit).
Fixes: bcrypt_hash rejects secrets over 72 bytes; the Keycloak group-removal helper uses the same resolver as the add helper and checks every response (the realm is no longer URL-encoded, matching the add helper), and a failed cleanup fails an otherwise passing test; the swarm certs generator removes a stale container before each attempt; vm.max_map_count is set before the early handler flush; securityadmin failed_when now needs rc 0 and the success marker.
The regenerated role README also picks up generator drift (mailu to stalwart).
docker stack deploy ignores depends_on, so manager, indexer and dashboard were scheduled together with certs_generator and rejected with 'bind source path does not exist' for the cert files. stack_ready.sh treats a Rejected newest task as fatal, so the converge failed on its first poll whenever certs_generator had not finished first.
In swarm, the certs_generator service is no longer rendered. Instead the stack host pulls the generator image and runs it once with the same entrypoint, env and mounts, before the stack deploys. The run is checked against a completion marker (sha256 of certs.yml plus tool version) and all eight cert files. Compose mode renders the same output as before.
Existing swarm hosts: a host whose Wazuh stack is already running has no marker yet, so the next deploy stops with an explicit error rather than rotating the CA under the live services. Either run 'docker stack rm wazuh' and redeploy, or write the expected marker value given in the error to <instance>/certs.complete, then redeploy. Compose hosts are not affected.
web-app-stalwart, merged on main, claims local.http 8095 (stalwart) and 8094 (webmail), the same ports the Wazuh indexer and dashboard held. Both new ports come from cli.contributing.network.ports.suggest against the rebased tree.
Per Kevin's review: replaced hardcoded usernames (wazuh-wui, kibanaserver,
admin) across wazuh.yml.j2, compose.yml.j2, internal_users.yml.j2,
roles_mapping.yml.j2, config.yml.j2 and 00_core.yml with
lookup('users', ...) references, following the same pattern already
used by web-svc-coturn and web-app-semaphore. New meta/users.yml
declares web-app-wazuh-dashboard (kibanaserver) and web-app-wazuh-api
(wazuh-wui) as service accounts, and references the existing shared
admin key as a dependency (admin: {}), matching the same pattern used
by openbao, pgadmin, checkmk, semaphore and openldap.
Per CodeRabbit's security finding (CWE-269): scoped readonly_auditor's
index_patterns from '*' to 'wazuh-*', matching the role's own stated
'all Wazuh indices' description. The wildcard previously granted read
access to every index on the cluster, including other users' saved
objects and plugin state, beyond what the role needs or describes.
playwright.env.j2's SUPER_ADMIN_PASSWORD/USERNAME lines looked up
web-app-keycloak's secrets unconditionally, with no check that
Keycloak is actually part of the deploy. This has been true since the
role's first commit, but was silently masked: the role's old
tor: enabled: true pulled in svc-net-tor, which declares Keycloak as
its onion-SSO provider, so Keycloak always ended up in the deploy
regardless of variant.
Removing tor (fix(web-app-wazuh): serve clearnet only) correctly
stopped that accidental masking - variant 1 (SSO/LDAP off) no longer
pulls Keycloak in at all, so the unconditional lookup now fails with
'Key administrator_password not found', confirmed on both a fedora
(compose) and centos (swarm) CI run.
Guards both lookups on WAZUH_OIDC_ENABLED (vars/main.yml, already
resolved from services.sso.enabled - i.e. whether Keycloak is in
group_names), matching the existing guards on PERSONA_ADMINISTRATOR_
BLOCKED and SSO_SERVICE_ENABLED a few lines above in the same file.
Falls back to an empty quoted string when SSO is off, so the lookup
never executes on that branch.
Verified: rendered both lines directly with a stub lookup that raises
if called - confirms the lookup is never evaluated when OIDC is off.
Confirmed variant 0 (SSO on) is unaffected. 22 passing tests across
the role's playwright/lint/mcp-secret suites.
Not run: a live compose/swarm deploy of variant 1 - the actual proof
this resolves the CI failure, pending next CI run.
While investigating, found the same unguarded-lookup pattern exists in
web-app-zammad (reads svc-db-postgres's password with no check that
postgres is in the deploy) - flagged for Kevin, not touched here since
it's unrelated to anything this branch changes.
Wazuh is a SIEM console and should never be served as an onion
service. Following web-app-peertube's established pattern for a
clearnet-only role:
- meta/services.yml: tor block set to enabled: false / shared: false,
with # nocheck: sso-variant-tor documenting why (the SIEM console is
never served over onion); bond and variants-coverage nocheck removed
since they no longer apply.
- meta/variants.yml: removed both tor: enabled: true entries.
- README.md and the root README index regenerated via make docs to
match (tor dependency removed from the diagram and index row).
roles/web-app-openbao (merged into main since our last sync) also
claims 8093 for its local.http port. Moved Wazuh's indexer to 8095,
confirmed free via the port-suggestion tool and a repo-wide check for
existing users. Nothing else in the role hardcodes the port number -
everything reads it from vars/main.yml's WAZUH_INDEXER_HTTP_PORT.
sys-svc-compose's shared 'compose up' handler resolves its chdir target
via lookup('container', application_id, 'directories.instance') fresh
at flush time, not frozen at the moment it's notified. Many roles set
their own identity via set_fact: application_id: <role> as their first
task (svc-db-openldap, web-app-gitlab, svc-ai-litellm, and many more -
a widespread platform convention), which permanently overwrites the
play-wide fact for every host it runs on. web-app-wazuh instead reads
application_id from vars/main.yml (a role var, lower precedence than
set_fact).
If any role using that set_fact pattern runs in the same play after
Wazuh's own notify: compose-up but before whichever later flush point
(sys-stk-front-proxy's meta: flush_handlers) actually fires the
handler - which happens whenever Wazuh's SSO dependency chain
(Keycloak, OpenLDAP, etc.) is deployed in the same play - the handler
silently targets that other role's directory instead of Wazuh's own.
Confirmed on a live CI run: compose up only ever created openresty's
containers, never any wazuh_* container, despite Wazuh's own
compose.yml.j2 rendering correctly beforehand.
This is a general, platform-wide hazard (any set_fact-identity role
downstream of a deferred shared-handler flush is exposed the same
way), not something fixable from inside this role alone - the proper
fix belongs in sys-svc-compose itself, as its own separate change.
This adds an explicit meta: flush_handlers immediately after the last
notify: compose-up task, before vm.max_map_count or the front-proxy
include - closing the vulnerability window entirely from Wazuh's own
task file: application_id is still guaranteed correct at that point,
since the render task immediately prior already required it to be
correct to write to the right path. Nothing after this point in
00_core.yml depends on the compose stack already being up, so the
earlier flush is semantically safe.
Verified via a repro mirroring the exact scenario (notifier role with
an early flush, then a dependency role doing set_fact: application_id)
using real ansible-playbook: the early flush consumes the notification
before the later role's set_fact can run, confirming the fix closes
the window.
plugins/action/uri_retry.py only bypasses its own internal retry loop
when until: is present on the task - but Ansible core's own retry
loop activates on retries: alone, independent of until:
(ansible/executor/task_executor.py:606-609). The 'Wait for Wazuh
indexer to answer HTTPS requests' task had retries: 60 with no
until:, so both loops fired: core's outer loop invoked the plugin
fresh on each of its own 61 attempts, and the plugin's own inner loop
ran its full 61-attempt cycle every time - up to 61x61 = 3,721 HTTP
requests instead of the intended 61, silently multiplying an
intended ~5-minute patience window into hours with zero log
visibility into the inner loop. This is what was actually consuming
the entire CI job timeout on debian/fedora (rc=124), not a genuine
indexer slow-boot or a resource-contention issue.
Added register: wazuh_indexer_ready + until: wazuh_indexer_ready is
succeeded, matching the existing until: <var> is succeeded idiom
already used elsewhere in the repo (svc-db-mariadb, svc-db-postgres).
This makes uri_retry delegate entirely to Ansible core's single retry
loop, as the plugin was actually designed to do.
Verified against a real HTTP server counting actual requests: 16
requests without the fix (exactly (retries+1)^2 = 4^2), 4 requests
with it (exactly retries+1) - both matching the theorized mechanism
precisely. Confirmed the role's only other uri_retry usage already
has until: and is unaffected.
Ansible validates a notified handler name against the play's handler
registry at the moment a task runs, not at flush time. Handler
registration here is play-global and execution-order-dependent (the
whole task graph is dynamic include_role, no static roles: list), so
compose-up (owned by sys-svc-compose, loaded transitively via
sys-stk-backend) only exists once something in the play has actually
included it.
00_core.yml's four notify: compose-up tasks ran before its own
include_role: name: sys-stk-backend, so on any host where nothing else
pre-loads the handler, the first notify hard-fails: 'The requested
handler compose-up was not found...'. This was masked in every prior
live-deploy verification because Keycloak/OpenLDAP (this role's own
run_after dependencies) are themselves compose-based and register the
handler first when deployed on the same host - but the per-role CI
matrix runs Wazuh standalone, with nothing to mask it.
Verified via a minimal two-role reproduction outside the project tree:
the exact failure reproduces with the original ordering and clears
with the fix, using real ansible-playbook, not just code inspection.
Moved the sys-stk-backend include immediately after the once-flag
guard, before all four notify-bearing tasks.
Seven real, branch-caused failures surfaced after rebasing onto main
(v14.0.0, ~900 commits):
- opensearch_dashboards_yml volume: type: config -> type: secret
(renders a live OIDC client secret; was world-readable at 0444 in a
volume type the platform treats as non-sensitive, same fix pattern
as web-app-gitlab's rails_secrets_yml)
- meta/networks.yml: local.subnet collided with web-app-litellm
- meta/services.yml: indexer/console ports collided with
svc-ai-litellm and web-app-openclaw (both added during the gap)
- compose.yml.j2: healthcheck interval/timeout/start_period now go
through the SPOT lookup instead of hardcoded values
- README.md: regenerated Quick Setup section, fixed credits link
formatting
- docs/requirements/035-mcp-proxy-expansion.md: added web-app-wazuh
to the adapter_candidate list (has a real, currently-unpublished
API surface - manager :55000, indexer :9200 - not yet evaluated
for an MCP adapter, same bucket web-app-checkmk/prometheus sat in
before promotion)
Two regressions caught by re-verification and fixed in the same pass:
meta/services.yml briefly reached into a role var directly (fixed to
use the lookup('config', ...) pattern already used elsewhere), and the
credits-link fix dropped required bold markdown.
Remaining test failures after this fix are confirmed pre-existing and
unrelated: svc-net-tor's nftables guard rule (kernel EOPNOTSUPP), an
INFINITO_SWARM_STEP_TIMEOUT_MINUTES Makefile/default.env drift, and a
CI chunk-budget drift.
Registers Wazuh's onion vhost with the front proxy and makes OIDC login
work correctly on either domain, following up on Kevin's request to
integrate this into the PR so CI can actually verify it.
- tasks/00_core.yml registers a second, guarded front-proxy include for
the onion domain (WAZUH_DASHBOARD_ONION_DOMAIN | length > 0), using
the same swarm-mode 'dashboard' compose-service-key fix already
applied to the clearnet include.
- The dashboard's OIDC redirect URL is now derived per request
(opensearch_security.openid.trust_dynamic_headers, driven by
X-Forwarded-Host) instead of a single static base_redirect_url that
could only ever match one of the two domains. Requires this role's
own templates/proxy.conf.j2 (forked from sys-svc-proxy's
basic.conf.j2) since the shared vhost template doesn't forward that
header.
- Restores WAZUH_DASHBOARD_ONION_DOMAIN in vars/main.yml, which had
been dropped during an earlier commit-splitting cleanup in this same
branch's history; removes the now-unused WAZUH_DASHBOARD_BASE_URL.
Trimmed on rebuild: the roles/sys-front-tls/ hunks are dropped; they
served only the dual-stack attempt.
Verification status: make lint-ansible / test-lint / test-unit-python
all pass. Not yet exercised against a live svc-net-tor deployment or
real CI onion-mode run - that's the actual verification this commit
unblocks, per Kevin's note that CI can't confirm the implementation
works while this sits uncommitted.
sys-stk-front-proxy's swarm branch resolves the upstream as
<entity>_<service_key>:<port>, and Docker Swarm assigns service DNS
aliases from the actual compose.yml.j2 service key - a Docker-level
naming rule independent of any Ansible meta/services.yml key. The
dashboard entity's meta/services.yml key is deliberately 'console'
(compose.yml.j2's service_name comment: avoids colliding with the
platform's own unrelated web-app-dashboard tile-integration bond), but
its compose service key stayed 'dashboard' by the same deliberate
design. The swarm-mode front-proxy include was passing 'console' as
webserver_proxy_service_key, producing a syntactically valid but
nonexistent DNS name (wazuh_console) - permanent NXDOMAIN, 502 with
zero variation, confirmed against real CI retry logs (three independent
~650s retry windows, never once succeeding).
The earlier swarm-proxy fix (d2142e8f9) verified resolve_upstream()
produced a well-formed string, but never cross-checked it against what
compose.yml.j2 actually registers with Docker - the one field where
Wazuh's entity-key/compose-key split (unique to this role; bluesky and
other precedents have no such split) actually matters.
TEST_E2E_PLAYWRIGHT_WAIT_RETRIES/_DELAY being redefined in web-app-wazuh's
vars/main.yml violated test_uppercase_constants_unique.py - ALL-CAPS
top-level constants must be defined exactly once, project-wide,
regardless of Ansible's include_vars runtime precedence (which does make
the override work correctly at runtime, just not in a way this lint rule
permits).
Follows the existing test_e2e_playwright_run_timeout /
TEST_E2E_PLAYWRIGHT_RUN_TIMEOUT_EFFECTIVE precedent already in this role:
moved wait_retries/wait_delay to lowercase, overridable defaults in
test-e2e-playwright/defaults/main.yml, updated 02_run_one.yml to read the
lowercase names directly. Wazuh's vars/main.yml now sets the lowercase
override (120/5) instead of shadowing the ALL-CAPS constant.
test-lint's test_comments_valid.py requires a single-line doc comment
for vars/*.yml entries. The explanatory comment above
TEST_E2E_PLAYWRIGHT_WAIT_RETRIES/_DELAY was 10 lines - condensed to one,
keeping the essential facts.
Five Exception:-labeled comments in vars/main.yml restated trip-wires
already documented at their true origin elsewhere (meta/services.yml's
console-vs-dashboard naming, tasks/00_core.yml's DinD sysctl guard,
plugins/filter/bcrypt_hash.py's passlib writeup, docs/contributing/
design/iam/rbac.md's rbac_group_path rules). Per review feedback,
Exception: is reserved for the one true origin, not every reference to
it - condensed each to a single-line plain doc comment pointing back.
Separately: CI's shared test-e2e-playwright readiness probe defaults to
a ~150s budget for every app. Wazuh's own indexer healthcheck already
declares a 390s worst-case boot ceiling, and swarm mode has no
depends_on/service_healthy gating, so the dashboard races a
still-booting indexer with no shield. Raised
TEST_E2E_PLAYWRIGHT_WAIT_RETRIES/_DELAY to 120/5s (600s), the first
per-role use of this override.
meta/volumes.yml's config-mount source: for certs-entrypoint.sh and
indexer-entrypoint.sh resolved via abs_role_path_by_application_id - a
controller-local checkout path. docker stack deploy runs delegate_to:
STACK_HOST (roles/sys-svc-compose/handlers/swarm.yml), so under swarm
mode that path doesn't exist on the remote host, and docker stack
deploy failed with 'no such file or directory' once the earlier
healthcheck himBHsescaping fix let it get that far.
Mirrors the existing 'Copy static Wazuh indexer roles.yml' pattern
already in this file: a new copy task ships both static, non-templated
scripts to WAZUH_INSTANCE_DIR on the actual deploy target, gated on
IS_STACK_HOST, before docker stack deploy/config ever runs. volumes.yml
now points source: at that instance-dir path instead.
Verified on a live swarm cluster (make swarm-zombie app=web-app-wazuh):
the copy task ran and succeeded, and the full stack came up healthy
(manager, indexer, dashboard all Up; certs_generator exited 0 as
expected for a one-shot init container; front proxy served a real
301 redirect through OpenResty).
The healthcheck test regex ended with an unescaped $ ('^(200|401)$'),
which docker compose config tolerates silently but docker stack config
(swarm mode's stricter validator) rejects outright: 'invalid
interpolation format ... you may need to escape any $ with another $'.
Escaped to $$. Verified locally: docker stack config now resolves the
correct literal ^(200|401)$ regex. Confirmed via the actual PR CI run
(#414, 33759579296) that this explains all 5 swarm-mode failures
(debian, ubuntu x2, fedora), regardless of TLS mode - manager/dashboard
have no healthchecks, and no other $-ending pattern exists elsewhere
in the role's templates.
sys-stk-front-proxy's swarm branch (resolve_upstream(), utils/networks/proxy.py)
derives the upstream port from services.<entity>.ports.internal.<kind>, where
<entity> is the bare holder derived from application_id ('wazuh'). Wazuh's
'wazuh' entity carries no ports (manager/indexer/console split the role), so
swarm mode hard-failed with 'needs services.wazuh.ports.internal.http (or
internal_port=)' - a gap invisible in compose mode, which only reads http_port.
web-app-bluesky hits the identical structural gap and works around it by
passing webserver_proxy_service_key/proxy_internal_port explicitly on every
front-proxy include, pointed at the real sub-entity. Applied the same pattern
here, pointed at 'console' (the entity that owns ports.internal.http: 5601).
Verified directly against utils.networks.proxy.resolve_upstream() with
Wazuh's real port config: the old call site reproduces the exact reported
CI error, the new call site resolves to wazuh_console:5601, and compose mode
is unaffected.
Add single-node Wazuh stack (manager, indexer, dashboard) as a new
Ansible role, integrated with the platform's Keycloak realm via native
OIDC and secondary LDAP authorization, with three-tier RBAC.
- Pins wazuh/wazuh-{manager,indexer,dashboard}:4.14.6 and
wazuh-certs-generator:0.0.4 (one-shot cert bootstrap job)
- No external database: wazuh.indexer (OpenSearch) is self-contained
- Auth: native OIDC against Keycloak, working end-to-end after two
infra fixes (JVM cacerts trust for the indexer's outbound calls to
Keycloak, front-proxy vhost wiring)
- LDAP wired as a secondary, authentication-independent authorization
backend (group-based backend_roles enrichment only)
- RBAC: three roles (administrator, security-analyst, readonly-auditor)
via the platform's standard meta/rbac.yml + rbac_group_path pipeline
- Documents a genuine upstream Wazuh bug: wz-home as the default
landing route throws an uncaught client exception for any
non-all_access role; default route moved to OpenSearch Dashboards'
own /app/home
Addressed two full rounds of review feedback (Kevin, Alejandro):
WEB_PROTOCOL usage, service-account naming, comment cleanup, shared
Keycloak-admin-helper extraction, a real TOCTOU race in RBAC test
setup, a Mermaid auth-flow diagram, meta/services.yml placement for
the certs-generator, CA-trust-path SPOT usage, CSP tightening
(including a genuine worker-src: blob: requirement found via live
testing), and a loud (non-silent) dev-bucket fallback warning.
Rebased onto main after a 3-month gap; brought the role in line with
~30 platform-convention changes introduced in that window (swarm
compose-template rules, Tor/onion contract, Playwright timeout/goto
wrappers, file layout, task numbering, comment policy, and more).
Live-deploy verification (9 iterations) found and fixed 6 further
real bugs invisible to lint/unit tests alone:
- credentials.* / meta/schema.yml -> secrets.credentials.* /
meta/secrets.yml (another platform-drift casualty)
- notify: "compose up" -> "compose-up" (day-one handler-name typo)
- local.subnet collision with web-app-mailu's /24
- missing primary: false alongside tor.exclusive: false, which let
the onion domain shadow the clearnet vhost entirely
- a second front-proxy include for the onion domain, later reverted
(see below)
- shared sys-stk-front-proxy/sys-front-tls TLS-lookup keying bug
(app-id-keyed vs. domain-keyed), which silently applied TLS to
Wazuh's onion vhost
The last two are related: enabling genuine dual-stack (clearnet +
onion) reachability for tor.exclusive: false surfaced a real bug in
two shared platform roles. That fix was verified safe against all 103
apps currently in the repo, but is deliberately NOT included here -
shared-role changes deserve their own PR and review, not to ride in
on a single app's feature branch. Wazuh's second front-proxy include
was reverted accordingly; the resulting onion-reachability gap is
documented in the README as a known, platform-scoped limitation.
make test: 5508 passed, 3 failed (all three trace to untracked local
scratch files - Dockerfile.bak, scripts/lab-*.sh - not present in any
CI checkout).
A root-owned, empty .ansible/ (collections/modules/roles) directory
skeleton can appear in the repo root, from some root-run containerized
process's Ansible invocation resolving its collections_paths default
against the wrong $HOME. Confirmed this is not the project's own
convention: every reference in the codebase (Makefile, scripts/install/
all.sh, .github/workflows/lint-ansible.yml) consistently targets
~/.ansible/collections outside the repo, and .ansible/ has never been
tracked in this repository's history. Regardless of the exact trigger,
this directory is untracked and root-owned, and being previously
unlisted caused 'git stash push -u' to abort mid-cleanup on it
(EACCES), silently leaving other unrelated working-tree changes
unreset.
- tasks/02_bootstrap.yml: set retries=4 delay=30 on owner probe to
stay within n8n's 5-attempt/5-minute rate limit window
- files/playwright: remove test-oidc-login.js - redundant with
test-login-administrator.js which already covers the full OIDC flow
- rebase onto latest main
- tasks/01_core.yml: add when: N8N_SSO_ENABLED | bool to hooks.js copy task
- templates/compose.yml.j2: wrap hooks.js volume mount in SSO conditional
Per Kevin's review - hooks.js should only be staged and mounted
when SSO is actually enabled, adding extra redundancy.
Remove relations: ['role'] from UserRepo.findOne() - in 1.95.3
user.role is a plain string column, not a relation. The relations
array caused EntityPropertyNotFoundError which produced 500 on
every SSO login attempt. Also removed dead RoleRepo/dbCollections.Role
references dropped in 1.95.3's DropRoleTable migration.
Verified: biber@infinito.example now auto-provisioned as global:member
after Keycloak SSO. All 5 Playwright tests pass locally.
- Add bond:1 to all 9 enabled+shared services in web-app-n8n/meta/services.yml
- Fix web-app-n8n/meta/variants.yml V3 postgres shared:false → shared:true (only V2 is standalone)
- Remove dead PIHOLE_SSO_PROXY_CA_CONTAINER_PATH var from web-app-pihole/vars/main.yml
- Add nocheck:url marker for dead link in web-app-mastodon/README.md
- Fix LDAP sign-in button selector in _shared.js (button[type=submit] → getByRole)
n8n is SSO-gated and redirects unauthenticated requests cross-origin to
Keycloak, causing ERR_ABORTED (405) in the headless Chromium probe. Adding
405 opts out of the probe, matching the web-app-prometheus pattern.
1. meta/info.yml: add homepage and video fields.
2. meta/server.yml: add status_codes.default [200, 302] so the CSP/webserver
health check accepts the oauth2-proxy redirect response.
3. Move self-signed CA trust for oauth2-proxy into the shared
oauth2-proxy-keycloak.cfg.j2 template (provider_ca_files, conditional on
TLS_MODE=self_signed). This fixes the root cause for all oauth2-proxy roles
(including web-app-pihole) without role-local lineinfile hacks.
4. Drop compose_handlers_flush: false + explicit meta: flush_handlers from
both web-app-n8n and web-app-pihole — now that no post-render file surgery
is needed, compose_handlers_flush: true lets sys-stk-backend flush normally.
5. Remove the now-unused N8N_SSO_PROXY_CA_CONTAINER_PATH and
PIHOLE_SSO_PROXY_CA_CONTAINER_PATH role variables.
6. README: document the V2 (no-auth) native administrator login flow.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Community edition with edge oauth2-proxy SSO (Decision #1b)
- PostgreSQL backend via svc-db-postgres
- Three variants: V1 sso+ldap, V2 no-auth, V3 ldap-only
- Bootstrap owner account via /api/v1/owner/setup
- LDAP config via direct PostgreSQL upsert (CE API limitation)
- Playwright: baseline, CSP, OIDC, administrator and biber personas
- All 5 active tests pass, 2 skipped correctly for this variant
Pi-hole v6 returns 308 (HTTP→HTTPS redirect) in addition to the
platform's DEFAULT_OK codes (200, 301, 302). Without 308 in the
accepted status_codes, the webserver health checker would flag
pihole.infinito.example as unhealthy.
- native.spec.js: update button selector from /sign in|login/i to
/log in|sign in/i — Pi-hole v6's submit button text is
'Log in (uses cookie)', not 'Sign in' or 'Login'
- 01_setup.yml: remove pihole setpassword CLI task — Pi-hole v6 blocks
this command when FTLCONF_webserver_api_password is set via env var
('set by environment variable; please unset it to use this function').
Password is now set at container startup via the env var, which is
the v6-idiomatic approach.
Three fixes:
1. guest.spec.js: skip instead of fail when OIDC_ISSUER_URL is unset
(variant-1 has SSO disabled so no Keycloak redirect is expected;
the old expect().toBeTruthy() caused a hard failure instead of skip)
2. playwright.env.j2: only emit OIDC_ISSUER_URL when SSO is proxy-gated
(was unconditionally set, causing native.spec.js to wrongly skip in
variant-1 and oauth2.spec.js/guest.spec.js to fail on missing redirect)
3. compose.yml.j2: Pi-hole v6 compatibility fixes
- Replace WEBPASSWORD with FTLCONF_webserver_api_password (v6 dropped
legacy env vars; WEBPASSWORD is silently ignored in v6)
- Add FTLCONF_webserver_domain={{ domain }} to whitelist the nginx
proxy Host header; Pi-hole v6 FTL returns 403 for unrecognised Host
headers (DNS rebinding protection) — this was causing the baseline
playwright.spec.js to fail with HTTP 403 in variant-1
Playwright's default testMatch only discovers files matching
*.spec.js / *.test.js. test-guest.js, test-oauth2.js, and
test-native.js never matched this pattern and were silently never
executed in any CI variant -- only playwright.spec.js's 2 baseline
tests ever ran (confirmed: 0 occurrences of these files in any
playwright-junit.xml across variant-0/variant-1, sync/async).
Renamed to guest.spec.js, oauth2.spec.js, native.spec.js so the SSO
persona scenarios (Keycloak redirect, admin/biber login, native login)
actually execute and validate the sso-proxy CA trust fix.
oauth2-proxy's singular 'provider-ca-file' flag has a known bug
(oauth2-proxy/oauth2-proxy#2802) and may not be a valid config key on
:latest, causing oauth2-proxy to fail config parsing and crash on
startup. This made nginx's auth_request to sso-proxy return 500 for
every pihole request (HTTP Error 500 on https://pihole.infinito.example/),
blocking even the pre-Playwright readiness check.
provider_ca_files (plural, array of strings) is the documented,
working config key for loading a custom CA for the OIDC provider
connection.
This block was added in an earlier (reverted-in-intent but never
actually removed) attempt and caused a duplicate 'sso-proxy' YAML key
(also defined via sys-svc-compose/base.yml.j2's sso_proxy include),
breaking compose_ca.py's compose config parsing entirely.
The CA trust fix now lives solely in 01_core.yml's lineinfile task,
which patches the rendered oauth2-proxy config file directly -- no
compose.yml changes needed.
- playwright.spec.js: remove duplicated SSO/persona scenarios now
covered by test-guest.js/test-oauth2.js/test-native.js. Reduce to
baseline checks (front page + TLS/HSTS, HTML content-type), matching
the pgadmin pattern.
- meta/main.yml: rewrite description to describe Pi-hole the software
(network-wide DNS sinkhole, DNS/DHCP management, web dashboard)
rather than the role/integration details.
penpot was merged using ports 8041/16506 after our last rebase.
Reassign pihole to 8078 (http) and 16507 (sso) via
infinito contributing network ports suggest.
oauth2-proxy (quay.io/oauth2-proxy/oauth2-proxy) is a distroless image
with no /bin/sh, so the platform's CA-trust wrapper (with-ca-trust.sh,
applied by compose_ca.py for services with a shell) is skipped for this
container. Go binaries ignore SSL_CERT_FILE/REQUESTS_CA_BUNDLE and use
only the OS system cert pool, so on self-signed TLS setups oauth2-proxy
fails OIDC discovery against Keycloak with
'x509: certificate signed by unknown authority'. nginx then falls back
to pihole's native /admin/login instead of redirecting to Keycloak,
breaking SSO entirely in CI and on any self-signed deployment.
Fix: append provider_ca_file to the rendered sso-proxy config after
web-app-keycloak/tasks/sso_proxy.yml renders it and before compose up
runs. oauth2-proxy loads this CA explicitly for the OIDC provider
connection, bypassing the system cert pool entirely.
Scoped to this role: avoids touching the shared
oauth2-proxy-keycloak.cfg.j2 template (would affect every
flavor: oauth2 role platform-wide). No-op on production
(TLS_MODE=letsencrypt).
oauth2-proxy is a distroless Go binary — the platform CA wrapper
(with-ca-trust.sh) is skipped because there is no /bin/sh, and Go
ignores SSL_CERT_FILE. Pass the CA cert directly via --provider-ca-file
so oauth2-proxy trusts the internal Keycloak OIDC endpoint.
Follows the penpot pattern of inline command override in compose.yml.j2
conditioned on TLS_MODE == self_signed. No-op on production (letsencrypt).
oauth2-proxy is distroless (no /bin/sh), so the platform CA wrapper
entrypoint is skipped. Go binaries ignore SSL_CERT_FILE and use the
system cert pool only, causing x509 errors on self-signed TLS setups.
Override sso-proxy command in compose.override.yml.j2 to append
--provider-ca-file pointing to the platform CA cert
(/tmp/infinito/ca/root-ca.crt, always bind-mounted by compose_ca.py)
when TLS_MODE is self_signed. No-op on production (letsencrypt).
inventory.py auto-generation only triggers when oauth2.enabled is a literal
boolean True. Pihole uses a Jinja2 expression for enabled, so auto-generation
does not trigger. The credential must be defined in schema.yml explicitly.
Other OAuth2-protected apps (decidim, espocrm, etc.) do not define this
credential in their schema.yml and work correctly. The platform handles
the cookie_secret for all apps through the web-app-oauth2-proxy role.
Defining it per-role was causing oauth2-proxy to crash on startup in CI
because the credential was not properly generated in the CI environment.
oauth2-proxy cannot verify Keycloak's self-signed certificate during OIDC
discovery because the platform CA cert is injected via env vars but Go's
TLS stack doesn't pick it up reliably in the DinD environment.
Adding --ssl-insecure-skip-verify allows oauth2-proxy to complete OIDC
discovery and serve requests instead of crash-looping with certificate errors.
This is acceptable for a self-signed lab environment.
oauth2-proxy cannot resolve auth.infinito.example from inside Docker in CI
environments where CoreDNS does not have this entry. Adding extra_hosts via
compose.override.yml.j2 (auto-included by compose_file_args lookup) maps
the Keycloak domain to host-gateway so OIDC discovery succeeds.
Pi-hole deploys immediately after Keycloak but Keycloak needs time to
initialize OIDC. Adding explicit wait for Keycloak OIDC discovery endpoint
ensures oauth2-proxy can complete OIDC discovery and serve 302 redirects
instead of 500 errors when the platform Playwright wait check runs.
- meta/services.yml: add noqa comment for email integration
- tasks/01_setup.yml: use uri_retry instead of legacy uri module
- tasks/01_setup.yml: add HTTPS endpoint check to prevent 500 in CI wait check
- tasks/01_setup.yml: increase oauth2-proxy wait timeout to 300s with ignore_errors
oauth2-proxy requires Keycloak to complete OIDC discovery before binding its port.
If Keycloak is still warming up, oauth2-proxy may not be ready within the timeout.
Playwright tests validate the actual functionality end-to-end.
oauth2-proxy needs time to complete OIDC discovery against Keycloak in CI.
120s was insufficient - increased to 300s to allow for slower CI environments.
- tasks/01_core.yml: set docker_compose_flush_handlers: true, remove manual flush_handlers task
- tasks/01_setup.yml: add wait for oauth2-proxy port to fix HTTP 500 in CI wait check
- meta/services.yml: use NETWORK_PUBLIC_DNS_RESOLVERS SPOT for upstream_dns
- vars/main.yml: remove default from upstream_dns lookup, join list with semicolon
Prageeth Panicker
