#1180 gave web-app-matrix a canonical `jitsi` domain for its own Jitsi
and listed it under `services.jitsi.domains`. When Jitsi is shared or
disabled nothing serves that domain, and the blackbox healthz job of
web-app-prometheus probed every canonical domain, so the probe of
https://jitsi.matrix.<domain>/healthz/ready failed with "tls:
unrecognized name" and `prometheus scrape: every consumer role reports
up=1 and passes its probe` went red (main run 40017166046: Matrix#2,
Nextcloud#12).
web_health_expectations already skipped such keys. The rule now lives
in utils/domains/unserved.py: a key claimed by a service entity that is
disabled, or enabled but shared, is not served by that app. The health
filter uses it unchanged, and prometheus.yml.j2 filters its targets
through the new `served_domains` filter, which also drops the
`<key>_onion` mirror of a skipped key.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The readiness module http_ready rendered insecure_skip_verify from
TLS_MODE and lookup('ca_injected', application_id) (#851). The lookup is
true only when Prometheus itself has a domain, TLS and the self-signed
mode, so in a self-signed deployment where Prometheus runs in another
mode the exporter had no CA file and skipped verification. All targets
share the module, so a target with a publicly trusted certificate was
probed without verification too, and anyone on the probe's path could
answer for a failed application.
http_ready now always verifies. A second module, http_ready_private_ca,
skips verification, and only the targets that need it select it: those
in the self-signed mode while the CA is not mounted into Prometheus,
through the __param_module label, which overrides the job's module for
that target. The default self-signed deployment mounts the CA and keeps
every target on the verifying module.
Verified on a local self-signed Tor round, where the CA is mounted:
the rendered blackbox.yml holds http_ready with insecure_skip_verify:
false and http_ready_private_ca with true, and no target of the scrape
configuration carries the module label. All targets of that round are
http onion names, so the https path itself ran only in the unit tests:
tests/unit/python/roles/web-app-prometheus pass, among them a publicly
trusted target that keeps the verifying module while the CA is missing,
a self-signed target that switches to the private-CA module, and the
verifying module rendering insecure_skip_verify: false without the CA.
Against the templates of main the three new cases fail.
Closes#851
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every blackbox probe of a Tor deployment answered 404, so AppDown fired for
all deployed apps although they were up. The job built its targets from
canonical_domains_map(DOMAIN_PRIMARY), which knows no onion domain. In a Tor
deployment the vhosts answer under <sub>.<onion> only, and the front proxy
served the clearnet names from its default server.
The targets now come from lookup('domains'), the map the vhosts are built
from, and each target takes its scheme from the domain it probes instead of
from the Prometheus vhost.
Which roles are probed follows lookup('roles_with_service', 'prometheus'),
the list the role's own spec expects metrics for. It leaves out the database
roles, which have a domain in the map and no vhost, and the roles that
declare scrape: false because their vhost carries no Prometheus block.
Two roles declared more than their vhost delivers. web-svc-mirror renders a
vhost of its own with no backend behind it, and on a live stack its probe
answered 404 under the onion name as well, so it declares scrape: false now.
web-app-peertube has a backend, but its vhost template never included the
Prometheus locations. It includes them now.
The spec could not see any of this in a compose deployment. dotenv_quote
wraps PROMETHEUS_TARGET_ROLES_JSON in quotes there, JSON.parse returned a
string, the role list came out empty and the per-role test was skipped. The
value is decoded now, and the test also requires probe_success == 1 for
every role in the list.
Closes#792
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A native-metrics app on a node-local force_bridge network cannot be joined or resolved by a swarm-wide prometheus: the stack deploy fails with the network scope error and the scrape job would flap as permanently down. native_metrics_apps now drops those apps at the source (DEPLOYMENT_MODE swarm + networks.local.force_bridge), covering all four call sites (network precreate, compose service networks, external-networks block, scrape include) without per-template guards. DEPLOYMENT_MODE is templated without a default so a missing value fails loudly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In swarm the sibling exporters (alertmanager, cadvisor, node-exporter, blackbox) and the healthz backend are unreachable on the compose container_name / host.docker.internal / 127.0.0.1 addresses, so prometheus marked every target down and /healthz/ready returned 502. Add a scrape_target lookup emitting <host>:<port> per mode (swarm tasks.<entity>_<service>:<internal_port>, compose container_name/compose_host), reading the internal port from services.<svc>.ports.internal; declare those ports on the four exporters; route both healthz probes through the proxy_pass lookup via one set variable. Compose output stays byte-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- web-app-opentalk: move CONTROLLER_HOST, INSECURE, BASE_URL, OIDC_*,
SHARED_FOLDER_OPENCLOUD_URL from the application service's compose
environment block into templates/env.j2 (already mounted as env_file
via sys-svc-container/base.yml.j2 on application + controller). OIDC
keys are guarded by OPENTALK_OIDC_ENABLED; SHARED_FOLDER_* by
OPENTALK_OPENCLOUD_ENABLED. INSECURE derives from
lookup('tls', application_id, 'enabled').
- web-app-prometheus, web-app-decidim: replace bare TLS_ENABLED Jinja
references with lookup('tls', application_id, 'enabled') so they
honor per-app TLS posture like the rest of the codebase.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hard-coded self-path references rot the moment the file moves: a header
comment ``# utils/foo/bar.py`` at the top of a module, a docstring
mentioning ``utils/foo/bar.py``, or a docs entry linking
``utils/foo/bar.py`` from inside ``utils/foo/bar.py`` itself all silently
desync from reality after the next ``git mv``.
The new lint walks every project file, computes its POSIX-style relative
path from PROJECT_ROOT, and reports any line whose text contains that
path as a substring. Files at the repo root (no ``/`` in their relative
path) and setuptools-generated ``*.egg-info/`` artefacts are exempt.
Per-line opt-out via ``# nocheck: self-path-reference`` on the offending
line, registered in the suppression catalog.
Existing offenders cleaned up:
- 72 redundant ``# <self-path>`` header lines removed across roles,
cli, plugins, scripts, tests, and utils.
- 4 prometheus templates that told consumers to "update
<self-template-path>" rewritten to drop the redundant self-reference.
- 8 docstring/comment self-references in cli, plugins, tests, utils,
and the dashboard oidc.js.j2 header rewritten to describe the layout
without naming the file's own path.
- 17 legitimate usage examples (shell scripts, test data lists, one
template self-include doc-block) marked with ``nocheck:
self-path-reference``.
- ``container.sh`` heredoc usage strings switched to ``${0}`` so the
marker stays out of the user-visible help output.
- ``healthcheck/http.yml.j2`` doc-block rewritten to prose because
``{# ... #}`` Jinja comments cannot host an inner ``{# nocheck #}``
(the inner ``#}`` closes the outer comment).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5.5
Alejandro Roman