HEALTH_CSP_SKIP_DOMAINS skipped every CURRENT_PLAY_REDIRECT_DOMAINS
source: the DOMAIN_PRIMARY apex, the node onion apex, inventory
redirect mappings and per-app aliases. The skip dates from when the
checker judged the 301 itself; since csp-checker 3.0.0 it follows the
redirect chain and judges the document it lands on. So a broken or
unserved alias target, and a redirect leaving its network, were never
loaded.
A redirect source now stays skipped only when its target is not among
CURRENT_PLAY_DOMAINS_ALL, i.e. lies off this node, like SOFTWARE_DOMAIN
for the apex without web-app-dashboard. Every other source is crawled.
Closes#929
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The probe dropped every canonical and alias domain of a role whose server.status_codes.default named anything at or above 400, so a single by-design 4xx removed the whole vhost from CSP inspection. The checker now takes --accept-status <domain>=<code>, which keeps the page navigated and its policy evaluated, so the declaration buys coverage instead of costing it.
csp_accept_status reads the existing SPOT in utils/roles/applications/status_codes.py rather than restating the rule. That resolves per vhost, because server.status_codes is keyed like domains.canonical, so two domains of one role can differ; the old filter looked only at default and was blind to seaweedfs declaring its api, filer and master keys separately. The threshold is the checker's own: anything below 300 passes unflagged, so a declared redirect is named too.
csp_skip_domains keeps only the half that has no vhost to probe at all, a canonical domain owned by a disabled service, and gains the tests that half never had.
Measured: 14 domains move from skipped to probed, among them jenkins, mattermost, n8n, prometheus, mirror and xmpp. 28 domains carry an entry in total. Across the CI diagnostics of this run 473 of 476 observed responses were 200, the single infinito vhost answering 403 is api.seaweedfs, and it declares that code, so it stays green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
csp-checker navigates every URL with a 20000 ms budget until
domcontentloaded. Over the Tor SOCKS proxy that expires before a page of
module scripts finishes: run 34614940508 lost Mastodon#1 and OpenProject#1
to it, 26 of 27 attempts logging "Unable to reach http://<onion>/
(Navigation timeout of 20000 ms exceeded)" while sys-service-terminator
retried the unit for 35 minutes. No policy was ever read; the checker never
got that far.
csp-checker 2.3.0 takes --timeout, so script.py takes --onion-timeout and
hands it to the onion batch alone, before the -- separator that would make
it a URL. Clearnet keeps the image default. The budget is
HEALTH_CSP_TIMEOUT_MS (20000, the image default) times the tor flavor's
multiplier, 100000 ms on the public network.
The unit's TimeoutStartSec had to follow. It was 30 + 25 s per domain --
130 s for the two onion vhosts Mastodon checks, while a 100 s budget needs
up to 200 s, so systemd would have killed the check mid-run. The filter now
takes onion_per_domain_seconds and counts .onion domains at that rate;
without the argument it computes exactly what it did before, so
sys-ctl-hlth-webserver is unchanged. The Mastodon case goes from 130 s to
450 s, a clearnet-only play stays at 130 s.
Tests pin that --timeout is appended only when set and stays ahead of the
separator, that main budgets the onion batch alone, that the role passes
both --onion-timeout and the onion rate for the start budget (both guards
fail against the previous tasks file), and the filter's split rate.
Not covered: the filter still caps at 3600 s, which binds beyond 34 onion
domains in one round.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drop the "web-opt-rdr-domains in deployed" gate in the
current_play_redirect_domains lookup. CURRENT_PLAY_REDIRECT_DOMAINS now
always carries DOMAIN_PRIMARY to DOMAIN_HOMEPAGE when both are set and
differ. This triggers sys-stk-front-base to auto-load web-opt-rdr-domains
so openresty actually renders a vhost matching its declared canonical, and
the bare apex stops hitting the default-server's ssl_reject_handshake.
CSP probing skips those redirect-source domains: pure 301 vhosts have no
CSP header to validate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`status` was too vague for what the lookup returns (apps in current
play). Rename to `deployment` and add two new fields:
* `runtime` — RUNTIME variable value
* `deployed` — runtime-aware authoritative set: equals `running`
(whitelist or groups) for ephemeral CI runtimes (act, github),
falls back to `groups` for persistent runtimes (host, dev). This
matches reality: in CI the container is fresh so only whitelisted
apps exist; in production a partial deploy still leaves prior
apps live, so groups is the right set.
Switch the two webserver/csp healthchecks to `lookup('deployment').deployed`
so they only probe domains for apps that actually exist in the current
deploy. In production behaviour is unchanged.
Update remaining consumers (cli/build/include, web-app-keycloak,
test-e2e-playwright) for the rename. Existing `status` semantics
(`whitelist`, `running`, `groups`, `all`) preserved under the new
name.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mirrors the contract that 0ef2a3481 established for sys-ctl-hlth-webserver
(reads status_codes.default from meta/server.yml) so the CSP probe stops
flagging federation-only roles like web-app-bridgy-fed as unhealthy.
Previously, sys-ctl-hlth-csp invoked `node health-csp.js` against every
nginx vhost's `/` and treated any non-2xx/3xx as "Unable to reach". For
web-app-bridgy-fed, whose canonical surfaces are the federation-protocol
endpoints (/robots.txt, /.well-known/host-meta, /web/<actor>) and whose
Flask app deliberately serves 404 on `/`, the probe failed, systemd
retried 10x over ~10 min, and sys-service-terminator aborted the deploy
with rc=2 (CI job 76714878060).
The Playwright spec already encoded the contract: 4xx-at-/ is by design,
5xx is the only unhealthy signal. The fix in 0ef2a3481 surfaced that
contract via status_codes.default for hlth-webserver but did not extend
it to hlth-csp.
This change adds:
- A new csp_skip_domains filter (filter_plugins/csp_skip_domains.py)
that walks every selected application, reads
server.status_codes.default, and returns the canonical+alias domains
of any role declaring a code >= 400. Follows the
web_health_expectations pattern.
- HEALTH_CSP_SKIP_DOMAINS in vars/main.yml, computed once via the
filter.
- --skip-domain repeatable CLI flag in files/script.py that filters
the discovered nginx-config domain list before URLs are built.
Side benefit: web-app-listmonk (default: 404) and web-svc-html
(default: [403, 200]) are now also correctly skipped if deployed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Swap role includes: sys-systemctl → sys-service in all roles
- Rename variables everywhere: systemctl_* → system_service_* (incl. systemctl_id → system_service_id)
- Templates: ExecStart now uses {{ system_service_script_exec }}; add optional RuntimeMaxSec via SYS_SERVICE_DEFAULT_RUNTIME
- Move SYS_SERVICE defaults into roles/sys-service/defaults (remove SYS_SERVICE_ALL_ENABLED & SYS_SERVICE_DEFAULT_STATE from group_vars/07_services.yml)
- Tidy group_vars/all/08_timer.yml formatting
- Introduce roles/sys-daemon:
- default manager timeouts (timeouts.conf)
- optional purge of /etc/systemd/system.conf.d
- validation via systemd-analyze verify
- handlers for daemon-reload & daemon-reexec
- Refactor sys-timer to system_service_* variables (docs and templates updated)
- Move filter_plugins/filetype.py under sys-service
- Update meta/README to point to official systemd docs
- Touch many roles (backup/cleanup/health/repair/certs/nginx/csp/wireguard/ssd-hdd/keyboard/update-docker/alarm compose/email/telegram/etc.) to new naming
BREAKING CHANGE:
- Role path/name change: use `sys-service` instead of `sys-systemctl`
- All `systemctl_*` vars are now `system_service_*` (e.g., on_calendar, state, timer_enabled, script_exec, id)
- If you have custom templates, adopt RuntimeMaxSec and new variable names
Chat context: https://chatgpt.com/share/68a47568-312c-800f-af3f-e98575446327
- Unified service templates into generic systemctl templates
- Introduced reusable filter plugins for script path handling
- Updated path variables and service/timer definitions
- Migrated roles (backup, cleanup, repair, etc.) to use systemctl role
- Added sys-daemon role for core systemd cleanup
- Simplified timer handling via sys-timer role
Note: This is a large refactor and some errors may still exist. Further testing and adjustments will be needed.
This commit restructures systemctl service definitions and category mappings.
Motivation: Alarm-related bugs revealed inconsistencies in service and role handling.
Preparation step: lays the groundwork for fixing the alarm issues by aligning categories, roles, and service templates.
kevinveenbirkenbachandClaude Opus 5.5