csp-checker 3.0.5 starts Chromium without HTTPS-Upgrades. Before it, every
http:// onion vhost was first tried as https://host:443 through the Tor SOCKS
proxy, which broke the port-80 requests to the same onion service with
net::ERR_SOCKS_CONNECTION_FAILED. hlth-csp then failed all 25 start attempts
and took the deploy down in 11 swarm/tor jobs of main run 38038541631
(Gitea#0, Hugo#0, Listmonk#0, Mattermost#2, Nextcloud#0/#4/#7, Penpot#0,
Pixelfed#0, YOURLS#0, odoo#0).
Verified against a local compose stack over Tor: image 3.0.4 as shipped hit
the SOCKS error on css and asset and reported extra redirects elsewhere; with
the 3.0.5 change no SOCKS error and only the redirects the server sends. The
csp-checker lint and all 16 E2E tests pass on the image built from 3.0.5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
3.0.3 has a tag and no image. Its release build failed the same 204 fixture
3.0.2 failed on, because the handling read the status the moment page.goto
rejected while the listener sat on a second CDP session, and the protocol
orders messages within a session rather than between two of them. The status
now gets awaited, so the verdict no longer depends on which session delivers
first, and the page close that the wait pushes into the navigation's commit
window is bounded rather than awaited forever.
A third number rather than a rebuilt 3.0.3, for the reason 3.0.3 itself got
one: the mirror copies by tag, so an overwritten tag turns every copy that has
already seen it into a silent lie, and the deploys pull through that mirror.
The reachability check answers 404 for 3.0.4 until the release publishes, the
way it did for the two before it. That clears when the release runs.
Worth stating plainly: the runner's failure was never reproduced locally. The
ordering hole is closed by construction and two full suites are green, but
whether it was the cause is decided by the release build, not here. If that
build fails on the 204 fixture again, this pin is worth as little as the last
one and the job log is what to read next.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3.0.2 never produced an image. Its release build failed the E2E suite on the
204 fixture itself: the handling listened for Network.responseReceived, and a
204 is never committed, so Chromium emits no such event and the checker kept
reporting the host as unreachable. The pin therefore addressed a tag that the
registry answers with 404, which the image-reachability check has been saying
since the bump landed.
csp-checker d827de1 reads the status from Network.responseReceivedExtraInfo,
the event that carries an uncommitted navigation's status, correlated with the
Document request id. Reproduced and fixed in one E2E run, with the download
fixture still failing so the change distinguishes a 204 rather than blessing
every aborted navigation.
A new number rather than a rebuilt 3.0.2, because the mirror copies by tag: an
overwritten tag turns every copy that already saw 3.0.2 into a silent lie, and
the deploys pull through that mirror.
Until the release publishes, the image does not exist and the reachability
check answers 404 for 3.0.3 the way it did for 3.0.2. That clears when the
release runs, not by a change here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
n8n ships isbot.spawn(['bot']) unconditionally and answers any user agent
matching /bot/i with a bodyless 204. This checker calls itself
CSP-CheckerBot, so the probe never reached the application. Chromium
cancels a navigation on 204, page.goto threw net::ERR_ABORTED, and the
checker reported the host unreachable. In CI job 105068914482 the unit
failed 25 times over 1551 seconds and took the job with it.
3.0.2 ends the visit as reachable when the document response is exactly
204, and prints that it checked nothing, because page.url is about:blank
afterwards. Exactly 204, not the 2xx range: a 200 carrying
Content-Disposition aborts the same way and is a download, not an empty
document. The checker's e2e suite pins both directions, with the download
fixture as the control that fails if the condition is ever widened.
Same shape as df4b7757bc took for 304: a status the checker's own contract
calls healthy, rejected at runtime over how the browser reacts to it.
The user agent keeps its name. The checker is a bot and a bot filter
blocking it is correct; what changes is that the consequence is now stated
rather than hidden. A vhost answering 204 reports that its CSP went
unchecked instead of reporting nothing at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
csp-checker 3.0.0 narrowed the healthy set to 2xx by failing every status at or above 300, which swept in 304. The health service then died on the first revalidated vhost: in CI job 104991065851 it reported element.<onion> unreachable with Status 304, and sys-service-terminator retried the unit 25 times over 1792 seconds before the job failed.
3.0.1 takes 304 out of the failure branch. The checker's own e2e suite verifies it in both directions: with the fix the second visit to a revalidating fixture reports 304 and stays healthy, without it the fixture reproduces the CI line verbatim.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The probe dropped every canonical and alias domain of a role whose server.status_codes.default named anything at or above 400, so a single by-design 4xx removed the whole vhost from CSP inspection. The checker now takes --accept-status <domain>=<code>, which keeps the page navigated and its policy evaluated, so the declaration buys coverage instead of costing it.
csp_accept_status reads the existing SPOT in utils/roles/applications/status_codes.py rather than restating the rule. That resolves per vhost, because server.status_codes is keyed like domains.canonical, so two domains of one role can differ; the old filter looked only at default and was blind to seaweedfs declaring its api, filer and master keys separately. The threshold is the checker's own: anything below 300 passes unflagged, so a declared redirect is named too.
csp_skip_domains keeps only the half that has no vhost to probe at all, a canonical domain owned by a disabled service, and gains the tests that half never had.
Measured: 14 domains move from skipped to probed, among them jenkins, mattermost, n8n, prometheus, mirror and xmpp. 28 domains carry an entry in total. Across the CI diagnostics of this run 473 of 476 observed responses were 200, the single infinito vhost answering 403 is api.seaweedfs, and it declares that code, so it stays green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
csp-checker navigates every URL with a 20000 ms budget until
domcontentloaded. Over the Tor SOCKS proxy that expires before a page of
module scripts finishes: run 34614940508 lost Mastodon#1 and OpenProject#1
to it, 26 of 27 attempts logging "Unable to reach http://<onion>/
(Navigation timeout of 20000 ms exceeded)" while sys-service-terminator
retried the unit for 35 minutes. No policy was ever read; the checker never
got that far.
csp-checker 2.3.0 takes --timeout, so script.py takes --onion-timeout and
hands it to the onion batch alone, before the -- separator that would make
it a URL. Clearnet keeps the image default. The budget is
HEALTH_CSP_TIMEOUT_MS (20000, the image default) times the tor flavor's
multiplier, 100000 ms on the public network.
The unit's TimeoutStartSec had to follow. It was 30 + 25 s per domain --
130 s for the two onion vhosts Mastodon checks, while a 100 s budget needs
up to 200 s, so systemd would have killed the check mid-run. The filter now
takes onion_per_domain_seconds and counts .onion domains at that rate;
without the argument it computes exactly what it did before, so
sys-ctl-hlth-webserver is unchanged. The Mastodon case goes from 130 s to
450 s, a clearnet-only play stays at 130 s.
Tests pin that --timeout is appended only when set and stays ahead of the
separator, that main budgets the onion batch alone, that the role passes
both --onion-timeout and the onion rate for the start budget (both guards
fail against the previous tasks file), and the filter's split rate.
Not covered: the filter still caps at 3600 s, which binds beyond 34 onion
domains in one round.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The required_by coverage verifier treated every flat 'required_by: {categories/roles}' as required in BOTH deploy modes. But sys-ctl-rpr-container-hard/soft repair compose compositions (DIR_COMPOSITIONS) and gate their body to compose; declaring required_by:[web] flat made the verifier expect them on swarm too, where they correctly skip, so web-opt-rdr-domains swarm CI failed 'required role(s) did not execute'. Let required_by nest under 'compose:'/'swarm:'; the verifier infers the deploy mode from the role set, mirroring DEPLOYMENT_MODE's own groups['svc-swarm-node'] definition, and enforces only the matching mode.
Migrate all 21 existing flat declarations to mirror under both modes (behavior unchanged) and mark sys-ctl-rpr-container-* compose-only. role_has_required_by reads both forms. New lint test_required_by_mode_keys requires the explicit compose:/swarm: form so the flat-vs-gate contradiction can't recur.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
274e4fd6f added required_by on the csp lifecycle entry and inadvertently
reset csp-checker.version back to 2.1.1, undoing the bump from 378fe140f.
Restore 2.1.2.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds `required_by: { categories: [web] }` to the four roles that MUST run
on every web-app/web-svc deploy: sys-svc-webserver-{core,https} (the
nginx/openresty stack) and sys-ctl-hlth-{webserver,csp} (the deploy-time
coverage checks).
Activates the cli.meta.roles.services.called verifier (added in the
previous commit). Without this annotation the verifier finds no required
roles and silently passes; with it, every web-* deploy round/pass asserts
that these four roles actually executed in the Ansible log, catching
silent coverage regressions like the one fixed in
158681821 (sys-svc-webserver-core: hlth-include moved out of once-flag).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous :stable tag points at a build whose with-ca-trust.sh
populated the host NSS DB at \$HOME/.pki/nssdb, which Chromium 113+ no
longer consults thanks to the Chrome Root Store. The CSP probe therefore
failed every internal target with ERR_CERT_AUTHORITY_INVALID, which
surfaced as `Final-run service failed: hlth-csp.…service` on every CI
deploy whose matrix topology happened to trigger sys-ctl-hlth-csp outside
the service-loader preload context (e.g. web-app-hugo, web-app-mini-qr
in their isolated round-2 variant).
csp-checker 2.1.1 bypasses cert validation at the Puppeteer layer
(ignoreHTTPSErrors + acceptInsecureCerts + --ignore-certificate-errors),
which restores reachability of self-signed internal domains and unblocks
the affected deploys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidates image declarations under a single schema: every role's
`<svc>.{image,version}` lives in `meta/services.yml` alongside compose
service entries. The `image` lookup now reads from disk per role,
`iter_role_images` collapses to a single source, and the mirror chain
(resolver + apply_mirror_overrides) drops the dead `images:` branch.
`test-e2e-playwright` resolves `TEST_E2E_PLAYWRIGHT_IMAGE` directly via
`lookup('image', 'playwright', 'ref')` in vars/main.yml; the runtime
set_fact + override-empty-string dance is gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Makefile lint integration:
* scripts/install/lint.sh: replace the prior checkmake binary install with
ensure_mbake (mbake is pip-installable, parses our Makefile correctly
whereas checkmake bailed on .ONESHELL: / .SHELLFLAGS: / ifneq).
* scripts/lint/makefile.sh: new runner — `mbake format --check Makefile`
reports formatting drift and missing .PHONY declarations.
* scripts/lint/autoformat.sh: add a two-pass `mbake format Makefile`
(mbake's formatter is not strictly idempotent on the first pass).
* Makefile: split the multi-line `.PHONY:` block into 18 single-line
`.PHONY:` declarations so mbake's parser can validate it; add the
previously-missing `install-python-dev` and `build-dependency` PHONY
targets; add `lint-makefile` aggregate-and-direct target.
* .pre-commit-config.yaml: new make-lint-makefile hook scoped to
`^Makefile$`.
* .github/workflows/lint-makefile.yml: new CI workflow (pip-install mbake
+ make lint-makefile).
* .github/workflows/ci-orchestrator.yml: wire lint-makefile into
lint-gate.needs.
Misc:
* .gitignore: add roles/importer_result.json (tooling artefact written by
galaxy-importer when it runs in roles/).
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
