HEALTH_CSP_SKIP_DOMAINS skipped every CURRENT_PLAY_REDIRECT_DOMAINS
source: the DOMAIN_PRIMARY apex, the node onion apex, inventory
redirect mappings and per-app aliases. The skip dates from when the
checker judged the 301 itself; since csp-checker 3.0.0 it follows the
redirect chain and judges the document it lands on. So a broken or
unserved alias target, and a redirect leaving its network, were never
loaded.
A redirect source now stays skipped only when its target is not among
CURRENT_PLAY_DOMAINS_ALL, i.e. lies off this node, like SOFTWARE_DOMAIN
for the apex without web-app-dashboard. Every other source is crawled.
Closes#929
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
csp-checker 3.0.5 starts Chromium without HTTPS-Upgrades. Before it, every
http:// onion vhost was first tried as https://host:443 through the Tor SOCKS
proxy, which broke the port-80 requests to the same onion service with
net::ERR_SOCKS_CONNECTION_FAILED. hlth-csp then failed all 25 start attempts
and took the deploy down in 11 swarm/tor jobs of main run 38038541631
(Gitea#0, Hugo#0, Listmonk#0, Mattermost#2, Nextcloud#0/#4/#7, Penpot#0,
Pixelfed#0, YOURLS#0, odoo#0).
Verified against a local compose stack over Tor: image 3.0.4 as shipped hit
the SOCKS error on css and asset and reported extra redirects elsewhere; with
the 3.0.5 change no SOCKS error and only the redirects the server sends. The
csp-checker lint and all 16 E2E tests pass on the image built from 3.0.5.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
3.0.3 has a tag and no image. Its release build failed the same 204 fixture
3.0.2 failed on, because the handling read the status the moment page.goto
rejected while the listener sat on a second CDP session, and the protocol
orders messages within a session rather than between two of them. The status
now gets awaited, so the verdict no longer depends on which session delivers
first, and the page close that the wait pushes into the navigation's commit
window is bounded rather than awaited forever.
A third number rather than a rebuilt 3.0.3, for the reason 3.0.3 itself got
one: the mirror copies by tag, so an overwritten tag turns every copy that has
already seen it into a silent lie, and the deploys pull through that mirror.
The reachability check answers 404 for 3.0.4 until the release publishes, the
way it did for the two before it. That clears when the release runs.
Worth stating plainly: the runner's failure was never reproduced locally. The
ordering hole is closed by construction and two full suites are green, but
whether it was the cause is decided by the release build, not here. If that
build fails on the 204 fixture again, this pin is worth as little as the last
one and the job log is what to read next.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3.0.2 never produced an image. Its release build failed the E2E suite on the
204 fixture itself: the handling listened for Network.responseReceived, and a
204 is never committed, so Chromium emits no such event and the checker kept
reporting the host as unreachable. The pin therefore addressed a tag that the
registry answers with 404, which the image-reachability check has been saying
since the bump landed.
csp-checker d827de1 reads the status from Network.responseReceivedExtraInfo,
the event that carries an uncommitted navigation's status, correlated with the
Document request id. Reproduced and fixed in one E2E run, with the download
fixture still failing so the change distinguishes a 204 rather than blessing
every aborted navigation.
A new number rather than a rebuilt 3.0.2, because the mirror copies by tag: an
overwritten tag turns every copy that already saw 3.0.2 into a silent lie, and
the deploys pull through that mirror.
Until the release publishes, the image does not exist and the reachability
check answers 404 for 3.0.3 the way it did for 3.0.2. That clears when the
release runs, not by a change here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
n8n ships isbot.spawn(['bot']) unconditionally and answers any user agent
matching /bot/i with a bodyless 204. This checker calls itself
CSP-CheckerBot, so the probe never reached the application. Chromium
cancels a navigation on 204, page.goto threw net::ERR_ABORTED, and the
checker reported the host unreachable. In CI job 105068914482 the unit
failed 25 times over 1551 seconds and took the job with it.
3.0.2 ends the visit as reachable when the document response is exactly
204, and prints that it checked nothing, because page.url is about:blank
afterwards. Exactly 204, not the 2xx range: a 200 carrying
Content-Disposition aborts the same way and is a download, not an empty
document. The checker's e2e suite pins both directions, with the download
fixture as the control that fails if the condition is ever widened.
Same shape as df4b7757bc took for 304: a status the checker's own contract
calls healthy, rejected at runtime over how the browser reacts to it.
The user agent keeps its name. The checker is a bot and a bot filter
blocking it is correct; what changes is that the consequence is now stated
rather than hidden. A vhost answering 204 reports that its CSP went
unchecked instead of reporting nothing at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
csp-checker 3.0.0 narrowed the healthy set to 2xx by failing every status at or above 300, which swept in 304. The health service then died on the first revalidated vhost: in CI job 104991065851 it reported element.<onion> unreachable with Status 304, and sys-service-terminator retried the unit 25 times over 1792 seconds before the job failed.
3.0.1 takes 304 out of the failure branch. The checker's own e2e suite verifies it in both directions: with the fix the second visit to a revalidating fixture reports 304 and stays healthy, without it the fixture reproduces the CI line verbatim.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The checker matches accepted codes by hostname, and a clearnet vhost
shares only its leftmost label with its onion twin, so --accept-status
mirror.<primary>=404 never applied to mirror.<host>.onion. That vhost
serves the same by-design 404, so the probe called it unreachable and
systemd failed the service -- with the identical message the missing
flag produced, while the status had already moved from 2/INVALIDARGUMENT
to 1/FAILURE.
--skip-domain carried this derivation all along, in is_skipped_domain;
replacing it with --accept-status dropped it. The expansion now runs on
the same label rule, in the wrapper rather than the lookup, because the
onion address does not exist yet when the unit is rendered -- only the
wrapper sees both names, and it reads them from the nginx configs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The systemd unit has passed --accept-status since f247b29fd0, but the
wrapper script never learned the flag. argparse rejected it, so the
service failed with status=2/INVALIDARGUMENT on every host that deploys
the role, in compose and in swarm alike.
The checker consumes every token after a list flag that does not start
with "--", and a URL does not, so both lists now share a single
separator placed before the URLs. The previous placement sat directly
behind --ignore-network-blocks-from and would have fed a second list
straight into the URL parser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The probe dropped every canonical and alias domain of a role whose server.status_codes.default named anything at or above 400, so a single by-design 4xx removed the whole vhost from CSP inspection. The checker now takes --accept-status <domain>=<code>, which keeps the page navigated and its policy evaluated, so the declaration buys coverage instead of costing it.
csp_accept_status reads the existing SPOT in utils/roles/applications/status_codes.py rather than restating the rule. That resolves per vhost, because server.status_codes is keyed like domains.canonical, so two domains of one role can differ; the old filter looked only at default and was blind to seaweedfs declaring its api, filer and master keys separately. The threshold is the checker's own: anything below 300 passes unflagged, so a declared redirect is named too.
csp_skip_domains keeps only the half that has no vhost to probe at all, a canonical domain owned by a disabled service, and gains the tests that half never had.
Measured: 14 domains move from skipped to probed, among them jenkins, mattermost, n8n, prometheus, mirror and xmpp. 28 domains carry an entry in total. Across the CI diagnostics of this run 473 of 476 observed responses were 200, the single infinito vhost answering 403 is api.seaweedfs, and it declares that code, so it stays green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The skip list gated its whole loop body on the group_names selection, so a role reaching a node through a runtime include_role never contributed its server.status_codes.default. web-svc-mirror arrives that way on every Tor node and declares 404 as healthy; on a node provisioned from the svc-net-tor README the probe read that 404 as unreachable and hlth-csp could not start.
Measured in CI job 104622514568: the unit rendered without --skip-domain and the checker reported ERR_SOCKS_CONNECTION_FAILED on the mirror onion 26 times. Only the service-disabled half depends on the selection now. The widened branch can contribute at most 14 of 266 applications, and naming a domain no vhost serves is inert because script.py filters an nginx-enumerated list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make quality's autoformat re-wrapped the line 6e98ecf66 added; the rendered expression is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
csp-checker navigates every URL with a 20000 ms budget until
domcontentloaded. Over the Tor SOCKS proxy that expires before a page of
module scripts finishes: run 34614940508 lost Mastodon#1 and OpenProject#1
to it, 26 of 27 attempts logging "Unable to reach http://<onion>/
(Navigation timeout of 20000 ms exceeded)" while sys-service-terminator
retried the unit for 35 minutes. No policy was ever read; the checker never
got that far.
csp-checker 2.3.0 takes --timeout, so script.py takes --onion-timeout and
hands it to the onion batch alone, before the -- separator that would make
it a URL. Clearnet keeps the image default. The budget is
HEALTH_CSP_TIMEOUT_MS (20000, the image default) times the tor flavor's
multiplier, 100000 ms on the public network.
The unit's TimeoutStartSec had to follow. It was 30 + 25 s per domain --
130 s for the two onion vhosts Mastodon checks, while a 100 s budget needs
up to 200 s, so systemd would have killed the check mid-run. The filter now
takes onion_per_domain_seconds and counts .onion domains at that rate;
without the argument it computes exactly what it did before, so
sys-ctl-hlth-webserver is unchanged. The Mastodon case goes from 130 s to
450 s, a clearnet-only play stays at 130 s.
Tests pin that --timeout is appended only when set and stays ahead of the
separator, that main budgets the onion batch alone, that the role passes
both --onion-timeout and the onion rate for the start budget (both guards
fail against the previous tasks file), and the filter's split rate.
Not covered: the filter still caps at 3600 s, which binds beyond 34 onion
domains in one round.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cosmos generator linked a consumer to its provider only through an
'<role>' in group_names reference or an explicit provides: name. A hard
dependency has neither -- pinning the provider on unconditionally is
exactly what removes the Jinja reference the generator was reading -- so
web-svc-asset's bond to web-svc-file vanished from the diagram the moment
it became binding. Match a service key against the provider's own primary
key from the registry, in both directions.
Services the role turns off drew a solid 1:1 edge, claiming a mandatory
relationship for something that carries nothing at deploy time. They now
render as a red 0..0 edge; edge merging gained a real precedence so a
provider consumed through two keys stays live when only one is off.
Named colour, not hex: a '#' opens an entity code in the mermaid lexer and
turns the whole diagram into a parse error on GitHub.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
default.env declares INFINITO_DOMAIN and everything is meant to derive from it.
Nothing checked that, so copies accumulated. A copy is a second source nobody
updates, and it fails in the least useful way: the code keeps running and
quietly talks about the wrong host. integration_mastodon is the worked example
-- it asserted the provisioned partner equalled a clearnet literal, so the spec
passed on a clearnet node and failed on an onion one while the product was
right in both.
tests/lint/repository/spot/test_no_hardcoded_primary_domain.py now fails on the
literal anywhere in tracked source. It reads the value it hunts for FROM
default.env, so the rule follows the SPOT instead of repeating it and would
otherwise be its own first violation.
Scope, and why each exclusion is not a loophole: default.env itself is the
source; .md is documentation; tests/ and scripts/tests/ hold fixtures that must
assert against a named host; inventories/ is where writing the real values is
the entire point; gitignored output carries the value because it correctly
derived it. Everything else is in.
Two real hardcodes are repaired rather than suppressed.
utils/domains/default_primary.py mirrors utils/roles/lifecycle.py -- one
function reading one key through parse_static_env -- and replaces the module
constant in utils/cleanup/nginx_vhosts.py and the argparse default in
cli/meta/domains/__main__.py. Both resolve to the same value as before.
Seven documentation sites become <primary-domain> placeholders instead of
carrying a nocheck: a Python docstring cannot host one without polluting the
docstring, and a placeholder does not rot when the default changes. The single
nocheck is scripts/system/tls/trust/wsl2.sh:61, where the marker is matched
verbatim against what an earlier run wrote into the Windows hosts file --
deriving it would orphan that block instead of replacing it.
Collateral, not intent: touching roles/sys-ctl-hlth-csp/files/script.py brought
it under the comment lint, which found five pre-existing narration comments.
Three deleted, `fn[:-5] # strip ".conf"` replaced by fn.removesuffix(".conf"),
and the root-user comment rewritten in the sanctioned Exception form. Its 33
tests stay green.
tests/lint/repository/test_no_hardcoded_container_name.py (05f425ea5) is the
same family for INFINITO_CONTAINER and could move into spot/ later; this change
leaves it where it is.
make test-lint: 326 passed. The rule found 18 sites before the cleanup and none
after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
22 commits, mostly dependabot and docker image-version bumps, plus two fixes:
5153f3030 loads keycloak-js 26 as an ES module, 87a053518 makes the coredns
Corefile readable to the container user.
One conflict, in roles/web-app-dashboard/templates/javascript/oidc.js.j2, resolved
as a union because the two sides touched the same line for unrelated reasons.
edb96e9ae had hoisted the asset URL into KC_JS_ASSET_URL and rewrote it onto the
onion mirror; 5153f3030 changed the path to lib/keycloak.js and declared a local
Keycloak binding, since the adapter now arrives as a module default export instead
of a window global. The onion rewrite stays at the conflict site, the local binding
is kept because loadKeycloakAdapter returns the constructor rather than setting a
global, and the path change moved to the constant's definition.
An import() fetch always runs in CORS mode, which is why 5153f3030 gave the CDN
aca_origin. The onion route reaches the same asset through web-svc-mirror, whose
proxy.conf.j2 already sets that header, so the onion flavor needs no counterpart.
make quality green on the merge result: test-external, test-integration, test-lint,
test-unit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_task_file_numbering.py: numbered task
files use exactly two leading digits, a numbered *_core.yml must be
00_core.yml, per directory the numbers are unique/consecutive and start
at 01 (00 with a core file), numbered includes appear in ascending
order, and shared helpers live in tasks/utils/ instead of the numbered
chain.
Apply it across every role: rename NN_core.yml to 00_core.yml with a
consecutive down-shift of the siblings, move letter-suffix children into
a subdirectory named after their parent task, relocate helpers included
from more than one place into tasks/utils/, and update every reference
(include_tasks, path_absolute, .ansible-lint-ignore, requirement docs).
Legacy narration comments in the touched files are deleted; the upstream
xwiki.properties reference block opts out via nocheck. The unused
MAILU_IP6_PUBLIC var (only ever named in now-removed comments) is dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rewrite all 254 role descriptions to a uniform noun-phrase schema
(function-first, no leading verb, no trailing period, <=120 chars) and
enforce it via the extended description lint. Rename 12 role README H1
titles for clarity (e.g. Backup Container to Local -> Backup Docker
Volumes, CSS Service -> Corporate Design, NGINX Legal -> Imprint) and
regenerate their Cosmos and Quick Setup sections. Overview table
regenerated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Roles declared under meta/main.yml dependencies now appear on both the
Dependencies and Dependents side, labelled with the glossary gear
symbol and connected by a fixed 1:1 edge; the legend explains it. All
role READMEs regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Edges now carry their relation: solid 1:1 for fixed consumption, dashed
0..1 for conditional (group-membership gated) consumption. Below every
diagram the generator renders a reading key for the edge kinds and the
node markers, with the glyphs drawn from the shared symbol glossary.
All role READMEs regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add cli/build/readme: derives each role README's managed sections from
templates/roles/README.md.j2.tmpl (the Cosmos mermaid diagram from
meta/services.yml, an invokable-gated Quick Setup, and Credits).
The Cosmos diagram shows dependencies, every declared service, and
dependents (roles that consume this one, matched by role name and by
provides provision-name in both directions). Role nodes carry deploy-mode
emoji markers (compose/swarm/host); disabled and shared services get a
stop mark; dependents are capped at 12.
test_readme enforces the Quick Setup body is byte-exact with the template
render. `make readme-generate` and `make cosmos` regenerate; `make cosmos`
also runs inside `make quality`.
Refactor: utils/symbol_glossary.py is the single source of truth for the
word/emoji mapping (unique project-wide), consumed by the complexity
matrix and the Cosmos builder. utils/roles/deploy.py holds role_has_stack
and role_deploy_modes, shared by both so the logic never drifts.
Add `make lint-mermaid` and its CI workflow, rendering every diagram via
mermaid-cli; the render step provisions chrome-headless-shell and unzip
on demand so it works in the minimal container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SeaweedFS is a server-side S3 backend for its consumers; only the
filer/master admin UIs are browser-facing. Split them into a `frontend`
service that stays off on onion nodes ('svc-net-tor' in group_names):
- frontend disabled: no nginx proxy container, no filer/master vhosts,
no oauth2 SSO sidecar; the S3 API vhost is unaffected
- stale filer/master vhosts from earlier frontend-enabled deploys are
removed so the site-wide CSP health probe stops hitting dead domains
- csp_skip_domains additionally skips canonicals a disabled service
declares via services.<name>.domains (defense in depth)
- UI personas (admin/biber/guest/storage-buckets) gate on the
FRONTEND_SERVICE_ENABLED flag; tor variants dropped (backend role)
Verified on the onion node: 4/4 UI specs skip, S3 API reachable, CSP
health check green, matrix S3 media consumer unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Role README Credits now name galaxy_info.author (plain bold or wrapped
in a Markdown link), matched byte-for-byte against the role's own
meta/main.yml, instead of a hard-coded maintainer name. Regenerate all
245 role README Credits blocks to the new 'Implemented by ...' format.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the csp, domains, and networks blocks out of roles/*/meta/server.yml
into dedicated meta files, each mapping to its own top-level applications key
via the recursive meta-walk:
csp: -> meta/csp.yml -> applications.<app>.csp
domains: -> meta/domains.yml -> applications.<app>.domains
networks: -> meta/networks.yml -> applications.<app>.networks
Residual nginx/proxy keys (status_codes, client_max_body_size, locations,
ip6_enabled, config_upstream_url) stay in server.yml; an emptied server.yml is
deleted. 98 server.yml -> 249 new files, 74 deleted.
- Consumers repointed applications.<app>.server.{csp,domains,networks} ->
applications.<app>.{csp,domains,networks} across templates, vars, tasks,
lookups and filters.
- Infra readers migrated: canonical_domains_map, application_domain_index,
build_ldap_role_entries, service-registry overlay discovery,
roles_with_service, and the file-based readers utils/domains/list.py +
utils/meta/scan.py (read the now-deleted server.yml directly).
- Detection + mapping: _META_TOPICS gains csp/domains/networks;
ROLE_FILE_META_{CSP,DOMAINS,NETWORKS} constants + ROLE_FILES entries.
- New self-expiring lint guard (test_server_meta_no_split_fields) fails if a
server.yml still carries a split field; auto-fails from 2026-12-28 as a
delete reminder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consumer lookup plugins now reach the merged config through the Ansible
lookup loader instead of importing get_merged_applications /
get_merged_domains from utils.cache. The merge stays one cached
implementation surfaced via the applications/config/domains/users lookups;
consumers call lookup_loader.get(...).run() (the compose_networks idiom).
- get_merged_applications walks the whole meta/ tree recursively
(meta/**/*.yml to applications.<app>.<path>, addons and any depth) and
folds the raw meta/volumes.yml subtree onto applications.<app>.volumes
after the templar render so the Jinja source: strings stay unrendered.
- 31 consumer lookups rerouted; providers (applications/config/
applications_current_play/users/domains) and the raw-volume accessors
(volume/nginx) keep the cache import behind # nocheck: lookup-cache-import.
- New lint test_no_cache_imports_in_lookups enforces the contract.
- nfs_flavor is now the flavor SPOT (svc-storage-nfs-server + nfs_client_src
resolve via lookup('nfs_flavor')); removed the unused STORAGE_NFS_FLAVOR
group_var. Extracted extract_default_credentials to
utils/cache/schema_credentials.py to keep applications.py under the cap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The required_by coverage verifier treated every flat 'required_by: {categories/roles}' as required in BOTH deploy modes. But sys-ctl-rpr-container-hard/soft repair compose compositions (DIR_COMPOSITIONS) and gate their body to compose; declaring required_by:[web] flat made the verifier expect them on swarm too, where they correctly skip, so web-opt-rdr-domains swarm CI failed 'required role(s) did not execute'. Let required_by nest under 'compose:'/'swarm:'; the verifier infers the deploy mode from the role set, mirroring DEPLOYMENT_MODE's own groups['svc-swarm-node'] definition, and enforces only the matching mode.
Migrate all 21 existing flat declarations to mirror under both modes (behavior unchanged) and mark sys-ctl-rpr-container-* compose-only. role_has_required_by reads both forms. New lint test_required_by_mode_keys requires the explicit compose:/swarm: form so the flat-vs-gate contradiction can't recur.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The CSP check derives its domain list + scheme from the locally-rendered nginx .conf files (os.listdir of the config dir), which exist only where the proxy renders them (the stack host); a swarm worker has no config dir to crawl. Gate the include on IS_STACK_HOST so it does not fail per-node. (Unlike sys-ctl-hlth-webserver, this is a genuine data-locality gate, not reachability.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the "web-opt-rdr-domains in deployed" gate in the
current_play_redirect_domains lookup. CURRENT_PLAY_REDIRECT_DOMAINS now
always carries DOMAIN_PRIMARY to DOMAIN_HOMEPAGE when both are set and
differ. This triggers sys-stk-front-base to auto-load web-opt-rdr-domains
so openresty actually renders a vhost matching its declared canonical, and
the bare apex stops hitting the default-server's ssl_reject_handshake.
CSP probing skips those redirect-source domains: pure 301 vhosts have no
CSP header to validate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- web-opt-rdr-www canonical: rdr-www.{DOMAIN_PRIMARY} -> w3redirect.{DOMAIN_PRIMARY}.
- New lookup `www_redirect_domains` (role-scoped): returns full {source, target}
mapping list, resolves DOMAIN_HOMEPAGE target for the role's own canonical via
lookup('domain', 'web-opt-rdr-www'). REDIRECT_WWW_DOMAINS now carries dicts;
tasks/main.yml and _01_cloudflare_edge_redirect.yml updated to consume dicts.
- New lookup `current_play_redirect_domains`: replaces the two inline
set_fact merges in tasks/stages/01_constructor.yml. Merges user-supplied
redirect_domain_mappings, primary-redirect (conditional on web-opt-rdr-domains
in deployed), and per-app domain mappings (inlined from the now-removed
domain_mappings filter).
- domain_redirect_mappings filter removed; web-app-keycloak RedirectUrisTest
pollution of utils.roles.applications.config in sys.modules fixed via
tearDownClass cleanup.
- csp_skip_domains lookup: csp.health.skip opt-out path removed (architecture
no longer needs it).
- Lint test_only_web_app_roles_integrate_dashboard expanded from web-app-* to
web-*; test_role_and_group_vars_used now scans .py files too so vars consumed
only inside plugins (DOMAIN_HOMEPAGE, AUTO_BUILD_ALIASES) are seen.
- New role-scoped variants.yml on web-opt-rdr-www with dashboard on/off.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
274e4fd6f added required_by on the csp lifecycle entry and inadvertently
reset csp-checker.version back to 2.1.1, undoing the bump from 378fe140f.
Restore 2.1.2.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`status` was too vague for what the lookup returns (apps in current
play). Rename to `deployment` and add two new fields:
* `runtime` — RUNTIME variable value
* `deployed` — runtime-aware authoritative set: equals `running`
(whitelist or groups) for ephemeral CI runtimes (act, github),
falls back to `groups` for persistent runtimes (host, dev). This
matches reality: in CI the container is fresh so only whitelisted
apps exist; in production a partial deploy still leaves prior
apps live, so groups is the right set.
Switch the two webserver/csp healthchecks to `lookup('deployment').deployed`
so they only probe domains for apps that actually exist in the current
deploy. In production behaviour is unchanged.
Update remaining consumers (cli/build/include, web-app-keycloak,
test-e2e-playwright) for the rename. Existing `status` semantics
(`whitelist`, `running`, `groups`, `all`) preserved under the new
name.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds `required_by: { categories: [web] }` to the four roles that MUST run
on every web-app/web-svc deploy: sys-svc-webserver-{core,https} (the
nginx/openresty stack) and sys-ctl-hlth-{webserver,csp} (the deploy-time
coverage checks).
Activates the cli.meta.roles.services.called verifier (added in the
previous commit). Without this annotation the verifier finds no required
roles and silently passes; with it, every web-* deploy round/pass asserts
that these four roles actually executed in the Ansible log, catching
silent coverage regressions like the one fixed in
158681821 (sys-svc-webserver-core: hlth-include moved out of once-flag).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous :stable tag points at a build whose with-ca-trust.sh
populated the host NSS DB at \$HOME/.pki/nssdb, which Chromium 113+ no
longer consults thanks to the Chrome Root Store. The CSP probe therefore
failed every internal target with ERR_CERT_AUTHORITY_INVALID, which
surfaced as `Final-run service failed: hlth-csp.…service` on every CI
deploy whose matrix topology happened to trigger sys-ctl-hlth-csp outside
the service-loader preload context (e.g. web-app-hugo, web-app-mini-qr
in their isolated round-2 variant).
csp-checker 2.1.1 bypasses cert validation at the Puppeteer layer
(ignoreHTTPSErrors + acceptInsecureCerts + --ignore-certificate-errors),
which restores reachability of self-signed internal domains and unblocks
the affected deploys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`lookup('image', '<role>', '<svc>', 'ref')` only consulted
`images_overrides.*` plus role defaults, which the mirror pipeline
never populates — the pipeline writes into
`applications.<role>.services.<svc>.{image,version}`. As a result,
`sys-svc-container`'s `BUSYBOX_IMAGE` / `NODE_IMAGE`,
`sys-ctl-hlth-csp`'s `HEALTH_CSP_IMAGE`, and `test-e2e-playwright`'s
`TEST_E2E_PLAYWRIGHT_IMAGE` always resolved to the raw upstream ref
and hit Docker Hub directly. In CI this surfaced as
`toomanyrequests: ...unauthenticated pull rate limit` on
`busybox:1.37`.
Switches every productive `lookup('image', ...)` to
`lookup('config', '<role>', 'services.<svc>.image|version')` — the
same path templates like `web-app-oauth2-proxy` already use, which
resolves through the merged `applications.*` map and so transparently
picks up the GHCR mirror override written by the inventory creator.
Removes the now-unused `plugins/lookup/image.py` plugin and its unit
test (zero remaining consumers across roles, scripts, and tests).
Rewrites the image-origin / image-mirror / dockerfile docs to teach
the `lookup('config', ...)` pattern, dropping references to the
deleted plugin and the dead `images_overrides.*` variable. Doc edits
follow the `documentation.md` rules (no em dashes, one sentence per
line, no historical phrasing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the contract that 0ef2a3481 established for sys-ctl-hlth-webserver
(reads status_codes.default from meta/server.yml) so the CSP probe stops
flagging federation-only roles like web-app-bridgy-fed as unhealthy.
Previously, sys-ctl-hlth-csp invoked `node health-csp.js` against every
nginx vhost's `/` and treated any non-2xx/3xx as "Unable to reach". For
web-app-bridgy-fed, whose canonical surfaces are the federation-protocol
endpoints (/robots.txt, /.well-known/host-meta, /web/<actor>) and whose
Flask app deliberately serves 404 on `/`, the probe failed, systemd
retried 10x over ~10 min, and sys-service-terminator aborted the deploy
with rc=2 (CI job 76714878060).
The Playwright spec already encoded the contract: 4xx-at-/ is by design,
5xx is the only unhealthy signal. The fix in 0ef2a3481 surfaced that
contract via status_codes.default for hlth-webserver but did not extend
it to hlth-csp.
This change adds:
- A new csp_skip_domains filter (filter_plugins/csp_skip_domains.py)
that walks every selected application, reads
server.status_codes.default, and returns the canonical+alias domains
of any role declaring a code >= 400. Follows the
web_health_expectations pattern.
- HEALTH_CSP_SKIP_DOMAINS in vars/main.yml, computed once via the
filter.
- --skip-domain repeatable CLI flag in files/script.py that filters
the discovered nginx-config domain list before URLs are built.
Side benefit: web-app-listmonk (default: 404) and web-svc-html
(default: [403, 200]) are now also correctly skipped if deployed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `image` lookup inferred `role_id` from `role_name` when only
`service_name` was passed. Inference broke lazy re-evaluation: a default
like `HEALTH_CSP_IMAGE: "{{ lookup('image', 'csp-checker', 'ref') }}"`
defined in sys-ctl-hlth-csp got re-evaluated inside sys-service's
systemctl.service.j2 template (via `system_service_tpl_exec_start`),
where `role_name='sys-service'`. The lookup then searched
`roles/sys-service/meta/services.yml` for `csp-checker` and aborted with
`no image mapping found for role 'sys-service' service 'csp-checker'`.
CI run 25976825584 surfaced this on six apps (web-app-{bridgy-fed,chess,
mig,hugo,mini-qr,postmarks}) whose deploy reached sys-svc-webserver-core
outside the service-loader preload context, which is what actually
triggers the sys-ctl-hlth-csp -> sys-service include chain.
Make `role_id` mandatory in the plugin signature, remove `_infer_role_id`,
and update the three callsites that relied on inference
(sys-ctl-hlth-csp, sys-svc-container, test-e2e-playwright) to pass the
owning role id explicitly. A migration guard catches the removed
`(service_name, want)` form and emits a clear error instead of the
confusing "missing services.yml" file-not-found message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidates image declarations under a single schema: every role's
`<svc>.{image,version}` lives in `meta/services.yml` alongside compose
service entries. The `image` lookup now reads from disk per role,
`iter_role_images` collapses to a single source, and the mirror chain
(resolver + apply_mirror_overrides) drops the dead `images:` branch.
`test-e2e-playwright` resolves `TEST_E2E_PLAYWRIGHT_IMAGE` directly via
`lookup('image', 'playwright', 'ref')` in vars/main.yml; the runtime
set_fact + override-empty-string dance is gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh: rename AUTOFORMAT_PARALLEL to PARALLEL,
declare it at file start with default `true`; still env-overridable.
Reword the worker comment so the literal `# shellcheck` token no
longer appears at start-of-line and trips SC1072/SC1073.
* scripts/lint/galaxy.sh: drop the GALAXY_FATAL / GALAXY_NAMESPACE /
GALAXY_PARALLEL env vars (over-engineered for a single repo). Failure
is always fatal. NAMESPACE is an internal constant. Add the same
PARALLEL env var (default `true`, env-overridable); falsy demotes the
xargs fan-out to one worker.
* docs/contributing/environment/variables.md: collapse the four
AUTOFORMAT_*/GALAXY_* rows into one `PARALLEL` row covering both
consumers.
* docs/contributing/tools/make.md: refer to `PARALLEL=0` everywhere;
add the lint-galaxy parallelism note.
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
