The compose-up handler resolves its chdir from lookup('container', application_id, 'directories.instance') at flush time. Bluesky passes compose_handlers_flush false to sys-stk-backend and notifies compose-build and compose-up afterwards, so the topics stayed pending until the next flush. That flush belongs to svc-prx-openresty, which the front proxy include pulls in, and it ran them with openresty's application_id.
Run 34653754175 shows the effect: the only compose up in the whole job created the openresty container in 0.52s, no pds container lifecycle line appears anywhere, and the wait for PDS XRPC then retried for 69s against a stack that was never brought up, ending on 'Status code was -1 and not [200]: Request failed: <urlopen error [Errno 111] Connection refused>'.
The flush now sits directly after the rebuild notifier, before the first front proxy include. sys-stk-backend has already rendered the stack there, so this is not a return to 293c67780d, whose registration sat before the staging and let the container runtime's first flush run the compose handlers before ca-inject and compose.yml existed.
The re-own include ahead of it repeats what sys-stk-backend and sys-svc-rdbms already do: it runs no task, it re-registers the handlers, and it binds them to this application_id instead of whatever the play happens to carry. The ordering test asserts the flush precedes the proxy and that the re-own precedes the flush.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pre-compose staging (social-app clone, login-broker copy, pdsadmin
download) ran before sys-stk-backend and notified compose-build and
compose-up. On a fresh host the container runtime's first flush ("Restart
and enable docker service") then ran the compose handlers before
/usr/bin/ca-inject and compose.yml existed, and "Generate CA trust
override" failed (run 34615516009, job 103327034609).
The staging tasks now only register their result. A changed-only notifier
after the sys-stk-backend include triggers compose-build and compose-up, so
no compose topic is pending at any earlier flush, and the handler-only
include that registered the owner early is gone. The ordering test asserts
both halves: the staging files notify no compose topic, and the rebuild
notifier sits after the backend include.
Verified locally in compose, variant 1: the reinstall full cycle passed
pass 1 with failed=0 and the rebuild task changed after the backend
include, pass 2 Playwright passed, and an update run finished failed=0 with
the rebuild task skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34421944328 Bluesky#1 stopped at "The requested handler 'compose-up'
was not found in either the main handlers list nor in the listening
handlers list", right after cloning the social-app build context. The
pre-compose staging (01_social_app.yml, 02_login_broker.yml) and the PDS
routine (03_pds.yml) notify compose-up and compose-build, but the
handlers only exist once sys-stk-backend loads sys-svc-compose, which
happens later in the role.
00_core.yml now includes sys-svc-compose with tasks_from: none.yml and
handlers_from: main right after the once flag, the same handler-only
registration sys-stk-backend and sys-svc-rdbms dedicated.yml already use.
The notifications still run at the stack's own flush, after the compose
files exist; the role has no earlier flush. A handler-order test pins
the registration ahead of the pre-compose block.
The oauth2-proxy cfg heal leaves its inline shell for
files/shell/heal_sso_proxy_cfg.sh, which receives the cfg path and the
container name through the environment. The blanket
"container stop ... || true" becomes a precondition: the container is
stopped only when container inspect finds it, so a directory is still
removed when the container is gone, and a failing stop of an existing
container now fails the task instead of vanishing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A handler name was doing two jobs at once: the line an operator reads in the
play output, and the identifier a `notify:` binds to. The two pull in opposite
directions -- the log wants "restart the SSH service, with diagnostics", the
binding wants a token that survives quoting, YAML flow style and a shell
round-trip -- and a name that changes for readability silently changes which
handler runs.
Every handler that a notify reached by name now carries a `listen:` with that
exact wording, so the binding is explicit before anything is renamed. The
topics themselves become identifiers: `compose up` reads `compose-up`,
`Restart NFS ganesha` reads `restart-nfs-ganesha`, 22 of them across both
sides of every notify. Where the topic is assembled at runtime, both branches
moved together: the conditional in sys-service carries two literals, and the
LDAP renderer interpolates `import-{{ folder }}-ldif-files`, which is what
makes those handlers reachable at all.
Freed from the binding, the names take the emoji prefix every other task line
already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_task_file_numbering.py: numbered task
files use exactly two leading digits, a numbered *_core.yml must be
00_core.yml, per directory the numbers are unique/consecutive and start
at 01 (00 with a core file), numbered includes appear in ascending
order, and shared helpers live in tasks/utils/ instead of the numbered
chain.
Apply it across every role: rename NN_core.yml to 00_core.yml with a
consecutive down-shift of the siblings, move letter-suffix children into
a subdirectory named after their parent task, relocate helpers included
from more than one place into tasks/utils/, and update every reference
(include_tasks, path_absolute, .ansible-lint-ignore, requirement docs).
Legacy narration comments in the touched files are deleted; the upstream
xwiki.properties reference block opts out via nocheck. The unused
MAILU_IP6_PUBLIC var (only ever named in now-removed comments) is dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
resource_filter now falls back service_name -> entity_name -> hard
default, so entity-level limits keep applying to per-service compose
entries. Compose service names drop the entity prefix (kix, jitsi,
zammad, bluesky); the docker container_name keeps it via compose_only.
A new lint forbids '<entity>-'-prefixed service names.
Service names and internal ports are declared once: vars/main.yml
holds <ROLE>_<SERVICE>_SERVICE constants (container names derived via
entity_name), meta/services.yml declares ports.internal consumed
through config lookups. sso-proxy upstreams stack-qualify the origin
host in swarm because short aliases are ambiguous on shared overlays.
Stale proxy.service_key entries fixed for erpnext/shopware/nextcloud.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pdsadmin binary is mounted from the volumes dir, but 02_pds.yml ran only
after the sys-stk-front-proxy include whose flush already deploys the stack, so
the deploy aborted on the missing source. Move the PDS staging ahead of the
front-proxy block. Also revert pdsadmin from a docker config back to a bind
mount: it is a ~9MB Go binary and swarm configs are capped at ~1MB, so type:
config fails stack deploy. The pds service is manager-pinned, so the
manager-staged bind source is always present on the node that runs it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In swarm the bluesky api/web/view vhosts proxied to the wrong upstream: the
front proxy used the app entity name as the service key and resolve_upstream
hard-failed without an internal port (bluesky has no
services.bluesky.ports.internal). Pass the swarm service keys (pds /
bluesky-broker / view) and the internal ports (added to
services.{api,web,view,broker}) per mode, and freeze + re-pass
proxy_internal_port in sys-stk-front-proxy so it survives the nested role loads
and reaches html.conf.j2 (mirrors the existing front_proxy_service_key freeze).
The sys-stk-front-proxy change is compose-neutral (resolve_upstream ignores
internal_port in compose) and a no-op for the 9 non-passing apps (internal_port
stays ''); it also co-fixes web-svc-coturn, whose deliberate
proxy_internal_port: 80 was being lost (coturn has no
services.coturn.ports.internal, so swarm resolve_upstream hard-failed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements docs/requirements/021-sso-flavor-migration.md atomically:
* Schema: every role's meta/services.yml replaces the parallel
`oauth2:` / `oidc:` top-level blocks with a single `sso:` block
carrying `enabled`, `shared`, and a `flavor` discriminator
(`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
entirety; its 5 sidecar templates move under
roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
`is_oidc_native`) and the oauth2 sub-values
(`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
Three lookup plugins consume the resolver: `sso` (single-app
predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
and plugins/filter/compose_volumes.py all gate on
`lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
`enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
→ `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
`OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
`oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
`OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
`SSO_SERVICE_ENABLED`; Playwright gating helpers
(isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
`flavor: oauth2` with their existing `oauth2.acl.blacklist`
(/user/login) and `oauth2.acl.whitelist` (federation + discovery
endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
(tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
test_acl_mutual_exclusive.py) are deleted per Decision #10;
test_oauth2_contract.py is renamed and rewritten to
test_sso_contract.py, asserting that `flavor: oauth2` roles carry
`oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
(web-app integration) now requires the unified `sso:` block with an
explicit flavor or a `# nocheck: sso` opt-out; new
tests/lint/repository/no_legacy_sso_paths/ guards the source tree
against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
docs/contributing/actions/testing/suppression.md are updated to the
unified SSO naming.
Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).
BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bridge the architectural gap that left bluesky without any usable login
path when services.oauth2.enabled is false. The login-broker stays the
upstream when OIDC is on (broker auto-provisions PDS accounts via the
encrypted-app-password handoff). When OIDC is off, the broker is bypassed
and the stock social-app is served directly; the PDS administrator is
provisioned by Ansible so the operator can sign in with handle+password.
Routing (tasks/01_core.yml):
- Front-proxy http_port now picks `BLUESKY_BROKER_PORT` when
services.oauth2.enabled is true, `BLUESKY_WEB_PORT` (social-app)
otherwise. Previously the broker was always upstream, so variant-1
deploys returned a `Refusing handoff` 401 on every request.
PDS provisioning (tasks/07_native_admin.yml, new):
- uri_retry waits on PDS `/xrpc/_health`, then POSTs createAccount with
`kc-administrator.<handle-domain>` (PDS rejects bare `administrator`
as a reserved handle, hence the `kc-` prefix the broker already uses
for OIDC-derived accounts).
- Idempotent: 200 = changed, 400 with "Handle already taken" = ok.
- Gated on `BLUESKY_WEB_ENABLED and not services.oauth2.enabled`.
Broker hardening (files/login-broker/server.js):
- 401 response (missing `X-Forwarded-User`) now `text/html` instead of
`text/plain`. Headless Chromium does not fire `DOMContentLoaded`
reliably on non-HTML 401 documents — Playwright `page.goto(...,
{ waitUntil: "domcontentloaded" })` hung the full 60s test timeout in
the variant-1 guest-persona test.
Playwright spec (files/playwright/):
- Split single-file playwright.spec.js (203 lines) into _shared.js +
seven test-*.js modules following the matrix/taiga/nextcloud/pixelfed
pattern. Each scenario is now individually grep-able from the
reporter output.
- Add test-native-admin.js: verifies broker bypass (landing body
contains `<title>Bluesky</title>`, NOT `Refusing handoff`) plus PDS
createSession returns 200 + accessJwt for the deployed admin handle.
Uses Playwright's APIRequestContext directly — driving the stock
social-app login UI (hosting-provider switcher + welcome modal +
React form state) was too brittle.
Vars + env (vars/main.yml, templates/playwright.env.j2):
- `BLUESKY_ADMIN_HANDLE` and `BLUESKY_PDS_INTERNAL_URL` added for the
Ansible task.
- `ADMIN_HANDLE` and `PDS_BASE_URL` exposed to Playwright env.
Verification:
- Variant 1 (live, in-container): 3 passed, 4 skipped, 0 failed.
- Variant 0 / Variant 2: spec bodies unchanged from pre-refactor,
Ansible conditional invariant on the OIDC-on branch — but not
re-deployed for live verification.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure 1:1 rename (69 insertions / 69 deletions across 66 files) to
finish the ``docker_*`` → ``compose_*`` alignment started in
8373468e9. The flag configures whether the compose include flushes
handlers right after run; the ``docker_`` prefix was misleading
(handlers are Ansible-side, not docker-side) and the variable is part
of the same ``sys-svc-compose`` SPOT cluster as ``compose_repository_*``.
Touches the role default, the implementation
(``sys-svc-compose/tasks/utils/up.yml``), all 60+ consumer roles,
the network-utility wrapper + its README, and the
``test_network_create_via_util.py`` lint (both the docstring guidance
and the asserted diff).
Breaking change for any external override that still sets the old
name — no alias is provided.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-locates every git-clone URL with its owning entity in the role's
`meta/services.yml` so the unified `image`/`version` ↔ `repository`/`ref`
naming convention covers secondary repos (plugins, build helpers,
bootstrap sources) just as cleanly as the primary entity source.
Migrations:
- sys-aur, pkgmgr (non-invokable bootstrap roles): `meta/services.yml`
gains `repository:`/`ref:` on the role's own entity; `vars/main.yml`
renders them via `lookup('config', role_name, …)` because these
roles never set `application_id`.
- web-app-{akaunting,bigbluebutton,bluesky,discourse,roulette-wheel,
taiga}: declare the secondary git URL on the most fitting entity
(`akaunting` itself for the docker-helper repo; `view` for bluesky's
social-app web client; `plugins.discourse-ldap-auth` for that
Discourse plugin; `oidc` for taiga's OIDC plugin). Each role gains
named `<ROLE>_REPOSITORY` / `<ROLE>_REF` constants in `vars/main.yml`
so tasks and templates stay readable instead of inlining a
`lookup('config', application_id, …)` string at every call site.
- svc-db-postgres `templates/Dockerfile.j2`: keep the `pgvector`
extension clone but mark the line with `{# nocheck: repository-declared #}`
since pgvector is an extension source, not the role's primary repo.
New integration test
`tests/integration/roles/meta/services/test_repository_declared_in_services.py`
walks every role's text files and fails if a literal `https://…/<…>.git`
URL is not declared as a `repository:` value somewhere in that role's
`meta/services.yml` (recursive — top-level entity, sub-entity, plugin
map). Legitimate exceptions opt out with `# nocheck: repository-declared`
on the same line or the preceding line; Jinja templates may use the
`{# … #}` form so a `\`-continued Dockerfile shell command stays valid.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`,
`requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from
non-Markdown source files. Comments anchored to requirement numbers
rot when requirements are renumbered or rewritten and conflate
present-state behaviour (which the comment should describe) with the
audit trail of how the code got there (which belongs in commit
messages, PR descriptions, and the requirement file itself).
* Document the rule in `docs/contributing/documentation.md` under
"Comments". Code MUST describe the *why* in present-state language;
contract enforcement lives in tests, not folklore.
* Clean 254 source files across `roles/`, `tests/`, `plugins/`,
`cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root
`eslint.config.js`. Pure comment / docstring / log-message edits;
no behavioural change.
* Add `tests/lint/repository/test_no_req_references_in_code.py` to
enforce the rule going forward across `.js`, `.py`, `.yml`,
`.yaml`, `.j2`, and `.sh` files. The legacy
`test_no_stale_requirement_refs.py` is exempt as the only file
whose purpose is to detect these very tokens.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh: rename AUTOFORMAT_PARALLEL to PARALLEL,
declare it at file start with default `true`; still env-overridable.
Reword the worker comment so the literal `# shellcheck` token no
longer appears at start-of-line and trips SC1072/SC1073.
* scripts/lint/galaxy.sh: drop the GALAXY_FATAL / GALAXY_NAMESPACE /
GALAXY_PARALLEL env vars (over-engineered for a single repo). Failure
is always fatal. NAMESPACE is an internal constant. Add the same
PARALLEL env var (default `true`, env-overridable); falsy demotes the
xargs fan-out to one worker.
* docs/contributing/environment/variables.md: collapse the four
AUTOFORMAT_*/GALAXY_* rows into one `PARALLEL` row covering both
consumers.
* docs/contributing/tools/make.md: refer to `PARALLEL=0` everywhere;
add the lint-galaxy parallelism note.
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Eliminates the last 102 violations of test_project_root_import by
making PROJECT_ROOT in each package's __init__.py the single source
and inlining every wrapper helper.
Migrations
==========
- ~70 files: ``Path(__file__).resolve().parents[N]`` → ``from <pkg>
import PROJECT_ROOT`` (single-expression shape, scripted).
- ~12 files: two-statement helpers (``T = Path(__file__).resolve();
R = T.parents[N]``) and multi-line ``os.path.abspath(os.path.join(...
os.pardir...))`` chains rewritten by a follow-up scripted pass.
- ~20 files migrated by hand for shapes the scripts couldn't safely
rewrite (helper functions like ``_repo_root_from_test_file``,
embedded ``os.pardir`` chains, etc.).
Helpers removed (inlined to PROJECT_ROOT)
=========================================
- All ``def repo_root_from_here() / _project_root_from_here()`` and
same-shape variants in cli/ (``cli/deploy/development/{down,common,
build,deploy}.py``, ``cli/meta/categories/invokable/command.py``,
``cli/meta/applications/resolution/{combined/repo_paths,
dependencies/__main__,run_after/__main__}.py``).
- All zero-arg ``def <name>() -> Path: return PROJECT_ROOT`` helpers
(utils/invokable.py, utils/roles/resource_validation.py, cli/meta/
runtime/__init__.py, cli/meta/callorder/__main__.py, plus a dozen
test files).
- 4 ``PROJECT_ROOT = PROJECT_ROOT`` self-assignments removed from
test files where the class attribute became dead after migration.
Test-mock updates
=================
``patch.object(repo_paths, 'repo_root_from_here', return_value=ROOT)``
→ ``patch.object(repo_paths, 'PROJECT_ROOT', ROOT)`` (27 patches across
8 unit-test files); same shape for ``patch('utils.invokable._repo_root',
return_value=…)``.
Suppressions documented
=======================
5 files marked ``# nocheck: project-root-import`` for the catalog's
documented exemptions:
- 2 lints walking from a discovered glob match (<role>/templates/...)
up to its role dir, NOT the repo root.
- ``cli/validate/inventory/__main__.py`` invoked as standalone
subprocess (relative imports unavailable).
- ``plugins/filter/value_generator.py`` loaded by Ansible outside the
project package (relative imports broken).
- ``roles/test-e2e-playwright/filter_plugins/discover_playwright_roles.py``.
Bugs surfaced and fixed in the same commit
==========================================
The migration uncovered two latent role-content violations that were
silently masked by buggy ``_repo_root()`` helpers in two integration
tests (the helpers walked upward looking for marker dirs and returned
``tests/integration/`` instead of the repo root, so the tests scanned
nothing under ``roles/`` for years):
- roles/web-app-minio: extracted 6-task OIDC block into 02_oidc.yml +
``include_tasks`` (the documented "Good" pattern from
test_block_size; parsing cost is paid only when OIDC is enabled).
- roles/web-app-bluesky/tasks/01_core.yml: collapsed 3 sequential
``when: BLUESKY_WEB_ENABLED | bool`` tasks into one ``block:`` with
the guard at block level (4 → 2 occurrences, under threshold).
Two semantic-bug fixes carried alongside
========================================
- cli/core/app.py: ``cli_dir`` was previously ``parents[1]`` (= cli/),
not the repo root; rewritten to ``PROJECT_ROOT / "cli"`` to match
the variable's intent.
- cli/make/__main__.py: pre-existing bug — ``repo_root`` was actually
the ``cli/`` directory (so ``cli/Makefile`` was the search target,
which never existed). Now correctly uses PROJECT_ROOT.
- scripts/lint/sort_claude_settings.sh: invokes utils/claude/
settings_sort.py as a module (``python -m``) so its
``from . import PROJECT_ROOT`` resolves.
Verification
============
- ``ruff check .`` → clean
- ``tests.lint.repository.test_project_root_import`` → OK (102 → 0)
- 1621 unit tests → all pass
- Integration suite: 44 failures (baseline, pre-existing, none from
this commit), 1 skipped
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Promotes web-app-bluesky's OIDC integration to a working Keycloak SSO
flow that ends up in the official Bluesky social-app web client
WITHOUT exposing the synthesised PDS app-password to the browser.
* New `login-broker` Node sidecar under
`roles/web-app-bluesky/files/login-broker/`. The broker sits behind
`web-app-oauth2-proxy` and in front of the official
`@bluesky-social/social-app` web client. On a user's first
OIDC-authenticated visit it:
1. Looks up the Keycloak user via the Admin REST API.
2. Provisions a Bluesky PDS account via createAccount if no
encrypted app-password is recorded yet, encrypts the
synthesised app-password with AES-256-GCM, and stores the
ciphertext as the `bluesky_app_password_enc` user attribute.
3. Decrypts the password and exchanges it for a PDS session via
createSession.
4. Renders an HTML handoff page that drops the session JWTs
into localStorage["BSKY_STORAGE"] (canonical social-app key)
and redirects to /.
For subsequent requests carrying the handoff cookie, the broker
reverse-proxies to the social-app upstream.
* `social-app` is built from a pinned upstream tag (1.121.0) so the
broker's BSKY_STORAGE schema stays in sync with the upstream
persistence layer.
* `roles/web-app-bluesky/tasks/05_keycloak_admin_client.yml` adds a
service-account Keycloak client for the broker (deterministic
secret from the credentials vault, view-users + manage-users
realm-management roles). kcadm output is filtered for the JVM
cgroup-detection warnings on cgroup-v2 hosts.
* `roles/web-app-bluesky/tasks/01_core.yml` switches from
sys-svc-compose to sys-stk-backend so the oauth2-proxy sidecar is
auto-included when services.oauth2.enabled is true. A pre-task
heals the oauth2-proxy bind-mount when a previous deploy left it
as an empty directory.
* `meta/schema.yml` adds `bridge_encryption_key` (AES-256-GCM key)
and `kc_admin_client_secret` (broker's KC service-account secret).
* `meta/services.yml` enables oauth2 with origin `bluesky-broker:8080`,
whitelist for /xrpc/ and /.well-known/ so federation traffic to
api.bluesky.<domain> stays open.
* `meta/variants.yml` adds the LDAP variant so the matrix-deploy
exercises both auth-flow legs.
* `templates/extra_locations.conf.j2` uses the new
NETWORK_PUBLIC_DNS_RESOLVERS SPOT and the on_off filter for the
nginx resolver directive in front of the public ip.bsky.app /
bsky.app upstreams.
* Documentation:
- `roles/web-app-bluesky/README.md` rewritten per documentation.md
and readme_md.md (software-focused description, current SSO
flow, no em-dashes, canonical Credits format).
- `roles/web-app-bluesky/files/login-broker/README.md` covers the
handoff sequence, env contract, and failure modes.
- `roles/web-app-bluesky/files/keycloak-bluesky-bridge/README.md`
cleaned up per documentation.md.
- `docs/requirements/013-alpha-to-beta-promotion.md` bluesky
per-role-notes updated to reference the variant A+ broker; new
"Future hardening" section tracks key rotation, an external
secrets store, the Joomla `?fallback=local` IP allowlist, and
the still-source-only Keycloak event-listener SPI.
* `.github/dependabot.yml` adds the npm ecosystem entry that covers
the broker's package.json metadata stub.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace all `docker compose build` invocations with the `compose` wrapper
to ensure automatic env-file detection and override handling. Update
handlers, notify targets, direct commands, and log labels consistently
across roles.
https://chatgpt.com/share/69866b76-ce6c-800f-ac9e-dba6bba27008
Renamed docker-compose and docker-compose-ca roles to sys-svc-compose and sys-svc-compose-ca, updated template includes, handlers, lookups, tests, and variables, and moved compose instances to /opt/compose.
https://chatgpt.com/share/6985ab2c-8344-800f-9e9a-768f562d138d
This change introduces the variable 'web_opt_rdr_domains_logout_enabled' allowing redirect-only domains to include the logout-proxy configuration when required (e.g., BlueSky canonical domain deactivation edge cases).
It ensures the logout endpoint is still properly served even when the domain is normally redirected.
Conversation: https://chatgpt.com/share/6932f3a5-3890-800f-a100-a2d36762ffd4
- Added new 01_core.yml orchestrating all Bluesky role sub-tasks with run-once logic
- Moved PDS and Social App routines to 02_pds.yml and 03_social_app.yml
- Updated DNS task to 04_dns.yml
- Reworked main.yml to delegate execution to 01_core.yml and prevent repeated runs
- Corrected BLUESKY_GEOLOCATION_PATH to new upstream location: src/geolocation/index.tsx
- Improved structure and clarity of the role, aligning with current Infinito.Nexus task layout
https://chatgpt.com/share/69321001-b8cc-800f-9589-2250b8a97fd3
Refactor proxy/webserver configuration variables to a consistent webserver_* naming scheme across roles. Replace legacy variables like proxy_extra_configuration, client_max_body_size, vhost_flavour, location_ws and ws_port with webserver_extra_configuration, webserver_client_max_body_size, webserver_vhost_flavour, webserver_websocket_location and webserver_websocket_port. Update NGINX vhost and location templates (html, upload, ws, basic, ws_generic) as well as callers (sys-front-inj-all, sys-stk-front-proxy, various web-app-* and web-svc-* roles) to use the new naming.
Tighten docker-compose Git repository handling by making docker_git_repository_pull depend on docker_git_repository_address being defined, a string and non-empty. This avoids accidental Git operations when the repository address is unset or of the wrong type.
Refactor the BigBlueButton role structure and fix deployment bugs: introduce 01_core.yml to orchestrate docker/proxy setup, database seeding, websocket map deployment, docker-compose overrides and admin/bootstrap logic in a single once-executed entrypoint. Rename supporting task files (02_docker-compose.yml, 03_administrator.yml, 04_dependencies.yml) and update tasks/main.yml to delegate via include_tasks with run_once_web_app_bigbluebutton. Improve Greenlight admin creation behavior by treating the 'Email has already been taken' error as a non-fatal, unchanged outcome and running user:set_admin_role as a fallback, both for the primary password and the OIDC starred-password path.
Also standardize vhost flavour selection for services like Mailu, Discourse, CDN, Collabora, Coturn, OnlyOffice, Simpleicons and web-svc-logout by explicitly passing webserver_vhost_flavour where needed and aligning client_max_body_size and websocket configuration with the new webserver_* variables.
Reference: ChatGPT conversation https://chatgpt.com/share/6931c530-bba8-800f-9997-dd61dc1d497b
- Un-commented `view.bluesky.{{ PRIMARY_DOMAIN }}` in config to allow
explicit AppView domain definition.
- Reworked `03_dns.yml` to build `cloudflare_records` list programmatically,
including conditional addition of AppView records only if the domain is
not `api.bsky.app`.
- Improved AAAA handling with `| default('')` and proper ternary
expressions for `present/absent`.
- Updated `vars/main.yml` to remove default port fallback for
`BLUESKY_VIEW_PORT`.
Refs: https://chatgpt.com/share/68cdde1d-1bd4-800f-a4bb-319372752fcd
- Added 'solo' flag support for A/AAAA, CNAME/MX/TXT, and SRV records in sys-dns-cloudflare-records.
- Simplified sys-svc-dns: removed NS management tasks and CLOUDFLARE_NAMESERVERS default.
- Renamed 03_apex.yml back to 02_apex.yml, adjusted AAAA task name.
- Updated web-app-bluesky DNS tasks: marked critical records with 'solo'.
- Updated web-app-mailu DNS tasks: removed cleanup block, enforced 'solo' on all records.
- Adjusted constructor stage to call domain_mappings with AUTO_BUILD_ALIASES parameter.
Conversation: https://chatgpt.com/share/68cd20d8-9ba8-800f-b070-f7294f072c40
- Switch certbot flag from MODE_TEST → MODE_DUMMY in dedicated certs
- Add sys-svc-dns defaults for CLOUDFLARE_NAMESERVERS
- Introduce 02_nameservers.yml for NS cleanup + enforce, adjust task ordering (apex now 03_apex.yml)
- Enforce quoting for Bluesky and Mailu TXT records
- Add cleanup of MX/TXT/DMARC/DKIM in Mailu role
- Normalize no_log handling in Nextcloud plugin
- Simplify async conditionals in Collabora role
Conversation: https://chatgpt.com/share/68cd20d8-9ba8-800f-b070-f7294f072c40
Replaced hardcoded path to src/state/geolocation.tsx with variable BLUESKY_GEOLOCATION_PATH pointing to src/state/geolocation/index.tsx.
This ensures BAPP_CONFIG_URL and IPCC_URL replacements work with the updated Bluesky code structure.
Ref: https://chatgpt.com/share/68cb16d5-d698-800f-97e5-cc7d9016f27c
- Add `server.config_upstream_url` default in `roles/web-app-bluesky/config/main.yml`
to define upstream for /config (defaults to https://ip.bsky.app/config).
- Introduce front-proxy injection `extra_locations.conf.j2` that:
- proxies `/config` to the upstream,
- sets SNI and correct Host header,
- normalizes CORS headers for same-origin consumption.
- Wire the proxy injection only for the Web domain in
`roles/web-app-bluesky/tasks/main.yml` via `proxy_extra_configuration`.
- Force fresh social-app checkout and patch
`src/state/geolocation.tsx` to `const BAPP_CONFIG_URL = '/config'`
in `roles/web-app-bluesky/tasks/02_social_app.yml`; notify `docker compose build` and `up`.
- Tidy and re-group PDS env in `roles/web-app-bluesky/templates/env.j2` (no functional change).
- Add vars in `roles/web-app-bluesky/vars/main.yml`:
- `BLUESKY_FRONT_PROXY_CONTENT` (renders the extra locations),
- `BLUESKY_CONFIG_UPSTREAM_URL` (reads `server.config_upstream_url`).
Security/Scope:
- Only affects the Bluesky web frontend (same-origin `/config`); PDS/API and AppView remain unchanged.
Refs:
- Conversation: https://chatgpt.com/share/68b8dd3a-2100-800f-959e-1495f6320aab
- Renamed role `srv-tls-core` → `sys-svc-certs`
- Renamed role `srv-https-stack` → `sys-stk-front-pure`
- Renamed role `sys-stk-front` → `sys-stk-front-proxy`
- Updated all includes, READMEs, meta, and dependent roles accordingly
This improves clarity and consistency of naming conventions for certificate management and proxy orchestration.
See: https://chatgpt.com/share/68b19f2c-22b0-800f-ba9b-3f2c8fd427b0
