At BBB boot, bbb-graphql-server's start.sh runs pg_terminate_backend and
drops bbb_graphql. When that kills apps-akka's first HikariCP connection
while the pool detects the driver's default transaction isolation,
HikariCP 4.0.3 swallows the 08xxx error, marks detection as done and
keeps isolation -1 for the pool's lifetime. Every later connection setup
then calls setTransactionIsolation(-1), which pgjdbc rejects with
"Transaction isolation level -1 not supported", so meetings never reach
the database and the client shows "Oops" (about 1 in 8 boots).
Pin the isolation explicitly so HikariCP never falls back to the
detected value: slick-hikaricp reads postgres.transactionIsolation,
prepends TRANSACTION_ and hands it to HikariCP, hence the value must be
exactly READ_COMMITTED (the PostgreSQL default, so normal boots behave
as before). The flag goes into apps-akka's own compose environment as
JAVA_OPTS, which the sbt-native-packager start script honours, instead
of env.j2, because compose_mods pins env.j2 onto every bbb service and
would hand the flag to bbb-web and fsesl-akka as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The version-source update in #1146 moved the bigbluebutton/docker ref from
v3.0.23 to v3.0.39. That tag changes three things the role did not
handle, and it exposed a fourth defect in compose_file_args.
1. generate-compose renders etherpad and bbb-pads only behind
USE_INSECURE_ETHERPAD and otherwise emits bbb-shared-notes-server with
a `build:` block and no `image:` (upstream adds the image only after
the tag, in 9a6a9a9a4c). compose_mods drops `build` only where an image
is set, so the guard "Fail if any service still builds locally" stopped
every BigBlueButton deploy (main run 39749899667, Nextcloud#11:
`rc: 0`, `stdout: ' build:'`, bootstrap_compose_project.yml:35).
meta/services.yml now pins alangecker/bbb-shared-notes-server:v3.0.39,
compose_mods takes `pinned` (BBB_PINNED_SERVICES, every service the
role config pins an image for) and drops their `build`, and the
etherpad leftovers go (ep_redis_publisher patch, etherpad postgres
entrypoint and initdb hook, etherpad database tasks, ETHERPAD_* env,
etherpad_api_key secret).
2. The notes server connects to postgres once at start and exits with 2
when it is not up yet; upstream gives it no restart policy. The
override sets the role's restart policy, so it retries until postgres
accepts connections.
3. Greenlight moves from v3.8.0 to v3.9.0.1, which sets
`config.hosts = URL_HOST`, and the template passes
`URL_HOST: https://${DOMAIN}`. Rails compares hostnames, so every
request was blocked (`Blocked hosts: bbb.talk.infinito.test`, 403).
Greenlight documents URL_HOST as the bare hostname; the override sets
it to the role domain.
4. compose_file_args decided whether a role ships a compose override by
templating "{{ application_id | abs_role_path_by_application_id }}"
through the templar. ansible-core 2.21 does not render an untrusted
string, so the check saw the literal and always returned False: the CA
override generation and the pull handler never read BBB's
compose.override.yml. That was harmless while compose.yml alone was
valid; with the notes-server image living in the override, `compose
config` failed ("has neither an image nor a build context"). The check
now resolves the role path for the application id it is given. BBB is
the only role that ships an override, so no other role's file list
changes.
Existing installations drop the etherpad and bbb-pads containers on the
next `compose up --remove-orphans`; the old `etherpad` database stays in
PostgreSQL untouched.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The welcome message linked to https://www.bigbluebutton.org/html5 for
"tutorial videos". That page is gone: it answers HTTP 404 with and
without www and with a trailing slash, while bigbluebutton.org itself
answers 200. tests/external/repository/test_urls_reachable.py reports
any other 4xx as a failure, so test-external went red for every branch.
Link the BigBlueButton documentation instead, which the welcome footer
already references, and name it as such.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The list form could publish a host:container pair and nothing else, so thirteen blocks stayed hand-written: the ones needing a protocol suffix, a port range, the swarm long syntax with mode host, or a container-port-only publish. Each of those duplicated numbers the role already declares in meta/services.yml, which is the drift the lookup exists to remove.
The lookup keeps the list form unchanged for its 93 existing call sites and gains a dict form: transport emits one item per protocol, container 'same' serves the services that cannot remap (STUN/TURN, media relays), mode 'host' emits the long block an edge role needs in swarm so the client address survives the hop, publish 'ephemeral' emits the container port alone, and key names the block for Discourse's launcher file, whose expose: list is what becomes docker run -p there. A declared {start, end} mapping renders as start-end on both sides, so a relay range needs no term of its own.
svc-prx-openresty had no ports block at all and bound 80 and 443 as literals in its template, while torrc.j2 carried a third literal 80 for the hidden service. None of the three referenced the others. openresty now declares the ports, the compose template reads them, and the onion's port 80 comes from ports.onion.http through TOR_ONION_EXTRA_PORTS.
Three blocks stay raw, each with the reason it cannot route through one call: keycloak's sso_proxy maps a published port owned by the consuming role to a container port owned by web-app-keycloak, and matrix's ansible flavor takes element on 8080 and federation on 8048 from the upstream playbook rather than from this repo's ports.internal.
The enforcing lint scanned only files with compose in the name and only double-quoted list items, so nine of the thirteen blocks were invisible to it. It now reads every role template that can carry a ports block, catches unquoted items, protocol suffixes, ranges, flow style and the long syntax, and tracks the block by the indentation of its key: keying on the next line closed the block on anything shaped like a key, which target, published, protocol and mode all are. Under expose: only a real mapping counts, because there the single-segment form documents a container port and publishes nothing.
Verified by rendering all 110 container_ports call sites through the real templar, the real lookup and the real merged applications view.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
42ebd9ea71 gave the healthcheck SPOT start_interval, the cadence docker uses
while start_period is still running. A block a template writes out itself
bypasses the SPOT and never gained it. Measured across the repository: 46 such
blocks, 40 of them already carrying start_period, not one carrying
start_interval.
Without it a probe fires only every interval, so a service ready after two
seconds is reported healthy at the next tick and a stack converges no faster
than its coarsest quantisation. That lands in sys-svc-compose's swarm converge
wait, 1551.7s per CI job over 38 stack deploys.
Twenty five of the blocks move into meta/services.yml and render through
container_healthcheck. Each keeps an explicit test rather than picking a flavor:
a flavor would replace the command instead of carrying it across.
The move is lossless by construction. probes.py block() writes
overrides.get(key, default) for interval, timeout, retries, start_period and
start_interval, so a declaration repeating the literal values renders the same
block plus the missing cadence. Checked per block against the old values: the
only difference is start_interval 5s, nothing changed, nothing lost.
Twenty one stay literal. Seventeen interpolate a template variable, whose value
resolves in a scope meta/services.yml does not share, so moving them would be a
rewrite rather than a transcription. Four interpolate nothing and still cannot
move: svc-db-memcached and svc-db-redis probe every second in their swarm
service partial, where a 5s start cadence would report healthy later than the
block already does, and the sys-svc-rdbms and keycloak sso_proxy partials render
for the consuming role's application_id, so their declaration has no single
meta/services.yml to live in. Each carries a nocheck naming its reason.
All twenty one gain start_interval. Three also gain start_period, without which
the key does nothing at all, since it only applies while the start period runs.
That widens their failure tolerance rather than narrowing it: bluesky from 90 to
120 seconds, magento from 50 to 80, the mariadb partial from 180 to 210.
pixelfed spelled its probe as a bare string and now spells it CMD-SHELL. Docker
reads both as a shell command; only the YAML shape changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mount pipeline was split across two layers: compose_volumes and
container_volumes lived as filter modules that ansible's filter loader
rejected on every start ("has no attribute 'FilterModule'", two warnings
per run), while the same-named lookups wrapped them thinly. The filter
bodies now live inside their lookups (the tor_socks precedent), the
filter files are gone, and the render tests moved next to the lookup
tests.
As lookups they see the templating context, so the application_id term
every call site passed back to its own role is gone too: both lookups
read application_id from context and fail loudly when it is missing.
107 templates and nfs_prep.yml drop the term; the calls collapse to
lookup('compose_volumes') and lookup('container_volumes', service_name).
The container_volumes call is now canonical - four leading spaces,
service_name in the service position, | indent(4) - enforced by a new
lint (test_container_volumes_call_shape). Normalizing the 85 literal
call sites surfaced that erpnext keyed its meta/volumes.yml mounts by
container name instead of compose service key, the only role to do so;
its meta now uses the service keys. wordpress's column-0 call, which
rendered its first line at top level, is indented like every other.
Template logic that only restated meta/volumes.yml is deleted: mailu fed
compose_volumes an extra_volumes dict whose eight names were looked up
from the very meta file compose_volumes reads itself; zammad's init and
railsserver extra mounts were the volumes.yml entries minus the two
services, which the meta now lists (the when-gated OIDC binds included).
Conditionality that meta cannot express stays in code, but in role-local
lookup plugins instead of template loops: matrix_bridge_mounts renders
the per-bridge registration mounts, nextcloud_extra_volumes the
feature-gated whiteboard/recording volumes (type: volume entries are
declared unconditionally by compose_volumes, so gating must happen
before it).
Parity is proven, not assumed: every mechanically rewritten template is
byte-identical to HEAD after the transformation regexes (95/95), and the
semantic roles render identically through the real functions against
HEAD's meta - zammad across six sso/tls polarities, all seven erpnext
services, mailu's top level. make test-lint passes complete.
Also: suitecrm's compose template pins container_extra_hosts with
host_alias (the SOCKS proxy URL host.docker.internal never resolved in
compose+tor), and the five disabled mautrix bridge addons use the
empty-default config lookup so templating the applications dict stops
emitting 125 errors per batch in every CI job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/volumes.yml says where a volume lands inside its container. Wherever a
second file spelled the same path out, the two drifted independently: the mount
moves, the consumer keeps pointing at the old place, and nothing fails until the
process looks for content that is not there. e031b7f61 fixed exactly this for
web-app-moodle; this generalises it and makes the rule hold.
The lookup gains what a caller needs. `targets` maps each mounting service to
its container path, and `target` is the one path an entry mounts at -- present
only when every mount agrees. An entry that lands at different paths per service
omits `target` rather than picking one, and the caller names the service:
"{{ lookup('volume', application_id, 'prometheus_config').target }}"
"{{ lookup('volume', 'web-app-prometheus', 'host_sys').targets['node-exporter'] }}"
66 literals across 48 role files now derive. Replacement only happened where the
path stood alone -- /srv/app inside /srv/app/data is a different path -- and
every rewritten file was re-parsed, Jinja and YAML alike. Three sites the tool
refused rather than guess were done by hand: zammad's path sat in a Jinja list
literal, seaweedfs' inside a quoted string feeding a filter, and openresty's
/var/run/docker.sock is not a mount target at all but the socket on the host,
which only happens to spell the same string. That one carries the single
nocheck.
tests/lint/repository/spot/test_volumes_path_single_spot.py keeps it that way.
roles/*/files/ is out of scope by construction: it is copied verbatim into an
image and executed there, so a Dockerfile or entrypoint has no Ansible to
evaluate a lookup with.
One defect this found, and it was in the tooling rather than the roles: deciding
ambiguity AFTER filtering short paths made web-app-prometheus' host_sys look
single-valued -- it mounts /sys for cadvisor and /host/sys for node-exporter,
and /sys is below the length filter. The template would have rendered an
undefined `.target`. Both the converter and the rule now decide ambiguity over
every target and filter afterwards, and that call reads targets['node-exporter'].
Verified by resolving rather than by reading: all 70 accesses in the diff were
run against the real registry and every one yields a non-empty path equal to the
literal it replaced. make test-lint: 328 passed.
Not verified: no Ansible render, so a call site whose surrounding context cannot
build the applications registry would still fail at deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Quote ETHERPAD_DB_PASSWORD with dotenv_quote so the generated env file is
mode-safe (the passwords lint requires it; compose strips the outer quotes,
swarm passes the value verbatim). Switch the etherpad entrypoint wrapper
mount from a single-file bind to a docker config, which swarm can place on
any node; the meta-volume lint rejects single-file binds as swarm-fragile.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Etherpad's ueberdb backend was pinned to the bbb redis service. During a
deploy that redis churns (RestartCount=2), taking etherpad's DB with it, so
etherpad crashes (RestartCount=1) and the health-container job fails. bbb's
own postgres stays stable (RestartCount=0) through the same deploy.
A boot wrapper rewrites settings.json dbType redis to postgres before the
upstream entrypoint runs; env carries the connection values; the etherpad
database is provisioned in the postgres password-sync task; compose adds a
postgres dependency. The ep_redis_publisher pub/sub client stays on redis
(its failures are non-fatal).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The p88f3f6b etherpad image ships ep_redis_publisher@0.0.3 whose retry() calls logger.error() with no logger in scope, so any redis reconnect throws ReferenceError and kills etherpad, which fails the shared hlth-container gate (observed on bbb-0 swarm and, collaterally, the nextcloud bundle that pulls the same image). Vendor the plugin index.js verbatim from the image plus a console-backed logger shim, and bind it read-only over the plugin file via meta/volumes.yml + stack_host_copy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New roles/sys-svc-container/templates/env.j2 sets TZ={{ HOST_TIMEZONE }} and LANG=C.UTF-8. Every role env.j2 includes it at the top; it renders via stack_host_template, the same engine that resolves the includes in compose.yml.j2, so it works in both compose and swarm. TZ/LANG are removed from the individual role env.j2 so each value stays single-source.
New lint tests/lint/ansible/test_env_base_include.py verifies every role env.j2 includes the base and never re-sets a base-owned key (keys derived from the base file).
mattermost and gitea meta/volumes.yml drop the /etc/timezone host-path bind: swarm hard-rejects a task whose bind source is missing on the node (Debian-trixie nodes have no /etc/timezone). TZ now comes from the base env. /etc/localtime bind also dropped; the container timezone follows TZ.
Non-directive comments were removed from the touched env.j2 (banners, restatements, reference URLs, narration) so the changed files pass the comment-lint; only tool directives remain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Postgres-dependent Hasura was still inside its 10s start_period
when the terminal hlth-container gate fired on a cold DinD node, so the
whole deploy failed on a container that was merely booting. 10m matches
the kix backend budget; start_period only defers the unhealthy verdict,
a passing probe still flips healthy immediately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The override declared prometheus-exporter unconditionally with only volumes, while the upstream-generated base compose emits the service (with its image) only when ENABLE_PROMETHEUS_EXPORTER is set. Deploying without web-app-prometheus in the group produced a merged project with a service that has neither image nor build, which compose config rejects. The guard mirrors the env.j2 gate, keeping override and base in lockstep like the greenlight and recording blocks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The override renders with lstrip_blocks=False (only compose.yml.j2 gets lstrip via the stack_host_template plugin), so an indented set line left dangling whitespace under trim_blocks, collapsing each service key to the same indent as its body, which fails with 'services.image/volumes must be a mapping'. Move the set statements to column 0 so they render correctly under both lstrip settings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The freeswitch build fetched libopusenc from launchpad.net, whose
librarian backend rate-limits CI (429s across 11 attempts); three more
build-time fetches carried the same risk. The maintainer publishes an
alangecker/* image for every buildable service, so compose_mods drops
build: wherever the service carries an image and compose pulls instead;
a guard task fails loudly if a future upstream service still builds.
Pin ref to the v3.0.23 release branch (the packaging repo has no tags)
and clone it without submodules via the new compose_repository_recursive
toggle: submodules only feed build contexts and DEV_MODE mounts, and
generate-compose falls back to the committed repos/tags file.
Declare the pulled images in meta/services.yml (updater-managed SPOT,
mirror pre-pull) and the named data volumes in meta/volumes.yml; the
override renders both through the standard container_image /
container_volumes / compose_volumes lookups with the service_name
pattern. Compose merges volumes by target, so the override's named
volumes verifiably replace the clone-local bind mounts (docker compose
config proven). This also heals a split-brain mount: apps-akka,
periodic and prometheus-exporter previously kept bind mounts while
their peers used the named volumes.
Teach container_image and compose_volumes to honour compose_mode_force
(the SPOT the other mode-aware lookups already read): bbb deploys as a
node-local compose stack in swarm, so it must get hub image refs and
plain local volumes there, not registry-prefixed refs and NFS
driver_opts. The unpinned-engine-volume lint learns the same two safe
conditions. Read the embedded Collabora port from web-svc-collabora's
services SPOT instead of hardcoding 9980.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Enable ENABLE_PROMETHEUS_EXPORTER and _OPTIMIZATION via a {% if services.prometheus.enabled %} guard so the bbb-exporter only runs when web-app-prometheus is present, instead of staying hard-commented. Add a head-level '# nocheck: comments-valid' opt-out so the vendored upstream sample.env comments stay valid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The compose/swarm split on `depends_on:` shape (map form with
`condition:` vs. list form, per `docker stack deploy` constraint)
was being hand-coded as a `{% if DEPLOYMENT_MODE == 'swarm' %}` gate
around every block. Repetitive, easy to forget, and templates that
did remember it had to repeat the same six-line pattern per service.
The new `depends_on` lookup encapsulates the shape switch:
{{ lookup('depends_on', {SVC: 'service_healthy', …}) }} # mapping
{{ lookup('depends_on', [SVC_A, SVC_B]) }} # all default
- Compose -> map form with explicit `condition:` per entry.
- Swarm -> list form (just names; swarm ignores depends_on
ordering semantically but accepts the syntax).
- Mapping `None` value uses the default condition (`service_started`,
matching Docker Compose's behaviour when no condition is given).
- Plain list of names uses the default for every entry.
The output is shaped so the call site reads naturally at the
service-member column:
some_service:
image: …
{{ lookup('depends_on', {DB: 'service_healthy'}) }} # at col 4
volumes: …
Concretely the lookup emits line 1 WITHOUT leading whitespace and
lines 2+ with the `indent` kwarg (default 4) plus the YAML-relative
nested indent. Jinja substitution applies the template's leading
spaces to line 1 only, so the column-4 call placement makes
`depends_on:` land at column 4 and the children at 6 / 8. Lines 2+
carry their own indent because Jinja does not propagate template-side
leading whitespace to continuation lines.
Plugin + tests:
- plugins/lookup/depends_on.py: ~190 lines. Mapping / list / empty
input handling, default_condition kwarg, condition validation
against the closed set `{service_started, service_healthy,
service_completed_successfully}`, DEPLOYMENT_MODE resolution via
templar with `mode=` test-only override, indent kwarg semantics
documented in the module docstring.
- tests/unit/plugins/lookup/test_depends_on.py: 27 cases covering
TestInputNormalisation (mapping / list / empty / typo / single
string rejection / non-string sequence rejection / unsupported
type rejection / arity errors), TestModeBehaviour (compose / swarm
/ unknown-falls-to-compose / missing-DEPLOYMENT_MODE-falls-to-compose
/ mode-kwarg override), TestOutputFormatting (line-1 unindented
exact-string assertion, indent=0 edge case, indent=6 edge case,
insertion-order stability for diff-stable output).
Lint:
- tests/lint/ansible/jinja/test_compose_template_uses_depends_on_lookup.py:
rule key `compose-depends-on-must-use-lookup`. Flags three patterns
in any `*compose*.yml.j2` under `roles/`:
1. literal `depends_on:` keyword line
2. flow-form `depends_on: [a, b]`
3. literal `condition: service_*` line
The pre-existing `lookup('container_depends_on', application_id)`
is intentionally not flagged: its call form contains no literal
`depends_on:` token in source. Scan target broadened to
`name.endswith('.yml.j2') and 'compose' in name` so
`compose.override.yml.j2` and `compose-inits.yml.j2` are covered
alongside `compose.yml.j2` and `*.compose.yml.j2`.
Migration (43 depends_on blocks across 17 templates):
- erpnext (6): configurator + backend + websocket + scheduler +
queue-short + queue-long + frontend chains.
- mailu (8): imap + smtp + oletools + antispam + antivirus + webdav +
fetchmail + webmail.
- zammad (4): rails + 3 worker shapes.
- shopware (4): web + worker + scheduler + opensearch, init-gated.
- jitsi (3): three single-key dependencies on JITSI_PROSODY_CONTAINER.
- kix (3): backend + frontend + proxy.
- bigbluebutton/compose.override.yml.j2 (2): bbb-web + etherpad,
inside the existing BBB_COLLABORA_ENABLED gate.
- moodle (2): nginx + cron, both depending on moodle.
- opentalk (2): application + recorder.
- taiga (2): events + gateway.
- matrix (1): loop-driven via `{% set deps = {} %} +
__setitem__` escape hatch over MATRIX_BRIDGES.
- matomo (1): bootstrap service.
- nextcloud (1): whiteboard sidecar.
- openproject (1): proxy.
- funkwhale (1): front.
- magento (1): nginx.
- flowise (1): postgres + redis.
Pre-existing `lookup('container_depends_on', application_id)` calls
(erpnext, nextcloud, moodle, zammad, taiga, matrix, etc.) are left
untouched -- that lookup auto-detects DB / redis from the application
config and emits its own block; it is the app-aware counterpart to
this structural lookup.
Every call placement repositioned to column 4 (standard
compose-service-member indent) so the new line-1-unindented semantic
renders correctly. Eight templates needed the column-4 reposition
(via a mechanical one-shot script that was deleted after).
`make test` green; new lint + 27 unit tests green; independent
re-grep across all `*compose*.yml.j2` finds zero residual raw
`depends_on:` / `condition:` lines.
Across 23 web-app env.j2 templates, drop the surrounding "..." from
plain KEY="value" entries. compose silently strips them at env_file
parse time, but docker stack deploy preserves them literally, so the
container then sees KEY="value" with the quotes embedded and parsers
that call int(os.environ['KEY']) or split URLs/DSNs break under swarm.
The change is mechanical: every removed line was `KEY="value"` and
every added line is the same `KEY=value` without surrounding quotes.
Lines that use `| dotenv_quote` are untouched because that filter
deliberately produces escaped values for env_file consumers; the
existing password-quote integration test continues to require it for
sensitive variables.
Affected: bigbluebutton, bluesky, bookwyrm, bridgy-fed, chess,
confluence, decidim, espocrm, friendica, funkwhale, gitea, jira,
magento, mailu, mobilizon, openproject, phpmyadmin, pixelfed, pretix,
shopware, snipe-it, suitecrm, xwiki.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements docs/requirements/021-sso-flavor-migration.md atomically:
* Schema: every role's meta/services.yml replaces the parallel
`oauth2:` / `oidc:` top-level blocks with a single `sso:` block
carrying `enabled`, `shared`, and a `flavor` discriminator
(`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
entirety; its 5 sidecar templates move under
roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
`is_oidc_native`) and the oauth2 sub-values
(`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
Three lookup plugins consume the resolver: `sso` (single-app
predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
and plugins/filter/compose_volumes.py all gate on
`lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
`enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
→ `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
`OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
`oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
`OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
`SSO_SERVICE_ENABLED`; Playwright gating helpers
(isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
`flavor: oauth2` with their existing `oauth2.acl.blacklist`
(/user/login) and `oauth2.acl.whitelist` (federation + discovery
endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
(tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
test_acl_mutual_exclusive.py) are deleted per Decision #10;
test_oauth2_contract.py is renamed and rewritten to
test_sso_contract.py, asserting that `flavor: oauth2` roles carry
`oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
(web-app integration) now requires the unified `sso:` block with an
explicit flavor or a `# nocheck: sso` opt-out; new
tests/lint/repository/no_legacy_sso_paths/ guards the source tree
against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
docs/contributing/actions/testing/suppression.md are updated to the
unified SSO naming.
Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).
BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The env.development marker file existed for two purposes that no longer
justify the wiring cost: a per-machine compose --env-file override layer
appended after auto-loaded .env, and a fallback signal for
detect_runtime() to return "dev" when the marker exists. With every
INFINITO_* default now collapsed into env/default.env -> .env,
persistent per-machine overrides are best expressed as
\`export INFINITO_FOO=…\` before invoking make (load.sh honors
caller-set values via setdefault). The marker fallback in
detect_runtime was redundant with the explicit RUNTIME=dev that every
dev-flow callsite already passes (fresh-*.sh, reset/inventory.sh).
Removed:
- Makefile mark-development target + .PHONY entry
- env.development from .gitignore / .dockerignore
- env_local conditional in compose.up()
- env_development conditional in down._compose_run()
- env.development conditional in scripts/meta/resolve/apps.sh and
scripts/meta/resolve/diff/affected_roles.sh
- env.development marker branch + the now-unused project_root parameter
from cli/meta/runtime/detect_runtime; __main__.py caller follows
- "Optional Development Marker" section in docs/contributing/environment/
setup.md and the marker reference in docs/administration/installation.md
Also drops two RUNTIME=='dev' guards in roles/web-app-bigbluebutton that
relied on the auto-detect path:
- the compose build etherpad task in tasks/02_compose.yml
- the freeswitch iptables-REJECT entrypoint override in
templates/compose.override.yml.j2
With auto-detection of "dev" gone the guards would otherwise only fire
on explicit RUNTIME=dev runs, silently regressing the local-dev path
operators rely on; both behaviors now apply unconditionally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/install/venv.sh
-----------------------
INFINITO_VENV_BASE is a static-default convention; the active VENV
may end up outside that base (e.g. \$HOME/.venvs/<name> when
/opt/venvs is not writable, or a wholly custom override). Probing
writability on the legacy base triggered a spurious sudo branch
whenever VENV lived elsewhere -- fatal in no_new_privs sandboxes
where sudo cannot escalate. Switch the writability + mkdir + chown
chain to \$(dirname "\${VENV}") so the script prepares the directory
it is actually about to create the venv in.
tests/lint/ansible/roles/web-app/playwright/test_app_base_url_uses_role.py
-------------------------------------------------------------------------
New lint: every role's templates/playwright.env.j2 must declare
APP_BASE_URL as a reference to the role's own surface. APP_BASE_URL
feeds the shared guest-persona helper and Playwright's global
baseURL; a role that hardcodes a sibling role name (most commonly
'web-app-dashboard') sends the guest test to the wrong site and
breaks any matrix variant where the sibling is not deployed
(ERR_SSL_UNRECOGNIZED_NAME_ALERT).
The 16 web-app role templates that violated the contract switch
from sibling lookups (`lookup('tls', 'web-app-dashboard', ...)`)
to the canonical `lookup('tls', application_id, ...)` form.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename env/static.env → env/default.env across the entire repo
(string label, doc links, Python file-path constants) — the
`parse_static_env*` function names stay as a stable API surface.
Sort all `*.env` / `*.env.j2` entries alphabetically (52 in
default.env, 4 in ci.env, 46 of 49 j2 templates; 3 templates
with Jinja2 `{% %}` control flow are skipped because conditional
blocks would tear apart under a naive sort). Sort all 83
Makefile target blocks alphabetically; .PHONY group headers
stay in place.
Collapse the two remaining multi-line Makefile recipe comments
(dotenv-force, test-signed) to single-line WHY comments.
Add four lint tests under tests/lint/filesystem/ to keep the
invariants enforced going forward:
- env/test_alphabetical_sort.py
- env/test_single_line_comments.py (# nocheck: markers exempt)
- makefile/test_alphabetical_targets.py
- makefile/test_single_line_comments.py (# nocheck: markers exempt)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the verbose
`(lookup('applications', application_id).server.domains.canonical | first)`
chain with the dedicated `lookup('domain', application_id)` plugin
across every playwright.env.j2 that renders `CANONICAL_DOMAIN`. The
helper already exists at plugins/lookup/domain.py and resolves the
canonical primary domain via the cached domain-merge pipeline, so the
inline canonical-list traversal is unnecessary noise that obscures
the actual contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`,
`requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from
non-Markdown source files. Comments anchored to requirement numbers
rot when requirements are renumbered or rewritten and conflate
present-state behaviour (which the comment should describe) with the
audit trail of how the code got there (which belongs in commit
messages, PR descriptions, and the requirement file itself).
* Document the rule in `docs/contributing/documentation.md` under
"Comments". Code MUST describe the *why* in present-state language;
contract enforcement lives in tests, not folklore.
* Clean 254 source files across `roles/`, `tests/`, `plugins/`,
`cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root
`eslint.config.js`. Pure comment / docstring / log-message edits;
no behavioural change.
* Add `tests/lint/repository/test_no_req_references_in_code.py` to
enforce the rule going forward across `.js`, `.py`, `.yml`,
`.yaml`, `.j2`, and `.sh` files. The legacy
`test_no_stale_requirement_refs.py` is exempt as the only file
whose purpose is to detect these very tokens.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cross-service surface assertions for dashboard, prometheus and matomo
no longer run in every consumer role's persona. They are owned by the
provider's own spec, parameterised once over a `<NAME>_TARGET_ROLES_JSON`
manifest rendered into the provider's `templates/playwright.env.j2` via
the new `lookup('roles_with_service', '<svc>')` Ansible lookup.
Changes
-------
- New `plugins/lookup/roles_with_service.py` enumerates roles that
declare `services.<svc>.{enabled, shared}` truthy in the merged
applications config, returning `[{id, canonical_domain}, …]`.
- `roles/web-app-{dashboard,prometheus,matomo}/files/playwright.spec.js`
parse `<NAME>_TARGET_ROLES_JSON` and parameterise the per-consumer
assertion (dashboard tile reachability, prometheus `up=1`, matomo
SitesManager registration). The matomo spec gains an explicit
biber-denial test (request body validated against admin DOM markers
per the body-marker rule); the prometheus admin-reach test drops
the dashboard-iframe path in favour of direct goto.
- Persona helpers `roles/test-e2e-playwright/files/personas/{admin,
biber,guest}.js` enter at `${APP_BASE_URL}/` directly. The
`clickRoleTileFromDashboard` step and the `assertPrometheus*` /
`assertMatomo*` cross-service probes are gone; `utils/{dashboard,
prometheus,matomo}.js` are deleted. `utils/csp.js` switches to
literal `process.env.<KEY>` reads so the env-keys-used parity guard
recognises matomo's CSP injector consumption.
- 42 consumer-role `templates/playwright.env.j2` templates drop
`DASHBOARD_BASE_URL`, `DASHBOARD_SERVICE_ENABLED`, `PROMETHEUS_BASE_URL`,
`PROMETHEUS_SERVICE_ENABLED`. `MATOMO_BASE_URL` and
`MATOMO_SERVICE_ENABLED` stay because the per-role CSP injection
helper still gates on matomo's tracker host.
- `tests/integration/roles/test_playwright_env_services_match.py`
globally exempts `dashboard` and `prometheus` from the
services.yml-to-env parity rule.
- `tests/unit/plugins/lookup/test_roles_with_service.py` adds 14
unit tests for the lookup, including an explicit-contract test
that pins the over-inclusive behaviour on raw Jinja strings (the
lookup expects a templar-rendered payload).
- Docs: `docs/requirements/019-playwright-meta-services-parity.md`
rewords Rules 1, 3, 12 and adds Rule 13 describing the SPOT shift.
`docs/contributing/artefact/files/role/playwright.specs.js.md`
reflects the new persona shape and per-service catalogue, and is
cleaned per `docs/contributing/documentation.md` (no SPOT self-
labels, no req-references, em dashes replaced with complete
sentences, multi-clause lines split onto separate lines, broken
anchor `#two-personas-fixed-flow-` corrected).
Verification
------------
- 418 passed / 1 skipped / 0 failed across `tests/lint/`,
`tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors.
- `markdownlint-cli2`: 0 errors on all touched docs.
- `make autoformat`: clean (FURB171 single-item-set inequality fixed).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a lint that fails when a role's templates/playwright.env.j2 declares
DASHBOARD_SERVICE_ENABLED= but does NOT render DASHBOARD_BASE_URL,
PROMETHEUS_BASE_URL, and MATOMO_BASE_URL. Renders those three URLs
across 42 role env templates so the lint passes.
Also reverts the matomo "role-closed" strikethrough in req 019 — the
URL wiring remains in scope for the persona rollout.
Reason: the persona helpers (roles/test-e2e-playwright/files/personas/
{biber,admin,guest}.js) collapse the persona scenario when their
required URLs are empty (req 019 persona-collapse exception). Without
this lint, a role that declares dashboard: enabled but forgets the
URL silently SKIPs the persona — the deploy goes green but no real
flow ran. Per-line opt-out is `# nocheck: persona-required-envs` for
auth-less svc-* roles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires runGuestFlow / runBiberFlow / runAdminFlow into bigbluebutton and
fediwall, plus matching meta/services.yml gate entries and
playwright.env.j2 env wiring. fediwall's adminInteraction drives a real
management surface so the admin-flow body is not a stub; bigbluebutton
imports isServiceEnabled for follow-on work.
Reason: continuation of d3e5ebe44 / 44d51744a / 735ae5e29 — finishing
the long-tail role list against the central persona library so the
new lint cannot find a stub-shaped body anywhere.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Moves the SYSTEM_EMAIL_* resolution logic out of group_vars/all/03_email.yml
into a dedicated Ansible lookup plugin, so every role template reads outgoing
mail settings via lookup('email') (global) or lookup('email', application_id)
(per-app). Per-app compose.services.email is merged on top of the resolved
defaults; SYSTEM_EMAIL_* variables remain usable as an inventory escape hatch.
Also splits the email docs: administrator-facing configuration stays under
docs/administration/configuration/, plugin contract and extension rules move
to docs/contributing/design/services/email.md. services.md is relocated to
services/base.md so the services/ subtree can host per-service design pages.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Covers CSP header enforcement, canonical-domain surfacing, and the
dashboard → Greenlight OIDC login flow. Passes against a reuse-kept
deploy.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Introduce a single global flag in group_vars/all/10_networks.yml and wire
the previously hardcoded IPv6 settings in nextcloud, bigbluebutton, and
matrix through it. Matches the host-level `make disable-ipv6` policy so
containers no longer stall on AAAA-first lookups (e.g. Nextcloud's
DnsPinMiddleware) while keeping a single knob to flip when v6 is wanted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces inline `applications`/`users` merging with dedicated lookup
plugins and runtime hydration. Introduces `add_csp_hash` filter,
integrates sort into `find_dock_val_by_bkp_entr`, splits the Keycloak
one-shot realm import into `05a_realm_import_run.yml`, and fixes the
Mastodon administrator task YAML. Removes obsolete `csp_hashes` filter,
`sys-token-store` Jinja filter, and empty CLI setup modules.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Apply documentation.md rules across staged files (emojis after headlines, RFC 2119 keywords, correct link text, Credits format)
- Remove expired ChatGPT share URLs from 39+ role READMEs and templates
- Remove orphaned text after link deletion: bare # @see, {# @see #}, Discussion: #}, empty Chat Conversation sections, dangling "Conversation is here available.", "Refer to the conversation with ChatGPT ()", "See ChatGPT Conversation", "ChatGPT Session Transcript" references
- Remove empty ## Further Resources and ## Credits sections where no links remain
- Fix URL reachability test: treat 401/403/405 as ok, 429/500/503 as warn, Timeout as warn; add # nocheck: url suppression (inline and line-above)
- Add lint test for broken relative and repo-root-absolute markdown links
- Add # nocheck: url to AUR git clone URLs, CSP allowlist entries, and other non-browser URLs
- Restore docs.infinito.nexus/setup in documentation.md MUST NOT example
- Fix Mobilizon geospatial endpoints (Addok, Photon) reverted to original values with # nocheck: url
- Add integration test rule: MUST NOT write unit tests for integration test logic
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename all docker-compose.yml templates to compose.yml
- Update override and CA override filenames accordingly
- Adjust compose wrapper, lookup plugins, filters and path helpers
- Update roles, tasks, handlers and scripts to use compose CLI
- Rewrite documentation and admin guides to reference compose
- Update unit and integration tests to expect compose.yml
- Keep internal variable keys for backward compatibility where needed
https://chatgpt.com/share/69870272-b168-800f-84c3-6417398e441f
The TLS lookup has been renamed from `tls_resolve` to `tls` to better
reflect its responsibility as a strict TLS state resolver.
This refactor updates all usages across the codebase, including:
- lookup plugins and internal helpers
- roles, vars, tasks, and templates
- tests, stubs, and error messages
- documentation and changelog references
Behavior, lookup API, and semantics remain unchanged.
https://chatgpt.com/share/6985e635-5a94-800f-b794-937e566ee308
- Remove deprecated filter_plugins/tls.py and all dependent filters
- Introduce strict lookup plugin lookup_plugins/tls_resolve.py as single TLS authority
- Migrate all roles, templates, vars, and tasks to use tls_resolve (protocols, ports, URLs, cert paths)
- Enforce explicit TLS context (domains, applications, TLS_ENABLED, TLS_MODE)
- Support letsencrypt and self_signed with SAN handling and per-app overrides
- Update TLS provider roles to assert resolver state instead of setting facts
- Add comprehensive unit tests for tls_resolve lookup
https://chatgpt.com/share/6972b49b-b074-800f-a323-dd9fe69d02eb
with explicit per-application TLS resolution.
Key changes:
- Introduce sys-front-tls-layer as the single TLS abstraction
- Centralize certificate paths via TLS_CERT_FILE / TLS_KEY_FILE / TLS_CA_FILE
- Replace sys-svc-letsencrypt SSL templates with generic TLS layer includes
- Add tls_* Jinja filters (enabled, mode, protocol, port, websocket)
- Resolve TLS mode per application (letsencrypt | self_signed | off)
- Remove implicit defaults and WEB_* globals to fail fast on misconfiguration
- Support self-signed certificates with SAN awareness
- Make TLS behavior explicit and consistent across all roles
This decouples application logic from certificate providers and
allows switching TLS flavor without touching app roles.
https://chatgpt.com/share/69727c61-8e68-800f-a267-473a7e627978
- Prebuild etherpad image in dev runtime to avoid local startup errors
- Override freeswitch entrypoint in dev to ensure missing iptables REJECT chain
does not crash the container (Docker-in-Docker compatibility)
https://chatgpt.com/share/6972262f-4318-800f-8bf8-f4166900f3a5
- add coturn enabled+shared → web-svc-coturn service resolution
- refactor BigBlueButton coturn logic to distinguish local vs shared coturn
- introduce BBB_COTURN_LOCAL_ENABLED flag
- split TURN secrets into local and shared variants
- update dependencies, env and compose templates accordingly
- enable shared coturn in BigBlueButton and Nextcloud configs
https://chatgpt.com/share/696d54a1-9dbc-800f-8507-088bad56bc48
- Switch BigBlueButton Docker source to upstream bigbluebutton/docker (main)
- Migrate coturn/collabora from internal to shared service model
- Make TURN protocol configurable (turn vs turns) via TLS_ENABLED
- Add BBB_TURN_PROTOCOL variable and use it in TURN_SERVER
- Only inject coturn TLS cert/key when TLS is enabled
- Normalize IPv6 flag handling with explicit boolean check
- Fix YAML formatting, trailing whitespace, and missing newlines
- Fix web-svc-legal dependency list formatting
- Add TODO entry for inventory dependency resolution
https://chatgpt.com/share/69687cb2-1550-800f-a5a4-394de22b21ec
- Replace DEFAULT_SYSTEM_EMAIL merge with flat SYSTEM_EMAIL_* variables
- Update templates across apps to new variable names
- Fix msmtp health check to use -t and local username fallback
- Improve postfix localhost config (localdomain, mydestination, timeouts)
- Extend sys-hostname to ensure /etc/hosts mapping once
https://chatgpt.com/share/695e8f39-0694-800f-b91c-a74f4b78d7c8
- Migrate Mailu credentials from legacy mailu_token to tokens['web-app-mailu']
- Update all affected application templates to use the new token structure
- Adjust inventory validation to no longer special-case mailu_token
- Minor typing and formatting cleanups in inventory creation and tests
https://chatgpt.com/share/69554915-87c8-800f-afd5-12118dbb1107
