funkwhale has two S3 surfaces sharing one private bucket. Audio goes out as X-Accel-Redirect and the front nginx fetches it server-side, but attachments (instance logo, avatars, covers) land in the API JSON as absolute URLs the BROWSER dials. AWS_S3_ENDPOINT_URL drives the boto3 client, so with the internal endpoint django-storages signed URLs for http://seaweedfs:8333, a host no browser can reach.
Point the client at the new bucket-less public_endpoint key. Measured against the live gateway: a URL signed for seaweedfs:8333 and sent to the public host answers 403 SignatureDoesNotMatch, while one signed for the public host reaches bucket lookup — query-string SigV4 binds the Host, so the signing host and the fetch host must be the same string. AWS_S3_CUSTOM_DOMAIN cannot serve this: storages/backends/s3.py:689 returns UNSIGNED without a CloudFront signer, and the bucket also holds the music library that music/views.py exists to gate.
AWS_CA_BUNDLE follows, because the public endpoint is TLS on the local CA and botocore reads its trust store only from that variable (configprovider.py:59) — sys-svc-compose-ca injects SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE and NODE_EXTRA_CA_CERTS, never this one. It takes the bundle rather than the bare root cert, since AWS_CA_BUNDLE replaces the default store.
AWS_QUERYSTRING_AUTH=True fixes a pre-existing break: upstream defaults it to `not PROXY_MEDIA`, which produced unsigned URLs against a private bucket. The new front nginx.conf.j2 serves /_protected/media, which the stock template resolves with `alias ${MEDIA_ROOT}/$1;` and therefore 404s for every S3-backed listen.
web-svc-seaweedfs keeps no onion vhost. Its tor-contract reason now states the mechanical cause instead of the old claim that the role is never a Tor-browser surface, which decidim and funkwhale have made false: the node hidden service publishes http only while sys-front-tls keys listen/ssl on application_id, so an onion vhost would listen on 443 against a port-80 hidden service. Attachment fetches therefore still fail on an onion pick, and the spec is left to fail there rather than record the miss and pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The entity name keys a role's instance directory, its compose project and its container name, and was derived only by stripping the longest matching category prefix. Two roles under different categories whose names share a suffix therefore collapse onto one entity and overwrite each other's rendered stack, which no provides: can repair.
A literal entity_name in vars/main.yml now wins over the derivation. web-app-seaweedfs-console takes its shorter name back and declares the entity seaweedfs-console, which frees the seaweedfs key in its own services map for the engine consumer and removes the provides:, covers: and alias entries that stood in for it. A new lint pairs the declaration with the primary key it has to match, because a drift between them would drop the role from the provider set in silence.
Compared across all 288 roles, exactly one entity changes, and it restores what main already keeps on disk, so no instance directory moves.
The function and the filter lose their get_ prefix. Two lints read that prefix as the discriminator between the variable and the filter that fills it; both discriminate on the pipe now. Ansible's templar resolves a filter and a same-named variable independently, verified against ansible-core 2.20.2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7b5833b37a assembled it in zammad's env template: urlencode both halves of the
credential by hand, patch them into the URL the lookup had already returned,
then append bucket and query. Three set-statements of string surgery on a value
the lookup builds from host and port in the first place.
The lookup now exposes credentialed_url next to url, built from the same
endpoint it already has, with urllib.parse.quote(safe='') on each half. The
template keeps only what is actually zammad's: the bucket path and the region /
force_path_style query.
This also removes the manual scheme concatenation that
test_onion_url_central_lookup refuses. That rule guards against a hand-built
app URL repeating the clearnet-vs-onion decision; the template was not doing
that, but it did name '://' and could therefore touch the scheme. Now it
cannot.
quote(x, safe='') is equivalent to the old
chain: checked over slashes, plus, equals, percent and space, no divergence.
Both branches of the lookup carry the key, the disabled one as the empty
string, so a consumer cannot hit a missing key.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-seaweedfs becomes two roles. web-svc-seaweedfs keeps the storage
engine: the container, the seaweedfs_data volume, the per-consumer S3
identities and the single canonical domain api. web-app-seaweedfs-console
takes the filer and master web UIs into its own compose stack behind the
admin-only oauth2 proxy, reaching the engine over the shared object-store
network. A node can now run the store without the consoles.
The console needs a distinct role name because get_entity_name strips the
category prefix: both halves would have resolved to the entity seaweedfs and
collided on the compose project, the instance directory and the container
name.
The filer and master ports move into the engine's services.seaweedfs.ports
.internal as their single owner; 01_init_consumer.yml consumes the master
port and would otherwise have resolved it only by role-var leakage from the
console, breaking bucket provisioning on exactly the console-less node the
split makes possible.
The engine's api vhost gains the prometheus locations. It never had them: the
override satisfied the cross-cutting-include lint only because the strings sat
in the SSO branch that the api domain never took.
Consumers, the objstore lookup and the catalogs follow the role id.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f713ba835 moved credentials to applications.<app>.secrets.credentials and
updated every role that reads them, but four lookup plugins kept addressing
the bare credentials.<key> root. They call get() with strict=False and
default="", so the miss was silent rather than loud: MariaDB received
MYSQL_ROOT_PASSWORD= and refused to initialise, which took friendica-database
down as an unhealthy dependency and failed compose up. Object-store consumers
and shared redis auth resolved to empty secrets for the same reason.
The five call sites now build the path from OVERRIDE_SECTION, the single point
of truth that commit introduced, instead of spelling it beside it.
The two lookup unit tests pinned the pre-rename fixture shape, which is why
they stayed green straight through the regression; their fixtures move to
secrets.credentials so they fail against the old code.
Evidence: run 31649386343, job 94302347256, artifact
rescue-diagnostics-instructions-web-app-friendica,
containers/infinito_nexus_debian/nested/.../friendica-database.log:29
"Database is uninitialized and password option is not specified".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extract the csp, domains, and networks blocks out of roles/*/meta/server.yml
into dedicated meta files, each mapping to its own top-level applications key
via the recursive meta-walk:
csp: -> meta/csp.yml -> applications.<app>.csp
domains: -> meta/domains.yml -> applications.<app>.domains
networks: -> meta/networks.yml -> applications.<app>.networks
Residual nginx/proxy keys (status_codes, client_max_body_size, locations,
ip6_enabled, config_upstream_url) stay in server.yml; an emptied server.yml is
deleted. 98 server.yml -> 249 new files, 74 deleted.
- Consumers repointed applications.<app>.server.{csp,domains,networks} ->
applications.<app>.{csp,domains,networks} across templates, vars, tasks,
lookups and filters.
- Infra readers migrated: canonical_domains_map, application_domain_index,
build_ldap_role_entries, service-registry overlay discovery,
roles_with_service, and the file-based readers utils/domains/list.py +
utils/meta/scan.py (read the now-deleted server.yml directly).
- Detection + mapping: _META_TOPICS gains csp/domains/networks;
ROLE_FILE_META_{CSP,DOMAINS,NETWORKS} constants + ROLE_FILES entries.
- New self-expiring lint guard (test_server_meta_no_split_fields) fails if a
server.yml still carries a split field; auto-fails from 2026-12-28 as a
delete reminder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consumer lookup plugins now reach the merged config through the Ansible
lookup loader instead of importing get_merged_applications /
get_merged_domains from utils.cache. The merge stays one cached
implementation surfaced via the applications/config/domains/users lookups;
consumers call lookup_loader.get(...).run() (the compose_networks idiom).
- get_merged_applications walks the whole meta/ tree recursively
(meta/**/*.yml to applications.<app>.<path>, addons and any depth) and
folds the raw meta/volumes.yml subtree onto applications.<app>.volumes
after the templar render so the Jinja source: strings stay unrendered.
- 31 consumer lookups rerouted; providers (applications/config/
applications_current_play/users/domains) and the raw-volume accessors
(volume/nginx) keep the cache import behind # nocheck: lookup-cache-import.
- New lint test_no_cache_imports_in_lookups enforces the contract.
- nfs_flavor is now the flavor SPOT (svc-storage-nfs-server + nfs_client_src
resolve via lookup('nfs_flavor')); removed the unused STORAGE_NFS_FLAVOR
group_var. Extracted extract_default_credentials to
utils/cache/schema_credentials.py to keep applications.py under the cap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Switch penpot, nextcloud, gitea, mattermost, openwebui, funkwhale, baserow, openproject, pixelfed, mastodon, bookwyrm, listmonk, peertube and opentalk to the engine-agnostic objstore lookup: the minio service entries are pinned off, a seaweedfs service (gated on the provider being in group_names, public for the media apps) replaces them, and the matrix variants pin the new flag. OpenTalk keeps object storage mandatory through an embedded fallback and provisions its bucket via weed shell.
Add objstore lookup SPOT keys consumed by these roles: endpoint (host:port), public_domain, public_bucket (public S3 host plus bucket) and public_url (scheme plus public_bucket, scheme derived from the provider TLS state). This removes the repeated lookup('config', lookup('objstore', ...), 'server.domains.canonical.api') / host:port concatenations from the consumer templates. Listmonk settings render their JSON values via the to_json filter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a database-style object-store abstraction: the objstore lookup resolves the active engine (seaweedfs|minio) per consumer and yields a uniform S3 payload, objstore_secret_key is auto-generated like database_password, sys-svc-objstore delegates per-consumer provisioning to the resolved provider's objstore_init entrypoint (lazy-loading the provider when it was not preloaded), and compose/network/volume plumbing supports both a shared central instance and an embedded storage-only instance per consumer.
Add the web-app-seaweedfs provider role: Filer and Master UIs behind the admin-only oauth2-proxy via a role-local nginx and proxy.conf.j2 (S3 API domain stays ungated, admin UIs deny with 403 when SSO is disabled), per-consumer S3 identities from s3.json, idempotent bucket provisioning via weed shell, a /24 on the central-provider subnet range, Playwright coverage for admin/biber/guest plus consumer bucket existence, and unit tests for the objstore lookup and the per-engine compose volumes.
Mark web-app-minio lifecycle: unsupported, move its mc client image into a dedicated services.client entry, align it with the objstore credential contract, and make its consumer provisioning idempotent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5