Standardize the private inventory dir on the fully spelled "production" (matching inventories/development). Update the role README template (SPOT) and regenerate 153 role READMEs plus the admin deploy guide; add inventories/production/ to .gitignore, propagated to .dockerignore via make fix-dockerignore.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The guide deploys to a .localhost target, where letsencrypt cannot issue (Cloudflare DNS-01 needs a real domain). Default TLS_MODE=self_signed in the generated Production block so the guide uses the working self-signed provider. Document every deploy parameter as a top-level shell var (APP, HOST, TLS_MODE, SSH_PUBLIC_KEY) instead of an inline placeholder.
The compose guide now extracts and runs the whole README block, sourcing params from it (no more -e injection); HOST is forced to localhost. Skip checkout/login under ACT so the guide workflow is iterable locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The README Production block went provision -> deploy with no admin-key step, so user-administrator failed early (authorized_keys empty) for both the Guide CI jobs and any real user copying the README. Add a visible 'provision --vars' step that sets users.administrator.authorized_keys (merged into host_vars) in both Production variants of the role README template, regenerate all role READMEs, and replace the invisible CI payload splice with a visible substitution of the documented <your-ssh-public-key> placeholder.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The role README Production section now runs provision and deploy in a
single docker call: APP and HOST are shell variables passed via -e,
INVENTORY is the in-container SPOT for the duplicated paths, and the
redundant --vars-file (optional, default None) and --id (the inventory
is already narrowed by --include) flags are gone. All 153 role READMEs
regenerated from the template; test-run.yml executes exactly this block.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rewrite all 254 role descriptions to a uniform noun-phrase schema
(function-first, no leading verb, no trailing period, <=120 chars) and
enforce it via the extended description lint. Rename 12 role README H1
titles for clarity (e.g. Backup Container to Local -> Backup Docker
Volumes, CSS Service -> Corporate Design, NGINX Legal -> Imprint) and
regenerate their Cosmos and Quick Setup sections. Overview table
regenerated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add cli.build.readme.overview (make readme-index / make docs) rendering a
roles overview in the root README: Name, Lifecycle, Homepage, Video,
Description, Integrated with, Install. New info-media lint requires
homepage + video in every invokable role's meta/info.yml (opt out via
'# nocheck: info-media' in meta/main.yml); backfilled 147 roles with
researched, adversarially-verified homepages and official videos.
Retire the 'unsupported' lifecycle value: fold it into 'eol' as a valid
live end-of-life status (shipped but untested/unmaintained) across the
stage lists, symbol glossary, allow-list lint and docs. Retag confluence,
jira, minio, phpldapadmin to eol and magento to pre-alpha.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Roles declared under meta/main.yml dependencies now appear on both the
Dependencies and Dependents side, labelled with the glossary gear
symbol and connected by a fixed 1:1 edge; the legend explains it. All
role READMEs regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Edges now carry their relation: solid 1:1 for fixed consumption, dashed
0..1 for conditional (group-membership gated) consumption. Below every
diagram the generator renders a reading key for the edge kinds and the
node markers, with the glyphs drawn from the shared symbol glossary.
All role READMEs regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add cli/build/readme: derives each role README's managed sections from
templates/roles/README.md.j2.tmpl (the Cosmos mermaid diagram from
meta/services.yml, an invokable-gated Quick Setup, and Credits).
The Cosmos diagram shows dependencies, every declared service, and
dependents (roles that consume this one, matched by role name and by
provides provision-name in both directions). Role nodes carry deploy-mode
emoji markers (compose/swarm/host); disabled and shared services get a
stop mark; dependents are capped at 12.
test_readme enforces the Quick Setup body is byte-exact with the template
render. `make readme-generate` and `make cosmos` regenerate; `make cosmos`
also runs inside `make quality`.
Refactor: utils/symbol_glossary.py is the single source of truth for the
word/emoji mapping (unique project-wide), consumed by the complexity
matrix and the Cosmos builder. utils/roles/deploy.py holds role_has_stack
and role_deploy_modes, shared by both so the logic never drifts.
Add `make lint-mermaid` and its CI workflow, rendering every diagram via
mermaid-cli; the render step provisions chrome-headless-shell and unzip
on demand so it works in the minimal container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rails boot (including db:migrate) dies with OpenIDConnect::Discovery::
DiscoveryFailed / TLS alert 112 on onion nodes: the CSP initializer's
sso_host performs OIDC discovery, and the swd/webfinger gems force
URI::HTTPS regardless of the issuer scheme, dialing the plaintext-only
onion issuer on :443.
Mount a 00_-prefixed initializer override into web and sidekiq (HTTP-only
deployments, same guard as the force_ssl override) that pins the
SWD/WebFinger url_builder to URI::HTTP for http issuers. The 00_ name is
load-bearing: initializers load alphabetically and the pin must precede
content_security_policy.rb, which triggers the first discovery at boot.
Verified: mastodon onion closure deploy runs migrations and E2E green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per-role deploy-mode opt-out moves from meta/tests.yml 'skip:' to meta/services.yml <entity>.modes.{compose,swarm}.enabled on the primary entity. get_role_skip now derives from it; get_role_mode_enabled added. A new lint requires every role shipping templates/*compose*.yml.j2 to declare both mode flags. Migrates 107 services.yml and drops 11 skip-only tests.yml. Also removes the variant_status deployability gate so swarm runs one job per variant unconditionally (variant_status.py and deployable_variant_indices deleted, complexity model simplified). Adds the i8matix alias.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New roles/sys-svc-container/templates/env.j2 sets TZ={{ HOST_TIMEZONE }} and LANG=C.UTF-8. Every role env.j2 includes it at the top; it renders via stack_host_template, the same engine that resolves the includes in compose.yml.j2, so it works in both compose and swarm. TZ/LANG are removed from the individual role env.j2 so each value stays single-source.
New lint tests/lint/ansible/test_env_base_include.py verifies every role env.j2 includes the base and never re-sets a base-owned key (keys derived from the base file).
mattermost and gitea meta/volumes.yml drop the /etc/timezone host-path bind: swarm hard-rejects a task whose bind source is missing on the node (Debian-trixie nodes have no /etc/timezone). TZ now comes from the base env. /etc/localtime bind also dropped; the container timezone follows TZ.
Non-directive comments were removed from the touched env.j2 (banners, restatements, reference URLs, narration) so the changed files pass the comment-lint; only tool directives remain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Over Tor the redirect chain behind a login/logout/OIDC click routinely exceeds Playwright's fixed 30s action timeout, so the click throws before the flow's scaled waits run. Sweep all role Playwright suites and the shared persona helpers: give every navigation/auth-transition click an onion-scaled timeout (.click({ timeout: resolveTimeout(30_000) })) and route the remaining raw millisecond literals (timeout options, waitForTimeout, sleep-promise setTimeout, positional wait args) through resolveTimeout. Local UI clicks (menus, toggles, modal dismissals, DOM clicks inside evaluate) stay untouched.
Two new lints keep it that way: test_no_raw_timeout_literals forbids raw numeric timeout literals in Playwright specs (opt-out: nocheck: raw-timeout), test_persona_clicks_scaled requires every click in the shared persona auth flows to carry a resolveTimeout timeout (opt-out: nocheck: unscaled-click).
Verified: node --check on all 133 changed JS files, all 13 playwright lints green, make autoformat clean, make test green (2745+196+163+4 tests OK).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename the role, its test tree, the primary entity key
(volume-2-local), the derived vars (BKP_VOLUME_2_LOCAL_*,
SYS_SERVICE_BACKUP_VOLUME_2_LOCAL), the run-once fact, and every
group_names consumer flag, include path, doc, and lint reference. The
container_backup service key and the on-disk backup-docker-to-local
tree keep their names: both are contracts with existing consumer
entries, snapshot trees, and the baudolo pull tooling.
Operators must rename the svc-bkp-container-2-local inventory group and
disable the previous versioned units on already-provisioned hosts; the
pre-deploy snapshot glob derives its base from the new role name and
will not find them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Variant 0 pins the backup consumer flag enabled so the matrix baseline exercises the in-group branch; every other variant pins it disabled (shared: false carries the static-override suppression since the base value stays literally true). The matrix and wordpress variant entries move their services mapping to the list-item head so the static-override line parser resolves the override paths; yaml semantics are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every invokable stack role that declares named volumes in meta/volumes.yml now carries the standard container_backup consumer entry: enabled resolves via 'svc-bkp-container-2-local' in group_names, shared stays literally true since there is no per-consumer backup instance. Playwright/dynamic-flag/bond-baseline suppressions ride on the block per the established idioms; stale explanatory comments in the touched files are dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New lint test_no_raw_goto.py forbids raw `page.goto(...)` in role playwright specs + persona flows (the gotoOnion wrapper in personas/utils is exempt) so navigation retries transient Tor-transport errors over onion. Migrates every raw call across ~75 roles (343 sites) to `gotoOnion(page, ...)` and teaches gotoOnion to detect relative onion URLs (`isOnionCanonical`). gotoOnion is a behaviour-identical drop-in on clearnet (single attempt, same return). Verified: no-raw-goto lint green, eslint 0 errors, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the global TIMEOUT_FACTOR and a shared timeouts.js helper: resolveTimeout(base) scales by TIMEOUT_FACTOR and, over a .onion target, an extra multiplier, so Tor's per-request latency no longer trips hard-coded timeouts. Migrate every raw timeout in the specs (timeout:/timeout=/setTimeout/waitForTimeout) to resolveTimeout, stage the helper in all three staging paths, and add test_no_raw_timeouts to forbid raw numeric timeouts. Clearnet behaviour is unchanged (factor 1: resolveTimeout(N) === N).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Indentation/whitespace normalization from make autoformat across role meta files. No semantic changes (git diff --ignore-all-space is empty for every file). Kept separate from the mailu notls-over-onion and Tor-router idempotency fixes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Indentation/whitespace normalization emitted by make autoformat across role meta/services.yml, meta/variants.yml and a task file. No semantic changes (verified: git diff --ignore-all-space is empty for every file). Kept separate from the Tor egress-router feature.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds tests/lint/ansible/roles/test_single_replica_justified.py: a bare replicas:1 in meta/services.yml is flagged so a single-replica pin is never applied silently; services that can scale should carry no replicas key and let compose_replicas pick the topology default. Marks the pins on genuinely single-instance services (audit-verified as unscalable): checkmk OMD/RRD single-writer, erpnext websocket/scheduler/configurator sidecars, jenkins controller, listmonk campaign-manager singleton, mastodon db:prepare one-shot, minio single-node (no erasure), opencloud decomposedfs+NATS single-writer, seaweedfs single-node, taiga admin sidecar, ejabberd standalone Mnesia.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each role's meta/variants.yml now pins the tor onion gate both ways so the matrix exercises it: variant 0 pins services.tor.enabled: true, every non-baseline variant pins enabled: false + shared: false (# nocheck: variants-static-override, since tor.shared is a static literal). This gives real test_coverage / test_bond_baseline coverage without the removed per-key code exemptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Domain-serving roles reference the svc-net-tor provider via a tor: block. Consumers declare enabled (dynamic group_names gate) and shared: true (static — onion is always the shared node model, inline nocheck: dynamic-flag). The provider svc-net-tor additionally declares exclusive/primary defaults that consumers inherit via resolve_service_config unless overridden.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the per-role service_replicas Jinja variable with a per-service
replicas: field in meta/services.yml. deploy.yml.j2 resolves it via
resource_filter (with entity fallback), the same path already used for
cpus/mem, so services.yml is the single source of truth for replica
overrides.
- 13 web roles migrated to per-service (or entity-level) replicas: 1
- matomo bootstrap -> replicas: 0, mastodon migrate -> replicas: 1
- minio/seaweedfs provider entries -> replicas: 1, which also pins the
central single-writer object-store providers instead of relying on the
topology default
- service_replicas removed entirely; dropped its dead reference from the
compose-structure lint allowlist and updated requirement 023 wording
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sys-svc-container healthcheck templates (curl/wget/nc/tcp/http) now
derive container_port from services.<service_name>.ports.internal.http,
so a role no longer needs to set container_port only to feed the shared
healthcheck.
Add the container-port-set-unused lint that fails a role setting
container_port without using it elsewhere, then resolve every flagged
role: drop the redundant assignment where the derivation reproduces the
value, or mark it with nocheck where the port is non-http, a constant, or
keyed on a service the derivation cannot reach.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HSTS (Strict-Transport-Security) is an HTTPS-only header: a browser ignores it
over plain http (RFC 6797) and OpenResty only emits it on the TLS vhost. On an
onion node TLS is off and every domain is served over http, so the inline
`expect(headers["strict-transport-security"]).toBeTruthy()` in 33 specs failed
with `Received: undefined`.
Route all 33 through the shared expectHstsWhenTls(headers, baseUrl, label)
helper (personas/utils/tls.js), which requires HSTS over https and asserts its
absence over http. On a TLS node the base URL is https, so the assertion is
identical to before; on an onion node the same spec now passes. HSTS logic
lives in one place (SPOT) instead of 34 inline copies.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Under swarm MASTODON_CONTAINER_ADDRESS expands to a resolve subshell
that can return the container id of a task living on a worker; the
manager then runs 'container exec <cid>' against its local daemon and
fails with 'No such container' (seen in the fediwall variant-2 job).
Reuse the resolve pattern 03_setup already proved in the same run:
exec MASTODON_LOCAL_CID delegated to MASTODON_HOST_NODE. The delegation
for 06_administrator is hoisted onto the include via apply. Compose is
unchanged: the resolver returns the compose container name and the
inventory host there.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The health wait inspected containers on the deploy host, but swarm can
schedule mastodon services on workers, so the wait always timed out.
Each service now resolves its node and container id via
resolve_host_cid and delegates the inspect loop there; compose passes
the local container id through unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every compose.yml.j2 now declares services as '{{ service_name }}:' with
its '{% set %}' knobs on two-space lines above the key, includes at four
spaces and if/for tags aligned to the content they wrap. The layout
renders correctly because stack_host_template enables lstrip_blocks for
compose.yml.j2; indent(4, True) in the container base and '{%+' guards
in whitespace-sensitive scalars keep the shared fragments identical
under both render modes. A new lint test enforces the skeleton.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
04_wait.yml resolves the sidekiq container via container_address, which requires services.sidekiq.name; it was unset (unlike mastodon/streaming), so the wait aborted with "services.sidekiq.name not set". Add name: mastodon-sidekiq (and mastodon-migrate) matching the already-built container names.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The role flushed handlers to start web/streaming/sidekiq and only then exec'd
'rails db:migrate' into the running web container, so on a fresh DB the app
booted against an unmigrated schema (swarm converge gate timed out) and the
compose migration was OOM-killed sharing the 2g web cgroup with the live puma
server. Add a dedicated one-shot migrate service (own 2g container, db:prepare,
replicas 1) that web/streaming/sidekiq depend on (service_completed_successfully
in compose; swarm converges via crash-restart + the run-once gate). Refactor the
four near-identical service blocks into a mastodon_service macro.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Role README Credits now name galaxy_info.author (plain bold or wrapped
in a Markdown link), matched byte-for-byte against the role's own
meta/main.yml, instead of a hard-coded maintainer name. Regenerate all
245 role README Credits blocks to the new 'Implemented by ...' format.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mastodon/mastodon:latest floated to a Rails 8.1 major whose boot contract changed (fatal without a VAPID key), breaking the swarm deploy without any change on our side. Pin server and streaming to v4.6.2 so the boot contract can't shift under us again.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mastodon on the current mastodon/mastodon:latest (Ruby 4.0 / Rails 8.1.3) aborts at Zeitwerk eager_load_all (task: non-zero exit 1) without a usable VAPID key. The two keys were declared 'plain', so under CI allow-empty-plain they render empty and the web+sidekiq services never converge. Omitting the env vars (0fc5408db) was not enough: empty and absent both crash identically, so a valid key must be present. Add 'vapid_private'/'vapid_public' credential algorithms that emit one linked P-256 keypair (private = unpadded base64url 32-byte scalar, public = unpadded base64url uncompressed point 0x04||X||Y), wire the mastodon schema to them, and render both keys unconditionally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
VAPID keys render empty (plain algorithm, allow-empty-plain); recent Mastodon hard-fails boot on empty VAPID env, crashing during Zeitwerk eager-load. Only emit the VAPID vars when both keys are set.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In swarm an app task can be scheduled on a worker, but the role exec'd
`container exec $(resolve-container-id ...)` on the manager -> "No such
container". Resolve the hosting node + node-local container id and delegate
each exec to that node.
- database.py: add a `service_name` want; resolve_host_cid/wait_converged
gain `*_service_name` for cross-stack services (mailu's shared mariadb)
- mailu wait-for-mariadb (DB probe + admin), mastodon db:migrate,
gitea migrate/admin-create, friendica manager-ops/patch/addons/bootstrap
(~13 exec sites) switch to resolve_host_cid + delegate_to
Compose-safe: resolve_host_cid degrades to HOST_NODE=inventory_hostname
(delegate is a no-op) + LOCAL_CID=bare compose container name, matching the
8 roles already using this pattern.
Note: friendica also needs service_replicas=1 (next bundle) so its resolved
container id does not churn across replicas.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 2daygeek 'check website availability' link returned HTTP 404 and tripped the external-URL reachability test; it was tangential to Mastodon anyway.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Autoformat (mbake/yamllint) re-indented sequence items under their mapping key
across the regenerated meta files and wrapped one long URL scalar in the matomo
inject task. Whitespace/formatting only, no semantic change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Block-sequence items under canonical/connect-src reindented two spaces to the autoformat-canonical style after the server.yml domains/csp split. Whitespace-only; git diff -w is empty, parsed YAML unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the csp, domains, and networks blocks out of roles/*/meta/server.yml
into dedicated meta files, each mapping to its own top-level applications key
via the recursive meta-walk:
csp: -> meta/csp.yml -> applications.<app>.csp
domains: -> meta/domains.yml -> applications.<app>.domains
networks: -> meta/networks.yml -> applications.<app>.networks
Residual nginx/proxy keys (status_codes, client_max_body_size, locations,
ip6_enabled, config_upstream_url) stay in server.yml; an emptied server.yml is
deleted. 98 server.yml -> 249 new files, 74 deleted.
- Consumers repointed applications.<app>.server.{csp,domains,networks} ->
applications.<app>.{csp,domains,networks} across templates, vars, tasks,
lookups and filters.
- Infra readers migrated: canonical_domains_map, application_domain_index,
build_ldap_role_entries, service-registry overlay discovery,
roles_with_service, and the file-based readers utils/domains/list.py +
utils/meta/scan.py (read the now-deleted server.yml directly).
- Detection + mapping: _META_TOPICS gains csp/domains/networks;
ROLE_FILE_META_{CSP,DOMAINS,NETWORKS} constants + ROLE_FILES entries.
- New self-expiring lint guard (test_server_meta_no_split_fields) fails if a
server.yml still carries a split field; auto-fails from 2026-12-28 as a
delete reminder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
