svc-net-tor aborts at roles/svc-net-tor/tasks/router.yml:2 when networks.internet.dns is empty, and rightly so: dnsmasq runs with no-resolv, so an empty upstream leaves the node answering .onion and nothing else. That value is set in exactly two places, both under inventories/development/, and the documented production path reaches neither - the provisioner writes no networks key at all. A reader following the role README verbatim therefore cannot deploy a Tor node.
CI has been proving this since 2026-08-31 without anyone seeing it: the README replay rides on the role's smallest variant and is skipped when the deploy is red, so the only row that ever reached it is one where svc-net-tor itself went green. That happened first on Ubuntu, where the local-service defect abb6f37297 fixes never fired.
The template now emits the DNS assignment, the -e passthrough and the networks entry in --vars for a role whose tasks name networks.internet.dns, which today is svc-net-tor alone; the generator derives that rather than taking a declaration, so a second role asking for the value documents it by doing so. scripts/github/guide/compose_deploy.sh substitutes the placeholder from INFINITO_DNS_IP, the same SPOT inventories/development/default.yml reads, next to the existing domain substitution.
This follows 972596a758, which added the administrator key to the same block for the same reason. No fallback inside the role: defaulting to NETWORK_PUBLIC_DNS_RESOLVERS would route a Tor node's clearnet queries to Cloudflare without the operator choosing it, which is what the assert exists to prevent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Production block never set one. inventory provision writes only
ansible_connection for a local host, so DOMAIN_PRIMARY stayed at the
group_vars default of infinito.localhost and the deployment served a zone its
own resolver knows nothing about. CoreDNS templates answers for
INFINITO_DOMAIN_RE and nothing else, so every lookup of the deployment's own
name came back NXDOMAIN.
Mastodon is where that surfaced. tootctl accounts create validates the address
through EmailMxValidator, which resolves MX and then, for the domain itself,
A and AAAA. All six queries returned NXDOMAIN in run 33574056480, the
validator recorded email :unreachable, and the admin account was never created.
Only the guide rows were affected: the matrix rows write
DOMAIN_PRIMARY: "{{ lookup('env', 'INFINITO_DOMAIN') }}" into their host_vars
and resolve fine.
The template now carries DOMAIN=<your-domain> in both the host and the
container form and threads it into --vars, with the container form passing it
through the docker run boundary as well. The two guide runners substitute it
the way they already substitute the SSH key, so the replay deploys under
INFINITO_DOMAIN like every other CI deploy. compose_deploy.sh moves its env
load ahead of the substitutions to have the value; host_deploy.sh had no env
load at all.
A production reader is asked for a domain now instead of silently inheriting
infinito.localhost, which no real deployment wants either. The shipped default
keeps its value: sys-svc-nss documents that *.localhost resolves through the
myhostname NSS module without any zone, and host-side tasks lean on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
House style bans em and en dashes. Established role READMEs carry none,
but the template had one in its Quick Setup line, so it had been added
after those roles were generated and reappeared in every regenerated
README since. Replacing it with a colon propagates to the 54 roles the
generator refreshed.
The Credits condition switches from an "is defined" test to a truthiness
test because the generator now always sets application_author_url,
passing None for an unknown author. Left as it was, the branch would have
rendered a broken [Name](None) link instead of falling back to the bare
name.
roles/categories.yml loses the same dashes from the svc.virt description.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Standardize the private inventory dir on the fully spelled "production" (matching inventories/development). Update the role README template (SPOT) and regenerate 153 role READMEs plus the admin deploy guide; add inventories/production/ to .gitignore, propagated to .dockerignore via make fix-dockerignore.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The guide deploys to a .localhost target, where letsencrypt cannot issue (Cloudflare DNS-01 needs a real domain). Default TLS_MODE=self_signed in the generated Production block so the guide uses the working self-signed provider. Document every deploy parameter as a top-level shell var (APP, HOST, TLS_MODE, SSH_PUBLIC_KEY) instead of an inline placeholder.
The compose guide now extracts and runs the whole README block, sourcing params from it (no more -e injection); HOST is forced to localhost. Skip checkout/login under ACT so the guide workflow is iterable locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The README Production block went provision -> deploy with no admin-key step, so user-administrator failed early (authorized_keys empty) for both the Guide CI jobs and any real user copying the README. Add a visible 'provision --vars' step that sets users.administrator.authorized_keys (merged into host_vars) in both Production variants of the role README template, regenerate all role READMEs, and replace the invisible CI payload splice with a visible substitution of the documented <your-ssh-public-key> placeholder.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A host role installs onto the machine itself, so its README Production block now shows the direct local-connection procedure (clone, scripts/install/package.sh, make install, source the env, then provision + deploy against localhost) instead of the container docker-run aimed at a remote server; stack roles keep the docker-run block. The mode is derived from role_has_stack. make docs now runs readme-generate quick_setup=true so the block stays in sync, and every host role README is regenerated accordingly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The role README Production section now runs provision and deploy in a
single docker call: APP and HOST are shell variables passed via -e,
INVENTORY is the in-container SPOT for the duplicated paths, and the
redundant --vars-file (optional, default None) and --id (the inventory
is already narrowed by --include) flags are gone. All 153 role READMEs
regenerated from the template; test-run.yml executes exactly this block.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Edges now carry their relation: solid 1:1 for fixed consumption, dashed
0..1 for conditional (group-membership gated) consumption. Below every
diagram the generator renders a reading key for the edge kinds and the
node markers, with the glyphs drawn from the shared symbol glossary.
All role READMEs regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add cli/build/readme: derives each role README's managed sections from
templates/roles/README.md.j2.tmpl (the Cosmos mermaid diagram from
meta/services.yml, an invokable-gated Quick Setup, and Credits).
The Cosmos diagram shows dependencies, every declared service, and
dependents (roles that consume this one, matched by role name and by
provides provision-name in both directions). Role nodes carry deploy-mode
emoji markers (compose/swarm/host); disabled and shared services get a
stop mark; dependents are capped at 12.
test_readme enforces the Quick Setup body is byte-exact with the template
render. `make readme-generate` and `make cosmos` regenerate; `make cosmos`
also runs inside `make quality`.
Refactor: utils/symbol_glossary.py is the single source of truth for the
word/emoji mapping (unique project-wide), consumed by the complexity
matrix and the Cosmos builder. utils/roles/deploy.py holds role_has_stack
and role_deploy_modes, shared by both so the logic never drifts.
Add `make lint-mermaid` and its CI workflow, rendering every diagram via
mermaid-cli; the render step provisions chrome-headless-shell and unzip
on demand so it works in the minimal container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removes the placeholder interpreter (utils/dict_renderer.py) in favor of
standard {{ lookup('config', ...) }} / {{ lookup('domain', ...) }} calls
inside role configs, and eliminates the `domains` set_fact stage in
tasks/stages/01_constructor.yml by introducing a cached get_merged_domains
helper plus a new `domains` lookup plugin (full map + by-app subtree).
The `domain` lookup no longer depends on a pre-computed `domains` variable;
it delegates to get_merged_domains, which respects inventory-level overrides
from both `applications:` (authoritative canonical source) and `domains:`
(direct override map). Caching mirrors the applications pipeline — keyed on
(roles_dir, variables signature, override fingerprint) with a re-entry guard.
Replaced `{{ domains[app_id] }}` / `{{ domains }}` usages across web-app-matrix,
-bluesky, -mastodon, -peertube, -keycloak, test-e2e-playwright, and
group_vars. Added two performance tests for lookup('domains') mirroring the
applications cache assertions (≥50x warm-to-hot ratio, 120s cold budget).
Replaces inline `applications`/`users` merging with dedicated lookup
plugins and runtime hydration. Introduces `add_csp_hash` filter,
integrates sort into `find_dock_val_by_bkp_entr`, splits the Keycloak
one-shot realm import into `05a_realm_import_run.yml`, and fixes the
Mastodon administrator task YAML. Removes obsolete `csp_hashes` filter,
`sys-token-store` Jinja filter, and empty CLI setup modules.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace legacy filter with the canonical lookup plugin across
nextcloud config, group_vars, dashboard template, and vars
template. Switches Talk TURN from onboard to standalone coturn
by default; gates talk container on spreed plugin flag.
Adds lint test to catch remaining get_app_conf usages.
Replace direct `docker` command usage, references, and documentation with
the `container` wrapper to ensure engine abstraction and future portability
(e.g. Docker → Podman).
This includes:
- Shell scripts, Ansible tasks, handlers, and Python helpers
- Documentation, comments, help texts, and meta descriptions
- Network, volume, system, and image operations
- Test expectations and comments
All runtime behavior remains unchanged while enforcing the repository-wide
rule to avoid raw Docker invocations.
https://chatgpt.com/share/698721c3-e270-800f-be7a-e4435fb54506
- Rename all docker-compose.yml templates to compose.yml
- Update override and CA override filenames accordingly
- Adjust compose wrapper, lookup plugins, filters and path helpers
- Update roles, tasks, handlers and scripts to use compose CLI
- Rewrite documentation and admin guides to reference compose
- Update unit and integration tests to expect compose.yml
- Keep internal variable keys for backward compatibility where needed
https://chatgpt.com/share/69870272-b168-800f-84c3-6417398e441f
Replace usages of the get_app_conf filter with the config lookup plugin
across roles and templates. Adjust quoting where required to keep YAML
and Jinja syntax valid. No functional changes intended.
https://chatgpt.com/share/69860678-cef8-800f-8402-7fe6231fdc2d
The TLS lookup has been renamed from `tls_resolve` to `tls` to better
reflect its responsibility as a strict TLS state resolver.
This refactor updates all usages across the codebase, including:
- lookup plugins and internal helpers
- roles, vars, tasks, and templates
- tests, stubs, and error messages
- documentation and changelog references
Behavior, lookup API, and semantics remain unchanged.
https://chatgpt.com/share/6985e635-5a94-800f-b794-937e566ee308
Renamed docker-compose and docker-compose-ca roles to sys-svc-compose and sys-svc-compose-ca, updated template includes, handlers, lookups, tests, and variables, and moved compose instances to /opt/compose.
https://chatgpt.com/share/6985ab2c-8344-800f-9e9a-768f562d138d
Replace attribute-style access (lookup('database', app).volume/.host/etc.)
with explicit want= lookups across docker-compose and matrix templates
to align with updated database lookup plugin interface.
https://chatgpt.com/share/6983ddfd-aee4-800f-bce9-18eca2144f25
Move TLS/ACME/CA vars out of 00_general into group_vars/all/02_tls.yml and renumber subsequent group_vars files. Add sys-ca-selfsigned to generate /etc/<software>/ca/root-ca.crt and make docker-compose-ca depend on it. Update setup script, templates, and tests to use the new applications/users file numbering.
https://chatgpt.com/share/697b8150-22ac-800f-98b1-e063a659b684
- Remove deprecated filter_plugins/tls.py and all dependent filters
- Introduce strict lookup plugin lookup_plugins/tls_resolve.py as single TLS authority
- Migrate all roles, templates, vars, and tasks to use tls_resolve (protocols, ports, URLs, cert paths)
- Enforce explicit TLS context (domains, applications, TLS_ENABLED, TLS_MODE)
- Support letsencrypt and self_signed with SAN handling and per-app overrides
- Update TLS provider roles to assert resolver state instead of setting facts
- Add comprehensive unit tests for tls_resolve lookup
https://chatgpt.com/share/6972b49b-b074-800f-a323-dd9fe69d02eb
with explicit per-application TLS resolution.
Key changes:
- Introduce sys-front-tls-layer as the single TLS abstraction
- Centralize certificate paths via TLS_CERT_FILE / TLS_KEY_FILE / TLS_CA_FILE
- Replace sys-svc-letsencrypt SSL templates with generic TLS layer includes
- Add tls_* Jinja filters (enabled, mode, protocol, port, websocket)
- Resolve TLS mode per application (letsencrypt | self_signed | off)
- Remove implicit defaults and WEB_* globals to fail fast on misconfiguration
- Support self-signed certificates with SAN awareness
- Make TLS behavior explicit and consistent across all roles
This decouples application logic from certificate providers and
allows switching TLS flavor without touching app roles.
https://chatgpt.com/share/69727c61-8e68-800f-a267-473a7e627978
Replace legacy volumes*.yml.j2 includes with the new compose_volumes filter across roles.
Add YAML-safe serialization for Ansible/Jinja proxy types to avoid PyYAML representer errors.
Remove obsolete volumes templates and extend unit tests to enforce required database_volume behavior.
https://chatgpt.com/share/696c2722-fd30-800f-bf01-dfcd3cb7ba72
- Drop deprecated docker_compose_file_creation_enabled flag (defaults + all consumers)
- Always run docker-compose file routines and bring stack up (caller can disable auto-flush via docker_compose_flush_handlers)
- Treat docker-compose.yml.j2 as optional via first_found (skip if missing)
- Clean up role template vars (remove removed flag, fix alignment)
- Record new E2E CI failures for Akaunting (502) and Baserow (timeout)
Fixes BigBlueButton CI failure caused by missing .env in repository:
Previously file routines could be skipped, so /opt/docker/bigbluebutton/services/repository/.env
was never created/linked and scripts/generate-compose failed with:
"cat: .env: No such file or directory" and "EXTERNAL_IPv4 is not set in .env".
With file management always enabled, the env file is copied/linked before compose generation.
https://chatgpt.com/share/69681be9-f944-800f-9b67-8afdc3a256fc
* Switch feature flag paths from docker.service.<name>.enabled to docker.services.<name>.enabled
* Update CSP builder and smart-default toggles and unit tests
* Update desktop cards iframe flag and desktop domain validation
* Update logout domain filter and tests
* Update recaptcha enable flags in Keycloak and EspoCRM
* Fix sys-ctl-cln-docker meta description formatting
* Follow-up: reorder imports in web_health_expectations so module_utils path injection runs before importing get_app_conf
https://chatgpt.com/share/6963f484-4cb4-800f-a9f4-8d70a66e4d32
- Add CSP3 support for style/script: include -elem and -attr directives
- Base (style-src, script-src) now unions elem/attr (CSP2/Safari fallback)
- Respect explicit base disables (e.g. style-src.unsafe-inline: false)
- Hashes only when 'unsafe-inline' absent in the final base tokens
- Nginx: set CSP only for HTML/worker via header_filter_by_lua_block; drop for subresources
- Remove per-location header_filter; keep body_filter only
- Update app role flags to *-attr where appropriate; extend desktop CSS sources
- Add comprehensive unit tests for union/explicit-disable/no-mirror-back
Ref: https://chatgpt.com/share/68f87a0a-cebc-800f-bb3e-8c8ab4dee8ee
Compute per-container CPU/RAM shares based on active services (web-/svc-*, enabled=true or undefined). Cast host facts to numbers, add safe min=1, and output compose-ready values. Include robust unit test.
Also: include resource.yml.j2 in base template and minor formatting tidy-up.
https://chatgpt.com/share/68d2d96c-9bf4-800f-bbec-d4f2c0051c06
