Four parse checks join the `lint-*` family, each using the language's own
parser rather than a reimplementation: `php -l`, `ruby -c`, `sqlfluff parse`,
and stylelint with an empty rule set (it raises CssSyntaxError whatever the
rules say, so the check stays syntax-only and grows no style opinion). Three
unit runners join `test-unit`: node:test, PHPUnit, minitest -- the first and
third ship with their runtime, so only PHPUnit adds a dependency.
Provisioning is best effort and the lint script decides whether it can run. A
machine without PHP must not fail the gate, but it must not look like success
either: the skip names how many files went unchecked. The first version of
these scripts read `git ls-files`, which inside the compose container returns
nothing (git refuses the bind-mounted repo as "dubious ownership") and reported
"No SQL files found." over 52 files -- green, and a lie. They use `find` now,
as the surrounding lint scripts already did.
sqlfluff needed two corrections before it parsed anything real. `%(name)s` is a
psycopg parameter marker, not SQL, so the placeholder templater is configured
rather than the files exempted; and the dialect follows each role's declared
engine (`meta/services.yml`), which is postgres for six roles and mysql for
svc-db-mariadb and pixelfed.
Recipes stay in scripts, not in the Makefile: the targets delegate, matching
every other target in the file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Tor-path credential tests tunnel their client through a local SOCKS5 shim.
When the CONNECT fails, handle() closed the client socket and returned, so the
client library saw nothing but an empty socket and reported whatever its own
generic read-EOF error is.
For MySQL that is `(2013, 'Lost connection to MySQL server during query')` —
which claims a query was in flight when none ever was. Fediwall 1 printed that
twenty times and it sent the investigation looking for a server that had hung
up, when in truth no HiddenServicePort existed behind the dialled onion port.
Postgres and ldap3 are less misleading ("server closed the connection
unexpectedly", "Connection reset by peer") but equally silent about the cause.
Reproduced for all three against a SOCKS5 proxy that completes the no-auth
greeting and rejects CONNECT with rep=5, running the real python block out of
each test.sh: every one now prints `socks connect failed: SOCKS5 CONNECT failed
rep=5` ahead of its library error.
This does not change any verdict — the root cause is fixed by 2ba3a14b9, where
tor_reachable requires both `services.tor.enabled` and an `exposed: true` that
actually earns a forward. This only stops the next occurrence from lying.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three svc-db-* credential tests each hand-rolled the same transport
gate in their test.env.j2:
(lookup('config', application_id, 'services.tor.enabled') | bool
and lookup('config', application_id, 'services.<entity>.exposed') | bool)
Three copies of one predicate, each naming its own entity key, and each
free to drift from the onion mapping that actually decides reachability.
Extract it into plugins/lookup/tor_reachable.py, split the same way as
its sibling tor_ports.py: a plain is_reachable() for callers holding the
merged applications view, plus a thin LookupModule resolving it through
lookup('applications'). Because collect_exposed_ports() scans
services.values(), the entity name disappears from the templates
entirely -- they collapse to
TOR_ENABLED={{ lookup('tor_reachable', application_id) | lower }}
Rework the integration guard accordingly: it now asserts that every
svc-db-* template delegates to the shared lookup, and drives its
base/variant agreement table through the real is_reachable() instead of
re-implementing the predicate. Add unit coverage for the new lookup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A cold onion connect alone measured 15.7s and tor holds SOCKS streams
up to 120s while fetching the descriptor; the 30s caps cut cold
circuits mid-handshake across all three forwarders and DB clients.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test.sh: dispatch auth via a run_auth wrapper calling auth_tor/auth_local directly — shellcheck SC2317 flagged the indirect ${attempt_fn} invocation as unreachable. test.env.j2: sort keys ASC for the env alphabetical-order lint.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add exposed: bool (default false) to svc-db-postgres/mariadb/openldap meta/services.yml with a variant-0 (exposed) / variant-1 (not) toggle, plus real credential e2e tests (files/test/test.sh + templates/test.env.j2): authenticate with actual credentials over the node Tor SOCKS proxy when the variant enables tor, locally in-container (password auth) otherwise. mariadb/openldap force the local publish (+ OPENLDAP_PORT var); secrets stay off the process argv (MYSQL_PWD, ldapwhoami -y /dev/stdin, PGPASSWORD env).
Depends on shared onion pieces landing in a follow-up commit: tor_ports exposed-port collection, svc-net-tor services.tor.node + services.tor.ports.local.socks SPOT, the tor_socks lookup, and test-e2e-cli 0600/stage-cleanup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DEPLOYMENT_MODE is now derived per role from group membership:
swarm iff len(groups[application_id]) > 1, else compose. The cluster
infrastructure flag SWARM_CLUSTER_AVAILABLE is true whenever any host
sits in svc-docker-swarm, so DB and proxy services render attachable
overlay networks even when their own group lists only one host.
All service_is_stateful flags are removed across svc-db-mariadb,
svc-db-postgres, svc-db-openldap, svc-prx-openresty, svc-docker-registry
and svc-db-mariadb's compose template; the inventory shape now expresses
the same intent. Cluster-wide handlers (swarm build/push/validate/deploy,
external network creation, DB healthcheck) gain run_once + delegate_to
the manager so they execute exactly once regardless of how many hosts
the role's play targets. CA-injection and compose validate are gated to
compose-mode only; their compose-config strictness breaks when the
deploy block coexists with container_name under swarm.
Dead per-host swarm-resolve infrastructure is removed: swarm_resolve_
container.sh, resolve_container.yml, resolve_self.yml, the MariaDB-
specific swarm_mariadb_cid.sh, the cross-host template replication
step, and the _swarm_retries / _swarm_delay defaults. With run_once +
delegate_to: manager replacing the old "render every template on every
swarm peer" pattern, every consumer becomes a noop on workers and
collapses to the manager's one-shot path.
The web-app-nextcloud memcache override mapping flips from the legacy
"single" sentinel to "compose" (semantically equivalent under the new
model). Requirement 023 is updated to reflect the new model and drops
the now-irrelevant per-role opt-out / placement-constraint clauses.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the svc-docker-swarm, svc-docker-swarm-manager, svc-storage-nfs-server
and svc-storage-nfs-client roles, the compose_volumes filter NFS support,
the test-deploy-swarm-nfs CI workflow with its swarm-nfs-pilot script set,
and the deploy_mode-aware compose/swarm wiring across mediawiki, mariadb
and sys-svc-* helpers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lint scanner now flags `community.docker.docker_container_exec` — it has
a 1:1 `compose exec`/`container exec` replacement and pinning to Docker
defeats engine-swappability. Other `community.docker.*` modules (network
create, host_info, container start with image+args) have no clean wrapper
equivalent and stay legal.
Migrations:
- svc-db-mariadb (01_core, 02_init, 03_reset_root_password): replace
module calls with `container exec -i` via `ansible.builtin.command`,
pipe SQL via stdin. Health probes share a single SELECT 1 file.
- web-app-decidim (01_setup): same migration, preserving env vars as
`-e KEY=VAL` argv elements and stdin for rails-runner snippets.
Extract all SQL to `roles/svc-db-mariadb/files/sql/*.sql` as plain SQL
(no Jinja). Values flow as MariaDB user variables (`@db_name`, `@db_pwd`,
`@root_pwd`, …) via a `SET` prefix in the task stdin. Identifier
substitution (db/user/encoding names) uses `PREPARE` + `CONCAT(@var, …)`
since `@vars` cannot bind identifiers directly. Password values use
`QUOTE(@var)` inside SQL and Jinja `replace("'", "''")` on the task side.
Pin MariaDB to "11.8" (current LTS major). Required for `CREATE DATABASE`
in prepared statements, supported since 10.5.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5
Environment TestandClaude Sonnet 4.6