1f870b1e5 left two marks the layout rules reject. friendica's utils/ held
patch_config.yml beside patch_db_credentials.yml, which is a shared
delimiter-bounded prefix, and its numbered files ran 00, 01, 02, 04, 05 with
nothing at 03 after the config patch moved out of the sequence.
The prefix becomes the folder it was standing in for, the way 8e8dbbd08c did it
for the AI files. All four roles move, not only the one the lint could see: the
path post_deploy.yml resolves a role's declaration through has to read the same
everywhere, so a folder in one role and a prefix in the other three would be no
domain at all.
utils/patch_db_credentials.yml -> utils/patch/db_credentials.yml (4 roles)
utils/patch_config.yml -> utils/patch/config.yml (friendica)
The numbers close up behind the file that left: 04_addons and 05_user_bootstrap
become 03 and 04.
Four references follow: the include in 01_manager_ops.yml, the lookup in
post_deploy.yml, the caller path in the unit test that reads each role's
declaration, and a requirement doc that linked the old addon file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3831c8f804 gave matomo, moodle and nextcloud a patch each, called from their
own 00_core.yml. That file only runs while the role's run-once flag is unset,
so the patch runs on the deploy's first pass and never on the second -- which is
the only pass where a rotated credential exists. The app keeps the password it
was installed with while the engine moves on, and every request answers 500.
Run 33317483326 shows it on friendica, which had the same shape: svc-db-mariadb
altered the user at 15:39 and again at 16:26, either side of the 15:58 rotation,
while the app's config was patched once. The engine said what followed:
[Warning] Access denied for user 'friendica'@'192.168.201.4' (using password: YES)
Five failed healthchecks in a row, swarm replaced the task, and the convergence
gate reported "tasks not running yet" for a service that started correctly every
time.
The call moves to post_deploy.yml, which runs after every deploy and sits behind
no role's flag. It includes roles/<app>/tasks/utils/patch_db_credentials.yml when
that file exists, so a role declares what to patch and nothing else: friendica
joins the other three with its volume, its config path and its expression.
It cannot live in the role. 00_core.yml has to set the run-once flag as its very
first task or load/app.yml re-includes the role and recurses, which
tests/lint/ansible/roles/test_service_core_first_task_run_once.py enforces, and
everything after that flag is closed to a second invocation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
383a59e9a1 gave the shared engine a way to be told a rotated superuser
password: swap pg_hba to scram, ALTER USER over the local socket, verify the
new password over TCP, put pg_hba back. An engine an app owns alone never got
that, so after the deploy rotates a credential the app holds the new password
and its own postgres still holds the old one, and every request answers 500
until someone recreates the volume.
The script is now told which role to alter instead of assuming postgres. Both
engines are reached the same way -- the container's local socket authenticates
by peer, which is the whole reason a password can be written into an engine
nobody can log into any more -- so one implementation covers both. The role
travels as an environment variable and is applied through psql's identifier
quoting rather than string interpolation.
MariaDB solves this at engine start with --init-file (b067d21ef5). Postgres has
no equivalent and /docker-entrypoint-initdb.d only runs on an empty PGDATA, so
this has to be an ansible step after the container is up. post_deploy.yml is
where it goes, right behind the health wait, and that gate is passable here for
a reason worth stating: the dedicated healthcheck is pg_isready, which does not
authenticate, so the container reports healthy on a stale password and leaves
the window this task needs.
Seven jobs of run 33279942812 fail this way -- mattermost, pretix, xwiki,
gitlab, opentalk, flowise, discourse -- each traced to the same rotation gate
that exists to prove a changed secret propagates.
Exercised against a stubbed container for both roles: postgres and an app role
behave identically, the role reaches all nine calls, ALTER USER fires. It still
refuses to run without a role, and still reports FAIL when authentication never
succeeds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A handler name was doing two jobs at once: the line an operator reads in the
play output, and the identifier a `notify:` binds to. The two pull in opposite
directions -- the log wants "restart the SSH service, with diagnostics", the
binding wants a token that survives quoting, YAML flow style and a shell
round-trip -- and a name that changes for readability silently changes which
handler runs.
Every handler that a notify reached by name now carries a `listen:` with that
exact wording, so the binding is explicit before anything is renamed. The
topics themselves become identifiers: `compose up` reads `compose-up`,
`Restart NFS ganesha` reads `restart-nfs-ganesha`, 22 of them across both
sides of every notify. Where the topic is assembled at runtime, both branches
moved together: the conditional in sys-service carries two literals, and the
LDAP renderer interpolates `import-{{ folder }}-ldif-files`, which is what
makes those handlers reachable at all.
Freed from the binding, the names take the emoji prefix every other task line
already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The msmtp_curl probe joined its mail branch to the curl probe with '&&', so a
relay that refuses the startup message short-circuited the whole command: curl
never ran, the container stayed unhealthy forever, and the swarm converge gate
burned all 150 retries before aborting the deploy. Friendica 0 and Fediwall 0
of run 31340965955 died that way, reporting only 'not converged:
friendica_friendica' with the real cause buried in a rescue artifact.
That contradicted the probe's own docstring, which promises an unrelated SMTP
outage cannot flap the container into unhealthy -- it only delivered that for
email_enabled=False. sys-ctl-hlth-msmtp/files/script.sh:23 has always tolerated
exactly these exit codes; the container probe was the outlier.
Liveness is now curl's verdict alone. The '&&' before touch stays, so a refused
mail leaves no marker and the next probe retries it rather than turning the
smoke test one-shot.
Detection does not disappear with the coupling: web-app-friendica now asserts
the delivery marker after convergence and fails with the relay's reason instead
of a timeout three layers away. sys-ctl-hlth-msmtp is a systemd timer and the
Mailu Playwright roundtrip submits through Roundcube, so neither covers a
container relaying over port 25 -- this assertion is what does.
The marker path moves into utils.docker.healthcheck.MAIL_MARKER and reaches
Ansible through lookup('container_mail_marker'), so the probe and the assertion
cannot drift apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_task_file_numbering.py: numbered task
files use exactly two leading digits, a numbered *_core.yml must be
00_core.yml, per directory the numbers are unique/consecutive and start
at 01 (00 with a core file), numbered includes appear in ascending
order, and shared helpers live in tasks/utils/ instead of the numbered
chain.
Apply it across every role: rename NN_core.yml to 00_core.yml with a
consecutive down-shift of the siblings, move letter-suffix children into
a subdirectory named after their parent task, relocate helpers included
from more than one place into tasks/utils/, and update every reference
(include_tasks, path_absolute, .ansible-lint-ignore, requirement docs).
Legacy narration comments in the touched files are deleted; the upstream
xwiki.properties reference block opts out via nocheck. The unused
MAILU_IP6_PUBLIC var (only ever named in now-removed comments) is dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The upstream entrypoint serialises its first-run init with flock on friendica-init-sync.lock inside the shared /var/www/html, which sits on NFSv4 on this branch; local_lock=flock is silently ignored on vers=4 mounts, the lock contends across replicas over server-side locking and the set -eu entrypoint dies with exit 65 on every worker, pushing all replicas onto the manager. Build a custom image whose entrypoint gate lets only task slot 1 run the init while other slots wait until the volume VERSION matches the image VERSION - the same pattern nextcloud already uses.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In swarm an app task can be scheduled on a worker, but the role exec'd
`container exec $(resolve-container-id ...)` on the manager -> "No such
container". Resolve the hosting node + node-local container id and delegate
each exec to that node.
- database.py: add a `service_name` want; resolve_host_cid/wait_converged
gain `*_service_name` for cross-stack services (mailu's shared mariadb)
- mailu wait-for-mariadb (DB probe + admin), mastodon db:migrate,
gitea migrate/admin-create, friendica manager-ops/patch/addons/bootstrap
(~13 exec sites) switch to resolve_host_cid + delegate_to
Compose-safe: resolve_host_cid degrades to HOST_NODE=inventory_hostname
(delegate is a no-op) + LOCAL_CID=bare compose container name, matching the
8 roles already using this pattern.
Note: friendica also needs service_replicas=1 (next bundle) so its resolved
container id does not churn across replicas.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce plugins/lookup/path_absolute.py as the single source for the
repo-root anchor: lookup('path_absolute', 'roles/...') returns an absolute
path under playbook_dir, joining every term. Replace all hand-built
playbook_dir path constructions across .yml/.yaml and .j2 templates with it:
[playbook_dir, '<rel>'] | path_join -> lookup('path_absolute', '<rel>')
playbook_dir ~ '/<rel>' -> lookup('path_absolute', '<rel>')
{{ playbook_dir }}/<rel> -> lookup('path_absolute', '<rel>')
Unrelated repo_path symbols in cli/meta/roles/applications/resolution and the
single bare discover_playwright_roles(playbook_dir) call are left untouched.
Add tests/lint/ansible/test_playbook_dir_via_lookup.py: fails the build if any
of the three construction forms reappears in .yml/.yaml/.j2, pointing callers
at the lookup. Unit test covers the plugin.
Touching these files pulled their pre-existing narration comments into
test_comments_valid scope; clean them (delete restatement/banners, keep real
trip-wires as Workaround/Upstream/Invariant markers). Drop caution, important
and pitfall from the comment-lint marker vocabulary and remove the comments
that used them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Surfaced by deep-inspecting the swarm matrix CI failures:
- sys-ctl-cln-faild-bkps: the systemd ExecStart resolved cleanback via the
command_path lookup, which runs on the Ansible controller; the binary is
pip-installed system-wide on the target node, so shutil.which returned None
("command not found in PATH: 'cleanback'"). The ExecStart already runs via
/bin/sh -c, so use the bare command and let the target shell resolve PATH.
- svc-db-rabbitmq: RABBITMQ_EXEC_SHELL used `sh -lc`; a login shell re-inits
PATH from /etc/profile and drops the image's /opt/rabbitmq/sbin, so
rabbitmq-diagnostics was not found (rc 127). Use `sh -c`.
- web-app-friendica: `{{ FRIENDICA_CONTAINER_ADDRESS | quote }}` single-quotes
the swarm $(resolve-container-id ...) subshell, killing the command
substitution ("No such container"). Drop the quote, matching 02_manager_ops.
- web-app-gitea: `gitea migrate` ran without `-c {{ GITEA_CONFIG }}` (admin-create
already has it), so MustInstalled() aborted against the custom app.ini. Add it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds tasks/addons hooks for the SSO/LDAP-coupled addons of the remaining web-app roles (exception-only; SPOT comments where the generic config + shared OIDC SPOT fully wire it). web-app-friendica gains the per-addon hook include in its 04_addons loader.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add tests/lint/ansible/tasks/test_include_uniform_apply_keyword.py: when a file pulled in via a dynamic include_tasks carries the same hoistable task keyword on every top-level task, that keyword should be set once via apply: on the include site instead of being repeated per task. The rule is blacklist-based: it flags any uniform Ansible task keyword except the per-task-binding ones (name, register, args, vars, when, loop), and never flags the module itself.
Hoist every existing occurrence to apply: across the affected roles (delegate_to, run_once, become, notify, no_log, changed_when, delay, async, poll), dropping the per-task duplication. Behaviour is identical because apply: propagates each keyword to every included task; no_log masking via apply: was empirically verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ldapauth addon bridges [ldap]; ldap+sso service blocks stay in meta/services.yml; tasks/04_addons.yml reads the new addons path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Long-string continuation breaks introduced by `make autoformat` on
the friendica addon-sync script command and the matomo bootstrap
failure message. Pure cosmetic, no semantic change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace compose-mode-specific `{{ BIN_COMPOSE }} exec ... {{ SERVICE }}`
+ `args.chdir: lookup('container', ..., 'directories.instance')`
invocations with `container exec -i {{ ..._CONTAINER_ADDRESS | quote }}`
(plus `--workdir` flag where chdir was load-bearing). The container
exec wrapper resolves the actual runtime container name on both
compose (compose-suffixed name) and swarm (stack-task-id-suffixed
name) via the `container_address` lookup, so the task no longer
assumes the compose-mode service-name address.
Pure-syntax adaptation only — no behavioural change, same register /
when / failed_when / no_log semantics preserved on every task.
Roles touched (12 task files):
- web-app-espocrm (1)
- web-app-friendica (2)
- web-app-listmonk (1)
- web-app-openproject (2)
- web-app-snipe-it (3)
- web-app-taiga (1)
- web-app-wordpress (2)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a mode-aware `container_address` lookup plugin and a
`resolve-container-id` helper script that resolve swarm task IDs at
exec time (compose mode returns bare names unchanged). Migrates ~25
roles + ~28 task files from `command:` to `shell:` so the lookup's
`$(...)` subshell evaluates correctly, and extends `database` lookup
with an `address` key for cross-role DB exec.
Three new lint tests enforce the contract: every container exec
target goes through the resolver, lives behind an IS_STACK_HOST gate
(walks the include graph one hop deep), and uses `shell:` rather than
`command:` when the target is a Jinja reference. Existing nocheck
suppressions on bare-name DB exec sites (nextcloud redis, listmonk,
moodle OIDC, mailu) are replaced with the lookup-based form.
Fixes the systemd Environment= injection in sys-ctl-cln-disc-space
(it never expanded $(...)) by passing STACK / SERVICE_KEY parts and
resolving inside the script. Extracts shopware admin and mediawiki
extension install bodies into dedicated .sh files invoked via env.
Rewrites password-bearing shell tasks to either `| quote` directly or
the `-e VAR=... | quote` + `sh -c '... ${VAR} ...'` env-bridge pattern.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Across roles/web-app-*, the typical body shape is:
- include_role: sys-stk-full (or sys-svc-compose) cluster prereqs
- <host-fs writes, container exec, compose exec> stack-host only
Under swarm-mode delegation the second group only makes sense on the
host that holds the compose stack (manager), but the role plays on
every host in groups[<role>] so workers abort on missing paths.
Two workflow sweeps (gate-manager-ops + gate-manager-ops-01core, 58
agents total, discover/plan/adversarial-verify per role) plus three
manual fixes surfaced 27 roles to refactor. Each role keeps
sys-stk-full ungated so worker-side prereqs (sys-svc-container, NFS
client) still bootstrap; the trailing manager-only block either moves
into a new tasks/02_manager_ops.yml included with
when: IS_STACK_HOST | bool (SUB_FILE pattern) or gets the gate
inlined on the one or two trailing tasks (INLINE pattern). In
compose mode IS_STACK_HOST stays true so single-host deploys are
unchanged.
Manual catches the workflow agents missed:
- web-app-fediwall: utils/once/flag.yml lives AT THE END of 01_core,
not at the start; preserved verbatim outside the gate so workers
still flip the run_once_<role> fact.
- web-app-xwiki: a pre-existing block(...) wraps sys-stk-full plus
_flush_config.yml; gated only the _flush_config.yml include so
sys-stk-full keeps running on workers.
- web-app-dashboard: 02_javascript.yml (creates /opt/compose/dashboard/
JS dirs) ran BEFORE sys-stk-full and was ungated; added the gate.
Roles touched (INLINE): akaunting, jenkins, magento, mobilizon
Roles touched (SUB_FILE): baserow, dashboard, decidim, erpnext,
fediwall, friendica, gitea, jitsi, keycloak, listmonk, littlejs,
mailu, matomo, moodle, odoo, openproject, opentalk, pixelfed,
shopware, suitecrm, taiga, xwiki, zammad
lookup('tls', ...).url.base keeps a trailing slash, so Friendica generated double-slash profile URLs (host//profile/<nick>) that broke local-follow contact resolution (no confirmation form rendered). Strip it via a shared FRIENDICA_BASE_URL var reused by env.j2 and the local.config.php patch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pure 1:1 rename (69 insertions / 69 deletions across 66 files) to
finish the ``docker_*`` → ``compose_*`` alignment started in
8373468e9. The flag configures whether the compose include flushes
handlers right after run; the ``docker_`` prefix was misleading
(handlers are Ansible-side, not docker-side) and the variable is part
of the same ``sys-svc-compose`` SPOT cluster as ``compose_repository_*``.
Touches the role default, the implementation
(``sys-svc-compose/tasks/utils/up.yml``), all 60+ consumer roles,
the network-utility wrapper + its README, and the
``test_network_create_via_util.py`` lint (both the docstring guidance
and the asserted diff).
Breaking change for any external override that still sets the old
name — no alias is provided.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Remove stale 0-byte ldapauth.config.php leftover when LDAP is disabled
in a matrix variant (docker bind-mount target survives in the html
volume after the bind mount is dropped, breaking bin/console).
- Force bin/console dbstructure update before addon sync; matrix variant
rounds can leave the friendica html volume's local.config.php in place
while the shared MariaDB has been re-provisioned without the friendica
tables, so addon enable then writes to non-existent tables and the
next playwright login times out.
- Add autoload-aware probe_hook_table.php; the previous inline PHP probe
always returned rc=255 on Friendica\\Module\\Register::OPEN and only
worked accidentally as a 120s sleep via retries.
- Reorganize role files into files/php/ + files/shell/ subdirs and drop
the multi-line explanatory comments now that the why-lines live in the
commit body.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The shell wrapper was a thin pipe around an inline PHP heredoc — only
needed because the actual count + retry logic lives in PHP land. Move
the logic into a standalone `verify_addon_hooks.php` so the PHP code
is lintable, syntax-highlightable, and free of double quoting.
The ansible task now pipes the file content via stdin into
`compose exec -T friendica php -- ldapauth 2`. PHP reads the script
from stdin and picks up addon + min-hooks from $argv. Logic
unchanged: count rows in `hook` for `%addon/<X>/<X>.php`, recycle
disable+enable on a miss, re-check, fail loud if still empty.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Empirically, `bin/console addon enable ldapauth` on a fresh stack
writes `config[cat=addons,k=ldapauth]` but the Hook::register() rows
from `_install()` are silently dropped — `addon list enabled` passes
while the `hook` table stays empty, so every authenticate hook is a
no-op and biber/admin login silently re-renders the form. The race
recovers on a second disable+enable cycle from a clean shell.
Add a post-sync verifier (verify_addon_hooks.sh) that PDO-queries
the hook table for `%addon/ldapauth/ldapauth.php` and, if fewer than
MIN_HOOKS rows are present, runs one disable+enable cycle and
re-checks. Fails loud on the second miss so the broken state surfaces
in the deploy instead of as a downstream Playwright login regression.
Wire it into 04_addons.yml after the existing addon sync loop,
gated on FRIENDICA_LDAP_ENABLED.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The patch task in 02_ldap.yml ran `compose exec friendica` before the
container was up (it sits before the meta: flush_handlers in 01_core.yml),
and even after a forced flush it raced the entrypoint rsync that copies
/usr/src/friendica/ into the /var/www/html volume. The script then saw
`addon/ldapauth/ldapauth.php` missing and exited 1 with the misleading
"NO-MATCH ldap_createaccount return-line shape changed upstream" — for
30 retries × 2s, then fatal. CI run 25967376208 (job 76334304707) hit
this on the first push that introduced the patch task (9763cd8a4).
Fix: move the patch into 04_addons.yml, after both readiness probes
(`bin/console list` for vendor/addon rsync, `addon list all` for DB)
and before the addon sync that enables ldapauth. Gated by
FRIENDICA_LDAP_ENABLED. Drops the retries/until — once those probes
pass, the file is guaranteed present, so a non-zero exit now means real
upstream drift and should fail loud.
02_ldap.yml is back to the pre-patch shape: template ldapauth.config.php
and notify compose up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause: upstream friendica-addons/ldapauth's `ldap_createaccount`
returns the raw `User::create()` result (`['user' => UserRow, 'password' => '']`),
but the `authenticate` hook only sets `authenticated = 1` when
`$user['uid']` is non-empty — never true for the autocreate path. First-
login LDAP users (e.g. biber) were inserted into `friendica.user` but
the session was never started, so /network kept serving the login form.
Encoded as an idempotent in-container patch task in `tasks/02_ldap.yml`
+ `files/patch_ldapauth_autocreate.sh` (reports PATCHED on first run,
ALREADY on subsequent runs). Runs whenever LDAP is wired, gated by
FRIENDICA_LDAP_ENABLED.
Other changes in the same stack:
- meta/services.yml: drop keycloakpassword addon (oauth2-proxy in front
owns SSO); ldap.enabled now follows svc-db-openldap group presence so
the variant matrix can disable it without a static override.
- tasks/01_core.yml + tasks/06_user_bootstrap.yml: pre-create the
administrator only in LDAP-disabled variants — when LDAP is wired,
ldapauth materialises the row on first admin login.
- files/playwright.spec.js: variant-aware spec replacing the parked
test.skip cases. Three tests now run: guest flow (always), admin
native login (picks double-login vs direct based on
OAUTH2_SERVICE_ENABLED), biber + admin side-by-side in isolated
contexts. PERSONA_* still blocked — friendica has no header-trusted
autocreate path.
- templates/playwright.env.j2: expose BIBER_* + service-gate flags.
- inventories/development/default.yml: biber email aligned with the
oauth2-proxy `email_domains = infinito.example` filter so biber's
Keycloak login isn't 403'd at the proxy.
- TODO.md: SAML migration outlined as the long-term replacement for
the double-login (currently second hop silently bypasses Keycloak 2FA).
Drive-by: trim double-space before `# nocheck` in
roles/sys-aur/meta/services.yml.
Verified on VARIANT=0 (oauth2 + ldap + oidc all enabled): three
playwright tests green, failed=0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh: rename AUTOFORMAT_PARALLEL to PARALLEL,
declare it at file start with default `true`; still env-overridable.
Reword the worker comment so the literal `# shellcheck` token no
longer appears at start-of-line and trips SC1072/SC1073.
* scripts/lint/galaxy.sh: drop the GALAXY_FATAL / GALAXY_NAMESPACE /
GALAXY_PARALLEL env vars (over-engineered for a single repo). Failure
is always fatal. NAMESPACE is an internal constant. Add the same
PARALLEL env var (default `true`, env-overridable); falsy demotes the
xargs fan-out to one worker.
* docs/contributing/environment/variables.md: collapse the four
AUTOFORMAT_*/GALAXY_* rows into one `PARALLEL` row covering both
consumers.
* docs/contributing/tools/make.md: refer to `PARALLEL=0` everywhere;
add the lint-galaxy parallelism note.
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Makefile lint integration:
* scripts/install/lint.sh: replace the prior checkmake binary install with
ensure_mbake (mbake is pip-installable, parses our Makefile correctly
whereas checkmake bailed on .ONESHELL: / .SHELLFLAGS: / ifneq).
* scripts/lint/makefile.sh: new runner — `mbake format --check Makefile`
reports formatting drift and missing .PHONY declarations.
* scripts/lint/autoformat.sh: add a two-pass `mbake format Makefile`
(mbake's formatter is not strictly idempotent on the first pass).
* Makefile: split the multi-line `.PHONY:` block into 18 single-line
`.PHONY:` declarations so mbake's parser can validate it; add the
previously-missing `install-python-dev` and `build-dependency` PHONY
targets; add `lint-makefile` aggregate-and-direct target.
* .pre-commit-config.yaml: new make-lint-makefile hook scoped to
`^Makefile$`.
* .github/workflows/lint-makefile.yml: new CI workflow (pip-install mbake
+ make lint-makefile).
* .github/workflows/ci-orchestrator.yml: wire lint-makefile into
lint-gate.needs.
Misc:
* .gitignore: add roles/importer_result.json (tooling artefact written by
galaxy-importer when it runs in roles/).
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* .pre-commit-config.yaml: add 6 scoped lint hooks (lint-action,
lint-ansible, lint-python, lint-shellcheck, lint-markdown, lint-galaxy)
with `files:` patterns so a markdown-only commit no longer triggers
ansible-lint or galaxy-importer. Aggregate make-test, make-test-lint
(md-only branch) and pre-push make-test-external are kept.
* Makefile: drop `lint` from the `make test` dependency chain. Per-file-
type lint stages now live exclusively in pre-commit; running them again
from `make test` would just duplicate work.
* scripts/lint/autoformat.sh: add markdownlint-cli2 --fix and
ansible-lint --fix passes alongside ruff/shfmt/shellcheck.
* roles/svc-db-openldap/meta/main.yml: collapse the wrapped description
into a single line (still under Galaxy's 255-char limit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_inline_script_block_size.py: scans
roles/, tasks/, playbooks/ and .github/ YAML for literal block scalars
(`|` / `|-` / `|+`) following the keys `shell`, `command`, `script`,
`raw` (Ansible) and `run` (GitHub Actions). Counts logical lines —
trailing-`\` continuations collapse to one line, blank lines are
ignored — and fails when a block exceeds 12.
FOLDED scalars (`>` / `>-` / `>+`) are exempt: they are one shell
statement after YAML folding, even if they span many source lines for
readability.
The walk goes through `utils.cache.files.iter_project_files(...)` per
`docs/agents/action/testing.md` so the shared project-tree cache is
reused across the lint suite.
The fail message tells the author exactly how to fix: Ansible tasks
get the built-in `script:` module (collapses the task to one line);
GitHub Actions workflows extract into `scripts/github/<name>.sh` and
call it via `run: bash scripts/github/<name>.sh` (or
`_helpers/scripts/github/<name>.sh` when the workflow checks out the
repo into `_helpers/`).
Extract the pre-existing literal-block offenders so the new lint
ships green:
Ansible (script: module, file under <role>/files/)
web-app-friendica sync_addon.sh
web-app-fider create_admin_user.sh
web-app-fider configure_oidc_provider.sh
web-app-mediawiki install_or_update.sh
web-app-mediawiki composer_install_extension.sh
web-app-xwiki debug_superadmin.sh
GitHub Actions (scripts/github/)
pick_distro.sh
guard_release.sh
resolve_release_inputs.sh
disk_pressure_report.sh
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New Ansible role rendering one Fediwall instance with one or more
configurable walls under https://fediwall.<DOMAIN_PRIMARY>/<slug>/. Each
wall's `wall-config.json` is forwarded VERBATIM from
meta/services.yml.fediwall.walls[*].config — the YAML there is the
single source of truth, the role's template applies no defaults. Two
matrix-deploy variants ship: variant 0 inherits the services.yml
default (one wall, polls all active Mastodon-API-compatible siblings via
the new `lookup('fediwall_active', ...)` plugin); variant 1 overrides
with two explicitly configured walls (default + mastodon-only) so the
per-wall server scoping is observable end-to-end.
End-to-end Playwright spec drives biber's first OIDC login on Mastodon
and ldapauth-mediated form login on Friendica through the same web UI a
human would use — biber's local accounts come into existence lazily on
first SSO login, exactly as in production. The cross-Fediverse test is
data-driven against each wall's `wall-config.json.servers`: a sibling's
post is visible iff its host is listed there.
Cross-cutting fixes that surfaced during the iteration ship alongside:
- utils/cache + utils/templating: type-preserving render path for
single-Jinja-expression scalars in `lookup('config', ...)`.
`_templar_render_best_effort` always coerced its output to `str`, so a
list-returning lookup like `{{ lookup('fediwall_active', 'domains') }}`
collapsed to its Python repr. `_render_scalar` now detects the
`{{ ... }}`-only shape and routes it through the new
`_templar_render_preserve_type` helper, which returns templar's
native value (list/dict/etc.) instead of stringifying.
- web-app-friendica/tasks/04_addons.yml: force a `disable` + `enable`
cycle for enabled addons. Friendica's `addon enable` only fires the
`_install` hooks on the disabled→enabled transition, so an
already-enabled-but-broken state (the addon shows up in `list
enabled` but its `_install`-registered hooks are missing from the
`hook` DB table) silently rejected every ldapauth login at runtime.
Cycling through disable+enable reliably re-registers the hooks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Marks each role's primary entity (`mastodon`, `friendica`, `pixelfed`)
as `shared: true; enabled: true` in meta/services.yml so the variant-
aware service registry resolves them. Consumer roles can now pull any
of the three in via `services.<entity>.enabled: true; shared: true`
(e.g. from a variant) without listing the role under
`meta/main.yml.dependencies` and without committing every deploy of
the consumer role to a heavyweight Fediverse pull-in.
To pass the service-role schema lint each tasks/main.yml now collapses
to a thin wrapper that includes a new tasks/01_core.yml under a
`run_once_<role_slug>` guard, and 01_core.yml ends with the role's
existing setup pipeline. The pre-existing step files are renumbered
to leave 01 free for the canonical entry point:
web-app-mastodon
00_overrides.yml -> 02_overrides.yml
01_setup.yml -> 03_setup.yml
02_wait.yml -> 04_wait.yml
03_cleanup.yml -> 05_cleanup.yml
04_administrator.yml -> 06_administrator.yml
web-app-friendica
01_ldap.yml -> 02_ldap.yml
02_patch_config.yml -> 03_patch_config.yml
03_addons.yml -> 04_addons.yml
web-app-pixelfed
00_bootstrap.yml -> 02_bootstrap.yml
01_administrator.yml -> 03_administrator.yml
02_oidc_mapping.yml -> 04_oidc_mapping.yml
Each role's 01_core.yml wires the renumbered files via include/import
exactly where the previous main.yml did. No behaviour change for
direct deploys (APPS=web-app-<role>) — load_app.yml already gates on
`run_once_<role>`, so the wrapper is invariant under the existing
direct-deploy path. Variant-driven service-registry pull-ins now work
end-to-end for these three roles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`bin/console addon enable` returns rc=0 with empty stderr in two failure
modes: (1) when the Friendica DB is mid-migration ("Database isn't ready or
populated yet" on stderr) and (2) occasionally on a fresh stack where the
command installs nothing — no config entry, no hooks registered. The
second mode leaves ldapauth silently inert and every LDAP login is
rejected.
Add a dedicated DB-ready probe that exercises the same code path as
enable/disable (`addon list all`) and only proceeds once stderr is clean.
Then replace the thin rc-based until with an effective-state check: run
enable/disable, swallow idempotent "already enabled/disabled" errors, and
verify the addon appears (or does not appear) in `addon list enabled`.
Retry the whole cycle — the second invocation reliably registers hooks.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the v1 wait-for-bin/console file-existence probe with a
`bin/console list` invocation, which boots Friendica's full Composer
autoloader and therefore only returns rc=0 once the entrypoint rsync
from /usr/src/friendica/ into /var/www/html has copied every required
vendor file. The v1 probe returned too early: bin/console lands before
vendor/ is complete, so the next invocation failed with
PHP Fatal error: Failed opening required
'.../vendor/composer/../symfony/deprecation-contracts/function.php'
Also adds defence-in-depth retries on the addon-sync loop itself, so
an individual addon command can still absorb a slow final piece of the
rsync or a just-reloaded autoloader.
Refs #161
The Friendica image copies /usr/src/friendica into the /var/www/html
volume via an entrypoint rsync. On fresh-purged deploys the addon sync
task raced that rsync and failed with `OCI runtime exec failed ...
"/var/www/html/bin/console": no such file or directory`. Add a polling
wait (60 retries / 2s) for the console binary so the role is resilient
to the rsync window regardless of image internals.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
docker exec resolves argv[0] against the container's default WORKDIR
instead of the Friendica app dir, so 'bin/console' fails with
"OCI runtime exec failed: no such file or directory". Add --workdir and
invoke console via its absolute path so the addon sync lands
deterministically regardless of the container's default cwd.
- Rename lookup plugin from `docker` to `container`
- Switch all lookups from `lookup('docker', ...)` to `lookup('container', ...)`
- Normalize compose file keys:
- docker_compose → compose
- docker_compose_override → compose_override
- docker_compose_ca_override → compose_ca_override
- Centralize compose path resolution via docker_paths_utils
- Update compose_file_args to always derive paths from DIR_COMPOSITIONS
- Align all roles, templates, and vars with container/compose naming
- Update and rename unit tests accordingly
This prepares the codebase for engine-agnostic container backends
(e.g. Docker ↔ Podman) and removes legacy docker-specific assumptions.
https://chatgpt.com/share/6987a7cc-d374-800f-99ad-bb4c8f92c59e
Replace usages of the get_app_conf filter with the config lookup plugin
across roles and templates. Adjust quoting where required to keep YAML
and Jinja syntax valid. No functional changes intended.
https://chatgpt.com/share/69860678-cef8-800f-8402-7fe6231fdc2d
The TLS lookup has been renamed from `tls_resolve` to `tls` to better
reflect its responsibility as a strict TLS state resolver.
This refactor updates all usages across the codebase, including:
- lookup plugins and internal helpers
- roles, vars, tasks, and templates
- tests, stubs, and error messages
- documentation and changelog references
Behavior, lookup API, and semantics remain unchanged.
https://chatgpt.com/share/6985e635-5a94-800f-b794-937e566ee308
