834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nextcloud's SSRF guard answers every server-side call to an RFC1918
address with 401 'violates local access rules'. integration_mastodon
registers its OAuth app dynamically against the partner, so in the swarm
lab -- where every partner resolves inside 192.168.244.0/24 -- the
integration never connects and the addon spec never leaves Nextcloud.
The switch is a service config entry rather than a hardcoded value, so a
deployment whose partners really do sit on a private network can turn it
on without the role stomping the setting on every deploy. Default is on
in development only; production keeps the guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nextcloud's log_type was never set, so the default 'file' applied and every
application exception went to data/nextcloud.log on the NFS volume. The rescue
diagnostics collect `docker logs` per service, which only ever saw the FPM
access log - so an exception is invisible in CI artifacts.
That cost a diagnosis. In run 30859490335, job 91840410448, the OIDC login
completed (302 /login -> 302 /apps/oidc_login/oidc -> 303 with the code) and
/apps/dashboard/ answered 500. The access log counts 15 x 303 against 15 x 500,
i.e. five addon specs times three attempts, every single authenticated request
failing - deterministic, not flaky. The trace holds only Nextcloud's generic
error page, "More details can be found in the server log", and that log is not
in the artifact.
'errorlog' sends the exception through PHP's error_log to FPM's stderr and from
there into `docker logs`, which the existing collection already captures. No new
collection code and no reads from the NFS volume. 'syslog' and 'systemd' would
both need a socket inside the container.
This does not explain the 500 in that run; it makes the next one explainable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Takes the revert of the rendered system config. The MCP work had added the
proxy service name and its swarm DNS name as trusted domains to
system.config.php.j2, which the revert deletes; both entries move back to
vars/system.yml as trusted_domains 1 and 2, where they lived before aeefb1407.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This reverts commit 41063206a.
It broke every swarm nextcloud deploy. system.config.php.j2 opened with
<?php
{%- macro php_str(value) -%}
and the leading dash strips the newline after `<?php`, so under Ansible's
trim_blocks the file renders as `<?phpreturn [`. Run 30848540165:
ParseError: syntax error, unexpected token "=>", expecting "]"
in /var/www/infinito/system.config.php:2
#0 /var/www/html/lib/private/Config.php(229): include()
Config->readData() includes the file, so occ upgrade, every later occ call and
every php-fpm request died on an uncatchable compile error. The template had
been verified by rendering it with plain Jinja defaults, which do not match the
ones Ansible uses - the check confirmed the escaping and missed the line above.
A one-character fix (`{% macro`) is correct and was verified end to end, but the
commit is reverted rather than repaired, because its premise did not survive
review:
- The failure it targeted is not deterministic. The same task ran 13 times
across the swarm jobs of run 30830377785 and passed 12 times, including a
second fedora job on the same commit and topology. It is an ~8% flake.
- It does not remove the mechanism. 03_upgrade.yml:8 runs
`occ maintenance:repair --include-expensive` unconditionally, and upstream
Repair.php sets the maintenance system value twice with no guard - two
exclusive locks on the same NFS config.php per deploy, unretried.
- The benefit is largely illusory. Config.php:279 var_exports the whole merged
cache back into config.php, so the nine values land in the locked file anyway
on the first maintenance:repair, with a larger payload.
- It traded away occ's exit-code confirmation that Nextcloud accepted the
values, and moved them from the shared NFS volume to a node-local bind.
The remaining exposure is the unretried occ calls in 03_upgrade.yml; a bounded
retry there is the smaller and better-targeted change and is left for its own
commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine sequential `occ config:system:set` calls each open config.php exclusively.
In swarm that file lives on the NFS share behind every replica, and the seventh
call could not take the lock:
In Config.php line 306:
Could not acquire an exclusive lock on the config file
/var/www/html/config/config.php
stdout shows six values written before the abort, so the run left a partial
configuration behind. That failed the swarm web-app-nextcloud 5 job in run
30830377785 while every PLAY RECAP on the other nodes reported failed=0.
The same nine values now come from templates/config/system.config.php.j2, which
rides the additional-config channel the role already uses for redis, ldap and
the rest: rendered to NEXTCLOUD_HOST_CONF_ADD_PATH, mounted read-only, merged by
zzz-infinito.config.php after config.php so its values win. That path never
takes an exclusive lock, and it runs before the stack starts, since all nine
values are statically derived and none needs a live container.
Two behaviours are deliberately not carried over. The --type flags become native
PHP types, which is stricter than the string coercion occ applied. And nothing
confirms the file was accepted any more, where occ's exit code did; a read-only
`config:system:get` would restore that assurance at the cost of an occ call, and
is left out until someone wants it.
php_str() escapes backslashes and single quotes, matching the idiom already used
for the redis password in memcache.config.php.j2. Without it a quote in any
value would be a PHP syntax error, and include_once would drop every additional
config file, not just this one.
vars/system.yml and 02_manager_ops/05_system_config.yml are gone; the later task
files shift down one number so test_task_file_numbering stays satisfied.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace usages of the get_app_conf filter with the config lookup plugin
across roles and templates. Adjust quoting where required to keep YAML
and Jinja syntax valid. No functional changes intended.
https://chatgpt.com/share/69860678-cef8-800f-8402-7fe6231fdc2d
- Mount includes.php read-only instead of docker cp
- Ensure include loader starts with proper PHP header
- Add overwriteprotocol, overwritehost, and trusted_proxies via system config
- Simplify env handling for protocol and trusted proxies
- Remove redundant permission and copy tasks
https://chatgpt.com/share/697f7aff-859c-800f-bfaa-0d4c2378fd85
This commit restructures systemctl service definitions and category mappings.
Motivation: Alarm-related bugs revealed inconsistencies in service and role handling.
Preparation step: lays the groundwork for fixing the alarm issues by aligning categories, roles, and service templates.
