834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The build script is copied into the CDN volume directory, and this makes its
existence explicit at the point of use rather than relying on an earlier task
having created it as a side effect of creating its child.
466f04503 replaced the container-run publish with a compose build service and
started copying build_assets.sh into <volumes>. ansible.builtin.copy does not
create parent directories, and nothing else creates <volumes> at that point:
sys-stk-full, which does, is included afterwards on purpose, because
meta/volumes.yml makes <volumes>/pkgsrc a bind source and
<volumes>/build_assets.sh a config source of the build service - both have to
exist before the stack is rendered.
<volumes> was therefore only ever created as a side effect of
utils/stage_manifest.yml:2-6, inside the loop that `manifest_role in groups`
filters. Run 30856149821, job "📖 Instructions": both candidates skipped, the
loop ran zero times, and the copy failed with
msg: Destination directory /opt/compose/cdn/volumes does not exist
The new task targets CDN_ASSET_STAGE rather than directories.volumes. That is
<volumes>/pkgsrc, so creating it with parents covers the build script's parent
AND the cdn_pkgsrc bind source. Aiming at directories.volumes would fix only the
task the log named: with no matched manifests, pkgsrc is missing too, and while
the compose daemon auto-creates a missing bind source, swarm does not.
The failure is not deterministic across reruns. cli/meta/ci/guide_select.py:70
picks the test role with random.choice, and of the three roles that require cdn
only web-app-dashboard ships a package-lock.json - a dashboard roll makes the
guard match and hides the bug. Pin inputs.role=web-svc-css to reproduce.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replace the ansible container-run plus copy publish with a proper compose
build service deployed via the role's own stack. The one-shot writes the
npm mirror straight into the node-local served CDN root that openresty
reads, so there is no DiND bind-layer mismatch and no separate publish step.
- meta/volumes.yml: build mounts (served root rw, staged manifests ro,
build script as a docker config)
- templates/compose.yml.j2 + env.j2: one-shot build service (restart no,
swarm condition none), CDN_WEB_ROOT and ROLES_DIR from the mount targets
- meta/services.yml: placement manager, replicas 1, resources, deploy modes
- meta/networks.yml: local subnet (the role now ships a stack)
- meta/variants.yml: internal and external flavor variants
- meta/tests.yml + test.env.j2 + files/test/test.sh: flavor-branching e2e
- tasks: internal flavor deploys the stack via sys-stk-full; drop the
container-run and ansible-copy publish; remove the unused dev-git include
- vars: pin CDN_FLAVOR to the role id (scope-safe) and compute host paths
directly (avoid the volume-lookup double-template)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The asset-mirror loop derived the staging dir and gate from a task var named
role_name, but role_name is Ansible's reserved role-magic variable pinned to the
executing role (web-svc-cdn); the vars override was silently ignored. So every
found package-lock.json (web-app-dashboard, web-app-littlejs) evaluated the gate
as 'web-svc-cdn in groups' and staged into CDN_ASSET_STAGE/web-svc-cdn/, the
second overwriting the first. No per-role dir was created, so keycloak-js (only
in dashboard's lock) was never mirrored and the CDN served 404 for
/npm/keycloak-js@24.0.5/dist/keycloak.js, breaking dashboard OIDC and the shared
universal-logout e2e across every app.
Rename the loop var to manifest_role so the override takes effect: each manifest
stages under its real role dir and the gate 'manifest_role in groups' mirrors
only deployed asset providers. Drop the redundant CDN_FLAVOR default (always set
in vars/main.yml).
Verified: build_assets.sh over dashboard's manifest produces
npm/keycloak-js@24.0.5/dist/keycloak.js; an ansible repro shows role_name stays
'web-svc-cdn' while manifest_role resolves to 'web-app-dashboard'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mirror task staged package manifests for every role in the repo that ships
a package-lock.json, so a penpot deploy also pulled in unrelated apps such as
web-app-littlejs (nothing references its assets) and coupled every app's deploy
to their npm builds. Move the three staging steps into tasks/utils/stage_manifest.yml
and drive them from a single looped include gated on 'role_name in group_names',
so only the apps and dependent services actually deployed here are mirrored.
Collapses the per-task loop/when duplication to one include.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The asset-mirror task copied package-lock.json / package.json to
<CDN_ASSET_STAGE>/<role>/ but ansible.builtin.copy does not create the
parent directory, so every deploy failed with 'Destination directory ...
does not exist'. web-svc-cdn runs in every stack, so all swarm priority
jobs failed at deploy. Ensure the per-role staging dir exists first; the
<role> layout matches build_assets.sh globbing ${ROLES_DIR}/*/package-lock.json.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_task_file_numbering.py: numbered task
files use exactly two leading digits, a numbered *_core.yml must be
00_core.yml, per directory the numbers are unique/consecutive and start
at 01 (00 with a core file), numbered includes appear in ascending
order, and shared helpers live in tasks/utils/ instead of the numbered
chain.
Apply it across every role: rename NN_core.yml to 00_core.yml with a
consecutive down-shift of the siblings, move letter-suffix children into
a subdirectory named after their parent task, relocate helpers included
from more than one place into tasks/utils/, and update every reference
(include_tasks, path_absolute, .ansible-lint-ignore, requirement docs).
Legacy narration comments in the touched files are deleted; the upstream
xwiki.properties reference block opts out via nocheck. The unused
MAILU_IP6_PUBLIC var (only ever named in now-removed comments) is dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5