Commit Graph
47 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5.5 a5de245bf9 fix(ci): let the ZFS data-root pools hand freed blocks back to their images
docker_dataroot.sh backs each node's ZFS pool with a sparse loop image
and created the pool without autotrim. ZFS never discarded freed blocks
to the loop device, so the image only grew: in WordPress#0 of run
40625679270 the runner still held 118G after the teardown had removed
every container ("Containers: 0") while only 47G of files remained, and
the pools held ~38G of live data. The step had already aborted at
"DISK EXHAUSTION IMMINENT (<6144M free on /)" with 139G of 145G used
(rc=75). The other ZFS swarm jobs of the run show the same df-minus-du
gap (wordpress1 73G, odoo1 73G, magento0 82G, weblate1 55G, gitlab2
44G); ext4 and btrfs jobs show none.

The pool is now created with -o autotrim=on, so ZFS discards freed
ranges to the loop device, which punches holes into the image file.

test_docker_dataroot runs the script's zfs path against stubs of every
host tool and asserts the single zpool create carries -o autotrim=on;
against the previous script that assertion fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 23:07:23 +01:00
kevinveenbirkenbachandClaude Opus 5.5 8a0e858f12 refactor(ci): group the rescue helpers under scripts/tests/deploy/utils/rescue
2bf329cd3b put rescue_window.sh next to rescue_index.sh, and
test_prefix_grouped_files rejects two siblings sharing the `rescue`
prefix (quality-high pass 67). Both move to utils/rescue/ as index.sh and
window.sh, their tests to the mirrored tests/.../utils/rescue/, and every
caller follows: dedicated.sh, the swarm teardown in 00_one.sh, the
workspace workflow, and the swarm and diagnostics READMEs.

The budget-kill drill gave the stand-in a 1 s budget; under the full
parallel unit run the SIGTERM arrived before its EXIT trap was installed,
so nothing was captured and the test failed. The two kill drills now use
KILL_BUDGET = 3 s; their assertions are unchanged. A full `make test-unit`
passes them under load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 20:01:54 +01:00
kevinveenbirkenbachandClaude Opus 5.5 2bf329cd3b fix(ci): fit the rescue capture into the kill grace of a budget kill
distros.sh stops a distro at its budget with `timeout -k 60`: SIGTERM at
the deadline, SIGKILL 60 s later. dedicated.sh's cleanup trap then ran
the in-container capture under `timeout 1500` (266 s in Baserow#0 of run
40625679270) and the host capture under `timeout 900`, so the SIGKILL
always landed mid-capture, the snapshot was never copied out and the
rescue-diagnostics artifact was lost. The inner `timeout` also sat in its
own process group and outlived the kill, holding the step open.

- distros.sh keeps the grace in one place, KILL_GRACE_SECONDS, and
  exports INFINITO_CI_DISTRO_KILL_EPOCH next to the deadline epoch.
- scripts/tests/deploy/utils/rescue_window.sh decides per trap step:
  past the deadline, the step gets half of the time left until the
  SIGKILL (at least 1 s) and an end epoch; before it, the nominal
  timeout and no end epoch. Inside the trap `$?` is 0 on a budget kill,
  so the deadline epoch is the only reliable signal.
- dedicated.sh runs both captures through `timeout --foreground` with
  that window, keeping them in the trap's process group, and passes the
  end epoch as RESCUE_DEADLINE.
- utils/diagnostics/container.py caps every command at RESCUE_DEADLINE,
  writes `[skipped: past RESCUE_DEADLINE]` for what no longer fits and
  hands the deadline to nested captures. Without the variable nothing
  changes, so the swarm and workspace callers behave as before.

-k 60, the distro budget and every timeout-minutes are unchanged. Drill
tests replay the trap with the real helper and container.py: a budget
kill ends with rc 124 and the snapshot copied out, the old shape ends
with rc 137 and nothing copied, an early failure keeps the full capture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 19:11:15 +01:00
kevinveenbirkenbachandClaude Opus 5 677a63931c Merge branch 'feature/network/multi-network-serving' into feature/store-i18n
Resolves 68 conflicted files. The governing rule was our structure with their
network semantics: the tor boolean becomes a network mode (clearnet | tor |
multi) throughout the variant layer, while this branch's architecture and mesh
axes, its extracted modules and its i18n content stay.

Where the two sides covered different ground, both are kept rather than one
picked:

- ca_injected answers a ca_client on the deployment's TLS mode OR on an SSO
  issuer that resolves self-signed. 3a3e9ed2c9 records why: an issuer is not
  an object store, so keeping only one side drops the other's coverage.
- get_merged_domains keeps the loud refusal when svc-net-tor is deployed
  without a provisioned node, now gated on their tor_serves_host; their
  node_onion_of returns an empty string there, which would have served every
  onion-exclusive app on clearnet in silence.
- sso_cookie_domains stays the single source for oauth2-proxy's cookie scope.
  Their per-network branch omitted the issuer domain on the onion arm, which
  breaks the callback leg.
- The clearnet CSP batch carries --clearnet-timeout again; their batch dict
  hardcoded zero while the role still passes the flag.
- The dashboard keeps its i18n content dir, image, version and design
  resolution; their side reduced the logo sources to the bare service
  provider values.

Their side wins where it is measurably better: the onion mirror rewrite is
gated per vhost rather than per app, the Playwright onion decision reads the
pass's own domain instead of the role's canonical one, and the matomo
reinjection and dashboard config render per network sibling.

Eight files were broken by the merge without ever carrying a conflict marker,
because git marks only the lines both sides touched:

- utils/cache/domains.py lost the onionize block that sat between two hunks
- roles/sys-front-inj-all/templates/body_filter.lua.j2 called it
- web-app-bluesky, keycloak sso_proxy: the get_entity_name filter was renamed
- trigger/__main__.py, playwright_coverage.py and its test: the deleted tor
  module
- cli/administration/deploy/ci/timings.py read Label.tor
- plugins/filter/csp_filters.py called _aligned_url with four of six arguments
- tests/unit/python/utils/cache/test_domains.py referenced a renamed class
- tests/.../sys-ctl-hlth-csp/files/test_script.py mocked a function main no
  longer calls

Verified: the variant layer (155 tests, 65 subtests), ca_injected (18),
the CI layer (175), the domains cache including the restored guard (32),
retrigger inputs (9), timings and the CSP health script (53). make
quality-high clears autoformat; its remaining findings are tracked separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-23 09:53:41 +02:00
kevinveenbirkenbach d0b45f8b2d Merge branch 'main' of github.com:infinito-nexus/core into feature/store-i18n
# Conflicts:
#	.github/workflows/call-images-build-ci.yml
#	.github/workflows/call-test-deploy.yml
#	.github/workflows/call-test-install-pkgmgr.yml
#	.github/workflows/call-test-workspace.yml
#	roles/svc-bkp-local-2-device/README.md
#	roles/svc-storage-nfs-client/README.md
#	roles/web-app-pixelfed/meta/services.yml
#	scripts/tests/deploy/act/workflow.sh
#	tests/unit/python/utils/recovery/test_base.py
2026-10-20 20:05:15 +02:00
kevinveenbirkenbach feb5630f19 Merge remote-tracking branch 'origin/main' into feature/network/multi-network-serving 2026-10-20 17:14:44 +02:00
kevinveenbirkenbachandClaude Opus 5.5 a6c3c1cc91 fix(apparmor): keep docker-default loaded when tearing AppArmor down
`security-apparmor-teardown` ran `aa-teardown`, which unloads every
profile including dockerd's in-memory `docker-default`. dockerd only loads
that profile when it starts, so every container start afterwards that
requests it fails until dockerd restarts. Since #1067 runs the teardown on
the GitHub runners, BuildKit failed each RUN of the swarm node image three
times (run 39332343490, 39 of 40 swarm jobs):

    runc run failed: unable to start container process: error during
    container init: unable to apply apparmor profile: apparmor failed to
    apply profile: write .../attr/apparmor/exec: no such file or directory

The teardown now removes each loaded profile by name through securityfs
and skips `docker-default` and child profiles, which leave with their
parent. Dev hosts gain the same: no dockerd restart is needed after
`make environment-bootstrap`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-20 12:11:14 +02:00
kevinveenbirkenbachandClaude Opus 5 1c950d0c5e fix(ci): widen the headroom test margin again and shorten an overlong task name
The full gate failed test_a_distro_needs_the_slowest_run_plus_headroom_to_start on a quiet machine, so the load is the gate own parallel workers rather than anything beside it. The branch under test needs d < B < 2.15*d for a measured first duration d, the rule 040955ebf2 wrote down when it widened the same fixture from sleep 2 / 3s to sleep 4 / 7s. At 4 / 7 the window admits a first iteration of 4 to 6 seconds, three seconds of slack, sized then against the single second CI had overshot by. Under the gate it overshoots further, takes the budget exhausted branch and never prints the line the test greps for. At 10 / 19 the window admits 10 to 18 seconds, nine seconds of slack. The assertion and the branch are unchanged and the suite pays six more seconds.

The second hunk shortens the Odoo reload task name to 60 characters; at 83 it broke the 80 character convention the name lint enforces.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-17 16:04:09 +02:00
kevinveenbirkenbach f8311d0780 Merge branch 'main' into feature/network/multi-network-serving 2026-10-16 22:36:07 +02:00
kevinveenbirkenbach de0c4e0e00 Merge branch 'feature/designer' into feature/store-i18n 2026-10-16 20:07:01 +02:00
kevinveenbirkenbachandClaude Opus 5.5 891d46d589 fix(ci): time the DNS53 sampler probes from /proc/uptime
The 🐳 Workspace (ubuntu:latest) job on main (run 38879556215, 1b36ef6120)
failed test-unit-python in the hook's second make test:

    AssertionError: Regex didn't match: '^ localhost=ok/\d+ms$'
    not found in ' localhost=ok/-1612931482679767779ms'

ubuntu:latest is Ubuntu 26.04, whose date is uutils coreutils. uutils
0.7.0 ignores the field width in %3N and prints all nine nanosecond
digits, so `date +%s%3N` yields a 19-digit value instead of epoch
milliseconds. Against main's sampler, uutils 0.7.0 date reports
localhost=ok/7081859ms, and an unpadded nanosecond value turns the
difference negative.

answer_once now takes both timestamps from /proc/uptime, which the
kernel prints with two decimals whatever date the image ships. The
sampler already reads /proc/net/udp, so it was Linux-only before. The
resolution is 10 ms; the same probe reports 10 to 20 ms with uutils
date on PATH.

The new test puts a date on PATH that does not understand %3N; main's
sampler dies on it, the fixed one does not call date for the latency.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 14:15:25 +02:00
kevinveenbirkenbachandClaude Opus 5 4a08d58e55 test(cache): point the CA test at the path the script was moved to
230d9a1e5e moved scripts/docker/cache/package-frontend/ca.sh to roles/svc-cache-package/files/shell/ca.sh as a pure rename. On origin/main the script stayed where it was, and 2756fd56f1 changed it there and added this test. The merge took the test without the move, so every case ran bash against a path that no longer exists and failed with exit 127.

Only the path changes. The script it exercises is the same file the rename produced, and every other PROJECT_ROOT path under tests/ was checked against git ls-files: registry-ca.sh is still at its old location and its test is correct, cert-gen.sh moved along but has no test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-16 00:22:13 +02:00
kevinveenbirkenbach 810c8bc413 Merge branch 'main' into feature/store-i18n 2026-10-15 23:51:34 +02:00
kevinveenbirkenbachandClaude Opus 5.5 2756fd56f1 fix(cache): install CA anchors where every update-ca-trust family reads them
The Fedora and CentOS Stream workspace jobs failed TLS to pypi.org,
registry.npmjs.org and repo.packagist.org (CERTIFICATE_VERIFY_FAILED,
curl error 7). package-frontend/ca.sh put the frontend CA into the Arch
anchor dir /etc/ca-certificates/trust-source/anchors, which p11-kit on
the RedHat family never reads.

Probing for the anchor dir does not fix it: with-ca-trust.sh creates
both /etc/pki/ca-trust/source/anchors and
/etc/ca-certificates/trust-source/anchors on every update-ca-trust host,
so the probe in registry-ca.sh already picks the RedHat dir on a reused
Arch runner. Both scripts now install into both dirs, rebuild only when
one of them changed, and take CA_ROOT so unit tests run them against a
fake root.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 23:45:53 +02:00
kevinveenbirkenbach 1da2cef95e Merge remote-tracking branch 'origin/main' into feature/network/multi-network-serving
# Conflicts:
#	plugins/lookup/email.py
#	tests/unit/python/roles/web-app-prometheus/test_blackbox_targets.py
2026-10-15 16:45:18 +02:00
kevinveenbirkenbachandClaude Opus 5 1e9936a9b1 fix(tests): point the distro loop test at the function that composes the parent image
efaed0dbb9 dropped pkgmgr_image from utils/distros.py and left this test importing it at line 23 and calling it at line 91, so the module raises ImportError at collection. pytest reports that as '7581 passed, 1 error' while test-unit-python exits 1, which is why the count looks healthy and the target is red.

The test's intent outlived the refactor. INFINITO_PARENT_IMAGE still exists and default.env documents it as composed by utils/env/handlers/infinito/parent_image.py; only the oracle the assertion compared against went away.

base_image(distro, owner, tag) at utils/distros.py:116 is the successor with a compatible signature, and it is what that handler itself calls - parent_image.py:20 is 'return base_image(distro, owner=owner, tag=tag)'. The assertion now checks the value against the function that produces it instead of against a parallel copy that could drift.

Collection goes from ImportError to 11 tests collected. The body shells out through distros.sh for every distro, so it gets its real exercise in the next full gate rather than here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-15 14:57:02 +02:00
kevinveenbirkenbach 6d68498b96 Merge branch 'feature/designer' into feature/store-i18n 2026-10-15 13:29:03 +02:00
kevinveenbirkenbach b1d38f49d6 Merge pull request #934 from infinito-nexus/fix/act/scope-outer-teardown
fix(act): remove only the own worktree's act containers
2026-10-15 10:51:48 +02:00
kevinveenbirkenbach b637742f96 Merge remote-tracking branch 'origin/main' into fix/env/distro-derived-keys/794 2026-10-15 01:07:53 +02:00
kevinveenbirkenbachandClaude Opus 5.5 87c9a5f41b test(act): mark the docker stub log read as outside the file cache
The teardown test reads the call log its docker stub writes into a
temporary directory. test_no_uncached_file_reads flagged that read in
CI; the path-keyed read cache does not fit a file written per run, so
the read keeps Path.read_text with the cache-read opt-out, as the other
stub-log tests do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-15 01:07:50 +02:00
kevinveenbirkenbachandClaude Opus 5 ac28c0b378 fix(ci): keep the backfill floor on the path the architecture gate takes
The main merge joined two correct rules into one broken script. The
architecture gate republishes the highest version when :latest misses an
architecture, because the published install instructions pull the image
untagged and only a release of that version rewrites latest. The backfill
floor refuses any tag below v9.0.2, which cannot build from its own tree.

The gate returned before the floor was ever consulted, so an incomplete
:latest would have asked the backfill for a tag it cannot build. Both
paths now ask below_floor, which also replaces the comparison the loop
carried inline.

The picker's own test could not have caught it: its docker stub answers
tags without emitting a manifest, and architectures.sh reads an empty
manifest as every architecture missing, so the gate fired on every case
and printed the newest tag. The stub now serves the whole pool, which is
the state the floor cases assume anyway.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-14 14:33:45 +02:00
kevinveenbirkenbachandClaude Opus 5.5 46373c1ba3 fix(act): remove only the own worktree's act containers
down_act_outer.sh removed every container named act-*, so starting or
releasing a swarm drill in one worktree killed the act container of a
drill another worktree had running.

workflow.sh now labels act's job containers with
infinito.worktree=${INFINITO_CONTAINER} through the default
ACT_CONTAINER_OPTIONS, and down_act_outer.sh removes only containers
carrying the own label. Both scripts refuse to run without
INFINITO_CONTAINER, so the filter never widens to every act container;
make exports BASH_ENV, which loads it from .env.

Worktrees without a custom.env share the default container name and
therefore still share one label among themselves.

Closes #933

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 13:50:07 +02:00
kevinveenbirkenbach e6b88662be Merge pull request #919 from infinito-nexus/fix/ci/release-backfill-floor
fix(ci): stop the release backfill below the oldest buildable tag
2026-10-14 10:02:04 +02:00
kevinveenbirkenbachandClaude Opus 5.5 fd9286b24f test(lint): mark the stub call log as written by the test
test_no_uncached_file_reads rejects Path.read_text in tests unless the file is
written in the test itself. The mermaid script test reads the call log its
stubs append to during the run, so that read carries `# nocheck: cache-read`.
The mermaid cli test is formatted with ruff.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 01:32:35 +02:00
kevinveenbirkenbachandClaude Opus 5.5 ee8d2bd983 test(env): check each distro's parent image against its own reference
The loop test only required three different parent images for three distros,
which a swap between two of them would still pass. Each iteration now also
records the owner and tag it sees, and its image must equal
pkgmgr_image(distro, owner, tag). With the loader of main the test fails on
arch with pkgmgr-debian:stable.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 01:32:24 +02:00
kevinveenbirkenbachandClaude Opus 5.5 1d970fe5bd fix(env): keep shellcheck and the cache-read lint green on the distro switch
The loader copies INFINITO_DISTRO into a loader-local variable before it
compares the value with the one in .env. With INFINITO_DISTRO read directly in
that condition, `shellcheck -x scripts/meta/resolve/apps.sh` reported SC2153
for INFINITO_DISTROS, which apps.sh takes from the sourced .env; with the copy
it reports nothing, as on main.

The loader test checks that .env is unchanged with filecmp against the file it
was copied from, instead of reading it twice with Path.read_bytes, which
test_no_uncached_file_reads rejects.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 01:32:24 +02:00
kevinveenbirkenbachandClaude Opus 5.5 d412b23409 fix(ci): stop the release backfill below the oldest buildable tag
The release backfill dispatches the highest version tag whose images are
missing. Every tag from v15.0.0 down to v9.0.2 has its images now, so since
2026-10-05 it picks v9.0.1 every day, and each of the nine runs failed in
"Build missing CI images / Image" at
`entry.sh: line 19: infinito: command not found`. The v9.0.1 tree cannot
build its own image; c680be12ea fixed that build in v9.0.2, and a tag cannot
take the fix.

highest_release_tag.sh now stops at v9.0.2: no tag below it is picked, so the
backfill reports nothing missing instead of retrying v9.0.1. v9.0.0 has the
same Dockerfile, Makefile and scripts/docker as v9.0.1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 01:32:05 +02:00
kevinveenbirkenbachandClaude Opus 5.5 c8cb7fda8c fix(lint): provision the mermaid browser in install-lint, not in the parallel lint run
make lint runs install-lint once and then dispatches its fifteen targets
in parallel. scripts/lint/mermaid.sh provisioned chrome-headless-shell
inside that parallel phase with `npx puppeteer browsers install
--install-deps`, whose `apt-get satisfy` carries no lock timeout. In a
fresh stack container another target held the apt lock, the shared
libraries stayed missing, the script only warned, and all 289 Markdown
files then failed on libnspr4.so (#795).

utils/install/lint/mermaid_cli.py now provisions the browser right after
mmdc, inside the serial install-lint step, as root with --install-deps
and as a user without it, because puppeteer refuses to install libraries
without root. A failed install deletes the chrome-headless-shell cache
folder and retries once under a 900 second timeout per attempt; a cut-off
download otherwise leaves a folder without its executable, and puppeteer
refuses every later install of that build. The install-lint stamp counts
as fresh only while the browser executable exists, so a recreated
container provisions it again.

mermaid.sh no longer calls apt-get or npx. It renders one minimal diagram
first and, when that fails, exits with mmdc's output instead of reporting
every Markdown file.

Verified: tests/unit/python/scripts/lint/test_mermaid.py stubs mmdc,
npx and apt-get. With a browser that cannot load libnspr4.so the script
exits 1 with that line, before any Markdown file renders, and mmdc is the
only tool it calls. Against main's mermaid.sh the same test fails: it
calls apt-get and npx and reports the files one by one. The installer and
stamp cases pass, shellcheck is clean.

Not verified: a full make lint in a recreated stack container. The race
reproduced in one of two runs in the issue.

feature/store-i18n changed the provisioning in mermaid.sh with 2f5f52bd1a
and dc67bce5f0 (timeout, deleting a cut-off browser). Both behaviours
moved into mermaid_cli.py here, so its merge takes this version of
mermaid.sh.

Closes #795

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:31:36 +02:00
kevinveenbirkenbachandClaude Opus 5.5 1079f48feb fix(env): derive the image and container names from the distro under test
A compose deploy job labelled centos or arch built its stack image from
pkgmgr-debian and ran it as infinito_nexus_debian (#794). distros.sh
exports INFINITO_DISTRO per iteration and sources load.sh again, but the
loader returned early because INFINITO_ENV_LOADED and the .env mtime were
unchanged. A full reload would not have helped either: the debian values
were already exported, so the restore step kept them as caller intent.

load.sh now records the distro it loaded for. Another INFINITO_DISTRO
defeats the early return and drops INFINITO_CONTAINER,
INFINITO_PARENT_IMAGE and INFINITO_IMAGE, the three keys the env handlers
derive from the distro. When the requested distro also differs from the
one .env was generated for, the loader sources a set that
`python -m cli.meta.env --output` generates for it in a private temp
directory and leaves .env untouched, so a local multi-distro run does not
switch the checkout's distro. A caller value set for the loaded distro
still wins.

Verified: tests/unit/python/scripts/meta/env/test_load.py and the new
distros.sh case pass. Against main's loader both fail with the symptom of
the issue: INFINITO_DISTRO=centos next to infinito_nexus_debian and
pkgmgr-debian:stable, and distros.sh handing debian, arch and centos the
same debian parent. shellcheck is clean.

Consequence: CI jobs labelled arch, ubuntu, fedora or centos now build
and deploy that distro, so a failure that shows up there was masked
before.

feature/store-i18n carries 3f973eb137, which resolves the parent image
for scripts/image/push.sh on its own, and 05e4dbafcb, which inserts the
INFINITO_INSTANCE slot check into the block this commit restructures. The
merge keeps the slot check ahead of the distro check.

Closes #794

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 21:30:48 +02:00
kevinveenbirkenbachandClaude Opus 5.5 7fa3400cc6 Merge remote-tracking branch 'origin/main' into feature/network/multi-network-serving
Conflicts resolved by keeping both sides' intent:

- call-test-deploy.yml: main's re-enabled deploy concurrency group, keyed
  on the network axis instead of the removed tor axis.
- sys-svc-proxy: main's shared headers/forwarded.conf.j2 now carries the
  branch's per-vhost semantics (X-Forwarded-Host, TLS port and state read
  from the vhost domain), so html, ws, upload and the Prometheus probes
  all answer per network sibling.
- plugins/lookup/email.py: main's provider-agnostic mail lookup asks the
  network registry instead of testing the .onion suffix, as the branch
  does everywhere; container_extra_hosts.py likewise. The provider host's
  NETWORK_MODE now decides whether it is served onion-first, since a Tor
  node defaults to multi and keeps clearnet canonical.
- workspace common.sh: main's INFINITO_DOMAIN fallback without the
  duplicated no-op line.
- README.md and requirement 035: both sides' rows (Wazuh plus main's
  Pi-hole and Stalwart entries); adapter_candidate count 39.

Adapted to main where the branch's rules now reach main's code:

- six tests from main read the primary domain from default.env instead
  of writing it out, as the hardcoded-domain lint requires;
- the chutney worker-check test passes ONION_SUFFIX from the network
  registry, which the branch's svc-net-tor test script requires;
- web-app-wazuh moves its local http ports to 8096/8097 (8094/8095 now
  belong to web-app-stalwart) and sizes the indexer heap through
  service_memory_mb, the same fixes #414 carries;
- web-app-wazuh's README cosmos follows Stalwart as the mail provider.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 00:14:42 +02:00
kevinveenbirkenbachandClaude Opus 5.5 e31b9557d2 fix(env): keep the host .env when the runner container runs make setup
The repository is bind-mounted into the infinito container, and every deploy
runs `make setup` there, which depends on `dotenv`. The generator therefore
rewrote the host's .env with the values the container sees: the host lost
INFINITO_RUNNING_ON_GITHUB=true that way, and the guide replay pulled
alpine:latest past the GHCR mirror.

compose.yml now marks the infinito service with INFINITO_DOTENV_HOST_OWNED,
and `python -m cli.meta.env` keeps an existing .env while that marker is
"true". A missing .env is still generated, and `make dotenv-force` (env -i)
still rewrites it. The key stays out of the env registry: only compose.yml
sets it, it is no configuration value, and a .env entry could freeze the
host .env.

test_distros now also strips INFINITO_ENV_LOADED. load.sh returns early
while that guard is set and the mtime of .env is unchanged, so the test only
reloaded the budget keys it strips because the container kept rewriting .env.

Verified: unit tests cover a kept .env, a missing one being generated, and a
rewrite without the marker. Local compose deploys print "Kept .env: the host
checkout owns it" where they printed "Wrote .env" before, the host .env
checksum stayed the same across a deploy, and the four keys holding host
paths resolve inside the container under the same paths.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-12 18:35:22 +02:00
kevinveenbirkenbachandClaude Opus 5.5 7c7b895ae9 fix(ci): match the privileged run of a fork PR by its number as well
The wait job picks the newest privileged run on the head commit of the
PR. Two fork PRs that share a head repository, branch and commit but
target different bases each start such a run, and the commit alone cannot
tell them apart. A failed run of the other PR would then end the wait of
this one while its images are still being built.

The run name of the workflow carries the PR number, so the match now
requires it next to the head commit. A run without that name matches
nothing, and the wait does what it does whenever no run is found: it
keeps polling for the images until they exist or the wait times out.

Raised by the CodeRabbit review of this PR.

Refs #784

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 12:07:14 +02:00
kevinveenbirkenbach 8e9fa797fe Merge remote-tracking branch 'origin/main' into fix/ci-fork-image-wait/784 2026-10-05 03:13:43 +02:00
kevinveenbirkenbachandClaude Opus 5.5 69fceafb5d fix(ci): limit the fingerprint shortcut to branches of this repository
The duplicate check accepted any open PR whose head branch started with
the dedupe prefix and ended in the fingerprint of the fresh proposal. The
listing also returns PRs from forks, and the fingerprint can be derived
from public data, so a fork PR with such a branch name would have made the
bot skip its own update PR without carrying the change.

The shortcut now only applies to PRs whose head branch lives in this
repository, which only accounts with write access can create. A fork PR
is still compared by the content of its head, as before.

Raised by the CodeRabbit review of PR #781.

Refs #780

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 03:06:57 +02:00
kevinveenbirkenbach f6228580be Merge remote-tracking branch 'origin/main' into fix/ci-fork-image-wait/784 2026-10-04 22:18:20 +02:00
kevinveenbirkenbachandClaude Opus 5.5 3ab0153856 fix(ci): find the privileged run of a fork PR by its head commit
The wait for fork CI images never found the privileged workflow run it
watches. It selected a run only when the run's pull_requests array held
the PR number, and for a run whose head repository is a fork the API
returns that array empty. The job only runs for fork PRs, so the lookup
matched in no case it exists for.

The early exit for a failed privileged run was therefore unreachable.
When that run failed, the job kept polling until its timeout of 1980
attempts and held a runner for about five and a half hours.

The lookup now asks the API for the runs of the PR head commit and
selects by head_sha. That also replaces a paginated listing of every run
of the workflow with one small request per attempt. Without a known head
commit the match by PR number stays.

Refs #784

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 20:55:54 +02:00
kevinveenbirkenbachandClaude Opus 5.5 53f71d85c8 fix(ci): keep waiting for fork images when a run lookup fails
The job that waits for the CI images of a fork PR ended on the first
failed GitHub API call. The lookup of the privileged workflow run is
assigned with run_json="$(find_matching_run)" under set -e, so a failing
gh ended the script, although the loop is built for 1980 attempts. PR #782
went red that way: GitHub answered one request with HTTP 504 in attempt
31, while the privileged run was still queued.

A failed lookup now counts as "no information in this attempt" and the
loop goes on waiting for the images, which are its success criterion.
Thirty failed lookups in a row end the job with a message that names the
cause, so a broken token does not hold a runner until the timeout.

The test runs the script against a gh that fails every lookup. The
previous script fails both cases at the first lookup.

Closes #784

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 20:55:54 +02:00
kevinveenbirkenbachandClaude Opus 5.5 9bdf33b89d fix(ci): treat an edited update branch as a duplicate of its own bump
The update bot recognised an open PR as a duplicate by hashing the blobs
of the files its commit changed, read at the PR head. A follow-up commit
that edits one of those files changes the hash, so the next run opened a
second PR with the identical bump: PR #778 next to PR #775 for
web-svc-collabora, after a fix had edited the bumped services.yml.

The branch name of a bot PR ends in the first seven characters of the
fingerprint of the change the bot proposed. An open PR whose branch ends
in the fingerprint of the fresh proposal now counts as a duplicate before
its head is compared. The head comparison stays as the second criterion.

The test runs pr.sh in a throwaway repository against a stubbed gh. It
fails on the previous script with exit 1, because that one goes on to
push.

Closes #780

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 14:36:57 +02:00
kevinveenbirkenbachandClaude Opus 5.5 e2beab9f1c test(domains): read the primary domain from default.env in every test
The hardcoded-primary-domain lint exempted tests/ and scripts/tests/, so test
fixtures kept their own copy of the domain and would drift from
INFINITO_DOMAIN the moment it changes. The lint now scans both trees, and a
unit test pins its scope: only default.env, Markdown, inventories/, .claude/
and build/ may name the domain.

Python tests take the value from default_domain_primary(), and the workspace
helper builds its dashboard and Matomo URLs from INFINITO_DOMAIN once the repo
.env is loaded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 22:53:54 +02:00
Alejandro Roman 4b9a834546 Merge remote-tracking branch 'upstream/main' into feature/role-stalwart-mail-swarm 2026-09-21 03:25:20 +04:00
Alejandro Roman e6596d51f2 fix(stalwart): add the fix PR into main PR 2026-09-09 00:21:58 +04:00
Alejandro Roman 8af17a6c3f fix(stalwart): fix TOr and testing properly, local pass, commit issues 2026-09-04 02:43:39 +04:00
kevinveenbirkenbachandClaude Opus 5 56b8f1716b fix(diagnostics): let the in-zone probes read the drop-in that holds the zone
d14354e892 added both probes reading only /etc/dnsmasq.conf, but svc-net-tor writes its onion zone to /etc/dnsmasq.d/tor-onion.conf and leaves the main file with nothing but a conf-dir pointer. Every Tor host of run 33698844864 therefore captured resolve-inzone.txt as '[no output, exit 2]' and all 471 sampler lines as inzone=unconfigured: during the resolver outage that ended Gitea#0, nobody ever asked dnsmasq for an onion name, and two investigations went without the one measurement that would have settled them.

Both probes now follow the conf-dir pointer, main file first so a lab node with its zone in the main config answers as before. Each is exercised by a test that runs it against real files rather than comparing its text, which the rescue constant had no test for at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-03 12:50:58 +02:00
kevinveenbirkenbachandClaude Opus 5 af605b049f refactor(domains): give local testing a .test domain, leave .example to docs
RFC 6761 reserves .test for testing and .example for documentation, and the
project's own domain was doing both jobs under the documentation name. It is
what default.env hands every local deploy and what the test fixtures assert
against, so it is a testing name.

Rename it wherever it stands for the running machine: the environment
defaults, the development inventory, the workspace and trust scripts, the test
suites, and the four documents that describe real behaviour - the DNS mapping
to loopback, the browser CA trust, WSL2 resolution. Those documents would have
started lying the moment default.env changed.

The sandbox allowlist moves with them, since egress is matched per pattern and
nothing resolves the old name any more, and the sandbox document that mirrors
that list follows it.

Two places keep .example on purpose: the link-style sample in the
documentation guide, which is documentation about documentation, and every
generic fixture name elsewhere in the suite - a.example, foo.example and their
kin. Those are a larger sweep of their own and are not touched here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 13:10:34 +02:00
kevinveenbirkenbachandClaude Opus 5 8230c187a2 fix(swarm-test): ask the swarm for the DB service
03_wait_converge built the DB service name from the engine type:
"${DB_DEP}_${DB_DEP}" -> mariadb_mariadb. That name is only correct when
the database is central. force_shared_db deliberately leaves a
manager-pinned role carrying a literal `shared: false` alone, so
web-app-magento keeps its embedded engine, and sys-svc-rdbms deploys it as
magento_database inside magento's own stack.

The name came from the engine, which is global, and was looked up with
`docker service ls` across the whole swarm instead of the role's stack, so
the `shared` flag never entered the check at any step. Both polarities were
wrong:

  variant #1 - no consumer pulls the central provider in, so
  mariadb_mariadb does not exist. Run 32630684501 job 97175522113 polled it
  ninety times and failed the drill, while the rescue artifact shows
  magento_database at 1/1 Running and the play at failed=0.

  variant #0 - matomo and mailu declare a Jinja `shared`, get force-shared
  and drag svc-db-mariadb in. Job 95662862942 then reported
  `db(mariadb): 1/1 Running 2 hours ago` beside `magento: Running 21
  minutes ago` - green on a database magento never touches.

resolve_db_service asks the deployed swarm which service carries this
role's database: <entity>_database first, else <dep>_<dep>. The two cannot
collide, because sys-svc-rdbms renders the _database sidecar only under
`lookup('database', application_id, 'local')`. Exact-line matching, not
docker's substring name filter, per 246558a13.

Only web-app-magento changes behaviour: it is the one role that is both
pinned by force_shared_db and seen by the DB_DEP grep. matrix and
bigbluebutton never reach the DB half - their entity key carries no image,
so skip_if_no_swarm_service exits first - and discourse resolves
postgres_postgres exactly as before, in every variant.

A role that declares a DB dep no service serves now says so and gates the
app half alone, instead of polling a name nobody deployed.

diagnostics.sh carried the same guess in its container filter.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-23 16:38:53 +02:00
kevinveenbirkenbachandClaude Opus 5 d14354e892 feat(diagnostics): sample what the resolver answers, not just its socket
Run 32118850138 lost the :53 listener, so the sampler counts listeners.
Run 32538854067 lost DNS with the listener still bound: sockets-udp.txt
showed dnsmasq holding 0.0.0.0:53 while every forwarded query timed out,
and the sampler reported that hour as healthy.

Each tick now also records what a glibc caller gets back — for the zone
this resolver owns, derived from its own address= line, and for one name
it must forward — with the latency that caller paid. The rescue collector
probes the in-zone name too: probing external names alone cannot separate
"forwarding is gone" from "dnsmasq is gone".

The sampler now also runs on the swarm lab DNS nodes; it only ever ran in
the compose path, which is why the Magento artifact carried no sampler log
at all. Its location comes from group_vars FILE_DNS53_SAMPLER_LOG and is
required at every call site, so a run cannot quietly collect nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 17:55:05 +02:00
kevinveenbirkenbachandClaude Opus 5 575a5a87ac refactor(layout): sort shipped scripts by language and split the unit suite
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).

This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.

Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:

  * 7 `src:` values built from a role variable holding a bare filename
    (svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
    The variable doubles as the destination name inside the container, so the
    language segment belongs at the call site, not in the variable.
  * `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
    through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
    18 roles ship exactly that and have no template fallback: the play would
    have died on each.
  * `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
    interface now reads `files/javascript/javascript.js`.
  * `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
    path through `path_absolute(... 'files', ...)`.
  * 4 `ansible.builtin.script:` invocations naming a bare command
    (pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
  * `dev-python/files/shell/install.sh` pointed at a sibling role's script.

Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 01:31:09 +02:00