Virgin pkgmgr containers ship without /opt/venvs, so the prior
'-d /opt/venvs && -w /opt/venvs' check fell through to ${HOME}/.venvs,
breaking the test-install-{make,pkgmgr} workflows that hardcode
'source /opt/venvs/infinito/bin/activate'. Probe the parent instead —
venv.sh mkdir -p's the leaf later.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
00_orchestrator.sh sources common.sh in a vanilla container before
01_install.sh has run; the eager load_repo_env then sources load.sh ->
detect.sh which hard-fails per the new SPOT contract. Skip the load
silently when python3 is absent — the post-install loop iteration
re-runs load_repo_env and picks up the freshly installed interpreter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pulls VENV/PYTHON/PIP/PYTHONPATH out of the dotenv handler pipeline
into scripts/meta/python/detect.sh. load.sh now sources detect.sh and
prepends ${VENV}/bin to PATH above the INFINITO_ENV_LOADED guard, so
every BASH_ENV re-entry re-resolves PYTHON the moment a venv
materialises mid-chain — the same fix as b79ed2869 but at the SPOT,
which retires the reload-trick from ansible.sh, python.sh and
pre-commit.sh.
Also renames INVENTORY_VARS_FILE → INFINITO_INVENTORY_VARS_FILE under
the new env-prefix lint, drops the 5 dead INFINITO_ENV_*_LOADED unsets
left over from the multi-loader era, and adds a lint guarding the
INFINITO_ prefix invariant on env/*.env keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The four test-deploy workflows (server/universal/workstation/local)
each started the full compose stack just so scripts/meta/resolve/
apps.sh could call its meta-CLI inside the container. The stack
spin-up dwarfed the actual discovery query.
Introduce INFINITO_APP_DISCOVERY_RUNNER (host|docker, default
`docker` for local dev) alongside the existing INFINITO_TEST_RUNNER
/ INFINITO_LINT_RUNNER dispatch knobs. In host mode apps.sh runs
the meta-CLI through the venv python directly; docker mode keeps
the existing `docker compose --profile ci exec` path.
Workflows switch to host mode and replace `make up` with
actions/setup-python@v5 + `make install`. make install was already
a prereq of make up, so the install cost is unchanged -- only the
docker pull + container start time disappears.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/docker/entry.sh's two compile-control reads (INFINITO_COMPILE
and INFINITO_COMPILE_SILENCE) crashed during Dockerfile RUN because
that context has no env-pipeline: env/default.env exists in the repo
but no make/BASH_ENV/load.sh chain materialises it into the running
shell. The bare reads under `set -u` triggered unbound-variable exit
even though the keys were registered in default.env + passthrough.
Move build/silence control from env to script flags so entry.sh is
self-contained at bootstrap time. Each flag now triggers its action
directly inside the parse loop -- no COMPILE/SILENCE state vars:
--compile -> run_local_build (streaming)
--compile-silent -> run_local_build >/dev/null 2>&1
-- -> end flag parsing, rest is the exec target
Drop INFINITO_COMPILE + INFINITO_COMPILE_SILENCE from the whole
registry surface: env/default.env, passthrough.STATIC_KEYS, the
infinito_compile handler (gone entirely), compose.yml's env
injection block, the dotenv-generator baseline test, the
build_env unit-test fixture, and the now-no-op `-e INFINITO_COMPILE=0`
in test-install-{make,pkgmgr} workflows.
Dockerfile RUN switches to `entry.sh --compile -- true`; runtime
ENTRYPOINT receives no flags so the default (no-compile) holds and
exec hands off to systemd via the CMD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/install/venv.sh
-----------------------
INFINITO_VENV_BASE is a static-default convention; the active VENV
may end up outside that base (e.g. \$HOME/.venvs/<name> when
/opt/venvs is not writable, or a wholly custom override). Probing
writability on the legacy base triggered a spurious sudo branch
whenever VENV lived elsewhere -- fatal in no_new_privs sandboxes
where sudo cannot escalate. Switch the writability + mkdir + chown
chain to \$(dirname "\${VENV}") so the script prepares the directory
it is actually about to create the venv in.
tests/lint/ansible/roles/web-app/playwright/test_app_base_url_uses_role.py
-------------------------------------------------------------------------
New lint: every role's templates/playwright.env.j2 must declare
APP_BASE_URL as a reference to the role's own surface. APP_BASE_URL
feeds the shared guest-persona helper and Playwright's global
baseURL; a role that hardcodes a sibling role name (most commonly
'web-app-dashboard') sends the guest test to the wrong site and
breaks any matrix variant where the sibling is not deployed
(ERR_SSL_UNRECOGNIZED_NAME_ALERT).
The 16 web-app role templates that violated the contract switch
from sibling lookups (`lookup('tls', 'web-app-dashboard', ...)`)
to the canonical `lookup('tls', application_id, ...)` form.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/install/ansible.sh inherited PYTHON=python3 from all.sh because
the first env-generation runs before the venv exists and the PYTHON
handler falls back to "python3" (literal). On a CI runner that resolves
to actions/setup-python's interpreter, which has no `ansible` distribution
even though pip just installed one into the venv -- `ansible-galaxy
collection install` then fails with ModuleNotFoundError.
Mirror the reload-after-venv pattern that scripts/install/python.sh
already uses: when ${VENV}/bin/python now exists and the inherited
PYTHON still points at the system fallback, drop INFINITO_ENV_LOADED
and re-source load.sh so it prepends ${VENV}/bin to PATH for the
remainder of this subshell.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new lint tests close gaps in the INFINITO_* SPOT contract:
* tests/lint/filesystem/env/test_default_env_keys_handled.py
Every default.env key must be wired through a handler.
* tests/lint/filesystem/env/test_used_keys_declared.py
Every INFINITO_* in .sh/.py must live in env/{default,ci}.env or
a handler; per-line `# nocheck:` suppresses runtime-only refs.
* tests/lint/filesystem/python/test_no_infinito_get_defaults.py
Forbid ANY default in `.get("INFINITO_*", ...)` -- including the
empty-string fallback (which silently masks an unset key). The
nocheck check walks every physical line the call spans so an
autoformat-split call still honours the marker.
Cache namespace flattens to a single prefix:
INFINITO_PACKAGE_CACHE_* -> INFINITO_CACHE_PACKAGE_*
INFINITO_REGISTRY_CACHE_* -> INFINITO_CACHE_REGISTRY_*
The five previously driver-injected paths (APT_LIST, NPMRC,
PIP_CONF, REGISTRY_PROXY_CONF, FRONTEND_CA_FILE) move into
env/default.env and passthrough.STATIC_KEYS; the
`cache_env_overrides()` helper and its two call sites are gone
along with the test-fixture override for FRONTEND_CA_DIR.
SPOT-strict rewrites:
- 19 sites: `something.get("INFINITO_X", "")` -> subscript
`something["INFINITO_X"]` (loud KeyError when env loader did
not run; all keys live in env/default.env).
- 2 handler-internal bootstrap reads suppressed with
`# nocheck: handler-bootstrap-read`.
- 30 runtime-only / driver-injected / test-fixture references
annotated with `# nocheck: <reason>` per-line.
- Unit tests that previously removed SPOT-guaranteed keys via
`{k: v ... if k \!= KEY}` or `clear=True` switch to setting
the key to "" explicitly, matching the new contract.
Side fixes:
- INFINITO_DNS_DOMAIN + INFINITO_DNS_HOSTS_FILE move from
shell-side defaults in common.sh to env/default.env and
passthrough.STATIC_KEYS.
- dns_systemd_is_operational() drops the dead
INFINITO_DNS_FORCE_HOSTS_FALLBACK override branch (the
variable was never set anywhere).
- INFINITO_WORKER_FETCH switches from `os.environ.get(KEY, "1")`
to `os.environ[KEY]`.
- scripts/tests/code/wrapper.sh exec args move into a bash
array so `# nocheck:` on INFINITO_TEST_TYPE can live on its
own element line (SC2215 fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename env/static.env → env/default.env across the entire repo
(string label, doc links, Python file-path constants) — the
`parse_static_env*` function names stay as a stable API surface.
Sort all `*.env` / `*.env.j2` entries alphabetically (52 in
default.env, 4 in ci.env, 46 of 49 j2 templates; 3 templates
with Jinja2 `{% %}` control flow are skipped because conditional
blocks would tear apart under a naive sort). Sort all 83
Makefile target blocks alphabetically; .PHONY group headers
stay in place.
Collapse the two remaining multi-line Makefile recipe comments
(dotenv-force, test-signed) to single-line WHY comments.
Add four lint tests under tests/lint/filesystem/ to keep the
invariants enforced going forward:
- env/test_alphabetical_sort.py
- env/test_single_line_comments.py (# nocheck: markers exempt)
- makefile/test_alphabetical_targets.py
- makefile/test_single_line_comments.py (# nocheck: markers exempt)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The URL-reachability probe now bounds its wall-clock end-to-end:
allow_redirects=False (a redirect chain previously stacked
timeouts), a 5h30m global deadline that marks in-flight probes
as warnings on expiry, and live `[done/total]` progress every 50
probes. Submission order is shuffled so same-host URLs no longer
batch at the front and self-DoS one origin.
The Docker image-version test drops its bespoke sequential
fetch loop and calls find_outdated_updates() from utils.update.docker,
so it now shares the INFINITO_WORKER_FETCH-sized worker pool with
the repository-refs check.
`scripts/tests/code/wrapper.sh` propagates BASH_ENV into the
container exec — without it INFINITO_WORKER_FETCH was unset
inside the container and the URL probe collapsed to 1 worker.
`make test` gains test-external; `make dotenv-force` regenerates
.env with a stripped environment so stale BASH_ENV-preloaded
INFINITO_* values cannot pin old numbers via setdefault.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the single `make -j ... --output-sync=target` invocation with
a shell-level fan-out that times each target individually and reports
a status table at the end. Output for each target is buffered to a
log file and dumped under a ✅/❌ header as soon as that target
finishes (preserving the per-target grouping `--output-sync` gave us).
Non-zero exit codes propagate; the run aborts at the first failure
once all already-launched children drain.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`make test` and `make lint` now delegate to a generic parallel-make
wrapper that caps `-j` at INFINITO_WORKER_CPU, syncs per-target
output, and prints wall-clock elapsed time. `make test` gains the
full `lint` sweep and drops the phantom `test-deploy` target (PHONY
without recipe); pre-commit hooks consolidate around a single
`make test` invocation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename INFINITO_PARALLEL -> INFINITO_WORKER_ENABLED and add two derived
keys so consumers no longer multiply by their own factor: the env
generator resolves the cascade once at `make dotenv` time and bakes
the integer counts into `.env`.
Chain:
- INFINITO_WORKER_ENABLED (operator toggle, default true)
- INFINITO_WORKER_CPU = cpu_count() when ENABLED truthy, else 1
- INFINITO_WORKER_FETCH = CPU * 20 for I/O-bound bulk fetches,
collapsing to 1 when CPU == 1
Handlers:
- utils/env/handlers/infinito_worker.py (the prior single key)
removed
- utils/env/handlers/infinito_worker_cpu.py + infinito_worker_fetch.py
added; registry orders them after `passthrough` so the cascade
reads its dependency from the env-builder state.
Consumers (now 1-liners):
- tests/external/repository/test_urls_reachable.py reads
INFINITO_WORKER_FETCH directly.
- utils/update/base.py::resolve_max_fetch_workers() reads it directly;
utils/update/{docker,repository}.py keep using that helper plus a
ThreadPoolExecutor for the actual bulk fetch (with `dict(pool.map(...))`
satisfying PERF403).
- scripts/lint/autoformat.sh + docs/contributing/tools/make.md
rename INFINITO_PARALLEL -> INFINITO_WORKER_ENABLED.
Tests:
- EXPECTED_BASELINE_KEYS in test_dotenv_generator updated to track the
three new keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
galaxy-importer's legacy-role mode was dropped upstream in 0.5.0
(CLI no longer accepts --legacy-role / --namespace), and the project
does not publish roles to Ansible Galaxy. The check enforces a
publishing schema that never gets exercised, against an abandoned API
that can only be kept alive via pinning galaxy-importer==0.4.39 +
setuptools<81 in an isolated venv. The cost has stopped buying
anything.
Removed:
- scripts/lint/galaxy.sh
- utils/install/lint/galaxy_importer.py
- install_isolated_pip_pkg() in utils/install/pip.py (only galaxy_importer
used it)
- Makefile lint-galaxy target + the entry in the `lint:` aggregate
- pre-commit make-lint-galaxy hook
- The galaxy_importer ensure() call in _install_ansible_tools
- Test assertion against cli.galaxy_importer in test_main.py
- Documentation rows mentioning lint-galaxy / galaxy-importer
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Refactor scripts/install/lint.sh (~870 LOC bash) into a Python package
under utils/install/. scripts/install/lint.sh becomes a 7-line wrapper
that delegates to `python3 -m utils.install.lint`.
Layout:
- utils/install/{primitives,pip,npm,github_release,system_pkg}.py:
channel-agnostic installers
- utils/install/lint/<tool>.py: one ensure() per tool (actionlint,
shfmt, mbake, ruff, ansible_lint, galaxy_importer, markdownlint_cli2,
eslint, shellcheck, ansible_commands, ansible_collections)
- utils/install/lint/__main__.py: CLI + group dispatch + stamp logic
INFINITO_LINT_RUNNER dispatch:
- New env var (host | docker, default docker), tracked in env/static.env
+ passthrough handler + drift test.
- scripts/install/wrapper.sh routes `make install-lint` to host or
container based on the runner.
- scripts/lint/wrapper.sh already dispatched; container branch now uses
bash (not --login) so /etc/profile does not reset PATH, and forwards
BASH_ENV so the .env keys land in the lint script's env.
Stamp logic:
- Per-interpreter stamp at build/install-lint-<sha256(sys.executable)>.stamp
so host venv and container venv track installs independently despite
bind-mounted build/.
- `--force` drops the stamp; group args bypass the stamp.
Tests + docs:
- 43 unit tests under tests/unit/utils/install/{,lint/} cover primitives,
pip/npm/github_release/system_pkg fallback chains, and the main CLI
(stamp logic, group dispatch, error path).
- utils/install/README.md + utils/install/lint/README.md document the
structure and the per-tool ensure() contract.
- docs/contributing/tools/make.md install-lint row updated to mention
the runner dispatch + per-env stamp + install-lint-force.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`export VAR=$(cmd)` masks the inner command's exit code with export's own.
Splitting the assignment lets a failing `dirname "$PYTHON"` propagate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace 8 bash env modules with a single Python generator
(cli.meta.env) that materialises env/static.env + runtime context
into a gitignored .env. Each variable lives in its own handler
under utils/env/handlers/; defaults carry inline one-line `# ...`
comments.
Rename to the INFINITO_* schema:
- workflow inputs: APPS, SERVICES_DISABLED, CMD, DEBUG, WHITELIST,
BUNDLES, FULL_CYCLE, VARIANT, RUN_FLAGS, LIMIT_HOST, PARALLEL
- inventory: INVENTORY_DIR / INVENTORY_FILE / HOST_VARS_FILE,
PW_FILE -> INVENTORY_PASSWORD_FILE
- venv: VENV_BASE / VENV_FALLBACK / VENV_NAME
- new SPOT defaults: COMPILE_SILENCE, CONTAINER_NO_CA,
WAIT_HEALTH_TIMEOUT_S, PACKAGE_CACHE_FRONTEND_INIT_IMAGE,
promoted IMAGE_TAG from GHA-only to general passthrough
compose.yml hardening:
- inline \${VAR:-default} removed; strict \${VAR:?...} everywhere
- new integration lint blocks regressions
Tooling:
- scripts/install/all.sh owns the stamp-file install logic
(Makefile target delegates)
- scripts/meta/env/load.sh is the single shell entry point;
8 bash modules deleted
- new shell lint: no INFINITO_* defaults outside env/static.env
Tests:
- 93 new unit tests for utils/env/{builder,parser,runtime,writer}
- restructure tests/lint into repository/ (file-content rules) vs
filesystem/{shell,python,env}/ (filesystem-shape rules)
- drift + smoke integration tests against the generator
Docs:
- variables.md is now a pointer; per-variable docs live next to
the key in env/static.env / env/ci.env
- compose.yml.md drops the Default column
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two new steps wrapped around the existing push/create flow:
1. Dedupe before push: hash the local diff vs origin/<base-branch> and
compare against gh pr diff of every open PR (any author, any branch)
targeting the same base. On a match, exit 0 without pushing and
without creating a PR. This is what stops the same image-version
bump from being opened repeatedly.
2. Supersede after create: close every other open PR whose branch
starts with UPDATE_BRANCH_PREFIX with a "Superseded by #N." comment.
The prefix filter keeps the bot from ever closing repository-refs,
skills, or human-authored PRs, even though dedupe itself sees them.
CRLF normalization on both sides of the hash compare so a runner that
checks out the patch with native-CRLF translation still produces a
stable hash.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Group the shell helpers under scripts/github/ into five subfolders
that match how the workflow files consume them:
runner/ host-runner hygiene (disk, swap, Docker daemon, diagnostics)
cancel/ run cancellation on PR-close and branch-delete events
resolve/ derivation of workflow inputs and outputs from repo state
update/ scheduled Update: Versions pipeline
release/ release-time gates
Drop the now-redundant subfolder prefix from each filename, e.g.
dump_runner_state.sh becomes dump_state.sh and cancel_pull_request_runs.sh
becomes pull_request_runs.sh. Each subfolder carries a README that
describes its scope per documentation.md rules; the top-level README
points at the subfolders without inventorying files.
Propagate the new paths through ten workflow files, three docs
(workflow.md, pipeline.md, secrets.md), and two tests. Fix a real
runtime bug in enlarge_swap.sh, which still referenced the
pre-rename dump_runner_state.sh path. Extend the inline-block-size
lint hint to describe the new subfolder convention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Workflow-driven update PRs opened with GITHUB_TOKEN do not start
further workflow runs, so the CI orchestrator never executed against
them. Each PR-opening job in update.yml now mints a short-lived
GitHub App installation token via actions/create-github-app-token@v2
and forwards APP_SLUG to open_update_pr.sh, which derives the
verified-commit identity from the App user.
The job-level permissions block drops to read-only since write access
is provided exclusively by the App token. Behaviour with the secrets
unset is fail-loud: the token step fails, the PR step is skipped,
and no PR with an unintended identity is ever created.
Requires the BOT_APP_ID and BOT_APP_PRIVATE_KEY repository secrets
documented in docs/contributing/tools/github/actions/secrets.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror scope.sh's is_markdown_or_rst skip-list in affected_roles.sh so a
PR that touches both a role and a doc file outside roles/<role>/ resolves
to that role only, not __ALL__. Covered by tests/integration/meta/workflows/test_diff_affected_roles.py.
Refines the explicit HTTPS default_server introduced in e9df1ae7b:
instead of aborting the TLS handshake (curl→000), complete TLS with
the primary-domain cert and return 421 Misdirected Request (RFC 9110
§15.5.20). When the cert SAN still covers the requested host but no
vhost is configured for it — the exact failure mode that
sys-svc-cln-domains produces between matrix-deploy variants — clients
now see a precise, RFC-correct signal instead of an opaque TLS error.
For hostnames not in any cert SAN (production with per-domain LE
certs and unrelated probes), curl still rejects the cert and reports
000, so the response is naturally dual-mode:
- SAN match + no vhost → 421 (deploy-incomplete signal)
- no SAN match → 000 (truly unknown host)
Updates scripts/tests/environment/08_deploy_performance.sh: the
matomo-only fresh-purged deploy keeps dashboard.infinito.example in
the shared CI cert's SAN list, so the assertion changes from
`assert_http_status 000` to `assert_http_status 421`. The matomo
assertion in 07_deploy_minimal.sh stays at 000 because matomo is
excluded from the cert SAN in that deploy's CA regen.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- new utils.inventory.bundle_apps resolver aggregates and dedupes role
groups from inventories/bundles/{servers,workstations}/<name>/inventory.yml
- new make targets deploy-bundles (fresh-purged) and redeploy-bundles
(reuse-kept), driven by BUNDLES=... env, wrapped by
scripts/tests/deploy/local/deploy/{bundles,redeploy-bundles}.sh
- unit tests for the resolver under tests/unit/utils/inventory/
- document the new targets in docs/contributing/tools/make.md,
docs/contributing/actions/testing.md, inventories/bundles/README.md
- collapse multi-line Makefile comments to one-liners for consistency
- disable nextcloud bbb plugin auto-enable (commented out)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per-app nginx gateways used proxy_pass with a literal docker container
hostname. nginx OSS resolves that hostname once at startup; if the
upstream container later restarts and gets a new IP (e.g. a partial
compose-up after credential change), the gateway proxies onto a dead
address until nginx itself reloads — the 502 we hit on taiga after a
reuse-kept replaced the taiga-back container.
Switch every per-app docker-hostname proxy_pass to a variable form so
the embedded docker DNS re-resolves at request time:
set $upstream_<rolename>_<purpose> http://<host>:<port>;
proxy_pass $upstream_<rolename>_<purpose>;
The full `resolver` directive (TTL + ipv6 toggle inherited from
NETWORK_IPV6_ENABLED) lives in one SPOT — NETWORK_DOCKER_DNS_NGINX_DIRECTIVE
in group_vars/all/08_networks.yml — so cache TTL and ipv6 stay
consistent across the stack.
In taiga the upstream `set` declarations hoist to server scope because
taiga-back is reused across /api/, /oidc/ and /admin/; in nextcloud
and kix each upstream is used exactly once so the declaration stays
local. Locations whose original proxy_pass stripped the location
prefix (`proxy_pass http://.../;` on /media/, /whiteboard/,
/standalone-signaling/) keep that semantics via an explicit
`rewrite ^<loc>(.*)$ /\$1 break;`, because variable-proxy_pass drops
the implicit prefix-strip.
kix's static files/kix-proxy.default.conf moves to
templates/kix-proxy.default.conf.j2 so it can reference the new
constant; the deploy task switches from copy to template accordingly.
Drive-by: scripts/tests/e2e/rerun-spec.sh docstring refresh — the
"only overwrites tests/playwright.spec.js" claim went stale when the
rerunner started restaging every role-local playwright/*.js.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`roles/__pycache__/` is created whenever Python imports the `roles`
package (it has an `__init__.py`), then `find` picks it up and
galaxy-importer fails the run with `FAIL __pycache__`. Same root cause
as the test_mapping.py fix in 635a0808d.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The early `/opt/compose/${entity}` check in apps.sh skipped redirect-only
entities (cdn, css, asset, file — no compose stack, only nginx vhost
files) before nginx.sh could run, leaving their per-domain vhosts
behind across matrix-deploy variant transitions.
Concretely chess CI Run 25997888664: variant-0 deployed web-svc-css
which rendered css.infinito.example.conf; the inter-round purge for
variant-1 skipped css with `\!\!\! WARNING: /opt/compose/css not found`;
the vhost survived; hlth-csp's headless browser then probed
https://css.infinito.example/ and got ERR_CERT_COMMON_NAME_INVALID
(the surviving vhost falls back to the default_server cert).
Moves the compose-dir guard from the entity-collection loop into the
purge loop and runs nginx.sh unconditionally before the
compose-dependent purges. The else branch prints the skip reason so
operators retain the trace signal for redirect-only entities.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Inner-loop iteration on a deployed stack often needs a one-off sidecar
container against the same docker daemon the stack lives on — typical
cases are rerunning a Playwright project with a patched .env, retrying
an arbitrary CLI image against an app container, or smoke-checking a
release image without going through Ansible. Without a wrapper the
caller has to spell out `docker exec <host-container> docker run --rm
<flags...> <image> <cmd>` by hand, which is verbose enough that agents
tend to fall back to redeploying instead of validating in-place.
Add `make run IMAGE=<ref> [CMD=...] [RUN_FLAGS=...]` plus the underlying
`scripts/tests/deploy/local/exec/run.sh`. The script reuses
`scripts/meta/env/all.sh` to resolve the host container name (same SPOT
as `make exec`), forwards RUN_FLAGS verbatim, and only wraps CMD in
`sh -lc` so default-entrypoint images and shell one-liners both work.
Wire both iteration loops to mention the wrapper next to `make exec`:
the Role Loop's Inspect block gets a bullet pointing at it, and the
Playwright Spec Loop gains a short "Debugging in the nested docker
daemon" section that also reminds the agent to mirror any successful
hand-patch back into the role's playwright.env.j2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three classes of drift cleaned up across 85 .md files:
- Em-dashes: replaced ` — ` with the contextually correct separator
(colon for `term: explanation`, period to split clauses across
sentences/source lines, parentheses for asides, semicolon for
tight-linked continuations). Two intentional exceptions remain
in `docs/contributing/documentation.md` (the rule literal) and
inside a fenced ```sh block in `docs/contributing/actions/debugging/ci.md`.
- SPOT self-labels: rewrote "This is the SPOT for X" to describe
the subject directly ("This directory holds X" / "This file
describes X"). Cross-references to other artefacts' canonical
role remain unchanged.
- Heading-emoji position: moved leading-emoji headings (`# 🧩 Title`)
to trailing position (`# Title 🧩`) per the headline rule that
emojis MUST come after the title text.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Folds the parallel Docker-image and git-ref version-update plumbing
into a single subsystem so the auto-update workflow, the external
freshness tests, and the unit tests all share one module layout.
utils/
update/
base.py # semver primitives shared by both backends
docker.py # OCI registry lookups + version: line rewriter
repository.py # git ls-remote walker + ref: line rewriter
cli/contributing/update/
docker/ # python -m cli.contributing.update.docker
repository/ # python -m cli.contributing.update.repository
scripts/github/update/
image_versions.sh
repository_refs.sh
Tests mirror the same shape under tests/{unit,external}/update/{docker,repository}/.
Also widens the Docker bump scope to all roles (previously gated on
`web-*`) so non-web roles like sys-svc-container get bumped too;
pins Bookwyrm's python image at 3.11-bookworm via
`# nocheck: docker-version` with a TODO until upstream supports
Python >= 3.12.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the no-op `docker restart` recovery with `POST
/service/rest/v1/repositories/{name}/invalidate-cache` against each
apt-format proxy repo (Debian, Ubuntu, …) discovered dynamically
from Nexus' REST API. Container restart never re-validated cached
content; the new path makes Nexus actually re-fetch from upstream
on the next request and keeps repo definitions, blobstore, and
admin credentials intact.
Co-located changes:
- fresh-purged-app.sh: source scripts/meta/env/all.sh so the cache
override env (incl. INFINITO_PACKAGE_CACHE_FRONTEND_IP) is set
for the deploy CLI — without this, apt traffic bypasses Nexus.
- utils/nexus/list_apt_proxy_repos.py: stdin JSON → stdout name list,
exit 1 on malformed input / no matches; used by the bash helper
for discovery (parser lives outside the heredoc for testability).
- tests/unit/utils/nexus/test_list_apt_proxy_repos.py: 10 cases
covering filter rules, empty/invalid input, non-dict entries,
missing/empty names, and a realistic live payload.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Step 08 (`deploy-fresh-purged-apps … FULL_CYCLE=true`) iterates matomo's
two variants and skips the final inter-round purge, so the host ends on
variant 1 (mariadb.shared=false → embedded `database` service). The
previous pin to VARIANT=0 forced a variant flip (shared=true) on a host
still wired for variant 1, which left the freshly-rendered matomo
compose.yml referencing `database` without an image and tripped the
`compose config --quiet` validation handler with:
service "database" has neither an image nor a build context specified:
invalid compose project
Pinning the reuse pass to VARIANT=1 redeploys onto the same variant the
matrix last left running, which is the intended `reuse of existing
state` semantic from variants.md ("the last state remains available for
inspection or follow-up specs").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`docker compose down --remove-orphans` only deletes networks that
compose itself created (those carrying
`com.docker.compose.network=default` + `com.docker.compose.project=<name>`
labels). When the per-entity default network ends up unlabeled or
project-mislabeled — typical after an inter-round matrix-deploy
purge, or when a `community.docker.docker_network` Ansible call
materialised the network before compose ran — compose deliberately
leaves it untouched, and a subsequent `compose up` aborts with
network <entity> was found but has incorrect label
com.docker.compose.network set to "" (expected: "default")
The matrix-deploy CI rounds for web-app-matrix, web-app-gitea and
others reliably reproduce this.
Add a new entity-keyed purge primitive
`scripts/container/purge/entity/network.sh <ENTITY>` that runs after
the compose / dir / nginx primitives. It is idempotent: no-op when
the network does not exist, skips removal while active container
endpoints prove the network belongs to a shared service still in
use (svc-db-mariadb, svc-db-postgres, svc-db-openldap, svc-ai-ollama,
…), and otherwise drops the orphan network with `docker network rm`.
`docker network rm` failures are logged and tolerated — the next
`compose up` will surface the same problem with an actionable error.
Wire the primitive into `scripts/container/purge/apps.sh` immediately
after the existing `nginx.sh` call, so every entity-keyed purge
collects its own default network on the same pass.
Finally, after the entity loop, run a single bare `docker network
prune -f` to sweep orphan networks whose name does NOT correspond
to a purged entity (left over from renamed roles, abandoned
matrix-deploy rounds, etc.). The sweep is bare on purpose — no
`|| true` guard — so a daemon-side prune failure aborts the purge
loudly via the script's `set -euo pipefail`. Performance: ~13× faster
than calling prune inside `network.sh` per entity (one ~200 ms
listing pass vs. N × 200 ms for N entities).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`INFINITO_PACKAGE_CACHE_MAX_AGE_MIN` was 129600 (90 days), 13× longer
than the 7-day `Valid-Until` window Debian and Ubuntu generate by
default in their apt `Release` files. Nexus held cached `Release`
manifests for 90 days, so once the upstream `Valid-Until` elapsed the
manifest stayed in the cache for another ~83 days and every Docker
build that ran `apt-get update` against the package-cache aborted with
E: Release file for http://deb.debian.org/debian/dists/.../InRelease
is expired (invalid since 5h 30min 54s).
Drop the default to 8640 minutes (6 days). Stays one day below the
7-day `Valid-Until` so Nexus always re-validates the cached `Release`
against upstream before apt would see it expired. The single value
still applies to every proxy repo (apt, yum, pypi, npm, gem, go, helm,
raw-*), but the `Valid-Until` enforcement only exists in apt — for
the other repos the change merely makes the cache 84 days fresher
instead of staler, no regression.
Update both docs sites that document the default:
- docs/contributing/artefact/files/compose.yml.md (env-var table)
- docs/contributing/environment/cache.md (operator narrative)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the registry-cache or package-cache (Nexus / nginx frontend)
serves apt repository metadata whose `Valid-Until` window has elapsed,
Docker builds inside an Ansible-driven deploy abort with
E: Release file for http://deb.debian.org/debian/dists/.../InRelease
is expired (invalid since 5h 30min 54s).
which surfaces as a generic non-zero exit from the wrapped deploy
command and forces the operator to manually wipe the caches before the
next attempt.
Add `scripts/tests/deploy/local/utils/cache-retry.sh` exposing
`deploy_with_cache_retry <label> -- <cmd> [args...]`. The helper runs
the wrapped command with combined stdout+stderr teed to a temp log,
captures the wrapped command's real exit code via PIPESTATUS (with a
disciplined `set +e` / restore-previous-state dance so the caller's
`set -e` does not abort the script before the helper can react), and
on non-zero exit grep's the captured log for the stale-cache
signature `Release file ... is expired` / `Valid-Until ... expired`.
If matched, the helper wipes both cache stacks via `make cache-clean`
(registry-cache + package-cache through
scripts/system/cache/clean.sh), runs `docker builder prune -af` and
`docker image prune -af` to drop cached build layers and unused base
images, and re-runs the wrapped command exactly once. Non-stale-cache
failures bubble up unchanged so the caller's existing error handling
is preserved. Live in the deploy/update script layer rather than the
Ansible handler because only the host-side scripts can reach the
registry-cache + package-cache containers and the host Docker engine.
Wire the helper into all five local deploy entrypoints:
- fresh-purged-app.sh (both `--full-cycle` and the sync-only branch)
- fresh-kept-app.sh
- fresh-kept-all.sh
- reuse-kept-app.sh (added SCRIPT_DIR/REPO_ROOT locally because the
script did not yet have them)
- reuse-kept-all.sh
CI deploy paths (scripts/tests/deploy/ci/*) are intentionally left
alone — CI runs do not share the developer's local cache stack and
they handle stale metadata differently.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
09_deploy_reuse.sh ran `make deploy-reuse-kept-apps APPS=web-app-matomo`
without VARIANT, which makes reuse-kept-app.sh fall back to the
unsuffixed `${INVENTORY_DIR}` path. The matrix-init in step 08 only
writes `${INVENTORY_DIR}-0` / `-1` because matomo declares 2 variants
(see roles/web-app-matomo/meta/variants.yml; planner contract:
cli/administration/deploy/development/inventory/planner.py:107), so
the reuse pass always missed the inventory and aborted with exit 2.
Also surface sibling inventory dirs as suggestions when the lookup
misses, so the next caller hitting this gets a hint instead of a bare
ENOENT.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Matrix-deploy variant transitions left per-domain vhost files behind:
when a service dropped out of a round's include set, sys-svc-compose's
docker prune removed its container but the vhost under
<NGINX_DIR>/conf.d/servers/{http,https}/<domain>.conf survived, making
the next round's HTTPS probe see 502 instead of TLS-error 000 (5 failed
test-development jobs on run 25705903504).
The inter-round purge in cli.administration.deploy.development.deploy
already invokes scripts/tests/deploy/local/purge/entity.sh with the
previous round's full include — the missing piece was an entity-keyed
nginx primitive next to db/compose/dir. Add it:
- utils.roles.entity_apps.apps_for_entity: reverse of get_entity_name,
walks roles/*/vars/main.yml to map entity -> [application_id]
- utils.cleanup.nginx_vhosts: resolves entity -> apps -> canonical+alias
domains (via build_applications_from_roles + iter_app_domains so it
stays ansible-free) and removes the matching vhost files
- scripts/container/purge/entity/nginx.sh: thin bash wrapper,
symmetric to db/compose/dir
- scripts/container/purge/apps.sh: drive nginx.sh inside the entity
loop, alongside db/compose/dir
Tests: 14 new unit tests covering the reverse-lookup edge cases and
the purge iterator (existing-only / aliases / multi-entity / no-op CLI).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Matrix-variant deploys write `${INVENTORY_DIR}-N/devices.yml` only —
the unsuffixed path doesn't exist, so the old `cat ${INVENTORY_FILE}`
killed every distro job in test-environment CI. Walk the suffixed
glob inside the container and fall back to the unsuffixed path.
Inner CMD runs under `sh -lc` (dash on Debian) so stay strictly
POSIX: no `shopt nullglob`, no arrays, no `(( ))`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops `MAX_LINES` from 12 to 11 in `test_inline_script_block_size.py`,
catching the five blocks that were sitting exactly on the previous
threshold. Pure-bash blocks move to dedicated `.sh` files invoked via
`ansible.builtin.script:` or `run: bash …`; the keycloak heredoc with a
templated JSON payload splits into a `set_fact` plus a folded
`printf | kcadm … -f -` one-liner.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`make install-lint` failed on every test-environment matrix runner
(manjarolinux/base, debian:bookworm, ubuntu:latest, fedora:latest,
quay.io/centos/centos:latest) with:
npm not found; install Node.js/npm first to get markdownlint-cli2.
make: *** [Makefile:227: install-lint] Error 1
The two npm-driven installers (`install_markdownlint_with_npm` and
`ensure_eslint`) bailed out as soon as `npm` was missing instead of
following the same auto-install pattern used elsewhere in the
script (e.g. `ensure_command shellcheck` / `shfmt` / `ruff`).
* Add an `npm:<manager>` mapping to `install_command`. pacman /
apt-get / dnf / yum get the candidate list `npm nodejs`; brew
gets `node`. `install_package_candidates` returns on the first
candidate that installs, which covers every matrix distro
(Arch's `npm` pulls `nodejs` as a hard dep; Debian/Ubuntu's `npm`
Depends: nodejs; Fedora/EL9+ ship `npm` as a direct package; brew's
`node` bundles npm).
* `install_markdownlint_with_npm` and `ensure_eslint` now call
`ensure_command npm` when `npm` is missing, before failing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`,
`requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from
non-Markdown source files. Comments anchored to requirement numbers
rot when requirements are renumbered or rewritten and conflate
present-state behaviour (which the comment should describe) with the
audit trail of how the code got there (which belongs in commit
messages, PR descriptions, and the requirement file itself).
* Document the rule in `docs/contributing/documentation.md` under
"Comments". Code MUST describe the *why* in present-state language;
contract enforcement lives in tests, not folklore.
* Clean 254 source files across `roles/`, `tests/`, `plugins/`,
`cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root
`eslint.config.js`. Pure comment / docstring / log-message edits;
no behavioural change.
* Add `tests/lint/repository/test_no_req_references_in_code.py` to
enforce the rule going forward across `.js`, `.py`, `.yml`,
`.yaml`, `.j2`, and `.sh` files. The legacy
`test_no_stale_requirement_refs.py` is exempt as the only file
whose purpose is to detect these very tokens.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires JavaScript linting into the project for the first time, scoped to
the Playwright specs and persona helpers under `roles/**/files/**/*.js`.
Setup
-----
- `package.json` + `package-lock.json` pin eslint v9, eslint-plugin-
playwright v2, @eslint/js, globals.
- `eslint.config.js` (flat config) loads `eslint:recommended` and
`playwright/flat/recommended`. Browser globals are injected so
`page.evaluate(() => window.foo)` callbacks lint cleanly. Project-
idiomatic patterns are explicitly turned off with rationale: the
service-gating contract (req 006/019) uses test.skip on disabled
services; persona helpers branch on isVisible/isServiceEnabled before
expect; some specs deliberately use waitForTimeout / networkidle for
OIDC settle. `no-empty` allows the empty-catch idiom; the autofixer
for `prefer-web-first-assertions` and `no-wait-for-navigation` is
unsafe in this codebase (broken diff in dashboard.js), so both are
off pending a hand-migration pass.
- `scripts/lint/javascript.sh` mirrors the markdown.sh shape (skip on
missing, exit 127 with "make install-lint" hint).
- `scripts/lint/autoformat.sh` gets a `run_eslint` worker (treats rc≤1
as OK, like markdownlint).
- `scripts/install/lint.sh` gets `ensure_eslint` (npm-local install in
the repo root, matching the markdownlint pattern), a `javascript`
group, and `install_javascript_tools` is wired into the `all` group.
- `Makefile` exposes `lint-javascript` and folds it into the aggregate
`lint` target.
- `.github/workflows/lint-javascript.yml` follows the lint-makefile.yml
shape (Node 20, npm cache, `npm ci`, `make lint-javascript`); the
ci-orchestrator wires it into the lint pipeline + lint-gate `needs:`.
- `.gitignore` adds `node_modules/`.
Real lint findings fixed in this commit
---------------------------------------
- `roles/web-app-decidim/files/playwright.spec.js` was using
`runBiberFlow` / `runAdminFlow` without ever requiring `./personas`
(would have failed at runtime). Adds the require.
- Same file: two nutzlose `\\'csrf-token\\'` escapes in a double-quoted
query selector — eslint flagged, fixed.
- `roles/web-app-moodle/files/playwright.spec.js`:
`expect(await locator.count()).toBe(0)` →
`await expect(locator).toHaveCount(0)` (Playwright auto-retrying
assertion; safe `prefer-to-have-count` autofix).
Lint baseline: 0 errors, 285 warnings (mostly `no-unused-vars` and
`playwright/expect-expect` — addressable as follow-up cleanup, not
blocking).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
shellcheck flagged three `[[ cond ]] && B || C` chains in
scripts/lint/autoformat.sh (ruff, shellcheck, mbake worker tails) as
SC2015: if `B` exits non-zero, `C` runs even when the test was true.
Rewrites each as `if [[ cond ]]; then B; else C; fi` so the success
and failure branches stay disjoint.
Reason: `make lint-shellcheck` aborted on these three info-level
findings; production-profile lint runs treat them as failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The reference at scripts/lint/makefile.sh:4 pointed at
github.com/mavyhq/bake which 404s; pypi metadata (mbake project) lists
the canonical homepage as github.com/EbodShojaei/bake.
Reason: tests/external/repository/test_urls_reachable.py was failing
with HTTP 404 on the stale URL.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces roles/test-e2e-playwright/files/personas/ — runGuestFlow,
runBiberFlow, runAdminFlow plus 10 utils (csp, dashboard, env, keycloak,
landing, logout, matomo, prometheus, interaction, index) — and wires
both the Ansible runner (run_one.yml) and the local rerun helper
(scripts/tests/e2e/rerun-spec.sh) to stage the tree alongside
service-gating.js so specs can `require("./personas")`.
First batch of role specs migrated to consume the shared library
(dashboard, keycloak, mailu, matomo, prometheus, web-svc-cdn,
web-svc-simpleicons) — each gets the spec.js persona scenarios plus
the matching meta/services.yml + playwright.env.j2 entries; matomo is
marked role-closed in req 019. dashboard/keycloak/mailu TODO.md
updated; docs/requirements/019 reflects matomo closure.
Reason: req 019 Rule 3 mandates that every role's guest/biber/
administrator flow runs the same canonical steps; centralising them
in one shared library keeps per-role drift from re-introducing the
stub-shaped bodies the new lint forbids.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removes 67 live references to docs/requirements/<NNN>-*.md files across
43 contributor docs, role READMEs, code comments, task files, and tests.
Requirement docs are initial-creation-only artefacts; live references
freeze a snapshot that rots once the per-feature SPOT (a contributor
doc, a code module, a lint test) takes over the live behaviour.
In the same pass, fixes documentation.md violations on the touched
files: drops em-dashes used as thought breaks, removes "SPOT for ..."
self-labels, strips historical phrasing ("the legacy", "previously",
"used to", "is no longer used", "matching the historical"), and
reflows soft-wrapped sentences so each sentence stays on one logical
line.
The `test_inventory_vars_file_spot.py` whitelist for the literal
inventory path drops the docs/requirements/003 entry and instead
skips docs/requirements/ entirely; future requirement docs that
mention the literal path are then ignored automatically.
Verified by tests/lint/repository/test_no_stale_requirement_refs.py
(green tree-wide).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
