os.environ at runtime does not reach sys.dont_write_bytecode: CPython
reads the variable once at interpreter start, so 498d9d1c12 protected
every child of the dedicated runner and never the runner itself.
Measured with inotifywait on cli/administration/deploy, same import and
same process: without the attribute a lazy import creates
CREATE,ISDIR cli/administration/deploy/__pycache__, with it nothing.
utils/i18n/libretranslate.py:264 and utils/tests/swarm/matrix.py:318
import into that tree from inside a function, which is the shape that
reaches it during a play.
This does not explain the tar failure in run 39186571513: the runner
blocks in subprocess.run while the snapshot reads, so the writer in that
window is still unnamed. A run that fails the same way again rules
bytecode out for good.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The snapshot task tars the live checkout from the repository root. A pristine published image carries no __pycache__, so loading each role's own filter_plugins or lookup_plugins writes one during the play, which moves a directory ctime under roles/. GNU tar answers a changed directory with exit 1 and set -o pipefail turns that into a failed task, which is how run 39113865427 lost docs#0, docs#1 and API#1.
The flagged set is not arbitrary: it is exactly the roles that own a plugin directory AND ran in that play. Three jobs, three different role sets, the same rule each time: 7 of 52 roles in docs#1, 3 of 25 in API#1, 5 in docs#0. web-app-keycloak owns a filter_plugins and was never flagged, because it never ran. The same task passes on the CI workspace with rc 0, where earlier Python steps had already populated those caches.
Set before the first child is spawned, so make setup, the inventory validator, ansible-playbook and every forked worker inherit it. scripts/tests/code/run.sh:31 already uses the same variable for the test suite. cli and utils sit in the snapshot paths too and role lookups import from utils during the play, so the same change closes that race as well.
Two callers still bypass this function and invoke ansible-playbook directly, scripts/tests/deploy/local/sync/role.sh and scripts/tests/design/sync.sh; they remain exposed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ruff 0.17.0 adds `datetime` -> `dt` to the default aliases of
flake8-import-conventions, and the project selects ICN, so lint-python
failed with ICN001 in seven files.
The files now import `datetime as dt`. ruff's unsafe fix also renamed
two local names in utils/github/playwright_summary.py into the alias:
the `_to_iso_z` parameter and the parsed value in
`_parse_iso_timestamp`. Both are now called `moment`; with the bare fix,
nine tests of test_playwright_summary.py failed with UnboundLocalError
and AttributeError.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`MODE_CLEANUP` says what to tidy on the deployed system: prune unused Docker
resources, clear Mastodon caches, switch an LDAP user list from `present` to
`exact`. Eight roles read it that way. The dedicated runner read the same flag
and ran `make clean` in the control repository instead, which is
`git clean -fdX` over every ignored path there.
`inventories/production/` is one of those paths. The published host install
guide provisions into it and then deploys against `$INVENTORY/.password`, so
the deploy deleted the vault password it was about to use:
Removing inventories/production/
[ERROR] The vault password file .../inventories/production/.password was
not found
Reproduced in CI on two unrelated rows, arm64/zfs/debian and
amd64/btrfs/centos, so it is neither architecture nor filesystem specific.
The coupling was a name collision, not a decision. 7e5990aa16 generated the
MODE_* flags from `01_modes.yml` and removed the legacy skip flags, and the
repository clean - which had had a flag of its own - was attached to the
generated `MODE_CLEANUP` because one now existed by that name. No other deploy
path does this; `container`, `development` and `swarm` never call `make` at
all.
`make clean` keeps its own entry points, `make setup-clean`, `make clean-sudo`
and `infinito --git-clean`, where a caller asks for it deliberately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three shapes of the same failure, taken from the build's own message list
rather than from a reading pass.
Eight unescaped inline characters in prose: a '*', '**', backtick or '|'
opens markup that never closes. 'roles/*/vars/main.yml', the regex
'^(web-|svc-).*', 'scripts/system/tls/trust/*.sh'. In ansible_log an inline
literal ended on a space, which reStructuredText forbids; the space carried
meaning, so it moved into the prose.
Thirteen example blocks after a single ':', which makes the indented block
a block quote instead of a literal one.
Three bullet lists with no blank line to the paragraph above, which
docutils reads as a continuation of that paragraph.
Two markdown files carried a '---' where reStructuredText allows no
transition: directly after a heading, and at the end of a document.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tasks/stages/03_destructor.yml includes test-e2e-cli only `when: RUNTIME in [dev, act, github]`, so a production host skips it by design. The coverage verifier demanded it anyway and failed every partial deploy there with "required role(s) did not execute". It stayed hidden because a full deploy passes no --id, which leaves allowed_applications empty and the guard unarmed.
required_by takes a `runtimes` list beside the existing compose/swarm blocks, read before the mode narrowing replaces rb. An unknown or empty runtime waives nothing, so the guard keeps its teeth wherever the runtime cannot be established; only a known runtime the list omits drops the requirement. This is 48753de136 one axis further: a role that a deploy correctly skips must not be demanded of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, each resolved by combining both sides rather than picking one.
compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.
daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.
The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The deploy CLI shipped every mode as -e KEY=value, which ansible-playbook parses as a string at the highest precedence. MODE_DEBUG=false therefore arrived as the non-empty, and so truthy, string false. Every when: MODE_X | bool was unaffected because the filter maps it back, but four sites read a flag bare and silently inverted: openldap wrote LDAP attributes with state exact on every deploy instead of only under MODE_CLEANUP, SYS_SERVICE_DEFAULT_STATE was restarted where it should have been omit, SYS_SERVICE_RUNNER_STATE was never started, and mailu always logged at DEBUG.
Send the modes as one JSON extra-var, which keeps the declared type, and give those four sites the | bool they were missing so the meaning of a flag no longer depends on how its value reached Ansible. A lint pins the rule: a MODE_* read in a truthiness position carries the filter or a nocheck naming a genuine string comparison.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sys-svc-compose-ca (required_by web) and sys-ctl-mtn-cert-deploy
(required_by web-app-mailu) were skipped at their include sites via a
TLS/self_signed `when:`. On a TLS-off node (e.g. .onion) the skipped
include_role leaves no role-named TASK block, so the post-deploy
required_by validator (cli/meta/roles/services/called) reports the role
as "did not execute" and exits 2.
Move the applicability decision into each role: an always-run debug marker
when not applicable (leaving the non-skip TASK block the validator needs)
plus the real CA/cert work gated behind the same condition. Clearnet
self_signed nodes still run the real work; TLS-off nodes pass with the
marker. Adds sys-svc-compose-ca/vars/main.yml (_compose_ca_applicable SPOT).
Also flush the deploy phase-header prints so they stream in order instead
of being buffered until after a failure (which misattributed the failing
pass).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New ``cli.administration.deploy.swarm`` package wraps the strict
``cli.administration.deploy.dedicated`` runner with a layered closure
so swarm deploys never require operators (or test scripts) to
enumerate every infra role:
effective --id = seed (operator --id, else inventory_role_groups)
∪ dep-walk(seed via applications_if_group_and_all_deps)
∪ default-placement safety net
The dep-walker is the same SPOT
(``utils.roles.applications.in_group_deps.applications_if_group_and_all_deps``)
the runtime constructor consumes via ``lookup('applications_current_play')``,
so the CLI-side closure and the in-play role set stay in sync.
The dedicated CLI keeps strict explicit-only semantics: ``main`` was
split into ``parse_args`` + ``run_from_args`` so the swarm wrapper
can share parser, mode wiring, validator and runner verbatim and
only mutate ``args.id``.
Meta-side: ``svc-db-postgres`` gains ``default_placement: manager``
(semantic symmetry with mariadb). ``default_placement`` is now a
PLACEMENT hint only ("if deployed, run on the manager"); deploy
decisions come from the dep-walk. Apps that declare ``mariadb:``
pull svc-db-mariadb; apps that declare ``postgres:`` pull
svc-db-postgres; apps that declare neither get no central DB.
Shared utils extracted so the four consumers stay aligned:
* ``utils.inventory.groups.inventory_has_group``: YAML + INI static
inventory group probe (extracted from ValidDeployId).
* ``utils.roles.meta_lookup.{get_role_default_placement,
iter_roles_with_default_placement}``: single source of truth for
``default_placement`` discovery (lookup plugin is now a thin wrapper).
Test pipeline (``scripts/tests/deploy/swarm/*``):
* Heredoc-Python shells converted to proper modules under
``utils.tests.swarm.*`` and invoked via ``python -m``:
- extend_inventory.py (was 08_extend_inventory.sh)
- write_extras.py (was 09_write_extras.sh dynamic half)
- derive_includes.py (replaces 06's hardcoded includes array)
- write_hosts_entries.py (replaces 02's hardcoded /etc/hosts heredoc;
derives every canonical + www-redirect domain from APP_ID via
canonical_domains_map + generate_all_domains, so the test
works for any app in the matrix without per-app maintenance)
* 01_create_network.sh deleted (single docker command inlined in workflow).
* 11_deploy.sh deleted (swarm CLI now called directly in workflow).
* 09_write_extras.sh: static half moved to inventories/development/swarm.yml.
* Remaining shell scripts renumbered 01-13 without gaps.
* 08_wait_converge.sh: case block on DB_DEP replaced with single
conditional (container name == DB_DEP).
* 06_provision_inventory.sh: hardcoded includes array replaced with
`mapfile -t includes < <(python -m utils.tests.swarm.derive_includes)`.
* Workflow YAML step references updated to new script names + module
invocations.
New tests (39 covering the closure surface):
* tests/unit/cli/administration/deploy/swarm/test_closure.py (14)
* tests/unit/utils/inventory/test_groups.py (11)
* tests/unit/utils/roles/test_meta_lookup.py (+3 for default_placement)
* tests/unit/utils/roles/validation/test_deploy_id.py (unchanged
8 tests still pass after refactor onto the shared util).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of step-08 deploy failures across all 4 test-development
distros in run 26572559126: `make compose-deploy mode=reinstall
apps=web-app-dashboard INFINITO_SERVICES_DISABLED=matomo` (step 07)
made make export the cmdline var into the recipe's shell env, which
reached `runner.py` → `make setup` → `make dotenv`. The dotenv
generator's `setdefault` then baked the runtime-only value into the
on-disk `.env`. Step 08 sourced that .env, filtered `web-app-matomo`
out of its primary apps, and exited with "All primary apps disabled".
Fix:
- Drop the 7 routing knobs (apps, mode, purge, bundles, disable,
full_cycle, variant) from `passthrough.STATIC_KEYS` and from
`default.env`. They are per-invocation; .env is the persistent SPOT
and must not capture them.
- Switch every consumer from `os.environ["KEY"]` strict access to
`os.environ.get("KEY", "")` so the missing-key path stops at "no
filter" instead of a KeyError.
- Rename the env-var surface to match the make-alias surface 1:1
(lowercase): INFINITO_APPS → apps, INFINITO_SERVICES_DISABLED →
disable, INFINITO_BUNDLES → bundles, INFINITO_FULL_CYCLE →
full_cycle, INFINITO_VARIANT → variant, INFINITO_DEPLOY_MODE → mode,
INFINITO_PURGE_ENTITIES → purge. The make alias was the user-facing
name in every doc; routing-knob env vars now use the same name so
the cmdline form, the recipe env-prefix, and the consumer reads
all line up.
- Collapse the Makefile `compose-deploy` recipe to just the two
knobs whose alias differs from the env name (`type` →
INFINITO_DEPLOY_TYPE, `debug` → INFINITO_DEBUG). GNU Make
auto-exports cmdline variables into the recipe's shell env, so the
identical-name `$(if $(apps),apps="$(apps)")` lines were pure
noise.
- Drop `disable: ${disable}` from compose.yml. The dev CLI builds
`extra_env` per `docker exec` call and forwards `disable` via
`-e`, so the container env never needs the value frozen at
compose-up time.
- Update the 3 actual leak call sites
(07/08/09_deploy_*.sh) to the make-alias form
(`disable=matomo`, `full_cycle=true`, `variant=1`).
- Add `SIM112` to ruff's ignore set: the lowercase env names are
the deliberate make/env parity, not a typo.
Repro before fix: `INFINITO_SERVICES_DISABLED=matomo make dotenv`
wrote `INFINITO_SERVICES_DISABLED=matomo` to `.env`. After fix:
`disable=matomo make dotenv` leaves `.env` untouched.
Tests: `make test` and `make autoformat` green (2711 tests, 8 lint
sub-targets).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Hooks the verifier into `cli.administration.deploy.dedicated.runner.
run_ansible_playbook` so every deploy path that reaches the dedicated
runner is checked — including `compose-deploy mode=update`, bundle
redeploys, and direct production calls, which previously bypassed the
dev-wrapper's verifier.
Runner pins ANSIBLE_LOG_PATH, captures pre-run log size, then verifies
the post-run slice host-locally (no more `docker exec tail` indirection
from the dev wrapper). Dev wrapper drops its redundant verifier call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace 8 bash env modules with a single Python generator
(cli.meta.env) that materialises env/static.env + runtime context
into a gitignored .env. Each variable lives in its own handler
under utils/env/handlers/; defaults carry inline one-line `# ...`
comments.
Rename to the INFINITO_* schema:
- workflow inputs: APPS, SERVICES_DISABLED, CMD, DEBUG, WHITELIST,
BUNDLES, FULL_CYCLE, VARIANT, RUN_FLAGS, LIMIT_HOST, PARALLEL
- inventory: INVENTORY_DIR / INVENTORY_FILE / HOST_VARS_FILE,
PW_FILE -> INVENTORY_PASSWORD_FILE
- venv: VENV_BASE / VENV_FALLBACK / VENV_NAME
- new SPOT defaults: COMPILE_SILENCE, CONTAINER_NO_CA,
WAIT_HEALTH_TIMEOUT_S, PACKAGE_CACHE_FRONTEND_INIT_IMAGE,
promoted IMAGE_TAG from GHA-only to general passthrough
compose.yml hardening:
- inline \${VAR:-default} removed; strict \${VAR:?...} everywhere
- new integration lint blocks regressions
Tooling:
- scripts/install/all.sh owns the stamp-file install logic
(Makefile target delegates)
- scripts/meta/env/load.sh is the single shell entry point;
8 bash modules deleted
- new shell lint: no INFINITO_* defaults outside env/static.env
Tests:
- 93 new unit tests for utils/env/{builder,parser,runtime,writer}
- restructure tests/lint into repository/ (file-content rules) vs
filesystem/{shell,python,env}/ (filesystem-shape rules)
- drift + smoke integration tests against the generator
Docs:
- variables.md is now a pointer; per-variable docs live next to
the key in env/static.env / env/ci.env
- compose.yml.md drops the Default column
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Module moves (with imports, paths, argv literals, scripts, docs, and
parents[N] formulas updated in lockstep):
- cli/deploy -> cli/administration/deploy
- cli/validate/inventory -> cli/administration/inventory/validate
- cli/vault -> cli/administration/secrets/vault
- cli/administration/inventory/encrypt -> cli/administration/secrets/seal
- cli/mirror -> cli/contributing/mirror
The new cli/administration/secrets cluster collocates the whole-file
ansible-vault wrapper next to the inventory-aware per-field sealer.
The user-facing CLI subcommand paths follow the package layout, e.g.
"infinito administration deploy dedicated <inv>" and
"infinito administration inventory validate <inv>".
Help system redesign in cli/core/{discovery,help,app}.py:
- Discovery now distinguishes runnable commands (directories with
__main__.py) from category folders (directories that only group
further commands). Empty support packages are filtered out.
- Each category folder MAY carry a README.md; its H1 heading is used
as the title and its first paragraph as the one-line description in
the help output.
- "infinito" / "infinito --help" prints only the top-level entries
(one level deep). "infinito <category>" navigates into a folder and
shows its sub-folders and commands one level deep, with the README
title as the page heading and the invocation as a dim subtitle.
- New "--tree [N]" flag walks the tree under the (optional) path
prefix recursively. N caps the depth (1-based); omit N for an
unbounded walk. The output uses Unicode tree connectors, an emoji
per entry (folder for categories, gear for commands), a globally
aligned description column, and multi-line wrapped descriptions
with tree-continuity bars on continuation lines.
- Category and command listings use the full invocation
("infinito meta callorder") rather than the bare leaf name, so the
output is copy-pasteable.
Convention doc + 17 CLI READMEs:
- New docs/contributing/artefact/files/cli/readme_md.md describes the
H1+paragraph layout the help system reads, and how it differs from
role/docs READMEs.
- README.md added for cli/{administration, build, contributing, meta}
and their immediate sub-folders, plus cli/meta/roles/applications
and its resolution sub-folder. Each follows the "title with trailing
emoji + single descriptive paragraph" convention.
Mirror resolver/sync description strings (the help system reads them
from argparse --help output) added so they no longer surface as
"options:" in the global index. cli/mirror/cleanup already had a
description and is unchanged.
Fixes uncovered by the move:
- import unittest.mock added to four dedicated-deploy tests so
unittest.mock.patch resolves under stdlib unittest discovery
(Python 3.13 no longer surfaces unittest.mock through "import
unittest" alone).
- parents[N] index in the new
cli/administration/inventory/validate/__init__.py corrected from
3 to 4 to match the deeper layout.
- test_paths.INVENTORY_VALIDATOR_PATH assertion and test_runner
substring assertions updated to match the new validator path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
