Commit Graph
20 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5 3b2761c15f perf(deploy): run the per-host setup once per host, not once per application
Two probes that describe a host were re-running per application. Measured in CI
run wp-swarm-tor2: 42 invocations of the pip probe and 44 of the NFS assert.

sys-pip-install guarded only its dependency include. Everything expensive sat
below that guard in main.yml, unguarded: resolving the interpreter, probing pip
for --break-system-packages, checking for `packaging` and bootstrapping it. All
four ran on every one of the 21 include sites. They move into 00_core.yml, where
the existing `when: run_once_sys_pip_install is not defined` covers them, and
main.yml keeps the guarded include and the pip task itself. The role recap reads
sys-pip-install ---- 87.35s.

The two facts this freezes per host are run constants: ansible_python_interpreter
is pinned once in group_vars/all/00_general.yml and no call site overrides it,
and all 21 sites pass global_install: true. main.yml now defends the probe fact
with `| default(false)` the way 00_core.yml already did, because the probe sits
behind `when: global_install | bool` and a first invocation with false would
otherwise leave it undefined for every later one.

The NFS assert emulated a per-host check with run_once plus delegate_to over
groups['svc-swarm-node'], and the include it lives in happens per application.
It becomes a plain per-host task in sys-svc-compose/00_core.yml, which main.yml
already gates once per host per play: 44 instances and 132 remote commands
become 4 and 4. Dropping run_once also removes a real fragility — at log line
35355 only wrk-01 and wrk-02 reach the include because mgr-01's flag is already
set, so run_once there designates a host that skipped.

It reaches the same hosts as the loop it replaces, measured rather than assumed:
sys-svc-compose/00_core fires only on mgr-01, wrk-01 and wrk-02, the backup play
never runs it, and nfs-server and bkp-01 never execute it. The two conditions
that do not travel are a per-application volume count, meaningless for a
per-host check, and an exemption for manager-placed roles — mgr-01 answered the
assert ok 14 times out of 14, so the exemption was hiding nothing.

The assert is appended to 00_core.yml rather than prepended: utils/once/flag.yml
must stay that file's first task for test_service_core_first_task_run_once.

Worth 128.6s: 84.6s from the pip probe and 44.0s from the assert.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 20:13:57 +02:00
kevinveenbirkenbachandClaude Opus 4.8 5e5d28aea1 fix(sys-pip-install): retry pip installs through transient egress resets
The packaging bootstrap and the main install/upgrade both download from
PyPI and fail hard on a connection reset. Wrap each in the standard
retry-until-succeeded loop (5 tries, 15s apart), matching the apt/dnf/
pacman/docker egress hardening already in place.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 20:01:32 +02:00
kevinveenbirkenbachandClaude Opus 4.8 4039313266 refactor(tasks): enforce and apply a single task-file numbering scheme
Add tests/lint/repository/test_task_file_numbering.py: numbered task
files use exactly two leading digits, a numbered *_core.yml must be
00_core.yml, per directory the numbers are unique/consecutive and start
at 01 (00 with a core file), numbered includes appear in ascending
order, and shared helpers live in tasks/utils/ instead of the numbered
chain.

Apply it across every role: rename NN_core.yml to 00_core.yml with a
consecutive down-shift of the siblings, move letter-suffix children into
a subdirectory named after their parent task, relocate helpers included
from more than one place into tasks/utils/, and update every reference
(include_tasks, path_absolute, .ansible-lint-ignore, requirement docs).
Legacy narration comments in the touched files are deleted; the upstream
xwiki.properties reference block opts out via nocheck. The unused
MAILU_IP6_PUBLIC var (only ever named in now-removed comments) is dropped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 20:01:31 +02:00
kevinveenbirkenbachandClaude Opus 4.8 cddb0dc59b feat(lint): require emoji-prefixed task/step/job names + convert existing
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.

- emit emoji names at the source: cli/build/include generator now writes
  "🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
  tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
  task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
  🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 22:45:06 +02:00
kevinveenbirkenbachandClaude Opus 4.7 951f6d36d2 chore(lint): unify autoformat/galaxy parallelism into one PARALLEL env var
* scripts/lint/autoformat.sh: rename AUTOFORMAT_PARALLEL to PARALLEL,
  declare it at file start with default `true`; still env-overridable.
  Reword the worker comment so the literal `# shellcheck` token no
  longer appears at start-of-line and trips SC1072/SC1073.
* scripts/lint/galaxy.sh: drop the GALAXY_FATAL / GALAXY_NAMESPACE /
  GALAXY_PARALLEL env vars (over-engineered for a single repo). Failure
  is always fatal. NAMESPACE is an internal constant. Add the same
  PARALLEL env var (default `true`, env-overridable); falsy demotes the
  xargs fan-out to one worker.
* docs/contributing/environment/variables.md: collapse the four
  AUTOFORMAT_*/GALAXY_* rows into one `PARALLEL` row covering both
  consumers.
* docs/contributing/tools/make.md: refer to `PARALLEL=0` everywhere;
  add the lint-galaxy parallelism note.

Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 00:59:24 +02:00
kevinveenbirkenbach d625962da1 fix(sys-pip-install): use isolated python check to fix packaging detection on CentOS
- Use 'python -I' (isolated mode) to check for 'packaging' library.
- This ensures the check fails if the library is missing in the strict environment used by Ansible modules, even if present in user site-packages.
- Fixes pip module failures on CentOS.
2026-03-23 20:21:27 +01:00
kevinveenbirkenbach 556782cc84 Unify Python interpreter usage for pip and LDAP tasks 2026-02-22 17:18:03 +01:00
kevinveenbirkenbach 70ecef3c1f Unify Python interpreter and pip install handling across distros 2026-02-22 15:50:37 +01:00
kevinveenbirkenbach dd9829cf0a refactor(sys-pip-install): split core run-once tasks 2026-02-22 01:39:41 +01:00
kevinveenbirkenbach 9766f8e8b3 feat(dev-python): install Python 3.11+ by default
- add cross-distro dev-python installer script and wire it into role + Dockerfile
- regenerate dockerignore from gitignore on make build and ignore logs/
- harden sys-pip-install break-system-packages support probe
- raise project requires-python to >=3.11
2026-02-21 23:45:41 +01:00
kevinveenbirkenbach 434dc84ce6 sys-pip-install: make break-system-packages flag conditional 2026-02-21 20:25:22 +01:00
kevinveenbirkenbach c27ab7c413 Set application_id: svc-db-openldap to solve reference bug 2026-01-12 01:10:32 +01:00
kevinveenbirkenbach 85a5a621df fix(sys-pip-install): install packages into ansible venv when not global_install
Use ansible_python_interpreter-derived virtualenv for pip installs by default and
only use system pip3 with --break-system-packages when global_install=true. This
ensures Ansible module dependencies (e.g. python-ldap) are available in the
runtime interpreter.

https://chatgpt.com/share/6964335a-29cc-800f-ae65-6a35c7cd2559
2026-01-12 00:33:36 +01:00
kevinveenbirkenbach 9e993c338e feat(pip): add global_install flag and fix python-ldap installation for Ansible interpreter
- Add global_install flag to sys-pip-install role (default: false)
- Allow optional system-wide installs via pip3 with --break-system-packages
- Install python-ldap via pip in Ansible interpreter for OpenLDAP role
- Refactor svc-db-openldap tasks into 01_core.yml with run-once gating
- Fix environment handling for conditional global pip installs
- Enable global pip installs for CLI tools (gitcon, ldapsm, backup tools, cleanup tools)

https://chatgpt.com/share/6963dd5d-cae0-800f-b8d7-82a3808be634
2026-01-11 18:26:49 +01:00
kevinveenbirkenbach a09e6d7592 fix(sys-pip-install): install packages via system pip3 instead of Ansible venv
Use system-wide pip3 for package installation to avoid installing CLI tools
into the Ansible virtualenv. This ensures binaries (e.g. dockreap) are placed
in a global PATH and are usable by systemd services.

https://chatgpt.com/share/69628897-4d98-800f-b4cf-b9a8f5fdaa6b
2026-01-10 18:12:46 +01:00
kevinveenbirkenbach bace99c9c6 Fix pip installation and stabilize cleanback service execution
- Switch sys-pip-install to ansible.builtin.pip with dynamically resolved pip executable
- Remove hardcoded python/pip paths to ensure cross-distro compatibility
- Separate cleanback package pinning from executed command
- Fix systemd ExecStart to call cleanback binary instead of package spec
- Improve robustness against PATH and interpreter mismatches

https://chatgpt.com/share/69554856-3920-800f-995e-8b6948e1d09c
2025-12-31 16:59:12 +01:00
kevinveenbirkenbach 735ae59cbb refactor: migrate cleanback installation from pkgmgr to pip
- replace pkgmgr-install with sys-pip-install for cleanback
- install cleanback as a regular pip package
- align roles with pyproject-based distribution
- keep system-wide, non-interactive installation semantics

https://chatgpt.com/share/69517d20-f850-800f-b6ff-6b983247888f
2025-12-28 20:42:12 +01:00
kevinveenbirkenbach cfceee7dbe Solved integration tests 2025-12-28 15:38:34 +01:00
kevinveenbirkenbach 5686a5ba29 fix(sys-pip-install,sys-service): avoid venv pip installs and handle oneshot services correctly
- Force sys-pip-install to use system python (/usr/bin/python3) instead of active venv
- Add explicit PATH for system-wide pip installs
- Add diagnostics task to verify installation context with system python
- Replace systemctl is-active check with Result/ExecMainStatus to correctly handle oneshot units
- Prevent false failures when oneshot services exit successfully

https://chatgpt.com/share/695123ba-8da4-800f-ae0d-2f09a5efb1e7
2025-12-28 13:33:53 +01:00
kevinveenbirkenbach 880d73e2ff refactor: replace pkgmgr with system pip installation roles
- Introduce sys-pip and sys-pip-install roles
- Install Python CLI tools system-wide via python3 -m pip
- Migrate cleanup and backup roles from pkgmgr to pip-based installs
- Update documentation to reflect Python-based installation
- Rename dev-python-pip to sys-pip for clearer responsibility

https://chatgpt.com/share/695107c4-f320-800f-b4ce-da953de9bb86
2025-12-28 12:41:59 +01:00